“Statistically significant” is one of the most quoted phrases in science and one of the most misunderstood. It sounds like a verdict — the result is real, the finding matters. But statistical significance answers a narrower question than most readers assume, and learning what it does and does not tell you is one of the genuinely empowering skills in reading research. It turns a headline number into something you can actually interpret.
What a p-value really says
Significance is usually reported as a p-value, and the p-value answers one specific question: if the null hypothesis were true, how compatible would the observed data — or data still more extreme — be with that assumption? In plainer terms: if there were genuinely no effect, how often would you see a result at least this large by chance alone? A small p-value (conventionally below 0.05) means the observed result would be relatively unlikely under the assumption of no effect, so researchers treat it as evidence that something is going on.
Notice what that does not include. The p-value says nothing about how big the effect is, or whether it matters. As the American Statistical Association emphasized in its formal statement on the subject, a p-value is not a measure of the size or importance of an effect. It is a statement about how compatible the observed data are with a specified statistical model — a useful thing to know, and only one piece of the picture.
Effect size: the number that got left out
The missing piece is effect size — how much of a difference an intervention actually made. This is where significance and importance come apart, and it is why statisticians urge reporting effect sizes alongside p-values rather than leaning on significance alone.
A helpful way to feel the distinction: imagine a study finding that a treatment lowers some marker by an amount so small no patient would ever notice — but the study is large enough that the tiny difference is statistically significant. The result may be statistically distinguishable from the null while still being clinically trivial. Both things can be true. Statistical significance tells you the data are relatively incompatible with the null model at the chosen threshold; effect size tells you how large the estimated difference is — and whether that magnitude is worth caring about.
Why study size changes everything
This is the key mechanism, and it cuts both ways.
Large studies can make trivial effects significant. The more participants a study has, the more sensitive it becomes — sensitive enough to detect differences so small they carry no practical meaning. In a very large trial, statistical significance can be a statement about sample size as much as about the intervention.
Small studies can make real effects look dramatic — and unstable. With few participants, estimates are noisy. A small study that reaches significance tends to overstate the effect, because only a large apparent difference can clear the bar in a small sample. That is why striking results from small studies so often shrink when a larger, more careful study repeats them. The finding was not necessarily wrong; it was imprecise.
This is why the size of a study belongs in your first glance at any result, right alongside whether it was significant.
Confidence intervals: significance with a sense of scale
A more informative companion to the p-value is the confidence interval — a range of values reasonably compatible with the observed data under the statistical model. Instead of a yes/no verdict, it gives you a sense of both the estimated magnitude of an effect and the precision around that estimate. A narrow interval indicates a more precise estimate; whether the effect is meaningful depends on its magnitude and context. A wide interval signals greater uncertainty, even when it excludes zero. Reading the interval, not just the p-value, is one of the fastest upgrades to your statistical literacy.
Absolute versus relative: the same number, two impressions
One more distinction quietly shapes how impressive a result sounds: absolute versus relative change. Suppose a treatment cuts the risk of some outcome from 2 in 1,000 to 1 in 1,000. That is a 50% relative reduction — a dramatic-sounding headline — and an absolute reduction of one-tenth of one percent. Both describe the same data. Relative changes tend to look large; absolute changes tell you how many people are actually affected. A careful reader always asks which one is being quoted, and looks for the other.
Surrogate endpoints: measuring the stand-in
Finally, significance is only as meaningful as what was measured. Many studies report surrogate endpoints — a lab value or biomarker that stands in for an outcome people actually care about, because it is faster or easier to measure. A significant improvement in a surrogate is encouraging, but it is a promise, not a delivery: the real question is whether moving the surrogate translates into the outcome that matters. Sometimes it does; sometimes it does not. Knowing whether a result is about the destination or merely a signpost along the way changes how much weight it deserves.
Why it matters
None of this is a reason to distrust statistics — quite the opposite. These tools are how science separates signal from noise, and they work beautifully when read in full. The trouble comes only from reading half of the picture: treating “significant” as the whole story instead of the beginning of it.
The upgrade is simple, and it makes following research far more rewarding. When you meet a new finding, ask not only “is it significant?” but “how big is the effect, how precise is the estimate, was it absolute or relative, and did they measure what actually matters?” Those questions turn a single word into real understanding — and they let you share in genuine excitement for the right results, for the right reasons.