Resources - RESEARCH LITERACY
What does statistically significant mean in health studies?
Statistically significant usually means p below 0.05: the data would be unusual if there were no effect. It does not mean an effect is big, important or proven.
In a health study, 'statistically significant' usually means the result had a p-value below 0.05. Put simply, if there were truly no effect, and the study's assumptions held, data at least this extreme would be expected less than 5% of the time. It does not mean the effect is large, important or definitely real, and it is not the probability that the finding happened by chance.
That gap between what the phrase sounds like and what it actually means is one of the most common sources of confusion in health news.
What is a p-value, in plain English?
Researchers start by assuming there is no difference, for example that a treatment and a placebo work equally well. This is called the null hypothesis. The p-value asks how surprising the data would be if that assumption were true. A small p-value says the data sit awkwardly with 'no difference'. By long-standing convention, a p-value under 0.05 is labelled statistically significant.
In 2016 the American Statistical Association took the unusual step of publishing a formal statement on p-values, with six principles. In short, a p-value can show how incompatible the data are with a particular statistical model. It does not tell you the probability that a hypothesis is true, nor the probability that the result arose by chance alone. Statistical significance does not measure how big an effect is or how much it matters, and conclusions should not rest only on whether a p-value crosses a threshold.
Does statistically significant mean the result is important?
No. With enough participants, even a tiny difference can produce a very small p-value. The aspirin arm of the US Physicians' Health Study, reported in the New England Journal of Medicine in 1989, followed 22,071 doctors for an average of about five years. Heart attacks fell by 44% in the aspirin group, with a p-value below 0.00001.
In absolute terms, the rate was 254.8 heart attacks per 100,000 people per year with aspirin against 439.7 with placebo, and there was no reduction in deaths from all cardiovascular causes. A 2012 article on effect size in the Journal of Graduate Medical Education uses this trial as a warning, putting the risk difference at 0.77%, a very small effect despite the tiny p-value. A highly significant result and a modest real-world benefit can sit side by side.
Does a non-significant result mean there is no effect?
Not necessarily. A p-value above 0.05 may simply mean the study was too small or too noisy to tell. In a 2019 comment in Nature, Valentin Amrhein, Sander Greenland and Blake McShane, joined by more than 800 signatories, called for an end to hyped claims and to the dismissal of possibly important effects. Their target included the habit of declaring 'no difference' between two groups just because a result was not statistically significant.
The Physicians' Health Study shows this side too. Its report said the evidence on stroke and on total cardiovascular deaths remained inconclusive because too few participants had those outcomes. The confidence intervals make the point. For deaths from all cardiovascular causes, the relative risk with aspirin was 0.96, with a 95% confidence interval from 0.60 to 1.54. That range runs from a 40% reduction to a 54% increase, so the data could not tell benefit from harm.
For haemorrhagic stroke, the relative risk was 2.14, with an interval from 0.96 to 4.77 and a p-value of 0.06. By the 0.05 rule that is 'not significant', yet the data were compatible with more than a doubling of risk. Inconclusive is not the same as no effect, and a confidence interval shows that far better than a p-value does.
Why is 0.05 the cut-off?
It is a convention, not a law of nature, and researchers disagree about it. In 2018, a group of 72 researchers writing in Nature Human Behaviour proposed changing the default threshold for claims of new discoveries from 0.05 to 0.005, a much stricter bar. Others, including the American Statistical Association, warn against basing conclusions only on whether a p-value crosses any fixed line at all.
How common is the focus on p below 0.05?
Very common. A 2016 JAMA study by David Chavalarias, John Ioannidis and colleagues text-mined more than 12 million MEDLINE abstracts, plus full-text articles, published between 1990 and 2015. The share of abstracts reporting p-values roughly doubled, from 7.3% in 1990 to 15.6% in 2014. Among those that reported p-values, 96% reported at least one at 0.05 or lower. When the team manually checked a random sample of 796 abstracts reporting empirical data, only 1.8% reported both an effect size and a confidence interval.
That matters because the effect size and confidence interval are what tell you how big a difference is and how precisely it was measured. A literature built around a single threshold tends to report whether something crossed the line, not how much it changed.
What is p-hacking?
P-hacking means collecting or analysing data in different ways until a non-significant result becomes significant, then reporting the version that crossed the line. A 2015 text-mining study in PLoS Biology by Megan Head and colleagues concluded that p-hacking is widespread across science, although its effect on the conclusions of meta-analyses appeared weak compared with the real effects being measured.
A 2016 guide in the European Journal of Epidemiology by Sander Greenland, Douglas Altman and five co-authors listed 25 common misinterpretations of p-values, confidence intervals and statistical power. It warns that choosing which analyses to present based on the p-values they produce can yield small p-values even when the hypothesis being tested is correct. This is one reason pre-registration, where researchers state their main outcome before collecting data, has become an important safeguard.
What should you look at instead of the p-value?
- The effect size: how big is the difference, in absolute terms as well as relative ones?
- The confidence interval: what range of effects are the data compatible with, and does it include effects too small to matter?
- The outcome: is it something people care about, such as illness or death, or a lab marker?
- The sample size: tiny p-values are easy in huge studies, and big effects are easy to overstate in small ones.
- Pre-registration: did the researchers say in advance which outcome they would test?
A p-value is one piece of evidence about how well data fit with 'no effect'. It is not a verdict, and on its own it says nothing about whether a result matters to patients.
This article is for education about research methods only. It is not medical advice, so speak to a qualified health professional about any treatment decision.
Frequently asked
- What does statistically significant mean in simple terms?
- It usually means the p-value was below 0.05: if there were truly no effect, data at least this extreme would turn up less than 5% of the time. The American Statistical Association has stressed that this does not tell you the probability the finding is true, or how large or important the effect is.
- Does statistically significant mean the result is important?
- No. Large studies can make tiny differences significant. In the aspirin arm of the US Physicians' Health Study, with 22,071 participants, heart attacks fell by 44% with a p-value below 0.00001, yet one analysis put the absolute risk difference at just 0.77%. Always check the size of the effect, not only the p-value.
- Does a non-significant result mean there is no effect?
- Not necessarily. A p-value above 0.05 can simply mean the study was too small or too noisy to tell. In 2019 more than 800 scientists backed a Nature comment criticising the habit of declaring 'no difference' between groups just because a result was not statistically significant. Inconclusive is not the same as no effect.
- What is p-hacking?
- P-hacking means collecting or analysing data in different ways until a non-significant result becomes significant, then reporting the version that crossed the line. A 2015 text-mining study in PLoS Biology concluded it is widespread across science, although its effect on the conclusions of meta-analyses appeared weak compared with the real effects being measured.
The studies and sources behind this page
- American Statistical Association. ASA statement on statistical significance and p-values (news release, 7 March 2016), presenting Wasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose. Am Stat 2016;70:129-33.
- Steering Committee of the Physicians' Health Study Research Group. Final report on the aspirin component of the ongoing Physicians' Health Study. N Engl J Med 1989;321:129-35.
- Sullivan GM, Feinn R. Using effect size - or why the P value is not enough. J Grad Med Educ 2012;4:279-82.
- Amrhein V, Greenland S, McShane B. Scientists rise up against statistical significance. Nature 2019;567:305-7.
- Benjamin DJ, Berger JO, Johannesson M, et al. Redefine statistical significance. Nat Hum Behav 2018;2:6-10.
- Chavalarias D, Wallach JD, Li AH, Ioannidis JP. Evolution of reporting P values in the biomedical literature, 1990-2015. JAMA 2016;315:1141-8.
- Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD. The extent and consequences of p-hacking in science. PLoS Biol 2015;13:e1002106.
- Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol 2016;31:337-50.
Lessons on this topic
Related reading
Educational content only. Not medical advice, diagnosis or treatment. Always consult a qualified healthcare professional before changing your health regimen.