Resources - RESEARCH LITERACY
How important is study sample size for meaningful results?
Sample size matters a lot: small studies often miss real effects and exaggerate the ones they find, while large studies give more precise, reliable answers.
Sample size matters a great deal. A study that is too small has a low chance of detecting a real effect, and when it does report a positive result, that result is more likely to be exaggerated or wrong. Large, well-designed studies give more precise and more trustworthy estimates, which is why dramatic early findings from small trials often shrink when bigger trials follow.
What does sample size actually change?
Every study measures a sample of people and uses it to estimate what would happen in everyone like them. Chance always adds noise. With a handful of participants, a few unusual people can swing the result. With thousands, those quirks tend to cancel out and the estimate settles closer to the truth. Statisticians call this precision: bigger samples give narrower confidence intervals, the range of values the data are compatible with.
Sample size also sets a study's statistical power, the chance it will detect an effect of a given size if that effect is real. Well-run trials work out the numbers they need before they start, based on the smallest effect that would matter to patients.
Can a small study miss a real effect?
Yes, and this has been documented for decades. In 1978 Jennie Freiman, Thomas Chalmers and colleagues re-examined 71 randomised trials that had been published as 'negative', meaning they found no difference between treatments. Writing in the New England Journal of Medicine, they reported that 67 of the 71 had more than a 10% risk of missing a true 25% improvement, and 50 could have missed a 50% improvement. Their conclusion was that many therapies labelled 'no different from control' had not received a fair test.
The authors also calculated confidence intervals for each trial. In 57 of the 71, the data were still compatible with a 25% improvement, and in 34 with a 50% improvement. In other words, many of these 'negative' trials could not rule out a worthwhile benefit. The lesson for readers is simple. 'No significant effect' from a small study often means 'we could not tell', not 'it does not work'.
Why do small studies often exaggerate results?
This is the less obvious problem. When a small study does find a significant result, the effect it reports tends to be too big. A 2013 analysis in Nature Reviews Neuroscience by Katherine Button, John Ioannidis, Marcus Munafò and colleagues found that the average statistical power of neuroscience studies was very low. They showed that low power does not only miss true effects. It also makes a statistically significant result less likely to reflect a true effect, inflates effect sizes and makes results harder to reproduce.
The same pattern shows up across medicine. A 2013 BMJ study by Agnès Dechartres and colleagues examined 93 meta-analyses containing 735 randomised trials. Compared with trials of 1,000 patients or more, treatment effects were on average 48% larger in trials with fewer than 50 patients. Comparing trials within each meta-analysis, the smallest quarter reported effects 32% larger than the largest quarter.
A 2012 JAMA study by Tiago Pereira, Ralph Horwitz and John Ioannidis searched the Cochrane Database of Systematic Reviews for very large treatment effects. These typically appeared in small trials, with a median of just 18 events in first trials. When further trials were added, 90% of the very large effects seen in first trials became smaller, and the median odds ratio fell from 11.88 to 4.20. Across the whole database they found only one intervention with a large, well-supported benefit on survival.
Do famous studies get overturned?
Sometimes, and smaller ones more often. In a 2005 JAMA paper, Ioannidis followed up 49 highly cited clinical studies published between 1990 and 2003, each cited more than 1,000 times. Of the 45 that claimed an intervention worked, 7 were later contradicted and 7 more had reported stronger effects than later studies found. Among randomised trials, the ones that were contradicted or overstated tended to be smaller than those that held up. Study design mattered even more: 5 of the 6 highly cited non-randomised studies had been contradicted or had found stronger effects, against 9 of the 39 randomised trials.
The most notorious example of a tiny study causing outsized harm is the 1998 Lancet paper by Andrew Wakefield and colleagues. It described 12 children and reported that parents linked the onset of their child's symptoms to the MMR vaccine in eight cases. A case series of 12 children with no comparison group cannot establish cause and effect. The paper's problems went well beyond its size, and The Lancet fully retracted it in 2010.
What does a large trial look like?
For contrast, the ISIS-2 trial, published in The Lancet in 1988, randomised 17,187 people with a suspected heart attack across 417 hospitals to streptokinase (a clot-dissolving drug), aspirin, both, or neither, with placebo control. Vascular deaths at five weeks were 9.2% with streptokinase against 12.0% with a placebo infusion, and 9.4% with aspirin against 11.8% with placebo tablets.
Differences of two or three percentage points are easy to miss, or to exaggerate, in a small study. With more than 17,000 people, ISIS-2 could measure them with real confidence.
Is a small study ever worth doing?
Yes, for the right purpose. Pilot studies test whether a design is workable, and studies of rare diseases may never be able to recruit large numbers. Some researchers defend small trials on the grounds that a later meta-analysis can combine them. In a 2002 JAMA article, Scott Halpern, Jason Karlawish and Jesse Berlin examined that argument and concluded that underpowered trials are ethical in only two situations: small trials for rare diseases with explicit plans to combine results in a prospective meta-analysis, and early-phase trials designed for purposes other than comparing treatments. In both cases, they said, participants should be told their involvement may only indirectly help future patients.
How can you judge sample size when reading a study?
- Look for the number of participants actually analysed, not just how many were recruited.
- Check whether the authors reported a sample size or power calculation made before the study began.
- Treat a dramatic effect from a small study as a question to be tested, not an answer.
- If a small study found 'no difference', ask whether it was big enough to detect one.
- Look for larger trials or a meta-analysis on the same question before drawing conclusions.
Size is not everything. A large study with poor design, no control group or biased measurement can still mislead. But size sets a limit on how much any single study can tell you, and a small sample should lower your confidence in its result accordingly.
This article is for education about research methods only. It is not medical advice, so speak to a qualified health professional about any treatment decision.
Frequently asked
- Why is sample size important in research?
- Sample size controls how precise a study's estimate is and how likely it is to detect a real effect. Small studies are easily swayed by chance. A 2013 BMJ analysis of 735 trials found treatment effects in trials with fewer than 50 patients were on average 48% larger than in trials with 1,000 or more.
- What is statistical power?
- Statistical power is the chance that a study will detect an effect of a given size if that effect is real. It depends mainly on sample size. A 1978 review of 71 'negative' trials found most had a high risk of missing a true 25% improvement, so many treatments labelled ineffective had never had a fair test.
- Can a small sample size still be useful?
- Yes, for the right purpose. Pilot studies test whether a design works, and rare diseases may never allow large trials. A 2002 JAMA article argued small trials are ethical mainly for rare diseases with plans to pool results in a meta-analysis, and for early-phase trials designed for purposes other than comparing treatments.
- Why do small studies show bigger effects?
- Chance swings results far more in small studies, so some produce striking estimates by luck alone. A 2012 JAMA study found very large treatment effects usually came from small trials, and 90% of those seen in first trials shrank once later trials were added to the evidence.
The studies and sources behind this page
- Freiman JA, Chalmers TC, Smith H Jr, Kuebler RR. The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial. Survey of 71 "negative" trials. N Engl J Med 1978;299:690-4.
- Button KS, Ioannidis JP, Mokrysz C, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci 2013;14:365-76.
- Dechartres A, Trinquart L, Boutron I, Ravaud P. Influence of trial sample size on treatment effect estimates: meta-epidemiological study. BMJ 2013;346:f2304.
- Pereira TV, Horwitz RI, Ioannidis JP. Empirical evaluation of very large treatment effects of medical interventions. JAMA 2012;308:1676-84.
- Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA 2005;294:218-28.
- Wakefield AJ, Murch SH, Anthony A, et al. Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. Lancet 1998;351:637-41. Retracted in Lancet 2010;375:445.
- ISIS-2 Collaborative Group. Randomised trial of intravenous streptokinase, oral aspirin, both, or neither among 17,187 cases of suspected acute myocardial infarction: ISIS-2. Lancet 1988;2:349-60.
- Halpern SD, Karlawish JH, Berlin JA. The continuing unethical conduct of underpowered clinical trials. JAMA 2002;288:358-62.
Lessons on this topic
Related reading
Educational content only. Not medical advice, diagnosis or treatment. Always consult a qualified healthcare professional before changing your health regimen.