Resources - RESEARCH LITERACY
What is publication bias, and how does it distort evidence?
Publication bias means positive studies get published more often than null ones, so the visible evidence makes treatments look better than the full data shows.
Publication bias is what happens when studies with positive or statistically significant results are more likely to be published than studies that found nothing. The published record then looks more favourable than the full set of research that was actually done, so treatments seem to work better, and more consistently, than they really do.
Nobody has to lie for this to happen. Journals like new, striking findings. Researchers know a positive result is easier to publish and better for their careers. Null results feel inconclusive, so they stay unpublished in what researchers call the file drawer. Each decision makes sense on its own. Added together, they bend the evidence.
How much can publication bias distort the evidence?
The clearest measurement comes from antidepressants. In 2008, a team at Oregon Health and Science University compared the US Food and Drug Administration's reviews of 74 registered trials of 12 antidepressants, involving 12,564 patients, with what appeared in medical journals. Of the trials the FDA judged positive, 37 were published and one was not. Of the trials it judged negative or questionable, 22 were never published and 11 were published in a way that, in the authors' view, conveyed a positive outcome. Reading the journals, 94% of the trials looked positive. The FDA's own analysis put the figure at 51%. Overall, the published literature inflated the apparent effect size by 32%.
Notice what this does and does not show. Half the trials were still positive by the regulator's count. The distortion was in how much better the drugs looked in print than in the complete data.
What happens when the missing trials are found?
Sometimes the hidden data change the verdict. In 2010, researchers including staff at Germany's Institute for Quality and Efficiency in Health Care (IQWiG) analysed 13 trials of the antidepressant reboxetine involving 4,098 patients. Data on 74% of those patients had never been published. Once the unpublished data were included, reboxetine showed no significant difference from placebo in remission rates and did worse than SSRI antidepressants. The published data had overestimated its benefit over placebo by up to 115% and underestimated its harms. The BMJ paper concluded that reboxetine was, overall, an ineffective and potentially harmful antidepressant.
Are positive trials really published more often?
Yes. A 2009 Cochrane methodology review pooled studies that followed groups of registered clinical trials to see which ones reached print. Trials with positive findings had almost four times the odds of being published (odds ratio 3.90). In plainer terms, if 41% of negative trials are published, you would expect about 73% of positive ones to be. Positive trials also tended to appear sooner, after four to five years, against six to eight years for negative ones. That delay means early summaries of any new treatment are the ones most likely to be flattering.
Can a study be published and still biased?
Yes, and this kind is harder to spot. A trial usually measures several outcomes. If only the ones that reached significance are reported in full, a mostly null study can read as a success. A 2013 review in PLoS One, led from the University of Liverpool, found that 40% to 62% of trials had at least one primary outcome that was changed, introduced or omitted between the protocol and the publication. Statistically significant outcomes had between 2.2 and 4.7 times the odds of being fully reported.
Spin adds another layer. A trial whose main outcome showed no difference can lead its abstract with a secondary result, or describe a non-significant trend as promising. Reading only the abstract is where readers are most easily caught.
How do researchers spot publication bias?
The classic tool is the funnel plot. Each study in a meta-analysis is plotted by its result against its size or precision. Large studies cluster tightly near the true effect, and small studies scatter more widely on both sides, so a complete set of studies looks like an upside-down funnel. If the small studies with disappointing results are missing, one corner of the funnel is empty and the plot looks lopsided.
In 1997 Matthias Egger and colleagues at the University of Bristol published a simple statistical test for that lopsidedness in the BMJ. Applying it, they found signs of bias in 14 of 37 meta-analyses (38%) from leading general medical journals and 5 of 38 (13%) Cochrane reviews. They also found eight cases where a meta-analysis could be compared with a later large trial. Four pairs disagreed, and in every one the meta-analysis had shown the bigger effect. They warned that the test has limited power when a meta-analysis rests on a few small trials, so a lopsided plot is a warning sign rather than proof.
Has anything reduced publication bias?
Registering trials before they start is the biggest change. Once a trial's main outcome is on public record in advance, it is much harder to bury the result or quietly swap the outcome. A 2015 analysis by Robert Kaplan and Veronica Irvin looked at 55 large trials of drugs or dietary supplements for cardiovascular disease, funded by the US National Heart, Lung, and Blood Institute. Of 30 published before 2000, 17 (57%) reported a significant benefit on their primary outcome. Of 25 published after 2000, only 2 (8%) did. Registering on ClinicalTrials.gov before publication was strongly associated with the shift, and the authors suggested that declaring outcomes in advance may have contributed.
Registered reports go a step further. The journal reviews the question and methods before any data exist and agrees to publish whatever the result. In 2021, Anne Scheel and colleagues compared 71 registered reports in psychology with 152 standard papers. In the standard papers, 96% reported a positive result for their first hypothesis. In the registered reports, 44% did. The authors suggested that less publication bias was a plausible part of the explanation.
How can you tell if a body of evidence is affected?
- Check whether a review searched trial registries and unpublished data, not just journals.
- Look for a funnel plot or a test for small-study effects, and see whether the authors discuss it.
- Be wary when the evidence is many small positive studies and no large ones.
- Compare the registered protocol with the paper. ClinicalTrials.gov and similar registries are free to search.
- Give more weight to pre-registered trials and registered reports.
- Watch whether the effect shrinks as studies get bigger and newer. Early, small studies are the most likely to be inflated.
Does publication bias mean published research is wrong?
No. It means the published literature is a selected sample, tilted towards positive results. The effect may be real but smaller and less certain than it looks. The practical response is not to distrust everything, but to ask what might be missing, and to give the most weight to evidence that could not have been filtered by its result: registered trials, registered reports and reviews that went looking for the unpublished data.
This article explains a research method for educational purposes and is not medical advice. Do not start, stop or change any medicine because of it; speak to your doctor or pharmacist.
Frequently asked
- What is publication bias in simple terms?
- Publication bias is the tendency for studies with positive or statistically significant results to be published more often, and sooner, than studies that found no effect. The visible research then overstates how well a treatment works. A Cochrane methodology review found that trials with positive findings had almost four times the odds of being published.
- Does publication bias mean published results are fake?
- No. The effect may be real but smaller and less certain than the published studies suggest. In antidepressant trials held by the US regulator, 51% were positive, but the journals made it look like 94%. The problem is usually what is missing from the record, not what is in it.
- How do researchers detect publication bias?
- The main tools are funnel plots, which show whether small negative studies seem to be missing, and statistical tests for asymmetry such as Egger's test. The most direct check is comparing trial registries and regulators' records with what was published. Funnel plot tests are weak when only a few small studies exist, so treat them as warning signs.
- What is outcome reporting bias?
- Outcome reporting bias happens when a published study fully reports only the outcomes that turned out significant, or changes its main outcome after the data are in. A 2013 review found that 40% to 62% of trials had at least one primary outcome changed, added or dropped between the protocol and the published paper.
- Has trial registration reduced publication bias?
- It appears to have helped. Among large US heart, lung and blood institute trials, 57% published before 2000 reported a significant benefit, against 8% of those published after, and registering in advance was strongly linked to the change. In psychology, registered reports show far fewer positive results than standard papers: 44% against 96%.
The studies and sources behind this page
- Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008;358(3):252-60.
- Eyding D, Lelgemann M, Grouven U, et al. Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ 2010;341:c4737.
- Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev 2009;(1):MR000006.
- Dwan K, Gamble C, Williamson PR, Kirkham JJ. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One 2013;8(7):e66844.
- Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997;315(7109):629-34.
- Kaplan RM, Irvin VL. Likelihood of null effects of large NHLBI clinical trials has increased over time. PLoS One 2015;10(8):e0132382.
- Scheel AM, Schijen MRMJ, Lakens D. An excess of positive results: comparing the standard psychology literature with registered reports. Adv Methods Pract Psychol Sci 2021;4(2).
Lessons on this topic
Related reading
Educational content only. Not medical advice, diagnosis or treatment. Always consult a qualified healthcare professional before changing your health regimen.