Resources - RESEARCH LITERACY
What peer review means in health research
Peer review means independent experts check a study before a journal publishes it. It is a useful filter, but trials show reviewers miss most planted errors.
Peer review is the check a study goes through before a journal publishes it. The editor sends the manuscript to independent experts in the field, who comment on its methods, analysis and conclusions and recommend whether it should be published. It is a useful filter, but it is not a guarantee: controlled experiments at medical journals show that reviewers typically spot only a few of the major errors in a paper.
So "peer reviewed" is best read as a minimum standard. It tells you that a few experts looked at the paper and the editor was satisfied. It does not tell you that the results are true, that the data are genuine or that other researchers will find the same thing.
How does peer review work?
The details vary between journals, but the usual route is the same. Researchers submit a manuscript. An editor decides whether it is worth sending out at all, and many papers are rejected at this stage. If it goes forward, the editor invites reviewers, usually unpaid academics working in the same area. They send back written comments and a recommendation: accept, revise or reject. The authors may be asked to answer the comments and resubmit, sometimes more than once, before a final decision.
Reviewers typically ask questions like these:
- Is the research question clear, and does the study design suit it?
- Were participants chosen and allocated fairly, for example by proper randomisation?
- Are the statistics appropriate, and are the results reported fully?
- Do the conclusions follow from the data, or do they go further?
- Was the study ethically approved and are conflicts of interest declared?
What reviewers usually do not do is just as important. They generally judge the paper in front of them. They rarely see raw data and do not repeat the experiment. Richard Smith, who edited the BMJ for 13 years, wrote in 2006 that peer review is not a reliable way to detect fraud because it works on trust. In the same essay he described it as slow, expensive, prone to bias and "something of a lottery".
Does peer review catch mistakes?
Journals have tested this directly by planting errors in papers. In a randomised trial published in JAMA in 1998, Godlee, Gale and Martyn took a paper already accepted for publication at a general medical journal, added eight weaknesses, and sent it to 420 of the journal's reviewers. Of the 221 who returned reports, the average reviewer commented on just two of the eight.
A larger trial reported in 2008 by Schroter and colleagues sent 607 BMJ reviewers three test papers, each with nine major and five minor errors inserted. On the first paper, reviewers found an average of 2.58 of the nine major errors. Short training packages made only a slight difference, and the figures for the second and third papers were 2.71 and 3.0. The authors concluded that editors should not assume reviewers will detect most major errors.
What about the bigger picture? A 2007 Cochrane review by Jefferson and colleagues gathered 28 studies of editorial peer review in biomedical journals. It concluded that little empirical evidence is available to support peer review as a way to ensure the quality of biomedical research, while noting that it may make papers more readable and improve the general quality of reporting. The authors were careful to add that a lack of evidence is not evidence that peer review does nothing.
Is anonymous or open peer review better?
Journals use several models. In single-anonymous review the reviewers know who the authors are but not the other way round. In double-anonymous review neither side knows. In open review the reviewers' names are shared, and some journals publish the reports alongside the paper.
The trials suggest the choice matters less than people expect. In the 1998 JAMA trial, neither hiding the authors' identity from reviewers nor asking reviewers to sign their reports changed how many weaknesses they found. In a 1999 BMJ trial by van Rooyen and colleagues, reviewers who were asked to be identified to the authors produced reviews of the same quality as anonymous reviewers. They were more likely to decline to review, though: 35% against 23%. The 2007 Cochrane review likewise found no clear-cut evidence that concealing identities improved the outcome.
Can a peer-reviewed study still be wrong?
Yes, and two famous cases in The Lancet show how. In 1998 the journal published a paper by Wakefield and colleagues describing 12 children with bowel and developmental problems. The parents of eight of the children linked the start of their child's symptoms to the MMR vaccine. The paper passed peer review at one of the best-known medical journals in the world. The Lancet fully retracted it in February 2010, 12 years later.
The second case moved much faster. In May 2020 The Lancet published a large registry analysis of hydroxychloroquine and chloroquine for COVID-19, based on data supplied by a company called Surgisphere. Concerns were quickly raised about whether the data were genuine. The authors arranged an independent audit, but Surgisphere would not transfer the full dataset, and in June 2020 the authors asked for the paper to be retracted, saying they could no longer vouch for the primary data. Normal peer review had judged the manuscript, not the database behind it, which is exactly the weakness Smith described.
Does "peer reviewed" mean a journal is trustworthy?
Not on its own. Any journal can say it uses peer review. In 2013, John Bohannon, writing in Science, described sending a deliberately flawed spoof paper to open-access journals. His report, titled "Who's afraid of peer review?", found little or no scrutiny at many of them. A paper in a journal you have never heard of, with no sign of who reviewed it or how, deserves extra caution.
How should you read a peer-reviewed study?
- Treat peer review as a minimum. It means a few experts looked at the paper, not that the result is settled.
- Check for retractions and corrections. PubMed labels retracted papers clearly, and journals publish retraction notices.
- Look at the study design. A large randomised trial tells you more than a small uncontrolled one, whatever journal it is in.
- Look for confirmation. One study is a single data point. Findings that have been repeated by independent groups, or pooled in a systematic review, are far more reliable.
- Be aware of preprints. Many studies now appear online before peer review. They can be useful, but they have not yet been through even this basic check.
Peer review is still worth having. It catches some errors, improves how papers are written and gives editors a second opinion. It is simply one layer of checking, and the evidence shows it is a thinner layer than many readers assume.
This article is for general education about how health research is published and checked. It is not medical advice.
Frequently asked
- What does peer reviewed mean?
- It means that before a journal published the study, the editor sent it to independent experts who checked its methods, analysis and conclusions and recommended whether it should be published. It is a minimum standard, not proof that the findings are right. Reviewers judge the manuscript in front of them and usually work on trust about the underlying data.
- Does peer review catch errors?
- Some, but not most. In a BMJ trial, 607 reviewers were sent papers with nine major errors deliberately inserted, and at the start they found an average of 2.58. In an earlier experiment with a paper containing eight weaknesses, reviewers commented on two on average. Peer review filters out some problems, but it is far from a full check.
- Can a peer-reviewed study be wrong?
- Yes. Wakefield's 1998 Lancet paper on 12 children passed peer review and was fully retracted in 2010. In 2020, a large Lancet study of hydroxychloroquine was retracted within weeks because independent reviewers could not get access to the full dataset. Check whether a paper has been retracted and whether other studies have confirmed it.
- Is open peer review better than anonymous review?
- The trials so far suggest it makes little difference to quality. A 1999 BMJ trial found no significant difference in review quality when reviewers were identified to authors, though more of them declined to review. A 1998 trial found that hiding authors' names or asking reviewers to sign their reports did not change how many errors they found.
The studies and sources behind this page
- Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev 2007;(2):MR000016.
- Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA 1998;280:237-240.
- Schroter S, Black N, Evans S et al. What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med 2008;101:507-514.
- van Rooyen S, Godlee F, Evans S et al. Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. BMJ 1999;318:23-27.
- Smith R. Peer review: a flawed process at the heart of science and journals. J R Soc Med 2006;99:178-182.
- Wakefield AJ, Murch SH, Anthony A et al. Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. Lancet 1998;351:637-641. Retracted: Lancet 2010;375:445.
- Mehra MR, Ruschitzka F, Patel AN. Retraction: Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet 2020;395:1820.
- Bohannon J. Who's afraid of peer review? Science 2013;342:60-65.
Lessons on this topic
Related reading
Educational content only. Not medical advice, diagnosis or treatment. Always consult a qualified healthcare professional before changing your health regimen.