Are observational studies useless?

No, but they show links rather than proof of cause, because people who choose a habit often differ in other ways. Observational data suggested hormone therapy protected the heart, yet the randomised Women's Health Initiative trial found more coronary heart disease and strokes in women given hormones. Observational findings are best treated as leads for trials to test.

How often are medical findings overturned?

Often enough to matter. Of 45 highly cited studies claiming a treatment worked, a 2005 JAMA analysis found 16% were later contradicted and another 16% had overstated the effect. A review of a decade of articles in one major journal found that 40% of studies testing an established practice reversed it.

Why does the news keep flip-flopping on food and health?

Because single nutrition studies often report large effects on weak statistics, and newspapers cover first findings far more than corrections. A review of 50 cookbook ingredients found most had been linked to cancer one way or the other, but pooled analyses showed effects close to zero. A French study found refutations of earlier claims were rarely reported.

Health studies seem to contradict each other because headlines report noisy single studies, and early results often shrink or reverse once bigger trials arrive.

Resources - RESEARCH LITERACY

Why do health studies contradict each other so often?

Health studies seem to contradict each other because headlines report noisy single studies, and early results often shrink or reverse once bigger trials arrive.

Health studies seem to contradict each other mostly because headlines report single studies, and single studies are noisy. Most of the contradictions you see are early, small or observational findings meeting later, larger and better designed ones, then reaching you through press releases and news stories that overstate both. The underlying science usually moves more slowly, and more consistently, than the coverage suggests.

That does not mean nothing is known. It means the newest study is rarely the best one to judge by. This article is about the steady drip of contradictory headlines on food, drink, supplements and everyday habits, rather than why two particular trials of the same compound disagree, which we cover separately. The examples below show where the flip-flopping comes from and how to tell a real reversal from noise.

Is everything really linked to cancer?

It can feel that way, and there is a study on exactly that. In 2013, Jonathan Schoenfeld and John Ioannidis picked 50 common ingredients from random recipes in a cookbook and searched for studies linking each one to cancer. Forty of the 50 had been studied. Of 264 individual study results, 191 (72%) concluded the food raised or lowered cancer risk, yet 75% of the risk estimates had weak or no statistical support. When the same foods were pooled in meta-analyses, the effects mostly shrank to nothing, with a median relative risk of 0.96.

This is the engine behind the eggs-are-good, eggs-are-bad cycle. Many small studies, each with a large-looking effect, point in both directions. Pooled together, they largely cancel out. If you only ever see the individual studies, one at a time, in the news, it looks like science keeps changing its mind.

Why do observational studies and trials disagree?

Because they answer different questions. An observational study watches what people already do and compares their outcomes. It cannot fully separate the habit from the kind of person who has it. A randomised trial decides by chance who gets the treatment, which is designed to remove that problem.

Hormone replacement therapy is the textbook case. The Nurses' Health Study followed 70,533 postmenopausal women and, in a 2000 report, found that current users of hormone therapy had a lower risk of major coronary events than women who had never used it, with a relative risk of 0.61. Two years later, the Women's Health Initiative randomly assigned 16,608 women aged 50 to 79 to combined hormone therapy or placebo. The trial was stopped early. Women given hormones had more coronary heart disease, strokes, blood clots in the lungs and invasive breast cancer, and the investigators concluded the regimen should not be started or continued to prevent heart disease. The trial tested one combined regimen in healthy women, so it does not answer every question about hormone therapy. But it showed how far an observational signal can mislead.

Beta-carotene followed the same pattern. Diets rich in carotenoid fruit and vegetables, and high blood levels of beta-carotene, had been associated with a lower risk of lung cancer. Then a Finnish trial randomly assigned 29,133 male smokers to supplements or placebo. Published in 1994, it found that the men given beta-carotene had an 18% higher incidence of lung cancer, and their total mortality was 8% higher.

In both cases the observational studies recorded something real: those groups really did have better outcomes. What the studies could not do was tell the treatment apart from everything else that differed about the people who used it.

How often do big medical findings get reversed?

More often than most people expect, though less often than the headlines imply. In 2005 John Ioannidis looked at 45 highly cited clinical studies, each cited more than 1,000 times, that had claimed a treatment worked. Later research contradicted 7 of them (16%) and found weaker effects for another 7 (16%). Twenty (44%) were replicated and 11 (24%) remained largely unchallenged. The pattern was uneven: five of the six non-randomised studies had been contradicted or had overstated their effect, against 9 of the 39 randomised trials, and among the trials, the smaller ones were more likely to be overturned.

A 2013 analysis by Vinay Prasad and colleagues reviewed ten years of original articles, from 2001 to 2010, in one high-impact medical journal. Of 363 articles that tested an established medical practice, 146 (40%) found the practice was no better than a lesser alternative. The authors called this medical reversal.

Reversals are not a sign that science is broken. They are science working, slowly. The trouble is that the first claim usually gets the publicity and the correction does not.

Why do news stories make it look worse?

Part of the problem starts before journalists see the study. In 2014, researchers at Cardiff University compared 462 health-related press releases from 20 leading UK universities with the original papers and 668 news stories. Of the press releases, 40% contained exaggerated advice, 33% made causal claims from research that only showed a correlation, and 36% inferred effects in humans from animal research. When a release exaggerated, the news usually followed: 81% of news stories carried an exaggerated causal claim when the release did, compared with 18% when the release was accurate.

The second problem is which studies get covered at all. A 2017 study from the University of Bordeaux traced 156 biomedical studies that had been reported in 1,561 newspaper articles and checked them against later meta-analyses. Only 48.7% of the covered studies were confirmed. Newspapers never covered initial studies that reported null findings. While 234 newspaper articles covered 35 initial studies that were later disconfirmed, only four press articles covered a subsequent null finding and mentioned that the original claim had been refuted.

How should you read a contradictory health headline?

Find out what kind of study it is. An observational study shows a link. A randomised trial is the better test of cause and effect.
Check who was studied. Mice, a small group of students and a large population of older adults are not interchangeable.
Look at the size of the effect, not just its direction. Linked to a higher risk can mean a very small change.
Ask whether it is the first study or one of many. First findings are the ones most likely to shrink.
Look for a systematic review or meta-analysis. In the cookbook study, pooled results were far more cautious than single studies.
Read past the headline to the paper's own wording. Press releases and news often go further than the researchers did.
Notice what is missing. Null results and corrections rarely make the news.

So can you trust anything?

Yes. The examples above point the same way. Randomised trials were overturned less often than non-randomised studies, small trials more often than large ones, and pooled analyses were more cautious than the single studies they combined. When a finding has held up across several independent, well-designed studies, one contrary headline is rarely a reason to change your mind. When a finding rests on one striking study, a contrary headline is a reminder that it was never settled in the first place.

This article is for education about how to read research and is not medical advice. Do not start, stop or change any treatment because of it; speak to your GP or another qualified professional.

Frequently asked

Why do health studies contradict each other?: Mostly because headlines report single studies, often small or observational, and single studies are noisy. Later, larger trials often find smaller effects or none. Press releases and news coverage then exaggerate the first claim and rarely report the correction, so the science looks far more changeable than it is. Pooled reviews usually tell a steadier story.
Are observational studies useless?: No, but they show links rather than proof of cause, because people who choose a habit often differ in other ways. Observational data suggested hormone therapy protected the heart, yet the randomised Women's Health Initiative trial found more coronary heart disease and strokes in women given hormones. Observational findings are best treated as leads for trials to test.
How often are medical findings overturned?: Often enough to matter. Of 45 highly cited studies claiming a treatment worked, a 2005 JAMA analysis found 16% were later contradicted and another 16% had overstated the effect. A review of a decade of articles in one major journal found that 40% of studies testing an established practice reversed it.
Why does the news keep flip-flopping on food and health?: Because single nutrition studies often report large effects on weak statistics, and newspapers cover first findings far more than corrections. A review of 50 cookbook ingredients found most had been linked to cancer one way or the other, but pooled analyses showed effects close to zero. A French study found refutations of earlier claims were rarely reported.

The studies and sources behind this page

Lessons on this topic

Correlation vs cause: reading observational studies (Heat & Sauna)
Sauna and longevity: what the cohort data shows (Heat & Sauna)
Seed oils: the claim vs the trials (Nutrition, Diet & Fasting)