Modern healthcare’s “dirty secrets”: 1. Results that cannot be reproduced
“Insanity is doing the same thing over and over again and expecting different results”
If modern medical research were to be held up to scrutiny on the basis of this popular quote (commonly, but erroneously, attributed to Einstein), it appears that it needs urgent help.
Reproducibility, a key tenet of Science, is the idea that when an experiment is repeated by other scientists, they will get the same result. There is a persistent buzz in the air of medical scientific research about a “reproducibility crisis”. Simply put, it means that when independent researchers attempt to reproduce the results of a previous study or research paper, they commonly fail to duplicate the original results.
In the publish or perish atmosphere of modern science, researchers prioritise “getting it published” over “getting it right”.
It requires no great effort of thought to appreciate the consequences of this problem. We take medicines and interventions with the trust that they will work reliably and that this reliability, like in all other areas of daily activity, has been vigorously tested. When you buy an iPhone or some other gadget, you are guaranteed that “it just works”. When it doesn’t, the furore is predictable. Surely, modern Medicine, which deals with the most important commodity in life: human health, wellness and longevity, has to give its users the same assurance.
It looks like, it cannot.
Numerous published research papers, many of high impact and readership, have not been supported by the efforts of other scientists who went about repeating these studies. Far from being an occasional event, modern medical research is full of these worrisome reports. Here’s a sample of the problem.
- A large-scale effort to closely replicate 100 studies in psychology, found that only 36% of them could be replicated.
- Another study showed that only 6 of 53 “landmark” cancer trials were reproducible in the community.
- Of 585 registered trials, 171 remained unpublished over the years. Close to 300,000 patients were involved in these trials.
- Between 2000 and 2010, 80,000 patients were included in trials that were subsequently retracted because the authors’ conclusions were not sustainable.
- A large body of research is industry-sponsored and appears to support a product or concept that is the money spinner for the concerned industry. The nexus is deep, often subtle. When tested by unbiased researchers, the conclusions fail to hold up.
The list is long and keeps growing over time.
The unreliability of risk factors
Hardly a week passes without the popular media reporting on common risk factors for diseases and disorders. Many of these associations are robust and have stood the test of time: smoking and cancer, alcohol abuse and liver disease, ultraviolet ray exposure and skin cancers in light skinned individuals, and many more.
Equally, many other well known risk factors have been found to have no basis or even disproved. For decades, the association of fats with heart disease was taken as gospel. Massive public education programmes were launched warning people of the dangers of high fat intake in the diet. In recent years, there has been a near total reversal of this idea. It appears that carbohydrate intake, not fats, are the real criminals. Large carb loads were consumed in an effort to cut back on fats and has resulted in the current epidemic of obesity and the problems related to it.
Just this week, an extensive review has shown that the much touted omega-3 supplement has no benefit: not in heart disease, cancer, dementia or any of the conditions for which billions have been spent in buying a chemical that appears to be useless.
Changes in direction are often 180 degree. The public remains totally confused and anxious.
Breakthrough news in healthcare: half of them are never confirmed on follow up
Stories of landmark advances in cancer, behavioural disorders like depression, heart attacks, strokes and other common conditions appear almost daily. A study of 199 newspapers showed that only half of the reports were sustained over time. The unfortunate part is that most refutations and failures are not reported.
An analysis of 156 reports in medical journals that were reported in 1475 newspaper articles showed some alarming facts.
- Newspapers never covered initial studies reporting null (no impact or difference) findings.
- They rarely reported subsequent null observations.
- 234 newspaper articles covered 35 initial studies that were later disconfirmed, but only 4 press articles covered a subsequent null finding and mentioned the refutation of an initial claim.
How do we handle this crisis?
The problem at hand is real and cannot be handled by a single strategy that throws a switch to turn on the light. Here’s a list of suggestions to counter this problem.
- Open Research: Scientific knowledge of all kinds should be openly shared as early as it is practical in the discovery process. We need to reward the publication of research outputs along the entire process, rather than just each journal article as it is published.
- Registered Reports was launched in 2013, where researchers register the proposal and how the research will be conducted, before any experimental work commenced. It allows editorial decisions to be based on the rigour of the experimental design and increases the likelihood that the findings could be replicated.
- Publications of reproduced research: Isolated findings don’t mean much until they have been replicated. Replication attempts are uncommon in science. Journals seek novel theories and findings, and view replications as treading over old ground which offers little incentive for career-minded academics to conduct replications where teams of researchers follow identical procedures independently and aim to replicate important findings from the literature. A single paper then collates and analyses them to establish the size and reproducibility of the original study.
- Being scooped is loosely defined as when two independent groups studying the same system produce the same or similar results, and one group publishes their work first. The resulting race to publish ahead of competitors is intense and to the detriment of the scientific endeavour. Being scooped is often considered to devalue the second, complementary study; many journals will reject it citing lack of novelty. However, there is a self-evident benefit to publishing complementary research. Just as summiting Everest second is still an incredible achievement, so too, is the scientific research resulting from a group who have (perhaps inadvertently) replicated the important findings of another group. Some scientific journals are defusing the fear of getting “scooped” by making it easier for scientists to publish results that have appeared elsewhere. PLoS Biology (Public Library of Science) is formalising a policy whereby manuscripts that confirm or extend a recently published study (“scooped” manuscripts, also referred to as complementary) are eligible for consideration at PLOS Biology.
- Undoing the publication bias: There is an excessive focus on the publication of groundbreaking results in prestigious journals. Science cannot only be groundbreaking, as there is a lot of important digging to do after new discoveries but there is not enough credit in the system for this work.
- We also need to publish “null” results — those that do not support the hypothesis — to prevent other researchers wasting time repeating work — the results of failed experiments. Researchers have to be rewarded for breadth of publications.
Considering the implications of the reproducibility crisis in healthcare, it is a matter of great importance that all those involved in this problem throw their weight behind these measures. We cannot afford to bury our heads in the sand and hope that some natural resolution will occur.