How can one researcher’s successful results be another researcher’s failure? Being able to replicate or reproduce another researcher’s experimental results is a key tenet of the scientific process. After all, if the study is valid, others should be able to duplicate the findings, right?
That assertion is drawing heated debate among members of the scientific community and even the public. Readers want to be able to believe what scientific journal insiders publish, but more studies are showing that replication is a big problem. Does that mean the original research is false or, worse, fraudulent?
According to a Nature survey of 1,576 researchers on scientific reproducibility, “More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments.” Do those findings mean readers can’t trust scientific research in general? Although the Nature data also showed that 52% of researchers believe there’s a “significant crisis of reproducibility,” less than 31% believe the lack of ability to reproduce published results indicates results are likely wrong, showing many still trust scientific literature.
Why is reproducing research so difficult?
The problem of reproducibility is not new. Aristotle’s experiments with mass and gravity, in which he dropped a rock and a feather and concluded that objects fall at speeds proportional to their masses held up until Galileo tried the experiment with a cannonball and a musket ball. They hit the ground at the same time, even though their weights were very different. Although Aristotle’s observation of the feather falling slower was correct, his conclusion was wrong.
A PBS article cites Aristotle’s example as an example of how scientific conclusions “can outrun the available evidence,” stating the reason so many scientific theories fail as fact is because no two experiments are identical. These inconsistent results don’t necessarily mean faulty science. They may be a result of noise in the system or variations as new measurements are taken.
What is scientific publishing’s role?
Scientific journal publishers and their peer review processes have long been the scientific gatekeepers for credibility. However, considering that retractions from scientific journals increased 10 times between 2001 and 2009, publishers need to be looking at their own practices. Although some reasons for the low reproducibility of scientific results can be assigned to sloppiness or researcher misconduct, some are due to the system.
For example, researchers looking to obtain grants or tenure may feel pressure to perform experiments that produce the kind of flashy or interesting results that are more likely to be published in prestigious journals. Researchers may subconsciously see only the results they want to see or tweak their experiments to show more significant findings.
Some journal publishers would rather publish significant results or new research rather than studies that simply attempt to replicate other experiments. Organizations that fund scientific research want to see the biggest bang for their bucks and tend to support the most exciting research and scientific breakthroughs. That often means that the results they publish are the most novel, unexpected, or eye-catching.
What’s the solution?
Although scientific community members are in general agreement that there is a problem, many are unsure how to solve it. Nature is one publication whose editors are taking steps to fight the problem of reproducibility. “Replication is something scientists should be thinking about before they write the paper,” said Ritu Dhand, Nature editorial director, in a BBC article interview. The publication introduced a reproducibility checklist for authors who submit their research, “designed to improve reliability and rigour.”
Edinburgh neuroscientist Malcolm Macleod is also taking a stance. He said, “Without knowing whether the published scientific literature is built on solid foundations or sand, we’re wasting both time and money.” He’s put forth “a new approach to animal studies that calls for independent, statistically rigorous confirmation of a paper’s central hypothesis before publication.”
Biotech company Amgen and biochemist Bruce Alberts have started an online journal to “lift the curtain on often hidden results in biomedicine: failed efforts to confirm other groups’ published papers.” The publication will let scientists share their replication methods and data so others won’t waste time researching false findings.
Academic publishers can help with such endeavors by ensuring published papers are fully transparent in their methodologies; ensuring a rigorous peer-review process; being more willing to publish replication studies; and, when others see problems in their published works, being quick with retractions.
|Contact your Sheridan representative or visit our contact page to ask how Sheridan experts can help streamline and simplify your publishing processes.|