Reproducibility is key
If reproducibility is essential to scientific investigation, why are so few researchers performing replication studies? The Science Exchange knows why and is working to change the status quo.
When children first learn about the scientific method, they master forming hypotheses, testing them, reporting results, and confirming their classmates’ results. These exercises represent how scientists do good science.
A major component of the scientific method is confirming experimental results. Ideally, independent researchers replicate studies to validate the original authors’ findings. Validation gives the first results extra weight, which brings science closer to explaining “truths” about the natural world. However, this step is often overlooked, even though most scientists believe it’s essential. So, what changed between those early days in the classroom and today at the lab bench? And what can scientists do to boost reproducibility?
“When I was an academic scientist, we were told not to do a study if someone else has already done it,” said Nicole Perfito, Director of Sourcing at the Science Exchange – a startup based in California and home to the largest validation initiative, the Reproducibility Project . “The incentive structure for scientists is to tell a clean story and focus on exploratory research because that’s what gets their papers into high-profile journals, [and leads to] tenure or promotions.”
The majority of the time, replication studies receive little to no funding. One reason is the public demand for innovation and discovery. The public expects high value results because “[they] are the ultimate recipients reaping the benefits” (1). This demand is also inflated by media coverage of scientific studies.
Huge government and public funding institutions such as the National Science Foundation and the National Institutes of Health realize this demand, so they too are pressured into funding the most novel and attractive studies. This situation causes increased competition and sometimes leads to issues such as fraud (uncommon) and subconscious biases (common). For example, p-hacking, a form of data mining to find statistically significant correlations to make a hypothesis “appear” correct, may seem like an act of fraud. Although it does happen occasionally, p-hacking frequently stems from confirmation bias. “Blinding” is one way of reducing or eliminating instances of bias (1), and replication studies will reveal biases that slip through the cracks.
Perfito said the Science Exchange researchers and partner academic institutions understand how prevalent bias is, so they do as much as possible to reduce the stress from lack of funding. The company received support from the Laura and John Arnold Foundation and the Gates Foundation—two major private funders that support many reproducibility initiatives—which helped make the Reproducibility Project a huge success in confirming major cancer and HIV-related research.
Some scientists believe a lack of funding, the idea of “publish or perish,” and biases have caused the current “reproducibility crisis.” According to a survey conducted by Nature in 2016, over 70 percent of 1500 researchers had tried and failed to reproduce other scientists’ experiments. Fifty percent even failed to replicate their own work, yet attempts to publish those results were minimal (2).
“One big issue with reproducibility is the lack of information from the original authors,” said Elizabeth Iorns, CEO of the Science Exchange. “There is a lot of variability in the performance of assays. As we understood that, we knew we could better interpret the robustness of original results if we got nearly all the raw data and materials. We also shared our replications so others could try reproducing the results.”
With the funders’ support and using the Center for Open Science’s (COS) Open Science Framework, Science Exchange made all of its data, protocols, and materials accessible to everyone. “New investigators often shy away from replication studies because of the current incentive structure. But I know younger scientists are much more interested in the open access philosophy, which is what we also promote,” explained Perfito.
Open access research and widely available data provide more transparency and may make it cheaper for young scientists to conduct replication experiments. COS even developed the Transparency and Openness Promotion (TOP) Guidelines that journals and academic institutions can adopt to encourage validation and scientific investigation (3). TOP has a total of eight transparency standards, which include: citation, data transparency, analytic methods (code) transparency, research materials transparency, design and analysis transparency, study preregistration, analysis plan preregistration, and replication.
Iorns said that with more resources and organizations trying to combat the reproducibility crisis, they can change the status quo: “There’s tremendous interest in reproducibility, and the scientific community often discusses its importance, as well as its difficulties. By having these discussions, funding more replication research, or even conducting open-access science, we’re taking a step in the right direction.”
