Reproduce or perish

The reproducibility crisis imposes new challenges for the scientific community, but what are the obligations of journals in solving the problem?

Decades of challenging research during the 1900s produced a cornucopia of scientific discoveries that revolutionized the way we diagnose and treat disease. While the 2000s have proven fruitful for discovery so far, they also exposed another challenge: replicating the arduous and technologically advanced experiments performed by others. The speed at which research is generated, along with a lack of published validation studies, leaves some worried that researchers are throwing time, money, and effort at false leads. While researchers, institutions and funding agencies are tackling the crisis from different angles, another important player in science often goes unnoticed: publishing.

“The replicability of papers that we publish is a very important feature of the quality and impact of our journal,” said John Krystal, editor of the journal Biological Psychiatry.

Scientific journals remain the major route for disseminating findings to the field at large. One concern regarding publishing replication studies for both researchers and publishers is the impact the paper will have on the field. Both researchers and publishers want to gain recognition for the work published. However, Krystal noted, “There are examples where the first replications of preliminary findings are extremely important.” He recalled one of his early seminal discoveries in 2000 that ketamine acts a rapid-acting antidepressant. “It had limited impact until the first replication by another group was published in 2006. I believe that the replication paper is cited more than the original finding!”

Furthermore, replication is inherent to particular fields. “In some areas, particularly genetics, we generally insist on replication samples,” said Krystal. However, when it isn’t the norm for a particular field to replicate aspects of previous work, journals can reach for other tools in their kit to evaluate the quality of submitted research.

Journals utilize peer review, which remains a critical institution in publishing. Journal editors rely on the expertise of a network of scientists to assess the content of submitted research. “We strive for rigorous statistics,” said Krystal. “We have a group of statisticians that routinely provide input on papers.”

While field-specific replication norms and peer review has helped to maintain scientific integrity during a time of rapid expansion and discovery, journals are now searching for novel ways to help scientists take on the reproducibility crisis. Nathan Blow, Editor in Chief of the journal BioTechniques, attributes part of the crisis to a lack of transparency in the experimental methods of papers. “The obligation that I think journals have is to enhance that transparency as much as possible,” said Blow. “Give researchers the space that they need when reporting on their methods and materials.” This would allow authors to expand on the resources used, including antibodies, reagents, kits, and equipment.

Providing researchers with more information on the specifics of a study will facilitate attempts to replicate the findings in independent labs. This inspired the creation of the Resource Identification Initiative, an online portal that allows journals and authors to incorporate Research Resource Identifiers (RRIDs) in papers to enhance resource authentication (e.g., antibodies, cell lines) and transparency.

RRIDs are just one way journals can catalyze replication studies. “What I think you’ll see coming up is a lot more people looking to include protocols, videos, and demonstrations within their supplementary materials,” said Blow.

All entities across the science pipeline play a role in ameliorating the replication crisis. A failure to diagnose and treat science’s own affliction could tarnish the public’s view of the field. “The reproducibility problem reduces the public confidence in science,” said Krystal. “As a member of society, I want to know that we can rely on the conclusions of biomedical research that influence medical care and that the government is making good investments in biomedical research through NIH and NSF.”

In order to ameliorate any skepticism in science that the reproducibility crisis may foster in the public eye, scientists, research agencies, and publishers must continue working towards improving the way scientific findings are reported. “I think science will come out at the end looking even better to the public. It will be stronger than what it was going in,” said Blow.