The rise of metascience

The reproducibility crisis may have caused a tsunami of doubt across peers, politicians and the public, but has also made way for the growth of metascience.

Metascience is defined as the “inquiry into the methodology and philosophical implications of scientific investigation,” and is referred to as the scientific study of science itself. It is believed to have originated in the 1930s, but has garnered some momentum over the course of recent years due to the emergence of the reproducibility crisis.

Reproducibility crisis

The reproducibility crisis has emerged as the volume of scientific studies that are difficult, or even impossible, to reproduce has escalated to a point where it constitutes being termed as a crisis. Many have asked the question of where the fault lies; is this in the hands of the scientist or the hands of the publisher?

There is the issue of publication bias, where journals are more likely to publish research that has produced desirable results. Anotherconcern is funding – funders want to invest in research that has promise and when results don’t live up to the expectations they promised, are unlikely to invest further.

However, it has been determined that fundamentally the blame must be placed onto scientists themselves. Science is relied upon to epitomize truth and integrity, and in many cases this is clearly failing. Whilst not all research is irreproducible due to hand-picked results, some methodologies are just incoherent and impossible to follow.

As an example, it is thought that as high as 85% of biomedical research may be irreproducible. This, of course, has implications in that the research itself cannot be relied upon and so is wasted, but if it did move forward, what would be the consequences for potential patients?

These statistics are produced by the field of metascience and it appears to be more important than ever, as research must undergo more scrutiny in order to regain trust in science and enable funding bodies to direct resources into research that is producing real results.

Metaresearchers

There are now a few metascience research groups, with one of the leading groups of metaresearchers based at Tilburg University (The Netherlands). The group was born after Diederik Stapel, the then dean of the university’s School of Social and Behavioral Sciences, confessed to committing academic fraud.

Stapel was investigated internally by Marcel van Assen, a fellow psychologist, who spent months looking into his data, methods and results. It was discovered that Stapel had committed intentional fraud in at least 55 of his papers, as well as 10 PhD papers written by students.

Jelte Wicherts, a researcher with the same passions as van Assen, then joined the same department. They went on to create their metascience research group with the motivation that they wanted to make science better and enable scientists to conduct more robust research.

“We’ve seen things we felt were not OK,” comments Wicherts. “The first way to deal with it, that’s our conviction, is to study these things.”

A major focus of van Assen and Wicherts’ work is on selective data reporting and molding of data into more desirable results – both key elements in the reproducibility crisis. These are often a result of confirmation bias, in which scientists search out or interpret data in a way that confirms their own hypotheses.

The research group hassince created different tools for checking scientific publications, including an algorithm named Statcheck, which checks for potential statistical errors on psychology studies. The team ran the algorithm on thousands of papers, publishing the results on PubPeer.

Whilst some researchers felt unfairly attacked, van Assen and Wicherts stand behind their research and believe that it is important to have statistical honesty across scientific studies, with the hope that the problems behind the reproducibility crisis are fixable. The pre-registration of studies and making data available immediately in order for others to check findings are both increasingly popular practices.

The controversy

We cannot claim that all researchers are committing fraud to same extent Stapel did; whilst this is not an isolated case, it is rare. However, inflated results and impossible-to-follow methods are becoming increasingly prominent issues.

Something many can relate to (over 70% of 1500 researchers in a survey conducted by Nature in 2016) is the frustration when failing to successfully replicate a methodology detailed by other researchers.

Scientists appear to be divided by the growing strength of metascience. Whilst many view it as an opportunity to prove the integrity of research, others – whose experiments may be caught in the cross-fire – see it as an unjustified, personal attack. Whether metascience will be the answer to the reproducibility crisis remains to be seen.