How can we increase reproducibility in neuroscience studies?
A new study says sample sizes of 30 people in fMRI experiments – the median number for these kinds of studies – are equally likely to challenge results as confirm them.
A team of scientists from the University of Illinois at Urbana-Champaign (IL, USA) has discovered that attempts to repeat task-based fMRI experiments with a sample size of approximately 30 people are equally likely to challenge the results as they are to confirm them. They also found that the same study type with sample sizes of less than 100 cannot be replicated either.
“This study is the largest investigation of the role of sample size in the reproducibility of task-based fMRI methods,” commented Aron Barbey, co-author and psychology professor at the University of Illinois.
Task-based fMRI experiments allow researchers to determine which parts of the brain are necessary to perform certain tasks, by tracking changes in blood oxygen levels. There are no set standards for the use of fMRI in neuroscience research. The team has found that this has caused a widespread problem, as the median number of participants in fMRI experiments across neuroscience journals in 2011–2014 was 30.
“We systematically examined the reproducibility of studies at different sample sizes, from as few as 16 to as many as 121 participants, and across many different task-based fMRI approaches,” Barbey explained.
“We found that reproducibility is modest in studies involving the standard number of 30 participants, and that much larger sample sizes may be needed to improve replicability.”
A key factor underpinning sample sizes used is the high expense of fMRI studies, as Erick Paul, co-author and former postdoctoral research at the University of Illinois, says: “even small studies can cost several tens of thousands of dollars, and funding systems are not generally set up to enable the routine collection of data from more than 100 participants.”
The team suggests that in order to overcome this issue, researchers could diversify their samples by pooling resources and conducting their fMRI studies over numerous sites. This may also garner a greater impact of their work.
“Replicability is the foundation of scientific progress,” the authors wrote. “Unfortunately, for a variety of reasons, many scientific fields are gripped by a crisis of irreproducibility. While some of the causes of the crisis are deeply woven into the academic landscape – incentives related to publication, funding and tenure – the most straightforward solution relates to statistical power.”
“Our study points to a major problem in the field of cognitive neuroscience, a field in which fMRI studies are a commonly used tool,” Barbey concluded. “But this is a problem that we can solve by increasing the size of the samples we study.”
