Same experiment but different results; now what?

CU-led task force calls on scientists across the nation to improve practices

Scientists are having trouble reproducing each other’s published findings. This growing problem has received national attention and is concerning policymakers, the public and scientists.

CU-Boulder biologist Mark Winey, a professor and chair of the Molecular, Cellular and Developmental Biology Department, wants to solve this problem.

He leads the Data Reproducibility Task Force, a group established by the American Society for Cell Biology (ASCB). This task force unites other prominent life scientists from around the nation in an effort to find solutions.

"The public wants to know that its tax dollars that are being spent on research are being used responsibly, and being used in a way that’s going to lead to cures and new technologies.“

Winey and the task force identified some major contributors to the reproducibility problem, including a competitive ‘publish or perish’ mentality, a lack of common procedural standards and communication issues among scientists. They recently published their findings and suggestions for improvement in a white paper.

“The public wants to know that its tax dollars that are being spent on research are being used responsibly, and being used in a way that’s going to lead to cures and new technologies,” Winey says, explaining the significance of the problem.

“The ability to replicate scientific findings is very important to the scientific method and getting to those cures and new technologies.”

A major goal of the task force was “to assemble a group of scientists to talk about the problem and to try to identify what the issue might really be.” Then, once they had an idea, to ask: “What would a bunch of scientists want to do about it? How can scientists address the problem and improve the current quality science?”

To answer these questions, the task force surveyed hundreds of members of the ASCB, mostly working scientists. Most of these scientists indicated that the fiercely competitive scientific culture is partially to blame.

With funding for biomedical research at a historic low, the pressure to publish one’s findings is immense. Jobs, salaries, and promotions rely almost exclusively on a scientist’s publication record.

Plus, more weight is given to scientists who publish in well-known journals. According to the survey, the demand to publish may lead to “hyping” results to create a high-profile story.

Another problem identified by the survey is lack of communication among scientists. In particular, researchers need to talk about creating standard ways to perform procedures.

The idea is simple: If researchers follow the same procedures, the likelihood that they will be able to reproduce one another’s work increases. In current practice, however, most scientists do experiments in slightly different ways.

Publishing practices also muddle communication. The sections of scientific research articles describing experimental methods formerly contained all the information other scientists would need to carry out an experiment in precisely the same way.

Today, “methods sections” are much shorter and often lack the details necessary to retest an experiment.

But the news isn’t all bad. Despite several well-publicized examples of fraud, lack of scientific reproducibility has little to do with deception, according to survey results. Scientists also report being able to solve their own problems.

“One piece of good news is that a lot instances of reproducibility problems are resolved, and they’re resolved amicably between the two labs, so we were really pleased with that,” says Winey.

The task force published findings from this survey in their white paper, along with 13 recommendations for how to improve scientific practices.

Among these recommendations, the paper highlights new initiatives, such as web-based forums, which encourage scientists to communicate with one another. It also emphasizes better publishing practices—including more detailed methods sections—and calls for scientists to share protocols and agree upon standard ways to perform procedures.

Finally, it aims to empower scientists to change the current, competitive culture, and recommends more focused education on responsible research and fair interpretation of data.

Winey and the task force met their initial goal of identifying some of the issues behind the reproducibility problem. According to task force member and UC Berkeley biologist Doug Koshland, it’s too early to tell if the task force has succeeded in improving reproducibility, though there’s a plan in place to track the initiatives proposed in the white paper.

“The ASCB and the [ASCB’s] public policy committee are working to follow up on the recommendations,” says Koshland.

Aggie Mika and Roni Dengler are writers for CU’s Science Buffs blog. Roni is a PhD. candidate in MCD Biology, and Aggie is fourth year graduate student in Integrative Physiology.