Robot Scientist ’Eve’ Illuminates Reproducibility in Breast Cancer Research

By combining automated text analysis with ‘Eve’ – a computer/robotic system that uses AI, researchers found that just a third of the results of a cohort of research papers in breast cancer cell biology were fully reproducible.

Ross King, Professor at Cambridge University and Wallenberg Guest Professor at Chalmers University of Technology, led the study which results was recently published in the journal Royal Society Interface. In a press release from Cambridge University, he says:

“The cancer literature is enormous, but no one ever does the same thing twice, making reproducibility a huge issue,” said King. “Given the vast sums of money spent on cancer research, and the sheer number of people affected by cancer worldwide, it’s an area where we urgently need to improve reproducibility.”

One third of the results found fully reproducible

In the study, an initial set of 12,000 published scientific papers were investigated through automated text mining techniques to find statements related to change in gene expression in response to drug treatment in breast cancer. From this screening, 74 papers were selected to undergo further analysis.

Two human teams, consisting of researchers from the UK, US, and Sweden, then used ‘Eve’ and two breast cancer cell lines to attempt to reproduce the results from the 74 selected papers.

Statistically significant evidence for repeatability was found for 43 papers, meaning that the results were replicable under identical conditions; and significant evidence for reproducibility or robustness was found in 22 papers, meaning the results were replicable by different scientists under similar conditions.

“There are lots of reasons why a particular result may not be reproducible in another lab,” explains King in the press release. “Cell lines can sometimes change their behaviour in different labs under different conditions, for instance. The most important difference we found was that it matters who does the experiment, because every person is different.”

The future of reproducing scientific results

King says that this work shows that automated and semi-automated techniques could be an important tool to help address the reproducibility crisis, and that reproducibility should become a standard part of the scientific process.

“It’s quite shocking how big of an issue reproducibility is in science, and it’s going to need a complete overhaul in the way that a lot of science is done,” said King. “We think that machines have a key role to play in helping to fix it.”

The full paper is available through the Journal of the Royal Society Interface: https://royalsocietypublishing.org/doi/10.1098/rsif.2021.0821