By combining automated text analysis with ‘Eve’ – a computer/robotic system that uses AI, researchers found that just a third of the results of a cohort of research papers in breast cancer cell biology were fully reproducible.

Ross King, Professor at Cambridge University and Wallenberg Guest Professor at Chalmers University of Technology, led the study which results was recently published in the journal Royal Society Interface. In a press release from Cambridge University, he says:

“The cancer literature is enormous, but no one ever does the same thing twice, making reproducibility a huge issue,” said King. “Given the vast sums of money spent on cancer research, and the sheer number of people affected by cancer worldwide, it’s an area where we urgently need to improve reproducibility.”

One third of the results found fully reproducible

In the study, an initial set of 12,000 published scientific papers were investigated through automated text mining techniques to find statements related to change in gene expression in response to drug treatment in breast cancer. From this screening, 74 papers were selected to undergo further analysis.

Two human teams, consisting of researchers from the UK, US, and Sweden, then used ‘Eve’ and two breast cancer cell lines to attempt to reproduce the results from the 74 selected papers.

Statistically significant evidence for repeatability was found for 43 papers, meaning that the results were replicable under identical conditions; and significant evidence for reproducibility or robustness was found in 22 papers, meaning the results were replicable by different scientists under similar conditions.

“There are lots of reasons why a particular result may not be reproducible in another lab,” explains King in the press release. “Cell lines can sometimes change their behaviour in different labs under different conditions, for instance. The most important difference we found was that it matters who does the experiment, because every person is different.”

The future of reproducing scientific results

King says that this work shows that automated and semi-automated techniques could be an important tool to help address the reproducibility crisis, and that reproducibility should become a standard part of the scientific process.

“It’s quite shocking how big of an issue reproducibility is in science, and it’s going to need a complete overhaul in the way that a lot of science is done,” said King. “We think that machines have a key role to play in helping to fix it.”

The full paper is available through the Journal of the Royal Society Interface:


Ross King

Wallenberg Chair in AI, Professor, Division of Systems and Synthetic Biology, Chalmers University of Technology

Published: April 27th, 2022

Latest news

We use cookies to personalise content and ads, to provide social media features and to analyse our traffic. We also share information about your use of our site with our social media, advertising and analytics partners. View more
Cookies settings
Privacy & Cookie policy
Privacy & Cookies policy
Cookie name Active
The WASP website uses cookies. Cookies are small text files that are stored on a visitor’s computer and can be used to follow the visitor’s actions on the website. There are two types of cookie:
  • permanent cookies, which remain on a visitor’s computer for a certain, pre-determined duration,
  • session cookies, which are stored temporarily in the computer memory during the period under which a visitor views the website. Session cookies disappear when the visitor closes the web browser.
Permanent cookies are used to store any personal settings that are used. If you do not want cookies to be used, you can switch them off in the security settings of the web browser. It is also possible to set the security of the web browser such that the computer asks you each time a website wants to store a cookie on your computer. The web browser can also delete previously stored cookies: the help function for the web browser contains more information about this. The Swedish Post and Telecom Authority is the supervisory authority in this field. It provides further information about cookies on its website,
Save settings
Cookies settings