How Much of Science Is Reproducible?

Reproducibility is the most fundamental yardstick in science. If a result can’t be replicated, it doesn’t count as science.

Yet in recent years, there has been much talk of a ‘replication crisis’. Many results that we assumed were robust simply cannot be replicated. The term is typically used in the context of psychology and medicine, though it may apply to other fields as well.

So how much of science is reproducible? One way of tackling this question is to select a large number of studies from a particular field and then attempt to replicate them. This has been done several times.

A 2012 paper was only able to replicate 11% of 53 studies from pre-clinical cancer. A 2015 paper was only able to replicate 36% of 97 studies from psychology. A 2018 paper did slightly better, replicating 54% of 28 studies from that field. A 2016 paper was able to replicate 60% of 60% of 100 economics experiments. Another 2018 paper was able to replicate 62% of 21 social science experiments.

These numbers are sobering. But there’s an important caveat: the ‘studies to be replicated’ were selected somewhat arbitrarily, so the corresponding percentage can’t be taken as representative of the entire field.

Another way of tackling the question above is to simply ask researchers what percentage of the studies in your field can be replicated – a sort of ‘wisdom of the crowds’ approach.

This was done in a 2016 survey by the journal Nature. They got 1,500 responses – the vast majority from currently-working scientists. Respondents were asked, “In your opinion, what proportion of published work in your field is reproducible?”

The highest figure – 72% – was found in physics. The lowest figure – 52% – was found in “other” (which I suspect is mostly social scientists). Environmental science and medicine had intermediate figures – both 58%. Chemistry was a little higher at 65%. (Answers did not differ substantially between students and working scientists.)

These figures are again sobering. According to researchers themselves, close to half of published work in medicine, social science and environmental science cannot be replicated. Unsurprisingly, more ‘objective’ fields like physics and chemistry are perceived to have higher rates of replicability.

Overall, the two methods yield similar findings: a large percentage of results in more ‘subjective’ – dare one say ‘politicized’ – fields are not reproducible.