Reproducibility

{{Philosophyscience}}
'''Reproducibility''' is a key part of the [[scientific method]] and refers to the possibility of reproducing an [[experiment]]al result by a third party following the same procedures as the original experimenter.

Having the experiment repeated by an independent investigator reduces the impact of a single study's experimental error and increases the chances of the [[hypothesis]] being accepted by the scientific community. Of course, reproducibility alone is not enough when even random numbers can reproduce prior results.<ref>NJL Brown, DA MacDonald, MP Samanta, HL Friedman, and JC Coyne (2014). [http://www.pnas.org/content/111/35/12705.full?sid=f2a4c103-fb98-446d-a4c3-184541f979a7 A critical reanalysis of the relationship between genomics and well-being]. Proceedings of the National Academy of Sciences of the United States of America, 111(35):12705-12709.</ref> 

One of the signs of [[pseudoscience]] is a lack of reproducibility in their experiments. Scientific experiments can also lack reproducibility,<ref>[https://www.sciencenews.org/article/redoing-scientific-research-best-way-find-truth Is redoing scientific research the best way to find truth? During replication attempts, too many studies fail to pass muster]</ref> but this just indicates either that the original experiment was flawed, that a better understanding of the problem is needed, or it is something that for its nature happens very often.<ref>For example the claimed detections of {{wpl|Magnetic monopole#Searches for magnetic monopoles|magnetic monopoles}}</ref>

==Replication crisis==
The '''replication crisis''' means that many scientific experiments fail to [[Reproducibility|replicate]]. Experiment replication attempts are part of the philosophy of good science. However, in the contemporary academic community, there are powerful disincentives against them. Academic journals are generally not interested in publishing replication attempts, and publications are important for the career of scientists. In contrast, academic journals are interested in novel experiments with significant results.

A failure to replicate an experiment does not necessarily mean that there was anything wrong with the original experiment or that questionable research practices took place; the scientific effect found in the original experiment may be real and the failed replication may be due to chance, or due to differences in methodology. Obtaining the same or similar results using different methodology does, however, increase confidence in the results.

The replication crisis is most pronounced in the field of [[psychology]].<ref>[https://www.sciencenews.org/article/closed-thinking Closed Thinking: Without scientific competition and open debate, much psychology research goes nowhere] (''Science News'', May 16, 2013)</ref> A large-scale replication project was published in 2015 of the results of replication attempts of original published psychology experiments from 2008. Among social psychology experiments, only 14 out of 55 (25%) replicated; among cognitive psychology experiments, only 21/41 (50%) replicated.<ref>[https://www.sciencenews.org/article/psychology-results-evaporate-upon-further-review Psychology results evaporate upon further review: Surprising reports, findings with marginal statistical significance least likely to be reproduced, study concludes] by Bruce Bower (2:00pm, August 27, 2015) ''Science News''.</ref><ref>[http://www.sciencemag.org/content/349/6251/aac4716 Estimating the reproducibility of psychological science] by B. A. Nosek et al. 28 August 2015: Vol. 349 no. 6251 ''Science'' DOI: 10.1126/science.aac4716.</ref> However, the replication crisis has also emerged in the biomedical field. The drug company Bayer could only replicate 20% to 25% of 67 peer-reviewed studies in oncology, women's health, and cardiovascular diseases that were performed between 2007 and 2011.<ref>Prinz, F., Schlange, T., & Asadullah, K. (2011). [http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html Believe it or not: how much can we rely on published data on potential drug targets?] <i>Nature reviews Drug discovery, 10</i>(9), 712. DOI:10.1038/nrd3439-c1</ref> Another drug company, Amgen, could only replicate 6 out of 53 "landmark" studies in cancer research.<ref>Begley, C. G., & Ellis, L. M. (2012). [http://www.nature.com/nature/journal/v483/n7391/full/483531a.html Drug development: Raise standards for preclinical cancer research]. <i>Nature, 483</i>, 531–533. DOI:10.1038/483531a</ref>

Notably, though, the most famous "failed replication" study supposedly had its own replicability issues. A group of Harvard researchers, led by Daniel Gilbert, claimed that its failure to replicate a given study tended to coincide with divergent methodology from the original.<ref>[http://news.harvard.edu/gazette/story/2016/03/study-that-undercut-psych-research-got-it-wrong/ Study that undercut psych research got it wrong], Harvard Gazette</ref> The authors of the original 2015 study in ''Science'' responded to these criticisms by arguing that the critics' "...very optimistic assessment is limited by statistical misconceptions and by causal inferences from selectively interpreted, correlational data."<ref>[http://science.sciencemag.org/content/351/6277/1037.3 Response to Comment on “Estimating the reproducibility of psychological science”], Anderson et al., 2016, ''Science''</ref> Gilbert et al.'s conclusions have been criticized for other reasons as well, including fundamental statistical errors.<ref>[http://daniellakens.blogspot.com/2016/03/the-statistical-conclusions-in-gilbert.html The statistical conclusions in Gilbert et al (2016) are completely invalid], Daniel Lakens, March 8, 2016</ref>

[[Economics]] also faces a replicability crisis, with a 2015 study found that 49% (29/59) of economics research is replicable with the assistance of the authors or only 33% (22/67) of papers without the assistance of the authors (or in the authors' own words: "economics research is ''usually'' not replicable.")<ref>[https://www.federalreserve.gov/econresdata/feds/2015/files/2015083pap.pdf Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say "Usually Not"] by Andrew C. Chang & Phillip Li (2015). Finance and Economics Discussion Series 2015-083. Board of Governors of the Federal Reserve System. doi:10.17016/FEDS.2015.083.</ref>

==See also==
*[[Falsifiability]]
*[[Confounding variable]]
*[[Scientific validity]]

==External links==
*http://www.theatlantic.com/notes/2015/09/sweeping-psychologys-problems-under-the-rug/403726/
*http://www.spsp.org/blog/resolving-replication-crisis-social-psychology
*[https://www.explainxkcd.com/wiki/index.php/583 xkcd understood reproducibility]

==References==
{{reflist|2|80%}}

[[Category:Philosophy of science]]
[[Category:Science]]

Categories: Philosophy of science, Science