Peer review

{{gold}}[[File:Peer review.png|thumb|right|Simple stuff, in theory.]]{{sciencenav}}
'''Peer review''' is the process of subjecting scholarly work to review by other experts in the field. The term "peer review" is typically used for scientific and academic publications. When an article is submitted, it is sent to the authors' "peers" (i.e., other experts in the same field) to assess the quality of the work. A similar approach is generally taken to evaluate research proposals submitted to agencies for funding, such as the National Science Foundation (U.S.) or Natural Sciences and Engineering Research Council (Canada), where the proposals are sent out to qualified scientists to assess whether the proposed projects merit funding.

==Initial and ongoing peer review==
The first and most notable step in peer review is when a piece of scientific work is looked at by reviewers (sometimes called "referees") for approval before [[Scientific publication|publication]]. Typically each paper is evaluated by three reviewers. The journal's editor then evaluates the reviews and decides whether the paper should be published or rejected. Acceptance usually is provisional on the authors making revisions to take the reviewers' comments into account. 

Approval by peer review is a necessary but ''not'' sufficient criterion for quality work. It mostly judges basic competence, such as minimizing the chances that [[pseudoscience]] can masquerade as [[science]], and filters out trivial or low-quality work that would not contribute anything of value. 

Following publication, peer review is an ongoing process where work is open to scrutiny by the scientific community — this is what publication allows a scientist to do. The process is designed to ensure that the work meets the standards of the field in question and science in general. A paper that survives initial peer review may be considered rubbish when examined more widely after publication.

===Selection of peer reviewers===
Being a peer reviewer is often a thankless task; one does not usually get paid for it, nor does one get much recognition except perhaps from the journal editor.<ref name=gasparyan>[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4366954/ Rewarding Peer Reviewers: Maintaining the Integrity of Science Communication] by Armen Yuri Gasparyan et al. (2015). ''J. Korean Med. Sci.'' Apr; 30(4):360–364.</ref> Nonetheless, scientists usually perform peer reviews when asked to do so. This is because it is considered a service to the profession:<ref name=gasparyan/> if I don't review my colleague's paper, who will review mine? Generally speaking, journal editors will seek out peer reviewers who:<ref name=gasparyan/><ref name=nature>[http://www.nature.com/authors/policies/peer_review.html Peer-review policy] ''Nature''</ref>
*are actively publishing in the field,
*have expertise relevant to the paper to be reviewed,
*and can give an independent review (i.e., no financial conflicts and not affiliated with the authors).
Peer reviews are sometimes [[double-blind]];<ref name=nature/> the reviewers do not know who the authors are, and the authors do not know who the reviewers are; only the journal editor knows. This helps to assure that the review is independent. The traditional method is single blind review, where the reviewer can see the author's identity, but not vice versa. A newly introduced style is open peer review, where the reviewers' identities are published along with the article.

Sometimes the level of expertise required to review the paper is so specific that the authors can guess who the reviewers are or vice versa. One method that can sometimes be used to address this issue is finding reviewers with expertise in different aspects of the paper (e.g., one for statistics and one for medicine).

==Relevance to the scientific community==
Peer review is a key part of the [[scientific method]], where the system's goal is to ensure that work is stripped of biases, unjustified assumptions, and other errors through the review by one's professional colleagues. Accordingly, '''peer''' in this context implies ''equals'': i.e., the reviewers or judges should have the same or reasonably similar qualifications that the work's [[author]] has — or ''claims'' to have. No ideology, other than the commitment to "rigorous empiricism… without which no man is a scientist," matters in asking whether a person is a scientific "peer".<ref name="kuhn">''The Structure of Scientific Revolutions'' by Thomas S. Kuhn (1996) University Of Chicago Press, 3<sup>rd</sup> ed. ISBN 0226458075.</ref>{{rp|42}}

[[Thomas Kuhn]], in the process of seeking to define the nature of science and the nature of "scientific [[progress]]", says in ''The Structure of Scientific Revolutions'' that the unique composition of the scientific community as composed of disinterested (read: unbiased) intellectual equals, is ''alone'' capable of generating scientific "progress."<ref name="kuhn"/>{{rp|168}} Therefore, Kuhn sees peer review as embodying "one of the strongest, if still unwritten, rules of scientific life… the prohibition of appeals to heads of state or to the populace at large in matters scientific."<ref name="kuhn"/> [[Politics]] has no place in science, and the goal of peer review is to strip politics out of the equation.

More prestigious [[Scientific journal|journals]] have a more thorough peer-review process and have tighter criteria for the level of work they can [[Scientific publication|publish]] — indeed, this is ''why'' they are prestigious, as only the most groundbreaking work is published in these journals. The two top journals in the world for science are ''Science'' and ''[[Nature (journal)|Nature]]'', and getting a paper in them is considered a great achievement.{{efn|An achievement worthy of 3 cases of champagne, a slap-up steak dinner at a fancy eatery and an all night drinking session at the pub later. All on departmental expenses, actually.}} The vetting process for these two journals is intense, with less than 10% of submitted manuscripts going forward for publication<ref>[http://www.nature.com/nature/authors/get_published/index.html Editorial criteria and processes] ''Nature''</ref> — that's 10% of the manuscripts scientists ''think'' are good enough. These top publications also tend to "coach" their submissions, improving the paper's readability, conciseness, and clarity (again, this is ''why'' they're considered the best).<ref>[http://www.phdcomics.com/comics/archive.php?comicid=1201 ''Science'' vs ''Nature'' Pt. 3] ''PhD Comics = Tales from the Road'' by Jorge Cham (2009).</ref> This isn't to belittle "lesser" journals at all; the publishers of ''Science,'' for example, openly acknowledge that articles may be rejected "not because the science isn't spectacular but because the area is no longer 'hot.'"<ref>[http://sciencecareers.sciencemag.org/career_development/previous_issues/articles/2450/academic_scientists_at_work_publishing_at_the_top_of_the_heap/ Publishing at the Top of the Heap] by Jeremy M. Boss & Susan H. Eckert (Jul. 11, 2003 , 8:00 AM) ''Science''.</ref> More typical journals in scientific fields ultimately accept perhaps 50-70% of submissions, though some revisions will almost always be required before publication.

==Possible problems==
Like all human activities, peer review can be subject to [[Cognitive bias|biases]] in certain situations or if insufficient care is taken in the selection of reviewers:
# In very competitive areas of sciences, reviewers may be tempted to gain an unfair advantage when they receive a rival's work for review long before publication and may even try to [[scoop]] the results. Good editors will be aware of the potential for such conflicts and consider them when making their final judgment on accepting a paper. 
# In the case of competing schools of thought, some reviewers might be influenced by their adherence to a certain point of view and give negative reviews to colleagues from an opposing school of thought. Again, the journal's editor should be able to judge this issue.
# If the field of experts for a given topic is small, there is a certain likelihood that the reviewer may have a relationship of animosity, rivalry, or friendship with the author that biases the review process.
#The peer-review process doesn't involve ''replicating'' experiments or studies to test their truth value. The reviews of submitted papers are only to detect glaring errors in methodology and to determine if the work is suitable for publication as it is presented. Therefore the process won't detect outright fraud immediately (unless it's blatantly obvious).
#In journals lacking real oversight from publishers, editorial board members may be able to publish their own articles by either bypassing the process or hand-picking their own peer reviewers. This created a small scandal in the mathematics community when ''Chaos, Solitons and Fractals'' was found to have carried hundreds of articles by its editor-in-chief, [[Mohamed El Naschie]].
#The editors of many journals can reject papers before they reach the peer review process.<ref>[https://books.google.com/books?id=lH3DVnkbTFsC&printsec=frontcover&dq=Editorial+peer+review:+its+strengths+and+weaknesses ''Editorial Peer Review: Its Strengths and Weaknesses''] by Ann C. Weller (2001). Information Today. ISBN 1573871001.</ref><ref>[http://xa.yimg.com/kq/groups/1097087/908376224/name/mahoney.pdf Publication Prejudices: An Experimental Study of Confirmatory Bias in the Peer Review System] by Michael J. Mahoney (1977) ''Cognitive Therapy and Research'' 1(2):161-175.</ref> While this can be good in that it avoids wasting reviewers' time on obvious junk, depending on the rules, this could lead to the unjustified dismissal of good science.
#In 2013 and 2014, the editors of the Journal of Vibration and Control and SAGE uncovered a peer-review ring involving assumed and fabricated identities, which led to the retraction of 60 published articles.<ref>[http://jvc.sagepub.com/content/20/10/1601.abstract Retraction notice] (July 8, 2014) ''Journal of Vibration and Control'' doi.org/10.1177/1077546314541924.</ref> As of January 2015, the number of fake-peer review articles exceeded 100.<ref>[http://retractionwatch.com/2015/01/02/fake-peer-review-fells-yet-another-paper/#more-24890 Fake peer review fells two more papers] by Ivan Oransky (January 2, 2015) ''Retraction Watch''.</ref> The individual record holder for retractions is currently "Yoshitaka Fujii, a Japanese anesthesiologist who fabricated his findings in at least 183 papers".<ref>[https://www.washingtonpost.com/opinions/more-science-than-you-think-is-retracted-even-more-should-be/2018/12/26/dc14fa98-0950-11e9-a3f0-71c95106d96a_story.html More science than you think is retracted. Even more should be] by Adam Marcus & Ivan Oransky (December 26 at 6:01 PM) ''The Washington Post''.</ref><ref>[http://www.anesth.or.jp/english/pdf/news20121019.pdf The Results of Investigation into Dr.Yoshitaka Fujii's papers] by Koji Sumikawa (2012) ''Anaesthesia''.</ref>

In all these cases, the responsibility to choose unbiased reviewers and to recognize a biased review rests on the shoulders of the journal's editors. Some journals allow the authors to suggest that certain colleagues not be used as reviewers. In the case of fraud prevention, the peer-review process does not end with the publication, as the article or paper remains available for all interested parties to view &mdash; if someone tries and fails to replicate the results, or finds that it contradicts research that they have done, then criticisms can be published and investigations made. Even in the face of these possible problems, the peer review process remains the most objective and qualified way to assess scientific work that has ever been developed.

==Peer review as a legal benchmark==
{{main|Evidence}}
Without a doubt, the twentieth century saw an uptick in the number of cases where specialized scientific knowledge is required to resolve a dispute: as a basic example, the plaintiff in a medical malpractice case must prove that a drug would have saved his father's life before he can prove the hospital's negligence in not administering the drug. Necessarily, if a party must prove a scientific argument to win his case, the court must have a way of judging the science.

American courts have met this challenge by adopting Federal Rule of Evidence 702 and the ''Daubert'' test, which require that "expert testimony" — a category that encompasses scientific testimony — be methodologically sound.<ref>See Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 590 (1993) (noting that the guarantee of "scientific knowledge" is one of "more than subjective belief" based in "the methods and procedures of science."</ref> In law student's shorthand, the Court in a ''Daubert'' inquiry asks, before considering the scientific testimony, if the scientific theory/method relied upon by a party (1) has been peer-reviewed, (2) has a low error rate, (3) is testable, and (4) is generally accepted by the scientific community. By adopting peer review as a benchmark for legal acceptance of scientific knowledge, the ''Daubert'' court validated the idea that science must be of sound, independently reviewed, unbiased methodology before it can be real science and agreed that the only cure for bad science is vigorous "cross examination."<ref>See Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 597 n.13 ("This is not to say that judicial interpretation, as opposed to adjudicative fact-finding, does not share basic characteristics of the scientific endeavor.").</ref>

The takeaway lesson is that temporal authorities, beyond the scientific community, recognize the necessity of unbiased peer review before science can be science.

==What peer review is not==
Many people think that the process of peer review is meant to settle the actual validity of the work and that in any paper that has passed peer review, the science is entirely correct. This is not the case. Peer review is an "entry-level" sort of test that weeds out the [[pseudoscience]] and obviously bad work but is not intended to be a catch-all for outright fraud or experimental error — reviewers simply challenge the rigor by which scientists are reporting their own work or challenge their conclusions if they haven't successfully eliminated competing hypotheses. Often enough, the demand for the right data and better conclusions made by reviewers is more than enough to ensure the work is valid enough, as the process is about making sure everything is submitted and out in the open with nothing hidden. Due to this, direct replication and validation aren't usually a priority or even a necessity for peer review. 

There are a few exceptions. For example, the American Chemistry Society won't accept computational chemistry papers unless the results have been verified. This is because taking parameters from a paper's supplemental material and running it on a computer for a few hours is ''practical''; rigorously replicating experiments that may have taken months to get right and require specialized and bespoke equipment is not.

Passing peer review and publication is indicative that (by the standards of the journal in question) the science is thorough, there are no glaring omissions, and the interpretation of the results presented is at least plausible, but this does not cement the science. Further publications and research can then use the data contained in the paper, and its conclusions can be amended (in worst-case scenarios, retracted) in later publications.<ref>"[S]cientists understand that peer review ''per se'' provides only a minimal assurance of quality, and that the public conception of peer review as a stamp of authentication is far from the truth." [http://www.nature.com/nature/peerreview/debate/nature05032.html Quality and Value: The True Purpose of Peer Review. What you can't measure, you can't manage: the need for quantitative indicators in peer review] by Charles Jennings (2006) ''Nature'' (doi:10.1038/nature05032); see also [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3404654/ Information Quality in Regulatory Decision Making: Peer Review versus Good Laboratory Practice] by Lynn S. McCarty et al. (2012) ''Environ. Health Perspect.'' Jul; 120(7): 927–934. doi:10.1289/ehp.1104277. ("It is difficult to extract from the extensive body of work and commentary published over the last 25-30 years that scientific journal peer review is a coherent, consistent, reliable, evaluative procedure… [T]he opposite conclusion may be more accurate.").</ref>

To make a legal analogy, if it is erroneously assumed that the peer review process is like a trial (the case is either proven true or dismissed), the actual process is more like an arraignment, only verifying that the case has enough merit to be heard. Indeed the "trial" part of scientific work is an ongoing and continuous process as other scientists cite the paper or attempt to replicate or use it in their own work.

It is also worth noting who those "peers" may be, as practitioners of ''pseudoscience'' might form a circle of ''[[Institute for Creation Research|pseudoscientists]]'' who start a ''[[Journal of 9/11 Studies|pseudoscientific journal]]''. It isn't the support of a claim that makes it true; it's the honest attempts to disprove a claim through experimentation that solidifies it.

==The problem with not peer reviewing==
Journals that do not have a peer-review process can give a veneer of respectability to otherwise bad science. Such publications can be picked up by the popular media and be a source for spreading irrationality, and even be responsible for causing hundreds of thousands of deaths. Such was the case with the journal ''Medical Hypotheses'' (published by Elsevier), which was not peer-reviewed before June 2010. The journal had published papers on [[HIV denial|AIDS denialism]]<ref>"Still Crazy After All These Years: The Challenge of AIDS Denialism for Science" by Nattrass, Nicoli (2009)''AIDS and Behavior'' 14(2):248–51. doi:10.1007/s10461-009-9641-z.</ref> and was the source of the [[Anti-vaccine_hysteria#Thiomersal|thiomersal anti-vaccine hysteria]], "the most damaging medical hoax of the last 100 years."<ref>"The Vaccine-autism Connection: A Public Health Crisis Caused by Unethical Medical Practices and Fraudulent Science" by D. K. Flaherty (2011). ''Ann. Pharmacother.'' 45(10):1302–4. doi:10.1345/aph.1Q318.</ref>

==Creationists and peer review==
===Allegations of bias in peer review===
[[Creationists]], whose "theories" have repeatedly been rejected by peer-reviewed journals,{{efn|See [[Intelligent design and academic freedom]].}} claim that the system is rigged against them since [[Theistic evolution|theistic explanations for natural phenomena]] are rejected out of hand because scientists assume [[methodological naturalism]] when working. The argument runs something like this: institutions of peer review assume that theistic explanations are invalid since a built-in requirement of naturalism constitutes atheistic bias.<ref>[https://web.archive.org/web/20090411220224/http://creation.com/creationism-science-and-peer-review Creationism, Science and Peer Review] by Andrew Kulikovsky (2 February 2008) ''Creation Ministries International'' (archived from April 11, 2009).</ref>

While ''prima facie'' is compelling and probably enough to dupe a few uninformed bystanders, this argument does not hold up to logic. The ''purpose'' of [[science]] (and rationalism) is to determine explanations of natural phenomena, with reference only to ''objective'' natural indicia, to create a body of knowledge that all of humanity can use and apply, if possible, in engineering and medicine. The purposes of science &mdash; to determine objective truth, free of a precondition of religious belief, and to create ''usable'' knowledge &mdash; are incompatible with theistic explanations.

Of course, naturalism has limits &mdash; if no objective explanation of a phenomenon exists, [[God of the gaps|perhaps]] some theistic agency is at work. Thus, naturalism does not displace theistic explanations; rather, it pushes back theistic explanations where they are incomplete and inaccurate. To argue that theism should be a ''start'' for science then completely discounts the ''nature'' of science: accusations of bias fall apart accordingly.

===Attempts by creationists to intelligently design peer review===
A common criticism of creationism and its intellectual kindred ([[baramin|baraminology]], [[intelligent design]], and [[creation science]]) is that these "sciences" can never pass peer review, as they fail to conform to scientific methodology. In addition to accusations of bias, disproven above, creationists have attempted to create their own "peer review publications" in a vain effort to create their own body of "scientific" knowledge and render neutral the "no peer review" criticism.

====First as tragedy: the creation science "mainstream"====
Of course, the "symptom" that creationism lacks peer review is indicative of the "condition" that it, and its brethren, lack real scientific merit: thus, by creating new "peer review" periodicals, "[[cdesign proponentsists]]" merely cure the symptom, not the underlying condition, i.e., the unscientific nature of their [[theology]]. In short, creationism does not gain merit by having an empty label ("Now Peer Reviewed!") attached to it.

However, several creationist groups have not gotten the memo: they continue to attempt to create sham peer review journals, each of which betrays the ultimate failing of their own discipline by failing to conform to the methodology of peer review and merely usurping the label. Several examples come to mind: [[Answers in Genesis]], [[CreationWiki]], [[JPANDS]], and more "mainstream" [[creation science|creation scientists]].

As one commentator has noted, creation "science" under its infantile peer review system approximates the level of depth of inquiry of 18<sup>th</sup>-century science.<ref>''God's Harvard: A Christian College on a Mission to Save America'' by Hanna Rosin (2007) Houghton Mifflin Harcourt. ISBN 0151012628.</ref> Since creation science cannot be tested without failing horribly ([[falsifiability|when it can even be tested]]), its intellectual discourse resembles, essentially, the creation of wild guesses, the discussion of said guesses around a fireplace, the assembly of these guesses in journals, the commitment to investigate them at some point, and then, the break for dinner. The process then repeats. The system entails no accountability because, ''by its nature, it cannot be accountable''.

====Then as farce: CreationWiki====
{{main|CreationWiki}}
[[CreationWiki]], "a free encyclopedia of apologetics that is being assembled by an international team of missionaries,"<ref>[https://web.archive.org/web/20110112133523/http://creationwiki.org/Main_Page CreationWiki] (archived from January 12 2011).</ref> has attempted to create a peer review system<ref>[http://creationwiki.org/CreationWiki:Peer_review Peer review] CreationWiki</ref> to vet CreationWiki articles, assuming that the result is something of "peer reviewed" quality. The premise is either deliberate deceit or unconscious misapplication of a scientific term of art; since science depends upon a group of "peers," those being individuals versed in high sciences, ''not [[apologetics]]'', nothing produced by CreationWiki could ever approximate peer review unless its membership base changed by nearly 100%.

{{Hatnote|'''Note:''' Parts of the following section refer to private conversations on matters of public import. If you are the quoted private author, please comment on the Talk page to request the removal of your words, but bear in mind that they are unattributed and public in scope.}}
	 
Evidence suggests that the conflation of "peer review" with "casual editing" by CreationWiki indicates a larger confusion, within creationist circles, about the meaning of "peer review." CreationWiki administrators, in particular, seem to believe that "peer review" is meant to be deliberately slanted: in their own words, "that is the goal of peer reviews in general — to uphold the consensus position."{{efn|Full quote:
{{cquote|Creationist disagree with each other on many issues. There are a variety of creationist views or opinions regarding almost every aspect of the cosmos. It is the goal of the CreationWiki to represent the main view held by creationists, and describe fringe views as such. Peer reviews are just what the phrase describes — reviews by peers. Atheists and creationists are not peers regarding theories formed from these worldviews. Only creationists can provide peer reviews of creationist views. Nevertheless, we allow for noncreationist reviews as well.|||CreationWiki Administrator<ref>discussion between AmesG ([[:User:User:AmesG]]) and CreationWiki Administrator, 10/20/2007.</ref>}}}} Apparently, creationists reconcile their own exclusion from academic journals not by conceding their own lack of merit but by asserting that peer review, in general, is meant to "censor" and responding in kind: in their own words, "peer review is done by professionals who hold to a POV and censor other views."<ref>CreationWiki Administrator, Discussion between AmesG and CreationWiki Administrator, 10/21/2007.</ref> In short, creationists see the denial of publication for lack of scientific merit as censorship for point of view (see [[balance fallacy]]): a rhetorical trick often used in creationist literature, as in ''[[Expelled: No Intelligence Allowed]]'', to make the cause out as a pariah.

====Latecomers: Answers in Genesis====
In 2008, [[Answers in Genesis]] attempted to found its own "academic" journal, ''[[Answers Research Journal]]''.<ref>[https://web-beta.archive.org/web/20080118021541/http://scienceblogs.com/pharyngula/2008/01/a_new_source_for_fake_science.php A new source for fake science] by PZ Myers (January 10, 2008 9:06 AM) ''Pharyngula'' (archived from January 18, 2008).</ref> Hilarity continues to accrue at a fairly rapid rate.

==What peer review looks like==
For an example of what ''real'' peer review looks like, see the peer review section of Clair, Carole, ''et al.''{{'}}s "Dose-dependent positive association between cigarette smoking, abdominal obesity and body fat", published in ''BioMed Central Public Health'' in 2011:<ref>[http://bmcpublichealth.biomedcentral.com/articles/10.1186/1471-2458-11-23/open-peer-review Dose-dependent positive association between cigarette smoking, abdominal obesity and body fat: cross-sectional data from a population-based survey] by Carole Clair et al. (2011). BMC Public Health 11:23. DOI: 10.1186/1471-2458-11-23.</ref>

[[File:Peer Review Clair 2011.png|center|400px]]

Notice that:
*The paper was reviewed 6 separate times over 11 months.
*The paper was rejected 5 separate times; the author revised it each time.
*The full peer review history ''and'' each of the 6 reviewer reports are publicly available
*The paper was reviewed by two qualified individuals from wholly unaffiliated universities<ref>[http://www.gla.ac.uk/schools/medicine/staff/catherinehankey/ Dr Catherine Hankey] School of Medicine, Dentistry and Nursing, University of Glasgow</ref><ref>[https://www.researchgate.net/profile/Christina_Bamia Christina Bamia] National and Kapodistrian University of Athens</ref>

==The humanities==
Fields in the humanities, such as [[history]], literary criticism, [[philosophy]], art history, cultural [[anthropology]], [[psychology]], [[women]]'s studies, [[African-American]] studies, classics, and poetics, differ from the hard [[science]]s regarding methodology and scholarly practice. In these cases, peer review exists to see if they hold to the [[academic]] and methodological standards of the specific disciplines, if they are internally consistent, if they engage with the established scholarship on the topic, if they account for long-existing problems, and if they make a meaningful contribution to the scholarship.

==See also==
*[[Bentham Science Publishers]] 
*[[Cohort study]]
*[[Vanity publishing]]
*[[Diederik Stapel‎]]
*[[Publication bias]]

==External links==
* [http://violentmetaphors.com/2013/08/25/how-to-read-and-understand-a-scientific-paper-2/ Jennifer Raff — How to read and understand a scientific paper: a guide for non-scientists]
*Pro: [https://retractionwatch.com/ Retraction Watch] — a searchable database of retracted published papers
*[[Charlatan|Con]]: [https://pdos.csail.mit.edu/archive/scigen/ SCIgen - An Automatic CS Paper Generator]

==Notes==
{{reflist|group=note}}

==References==
{{reflist|2|90%}}

[[Category:Research terms]]
[[category:Science]]
[[category:Philosophy of science]]

Categories: Research terms, Science, Philosophy of science