Articles about issues like this have been submitted several times to Hacker News, and I'm glad to see that this article is getting some discussion. Right after I saw the submission here, I looked up articles on the replication problem in psychology, and I came back to the PsychFileDrawer.org top 20 list of studies that are nominated as needing more replication.[1] I'd like to see more replication of several of those studies myself.

Another recent commentary that I don't recall seeing submitted to Hacker News is "Psychology's real replication problem: our Methods sections,"[2] which suggests (quite plausibly to me) that many publications in psychology journals describe the methods of the study so inadequately that it is hard to know whether or not the study can be replicated.

A scholar of how scientific research is conducted and of statistical errors that show up in many peer-reviewed scientific publications, Uri Simonsohn, has a whole website about "p-hacking" and how to detect it.[3] Simonsohn is a professor of psychology with a better than average understanding of statistics. He and his colleagues are concerned about making scientific papers more reliable. You can use the p-curve software on that site for your own investigations into p values found in published research. Many of the interesting issues brought up by comments on the article kindly submitted here become much more clear after reading Simonsohn's various articles[4] about p values and what they mean, and other aspects of interpreting published scientific research. And I think Hacker News readers who have thought deeply about statistics will be delighted by the sense of humor while making pointed remarks about experimental methods that you can find in the papers of Simonsohn and his colleagues.

Simonsohn provides an abstract (which links to a full, free download of a funny, thought-provoking paper)[5] with a "twenty-one word solution" to some of the practices most likely to make psychology research papers unreliable. He also has a paper posted on evaluating replication results[6] with more specific tips on that issue.

"Abstract: "When does a replication attempt fail? The most common standard is: when it obtains p > .05. I begin here by evaluating this standard in the context of three published replication attempts, involving investigations of the embodiment of morality, the endowment effect, and weather effects on life satisfaction, concluding the standard has unacceptable problems. I then describe similarly unacceptable problems associated with standards that rely on effect-size comparisons between original and replication results. Finally, I propose a new standard: Replication attempts fail when their results indicate that the effect, if it exists at all, is too small to have been detected by the original study. This new standard (1) circumvents the problems associated with existing standards, (2) arrives at intuitively compelling interpretations of existing replication results, and (3) suggests a simple sample size requirement for replication attempts: 2.5 times the original sample."

I should add that slamming the entire discipline of psychology as a discipline with sloppy methodology goes a bit too far. I have learned about most of the publications that take psychology most to task from working psychology researchers. There are whole departments of psychology[7] that largely have a scientific orientation and are trying to improve the discipline's methodology. Crap psychology abounds, but it is gradually being displaced by science-based psychology based on sound methodologies. It is of course more methodologically difficult to study the behavior of our fellow human beings than to study clouds or volcanoes or insects, but many scientifically oriented psychologists are working on the problem with good methods and sound statistical analysis. Some thoughtful psychologists have been prompted to stress careful replication by the failed studies that have come before.[8]

[1] http://www.psychfiledrawer.org/top-20/

[2] http://psychsciencenotes.blogspot.com/2014/05/psychologys-re...

[3] http://www.p-curve.com/

[4] http://opim.wharton.upenn.edu/~uws/

[5] http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2160588

[6] http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2259879

[7] http://www.psych.umn.edu/research/areas/pib/

[8] http://www.psychologicalscience.org/index.php/publications/o...