BONUS - Advances in Methods and Practices in Psychological Science May 2018

Wilson and Wixted (a contributor to my very own GE reading list!) recently published an interesting analysis of the odds of a tested effect being true in experimental psychology. That is, we understand power and type 1 error rate to be primary contributors to whether we observe a significant effect, but an overlooked aspect is the base rate for that effect being real. I think we (students especially) tend to sort of assume that this rate sort of washes out across studies, and so we focus on the things that have more obvious prescriptions - like seeking a higher n for more power to detect whatever the effect is that we seek.
But it probably isn't really true that that rate of a tested effect being found to be real (R) is equivalent. Using a corpus of data from the Open Science Foundation's large-scale replication attempts, Wilson and Wixted attempt to estimate those R values. For illustration purposes, they consider the widely reported finding that cognitive and social psychologies have different rates of replication. Are social psychology studies replicated less frequently because they take less care with experimental design, use underpowered analyses, and p hack the crap out of their data? Probably :p. But another consideration is that the base rate of the effect being real is probably much lower. In any discipline of psychology, experiments that are based on private hunches, rather than a grounded base of established knowledge or shared experience, will suffer from low R. To be clear, low R is not necessarily bad science, as there will simply be hard to detect, unexpected effects that are nevertheless incredibly important. The real takeaway should be that the way that you came up with an idea for an experiment informs how you should interpret its data. To parrot the authors' example, if you like balloons, put balloons outside of kids' classrooms, and collect data that suggest kids perform better in school when there are balloons outside their classroom, you might still keep in mind that there are reasons to pump the brakes on your new balloon-based research program. The fact that you conducted the experiment because you liked balloons, and that your results are tremendously surprising, both inform the kind of inference that you can make from the experiment, regardless of power and Type 1 error considerations.
Finally, their estimates - in the Open Science Collaboration dataset, R in cognitive psychology was .27 and R in social psychology was .06. In cognitive psychology, one real effect is tested for every four noneffects that are tested. **In social psychology, one real effect is tested for every 16 noneffects that are tested.** Disciplines have to set their own standard for what is an acceptable R (at 0 - everything is exciting! ESP is real!, at 1 - every effect is guaranteed to replicate because it is exactly what we already knew to be true). Social psychology's, uh, might be a little low.
There was other interesting stuff in the paper, including asking independent raters to rate how surprising / exciting different kinds of findings were, more comparisons of power and R across psychologies, how journals have responded to this kind of reasoning, etc. But this impromptu write-up is long enough as-is :)