We analyze what can be learned from tests for p-hacking based on distributions of t-statistics and p-values across multiple studies. We analytically characterize restrictions on these distributions that conform with the absence of p-hacking. This forms a testable null hypothesis and suggests statistical tests for p-hacking. We extend our results to p-hacking when there is also publication bias , and also consider what types of distributions arise under the alternative hypothesis that researchers engage in p-hacking. We show that the power of statistical tests for detecting p-hacking is low even if p-hacking is quite prevalent .
We analyze theoretically the problem of testing for p-hacking based on distributions of p-values across multiple studies. We provide general results for when such distributions have testable restrictions under the null of no p-hacking. We find novel additional testable restrictions for p-values based on t-tests. Analytical characterizations of the distributions of p-values under the null of no p-hacking and the alternative where there is p-hacking allow us to both analyze the power of existing tests and provide new more powerful statistical tests for p-hacking. Results are extended to practical situations where there is publication bias and when reported p-values are rounded. We also show that tests for p-hacking based on distributions of t-statistics can be problematic and may not control size. Our proposed tests are shown to have good properties in Monte Carlo studies and are applied to two datasets of p-values .
R
what can be learned from tests
theoretically the problem of testing
D
t-statistics and
R
analytically characterize restrictions on these distributions that conform with the absence of
provide general results for when such distributions have testable restrictions under the null of no
R
This forms a testable null hypothesis and suggests
We find novel additional testable restrictions for p-values based on t-tests. Analytical characterizations of the distributions of p-values under the null of no p-hacking and the alternative where there is p-hacking allow us to both analyze the power of existing tests and provide new more powerful
R
We extend our results to
Results are extended to practical situations where there is publication bias and when reported p-values are rounded. We also show that tests for
R
when there is also publication bias , and also consider what types of distributions arise under the alternative hypothesis that researchers engage in p-hacking. We show that the power of statistical tests for detecting p-hacking is low even if p-hacking is quite prevalent
based on distributions of t-statistics can be problematic and may not control size. Our proposed tests are shown to have good properties in Monte Carlo studies and are applied to two datasets of p-values