We analyze theoretically the problem of testing for p-hacking based on distributions of p-values across multiple studies. We provide general results for when such distributions have testable restrictions  under the null of no p-hacking. We find novel additional testable restrictions for p-values based on t-tests. Analytical characterizations of the distributions of p-values under the null of no p-hacking and the alternative where there is p-hacking allow us to both analyze the power of existing tests and provide new more powerful statistical tests for p-hacking . Results are extended to practical situations where there is publication bias and when reported p-values are rounded. We also show that tests for p-hacking based on distributions of t-statistics can be problematic and may not control size. Our proposed tests are shown to have good properties in Monte Carlo studies and are applied to two datasets of p-values .
We theoretically analyze the problem of testing for p-hacking based on distributions of p-values across multiple studies. We provide general results for when such distributions have testable restrictions (are non-increasing) under the null of no p-hacking. We find novel additional testable restrictions for p-values based on t-tests. Specifically, the shape of the power functions results in both complete monotonicity as well as bounds on the distribution of p-values . These testable restrictions result in more powerful tests for the null hypothesis of no p-hacking . A reanalysis of two prominent datasets shows the usefulness of our new tests .
R
analyze theoretically
theoretically analyze
A
(are non-increasing)
R
Analytical characterizations of the distributions of
Specifically, the shape of the power functions results in both complete monotonicity as well as bounds on the distribution of
R
under the null of no
. These testable restrictions result in more powerful tests for the null hypothesis of no
D
and the alternative where there is p-hacking allow us to both analyze the power of existing tests and provide new more powerful statistical tests for p-hacking
R
Results are extended to practical situations where there is publication bias and when reported p-values are rounded. We also show that tests for p-hacking based on distributions of t-statistics can be problematic and may not control size. Our proposed tests are shown to have good properties in Monte Carlo studies and are applied to two datasets of p-values
A reanalysis of two prominent datasets shows the usefulness of our new tests