We present  an exact probability distribution (meta-distribution) for p-values across ensembles of statistically identical phenomena , as well as the distribution of the minimum p-value among m independents tests . We derive the distribution for small samples 2<n \leq n^*\approx 30 as well as the limiting one as the sample size n becomes large. We also look at the properties of the "power" of a test through the distribution of its inverse for a given p-value and parametrization.  P-values are shown to be extremely skewed and volatile, regardless of the sample size n, and vary greatly across repetitions of exactly same protocols under identical stochastic copies of the phenomenon; such volatility makes the minimum p value diverge significantly from the "true" one. Setting the power is shown to offer little remedy unless sample size is increased markedly or the p-value is lowered by at least one order of magnitude . The formulas allow the investigation of the stability of the reproduction of results and "p-hacking" and other aspects of meta-analysis. From a probabilistic standpoint, neither a p-value of .05 nor a "power" at .9 appear to make the slightest sense .
We present the expected values from p-value hacking as a choice of the minimum p-value among m independents tests, which can be considerably lower than the "true" p-value, even with a single trial, owing to the extreme skewness of the meta-distribution. We first present an exact probability distribution (meta-distribution) for p-values across ensembles of statistically identical phenomena . We derive the distribution for small samples 2<n \leq n^*\approx 30 as well as the limiting one as the sample size n becomes large. We also look at the properties of the "power" of a test through the distribution of its inverse for a given p-value and parametrization. The formulas allow the investigation of the stability of the reproduction of results and "p-hacking" and other aspects of meta-analysis. P-values are shown to be extremely skewed and volatile, regardless of the sample size n, and vary greatly across repetitions of exactly same protocols under identical stochastic copies of the phenomenon; such volatility makes the minimum p value diverge significantly from the "true" one. Setting the power is shown to offer little remedy unless sample size is increased markedly or the p-value is lowered by at least one order of magnitude .
A
the expected values from p-value hacking as a choice of the minimum p-value among m independents tests, which can be considerably lower than the "true" p-value, even with a single trial, owing to the extreme skewness of the meta-distribution. We first present
D
, as well as the distribution of the minimum p-value among m independents tests
A
The formulas allow the investigation of the stability of the reproduction of results and "p-hacking" and other aspects of meta-analysis.
D
. The formulas allow the investigation of the stability of the reproduction of results and "p-hacking" and other aspects of meta-analysis. From a probabilistic standpoint, neither a p-value of .05 nor a "power" at .9 appear to make the slightest sense