Evolution of Reporting<i>P</i>Values in the Biomedical Literature, 1990-2015
Authors: David Chavalarias, Joshua David Wallach, Alvin Ho Ting Li, John P. A. Ioannidis
Journal: JAMA (2016)

<h3>Importance</h3> The use and misuse of<i>P</i>values has generated extensive debates. <h3>Objective</h3> To evaluate in large scale the<i>P</i>values reported in the abstracts and full text of biomedical research articles over the past 25 years and determine how frequently statistical information is presented in ways other than<i>P</i>values. <h3>Design</h3> Automated text-mining analysis was performed to extract data on<i>P</i>values reported in 12 821 790 MEDLINE abstracts and in 843 884 abstracts and full-text articles in PubMed Central (PMC) from 1990 to 2015. Reporting of<i>P</i>values in 151 English-language core clinical journals and specific article types as classified by PubMed also was evaluated. A random sample of 1000 MEDLINE abstracts was manually assessed for reporting of<i>P</i>values and other types of statistical information; of those abstracts reporting empirical data, 100 articles were also assessed in full text. <h3>Main Outcomes and Measures</h3> <i>P</i>values reported. <h3>Results</h3> Text mining identified 4 572 043<i>P</i>values in 1 608 736 MEDLINE abstracts and 3 438 299<i>P</i>values in 385 393 PMC full-text articles. Reporting of<i>P</i>values in abstracts increased from 7.3% in 1990 to 15.6% in 2014. In 2014,<i>P</i>values were reported in 33.0% of abstracts from the 151 core clinical journals (n = 29 725 abstracts), 35.7% of meta-analyses (n = 5620), 38.9% of clinical trials (n = 4624), 54.8% of randomized controlled trials (n = 13 544), and 2.4% of reviews (n = 71 529). The distribution of reported<i>P</i>values in abstracts and in full text showed strong clustering at<i>P</i>values of .05 and of .001 or smaller. Over time, the “best” (most statistically significant) reported<i>P</i>values were modestly smaller and the “worst” (least statistically significant) reported<i>P</i>values became modestly less significant. Among the MEDLINE abstracts and PMC full-text articles with<i>P</i>values, 96% reported at least 1<i>P</i>value of .05 or lower, with the proportion remaining steady over time in PMC full-text articles. In 1000 abstracts that were manually reviewed, 796 were from articles reporting empirical data;<i>P</i>values were reported in 15.7% (125/796 [95% CI, 13.2%-18.4%]) of abstracts, confidence intervals in 2.3% (18/796 [95% CI, 1.3%-3.6%]), Bayes factors in 0% (0/796 [95% CI, 0%-0.5%]), effect sizes in 13.9% (111/796 [95% CI, 11.6%-16.5%]), other information that could lead to estimation of<i>P</i>values in 12.4% (99/796 [95% CI, 10.2%-14.9%]), and qualitative statements about significance in 18.1% (181/1000 [95% CI, 15.8%-20.6%]); only 1.8% (14/796 [95% CI, 1.0%-2.9%]) of abstracts reported at least 1 effect size and at least 1 confidence interval. Among 99 manually extracted full-text articles with data, 55 reported<i>P</i>values, 4 presented confidence intervals for all reported effect sizes, none used Bayesian methods, 1 used false-discovery rates, 3 used sample size/power calculations, and 5 specified the primary outcome. <h3>Conclusions and Relevance</h3> In this analysis of<i>P</i>values reported in MEDLINE abstracts and in PMC articles from 1990-2015, more MEDLINE abstracts and articles reported<i>P</i>values over time, almost all abstracts and articles with<i>P</i>values reported statistically significant results, and, in a subgroup analysis, few articles included confidence intervals, Bayes factors, or effect sizes. Rather than reporting isolated<i>P</i>values, articles should include effect sizes and uncertainty metrics.