AB1222 HOW DOES CHANGING THE TRADITIONAL P VALUE SIGNIFICANCE THRESHOLD TO .005 EFFECT OBSERVATIONAL STUDIES?
Authors: M. Oztas, Z. Toker Dincer, H. Yazici
Journal: Annals of the Rheumatic Diseases (2020)

Background: Changing the traditional p value significance level from 0.05 to 0.005 has been proposed as one remedy to the common ill understanding and use of the p values (1-2) especially leading to poor reproducibility. It was then reported that this change would decrease the number of statistically significant primary end points in randomized controlled trials (RCT) reported during 2017 in 3 main general medical journals by 1/3 (3). Objectives: We reasoned that the same change would affect in the right direction the significance conclusions more in observational studies (OS) in that OS generally have retrospective or cross-sectional designs, looser study inclusion criteria, many potential confounders and post – hoc analyses. Methods: We surveyed how the 0.05 to 0.005 change would affect the OS in the same 3 journals in reference 3 for the same year (2017) and in another year (2002), included to better assess the validity of our findings. RCT were also surveyed for both time periods. Two authors (MO, ZTD) surveyed all studies. Disagreements were settled by HY. Differing from the previous study (3) we analyzed the p values as related to the primary end points only in the abstracts. The RCT and OS between January 1, 2002 – December 31, 2002 and January 1, 2017 -December 31, 2017 in N Engl J Med, Lancet and JAMA were included. Results: Among the RCT in 2017 the decrease in the number of significant p values was quite similar between what was reported (3) and the current study (Table) (29.3 vs 30.0%) supporting the validity of the slightly different methodology we used. We were however surprised to see that changing the significance level to 0.005 did not decrease the number of significant p values in OS as compared to the RCT. In fact the percentage decrease in RCT was statistically significantly less in 2002 and numerically in 2017 (Table). Table. OS and RCT in 2002 and 2017 OS (2002 ) RCT (2002 ) Number of articles 305 211 Total number of p values related to primary outcome(s) 323 322 p&lt;0.05 n(%) 260(80.4) 240(74.5) p&lt;0.005 n(%) 190(58.8) 121(37.5) Reduction in significant number of p values* (%) 70(26.9) P=0.011 119(49.5) OS (2017 ) RTC (2017 ) Number of articles 167 200 Total number of p values related to primary outcome(s) 187 257 p&lt;0.05 n (%) 162 (86.6) 173(67.3) p&lt;0.005 n (%) 125(66.8%) 121(47.0%) Reduction in significant number of p values* (%) 37(22.8) P=0.55 52(30.0) RCT:Randomized Controlled Trials OS: Observational Studies *Number of p values &lt;0.05 -Number of p values&lt;0.005/(Number of p values&lt;0.05 Conclusion: Contrary to our initial reasoning we now realize that changing the traditional p threshold will not decrease the number of significant p values more in OS than in RCT. A limitation of our work was that we surveyed only 3, high impact factor, general medicine journals. We envisage that in studies with smaller sample sizes more commonly published in journals with lower impact factors, the proposed change in the significance threshold would be more useful. Smaller sample sizes lead to more unstable, fickle p (4) values and changing the threshold would conceivably more or less stabilize this fickleness both in OS and RCT. References: [1]Ioannidis JPA. The proposal to lower p value thresholds to .005. JAMA. 2018;319:1429-30. [2]Benjamin DJ, Berger JO, Johannesson M, Nosek BA, Wagenmakers EJ, Berk R,et al. Redefine statistical significance. Nat Hum Behav. 2018;2:6-10. [3]Wayant C, Scott J, Vassar M. Evaluation of lowering the p value threshold for statistical significance from .05 to .005 in previously published randomized clinical trials in major medical journals. JAMA. 2018;320:1813-15. [4]Halsey, Lewis G., et al. The fickle P value generates irreproducible results. Nature methods. 2015;12:179-185. Disclosure of Interests: None declared