Inappropriate Statistical Analysis and Reporting in Medical Research: Perverse Incentives and Institutional Solutions
Authors: A. Russell Localio, Catharine B. Stack, Anne R. Meibohm, Eric A. Ross, Eliseo Guallar, John B. Wong, John E. Cornell, Michael E. Griswold, Steven N. Goodman
Journal: Annals of Internal Medicine (2018)

Editorials16 October 2018Inappropriate Statistical Analysis and Reporting in Medical Research: Perverse Incentives and Institutional SolutionsA. Russell Localio, PhD, Catharine B. Stack, PhD, Anne R. Meibohm, PhD, Eric A. Ross, PhD, Eliseo Guallar, MD, DrPH, John B. Wong, MD, John E. Cornell, PhD, Michael E. Griswold, PhD, and Steven N. Goodman, MD, MHS, PhDA. Russell Localio, PhDUniversity of Pennsylvania, Philadelphia, Pennsylvania (A.R.L.)Search for more papers by this author, Catharine B. Stack, PhDAmerican College of Physicians, Philadelphia, Pennsylvania (C.B.S., A.R.M.)Search for more papers by this author, Anne R. Meibohm, PhDAmerican College of Physicians, Philadelphia, Pennsylvania (C.B.S., A.R.M.)Search for more papers by this author, Eric A. Ross, PhDFox Chase Cancer Center, Philadelphia, Pennsylvania (E.A.R.)Search for more papers by this author, Eliseo Guallar, MD, DrPHJohns Hopkins University, Baltimore, Maryland (E.G.)Search for more papers by this author, John B. Wong, MDTufts University School of Medicine, Boston, Massachusetts (J.B.W.)Search for more papers by this author, John E. Cornell, PhDUniversity of Texas Health Science Center, San Antonio, Texas (J.E.C.)Search for more papers by this author, Michael E. Griswold, PhDUniversity of Mississippi Medical Center, Jackson, Mississippi (M.E.G.)Search for more papers by this author, and Steven N. Goodman, MD, MHS, PhDStanford University School of Medicine, Stanford, California (S.N.G.)Search for more papers by this authorAuthor, Article, and Disclosure Informationhttps://doi.org/10.7326/M18-2516 SectionsAboutFull TextPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinkedInRedditEmail Wang and colleagues (1) present a sobering report of a national survey of nearly 400 consulting statisticians about requests from investigators to engage in inappropriate statistical practices. Framed as an exploration of bioethical issues, the report implicitly adopts Doug Altman's mantra: “Misuse of statistics is unethical” (2). Although the survey did not ask statisticians whether they fulfilled these requests, the inappropriate methods described in this report are still used in the published literature, and thus contribute to the problem of nonreproducible research.Practices like these are extraordinarily difficult to detect in published work; identification takes either unusual transparency or a ...References1. Wang MQ, Yan AF, Katz RV. Researcher requests for inappropriate analysis and reporting: a U.S. national survey of consulting biostatisticians. Ann Intern Med. 2018;169:554-8. doi:10.7326/M18-1230 LinkGoogle Scholar2. Altman DG. Statistics and ethics in medical research. Misuse of statistics is unethical. Br Med J. 1980;281:1182-4. [PMID: 7427629] CrossrefMedlineGoogle Scholar3. Baker M. 1,500 scientists lift the lid on reproducibility. Nature. 2016;533:452-4. [PMID: 27225100] doi:10.1038/533452a CrossrefMedlineGoogle Scholar4. Rubin DB. For objective causal inference, design trumps analysis. Annals of Applied Statistics. 2008;2:808-40. CrossrefGoogle Scholar5. Rosenbaum PR. Design sensitivity and efficiency in observational studies. Journal of the American Statistical Association. 2010;105:692-702. CrossrefGoogle Scholar6. Rubin DB. The design versus the analysis of observational studies for causal effects: parallels with the design of randomized trials. Stat Med. 2007;26:20-36. [PMID: 17072897] CrossrefMedlineGoogle Scholar7. National Academies of Sciences, Engineering, and Medicine. Fostering Integrity in Research. Washington, DC: National Academies Pr; 2017. Google Scholar8. Patient-Centered Outcomes Research Institute (PCORI) Methodology Committee. PCORI Methodology Report. July 2017. Accessed at www.pcori.org/sites/default/files/PCORI-Methodology-Report.pdf on 12 September 2018. Google Scholar9. National Institutes of Health. Clinical e-Protocol Writing Tool. Accessed at https://e-protocol.od.nih.gov/#/home on 13 July 2018. Google Scholar10. Moher D, Naudet F, Cristea IA, Miedema F, Ioannidis JPA, Goodman SN. Assessing scientists for hiring, promotion, and tenure. PLoS Biol. 2018;16:e2004089. [PMID: 29596415] doi:10.1371/journal.pbio.2004089 CrossrefMedlineGoogle Scholar Author, Article, and Disclosure InformationAffiliations: University of Pennsylvania, Philadelphia, Pennsylvania (A.R.L.)American College of Physicians, Philadelphia, Pennsylvania (C.B.S., A.R.M.)Fox Chase Cancer Center, Philadelphia, Pennsylvania (E.A.R.)Johns Hopkins University, Baltimore, Maryland (E.G.)Tufts University School of Medicine, Boston, Massachusetts (J.B.W.)University of Texas Health Science Center, San Antonio, Texas (J.E.C.)University of Mississippi Medical Center, Jackson, Mississippi (M.E.G.)Stanford University School of Medicine, Stanford, California (S.N.G.)Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M18-2516.Corresponding Author: A. Russell Localio, PhD, Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, 617 Blockley Hall, 423 Guardian Drive, Philadelphia, PA 19104; e-mail, [email protected]upenn.edu.Current Author Addresses: Dr. Localio: Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, 617 Blockley Hall, 423 Guardian Drive, Philadelphia, PA 19104.Drs. Stack and Meibohm: American College of Physicians, 190 N. Independence Mall West, Philadelphia, PA 19106.Dr. Ross: Fox Chase Cancer Center, 333 Cottman Avenue, Philadelphia, PA 19111.Dr. Guallar: Welch Center for Prevention, Epidemiology, and Clinical Research, Johns Hopkins Medical Institutions, Room 2-645, Baltimore, MD 21205.Dr. Wong: Tufts Medical Center, 800 Washington Street, #302, Boston, MA 02111.Dr. Cornell: 14546 Indian Woods, San Antonio, TX 78249.Dr. Griswold: Department of Data Science, John D. Bower School of Population 08alth, University of Mississippi Medical Center, 2500 North State Street, Jackson, MS 39216.Dr. Goodman: Stanford University School of Medicine, 259 Campus Drive, T265 Redwood Building/Health Research and Policy, Stanford, CA 94305.This article was published at Annals.org on 9 October 2018. PreviousarticleNextarticle Advertisement FiguresReferencesRelatedDetailsSee AlsoResearcher Requests for Inappropriate Analysis and Reporting: A U.S. Survey of Consulting Biostatisticians Min Qi Wang , Alice F. Yan , and Ralph V. Katz Metrics Cited byLes clefs d’une publication réussie dans les Eur Ann Otorhinolaryngol Head Neck Dis : analyse STROBE de la relecture des articles scientifiques soumis en 2020–2021Keys for successful publication in Eur Ann Otorhinolaryngol Head Neck Dis: A STROBE analysis of peer reviews of articles submitted in 2020–2021Collaborative biostatistics and epidemiology in academic medical centres: A survey to assess relationships with health researchers and ethical implicationsMaîtriser les statistiques descriptives utilisées en otorhinolaryngologieMastering the descriptive statistics used in otorhinolaryngologyComprehensive review of statistical methods for analysing patient-reported outcomes (PROs) used as primary outcomes in randomised controlled trials (RCTs) published by the UK’s Health Technology Assessment (HTA) journal (1997–2020)Détente: A Practical Understanding of P values and Bayesian Posterior ProbabilitiesLes statistiques des articles scientifiques publiés dans les European Annals of Otorhinolaryngology Head & Neck DiseasesStatistics in scientific articles published in the European Annals of Otorhinolaryngology Head & Neck DiseasesInaccurate Use of the Upper Extremity Fugl-Meyer Negatively Affects Upper Extremity Rehabilitation Trial Design: Findings From the ICARE Randomized Controlled TrialEnhancing Collaboration between Clinician-Researchers and Methodologists in Clinical ResearchShortcoming of Visual Interpretation of Cardiotocography: A Comparative Study with Automated Method and Established Guideline Using Statistical Analysis« Suggestif » et « significatif » sont dans un bateau…“Suggestive” and “Significant”: You can’t always get what you want…The Importance of Predefined Rules and Prespecified Statistical Analyses 16 October 2018Volume 169, Issue 8Page: 577-578KeywordsBiostatisticsClinical epidemiologyDisclosureEpidemiologyMotivationReproducibilityResearch and reporting methodsResearch designResearch reporting guidelinesStatistical data ePublished: 9 October 2018 Issue Published: 16 October 2018 Copyright & PermissionsCopyright © 2018 by American College of Physicians. All Rights Reserved.PDF downloadLoading ...