Does the Net Reclassification Improvement Help Us Evaluate Models and Markers?
Authors: Andrew J. Vickers, Margaret Pepe
Journal: Annals of Internal Medicine (2014)

Editorials21 January 2014Does the Net Reclassification Improvement Help Us Evaluate Models and Markers?Andrew J. Vickers, PhD and Margaret Pepe, PhDAndrew J. Vickers, PhDFrom Memorial Sloan-Kettering Cancer Center, New York, New York; and Fred Hutchinson Cancer Research Center, Seattle, Washington.Search for more papers by this author and Margaret Pepe, PhDFrom Memorial Sloan-Kettering Cancer Center, New York, New York; and Fred Hutchinson Cancer Research Center, Seattle, Washington.Search for more papers by this authorAuthor, Article, and Disclosure Informationhttps://doi.org/10.7326/M13-2841 SectionsAboutFull TextPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinkedInRedditEmail Few statistical methods have been as rapidly adopted as the net reclassification improvement (NRI). The reasons that the NRI has become so popular so quickly are not hard to fathom. First, it is an appealing concept to investigators. Instead of reporting that their new marker improved the area under the receiver-operating characteristic curve (AUC)—a statistical abstraction at the best of times—they can say that their marker led to “reclassification,” suggesting clinical value. The popularity of the NRI also stems from a belief that the AUC is insensitive. For example, in an early paper on the NRI, Pencina and colleagues stated ...References1. Pencina MJ, D'Agostino RB, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med. 2011;30:11-21. [PMID: 21204120] CrossrefMedlineGoogle Scholar2. Begg CB, Gonen M, Seshan VE. Testing the incremental predictive accuracy of new markers. Clin Trials. 2013;10:690-2. [PMID: 23881367] CrossrefMedlineGoogle Scholar3. Leening MJG, Vedder MM, Witteman JCM, Pencina MJ, Steyerberg EW. Net reclassification improvement: computation, interpretation, and controversies. A literature review and clinician's guide. Ann Intern Med. 2014;160:122-131. LinkGoogle Scholar4. Vickers AJ, Elkin EB, Steyerberg E. Net reclassification improvement and decision theory [Letter]. Stat Med. 2009;28:525-6. [PMID: 17907248] CrossrefMedlineGoogle Scholar5. Kerr KF, Wang Z, Janes H, McClelland RL, Psaty BM, Pepe MS. Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology. 2014;25:114-21. [PMID: 24240655] CrossrefMedlineGoogle Scholar6. Hilden J, Gerds TA. A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index. Stat Med. 2013;:. [PMID: 23553436] MedlineGoogle Scholar7. Pepe MS, Janes H, Kerr KF, Psaty BM. Net Reclassification Index: a Misleading Measure of Prediction Improvement. UW Biostatistics Working Paper Series; Working Paper 394. 2013. Google Scholar8. Vickers AJ. Decision Curve Analysis. New York: Memorial Sloan-Kettering Cancer Center; 2013. Accessed at www.mskcc.org/research/epidemiology-biostatistics/health-outcomes/decision-curve-analysis-0 on 6 December 2013. Google Scholar9. Localio AR, Goodman S. Beyond the usual prediction accuracy metrics: reporting results for clinical decision making [Editorial]. Ann Intern Med. 2012;157:294-5. [PMID: 22910942] LinkGoogle Scholar10. Van Calster B, Vickers AJ, Pencina MJ, Baker SG, Timmerman D, Steyerberg EW. Evaluation of markers and risk prediction models: overview of relationships between NRI and decision-analytic measures. Med Decis Making. 2013;33:490-501. [PMID: 23313931] CrossrefMedlineGoogle Scholar Author, Article, and Disclosure InformationAffiliations: From Memorial Sloan-Kettering Cancer Center, New York, New York; and Fred Hutchinson Cancer Research Center, Seattle, Washington.Grant Support: In part by funds from David H. Koch provided through the Prostate Cancer Foundation, the Sidney Kimmel Center for Prostate and Urologic Cancers, and SPORE grant P50-CA92629 from the National Cancer Institute to Dr. Howard Scher. This work was also supported in part by a National Institutes of Health/National Cancer Institute Cancer Center Support Grant to Memorial Sloan-Kettering Cancer Center under award P30 CA00874 and by grant R01CA152089 from the National Institutes of Health.Disclosures: None disclosed. Forms can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M13-2841.Corresponding Author: Andrew J. Vickers, PhD, Department of Epidemiology & Biostatistics, Memorial Sloan-Kettering Cancer Center, 307 East 63rd Street, 2nd Floor, New York, NY 10021; e-mail, [email protected]org.Current Author Addresses: Dr. Vickers: Department of Epidemiology & Biostatistics, Memorial Sloan-Kettering Cancer Center, 307 East 63rd Street, 2nd Floor, New York, NY 10021.Dr. Pepe: Fred Hutchinson Cancer Research Center, 1100 Fairview Avenue North, PO Box 19024, Seattle, WA 98109-1024. PreviousarticleNextarticle Advertisement FiguresReferencesRelatedDetailsSee AlsoNet Reclassification Improvement: Computation, Interpretation, and Controversies Maarten J.G. Leening , Moniek M. Vedder , Jacqueline C.M. Witteman , Michael J. Pencina , and Ewout W. Steyerberg Net Reclassification Improvement: Computation, Interpretation, and Controversies Maarten J.G. Leening , Moniek M. Vedder , Jacqueline C.M. Witteman , Michael J. Pencina , and Ewout W. Steyerberg Metrics Cited byReclassification prediction of first-year protocol biopsy on active surveillance of prostate cancer by p2PSA-related parameters: from PRIAS-JAPANNet benefit index: Assessing the influence of a biomarker for individualized treatment rulesUtility of the “omics” in kidney disease: Methods of analysis, sampling considerations, and technical approaches in renal biomarkersGraphical representations and summary indicators to assess the performance of risk predictorsApplication of net reclassification index to non-nested and point-based risk prediction models: a reviewStatistical methods for building better biomarkers of chronic kidney diseaseHematuria as a risk factor for progression of chronic kidney disease and death: findings from the Chronic Renal Insufficiency Cohort (CRIC) StudyDevelopment and Validation of a Prediction Model for Postoperative Peritoneal Metastasis After Curative Resection of Colon CancerEvent rate net reclassification index and the integrated discrimination improvement for studying incremental value of risk markersFirst things first: risk model performance metrics should reflect the clinical applicationClinical utility of serum 25-hydroxyvitamin D in the diagnosis of insulin resistance and estimation of optimal 25-hydroxyvitamin D in U.S. adultsThe Brier score does not evaluate the clinical utility of diagnostic tests or prediction modelsHow good are we at predicting the fate of someone with acute myeloid leukaemia?Prediction modelsThe clinical utility curve: a proposal to improve the translation of information provided by prediction models to cliniciansPrognostic impact of nomogram based on whole tumour size, tumour disappearance ratio on CT and SUVmax on PET in lung adenocarcinomaPredicting Progression in CKD: Perspectives and PrecautionsIncorporating Clinical Considerations into Statistical Analyses of Markers: A Quiet Revolution in How We Think About DataShould Patients With Atrial Fibrillation and 1 Stroke Risk Factor (CHA 2 DS 2 -VASc Score 1 in Men, 2 in Women) Be Anticoagulated?25-hydroxyvitamin D Levels and Coronary Heart Disease Risk Reclassification in Hypertension – Is it worth the “hype”?How many longitudinal covariate measurements are needed for risk prediction?Evaluation of the Usefulness of Consensus Definitions of Sarcopenia in Older Men: Results from the Observational Osteoporotic Fractures in Men Cohort StudyThe Net Reclassification Index (NRI): A Misleading Measure of Prediction Improvement Even with Independent Test Data SetsStatistical models for respiratory disease diagnosis and prognosisPrognostic utility of serum CRP levels in combination with CURB-65 in patients with clinically suspected sepsis: a decision curve analysisTailoring the Implementation of New Biomarkers Based on Their Added Predictive Value in Subgroups of IndividualsTransparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and ElaborationKarel G.M. Moons, PhD, Douglas G. Altman, DSc, Johannes B. Reitsma, MD, PhD, John P.A. Ioannidis, MD, DSc, Petra Macaskill, PhD, Ewout W. Steyerberg, PhD, Andrew J. Vickers, PhD, David F. Ransohoff, MD, and Gary S. Collins, PhDNovel metabolic biomarkers of cardiovascular diseaseBiomarkers and Risk Models in Cardiac SurgeryNet reclassification improvement and integrated discrimination improvement require calibrated models: relevance from a marker and model perspectiveKey Concepts and Limitations of Statistical Methods for Evaluating Biomarkers of Kidney DiseaseAssessment of a University of California, Los Angeles 4‐Variable Risk Score for Advanced Heart Failure 21 January 2014Volume 160, Issue 2Page: 136-137KeywordsBiopsyBiostatisticsCancer epidemiologyDecision analysisExerciseGastroesophageal reflux diseaseMedical risk factorsProstate cancerStatistical methods ePublished: 21 January 2014 Issue Published: 21 January 2014 Copyright & PermissionsCopyright © 2014 by American College of Physicians. All Rights Reserved.PDF downloadLoading ...