Prognostic models in first-episode psychosis
Authors: Daniel Whiting, Seena Fazel
Journal: The Lancet Digital Health (2020)

Samuel P Leighton and colleagues1Leighton SP Upthegrove R Krishnadas R et al.Development and validation of multivariable prediction models of remission, recovery, and quality of life outcomes in people with first episode psychosis: a machine learning approach.Lancet Digital Health. 2019; 1: e261-e270Summary Full Text Full Text PDF Scopus (10) Google Scholar place a welcome focus on first-episode psychosis as an area in which prediction models could assist in developing treatment pathways on the basis of more individualised prognostic assessments. However, whether machine learning is the most appropriate approach, as well as the reporting and interpretation of model performance, will need clarification before these models can be clinically used. First, machine learning has strengths, such as the capacity for complex, high-dimensional data. However, the presumption that it will be superior to simpler regression approaches is not supported by a systematic review of prognostic models, published in 2019.2Christodoulou E Ma J Collins GS Steyerberg EW Verbakel JY Van Calster B A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models.J Clin Epidemiol. 2019; 110: 12-22Summary Full Text Full Text PDF PubMed Scopus (282) Google Scholar Therefore, the use of machine learning needs stronger justification. Additional challenges include reduced interpretability and face validity compared with regression models. Such models, if they incorporate evidence on prognostic factors as a starting point, are likely to be more acceptable for clinical services than exploratory, data-driven approaches and can be made more easily available for external validation.3Fazel S Wolf A Larsson H Lichtenstein P Mallett S Fanshawe TR Identification of low risk of violent crime in severe mental illness with a clinical prediction tool (Oxford Mental Illness and Violence tool [OxMIV]): a derivation and validation study.Lancet Psychiatry. 2017; 4: 461-468Summary Full Text Full Text PDF PubMed Scopus (29) Google Scholar Furthermore, because machine learning uses many predictors, sample size needs to be proportionately large, with at least ten outcomes per predictor. Although Leighton and colleagues highlighted sample size as a study strength, their training data considered 163 predictor variables with 320 to 436 outcome events for the models developed, suggesting that they are underpowered.4Moons KGM Wolff RF Riley RD et al.PROBAST: a tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration.Ann Intern Med. 2019; 170: W1-W33Crossref PubMed Scopus (133) Google Scholar, 5Whiting D Fazel S How accurate are suicide risk prediction models? Asking the right questions for clinical practice.Evid Based Ment Health. 2019; 22: 125-128Crossref PubMed Scopus (13) Google Scholar Second, although some performance measures are presented in tables, the discussion of model performance is confined to the area under the receiver operating characteristic curve (AUC). The range of AUC 0·56–0·87 for the models is interpreted as reliable (although AUC is an omnibus measure of discrimination rather than reliability) and compared with other AUCs. This is a narrow interpretation of predictive performance and needs to be balanced with false positive and negative rates at the recommended cutoff. Importantly, if the model is to be used to inform specific treatment decisions for individual patients, calibration should be presented, and paired measures of misclassification should be used to consider the potential harms in a manner sensitive to the clinical setting and decision pathway. Another discrimination metric, the positive predictive performance for vocational recovery, was 0·49 in one validation. This means that, for every ten individuals predicted to recover, five will not. Is this sufficiently accurate to withhold resources from those predicted to do well, if this is one potential application of the model? The other validation reported a positive predictive performance of 0·77. This is a large range for a metric that greatly affects whether and how clinical services will use such models. This context is particularly important in first-episode psychosis services, in which interventions for and decisions about prognosis involve the consideration of many factors. Embedding such clinical considerations at each stage of model development and validation will be important to translate advances in data science into patient benefit. We declare no competing interests. DW is funded by a National Institute for Health Research (NIHR) Doctoral Research Fellowship for this research project. This publication presents independent research supported by the NIHR. The views expressed are those of the authors and not necessarily those of the National Health Service, the NIHR, or the Department of Health and Social Care. Professor Seena Fazel is funded by the Wellcome Trust. Development and validation of multivariable prediction models of remission, recovery, and quality of life outcomes in people with first episode psychosis: a machine learning approachIn our machine learning analysis, we showed that prediction models can reliably and prospectively identify poor remission and recovery outcomes at 1 year for patients with first-episode psychosis using baseline clinical variables at first clinical contact. Full-Text PDF Open AccessPrognostic models in first-episode psychosis – Authors' replyWe thank Daniel Whiting and Seena Fazel for their insightful comments about our work and important suggestions for improvement in our analyses. Full-Text PDF Open Access