Toward clinically useful models for individualised prognostication in psychosis
Authors: Nikolaos Koutsouleris
Journal: The Lancet Digital Health (2019)

On the way toward clinical application, tools for individualised outcome prediction in patients with mental illnesses must overcome three major challenges.1Chekroud A Koutsouleris N The perilous path from publication to practice.Mol Psychiatry. 2017; 23: 24-25Crossref PubMed Scopus (18) Google Scholar First, they must be generalisable to new patient cohorts. Second, they must prove useful for informing efficacious preventive interventions that adapt to the risk profile of the given patient. Finally, their clinical real-world implementation must be practical, acceptable, safe, and economically viable so that patients, families, professionals, and health-care systems can finally adopt them. Unfortunately, given these requirements, very little progress has been achieved so far in translating predictive research into prognostication tools for patients in the early stages of psychotic disorders. In my view, a key bottleneck lies in the beginning of the translational chain—ie, the phase of model derivation and validation. Here, the still widespread and combined use of insufficient statistical approaches has hampered substantial progress.2Dwyer DB Falkai P Koutsouleris N Machine learning approaches for clinical psychology and psychiatry.Annu Rev Clin Psychol. 2018; 14: 91-118Crossref PubMed Scopus (234) Google Scholar Specifically, the research-driven selection of purified, unrepresentative patient cohorts, the use of univariate analytics to tackle the complexity of psychiatric phenotypes, or the lack of thorough cross-validation to benchmark predictive patterns in new patients represent widespread shortcomings of current modelling practice. It is for these reasons that the work of Samuel P Leighton and colleagues3Leighton SP Upthegrove R Krishnadas R et al.Development and validation of multivariable prediction models of remission, recovery, and quality of life outcomes in people with first episode psychosis: a machine learning approach.Lancet Digital Health. 2019; (published online Sept 12.)http://dx.doi.org/10.1016/S2589-7500(19)30121-9Summary Full Text Full Text PDF PubMed Scopus (14) Google Scholar published in The Lancet Digital Health represents as a crucial step toward the next phase of clinical translation in that the authors delivered prognostication algorithms ready to be tested for clinical effectiveness. The paper follows the approach of Koutsouleris and colleagues4Koutsouleris N Kahn RS Chekroud AM et al.Multisite prediction of 4-week and 52-week treatment outcomes in patients with first-episode psychosis: a machine learning approach.Lancet Psychiatry. 2016; 3: 935-946Summary Full Text Full Text PDF PubMed Scopus (90) Google Scholar by analysing easily recordable, patient-reported, and sociodemographic baseline information using machine learning. Leighton and colleagues take this approach to the next stage by deriving predictive models not just for one (functional) outcome but for a portfolio of clinical, social, occupational, and quality of life-related endpoints in a larger sample of 1027 first-episode patients drawn from the English Evaluating the Development and Impact of Early Intervention Services (EDEN) study. A first key finding of this work is that these different outcomes can be predicted with an area-under-the-curve (AUC) of 0·70 for symptom remission to 0·74 for vocational recovery following the internal-external leave-one-site-out cross-validation approach proposed by Steyerberg and Harrell.5Steyerberg EW Harrell FE Prediction models need appropriate internal, internal–external, and external validation.J Clin Epidemiol. 2016; 69: 245-247Summary Full Text Full Text PDF PubMed Scopus (312) Google Scholar Furthermore, due to the multi-collinearity of predictors and predictands, each model estimated the targets of the other models with a reasonable, albeit lower, AUC (>0·6). The paper also achieved a higher level of scientific rigour than the approach of Koutsouleris and colleagues by adding an external validation step to the internal-external leave-one-site-out cross-validation approach. Among the most prognostically relevant variables, the authors first identified those that were shared between EDEN and two smaller independent datasets (two Scottish studies and the Danish OPUS sample). Then, they trained generalised linear models on the overlapping variables from the EDEN cohort and benchmarked the predictive performance of these models in the two independent cohorts. Interestingly, Leighton and colleagues observed similar predictive performance of their models in the EDEN and Scottish cohorts, but reported a significant lower degree of generalisability in the OPUS dataset. This discrepancy represents a key finding in the rich set of results of the authors' work in that it highlights the remaining challenges when it comes to the fair and unbiased external validation of predictive models: post-hoc validation might give rise to an increased type II error given the inevitable mismatch of variables recorded, recruitment or follow-up schemes applied, and purposes served by different patient cohorts. Thus, this retrospective validation approach—although highly commendable compared with the state of the art approaches—risks to produce over-conservative estimates of the true model generalisability as it might inflate the heterogeneity of study populations, and reduce the available features for prediction to the least common denominator. This risk is compounded by the authors' choice to not use the same machine learning algorithm in the external validation step and instead retrain a top-hits predictive tool based on generalised linear models. Furthermore, Leighton and colleagues leave the question unanswered whether algorithms developed for better generalisation to new patients, such as non-linear support vector machines,6Noble WS What is a support vector machine?.Nat Biotechnol. 2006; 24: 1565-1567Crossref PubMed Scopus (1083) Google Scholar or ensemble-based methods,7Polikar R Ensemble based systems in decision making.IEEE Circuits Syst. 2006; 6: 21-45Crossref Scopus (1841) Google Scholar such as random forests, would have led to more reproducible prediction results in the independent validation samples, although the authors found no improvement in the performance of their model when scoping their training dataset with more complex classifiers. What conclusions can we draw from the major strengths, as well as the inevitable and evitable limitations of this work? Leighton and colleagues have most likely provided the field with a set of models generalisable enough to warrant a thorough exploration of their therapeutic effectiveness in clinical trials and their implementability in real-world clinical care. To move, however, to a more reproducible and translatable model-based prognostication in psychiatry, we will need a bold move toward data sharing, mandatory study registration, and a greater degree of harmonised data acquisition across prognostic and diagnostic studies. External validation will also require prospective replication steps built into new study designs and honest broker infrastructures to manage discovery and validation steps in a way that excludes any information leakage between the two. Finally, researchers, editors, and reviewers might need to reconsider the value of scientific originality against the need to develop reliable digital tools that substantially improve patients' disease courses through more personalised preventive care. I have a patent issued related to adaptive pattern recognition for psychosis risk modelling (US20160192889A1). Development and validation of multivariable prediction models of remission, recovery, and quality of life outcomes in people with first episode psychosis: a machine learning approachIn our machine learning analysis, we showed that prediction models can reliably and prospectively identify poor remission and recovery outcomes at 1 year for patients with first-episode psychosis using baseline clinical variables at first clinical contact. Full-Text PDF Open Access