Progress towards clinically informative data-driven decision support tools in psychotherapy
Authors: Christian A Webb, Zachary D Cohen
Journal: The Lancet Digital Health (2021)

Timely identification of patients receiving psychotherapy who are elevated risk of a poor outcome can enable clinicians to intervene early, make appropriate treatment adjustments, and ultimately improve outcomes. In many real-world settings, this process is done intuitively and idiosyncratically, but studies have shown that providing therapists with regular, structured feedback about patients' treatment progress can improve outcomes, in particular for patients who are classified as not on track and therefore at increased risk of poor outcomes.1Delgadillo J de Jong K Lucock M et al.Feedback-informed treatment versus usual psychological treatment for depression and anxiety: a multisite, open-label, cluster randomised controlled trial.Lancet Psychiatry. 2018; 5: 564-572Summary Full Text Full Text PDF PubMed Scopus (62) Google Scholar Within these studies, prognosis (ie, the expected outcome trajectory or endpoint) is typically based only on pre-treatment symptom severity, which excludes other information about patients that could improve the prediction. Additionally, previous studies have typically used so-called fixed predictions based on baseline symptom severity, which are not dynamically updated over time as more information about a patient's progress becomes available. There is a crucial need to develop, rigorously test, and disseminate data-driven approaches for predicting treatment response, and systems (eg, clinical decision tools) that can leverage these predictions to help inform the shared clinical decision making process by adapting treatment and optimising outcomes.2Cohen ZD, Delgadillo J, DeRubeis RJ. Personalized treatment approaches. In: Barkham M, Lutz W, Castonguay LG, eds. Bergin and Garfield's Handbook of Psychotherapy and Behavior Change, 7th edn. New Jersey, NJ: Wiley Blackwell (in press).Google Scholar In this issue of The Lancet Digital Health, Claire Bone and colleagues3Bone C Simmonds-Buckley M Thwaites R et al.Dynamic prediction of psychological treatment outcomes: development and validation of a prediction model using routinely collected symptom data.Lancet Digit Health. 2021; 3: e231-e240Summary Full Text Full Text PDF PubMed Scopus (8) Google Scholar provide a valuable contribution to the literature by addressing several of these limitations and developing a dynamic algorithm (Oracle) to predict psychotherapy outcome. A key innovation in this study3Bone C Simmonds-Buckley M Thwaites R et al.Dynamic prediction of psychological treatment outcomes: development and validation of a prediction model using routinely collected symptom data.Lancet Digit Health. 2021; 3: e231-e240Summary Full Text Full Text PDF PubMed Scopus (8) Google Scholar is that predictions were continuously updated on the basis of incoming patient symptom data. Predictive performance was relatively modest at session one (approximate area under the curve [AUC] across models 0·60), and steadily improved, reaching high accuracy by session seven (AUC >0·80). The predictive algorithm incorporated both between-patient (ie, comparing a specific patient's current symptom severity with the sample mean) and within-patient (ie, variability in a specific individual's symptoms over time) information. The authors also evaluated the benefit of integrating information from the previously validated Leeds Risk Index,4Delgadillo J Appleby S Booth S et al.The Leeds Risk Index: field-test of a stratified psychological treatment selection algorithm.Psychother Psychosom. 2020; 89: 189-190Crossref PubMed Scopus (6) Google Scholar which uses weighted risk scores for factors such as unemployment, disability, and treatment expectancy to classify patients according to their risk of having poor treatment response, and also compared these models to similarly complex, alternative modelling approaches. This study has several methodological strengths compared with much of the previous literature on psychotherapy prediction, including its use of large sample sizes, rigorous external validation procedures, and evaluation of the degree to which the complex modelling approaches outperformed more simple or standard models. In particular, this methodological rigor revealed that the models maintained predictive accuracy when evaluated in the held-out test samples (ie, new sets of patients the model has not yet seen) and evidenced generalisability in samples obtained from different settings (ie, different clinics or years). These findings have important implications for the potential utility of these models in real-world clinical settings, since they suggest that the models could be used in services across the UK, and thus the creation of new service-specific models at each unique site might not be required. Two important findings require consideration. First, the performance of a model that simply included baseline and current session symptom scores was comparable to the Oracle models, including Oracle2, which incorporated more data (eg, Leeds Risk Index). Second, the more complex modelling approaches (eg, machine learning) did not significantly improve prediction accuracy when compared with simpler approaches (eg, logistic regression), which conforms to the findings reported in previous meta-analyses of the medical literature.5Christodoulou E Ma J Collins GS Steyerberg EW Verbakel JY Van Calster B A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models.J Clin Epidemiol. 2019; 110: 12-22Summary Full Text Full Text PDF PubMed Scopus (481) Google Scholar With continued exploration of systems that use predictive models to inform treatment decisions, the benefits, costs and risks of relying on these more complex modelling approaches that are currently popular in many fields,6Wilkinson J Arnold KF Murray EJ et al.Time to reality check the promises of machine learning-powered precision medicine.Lancet Digit Health. 2020; 2: e677-e680Summary Full Text Full Text PDF PubMed Scopus (50) Google Scholar must be carefully considered. Several areas of research could be explored moving forward. The predictive power of algorithms, and the added value of more sophisticated modelling approaches, might increase in the context of a more broad or complex set of predictor variables. For example, longitudinal data about treatment mechanisms (eg, acquisition and use of psychotherapy skills7Webb CA, Forgeard M, Israel E, Lovell-Smith NM, Björgvinsson T, Beard C. Personalized prescriptions of therapeutic skills from patient characteristics: an ecological momentary assessment approach. J Consult Clin Psychol (in press).Google Scholar) collected via self-report, ecological momentary assessment, or use-data from digital therapy apps could improve the algorithms' performance. Furthermore, passively collected data from smartphone sensors (eg, accelerometer, global positioning system data, metadata on calls and texts, phone usage) could be harnessed to predict states of elevated symptoms or affective distress when interventions are most needed (ie, just-in-time adaptive interventions8Nahum-Shani I Smith SN Spring BJ et al.Just-in-time adaptive interventions (JITAIs) in mobile health: key components and design principles for ongoing health behavior support.Ann Behav Med. 2018; 52: 446-462Crossref PubMed Scopus (625) Google Scholar). Ultimately, the goal is to translate this research into clinical decision support tools that clinicians and patients are willing to use in real-world treatment settings and, crucially, that improve outcomes for patients.9Lutz W, Deisenhofer A-K, Rubel J, et al. Prospective evaluation of a clinical decision support system in psychological therapy. J Consult Clin Psychol (in press).Google Scholar How exactly these predictive algorithms would be used systematically, and evidence that such an approach would actually improve outcomes relative to current best practices, has yet to be determined. The work of Bone and colleagues and others in the field, represent important steps towards this ambitious and exciting endeavour. ZDC has previously collaborated with a subset of the Article authors (JR, WL, JD) on published work. CAW declares no competing interests. Dynamic prediction of psychological treatment outcomes: development and validation of a prediction model using routinely collected symptom dataDynamic prediction models using sparse and readily available symptom measures are capable of predicting psychotherapy outcomes with high accuracy. Full-Text PDF Open Access