Predicting Treatment Outcome from Patient Texts:The Case of I nternet-Based Cognitive Behavioural Therapy (2021)

Abstract: We investigate the feasibility of applying standard text categorisation methods to patient text in order to predict treatment outcome in Internet-based cognitive behavioural therapy. The data set is unique in its detail and size for regular care for depression, social anxiety, and panic disorder. Our results indicate that there is a signal in the depression data, albeit a weak one. We also perform terminological and sentiment analysis, which confirm those results.

## introduction the internet psychiatry clinic in stockholm offers internet-based cognitive behavioral therapy (icbt) for depression, social anxiety, and panic disorder with documented significant treatment effects @xcite . the treatment is delivered on a secure online platform that enables patients to submit answers to weekly questionnaires of primary symptoms, depression, and suicidal ideation, as well as interactive worksheets. because patients normally are not in contact with the psychologists away from keyboard during treatment, the infrastructure platform logs a large amount of data for each patient, yielding a unique data set. because sweden was the first country to introduce this kind of icbt, the clinical experience is considerable @xcite , and experimental research of the kind reported in this paper rests on well-established procedures for sensemaking of machine learning at the clinic @xcite . the overarching problem is how to be able to offer accelerated care, via identification of those predicted not to succeed with treatment, or at least not with enough decrease in symptoms, and giving these people more attention, as early as possible. we here investigate the possibility of predicting outcome based only on patient texts, in various points in treatment. we formulate this as a unimodal binary text categorisation problem, where the categories are simplified to either success or failure. in this initial feasibility study, we evaluate the performance of a number of text categorisation methods for predicting treatment outcome. even if our methodology is mostly exploratory, we also relate terminological and sentiment analysis of patients' text to the treatment outcome. ## data set the total data covers 6, 821 patients enrolled in a 12-week treatment for three different psychological disorders: major depressive disorder (hereafter referred to as mdd, or depression), social anxiety, or panic disorder. each treatment consists of weekly exercises, referred to as homework reports. for predicting treatment outcome, we focus on the depression treatment programme, which consists of 10 homework reports distributed over the 12 weeks. the homework contains questions about the progress made by the patient during treatment, such as "what is the most important thing you bring from module 1?" the questions can be both closed questions, where the patient selects pre-defined answers from a list, and open-ended questions, where the patient is invited to type in free text. the depression data covers 3, 179 patients. we remove patients who completed less than five homework reports (out of ten), patients without treatment outcome (who did not fill out the self-assessment at the end of the treatment), and empty texts. the resulting filtered data set contains 16, 379 texts from homework reports completed by 1, 986 patients. each text is a concatenation of all answers written by a patient in one single weekly homework. we also use patients' texts from the other treatments in order to enhance the domain pre-training for the embeddings. this data covers all 6, 821 patients participating in any of the three treatments, including 180, 017 free texts answers from the homework reports and 146, 398 direct messages, totalling almost 29 million (28, 993, 089) tokens. as target scores for predicting treatment outcome for the depression patients, we use the self-assessment madrs-sr method @xcite , which consists of ten questions scored between zero and six, and amounting to each patient having done between five and ten modules, yielding a total score from zero to 54. therapists commonly use the cut-off score of 10 to partially define successful treatment. all diagnosed patients started the treatment above this score, and the treatment is defined as successful if the score drops below 10 at the end of the treatment (remission) @xcite . when patients show a symptom reduction of at least 50%, this is defined as a response, and the treatment is also considered successful, even if the final score is above the cutoff value. in other cases, the treatment is considered a failure. regarding the choice of 50% cut-off for the definition of the responder patients, it is supported by previous work @xcite , while the same value has been previously used by clinical researchers @xcite . all patients are labelled as being subject to a treatment success or a treatment failure, as shown in ## methodology we formulate the task of predicting outcome as a simple binary text classification problem, where the classes are either success or failure. our goal is to assist clinicians in defining the best possible adaptive treatment strategy. therefore, the class definitions used here are inherited from the treatment model employed at the clinic. we compare four different text representations: tf-idf (jones, 1972), foot_0 word2vec @xcite , fasttext @xcite , and doc2vec @xcite , all of which are fed into a simple linear classifier trained with binary cross entropy loss using pytorch. 2 all word embeddings are pro-duced with the gensim implementation 3 and are pre-trained on patients' texts from all three treatments, as described in section 3. we use a naïve classification model that generates stratified predictions based on the class distributions as the baseline. the training set includes 1, 588 patients, with 860 labelled as success and 728 as failure, from the depression programme who have completed at least five weeks of homework reporting. all models are trained with 3-fold cross validation and early stopping, to avoid overfitting. after training, we evaluate each model on the test set, which consists of 398 patients (215 labelled as success and 183 as failure). model performance is measured using the f1 score metric. ## results the results from the classification experiment are presented in as indicated by the classification results, simple lexical representations perform best in our experimental setup. a concrete follow-up question then becomes: which vocabulary terms have a strong positive or negative correlation with the success class? to answer the question, we perform a linear regression analysis on the complete set of 1, 986 tf-idf patient embeddings from our prediction data set, using the first three homework sets, and then compute the dot product between each word in the patient vocabulary and the regression coefficient. vocabulary words with positive value of dot product contribute positively to the prediction of the success class and vice versa. examples of terms with high positive or negative correlation with the success class are presented in there seems to be a vague relation between terminology and treatment outcome, which corroborates previous work by @xcite that reports positive correlation between positively signed written language and successful counseling conversations. we therefore perform sentiment analysis on the complete list of homework reports written by 2, 611 depression patients, with at least one homework report completed. since we are interested in the overall progression of sentiment in patients' answers, we are not constrained to include only patients with a specified number of homework reports completed. we train a multinomial naïve bayes classifier for predicting the sentiment class (positive, neutral or negative) on the sensaldo swedish sentiment lexicon. 4 all words are represented with tf-idf embeddings. the sentiment classifier is then used to identify the sentiment class of all words in the vocabulary, excluding stopwords and words with document frequency lower than 0.2. for each patient and concatenated set of homework reports, the number of word occurrences per sentiment class is calculated. we group the patients by the label outcome and compute the average relative frequency of the number of positive words divided by the number of negative words for each homework report and group. the progression of the average relative frequency of words with positive sentiment throughout the homework reports is presented in figure 1 . similar to @xcite , we observe that more positive sentiment in the homework answers is related to successful treatment outcome. this reflects the treatment progress for patients that in the end are classified as successful. specifically, homework report 8 had questions concerned with the overall opinion of the treatment, and the usefulness of the tools that the patients had been using. the large differences observed at homework report 8 confirm our intuition that patients showing substantial progress during the treatment, would evaluate its usefulness in more positive terms. whereas the analysis presented in @xcite concerns open-ended counseling conversations, our data set contains patients' answers to targeted ques-tions with positive or negative underlying sentiment. two indicative questions are "what's the most important thing you have learned from the past week?" and "tell us about the activities you had planned for this week that you followed through. how was it?". the variation in sentiment in homework questions is reflected in the saturation of the relative frequency throughout the progression of homework reports. ## discussion the results demonstrate that the task of predicting treatment outcome based on patient text is very difficult. it is interesting to note that embedding-based representations in some cases fail to perform better than random guessing, which we partly attribute to the comparably small amounts of training data for both domain adaptation and classification, and partly to the fact that such models tend to conflate paradigmatically similar terms, which in this application can signal very different treatment effects. this is demonstrated by terminological and sentiment analysis, which indicate a small but noticeable difference in vocabulary between successful and not successful patients. one hypothesis may be that antonyms are predictive of different outcomes, but have similar representations in the word embeddings, which makes it difficult for a simple linear classifier to separate them. the bag of words does not have this problem, which may be the reason it performs better in this specific application. the selection of patients with a minimum predefined number of homework reports completed for the classification experiments serves the purpose of identifying the best prediction point in the home-positive words forward, away, day, thoughts, initiative, act, wanted, feeling, together, energy, later negative words sleep, rarely, unfortunately, sad, friend, rewarding, despite, life, walk, fill, different our current analysis ignores the set of homework assignment questions. since much of the patient text in icbt is generated on the prompt of a question, future work could study the correlation of question-answer pairs with the treatment outcome. ## conclusions we have analysed the potential of using patient text from internet-based cognitive behavioural therapy as the only signal for predicting treatment outcome. when framing the problem as a binary classification task between treatment success and failure, we manage to beat stratified random guessing using a simple bag of words classifier. this demonstrates the feasibility of our approach. interestingly, word embeddings fail to improve on this simple approach, and our best results are achieved using only data from the first couple of weeks of treatment. this has clinical significance because interventions during treatment are still meaningful, and could prove crucial, in the first four weeks of treatment. additionally, we provide simple terminological and sentiment analysis, which also indicate that there is a signal in the patient text data, albeit a weak one. the work reported on here should be seen as a feasibility study in that it shows the potential to predict treatment outcome based on patient text, even in a very small subset of the set of patients reporting and reflecting on what they do in treatment, and how treatment progresses. our next step is to look into larger subsets than the one employed here, but that step will be taken in conjunction with moving from the well-established albeit simple methods used here to contextualised language models.