Identifying Distorted Thinking in Patient-Therapist Text Message Exchanges by Leveraging Dynamic Multi-Turn Context (2022)

Abstract: There is growing evidence that mobile text message exchanges between patients and therapists can augment traditional cognitive behavioral therapy. The automatic characterization of patient thinking patterns in this asynchronous text communication may guide treatment and assist in therapist training. In this work, we automatically identify distorted thinking in text-based patient-therapist exchanges, investigating the role of conversation history (context) in distortion prediction. We identify six unique types of cognitive distortions and utilize BERT-based architectures to represent text messages within the context of the conversation. We propose two approaches for leveraging dynamic conversation context in model training. By representing the text messages within the context of the broader patient-therapist conversation, the models better emulate the therapist’s task of recognizing distorted thoughts. This multi-turn classification approach also leverages the clustering of distorted thinking in the conversation timeline. We demonstrate that including conversation context, including the proposed dynamic context methods, improves distortion prediction performance. The proposed architectures and conversation encoding approaches achieve performance comparable to inter-rater agreement. The presence of any distorted thinking is identified with relatively high performance at 0.73 F1, significantly outperforming the best context-agnostic models (0.68 F1).

## introduction cognitive behavioral therapy (cbt) is an evidence based treatment applicable to a wide range of mental health conditions including depression, anxiety, addiction, bipolar disorder, and schizophrenia spectrum disorders @xcite . one primary clinical activity of cbt is the identification and re-framing of systematic errors in thinking, termed cognitive distortions, that create a skewed perception of reality @xcite . cognitive distortions are known to exacerbate psychiatric symptoms without intervention @xcite ; however, there are many types of cognitive distortions (e.g., overgeneralization or catastrophizing), which can make identification and appropriate intervention by clinicians more complicated @xcite . cbt has traditionally been administered through in-person office visits; however, there is increasing need for remote therapy options, to extend provider reach and increase access @xcite . remote therapy options include internetdelivered therapy, application-based therapy, teletherapy, and text messaging @xcite . there is growing evidence that asynchronous text-message-based exchanges between patients and therapists can augment conventional synchronous therapy and improve patient outcomes (d' @xcite . the expansion of text-message-based cbt provides an opportunity to develop clinician supports via novel natural language processing (nlp) methods that can guide patient treatment and assist in therapist training. in this work, we explore the automatic identification and categorization of cognitive distortions in a corpus of text-message conversations between patients with serious mental illness and their therapists. prior work identifying cognitive distortions in text treats each text sample (e.g. sentence or message) as an independent event without context. however, in this conversational paradigm, the preceding turns in the conversation may provide important contextual cues for recognizing distorted thinking. here, we utilize state-of-the-art deep learning nlp methods to explore the role of conversation history in identifying cognitive distortions in patient-therapist text message exchanges. by identifying distorted thinking in text messages within the broader context of the dialogue, the dialogue-based prediction architectures emulate the real-world process of mental health clinicians who account for conversation context when assessing for distortions. the dialogue-based architectures also mirror the cognitive distortion annotation process associated with the data set used in this work. we present multiple bert-based architectures for identifying distortions in multi-turn conversations and propose methods for dynamically representing the conversation context. we demonstrate that leveraging the dialogue context and incorporating the proposed dynamic conversation context yields statistically significant performance improvement, reaching performance levels comparable to interrater agreement. distorted thinking is identified in the text messages at 0.73 f1. ## related work there is a relatively small body of work exploring the automatic identification and categorization of cognitive distortions in user-generated text. @xcite explored the identification of dysfunctional thoughts in 188 text examples from the cognitive distortion literature. the authors manually curated linguistic features that were used in a decision tree. @xcite annotated 459 tumblr blogs for the presence of cognitive distortions. features were extracted using the linguistic inquiry and word count (liwc) tool (tausczik and pennebaker, 2010) 1 , and several classifiers were explored with logistic regression (lr) achieving the best performance. @xcite investigated the identification of cognitive distortions in online journal entries from college students and samples from crowdsourced participants prompted to give examples of defined distortion types. the authors investigated many classification architectures, including lr, support vector machines (svm), recurrent neural networks (rnn), convolutional neural networks (cnn), and bidirectional encoder representations from transformers (bert) @xcite . the authors reported the highest performance using lr with with term frequency-inverse document frequency (tf-idf) features. @xcite explored the classification of 10 distinct cognitive distortions in 3,000 therapist question-answer samples. several classifiers and feature encoding approaches were explored, and the best performance was achieved by an svm operating on the sen-tencebert encoding, without fine-tuning bert. in this work, we identify cognitive distortions in text-based conversations, exploring the role of conversation history. this distortion prediction task shares similarities with other multi-turn conversational tasks, including retrieval-based dialogue response generation and question answering. dialogue response and question answering are often approached using hierarchical architectures that first encode each turn, then aggregate the turn embeddings to create a conversation embedding, and lastly generate predictions using the conversation embedding. conversation turns are frequently mapped to a vector embedding using cnn, rnn, and transformers (e.g. bert), and conversation embeddings are derived from the turn embeddings using approaches like self-attention, rnn, markov models, and graphical models @xcite @xcite @xcite . drawing inspiration from these hierarchical approaches, we experiment with an approach where each turn is encoded using bert and then the sequence of turn encodings is mapped to a fixed length vector using a uni-directional rnn. there is also conversation modeling work that encodes multiple turns as a single input sequence to bert, separating the turns with the [sep ] token @xcite , which we also explore here. @xcite explored a retrieval-based response generation task and proposed a data augmentation technique for model training. the authors created additional positive samples by sampling contiguous multi-turn excerpts from conversations and assuming the last turn is a correct response. additional negative samples were created by sampling contiguous multi-turn excerpts, randomly removing intermediate turns, and assuming the last turn is an incorrect response. we adapt this turn masking approach to our cognitive distor-tion task to create dynamic conversation context in training, as described in section 3.2. ## discussion we explored the automatic identification of cognitive distortions in text-based exchanges between patients and therapists, focusing on the role of conversation context. we utilized multiple transformerbased classification architectures and proposed two methods for dynamically utilizing conversation context in training, random length and random mask. our results demonstrate that the inclusion of context improves cognitive distortion prediction performance for several distortion types, with the best performing architecture encoding the target message and context messages as a single input sequence to bert (bert-only). results also demonstrate that using random length for the context during training improves performance over using a fixed length context, for several distortion types. the performance of the context-aware models approaches the inter-rater agreement for a majority of the distortion types. bert-only with random length context identifies any distortion with relatively high performance at 0.73 f1; however, lower performance (@xmath0.5) is achieved in resolving specific distortion types (e.g. catastrophizing or jumping to conclusions). the error analysis suggests that at least a portion of the performance gains associated with the inclusion of context messages is attributable to the tendency for messages expressing cognitive distortions to cluster in time. this work presents context-aware classification approaches that improve performance in identifying cognitive distortions in text messages. the improved performance associated with the inclusion of context will benefit downstream clinical applications, including clinical decision-support systems, therapist training, and clinical research. in the community health setting, the adoption of new treatment modalities and technology for serious mental illness is hindered by the availability of training and expertise among community-based clinicians @xcite . the adoption of new interventions is resource intensive, and training and supervision for novel interventions may improve the adoption of new interventions, like texting @xcite . our work exploring the automatic identification of cognitive distortions could mediate the development of clinician training and support tools that improve the uniformity and quality of care and reduce required human resources, by flagging patient content that requires intervention. in terms of clinical research, this work may support the implementation of interventions that target cognitive distortions, assess the extent to which such interventions are effective in reducing distortion frequency, and improve understanding of the relationships between distorted thinking, symptom severity and mental status. this study is limited by the number of participating patients and therapists. text-based therapy conversations are likely heterogeneous and vary by patient-therapist dyad, patient clinical condition, and other factors. due to the size of the annotated corpus, the data set was split such that each patient appears both in the train and test partitions, although there is no overlap between the messages in the train and test partitions. additional work with an expanded data set is needed to assess the generalizability of the classifiers to a diverse patient population, including patients not represented in the training data. similar to prior cognitive distortion work @xcite , classification performance is limited by the challenge of manually annotating distortions, including the soft boundaries between distortion types. we are currently adding additional cognitive distortion type labels to the text-message corpus to include more fine-grained distortion categories that can be condensed into functionally related higher-level categories. the inclusion of additional cognitive distortion types and aggregation of individual distortion types into higher-level thought patterns may improve annotation consistency. as part of this annotation effort, we are expanding the annotation guidelines and providing additional annotator training to improve annotation detail and quality. this work investigates the use of preceding conversational turns as context for prediction. there are many other forms of context, and mechanisms for representing it, that may be considered in future work. with a sufficiently large corpus of text con-versations, it may be feasible to learn patient representations that capture important linguistic patterns, thinking styles, and other information relevant to characterizing thought patterns and mental state. the patient representations could take the form of learned patient embeddings, for example special patient-specific bert tokens. additional contextual information could include message metadata (e.g. time of day or time between responses) or patient demographics/attributes (e.g. age, gender, tech literacy, or diagnoses). models incorporating such information may add to our understanding of the contexts in which distortions occur and further improve automated methods to detect them. ## conclusions the improvements in performance shown in this work demonstrate that modeling conversational context is important for identifying cognitive distortions in text-based exchanges between patients and therapists. by identifying cognitive distortions in patient messages within the larger context of the conversation, the modeling better emulates the process mental health clinicians use to assess for distortions. distorted thinking in the patient messages tends to cluster in time, such that distortions are more likely to occur in context messages, if there are distortions in the target message @xcite . some of the improved performance associated with the inclusion of context is likely attributable to the model implicitly identifying distortions in the context messages. additionally, the inclusion of context also captures important cues in therapist messages for the presence of distorted thinking in patient messages. conversational context is likely to improve performance in identifying cognitive distortions, with implications for the development of decision support tools, and quantification of distortions in observational data.