Psychotherapy is Not One Thing: Simultaneous Modeling of Different Therapeutic Approaches (2022)

Abstract: There are many different forms of psychotherapy. Itemized inventories of psychotherapeutic interventions provide a mechanism for evaluating the quality of care received by clients and for conducting research on how psychotherapy helps. However, evaluations such as these are slow, expensive, and are rarely used outside of well-funded research studies. Natural language processing research has progressed to allow automating such tasks. Yet, NLP work in this area has been restricted to evaluating a single approach to treatment, when prior research indicates therapists used a wide variety of interventions with their clients, often in the same session. In this paper, we frame this scenario as a multi-label classification task, and develop a group of models aimed at predicting a wide variety of therapist talk-turn level orientations. Our models achieve F1 macro scores of 0.5, with the class F1 ranging from 0.36 to 0.67. We present analyses which offer insights into the capability of such models to capture psychotherapy approaches, and which may complement human judgment.

## introduction a typical psychotherapy session involves a clienttherapist dialog with the aim of diagnosing and assuaging a client's mental health condition. psychotherapists, generally, rely on certain approaches (e.g., cognitive behavioral or interpersonal therapy) and interventions differ across these approaches. 1 . for example, a therapist might focus on a client's interpersonal relationships, their emotions, or help develop behavioral activities designed to reduce symptoms (or all of the above). a key goal of psychotherapy research is to categorize such approaches and study them to determine the effectiveness of each approach in any given scenario. we refer to this process of categorizing and detecting approaches based on an overarching theory as 'evaluation'. in this paper, we study an application of natural language processing (nlp) to mental health, and focus on therapists' approach to psychotherapy @xcite . past nlp research has developed tools for evaluating specific types of interventions like motivational interviewing @xcite or cognitive behavioral therapy @xcite . however, psychotherapists differ from each other in the approaches they take. furthermore, they can also vary in the interventions they use within and between sessions. the lines of work mentioned before assume that a session is comprised of exactly one approach, and consequently do not attempt to automatically evaluate different psychotherapy approaches that may coexist in the same session. mccarthy and @xcite proposed one multiple-approach evaluation methodology-the multitheoretical list of therapeutic interventions (multi), which is a list of 60 interventions @xcite against which a psychotherapy session as a whole is evaluated post-session. the multi items are grouped into eight approaches. note the multi is a session-level measure and thereby limited in specificity because it does not record therapist language that informs a given item's presence. @xcite extend the scheme to the evaluation of therapist monologues, talk-turn by talk-turn, in addition to the session-level evaluation. such a scheme provides additional detail over time in a session. evaluating sessions with the multi requires a certain amount of time to be set aside postsession. evaluating talk-turns manually for every session would be even more onerous and inefficient. this calls for a better automatic/semi-automatic method(s) to evaluate talk-turns. these methods serve two advantages: i) reducing the amount of effort required in manual classification for research and quality assurance, and ii) creating applications to analyze approaches deemed helpful on out-ofsession platforms (e.g., social media). to that end, we present a neural machine learning model which aims to automate talk-turn level approach annotation. the task is set up in the following fashion: given a therapist input talk-turn, does the input (or part of the input) correspond to one or more approaches. a talk-turn might only represent one approach, or might have different parts that correspond to different approaches. it is also possible that a therapist talk-turn does not fall within a specific therapeutic approach (e.g., minimal encouragers, small talk, etc.). examples are shown in the state-of-the-art in natural language processing (nlp) has seen significant improvements with the advent of transformer-based models @xcite . in this paper, we show the performance of one such pre-trained transformer based language model on three paradigms, and experiment with changing context windows. our models achieve around 0.5 f1 macro scores with the class f1 ranging from 0.36 to 0.67. our analyses reveal that while our models mispredict on certain talk-turns during a session, they capture the dominant approaches when viewed from a session-level perspective. furthermore, we show that certain decisions rely on inter-session context, and even common-sense knowledge which sets up a challenge for current models. each subscale was defined by a psychotherapeutic theoretical orientation. we describe each subscale briefly here. data source. psychotherapy audio data was collected from a university counseling center at large public school in the western united states. there were 243 unique sessions transcribed, some of which were annotated more than once, totaling to 473 sessions. these sessions were annotated using a talk-turn level version of the multi-30 (caperton, 2021). coding procedure and reliability. seven graduate students in mental health fields annotated session content for their varying use of theoretical interventions. each coder received approximately 18 hours of training during in-person meetings and practiced coding sessions for an additional 36 hours before annotating session data used in this study. to minimize coder drift over time, case example talk-turns approach(es) okay let's set you up with an appointment. no code non-approach so, all of us trainees get to have a break as well. no code you're scared. process-experiential single approach i definitely notice a lot of progress that you've made. common factors unfortunately, it's very normal. but i want you to continue practicing that exercise. coders met together with their team leader every two weeks to discuss difficult talk-turns, items, and areas of disagreement. coders were tasked with identifying the presence or absence of theory-derived content in therapists' language at every therapist talk-turn (i.e., a string of words or statements uninterrupted by client speech). a given talk-turn could be identified with one, multiple, or no interventions. of the 243 unique sessions, 102 were annotated by multiple coders, resulting in 270 codings for interrater analysis. the statement-level interrater reliability of the eight theoretical orientations (subscales) was calculated using cohen's kappa. kappa was calculated for every possible coder pair who rated the same session and weighted according to the number of comparisons. subscale kappa scores ranged from .37 ('fair' reliability; @xcite ) to .63 ('substantial'). the dataset was split by client randomly into train/dev/test sets containing 70%, 15% and 15% of the clients respectively. the splits contain 338, 66, and 76 sessions respectively containing 74k, 14k, and 17k talk-turns in total. dataset statistics for the training split are presented in ## models while we want to model the eight subscales (plus the 'no code' class) conventionally used in literature, we deviate from these eight classes for the implementation. the behavioral, cognitive, and dialectical-behavioral subscales contain overlapping items (e.g., items 1 and 10 are shared by all three subscales). we break these subscales into four categories such that each of these categories contains mutually exclusive items. note that the other subscales(psychodynamic, interpersonal, etc.) remain the same. hence, in total, we obtain ten modified model classes (including the 'no code' class). we refer the reader to tables 7 and 8 in appendix a for further details on the breakdown. this method can aid downstream analysis by allowing credit/blame assessment on a smaller set of items. in our setup, each therapist talk-turn u i has a corresponding binary label vector y i . the binary label vector is ten dimensional, one decision each for the nine model classes (i.e., modified subscales) and one additional class indicating the absence of any code (nc). for all our experiments, we consider the roberta-base @xcite model as the language model of choice. this model takes in a talk-turn u i as input to produce contextual representations for its words. we take the pooler output of these contextual representations which gives us a vector representation h i for the talk-turn. we consider three modeling paradigms for our experiments. stand-alone (sa) model. this model is the vanilla multi-label classifier. talk-turn representations are passed through a linear layer with the number of output nodes equal to the model classes. the result is passed through a sigmoid layer resulting in a vector of presence probabilities for each label ŷi . that is for inference, a probability of 0.5 or above indicates label presence for a particular class. pipeline model. a heavily imbalanced dataset can hinder model performance for the underrepresented categories. as seen in if this binary classifier predicts that the talk-turn supports at least one orientation, then the talk-turn is given to a multi-label model to predict over the nine model classes. the multi-label model will be similar to the one mentioned in the stand-alone model, except nine classes are considered since predicting a "no code" would be redundant. the multi-label model has the flexibility, nonetheless, to predict an absence of orientation by predicting that none of the codes are present (i.e., a zero vector). note that two separate roberta models are used for the binary and the multi-label classifiers. multi-task model. the pipeline model trains two separate roberta models -one for the binary classifer and one for the multi-label model. a major drawback of this system is that training two roberta models is computationally expensive and memory-intensive. an alternative method is to share the roberta layer between the two tasks and have two separate linear layers for the respective binary and multi-label classification. this strategy of multi-task or joint learning has shown to be of promise in literature @xcite and allows for better shared representation. the losses for both the tasks are combined as a weighted sum for learning. we consider two variants of the model based on the number of output classes for the multi-label classifier. the multitask 10 variant considers all the classes including "no code" while multitask 9 excludes the "no code" class. the inference is identical to the pipeline model. the multi-task and stand-alone model paradigm can be thought of as fairly similar architectures. however, the multi-task model assigns a higher loss weight to the binary classifier, uses a different optimization metric and utilizes a pipelined inference approach as opposed to the one-shot prediction by the stand-alone model. so far, we explained that we break the conventional eight subscales into nine which have mutually exclusive items. while this approach allows us for better analysis, it is essential to present performance on the original theoretical subscales. to that end, we aggregate binary vector model predictions to the conventional eight multi subscales during evaluation. the output of the model, a ten-dimensional vector, will be mapped to a nine-dimensional vector(eight subscales plus 'no code'). we use these nine-dimensional vectors to perform model evaluation. ## analysis in this section, we present analyses on the development set. we choose the best performing multi-task 9 model for our analysis. do our models capture the global prevalence of approaches? the multi, to begin with, was intended to capture approaches at the session level. we investigate whether our models replicate the trends at a session-level. the comparative analysis for a randomly chosen session is shown in figure 1 . we see that despite making mistakes locally, the model captures approaches over therapist talk-turns. in this case, we see that the therapist scarcely uses a psychodynamic or interpersonal intervention and the model prediction shows similar behavior. on the other hand, the other subscale interventions are used almost uniformly over the length of the session. the model again captures this pattern. which categories are confused with each other? figure 2a presents which categories tend to cooccur with each other. we observe a category process-experiential (pe) co-occurs with person-centered (pc) almost every third instance. similarly, psychodynamic (pd) approach almost always co-occurs with process-experiential (pe). note that this is not commutative, i.e., pe co-occurs with pd about every fourth instance. figure 2b shows the same, however, between gold labels and model prediction. here we ask the question: for a certain category that exists in the gold data, what are the categories predicted by the model? figures 2a and 2b should be identical if our model is ideal. studying these figures in conjunction, gives us an idea of where the model confuses predictions the most. for example, a lot of cognitive (ct) instances get misclassified as person-centered (pc), a trend which is not reflected in figure 2a . we also observe that psychodynamic (pd) items get significantly mispredicted as process-experiential (pe). a large number of approach-labeled instances get classified as 'no code'. we expected this observa ## qualitative analysis f1 scores and cohen's kappa scores cannot be compared directly. we analyze some model error examples to assess examples in a fair manner. we selected 22 examples at random with the constraint of selecting different combinations of labels. out of the 22 examples chosen, five were ones which had an 'nc' gold label and a non-'nc' model prediction, while five had the opposite. the remaining twelve examples were mis-predictions between approach classes. of the twelve, four were cases in which the talk-turn had a single gold approach and a single model prediction which did not match, while four each were cases in which there were multiple gold approaches but a single model predicted approach, and vice-versa. we made sure that the cases were diverse. we present five of these examples. we consider the best mutlitask 9 model which is trained on just the therapist talk-turn (va) for this analysis. example 1 "interaction with your ex, like that's better for you" human annotation: nc model prediction: ip here the human assessed that the talk-turn was not structured or specific enough to earn a code, despite the presence of interpersonal content. however, the model identified interpersonal language which may or may not be linked to client distress. in this case, the human seems to have been more conservative than the model in applying a code. example 2 "and did you journal? or keep a log?" human annotation: bt, ct, dbt model prediction: nc here, journaling and log-keeping likely refers to reviewing homework, so the annotator marked an item 10. this item, subsequently, maps onto three speaker talk-turn gold sa pipeline multitask 9 client okay, sounds good, thank you. ----therapist yeah, so i just want to check in again, see how you're feeling in the room. pd,pei pei pei pei,pc client um i still feel fine, um, yeah i feel pretty good i guess. ----therapist okay, and that's also okay if you don't feel good, if you feel anxious. i still feel a little anxious as we're getting to know each other. pei, pc, cf pei, cf pei, cf pei client yeah. ----therapist i just want to acknowledge that we have about twenty minutes left in our session. i'm curious is there anything you want to bring up, anywhere you want to start exploring? cf cf cf cf human annotation: pc, bt, ct, dbt model prediction: pc the therapist clearly asks about the client's experience, justifying a pc label. the phrase "last week of paying attention to your thoughts", however, sounds like a homework check-in (item 10). similar to example 2, item 10 triggers three subscales and the human annotation of bt, ct, and dbt subscales seems appropriate and highlights a case which the model does not capture. this is an interesting case of annotation based on common-sense knowledge with which nlp models still struggle. we should emphasize again that the humans do not annotate eight subscales directly; rather, they annotate based on the 30-item inventory. for instance, in example 2, the human annotator does not annotate the bt, ct, and dbt categories individually. they, instead, might have just annotated a single item (item 10) which maps to the three subscales. hence, it should not be misconstrued that the human has over-labeled in that scenario. in general, after analyzing the 22 examples, we find that in many such erroneous cases, prior intrasession (short or long range) and even inter-session contextual information might be relevant to determine the correct context. we leave this as a possible direction for future research. ## related work to the best of our knowledge, this will be the first work to automate the multi subscale assignment of therapist talk-turns. ## conclusion the expanding awareness and need for mental health improvement demands the ubiquity of such resources. therapeutic evaluation becomes increasingly important as more people leverage mental health resources. we consider one such evaluation strategy -a talk-turn level adaptation of the multi -which evaluates therapist orientations. a major downside of such strategies remains their time-intensive nature. in this paper, we propose using pre-trained language models, which have proven to be high performance systems, to automate this evaluation. we experiment across three modeling paradigms using a pre-trained language model -roberta. in addition, we show substantial analyses to understand the results. our experiments are encouraging, however, we stress that substantial gaps in performance remain. we see this work as a significant stepping stone towards improving therapeutic feedback using nlp tools.