Analysis of Behavior Classification in Motivational Interviewing (2021)

Abstract: Analysis of client and therapist behavior in counseling sessions can provide helpful insights for assessing the quality of the session and consequently, the client’s behavioral outcome. In this paper, we study the automatic classification of standardized behavior codes (annotations) used for assessment of psychotherapy sessions in Motivational Interviewing (MI). We develop models and examine the classification of client behaviors throughout MI sessions, comparing the performance by models trained on large pretrained embeddings (RoBERTa) versus interpretable and expert-selected features (LIWC). Our best performing model using the pretrained RoBERTa embeddings beats the baseline model, achieving an F1 score of 0.66 in the subject-independent 3-class classification. Through statistical analysis on the classification results, we identify prominent LIWC features that may not have been captured by the model using pretrained embeddings. Although classification using LIWC features underperforms RoBERTa, our findings motivate the future direction of incorporating auxiliary tasks in the classification of MI codes.

## introduction motivational interviewing (mi) is a psychotherapy treatment style for resolving ambivalence toward a problem such as alcohol or substance abuse. mi approaches focus on eliciting clients' own intrinsic reasons for changing their behavior toward the desired outcome. mi commonly leverages a behavioral coding (annotation) system, motivational interviewing skills code (misc) @xcite , which human annotators follow for coding both client's and therapist's utterance-level intentions and behaviors. these codes have shown to be effective means of assessing the quality of the session, training therapists, and estimating clients' behavioral outcomes @xcite @xcite . due to the high cost and labor-intensive procedure of manually annotating utterance-level behaviors, existing efforts have worked on automatic coding of the mi behaviors. the client utterances throughout the mi session are categorized based on their expressed attitude toward change of behavior: (1) change talk (ct): willing to change, (2) sustain talk (st): resisting to change, and (3) follow/neutral (fn): other talk unrelated to change. an example conversation between a therapist (t) and a client (c) is shown below. in this paper, we study the quality of automatic mi coding models in an attempt to understand what distinguishes language patterns in change talk, sustain talk, and follow/neutral. we develop a system for classifying clients' utterancelevel mi codes by modeling the client's utterance and the preceding context history from both the client and the therapist. we compare the effectiveness and interpretability between contextualized pretrained embeddings and hand-crafted features, by training classifiers using (1) pretrained roberta embeddings @xcite , (2) an interpretable and dictionary-based feature set, linguistic inquiry word count (liwc) @xcite . our best-performing model outperforms the baseline model from previous work on the same dataset @xcite , reaching @xmath0.66 from @xmath1.63. in examining misclassifications by both models, we identify features that are significant across classes. our findings suggest that large pretrained embeddings like roberta, despite their high representation power, might not necessarily capture all the salient features that are important in distinguishing the classes. we identified prominent features that are statistically significant across classes on the entire dataset, as well as the misclassified samples. theses findings suggest that our systems might benefit from fine-tuning pretrained embeddings, adding auxiliary tasks (e.g sentiment classification), and better context modeling. ## data we use two clinical datasets @xcite collected in college campuses from real mi sessions with students having alcohol-related problems. the data consists of transcripts and audio recordings from the client-therapist in-session dialogues. the sessions are manually transcribed, and labelled per utterance using misc codes. the dataset includes 219 sessions for 219 clients, consisting of about 93k client and therapist utterances; the client-therapist distribution of utterances is 0.44-0.54. the dataset is highly imbalanced, with a class distribution of [0.13, 0.59, 0.28] for [sustain talk, follow/neutral, change talk]. in addition to the in-session text and speech data, the dataset consists of session-level measures regarding clients' behavioral changes toward the desired outcome. additional metadata includes session-level global metrics such as therapist empathy, mi spirit, and client engagement. ## results and discussions for training, we use a 5-fold subject-independent cross validation. 10% of the train data from each fold is randomly selected in stratified fashion, and held out as the validation set. we optimize the network using adamw @xcite , with a learning rate of 10 -4 and batch size of 32. we train our model for 25 epochs with early stopping after 10 epochs, and select the model with the highest macro f1 on the validation set. to handle class imbalance, we use a cross-entropy loss with a weight vector inversely proportional to the number of samples in each class. the gru hidden dimension is 256 and 32 when running on roberta and liwc representations, respectively. we compare our work to the best performing model from previous work @xcite , trained on the same dataset and under the same evaluation protocol. briefly, this baseline model differs from our current model in several aspects: bert embeddings were used as input; the representation vector for the current client utterance is fed into a linear layer. the client and therapist utterances within the context window are separated, mean-pooled and fed individually to two different linear layers. the output encodings from the three linear layers are merged and fed into another linear layer before being passed to the classification layer. we perform statistical analysis to identify prominent liwc features across pairs of classes, as well as misclassified samples from each classifier. since the classifiers encode context, we incorporate the context in the statistical analysis by averaging the feature vectors along utterances within the input window. ## conclusion we developed models for the classification of clients' mi codes. we experimented with pretrained roberta embeddings and interpretable liwc features as our model inputs, where the roberta model outperformed the baseline from previous work, reaching @xmath2.66. through statistical analysis, we investigated prominent liwc features that are significantly different across pairs of classes. we further looked into misclassified samples across the classifiers, and identified prominent features that may have not been captured by the roberta model. this finding motivates the use of auxiliary tasks like sentiment and affect prediction, in addition to fine-tuning the model with domain-specific data and better context modeling. with this work, we aim to develop systems for enhancing effective communication in mi, which can potentially generalize to other types of therapy approaches. identifying patterns of change language can lead to mi strategies that will assist clinicians with treatment, while facilitating efficient means for training new therapists. these steps contribute to the long-term goal of providing cost-and time-effective evaluation of treatment fidelity, education of new therapists, and ultimately broadening access to lower-cost clinical resources for the general population.