PAIR : Prompt-Aware marg I n Ranking for Counselor Reflection Scoring in Motivational Interviewing (2022)

Abstract: Counselor reflection is a core verbal skill used by mental health counselors to express understanding and affirmation of the client’s experience and concerns. In this paper, we propose a system for the analysis of counselor reflections. Specifically, our system takes as input one dialog turn containing a client prompt and a counselor response, and outputs a score indicating the level of reflection in the counselor response. We compile a dataset consisting of different levels of reflective listening skills, and propose the Prompt-Aware margIn Ranking (PAIR) framework that contrasts positive and negative prompt and response pairs using specially designed multi-gap and prompt-aware margin ranking losses. Through empirical evaluations and deployment of our system in a real-life educational environment, we show that our analysis model outperforms several baselines on different metrics, and can be used to provide useful feedback to counseling trainees.

## introduction counselor training is an expensive and time consuming process due to the extensive expert supervision involved @xcite . current strategies for counselor training usually rely on either role playing or monitoring and recording live video interactions, which are then manually evaluated to provide constructive feedback, thus limiting the opportunities for training counselors to practice and receive timely evaluative feedback. while several promising approaches have been proposed to automatically provide evaluative feedback to counselors @xcite , providing detailed feedback in real time remains a challenge. this is particularly the case in educational settings, where counseling trainees could benefit from a supportive learning environment that allows them to make mistakes and learn at their own pace while acquiring counseling skills. seeking to address this need, we introduce the task of scoring the language of counseling trainees when learning to formulate responses to clients' statements. we focus on responses containing counseling reflections, e.g., the ability to understand and reflect on what the client is saying. our goal is to create a computational model for scoring counselor reflections by leveraging existing counseling behavioral annotation schemes and learningto-rank approaches. we believe that developing an natural language processing (nlp) model capable of assessing the quality of verbal behavior observed in counselor language can improve the quality and efficiency of motivational interviewing (mi) training by allowing counselors to practice in real-time their reflective listening skills, and providing them with immediate feedback. to build our scoring system, we compile a new dataset of counseling reflections using expert and non-expert annotations. the dataset consists of pairs of client prompts, i.e., situations from a client that prompt a reflective statement, and counselor responses showing different levels of counseling skill, as shown in figure 1 . we also introduce pair (prompt-aware margin ranking), a novel margin ranking-based approach that can output a continuous score learned from discrete annotations of counseling reflections. we conduct a set of experiments showing that our model is able to learn the correct ranking of counseling responses. in addition to testing the model on the collected data, we deploy our models in a real world education setting with graduate counseling students, and conduct quantitative and qualitative evaluations showing that our system is a viable alternative to manual human feedback. our main contributions include: (1) the formulation of the reflection scoring problem and a counseling dataset for this task; (2) a novel contrastive learning-inspired framework pair (prompt-aware margin ranking) for automatically scoring the quality of a reflection statement; (3) quantitative and qualitative assessments of our model on the annotated dataset and through in-the-wild deployment and feedback. ## related work automated analysis and evaluation of verbal strategies used in mental health conversations has emerged as a promising intersection of psychotherapy and nlp @xcite . work has been done to measure the fidelity of treatment via behavioral coding or analysis of counselors' language @xcite @xcite , and also to evaluate empathy and verbal mimicry expressed by counselors @xcite . more recently, dialog-based systems have been explored to assist the development of basic counseling skills. @xcite developed a patient-like conversational agent that interacts with counselors while practicing open questions and reflections and categorizes their responses to show percentages of questions and reflections used during the interaction. the task proposed in our work is related to behavioral coding, but our focus is on detecting the overall quality of a specific verbal behavior (a reflection), rather than identifying their type. our scoring system can also be used as a component in larger systems for writing or rewriting counselor utterances, similar to the framework in @xcite . finally, @xcite show that high reflection frequency is associated with high-quality, demonstrating the importance of practicing reflective be-havior in mi. ## motivational interviewing reflection dataset to build our reflection scoring models, we compile a dataset consisting of brief interactions between counselors and clients portraying different levels of reflective listening skills. each interaction is in english and includes a client prompt, i.e., a client's statement that is usually given to the counseling trainee, paired with counseling responses portraying different levels of reflections skill, i.e., low quality, medium quality, and high quality. we build the dataset using both expert and crowd-sourced annotators and also leverage conversational data from an mi dataset to obtain additional prompt-response pairs from conversations snippets containing reflections. ## prompt-aware margin ranking (pair) our reflection scoring task consists of assigning a reflection quality score s between [0, 1] to a interaction pair containing a client prompt p and a candidate reflection by a counselor r. while this task can be viewed as regression, obtaining ground truth labels is expensive and noisy, even for expert annotators. instead, we formulate the scoring problem as a learning-to-rank task, in which the training data labels are pairwise relevance levels based on the ground truth reflection, i.e, nr, sr and cr @xcite . we develop a scoring framework inspired by contrastive and metric learning strategies, where binary contrastive estimations are computed between examples for consecutive reflection quality levels. our model combines two metric-learning based objectives that enforce the correct ranking along with prompt relevance. figure 2 shows an overview of our training process. for our model backbone, we adopt a transformerbased encoder architecture @xcite , but our learning objectives can be extended to other neural models, such as recurrent neural networks. we use the roberta variant of the transformerbased encoder @xcite . we implement a simple cross encoder that takes the concatenated sequence of a prompt and a response pair as input. since this design choice allows to directly model the interaction of prompt and response tokens, we classify our main model as a cross encoder-based model, following the characterization of encoder provided by @xcite . multi-level margin ranking objective. we introduce a margin ranking loss term to ensure a distance gap between quality levels of reflections taking inspiration from @xcite . the ranking objective uses a margin parameter µ or 2µ, depending on the examples being compared in the loss term, where µ corresponds to the "gap" in scores between each neighboring quality level pair. next, we use µ when the quality gap is within one level i.e, distinguishing between medium quality and high quality pairs (sr, cr) or low quality and medium quality pairs (nr, sr), and 2µ when the gap is within two levels i.e, low quality and high quality pairs (nr,cr). the loss is calculated using the equation below, where p is the client prompt and r cr , r sr , r n r respectively denote cr, sr, and nr responses to p. prompt-aware margin ranking objective. in preliminary experiments using l gap we find that the model tended to ignore the client prompt when making predictions. this can result in incorrect scoring for cases where responses are not related to the client prompt but do follow mi style. to avoid this problem, we design a prompt-aware objective to penalize the model against this scenario. to provide examples of such cases to the model, we build an additional set of pairs by sampling cr and sr responses (m cr , m sr ) from the training batch and matching them with random prompts from the same batch (p), with the condition that the matched prompts must be different from the original pairs. then, we treat the constructed pairs of prompt and mismatched responses as low-quality examples (nr). we thus formulate the following prompt-aware ranking objective, where r cr , r sr , µ, and 2µ are defined as in l gap , while m cr , m sr refer to the mismatched responses. our scoring function is the transformer encoder model, followed by a pooling layer and a sigmoid activation. for our final model, we combine the l gap and l prompt objectives with equal weights: 5 experiments ## results for tables 3 and 4 , the recall@1 results are identical, indicating that even after randomly matched responses are added, all the models are able to correctly identify response with the highest reflection level (complex reflection). moreover, we note that for the naive classifier models spearman correla-tion scores higher than pearson correlation, likely due to the fact that spearman correlation measures monotonic relationships and the naive classifier predictions contain frequent ties, since unlike other models the classifier outputs discrete integers. baseline comparison. we compare the performance of our model against several baselines. when the models are tested against data without randomly matched responses ( however, in ## ablation study we study the effects of the prompt-aware learning objective by conducting a set of ablation experiments. specifically, we evaluate our scorer models and baselines with or without the prompt-aware learning objective. in when we test our models on a dataset augmented with randomly-matched negative responses, we find that the prompt-aware loss leads to improved performance on data that includes random responses. in ## user study in addition to testing our model on annotated data, we also deploy our final model (pair) in a real-life education setting -a graduate-level mi training course taught by one of the authors of this paper. we collaborated with mi experts at the university of michigan school of public health and implemented a web application for a graduate-level mi training course 1 . we develop and evaluate a web based application that uses the pair model to provide real-time scoring feedback to students while learning to create reflective responses to a given client prompt. we plan to make the system website available for demonstration. ## ethics statement privacy and data protection. we ensure that users of our systems are informed of our data collection practices. moreover, we conduct data cleaning and anonymization to remove any personal or sensitive information from the collected data. bias and impact of the model since our model provides feedback to human behavior, there is a risk that the model may have negative consequences. for instance, biases or artifacts contained in expert annotation can be encoded in such models and may exert influence on students who are trying to mimic or learn from the model. although we have not detected any such examples or trends during the model testing and deployment, we plan to further study and evaluate the impact of our models as future work. ## limitations our work has several limitations, which we aim to address in our future work. first, our model is finetuned using only synthetic data, annotated by a group of experts, for a prede-fined collection of simulated client prompts. we included real counseling data in our framework through pretraining, but this data is not directly used in the supervised training or downstream evaluation of the model. although we evaluate our model in the wild through system deployment and user evaluation, we hope to further understand and bridge the gap between models trained using observed data and models trained using synthetic data. second, our models rely on linguistic and reflection style to score reflections but do not account for conversational aspects such as empathy, which is also considered an important counseling strategy while generating reflections. in one of our preliminary experiments where we compared our reflection scores and empathy levels computed by sharma et al's empathy model @xcite , we observed that emotional reactions to mental health posts tend to have low reflection levels, indicating that counseling reflection is related but not identical to empathy in counseling. finally, the reflection scoring system proposed in this paper mainly provides numerical scoring feedback to trainees along with good reflection feedback that has been designed by the course instructor. we are working on expanding the system to include models for different types of feedback, beyond mere reflection level scoring. for instance, by exploring generative models to automatically create counselor responses, reference responses can be provided for students, even when annotated ground truth is unavailable. additionally, rewriting models can provide more valuable feedback by presenting improved versions of students' own responses. ## conclusion and future work in this work, we introduced the task of reflection scoring and developed a prompt-aware margin ranking approach, pair, to tackle this problem. our model learns to predict continuous scores from discrete label training data and outperforms simple baselines on several metrics, and we showed its deployment in an educational setting with real students and instructors. we plan to extend our model to incorporate diverse information that can assist counselors in understanding their clients, such as dialog context, client background, or medical knowledge. we make our data available at https: //lit.eecs.umich.edu/downloads. html#pair.