Towards Low-Resource Real-Time Assessment of Empathy in Counselling (2021)

Abstract: Gauging therapist empathy in counselling is an important component of understanding counselling quality. While session-level empathy assessment based on machine learning has been investigated extensively, it relies on relatively large amounts of well-annotated dialogue data, and real-time evaluation has been overlooked in the past. In this paper, we focus on the task of low-resource utterance-level binary empathy assessment. We train deep learning models on heuristically constructed empathy vs. non-empathy contrast in general conversations, and apply the models directly to therapeutic dialogues, assuming correlation between empathy manifested in those two domains. We show that such training yields poor performance in general, probe its causes, and examine the actual effect of learning from empathy contrast in general conversation.

## introduction as a pillar of psychotherapy, empathy is crucial to effective counselling, owing to its importance in building counsellor foot_0 -client rapport @xcite that can enable more effective interventions and better outcomes @xcite . in particular, "listening with empathy" is considered a guiding principle @xcite for motivational interviewing @xcite (mi), a psychotherapeutic approach widely adopted to elicit positive behaviour change by evoking motivation from clients. gauging counsellor-side empathy is, therefore, essential to assessing mi integrity @xcite . empathy assessment for mi has conventionally been conducted manually by trained annotators, which requires extensive annotator training and transcript review. since such a time-consuming and costly setup is difficult to scale up, recent years have seen attempts of automating the process with machine learning, including transcriptbased @xcite @xcite , speech-based @xcite , and multimodal @xcite methods. those works are, however, limited in that 1) therapist empathy is only assessed at session-level rather than utterance-level; 2) classical machine learning with heuristic feature engineering is used, while recent deep-learning frameworks have not been utilised for this purpose; 3) the machine-learning-based approaches all assume access to privately-owned sizeable corpora of therapeutic dialogues with empathy annotation at session level, but in reality such well-annotated data are often very limited, even more so at utterance level; and 4) the link between empathy manifested in general conversation and in mi counselling remains unexplored. in this work, we make the first attempt (to the best of our knowledge) at addressing those limitations while probing the correlation between empathy manifestations in different domains. specifically, we employ pre-trained language models such as bert @xcite for text-based binary classification of utterance-level therapist empathy, optionally taking the conversation context as input. we consider any counsellor utterance to be empathetic if it shows empathy, and non-empathetic if it does not (ranging from neutral to apathetic). our models have no access to counselling conversations during their training and validation, as we experiment with learning from contrast of empathy vs. non-empathy in out-of-domain (ood) training data. to that end, we leverage publicly available datasets of general conversations with heuristic empathy labels @xcite for ood training, investigating the connections between general-conversational empathy and therapeutic empathy, as illustrated in figure 1 . to benchmark the models, we manually annotated utterance-level empathy for a subset of transcribed high-vs. low-quality counselling demonstrations (pérez-rosas et al., 2019) that are publicly available. we also build unsupervised baselines for the task by a) formulating binary empathy classification as natural language inference (nli), as proposed by @xcite , and b) tackling the surrogate task of client-counsellor agreement via nli, under the assumption that an empathetic reply from the counsellor tends to show accordance with the client utterance in the preceding turn. our experiments show that models trained on ood empathy contrast are not sufficiently accurate predictors of mi empathy/non-empathy, even though the benefit of such training can be observed when compared to training on ood data without empathy contrast. upon probing, we argue that more fine-grained (e.g. sentence-level) empathy annotation and prediction could yield better results. ## data we leverage foot_2 two types of data: general conversations and transcripts of mi demonstration videos. we define an utterance as everything said by an interlocutor in their turn in a 2-person conversation, which is the most widely used definition of utterance in the literature of deep-learning-based conversational intelligence. this differs from some utterance definitions in psychotherapy. for example, an "utterance" in this work is identical to a "volley" as defined in the motivational interviewing skill code (misc) @xcite , while an "utterance" in misc is "a complete thought" that "ends either when one thought is completed or a new thought begins with the same speaker, or by an utterance from the other speaker". ## binary empathy classification in this section, we first define the task of binary empathy classification, then lay out the out-of-domain empathy contrast strategy behind our supervised models for the task, and finally describe our unsupervised baselines driven by nli. ## clinical application & impact the motivation for this work was to minimise the annotation effort needed for training an utterance-level classifier of therapeutic empathy/non-empathy, based on the assumption that 1) pre-trained language models can be finetuned to distinguish between empathy and nonempathy in general conversations, and 2) the finetuned model can be leveraged to directly predict therapeutic empathy/non-empathy. figure 3 : test set performances (in mcc) of all ood models. the first subplot on the left shows the test set performances of the 1-and 2-utterance ood models trained on data with empathy contrast, and the second shows the test set performances of the 1-and 2utterance models trained on data without empathy contrast. as explained in figure 2 , each ood pair (e.g. r/h vs. r/cc (1) / ( 2 )) corresponds to 5 groups of randomly sampled {train, dev, test} data and hence 5 trained models. thus, the model trained on the training data of a group has a test set score associated with the test data of the group. therefore, the value of each rectangular bar indicates the mean of the test set scores of the 5 models from the same ood pair, and the error bar shows +/-one standard deviation from the mean. our results, for the most part, show that this simple ood training approach did not sufficiently perform accurate classification, which limits its application in clinical settings. compared to supervised learning of session-level empathy on sizeable corpora of well-annotated therapeutic conversations @xcite , the task of utterance-level empathy classification with no in-domain training is more challenging and the models unsurprisingly fared worse. as discussed, the coarse, heuristic empathy labelling for the utterances in the training data and the domain gap between general conversation and therapeutic dialogue may have contributed considerably to the sub-optimal performance. nevertheless, we believe that this work is a meaningful step towards low-resource real-time assessment of empathy in counselling, and that the idea of utilising pre-trained language models for low-resource scenarios related to clinical psychology is still relevant. with smoothed domain gaps and more fine-grained annotation, future work can still use pre-trained language models to leverage parallels between empathy manifestations in general conversation and therapeutic dialogue. for instance, knowledge of empathy vs. non-empathy learned from well-annotated general conversations can serve as a bootstrapping step for empathy vs. non-empathy training on a minimal amount of wellannotated therapeutic conversations, since there can be a small to modest amount of therapeutic dialogue data available for a specialised domain instead of no data at all, which can take advantage of ood empathy knowledge as a starting point for in-domain fine-tuning and thus maximise the benefit of ood empathy training. ## conclusion we find that our models trained to learn from empathy vs. non-empathy contrast in general conversation (i.e. out-of-domain w.r.t. counselling) are generally not reliable predictors of empathy/nonempathy in motivational interviewing. upon probing, we observe that ood empathy contrast learning is still marginally better than ood learning without empathy contrast, particularly when more conversation context is available. in future work, we plan to investigate more finegrained empathy annotation and prediction, such as at sentence level, where we expect less noise and more accurate predictions. in addition, we will explore few-shot methods for the empathy classification task with out-of-domain empathy contrast training as a bootstrapping step.