Counseling-Style Reflection Generation Using Generative Pretrained Transformers with Augmented Context (2020)

Abstract: We introduce a counseling dialogue system that seeks to assist counselors while they are learning and refining their counseling skills. The system generates counselors’reflections – i.e., responses that reflect back on what the client has said given the dialogue history. Our method builds upon the new generative pretrained transformer architecture and enhances it with context augmentation techniques inspired by traditional strategies used during counselor training. Through a set of comparative experiments, we show that the system that incorporates these strategies performs better in the reflection generation task than a system that is just fine-tuned with counseling conversations. To confirm our findings, we present a human evaluation study that shows that our system generates naturally-looking reflections that are also stylistically and grammatically correct.

## introduction a recent survey on mental and behavioral healthcare showed that while there is an increasing need for counseling services, the available mental health workforce is barely coping with this demand. 1 an important reason behind this unmet need is that the training of counselors requires a lot of time and effort. typically, counselor training involves refining counseling skills through practice and feedback using role-play activities, simulated patients, or real patients, thus heavily relying on human supervision and interaction. in clinician training, feedback and coaching can significantly improve the post-training counselor proficiency @xcite . however, the standard way of providing systematic feedback relies on human coding of the counseling sessions. this process can take up to ten times as long as the duration of the session itself, and thus it does not scale up @xcite . previous work has focused on developing automatic tools for counseling evaluation and training tasks, including automatic coding (i.e., recognizing a counselor behavior) and forecasting (i.e., predicting the most appropriate behavior for the next counselor's utterance) @xcite @xcite . these tools aim to facilitate the evaluation of a counseling encounter and, to some extent, provide generic guidance during the conversation. although these systems help counselors by suggesting the timing of a certain counseling behavior, they do not offer any help on how to acomplish it. among the different skills to be learned by counselors, reflective listening has been shown to be an important skill related to positive therapeutic outcomes @xcite . reflective listening is a conversational strategy used by counselors to show that they understand their clients' perspectives, feelings, and values @xcite . during this process, the counselor listens to the client's statements and then makes a statement (reflection) that is a reasonable approximation of the meaning of what the client has said. thus, the main role of reflections is to keep the conversation focused on the client and to move the conversation forward. for example, considering the following utterance by the client, a counselor could make reflections (a) or (b) to show an understanding of the client's feelings and concerns. motivated by the importance of reflective listening skills and the significance of real-time feedback in the success of a counseling encounter, we envision our system as an automatic assistant that provides counselors with sample reflection language that is appropriate to the conversation context, thus helping counselors to acquire or improve reflective listening skills by emulating traditional psychotherapy training, but without the need of close human supervision. we present a reflection generation system that leverages state-of-the-art language models, and further improve it with context augmentation techniques inspired by traditional counselor training. specifically, we (1) identify previously used reflections from related sessions based on the current context, similar to how trainee counselors are exposed to several types of reflections on the same topic before they have to produce their own; and (2) we expand the content with synonyms for verbs and nouns, similar to how counselors are advised to use rephrasing strategies such as synonym rewording @xcite . we perform a domain adaptation on an additional counseling corpus containing a variety of counseling styles, and fine-tune our system on a corpus of successful counseling interactions with labels available. thus, it allows the system to benefit from successful counseling patterns derived from the cumulative experience of a large number of professionals. we conduct several comparative experiments, and perform evaluations using automatic metrics for language generation, including n-gram based, embedding-based and language diversity metrics. in addition, given the subjective nature of our task and the inability of automatic metrics to capture other relevant aspects of reflection generation, we conduct a human evaluation to assess the ability of our system to generate counseling reflections that are grammatically correct, fluent, and relevant to the conversation context. ## related work there have been significant efforts put in building automatic tools that provide support for mental and behavioral health. in particular, for dialogue-based counseling most of the existing work has focused on generating conversational agents that emulate the counselor in chat-bot like settings. for instance, @xcite built a system that extracts 5w1h (who, what, when, where, why, and how) information and user emotions @xcite to recognize what the user says, predict the conversation context and generate suitable responses based on utterance templates developed to encode three basic counseling techniques (paraphrasing, asking open questions, and reflecting feelings). a similar system is presented in @xcite , where authors first detect the user emotion and intention (e.g., greeting, self-disclosure, informing, questioning) and then extract the entities present in the utterance as well as related information (from an external knowledge base) to generate an appropriate response using language templates. while these studies have focused on the delivery of health interventions via conversational agents (i.e., virtual counselors), we seek to build an automatic dialogue generation system that can help training counselors to improve their everyday practice. this is in line with a recent study on the impact of technology in psychotherapy, which has identified the development of technologies for counselor's training and feedback and technologymediated treatment as important needs in this domain @xcite . initial work in this direction is presented in @xcite , where authors present a system that implements an artificial standardized client that interacts with the counselor and provides trainees with real-time feedback on their use of specific counseling skills by providing suggestions on the type of skills to use. following the same line of work, our goal is to aid counselors while training specific skills, more specifically reflective listening skills. however, different from previous work, we focus on presenting the counselor with automatically generated samples for potential reflections that can be used immediately in the conversation. finally, potential applications of our proposed system include supporting counselor training in counseling platforms such as talkspace 2 , which currently has over a million users and five thousand therapists, and crisis text line, 3 with 20 thousand counselors, handling over three thousand conversations a day, allowing users to connect with licensed therapists and to seek help via text messaging. the ability to automatically generate reflections given a conversation context can assist these counselors in formulating what they are going to say, thus improving the efficiency and quality of their reflections, with the final goal of increasing the number of people they can help and the effectiveness of their interaction on patient outcomes. ## model overview to build an automatic reflection generation system, we rely on the generative pretrained transformer 2 (gpt-2) architecture @xcite as a base model. gpt-2 is a state of the art transformer-based general purpose language model that has been found useful for dialogue generation tasks @xcite . our choice is motivated by its ability to produce language that closely emulates text written by humans @xcite . our model learns how to generate a counselor reflection using a gpt-2 architecture by operating entirely in a sequence-to-sequence way. in order to condition the generation on the counseling dialogue context and to generate reflections that are stylistically correct, we fine-tune the model with conversations in the counseling domain. below, we describe important elements of the model architecture related to the reflection generation task. input representation. the input sequence for the model consists of a counselor's utterance and a dialogue context including previous utterances from either the client or counselor. the window size of the dialogue context is set to five utterances, as a larger window size did not improve performance in preliminary experiments. embeddings. besides learning word and positional embeddings, we also learn type embeddings to indicate whether the current token is part of the utterance from the client, counselor, or the reflection response. we use a trainable embedding matrix to map each location or type into a vector with the same size as the token embeddings. separation tokens are also added to further delimit these elements in the dialogue. decoding details. the generator model consists of a transformer decoder with a similar structure to the decoder in @xcite but only keeping the self-attention blocks. during the decoding stage, we assume we only have access to the augmented input and dialogue context and not the response. at each time-step, the model chooses a token from the output distribution conditioned on the context and the previously decoded tokens. the chosen token will be added into the input in the next time-step. to generate more diverse and expressive reflections, we adopted the top-k random sampling method @xcite , where the model samples from the k options with the highest probabilities. ## counseling-style reflection generation our goal is not only to generate natural-looking text that is relevant to the prompt but also to resemble the language style that counselors use while generating reflections. thus, we extend the base model to incorporate two strategies that are commonly used by counselors while generating reflective statements. first, we consider a training scenario where trainees are first shown sample reflections made while discussing different behavioral change goals (e.g. smoking cessation or weight management). after they have been exposed to several types of reflections, trainees are usually asked to construct alternative reflections for a given scenario as a way to reinforce what they have learned. in this case, trainees might associate previous reflections with the same behavioral change target as potential examples to generate their own. we attempt to use the same strategy to improve our system's responses. thus, we devise a retrieval-based method to obtain a reflection to be used to expand the dialogue context. second, considering that counselors generate reflections using rephrasing strategies such as rewording with synonyms and verb tense changes, we design a content expansion method that augments the system input with verb and nouns synonyms. these methods are described in detail below. ## conclusion we presented a system based on a state of the art language model that generates counseling reflections based on the counselor-client dialogue context. we first conducted domain adaptation and subsequently fine-tuned the system with motivational interviewing conversations. we then improved the system by augmenting the dialogue context using retrieval and content expansion methods that implement actual strategies used by counselors while generating reflections. we conducted comparative experiments between systems implementing these strategies and demonstrated their effectiveness in generating improved reflections as measured by standard language generation metrics such as rouge as well as embeddingbased and diversity metrics. to further validate our models, we conducted a human evaluation study on the generated responses. the evaluation showed that humans scored our proposed systems higher than the baseline model on quality, relevance, and reflection-likeness. we believe that counselors could benefit from the proposed system by using the automatically generated reflections as reference while learning to formulate reflective statements.