Automatic Reflection Generation for Peer-to-Peer Counseling (2023)

Abstract: Online peer counseling platforms enable conversations between millions of people seeking and offering mental health support. Among counseling skills, reflective listening, i.e., capturing and returning to the client something the client has said, is important for positive therapeutic outcomes. We introduce a reflection generation system for online mental health support conversations leveraging GPT-3, a large language model. We compare few-shot learning against fine-tuning and assess the impact of the quality of training examples as measured by fluency, reflection resemblance, and overall preference. Fine-tuned GPT-3 generates responses that human evaluators rate as comparable in reflection quality to responses used for tuning. Models based on high-quality responses generate substantially better reflections than ones tuned on actual responses from a large online counseling service–and better reflections than the actual counselor responses. These results suggest the care needed in selecting examples for tuning generative models.

## introduction online mental health support platforms, from talkspace to 7 cups to crisis text line, are used by millions of users for expressing challenges and receiving peer support. these platforms can help improve access to mental health support, as such care remains a global challenge with workforce shortages and limited affordable options @xcite . helping counselors with feedback, suggestions, and training, for instance through machinein-the-loop writing systems @xcite , has the potential to aid counselors in improving the quality of their responses and in turn improve the effectiveness of these platforms @xcite . moreover, training counselors can require substantial time and effort. often, training incorporates didactic instruction and experiential exercises (e.g., ## related work there is a growing body of related work aimed at building automatic tools in the form of dialogue systems for online mental health support. one significant line of work has focused on delivery of psychotherapy interventions for which conversational agents serve as counselors. @xcite created a system recognizing what users say, predicting conversation context, and following users' feelings to generate responses based on templates designed for three counseling techniques (paraphrasing, asking open questions, and reflecting feelings). @xcite presented a similar system, generating appropriate responses using templates by identifying user emotion and intention and extracting entities and related information from a knowledge base. in contrast to these works, our system is intended to serve as a resource for counselors in creating responses to client messages. in line with our goal of augmenting counselors' everyday practice, related work has built technology for counselor training and feedback. such work has explored the creation of a dialogue observer that categorizes therapist and client motivational interviewing behavioral codes and forecasts codes for upcoming utterances to help guide conversation @xcite . other systems have used machine learning-based feedback for training with artificial standardized clients, providing real-time suggestions on skills to use @xcite . although they aid in the evaluation of counselor language, such tools are limited in providing easily implementable suggestions. our goal is to present the counselor with an actionable suggestion at a particular point in the conversation. generative applications in peer-to-peer counseling include empathic rewriting @xcite and motivational interviewing response rephrasing (welivita and pu, 2023), i.e., making sentence-level edits to increase empathy while maintaining conversation quality or increase motivational interviewing adherence respectively; an ai-based tool to empower peer counselors through automatic suggestion generation for a range of counseling strategies @xcite ; and related work in generating reflections using gpt-2 and motivational interviewing conversations @xcite . the latter reflects the most relevant work. shen et al. (2020) evaluated gpt-2 models' ability to generate reflections given dialogue history, exploring how augmenting input with reflections from similar conversations and content expansion impact quality of generated reflections. they found the gpt-2 models perform better than the baseline seq2seq model. based on automated metrics, models with context augmentation outperform the fine-tuned gpt-2 model; however, while the systems perform on-par or above reference reflections (ground truth), there appears little difference between the gpt-2 model and models leveraging additional context expansion strategies. shen et al. (2022) addresses a similar task but enhances generation by infusing commonsense and domainspecific knowledge. expanding on these works, we explore additional techniques including few-shot learning and evaluate the impact of the quality of examples for fine-tuning. ## discussion we introduced a system for generating reflections in peer-to-peer counseling and explored the impacts on generated output quality of few-shot learning and fine-tuning, fine-tuning on a larger training set, and fine-tuning on higher-quality instances. our human evaluation shows humans rate responses generated by a model fine-tuned on higher-quality examples as more reflection-like, and there is a significant difference between overall preference for human and computer-generated responses. there is no evidence to suggest fine-tuning gpt-3 results in better reflections than few-shot learning. our results highlight the care needed in selecting examples for fine-tuning a model to generate reflections. evidently, input that reveals the nature of the counselor-client dialogue context alone is not enough to produce high-quality model responses, but also reflections epitomizing effective responses. our results also suggest generative models tuned on high-quality reflections generate better reflections than the actual peer counselors. however, with respect to the difference in quality between actual counselor and reference reflections, it is important to recognize that the author knew the reference reflections were being evaluated given minimal context, whereas naturally, the actual counselors may not have such a focus on creating a well-crafted response for any given moment in conversation. the global burden of mental illness is significant @xcite . online mental health support platforms with peer counselors are a means of scaling up support, but the challenge of effective training remains. our work represents how natural language generation can be used to help support peer counselors. such a system with a machinein-the-loop approach can provide actionable suggestions to counselors and in turn potentially offer those seeking help more reflective support. our results also have potential implications for the wider context of online peer interactions, e.g., peer grading and customer support. many contexts require responses to be consistent with a particular style; our results indicate higher-quality examples are critical for fine-tuning. given we did not find evidence to suggest fine-tuning gpt-3 results in better quality reflections than few-shot learning, we suggest future work further explore the trade-off between quality and quantity of examples provided to orient models toward the domain of interest. future work could build a collaborative writing tool, e.g., @xcite , for reflections and study the extent to which counselors accept, modify, or abandon suggestions. future work could explore inclusion of an additional input for the counselor to provide conversational redirection. our approach could also be applied to build and evaluate generative systems for other clinical skills. ## limitations as a result of our decision to limit conversational context to most recent messages, sometimes actual counselor responses took into account more of the conversation than was captured in the prompt. in shen et al. (2020), the context window size was five utterances, and a larger window size did not improve performance in preliminary experiments. however, it may be worth further exploring how greater context could enable more complex reflection statements. current language models lack the ability to account for the broader context. another technical limitation of this work entails the use of only one type of large language model for the reflection task. the experiments in this work are targeted at comparing few-shot learning and fine-tuning as well as assessing the impact of the quality of examples provided for tuning, and so in the interest of narrowing the focus, this work lacks a comparison of the quality of responses using various large language models in the counseling setting. the evaluation criteria used in the human evaluation have their own limitations in that these dimensions do not necessarily reflect what is most therapeutically beneficial in the counseling setting and what offers the best experience for clients. the models were evaluated on their ability to generate fluent reflections and not on true therapeutic impact. moreover, given the low agreement among annotators, the criteria's limited specificity likely introduced ambiguity and different interpretations in rating responses. future work should consider having annotators go through a first round of annotation followed by discussion of disagreements with opportunity to clarify judgments and resolve different interpretations of the criteria, thus offering a means of potentially reducing disagreement for the subsequent annotation process. the reference reflections in this work were created by one of the authors, whose experience with counseling and motivational interviewing derives from over one hundred hours of training at a crisis hotline and textline service and experience through a fellowship developing and user testing a platform for nurses to practice and grow their motivational interviewing skills. therefore, these reflections may not be as clinically precise as are possible from a medical professional, and the diversity of reflections is inherently limited. additionally, this work examined one mental health support community; peer supporters of this counseling service receive more training than some online support groups, where members do not receive training, but substantially less training than suicide hotline volunteers may receive. ## ethics gpt-3 was trained on over 45 terabytes of data from the internet and books, and large volumes of data collected from online sources will inevitably contain biases. there may thus be inadvertent discrimination against subclasses of particular pro-tected groups. using generated responses as a source of guidance rather than using generative systems as the counselors themselves may be able to help balance the benefits and risks of using artificial intelligence in delicate mental health settings. it is critical such systems are not misused by companies seeking to maximize efficiency and minimize cost. such a tool cannot replace counselor training, as it remains critical for counselors to be able to adequately assess responses prior to using them, particularly so that if generated text is biased or careless, it is reviewed and discarded. thus, it is necessary counselors continue to receive sufficient training to ensure they can identify clearly inappropriate generated text. when such technology is introduced to counselors, its limitations should be clearly communicated and its use monitored. additionally, it is imperative deployment of and subsequent experimentation with such a tool is done only with informed consent of users of an online counseling service. importantly, we see such automated tools as a way of assisting online counselors, especially peer counselors, not as replacing humans. ashish sharma, inna w. lin, adam s. miner, david c. atkins, and tim althoff. 2021. towards facilitating empathic conversations in online mental health support: a reinforcement learning approach. in proceedings of the web conference 2021, www '21, page 194-205, new york, ny, usa. association for computing machinery. siqi shen, veronica perez-rosas, charles welch, soujanya poria, and rada mihalcea. 2022. knowledge enhanced reflection generation for counseling dialogues. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 3096-3107, dublin, ireland. association for computational linguistics. siqi shen, charles welch, rada mihalcea, and verónica pérez-rosas. 2020. counseling-style reflection generation using generative pretrained transformers with augmented context. in proceedings of the 21th annual meeting of the special interest group on discourse and dialogue, pages 10-20, 1st virtual meeting. association for computational linguistics. eric smith, orion hsu, rebecca qian, stephen roller, y-lan boureau, and jason weston. 2022. human evaluation of conversations is an open problem: comparing the sensitivity of various methods for evaluating dialogue agents. in proceedings of the 4th workshop on nlp for conversational ai, pages 77-97, dublin, ireland. association for computational linguistics. michael j. tanana, christina s. soma, vivek srikumar, david c. atkins, and zac e. imel. 2019. development and evaluation of clientbot: patient-like conversational agent to train basic counseling skills. journal of medical internet research, 21(7):e12529. anuradha welivita and pearl pu. 2023. boosting distress support dialogue responses with motivational interviewing strategy. in findings of the association for computational linguistics: acl 2023, pages 5411-5432, toronto, canada. association for computational linguistics. yizhe zhang, siqi sun, michel galley, yen-chun chen, chris brockett, xiang gao, jianfeng gao, jingjing liu, and bill dolan. 2020. dialogpt : large-scale generative pre-training for conversational response generation. in proceedings of the 58th annual meeting of the association for computational linguistics: system demonstrations, pages 270-278, online. association for computational linguistics. chujie zheng and minlie huang. 2021. exploring prompt-based few-shot learning for grounded dialog generation. arxiv preprint arxiv:2109.06513.