Empathetic Response Generation for Distress Support (2023)

Abstract: AI-driven chatbots are seen as an attractive solution to support people undergoing emotional distress. One of the main components of such a chatbot is the ability to empathize with the user. But a significant limitation in achieving this goal is the lack of a large dialogue dataset containing empathetic support for those undergoing distress. In this work, we curate a large-scale dialogue dataset that contains ≈1.3M peer support dialogues spanning across more than 4K distress-related topics. We analyze the empathetic characteristics of this dataset using statistical and visual means. To demonstrate the utility of this dataset, we train four baseline neural dialogue models that can respond empathetically to distress prompts. Two of the baselines adapt existing architecture and the other two incorporate a framework identifying levels of cognitive and emotional empathy in responses. Automatic and human evaluation of these models validate the utility of the dataset in generating empathetic responses for distress support and show that identifying levels of empathy in peer-support responses facilitates generating responses that are lengthier, richer in empathy, and closer to the ground truth.

## introduction psychological distress refers to a state of extreme sorrow, pain, or suffering, both emotional and physical. it is often associated with feelings of discomfort, anxiety, or anguish. the world health organization estimates that psychological distress affects 29% of people in their lifetime @xcite . despite the availability of mental health services, people hesitate to reach them because of the public stigma associated with mental health. there is also a severe shortage of mental health workers @xcite . thus, recent work investigates how technology can be utilized to meet the needs of people suffering from distress. one * these authors contributed equally to this work. such solution is the development of conversational agents or chatbots to deliver distress support @xcite @xcite . deep neural networks work very effectively in the development of task-oriented and opendomain conversational agents @xcite @xcite . most of such dialogue agents can generate syntactically correct and contextually relevant responses. but a major challenge faced by these systems is identifying human emotion and responding in an empathetic manner @xcite . this is very important when developing chatbots to support distress as one of the major components that contributes to the success of such interaction is the ability to empathize @xcite . recently, researchers have curated emotion-labeled and empathetic datasets such as emotionlines @xcite , emocontext @xcite , @xcite , and es-conv @xcite to enable training dialogue systems that can generate emotion-aware and empathetic responses. however, the above datasets include only a limited amount of dialogues dealing with distress. the dialogues in the first three datasets are more open-domain and span across topics less related to distress. the esconv dataset that is more focussed on distress contains only 1.3k dialogues covering only 13 distress-related topics. recent research has curated and conducted analysis on real counseling conversations @xcite . but these datasets are not publicly accessible due to privacy and ethical reasons. to address the above limitations, we curated a large-scale dialogue dataset, named red (reddit emotional distress), containing ≈1.3m dialogues spanning across more than 4k distress-related topics. the dialogues are scraped from the popular peer support forum, reddit. peers are seen to actively engage in such forums to support others undergoing distress and thus they contain distressrelated dialogues in abundance spanning a wide range of topics. prior work has also found that responses from peers contain higher empathic concern for posts seeking help as many peers share similar distressful experiences @xcite . but as these conversations are available as long threads, the turn-taking structure has to be explicitly extracted and the conversations have to undergo a rigorous pre-processing pipeline including the removal of profanity before they are used to train chatbots. even then, the dataset can still possess less ideal responses to distress since peers are not trained in delivering distress support as professionals. we take steps to address this by making use of existing empathetic frameworks based on psychology that can be used to identify highly empathetic responses in such dialogues and enabling chatbot models to favor such responses over others. empathy is a complex multi-dimensional construct with two broad aspects related to emotion and cognition. the emotion aspect refers to the ability to share the feelings of another person and the cognition aspect refers to the ability to understand and acknowledge how a person feels. in mental health therapy, both emotional and cognitive empathy are equally important @xcite . thus, for the development of distress support chatbots, it is vital to understand these types of empathy and the techniques by which these different types of empathy can be elicited. we apply such empathy recognition frameworks on red to develop several distress support chatbots models. figure 1 shows an example. in the first instance, identification of the context emotion enables the chatbot to produce a suitable generic response. in the second instance, the chatbot's response is conditioned on a specific empathetic response intent that helps to generate a diversified response. in the third instance, training the model to favour more cognitive and emotional empathy helps in generating lengthier responses containing specific cognitive and emotional empathy communication strategies. our contributions are three folds. 1) we curate a large-scale dialogue dataset containing ≈1.3m distress support dialogues spanning across more than 4k distress topics, from a set of carefully selected subreddits. 2) we describe the empathetic dialogue characteristics between the speakers and the listeners in this dataset using statistical and visual means. 3) using this dataset as a benchmark, we develop four baseline chatbot models. the first two baseline models adapt existing empathetic response generation architectures. on top of them, we develop two new baselines by incorporating a framework that can identify levels of emotional and cognitive empathy in responses contained in red. automatic and human evaluation of the models' responses validate the utility of the red dataset in facilitating empathetic response generation and show that identifying different levels of emotional and cognitive empathy enables generating responses that are lengthier, richer in empathy, and closer to the ground-truth. the code and the datasets are available at https: //github.com/yehchunhung/epimeed 2 related work many dialogue datasets such as iemocap @xcite , @xcite , and meld @xcite are developed to make chatbots understand users' emotions and respond appropriately. these datasets contain visual, acoustic, and textual signals. more recent work such as emotionlines @xcite , open-subtitles @xcite , and edos @xcite are conversation datasets containing tv and movie transcripts translated from voice to text. though these works intend to build dialogue datasets by improving the sentence quality, they are still unable to fully model interactions occurring only via text. and most of the dialogues contained in these datasets represent generic day-to-day situations and not psychological distress in particular. @xcite developed the empathet-icdialogues dataset, inclusive of 25k dialogues grounded on 32 positive and negative emotions. @xcite developed the esconv dataset, containing ≈1.3k dialogues discussing emotional distress and whose responses are grounded on the helping skills theory @xcite . but the crowdsourced artificial setting used to curate them makes the dialogue prompts less authentic and the responses less genuine. because of the cost of crowdsourcing, it also limits the size of these datasets as well as their topic coverage. thus, a large-scale topically diverse dataset focused on textual conversations between speakers who are emotionally distressed and listeners who actively offer emotional support is lacking in the literature. this type of conversation could be available as recorded therapy sessions between psychologically distressed patients and therapists. however, such counseling datasets used to conduct recent research @xcite are not directly accessible to the public due to ethical reasons. to address these limitations, we curate a large dataset containing peer support dialogues related to a variety of distress-related topics and validate that combined with existing empathyidentifying frameworks, it can potentially be used to develop chatbots that can offer empathetic support to distressful user prompts. ## conversational baselines using the red dataset as a benchmark, we trained four baseline dialogue models. the first two baselines adapted the architecture of emoprepend @xcite and meed @xcite , which are state-of-the-art empathetic chatbot models. we also examined different ways existing models can be combined to produce more empathetic responses for distress prompts. for this purpose, we developed another two experimental baselines, epimeed and epimeed+, by combining meed with epitome @xcite , which is a theoretically-grounded framework that can identify levels of cognitive and emotional empathy in text-based conversations and extract rationales underlying its predictions. all the models were trained on 80% of red conversations, leaving 10% of the conversations each for validation and testing. figure 4 show the architecture of the different models we used for evaluation. emoprepend: top-k predicted emotion labels from a supervised classifier for the corresponding dialogue context is prepended to the beginning of the token sequence as encoder input. we initialized the encoder of this model with weights from the pre-trained language model roberta @xcite and trained it on red, prepending the top-1 emotion or intent predicted by the bert transformer-based classifier proposed by @xcite . meed: this model proposed by xie and pu (2021) consists of two modules: 1) a response emotion/intent prediction module; and 2) a response generation module. the response generation module is an encoder-decoder model that uses the transformer architecture, in which the encoder is initialized with weights from roberta. the response emotion/intent prediction module takes the dialogue context as input and predicts what the emotion or intent of the response should be. this prediction is used to condition the response generated by the decoder in the first module. epimeed: in therapy, interacting empathetically with clients is fundamental to success @xcite . even though empathy can be interpreted as reacting with emotions of warmth and compassion @xcite , a separate but key aspect of empathy is also to communicate a cognitive understanding of others, referred to as cognitive empathy. for mental health support, both emotional and cognitive empathy are equally important @xcite . thus, it is important to identify such emotional and cognitive empathetic responses amongst other responses that appear in red and train models in such a way that they favor such responses that reflect cognitive and emotional empathy over others. to support this, we experimented with a new epimeed+: to enable the model to favour responses containing stronger emotional reactions, interpretations, and explorations while decoding, we further tweaked the loss function associated with meed such that it incorporates levels of emotional and cognitive empathy predicted by epitome. we modified the loss function to be the dot product between the cross entropy loss and the rationale mask predicted by epitome. the rationale mask predicted by epitome may assign 1 to each token in a text subsequence that may be considered more empathetic than the rest of the text. it acts as an amplifier to the loss so that the model will predict better the tokens with larger empathetic values as predicted by epitome. compared to the original loss l old (i) , the new loss l new (i) given an input sequence i can be written as: ## automatic evaluation automatic evaluation of the models was conducted using a variety of automatic metrics used in evaluating chatbots. they are grouped into diversity-based, word-overlap-based, and embedding-based metrics (details in appendices). the levels of emotional reactions, interpretations, and explorations computed by epitome in the responses generated by the four models are denoted in ## human evaluation a human evaluation experiment was designed to evaluate the empathetic appropriateness of the responses generated by the four models, by recruiting workers from amazon mechanical turk. we randomly selected 200 dialogue prompts from the red test dataset and the responses generated by the four models for these prompts to be evaluated by the crowdworkers. the workers were asked to drag and drop the responses generated by the models into areas good, okay, and bad, depending on how empathetically appropriate those responses were to the given prompt. this new way of rating makes it easy to compare many models at once instead of traditional a/b testing, which only allows the comparison of a pair of models at a time. three workers rated the same response and the final results were computed based on the majority vote. the human evaluation scores for each of the models is denoted in ## discussion and conclusion in this paper, we addressed the problem of responding empathetically to distress by curating dialogues from the peer support forum, reddit, and develop-ing models by adapting and hybridizing existing empathetic response generation architectures and empathy identifying frameworks. the red dataset can be used as benchmark to develop similar and better performing chatbot models that can respond to distress. the results of the emotion and intent analysis as well as the automatic and human evaluation results of the experiments conducted on the four baseline chatbot models validate the utility of this dataset in generating emotionally supportive and empathetic responses for distress-related dialogue prompts. but there are some limitations to this work. since users responding to distress-related posts in reddit are not professionals, caution must be taken if these conversations are directly used for training automatic systems that can offer emotional support. removal of profanity is one step that we have taken towards making such systems reliable and fail-safe. the shift in the emotion of the speaker towards more positive emotions such as gratefulness is also another indicator that the responses do help the speaker lift his/her mood. but deeper analysis such as measuring the level of speaker satisfaction in subsequent dialogue turns and identifying the specific communication techniques that lead to positive outcomes are required when developing an emotionally supportive chatbot based on these conversations. we showed that incorporating existing empathetic frameworks such as epitome @xcite and conditioning the response on specific empathetic response intents such as in meed @xcite are good advances in addressing such limitations.