A Large-Scale Dataset for Empathetic Response Generation (2021)

Abstract: Recent development in NLP shows a strong trend towards refining pre-trained models with a domain-specific dataset. This is especially the case for response generation where emotion plays an important role. However, existing empathetic datasets remain small, delaying research efforts in this area, for example, the development of emotion-aware chatbots. One main technical challenge has been the cost of manually annotating dialogues with the right emotion labels. In this paper, we describe a large-scale silver dataset consisting of 1M dialogues annotated with 32 fine-grained emotions, eight empathetic response intents, and the Neutral category. To achieve this goal, we have developed a novel data curation pipeline starting with a small seed of manually annotated data and eventually scaling it to a satisfactory size. We compare its quality against a state-of-the-art gold dataset using both offline experiments and visual validation methods. The resultant procedure can be used to create similar datasets in the same domain as well as in other domains.

## introduction researchers are increasingly inclined towards refining pre-trained language models with domainspecific datasets to achieve certain tasks @xcite @xcite . one such area is the development of empathetic conversational agents that can understand human emotions and respond appropriately. the aim of the empathetic response generation task is to generate syntactically correct, contextually relevant, and more importantly emotionally appropriate responses following previous dialogue turns. such tasks require the creation and availability of large dialogue datasets, in which each utterance is annotated with the correct intents and emotions. though many such datasets have been developed in the past @xcite @xcite , due to the cost of manual labor, they are limited in size, thus insufficient to train robust conversational agents. since collecting and manually annotating such gold standard data is expensive, replacing them with automatically annotated silver standard data has become a rising interest @xcite . we show how such a large-scale silver standard dataset with sufficient quality can be curated and used to fine-tune pre-trained language models for the generation of empathetic responses. emotions revealed in social chitchat are rather complex. it has many categories of emotions to distinguish due to subtle variations present in human emotion. for example, sadness and disappointment are pursued and dealt with differently in human conversations even though both of them are negative emotions. also, the listener's reaction to emotion is not always a straightforward mirroring effect of the speaker's emotion. rather it can be more neutral and convey a specific intent, as is evident from the dialogue example in to fill the above gap, we curate a novel largescale silver dialogue dataset, edos (emotional dialogues in opensubtitles), containing 1m emotional dialogues from movie subtitles, in which each dialogue turn is automatically annotated with 32 fine-grained emotions, eight plus categories as well as the neutral category. movie subtitles are extensively used for emotion analysis in text in earlier and recent research @xcite @xcite . the nature article "how movies mirror our mimicry" @xcite states "screenwriters mine everyday discourse to make dialogues appear authentic" and "audiences use language devices in movies to shape their own discourse". hence, it can be one of the major sources to train chatbots and learn emotional variations and corresponding response strategies in dialogues. to reduce the cost of human labeling and the complexity of labeling dialogues with fine-grained emotions and intents, we devised a semi-automated human computation task to collect fine-grained emotion and intent labels for a small set of movie dialogues (9k). we then followed automatic data augmentation techniques to expand the labeled data and trained a dialogue emotion classifier to automatically annotate 1m emotional dialogues. the process of curating the dataset involved several stages. first, we applied automatic turn and dialogue segmentation methods, data cleaning and removal of duplicates on movie subtitles in the opensubtitles (os) corpus @xcite and obtained close to 4m dialogues. then, we applied a weak labeler (a bert-based sentence-level classifier) trained on the empatheticdialogues dataset @xcite , to label utterances in os dialogues and filtered 1m emotional dialogues (edos initial). thereafter, we applied data augmentation techniques on a small set of human-annotated data and used the manually annotated and extended labels to train a strong labeler that is used to annotate dialogues in edos initial and obtained the final 1m edos dataset. we evaluated the quality of the resultant dataset by comparing it against the empatheticdialogues dataset by means of offline experiments and visual validation methods. figure 1 summarizes the process of creating edos. the data curation pipeline we followed substantially reduced the cost of human labor while ensuring quality annotations. our contributions in this paper are three-fold. 1) we curate a large-scale dialogue dataset, edos, containing 1m emotional dialogues labeled with 32 fine-grained emotions, eight empathetic response intents (the plus categories), and neutral. compared to existing dialogue datasets tagged with emotions, edos is significantly larger (≈ 40 times larger than empatheticdialogues), and contains more fine-grained emotions and empathetic response strategies. 2) we outline the complex pipeline used to derive this dataset. 3) we evaluate the quality of the dataset compared to a state-of-theart gold standard dataset using offline experiments and visual validation methods. the empatheticdialogues dataset @xcite ) contains 25k open-domain dialogues grounded on 32 emotions. the 32 emotions range from basic emotions derived from biological responses @xcite to larger sets of subtle emotions derived from contextual situations @xcite . @xcite manually analyzed a subset of the listener turns in empatheticdialogues and identified eight listener-specific response intents. they developed a sentence-level weak labeler using which they annotated the entire dataset with 32 emotions, eight empathetic response intents, and the neutral category. however, due to the limited size of empatheticdialogues, it is difficult to be used for data-intensive applications. to address the above limitations, we curate edos containing 1m movie dialogues. we label each dialogue turn with 32 emotions, eight empathetic response intents, and neutral using our own dialogue emotion and intent classifier. ## methodology this section describes the dialogue selection process, the design of the human annotation task, the data augmentation techniques used to expand human-labeled dialogues, and the development of a strong labeler to annotate the dataset. ## edos quality analysis and comparison with the state-of-the-art gold standard we also conducted a qualitative comparison of the annotations in the edos dataset with empa-theticdialogues @xcite , a state-of-the-art gold standard dataset for empathetic conversations. figure 2 compares the distributions of emotions and intents in the two datasets. it is observed that in both datasets, intent categories take prominence over individual emotion classes. this is in par with observations of welivita and pu (2020), where they notice that one or more intents from the taxonomy of empathetic intents are mostly utilized when responding to emotions in dialogue, rather than similar or opposite emotions. especially, the intent questioning takes the highest percentage among the annotations in empatheticdialogues and edos. we also computed the kl-divergence (≥ 0) of the emotion and intent distribution of edos with respect to that of empatheticdialogues, which measures how one probability distribution is different from a second, reference probability distribution @xcite . it resulted in a kl-divergence value of 0.2447, which indicates a considerable similarity between the two distributions (the lower the kl divergence, the more similar the distributions are). figure 3 compares the emotion-intent flow patterns in empatheticdialogues and edos. in the visualization corresponding to empatheticdialogues, the 1 st and 3 rd dialogue turns correspond to the speaker and the 2 nd and 4 th dialogue turns correspond to the listener. however, in edos, we cannot distinguish the dialogue turns as speaker and listener turns due to the absence of speaker annotations. though this is the case, we could still observe some conversational dynamics present in empatheticdialogues are preserved in edos. for example, in both datasets, the speaker mostly starts the conversation with some emotional statement and in the subsequent turn, the response tends to be of the intent questioning. in both datasets, intents agreeing and acknowledging follow emotions seen in the first turn irrespective of whether they are positive or negative. as the dialogues proceed, it could be seen in both datasets the emotions deescalate as more empathetic response intents emerge. ## experimental baselines we propose some experimental baselines using the curated dataset for empathetic response generation and compare the performance against a dialogue model trained on the empatheticdialogues dataset. for this purpose, we trained a transformer @xcite model with various training settings. specifically, the following datasets were involved: 1) os dialogues (as described in section 3.1, these dialogues were obtained by segmenting the movie subtitles. note that for the purpose of pre-training, we excluded the edos dialogues, resulting in around 3m dialogues.); 2) edos (1m dialogues); and 3) empatheticdialogues (25k dialogues). all (a) empatheticdialogues dataset (b) edos dataset figure 3: comparison of emotion-intent flow patterns in the empatheticdialogues and edos datasets. for simplicity, only the first four dialogue turns are visualized. to evaluate the performance of the dialogue models, we adopted the following metrics: 1) perplexity; 2) distinct-1 and -2 metrics @xcite , which measure the diversity of the generated responses; 3) sentence embedding similarity-we used sbert @xcite to obtain an embedding for the generated response as well as the ground-truth and then calculated the cosine similarity between the two embeddings. the performance of the dialogue models was tested in held-out and zero-shot settings. the evaluation results are shown in in the held-out setting, where the model is evaluated on data from the same domain as the training data, all three models achieved good performance, and the perplexity values are much lower compared with the zero-shot setting, where the model is evaluated on data from a different domain. we also observe that the model fine-tuned on os and edos dialogues achieves much higher distinct-1 and -2 scores, even in the zero-shot setting when evaluated on empatheticdialogues. this indicates that by training on our curated opensubtitles dialogues, the model gains more diversity in the generated responses. it might be due to the larger size of the datasets containing many diverse responses. out of the two, edos performs the best in terms of diversity, which reflects the quality of dialogues filtered from opensubtitles. ## discussion and conclusion in this work, we curated a large-scale dialogue dataset, edos, comprising of 1m emotional dialogues from movie subtitles. this dataset is significantly larger in size and contains more fine-grained emotion categories and empathetic response intents than the existing emotional dialogue datasets. to facilitate annotation, we utilized data augmentation techniques to extend a small set of manually annotated data and trained a dialogue emotion classifier having comparable accuracy to the state-of-the-art. the data augmentation and automatic annotation procedure we employed significantly reduced the manual annotation cost and time. obtaining a large dataset is important only if the quality can be assured. the qualitative comparison conducted between edos and the state-of-the-art empatheticdialogues dataset by means of visual validation was one way to confirm that. the results of the comparison confirmed that most of the conversational dynamics present in empatheticdiathe results showed that the model fine-tuned on edos scored the best in terms of diversity metrics. this dataset can be readily utilized to develop empathetic conversational agents and for fine-grained emotion analysis in dialogues. the pipeline we present can be used when creating similar largescale datasets in similar or even different domains. as future work, we plan to utilize this dataset to further conduct experiments on empathetic response generation. since it is annotated with emotions and intents, we will use it for experiments involving controllable and interpretable response generation. particularly, the plus categories present in the dataset can be utilized to condition the chatbot's response generation process, making it possible to control and interpret the generated responses. the dataset can also be used to train state-of-the-art dialogue emotion classifiers.