Does GPT -3 Generate Empathetic Dialogues? A Novel In-Context Example Selection Method and Automatic Evaluation Metric for Empathetic Dialogue Generation (2022)

Abstract: Since empathy plays a crucial role in increasing social bonding between people, many studies have designed their own dialogue agents to be empathetic using the well-established method of fine-tuning. However, they do not use prompt-based in-context learning, which has shown powerful performance in various natural language processing (NLP) tasks, for empathetic dialogue generation. Although several studies have investigated few-shot in-context learning for empathetic dialogue generation, an in-depth analysis of the generation of empathetic dialogue with in-context learning remains unclear, especially in GPT-3 (Brown et al., 2020). In this study, we explore whether GPT-3 can generate empathetic dialogues through prompt-based in-context learning in both zero-shot and few-shot settings. To enhance performance, we propose two new in-context example selection methods, called SITSM and EMOSITSM, that utilize emotion and situational information. We also introduce a new automatic evaluation method, DIFF-EPITOME, which reflects the human tendency to express empathy. From the analysis, we reveal that our DIFF-EPITOME is effective in measuring the degree of human empathy. We show that GPT-3 achieves competitive performance with Blender 90M, a state-of-the-art dialogue generative model, on both automatic and human evaluation. Our code is available at https://github.com/passing2961/EmpGPT-3 .

## introduction empathy refers to the ability to understand another person's experiences and feelings. this is important for increasing social bonding (rapport) with conversation partners @xcite . empathy is a multi-dimensional concept consisting of two main aspects: cognitive and affective @xcite . since @xcite released the empatheticdialogues dataset for empathetic dialogue generation task, previous studies have improved their dialogue agents to generate more empathetic dialogues @xcite @xcite @xcite . depending on how the dialogue agents are trained, these approaches are largely divided into two categories depending on how to train own dialogue agents: (i) training from scratch or (ii) fine-tuning a pretrained dialogue generative model. however, neither of these approaches uses the prompt-based in-context learning paradigm in zero-shot and fewshot settings. recently, many researchers have attempted to build large-scale language models (llms), such as gpt-3 @xcite , opt @xcite , and hyperclova @xcite . these models have shown surprising performance in various nlp tasks via prompt-based in-context learning, which is a new paradigm learning technique. previous studies have explored the effect of fewshot in-context learning on dialogue generation tasks @xcite . @xcite explored fewshot in-context learning for empathetic dialogue generation, they did not conduct an in-depth analysis of their ability to generate empathetic dialogues. moreover, they did not leverage gpt-3 as a prompting language model. in this study, we explore whether gpt-3 generates empathetic dialogues using prompt-based in-context learning in both zero-shot and few-shot settings. we simply designed the prompt, which is a modified version of the basic prompt provided by openai. as pointed out in @xcite , it is important to carefully choose in-context examples to enhance few-shot performance. inspired by empathy being a multi-dimensional concept @xcite , we propose sitsm and emositsm selection methods that choose in-context examples based on emotion and situation information. to reflect the human tendency to express empathy, we also propose a new automatic evaluation method called diff-epitome, which is an extended version of epitome @xcite . our main contributions are as follows. in human evaluation, regardless of the dialogue turn setting, we also show that human annotators prefer gpt-3's responses on both human rating and a/b test. empathetic dialogue generation rashkin et al. (2018) first introduced the empatheticdia-logues dataset. lin et al. (2019) proposed a mixture of empathetic listeners (moel), where each listener is specialized in how to understand and respond appropriately to each emotion. majumder et al. (2020) generated empathetic responses by mimicking human emotions, grouping emotions, and imposing stochasticity into each emotion group. sharma et al. (2020) introduced a conceptual framework epitome (described in §3.3.1). welivita and pu (2020) proposed a taxonomy of empathetic response intents, consisting of nine categories (in appendix d). for convenience, this is referred to as empintent. zheng et al. (2021) proposed a multifactor hierarchical framework (comae), which considers epitome, empintent, and emotion. kim et al. (2021b) generated more specific empathetic responses focused on emotion cause words by utilizing the rational speech acts (rsa) framework (frank and goodman, 2012). sabour et al. (2021) leveraged commonsense to generate more empathetic responses. li et al. (2022) also leveraged external knowledge, such as commonsense knowledge, to explicitly generate empathetic responses. prompt-based in-context learning since brown et al. ( 2020 ) first introduced prompt-based in-context learning, many studies have shown that large-scale language models (e.g., gpt-3) itself has the ability to solve various nlp tasks in both zero-shot and few-shot settings @xcite @xcite @xcite @xcite @xcite . some studies have shown that prompt-based few-shot in-context learning can also be successfully applied in dialogue generation tasks @xcite . the advantage of in-context learning is that it does not require any additional training. however, one problem is that gpt-3 achieves unstable performance depending on in-context examples. to mitigate this problem, @xcite proposed a knn-augmented in-context example selection approach called the kate. in this study, we extended this method to empathetic dialogue generation by selecting relevant in-context examples based on the situation and emotion (in §3.2). ## conclusion in this study, we explore the zero-shot and few-shot performance of gpt-3 in an empathetic dialogue generation task on various metrics with respect to diversity, fluency, and empathy. we introduce a new in-context example selection method, sitsm and emositsm. we also propose a novel automatic evaluation method, diff-epitome, for empathetic dialogue generation. we show that gpt-3 achieves competitive performance with blender 90m on the empatheticdialogues test set on both automatic and human evaluations. from the correlation analysis, we reveal that diff-epitome correlates more with human ratings. in future work, we will apply opt @xcite with an optimized prompt. in addition, we reflect on the overall human tendency to express empathy in the modeling.