Harnessing the Power of Large Language Models for Empathetic Response Generation: Empirical Investigations and Improvements (2023)

Abstract: Empathetic dialogue is an indispensable part of building harmonious social relationships and contributes to the development of a helpful AI. Previous approaches are mainly based on fine small-scale language models. With the advent of ChatGPT, the application effect of large language models (LLMs) in this field has attracted great attention. This work empirically investigates the performance of LLMs in generating empathetic responses and proposes three improvement methods of semantically similar in-context learning, two-stage interactive generation, and combination with the knowledge base. Extensive experiments show that LLMs can significantly benefit from our proposed methods and is able to achieve state-of-the-art performance in both automatic and human evaluations. Additionally, we explore the possibility of GPT-4 simulating human evaluators.

## introduction empathetic dialogue plays an essential role in building harmonious social relationships @xcite . the task of empathetic response generation involves understanding the user's experiences and feelings, and generating appropriate responses @xcite . using dialogue systems to provide empathetic responses has advantages such as easy access and no time constraints @xcite . figure 1 shows an example of the empathetic dialogue from the benchmark dataset. most previous researchers have established elaborately designed models based on reliable theoretical knowledge @xcite @xcite @xcite . however, the basic models used are mostly small in scale. recently, large language models (llms) @xcite @xcite have been widely used in natural language processing (nlp) with superior performance. in particular, the emergence of chatgpt has elicited substantial attention and interest in academia and industry, and it has demonstrated extraordinary performance in a variety of tasks, especially dialogue generation. these llms are trained on a large amount of corpora, encompassing a wealth of knowledge. in specific tasks, even without fine-tuning, outstanding performance can be achieved by adopting some gradient-free techniques @xcite (e.g., in-context learning (icl)). therefore, it is necessary to empirically explore the performance of llms on specific domains, as the methods of solving problems may undergo significant changes. there have been some initial attempts @xcite to apply llms to empathetic response generation. however, their approaches mainly focus on pre-training or fine-tuning on the training data, or simply exploring the capability of a single model. to investigate the capability of llms in empathetic response generation, this work empirically studies the performance of llms on the empathetic dialogue benchmark dataset. we first compare llms in the zero-shot and few-shot icl set-tings with a large number of baseline models. surprisingly, the performance of the gpt-3.5 series of llms with in-context learning settings has comprehensively surpassed state-of-the-art models. this reveals that the paradigm shift brought by llms also applies to empathetic dialogue. furthermore, based on the best performance llm setting, we propose three possible methods to improve its performance. specifically, improvement via semantically similar in-context learning, two-stage interactive generation, and combination with the knowledge base. extensive automatic and human evaluation experiments show that llms can benefit from our proposed methods, which can generate more empathetic, coherent, and informative responses. in addition, although human evaluation is crucial in empathetic dialogue, its associated costs and time consumption are enormous. in view of the outstanding performance of llms on empathetic response generation, we attempt to use gpt-4 (ope-nai, 2023) to simulate human evaluators to evaluate the results. the spearman and kendall-tau correlation results indicate that gpt-4 has the potential to be a substitute for human evaluators. our contributions are summarized as follows: ## conclusion and future work in this work, we empirically study the performance of llms on empathetic response generation and propose three improvement methods. empirical automatic and human evaluation results show that llms significantly outperform state-of-the-art models, and verify the effectiveness of our proposed improvements of llms. in the future, our work can contribute to deeper comprehension and the application of llms for empathetic dialogue, and provide some insights for similar tasks.