Towards Boosting the Open-Domain Chatbot with Human Feedback (2023)

Abstract: Many open-domain dialogue models pre-trained with social media comments can generate coherent replies but have difficulties producing engaging responses. This phenomenon might mainly result from the deficiency of annotated human-human conversations and the misalignment with human preference. In this paper, we propose a novel and efficient framework Diamante to boost the open-domain chatbot, where two kinds of human feedback (including explicit demonstration and implicit preference) are collected and leveraged. By asking annotators to select or amend the model-generated candidate responses, Diamante efficiently collects the human demonstrated responses and constructs a Chinese chit-chat dataset. To enhance the alignment with human preference, Diamante leverages the implicit preference in the data collection process and introduces the generation-evaluation joint training. Comprehensive experiments indicate that the Diamante dataset and joint training paradigm can significantly boost the performance of pre-trained dialogue models. The overall engagingness of the previous state-of-the-art model has been improved remarkably by 50% in Chinese open-domain conversations.

## introduction in recent years, the self-supervised pre-training based on tremendous unlabeled data has brought great success for many natural language processing tasks @xcite . in dialogue generation, the pre-training is usually carried out with massive social media comments, acting as human-like conversations @xcite @xcite . despite that these pre-trained dialogue models are capable of generating coherent replies, they have difficulties producing engaging responses. the main reasons for this phenomenon might be twofold. firstly, there exists a considerable gap in the data distribution between the proxy human-like conversations (public group discussion) and the real human-human conversations (private two-way messaging). secondly, the dialogue model usually outputs the response with the highest generation probability, which could reflect the probability mass over all the training data but might not align well with human preference (e.g., some biased or unsafe statements). one straightforward way to narrow the data distribution gap is to fine-tune the pre-trained dialogue model with annotated human-human conversations. for instance, blender @xcite employs four annotated datasets @xcite @xcite to emphasize the conversational skills of personality, knowledge, empathy, and engagingness. as for the alignment with human preference, lamda @xcite defines and quantifies some critical metrics for dialogue evaluation, including safety, interestingness, and so on. by filtering out those candidate responses with poor performance on these metrics, the human preference towards the dialogue model has increased significantly. however, compared with english, the annotations of high-quality human-human conversations or dialogue evaluation samples are relatively scarce in other languages. as a result, even the state-of-theart chinese chatbot -plato-xl @xcite , is only pre-trained with social media comments and not involved with advanced response evaluation. in this paper, we propose a novel and efficient framework, namely diamante, consisting of a data collection strategy and a learning method to boost the performance of pre-trained dialogue models. two kinds of human feedback are collected and leveraged in diamante, including explicit demonstration and implicit preference. firstly, to bridge the gap in data distribution, diamante collects ## diamante dataset in this paper, we collect an open-domain chit-chat dataset in chinese with the assistance of a pretrained dialogue model. in the following, we will describe the creation of the diamante dataset. ## generation-evaluation joint training in this paper, we propose to leverage not only the explicit human demonstrations but also the implicit human preference that appeared in the data collection to boost the open-domain chatbot comprehensively. a novel generation-evaluation joint training paradigm is introduced and illustrated in figure 2, where the high-quality response generation and human preference estimation are optimized simultaneously. the classical training objective of dialogue generation is to minimize the negative log-likelihood (nll) loss: transformer blocks transformer blocks context ⋅ [sep] ⋅ ⋅ ⋅ human response context ⋅ [sep] ⋅ ⋅ ⋅ model response transformer blocks context ⋅ [sep] ⋅ ⋅ ⋅ random response nll pe 𝑠(𝑐, 𝑟 ℋ ) 𝑠(𝑐, 𝑟 ℳ ) 𝑠(𝑐, 𝑟 ℛ ) figure 2 : overview of the generation-evaluation joint training in diamante. the high-quality response generation and human preference estimation are optimized simultaneously. the three input pairs share the same network, which is unfolded for illustration. the total objective of the generation-evaluation joint training is to minimize the following integrated loss: the first term helps the model learn to mimic human demonstrations and generate high-quality candidate responses. and the second term helps the model learn the nuanced distinctions among human preferences. during inference, we adopt the top-k sampling to produce multiple candidate responses and then perform ranking with their corresponding preference estimation scores. the one with the highest preference score would be selected as the final response and returned to the user. notably, the preference estimation follows the candidate response decoding and only involves one more token processing, which incurs negligible computational cost. one similar work to diamante's joint training is lamda @xcite , where a single model functions as both a generator and a discriminator. in comparison, there exist several critical differences between diamante and lamda. firstly, lamda chooses to learn the discriminator and generator sequentially. by contrast, diamante optimizes generation and evaluation simultaneously, trying to avoid the catastrophic forgetting issue of the two-stage training @xcite . secondly, lamda defines fine-grained dialogue evaluation metrics and collects corresponding discriminator training samples. considering the expensive cost of data collection and the difficulty of reaching an agreement in fine-grained dialogue evaluation @xcite , diamante leverages the implicit human preference as the overall evaluation and gets rid of additional annotations. thirdly, as suggested in the works of human alignment @xcite , the ranked preference evaluation adopted in diamante performs better than the binary discrimination used in lamda. ## conclusion in this paper, we propose to collect and leverage human feedback to boost the open-domain chatbot. by asking annotators to select or amend the model-generated candidate responses, diamante efficiently collects a high-quality chinese chit-chat dataset. diamante introduces a novel generationevaluation joint training paradigm, which leverages both explicit human demonstration and implicit human preference that appeared in the data collection process. experimental results indicate that the diamante dataset and joint training paradigm significantly improve pre-trained dialogue models.