Knowledge-Grounded Dialogue Generation with Pre-trained Language Models (2020)

Abstract: We study knowledge-grounded dialogue generation with pre-trained language models. To leverage the redundant external knowledge under capacity constraint, we propose equipping response generation defined by a pre-trained language model with a knowledge selection module, and an unsupervised approach to jointly optimizing knowledge selection and response generation with unlabeled dialogues. Empirical results on two benchmarks indicate that our model can significantly outperform state-of-the-art methods in both automatic evaluation and human judgment.

## introduction with advances in neural machine learning @xcite @xcite and availability of the huge amount of human conversations on social media @xcite , building an open domain dialogue system with data-driven approaches has attracted increasing attention from the community of artificial intelligence and natural language processing. in this work, we are interested in generative approaches. generative models for open domain dialogues are notorious for replying with generic and bland responses, resulting in meaningless and boring conversations @xcite . such deficiency is particularly severe when human participants attempt to dive into specific topics in conversation @xcite . as a result, there is still a big gap between conversation with existing systems and conversation with humans. very recently, there emerge two lines of research that seem promising to bridge the gap. one is to apply large-scale pre-trained language models, such as gpt-2 @xcite . encouraged by the results on pre-training for dialogue generation and knowledge-grounded dialogue generation, and motivated by the problems in both sides, we consider bringing the two together in this work. specifically, we propose knowledgegrounded dialogue generation with pre-trained language models in order to endow a generative model with both rich knowledge and good generalization ability foot_0 . the challenge is that pre-trained language models often set constraints on the maximum number of tokens they can handle (e.g., the maximum number for gpt-2 @xcite is 1024), and thus hinders exploitation of the knowledge text which could be rather long and redundant (e.g., in wizard of wikipedia @xcite , on average each conversation context is associated with 61.2 sentences retrieved from wiki articles, and the average number of tokens in the extra knowledge is 1625.6). indeed, the conflict between model capacity and the ability required for processing long knowledge input represents an essential obstacle for applying pre-trained language models to knowledge-grounded dialogue generation, since on the one hand we always have to set up an upper bound to the capacity of pre-trained models in order to handle massive text corpus, and on the other hand we need to keep sufficient candidates with rich enough content in the procedure of response generation in order to guarantee the recall of relevant knowledge. to overcome the challenge, we consider equipping the pre-trained response generation model with a knowledge selection module whereby the redundant knowledge input is slimmed with relevant information (regarding to conversation contexts) kept to meet the capacity constraint. while some recent papers on knowledge-grounded dialogues have paid attention to the problem of knowledge selection @xcite @xcite , the knowledge selection module is either deeply coupled with the specially configured models @xcite and thus is incompatible with the pre-trained language models, or it is learned with human annotations @xcite which are difficult to obtain in practice (e.g., the dataset in @xcite does not contain annotations for knowledge selection). therefore, we propose an unsupervised approach where learning of knowledge selection and fine-tuning of response generation are jointly conducted with unlabeled dialogues. specifically, we build the knowledge selection module on the basis of bert, and formalize knowledge selection as a sequence prediction process, by which the model can take advantage of the pre-training techniques and dynamically determine the relevant knowledge for a given context. the learning algorithm starts from training with pseudo ground-truth that is constructed by making full use of responses as an alternation of human annotations, and then alternatively updates the knowledge selection model and the response generation model through a reinforcement learning approach and a curriculum learning approach respectively. thus, knowledge selection is further optimized with the feedback from response generation, and the knowledge used for fine-tuning the response generation model gradually moves from the pseudo ground-truth to the prediction of the knowledge selection module. we test the proposed method on two benchmarks of knowledge-grounded dialogue generation: wizard of wikipedia @xcite and cmu document grounded conversations @xcite . evaluation results indicate that our model can significantly outperform state-of-the-art methods as well as a few pre-trained models used in heuristic ways, and thus achieves new state-of-theart on the benchmarks. moreover, as a byproduct, the knowledge selection module also outperforms the state-of-the-art model in terms of accuracy of knowledge selection on wizard of wikipedia, implying that other models could also benefit from the component. our contributions in this paper are three-fold: ## related work early work on end-to-end open domain dialogue generation is inspired by the research of machine translation @xcite @xcite . later, the vanilla encoderdecoder architecture is widely extended to improve diversity of responses @xcite @xcite ; to model the structure of conversation contexts @xcite @xcite ; to control attributes of responses @xcite @xcite @xcite ; and to bias responses to some specific personas @xcite . recently, grounding dialogue generation by extra knowledge is emerging as an important step towards human-like conversational ai, where the knowledge could be obtained from knowledge graphs @xcite @xcite , retrieved from unstructured documents @xcite @xcite , or extracted from visual background @xcite @xcite . in this work, we study document-grounded dialogue generation. rather than learning from scratch like most existing work, we take advantage of the pre-trained language models and achieve new stateof-the-art on the benchmarks of the task. big, deep neural language models pre-trained on huge unlabeled text corpus have led to strong improvements on numerous natural language understanding and natural language generation benchmarks @xcite @xcite @xcite @xcite , and therefore are revolutionizing almost the full spectrum of nlp applications @xcite @xcite @xcite and some interdisciplinary applications in nlp and computer vision @xcite @xcite . in the context of dialogue generation, by fine-tuning gpt-2 @xcite in different sizes on social media data, recent work has @xcite shown promising progress on conversation engagement and commonsense questionanswering. in this work, we further explore the application of pre-training to the task of open domain dialogue generation by equipping the pre-trained language models with external knowledge. differ-ent from a very recent paper on pre-training for low-resource knowledge-grounded dialogue generation @xcite , the work presents an in-depth investigation on how to release the power of the existing pre-trained language models on the task when input exceeds the capacity of the models. ## approach heading for learning a g(u, d) for applying gpt-2 to the task of knowledge-grounded dialogue generation, we need to deal with several challenges: (1) how to model the correlation between a context and the external knowledge; (2) how to learn g(u, d) when labels of ground-truth knowledge are absent; and (3) how to jointly optimize g(u, d) and the gpt-2 model with d, and thus the two can boost each other. figure 1 illustrates the architecture of the model. on the basis of the transformer architecture, the knowledge selection module is made up of a context-aware knowledge encoder and a sequential knowledge selector. the former captures interaction patterns between a context u and each sentence in d through a stack of selfattention layers, and the patterns are then fed to the latter to decode useful knowledge one sentence per step. since human annotations are not accessible, the learning method begins with pseudo groundtruth constructed by making full use of responses, and optimization of g(u, d) and optimization of the gpt-2 generation model are alternatively conducted with a reinforcement learning approach and a curriculum learning approach respectively. ## experiments we conduct experiments on wizard of wikipedia (wizard) and cmu document grounded conversations (cmu dog) @xcite . ## conclusions we apply large-scaled pre-trained language models to the task of knowledge-grounded dialogue generation. to this end, we devise a knowledge selection module, and propose an unsupervised approach to jointly optimizing knowledge selection and response generation. evaluation results on two benchmarks indicate that our model can significantly outperform state-of-the-art methods.