Reciprocal Learning of Knowledge Retriever and Response Ranker for Knowledge-Grounded Conversations (2022)

Abstract: Grounding dialogue agents with knowledge documents has sparked increased attention in both academia and industry. Recently, a growing body of work is trying to build retrieval-based knowledge-grounded dialogue systems. While promising, these approaches require collecting pairs of dialogue context and the corresponding ground-truth knowledge sentences that contain the information regarding the dialogue context. Unfortunately, hand-labeling data to that end is time-consuming, and many datasets and applications lack such knowledge annotations. In this paper, we propose a reciprocal learning approach to jointly optimize a knowledge retriever and a response ranker for knowledge-grounded response retrieval without ground-truth knowledge labels. Specifically, the knowledge retriever uses the feedback from the response ranker as pseudo supervised signals of knowledge retrieval for updating its parameters, while the response ranker also receives the top-ranked knowledge sentences from knowledge retriever for optimization. Evaluation results on two public benchmarks show that our model can significantly outperform previous state-of-the-art methods.

## introduction human-machine communication is one of the ultimate goals of artificial intelligence. recently, building a dialogue system with intelligence has sparked increased attention in both academia and industry. advanced work includes retrieval-based methods @xcite @xcite and generation-based methods @xcite @xcite . in this paper, we focus on the retrieval-based approaches since they are superior in providing informative and fluent responses to a human input by * corresponding author: dongyan @xcite selecting a proper response from a pre-built index. however, such models are still limited in their ability to fully understand the human query and predict a more engaging response. to this end, some researchers have begun to ground dialogue agents with knowledge @xcite @xcite since humans can naturally associate the content of the conversation with the background knowledge in his/her mind, which has led to improved performance. two prominent lines of research have evolved for this task. one is to build retrieval-based knowledgegrounded dialogue models by directly attending to all available knowledge entries @xcite @xcite . the other is to separate the knowledgegrounded response retrieval process into two stages: knowledge retrieving and response ranking @xcite @xcite , in which a knowledge retriever first selects relevant knowledge sentences from grounded documents, and then a response ranker incorporates the retrieved knowledge sentences from the knowledge retriever and ranks the candidate responses regarding the dialogue context. however, a long-standing issue on this task is that it is nontrivial to collect large-scale dialogues that are naturally grounded on a small set of knowledge sentences. to train such models, one should first collect pairs of dialogue context and the corresponding list of knowledge sentences that contains the information corresponding to the dialogue context. unfortunately, hand-labeling data to that end is time-consuming, and many data sets and applications lack such knowledge annotations 1 . therefore, the above two research lines both suffer from insufficient knowledge supervision. the former is prone to be affected by noise from irrelevant and redundant knowledge when conducting response retrieval, and the knowledge retrieving process of the latter suffers from the lack of labels indicating the ground-truth knowledge sentences. hence, the challenge we consider is: how to better optimize the knowledge retriever and response ranker jointly without ground-truth knowledge labels? to address the challenge, we follow the twostage paradigm and propose a reciprocal learning approach to jointly optimize knowledge retriever and response ranker for response retrieval in knowledge-grounded conversations. we name our model as reckgc. in reciprocal learning, the knowledge retriever uses the feedback from the response ranker as pseudo supervised signals of knowledge retrieval for updating its parameters, while the response ranker also receives the topranked knowledge sentences from the knowledge retriever to optimize itself. we use the posterior estimate to train the knowledge retriever, and use the prior information to train the response ranker. by this means, the knowledge retriever and response ranker can be jointly optimized without groundtruth knowledge labels. we conduct experiments on two public benchmarks including wizard of wikipedia @xcite and cmu_dog @xcite . evaluation results indicate that our model can significantly outperform the existing methods, and achieve new state-of-the-art performance on both data sets. our contributions in this paper are twofold: (1) proposal of a reciprocal learning of knowledge retriever and response ranker for knowledgegrounded response retrieval without ground-truth knowledge label; (2) empirical verification of the effectiveness of the proposed learning approach on two public benchmarks. ## related work early studies of retrieval-based dialogue systems focused on building single-turn context-response matching models that consider only a single utterance or several utterances in the context that are concatenated into a long sequence for response selection @xcite . recently, more emphasis has been placed on response retrieval with multi-turn dialogue context and lots of impressive results have been obtained, including the dual lstm model @xcite , the sequential matching network (smn) @xcite , the deep attention matching network (dam) @xcite , the multi-hop selector network (msn) @xcite . with advances in pre-trained language models @xcite , some researchers also attempt to apply them on response selection: to represent each utterance-response pair with bert and fuse these representations to compute the context-response matching score @xcite ; to directly treat the context as a long sequence and conduct context-response matching with bert @xcite ; to leverage finegrained post-training for improving retrieval-based dialogue systems @xcite . inspired by the ability of human beings to associate dialogue content with background knowledge in his/her mind, researchers have begun to ground dialogue agents with knowledge. zhang et al. (2018) collect a persona-based dialogue corpus which utilizes the interlocutor's profile as background knowledge; zhou et al. (2018a) publish a corpus which contains conversations grounded in articles about popular movies; dinan et al. (2019) release another corpus with wiki articles as grounded documents which cover a wide range of topics. at the same time, lots of representative models have been obtained. zhao et al. (2019); gu et al. (2019); hua et al. (2020) successively put forward document-grounded matching network (dgmn), dually interactive matching network (dim), and rsm-dck which let the dialogue context and all knowledge sentences interact with candidate responses respectively with crossattention mechanism. gu et al. (2020) propose a document-grounded model named fire which first compute the importance score for each context turn and knowledge sentence, then further use them to weigh the corresponding representation. dinan et al. 2019 also propose to joint learn the knowledge selection and response matching in a multi-task manner or a two-stage training procedure. this strategy, however, requires ground-truth knowledge labels annotated by human wizards, which is presumed absent in our paper. recently, @xcite study response matching in knowledge-grounded conversations under a zero-resource setting. in particular, they propose decomposing the training of the knowledge-grounded response selection into three tasks and jointly training all tasks in a unified pre-trained language model. ## methodology: reckgc in this section, we first formalize the task of knowledge-grounded response retrieval and then introduce our model from overview to several components to reciprocal learning of them. ## experiments to demonstrate the effects of the proposed models, we conduct experiments on two public data sets. ## conclusion in this paper, we study the retrieval-based knowledge-grounded dialogues. to effectively optimize the knowledge retriever and response ranker without ground-truth knowledge labels, we propose a reciprocal learning approach to jointly optimize the two components in an end-to-end way. concretely, the knowledge retriever takes the feedback from the response ranker as pseudo supervised signals of knowledge retrieval, while the response ranker receives the top-ranked knowledge sentences from the knowledge retriever to optimize itself. by this means, our model can be trained without ground-truth knowledge labels. evaluation results on two benchmarks indicate that our model can significantly outperform state-of-the-art methods.