External Knowledge Acquisition for End-to-End Document-Oriented Dialog Systems (2023)

Abstract: End-to-end neural models for conversational AI often assume that a response can be generated by considering only the knowledge acquired by the model during training. Document-oriented conversational models make a similar assumption by conditioning the input on the document and assuming that any other knowledge is captured in the model’s weights. However, a conversation may refer to external knowledge sources. In this work, we present EKo-Doc, an architecture for document-oriented conversations with access to external knowledge: we assume that a conversation is centered around a topic document and that external knowledge is needed to produce responses. EKo-Doc includes a dense passage retriever, a re-ranker, and a response generation model. We train the model end-to-end by using silver labels for the retrieval and re-ranking components that we automatically acquire from the attention signals of the response generation model. We demonstrate with automatic and human evaluations that incorporating external knowledge improves response generation in document-oriented conversations. Our architecture achieves new state-of-the-art results on the Wizard of Wikipedia dataset, outperforming a competitive baseline by 10.3% in Recall@1 and 7.4% in ROUGE-L.

## introduction recent advances in natural language processing (nlp) allowed us to build complex conversational systems for various scenarios, such as task-oriented @xcite , chit-chat @xcite , and even for guiding users in performing complex real-world tasks @xcite . when using deep neural networks for implementing a conversational agent, a common practice is to use many historical conversations to train the model to pro- * * work done during internship at amazon. duce responses that are related to a dialog context. in this setting, a response is generated solely from the knowledge acquired during the training of the network @xcite , and no other knowledge sources are used at inference time. a more effective way of producing dialog responses would be to incorporate external knowledge into the model. this is, for example, the case of systems that make use of document/passage retrieval in qa @xcite . in this work, we consider the setting in which a conversation is grounded in a target topic but also in external knowledge, in the form of documents. for example, figure 1 shows a conversation with respect to a target topic (i.e., san diego comic-con) represented by a topic document. during the conversation, some turns may refer to other documents from an external knowledge base (kb) providing additional information (e.g., shel dorf ). in our preliminary studies, we estimated that about 36% of the dialogs in the popular wizard of wikipedia dataset @xcite require knowledge beyond the topic document. this setting poses additional challenges to the generation of adequate responses in a dialog. while a model could possibly memorize a vast amount of knowledge in its weights during training, the model will likely be applied to new dialog contexts that refer to unseen knowledge. if enriched with external knowledge, the model input could be better conditioned to produce accurate outputs. from a technical perspective, the model needs to learn i) to retrieve the relevant knowledge and ii) to incorporate it into the generated response. to address these challenges, we propose eko-doc, an end-to-end conversational agent designed to model a target topic document and a set of external knowledge documents. eko-doc integrates a dense passage retriever (dpr) @xcite and a re-ranker (rr) into a response generation model. to reduce the need for manual annotations, we automatically acquire labels for the training of the dpr and rr components. in particular, we leverage the attention weights of the generation model over the retrieved documents to generate a set of silver labels. our generation model is a sequence-to-sequence model that generates a response by taking as input the entire concatenation of the dialog context and related documents retrieved from a kb. experimental results demonstrate that eko-doc is able to correctly use both the topic document and the external knowledge to produce better responses in a conversation. we report competitive results on two public datasets, wizard of wikipedia (wow) @xcite and multidoc2dial @xcite . in particular, eko-doc achieves new stateof-the-art (sota) results on wow, outperforming a strong baseline by 10.3% in recall@1 and 7.4% in rouge-l. finally, we conduct a human study to verify the quality of the generated responses. compared to models that do not condition on the topic document or external knowledge, our model produces responses that are more fluent, more ontopic, and more interesting. to summarize, our contributions are: i) we study the setting where we model both a target topic document and external knowledge for a conversation; ii) we propose eko-doc, a novel end-to-end response generation architecture for the task; iii) we propose an automatic annotation procedure to acquire the labels for training the retrieval engine. the dialog is mostly centered around d t , but external knowledge from k kb is sometimes needed to generate an informative and relevant response (see figure 1 for an example). to this end, the task can be viewed as building a model of p (r|c, d t , k kb ). ## conclusions this work proposes and studies a new problem setting that combines document-oriented conversations and open-domain qa. we introduce a new architecture for the problem that includes a dense passage retriever, a re-ranker, and a response generation model. we train these three components endto-end and use cross-attention scores as pseudolabels to update the retrieval engine. extensive experimental results on two public datasets demonstrate the effectiveness of our method. in the future, we plan to reduce the computational complexity of our model by using compression techniques.