Retrieval-free Knowledge Injection through Multi-Document Traversal for Dialogue Models (2023)

Abstract: Dialogue models are often enriched with extensive external knowledge to provide informative responses through a retrieval-augmented pipeline. Nevertheless, retrieval-augmented approaches rely on finely annotated retrieval training data and knowledge-grounded response generation data, making it costly to transfer. To tackle this challenge, this paper proposed a retrieval-free approach, KiDG, by automatically turning knowledge documents into simulated multi-turn dialogues through a Multi-Document Traversal algorithm. The simulated knowledge-intensive dialogues constructed by KiDG in one domain can be easily used to train and enhance pre-trained dialogue models’ knowledge w.r.t. this domain without costly annotation. We conduct extensive experiments comparing retrieval-augmented models and a variety of retrieval-free models. We found that dialogue models enhanced with data simulated with KiDG largely outperform state-of-the-art retrieval-free methods, and it achieves comparable performance compared to retrieval-augmented methods while being better, and cheaper at domain transfer.

## introduction knowledge plays a crucial role in dialogue systems, which is helpful in improving the informativeness, logicality, and reliability of generated responses. to encourage pretrained dialogue models (pdms) to produce knowledge-grounded responses, existing research mainly follows two lines: retrievalbased @xcite and retrieval-free @xcite . ## experiment in this section, we show that simulated dialogues from kidg could boost pdms' performance in knowledge-grounded response generation tasks. from our exhaustive experiments, we found that the enhanced pdms obtain state-of-the-art performance under retrieval-free settings and even achieve comparable performance compared with retrieval-based approaches. in addition, as the simulation dialogue scale increases, the pdms tend to generate more proactive and precise responses. ## related work knowledge-grounded dialogues are helpful for enhancing pre-trained dialogue models (pdms) @xcite @xcite @xcite to be more knowledgeable. existing research can be classified into two directions: retrieval-based @xcite @xcite and retrieval-free @xcite paradigms. the retrieval-based paradigm is composed of a knowledge retriever and a generator. the retriever uses sparse or dense representations @xcite to obtain relevant knowledge for response generation. as the input length of pdms is limited, the retriever is responsible to fulfill finegrained knowledge retrieval rather than a batch of relevant documents and more information is not always better, since there exists much noise. hence the retrieval-based methods need knowledge and dialogue utterances aligned dataset to learn how to fetch knowledge and how to incorporate it into responses. however, the data annotation process is tedious and labor-intensive. to alleviate this problem, xu et al. (2022) proposed the retrieval-free paradigm. they first train topic experts with documents from several topics. then the model is fine-tuned with knowledge-grounded dialogues for adaption. however, they utilize topic relations at a coarse granularity, i.e., document-level only. recently, @xcite devised the dialogue inpainting to produce knowledge-grounded dialogues: transforming documents into two-person conversations with t5 @xcite . but the generated dialogues only contain sentences in the same document and always in the original order as the source document, which hurts the generalization abilities of pdms. ## conclusion in this paper, we propose kidg, a retrieval-free approach to incorporate knowledge into pdms by automatically turning knowledge documents into simulated dialogues. kidg exploits both the finegrained aspect relations in a single document and the coarse-grained topic relations between documents though multi-document traversal. our experiments show that the kidial generated by kidg can improve the pdms to achieve state-ofthe-art performance under retrieval-free settings and achieve performance comparable to retrievalbased methods. our further analysis proves that a larger kidial can enhance the pdms to generate more proactive and informative responses.