Large Language Models as Source Planner for Personalized Knowledge-grounded Dialogues (2023)

Abstract: Open-domain dialogue system usually requires different sources of knowledge to generate more informative and evidential responses. However, existing knowledge-grounded dialogue systems either focus on a single knowledge source or overlook the dependency between multiple sources of knowledge, which may result in generating inconsistent or even paradoxical responses. To incorporate multiple knowledge sources and dependencies between them, we propose SAFARI, a novel framework that leverages the exceptional capabilities of large language models (LLMs) in planning, understanding, and incorporating under both supervised and unsupervised settings. Specifically, SAFARI decouples the knowledge grounding into multiple sources and response generation, which allows easy extension to various knowledge sources including the possibility of not using any sources. To study the problem, we construct a personalized knowledge-grounded dialogue dataset Knowledge Behind Persona (KBP), which is the first to consider the dependency between persona and implicit knowledge. Experimental results on the KBP dataset demonstrate that the SAFARI framework can effectively produce persona-consistent and knowledge-enhanced responses.

## introduction knowledge enhancement techniques @xcite have significantly empowered machines to deepen their understanding of the underlying knowledge in open-domain dialogues @xcite , surpassing what can be solely acquired from conversational corpora. recent years have witnessed various open-domain dialogue systems relying on different types of knowledge sources, such as topic @xcite , external documents (e.g., wikipedia) @xcite @xcite , persona @xcite @xcite , user memory @xcite @xcite , and more. realizing the limitations of using single-source knowledge, some recent studies further develop dialogue systems with access to multi-source knowledge @xcite @xcite . despite their effectiveness of enriching the dialogue responses with multi-source knowledge, existing methods typically design models to incorporate all sources indiscriminately, resulting in a cumbersome process that struggles to handle cases dependent on the interaction between some specific sources instead of all @xcite . moreover, the importance of comprehending the potential dependency between knowledge sources is overlooked in previous works, which may result in generating paradoxical responses @xcite . for example, humans often express their persona with the assistance of external knowledge. as shown in @xcite , for responding to the question "hi, what do you like to eat?", it is inadequate to only incorporate single-source knowledge from user persona, e.g., "i am a vegetarian", since relevant information from external documents is also required, e.g., vegetarian (vegetarians like to eat fruits and vegetables). however, being unaware of the dependency between these two different sources of knowledge (persona and documents), dialogue systems may select the document implying inconsistent personas (e.g., "food contains meat, fruits, ..."), leading to responses conflicting with defined personas (e.g., "i like to eat meat"). therefore, it attaches great importance in modeling the interaction and dependency of different sources foot_0 in building knowledgegrounded dialogue systems. the absence of problem definitions and wellthere are three situations of responses in our datasets: 1) response without the need for any sources (null), 2) response using only personae description (from persona source), and 3) response using both persona and knowledge (from persona, documents sources). the example here presents the first and third situations. we highlight the response and used knowledge with the same color. in order to address the interaction and dependency issue between specific sources, we propose a novel framework, named source planner for personalized knowledge-grounded dialogues (safari). seeing the potential of large language models (llms) in planning the use of external information @xcite @xcite , we explore llms' capability of connecting different sources in the context of the personalized knowledge-grounded dialogue system. as illustrated in figure 1(b) , the whole response generation can be modeled into three steps in sa-fari: 1) planning to make a series of decisions of whether to use a specific knowledge source given relationship descriptions between different sources; 2) retrieval to retrieve top-n results from external databases according to the decisions; 3) assembling to incorporate all retrieved knowledge into the final response generation. benefiting from decoupling source selection and response generation, our framework is more flexible and scalable, allowing independent modification of each component. additionally, our framework can easily accommodate scenarios where multiple or no sources are required. to sum up, our contributions are listed below: ## analysis in this section, we analyze the effects of different components and the choice of the number of retrieved results, based on chatglm under the supervised setting. in addition, we conduct human evaluations to verify the quality of automatic evaluations. ## conclusion in this paper, we propose a novel framework sa-fari to incorporate multiple sources of knowledge 16 more analysis can be found bases and further address the dependency issue between them. unlike previous works, safari can be extended to multiple sources easily and it can handle cases that do not require any sources or require some instead of all sources between them. we build the first personalized knowledge-grounded dialogue (kbp) dataset, and experimental results prove the effectiveness and robustness of safari.