Multi-Source Multi-Type Knowledge Exploration and Exploitation for Dialogue Generation (2023)

Abstract: Open-domain multi-turn dialogue generation encounters the significant challenge of lacking various types of knowledge from diverse sources. Existing models typically focus on identifying specific types of dialogue knowledge and utilize corresponding datasets for training. However, this approach often leads to limited generalization capabilities and increased computational resource requirements. Recently, large language models (LLMs) have shown impressive performance on natural language processing tasks. To harness the knowledge storage of LLMs, we propose a framework named KnowEE that explores multi-source multi-type knowledge from LLMs by leveraging diverse datasets and then exploits the obtained knowledge for response generation. Our framework comprises two phases: First, we leverage five external datasets encompassing various types of knowledge to extract the most relevant samples to the dialogue context which are served as prompts to generate corresponding type of knowledge; Second, we inject the acquired knowledge into the ongoing dialogue context in fine-grained and coarse-grained manners, which is then fed into LLMs to generate the final dialogue response. Both automatic and manual evaluation results validate the effectiveness of our framework in exploring and exploiting multi-source multi-type knowledge to generate coherent, informative, and fluent responses.

## introduction open-domain multi-turn dialogue generation necessitates models capable of producing high-quality responses that are coherent, consistent, and informative @xcite @xcite . in human conversations, we naturally connect various types of knowledge such as emotion, topic, event, persona, general world knowledge, etc. to facilitate smooth and high-quality communication. and this knowledge is gradually acquired through various sources in the process of learning after birth. for example, as depicted in figure 1 , when asked if lu xun and zhou shuren are the same person, humans draw upon various knowledge sources to comprehend different types of knowledge related to the dialogue history, including the speaker's emotion and intent, the overall topic of the dialogue, and relevant background knowledge about lu xun and zhou shuren. finally, we can provide a appropriate and correct response. therefore, enabling a model to explore and aggregate various types of knowledge from multiple sources, similar to how humans do, and exploit-ing them effectively will play a crucial role in improving the quality of intelligent human-machine conversations @xcite . previous studies have utilized external knowledge sources to bridge the knowledge gap between machines and humans in conversation by injecting knowledge that is difficult to learn purely from the given dialogue training dataset @xcite @xcite . however, these existing knowledge-enhanced dialogue generation approaches have several notable problems: (1) limited knowledge types, (2) monotonous knowledge sources, and (3) complicated and inefficient knowledge mining and fusion strategies. for example, incorporating external knowledge of specific type into the training process is a common strategy, such as commonsense and emotion @xcite , topic @xcite , persona @xcite , wikipedia knowledge @xcite , etc. nevertheless, these approach often suffers from poor generalization due to constraints in the knowledge types and sources. furthermore, the process of modeling knowledge for training or inference introduces additional computational resource overhead. another approach is to use search engines to generate responses based on internet search results @xcite . while this reduces training and inference costs, it fails to enable the model to fully understand the dialogue history as the knowledge is not stored internally. in addition, several researchers have endeavored to utilize large language models (llms) for dialogue generation tasks. these models, such as @xcite , have exhibited exceptional performance across various natural language processing tasks @xcite @xcite , and show two significant capabilities to (1) store and access various types of knowledge from extensive training sources @xcite , and (2) perform different downstream tasks without fine-tuning. intuitively, the knowledge storage and inference capacity as well as the text understanding and generation capabilities of llms can help generate better conversations, even for the zero-shot or few-shot scenarios. nevertheless, despite these superior capabilities, even the most advanced llms such as chatgpt, often encounter challenges in dialogue generation tasks, as depicted in figure 1 . although chatgpt "knows" that lu xun is a pseudonym for zhou shuren, it fails to accurately distinguish between the two persons when directly input the dialogue context into the chatgpt system. we analyze that relying solely on the dialogue historical context cannot effectively stimulate and explore the rich knowledge stored within the llms when generating responses. to maximize the utilization of the rich knowledge stored in llms for dialogue generation system, we propose a framework named knowee, which can conduct multi-source multitype knowledge exploration from llms and then conduct knowledge exploitation for better dialogue understanding and response generation. as mentioned above, in our work we identify the dialogue knowledge into five types: emotion, topic, persona, event, and general world knowledge. our framework knowee includes two phases: multisource multi-type knowledge exploration via incontext learning and fine/coarse-grained knowledge exploitation for response generation. for the first phase, given the dialogue context, we leverage five external source datasets with knowledge labels corresponding to the above five types of knowledge, to extract the most relevant samples to the given context. these samples will serve as prompts to explore the llms to generate knowledge for the corresponding type. next, we inject the generated knowledge into ongoing dialogue context using two different approaches: utterancelevel fine-grained knowledge injection (fgki) and dialogue-level coarse-grained knowledge injection (cgki). the combined information is then fed into llms to obtain the final response. we evaluated our proposed framework knowee on four dialogue generation tasks: empatheticdialogues (ed), dailydialog (dd), persona-chat (pc) and opendialkg (odkg), and the experimental results demonstrate that it outperforms the baseline methods in terms of both automatic and manual evaluation metrics. our contributions can be summarized as follows: additionally, we propose a fine-grained and a coarse knowledge injection approaches to combine the generated knowledge with the dialogue context. ## conclusion in this paper, we study the task of open-domain knowledge generation and identifies challenges of lacking various types of knowledge from diverse sources. to address these challenges, we identify five key type of dialogue knowledge, and propose a framework called knowee that explores multisource multi-type knowledge from large language models using external datasets, and exploits the knowledge in fine-grained and coarse-grained manners for response generation. it explores knowledge from llms and injects knowledge into dialogue context to generate final responses. experiments on four dialogue datasets show that knowee enhances llm's understanding of dialogue context and improves generated responses in terms of coherence, informativeness, and fluency. moreover, knowee exhibits robust generalization capabilities, making it applicable to multiple llms. in our future endeavors, we intend to refine the dialogue knowledge further and explore the untapped potential of large language models in tasks that extend beyond dialogue response.