Graph vs. Sequence: An Empirical Study on Knowledge Forms for Knowledge-Grounded Dialogue (2023)

Abstract: Knowledge-grounded dialogue is a task of gener- ating an informative response based on both the dialogue history and external knowledge source. In general, there are two forms of knowledge: manu- ally annotated knowledge graphs and knowledge text from website. From various evaluation viewpoints, each type of knowledge has advantages and downsides. To further distinguish the principles and determinants from the intricate factors, we conduct a thorough experiment and study on the task to answer three essential questions. The ques- tions involve the choice of appropriate knowledge form, the degree of mutual effects between knowl- edge and the model selection, and the few-shot performance of knowledge. Supported by statistical shreds of evidence, we offer conclusive solutions and sensible suggestions for directions and standards of future research.

## introduction consistent human knowledge is compulsory to achieve an intelligent ai, except for a massive amount of data provided, whether a well-formed structural knowledge @xcite or human instruction @xcite . however, many dialogue systems are not proficient in integrating knowledge and cannot engage humans in a thorough discussion about particular subjects. in order to incorporate knowledge into the conversation, knowledge-grounded dialogue systems have gained popularity @xcite @xcite . the goal is to generate informative and meaningful responses based on dialogue history and external knowledge. so far, researchers have gathered datasets with various knowledge forms, such as the recommendation dialogues with knowledge graph @xcite and the open-domain dialogues with wikipedia @xcite . generally, there are two forms of knowledge: knowledge graphs and knowledge text. knowledge graphs are annotated by human experts and saved as triples @xmath0, relationship, tail en-tity>. because the knowledge is structured, the system can retrieve and use it to facilitate reasoning @xcite . however, a high-quality knowledge graph is quite labor expensive. instead, the text naturally serves as a medium of knowledge. knowledge text is derived from knowledgeintensive texts and easily parameterized by large language models. textual knowledge is unstructured, easy to collect, yet challenging to reason and explain directly. different forms of knowledge tend to serve different knowledge-intensive language tasks (kilt) @xcite @xcite @xcite . specifically, knowledge-grounded dialogue systems can receive either forms of knowledge graph or knowledge text to facilitate response generation; nevertheless, their performances are quite different regarding response quality and factual consistency, etc. the reason is closely tied to knowledge's characteristics. for instance, open-dialkg @xcite is a typical knowledge graph dataset, which annotates the reference triples based on a large knowledge graph for recommended conversations. however, the knowledge in opendialkg comes from freebase @xcite , which requires large-scale data annotation, and the knowledge is sparse (1-2 triples) for generation. on the contrary, wow dataset @xcite retrieves wikipedia passages for open-domain dialogue to assist intelligent agents in generating informative utterances. the information retrieved in wow is too dense (i.e., 7 passages) and noisy for models to figure out the accurate knowledge. thus, an extra selection module is re-quired to polish the pertinent knowledge. @xcite . therefore, determining the forms of knowledge is crucial and merits extensive discussion in light of the task's features. with a particular form of knowledge in hand, we have to choose an appropriate approach to incorporate the knowledge to support relevant tasks. though knowledge-grounded tasks are extensively studied, the majority aiming for advanced performance, few of them have exhaustively compared the determinants in the model and knowledge. @xcite proposed a largesize dual-encoders model on wow. however, according to our experiments, the base-size dual-encoders model may perform better on wow data. @xcite experimented with an encoder-decoder model on multiple datasets but ignored the dual-encoders are more compatible with the task. @xcite experimented on sentencelevel knowledge datasets, neglecting the effect of knowledge size and granularity. nevertheless, comparisons from multiple perspectives are needed to support the superiority and contribution of the model theoretically. however, considering all factors every time in each specific research is not feasible. therefore, to aid future research, we comprehensively analyze the influencing factors, such as model architecture, model size, pretrained models, etc., and then summarize several determinant conclusions. concretely, we leverage the scientific strategy of controlling variables to investigate how intricate factors affect and to what extent the model and knowledge should be mutually adaptive. furthermore, we also explore how various knowledge performs in few-shot settings as fewshot learning with the knowledge-auxiliary is an important application @xcite . to answer the above question comprehensively, we employ a unified transformer @xcite framework to observe the effects of different knowledge and serialize the knowledge graph to adapt to the input. in this paper, our investigation mainly focuses on three points: ## literature review knowledge-grounded dialogue is the task of generating an informative response r based on both dialogue history c and external knowledge k. it is becoming an increasingly important topic in our community. the involved knowledge can be in different forms, such as documents for open-domain conversation @xcite , movie database for movie recommendation conversation @xcite , and knowledge graph for recommendation conversation @xcite . recent research investigates several techniques for improving knowledge representation to fuse knowledge in response generation. @xcite encoded the dialogue history and knowledge documents separately to infuse the response with external world facts. @xcite added a particular knowledge graph representation in the response generation module. another line of work investigates pre-training methods for the knowledge-grounded dialogue to improve the system's performance on unseen topics and train in a low-resource setting. @xcite pre-trained the dialogue generation module and the knowledge encoder with ungrounded dialogues and the wikipedia dump separately. @xcite devised a variational network that can effectively estimate a generation model from a dialogue corpus and an independent knowledge corpus. @xcite proposed a three-stage learning framework to separately pre-train the dialogue module and knowledge module. inspired by the accomplishments of pre-trained language models (plm) for a range of natural language processing (nlp) tasks, researchers investigate learning knowledge through plm' parameters. @xcite equipped a pre-trained language model with a knowledge selection module and presented an unsupervised jointly optimizing technique. prabhumoye et al. (2021) focused on the attention mechanism in transformers and proposed a strong baseline by a pre-trained model. @xcite increased the faithfulness in knowledge-grounded dialogue with controllable features of pre-trained models. however, the above studies only considered a single form of knowledge, neglecting the impact of knowledge forms. @xcite proposed a pre-trained model for data-to-text tasks by unifying the knowledge format in the pre-training data and downstream tasks. @xcite used unified essential knowledge triples instead of documents in pre-training, making their model more generalizable. motivated by these methods, we conduct a thorough investigation of knowledge-grounded dialogue in various knowledge forms to explore the impact of knowledge. we focus on how to choose a knowledge form (sequence or graph) for nlg and investigate questions about which knowledge form is better and how it affects performance. our study will facilitate the subsequent data collection and model construction and even influence the direction of knowledge-intensive language tasks. ## experimental settings in order to investigate the effect of various forms of knowledge, we comprehensively analyze the performance of different knowledge. further, we also consider the factors of models to identify the interrelation between models and knowledge. this section introduces the detailed experimental settings for a clearer illustration foot_0 . ## experimental analyses this section presents our detailed investigation of how the knowledge forms affect the knowledgegrounded dialogue, focusing on the better forms of knowledge, the mutual adaption of model and knowledge, and the performance under few-shot settings. in this section, large pre-trained models (gpt2-large and bart-large) are leveraged for initialization unless otherwise highlighted. we use sequence-or graph-based models to refer to models that use knowledge text or knowledge graph as knowledge, respectively. to facilitate the following analyses, we highlight the knowledge's characteristics of different datasets. opendialkg provides manually annotated knowledge graphs and knowledge text, which are short and accurate knowledge. faithdial and wow both provide knowledge text and the corresponding knowledge graphs are extracted by automated tools. for knowledge text, faithdial's knowledge is sentence-level knowledge annotated by humans, while the knowledge in wow ranges over long passages retrieved from wikipedia. in comparison, the extracted knowledge graph in both faithdial and wow is not as good quality as manually annotated opendialkg. therefore, the results of graph-based models on opendialkg can be regarded as the upper-bound of knowledge graph grounded dialogue and the results of sequencebased models on faithdial and opendialkg can be regarded as the upper-bound of knowledge text grounded dialogue. however, as retrieving the precise knowledge sentence or triples is difficult in real-world practice, the results on wow is much closer to reality. 4.1 q1: graph v.s. sequence, which form of knowledge form is better? our analyses examine the validity of the knowledge text and graph in knowledge-grounded dialogue and attempt to determine the better form. response quality our observations indicate that in the faithdial dataset, knowledge graphs improve response diversity in the "seen" setting, while knowledge text enhances diversity in the "unseen" setting. conversely, in the wow dataset, knowledge text improves diversity in the "seen" setting, whereas knowledge graph improves it in the "unseen" setting. these findings suggest that lower-quality knowledge can lead to greater diversity when the domain/topic is familiar during training, while higher-quality knowledge may be more beneficial when the domain/topic is unfamiliar. furthermore, we speculate that the variation in diversity across datasets and knowledge formats may be attributed to differences in modeling difficulty and generalization capabilities. for instance, decoder-only architecture can readily model fine-grained sentences and generalize to unseen examples in faithdial but may struggle with complex knowledge passages in wow.