Bridging Information Gaps in Dialogues with Grounded Exchanges Using Knowledge Graphs (2024)

Abstract: Knowledge models are fundamental to dialogue systems for enabling conversational interactions, which require handling domain-specific knowledge. Ensuring effective communication in information-providing conversations entails aligning user understanding with the knowledge available to the system. However, dialogue systems often face challenges arising from semantic inconsistencies in how information is expressed in natural language compared to how it is represented within the system’s internal knowledge. To address this problem, we study the potential of large language models for conversational grounding, a mechanism to bridge information gaps by establishing shared knowledge between dialogue participants. Our approach involves annotating human conversations across five knowledge domains to create a new dialogue corpus called BridgeKG. Through a series of experiments on this dataset, we empirically evaluate the capabilities of large language models in classifying grounding acts and identifying grounded information items within a knowledge graph structure. Our findings offer insights into how these models use in-context learning for conversational grounding tasks and common prediction errors, which we illustrate with examples from challenging dialogues. We discuss how the models handle knowledge graphs as a semantic layer between unstructured dialogue utterances and structured information items.

## introduction conversational grounding is an integral aspect of dialogues where interlocutors share information and build up a common understanding. this mutually established knowledge serves as context for subsequent interactions. for building effective dialogue systems, the natural language processing (nlp) community has long focused on conversational grounding, which involves inferential reasoning, dynamic feedback, and repair strategies @xcite . despite extensive research, challenges remain in adapting to different conversation domains, addressing semantic vocabulary mismatches, overcoming information gaps between user knowledge and the system's internal knowledge model, as well as the lack of appropriate training data @xcite . owing to rapid technical advances regarding large language models (llms), novel opportunities arise to comprehend contextual intricacies within dialogues and reconcile information expressed in natural language with that stored in machine-readable data structures. recognizing the limited research on llm-based conversational grounding, we investigated the capabilities of llms on knowledge grounding tasks. this involved annotating an existing corpus containing dialogues about different domain-specific tabular datasets. in addition to labeling grounding acts, we annotated grounded knowledge items in a knowledge graph structure, a powerful representation of complex relationships between entities and their attributes. knowledge graphs have proven valuable in various nlp tasks, such as disambiguating ambiguous utterances by providing contextual information @xcite . for example, in dialogue systems, knowledge graphs can help identify the correct meaning of a word with multiple senses or resolve references to specific entities, enhancing the overall understanding and coherence of conversations. we opted for the json-ld format due to its simplicity and acceptance as a web standard, allowing interoperability by reusing existing namespaces with shared vocabularies to model knowledge from different sources and domains. while json-ld primarily uses a tree-like structure, it can represent more complex graph structures by linking nodes using identifiers like @id and @type. as a serialization format for resource description framework (rdf) data, json-ld can be transformed into other formats, such as n-triples, rdf/xml, or turtle. this flexibility allows json-ld to be integrated with graph databases and other rdf tools, enhancing its utility in various applications. our contributions include (1) creating a novel dialogue corpus called bridgekg with over 250 conversational grounding annotations across five knowledge domains, (2) conducting a range of zeroand few-shot experiments by evaluating four llms on two grounding tasks, and (3) summarizing common prediction errors and prompting techniques for improving model performance. to ensure the reproducibility of our experiments, we provide the bridgekg dataset, source code, and evaluation outputs in a public github repository. 1 2 related work in regard to the literature on grounding in nlp, it is essential to first define the broadly used term. grounding can be categorized into three main types. conversational grounding ensures a common understanding of shared knowledge within a conversation @xcite . perceptual grounding links language to sensory experiences of the real world like visual information @xcite . knowledge grounding incorporates external information sources to support nlp systems, such as providing factual knowledge to generative language models @xcite . our study focuses solely on conversational grounding by employing llms, a topic addressed in only a few recent studies. one related work by @xcite examines whether llm generations contain grounding acts, simulating turntaking from various conversation datasets. they found that llms generate language with less conversational grounding than humans, often producing text that appears to assume common ground. both their study and ours focus on the three grounding acts: explicit grounding, implicit grounding, and clarification, as proposed by @xcite . two other closely related studies, conducted by @xcite and @xcite , involve annotating dialogue corpora and employing language models to classify grounding acts and extract grounded knowledge items. while the former conducts preliminary experiments on two conversations with gpt-3.5-turbo, the lat-1 github.com/philotron/bridge-kg ter presents two annotated dialogue corpora with grounding acts, grounding units, a measure of their degree of grounding, and a baseline evaluation with the open-source t5 model @xcite . unlike the mentioned related work, we are the first to conduct a series of llm experiments aimed at knowledge identification in information-seeking conversations utilizing an in-context knowledge graph structure for identifying referenced and grounded knowledge items in dialogues. ## results and discussion classification of grounding acts ## conclusion and future work our study examined llms for handling groundingrelated knowledge in information-sharing dialogues. we found that classifying grounding acts was feasible for both open-and closed-source llms, with open-source llms performing on par compared with leading proprietary ones. however, identifying grounded knowledge proved to be a distinctly more complex task. for the latter, the proprietary llms had a competitive edge, and the open-source models underperformed due to their higher predisposition to generate erroneous output. the experiment results from our newly created dataset highlight common prediction issues and demonstrate how few-shot prompting can enhance model outputs, offering valuable insights to advance research on conversational grounding. future work should concentrate on developing llm-based dialogue systems that handle conversational grounding through a multi-component pipeline approach for recognizing groundingspecific dialogue acts as well as grounded knowledge @xcite . in previous studies, we have shown that llms can augment dialogue systems by performing semantic parsing for conversational question answering over knowledge graphs @xcite and by verbalizing retrieved semantic triples into text responses @xcite . we believe conversational grounding is essential as it links the processes of semantic parsing of dialogue utterances, knowledge identification, and response generation, aligning the user's prior knowledge with the system's available knowledge base while maintaining the relevance and coherence of conversations.