ODA : Observation-Driven Agent for integrating LLM s and Knowledge Graphs (2024)

Abstract: The integration of Large Language Models (LLMs) and knowledge graphs (KGs) has achieved remarkable success in various natural language processing tasks. However, existing methodologies that integrate LLMs and KGs often navigate the task-solving process solely based on the LLM’s analysis of the question, overlooking the rich cognitive potential inherent in the vast knowledge encapsulated in KGs. To address this, we introduce Observation-Driven Agent (ODA), a novel AI agent framework tailored for tasks involving KGs. ODA incorporates KG reasoning abilities via global observation, which enhances reasoning capabilities through a cyclical paradigm of observation, action, and reflection. Confronting the exponential explosion of knowledge during observation, we innovatively design a recursive observation mechanism. Subsequently, we integrate the observed knowledge into the action and reflection modules. Through extensive experiments, ODA demonstrates state-of-the-art performance on several datasets, notably achieving accuracy improvements of 12.87% and 8.9%.

## introduction large language models (llms) @xcite @xcite have exhibited extraordinary capabilities across a variety of natural language processing tasks. despite their impressive accomplishments, llms often struggle to provide accurate responses to queries that necessitate specialized expertise beyond their pre-training content. in response to this limitation, a natural and promising approach involves the integration of external knowledge sources, such as knowledge graphs (kgs), to augment llm reasoning abilities. kgs provide structured, explicit, and explainable knowledge representations, offering a synergistic method to overcome the intrinsic constraints of llms. the fusion of llms with kgs has garnered significant interest in recent research @xcite , underlying a vast array of applications @xcite @xcite . existing methodologies for solving tasks that integrate kgs with llms can be categorized into two groups. the first one involves retrieving relevant triples from kgs in response to specific questions @xcite @xcite . the second part adopts an exploreexploit strategy, directing the knowledge utilization process within the graph according to the question @xcite . however, both categories navigate the task-solving process by merely relying on the llm's analysis of the question, overlooking the rich cognitive potential inherent in the abundant knowledge encapsulated 1 7417 in kgs. kgs, which store a wealth of informative and symbolic facts, should deeply participate in the reasoning process together with llm rather than being merely treated as a static repository of knowledge @xcite . as the example in the upper panel of figure 1 , llm analyzes the question and navigates towards narrative location relation of entity the call of the wild. however, this entity has many neighboring entities with that relation, leading llm to incorrectly infer canada as the answer. in contrast, the bottom panel demonstrates how kg provides key patterns that reveal both the call of the wild and white fang share the location yukon. if llm could observe this information beforehand, it would precisely guide its reasoning process towards the correct answer (as shown in the bottom panel). therefore, llm should adopt an overall observation to incorporate the extensive knowledge and intricate patterns embedded within the kg. achieving this objective presents two primary challenges: firstly, a global observation of the kg can result in an exponential growth in the number of triples. as shown in the upper panel of figure 1 , fully processing all 3-hop connections for the call of the wild is impractical. secondly, the integration of such comprehensive observation into the existing reasoning paradigms of llms presents another challenge. how to combine the observation with the reasoning process of llm matters for solving the tasks. motivated by this, we introduce a novel framework, the observation-driven agent (oda), aimed at sufficiently and autonomously integrating the capabilities of both llm and kg. oda serves as an ai agent specifically designed for kg-centric tasks. oda engages in a cyclical paradigm of observation, action, and reflection. within oda, we design a novel observation module to efficiently draw autonomous reasoning patterns of kg. our observation module avoids the problem of exponential growth of triples via recursive progress. this approach ensures oda integrating abilities of kg and llm while mitigating the challenges associated with excessive data in kg, improving the efficiency and accuracy. following the observation phase, oda takes action by autonomously amalgamating insights derived from llm inferences with the observed kg patterns. oda can perform actions of three distinct types: neighbor exploration, path discovery, and answering. subsequently, oda reflects on its internal state, considering both the outcomes of its actions and the prior observations. this iterative process continues until oda accomplishes the task at hand. we conduct extensive experiments to testify to the effectiveness of oda on four datasets: qald10-en, t-rex, zero-shot re and creak. notably, our approach achieved state-of-the-art (sota) performance compared to competitive baselines. specifically, on qald10-en and t-rex datasets, we observed remarkable accuracy improvements of 12.87% and 8.9%, respectively. we conclude the contributions as follows: ## methods in this work, we aim to solve tasks associated with kg. let q represent a user question. the task t can be defined as generating an answer y given a question q, task-relevant entities @xmath0, e 1 , ..., e k }, and a kg denoted as g. formally, the task t can be expressed as: employing an iterative approach, oda tackles the challenges inherent in kg-centric tasks. in contrast to existing methods that couple llms and kgs and rely solely on analyzing the llm's query, oda autonomously integrates observed knowledge from the kg into the entire reasoning process, resulting in more informed decisions. to achieve this objective, our oda system, illustrated in figure 2, primarily comprises three key modules for task resolution: through this iterative process, oda dynamically updates its observation subgraph o i and memory m i at each iteration i. each module is discussed in detail in the following sections. ## discussion to better understand the key factors influencing our oda, we conducted extensive analysis experiments. to conserve computational resources, we kept the previously mentioned datasets (qald10-en, creak, t-rex, and zero-shot re) but randomly sampled 400 examples each for creak, t-rex, and zero-shot re. ## conclusion in this work, we design oda framework for kgcentric tasks. in oda, we introduce kg observation mechanism to autonomously combine the reasoning abilities of kg with llm. we first propose the observation method to mitigate the problem of explosive number of triples in kg when tackling complex tasks. then we fuse the observation into the action and reflection modules to further enhance the overall performance. we conduct extensive experiments, and the results clearly illustrate the effectiveness of our framework, highlighting its capability to enhance performance across four kbqa datasets, particularly in handling complicated questions.