Triad: A Framework Leveraging a Multi-Role LLM -based Agent to Solve Knowledge Base Question Answering (2024)

Abstract: Recent progress with LLM-based agents has shown promising results across various tasks. However, their use in answering questions from knowledge bases remains largely unexplored. Implementing a KBQA system using traditional methods is challenging due to the shortage of task-specific training data and the complexity of creating task-focused model structures. In this paper, we present Triad, a unified framework that utilizes an LLM-based agent with multiple roles for KBQA tasks. The agent is assigned three roles to tackle different KBQA subtasks: agent as a generalist for mastering various subtasks, as a decision maker for the selection of candidates, and as an advisor for answering questions with knowledge. Our KBQA framework is executed in four phases, involving the collaboration of the agent’s multiple roles. We evaluated the performance of our framework using three benchmark datasets, and the results show that our framework outperforms state-of-the-art systems on the LC-QuAD and YAGO-QA benchmarks, yielding F1 scores of 11.8% and 20.7%, respectively.

## introduction a question-answering system is designed to extract information by converting a natural language question into a structured query that can retrieve precise information from an existing knowledge base @xcite . the resolution of knowledge base question answering (kbqa) typically involves phases including question understanding, uri linking, and query execution. traditional kbqa systems require the use of specialized models trained with domain datasets for question parsing and entity linking @xcite @xcite . large language models (llms), however, have shown promising competencies in in-context learning using task-specific † corresponding authors. in this study, we introduce triad, a unified framework that leverages an llm-based agent with three roles to address kbqa tasks. specifically, we implement the agent consisting of an llm as the core, supplemented by various task-specific modules such as memory and executing functions. the agent is assigned three distinct roles: a generalist (g-agent) adept at mastering numerous small tasks by the given examples, a decision maker (d-agent) proficient at identifying options and selecting candidates, and an advisor (a-agent) skilled at providing answers using internal and external knowledge. the cooperation of these agent roles composes a kbqa process containing four phases: question parsing, uri linking, query construction, and answer generation. we evaluate our framework on three benchmark datasets in various difficulties. the results show that our framework outperforms state-of-the-art systems, demonstrated by 11.8% and 20.7% f1 scores on the lc-quad and yago-qa benchmarks, respectively 1 . the contributions of this study can be summarized as follows: ## triad framework the overall architecture of triad is shown in figure 2 . each role of the llm-based agent, along with its associated subtasks, is illustrated as follows. ## conclusion in this study, we aim to bridge the gap between kbqa tasks and the investigation of llm-based agents. we introduce triad, a framework to address the kbqa task through an llm-based agent acting as multiple roles, including a generalist capable of mastering diverse tasks given minimal examples, a decision-maker concentrating on option analysis and candidate selection, and an advisor skilled in answering questions with the aid of both external and internal knowledge. triad achieves the best or competitive performance across three benchmark datasets compared to traditional kbqa systems and pure llm models. in future research, we plan to broaden our framework to handle more intricate questions, such as multi-hop reasoning, and exploring the integration between our framework and retrieval-augmented generation.