S truct GPT : A General Framework for Large Language Model to Reason over Structured Data (2023)

Abstract: In this paper, we aim to improve the reasoning ability of large language models (LLMs) over structured data in a unified way. Inspired by the studies on tool augmentation for LLMs, we develop an Iterative Reading-then-Reasoning (IRR) framework to solve question answering tasks based on structured data, called StructGPT. In this framework, we construct the specialized interfaces to collect relevant evidence from structured data (i.e., reading), and let LLMs concentrate on the reasoning task based on the collected information (i.e., reasoning). Specially, we propose an invoking-linearization-generation procedure to support LLMs in reasoning on the structured data with the help of the interfaces. By iterating this procedure with provided interfaces, our approach can gradually approach the target answers to a given query. Experiments conducted on three types of structured data show that StructGPT greatly improves the performance of LLMs, under the few-shot and zero-shot settings.

## introduction recently, large language models (llms) @xcite have made remarkable advancements in the nlp field. existing work @xcite has demonstrated that llms (e.g., chatgpt or gpt-4 (openai, 2023)) have strong zero-shot capability to solve a broad range of tasks using specially designed prompts, without task-specific fine-tuning. despite the successes, recent work has also revealed that llms may generate unfaithful information in conflict with the factual knowledge @xcite , and also fall short of mastering domainspecific or real-time knowledge @xcite . a direct solution to the above issues is to augment llms with external knowledge resources, so as to amend the incorrect generations. among these resources, structured data (e.g., knowledge graphs and databases), has been widely used as the carrier of the required knowledge for llms. unlike plain text, structured data is organized in a standardized format, conforming to some logical data model. for example, knowledge graphs (kgs) are often organized as fact triples that state the relations between head entities and tail entities, and data tables are organized in the form of column-indexed records by rows. however, as structured data has special data formats or schemas that llms have not seen during pretraining, they may be not fully grasped or understood by llms @xcite . a straightforward way to solve this problem is to linearize the structured data into a sentence that llms can well understand. while, the amount of structured data is often vast, making it infeasible to include all the data records in the input prompt. regarding the above challenges, we are inspired by the tool manipulation strategy for augmenting the abilities of llms @xcite . our basic idea is to incorporate specialized interfaces (e.g., extracting columns from tables) to manipulate the structured data records. with these interfaces, we can effectively reduce the search space of the data records, and more accurately identify the required evidence to fulfill specific tasks. in this way, llms can concentrate on reasoning based on the evidence obtained from the interfaces. to implement the interface-augmented approach, there remain two key problems, namely how to design suitable interfaces for specific tasks and how to utilize them for reasoning by llms, which are the focus of this work. to design suitable interfaces, we regard multiple types of structured data as black-box systems, and design the interfaces to provide accurate, efficient data access and filtering for llms. for each interface, its implementation is dependent on the characteristic of the structured data, while its functionality is general to all llms, with just a few arguments for specifying the data requirements. based on these interfaces, we propose an iterative reading-then-reasoning (irr) framework for llms to utilize the interfaces to solve the tasks based on structured data, namely structgpt. this framework considers two major functions to fulfill different tasks, namely collecting relevant evidence (reading) and inferring the answer or planning subsequent steps (reasoning). specifically, we propose an invoking-linearization-generation procedure to support llms in reading and reasoning on the structured data with the help of the external interfaces. by iterating this procedure with provided interfaces, we can gradually approach the target answer to a given question. to our knowledge, this is the first work that explores how to support llms in reasoning on multiple types of structured data (including tables, kgs, and dbs) in a unified paradigm. to evaluate the effectiveness of our approach, we conduct extensive experiments on a wide range of tasks (e.g., kgbased question answering (kgqa), table-based question answering (tableqa), and db-based textto-sql). experimental results on 8 datasets demonstrate that our approach can effectively enhance the reasoning performance of llms on structured data in zero-shot and few-shot settings, even comparable with competitive full-data supervised-tuning methods. for example, in kgqa, tableqa, and text-to-sql tasks, our approach yields an increase of 11.4% of hits@1 on webqsp, 4.2% of accuracy in tabfact, and 4.7% of execution accuracy in spider respectively, compared to directly using chatgpt in the zero-shot setting. ## related work reasoning over structured data. structured data (e.g., knowledge graphs, tables, and databases) is an important knowledge carrier for a variety of qa and reasoning tasks. early work focuses on designing specific model architectures tailored for each type of structured data, such as graph neural networks @xcite , table transformers @xcite , and tree-structured decoder @xcite . while achieving remarkable performance, these approaches lack generality for various types of structured data and are hard to be transferred across different tasks. re-cently, with the success of pre-trained language models (plms) (e.g., t5 @xcite , bart @xcite ), several methods @xcite ) have adopted plms as the general encoder or solver for different structured data and tasks. among them, unified-skg @xcite unifies a number of reasoning tasks over structured data into a text-to-text format, which concatenates the question and the linearized structured data as input, and then finetunes t5 to learn to generate the answer. however, unifiedskg also requires to tune the model parameters, and is unable to handle large-scale structured data under the limitation of the maximum input length. instead, our method can utilize the llm to perform reasoning on structured data without training, and also leverage the interfaces of structured data to better manipulate vast structured data. llms for structured data. benefitting from the strong few-shot and zero-shot capability, recent studies have leveraged llms to perform reasoning over structured data @xcite @xcite . existing work can be roughly divided into two types. the first type of method linearizes the structured data into a sentence (e.g., table rows), and feeds it into the llms to generate the answer according to in-context exemplars @xcite . for complex questions or structured data, they first decompose it into multiple simple and short ones and then perform linearization and generation @xcite . another type of method leverages llms to evaluate the plausibility of the solution plan based on the knowledge base @xcite , or first generate a solution draft with in-context exemplars and then revise the draft grounding on the knowledge base @xcite . however, most of them only focus on a specific type of structured data, and are lack of generality across various data and tasks. in structgpt, we provide a unified paradigm that is general to various structured data and downstream tasks. ## preliminary in this section, we introduce the definition of structured data, which mainly consists of three commonly used types. then we present the unified problem statement. structured data. structured data (e.g., data tables and knowledge graphs) refers to the data that is in a standardized format, conforming to some logical data model @xcite . due to the formal structure, it is easy and efficient to access and query structured data using formal languages (e.g., sql and sparql for databases) or specific algorithms (e.g., triples search for knowledge graphs). in this work, we mainly focus on three types of structured data, namely knowledge graphs (kg), data tables (table), and databases (db), since they play an important role as the knowledge source in helping solve complex reasoning tasks, described as follows. problem statement. this work mainly focuses on using llms to solve complex reasoning tasks based on structured data. formally, it can be described as a question answering task: given a natural language question q and an accessible structured data s (e.g., a knowledge graph, a table, or database), the llm needs to extract useful evidence from s and then generates the expected result to answer the question q based on the extracted evidence. according to the task requirement, the generated result can be either free-form answers in natural language or structured expressions (e.g., sql statements) to be executed for obtaining the answer from s. since we consider three types of structured data (section 4), our tasks can be instantiated as follows: ## experiment we conduct experiments on three complex reasoning tasks over structured data, i.e., kgqa, tableqa, and db based text-to-sql. ## conclusion in this work, we proposed a general framework for improving the zero-shot reasoning ability of llms over structured data, namely structgpt. in our approach, we first constructed the specialized interfaces that support accurate and efficient data access, and then proposed an invoking-linearizationgeneration procedure that leverages llms to read and perform reasoning based on the interface. by iterating the above procedure using the interfaces sequentially, llms can progressively capture more useful and detailed evidence and finally generate the answer. to verify the effectiveness of our approach, we implemented our approach on kg based qa, table based qa and db based semantic parsing tasks. experimental results on 8 datasets show that our approach can boost the zero-shot performance of llms by a large margin, and achieve comparable performance as full-data supervisedtuning methods. we also provide detailed error analysis to point out the weakness of our approach, for enlighting other researchers in related areas.