SPINACH : SPARQL -Based Information Navigation for Challenging Real-World Questions (2024)

Abstract: Large Language Models (LLMs) have led to significant improvements in the Knowledge Base Question Answering (KBQA) task. However, datasets used in KBQA studies do not capture the true complexity of KBQA tasks. They either have simple questions, use synthetically generated logical forms, or are based on small knowledge base (KB) schemas.We introduce the SPINACH dataset, an expert-annotated KBQA dataset collected from discussions on Wikidata’s “Request a Query” forum with 320 decontextualized question-SPARQL pairs. The complexity of these in-the-wild queries calls for a KBQA system that can dynamically explore large and often incomplete schemas and reason about them, as it is infeasible to create a comprehensive training dataset. We also introduce an in-context learning KBQA agent, also called SPINACH, that mimics how a human expert would write SPARQLs to handle challenging questions. SPINACH achieves a new state of the art on the QALD-7, QALD-9 Plus and QALD-10 datasets by 31.0%, 27.0%, and 10.0% in F_1 , respectively, and coming within 1.6% of the fine-tuned LLaMA SOTA model on WikiWebQuestions.On our new SPINACH dataset, the SPINACH agent outperforms all baselines, including the best GPT-4-based KBQA agent, by at least 38.1% in F_1 .

## introduction wikidata @xcite , one of the largest publicly available knowledge bases, currently contains 15 billion facts and is estimated to grow at a rate of 1 billion triples per year @xcite . it is of significant value to * equal contribution § work conducted while at the wikimedia foundation 1 code and data available at https://github.com/ stanford-oval/spinach . the spinach agent is publicly available at https://spinach.genie.stanford.edu/ and on wikidata as spinachbot. question: who are the doctoral advisors of leonhard euler, and their advisors, and so on? in addition, who are his doctoral student, grand-student, great-grand-student, and great-great-grand-students? each tuple in the result should contain both the student and the doctoral advisor. sparql: select ?doctor ?doctormaster where { { ?root (wdt:p184*) ?doctor. } union { ?root (wdt:p185/(wdt:p185?)/(wdt:p185?)/(wdt:p185?)) ?doctor. } ?doctor wdt:p184 ?doctormaster. values ?root { wd:q7604 } } figure 1(b): the corresponding decontextualized example included in the validation set of the spinach dataset. the sparql is based on the "refined sparql" proposed by the user tomt0m. projection fields asking for labels and the optional image clause are removed in accordance with our methodology in section 3.1. many scientific communities, including mathematics @xcite , biology @xcite , education @xcite , linguistics @xcite , and the social sciences @xcite , among many others @xcite . effective access to wikidata data can be challenging. to address this, the wikidata request a query forum foot_0 has been created so that users can ask questions, propose solutions, and participate in follow-up conversations. figure 1 (a) shows one such conversation, in which a user wants to obtain the academic genealogy of leonhard euler. note that in sparql, entities and properties are uniquely identified by qids and pids, respectively. here p184 and p185 represent the "doctoral advisor" and "doctoral student", respectively, and q7604 is leonhard euler. readers are referred to @xcite for more information. although numerous datasets have been proposed for knowledge base question answering (kbqa) task, they either contain only simple questions @xcite @xcite @xcite @xcite or synthetically generated complex logical forms @xcite @xcite @xcite . datasets with synthetically generated logical forms often lead to an overestimate of the performance of kbqa systems; performing well on them does not translate to real-world queries @xcite . thus, the community needs a highquality dataset with organic, real-world queries that capture the true complexity of kbqa tasks. in this paper, we propose using the wikidata request a query forum to build and evaluate next-generation kbqa systems. we introduce the spinach dataset, a new, expertannotated kbqa dataset featuring decontextualized question-sparql pairs derived from complex discussions on a real-world forum. figure 1(b) is the example obtained from the discussion in figure 1(a) . as we show in this paper, current kbqa approaches @xcite cannot handle the complexity of these real-world queries. the state-of-the-art (sota) approach, tog @xcite , which integrates llms with kg reasoning, scores only 1.8 em and 7.2 f1 on this data set. therefore, we propose a new llm-augmented kbqa approach, spinach: sparql-based information navigation for challenging real-world questions, which is designed with the primary goal of mimicking how an expert would tackle the task. we show that spinach establishes a new state of the art on popular datasets such as qald-7, qald-9, and qald-10 and comes within 1.6% of the fine-tuned sota on wikiwebquestions. on the spinach dataset, our agent outperforms all baselines, including the best gpt-4-based kbqa agent, by at least 38.1% in f1. ## the spinach dataset the archives of the wikidata request a query from july 2016 to the present date are available foot_2 . these conversations are real and organic, reflecting the kind of queries that practitioners are interested in. additional details on the forum and analysis of the conversations can be found in appendix c.1. out of the discussions up to may 2024, 2780 discussions include at least one valid sparql query. we keep conversations whose last-mentioned sparql query returns non-empty results under 10 mb. of the 2171 discussions left, we randomly sample 920 conversations spanning many domains for consideration. these conversations typically do not specify the exact natural language corresponding to the sparqls. thus, manual processing is required to convert them to a kbqa dataset. three wikidata experts among the authors of this paper manually inspected these conversations to produce a dataset with 155 examples in the validation and 165 examples in the test set, as described below. additional discussions on the size and statistical power of spinach dataset can be found in appendix c.2. more information on the process of converting source conversations and sparqls into a kbqa dataset can be found in appendix c.3. ## the spinach agent the spinach agent is an llm-augmented knowledge graph exploration agent, where an llm is instructed to explore the knowledge graph until an appropriate answer is found or another stop condition is met. however, unlike prior work, we design spinach with the primary goal of mimicking a human expert writing a sparql query. an expert starts by writing simple queries and looking up wikidata entity or property pages when needed, all to understand the structure of the knowledge graph and what connections exist. this is especially important for wikidata due to its anomalous structure @xcite . an expert then might add new sparql clauses to build towards the final sparql, checking their work along the way by executing intermediate queries and eyeballing the results. potentially, they might go back to the drawing board and double-check their assumptions if a query fails. following this design principle, spinach improves upon the following aspects of previous agent-based kbqa systems: first, spinach uses the full expressiveness of sparql for exploration. this contrasts with @xcite ; @xcite ; @xcite , which explore the knowledge graph one edge at a time. that is, spinach weaves together the exploration and the writing of sparql. it is instructed to try many sparqls and fail early. it learns about the structure of the knowledge graph from the results of its queries (or lack thereof), or realizes its incorrect assumptions, and decides to revisit them. this is a key difference from @xcite , where the answer is generated only when exploration is done. second, during its exploration, spinach does not keep track of a subgraph as its state. instead, we define the state to be the full history of actions and their results so far. limiting the state to a subgraph @xcite means that the agent would categorically not support questions whose answer contains a large number of entities (e.g. "which actors graduated from harvard?") or computation (e.g. "what is the tallest mountain?"). third, while most kbqa agents start their exploration from entities they detect from the question, spinach does not assume access to entities. this, for example, gives the agent the flexibility to start by searching for relevant relations if that is a more suitable way to explore. concretely, spinach agent runs for n time steps. at time step n, it takes the current state as input and outputs a thought @xcite t n and one of the possible actions a n from the set of all possible actions. the action is then executed outside of the agent, and the results are returned to the agent as observation o n . in the rest of this section, we go into the details of the spinach agent. figure 2 shows an example of how spinach answers a question.