Fine-tuned LLM s Know More, Hallucinate Less with Few-Shot Sequence-to-Sequence Semantic Parsing over W ikidata (2023)

Abstract: While large language models (LLMs) can answer many questions correctly, they can also hallucinate and give wrong answers. Wikidata, with its over 12 billion facts, can be used to ground LLMs to improve their factuality. This paper presents WikiWebQuestions, a high-quality question answering benchmark for Wikidata. Ported over from WebQuestions for Freebase, it consists of real-world data with SPARQL annotation. This paper presents a few-shot sequence-to-sequence semantic parser for Wikidata. We modify SPARQL to use the unique domain and property names instead of their IDs. We train the parser to use either the results from an entity linker or mentions in the query. We fine-tune LLaMA by adding the few-shot training data to that used to fine-tune Alpaca. Our experimental results demonstrate the effectiveness of this methodology, establishing a strong baseline of 76% and 65% answer accuracy in the dev and test sets of WikiWebQuestions, respectively. By pairing our semantic parser with GPT-3, we combine verifiable results with qualified GPT-3 guesses to provide useful answers to 96% of the questions in dev. We also show that our method outperforms the state-of-the-art for the QALD-7 Wikidata dataset by 3.6% in F1 score.

## introduction large language models (llms) such as gpt-3 can answer open-domain questions without access to external knowledge or any task-specific training examples. however, llms are prone to hallucinate @xcite , while using a convincing and confident tone. this may cause significant harm as people increasingly accept llms as a knowledge source @xcite . an entity linker is used to link entities in the user query to their unique id in wikidata; e.g. "a bronx tale" is linked to entity id "q1130705". the query and entity linker outputs are fed to the wikisp semantic parser to produce a modified version of sparql, where property ids (e.g. "p915") are replaced by their unique string identifiers (e.g. "film-ing_location"). if applying the query to wikidata fails to return a result, we default to gpt-3, labeling the result as a gpt-3 guess. returned answers are presented in the context of the query, so the user can tell if the answer is acceptable; if not, we also show the guess from gpt-3. here wikisp mistakenly uses "filming_location" instead of "narrative_location"; the user detects the mistake, thumbs down the answer, and the gpt-3 answer is provided. in contrast, traditional knowledge base question answering (kbqa) is grounded with a given knowledge base. semantic parsing (sp) has been widely used to tackle this challenging task, where the questions are first parsed into a logical form and then executed to retrieve answers from the knowledge base. it has better interpretability than gpt-3 and other information-retrieval-based approaches @xcite @xcite where answers are predicted directly. to handle large knowledge bases, previous spbased approaches tend to use a multi-stage pipeline of sub-tasks, starting with extracting the relevant subgraph based on entities detected in the questions @xcite . such an approach struggles with questions that have a large search space and fails to understand questions that refer to information missing in the knowledge graph. having to retrieve the relevant subgraphs to create the logical form conflates query resolution with semantic parsing, rendering classical query optimization inapplicable. end-to-end seq2seq translation, on the other hand, has mainly been used on schemas of relatively small relational databases @xcite and web apis @xcite . to handle large knowledge graphs, recent work proposed retrieving (1) information on linked entities, (2) exemplary logical forms relevant to the query @xcite , and (3) schemas as context to semantic parsing @xcite . others use induction or iterative methods to generate complex logical forms @xcite . ## semantic parsing for wikidata wikidata is the largest public knowledge base with over 12 billion facts represented by subjectpredicate-object triples using 100+ million entities and 10,000 properties. 3,000 of the properties are useful for answering natural language questions, whereas the rest are used to link data in wikidata with external library catalogs and database ids. entities and properties are given unique identifiers, qids and pids, respectively. for example, the fact that joe biden is the president of the us can be represented as a triple (q6279, p39, q11696), where p39 is the pid for property position held, q6279 and q11696 are qids for joe biden and the president of the united states, respectively. ## wikiwebquestions (wwq) dataset despite being the most popular large knowledge base for a long time, existing benchmarks on wikidata with labeled sparql queries are unfortunately either small or of low quality. on the other hand, benchmarks over the deprecated freebase still dominate the kbqa research with betterquality data. for example, the webquestions @xcite dataset was collected by using google search api instead of human paraphrasing or synthesis. as a result, it is much more natural and truly reflects the real-world questions users may ask. this dataset is later annotated with sparql over freebase, named webquestionssp @xcite . examples with no legitimate sparql to retrieve answers from freebase are dropped. in total, webquestionssp consists of 3098 examples in the training set and 1639 in the test set. we migrated webquestionssp, the best collection of natural language questions over a general knowledge graph, from freebase to wikidata, with the help of an automatic tool we developed, based on google's entity mapping 2 and wikidata's relation mapping 3 . about 60% of the dataset was automatically converted. one of the authors of this paper, who did not participate in model tuning, manually converted those instances that failed to convert automatically. ## implementation this section discusses the implementation details of the entity linker and the wikisp semantic parser. ## experiments in this section, we evaluate wikisp on wikiweb-questions and demonstrate how it can be used to complement large language models such as gpt-3. ## experiment with qald-7 for another evaluation of wikisp, we apply our model on task 4 from qald-7 @xcite dataset. qald-7 is part of the qald (question answering over linked data) which is a series of challenges started in 2011 known for their complex, manually created questions. it mainly focuses on dbpedia, but qald-7's task 4 is engineered for wikidata. the task includes 100 train examples, which we use to fine-tune our model and 50 test examples. there is no dev set. we choose qald-7 as it is a manually crafted dataset with complex questions. we avoid datasets built on synthetic or human-paraphrased data, such as csqa @xcite and kqa-pro @xcite . as they have limited natural language variety between the training and evaluation em f1 stagg (yih et al., 2016) -19.0 ggnn (sorokin and gurevych, 2018) -21.3 wdaqua (diefenbach et al., 2017) -40.0 wikisp (ours) 38.0 43.6 the qald-7 test set provides both the sparql queries as well as the answers. to double-check the correctness of the qald-7 dataset, we applied the 50 gold queries of the test set to wikidata and found that 4 did not return an answer. we hypothesize that the discrepancy is caused by the change in wikidata structure/quantity of information. we evaluate wikisp by comparing the answers where possible, and by comparing the generated sparql syntactically otherwise. for this experiment, we use the same hyperparameters and data format as described in section 5.3. in addition to the training data for wikisp, we also include the qald-7 train samples, upsampled 20 times. ## conclusion we have created a new high-quality benchmark, wikiwebquestions, for large knowledge-base question answering. the dataset is based on the popular webquestionssp dataset with natural questions, annotated with sparql for wikidata. we establish a first, strong baseline of 65% answer accuracy and 72% f1 score for wikiweb-questions. this is achieved by fine-tuning llama with a few-shot training data set using a sparql query format modified for semantic parsing. we show that we can reduce the hallucination of large language models like gpt-3 by grounding it with a semantic parser. for the dev set of wiki-webquestions, this combination approach provides useful information for 96% of the questions in the dev set of the benchmark. more importantly, it generates verifiable answers for 76% of the questions.