SPAGHETTI : Open-Domain Question Answering from Heterogeneous Data Sources with Retrieval and Semantic Parsing (2024)

Abstract: We introduce SPAGHETTI: Semantic Parsing Augmented Generation for Hybrid English information from Text Tables and Infoboxes, a hybrid question-answering (QA) pipeline that utilizes information from heterogeneous knowledge sources, including knowledge base, text, tables, and infoboxes. Our LLM-augmented approach achieves state-of-the-art performance on the Compmix dataset, the most comprehensive heterogeneous open-domain QA dataset, with 56.5% exact match (EM) rate. More importantly, manual analysis on a sample of the dataset suggests that SPAGHETTI is more than 90% accurate, indicating that EM is no longer suitable for assessing the capabilities of QA systems today.

## introduction open-domain question answering (qa) grounded in knowledge corpora has long been an active topic of research in natural language processing @xcite @xcite @xcite @xcite . with the rise of llms, new state of the art has been established with qa separately on free-text documents @xcite @xcite , databases @xcite @xcite , and graph databases @xcite @xcite . in practice, we need to fully leverage hybrid data sources. for instance, wikipedia alone offers a wealth of knowledge through nearly 7m free-text articles; many of these articles contain structured information in tables and infoboxes; wikidata is a knowledge graph containing over 17 billion triples. this paper investigates how to leverage llms to answer questions on all the different types of data. figure 1 : given an input query, spaghetti gathers factual information from four sources to generate a prediction. in parallel, we parse the query to logical form to query wikidata (left), run retrieval to find information from wikipedia text, tables, and infoboxes (middle), and generate a response using llm, only keeping a claim if it is verified (right). finally, evidences are gathered to generate an answer. the premise of this paper is that we need hybrid data and we need hybrid access methods. our main contribution is twofold. first, we introduce a hybrid llm-based system (fig. 1 ), spaghetti, that combines information retrieval with semantic parsing for question answering and achieves sota of 56.5% exact match rate on compmix, the most comprehensive open-domain qa dataset on heterogeneous sources. second, we show via careful evaluation and analysis that measuring the accuracy of llm-based qa systems with the exact-match metric against hand-annotated answers is obsolete. by using evaluation methods closer to human judgment, we find that spaghetti is more than 90% accurate on compmix, surprisingly higher than traditional methods would suggest. ## related work textqa, tableqa. and kbqa have all been individually studied extensively @xcite @xcite . however, the task of answering questions from two or more sources, known as heterogeneous qa, is under-studied. some literature investigate two of the three sources, including those on closed domain @xcite @xcite @xcite and open domain @xcite @xcite @xcite , but very limited existing work experiments on all three. convmix @xcite collected the first conversational qa dataset that requires knowledge from all three heterogeneous sources. crowdworkers were asked to pick an entity of their interest and find the answer from one of the ## spaghetti spaghetti is a hybrid qa pipeline that takes advantage of both structured and unstructured information. we obtain evidence from heterogeneous sources in parallel, including structured knowledge bases, plain text, linearized tables / infoboxes, and llm-generated claims that are verified, and gather those evidence to generate the final answer using a few-shot llm (fig. 1 ). how many nominations for the academy award did the wizard of oz receive? ('the wizard of oz', 'q193695') refined only misses entity "academy award" 1. the wizard of oz is a 1939 american musical fantasy film. 2. academy award is an annual awards ceremony … refined ('the wizard of oz', 'q193695') ('academy awards', 'q19020') ## experiments we evaluate spaghetti on the compmix development and test sets, which contain 1680 and 2764 questions respectively. for querying wikidata, we use the llama-7b semantic parser from @xcite trained on both wikiwebquestions and qald-7 @xcite . we use gpt-3.5 as the llm in our entity linking module. we experiment with llama-7b, gpt-3.5-turboinstruct, and gpt-4 foot_3 , respectively, as the llm backbone in all the stages for handling retrieved evidences and for answer generation. we use fewshot prompts for gpt-3.5 and gpt-4, and use the llama model from @xcite , which is distilled from the teacher gpt-4. to fine-tune the colbert table retriever, we obtain training data from the nq-tables dataset @xcite , where each example matches one gold table to a query. for each positive example, we sample 10 negative tables to obtain a total of 95k training triplets. we confirmed on the nq-tables dataset that the fine-tuned version improves table retrieval recall@3 by 10%. evaluation metrics. @xcite and @xcite have established that exact match (em) against gold answers, which is commonly used for evaluating qa systems, cannot evaluate generative models properly as they often generate lexically different, but semantically equivalent answers. to properly assess our approach, we introduce two additional evaluation metrics: (1) superset: whether the gold answer is a substring of the generated answer, as the latter tends to spell out the answer in long form and may include a more ablations. we evaluate the contribution of each knowledge source by ablating different parts of the system ( human evaluation we examine how our human "platinum" evaluation (92%) differs from the em metric (60%) on our sample of 100 cases. we report findings on the spaghetti (gpt-4) responses, specifically in 32 annotated examples where em fails but the response is indeed correct (figure 3 ). out of the 32 discrepancies, the unsophisticated "superset" metric resolves 7, and gpt-4 matching resolves an additional 14. platinum evaluation identifies that 4 questions have incorrect gold labels, and 7 questions are ambiguous and the generated answers are correct though different from the gold. we include examples for each of these resolved cases in appendix b. of the 5 true errors, one is because spaghetti cannot find the answer in any of the four information sources; in the other 4 cases, the answer generator cannot identify the correct answer re- superset 21.9% format 12.5% paraphrasing 31.2% multiple correct answers 21.9% gold wrong 12.5% trieved due to conflicting or misleading evidence. see appendix c for details on each error case. ## conclusion we propose spaghetti, a hybrid open-domain question-answering system that combines semantic parsing and information retrieval to handle structured and unstructured data. spaghetti achieves an exact match rate improvement of 6.3% over the prior state-of-the-art on the compmix dataset. more importantly, we show that our approach is likely to reach an accuracy of over 90%, if we account for differences in the answer wording and incompleteness/errors in gold labels. this, however, does not mean opendomain qa is solved. further research is needed to handle open-domain questions that require complex structured queries or composition of answers from multiple information sources, none of which are included in compmix.