Dual Reader-Parser on Hybrid Textual and Tabular Evidence for Open Domain Question Answering (2021)

Abstract: The current state-of-the-art generative models for open-domain question answering (ODQA) have focused on generating direct answers from unstructured textual information. However, a large amount of world’s knowledge is stored in structured databases, and need to be accessed using query languages such as SQL. Furthermore, query languages can answer questions that require complex reasoning, as well as offering full explainability. In this paper, we propose a hybrid framework that takes both textual and tabular evidences as input and generates either direct answers or SQL queries depending on which form could better answer the question. The generated SQL queries can then be executed on the associated databases to obtain the final answers. To the best of our knowledge, this is the first paper that applies Text2SQL to ODQA tasks. Empirically, we demonstrate that on several ODQA datasets, the hybrid methods consistently outperforms the baseline models that only takes homogeneous input by a large margin. Specifically we achieve the state-of-the-art performance on OpenSQuAD dataset using a T5-base model. In a detailed analysis, we demonstrate that the being able to generate structural SQL queries can always bring gains, especially for those questions that requires complex reasoning.

## introduction open-domain question answering (odqa) is a task to answer factoid questions without a prespecified domain. recently, generative models @xcite @xcite have achieved the state-of-the-art performance on many odqa tasks. these approaches all share the common pipeline where the first stage is retrieving evidence from the free-form text in wikipedia. however, a large amount of world's knowledge is not stored as plain text but in structured databases, and need to be accessed using query languages such as sql. furthermore, query languages can answer questions that require complex reasoning, as well as offering full explainability. in practice, an ideal odqa model should be able to retrieve evidence from both unstructured textual and structured tabular information sources, as some questions are better answered by tabular evidence from databases. for example, the current state-of-the-art odqa models struggle on questions that involve aggregation operations such as counting or averaging. one line of research on accessing databases, although not open domain, is translating natural language questions into sql queries @xcite @xcite @xcite @xcite @xcite . these methods all rely on knowing the associated table for each question in advance, and hence are not trivially applicable to the open-domain setting, where the relevant evidence might come from millions of tables. in this paper, we provide a solution to the aforementioned problem by empowering the current generative odqa models with the text2sql ability. more specifically, we propose a dual readerparser (durepa) framework that can take both textual and tabular data as input, and generate either direct answers or sql queries based on the context 1 . if the model chooses to generate a sql query, we can then execute the query on the corresponding database to get the final answer. overall, our framework consists of three stages: retrieval, joint ranking and dual reading-parsing. first we retrieve supporting candidates of both textual and tabular types, followed by a joint reranker that predicts how relevant each supporting candidate is to the question, and finally we use a fusion-in-decoder model @xcite for our readerparser, which takes all the reranked candidates in addition to the question to generate direct answers or sql queries. to evaluate the effectiveness of our durepa, we construct a hybrid dataset that combines squad @xcite and wikisql @xcite questions. we also conduct experiments on naturalquestions (nq) @xcite and ott-qa @xcite to evaluate durepa performance. as textual and tabular open-domain knowledge, we used textual and tabular data from wikipedia via wikidumps (from dec. 21, 2016) and wikitables @xcite . we study the model performance on different kinds of questions, where some of them only need one supporting evidence type while others need both textual and tabular evidence. on all question types, durepa performs significantly better than baseline models that were trained on a single evidence type. we also demonstrate that durepa can generate humaninterpretable sqls that answer questions requiring complex reasoning, such as calculations and superlatives. our highlighted contributions are as follows: • we propose a multi-modal framework that incorporates hybrid knowledge sources with the text2sql ability for odqa tasks. to the best of our knowledge, this is the first work that investigates text2sql in the odqa setting. with that, we achieve the state-of-the-art performance on opensquad using a t5-base model. open domain question answering odqa has been extensively studied recently including extractive models @xcite @xcite @xcite that predict spans from evidence passages, and generative models @xcite @xcite @xcite ) that directly generate the answers. @xcite ; @xcite proposed to rerank the retrieved passages to get higher top-n recall. table parsing text2sql is a task to translate natural questions to executable sql queries. @xcite proposed senlidb dataset which only contains 29 tables and lacks annotation in their training set. recently, with datasets like wik-isql @xcite , spider @xcite and cosql @xcite being introduced, many works have shown promising progress on these dataset @xcite @xcite @xcite @xcite @xcite @xcite . another line of work proposes to reason over tables without generating logical forms @xcite @xcite . however, they are all closed-domain and each question is given the associated table. hybrid @xcite also proposed an open-domain qa problem with textual and tabular evidence. unlike our problem, they generate an answer directly from the tabular evidence instead of generating an sql query. in addition, they assume some contextual information about table is available during retrieval stage (e.g. their fusion-retriever is pretrained using hyperlinks between tables and paragraphs), whereas we don't use any link information between tables and passages. moreover, @xcite proposed a closed-domain hybrid qa dataset where each table is linked to on average 44 passages. different from ours, their purpose is to study multi-hop reasoning over both forms of information, and each question is still given the associated table. ## method in this section, we describe our method for hybrid open-domain question answering. it mainly consists of three components: (1) a retrieval system; (2) a joint reranker and (3) a dual seq2seq model that uses fusion-in-decoder @xcite to generate direct answer or sql query. ## experiments in this section, we report the performance of the proposed method on several hybrid open-domain qa datasets. ## discussion and future work our experiments consistently show that the proposed framework durepa brings significant improvement on answering questions using hybrid types of evidence. especially on the questions that can be answered by both supporting evidence types, our multi-modal method still shows clear advantage over models using single-type knowledge, implying that our approach could figure out the most relevant evidence to answer a question. we also demonstrate that the dual reader-parser is essential to the good performance of durepa; the ability of generating both direct answers and structural sql queries help durepa perform much better than fid+ and other baselines on questions that require complex reasoning like counting or averaging. we believe that our methods can be improved in two aspects. first, our general framework fig. 1 can be improved by a better retrieval system. for example, instead of using bm25, we can use more powerful neural retrieval models @xcite . on the hybrid evidence, one can also use an entity linking module to link the entities between the tables and passages @xcite ) and utilize the structure information for better multi