Open Domain Question Answering over Tables via Dense Retrieval (2021)

Abstract: Recent advances in open-domain QA have led to strong models based on dense retrieval, but only focused on retrieving textual passages. In this work, we tackle open-domain QA over tables for the first time, and show that retrieval can be improved by a retriever designed to handle tabular context. We present an effective pre-training procedure for our retriever and improve retrieval quality with mined hard negatives. As relevant datasets are missing, we extract a subset of Natural Questions (Kwiatkowski et al., 2019) into a Table QA dataset. We find that our retriever improves retrieval results from 72.0 to 81.1 recall@10 and end-to-end QA results from 33.8 to 37.7 exact match, over a BERT based retriever.

## introduction models for question answering (qa) over tables usually assume that the relevant table is given during test time. this applies for semantic parsing (e.g., for models trained on spider @xcite ) and for end-to-end qa @xcite . while this assumption simplifies the qa model, it is not realistic for many use-cases where the question is asked through some open-domain natural language interface, such as web search or a virtual assistant. in these open-domain settings, the user has some information need, and the corresponding answer resides in some table in a large corpus of tables. the qa model then needs to utilize the corpus as an information source, efficiently search for the relevant table within, parse it, and extract the answer. recently, much work has explored open-domain qa over a corpus of textual passages @xcite @xcite . these approaches usually follow a two-stage framework: (1) a retriever first selects a small subset of candidate passages relevant to the * work completed while interning at google. s read ("chlorine") s read (...) s read ("latin") which element is named for the greek word for green? tapas q (q) list of chemical element name etymologies element origin meaning ... fluorine latin a flowing ... chlorine greek pale green ... ... ... ... ... ## setup we formally define open domain extractive qa over tables as follows. we are given a training set of n examples d @xmath0, t i , a i )} n @xmath1, where q i is a question, t i is a table where the answer a i resides, and a corpus of m tables @xmath2 our task shares similarities with open domain qa over documents @xcite @xcite , where the corpus c consists of textual passages extracted from documents instead of tables, and the answer is a span that appears in some passage in the corpus. as in these works, dealing with a large corpus (of tables in our setting), requires relevant context retrieval. naively applying a qa model, for example tapas @xcite , over each table in the large corpus is not practical because inference is too expensive. to this end we break our system into two independent steps. first, an efficient table retriever component selects a small set of candidate tables c r from a large corpus of tables c. second, we apply a qa model to extract the answer a given the question q and the candidate tables c r . ## dense table retrieval in this section we describe our dense table retriever (dtr), which retrieves a small set of k candidate tables c r given a question q and a corpus c. in this work we set @xmath3. as in recent work for open domain qa on passages @xcite @xcite @xcite , we also follow a dense retrieval architecture. as tables that contain the answer to q do not necessarily include tokens from q, a dense encoding can better capture similarities between table contents and a question. for training dtr, we leverage both in-domain training data d train , and automatically constructed pre-training data d pt of text-table pairs (see below). ## question answering over tables a reader model is used to extract the answer a given the question q and k candidate tables. the model scores each candidate and at the same time extracts a suitable answer span from the the answer span extraction is modeled as a softmax over all possible spans up to a certain length. spans that are located outside of a table cell or that cross a cell are masked. following @xcite , the span representation is the concatenation of the contextual representation of the first and last token in the span s: the training and test data are created by running a retrieval model. we extract the @xmath4. at training time we add the reference table if it is missing from the candidates. at inference time all table candidates are processed and the answer of the candidate with the highest score is returned as the predicted answer. ## dataset we create a new english dataset called nq-tables from natural questions @xcite . concurrently with this work, @xcite study a similar subset of nq but without the retrieval aspect. nq was designed for question answering over wikipedia articles. the 320k questions are mined from real google search queries and the answers are spans in wikipedia articles identified by annotators. although the answers for most questions appear in textual passages, we identified 12k examples where the answer resides in a table, and can be used as a qa over tables example. to this end, we form nq-tables that consists of (q, t, a) triplets from these examples. tables are extracted from the article's html, and are normalized by transposing infobox tables. we randomly split the original nq train set into train and dev (based on a hash of the page title) and use all questions from the original nq dev set as our test set. to construct the corpus c, we extract all tables that appear in articles in all nq sets. nq can contain the same wikipedia page in different versions which leads to many almost identical tables. we merge close duplicates using the following procedure. for all tables that occur on the same wikipedia page we flatten the entire table content, tokenize it and compute l 2 normalized unigram vectors of the token counts of each table. we then compute the pair-wise cosine similarity of all tables. we iterate over the table pairs in decreasing order of similarity and attempt to merge them into clusters. this is essentially a version of single link clustering. in particular, we will merge two tables if the similarity @xmath5.91, they do not occur on the same version of the page, their difference is rows is at most 2 and they have the same number of columns. ## conclusion in this paper we demonstrated that a retriever designed to handle tabular context can outperform other textual retrievers for open-domain qa on tables. we additionally showed that our retriever can be effectively pre-trained and improved by hard negatives. in future work we aim to tackle multimodal open-domain qa, combining passages and tables as context.