LI - RAGE : Late Interaction Retrieval Augmented Generation with Explicit Signals for Open-Domain Table Question Answering (2023)

Abstract: Recent open-domain TableQA models are typically implemented as retriever-reader pipelines. The retriever component is usually a variant of the Dense Passage Retriever, which computes the similarities between questions and tables based on a single representation of each. These fixed vectors can be insufficient to capture fine-grained features of potentially very big tables with heterogeneous row/column information. We address this limitation by 1) applying late interaction models which enforce a finer-grained interaction between question and table embeddings at retrieval time. In addition, we 2) incorporate a joint training scheme of the retriever and reader with explicit table-level signals, and 3) embed a binary relevance token as a prefix to the answer generated by the reader, so we can determine at inference time whether the table used to answer the question is reliable and filter accordingly. The combined strategies set a new state-to-the-art performance on two public open-domain TableQA datasets.

## introduction tabular data is ubiquitous on the web. opendomain table question answering (tableqa), the task of answering questions grounded in tables, is increasingly attracting attention of both public and commercial research, for its value in real-world applications. research tableqa pipelines are typically implemented with two components: a retriever and a reader. the retriever chooses a small set from the entire pool of table candidates, while the reader generates answers processing each table candidate. state-of-the-art implementations use transformer-based models for both components. in particular, the retriever is built with variants of dense passage retriever @xcite , which computes question-table similarity by using single vector representations of the question and the table. retriever and reader can be trained separately @xcite or jointly @xcite via retrieval augmented generation loss @xcite . we observe three limitations which we address in this paper. first, a table can be very large and might contain heterogeneous information across rows/columns; encoding into a fixed size vector risks information loss, which can have an impact in qa quality. one way to alleviate this issue is to replace dpr with a latent interaction (li) model, which encodes text into token-level representations. in particular, we find colbert @xcite to be very effective, even if not pretrained for tables. second, rag uses only an implicit signal to guide the retriever. recently, @xcite proposed rage loss (retrieval augmented generation with explicit signals) for visual qa, which in our setting rewards the retriever with table-level signals from the reader model in joint training. third, we observe empirically that the reader does not always rank answers coming from the gold table at the top. as our reader is a sequence-tosequence model, we propose a simple modification to the training data: we prepend binary relevance tokens @xcite ) to the answer itself. the reader learns to generate a first token indicating whether the table is relevant to the question or not. using these techniques, we build an end-to-end framework, li-rage, and achieve state-of-theart results on two benchmarks for open-domain tableqa, nq-tables @xcite and e2e-wtq @xcite . foot_0 ## related work while open-domain tableqa is yet a relatively unexplored problem, with only a few applications in the past couple of years, there has been extensive work on table retrieval and tableqa separately. in table retrieval, recent advances in ma-chine learning have enabled extracting deep features for tables with transformers @xcite , by designing models to parse complex tabular structure @xcite , or by simply linearizing tables with interleaving tokens to preserve its structure @xcite . in tableqa, until recently researchers assumed gold tables were given and focused on developing models that understood and answered questions over tables, i.e. the readers. earlier models generated commands in logical forms (e.g. sql queries) that were executable over tables @xcite @xcite , while recent state-of-the-art models directly predict the answers from the input question and table by either classification @xcite or autoregressive generation @xcite . following these advances, in open-domain tableqa the best performing systems are based on a retriever-reader pipeline @xcite . @xcite leverages tapas @xcite to both initialize a dpr-like retriever and the reader. t-rag @xcite uses dpr as retriever of rows/columns by decomposing the table and generates the answer via a sequence-tosequence reader @xcite , applying the rag loss to refine the retriever with implicit signals during end-to-end tableqa fine-tuning. unlike dtr and t-rag, cltr @xcite employs only retrieval of rows and columns and obtains the answer cell by intersecting the top-scored ones. in this work we focus mainly on the retriever, and unlike previous work that relies on single vector embeddings, we leverage late interaction retrievers @xcite to achieve a finer-grained interaction between questions and tables. in contrast to t-rag and cltr, we do not need to decompose the table into rows and columns, but retrieve a whole table from the corpus, ensuring that the reader is given all the relevant information. in addition, we explore different techniques for explicitly refining the retriever during end-to-end tableqa achieving superior performance. ## methodology given a question q, the tasks are to find the gold table t * from a table corpus t, i.e. table retrieval ( § 3.1), and to derive the answer denotations s (1 or more cells from the table), i.e. question answering over the retrieved tables ( § 3.2). we assume that labeled datasets consisting of triples {(q, s, t * )} are available to us. we flatten the tables into sequences with interleaving special tokens that encode its structure (see appendix a). ## experimental setup datasets and metrics. we evaluate our system on two benchmarks, i.e. nq-tables @xcite and e2e-wtq @xcite . 3 nq-tables contains generally hard questions extracted from the naturalquestions @xcite dataset, comprising the questions that can be answered from tables rather than plain text. for this benchmark, we evaluate the models using: token f1, i.e. token-wise f1 score; and exact match (em) or accuracy, i.e. whether predictions match the annotations. e2e-wtq contains look-up questions that require cell selection operation and is a subset of wik-itablequestions @xcite . in e2e-wtq train/valid/test splits are the same as in wikitablequestions, with questions limited to those that do not aggregations across multiple table cells. we evaluate models via accuracy 4 . in addition, we report recall@k for the retrieval performance in both, which measures whether the gold table is among the top-k retrieved tables. 5 system configurations. for the table retrieval component, we conduct contrastive experiments using both dpr and li. we first fine-tune the official pretrained dpr or colbertv2 model on each dataset before using them in the joint retrieverreader training. we do not train the tableqa model from scratch, instead we warm-start the training with tapex, a state-of-the-art pre-trained model for tabular data understanding based on bart @xcite . since the e2e-wtq is very small and not enough for learning a robust tableqa model, we additionally fine-tune tapex on its superset, i.e. wikitablequestions. note that no test samples are leaked due to this as the dataset splits of e2e-wtq are the same as wikitablequestions. we select the best checkpoints based on the validation set. we set @xmath0. ## conclusion we introduce a novel open-domain tableqa framework, li-rage, that leverages late interaction retrievers to enable finer-grained interaction between questions and tables. additionally, li-rage incorporates the rage loss and binary relevance tokens which enable significant improvements over the state-of-the-art in two challenging tableqa tasks.