An Inner Table Retriever for Robust Table Question Answering (2023)

Abstract: Recent years have witnessed the thriving of pretrained Transformer-based language models for understanding semi-structured tables, with several applications, such as Table Question Answering (TableQA).These models are typically trained on joint tables and surrounding natural language text, by linearizing table content into sequences comprising special tokens and cell information. This yields very long sequences which increase system inefficiency, and moreover, simply truncating long sequences results in information loss for downstream tasks. We propose Inner Table Retriever (ITR), a general-purpose approach for handling long tables in TableQA that extracts sub-tables to preserve the most relevant information for a question. We show that ITR can be easily integrated into existing systems to improve their accuracy with up to 1.3-4.8% and achieve state-of-the-art results in two benchmarks, i.e., 63.4% in WikiTableQuestions and 92.1% in WikiSQL. Additionally, we show that ITR makes TableQA systems more robust to reduced model capacity and to different ordering of columns and rows. We make our code available at: https://github.com/amazon-science/robust-tableqa .

## introduction tables offer a systematic way of storing information in the web. extracting information from web tables poses different challenges than extracting information from relational databases with logical queries, especially when queried via natural language (nl) user questions. table question answering (tableqa) is the task of answering such questions with factoid answers extracted from table content. this requires developing models with the ability to reason over and understand tables. following the success of the pre-training paradigm for understanding nl text @xcite , some recent research has focused on pretraining transformer models @xcite c 1 c 2 c 3 c 4 r 1 r 2 r 3 r 4 mountain peak elevation red slate mountain 13,162 ft mount ritter 13,149 ft c 2 c 4 r 3 r 4 (a) (b) answer: mount whitney answer: mount ritter on large corpora of linearized tables in a selfsupervised fashion using encoder-only architectures @xcite @xcite , or encoder-decoder architectures @xcite . these so-called tabular language models @xcite were fine-tuned on the tableqa downstream taskamong others-to achieve state-of-the-art performance @xcite . however, the self-attention mechanism in talms has a quadratic complexity on the dimensionality of input tables, which might consist of tens or even hundreds of rows and columns, thus yielding longer sequences than talms can easily handle. state-ofthe-art tableqa models handle this limitation by truncating the linearized table to fit an input length budget, e.g., of 512 and 1024 tokens for @xcite and @xcite , respectively. in other applications, simple sequence truncation might be reasonable, e.g., encoding only the initial paragraphs of a wikipedia document pre-suming it comprises a summary, or dropping earlier turns in conversational qa to focus on the recent ones. however, in tableqa it is not realistic to assume that relevance depends on the position within the linearized sequence, especially because different questions require various table regions to be properly answered. for example, even for a standard dataset such as wikitablequestions, naive truncation allows information loss affecting 18.1%-44.9% of tables, which limits qa accuracy (see §4.1 for more details). this is also an important limitation in latency-constrained realistic use cases that use big tables, while limiting transformer models even down to 64 input tokens. to this end, a content-driven strategy is needed to avoid information loss. we refer to tables exceeding the input length budget as overflow tables, as opposed to compact tables, which fit within the budget. an example of an overflow table is shown in figure 1 (a), where naive truncation leads to the wrong answer. in this work, we propose itr to improve on this problem by creating smaller sub-tables, i.e., within a length budget, based on dense retrieval of table rows and columns according to the relevance to the question. an example is shown in figure 1 (b). our method is flexible and can be integrated off-theshelf into virtually any existing tableqa system. to the best of our knowledge, our work is the first to propose a neural-based sub-table selection in the context of tableqa that improves denotation accuracy @xcite , especially for the overflow tables, setting a new state of the art. other input selection strategies, mainly heuristics-based, have also been proposed in the literature @xcite @xcite ), which we discuss further in section 2. to summarize, the contributions of this work are the following: ## related work the works most related to ours employed different pruning strategies to handle large tables. yin et al. (2020, tabert) introduce the concept of content snapshot as encoder input. this snapshot is composed of a small number of rows which are chosen based on the n-gram similarity between the question and column headers and cell values. in a similar fashion, eisenschlos et al. (2020) explore jaccard similarity to obtain the most similar columns. in addition, they leverage model tokenizer to reduce cell tokens to their first token, when necessary, and dropping entire rows that do not fit the length budget. however, lexical similarity and naive truncation are not flexible and lead to information loss, which has a drastic effect on tableqa performance, as we show in our experiments. another line of work focuses on balancing model efficiency and accuracy when handling long tables. krichene et al. (2021, dot) first uses a smaller pruning transformer to selects top-k tokens from the input table, and then a larger second task-specific transformer takes into consideration only the selected k tokens and their pruning scores; eisenschlos et al. (2021, mate) can accept more tokens while not significantly increasing latency. authors apply sparse self-attention and use different attention heads for tokens in the same column and row. however, their proposed mechanisms are deeply integrated with the task-specific model and must be jointly trained. in contrast, our itr method, as a flexible plug-in process, can work independently of any underlying tableqa model. moreover, our approach is complementary to theirs since itr can drop irrelevant information in tables efficiently and pass the trimmed compact table to the underlying model. this can further exploit the potentials of virtually any tableqa models and improves their performance. although not specific to tableqa, works such as wang et al. (2021) and chen et al. (2021) employ chunking strategies, i.e., encoding table chunks separately and then aggregate them together. however chunking is not widely employed in the literature @xcite , due to encoding overhead requiring multiple inference calls for each chunk. algorithm 1 creating n sub-tables from t for q. 3: e q ← e q (q) 4: for each @xmath0: e t ← e t (i) 6: z ← z ∪ {sim(e q , e t ), i} 7: for each @xmath1: t aux ← t aux ∪ i 9: if checkvalid(t aux ) then 10: t sub ← t sub ∪ t aux 11: for each @xmath2: l ← l ∪ {length(t), t} 13: t sub ← [] 14: for each (l, t) ∈ sorted(l) do ▷ ↓ length 15: if @xmath3: t sub ← t sub ∪ t 17: return t sub [ : n ] 3 methodology 3.1 task given a question q and a table t , tableqa systems return an answer denotation a, either by performing table cell selection or as the result of operations (such as counting) carried out over an aggregation of table cells. this work aims to find one or more sub-tables t sub containing the most relevant information from t needed to answer q; t sub can replace t as the input to virtually any existing tableqa system. to this end, we map a table t into a set of items, where an item is either a complete row or a complete column. for an n × m table, this gives a set of items {r 1 , . . . , r n , c 1 , . . . , c m }. then, we construct sub-tables by specifying subsets of these rows and columns: a sub-table consists of the cells at the intersection of the selected rows and columns. we refer to each such set of rows and columns as a mix, and note that a mix must contain at least one row and one column to specify a valid sub-table. the table in figure 1 (a) is defined as {r 1 , r 2 , r 3 , r 4 , c 1 , c 2 , c 3 , c 4 }, and the sub-table in figure 1 (b) is specified as the mix {r 3 , r 4 , c 2 , c 4 }. {c 2 , c 4 } is not a valid sub-table, since no cells will be intersected with only column-wise items. ## conclusions in this paper we presented itr for tableqa, an approach to create the most relevant sub-table(s) to efficiently answer a given question via tableqa. itr is based on a dense retrieval component, which selects relevant rows and columns and combines them into a compact sub-table that satisfies length budget constraints. we combined itr with different tableqa models from the literature, at training and/or inference time, and showed that itr indeed captures the most relevant information, which enabled underlying models to perform better overall and become more robust, thus attaining state-ofthe-art results on wikisql and wikitq benchmarks. itr is flexible, does not depend on the underlying model and can be easily integrated in future model developments to increase their robustness. as future work we can combine itr with computational operations over different table elements @xcite to collapse its information in a more compact format, to benefit also questions that rely on table completeness.