Capturing Row and Column Semantics in Transformer Based Question Answering over Tables (2021)

Abstract: Transformer based architectures are recently used for the task of answering questions over tables. In order to improve the accuracy on this task, specialized pre-training techniques have been developed and applied on millions of open-domain web tables. In this paper, we propose two novel approaches demonstrating that one can achieve superior performance on table QA task without even using any of these specialized pre-training techniques. The first model, called RCI interaction, leverages a transformer based architecture that independently classifies rows and columns to identify relevant cells. While this model yields extremely high accuracy at finding cell values on recent benchmarks, a second model we propose, called RCI representation, provides a significant efficiency advantage for online QA systems over tables by materializing embeddings for existing tables. Experiments on recent benchmarks prove that the proposed methods can effectively locate cell values on tables (up to ~98% Hit@1 accuracy on WikiSQL lookup questions). Also, the interaction model outperforms the state-of-the-art transformer based approaches, pre-trained on very large table corpora (TAPAS and TaBERT), achieving ~3.4% and ~18.86% additional precision improvement on the standard WikiSQL benchmark.

## introduction tabular data format is a commonly used layout in domain specific enterprise documents as well as open domain webpages to store structured information in a compact form @xcite . in order to make use of these resources, many techniques have been proposed for the retrieval of tables @xcite @xcite @xcite . given a large corpus of documents, the goal in these studies is to retrieve top-k relevant tables based on given keyword(s). the user is then expected to skim through these tables and locate the relevant cell values which is a tedious and time consuming task. more recently, popular search engines made significant improvement in understanding natural language questions and finding the answers within passages, owing to the developments in transformer based machine reading comprehension (mrc) systems @xcite @xcite @xcite . one natural extension of these systems is to answer questions over tables. these questions are broadly classified into two types: lookup and aggregation. lookup questions require returning exact strings from tables such as cell values whereas aggregation questions are executed by performing an arithmetic operation on a subset of the column cells, such as min(), max(), average() and count(). for look-up questions, the users can verify if the returned cell values from the table(s) are correct, while this is not applicable for aggregation questions because a scalar value is returned as an answer. our primary focus in this paper is on lookup questions since the answers are verifiable by users although our proposed techniques outperform the state-of-the-art (sota) approaches on both question types. in this paper, we propose a new approach to table qa that independently predicts the probability of containing the answer to a question in each row and column of a table. by taking the row and column intersection (rci) of these probabilistic predictions, rci gives a probability for each cell of the table. these probabilities are either used to answer questions directly or highlight the relevant regions of tables as a heatmap, helping users to easily locate the answers over tables (see figure 1 for a question answered with the help of a heatmap). we developed two models for rci, called rci interaction and rci representation. in order to evaluate these approaches, we also propose a weakly supervised mrc system as a strong baseline to identify / "read" relevant cells of a table. in this baseline approach, we convert tables into passages and extract a relevant span of text within these passages. the interaction model is designed to provide very high accuracy on finding cell values over tables for a given natural language question. we demonstrate that without even using any specialized pre-trained models, we can achieve up-to ∼98% hit@1 accuracy on finding cell values of tables for lookup questions from the wikisql benchmark. also, the interaction model outperforms the state-of-the-art transformer based approaches, tapas @xcite and tabert @xcite , achieving ∼3.4% and ∼18.86% additional precision improvement on the standard wik-isql benchmark, containing both lookup and aggregation questions. while the interaction model yields very high accuracy on the benchmarks, the representation model has the advantage of pre-computing the embeddings for all tables in a corpus and storing them for online query processing. once a user query is received, the most relevant tables can be retrieved from a table retrieval system and relevant cell values can be highlighted using the existing embeddings of the tables, resulting in less computation per received user query, as opposed to running tables over expensive transformer architecture for every received query. the specific contributions of this paper are as follows: in the following sections, we first review the prior work on qa systems over tables as well as table search from large corpora in section 2. we then describe a weakly supervised machine reading comprehension (mrc) system as a baseline that is capable of answering questions over tables in section 3. in section 4, we introduce two models that decompose tableqa as the intersection between rows and columns of a table using a transformer architecture. experimental results are reported and discussed in section 5 and finally section 6 concludes the paper and discusses the future work. ## related work qa from text: there is plenty of work on qa from plain text @xcite @xcite @xcite . typical strategies rely on token overlap between the question and passage text either based on a bag of word statistics or contextualized language model representations. in either case, tabular structure is not leveraged to capture semantic relationships between rows and columns. as we show in section 5, these strategies are insufficient for answering questions over tables with high precision. qa over tables: our work mostly relates to the previous research on qa over tables @xcite @xcite . they center around answering factoid questions and return the exact cell of a table that answers the query. we briefly describe here how these works are different. @xcite assume access to the 'gold' table that contains the answer to the input question. they build a semantic parser that parses the query to a logical form. they likewise convert the table into a knowledgegraph and execute the logical form on it to get the answer. a more advanced semantic parsing based methodology has been recently proposed by @xcite . this system is pre-trained on wik-itablesquestions @xcite . the proposed approach leverages an lstm encoderdecoder model where tables are first converted to a knowledge-graph and word tokens in the questions are linked to table entities (columns and cells). the questions and linked table entities are then encoded into representation vectors which are decoded to executable λ-dcs logical forms. this logical forms are executed over a knowledge graph to get answer predictions. our approach is different, since we do not convert natural language questions into logical forms and execute them on tables. instead, we leverage transformer architectures pre-trained on large corpora and further trained on finding cell values on tables. in section 5, we show that we achieve significant improvement over this approach without using any semantic parser technique. @xcite focus on the table retrieval problem over table corpora by leveraging the content of cell values and headers. for a given query, they extract answers from millions of tables in the provided corpus. they construct a unified chain representation of both the input question and the table cells and then find the table cell chain that best matches the question chain. as opposed to this work, we primarily focus on answering questions over a single table rather than the retrieval of top-k tables from a corpus. more recently, transformer based pre-training approaches have been introduced in tabert @xcite and tapas @xcite to improve accuracy for table qa. tabert has been pre-trained on 26 million tables and nl sentences extracted from wikipedia and wdc webtable corpus @xcite . the model can be plugged into a neural semantic parser as an encoder to pro-vide contextual embeddings for tables. herzig et al. on the other hand, claim that semantic parsers incur an extra overhead of computing intermediate logical representations which can be avoided by leveraging fine-tuned models to answer questions over tables. the model in tapas has been pretrained on about 6 million tables extracted from wikipedia content. our work is different from both tapas and tabert. first and foremost, our focus in this paper is not on pre-training a new model for table qa, but rather on leveraging the existing language models to find the connection between a question and table columns/rows with very high accuracy. second, our goal is to provide a heatmap over tables on an end-to-end table retrieval system to help users to quickly identify the regions of tables where the answers would most likely appear. because the transformer architectures are quite expensive to query, the representation model we propose radically reduces the computational overhead during online query processing. table search over the web: another active research area in nlp is searching over web tables. there are numerous search algorithms that have been explored such as keyword search @xcite @xcite , retrieve similar tables @xcite , retrieve tables based on column names @xcite and adding new columns to existing entity lists @xcite . this thread of work focuses on retrieval of top-k tables with high precision from large corpora, rather than finding relevant rows and columns within tables. ## mrc model we provide a brief description of our underlying machine reading comprehension (mrc) model architecture, which we use as a strong baseline. the architecture is inspired by @xcite @xcite and direct interested readers to their papers for more details. our mrc model follows the approach introduced by @xcite of starting with a pretrained transformer based language model (lm) and then fine-tuning mrc specific feed-forward layers on both general question answering datasets (squad 2.0 and nq) as well as the table specific question answers associated with the datasets in section 5. we use albert @xcite as the underlying lm similar to models which achieve sota on the squad 2.0 leaderboard @xcite at the time of writing. more specifically, we show results starting from the weights and dimensions of the base v2 version (25m parameters) of the lm shared by @xcite . we also experiment with the xxlarge v2 version (235m parameters) as well. the input to the model is a token sequence (x) consisting of a question, passage, and special markers (a [cls] token for answerability classification and [sep ] tokens to dileneate between the query and passage). the input token sequence is passed through a deep transformer @xcite network to output a sequence of contextualized token representations h. mrc then adds two dense layers followed by a softmax: the model is trained using binary cross-entropy loss at each token position based on whether or not the annotated correct answer begins or ends at the t th token. unanswerable questions have their begin and end offsets set to the [cls] token position. at prediction time, a score is calculated for each possible span by summing the α t j b and α t i e at each possible i and j combination to identify the max scoring answer span. the sum of the α we fine-tune the model with the squad 2.0 dataset and then the nq dataset in line with @xcite , to produce a generic rc model comparable to the current sota. we then train for an additional epoch on the subset of nq which consists of short answer questions that need to be answered by lookup inside an html table. this is about 5% of the total nq data (∼ 15, 500 question-answer pairs). note that in these cases, the input "passage" text consists of textual representation of tables (i.e., we introduce tabs between columns and new line characters between rows); so it is devoid of true row and column structure. this pre-training and task adaptation strategy is inline with prior art @xcite in adapting transformers. simpler pre-training strategies (e.g. relying only on squad 2.0 or skipping the table specific epoch of training) were tried and found to provide similar, but generally worse, performance. so those are excluded from section 5 for brevity. finally, we fine-tune (i.e., train for an additional epoch) on the training examples (table-question pairs) associated with the appropriate evaluation data sets described in section 5. during this step we do not have access to exact span offsets in the ground truth annotations and, instead, use weak supervision by matching the first occurrence of the answer text within the textual representation of the ## rci model architecture the row-column intersection model (rci) is motivated by the idea of decomposing lookup table qa into two operations: the column selection and the row selection. combining the predicted answer probability of each row and the probability of each column gives a score for all cells in the table. the highest scoring cell may then be returned as an answer, or highlighting may be applied to the table to aid a user in locating the relevant information. unlike the pointer network of an adapted machine reading comprehension system (described in section 3), the rci model always gives a ranked list of cells rather than answer spans that may cross cell boundaries. we observe that the process of identifying the correct column is often about matching the column header and the type of values in the column to the expected answer type of the question. for example in identifying the correct row is often more difficult. in the example given in representation models independently project each sequence of the sequence-pair to a vector, then compare those vectors. representation models are motivated by the need to improve efficiency for a practical system. considering the column classifier, the interaction model requires running a transformer over each question plus column sequence. in contrast, the representation model can pre-process the collection of tables, producing a vector representation of each column for each table, independent of any query. then, at query time, the query is projected to a vector which is then combined with the vector for each column and classified with a single-layer network. on the wikitablequestions-lookup dev set, we see the column model's time drop from 40 seconds to 0.8 seconds on a k80 gpu when ten queries are batch processed at once. let a table with m rows and n columns be defined as a header, @xmath0, h 2 , ..., h n ] and cell values @xmath1,j ], 1 ≤ @xmath2, 1 ≤ @xmath3. a tableqa instance consists of a table, a question and a ground truth set of cell indices, t ⊆ i × j, @xmath4, 2, ..., m, @xmath5, 2, ..., n. in principle, these ground truth cell positions could be annotated with the correct occurrences of the correct values. however, this form of supervision may be too difficult to obtain. we use weak supervision: the ground truth cell indices are found by matching the ground truth answer strings in the table. to train the row and column classifier we find ground truth row and column indices: although it is possible to naïvely construct a sequence representation of columns and rows by simply space separating the contents of each row or column, better performance can be achieved by incorporating the table structure in the sequence representation. we focus on tables with a single header for columns, but this method could also be applied to tables with a hierarchical header, by first flattening the header. the row (s r i ) and column (s c j ) sequence representations are formatted as: where ⊕ indicates concatenation and the functions ζ h and ζ v delimit the header and cell value contents. for ζ h we append a colon token (':') to the header string, and for ζ v we append a pipe token ('|') to the cell value string. the particular tokens used in the delimiting functions are not important. any distinctive tokens can serve since the transformer will learn an appropriate embedding to represent their role as header and cell value delimiters. considering again the example in in the representation model shown in figure 2 the representations of the question (r q ) and the jth column sequence (r c ) are first computed independently. the representations are taken from the vector that the transformer model produces for the [cls] input token. these vectors are then concatenated (indicated as :) with their element-wise product (indicated as ⊗) and the element-wise square of their differences. the probability that this column is the target for the question is then given by a softmax over a linear layer. layer and softmax 𝒑(𝒚𝒆𝒔) 𝒑(𝒏𝒐) 𝐿 !" weak supervision label figure 2: rci representation model extension to aggregation questions: although our focus is on lookup questions, the rci model can be extended to aggregation questions with the addition of a question classifier. another transformer is trained to classify the sequence-pair of the question and the table header into one of six categories: lookup, max, min, count, sum and average. the table header is relevant because a question such as "how many wins do the cubs have?" can be lookup, count or sum depending on the structure of the table. taking a threshold on the cell level confidences of the rci model and aggregating by the predicted question type produces the final answer, either a list of cells for lookup questions or a single number for aggregation questions. this approach requires full supervision, we must know the cells to be aggregated to train the rci row and column classifiers as well as the type of aggregation to train the question classifier. this type of supervision is available in the wikisql dataset, but not in wikitablequestions. ## evaluation to evaluate these three approaches, we adapt three standard tableqa datasets: wikisql @xcite , wikitablequestions @xcite and tabmcq @xcite . wik-isql and wikitablequestions include both lookup questions as well as aggregation questions. as mentioned in section 1, our primary focus in this paper is on lookup questions that require selection and projection operations over tables (i.e., identifying the row and column of a table with very high precision for a given natural language question). we are releasing the processing and evaluation code for the datasets to support reproducibility foot_2 . in wikisql, the ground truth sql query is provided for each question, so questions involving an aggregation operation can be automatically excluded. the lookup questions are 72% of the wik-isql benchmark. wikisql has some questions (< 3%) with multiple answers. we treat these as a list of relevant items and use information retrieval metrics to measure the quality of a predicted ranked list of cells. tabmcq is a multiple-choice, lookup tableqa dataset over general science tables. we discard the multiple-choice setting and treat it as a standard open-ended qa task. however, some tabmcq tables are very large. of the 68 tables, 17 have more than 50 rows, with two tables containing over a thousand rows. we down-sample the rows that are not relevant for a given question, limiting the largest table size to 50 rows. unlike the other two datasets, these tables are not wikipedia tables and have an unusual format. a sample tabmcq table is provided in the appendix. wikitablequestions does not provide a definitive indication for what questions are lookup questions. to identify these questions we first filter questions with words indicating an aggregation, such as 'average', 'min', 'max', etc. these questions were further filtered manually to get the wikitablequestions-lookup set. in order to evaluate our proposed approaches on these datasets we built three different systems and also used three existing models: is-sp, provided by @xcite , tabert @xcite and tapas @xcite . is-sp is a semantic parsing based model trained on wikita-blesquestions @xcite dataset (see section 2 for the details of this work). for building their model we used the code provided in @xcite . for tabert we trained the model for wikisql using the lookup subset, and for wikitablequestions we used the full training set and applied to the lookup subset. for tapas we used the trained base (reset) models 4 for wik-isql and applied to the lookup subsets of the dev and test sets. the mrc and mrc xxl models are based on machine reading comprehension, using the base v2 and xxlarge v2 versions of albert. because this model returns a span rather than a cell prediction, we match each of the top-k span predictions to the closest cell, the cell with the lowest difference in its character offsets. in case multiple of the top-k predictions map to the same cell, these predictions are merged. we also evaluate the two approaches to rci: interaction (rci inter ) and representation (rci repr ). ## conclusion in this paper we propose two novel techniques, rci interaction and rci representation, to tackle the problem of locating answers over tables for given natural language questions. these transformer based models are fine-tuned on ground truth tables to predict the probability of containing the answer to a question in the rows and columns of tables independently. these probabilities are either used to answer questions directly or highlight the relevant regions of tables as a heatmap, helping users to easily locate the answers over tables. our experiments prove that the rci model outperforms the state-of-the-art transformer based approaches pre-trained on very large table corpora (tapas @xcite and tabert @xcite ), achieving ∼3.4% and ∼18.86% additional precision improvement on the standard wik-isql benchmark including both lookup and aggregation questions. the representation model, on the other hand, enables pre-processing the tables and producing the embeddings to store and further use during online query processing, providing significant efficiency advantages without compromising much on the accuracy of finding cell values in tables. as for the future work, we plan to explore the exploitation of domain-specific taxonomies and embeddings generated for domain-specific corpora to tackle the problem of answering natural language questions over tables in domains such as finance, aviation and health care.