T a P as: Weakly Supervised Table Parsing via Pre-training (2020)

Abstract: Answering natural language questions over tables is usually seen as a semantic parsing task. To alleviate the collection cost of full logical forms, one popular approach focuses on weak supervision consisting of denotations instead of logical forms. However, training semantic parsers from weak supervision poses difficulties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we present TaPas, an approach to question answering over tables without generating logical forms. TaPas trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TaPas extends BERT’s architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TaPas outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WikiSQL and WikiTQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WikiSQL to WikiTQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.

## introduction question answering from semi-structured tables is usually seen as a semantic parsing task where the question is translated to a logical form that can be executed against the table to retrieve the correct denotation @xcite @xcite . semantic parsers rely on supervised training data that pairs natural language questions with logical forms, but such data is expensive to annotate. in recent years, many attempts aim to reduce the burden of data collection for semantic parsing, including paraphrasing @xcite , human in the loop @xcite and training on examples from other domains @xcite . one prominent data collection approach focuses on weak supervision where a training example consists of a question and its denotation instead of the full logical form @xcite @xcite . although appealing, training semantic parsers from this input is often difficult due to the abundance of spurious logical forms @xcite and reward sparsity @xcite . in addition, semantic parsing applications only utilize the generated logical form as an intermediate step in retrieving the answer. generating logical forms, however, introduces difficulties such as maintaining a logical formalism with sufficient expressivity, obeying decoding constraints (e.g. wellformedness), and the label bias problem @xcite . in this paper we present tapas (for table parser ), a weakly supervised question answering model that reasons over tables without generating logical forms. tapas predicts a minimal program by selecting a subset of the table cells and a possible aggregation operation to be executed on top of them. consequently, tapas can learn operations from natural language, without the need to specify them in some formalism. this is implemented by extending bert's architecture @xcite with additional embeddings that capture tabular structure, and with two classification layers for selecting cells and predicting a corresponding aggregation operator. importantly, we introduce a pre-training method for tapas, crucial for its success on the end task. we extend bert's masked language model objective to structured data, and pre-train the model over millions of tables and related text segments crawled from wikipedia. during pre-training, the model masks some tokens from the text segment and from the table itself, where the objective is to predict the original masked token based on the textual and tabular context. finally, we present an end-to-end differentiable training recipe that allows tapas to train from weak supervision. for examples that only involve selecting a subset of the table cells, we directly train the model to select the gold subset. for examples that involve aggregation, the relevant cells and the aggregation operation are not known from the denotation. in this case, we calculate an expected soft scalar outcome over all aggregation operators given the current model, and train the model with a regression loss against the gold denotation. in comparison to prior attempts to reason over tables without generating logical forms @xcite @xcite , tapas achieves better accuracy, and holds several advantages: its architecture is simpler as it includes a single encoder with no auto-regressive decoding, it enjoys pre-training, tackles more question types such as those that involve aggregation, and directly handles a conversational setting. we find that on three different semantic parsing datasets, tapas performs better or on par in comparison to other semantic parsing and question answering models. on the conversational sqa @xcite , tapas improves stateof-the-art accuracy from 55.1 to 67.2, and achieves on par performance on wikisql @xcite and wikitq @xcite . transfer learning, which is simple in tapas, from wikisql to wikitq achieves 48.7 accuracy, 4.2 points higher than state-of-the-art. our code and pre-trained model are publicly available at https: //github.com/google-research/tapas. ## tapas model our model's architecture (figure 1 ) is based on bert's encoder with additional positional embeddings used to encode tabular structure (visualized in figure 2 ). we flatten the table into a sequence of words, split words into word pieces (tokens) and concatenate the question tokens before the table tokens. we additionally add two classification layers for selecting table cells and aggregation operators that operate on the cells. we now describe these modifications and how inference is performed. question [cls] tok 1 tok n [sep] tok 1 tok m ... ... e [cls] e 1 e n e [sep] e' 1 e' m ... ... [cls] t 1 t n [sep] t' 1 t' m ... ... 0.9 0.9 0 0.2 0 p s flattened table ... aggregation prediction cell selection rank ... days 1 ... 37 2 ... 31 3 ... 17 4 ... 15 ... ... ... op p a (op) compute(op,p s ,t) none 0 -count 0.1 .9 + .9 + .2 = 2 sum 0.8 .9×37 + .9×31 + .2×15 = 64.2 avg 0.1 64.2 ÷ 2 = 32.1 s @xmath0.1×2 + .8×64.2 + .1×32.1 = 54.8 figure 1: tapas model (bottom) with example model outputs for the question: "total number of days for the top two". cell prediction (top right) is given for the selected column's table cells in bold (zero for others) along with aggregation prediction (top left). ## pre-training following the recent success of pre-training models on textual data for natural language understanding tasks, we wish to extend this procedure to structured data, as an initialization for our table parsing task. to this end, we pre-train tapas on a large number of tables from wikipedia. this allows the model to learn many interesting correlations between text and the table, and between the cells of a columns and their header. we create pre-training inputs by extracting texttable pairs from wikipedia. we extract 6.2m tables: 3.3m of class infobox foot_0 and 2.9m of class wikitable. we consider tables with at most 500 cells. all of the end task datasets we experiment with only contain horizontal tables with a header row with column names. therefore, we only extract wiki tables of this form using @xmath1. we furthermore, transpose infoboxes into a table with a single header and a single data row. the tables, created from infoboxes, are arguably not very typical, but we found them to improve performance on the end tasks. as a proxy for questions that appear in the end tasks, we extract the table caption, article title, article description, segment title and text of the segment the table occurs in as relevant text snippets. in this way we extract 21.3m snippets. we convert the extracted text-table pairs to pretraining examples as follows: following @xcite , we use a masked language model pre-training objective. we also experimented with adding a second objective of predicting whether the table belongs to the text or is a random table but did not find this to improve the performance on the end tasks. this is aligned with @xcite that similarly did not benefit from a next sentence prediction task. for pre-training to be efficient, we restrict our word piece sequence length to a certain budget (e.g., we use 128 in our final experiments). that is, the combined length of tokenized text and table cells has to fit into this budget. to achieve this, we randomly select a snippet of 8 to 16 word pieces from the associated text. to fit the table, we start by only adding the first word of each column name and cell. we then keep adding words turn-wise until we reach the word piece budget. for every table we generate 10 different snippets in this way. we follow the masking procedure introduced by bert. we use whole word masking foot_1 for the text, and we find it beneficial to apply whole cell masking (masking all the word pieces of the cell if any of its pieces is masked) to the table as well. we note that we additionally experimented with data augmentation, which shares a similar goal to pre-training. we generated synthetic pairs of questions and denotations over real tables via a grammar, and augmented these to the end tasks training data. as this did not improve end task performance significantly, we omit these results. ## fine-tuning overview we formally define table parsing in a weakly supervised setup as follows. given a training set of n examples {(x i , t i , y i )} n @xmath2, where x i is an utterance, t i is a table and y i is a corresponding set of denotations, our goal is to learn a model that maps a new utterance x to a program z, such that when z is executed against the corresponding table t , it yields the correct denotation y. the program z comprises a subset of the table cells and an optional aggregation operator. the table t maps a table cell to its value. as a pre-processing step described in section 5.1, we translate the set of denotations y for each example to a tuple (c, s) of cell coordinates c and a scalar s, which is only populated when y is a single scalar. we then guide training according to the content of (c, s). for cell selection examples, for which s is not populated, we train the model to select the cells in c. for scalar answer examples, where s is populated but c is empty, we train the model to predict an aggregation over the table cells that amounts to s. we now describe each of these cases in detail. ## related work semantic parsing models are mostly trained to produce gold logical forms using an encoder-decoder approach @xcite . to reduce the burden in collecting full logical forms, models are typically trained from weak supervision in the form of denotations. these are used to guide the search for correct logical forms @xcite . other works suggested end-to-end differentiable models that train from weak supervision, but do not explicitly generate logical forms. neelakantan et al. (2015) proposed a complex model that sequentially predicts symbolic operations over table segments that are all explicitly predefined by the authors, while yin et al. (2016) proposed a similar model where the operations themselves are learned during training. müller et al. (2019) proposed a model that selects table cells, where the table and question are represented as a graph neural network, however their model can not predict aggregations over table cells. cho et al. ( 2018 ) proposed a supervised model that predicts the relevant rows, column and aggregation operation sequentially. in our work, we propose a model that follow this line of work, with a simpler architecture than past models (as the model is a single encoder that performs computation for many operations implicitly) and more coverage (as we support aggregation operators over selected cells). finally, pre-training methods have been designed with different training objectives, including language modeling @xcite @xcite and masked language modeling @xcite . these methods dramatically boost the performance of natural language understanding models @xcite . recently, several works extended bert for visual question answering, by pre-training over text-image pairs while masking different regions in the image @xcite . as for tables, @xcite experimented with rendering a table into natural language so that it can be handled with a pre-trained bert model. in our work we extend masked language modeling for table representations, by masking table cells or text segments. ## conclusion in this paper we presented tapas, a model for question answering over tables that avoids generating logical forms. we showed that tapas effectively pre-trains over large scale data of text-table pairs and successfully restores masked words and table cells. we additionally showed that the model can fine-tune on semantic parsing datasets, only using weak supervision, with an end-to-end differentiable recipe. results show that tapas achieves better or competitive results in comparison to stateof-the-art semantic parsers. in future work we aim to extend the model to represent a database with multiple tables as context, and to effectively handle large tables. in the example, the model correctly restores the draw (d) and loss (l) numbers for spain. it fails to restore the points for (pf) and points against (pa) for zimbabwe, but gives close estimates. note that the model also does not produce completely consistent results for each row we should have pa + @xmath3.