Understanding tables with intermediate pre-training (2020)

Abstract: Table entailment, the binary classification task of finding if a sentence is supported or refuted by the content of a table, requires parsing language and table structure as well as numerical and discrete reasoning. While there is extensive work on textual entailment, table entailment is less well studied. We adapt TAPAS (Herzig et al., 2020), a table-based BERT model, to recognize entailment. Motivated by the benefits of data augmentation, we create a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning. This new data is not only useful for table entailment, but also for SQA (Iyyer et al., 2017), a sequential table QA task. To be able to use long examples as input of BERT models, we evaluate table pruning techniques as a pre-processing step to drastically improve the training and prediction efficiency at a moderate drop in accuracy. The different methods set the new state-of-the-art on the TabFact (Chen et al., 2020) and SQA datasets.

## introduction textual entailment @xcite , also known as natural language inference @xcite , is a core natural language processing (nlp) task. it can predict effectiveness of reading comprehension @xcite , which argues that it can form the foundation of many other nlp tasks, and is a useful neural pre-training task @xcite . textual entailment is well studied, but many relevant data sources are structured or semi-structured: health data both worldwide and personal, fitness trackers, stock markets, and sport statistics. while some information needs can be anticipated by handcrafted templates, user queries are often surprising, and having models that can reason and parse that structure can have a great impact in real world applications @xcite . a recent example is tabfact @xcite , a dataset of statements that are either entailed or refuted by tables from wikipedia (figure 1 ). because solving these entailment problems requires sophisticated reasoning and higher-order operations like arg max, averaging, or comparing, human accuracy remains substantially (18 points) ahead of the best models @xcite . the current models are dominated by semantic parsing approaches that attempt to create logical forms from weak supervision. we, on the other hand, follow @xcite and @xcite and encode the tables with bert-based models to directly predict the entailment decision. but while bert models for text have been scrutinized and optimized for how to best pre-train and represent textual data, the same attention has not been applied to tabular data, limiting the effectiveness in this setting. this paper addresses these shortcomings using intermediate task pretraining @xcite , creating efficient data representations, and applying these improvements to the tabular entailment task. our methods are tested on the english language, mainly due to the availability of the end task resources. however, we believe that the proposed solutions could be applied in other languages where a pre-training corpus of text and tables is available, such as the wikipedia datasets. our main contributions are the following: i) we introduce two intermediate pre-training tasks, which are learned from a trained mask-lm model, one based on synthetic and the other from counterfactual statements. the first one generates a sentence by sampling from a set of logical expressions that filter, combine and compare the information on the table, which is required in table entailment (e.g., knowing that gerald ford is taller than the average president requires summing all presidents and dividing by the number of presidents). the second one corrupts sentences about tables appearing on wikipedia by swapping entities for plausible alternatives. examples of the two tasks can be seen in figure 1 . the procedure is described in detail in section 3. we release the pre-training checkpoints, data generation and training code at github.com/googleresearch/tapas. ## model we use a model architecture derived from bert and add additional embeddings to encode the table structure, following the approach of @xcite to encode the input. the statement and table in a pair are tokenized into word pieces and concatenated using the standard [cls] and [sep] tokens in between. the table is flattened row by row and no additional separator is added between the cells or rows. 1 based on six types of learnable input embeddings are added together as shown in appendix b. token embeddings, position embeddings and segment embeddings are analogous to the ones used in standard bert. additionally we follow @xcite and use column and row embeddings which encode the two dimensional position of the cell that the token corresponds to and rank embeddings for numeric columns that encode the numeric rank of the cell with respect to the column, and provide a simple way for the model to know how a row is ranked according to a specific column. recall that the bi-directional self-attention mechanism in transformers is unaware of order, which motivates the usage of positional and segment embeddings for text in bert, and generalizes naturally to column and row embeddings when processing tables, in the 2-dimensional case. let s and t represent the sentence and table respectively and e s and e t be their corresponding input embeddings. the sequence @xcite denoted f and a contextual representation is obtained for every token. we model the probability of entailment p (s|t ) with a single hidden layer neural network computed from the output of the [cls] token: ## methods the use of challenging pre-training tasks has been successful in improving downstream accuracy @xcite . one clear caveat of the method adopted in @xcite which attempts to fill in the blanks of sentences and cells in the table is that not much understanding of the table in relation with the sentence is needed. with that in mind, we propose two tasks that require sentence-table reasoning and feature complex operations performed on the table and entities grounded in sentences in non-trivial forms. we discuss two methods to create pre-training data that lead to stronger table entailment models. both methods create statements for existing wikipedia tables foot_0 . we extract all tables that have at least two columns, a header row and two data rows. we recursively split tables row-wise into the upper and lower half until they have at most 50 cells. this way we obtain 3.7 million tables. ## experimental setup in all experiments, we start with the public tapas checkpoint, 6 train an entailment model on the data from section 3 and then fine-tune on the end task (tabfact or sqa). we report the median accuracy values over 3 pre-training and 3 fine-tuning runs (9 runs in total). we estimate the error margin as half the interquartile range, that is half the difference between the 25th and 75th percentiles. the hyper-parameters, how we chose them, hardware and other information to reproduce our experiments are explained in detail in appendix a. the training time depends on the sequence length used. for a bert-base model it takes around 78 minutes using 128 tokens and it scales almost linearly up to 512. for our pre-training tasks, we explore multiple lengths and how they trade-off speed for downstream results. ## analysis salient groups to obtain detailed information of the improvements of our approach, we manually annotated 200 random examples with the complex operations needed to answer them. we found 4 salient groups: aggregations, superlatives, com-paratives and negations, and sort pairs into these groups via keywords in the text. to make the groups exclusive, we add a fifth case when more than one operation is needed. the accuracy of the heuristics was validated through further manual inspection of 50 samples per group. the trigger words of each group are described in appendix g. for each group within the validation set, we look at the difference in accuracy between different models. we also look at how the total error rate is divided among the groups as a way to guide the focus on pre-training tasks and modeling. the error rate defined in this way measures potential accuracy gains if all the errors in a group s were fixed: among the groups, the intermediate task data improve superlatives (39% error reduction) and negations (31%) most ( in figure 4 we show examples in every group where our model is correct on the majority of the cases (across 9 trials), and the mask-lm baseline is not. we also show examples that continue to produce errors after our pre-training. many examples in this last group require multi-hop reasoning or complex numerical operations. model agreement similar to other complex binary classification datasets such as boolq @xcite , for tabfact one may question whether models are guessing the right answer. to detect the magnitude of this issue we look at 9 independent runs of each variant and analyze how many of them agree on the correct answer. figure 5 shows that while for mask-lm only for 24.2% of the examples all models agree on the right answer, it goes up to 55.5% when using using the counterfactual and synthetic pre-training. this suggests that the amount of guessing decreases substantially. ## conclusion we introduced two pre-training tasks, counterfactual and synthetic, to obtain state-of-the-art results on the tabfact @xcite ) entailment task on tabular data. we adapted the bert-based architecture of tapas @xcite to binary classification and showed that pre-training on both tasks yields substantial improvements on tab-fact but also on a qa dataset, sqa @xcite , even with only a subset of the training data. we ran a study on column selection methods to speed-up training and inference. we found that we can speed up the model by a factor of 2 at a moderate drop in accuracy (≈ 1 point) and by a factor of 4 at a larger drop but still with higher accuracy than previous approaches. we characterized the complex operations required for table entailment to guide future research in this topic. our code and models will be opensourced.