TAPAS at S em E val-2021 Task 9: Reasoning over tables with intermediate pre-training (2021)

Abstract: We present the TAPAS contribution to the Shared Task on Statement Verification and Evidence Finding with Tables (SemEval 2021 Task 9, Wang et al. (2021)). SEM TAB FACT Task A is a classification task of recognizing if a statement is entailed, neutral or refuted by the content of a given table. We adopt the binary TAPAS model of Eisenschlos et al. (2020) to this task. We learn two binary classification models: A first model to predict if a statement is neutral or non-neutral and a second one to predict if it is entailed or refuted. As the shared task training set contains only entailed or refuted examples, we generate artificial neutral examples to train the first model. Both models are pre-trained using a MASKLM objective, intermediate counter-factual and synthetic data (Eisenschlos et al., 2020) and TABFACT (Chen et al., 2020), a large table entailment dataset. We find that the artificial neutral examples are somewhat effective at training the first model, achieving 68.03 test F1 versus the 60.47 of a majority baseline. For the second stage, we find that the pre-training on the intermediate data and TABFACT improves the results over MASKLM pre-training (68.03 vs 57.01).

## introduction recently, the task of textual entailment (te) @xcite or natural language inference (nli) @xcite has been adapted to a setup where the premise is a table @xcite . the shared task on statement verification and evidence finding with tables (semeval 2021 task 9, @xcite follows this line of work and provides a new dataset consisting of tables extracted from scientific articles and natural language statements written by crowd workers. in this paper, we discuss a system for tackling task a, which is a multi-class classification task that requires finding if a statement is @xcite and then fine-tune on tabfact @xcite . a successful line of research on table entailment @xcite @xcite has been driven by bert-based models @xcite . these approaches reason over tables without generating logical forms to directly predict the entailment decision. such models are known to be efficient on representing textual data as well reasoning over semi-structured data such as tables. in particular, tapas-based models @xcite ) that encode the table structure using additional embeddings, have been successfully used to solve binary entailment tasks with tables @xcite . to address multi-class classification entailment we decompose the main task into two sub-tasks and use two tapas models as described in figure 2 . a first model classifies the statement into neutral or non-neutral, and a second into entailed or refuted. the two models are learned separately: we created artificial neutral statements to fine-tune the first model. examples are extracted by randomly pairing statements and tables from the semtabfact training set. we also generate harder examples by creating new tables from the original tables by removing columns that contain evidence to refute or entail the statement. this procedure is discussed in section 3. we follow @xcite and pre-train the two tapas models with a masklm objective @xcite and then with counterfactual and synthetic data as shown in figure 1 . we additionally fine-tune both models on the tabfact dataset. details are given in section 4. we find that our artificial neutral statement creations out-performs a majority baseline and that pre-training help for both the first and the second stage. our best models achieve 68.03 average micro f1-score on the test set. the tabfact dataset @xcite for example, uses tables as the premise, or source of information to resolve whether a statement is entailed or refuted. the tapas architecture introduced by @xcite can be used to obtain transformer-based baselines, as shown in eisenschlos et al. (2020), by using special embeddings to encode the table structure. zhang et al. (2020); chen et al. (2020) also use bert like models but obtain less accurate results due possibly to not using table-specific pre-training. intermediate pre-training our system relies on intermediate pre-training, a technique that appears in different forms in the literature. language model fine-tuning (howard and ruder, 2018), or domain adaptive pre-training (gururangan et al., 2020) are useful applications for domain adaptation. in a similar manner than pruksachatkun et al. (2020), we use the counterfactual+synthetic tasks from eisenschlos et al. (2020) to improve the discrete and numeric reasoning capabilities of the model for table entailment. synthetic data the use of synthetic data to improve learning in nlp is ubiquitous @xcite @xcite . @xcite al. (2019 ) focus on textual entailment and probes models with synthetic examples. in semantic parsing wang et al. (2015); iyer et al. (2017); weir et al. (2020) use templates to augment the training data for text-to-sql tasks and geva et al. (2020) do so to improve numerical reasoning, as do eisenschlos et al. ( 2020 ) on tabular data. they also create minimal contrastive examples @xcite by automatically swapping entities in the statements by plausible alternatives that exists elsewhere in the table. ## system our system is a two stage process that first decides whether a statement is neutral, and then decides if non-neutral statements are entailed or refuted. both stages are implement using a binary tapas @xcite statistics. the training data for the first stage was created from the crowdsourced training data using artificial neutral statements created by deleting columns with evidence or swapping statements randomly. for the second stage we use the crowdsourced training data. ## experimental setup in this section we explain the semtabfact task and dataset and give additional details about the experimental setup we used. the semtabfact dataset consists of statements and tables from the scientific literature. it is much smaller than similar datasets such as tab-fact @xcite and infotabs @xcite . it is note-worthy that the training set only contains entailed and refuted statements while the dev and test set also contain neutral (unknown) statements. the statements were written by crowd workers, which presented with 7 different types of statements were instructed to write one statement of each type. the types of statements were using aggregation, superlatives, counting, comparatives, unique counting and the usage of the caption or common-sense knowledge. the main metric of the task is the micro f1-score computed over the statements belonging to a table. the 3-way score takes all statements into account while the 2-way score is restricted to refuted and entailed statements. ## results looking at the first stage of the system in with respect to the data generation we observe that any kind of neutral data generation outperforms the majority baseline. combining the column removal and random statements yields the best results. the drop in the 2-way metrics going from the majority stage 1 model to a learned model is expected as that metric ignores all neutral statements in the eval set. ## analysis in by specific keywords appearing in the statement, for example comparatives must contain "higher", "better", "than", etc. the full list is defined in the appendix of @xcite . we observe that comparatives and aggregations have the largest total error rates, meaning that the biggest gains in overall accuracy can be made by improving those reasoning skills. between these two, comparatives have the lowest in-group accuracy. another class of examples with relatively low accuracy are statements around unique counting. we find that statements containing the word different have an accuracy of 51.3 (vs. 71% overall) and account for 3.4 percentage points of the total error rate. examples include "there are six different classes" and "they have ten different parameters". ## conclusion we presented our contribution to the semtabfact task @xcite on table entailment. our system consists of two stages that classify statements into non-neutral or neutral and refuted or entailed. our model achieves 68.03 average micro f1-score on the test set. we showed that our procedure for creating artificial neutral statements improves the system over a majority baseline but results in a relatively low recall of 37.6. other methods for creating harder neutral statements might further improve this value. in line with @xcite , we find that pre-training on intermediate data improves the system accuracy over a system purely pre-trained with a masklm objective. while these initial results look promising, we find that the model struggles with statements that involve complex operations such as comparisons and unique counting.