Enhancing Tabular Reasoning with Pattern Exploiting Training (2022)

Abstract: Recent methods based on pre-trained language models have exhibited superior performance over tabular tasks (e.g., tabular NLI), despite showing inherent problems such as not using the right evidence and inconsistent predictions across inputs while reasoning over the tabular data (Gupta et al., 2021). In this work, we utilize Pattern-Exploiting Training (PET) (i.e., strategic MLM) on pre-trained language models to strengthen these tabular reasoning models’ pre-existing knowledge and reasoning abilities. Our upgraded model exhibits a superior understanding of knowledge facts and tabular reasoning compared to current baselines. Additionally, we demonstrate that such models are more effective for underlying downstream tasks of tabular inference on INFOTABS. Furthermore, we show our model’s robustness against adversarial sets generated through various character and word level perturbations.

## introduction natural language inference (nli) is the problem of categorizing a hypothesis into entailment, contradiction, or neutral based on the given premise @xcite . large language models such as bert @xcite , roberta @xcite have been applied to large datasets like snli @xcite , multinli @xcite , where they have shown performance comparable to that of humans. however, the existing methods based on language models are ineffective for reasoning over semi-structured data @xcite . these models often ignore relevant rows and use spurious correlations in hypothesis or pre-training information for making inferences @xcite @xcite @xcite . due to existing biases in human curated datasets @xcite in this paper, we investigate the current model's reasoning capability, particularly whether they can extract the right knowledge and correctly make rational inferences from that extracted knowledge. we focus on the task of tabular reasoning through table inference on infotabs @xcite . for instance, in it has been proven that lms pre-trained without explicit supervision on a huge corpus of free web data implicitly incorporate several types of knowledge into their parameters @xcite . for extracting this knowledge from language models (lm), various methods utilize probing @xcite , attention @xcite , and prompting @xcite strategies. this internalized knowledge cannot be retrieved when fine-turning for a subsequent task. one explanation is that the objectives of pre-training and fine-tuning are vastly different. this variation in training objectives also diminishes the expected performance gains of the task, hence necessitating further pre-training on training data @xcite @xcite . therefore, reframing the subsequent task as a joint pre-training objective becomes essential. hence, we reformulate the tabular nli, i.e., our downstream task as a clozestyle problem, a.k.a, a mask language modeling (mlm) problem. for fine-tuning, we utilize the efficient pattern-exploiting training (pet) technique @xcite . pet entails establishing pairs of cloze question patterns and verbalizers that enable subsequent tasks to utilize the knowledge of the pre-trained language models. in addition, pet does not need model upgrades, such as adding more layers or parameters during pre-training. compared to direct fine-tuning-based techniques, i.e., training a classifier layer on top of lm, our method improved +8.1 and +25.8 on factual and relational knowledge evaluation tasks, respectively (see our main contributions are the following: the dataset and associated scripts, are available at https://infoadapet.github.io/ . ## motivation case for reasoning on semi-structured data. reasoning semi-structured data acquire skills such as arithmetic and commonsense, understanding the text types in the tabular cells, and aggregating information across numerous rows if necessary. for example, to judge the h1 in ## our approach in this section we describe our method to (a) evaluate pre-trained lm knowledge for tabular reasoning, (b) enhance model tabular reasoning capability using pet training, (c) and assess model robustness to input perturbations. ## experiments and analysis dataset. our experiments we use infotabs, a tabular inference dataset introduced by @xcite . the dataset is diverse in terms of the tables domains, categories, and corresponding keys (entity types and forms) it contains, as illustrated in examples in addition to the conventional development set and test set (referred to as α 1 ), an adversarial test set (α 2 ) lexically equivalent to α 1 but with minor changes in the hypotheses to flip the entailcontradict label and a zero-shot cross-domain test set (α 3 ) containing large tables from other domains that are not in the training set are used for evaluation. for all of our experiments, we use the accuracy of classifying the labels as our primary metric for evaluation. the domain of tables in training sets and α 1 ,α 2 are similar. however, the training and fine-tuning tables are exclusive. each of the test sets α 1 , α 2 , α 3 has 200 unique tables paired with 9 hypothesis sentences (3e, 3c, 3n), totalling 1800 table-hypothesis pairs. model. we use the pre-trained roberta-large (roberta l ) @xcite language model from huggingface @xcite ) for all of our investigations. we employ various configurations of language models to assess knowledge in two different cases. these configurations include roberta l , roberta l finetuned on infotabs (roberta l +cls), roberta l trained for tabular inference using pet (adapet), and finetuning infotabs on adapet (adapet+cls). here we define finetuning as training a classifier head (cls). we also investigate the effect of nli pre-training using roberta l pretrained on mnli @xcite , and mixed dataset (mixnli) containing anli+mnli+snli+fevernli 3 @xcite @xcite . all models are trained on 16538 table-hypothesis pairs (1740 tables) for 10 epochs with a 1e-5 learning rate. table representation. we explored two ways to represent table (a.) table as paragraph uses better paragraph representation for table representation, @xcite and distracting row removal prunes tables based on the similarity between hypothesis and tables rows. we investigated the pruning of top 4 (drr@4) and top 8 (drr@4) rows for our experiments. both representation methods are adapted from @xcite . for more details on table representation, refer to appendix a.1. ## conclusion in this work, we have validated the effects of factual and relational knowledge in the language model via handcrafted prompts for tabular reasoning. through prompt learning, i.e., pattern-exploiting training, we extracted knowledge from semi-structured tables and further improved the model's reasoning capabilities. our intensive experiments on the infotabs demonstrate that our approach can conserve knowledge and enhance tabular nli performance. the conclusions hold up well when tested against carefully crafted adversarial test sets based on character and word-level perturbations. method limitations: entity tables are the focus of our solution. its scalability in constructing prompts and other tables with different structures is limited by the idea that manually identified pattern from the specific dataset and template-based prompts. in addition, as not different from other nlp tasks, automatically detecting knowledge patterns and bridging patterns to prompts, especially for semi-structured tables, is under-explored. furthermore, investigating prompting for sophisticated structured tables such as nested structures (e.g., lists inside tables), hierarchical tables (e.g., table inside a table), and multi-modal tables (pictures within table) will necessitate substantial effort.