Realistic Data Augmentation Framework for Enhancing Tabular Reasoning (2022)

Abstract: Existing approaches to constructing training data for Natural Language Inference (NLI) tasks, such as for semi-structured table reasoning, are either via crowdsourcing or fully automatic methods. However, the former is expensive and time consuming and thus limits scale, and the latter often produces naive examples that may lack complex reasoning. This paper develops a realistic semi-automated framework for data augmentation for tabular inference. Instead of manually generating a hypothesis for each table, our methodology generates hypothesis templates transferable to similar tables. In addition, our framework entails the creation of rational counterfactual tables based on human written logical constraints and premise paraphrasing. For our case study, we use the INFOTABS (Gupta et al., 2020), which is an entity centric tabular inference dataset. We observed that our framework could generate human-like tabular inference examples, which could benefit training data augmentation, especially in the scenario with limited supervision.

## introduction natural language inference (nli) is a natural language processing task of determining if a hypothesis is entailed or contradicted given a premise or is unrelated to it @xcite . the nli task has been extended for tabular data where it takes tables as the premise instead of sentences, namely tabular inference task. two popular human-curated datasets for tabular reasoning, tabfact @xcite and infotabs @xcite datasets, have enhanced recent research in this area. however, human-generated datasets are limited in scale and thus insufficient for learning with large language models (e.g., @xcite . since curating these datasets requires expertise, huge annotation time, and expense, they cannot be scaled. furthermore, it has been shown that these datasets suffer from annotation bias and spurious correlation problem (e.g., @xcite @xcite . in contrast, automatically generated data lacks diversity and have naive reasoning aspects. recently, use of large language generation model (e.g., radford et al.; @xcite is also proposed for data generation (e.g., @xcite @xcite . despite substantial improvement, these generation approaches still lack factuality, i.e., suffer hallucination, have poor facts coverage, and also suffer from token repetition (refer to appendix §e analysis). recently, @xcite shows that automatic tabular nlg frameworks cannot produce logical statements and provide only surface reasoning. to address the above shortcomings, we propose a semi-automatic framework that exploits the patterns in tabular structure for hypothesis generation. specifically, this framework generates hypothesis templates transferable to similar tables since tables with similar categories, e.g., two athlete tables in wikipedia, will share many common attributes. in to show the effectiveness of our proposed framework, we conduct a case study with infotabs dataset. infotabs is an entity-centric dataset for tabular inference, as shown in example to summarize, we make the following contributions in this paper: the dataset and associated scripts, are available at https://autotnli.github.io . ## proposed framework our framework includes four main components: (a.) hypothesis template creation, (b.) rational counterfactual table creation, (c.) paraphrasing of premise tables, and (d.) automatic table-hypothesis generation. . ## the auto-tnli dataset we apply our framework as described in §2 on an entity specific tabular inference dataset infotabs to construct auto-tnli. infotabs @xcite consists of pairs of nli instances: a hypothesis statement grounded and inferred on premise as previously reported in the original in-fotabs paper by @xcite , annotators are biased towards specific keys over others. for example, for the category company, annotators would create more sentences for the @xmath1, resulting in an inherent hypothesis bias in the dataset. while creating the templates for auto-tnli, we ensure that each key has a minimum of two hypotheses and a minimum of three (> 3) premise paraphrases, which helps mitigate hypothesis bias. to address the inference class imbalance labeling issue, we construct approximately 1:1 entail to contra-dict the hypothesis. we observe that most additional human labor required to build such sentences is spent on the set of key-specific rules and constraints that ensure the sentences are grammatically accurate. the counterfactual tabular data is logically consistent, i.e., not self-contradictory. we have chosen infotabs as it has three evaluation sets α 1 , α 2 , and α 3 , in addition to the regular training and development sets. the α 1 set is lexically and topic-wise similar to the train set, and in α 2 the hypothesis is lexically adversarial to the train set. and in α 3 the tables are from topics not in the train set. moreover, it has multiple reasoning types such as multi-row reasoning, entity type, negation, knowledge & common sense, etc. in-fotabs has all three labels entail, neutral, and contradict compared to just two labels in other datasets such as tabfact. human verification to evaluate the quality and correctness of our data, we requested one of our human annotators (expert nlp ph.d. grad student) to assign a label to the generated hypothesis and select a score from 1 to 5 for the grammar and complexity of the sentences. the grammar score reflects how meaningful and lexically accurate the data is, and the complexity score indicates how difficult it is to label the hypothesis correctly. this was done for about 1300 premise-hypothesis pairs from auto-tnli. ## experiments and analysis overall, we address the following two research questions through our experiments: rq1: (a) taking auto-tnli as an evaluation set, how challenging is the tnli task? (b) if finetuning on auto-tnli beneficial? rq2: (a) is it beneficial to use auto-tnli as data augmentation for the tnli task? (b) if so, will it also be useful in little supervision scenario? experiment settings. we use roberta base @xcite ) (12-layer, 768-hidden, 12-heads, 125m parameters) and albert base @xcite (12-layer, 768-hidden, 12-heads, 12m parameters) as our model for all of our experiments foot_0 . @xcite shows data augmentation techniques that uses mnli data for pre-training acts as implicit knowledge and enhances the model performance for infotabs. therefore, we also explore implicit knowledge addition via data augmentation. in particular, we explored the following models: (a) roberta base fine-tuned using the auto-tnli dataset (b) roberta base , fine-tuned on the mnli dataset and the auto-tnli dataset (mnli + auto-tnli). additionally, we also explore performance with roberta base model finetuned sequential on all three mnli, auto-tnli and infotabs dataset. due to limited space, we report all albert foot_1 findings in appendix f. ## related work tabular reasoning. there has been considerable work on solving nlp tasks on semi-structured tabular data, such as tabular nli @xcite @xcite , question-answering task @xcite @xcite @xcite @xcite @xcite and table-to-text generation @xcite @xcite @xcite . similar to our data setting, some recent papers have also proposed ideas for representing wikipedia relational tables, some such papers are tapas @xcite , @xcite , table2vec @xcite , tabert @xcite , tabbie @xcite ,tabstruc @xcite , tabgcn @xcite , rci @xcite , turl @xcite and @xcite . some papers such as @xcite @xcite @xcite study the improvement of tabular inference by pre-training. tabular datasets. synthetic creation of dataset has long been explored @xcite @xcite . for tabular nli in particular, the datasets can be categorized into 1) manually created datasets @xcite with manually creates both hypothesis and premise, @xcite manually creates the hypothesis while premise is automatically generated 2) synthetically created semi-automatically generated datasets which completely automate data generation requires manual designing table-dependent contextfree grammar (cfg) @xcite , or require logical forms to be annotated @xcite . several works such as poliak et al. (2018 2020 ) authors investigate impacts of artifacts in dataset by creating adversarial test sets. however, semi-automatic systems requiring a cfg or logical forms contains reasoning which is often limited to certain types. creating sentences that contain other reasonings (like lexical reasoning, knowledge, and common sense reasoning) is challenging using cfg and logical forms. our paper requires subject matter experts to create entity specific templates for each category which leads to creating sentences with multiple reasonings as well as complex reasonings. ## discussion why counterfactual table generation? tabular dataset is inherently semi-structured. therefore, each category table has a specific set of keys. this enables us to create key-specific templates based on the entity-types of keys @xcite , which could be applied to millions of tables of a given category. furthermore, as explained in §3, the templates also generalize across keys with similar value types across categories. all this is only possible due to the semi-structured nature of tabular data. using counterfactual tables equips the model with more linguistically comparable. but oppositely labeled data to learn from, guaranteeing that the model can learn beyond the superficial textual artifacts and so becomes more resilient as shown by @xcite . as a result, when counterfactual data is included in the auto-tnli, we observe performance improvement throughout all experimental settings. this learning is further verified by the findings for better gains in α 2 , which comprises instances of linguistically comparable but oppositely labeled data instances. why semi-automatic approach? by examining the two diametrically opposed frameworks, namely a human and an automatic annotation framework, we may see many issues with both. manually created data is prohibitively expensive and demands much human effort, limiting the ability to develop large-scale databases. additionally, humans have a propensity to establish artificial patterns when manually creating a dataset, such as not giving all keys the same importance (explained in §3). while autonomous data generation is computationally efficient, it has many limitations. e.g., the inability to generate linguistically complex sentences and the difficulty of incorporating reasoning into the dataset. because most deep learning models perform better with more data, producing large-scale datasets at a reasonable cost is critical while retaining data quality. with this in mind, we presented a "semi-automatic" architecture with the following benefits: (a.) it simplifies the creation of large-scale datasets. using only 660 templates, we can generate 1,478,662 premise-hypothesis pairings from around 10,182 tables. (b.) the framework may be reused with additional tabular data as long as the structure is preserved. (c.) it enables the creation of linguistically and lexically diverse datasets. (d.) as shown in §3, hypothesis bias can be minimized by establishing an adequate number of diverse templates for all keys of each category. (e.) the premises have been paraphrased in three ways to bring the required lexical diversity. ## conclusion we introduced a semi-automatic framework for generating data from tabular data. using a templatebased approach, we generate auto-tnli. we utilized auto-tnli for both tnli evaluation and data augmentation. our experiments demonstrate the effectiveness of auto-tnli and, by implication, our framework, especially for adversarial settings. for the future work, we aim to involve the creation of additional lexically varied and robust datasets and investigate whether the addition of neutrals could improve these datasets.