Leveraging Data Recasting to Enhance Tabular Reasoning (2022)

Abstract: Creating challenging tabular inference data is essential for learning complex reasoning. Prior work has mostly relied on two data generation strategies. The first is human annotation, which yields linguistically diverse data but is difficult to scale. The second category for creation is synthetic generation, which is scalable and cost effective but lacks inventiveness. In this research, we present a framework for semi-automatically recasting existing tabular data to make use of the benefits of both approaches. We utilize our framework to build tabular NLI instances from five datasets that were initially intended for tasks like table2text creation, tabular Q/A, and semantic parsing. We demonstrate that recasted data could be used as evaluation benchmarks as well as augmentation data to enhance performance on tabular NLI tasks. Furthermore, we investigate the effectiveness of models trained on recasted data in the zero-shot scenario, and analyse trends in performance across different recasted datasets types.

## introduction given a premise, natural language inference (nli) is the task of classifying a hypothesis as entailed (true), refuted (false) or neutral (cannot be determined from given premise). several large scale datasets such as snli @xcite , multinli @xcite , and squad @xcite explore nli with unstructured text as the premise. while textual inference is commonly researched, structured data (e.g. tables, knowledge-graphs and databases) enables the addition of more complex types of reasoning, such as ranking, counting, and aggregation. creating challenging large scale supervision data is vital for research in tabular reasoning. in recent years, many initiatives to include tabular semi-structured data as the premise have been introduced e.g. tabular inference (tnli) datasets such as tabfact @xcite , infotabs @xcite and shared tasks like semeval 2021 task 9 @xcite and feverous @xcite . tabular data differs from unstructured text in the way that it can capture information and relationships in a succinct manner through underlying structure @xcite . despite fluent, diverse, and creative, these human-annotated datasets are limited in scale owing to the costly and time-consuming nature of annotations. furthermore, @xcite and @xcite show that many humanannotated datasets for nli contain annotation biases or artifacts. this allows nli models to learn spurious patterns @xcite , which enables models to predict the right label for the wrong reasons, sometimes even with noisy, incorrect or incomplete input @xcite . @xcite revealed that tabular inference datasets also suffer from comparable challenges. furthermore, @xcite ; @xcite show that annotators introduce their own bias during annotation. for example, @xcite demonstrates that annotators only generate hypothesis sentences from keys having numerical values, implying that some keys are either over or underutilized. on the other hand, automatic grammar-based strategies , despite their scalability, lack linguistic diversity (both structural and lexical) and necessary inference complexity. these approaches create examples with only naive reasoning. recently, @xcite , create context-free-grammar and templates to generate augmentation data for tabular nli. additionally, the use of a large language generation model (e.g., @xcite @xcite for data generation has been proposed as well (e.g., @xcite @xcite . however, such generation systems lack factuality, leading in hallucinations, inadequate fact coverage, and token repetition problem. can we generate challenging supervision data that is as scalable as synthetic data and yet contains human-like fluency and linguistic diversity? in this work, we attempt to answer the above question through the lens of data recasting. data recasting refers to transforming data intended for one task into data intended for another distinct task. although data recasting has been around for a long time, for example, qa2d @xcite and scitail @xcite effectively recast question answering data for inference (nli), no earlier study has applied it to semi-structured data. therefore, we propose a semi-automatic framework for tabular data recasting. using our framework, we generate large-scale tabular nli data by recasting existing datasets intended for non nli tasks such as table2text generation (t2tg), tabular question answering (tqa), and semantic parsing on tables (spt) . this recasting strategy is a middle road technique that allows us to benefit from both synthetic and human-annotated data generation approaches. it allows us to minimise annotation time and expense while maintaining linguistic variance and creativity via human involvement from the original source dataset. our recasted data can be used for both evaluation and augmentation purposes for tabular inference tasks. models pre-trained on our data show an improvement of 17% from the tabfact baseline @xcite and 1.1% from @xcite , a synthetic data augmentation baseline. additionally, we report a zero-shot accuracy of 71.1% on the tabfact validation set, which is 5% percent higher than the supervised baseline accuracy reported by @xcite . our main contributions are the following: we demonstrate the efficacy of our generated data for data augmentation on tabfact. the dataset and associated scripts are available at https://recasting-to-nli.github. io 2 why table recasting? tables are structured data forms. table cells define a clear boundary for a standalone independent piece of information. these defined table entries facilitate the task of drawing alignments between relevant table cells and given hypotheses. if both the premise and hypothesis were plain text, any ngram in the premise could be aligned to any n-gram in the hypothesis. finding alignments between the premise and hypothesis is crucial in the recasting process. this is because, in order to modify statements, we must understand which entities influence their truth value. moreover, in tables, entries of the same type (same part of speech type, named entity type, domain etc) are clubbed under a common column header. this allows us to easily identify a group of candidates which are interchangeable in a sentence without disrupting its coherence. frequently, the column header also indicates the data type (e.g. name, organization, year etc). this is incredibly beneficial when modifying source data by substituting entities. ## tabular recasting framework in this section, we describe a general semiautomatic framework to recast tabular data for the task of table nli . by recasting, we imply converting data meant for one task into a format that satisfies the requirements of a different task. original table (og) party votes(thou) seats party a 650 120 party b 570 89 party c final count tba 89 total 1235 298 counterfactual table (cf -after cells swaps) party votes(thou) seats party a party b 650 120 party b party a 570 89 party c final count tba 89 total 1235 298 base entailmentog party a won 120 out of 298 seats. party a won the most seats. new entailmentog party b won 89 out of 298 seats. party b won the second most seats. paraphraseog out of a total of 298 available seats, party b secured the second largest party b won 89. number of seats. contradictionog party a party b won 120 out of 298 party a party b won the most seats. seats. party a won the most least seats. we swap party a and party b to create a counterfactual table. the contradictions mentioned above become the new base annotations (annotationct) base entailmentcf party b won 120 out of 298 seats. party b won the most seats. new entailmentcf party a won 89 out of 298 seats. party a won the second most seats. paraphrasecf 89 of the 298 available seats were party a won next to the maximum secured by party a number of seats. contradictioncf party b party a won 120 out of 298 party b party a won the most seats. seats. party b won the most least seats. ## addressing tabular recasting constraints dataset specific implementations pose several challenges. we address them in the following ways: table orientation. as stated in section 3, we conduct all experiments assuming the tables are vertically aligned. we observe several horizontally aligned tables (with the first column containing headers) in source datasets. as a preliminary processing step, we employ heuristics to automatically recognise such tables and flip them. for example, we check for frequently occurring header names in the first column or consistency in data types (numeric, alpha, etc.) across rows rather than columns. partial matching. we observe that some datasets provide relevant cells, but do not provide their alignments with the base entailment. we attempt to match every relevant cell with n-grams in the base entailment. of particular interest is the sample row shown in us presidential inaugurations (a table row) president # 44 name barack obama inauguration date january 20, 2009 location west front, us capitol base entailment : obama's inauguration as the forty fourth president took place at the united states capitol in 2009. irreplaceable entities. we observe that not all relevant entities are replaceable by potential candidates. similar observation is made while swapping table cells to create counterfactual tables. suppose we swap the aggregate cell 298 with 120 . the resultant table would be logically flawed since the "seats" column won't add up to its total. to prevent this, aggregate rows and header cells are marked as non-replaceable entities. ## dataset recasting using the framework outlined in section 3, we recast the five datasets listed in ## experiments and analysis in this section, we examine the relevance of our recast data across various settings. overall, we aim to answer the following research questions: ## related work inference on semi-structured data. in recent times, inference tasks such as nli, question answering and text generation have been applied to structured data sources like tables. tabfact @xcite and @xcite explore inference as an entailment task. wikitable-questions @xcite , wikiqaa @xcite , finqa @xcite and hybridqa @xcite perform question answering on tables. totto @xcite , logicnlg @xcite and logic2text @xcite explore logical text generation on tables. most of these datasets derive tables from wikipedia. early work on structured data modeling classify tables into structural categories and embed tabular data into a vector space @xcite @xcite . recent work like tapas @xcite , tapas-row-col-rank @xcite , tabert @xcite , tabbie @xcite , tables with sat @xcite , tabgcn @xcite and rci @xcite use more sophisticated methods of encoding tabular data. tapas @xcite encodes row/column index and order via specialized embeddings and pre-trains a mask-lm model on co-occurring wikipedia text and tables. @xcite decomposes nli statements into subproblems to enhance inference on tabfact. ## conclusion in this paper we introduced a semi-automatic framework for recasting tabular data. we made our case for choosing the recasting route due to its cost effectiveness, scalability and ability to retain human-alike diversity in the resultant data. finally, we leveraged our framework to generate nli data for five existing tabular datasets. in addition, we demonstrated that our recasted datasets could be utilized as evaluation benchmarks as well as for data augmentation to enhance performance on the tabular nli task on tabfact @xcite .