T ab P ert : An Effective Platform for Tabular Perturbation (2021)

Abstract: To grasp the true reasoning ability, the Natural Language Inference model should be evaluated on counterfactual data. TabPert facilitates this by generation of such counterfactual data for assessing model tabular reasoning issues. TabPert allows the user to update a table, change the hypothesis, change the labels, and highlight rows that are important for hypothesis classification. TabPert also details the technique used to automatically produce the table, as well as the strategies employed to generate the challenging hypothesis. These counterfactual tables and hypotheses, as well as the metadata, is then used to explore the existing model’s shortcomings methodically and quantitatively.

## introduction given factual evidence, a crucial part of nlp model reasoning capacity is evaluating whether a given hypothesis is an entailment (true), a contradiction (false), or is neutral (cannot be determined). current transformers-based models have been shown to outperform humans on these tasks when the evidence is presented as simple unstructured text @xcite ; however, when tested with semi-structured evidence @xcite , such as tables, as shown in figure 1 , the very same models struggle to match human accuracy @xcite @xcite . furthermore, there can be several reasons for a model's correct predictions on a particular example. for example, @xcite ; @xcite show that multiple nli datasets such as the snli and mnli datasets @xcite exhibit hypothesis bias, i.e., the hypothesis-only model performs significantly better than the majority label baseline. in the context of tables, @xcite ; neeraja et al. new york stock exchange type stock exchange location new york city, new york, u.s. founded may 17, 1792; 226 years ago currency united states dollar no. of listings 2,400 volume us@xmath010 trillion. figure 1: a tabular premise example. the table's first and second columns correspond to the keys and their associated values, respectively. the hypothesis h1 is entailed by the data in the table, h2 is a contradiction, and h3 is neutral, i.e., neither entailed nor contradictory. recently, @xcite have shown that one can deploy simple tricks to semi-automate this process of altering existing data. this semiautomated approach can then generate difficult adversarial counterfactual contrast sets, which can subsequently be utilised to perform behavioural testing of existing nlp models. however, such methods are currently only applicable to unstructured data and cannot be directly used for semistructured text such as tables. to fill this gap, in this work, we present tabpert. tabpert is an annotation platform specifically designed to work on semi-structured tabular data. for example, tabpert can support the semi-automatic creation of tabular counterfactual data. through tabpert, annotators can modify tables in several ways, such as (a) deleting information: deleting an attribute-value pair or an existing row completely, (b) inserting information: inserting an attribute-value pair for an existing row or creating a fresh row, (c) modifying information: editing the attribute or values cells of an existing row, and (d) modifying hypotheses or labels: modifying an existing hypothesis and its inference label. furthermore, each component of tabpert can be customized to meet the individual needs of a project that necessitates tabular perturbations. tabpert additionally logs the strategy used to modify each attribute-value of the table. in addition to the gold label, users can manually log information about the technique used for perturbing a table-hypothesis pair and the rows relevant to the hypothesis. this information is crucial in understanding the challenges annotated data poses to the existing model, and therefore, can be utilized to probe a model's yet-unknown shortcomings systematically. the contributions of our work can be summarised as below: this method, however, does not automate the updation of associated hypotheses and inference labels. furthermore, such automated shuffling quite often flagrantly violates common-sense logical constraints. for example, a person's 'birth date' must be before their 'died date', a person's 'marriage date' should be after their 'birth date' and before their 'died date', an album's 'released date' should be after its 'recording date' and so on. the updated table may be self-contradictory if these constraints are not enforced. while some of these constraints can be automatically met and therefore not violated, the vast majority of them inevitably sneak through due to their enormous diversity and variance foot_0 . furthermore, due to the domain-specific nature of these constraints, enforcing them automatically during perturbation is a challenging task. keeping this in mind, automated perturbations like these are only appropriate for table initialization. human annotators can then manually analyze and modify the initialized tables for self-consistency, i.e., no logical common sense constraint violation. ## tabpert functions, aspects, and usability tabpert is currently supported on common web browsers such as google chrome and can be installed to run locally foot_1 . we start with a dataset of tables along with already annotated labelled hypotheses. we utilize the infotabs dataset for the case study provided in this research. infotabs is a semi-structured natural language inference dataset that consists of entity tables and human-written hypotheses. we create three counterfactual tables (labelled a, b, and c) for each original table in the dataset. there are three main steps required for successful annotation, as described below. ## tabpert utility main platform tabpert is a tool designed for efficient and accurate table perturbation. one such case is creating tabular data for tabular inference tasks, as demonstrated through this paper. tabpert supports several features which aid in the creation of effective counterfactual tabular data. it has numerous optimizations with a friendly user interface to ensure fast annotation of data. this ensures huge data collection, leading to scalability. tabpert enables a larger range of services compared to using spreadsheets or the mturk platform. for example, the drag-and-drop functionality simplifies annotation and helps easily visualise a complicated job. all the tabular data can be examined in a single view. the automatic type validation during initialisation and modification reduces the chances of unintended errors. customizing tabpert functionality the initialization source code, as well as the platform, are designed to be modular. this facilitates component addition, deletion, and updation. for example, the ability to reorganize table parts, copy values across table triplets (in addition to cut-paste), auto-save work 12 , and an undo option, as well as checkpoints added to reverse mistakes, may all be readily implemented. the augmentation initialization code can also be configured to suit the requirements of the task. metadata counterfactual data can be utilized as a difficult adversarial test set to assess tabular model reasoning. for example, as demonstrated in section 4, the model performs poorly on counterfactual hypotheses with flipped labels on tables with relevant rows drawn from the external set (in our case, the t rain set), indicating model overfitting on pre-trained knowledge. the recorded metadata can also be utilized to guide annotators in creating increasingly difficult data. for example, annotators can be encouraged to focus more on label flip methods with external set-initialised tables rows to generate more challenging counterfactual data. label flipping techniques can also be used to test a model for hypothesis bias. the metadata associated with hypotheses-specific relevant rows assists in pruning premise tables, which improves inference model reasoning and interpretability, as shown in section 4. these are only a handful of the countless possible application scenarios. in section b of the appendix, we also compare and contrast tabpert with spreadsheets on effectiveness, visual benefits, and metadata aspects. ## tabpert limitations and future during our pilot study, the platform was run locally by three annotators. this was not an issue because the number of annotators was limited, and the tables were divided among them manually. if we want numerous annotators to be able to make simultaneous modifications for large-scale distribution, we must host our platform on a centralized server. this is something we intend to accomplish in the not-too-distant future. finally, the counterfactual data generated by modifications has to be manually stored by pressing a button. this was done so that if the user made a mistake, the original data would not be erased, and the user may save the data after they are satisfied with the modifications. to accommodate both of these circumstances, we would like to include an auto-save function along with an undo option. ## conclusions tabpert is an effective platform for examining by pressing the button several times at regular intervals. semi-structured tabular data and generating counterfactual tabular perturbations. annotators can use the platform to alter tables and hypothesis phrases, as well as collect related metadata information, in order to produce tabular counterfactual data. the metadata collected can be utilised to analyze the unknown vulnerabilities with existing nlp systems quantitatively and methodically. we believe that tabpert will be helpful to academics that work with semi-structured data such as tables. many non-academic industrial scenarios that require table modification, such as e-commerce product specification tables, financial and tax statements tables, and so on, may also leverage tabpert. table row perturbation metadata accuracy drop (relative %) 0 10 20 30 40 50 230 235 73 176 163 74 13 hypothesis perturbation strategy number of examples 0 50 100 150 200 250 t c + l f h c + l f t c + h c + l f h y p o p r o m p t o v e r la p n e w h y p o o th e r si figure 6: number of examples of each hypothesis perturbation strategy. in addition to this, the lack of scrolling required while dragging and dropping on our platform saves annotators time. to discover the relevant cells in a spreadsheet, one must execute numerous scrolling operations to the up, down, left, or right. finally, in tabpert, the cell size is set to exactly fit its contents, but in a spreadsheet, cells in each row and columns have the same height and width, making it quite problematic to view text properly. metadata collection tabpert makes it simple to gather information such as methods used to change a hypothesis and rows utilized to answer each hypothesis, using checkboxes. in a spreadsheet, this would require 9 columns of checkboxes for each table or manually writing the metadata, which is now automatically done with a single click, thus making the process simple and efficient. moreover, automatic metadata collection about a drag and drop location is not possible in a spreadsheet.