Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning (2022)

Abstract: When pre-trained contextualized embedding-based models developed for unstructured data are adapted for structured tabular data, they perform admirably. However, recent probing studies show that these models use spurious correlations, and often predict inference labels by focusing on false evidence or ignoring it altogether. To study this issue, we introduce the task of Trustworthy Tabular Reasoning, where a model needs to extract evidence to be used for reasoning, in addition to predicting the label. As a case study, we propose a two-stage sequential prediction approach, which includes an evidence extraction and an inference stage. First, we crowdsource evidence row labels and develop several unsupervised and supervised evidence extraction strategies for InfoTabS, a tabular NLI benchmark. Our evidence extraction strategy outperforms earlier baselines. On the downstream tabular inference task, using only the automatically extracted evidence as the premise, our approach outperforms prior benchmarks.

## introduction reasoning on tabular or semi-structured knowledge is a fundamental challenge for today's natural language processing (nlp) systems. two recently created tabular natural language inference (nli) datasets, tabfact @xcite on wikipedia relational tables and infotabs @xcite on wikipedia infoboxes help study the question of inferential reasoning over semi-structured tables. today's state-of-the-art for nli over unstructured text uses contextualized embeddings (e.g., @xcite . when adapted for tabular nli by flattening tables into synthetic sentences using heuristics, these models achieve remarkable performance on the datasets. however, a recent study @xcite demonstrates that these models fail to reason prop- * work done during an internship at bloomberg breakfast in america relevant released 4 29 march 1979 4 h3 recorded 3,4 may-december 1978 3,4 h2, h3 studio the village recorder in los angeles 3 genre pop, art rock, soft rock length 2 figure 1 : a semi-structured premise (the table 'breakfast in america') example from @xcite . hypotheses h1 are entailed by it, h2 is neither entailed nor contradictory, and h3 is a contradiction. the relevant column shows the hypotheses that use the corresponding row. the colored text (and superscripts) in the table and hypothesis highlights relevance token level alignment. to illustrate this task, consider an example from the infotabs dataset in figure 1 , which shows a premise table and three hypotheses. the figure also marks the rows needed to make decisions about each hypothesis, and also indicates the relevant tokens for each hypothesis. for trustworthy tabular reasoning, in addition to predicting the label entail for h1, contradict for h2 and neu-tral for h3, the model should also identify the evidence rows-namely, the rows producer and length for hypothesis h1, recorded for hypothesis h2, released and recorded for hypothesis h3. as a first step, we propose a two-stage sequential prediction approach for the task, comprising of an evidence extraction stage, followed by an inference stage. in the evidence extraction stage, the model extracts the necessary information needed for the second stage. in the inference stage, the nli model uses only the extracted evidence as the premise for the label prediction task. we explore several unsupervised evidence extraction approaches for infotabs. our best unsupervised evidence extraction method outperforms a previously developed baseline by 4.3%, 2.5% and 5.4% absolute score on the three test sets. for supervised evidence extraction, we annotate the in-fotabs training set (17k table-hypothesis pairs with 1740 unique tables) with relevant rows following the methodology of @xcite , and then train a roberta large classifier. the supervised model improves the evidence extraction performance by 8.7%, 10.8%, and 4.2% absolute scores on the three test sets over the unsupervised approach. finally, for the full inference task, we demonstrate that our two-stage approach with best extraction, outperforms the earlier baseline by 1.6%, 3.8%, and 4.2% on the three test sets. in summary, our contributions are as follows foot_1 : ## task formulation we begin by introducing the task and the datasets we use. tabular inference is a reasoning task that, like conventional nli @xcite @xcite , asks whether a natural language hypothesis can be inferred from a tabular premise. concretely, given a premise table t with m rows {r 1 , r 2 , . . . , r m }, and a hypothesis sentence h, the task maps them to entail (e), contradict (c) or neutral (n ). we can denote the mapping as trustworthy tabular inference is a table reasoning problem that seeks not just the nli label, but also relevant evidence from the input table that supports the label prediction. we use t r , a subset of t, to denote the relevant rows or evidence. then, the task is defined as follows. in our example table, this task will also indicate the evidence rows t r of producer and length for hypothesis h1, recorded for hypothesis h2, and released and recorded for hypothesis h3. while the notion of evidence is well-defined for the entail and contradict labels, the neu-tral label requires explanation. to decide on the neutral label, one must first search for relevant rows (if any), i.e., identify evidence in the premise tables. in fact, this is a causally correct sequential approach. indeed, infotabs has multiple neutral hypotheses that are partly entailed by the table ; if any part of a hypothesis contradicts the table, then the inference label should be contradict. for example, in our example table, the premise table indicates that the album was recorded in 1978, emphasizing the importance of the recorded row for the hypothesis h2. for neutral examples, we refer to any such pertinent rows as evidence. dataset details. there are several datasets for tabular nli: tabfact, infotabs, and the se-meval'21 task 9 @xcite and the feverous'21 shared task @xcite datasets. we use the infotabs data in this work. it contains finer-grained annotation (e.g., tabfact lacks neutral hypotheses) and more complex reasoning than the others foot_2 . the dataset consists of 23, 738 premisehypothesis pairs collected via crowdsourcing on amazon mturk. the tabular premises are based on 2, 540 wikipedia infoboxes representing twelve diverse domains, and the hypotheses are short statements paired with nli labels. all tables contain a title followed by two columns (cf. figure 1 ); the left columns are keys and the right ones are values). in addition to the train and development sets, the data includes multiple test sets, some of which are adversarial: α 1 represents a standard test set that is both topically and lexically similar to the training data; α 2 hypotheses are designed to be lexically adversarial foot_3 ; and α 3 tables are drawn from topics unavailable in the training set. the dev and test set, comprising of 7200 table-hypothesis pairs, were recently extended with crowdsourced evidence rows @xcite . as one of our contributions, we describe the evidence rows annotation for the training set in the next section 3. ## crowdsource evidence extraction this section describes the process of using amazon mturk to annotate evidence rows for the 16, 538 premise-hypothesis pairs that make the training set of infotabs. we followed the protocol of @xcite : one table and three distinct hypotheses formed a hit. for each of the hypotheses, five annotators would select the evidence rows. we divide the tasks equally into 110 batches, each batch having 51 hits each having three examples. to reduce bias induced by a link between the nli label and row selection, we do not reveal the labels to the annotators. the quality control details are provided in the appendix §b. ## trustworthy tabular inference trustworthy inference has an intrinsic sequential causal structure: extract evidence first, then predict the inference label using the extracted evidence data, knowledge/common sense, and perhaps formal reasoning @xcite foot_6 . to operationalize this intuition, we chose a two-stage sequential approach which consists of an evidence extraction followed by the nli classification, as shown in figure 2 . notation. the function f in eq. 2 can be rewritten with functions g and h, f (. here, g extracts the evidence rows t r subset of t, and h uses the extracted evidence t r and the hypothesis h to predict the inference label y, as to obtain f , we need to define the functions g and h, and a flexible representation of a semi-structured ## experimental evaluation our experiments assess the efficacy of evidence extraction ($4) and its impact on the downstream nli task by studying the following questions: ## evidence extraction: human versus model we perform an error analysis of how well our proposed supervised extraction model (hard negative(3x)) performs compared to the human annotators. the model makes two types of errors: a type i error occurs when an evidence row is marked as irrelevant, whereas type ii error occurs when an irrelevant row is marked as evidence. a type i error will reduce the model's precision for the extraction model, whereas a type ii error will decrease the model's recall. type i errors are especially concerning for the downstream nli task. since mislabeled evidence rows will be absent from the extracted premise, necessary evidence will be omitted, leading to inaccurate entailment labels. on the other hand, with type ii errors, when an irrelevant row is labeled as evidence, the model has to deal with from extra noise in the premise. however, all the required evidence remains. ## discussion why sequential prediction? our choice of the sequential paradigm is motivated by the observation that it enforces a causal structure. of course, a joint or a multi-task model may make better predictions. however, these models ignore the causal relationship between evidence selection and label prediction @xcite . ideally, each row is independent and, its relevance to the hypothesis can be determined on its own. in a joint or a multi-task model that exploits correlations across rows and the final label, irrelevant rows and the nli label, can erroneously influence row selection. ## conclusion and future work in this paper, we introduced the problem of trustworthy tabular inference, where a reasoning model both extracts evidence from a table and predicts an inference label. we studied a two-stage approach, comprising an evidence extraction and an inference stage. we explored several unsupervised and supervised strategies for evidence extraction, several of which outperformed prior benchmarks. finally, we showed that by using only extracted evidence as the premise, our approach outperforms previous baselines on the downstream tabular inference task. hongzhi zhang, yingyao wang, sirui wang, xuezhi cao, fuzheng zhang, and zhongyuan wang. 2020a. table fact verification with structure-aware transformer. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 1624-1629, online. association for computational linguistics. shuo zhang and krisztian balog. 2019. autocompletion for data cells in relational tables. in proceedings of the 28th acm international conference on information and knowledge management, cikm '19, pages 761-770, new york, ny, usa. acm. shuo zhang and krisztian balog. 2020. web table extraction, retrieval, and augmentation: a survey. acm trans. intell. syst. technol., 11(2):13:1-13:35. shuo zhang, zhuyun dai, krisztian balog, and jamie callan. 2020b. summarizing and exploring tabular data in conversational search. in proceedings of the 43rd international acm sigir conference on research and development in information retrieval, sigir '20, pages 1537-1540, new york, ny, usa. association for computing machinery. zhengli zhao, dheeru dua, and sameer singh. 2018. generating natural adversarial examples. in international conference on learning representations.