V olta at S em E val-2021 Task 9: Statement Verification and Evidence Finding with Tables using TAPAS and Transfer Learning (2021)

Abstract: Tables are widely used in various kinds of documents to present information concisely. Understanding tables is a challenging problem that requires an understanding of language and table structure, along with numerical and logical reasoning. In this paper, we present our systems to solve Task 9 of SemEval-2021: Statement Verification and Evidence Finding with Tables (SEM-TAB-FACTS). The task consists of two subtasks: (A) Given a table and a statement, predicting whether the table supports the statement and (B) Predicting which cells in the table provide evidence for/against the statement. We fine-tune TAPAS (a model which extends BERT’s architecture to capture tabular structure) for both the subtasks as it has shown state-of-the-art performance in various table understanding tasks. In subtask A, we evaluate how transfer learning and standardizing tables to have a single header row improves TAPAS’ performance. In subtask B, we evaluate how different fine-tuning strategies can improve TAPAS’ performance. Our systems achieve an F1 score of 67.34 in subtask A three-way classification, 72.89 in subtask A two-way classification, and 62.95 in subtask B.

## introduction there has been extensive work on verifying if a given textual context supports a given statement. even though tables are also widely used to convey information, especially in scientific texts, there has been comparatively less work on verifying if a given table supports a statement. to this end, se-meval 2021 task 9 @xcite focuses on statement verification and evidence finding for tables from scientific articles in the english language. the task is divided into two subtasks -a and b. the aim of subtask a is to classify whether a given statement is entailed or refuted according to the given table and associated table metadata (such as captions and legends) or whether the statement's truth is unknown as it cannot be determined from the table. the aim of subtask b is to classify each cell in the table as relevant or irrelevant in determining whether the statement is entailed or refuted from the tabular evidence (the truth value of the statement is also provided). our systems use tapas @xcite trained with intermediate pre-training @xcite for both the subtasks. for subtask a, we fine-tune tapas after adding a three-way classification head on top for classifying the statement as entailed/refuted/unknown. we also evaluate how transfer learning and standardizing tables to have a single header row can improve tapas' performance. due to the similarity between subtask b and table question-answering (which involves cell selection or cell selection followed by aggregation), we use the tapas architecture previously used for table question-answering and fine-tune it to select the relevant cells. we also evaluate how different fine-tuning strategies can improve tapas' performance on evidence finding. our systems achieve an f1-micro score of 67.34 in subtask a and 72.89 in subtask a if the unknown statements are not considered while calculating the metrics (however, classifying entailed/refuted statements as unknown is still penalized). our submitted system achieves an f1 score of 62.95 in subtask b. during the post-evaluation phase, we modified our system and achieved an f1-score of 65.48 in subtask b. the code for our systems is available at https: //github.com/devanshg27/sem-tab-fact. ## background verifying if the given textual evidence supports a given statement is a fundamental natural language processing problem. it has been extensively studied under different tasks such as rte (recognizing textual entailment) @xcite , nli (natural language inference) @xcite , fever (fact extraction and verification) @xcite . in recent years, largescale pre-trained models @xcite @xcite have dominated these tasks and have achieved close-tohuman performance. nlvr @xcite and nlvr2 @xcite focus on verifying a statement given an image as evidence. tab-fact @xcite focuses on verifying a statement given a table from wikipedia foot_0 as evidence. along with releasing tabfact, chen et al. (2020) also discuss two promising approaches for tabular fact verification, latent program algorithm(lpa) and table-bert. lpa is a semantic parsing approach that parses statements into programs (logical forms) and executes the programs against the table to predict the entailment decision. most of the current models @xcite @xcite for tabfact are semantic parsing approaches similar to lpa. table-bert encodes the linearized tables and statements using bert-based models and directly pre-dicts the entailment decision. zhang et al. (2020) inject table structural information into the mask of the self-attention layer of bert-based models, which helps the model learn better table representations. tapas @xcite ) extends bert's architecture to capture the tabular structure, and it showed competitive performance on various table question answering datasets: sqa @xcite , wtq @xcite and wikisql @xcite . @xcite add an intermediate pre-training step before the fine-tuning step to tapas and show that it achieves state-of-the-art results on tabfact and sqa @xcite . their model is still 8 points behind human performance on tabfact since tabular fact verification involves table understanding and complex reasoning. while tabfact also focuses on fact verification using tables as evidence, it focuses on tables from wikipedia, whereas semeval-2021 task 9 (sem-tab-facts) instead focuses on tables from scientific articles and has a subtask related to evidence finding. also, tabfact did not have a neutral/unknown class, which they left out because of low inter-worker agreement due to confusion with refuted class. figure 1 shows an example of a table from the sem-tab-facts dataset and the labels for the two subtasks. ## system overview in this section, we provide a general overview of our systems for the two subtasks. we use tapas for both subtasks. ## conclusion in this paper, we presented our approach for fact verification and evidence finding for tabular data in scientific documents. we show that transfer learning from tabfact and standardization of the tables to have a single header helps improve our system's performance. we also show that having separate evidence finding models for entailed/refuted statements helps improve our system's performance in the second subtask. we also find that our model has a significant drop in performance on large tables since they are truncated to fit in the 512 tokens, the maximum number of tokens supported by tapas. in future work, we would like to experiment with table pruning methods like heuristic entity linking @xcite or heuristic exact match @xcite so that the statement and table can fit in 512 tokens. our systems did not use the table metadata while making the predictions. in the future, we would also like to explore extending the model to encode table metadata along with the table.