A ttes T able at S em E val-2021 Task 9: Extending Statement Verification with Tables for Unknown Class, and Semantic Evidence Finding (2021)

Abstract: This paper describes our approach for Task 9 of SemEval 2021: Statement Verification and Evidence Finding with Tables. We participated in both subtasks, namely statement verification and evidence finding. For the subtask of statement verification, we extend the TAPAS model to adapt to the ‘unknown’ class of statements by finetuning it on an augmented version of the task data. For the subtask of evidence finding, we finetune the DistilBERT model in a Siamese setting.

## introduction tables provide a compact and structured way of presenting information. they are easily interpretable by humans and are widely used in scientific papers, business articles, and even in government reports. at the same time, it is easy to misinterpret tabular data and even use it maliciously to spread misinformation. thus, systems that can verify facts from tables and locate evidence in them can go a long way in ameliorating issues related to table interpretation. such systems can also be used for question-answering over large tables, that are difficult to analyze manually, since the task of fact verification with evidence finding is closely related to that of question answering @xcite . table entailment refers to the task of finding whether a sentence is refuted or supported by a given table. traditionally, it has been considered a binary classification task. however, there could be instances where the table is not capable enough to either refute or accept the given statement, meaning the statement context is "unknown" for the table. in this paper, we have presented a table entailment setup that extends to three classes: refuted, entailed, and unknown, as a part of the sem-tab-fact task @xcite . the task could be divided into two parts: statement verification, and evidence finding. given a table and a statement, one has to first determine whether the table entails the statement, refutes the statement, or has insufficient information about it. this forms the subtask of statement verification. if the table entails or refutes the statement, one would also like to know which cells in the table provided evidence for the same. this is the subtask of evidence finding, that can be formulated as a binary classification problem. each cell in the table is assigned one of the 'relevant' or 'irrelevant' labels depending on whether it provided evidence for the given statement. recent years have seen a rise in the use of transfer learning in language processing @xcite owing to their superior performance. models based on underlying concepts like attention mechanism and transformers are seeing widespread use across a range of tasks @xcite . our findings concurred with this trend as we used similar kinds of architectures as the fundamental blocks in the systems for both the subtasks. for the subtask of statement verification, we modify the recently released tapas model @xcite that is pre-trained on the tabfact dataset @xcite . the pre-trained model is trained on only two classes: 'entailed' and 'refuted', and is not capable of classifying the 'unknown' samples. the training data provided to us for our task also contains the same two labels. to adapt to the 'unknown' statements, we augment the given data by including random statements from other tables as unknown. we then fine-tune the tapas model by extending it to three classes on this augmented data. using this approach, we achieve a three-way f1 score of 65.59 and a two-way f1 score of 71.72 on the test data. we ranked 8 th on the official leaderboard for this subtask. for the subtask of evidence finding, we finetune the distilbert model @xcite in a siamese setup @xcite . the model is pre-trained on the snli dataset @xcite , the multinli dataset @xcite and the sts benchmark @xcite . for finetuning, we use the contrastive loss @xcite as our loss function. for making a prediction, we calculate the cosine similarity between the embeddings computed by the model for both the statement and the cell value. if the cosine similarity value crosses a set threshold, we consider the cell to be relevant to the given statement. using this approach, we achieve an f1-score of 43.02 on the test data and secured the 7 th rank on the official leaderboard for this subtask. the source code of our proposed approach for both the subtasks has been made public foot_0 to encourage usage and improve the reproducibility of the results. ## related work recently, the tabfact dataset was released, which contained 118k human-annotated statements, related to about 16k wikipedia tables. these statements were annotated for only two labels, 'entailed' and 'refuted', leaving out the 'unknown' cases. a system's ability to distinguish 'unknown' statements from 'entailed' and 'refuted' is quite critical, as one may elusively create seemingly believable statements that are actually 'unknown'. recently, there has been work in areas related to table fact verification, especially since the release of the tabfact dataset. many of these approaches are graph-based in nature @xcite @xcite . the current stateof-the-art on the tabfact dataset is the recently released tapas model, which outperforms its predecessor by approximately 6%. unfortunately, all of these models are quite far from human-level performance, suggesting ample scope for improvement. little to no work has been done previously on the task of evidence finding, with the closest approaches being related to table-based question answering like wikitablequestions @xcite , wikisql @xcite and sqa @xcite where getting the answer for a question can be considered analogous to getting evidence for a statement. ## conclusion thus we have presented a statement verification and evidence finding setup for tables. for subtask-a, we extended the tapas model to adapt to the 'unknown' statements. for subtask-b, we used a semantic approach for evidence finding. our results for subtask-a show the problems encountered while generating and working with the 'unknown' statements. for subtask-b, our results show the effect of taking only the semantic information into 6 training the large variant on augmented data exceeds the time limit of google colab ( 12 hours) account. an important future prospect for subtask-a would be to find a more effective way of generating the 'unknown' statements. for subtask-b, utilizing the table's available structural information to improve the results seems to be a promising prospect.