B reaking BERT @ IITK at S em E val-2021 Task 9: Statement Verification and Evidence Finding with Tables (2021)

Abstract: Recently, there has been an interest in the research on factual verification and prediction over structured data like tables and graphs. To circumvent any false news incident, it is necessary to not only model and predict over structured data efficiently but also to explain those predictions. In this paper, as the part of the SemEval-2021 Task 9, we tackle the problem of fact verification and evidence finding over tabular data. There are two subtasks, in which given a table and a statement/fact, the subtask A is to determine whether the statement is inferred from the tabular data and the subtask B is to determine which cells in the table provide evidence for the former subtask. We make a comparison of the baselines and state of the art approaches over the given SemTabFact dataset. We also propose a novel approach CellBERT to solve the task of evidence finding, as a form of Natural Language Inference task. We obtain a 3-way F1 score of 0.69 on subtask A and an F1 score of 0.65 on subtask B.

## introduction textual inference, also known as natural language inference @xcite , plays an important role in the study of natural language understanding and semantic representation. due to the unprecedented amount of information generated over the internet, it becomes essential for machines to comprehend new information based on previous knowledge. recent social events like political elections and pandemic spread have also shown the need for intelligent fact-checking systems that majorly depends on textual inference over the scientific data. though textual inference is well explored, the current works mainly deal with unstructured evidence in the form of sentences @xcite . verification under structured and semi-structured evidence, such as tables, graphs, and databases, remains unexplored. tables are ubiquitous in documents and presentations for concisely conveying important information; however, inference on structured data like tables or graphs is much more difficult than simple text format due to complex structure and non-universal schema for the representation of data. though recently, there has been work on tabular inference problems @xcite @xcite @xcite @xcite explaining the prediction, evidence finding is still an unexplored area. through the semeval-2021 task 9 @xcite we have tried to solve the tabular inference problem over scientific tables by providing an answer as well as a solution to our reasoning. in other words, given the structured table data and statement, we aim to classify the statement as entailed, unknown (neutral), or contradiction. in addition, we also aim to classify each cell of the table whether it is relevant or irrelevant in making the aforementioned prediction. our contribution is three-fold: to handle large size tables, table pruning is done using heuristic exact match (hem). in this method, the columns are ranked by a relevance score and added in order of decreasing relevance. columns that exceed the maximum input length are skipped. our task for statement verification differs from the works mentioned above in the aspect that we have a third label, 'unknown', where the table fails to provide any information to infer the statement. moreover, no prior work has been performed concerning the task of evidence finding. ## experiments 4.1 subtask a following the tabfact's tablebert, we fine-tuned our own tablebert model on the semtabfact dataset for subtask a. we also experimented with mutations of tablebert by using roberta (tableroberta), xlnet (tablexlnet), and scib-ert (tablescibert) as well. we also experimented with implementing bigru layers on top of these table transformers. all our experiments were conducted using pytorch @xcite deep learning library. we experimented with a dataset (d 1 ) which contained the manual dataset along with the autogenerated statements. a caveat to d 1 was that autogenerated statements that have common tables with the manual dataset were only used. this was done because every model we trained upon only on the manual dataset was overfitting. the overfitting was due to an insufficient number of statements, i.e., 4056 in the manual dataset. after preparing the dataset d 1 , we had a total of 72k statements. for the similarity-based approach, we manually experimented with various non-semantic similarity approaches like edit distance and binary-matching foot_1 as well as embedding space-based semantic similarity approaches by first computing the word vector. we computed word vectors using glove @xcite , bert @xcite and roberta @xcite . cosine similarity and euclidean distance was used to compute similarity between two vectors. the non-semantic based binary-matching approach outperformed others upon the validation set; therefore, we used it to evaluate our results on the test set. in our final submitted model, scibert was fine-tuned for 3 epochs upon the combined dataset d 1 and development dataset, with learning rates as 5e -5, 5e -6 and 1e -6 for each epoch. batch size was kept as 6 with maximum sequence length of 512 tokens. tapas was fine tuned upon the auto-generated, manual and development datasets separately for 6, 12 and 5 epochs respectively. learning rate was kept the same as 2e -5 for each epoch with maximum sequence length as 512. dropout probability was set to 0.07. for ensemble, we used weights 0.7 and 0.3 for tapas and tablescibert respectively. we set the hyperparameter λ a as 2 in our similarity heuristic. ## conclusion this paper attempts a solution to an under-explored but essential problem: statement verification and evidence finding with tables. there have been various works related to a binary classification of statements. still, evidence finding for these classifications is a difficult and novel challenge. we are successfully able to present an ensemble of tapas model, table transformer-based tablescibert, and similarity heuristic trained for subtask a with 2way f1 of 0.768 and 3-way f1 of 0.693. in subtask b, we implemented the cellbert -devt+ similarity ensemble method as our best model with an f1 score of 0.652. in the future, we plan to progress in implementing new models that can tackle both linguistic and symbolic reasoning. we aim to extend the tapas model to 3 labels, requiring large data for training in unknown labels for good results. in the case of subtask b, we are planning to experiment with other nli techniques and models. besides, we will be looking into using more data for training cellbert and other nli models as well.