Combining sentence and table evidence to predict veracity of factual claims using T a P a S and R o BERT a (2021)

Abstract: This paper describes a method for retrieving evidence and predicting the veracity of factual claims, on the FEVEROUS dataset. The evidence consists of both sentences and table cells. The proposed method is part of the FEVER shared task. It uses similarity scores between TF-IDF vectors to retrieve the textual evidence and similarity scores between dense vectors created by fine-tuned TaPaS models for tabular evidence retrieval. The evidence is passed through a dense neural network to produce a veracity label. The FEVEROUS score for the proposed system is 0.126.

## introduction until recently, fact checking has been done solely by journalists and other human labor. with the large spread of information on the internet, the human labor component of fact checking has become a bottleneck. vlachos and riedel @xcite introduced the problem of automatic fact checking. since then a lot of progress has been made, most notably with the release of the fever @xcite dataset, and succeeding systems using that dataset. the feverous shared task @xcite ) consists of retrieving evidence and predicting the veracity of a given claim, based on the retrieved evidence. this task can be divided into two separate tasks. the first is to retrieve the most relevant evidence from the given wikipedia dataset. here the evidence can be any form, such as sentences, table cells or list items. the second task is to predict the veracity for the claim, given the retrieved evidence. in this paper a method for solving this task is proposed. the method can be divided into the following parts: each of the parts will be explained in detail later in this paper. the document retrieval and sentence extraction models are based on tf-idf representation and cosine distances to get the most relevant documents and sentences. the table extraction part uses the model proposed by @xcite , which uses dense representations from pre-trained tapas @xcite models to retrieve the most relevant tables. similarly, the table cell extraction part uses a tapas model to retrieve the most relevant table cells from the previously retrieved tables. lastly the claim veracity prediction model uses tapas and roberta @xcite representations of the retrieved evidence as input to a dense neural network that outputs the predicted veracity label of the given claim. ## related work most previous research has been done on verification of claims based on textual evidence. however, a large amount of information on the internet is not in textual form, such as tables and databases. chen et al introduced a dataset, tabfact @xcite , containing textual claims paired with tables and labelled either entailed or refuted. two methods for determining the veracity of the claim are also proposed, @xcite . this paper describes a closed setting, where the table to use is given. in a real application an open setting would be encountered, where the evidence needs to be retrieved first, before veracity prediction can take place. the open setting scenario was studied by schlichtkrull et al @xcite ) which introduced a model for retrieving tables before using these to predict a veracity label for a claim. the previously mentioned research has been investigating fact verification with either text or tables as evidence, but not both. unik-qa @xcite is a question answering system that incorporates lists, tables and knowledge graphs. this is done by flattening the structured data into plain text using simple heuristics. chen et al introduced a dataset called hybridqa @xcite , consisting of questions paired with both tables and text that contain the answer. thus, to get a good score on this dataset the provided model needs to be able to represent both tabular and textual data. pre-trained language models have turned out to be useful on downstream nlp tasks. when bert @xcite was introduced it showed that a model can be trained on one task then be used to perform on another with good performance. a lot of other pre-trained models have been created after the introduction of bert, one of them being roberta @xcite . roberta has the same architecture as bert but is trained in a different way and on more data, which yields a better performance. most of the language models are pre-trained on textual data, but the feverous dataset requires reasoning over tabular data for some of the samples. tapas @xcite ) is one language model that is pre-trained on tables, instead of plain text. the input to the model includes tokens that represent the structure of the tables, such as row and column number. ## model in this section the different parts of the model is described in detail. a graphical overview of the model is shown in figure 1 . ## discussion one improvement to the algorithm could be to incorporate the table caption into the table representation. currently that information is discarded, but for some tables it could be crucial to know that to make a valid interpretation of the table. by studying the dataset, one could find that some required evidence are the table captions. another extension would be to add the page title or surrounding context into the representation of the table . tables are rarely presented without some context around them, and for the algorithm to give a valid prediction, it might need to know the surrounding context, the same way as human beings would. ## conclusion this article describes a method for solving the feverous shared task introduced by @xcite . the system consists of two main parts, the retrieval part and the verification part. in the retrieval part, documents are first retrieved, then sentences and table cells are extracted from these documents. for the verification part, the previously retrieved evidence, in form of sentences and ta-ble cells, are used to make a prediction about the veracity of a given claim. while the results show that this method may not be feasible in a real world application, there are parts that show promising results. the problem of automatic fact checking is difficult and includes several nlp topics, such as information extraction and natural language inference. certainly there will be further advances in this topic, which may result in a system that is usable in real world applications.