Automatic Fact-Checking with Document-level Annotations using BERT and Multiple Instance Learning (2021)

Abstract: Automatic fact-checking is crucial for recognizing misinformation spreading on the internet. Most existing fact-checkers break down the process into several subtasks, one of which determines candidate evidence sentences that can potentially support or refute the claim to be verified; typically, evidence sentences with gold-standard labels are needed for this. In a more realistic setting, however, such sentence-level annotations are not available. In this paper, we tackle the natural language inference (NLI) subtask—given a document and a (sentence) claim, determine whether the document supports or refutes the claim—only using document-level annotations. Using fine-tuned BERT and multiple instance learning, we achieve 81.9% accuracy, significantly outperforming the existing results on the WikiFactCheck-English dataset.

## introduction as the volume of information present on the internet steeply increases, automatic fact-checking has become a promising approach to identify and stop the spread of misinformation. active research in this area has been supported in part by a handful of carefully curated datasets @xcite @xcite . while these datasets have been playing a crucial role in the development of the latest fact-checkers, they do not faithfully represent the reality in certain aspects. first, these datasets come with short evidence snippets for the claims; most existing factcheckers rely on such sentence-level evidence annotations to build the natural language inference (nli) component of their systems-e.g. given an evidence sentence and a claim sentence, determine if the claim is supported @xcite . second, most of the datasets consist of synthetic claims written by annotators based on snippets of in this paper, we tackle the nli subtask-given a document and a (sentence) claim, determine whether the document supports or refutes the claim-only using document-level annotations. we improve on existing systems trained and tested on the wikifactcheck-english dataset during both steps of the 2-step pipeline: evidence retrieval and support verification. we find that fine-tuned bert with multiple instance learning (mil)to use multiple candidate evidence sentencesresults in about 13% increase in accuracy over the baseline. however, incorporating sentence-bert (sbert) @xcite to identify candidate evidence sentences during evidence retrieval does not lead to a noticeable improvement. the work we build off of in this paper is from @xcite . the authors released a new fact-checking dataset with only document-level annotations that we use in this work. along with it, they presented baseline systems consisting of a two-step pipeline: evidence retrieval and support verification. in our work, we adopt the same pipeline but make improvements to each step using bert-based approaches. also, unlike their approach, we retrieve multiple sentences during the evidence retrieval step and compute the final aggregated prediction using multiple instance learning. ## method we tackle the nli subtask of automatic fact checking-given a relevant document (a list of sentences e c ) and a claim c, determine whether the e c supports or refutes c. this is done in two-steps: (1) evidence retrieval and (2) support verification, as shown in figure 1 . the same pipeline was used in @xcite ; we improve both steps using bert-based approaches and multiple instance learning to make use of multiple candidate evidence sentences. ## conclusion and future work we presented a 2-step pipeline to determine if a document supports or refutes a claim only using document-level annotations. fine-tuned bert with mil to use multiple candidate evidence sentences resulted in about 13% increase in accuracy over the baseline. however, incorporating sbert in evidence retrieval did not lead to additional gains that are noticeable. in the future, we want to leverage the context, as sentences in this dataset, and in the wild, are often not self-contained. we expect the context to be useful for disambiguating keywords in both evidence retrieval and support verification. note that both the claim and evidence sentences have contexts, and the context of an evidence sentence is a potential evidence sentence itself. both opens up many possibilities to be explored.