Assisting the Human Fact-Checkers: Detecting All Previously Fact-Checked Claims in a Document (2022)

Abstract: Given the recent proliferation of false claims online, there has been a lot of manual fact-checking effort. As this is very time-consuming, human fact-checkers can benefit from tools that can support them and make them more efficient. Here, we focus on building a system that could provide such support. Given an input document, it aims to detect all sentences that contain a claim that can be verified by some previously fact-checked claims (from a given database). The output is a re-ranked list of the document sentences, so that those that can be verified are ranked as high as possible, together with corresponding evidence. Unlike previous work, which has looked into claim retrieval, here we take a document-level perspective. We create a new manually annotated dataset for the task, and we propose suitable evaluation measures. We further experiment with a learning-to-rank approach, achieving sizable performance gains over several strong baselines. Our analysis demonstrates the importance of modeling text similarity and stance, while also taking into account the veracity of the retrieved previously fact-checked claims. We believe that this research would be of interest to fact-checkers, journalists, media, and regulatory authorities.

## introduction recent years have brought us a proliferation of false claims, which spread fast online, especially in social media; in fact, much faster than the truth @xcite . to deal with the problem, a number of fact-checking initiatives have been launched, such as factcheck, fullfact, politi-fact, and snopes, where professional fact-checkers verify claims. yet, manual fact-checking is very time-consuming and tedious. figure 1 : the architecture of our system. given an input document, it aims to detect all sentences that contain a claim that can be verified by some previously factchecked claims (from a given database). the output is a re-ranked list of the document sentences, so that those that can be verified are ranked as high as possible, together with corresponding evidence. thus, automatic fact-checking has been proposed as an alternative @xcite @xcite @xcite @xcite @xcite @xcite @xcite . while it scales better and works faster, it lags behind in quality, credibility, transparency, and explainability. manual and automatic fact-checking can benefit from each other as automatic methods are trained on data that human fact-checkers produce, while human fact-checkers can be assisted by automatic tools. a middle ground between manual and automatic fact-checking is to verify an input claim by finding a previously fact-checked claim that allows us to make a true/false judgment on the veracity of the input claim. this is the problem we will explore below. previous work has approached the problem at the sentence level: given an input sentence/tweet, produce a ranked list of relevant previously factchecked claims that can verify it @xcite . however, this formulation does not factor in whether the factuality of the input sentence/tweet can be determined using the database of previously fact-checked claims, as it is formulated as a ranking task. for example, in a us presidential debate that has 1,300 sentences on average, only a small fraction would be verifiable using previously factchecked claims from politifact. therefore, we target a more challenging reformulation at the document level, where the system needs to prioritize which sentences are most likely to be verifiable using the database of previously fact-checked claims. this is still a ranking formulation, but here we rank the sentences in the input document (by verifiability using the database of claims), as opposed to ranking database claims for one input sentence (by similarity with respect to that sentence). in our problem formulation, given an input document, the system needs to detect all sentences that contain a claim that can be verified by a previously fact-checked claim (from a given database of such claims). the output is a re-ranked list of the document sentences, so that those that can be verified are ranked as high as possible, as illustrated in figure 1. the system could optionally further provide a corresponding fact-checked claim (or a list of such claims) from the database as evidence. note that we are interested in returning claims that would not just be relevant when fact-checking the claims in the input sentence, but also would be enough to decide on a verdict for its factuality. this novel formulation of the problem would be of interest to fact-checkers not only when they are facing a new document to analyze, but also when they want to check whether politicians keep repeating claims that have been previously debunked, so that they can be approached for comments. it would also be of interest to journalists, as it could bring them a tool that can allow them to put politicians and public officials on the spot, e.g., during a political debate, a press conference, or an interview, by showing the journalist in real time which claims have been previously fact-checked and found false. finally, media outlets would benefit from such tools for self monitoring and quality assurance, and so would regulatory authorities such as ofcom. our contributions can be summarized as follows: the vast majority of previous work has focused on the latter problem, while the other three problems remain understudied, even though there is an awareness that they are integral steps of an end-toend automated fact-checking pipeline @xcite . this situation is gradually changing, and the research community has recently started paying more attention to all four problems, in part thanks to the emergence of evaluation campaigns that feature all steps such as the clef checkthat! lab. @xcite proposed a claim-focused task formulation, and released two datasets: one based on politifact, and another one based on snopes. they had a ranking formulation: given a claim, they asked to retrieve a ranked list of previously fact-checked claims from a given database of such claims; the database included the verified claims together with corresponding articles. one can argue that this formulation falls somewhere between (ii) detecting relevant previously fact-checked claims and (iii) retrieving relevant evidence to fact-check a claim. the same formulation was adopted at the clef checkthat! lab in 2020, where the focus was on tweets, and in 2021-2022, which featured both tweets and political debates @xcite @xcite @xcite . the best systems at the clef checkthat! 2021 lab used bm25 retrieval, semantic similarity using embeddings, and reranking @xcite @xcite . a follow-up work used a batch softmax contrastive loss to better fine-tune bert for the task @xcite . it has been further shown that it is important to match not only against the target claim, but also using the full text of the associated article that factcheckers wrote to explain their verdict. thus, in a follow-up work, @xcite focused on modeling the context when checking an input sentence from a political debate, both on the source side and on the target side, e.g., by looking at neighboring sentences and using co-reference resolution. @xcite proposed a re-ranker based on memory-enhanced transformers for matching (mtm) to rank fact-checked articles using key sentences selected using lexical, semantic and patternbased similarity. @xcite modeled claimmatching using topic-aware evidence reasoning and stance-aware aggregation, which model semantic interaction and topical consistency to learn latent evidence representation. @xcite developed two datasets (one consisting of claim-like statements and the other one using annotation of claim similarity) covering four languages. @xcite used sequence-to-sequence transformer models for sentence selection and label prediction. @xcite proposed a deep q-learning network, i.e., a reinforcement learning approach, which computes candidate pairs of precise evidence and their labels, and then uses postprocessing to refine the candidate pairs. vo and lee (2020) looked into multimodality. they focused on tweets that discuss images and tried to detect the corresponding verified claim by matching both the text and the image against the images in the verified claim's article. they mined their dataset from pairs of tweets and corresponding fact-checking articles proposed by twitter users as a response. @xcite used a similar crowd-checking idea, and further proposed how to learn from potentially noisy data. finally, the task was also addressed in a reverse formulation, i.e., given a database of fact-checked claims (e.g., a short list of common misconceptions about covid-19), find social media posts that make similar claims @xcite . unlike the above work, our input is a document, and the goal is to detect all sentences that contain a claim that can be verified by some previously fact-checked claim (from a given database). ## task definition we define the task as follows (see also figure 1 ): given an input document and a database of previously fact-checked claims, produce a ranked list of its sentences, so that those that contain claims that can be verified by a claim from the database are ranked as high as possible. we further want the system to be able to point to the database claims that verify a claim in an input sentence. note that we want the input sentence to be verified as true/false, and thus we want to skip matches against verified claims with labels of unsure veracity such as half-true. note also that solving this problem requires going beyond stance, i.e., whether a previously fact-checked claim agrees/disagrees with the input sentence @xcite . in certain cases, other factors might also be important, such as, (i) whether the two claims express the same degree of specificity, (ii) whether they are made by the same person and during the same time period, (iii) whether the verified claim is true/false or is of mixed factuality, etc. 4 dataset ## evaluation measures given a document, the goal is to rank its sentences, so that those that can be verified (i.e., with a true/false verdict; verdict-input in note that the simple ap can also be represented as ap r 1 , as it increments by 1 regardless of whether a relevant verified claim is in the top-r of the list of verified claims. we compute m ap , m ap r h , m ap r 0 , and m ap r 0.5 by averaging ap , ap r h , ap r 0 , and ap r 0.5 , respectively, over the test transcripts. we also compute m ap inner by averaging the ap inner on the verified claims: we compute ap inner for a given input sentence, by scoring the rankings of the retrieved verified claims as in the task presented in @xcite . ## model the task we are trying to solve has two subtasks. the first sorts the input sentences in the transcript in a way, so that the input sentences that can be verified using the database are on top. the second one consists of retrieving a list of matching verified claims for a given input sentence. while we show experiments for both subtasks, our main focus is on solving the first one. ## experiments and evaluation we performed a 7-fold cross-validation, where we used 6 out of the 7 transcripts for training and the remaining one for testing. we first computed 19 similarity measures and then used them to test the baselines and to train pairwise learning-to-rank models. the results are shown in ## conclusion and future work we introduced a new challenging real-world task formulation to assist fact-checkers, journalists, media, and regulatory authorities in finding which claims in a long document have been previously fact-checked. given an input document, we aim to detect all sentences containing a claim that can be verified by some previously fact-checked claims (from a given database of previously fact-checked claims). we developed a new dataset for this task formulation, consisting of seven debates, 5,054 sentences, 16,636 target verified claims to match against, and 75,810 manually annotated sentenceverified claim pairs. we further defined new evaluation measures (variants of map), which are better tailored for our task setup. we addressed the problem using learning-to-rank, and we demonstrated sizable performance gains over strong baselines. we offered analysis and discussion, which can facilitate future research, and we released our data and code. in future work, we plan to focus more on detecting the matching claims, which was our second objective here. we also plan to explore other transformer architectures and novel ranking approaches such as multi-stage document ranking using monobert and duobert @xcite . @xcite we have developed a dataset and proposed and evaluated a model using data from politifact, which consists of political statements. we have not evaluated our approach on other topics, e.g., factual claims appearing on social media, which is out of the scope of the present work.