The Role of Context in Detecting Previously Fact-Checked Claims (2022)

Abstract: Recent years have seen the proliferation of disinformation and fake news online. Traditional approaches to mitigate these issues is to use manual or automatic fact-checking. Recently, another approach has emerged: checking whether the input claim has previously been fact-checked, which can be done automatically, and thus fast, while also offering credibility and explainability, thanks to the human fact-checking and explanations in the associated fact-checking article. Here, we focus on claims made in a political debate and we study the impact of modeling the context of the claim: both on the source side, i.e., in the debate, as well as on the target side, i.e., in the fact-checking explanation document. We do this by modeling the local context, the global context, as well as by means of co-reference resolution, and multi-hop reasoning over the sentences of the document describing the fact-checked claim. The experimental results show that each of these represents a valuable information source, but that modeling the source-side context is most important, and can yield 10+ points of absolute improvement over a state-of-the-art model.

## introduction the fight against dis/mis-information has become an urgent social and political matter. online media have been widely used not only for social good, but also to mislead entire communities. many fact-checking organizations, such as factcheck.org, foot_0 snopes, 2 politifact, foot_2 and full-fact, 4 as well as some broader international initiatives such as the credibility coalition foot_4 and eufactcheck, 6 have emerged to address the problem @xcite . there have also been efforts to develop automatic systems to detect such content @xcite @xcite @xcite @xcite , including the development of datasets @xcite , systems @xcite , and evaluation campaigns @xcite @xcite . an important issue with automatic systems is that journalists and fact-checkers often question their credibility for reasons such as (perceived) insufficient accuracy given the state of present technology, but also due to the lack of explanation about how the system has made its decision. on the other hand, manual fact-checking is time-consuming and does not scale. yet, time is precious: it has been reported in the literature that fake news travels faster than real news @xcite , and that 50% of the spread of some very viral false claims has happened within the first ten minutes after they got published @xcite . such findings show the importance of real-time fake news detection, which can enable a timely intervention. as both manual and automatic systems have their limitations, there have been proposals for human-in-the-loop settings, aiming to bring the best of both worlds. in order to enable such an approach, one question that arises is how to facilitate fact-checkers and journalists with automated systems @xcite ). an immediate problem is to know whether a given input claim has been previously fact-checked by a reputable fact-checking organization. this would give the journalist a credible reference and could save her significant amount of time, as manually fact-checking a single nontrivial claim may take from 1-2 days to 1-2 weeks. while earlier studies have suggested that such a mechanism should be part of an end-to-end automated system, there has been limited work in this direction @xcite . figure 1 : a pipeline of retrieving and ranking previously fact-checked claims. s i is the claim (source), t t is the title of the target, t j is a sentence from the target. at the time of covid-19, there are a number of false claims and conspiracy theories spreading online, e.g., about bill gates and his chips in the covid-19 vaccine, about garlic water as a cure, etc. many such claims have already been debunked, but this does not stop them, as they keep being repeated, potentially in a slightly different form but with the same meaning. thus, it is important to recognize such variations quickly, and possibly to post a reply in social media with a link to a factchecking article. similarly, in a scenario where a politician is being interviewed or is taking part in a debate, a quick check against a collection of previously fact-checked claims would make it possible to put him/her on the spot in real time. however, the problem in a real-time scenario is that, unlike written text, interviews, debates, and speeches are more spontaneous, and the claims that are being made are often not clearly formulated in a single sentence. this is illustrated in figure 1 , where we can see a fragment from a democratic debate as part of the 2016 us presidential election, where hillary clinton said: "i waited until it had actually been negotiated because i did want to give the benefit of the doubt to the administration." understanding this claim requires pronominal coreference resolution (e.g., what does it refer to, is it cafta or is it tpp, as both are mentioned in the previous sentences?), more general co-reference (e.g., that the administration being discussed is the obama administration), as well as general understanding of the conversation so far, and possibly general world knowledge about us politics at the time of the debate (e.g., that hillary clinton was secretary of state when tpp was being discussed). moreover, previous research has shown that it is beneficial to match the input claim not only against the canonical verified claim that fact-checkers worked with, but against the entire article that they wrote explaining why the claim was ultimately judged to be true/false @xcite . this is because, in the fact-checking article, the claim is likely to be paraphrased in different ways, and there could also be background information and related terms, which can facilitate claim matching, and thus improve recall. this means that we need to make use of the global contextual information contained within the full text of the fact-checking article or at least the sentences next to the claim, i.e., the local context. similarly, for the fever fact-checking task, which asks to factcheck against wikipedia, it has been shown that multi-hop reasoning (transformer-xh) over the sentences of the target article can help @xcite , an observation that was further confirmed in the context of fact-checking political claims @xcite . transformer-xh uses a novel attention mechanism that naturally "hops" across the connected text sequences in addition to attending over tokens within each sequence. as claims and reasoning about them are manifested across documents, this hop-based attention mechanism constructs global contextualized representation to provide better joint multi-evidence reasoning. in the present work, we rely on transformer-xh to extract and use global contextual information. based on the above considerations, we propose a framework that focuses on modeling coreference, local context (representation from neighboring sentences; see section 4.2.2), and global context (representation from transformer-xh; see section 4.2.3), both on the source and on the target side, while also using multi-hop reasoning over the target side. our contributions can be summarized as follows: ## dataset here, we focus on the problem of detecting previously fact-checked claims, using the task formulation and an adaptation of data from @xcite . they had two datasets: one on matching tweets against snopes claims, and another one on matching claims in the context of a political debate to politifact claims. here, we focus on the latter, 8 and we perform a close analysis of the claims and what makes them easy/hard to match. we experimented with their politifact dataset, which targets claims related to us politics. after a us political debate, speech, or interview, factchecking journalists from politifact would select few claims made in the event and would verify them either from scratch or by linking them to a previously fact-checked claim. each fact-checked claim has an associated article stating its degree of factuality along with an explanation of how the fact-checkers arrived at their verdict. the dataset has two parts: (i) verified claims {normalized ver-claim, article title, and article text}, (ii) transcripts of the political events (e.g., debates). @xcite annotated the data by linking sentences from the transcript (inputclaim) to one or more verified claims (out of 16,636 claims in politifact). says hillary clinton "wanted the wall." 695 part-of c. wallas: and since then, as we all know, nine women have come forward and have said that you either groped them or kissed them without their consent. the stories from women saying he groped or forced himself on them "largely have been debunked." the stories from women saying he groped or forced himself on them "largely have been debunked to further analyze the dataset, we looked at the inputclaim-verclaim pairs, and we manually categorized them into one of the following categories: the above categories include all types of pairs we have seen. moreover, since the dataset is constructed from speeches, debates, and interviews, the structure of the inputclaim-verclaim pairs differs. for example, in debates, we see more part-of examples, as there are multiple question-answer claim pairs, as well as back-and-forth arguments splitting the claims into multiple sentences. the annotations were performed by three annotators who are experts in fact-checking (and coauthors of this paper), using the above definitions for the categories. we consolidated their annotations using majority voting, and they had a consolidation discussion for cases with no majority. the fleiss kappa inter-annotator agreement was 0.5, which corresponds to moderate agreement, which is reasonable for such a complex annotation task. finally, we should note while annotating the data into the above four categories, we found out that a small number of inputclaim-verclaim pairs in @xcite were false matches (which happened, as they did the matching automatically, without manually double-checking every single example). we removed these pairs, and thus our reported number of pairs is slightly lower than theirs. we further observed that shaar et al. (2020a) dealt with each inputclaim independently, i.e., at the sentence level. this is problematic because for part-of claims we could end up splitting them and putting them in different sets: one in training, and one in testing. moreover, splitting the dataset in this way means that the examples for a given topic can split between training and test, and thus information can leak, e.g., a claim can be repeated. therefore, we considered new splits for the data: in our experiments, we chose to use the most realistic, but also the hardest setup: debate-level chrono. as a result, our map score, when experimenting with the state-of-the-art model of @xcite , decreases from 0.602 to 0.429. ## experimental setup below, we first introduce the experimental setup for our baseline, and then we describe our proposed model that takes the context of the input claim into account, both on the source and on the target side. ## conclusion and future work we have presented our work on the important but under-studied problem of detecting previously factchecked claims in political debates and speeches. we studied the impact of modeling the context: both on the source side, i.e., in the debate, as well as on the target side, i.e., in the fact-checking document that explains how human fact-checkers have arrived at their decision about the factuality of the claim. in particular, we modeled the local context, the global context, and we further used coreference resolution and multi-hop reasoning over the target text using transformer-xh. the experimental results have shown that each of these components represents a valuable information source, but modeling the source-side context is more important, and can yield 10+ points of absolute improvement over a context-free state-of-the-art baseline. in future work, we want to try other multi-hop reasoning frameworks for context modeling. we also plan to experiment with other kinds of conversations, e.g., in community forums and in social media, including for other languages.