Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation (2022)

Abstract: Misinformation emerges in times of uncertainty when credible information is limited. This is challenging for NLP-based fact-checking as it relies on counter-evidence, which may not yet be available. Despite increasing interest in automatic fact-checking, it is still unclear if automated approaches can realistically refute harmful real-world misinformation. Here, we contrast and compare NLP fact-checking with how professional fact-checkers combat misinformation in the absence of counter-evidence. In our analysis, we show that, by design, existing NLP task definitions for fact-checking cannot refute misinformation as professional fact-checkers do for the majority of claims. We then define two requirements that the evidence in datasets must fulfill for realistic fact-checking: It must be (1) sufficient to refute the claim and (2) not leaked from existing fact-checking articles. We survey existing fact-checking datasets and find that all of them fail to satisfy both criteria. Finally, we perform experiments to demonstrate that models trained on a large-scale fact-checking dataset rely on leaked evidence, which makes them unsuitable in real-world scenarios. Taken together, we show that current NLP fact-checking cannot realistically combat real-world misinformation because it depends on unrealistic assumptions about counter-evidence in the data.

## introduction according to van der linden (2022), misinformation is "false or misleading information masquerading as legitimate news, regardless of intent". misinformation is dangerous as it can directly impact human behavior and have harmful real-world consequences such as the pizzagate shooting @xcite , interfering in the 2016 democratic us election @xcite , or the promotion of false covid-19 cures (aghababaeian et al., 2020). surging misinformation during the covid-19 pandemic, coined "infodemic" by who (zarocostas, 2020), exemplifies the danger coming from misinformation. to combat misinformation, journalists from fact-checking organizations (e.g., poli-tifact or snopes) conduct a laborious manual effort to verify claims based on possible harms and their prominence @xcite . however, manual factchecking cannot keep pace with the rate at which misinformation is posted and circulated. automatic fact-checking has gained significant attention within the nlp community in recent years, with the goal of developing tools to assist fact-checkers in combating misinformation. for the past few years, nlp researchers have created a wide range of factchecking datasets with claims from fact-checking organization websites @xcite @xcite @xcite @xcite . the fundamental goal of fact-checking is, given a claim made by a claimant, to find a collection of evidence and provide a verdict about the claim's veracity based on the evidence. the underlying technique used by fact-checkers, and journalists in general, to assess the veracity of a claim is called verification @xcite . in a comprehensive survey, @xcite proposed an nlp fact-checking framework (fcnlp) that aggregates existing (sub)tasks and approaches of automated fact-checking. fcnlp reflects current research trends on automatic factchecking in nlp and divides the aforementioned process into evidence retrieval, verdict prediction, and justification production. in this paper, we focus on harmful misinformation claims that satisfied the professional factcheckers' selection criteria and refer to them as real-world misinformation. our goal is to answer the following research question: can evidencebased nlp fact-checking approaches in fc-nlp refute novel real-world misinformation? fcnlp assumes a system has access to counterevidence (e.g., through information retrieval) to refute a claim. consider the false claim "telemundo is an english-language television network" from fever @xcite : a system following fcnlp must find counter-evidence contradicting the claim (i.e., telemundo is a spanish company) to refute the claim. this may require more complex reasoning over multiple documents. we contrast this example to the real-world false claim that "half a million sharks could be killed to make the covid-19 vaccine" (figure 1 ). if true, credible sources would likely report this incident, providing supporting evidence. as it is not, before being factchecked, there is no refuting evidence stating that covid-19 vaccine production will not kill sharks. only after guaranteeing that the claim relies on the false premise of covid-19 vaccines using squalene (harvested from sharks), it can be refuted. after the claim's verification, fact-checkers publish reports explaining the verdict and thereby produce counter-evidence. relying on counter-evidence leaked from such reports is unrealistic if a system is to be applied to new claims. in this work, we identify gaps between current research on fcnlp and the verification process of professional fact-checkers. via analysis from different perspectives, we argue that the assumption of the existence of counter-evidence in fcnlp is unrealistic and does not reflect real-world requirements. we hope our analysis sheds light on future research directions in automatic fact-checking. in summary, our major contributions are: • we identify two criteria from the journalistic verification process, which allow overcoming the reliance on counter-evidence (section 2). ## how humans fact-check to motivate our distinct focus on misinformation, we investigate what claims professional factcheckers verify. we crawl 20,274 fact-checked claims from politifact foot_1 ranging from 2007-2021. figure 2 shows the ratio of different verdicts 3 per year. after 2016, fact-checkers increasingly select false claims as important for fact-checking. in 2021 less than 10% of the selected claims were correct. some claims can be refuted via counter-evidence (as required by fcnlp). for example, official statistics can contradict the false claim about the u.s. that "in the 1980s, the lowest income people had the biggest gains". if the evidence makes it claim based upon (1) if you were forced to use a sharpie to fill out your ballot, that is voter fraud. professional fact-checkers refute misinformation even if no global counter-evidence exists, e.g., by rebutting underlying assumptions (figure 1 ). both criteria are challenging for computers but naturally satisfied by human fact-checkers. @xcite defines the question "how do you know that?" to be at the heart of verification. after selecting a claim, finding provenance and sourcing are the first steps in journalistic verification. provenance provides crucial information about context and motivation @xcite . journalists must then identify solid sources to compare the claim with @xcite . ideally, the claimant provides sources, which must be included and assessed in the verification process. during verification, journalists rely, if possible, on relevant primary sources, such as uninterpreted and original legislation documents (for claim 2, in this section, we first analyze human verification strategies based on an analysis of 100 misinformation claims. we then contrast human verification strategies with fcnlp. ## nlp fact-checking datasets based on our observations in section 3 and fc-nlp's reliance on global counter-evidence, we hypothesize that evidence in existing fact-checking datasets does not fully satisfy real-world demands. we, hence, investigate how fcnlp's assumptions affect fact-checking datasets and if they constitute realistic test beds for real-world misinformation. for real-world scenarios, datasets must contain realworld misinformation claims and realistic counterevidence. for evidence, we define the following two requirements: the issue of leaked evidence was also mentioned very recently by @xcite . unlike us, they did not comprehensively analyze existing datasets, evaluate the impact on trained systems (section 5), or consider the complementary criterion of sufficient (counter-) evidence. relying on leaked evidence is related to the important yet different task of detecting already-verified claims @xcite , but is unrealistic for novel claims. we survey nlp fact-checking datasets with natural input claims foot_7 that assume access to trusted evidence. we find that no dataset with real-world misinformation (7-16) satisfies both evidence criteria. we identify four categories: first, datasets that consider (parts of) a fact-checking article as evidence contain sufficient, yet leaked evidence @xcite . second, annotators estimate claim veracity based on evidence such as wikipedia or scientific publications. the authors find that evidence often only covers parts of these realistic, complex claims, which yield low annotator agreement (9), or a weakened task definition for stances @xcite . detected via claim & evidence phrase & claim: google earth finds sos from woman stranded on deserted island snippet url evidence snippet: the truth: the story is a hoax. ... google earth finds woman trapped on deserted island for 7 years ... other end "how did you find me" to which they replied "some kid from minnesota found your sos sign on google earth"; from truth or fiction phrase claim: country music singer merle haggard left his entire estate to an lgbt group. evidence snippet: discover ideas about country singers. fake news reports that recently-deceased country music legend merle haggard left his entire estate to an lgbt group; from pinterest articles link to the claim's context and possibly other similar claims (likely supporting). often (e.g. during covid-19 @xcite )), claims are not completely fabricated. fact-checkers identify documents and their interdependence when investigating the claimant's reasoning for the claim (likely not refuting). documents used to disprove the claimant's reasoning may have no or little relevance to the original claim (as in figure 1 ). each step is non-trivial and may rely on numerous documents (or expert statements). relying on premise evidence without considering the verification process and how these documents relate, is insufficient. both other datasets (12 & 13) in this category provide no annotations and are limited to freely available evidence documents (as opposed to paywalled web pages or e-mails). fourth is using a search engine during dataset construction to expand the accessible knowledge. even when excluding search results that point to the claim's fact-checking article, leaked evidence persists: different organizations may verify the same claims, or disseminate the fact-checkers verification. only @xcite provide stance annotation for arabic claim and evidence pairs. for false claims, they found that only a few documents disagree, and more agree, with the claim. a possible explanation is that misinformation often emerges when trusted information or counterevidence is scarce. fact-checking articles fill this deficit. partially excluding them during dataset generation reduces the found counter-evidence. lacking counter-evidence is not a problem of the dataset generation, but the underlying nature of misinformation, and should be considered by the task definition. we rate evidence in this category (14-16) leaked and insufficient, and back it up in section 5. ## a case study of leaked evidence we view multifc @xcite , the largest dataset of its group, as an instantiation of fcnlp applied to the real world: it contains realworld claims and professionally assessed verdicts as labels from 26 fact-checking organizations (like snopes or politifact). the authors use the google search engine to expand evidence retrieval to the real world during dataset construction. we abstract from the fact that multifc only provides incomplete evidence snippets and consider (if possible) the underlying article in its entirety. ## conclusion in this work, we contrasted nlp fact-checking approaches with how professional fact-checkers combat misinformation. we identified that reliance on counter-evidence hinders current fact-checking systems to refute real-world misinformation. using multifc we find that most evidence is insufficient, or leaked and exploited by trained models. moving forward, we suggest to align nlp approaches with the human verification process, and task definitions with smaller and well-defined verification strategies.