W at C laim C heck: A new Dataset for Claim Entailment and Inference (2022)

Abstract: We contribute a new dataset for the task of automated fact checking and an evaluation of state of the art algorithms. The dataset includes claims (from speeches, interviews, social media and news articles), review articles published by professional fact checkers and premise articles used by those professional fact checkers to support their review and verify the veracity of the claims. An important challenge in the use of premise articles is the identification of relevant passages that will help to infer the veracity of a claim. We show that transferring a dense passage retrieval model trained with review articles improves the retrieval quality of passages in premise articles. We report results for the prediction of claim veracity by inference from premise articles.

## introduction the rise of social media has led to a democratization of news, but it has also amplified issues related to fake news and misinformation. to that effect, many fact checking organizations (e.g., politifact, snopes, afp fact check, alt news, factcheck.org, africa check, etc.) have emerged around the globe. they investigate debatable claims made by authorities, politicians, celebrities and the public. for each claim, they publish a review article with links to sources that support a verdict (e.g., true, partly true/false, false) about the veracity of the claim. those reviews debunk false claims and mitigate the spread of misinformation. we consider a key nlp challenge in the context of automated fact checking: claim inference from premise articles. note that determining the veracity of a claim without additional information is nearly impossible since claims are selected by professional fact checkers in part because their veracity is far from obvious and also because of their degree of controversy. to that effect, professional fact checkers invest a fair amount of time to research each claim by finding relevant sources and publishing a review article that explains their verdict of the claim. hence there is a natural entailment problem, whereby anyone who reads a review article should be able to arrive at the same verdict as the professional fact checker regarding the claim. unlike many entailment tasks that consist of short text (e.g., pairs of utterances) that may be artificially generated or extracted, this is a natural and challenging entailment task that involves an entire document (review article) with an utterance (claim) that requires a certain degree of reading comprehension. we note that this entailment problem has been tackled in some previous work @xcite @xcite and although it is a challenging nlp problem, it does not correspond to the problem that professional fact checkers need to solve. in this paper, we focus on the harder problem of claim inference from premise articles. this is part of the challenge that professional fact checkers face. they find premise articles that contain relevant facts and then infer the veracity of the claim based on those facts. unlike many existing inference tasks where it is sufficient to use one or a few facts in a few sentences @xcite , information from a set of premise articles must be distilled and combined in non trivial ways to infer the veracity of a claim. we assembled a dataset of 33,697 claims made between december 1996 and july 2021 with associated review articles, premise articles and claim verdicts. many other datasets for claim verification are listed in since there are several premise articles for a given claim and each premise article may be long, a simple two-stage approach to identify relevant passages would consist of a lightweight retrieval technique in a first stage, followed by a heavyweight inference technique applied to those passages. when the first stage fails to retrieve key passages, then the inferred verdict will be negatively affected regardless of how good the second stage is. to that effect, several supervised dense passage retrieval techniques have been proposed for question-answering @xcite @xcite . unfortunately, we cannot directly apply those techniques since we do not have labels for the relevant passages. instead, we show how to use the review articles to train a supervised dense retrieval technique that is then transferred to premise articles. the contributions of the paper can be summarized as follows: the paper is organized as follows. sect. 2 reviews previous work related to automated fact checking and claim verification. sect. 3 describes the new dataset and summarizes the differences with previous datasets for claim verification. sect. 4 describes a two-stage process to i) extract evidence sentences from premise articles and ii) infer the veracity of claims. this section also explains how to transfer a dense passage retrieval technique trained with review articles to premise articles. sect. 5 reports the results for the claim veracity inference task. finally, sect. 6 concludes and discusses possible future work. ## related work there is an important line of work that focuses on claim verification @xcite . this includes techniques that predict the veracity of a claim based on the text of the claim only @xcite , linguistic features @xcite , meta information about the claimant (e.g., name, job, party affiliation, veracity history) @xcite , review articles @xcite @xcite , relevant articles returned by a search engine @xcite @xcite as well as premise articles @xcite . there is an important distinction between articles returned by a search engine and premise articles. the techniques that use a search engine to find articles related to a claim query the search engine after a fact checking website has published a review article and therefore end up retrieving articles that include the review article as well as other articles that summarize and/or discuss the verdict of the fact checking website. hence they are tackling an entailment problem. in contrast, the premise articles that we consider are the source articles used by a fact checker before publishing a review article. those articles contain relevant facts, but not a summary or discussion of the review article since they are published before the review article and in fact serve as premises for the review article. closely related to claim verification is the problem of fake news detection. in this problem, the credibility of an entire news article is evaluated. the credibility can be estimated based on linguistic and textual features @xcite @xcite , discourse level structure @xcite , network analysis @xcite , knowledge graphs @xcite , inter-user behaviour dynamics @xcite or a combination of multiple modalities @xcite . some techniques reorder the articles returned by a search engine based on their degree of credibility @xcite . an important task that can help the detection of fake news is stance detection @xcite , i.e., does the content of an article agree or disagree with the title of the article? the following surveys summarize advances in fake news detection: @xcite . ## models we develop a two-stage system to perform evidence based veracity detection. the first stage selects relevant sentence level evidence from the premise articles associated with a claim and the second stage performs claim veracity inference using the claim text and selected evidence sentences. for the first stage, we evaluate two different approaches. the first approach is term frequency inverse document frequency (tf-idf), which is typically used by fact checking methods for sentence based retrieval @xcite . for the second approach, we propose a novel way to adapt dense passage retrieval techniques using the review articles for evidence sentence selection. in our experiments, the aforementioned dense passage retrieval technique outperforms tf-idf text retrieval and leads to overall system performance improvements. the second stage consists of training deep learning models to perform claim veracity inference using the claim text and selected evidence. we utilize multiple deep learning models to perform claim veracity inference ranging from basic bi-directional recurrent networks to state of the art transformers. ## experiments we evaluate the two-stage process and the algorithms described in the previous section on the claim inference problem with our new dataset. ## conclusion this paper introduces a new dataset for automated fact checking. watclaimcheck includes premise articles used by professional fact checkers and therefore corresponds more closely to the task of claim veracity inference in automated fact checking. an important challenge is the extraction of relevant facts from the premise articles since it is not generally possible to apply heavyweight models on the entire content of all premise articles. to that effect, we described how to train the encoders of a dense passage retrieval technique with the review articles and then transfer the resulting retrieval technique to the premise articles. this increased the overall performance of the claim verification algorithms. we also performed a prequential evaluation that highlighted an important distribution shift that caused a significant drop in accuracy for all algorithms. we strongly recommend that future algorithms be evaluated in prequential mode. in fact, an important direction for future research would be to design algorithms based on transfer learning or domain generalization that can cope better with this distributional shift. we also note that the tech-niques that we evaluated are black boxes and therefore it is not clear how they do inference. hence, another direction for future research would be to develop inference techniques that are explainable in the sense that they could provide explanations to the users to justify their veracity prediction for a claim.