COVID -Fact: Fact Extraction and Verification of Real-World Claims on COVID -19 Pandemic (2021)

Abstract: We introduce a FEVER-like dataset COVID-Fact of 4,086 claims concerning the COVID-19 pandemic. The dataset contains claims, evidence for the claims, and contradictory claims refuted by the evidence. Unlike previous approaches, we automatically detect true claims and their source articles and then generate counter-claims using automatic methods rather than employing human annotators. Along with our constructed resource, we formally present the task of identifying relevant evidence for the claims and verifying whether the evidence refutes or supports a given claim. In addition to scientific claims, our data contains simplified general claims from media sources, making it better suited for detecting general misinformation regarding COVID-19. Our experiments indicate that COVID-Fact will provide a challenging testbed for the development of new systems and our approach will reduce the costs of building domain-specific datasets for detecting misinformation.

## introduction the proliferation of disinformation and misinformation on the web is increasing at a scale that calls for the automation of the slow and laborintensive manual fact-checking process @xcite . new york times reports that "physicians say they regularly treat people more inclined to believe what they read on facebook than what a medical professional tells them." disinformation is even more acute around the recent covid-19 pandemic. as a result, there is a need for automated fact-checking tools to assist professional fact-checkers and the public in evaluating the veracity of claims that are propagated online in news articles or social media. ideally, a fact-checking pipeline will address several tasks: 1) consider real-world claims, 2) retrieve relevant documents not bounded to a known document collection (e.g., wikipedia) and which contain information to validate the claim, 3) select evidence sentences that can support or refute the claim and 4) predict the claim veracity based on this evidence. recent work on end-to-end factchecking, including models and datasets, has advanced the field by addressing several tasks in the pipeline, but not all @xcite @xcite @xcite . one line of work that includes fever @xcite and scifact @xcite addresses tasks 2, 3 and 4, but assumes a given document collection for task 2 (wikipedia or cord-19, respectively) and does not address task 1. moreover, the refuted claims in these datasets are manually generated by asking humans to produce counter-claims for a given claim supported by a source document. another line of work that includes multi-fc @xcite addresses tasks 1, 2 and 4, but not 3. it provides real-world claims collected from fact-checking websites and evidence documents and other metainformation, but it does not provide evidence sentences. we propose a novel semi-automatic method to build a fact-checking dataset for covid-19 (covid-fact) with the goal of facilitating all the above tasks. we make the dataset and code available for future research at https://github.com/ asaakyan/covidfact . our contributions are as original claim closed environments facilitate secondary transmission of coronavirus disease 2019 counter-claim closed environments prevent secondary transmission of coronavirus disease 2019 gold document https://www.medrxiv.org/content/10.1101/2020.02.28.20029272v2 ## covid-fact dataset construction the covid-fact dataset contains 4, 086 real-world claims with the corresponding evidence documents and evidence sentences to support or refute the claims. there are 1, 296 supported claims and 2, 790 automatically generated refuted claims. in this section, we present the three main steps to semiautomatically construct this dataset: 1) real-world true claim and trustworthy evidence document selection (section 2.1), 2) automatic counter-claim generation (section 2.2) and 3) evidence sentence selection (section 2.3). ## experimental setup ## results and analysis error analysis. we observe that errors in veracity prediction can be attributed to three factors: cause and effect, commonsense or scientific background. for instance, in the first (c1, ev1) pair in on statements fact-checked by journalists or organizations such as politifact.org @xcite , news articles (pomerleau and rao, 2017), or answers in community forums @xcite . mixeddomain large scale datasets such as ukp snopes @xcite , multifc @xcite , and fever @xcite rely on wikipedia and fact-checking websites to obtain evidences for their claims. even though these datasets contain many claims, due do domain mismatch they may be difficult to apply for covid-19 related misinformation detection. scifact @xcite introduced the task of scientific fact-checking, generating a dataset of 1.4k scientific claims and corresponding evidences from paper abstracts annotated by experts. however, the dataset does not contain simplified scientific claims encountered in news and social media sources, making it difficult to optimize for a misinformation detection objective. another approach to misinformation detection similar to ours is climate-fever @xcite . they adapted fever methodology to create a dataset specific to climate change factchecking. however, due to difficult and expensive methods employed for generation of fever, it can be difficult to extrapolate this method to assemble a covid-19 specific dataset. covid-19 related nlp tasks. numerous nlp approaches were employed to aid the battle with the covid-19 pandemic. notably wang et al. (2020) released cord-19, a dataset containing 140k papers about covid-19 and related topics while zhang et al. (2020a) created a neural search engine covidex for information retrieval. to combat misinformation lee et al. (2020) proposed a hypothesis that misinformation has high perplexity. hossain et al. (2020) released covidlies: a dataset of 6761 expert-annotated tweets matched with their stance on known covid-19 misconceptions. the dataset provides a comprehensive evaluation of misconception retrieval but does not analyze evidence retrieval and prediction of veracity of claims based on presented evidence. poliak et al. (2020) collected 24,000 question with expert answers from 40 trusted websites to help nlp research with covid related information. covid-fact, on the other hand, deals with real world claims and presents an end-to-end fact checking system to fight misinformation. ## conclusion we release a dataset of 4,086 claims concerning the covid-19 pandemic, together with supporting and refuting evidence. the dataset contains real-world true claims obtained from the r/covid19 subreddit as well as automatically generated counterclaims. our experiments reveal that our dataset outperforms zero-shot baselines trained on popular fact-checking benchmarks like scifact and fever. this goes on to prove how domain-specific vocabulary may negatively impact the performance of popular nlp benchmarks. finally, we demonstrate a simple, scalable, and cost-efficient way to automatically generate counter-claims, thereby aiding in creation of domain-specific fact-checking datasets. we provide a detailed evaluation of the covid-fact task and hope that our dataset serves as a challenging testbed for end-to-end fact-checking around covid-19.