Check- COVID : Fact-Checking COVID -19 News Claims with Scientific Evidence (2023)

Abstract: We present a new fact-checking benchmark, Check-COVID, that requires systems to verify claims about COVID-19 from news using evidence from scientific articles. This approach to fact-checking is particularly challenging as it requires checking internet text written in everyday language against evidence from journal articles written in formal academic language. Check-COVID contains 1, 504 expert-annotated news claims about the coronavirus paired with sentence-level evidence from scientific journal articles and veracity labels. It includes both extracted (journalist-written) and composed (annotator-written) claims. Experiments using both a fact-checking specific system and GPT-3.5, which respectively achieve F1 scores of 76.99 and 69.90 on this task, reveal the difficulty of automatically fact-checking both claim types and the importance of in-domain data for good performance. Our data and models are released publicly at https://github.com/posuer/Check-COVID .

## introduction throughout the covid-19 pandemic, misinformation on the internet has proven to be exceptionally dangerous, undercutting containment efforts by public health officials around the world @xcite . in future pandemics, the ability to automatically detect and debunk such misinformation has the potential to save many lives. however, an automated system cannot rely solely on surface forms or linguistic features to identify misinformation @xcite . such a system must be capable of checking claims written in everyday language against jargon-laden evidence from an evolving set of scientific articles. in this work, we formalize this challenge as a new fact-checking benchmark that requires verifying claims about covid-19 from news using evidence from scientific articles. this task is particularly challenging as it requires grounding everyday claim (extracted) a relatively insensitive test, rolled out twice a week, vastly outperformed a more accurate test, administered once every two weeks, in curbing the spread of disease. ## related work there has been tremendous interest in developing fact-checking benchmarks for covid-19, however many of these datasets only contain claims @xcite ) or claims with veracity labels and no evidence @xcite ; elhadad et al. (2020); shahi and nandini (2020); cui and lee (2020); vijjali et al. (2020); alam et al. (2020) ). others primarily source claims from social media @xcite al. (2021); mohr et al. (2022); sundriyal et al. (2022) ) or scientific texts @xcite ). some datasets, like ours, source claims from colloquial news sources (or other online sources) @xcite ; @xcite ; @xcite ) but of these @xcite and @xcite have just a few hundred examples, a challenge for use in training deep learning systems. @xcite is the most similar to our dataset, however, our evidence comes from citations in the news articles the claims are drawn from, whereas in @xcite the evidence is retrieved using the extracted claims as search queries. thus, our claim-evidence pairings are closer to those the average reader encounters online. additionally we include a third veracity label (notenoughinfo) which some of the above datasets -and many general fact-checking corpora -do not. in the wild, a system will not always be able to fact-check a claim. thus, modeling these cases is critical to real-world performance. ## check-covid dataset we introduce check-covid, a dataset of 1, 504 claims drawn from news paired with sentence-level evidence from scientific journal articles and accompanying veracity labels: {support, refute, notenoughinfo}. the number of examples across the three labels is balanced (505, 504, 495). the claims are categorized into composed or extracted based on whether they are written by our annotators or drawn from news articles. we also provide a corpus containing the abstracts of the journal articles from which the evidence is drawn, a subset of the cord-19 corpus. we randomly split the dataset into three balanced subsets with no overlapping abstracts: train (70%), dev (15%) and test (15%). ## check-covid task and challenges given a claim about covid-19, either written by an annotator or drawn from a news article, the task is to retrieve related scientific abstracts from cord-19, choose the most relevant evidence sentences from those abstracts and decide whether that evidence supports, refutes, or does not provide enough information about the claim. this formulation matches the scifact task format. the claims in our corpus pose many unique challenges: medical knowledge and terminology many examples in our corpus require knowledge of medicine and its terminology to perform successful fact-checking. to fact-check the first example in figure 1 , a model must recognize that the alveolar region is where blood exchanges oxygen and carbon dioxide foot_2 . this implies that here, the virus can enter someone's bloodstream and infect them. ## baseline system our fact-checking pipeline is adapted from the systems presented in deyoung et al. (2019), wadden et al. (2020), and wang et al. (2021) , whose use is consistent with their intended use. it consists of three modules, trained in a supervised fashion on different combinations of existing fact-checking corpora and check-covid. we describe the individual components of this baseline system below. abstract retrieval we query for covid-19 scientific literature through a cord-19 search engine, vespa foot_3 . given a claim, vespa retrieves relevant ab-stracts from cord-19 and ranks them with the bm25 scoring function @xcite . by integrating vespa instead of using a retrieval model with a fixed corpus, we enable the pipeline to retrieve evidence from a corpus of covid-19 scientific articles that could be continually updated. the three abstracts with the highest bm25 scores are passed to the next step of rationale selection. rationale selection we fine-tune a roberta @xcite model with transformers @xcite to predict whether each sentence e i from the selected abstracts is relevant to the given claim c, where i refers to the index within the abstract. we encode each sequence [c sep e i ] and then feed the final [cls] representation to a linear classification layer to predict a binary relevance label. sentences that score over 0.5 are passed as a (possibly empty) set to label prediction. ## experiments to establish a baseline score for check-covid, we first evaluate the dev-set performance of our rationale selection and label prediction modules. we then choose the best-performing model for each to build the full end-to-end pipeline system which we evaluate on the test split. given the material differences between composed and extracted claims, we evaluate training on each claim type separately and in tandem. as vespa could return many relevant abstracts from cord-19 beyond the ones selected for our corpus, we omit evaluation of the vespa-based abstract retrieval module by itself. training datasets to build our rationale selection and label prediction modules, we fine-tune roberta-large models sequentially on fever and/or scifact and then on check-covid. we note that though fever and scifact contain claims and evidence unrelated to covid-19, their inclusion allows us to evaluate the transfer potential of additional task-related data from different domains. for label prediction, we also explore fine-tuning on covid-fact (which we cannot use for ratio- as we only use these corpora for research, it is consistent with their intended use. training setting while training the rationale selection module, we pair each claim with its labeled evidence sentences to produce positive examples from fever, scifact, or check-covid; we create negative examples by pairing claims and non-evidence sentences from the same abstracts that contained the labeled evidence. since notenoughinfo claims are not paired with relevant evidence, we only use support/refute data points for training this module. for the label prediction module, we train on each claim and the concatenation of its gold-label evidence sentences. hyperparameter settings for the rationale and label prediction modules, we use batch sizes of 256 and 16 respectively. we use learning rates of 1e-5 for our roberta encoders and 1e-4 for our linear classifier heads. we train on 4 v100 gpus up to 20 epochs with early stopping (within 6 epochs). 5 ## conclusion we present check-covid, a new corpus for factchecking everyday claims about covid-19 against evidence from scientific journal articles. while our experimental results establish a strong baseline, they also demonstrate the difficulty in transferring learning from existing fact-checking corpora to this new dataset. in future work, we plan to train a unified model to perform rationale selection and label prediction, mitigating error propagation.