Fact or Fiction: Verifying Scientific Claims (2020)

Abstract: We introduce scientific claim verification, a new task to select abstracts from the research literature containing evidence that SUPPORTS or REFUTES a given scientific claim, and to identify rationales justifying each decision. To study this task, we construct SciFact, a dataset of 1.4K expert-written scientific claims paired with evidence-containing abstracts annotated with labels and rationales. We develop baseline models for SciFact, and demonstrate that simple domain adaptation techniques substantially improve performance compared to models trained on Wikipedia or political news. We show that our system is able to verify claims related to COVID-19 by identifying evidence from the CORD-19 corpus. Our experiments indicate that SciFact will provide a challenging testbed for the development of new systems designed to retrieve and reason over corpora containing specialized domain knowledge. Data and code for this new task are publicly available at https://github.com/allenai/scifact . A leaderboard and COVID-19 fact-checking demo are available at https://scifact.apps.allenai.org .

## introduction due to rapid growth in the scientific literature, it is difficult for researchers -and the general public even more so -to stay up to date on the latest findings. this challenge is especially acute during public health crises like the current covid-19 pandemic, due to the extremely fast rate at which new findings are reported and the risks associated with making decisions based on outdated or incomplete information. as a result, there is a need for automated tools to assist researchers and the public in evaluating the veracity of scientific claims. figure 1 : a scientific claim, supported by evidence identified by our system. to correctly verify this claim, the system must possess background knowledge that troponin is a protein found in cardiac muscle and that elevated levels of troponin are a marker of cardiac injury. in addition, it must be able to reason about directional relationships between scientific processes: replacing higher with lower would cause the rationale to refute the claim rather than support it. finally, the system should interpret @xmath0.001 as an indication that the reported finding is statistically significant. fact-checking -a task in which the veracity of an input claim is verified against a corpus of documents that support or refute the claim -has been studied to combat the proliferation of misinformation in political news, social media, and on the web @xcite . however, verifying scientific claims poses new challenges to both dataset construction and effective modeling. while political claims are readily available on fact-checking websites and can be verified by crowd workers, annotators with extensive domain knowledge are required to generate and verify scientific claims. in addition, nlp systems for scientific claim verification must possess additional capabilities beyond those required to verify factoid claims. for instance, to verify the claim shown in figure 1 , a claim 1: lopinavir / ritonavir have exhibited favorable clinical responses when used as a treatment for coronavirus. supports: . . . interestingly, after lopinavir/ritonavir (kaletra, abbvie) was administered, β-coronavirus viral loads significantly decreased and no or little coronavirus titers were observed. refutes: the focused drug repurposing of known approved drugs (such as lopinavir/ritonavir) has been reported failed for curing sars-cov-2 infected patients. it is urgent to generate new chemical entities against this virus . . . claim 2: the coronavirus cannot thrive in warmer climates. supports: ...most outbreaks display a pattern of clustering in relatively cool and dry areas...this is because the environment can mediate human-to-human transmission of sars-cov-2, and unsuitable climates can cause the virus to destabilize quickly... refutes: ...significant cases in the coming months are likely to occur in more humid (warmer) climates, irrespective of the climate-dependence of transmission and that summer temperatures will not substrantially limit pandemic growth. in this paper, we introduce the task of scientific claim verification to evaluate the veracity of scientific claims against a scientific corpus. we establish performance baselines on scifact with an approach similar to deyoung et al. (2020a), which achieves strong performance on the fever claim verification dataset @xcite . our baseline is a pipeline system which retrieves abstracts related to an input claim, uses a bertbased @xcite sentence selector to identify rationale sentences, and labels each abstract as supports, refutes, or noinfo with respect to the claim. we demonstrate that our baseline can benefit from training on claims from domains including wikipedia articles and politics. we showcase the ability of our model to verify expert-written claims concerning the novel coronavirus covid-19 against the newly-released cord-19 corpus @xcite . expert annotators judge retrieved evidence to be plausible for 23 of 36 claims. 1 our results and analyses demonstrate the importance of the new task and dataset to support significant future research in this domain. in summary, our contributions include: (1) we introduce and formalize the scientific claim verification task. (2) we develop a novel annotation protocol to generate and verify 1.4k naturallyoccurring claims about scientific findings. (3) we establish strong baselines on this task, and identify substantial opportunities for improvement at all stages of the modeling pipeline. (4) we demonstrate the efficacy of our system in a real-world case study verifying claims about covid-19 against the research literature. ## background and task definition as illustrated in figure 1 , scientific claim verification is the task of identifying evidence from the research literature that supports or refutes a given scientific claim. claims in scifact are natural -they are derived from citation sentences, or citances @xcite , that occur naturally in scientific articles. this is similar to political fact-checking datasets such as ukp snopes @xcite , which use political fact-checking websites as a source of natural claims. on the other hand, claims in the popular fever dataset @xcite are synthetic, since they are created by annotators by mutating sentences from the wikipedia articles that will serve as evidence. supporting and refuting evidence in most factchecking work, claims are assigned a global truth label based on the entirety of the available evidence. for example in fever, the claim "barack obama was the 44 th president of the united states" can be verified using wikipedia as an evidence source. while scifact claims are indeed verifiable assertions about scientific findings, accurately assigning a global truth label to a scientific claim (given a fixed scientific corpus) requires a systematic review by a team of experts. in this work we focus on the simpler task of assigning supports or refutes relations to individual claim-abstract pairs. each supports or refutes relation between claim and abstract must be justified by at least one rationale. a rationale is a minimal collection of sentences which, taken together as premises in the context of the abstract, can reasonably be judged by a domain expert as implying the claim. rationales facilitate the development of interpretable models which not only have the ability to make label predictions, but can also identify the exact sentences that are necessary for their decisions. ## the scifact dataset the scifact dataset consists of 1,409 scientific claims 3 verified against a corpus of 5,183 abstracts. 2 requiring annotators to search multiple sources increases cognitive burden and decreases annotation quality. ## the scifact task task formulation the inputs to our task are a scientific claim c and a corpus of abstracts a. all abstracts @xmath1, a) ∈ {supports, refutes, noinfo } with respect to a claim c. we denote the set of all rationales as r(c, a) = {r 1 (c, a), . . . , r n (c, a)}. given a claim c and a corpus a, the system must predict a set of evidence abstracts e(c). for each abstract @xmath3, it must predict a label y(c, a), and a collection of rationale sentences s(c, a) = { s 1 (c, a), . . . , s ℓ (c, a)}. note that although the gold annotations may contain multiple separate rationales, to simplify the prediction task we only require the model to predict a single collection of rationale sentences; these sentences may encompass multiple gold rationales. ## verisci: baseline model we develop a baseline (referred to as verisci) that takes a claim c and corpus a as input, identifies evidence abstracts e(c), and predicts a label y(c, a) and rationale sentences s(c, a) for each @xmath4. rationale selection given a claim c and abstract a, we train a model to predict z i 1[a i is a rationale sentence] for each sentence a i in a. for each sentence, we encode the concatenated sequence w @xmath5, sep, c] using a bertstyle language model and predict a score @xmath6, where σ is the sigmoid function, f is a linear layer and cls(w i ) is the cls token from the encoding of w i . we train the model on pairs of claims and their cited abstracts and minimize cross-entropy loss between z i and zi . for each claim, we use cited abstracts labeled noinfo, as well as non-rationale sentences from abstracts labeled supports and refutes as negative examples. to make predictions, we select all sentences a i with @xmath7, where @xmath8, 1] is tuned on the dev set (appendix a.1). label prediction sentences identified by the rationale selector are passed to a separate bertbased model to make the final labeling decision. given a claim c and abstract a, we concatenate the claim and the predicted rationale sentences @xmath9, a), . . . s ℓ (c, a), sep, c] foot_2 , and predict ỹ(c, a) = φ[f (cls(u))], where φ is the softmax function, and f is a linear layer with three outputs representing the {supports, refutes, noinfo } labels. we minimize the cross-entropy loss between ỹ(c, a) and the true label y(c, a). we train the model on pairs of claims and their cited abstracts using gold rationales as input. for cited abstracts labeled noinfo, we choose the k sentences from the cited abstract with highest tf-idf similarity to the claim as input rationales. for prediction, we use the predicted rationale sentences s(c, a) as input and predict ŷ(c, a) = argmax ỹ(c, a). noinfo is predicted for abstracts with no rationale sentences. we experimented with a label prediction model which encodes entire abstracts via the longformer @xcite , and makes predictions using the document-level cls token. performance was not competitive with our pipeline setup, likely because the label predictor struggles to identify relevant information when given full abstracts. ## experiments in our experiments, we (1) analyze the performance of each individual component of verisci, (2) evaluate full task performance in both the "oracle abstract" and "open" settings, (3) present promising results verifying claims about covid-19 using rational-select. label-pred. verisci, and (4) discuss some modeling challenges presented by the dataset. ## conclusion and future work claim verification allows us to trace the sources and measure the veracity of scientific claims. these abilities have emerged as particularly important in the context of the current pandemic, and the broader reproducibility crisis in science. in this article, we formalize the task of scientific claim verification, and release a dataset (scifact) and models (verisci) to support work on this task. our results indicate that it is possible to train models for scientific fact-checking and deploy them with reasonable efficacy on real-world claims related to covid-19. scientific claim verification presents a number of promising avenues for research on models capable of incorporating background information, reasoning about scientific processes, and assessing the strength and provenance of various evidence sources. this last challenge will be especially crucial for future work that seeks to verify scientific claims against sources other than the research literature -for instance, social media and the news. we hope that the resources presented in this paper encourage future research on these important challenges, and help facilitate progress toward the broader goal of scientific document understanding. distractor abstracts in §3.1, we mention how we increase the size of the corpus by adding distractor abstracts. the reason why we do not use the entirety of a large research corpus like s2orc as our fact-checking corpus is that doing so would introduce many false negative retrievals: abstracts containing evidence relevant to a given claim, but not mentioned in the claim's source citance. this can occur either because the citance authors simply were not aware of these abstracts, or because the abstracts were published after the citance was writ-"future studies are also warranted to evaluate the potential association between wnt5a/pcp signaling in adipose tissue and atherosclerotic cvd, given the major role that il-6 signaling plays in this condition as revealed by large mendelian randomization studies 44, 45 ." il-6 signaling plays a major role in atherosclerotic cardiovascular disease.