S ci F act-Open: Towards open-domain scientific claim verification (2022)

Abstract: While research on scientific claim verification has led to the development of powerful systems that appear to approach human performance, these approaches have yet to be tested in a realistic setting against large corpora of scientific literature. Moving to this open-domain evaluation setting, however, poses unique challenges; in particular, it is infeasible to exhaustively annotate all evidence documents. In this work, we present SciFact-Open, a new test collection designed to evaluate the performance of scientific claim verification systems on a corpus of 500K research abstracts. Drawing upon pooling techniques from information retrieval, we collect evidence for scientific claims by pooling and annotating the top predictions of four state-of-the-art scientific claim verification models. We find that systems developed on smaller corpora struggle to generalize to SciFact-Open, exhibiting performance drops of at least 15 F1. In addition, analysis of the evidence in SciFact-Open reveals interesting phenomena likely to appear when claim verification systems are deployed in practice, e.g., cases where the evidence supports only a special case of the claim. Our dataset is available at https://github.com/dwadden/scifact-open.

## introduction the task of scientific claim verification @xcite ) aims to help system users assess the veracity of a scientific claim relative to a corpus of research literature. most existing work and available datasets focus on verifying claims against a much more limited context-for instance, a single article or text snippet @xcite @xcite or a small, artificiallyconstructed collection of documents @xcite . current state-of-the-art models are able to achieve very strong performance on these datasets, in some cases approaching human agreement @xcite . ## background and task overview we review the scientific claim verification task, and summarize the data collection process and modeling approaches for scifact, which we build upon in this work. we elect to use the scifact dataset as our starting point because of the diversity of claims in the dataset and the availability of a number of state-of-the-art models that can be used for pooled data collection. in the following, we refer to the original scifact dataset as scifact-orig. ## the scifact-open dataset in this section, we describe the construction of scifact-open. we report the performance of claim verification models on scifact-open in §4, and perform reliability checks on the results in §5. our goal is to construct a test collection which can be used to assess the performance of claim verification systems deployed on a large corpus for our corpus, we filter the s2orc dataset @xcite for all articles which (1) cover topics related to medicine or biology and (2) have at least one inbound and one outbound citation. from the roughly 6.5 million articles that pass these filters, we randomly sample 500k articles to form the corpus for scifact-open, making sure to include the 5k abstracts from scifact-orig. we choose to limit the corpus to 500k abstracts to ensure that we can achieve sufficient annotation coverage of the available evidence. additional details on corpus construction can be found in appendix a. unlike scifact-orig (which is skewed toward highly-cited articles from "high-impact" journals), we do not impose any additional quality filters on articles included in scifact-open; thus, our corpus captures the full diversity of information likely to be encountered when scientific fact-checking systems are deployed on real-world resources like s2orc, arxiv, 3 or pubmed central. 4 ## dataset reliability the total number of annotations collected during pooling ( §3.1) is determined by two parameters: the number of annotations per system d, and the number of systems n for which we collect annotations. these parameters must be large enough that increasing them further is unlikely to ( 1 ) lead to the discovery of a large number of additional ecaps or (2) alter the performance metrics of models evaluated on the dataset. following zobel (1998), we conduct checks to ensure that conditions (1) and ## discussion & conclusion in this work, we introduced a new test collection, scifact-open, to support performance evaluation for open-domain scientific claim verification. the construction of scifact-open was enabled by our adaptation of the pooling strategy from ir for identification and annotation of evidence from a corpus of 500k documents. we hope such methodology can see further usage on other nlp tasks for which exhaustive annotation is infeasible. in analyzing the evidence in scifact-open ( §3.3), we found that some claims possess a large amount of conflicting evidence, and that evidence may not always match the specificity of the claims as written. we consider two future directions to improve the expressiveness of scientific claim verification. (1) as discussed in §3.3, one could still require systems to label each ecap, but also to generate a revised claim matching the specificity of each evidence abstract. this output would provide users with fine-grained information indicating the conditions under which an input claim is likely to hold. we release 91 claim revisions, which can be used to facilitate exploratory research in this direction. overall, our analysis indicates that evaluations using scifact-open can provide key insights into modeling challenges associated with scientific claim verification. in particular, the performance of existing models declines substantially when evaluated on scifact-open, suggesting that current claim verification systems are not yet ready for deployment at scale. it is our hope that the dataset and analyses presented in this work will facilitate future modeling improvements, and lead to substantial new understanding of the scientific claim verification task.