Generating Scientific Claims for Zero-Shot Scientific Fact Checking (2022)

Abstract: Automated scientific fact checking is difficult due to the complexity of scientific language and a lack of significant amounts of training data, as annotation requires domain expertise. To address this challenge, we propose scientific claim generation, the task of generating one or more atomic and verifiable claims from scientific sentences, and demonstrate its usefulness in zero-shot fact checking for biomedical claims. We propose CLAIMGEN-BART, a new supervised method for generating claims supported by the literature, as well as KBIN, a novel method for generating claim negations. Additionally, we adapt an existing unsupervised entity-centric method of claim generation to biomedical claims, which we call CLAIMGEN-ENTITY. Experiments on zero-shot fact checking demonstrate that both CLAIMGEN-ENTITY and CLAIMGEN-BART, coupled with KBIN, achieve up to 90% performance of fully supervised models trained on manually annotated claims and evidence. A rigorous evaluation study demonstrates significant improvement in generated claim and negation quality over existing baselines

## introduction scientific documents contain complex assertions about scientific processes, making it difficult to automate important tasks such as claim extraction and scientific fact checking. additionally, the collection of manually annotated labels to train models on tasks with scientific data is time consuming and expensive due to the need for domain expertise @xcite @xcite @xcite . as such, methods which require less manual annotation are especially useful in this domain. this work addresses this challenge by exploring how automatic generation of scientific claims can assist with dataset creation and zero-shot fact checking in the biomedical domain. being able to reduce scientific text to atomic assertions has numerous possible applications, and is known to be helpful for scientific communication and machine processing of scientific concepts @xcite . claim generation can enable zero-shot fact checking, reducing the need for expert-labeled data @xcite , and can be used to expand existing datasets such as @xcite and @xcite without additional manual annotation. in this work we focus on the use of claim generation in scientific fact checking, demonstrating that claim generation enables zero-shot biomedical fact checking. generating scientific claims involves distilling a complex scientific sentence into one or more valid claims (see examples in figure 1 ). as in previous work, we focus on biomedical claims as biomedical literature has long been a major focus in scientific natural language processing, as well as scientific fact checking @xcite @xcite . while in @xcite , claims were rewritten by domain experts from complex citation sentences (citances), we propose methods for automatically generating claims and claim negations from this source. similar to other generation tasks, evaluating the quality of generated output requires multiple judgements beyond the fluency of the generated text, e.g., whether each claim is faithful to the source sentence, and is understandable on its own @xcite . however, there are also other quality attributes that are important to assess specifically for scientific claims, such as whether each claim is atomic or check-worthy @xcite . given this, we propose a set of manual evaluation criteria and annotation guidelines for evaluating claim generation ( §5.2). additionally, when generating claims to build datasets for tasks such as fact checking, a major challenge is creating refuted claims as negative training instances. previous work has proposed automatic ways of generating refutations based on negating existing claims or creating claim variants via entity-replacement @xcite ) and text-infilling using a pre-trained masked language model @xcite . we improve upon this by introducing knowledge base informed negations (kbin), a principled method to generate refutations that performs entity-replacement using the relations and learned embeddings of entities in a domain-specific knowledge base. contributions in sum, our contributions are: • the first study on scientific claim generation, comparing both unsupervised (claimgen-entity) and fully supervised (claimgen-bart) generation on biomedical text. • kbin, a novel method for generating refuted scientific claims which produces more convincing negations than previous work. • application of our claim generation methods on zero-shot scientific fact checking resulting in 90% of the performance of a model trained on in-domain manually written claims. additionally, a rigorous evaluation study showing that claimgen-bart and kbin produce significantly higher quality claims and more convincing negations than previous work. ## preliminaries valid claims in this work, we define a valid claim as one which is fluent, atomic, decontextualized, and accurately reflects the meaning of the original sentence. fluency is concerned with a claim being a generally well-formed english sentence, and atomicity with a claim being a "verifiable statement expressing a finding about one aspect of a scientific entity or process, which can be verified from a single source" @xcite . de-contextualilzation is concerned with a sentence being interpretable on its own, requiring none of the original surrounding text to resolve aspects of the sentence such as pronouns, abbreviations, etc., and can be handled by either directly de-contextualizing a sentence @xcite or by ensuring that all of the context sentences are available to a model @xcite . checkworthy claims in the wild may not be fluent, atomic, or de-contextualized, however it is useful to generate such claims as they have been shown to be useful for automated processing of science concepts @xcite and scientific fact checking @xcite . ## generating supported claims we experiment with two generation methods designed to produce claims which are supported by the source sentence. the first method is an entitycentric unsupervised method adapted from pan et al. (2021) which requires @xmath0, @xmath1. we also introduce a new exergames improve function and reduce the risk of falls. umls c0184511 'c0426422' 'c1998348' 'c0024103' 'c0043094' 'c1457868' 'c1827505' 'c1719838' c1457868 t033: "finding" exergames worse function and reduce the risk of falls. exergames deteriorating function and reduce the risk of falls. exergames worsened function and reduce the risk of falls. ## experiments we investigate three primary research questions: rq1 do automatically generated claims enable zero-shot scientific fact checking? rq2 what is the percentage of high-quality claims generated using our methods? rq3 how does kbin compare with previous work for claim negation in terms of generating contradictions? for rq1, we use claimgen-entity and claimgen-bart generated claims to train a fact checking model, evaluating on the scifact dataset @xcite and comparing to relevant baselines. to answer rq2 and rq3, we design annotation criteria and perform manual evaluations with a group of expert annotators (details in §5.2). ## related work scientific fact checking our work follows a line of recent literature on scientific fact checking @xcite . the goal of this task is to determine the veracity of claims related to scientific topics by retrieving appropriate documents from scientific literature, finding evidentiary sentences from those documents, and determining whether claims are supported, refuted, or there is not enough evidence to make a judgement. the task closely resembles the task of general domain factchecking @xcite . well-performing systems on this task use large language models to perform neural document retrieval @xcite or multi-task learning of rationale prediction and stance prediction @xcite . recent work on general domain fact checking has also introduced methods for adversarial generation of claims which are particularly difficult to fact-check @xcite , and for performing the task without any labeled data @xcite . our proposed methods extend zero-shot fact checking to the scientific domain, demonstrating that one can achieve 90% of the inference performance of state-of-the-art systems without domain-specific labeled data. generating training data our work is also related to methods for the automatic generation of training data. generation of synthetic data has been used for multiple tasks, for example question answering @xcite , knowledge-base completion @xcite , and fact-checking @xcite . most similar to our setting, the covid-fact dataset @xcite contains claims related to covid-19 crawled from reddit, and is constructed semiautomatically. claims which are supported by evidence are extracted from reddit and verified by human annotators, while negations of these claims are generated automatically via masked language model infilling. kbin improves upon the negation method proposed in this work by leveraging in-domain structured knowledge via umls. ## conclusion in this work, we propose the task of scientific claim generation, presenting claimgen-bart, claimgen-entity, and kbin to perform the task. we demonstrate that generated claims can be used to train a model for zero-shot scientific fact checking and obtain within 90% of the performance of a model trained on human-written claims. through a rigorous user study we demonstrate that claimgen-bart produces higher quality claims than claimgen-entity, and that kbin produces more fluent and more convincing negations than previous work. work remains to improve claim generation quality and assess the impacts of generated claims in other domains of science, as well as how generated claims can be used in the evidence retrieval component of fact checking systems. we hope that our methods will be used to facilitate future work by enabling faster creation of training datasets and improving the performance of models on the timely and important task of scientific fact checking. curie grant agreement no 801199, and by the united states national science foundation grant oia-2033558. we thank doug downey, hannaneh hajishirzi, the reviewers, and members of the semantic scholar research team for their valuable feedback.