Zero-shot Scientific Claim Verification Using LLM s and Citation Text (2024)

Abstract: Due to rapidly changing and advancing science, it is important to check the veracity of scientific claims and whether they are supported by research evidence. Previous versions of this task depended on supervised training, where labeled datasets were constructed through manual claim writing and evidence identification, sometimes coupled with mining citation relationships in papers. In this work, we investigate whether zero-shot scientific claim verification could be enabled using large language models (LLMs) and distant supervision examples taken directly from citation texts. We derive an in-context learning (ICL) dataset, SCitance, consisting of citation sentences (“citances”), LLM-generated negations, evidence documents, and veracity labels, and find that prompting GPT-4 with ICL examples from this dataset yields comparable performance (within 1 point F1) to previous finetuned models trained on manually curated claim-evidence pairs. Our results suggest that prompting LLMs with citance-evidence pairs directly poses a viable alternative to finetuning scientific claim verification models with manually-curated data.

## introduction verifying scientific claims is important for assessing the rigor of the research enterprise and for addressing concerns such as misinformation or misinterpretation of scientific output. prior work in scientific claim verification relies on manually annotated supervision data (expert-written claims verified against documents) @xcite @xcite , which can be expensive to curate and difficult to scale. this has inspired work in zero-or few-shot settings, which have demonstrated success by extracting claims using supervised models, then training claim verification models on the extracted claims @xcite . here, we investigate two questions: (i) whether expert-written claims are needed at all, and (ii) whether large language models (llms) can verify scientific claims in zero-or few-shot settings with no need for supervision labels or model finetuning. towards (i), we investigate whether citation sentences ("citances") and citing relationships from the scientific literature could be used directly as noisy labeled data, without converting citances to claims. we construct a dataset of citance-evidence pairs, scitance, based on the claim-evidence pairs from scifact @xcite , and use these as in-context learning (icl) examples for prompting. to derive contradictory examples, we use gpt-3.5 to generate negations of citances. towards (ii), we design zero-and few-shot prompts for claim verification using gpt-3.5 and gpt-4, and find that gpt-4 with icl performs comparably at abstractlevel verification compared to supervised models trained on expert-annotated claim-evidence pairs (within 1 point f1). this result indicates that contemporary llms could be adapted to perform scientific claim verification with few supervision labels, which could dramatically lower the costs associated with domain transfer of these models. tific claims using citation contexts from existing literature directly. since the emergence of llms, many works have explored prompting for fact/claim verification. zhang and gao (2023) experimented with icl for news claim verification, finding that prompting with a few examples achieves performance comparable to that of supervised models. they and others @xcite also demonstrated the ability of llms to support explainable claim verification, where rationales are provided alongside veracity judgements or grounded to knowledgebases. llms have also been used to coordinate fact-checking of complex claims @xcite , which are similar to citances in scientific text. ## dataset we derive our dataset, scitance, from scifact @xcite , a dataset of 1.4k manuallywritten scientific claims from citances in biomedical papers verified against over 5000 evidence documents. for each claim-evidence pair, trained annotators provided labels for whether the evidence fold support refute nei all train 178 155 134 467 dev 38 31 29 98 test 35 39 17 91 all 251 225 180 656 we keep all citance-evidence pairs from the train and dev set with support and nei labels, yielding 251 supporting and 180 nei entries. these numbers are notably lower than scifact's due to: (i) in scifact, multiple claims could be written based on a single citance; and (ii) each claim from the same citance with the same supporting evidence document translates to only one instance in our dataset. this also carries over to nei examples. ## experiments we adopt the abstract-level scientific claim verification task definition from @xcite . given a claim and retrieved evidence abstract, the goal is to determine whether the evidence supports, refutes, or offers insufficient information towards 3 in rare cases, entire clauses are removed in the negation. the claim. here, instead of only claims, we provide either citances or claims as our verification objection. we report performance on scitance and scifact (via the public leaderboard foot_2 ). ## results performance on scitance model performance on scitance is provided in performance on scifact on the scifact test set, we see comparable performance between few-shot prompting with gpt-4 and multivers ( ## discussion & conclusion this study explores zero-shot scientific claim verification using llms and citation texts directly, demonstrating effective transfer to verifying scientific claims. gpt-4, when prompted with icl examples from scitance, performs within 1-pt f1 of finetuned models that rely on manually curated claim-evidence pairs. thus, using citation sentences directly as noisy labeled data and prompting llms to produce small numbers of negations and counter-examples can serve as a potential alternative to manual data curation. such datasets would be much easier to create in a novel scientific domain due to the common occurrence of citation relationships in scientific literature that could be used to bootstrap annotations. however, there are limitations to our approach. negating or generating variants of complex sentences while preserving logical internal relationships remains a challenge. our work also does not contend with explainability, an important facet of claim verification that has received ample attention in recent years. while we did not conduct experiments to this effect, prior efforts in news and general domain claim verification @xcite suggest that llms excel at rationalizing verification decisions, which could also be tested in the scientific domain. scitance was based on claim-evidence relationships validated by annotators for scifact, so they are not truly devoid of any manual curation. followup work should investigate whether a dataset like scitance could be constructed directly from citation relationships from independent sets of scientific documents. this would demonstrate the feasibility of citation-based dataset construction to support task transfer to other scientific disciplines.