Comparing Knowledge Sources for Open-Domain Scientific Claim Verification (2024)

Abstract: The increasing rate at which scientific knowledge is discovered and health claims shared online has highlighted the importance of developing efficient fact-checking systems for scientific claims. The usual setting for this task in the literature assumes that the documents containing the evidence for claims are already provided and annotated or contained in a limited corpus. This renders the systems unrealistic for real-world settings where knowledge sources with potentially millions of documents need to be queried to find relevant evidence. In this paper, we perform an array of experiments to test the performance of open-domain claim verification systems. We test the final verdict prediction of systems on four datasets of biomedical and health claims in different settings. While keeping the pipeline’s evidence selection and verdict prediction parts constant, document retrieval is performed over three common knowledge sources (PubMed, Wikipedia, Google) and using two different information retrieval techniques. We show that PubMed works better with specialized biomedical claims, while Wikipedia is more suited for everyday health concerns. Likewise, BM25 excels in retrieval precision, while semantic search in recall of relevant evidence. We discuss the results, outline frequent retrieval patterns and challenges, and provide promising future directions.

## introduction the fast promulgation of knowledge in the digital world has made keeping track of information trustworthiness a challenging endeavor. in particular, science and health have become popular talking points and brought with it an abundance of medical advice that permeate online resources @xcite . a report by the pew research center @xcite found that over one-third of american adults have searched the internet for medical conditions and asked it medical questions before going to a medical professional. the sought information ranged from self-diagnosis to finding medications. automated solutions for claim verification based on natural language processing (nlp) have emerged as a potential aid to help with bringing light into the information overload @xcite . while most work in the automated factchecking domain is concerned with claims related to politics, society, rumors, and general misinformation, there has been an increasing interest in fact-checking of scientific and biomedical claims @xcite . the task of automated claim verification consists of retrieving evidence for a claim being checked and then predicting a veracity label based on the discovered evidence. the most common setting for this task either already provides the source document that will contain evidence for the claim or works over a limited, manually constructed collection of documents @xcite . while this is an important step in developing models capable of reading comprehension and detecting which spans provide evidence in a given context, this is not a realistic setting for automated claim verification systems deployed in the real world. in such a scenario, the documents containing evidence are not known and knowledge bases containing them can possibly contain millions of documents. moreover, with the rise of medical assistants and conversational agents in healthcare, many users are turning to these systems as a source of health-related information and medical support @xcite . to address these research gaps, we perform an array of experiments that test the performance of nlp systems for claim verification in the open domain. in the experiments, we keep the parts of the verification pipeline concerned with evidence sentence selection and verdict prediction fixed, and vary the knowledge source being used and information retrieval techniques being deployed to query the databases. since the final goal of factverification systems is to provide a verdict on the correctness of a claim, we measure the usefulness of knowledge sources and retrieval techniques by looking at verdict prediction scores. for this purpose, we leverage four english datasets of biomedical and health claims that contain gold annotations stemming from domain experts. we use the veracity labels of claims in datasets as ground truth. we opt for three large-scale knowledge sources: pubmed, the cardinal collection of biomedical research publications; wikipedia, as the largest publicly curated encyclopedia of human knowledge; and google search results (representing "the whole web"), which is a straightforward and intuitive way how users seek information. finally, we perform a qualitative analysis of retrieved evidence for some interesting example claims, present the insights from results, and provide future directions. our contributions are as follows: we make the data and code of the experiments available in a github repository. foot_0 2 foundations ## results for both knowledge sources, the evidence from documents retrieved using semantic search outperformed the standard sparse metric bm25. still, bm25 fares well compared to the relatively recent semantic approaches. it is also notable to observe in when looking at the performance of claim verification systems over wikipedia in at first glance, the performance with google search seems impressive, especially considering that for the two most challenging datasets, sci-fact and pubmedqa, the performance is improved when compared to the first two tables. a more careful look reveals this to be an artefact claim pubmed (semantic) wikipedia (semantic) can regular intake of vitamin c prevent colds? (refuted) nevertheless, given the consistent effect of vitamin c on the duration and severity of colds in the regular supplementation studies, and the low cost and safety, it may be worthwhile for common cold patients to test on an individual basis whether therapeutic vitamin c is beneficial for them. @xcite (supported) according to the most recently published cochrane review on vitamin c and the common cold, one gram per day or more of vitamin c does not influence common cold incidence in the general community, i.e., it does not prevent colds. (en wiki: vitamin c and the common cold) (refuted) can lung cancer screening by computed tomography (ct) also do harm? lung cancer screening with low dose computed tomography (ct) is the only method ever proven to reduce lung cancer-specific mortality in high-risk current and former cigarette smokers. we aim to explain why the risks associated with radiation exposure from lung cancer screening are very low and should not be used to avoid screening or dissuade... @xcite low-dose ct screening has been associated with falsely positive test results which may result in unneeded treatment. in a series of studies assessing the frequence of false positive rates, results reported that rates ranged from 8-49%.(en wiki: lung cancer screening) (supported) can ginkgo extract relieve the symptoms of tinnitus? ginkgo biloba is a plant extract used to alleviate symptoms associated with cognitive deficits, e.g., decreased memory performance, lack of concentration, decreased alertness, tinnitus, and dizziness. pharmacologic studies have shown that the therapeutic effect of ginkgo... @xcite (supported) ginkgo leaf extract is commonly used as a dietary supplement, but there is no scientific evidence that it supports human health or is effective against any disease. systematic reviews have shown there is no evidence for effectiveness of ginkgo in treating high blood pressure, menopause-related cognitive decline, tinnitus, post-stroke recovery, or altitude sickness. (en wiki: gingko bilboa) (refuted) the most prevalent adverse events to semaglutide are gastrointestinal. (supported) we evaluated gastrointestinal (gi) adverse events (aes) with once-weekly semaglutide 2.4 mg in adults with overweight or obesity and their contribution to weight loss (wl). gi aes were more common with semaglutide 2.4 mg than placebo, but typically mild-to-moderate and transient. @xcite (supported) possible side effects include nausea, diarrhea, vomiting, constipation, abdominal pain, headache, fatigue, indigestion/heartburn, dizziness, abdominal distension, belching, hypoglycemia (low blood glucose) in patients with type 2 diabetes, flatulence, gastroenteritis, and gastroesophageal reflux disease (gerd) (en wiki: semaglutide) (refuted) macrolides protect against myocardial infarction. (refuted) our findings indicate that macrolide antibiotics as a group are associated with a significant risk for mi but not for arrhythmia and cardiovascular mortality. @xcite macrolides are a class of natural products that consist of a large macrocyclic lactone ring to which one or more deoxy sugars, usually cladinose and desosamine, may be attached. (en wiki: macrolide) (supported) ## discussion in this section, we provide further insights and a deeper look into the performance of our pipeline for open-domain claim verification of scientific claims in large knowledge sources. we do this with a qualitative analysis where we looked at what kind of documents and sentences are retrieved from dif-ferent knowledge sources with different retrieval techniques and outline some common patterns with representative examples. ## conclusion in this paper, we conducted a number of experiments assessing the performance of a factverification system in an open-domain setting. moving away from the standard setup of a small evidence corpus, we expand the knowledge sources to two large document bases (pubmed and wikipedia), searching the whole web via google search api, and experiment with two retrieval techniques (sparse and dense). we measured the verdict prediction performance over four established factchecking datasets. our results show that searching for evidence in the open domain provides satisfyingly high f1 performance, not far from the closed-domain setting, with a room for further improvement. we conclude that the knowledge source perform comparably, with wikipedia being better for popular and trending claims and pubmed for technical inquiries. we demonstrate the general superiority of dense retrieval techniques, with examples of where it falls short and bm25 retrieval would be beneficial. we hope our research will encourage more exploration of the open-domain setting in the nlp fact-checking community and addressing real-world misinformation scenarios.