M ulti2 C laim: Generating Scientific Claims from Multi-Choice Questions for Scientific Fact-Checking (2023)

Abstract: Training machine learning models to successfully perform scientific fact-checking tasks is challenging due to the expertise bottleneck that limits the availability of appropriate training datasets. In this task, models use textual evidence to confirm scientific claims, which requires data that contains extensive domain-expert annotation. Consequently, the number of existing scientific-fact-checking datasets and the sizes of those datasets are limited. However, these limitations do not apply to multiple-choice question datasets because of the necessity of domain exams in the modern education system. As one of the first steps towards addressing the fact-checking dataset scarcity problem in scientific domains, we propose a pipeline for automatically converting multiple-choice questions into fact-checking data, which we call Multi2Claim. By applying the proposed pipeline, we generated two large-scale datasets for scientific-fact-checking tasks: Med-Fact and Gsci-Fact for the medical and general science domains, respectively. These two datasets are among the first examples of large-scale scientific-fact-checking datasets. We developed baseline models for the verdict prediction task using each dataset. Additionally, we demonstrated that the datasets could be used to improve performance with respect to the F 1 weighted metric on existing fact-checking datasets such as SciFact, HEALTHVER, COVID-Fact, and CLIMATE-FEVER. In some cases, the improvement in performance was up to a 26% increase.

## introduction learning to verify the claims in scientific papers and "science releases" (media announcements of scientific findings) is a difficult task for both artificial intelligence (ai) systems and humans. however, this task is crucial because learning to separate verified facts from speculation or falsehoods has important consequences. success at this task can help the reader understand scientific topics and promote science. conversely, failure at this task leads to the 1 https://github.com/taneset/multi2claim . spread of misinformation and exaggeration, which can cause distortion in scientific communication and undermine public confidence in science. unfortunately, several studies have revealed that science releases and scientific articles can contain significant exaggeration and misinformation @xcite , @xcite , @xcite , @xcite . this misleading information can directly impact people's lives, as was the case for media releases concerning the covid-19 pandemic @xcite . in 2014, some estimates claimed that 40% of the press releases contained exaggerated advice and 33% contained exaggerated claims in science-related news @xcite . considering that this problem has not disappeared over the last eight years and has possibly been exasperated by the continued growth of social media and the number of published scientific papers, there is a need for automated systems that can aid both academics and the public in judging the veracity and credibility of scientific claims. in this context, attempts to automate searching for distortions of findings, exaggerations, and misrepresentations may contribute to the trustworthiness verification of science releases and articles. the fact verification task, commonly known as "fact-checking", is verifying claims in natural language against a collection of information that contains facts. the pipeline for fact-checking usually includes two subtasks: 1) retrieve-rank and 2) veracity prediction. the retrieve-rank operation is typically carried out using information retrieval, and ranking models that rely on a combination of lexical (bm25) and semantic (using word embeddings in pre-trained language models) similarities between the claim and textual evidence candidates @xcite . the veracity prediction task, which has been studied less than the retrieve-rank task, can be seen as a classification task that predicts the type of verdict given a claim-evidence pair. typically there are three classes of verdicts: "support", "refute", and "not enough information". the veracity prediction problem can also be formulated as a task of identifying textual entailment, in which a model predicts if the provided evidence entails a given claim. recent work shows that these two subtasks could be combined in an end-to-end manner @xcite or could be carried out separately @xcite . most studies and datasets used in fact-checking are designed to verify claims in general domains such as news, forums, popular wikipedia passages, and social media posts @xcite @xcite @xcite . it is typically more challenging to automatically verify a scientific claim compared to a claim in the general domain. this is because scientific assertions can be much more complex, and it requires deep domain knowledge to create datasets for scientific claims. the required domain knowledge is a major bottleneck, making the annotation process expensive and time-consuming. as a result of these difficulties, there are only a few scientificfact-checking datasets in the literature, and the sizes of those datasets are limited. however, the recent deep-learning-based approaches to performing fact-checking require large amounts of annotated training data to generalise well on unseen data. therefore, there is an urgent need for large-scale scientific-fact-checking datasets, and methods for automatically creating such datasets. as a step towards addressing the problem of lack-ing large-scale datasets in scientific-fact-checking, this paper makes the following contributions: ## claim generation from multi-choice questions claim generation can be defined as the process of generating claims that can be classified as "supported", "refuted", or "not enough information" based on evidence in associated texts. a multiplechoice question typically consists of a question, a correct option, and multiple distractors. some multiple-choice qa datasets even provide an explanation of the correct answer. this section describes a pipeline to automatically generate all three types of claims that are commonly found in the existing fact-checking datasets by taking advantage of such multiple-choice qa datasets. the pseudocode of the generation process is described in algorithm 1. ## experiments we conducted experiments to answer the following research questions: we formulate verdict prediction as a multi-class classification task. for a given claim c and a document d, the model must determine a label l(c, d) ∈ {supported, refuted, not-enough-info}. we concatenate a claim and its document (explanation) together as input, and the model is trained to predict the claim type (supported, refuted, or not-enough-info) in a supervised manner. ## related work recent work in automated fact-checking has made progress in the battle against the spread of false information in the news @xcite , social media posts, online forums @xcite , and popular wikipedia articles @xcite . there has also been substantial work done that uses statements of fact-checking organizations @xcite . all the work mentioned so far has focused on claims and related documents from the general domain. however, the proposed models and datasets created can be difficult to apply to scientific domains because of the domain mismatch. limited research has been conducted with datasets and models that focus on verifying claims made in scientific releases against scientific documents. for exam- in recent years, a variety of techniques have been developed to enhance fact-checking abilities. one such model is based on multi-layer perceptrons (mlp) @xcite , and another is based on attention mechanisms @xcite . both models were used as baselines in claim verification on the fever dataset @xcite . additionally, a graph neural network (gnn) based approach @xcite ) was used for propagating nodes represented by evidence @xcite . semantic role labeling and logical reasoning tools can also improve gnn-based approaches @xcite . in @xcite , a graph-based evidence aggregating and reasoning (gear) framework was employed to aggregate multi-evidence data. recently, transformer-based language models have produced the best performance on factchecking in general and scientific domains. these claim-verification models usually take concatenated claim and evidence pairs and process them with multi-layer transformer-based models to obtain representations for classifying relationships between the claim and the evidence. pre-trained bert models have often been used for classification (supported, refuted, and not enough info). for claim verification, bert-based models are prevalent @xcite @xcite @xcite , while longformer has been used for verdict prediction @xcite . advancements in pre-trained transformer-based sequence-to-sequence language models, such as t5 @xcite and bart @xcite ), have allowed researchers to create fully automatic pipelines for claim generation. in particular, negation generation and explanation generation of claims have been studied in the generaldomain fact-checking task @xcite . entity-centric approach @xcite uses entities in the scientific text to generate claims and has been applied to scientific claim generation by @xcite . ## discussion because of the annotation cost, most large-scale fact-checking datasets are synthetically generated. for example, the popular fact-checking dataset, fever @xcite , contains a limited number of claims annotated by humans. the rest of the claims were synthetically augmented using paraphrasing, negating, and substituting the original claims. however, such methods have been found to weaken results in several studies since real-world claims have different structures than the synthetically augmented claims @xcite . when generating a refuted claim, simply adding "not" to a supported claim does not always reflect the structure of real-world refuted claims. in our approach, we use the expertselected distractors to generate refuted claims. the quantity of refuted claims is another weak point of the existing datasets. the amount of "refuted" claims is dramatically less than that of the "supported" claims in the existing datasets. this results in unbalanced fact-checking datasets (see the it is worth mentioning that the supporting documents for the claims that cannot be judged by a given document have been left as empty in @xcite and fact-checking datasets that are designed for binary labels ("supported" and "refuted") @xcite . in our proposed methods, we were able to retrieve related documents by using their dense-vector representations and consider them as documents for "not-enoughinfo" type of claims. finally, we want to discuss the entity-centric claim generation process @xcite . the first step of this process is selecting entities in a text to generate claims from them. these entities become the main objects of the generated claims. the weakness of this approach, according to @xcite is that the generated claims can be superficial, and the verification of these claims can be done without the models' reasoning capabilities or knowledge of common sense. the degree of importance of these entities can differ for a scientific text that contains entities from multiple domains. since there is no mechanism to indicate how important the selected entity is compared to the other entities in the given text, this approach might result in a poor selection of entities and, therefore, poor claim data. in our proposed method, using the multiple choices/entities of the question likely avoids that problem, because domain experts select the entities in the questions. ## conclusion we have presented multi2claim, a pipeline that converts multiple-choice questions to fact-checking datasets. we specifically focus on challenging scientific domains, where the claim verification process can be complicated due to scientific jargon and complex assertions about fields. we presented two large-scale scientific fact-checking datasets created with this pipeline in biomedical (med-fact) and general science domains (gsci-fact). we testified these dataset for possible biases and the generated datasets were evaluated from various perspectives. baseline models for these balanced large-scale scientific fact-checking datasets were also presented. we conducted extensive experiments to examine the benefits of the generated datasets on the existing scientific fact-checking dataset, which suffer from low numbers of samples and unbalanced labels. we consistently obtained improvements for all scientific fact-checking datasets, including binary-labeled datasets such as covid-fact. we hope this work will lead to more breakthroughs in scientific fact-checking, which has received little attention due to the expensive and timeconsuming annotation process that has to be done by domain experts. we also hope that the proposed pipeline and baseline models will help develop reliable models that will play an essential role in the scientific claim verification process.