Multiple Evidence Combination for Fact-Checking of Health-Related Information (2023)

Abstract: Fact-checking of health-related claims has become necessary in this digital age, where any information posted online is easily available to everyone. The most effective way to verify such claims is by using evidences obtained from reliable sources of medical knowledge, such as PubMed. Recent advances in the field of NLP have helped automate such fact-checking tasks. In this work, we propose a domain-specific BERT-based model using a transfer learning approach for the task of predicting the veracity of claim-evidence pairs for the verification of health-related facts. We also improvise on a method to combine multiple evidences retrieved for a single claim, taking into consideration conflicting evidences as well. We also show how our model can be exploited when labelled data is available and how back-translation can be used to augment data when there is data scarcity.

## introduction in today's age of easy access to the internet, information exchange among people has increased rapidly, which has also resulted in the spread of misinformation @xcite within the society. misinformation has been found to spread faster than real news, and the rise of social media popularity has aided the spread of misinformation @xcite . research on health misinformation is still an ongoing area, as it is different from political misinformation on the basis of the complexity level of fact-checking @xcite . manual fact-checking of health information requires domain-specific experts, which increases both time taken and cost incurred. automated fact-checking of health information found online has been aided by the release of datasets such as scifact @xcite , healthver @xcite , covid-fact @xcite . fact-checking of health information comprises of retrieving evidences from reliable resources which either supports or refutes the key claim @xcite . recent works have focused on building end-to-end fact-checking models evaluating on the aforementioned datasets @xcite @xcite . however, they do not take into account conflicting evidences retrieved for a single claim. any claim can have more than one evidence, and these evidences can be conflicting in real-world scenarios wherein one evidence would be supporting the claim and another evidence may be refuting the claim. ## related work in this section, we will discuss the research work that has been done for fact-checking scientific claims using evidences retrieved from existing medical article repositories. with the release of the sci-fact dataset, various transformer-based methods of predicting the veracity labels using evidences for scientific claims have been proposed and evaluated using the dataset. @xcite ) established a pipeline model using a roberta-large @xcite model to retrieve evidences from pubmed abstracts. the retrieved evidence sentences are then passed along with the claims to predict whether the evidences support or refute the claims using a roberta-large model fine-tuned over the training set of scifact. vert5erini @xcite uses a t5 (raffel et al., 2020) model-based pipeline for their work. for the evidence sentence selection task for the claims, as well as for label prediction from pubmed abstracts, they used two different t5 models. for the sentence selection task, the t5 model used is fine-tuned over the ms-marco @xcite dataset and then further trained on scifact. for the label prediction task, the t5 model is trained on the scifact dataset. paragraph-joint @xcite uses a roberta-large model similar to @xcite for both the evidence sentence selection as well as the label prediction task which is fine-tuned over scifact. however, the training approach is different, as @xcite uses a multitask learning approach for model training. both the tasks of sentence selection and label prediction are done using a joint cross-entropy loss as the training objective. for the label prediction task, the authors have also used two different approaches which includes a simple sentence-level attention and kgat which is a kernel graph attention network @xcite . similarly, arsjoint @xcite also uses a joint approach where their proposed method jointly learns the three tasks of abstract retrieval, sentence selection, and label prediction. similar to @xcite , they have also used roberta-large for their work together with biobert-large @xcite . all the above works focus on the three tasks of abstract retrieval, evidence sentence selection and label prediction as a pipeline approach. however, there is a difference in the sentence selection task as well as the label prediction. vert5erini selects sentences independently, whereas paragraph-joint and arsjoint use the abstracts to select the sentences. the label prediction also differs as both paragraph-joint and arsjoint use a joint approach unlike vert5erini. the models used in the tasks also differ as paragraph-joint and arsjoint use bert-based models whereas vert5erini uses a much larger t5 model having superior performance. however, the current stateof-the-art method, multivers @xcite differs from these works in the approach and the transformer model used. multivers uses a longformer @xcite architecture to encode both claims and abstracts together so that there is a minimum loss of information. the authors have used a weak supervision approach, in which the longformer model is trained on available scientific data before fine-tuning on scifact. however, the overall pipeline training method is a multi-task approach similar to @xcite . it outperforms the other approaches in the label prediction task of scifact. contrary to the above works, our approach is different in the way that for a given pair of claim and evidence, our model can predict the labels in a zero-shot approach, surpassing the state-of-theart results without the need for any supervision. the above mentioned works have the end goal of predicting the labels of the claim-evidence pairs. however, to have a final prediction for the claims whether it is a "true" claim or "false" claim, we need to have a combined judgement of all the evidence sentences for that claim which is not addressed by the above works. we have extended our approach to include the final prediction for the claims taking into consideration conflicting evidences as well. ## task formulation in this section, we will first discuss the problem statement and then proceed with the formulation of the tasks. the problem statement is "given a claim and a number of evidence sentences, determine whether the claim is true or false or neutral". we can formulate the problem as two tasks: ## methodology in this section we will describe in detail the proposed methods we have adopted for the formulated tasks. ## experimental details in this section, we will describe the experimental details for our tasks. ## transferring over other datasets in order to know how well our model generalizes over other similar data, we experimented with two similar fact-checking datasets on biomedical data, healthver @xcite and covid-fact @xcite . both datasets focus on covid-19 data, however, the way claims and evidences are collected in both these datasets differ. healthver claims are collected from cord-19 @xcite corpus article snip-pets which were retrieved to answer questions for trec-covid @xcite . the claims in healthver are complex and evidences are provided for each claim. the dataset has three labels, support, refute and neutral based on the evidences which are collected from the article snippets itself. covidfact, on the other hand, has claims collected from covid-19 subreddit and evidences are collected from linked scientific papers and documents collected from google search. the claims in covidfact are also complex and has two labels for the evidences collected, support and refute. both healthver and covidfact have one annotated evidence per claim, whereas in scifact, there may be more than one evidence for one claim. also, in sci-fact, relevant abstracts are needed to be retrieved first from the corpus provided and then evidence sentences are needed to be retrieved from those abstracts. although our model has not been trained over covid-19 specific text, we wanted to experiment how well it generalizes over such data by performing two different experiments. in the first experiment, we applied our model to the test set of both healthver and covidfact without using the training set in a zero-shot approach. for scifact, we have used the dev set. we can see from for the second experiment, we have transferred a trained model over one dataset to the other two to see how well models trained on one dataset generalize to other datasets. we use the train sets of the datasets to train the model and then use the test sets of the other datasets to evaluate the model per-formance. since the healthver dataset has the neutral label, we have dropped instances from its test set having that label in order to maintain consistency over all the datasets when the model was trained over scifact and covidfact train sets since these datasets only have support and refute labels. the results of the experiment are shown in we have explored the prediction of veracity for health-related fact-checking tasks that can be learned from nli data. by doing experiments, we showed that training domain-specific bert-based models on domain-specific nli data improves the model performance for fact-checking task. we also explored a method that can be used to combine different evidences for a claim, even for situations that have conflicting evidences for the same claim. we have also shown by experiments that augmenting data using back-translation helps in situations where there is a lack of training data. although fact-checking of scientific claims is still a new task, there is a potential for improvement of the current methods being used for the task. with the advent of more capable large language models, new research direction such as prompt based methods can also be explored. as future work, we are interested in exploring such prompt-based approaches along with multimodal data in this space.