Document Retrieval and Claim Verification to Mitigate COVID -19 Misinformation (2022)

Abstract: During the COVID-19 pandemic, the spread of misinformation on online social media has grown exponentially. Unverified bogus claims on these platforms regularly mislead people, leading them to believe in half-baked truths. The current vogue is to employ manual fact-checkers to verify claims to combat this avalanche of misinformation. However, establishing such claims’ veracity is becoming increasingly challenging, partly due to the plethora of information available, which is difficult to process manually. Thus, it becomes imperative to verify claims automatically without human interventions. To cope up with this issue, we propose an automated claim verification solution encompassing two steps – document retrieval and veracity prediction. For the retrieval module, we employ a hybrid search-based system with BM25 as a base retriever and experiment with recent state-of-the-art transformer-based models for re-ranking. Furthermore, we use a BART-based textual entailment architecture to authenticate the retrieved documents in the later step. We report experimental findings, demonstrating that our retrieval module outperforms the best baseline system by 10.32 NDCG@100 points. We escort a demonstration to assess the efficacy and impact of our suggested solution. As a byproduct of this study, we present an open-source, easily deployable, and user-friendly Python API that the community can adopt.

## introduction the escalating drift of online social media platforms has led to a massive rise in online content consumers. participation in these platforms has swung into another correspondence, which is no longer limited by physical barriers. because of their speed and focused information, these platforms facilitate the dissemination of personal thoughts and information to a much larger audience. however, at the same time, these platforms have enriched an equally docile environment for malicious users to promulgate fake news, bogus claims, rumors and misinformation. there have been numerous cases where the propagation of malicious unverified content has influenced the entire society. one such concrete example is the 2016 presidential elections in the united states, which witnessed the alarming impact of false news, with many citizens swayed by a fraudulent website @xcite . @xcite revealed that nearly 25% of american citizens visited a fake news website that aimed at manipulating the general public's cognitive process and consequently clouted the eventual conclusion of the election. another recent example is the global pandemic of covid-19. when the entire world went into lockdown, the virtual world encountered a great closeness transforming social media platforms into the primary conduits for information consumption and dissemination. consequently, there has been an accretion of 50%-70% in total internet hits in the year 2020 @xcite . around the same time, enormous social media posts with unverified bogus claims about the pandemic began to arise, frequently spurring life-threatening remedies @xcite . such claims had an unprecedented impact, resulting in monetary damage and the loss of priceless human lives. a study revealed that at least 800 individuals died worldwide in the first quarter of 2020 due to misinformation about covid-19 @xcite . motivation: a slew of such incidents has continued to emerge from the worldwide community in recent years. thousands of people read these unverified claims online and spread misinformation if the claims' integrity is not corroborated. as a result, a variety of manual fact-checking organizations have evolved to address this concerning issue. unfortunately, the enormity of misinformation floating around on the internet has developed into a global infodemic foot_0 making their efforts untenable. to alleviate this bottleneck, the process of automating fact-checking has recently garnered a lot of consideration in the research world. @xcite formalized the task of fact-checking and claim verification as a series of components -identifying claims to be evaluated, extracting relevant shreds of evidence, and delivering verdicts. as a result, this facilitated the establishment of automated fact-checking pipelines composed of subcomponents that can be mapped to tasks well-studied in the nlp community. the task of retrieving relevant information has gained a lot of impetus in recent years, especially with the introduction of tools like pyserini foot_1 and beir foot_2 . furthermore, advancements were made by establishing datasets of either claims acquired from fact-checking websites (wang, 2017) or datasets curated specifically for research @xcite . the recent release of the cord-19 dataset foot_3 , consisting of more than 500,000 articles, has provided access to thousands of scientific articles on the prevention techniques, spread, transmission, and cures of the covid-19. the dataset consists of more than 500,000 articles. ## related work the challenge of verifying claims on online social media has garnered considerable attention in the last several years. initially, the task of automatic claim verification and fact-checking were investigated in the context of computational journalism @xcite , and journalists and professional fact-debunkers manually verified claims utilizing various information sources. however, that was not just time-consuming but also introduced substantial human bias in it. the recent advancement in nlp and information retrieval (ir) has equipped journalists and online social media users with tools enabling automatic claim verification. in the past few years, plenty of work has been proposed to fact-check online claims. @xcite presented the initial pioneering work in this domain. they published the first claim verification dataset, which included 106 statements taken from fact-checking websites like politifact. however, they lacked justification for the verdict, which verification systems typically require. to address this issue, wang (2017) prolonged this approach by introducing 12.8k claims from politifact along with their explanations. the fact extraction and verification (fever) shared task was launched to advance research in this direction @xcite . the organizers of the fever shared task constructed a large-scale dataset of 185445 claims based on wikipedia articles, each of which comes with several evidence sets. traditionally, the existing claim verification systems primarily rely on textual content and/or social context. the content-based methods essentially acquire the n-grams (wang, 2017), semantics @xcite , writing styles @xcite , etc. besides textual-content, auxiliary knowledge around social-context has also been extensively examined for verification tasks. these context-based methods emphasize collecting user profile-based @xcite , propagation structure-based @xcite , source-based @xcite introduced claimverif that provides a credibility score for a user given a claim and also gives supporting evidences that justify the credibility score. @xcite presented their approach to the fever task @xcite which was introduced to expedite the development of fact verification systems, in which they used entity linking for document retrieval and enhanced sequential inference model for determining the entailment. ma et al. (2019) used hierarchical attention networks with sentence-level evidence embeddings. despite the fact that these tactics produce good performance results, it is challenging for these approaches to provide adequate reasons for claim verification outcomes. as a result, current research has focused on interpretable claim verification, which develops interactive models to examine the distinction. attentionbased interaction models @xcite , gate fusion interactive models @xcite , coherence modelling interactive models @xcite , and graph-aware interaction models are among the interactive models. the granularity of captured semantic conflicts involves word-level @xcite , sentence-level @xcite , and multi-feature (wu and rao, 2020) conflicts. @xcite came up with a question-answeringbased model that mines relevant articles from the cord-19 dataset and summarizes them to answer pressing questions about the covid-19 pandemic. recently, @xcite proposed a t5 foot_5 transformer-based architecture for abstract retrieval, sentence selection and label prediction and perform claim verification. similar to us, they also utilized the cord-19 (wang et al., 2020) corpus as the knowledge base to retrieve shreds of evidences. these methods, which employ semantic conflicts to verify claims, reflect a certain degree of interpretability. but not all conflicts can be used as valid evidence to reasonably explain the results, and they also include considerable conflicts unrelated to claims or even interfere with the verified results. it is difficult for automatic claim verification to provide reasonable explanations for the claim: 1 @cnn boosting our immune systems will help deter the virus. it's our only defense aside from n95 masks and goggles evidence label first, there's the not-so-great news. despite claims you may have seen on the internet, there's no magic food or pill that is guaranteed to boost your immune system and protect you against coronavirus...there are ways to keep your immune system functioning optimally, which can help to keep you healthy and give you a sense of control in an uncertain time...for a starter dose of immune-boosting vitamins, minerals and antioxidants, fill half of your plate with vegetables and fruits. ## description of the datasets for our experiments, we adopt two datasets. their details are shown as follows: ## our approach adhering to the standard of automated claim verification and fact-checking systems @xcite , our proposed approach also consists of a two-step pipeline -document retrieval and veracity prediction. in this section, we present the techniques employed for retrieval and veracity prediction components. besides the current approach, we had also employ alternative techniques using rapid automatic keyword extraction or rake @xcite and scispacy @xcite for keyword extraction and searching our corpus using the extracted keywords. figure 1 illustrates the general architecture of our proposed claim verification approach. once a textual claim is submitted, the document retrieval module extracts the top-k relevant documents from the knowledge base. the retrieved documents are then passed to the veracity prediction module that figures out an entailment decision for the claim with respect to the retrieved evidences. ## evaluation we compare the findings of our retrieval system bm25 to those of other existing systems. we employ a collection of claims and ground-truth labels from our claver dataset for quantitative evaluation. the test data set consists of claims excluded from the knowledge base in the retrieval phase. for this, we develop a manually annotated dataset with ∼ 1000 claims obtained from twitter and build a knowledge-base of ∼ 400 articles from reliable sources, equipping a testing ground to validate the results. ## demonstration in this section, we demonstrate how our proposed claim verification pipeline works. figure 2 depicts an example claim as well as the model's output results. users enter a claim into our system as a query, and the system evaluates whether or not it is a validated claim. in practice, the system takes somewhere around 20 and 80 seconds to execute a single user query, depending on the number and length of articles obtained by the search engine. the input section of our tool, as shown in figure 2 , provides a query text box where the user can enter any natural language text as an input claim for evaluation, as well as a specific configuration to 8 links of article sources can be found at: https:// cutt.ly/lfwsxxa limit the number of articles to be retrieved. following the submission of the claim, the tool's back-end server does its analysis. it returns three sets of outputs: (i) a set of articles employing the various approaches, (ii) a claim category, and (iii) a veracity score. the output also presents the technique utilized for retrieval (pink) and from which knowledge base the shreds of evidence were extracted (blue). the most intriguing aspect of the system is that it links resources from the web, where the article was retrieved, allowing individuals to make their own decisions based on them. not all information is equally reliable, and sometimes even the trusted sources contradict one another. this calls into question the assumptions behind most current fact-checking research, which relies on a single authoritative source. as a result, we offer results for a common claim from several models and knowledge bases. for demonstration, we practice the widely spread claim "vaccines are not effective against covid-19" as an input as shown in figure 2 , and the tool returned the top-ranked shreds of evidence. the first two pieces of evidence come from the bm25 model, which was run on the cord-19 dataset and our data, respectively. furthermore, evidences 3 and 4 collected articles from our dataset using a dense and hybrid retrieval strategy, respectively. we can see that all four pieces of evidence assigned the same label to the claim, but their truthfulness scores differed from each other. ## conclusion in this work, we verged upon claim verification on online social media towards coping with misinformation. we bestowed a claim verification system that evaluates the authenticity of a usersupplied query claim and justifies the verdict corroborating evidence. we explored multiple retrieval methodologies and published user research findings, demonstrating the utility of the bm25 method. unlike other tools, our system learns the distributed representations to encapsulate the semantic relations between the claim and the evidence. our approach uses a two-step training process to provide a high-quality veracity score as well as best-suited articles, leveraging data from formal articles and web-based informal texts. we have made the source codes and the dataset public at the following link: https://github.com/ lcs2-iiitd/claim_verification .