Natural Language Inference with Self-Attention for Veracity Assessment of Pandemic Claims (2022)

Abstract: We present a comprehensive work on automated veracity assessment from dataset creation to developing novel methods based on Natural Language Inference (NLI), focusing on misinformation related to the COVID-19 pandemic. We first describe the construction of the novel PANACEA dataset consisting of heterogeneous claims on COVID-19 and their respective information sources. The dataset construction includes work on retrieval techniques and similarity measurements to ensure a unique set of claims. We then propose novel techniques for automated veracity assessment based on Natural Language Inference including graph convolutional networks and attention based approaches. We have carried out experiments on evidence retrieval and veracity assessment on the dataset using the proposed techniques and found them competitive with SOTA methods, and provided a detailed discussion.

## introduction in recent years, and particularly with the emergence of the covid-19 pandemic, significant efforts have been made to detect misinformation online with the aim of mitigating its impact. with this objective, researchers have proposed numerous approaches and released datasets that can help with the advancement of research in this direction. most existing datasets (d' @xcite focus on a single medium (e.g., twitter, facebook, or specific websites), a unique information domain (e.g., health information, general news, or scholarly papers), a type of information (e.g., general claims or news), or a specific application (e.g., verifying claims, or retrieving useful information). this inevitably results in a limited focus on what is a complex, multi-faceted phenomenon. with the aim of furthering research in this direction, the contributions of our work are twofold: (1) creating a new comprehensive dataset of misinformation claims, and (2) introducing two novel approaches to veracity assessment. in the first part of our work, we contribute to the global effort on addressing misinformation in the context of covid-19 by creating a dataset for pandemic ai claim veracity assessment, called the panacea dataset. it is a new dataset that combines different data sources with different foci, thus enabling a comprehensive approach that combines different media, domains and information types. to this effect our dataset brings together a heterogeneous set of true and false covid claims and online sources of information for each claim. the collected claims have been obtained from online fact-checking sources, existing datasets and research challenges. we have identified a large overlap of claims between different sources and even within each source or dataset. thus, given the challenges of aggregating multiple data sources, much of our efforts in dataset construction has focused on eliminating repeated claims. distinguishing between different formulations of the same claim and nuanced variations that include additional information is a challenging task. our dataset is presented in a large and a small version, accounting for different degrees of such similarity. finally, the homogenisation of datasets and information media has presented an additional challenge, since fact-checkers use different criteria for labelling the claims, requiring a specific review of the different kinds of labels in order to combine them. in the second part of our work, we propose nli-san and nli-graph, two novel veracity assessment approaches for automated fact-checking of the claims. our proposed approaches are centred around the use of natural language inference (nli) and contextualised representations of the claims and evidence. nli-san combines the inference relation between claims and evidence with attention techniques, while nli-graph builds on graphs considering the relationship between all the different pieces of evidence and the claim. specifically we make the following contributions: covid-19 and misinformation datasets. comprehensive information on covid-19 datasets is provided in appendix a. such datasets include the coronavirusfacts/datoscoronavirus alliance database, the largest existing collection of covid claims and the largest existing network of journalists working together on covid misinformation, an essential reference for our work; covid-19-tweetids (chen et al., 2020) the widest dataset of covid tweets with more than 1 billion tweets; cord-19: the covid-19 open research dataset @xcite , the largest downloadable set of scholarly articles on the pandemic with nearly 200,000 articles. general misinformation datasets linked to our verification work include: emergent @xcite collection of 300 labeled claims by journalists; liar @xcite with 12,836 statements from politifact with detailed justifications; fakenewsnet @xcite collecting not only claims from news content, but also social context and spatio-temporal information; nela-gt-2018 @xcite with 713,534 articles from 194 news outlets; fakehealth @xcite collecting information from healthnewsreview, a project critically analysing claims about health care interventions; pubhealth @xcite with 11,832 claims related to health topics; fever @xcite as well as its later versions fever 2.0 @xcite and fever-ous @xcite , containing claims based on wikipedia and therefore constituting a well-defined, informative and non-duplicated information corpus; scifact @xcite also from a very different domain, containing 1,409 scientific claims. our dataset is a real-world dataset bringing together heterogeneous sources, domains and information types. approaches to claim veracity assessment. we employ our dataset for automated fact-checking and veracity assessment @xcite . researchers such as hanselowski et al. (2018); yoneda et al. (2018); luken et al. (2018); soleimani et al. (2020); pradeep et al. ( 2021 ) analysed the veracity relation between the claim and each piece of evidence independently, combining this information later. other authors considered multiple pieces of evidence together @xcite @xcite . different pieces of evidence have been previously combined using graph neural networks @xcite @xcite . many of these authors have centred their techniques on the use of nli @xcite @xcite to verify the claim. in our work we also make use of nli results of claim-evidence pairs, but propose alternative approaches built on a self-attention network and a graph convolutional network for veracity assessment. ## dataset construction this section describes our dataset construction by selecting covid-19 related data sources ( §3.1), and applying information retrieval and re-ranking techniques to remove duplicate claims ( §3.2). ## claim veracity assessment we develop a pipeline approach consisting of three steps: document retrieval, sentence retrieval and veracity assessment for claim veracity evaluation. given a claim, we first retrieve the most relevant documents from covid-19 related sources and then further retrieve the top n most relevant sentences. considering each retrieved sentence as evidence, we train a veracity assessment model to assign a true or false label to the claim. ## conclusions we have presented a novel dataset that aggregates a heterogeneous set of covid-19 claims categorised as true or false. aggregation of heterogeneous sources involved a careful deduplication process to ensure dataset quality. fact-checking sources are provided for veracity assessment, as well as additional information sources for true claims. additionally, claims are labelled with sub-types (multimodal, social media, questions, numerical, and named entities). we have performed a series of experiments using our dataset for information retrieval through direct retrieval and using a multi-stage re-ranker approach. we have proposed new nli methods for claim veracity assessment, attention-based nli-san and graph-based nli-graph, achieving in our dataset competitive results with the gear and kgat state-of-the-art models. we have also discussed challenging cases and provided ideas for future research directions.