PHEMEP lus: Enriching Social Media Rumour Verification with External Evidence (2022)

Abstract: Work on social media rumour verification utilises signals from posts, their propagation and users involved. Other lines of work target identifying and fact-checking claims based on information from Wikipedia, or trustworthy news articles without considering social media context. However works combining the information from social media with external evidence from the wider web are lacking. To facilitate research in this direction, we release a novel dataset, PHEMEPlus, an extension of the PHEME benchmark, which contains social media conversations as well as relevant external evidence for each rumour. We demonstrate the effectiveness of incorporating such evidence in improving rumour verification models. Additionally, as part of the evidence collection, we evaluate various ways of query formulation to identify the most effective method.

## introduction the harm and prevalence of online misinformation made research into automated methods of information verification an important and active research area. this includes various tasks like fact-checking, social media rumour detection, stance classification and verification. in this work we are concerned with social media rumour verification, the task of identifying whether a rumour (i.e check-worthy claim circulating on social media whose veracity status is yet to be verified @xcite ), is true, false or unverified. although a significant amount of work has been done towards evaluating the veracity of social media rumours @xcite @xcite , there is still a dearth of works and datasets combining the information from social media with external evidence from the wider web. while recent works focusing on rumours around the covid-19 pandemic have been collecting data from a wide range of sources from news and social media to scientific publications @xcite @xcite , these are not sufficient for the creation of generalisable verification models as they only focus on a single topic. at the same time works on fact-checking, which do not focus on social media content, but use claims from debunking websites @xcite , as well as recent work by @xcite have shown the benefits of utilising stance of evidence for verification. here we aim to further enable research in this direction and release an enriched version of a popular benchmark dataset pheme @xcite with timely evidence for each of the rumours, obtained from a wide range of web sources. although a few works use web search for evidence retrieval @xcite , to our knowledge, only the work of @xcite touches upon the topic of the search query formulation. here we analyse several query formulation strategies to find the most effective one. in this work we make the following contributions: ## evaluating the effectiveness of evidence for rumour verification we conduct experiments to evaluate the effectiveness of our retrieved evidence for twitter rumour veracity classification. ## conclusions and future work after experimentation with various searching strategies for retrieving evidence from the web, we have constructed the phemeplus dataset, which will facilitate further work on using evidence from wide range of sources for rumour veracity classification. the best such strategies, according to our evaluation metrics, are those which leave the grammatical structure of the claim relatively intact. there is much potential to improve existing rumour veracity classification systems by augmenting them with, or with a broader range, or better quality of evi-dence. we plan to build upon these findings in the future, working on identifying ways of incorporating the evidence from heterogeneous sources into more complex rumour verification models to maximise the gains from this information and achieve state-of-the-art results.