A u RED : Enabling A rabic Rumor Verification using Evidence from Authorities over T witter (2024)

Abstract: Diverging from the trend of the previous rumor verification studies, we introduce the new task of rumor verification using evidence that are exclusively captured from authorities, i.e., entities holding the right and knowledge to verify corresponding information. To enable research on this task for Arabic low-resourced language, we construct and release the first Authority-Rumor-Evidence Dataset (AuRED). The dataset comprises 160 rumors expressed in tweets and 692 Twitter timelines of authorities containing about 34k tweets. Additionally, we explore how existing evidence retrieval and claim verification models for fact-checking perform on our task under both the cross-lingual zero-shot and in-domain fine-tuning setups. Our experiments show that although evidence retrieval models perform relatively well on the task establishing strong baselines, there is still a big room for improvement. However, existing claim verification models perform poorly on the task no matter how good the retrieval performance is. The results also show that stance detection can be useful for evidence retrieval. Moreover, existing fact-checking datasets showed a potential in transfer learning to our task, however, further investigation using different datasets and setups is required.

## introduction the spread of rumors and fake news on social media causes anxiety and panic in communities, forming persistent challenges for platforms, policymakers, and researchers. to address this, several rumor verification studies on social media incorporate the propagation networks as a key source of evidence. they either utilize the stance of replies @xcite @xcite , the structure of the replies @xcite @xcite , or the users' metadata @xcite . on the other hand, evidence are extracted from the web to augment signals from the propagation networks @xcite . however, studies on arabic rumor verification incorporating evidence are scarce. @xcite and @xcite exploited the tweet replies, while @xcite leveraged the images and videos embedded in the rumor tweet. authorities (i.e., entities having the real knowledge or power to verify or deny a specific rumor @xcite ) can also be a valuable source of evidence that augments other sources for verifying rumors, either by automated verification systems or more specifically by human fact-checkers. detecting the stance of authorities towards rumors in twitter 1 was indeed introduced recently as a potential signal for better rumor verification @xcite . however, to the best of our knowledge, no study to date has explored the incorporation of evidence tweets retrieved from the timelines of authorities for rumor verification over social media in general and for arabic rumor verification in particular. additionally, there is no available dataset for that task to support such research. to bridge this gap, in this paper, we introduce the problem of rumor verification using evidence from authorities over twitter and a dataset that enables research tackling that problem. the problem is defined as follows: given a rumor expressed in a tweet and a set of twitter accounts of authorities for that rumor, the system should retrieve evidence tweets posted by any of those authorities. based on the retrieved evidence, the system should determine if the rumor is supported, refuted, or unverifiable. figure 1 illustrates the setup of the problem. to facilitate the research on this task, we introduce aured, the first authority-rumor-evidence dataset. aured covers 160 arabic rumors an-1 "twitter" is the former name of "x," however we will use "twitter" for clarity. notated with tweet-level evidence from their corresponding 692 authority timelines, comprising about 34k annotated tweets in total. the dataset was constructed by annotating a set of rumors, selected from two existing datasets @xcite , following two main steps (1) finding authorities that can help verify the rumors @xcite , and (2) collecting the timelines of those authorities, and annotating those timelines to find evidence tweets. our contribution in this work is five-fold: the remainder of the paper is organized as follows. we review the literature in section 2 and formally define our task in section 3. in section 4, we discuss our dataset construction approach. our experimental design and setup are presented in sections 5 and 6, respectively. we analyze our experimental results and answer our research questions in section 7. we conclude and suggest some future directions in section 8. finally, we discuss the limitations of our work in section 9. ## related work in this section, we review previous studies on rumor verification in social media and fact-checking. ## task definition we propose the task of rumor verification using evidence from authorities with two subtasks: ## aured dataset to expedite the development of automatic verification systems and to evaluate proposed models for our task, we introduce the first authority-rumor-evidence dataset (aured). we target arabic as it is one of the most used languages in twitter @xcite , yet under-explored for rumor verification. as presented in figure 2 , the dataset was constructed by annotating a set of rumors, selected from two existing datasets (section 4.1) following two main steps (1) finding authorities that can help verify the rumors (section 4.2), and (2) collecting and annotating the timelines of those authorities to find evidence tweets (section 4.3). ## experimental design our task is closely related to the general task of fact-checking @xcite . in fact, it can be viewed as a special case of the fact-checking task, where evidence for verification is exclusively retrieved from authorities rather than from any other source, e.g., web pages, or posts from layman users or propagation networks on social media. with a large body of existing research on the fact-checking task @xcite , it is intriguing to investigate how existing evidence retrieval and claim verification models, originally designed for the general fact-checking task, perform on our specific task. moreover, with the availability of datasets for the general task in other languages (e.g., fever @xcite )), it is then intuitive to explore the potential of cross-lingual transfer learning. accordingly, we address the following research questions: ## experimental setup in this section, we present our detailed experimental setup. we discuss our adopted evidence retrieval and rumor verification models in sections 6.1 and 6.2, respectively. we also discuss how we evaluate those models in section 6.3. ## results and discussion in this section, we present and discuss the results of our experiments which address the two research questions introduced in section 5. ## conclusion and future work in this paper, we introduced the new task of rumor verification using evidence from authorities over twitter. we constructed and released the first authority-rumor-evidence dataset (aured) which consists of 160 rumors expressed in tweets and 692 timelines of authorities twitter accounts comprising about 34k annotated tweets in total. we explore existing fact-checking models to set up the baseline systems for our two substasks namely evidence retrieval and rumor verification. our experiments show that evidence retrieval models for fact-checking achieved competitive benchmark results even under cross-lingual zero-shot setup, however the performance on rumor verification is still far from enough. for future work, we plan to ( 1 ) consider the multimodality of evidence tweets to improve the evidence retrieval, (2) augment the dataset to expand the number of rumors to improve the rumor veracity prediction, (3) propose models to improve the performance achieved on both subtasks, and (4) construct a similar dataset in english to facilitate and encourage research on the task.