A r COV 19-Rumors: A rabic COVID -19 T witter Dataset for Misinformation Detection (2021)

Abstract: In this paper we introduce ArCOV19-Rumors, an Arabic COVID-19 Twitter dataset for misinformation detection composed of tweets containing claims from 27th January till the end of April 2020. We collected 138 verified claims, mostly from popular fact-checking websites, and identified 9.4K relevant tweets to those claims. Tweets were manually-annotated by veracity to support research on misinformation detection, which is one of the major problems faced during a pandemic. ArCOV19-Rumors supports two levels of misinformation detection over Twitter: verifying free-text claims (called claim-level verification) and verifying claims expressed in tweets (called tweet-level verification). Our dataset covers, in addition to health, claims related to other topical categories that were influenced by COVID-19, namely, social, politics, sports, entertainment, and religious. Moreover, we present benchmarking results for tweet-level verification on the dataset. We experimented with SOTA models of versatile approaches that either exploit content, user profiles features, temporal features and propagation structure of the conversational threads for tweet verification.

## introduction in addition to being a medium for the spread and consumption of news, twitter has been shown to capture the dynamics of real-world events including the spread of diseases such as the seasonal influenza @xcite or more severe epidemics like ebola @xcite . since the first reported case of the novel coronavirus in china, in november 2019, the covid-19 topic has drawn the interest of many arab users over twitter. their interest, reflected in the arabic content on the platform, has reached a peak after two months when the first case was reported in the united arab emirates late in jan-uary 2020. this ongoing pandemic has, unsurprisingly, spiked discussions on twitter covering a wide range of topics such as general information about the disease, preventive measures, procedures and newly-enforced decisions by governments, upto-date statistics of the spread in the world, and even the change in our daily habits and work styles. with the great importance and spread of covid-19 information, misinformation and fake news have infected the twitter stream. an early study quantifying covid-19 medical misinformation on twitter found that 25% of collected tweets contained misinformation @xcite . during covid-19 pandemic, we observed that misinformation stretched beyond spreading fake and potentiallyharmful medical information, to information that can have adverse negative political effects (example: "in light of the unresponsiveness of yemeni government to requests of evacuation from yemeni students in wuhan, sultan of oman orders their evacuation.") and economical effects too (example: "kuwaitis boycott almarai saudi dairy company after reports on coronavirus infected employees."). combating the spread of such claims and verifying them becomes essential during this sensitive time. in this work, we aim to facilitate research on misinformation detection on social media during this complex and historical period of our time by introducing a manually-annotated arabic dataset, arcov19-rumors, that covers tweets spreading covid-19 related claims. to construct arcov19-rumors, we start from an existing covid-19 arabic dataset @xcite , arcov-19, that is the first arabic covid-19 twitter dataset with propagation networks. our proposed arcov19-rumors includes a set of 138 covid-19 verified claims and 9.4k corresponding relevant tweets that were manually-annotated to support both claimlevel and tweet-level verification tasks. claim-level verification is defined as follows: given a short free-text claim (in 1 or 2 sentences) and its corresponding relevant tweets, predict whether the claim is true or false. tweet-level verification is defined as follows: given a tweet containing a claim, detect whether it is true or false. to our knowledge, arcov19-rumors is the only arabic dataset made available to support both claim-level and tweet-level verification tasks in twitter given the propagation networks of the tweets in general and on covid-19 in particular. some related twitter datasets were recently released by @xcite , @xcite , @xcite @xcite . none of these datasets has the propagation networks of the tweets, and they either support the tweet verification task @xcite , misinformation detection @xcite or multi-class categorization including rumors as one category @xcite . differently from arcov19-rumors, elhadad et al. (2020a) used an automatic approach to annotate tweets, while alsudias and rayson (2020) only cover covid-19 health-oriented claims. starting from some health misinformation reported by the ministry of health in saudi arabia and the world health organization (who), alqurashi et al. (2021) annotated covid-19 tweets as misinformation or not. differently, our dataset covers in addition to health, other types of claims that were influenced by covid-19, namely, social, politics, sports, entertainment, and religious. the contribution of this paper is three-fold: the remainder of the paper is organized as follows. we present studies related to covid-19 misinformation analysis and datasets in section 2. the construction of arcov19-rumors is presented in section 3. several use cases supported by our dataset are discussed in section 4. our benchmarks and their performances are presented in section 5. we detail the released components of the dataset in section 6, and conclude in section 7. ## related work the negative effects and spread of misinformation about covid-19, triggered efforts to understand this phenomenon. many studies analyzed misinformation spreading on twitter related to @xcite @xcite @xcite . @xcite analyzed the spread of five health misinformation over time. @xcite analyzed false and partially false tweets that have been previously fact-checked by a factchecking platform. @xcite focused only on health misinformation and analyzed tweets negating health information by trusted sources (e.g., who). cui and lee (2020) and @xcite published an english dataset for claim verification, and tweet verification respectively, while @xcite released a twitter dataset for health risk assessment of covid-19-related english tweets. in this work, we extend an existing raw arabic covid-19 twitter dataset, arcov-19 @xcite , to include manually-annotated tweets to support both claim and tweet verification. to our knowledge, there is no work that released a large manually-annotated arabic dataset for misinformation detection on covid-19 at both claim and tweet levels. the closest work to ours is that done by @xcite and @xcite . however, neither have the propagation networks of the annotated tweets, and they both solely support the tweet verification task. moreover, arcov19-rumors has the largest number of manually annotated tweets. claim verification is a widely studied problem in general. several non-covid-19 datasets exist for claim verification (e.g., @xcite ), however, they either support english language @xcite @xcite or chinese language @xcite . among these studies, some only focused on tweet-level verification @xcite @xcite while others addressed claim-level verification @xcite . there are few initiatives targeting arabic online content, but they support rumors detection (i.e., rumor or non-rumor) (alzanin and azmi, 2019; @xcite , tweet verification without propagation networks @xcite , misinformation detection @xcite , or tweet credibility @xcite . another notable work is a shared task by clef checkthat! lab for claim verification @xcite @xcite . to the best of our knowledge, this is the first arabic dataset that supports both claim and tweet verification on social media using the propagation networks. ## methodology we extended arcov-19, an arabic twitter dataset about covid-19 @xcite , by annotating a subset of the tweets to support research on misinformation detection, which is one of the major problems faced during a pandemic. we aim to support two classes of the misinformation detection problem (with variants) over twitter: verifying free-text claims (called claim-level verification) and verifying claims expressed in tweets (called tweet-level verification), covering two com-mon use cases. to that end, we need to collect covid-19 verified claims, then, for each claim, we need to find, in arcov-19, corresponding relevant tweets, and finally, among the relevant tweets, identify those that are either expressing the claim or negating it (to easily propagate the veracity of the claims to the tweets). this section details how that pipeline was implemented (sections 3.1-3.6). ## experiments and evaluation in this section, we present benchmarking results using sota models on the tweet-level verification task to facilitate future research. we experimented with variant models that either exploit content only, user profile features, temporal features, or propagation structure of the conversational threads for tweet verification. we present our preprocessing approach in section 5.1, the sota models used for benchmarking in section 5.2, and finally a discussion of the results in section 5.3. ## data release in summary, we release the following resources as arcov19-rumors dataset, taking into consideration twitter content redistribution policy: 17 • verified claims: 138 verified claims, each labeled as true or false. ## conclusion in this paper, we presented arcov19-rumors, which is an arabic twitter dataset that supports both claim-level and tweet-level verification tasks given the propagation networks. we released 138 verified claims associated with 9.4k relevant tweets. our dataset covers, in addition to health, other types of claims that were influenced by