Kawarith: an A rabic T witter Corpus for Crisis Events (2021)

Abstract: Social media (SM) platforms such as Twitter provide large quantities of real-time data that can be leveraged during mass emergencies. Developing tools to support crisis-affected communities requires available datasets, which often do not exist for low resource languages. This paper introduces Kawarith a multi-dialect Arabic Twitter corpus for crisis events, comprising more than a million Arabic tweets collected during 22 crises that occurred between 2018 and 2020 and involved several types of hazard. Exploration of this content revealed the most discussed topics and information types, and the paper presents a labelled dataset from seven emergency events that serves as a gold standard for several tasks in crisis informatics research. Using annotated data from the same event, a BERT model is fine-tuned to classify tweets into different categories in the multi- label setting. Results show that BERT-based models yield good performance on this task even with small amounts of task-specific training data.

## introduction recent studies have revealed the significant role of twitter during emergency events @xcite @xcite . in particular, twitter messages have been shown to contain valuable and timely crisis-related information that serves to enhance situational awareness and supports humanitarian response efforts. accessible datasets enable crisis informatics researchers to leverage this valuable user-generated data. the present study focuses on crisis-related arabic twitter content. @xcite reported that arabic is one of the most used languages on twitter. nevertheless, this content has attracted relatively little research attention. arabic dialects, which differ in their morphology, phonology and syntax, have limited resources. the lack of sufficient resources makes developing natural language processing (nlp) tools utilising arabic sm messages more challenging. to the best of our knowledge, there is no accessible arabic twitter corpus of multi-crisis events. for that reason, we assembled a large-scale crisis-related corpus foot_1 of more than a million tweets from 22 middle east crises involving different hazard types. using nlp techniques, our detailed analysis of the corpus identified the main information types shared on twitter, and we used these to label a subset of the data. the dataset can facilitate information extraction from sm to support situational awareness and enable decision-makers to respond effectively during mass emergencies. the paper makes a number of contributions to the existing literature. ## related work in the publicly available twitter crisis-related corpora in the crisis informatics literature, supervised approaches have been used to extract messages of interest from human-labelled datasets, including actionable information that enhances situational awareness. in the present context, accessible unlabelled corpora have been used for a range of purposes, including key topic extraction, social analytics and public sentiment assessment during emergency events. @xcite shared two annotated datasets labelled for two tasks: identifying informative messages that contribute to situational awareness and assigning these to information categories such as cautions and donations. the first set, iscram2013, comprises 3617 tweets about the joplin tornado, and the second includes 2987 tweets collected during hurricane sandy in 2012. @xcite collected and published 2k spanish tweets from the chilean earthquake in 2010, which were human-labelled as relevant or irrelevant to the crisis. @xcite created the sositalyt4 dataset, comprising 5.6k italian tweets collected during four natural disasters in italy between 2009 and 2014, which were annotated for the purposes of damage assessment as damage, no damage or irrelevant. one of the largest labelled and publicly accessible crisis datasets is crisislex, which incorporates two collections: crisislext6 @xcite and crisislext26 @xcite . cri-sislext6 includes 60k english tweets from six crisis events during 2012 and 2013. the messages were annotated by relatedness to the event in question (related vs. not related). crisislext26 contains tweets collected during 26 crises that also occurred in 2012 and 2013. about 28k posts were annotated in terms of informativeness (informative vs. uninformative), information type and information source, and most of the subsequent research followed the proposed taxonomies. @xcite released crisisnlp, a corpus of 52k annotated tweets collected during 19 crisis events between 2013 and 2015. the tweets were manually labelled by information type by paid workers and volunteers. for the purpose of coordinating humanitarian response efforts, different annotation schemes (labels) were used for different event types. most messages in crisislex and crisisnlp were written in english but they also include posts written in other languages such as italian, spanish and french, and both datasets have been widely used in the twitter crisis detection literature. @xcite created crisismmd, a multimodal dataset of 16k english tweets with attached images collected in 2017 from seven natural disasters. entries were labelled according to three dimensions: informativeness, information categories and damage severity. trec-is foot_3 (mc-creadie et al., 2020) provided crisis-related twitter datasets from 48 past emergency events, which were manually labelled by information types and priority levels. @xcite shared a dataset of 4k arabic tweets that refer to four highrisk floods in three middle eastern countries during october and november 2018, annotated by relatedness and information type. @xcite presented flodusta data, comprises 9k tweets from floods, dust storms, and traffic accident events that occurred in saudi arabia. the messages were labeled by event type and time of occurrence (historical, immediate, future or irrelevant). to identify different types of eyewitness report during crises, @xcite published a labelled dataset of 14k english tweets gathered during four natural disasters. @xcite built a dataset of 13k french tweets collected from different ecological crises, labelled for relatedness, urgency and intention to act as information types that reflect categories in other studies. several large published twitter crisis corpora that provide no human labels can only be reused by reassembling the data from tweet ids. unlike small datasets, twitter's developer policy did not allow the distribution of tweets for large-scale datasets @xcite . examples include 6m geo-tagged tweet ids from hurricane sandy @xcite , 7m english tweets from hurricane harvey (phillips, 2017) and 35m tweet ids related to hurricanes irma and harvey. alam et al. (2018b) also published a twitter corpus of more than 8m message ids collected in 2017 from hurricanes irma, harvey and maria. more recently, research interest has focused increasingly on the covid-19 pandemic, and several crisis informatics studies have released large-scale twitter datasets. while some of these are limited to a single language like english @xcite or ara-bic @xcite @xcite , others have created multi-lingual datasets @xcite @xcite @xcite . @xcite created epic, an epidemic corpus consisting of 20m tweets related to various diseases. to contribute to this body of research, the present study introduces a large-scale arabic twitter corpus and a subset of tweets from seven crises, manually labelled in terms of their content. in contrast to previous presented work, we consider a multi-label annotation scheme, which is described in detail below. ## crisis events and data collection the kawārith corpus comprises arabic tweets from 22 crisis events that occurred between october 2018 and september 2020. kawarith focuses on high-to medium-risk events that are most likely to trigger substantial twitter activity and encompasses a wide range of hazard types, including floods, shootings, bombing, wildfires, pandemics, sandstorms and explosions. using the twitter search api foot_4 , these data were collected iteratively. during each crisis, we began by using trending crisis-related hashtags or keywords as query terms. if no relevant trends were found during this initial phase of data collection, we used the api to search twitter using a logical and combination of the terms hazard type and crisis location. additionally, as an alternative search term, we linked the two terms in hashtag form, as we observed that people tended to use crisis-related hashtags like # "#kuwait floods" and # "#corona pandemic" for the kuwait floods and covid-19 crises, respectively. this first step led us to crawl an initial set of tweets, which was manually inspected to identify any new hashtags that related strongly to the event. the dataset was then expanded by tracking these hashtags, and this step was repeated until no new relevant hashtags could be found. finally, we updated our query to include all manually selected keywords linked by logical or to extract crisis messages in the next timeframe. concurrently, we updated the query with any new relevant keywords emerging as trends on twitter. keywords could be in the form of observed hashtags, phrases or single words. we adopted a cautious approach to keyword selection, often using event-specific (discriminative) terms rather than hazard descriptors such as "heavy rain" or "drowns" to reduce false positives-especially for flood events, which usually occurred simultaneously. terms such as country name hashtags were generally disregarded, especially if the event had little impact on that country. the decision to use such terms as search queries was generally based on recently retrieved tweets; a candidate term was added to the query if it retrieved event-relevant messages. importantly, as we favoured precision rather than recall, many relevant tweets were likely to be missed. however, we are satisfied that our data captured the key aspects of the crises. in the case of covid-19, we tracked only nine keywords referring to the crisis by name because the event has triggered many other topics (such as conspiracy theories) that were not immediately relevant to our purposes. as our study focused on building an arabic dataset, data collection was confined to tweets that twitter tagged as arabic, and this language parameter necessarily excluded tweets in other languages. in lebanon, for instance, people also tweeted in arabizi (romanised arabic), english and other languages, which may account for the relatively small volume of arabic data crawled for those events despite their severity and relevance. data gathering continued from the first day of a crisis until the end, which we chose to define as the point at which it no longer triggered conversations on twitter and related keywords no longer appeared in the twitter trending list for that geographical area. we treated long-term crises like the covid-19 as exceptions to this rule. in the case of covid-19, data collection was delayed until near the peak of the epidemic in the middle east. in other words, the goal was to obtain representative rather than comprehensive samples. lists of the query terms and collection dates have been included in the published data. in total, we 4 data description ## tweets classification to benchmark the dataset, we fine-tuned the arabic bidirectional encoder representation from transformer (arabert) base model @xcite . for reproducibility, we split the data into stratified train and test sets (80% and 20%, respectively). we fine-tuned a bert model for each event using its training data by adding a linear classification layer on top of the bert model, preceded by a dropout layer of a probability (0.2) to prevent the model from over-fitting. the learning rate of adam optimizer was set to 5e-5 and the loss function to binary cross-entropy. we set the maximum sequence length to 60 tokens as the longest tweet in kawārith has 60 words. all models were trained for 5 epochs, which was empirically chosen. we experimented with batch sizes of 8 and 32. each experiment was repeated three times with random seeds of {a, b, c} because different seeds can yield considerably different results @xcite . the hl captures the fraction of labels that are incorrectly predicted. the binary covid-19 data was evaluated using the accuracy and macro f1. the average scores of the three runs for each experiment are displayed in results show that bert achieved good accuracy and micro-f1 scores despite the relatively small training data used to fine-tune the models. obtaining low hl scores indicates that a few labels were incorrectly predicted. while achieving a high macro-f1 is challenging with a skewed class distribution, bert achieved macro-f1 scores higher than 70% in almost all cases without handling the data imbalance problem. investigating different data augmentation techniques to improve the models' performance is left for future work. generally, we found that fine-tuning in batches of 32 yields small gains in performance. in the case of the cairo bombing, batch size of 8 achieved better results in all random runs, which we relate to the small number of training data (only 565 examples). when repeating the experiments for the other events using less than (randomly selected) 600 training examples, we found that it is more effective to fine-tune the models in batches of 8 instead of 32, which resulted in improvements in most cases. looking at the classification errors of the best models, we found that the models' mispredictions are not associated with a specific information category, and error types vary across events. we observed that many of the irrelevant tweets (especially from flood crises) were mistakenly classified as opinions. most of these misclassified irrelevant tweets are ambiguous messages that express negative opinions about some topics that could be related to the crisis in some way. it is important to note that the two main annotators disagreed on most of such ambiguous posts. for many events, the models confused some messages from 'opinions & criticism' class with other classes. classifying covid-19 data obtained high accuracy and f1 scores. performing such binary classification to filter out spam and other irrelevant posts prior to message categorisation could alleviate errors of classifying them as opinions. for future work, we will explore the bert's performance when classifying cross-event data. ## conclusion this paper introduced kawārith, an arabic twitter corpus for 22 crises. we also reported a preliminary analysis of tweet content and provided a gold-standard multi-label dataset comprising 12k unique tweets. we believe this corpus can be leveraged for several tasks, including crisis detection and crisis type classification. assigning messages to categories in order to identify informative posts can enhance situational awareness and assist emergency responders in organising effective relief efforts. the labelled dataset can also be utilised to gauge public opinion and sentiment during crises.