Civil Unrest on T witter ( CUT ): A Dataset of Tweets to Support Research on Civil Unrest (2020)

Abstract: We present CUT, a dataset for studying Civil Unrest on Twitter. Our dataset includes 4,381 tweets related to civil unrest, hand-annotated with information related to the study of civil unrest discussion and events. Our dataset is drawn from 42 countries from 2014 to 2019. We present baseline systems trained on this data for the identification of tweets related to civil unrest. We include a discussion of ethical issues related to research on this topic.

## introduction from the tomb-builder strikes in 1159 bce egypt 1 to the black lives matter protests in 2020 ce u.s. 2 , humanity has utilized protests and other demonstrations to register grievances and affect change in government or society. while some basic elements of protest remain unchanged for thousands of years, the ability to organize and execute civil unrest activities has been transformed by social media. twitter has played a central role in several recent civil unrest related activities, with the most noted being the arab spring, a series of pro-democratic uprisings, protests, and armed rebellions from 2010 to 2012 in tunisia, morocco, syria, libya, egypt and bahrain, that led to regime changes 3 . some sociologists believe that the distrust in official country press due to censorship led to civilians turning to each other on social media for independent news @xcite . in addition to spreading news, civilians also use social media to share opinions on new policies or recent events (e.g. political debates), and to share information for upcoming events (e.g. protests). the use of social media to plan protests was a key motivator for the planned protest module in the 1 https://www.ancient.eu/article/1089/the-first-labor-strike-in-history/ embers civil unrest forecasting system @xcite . we study civil unrest discussions on twitter for the exact same reason tunisians turned to twitterwe believe civilian voices are an important source of information about the state of a country. @xcite hinted at this with their paper title, "beating the news," for what is news other than reports of the people to the people? while news articles can inform about the presence of civil unrest, we (and other researchers who utilize twitter data) seek to find information about events and opinions from twitter before official news reports @xcite . we discuss ethical concerns of analyzing civil unrest data in §5. a challenge to studying civil unrest in social media is finding it. tweets cover a wide range of topics, and identifying those directly relevant to a protest can be challenging. therefore, to support the study of civil unrest from twitter, we present the civil unrest on twitter (cut) dataset: a collection of 4,381 tweets with annotations for a variety of information related to civil unrest. examples of annotated tweets are shown in the appendix (figure 1 ). tweets are labeled for the following: whether a tweet refers to a protest/strike/riot, if general unrest/dissatisfaction is conveyed, the time of the tweet with respect to the event, the user's stance, whether the event topic is present, if the user intends to participate, and event-specific hashtags. these annotations are useful for a variety of tasks, such as stance detection and event extraction @xcite . as an example use case, we create a model that distinguishes between tweets that discuss civil unrest events from those that do not. this filtration model is a popular processing step in civil unrest detection and forecasting pipelines @xcite @xcite @xcite . we make the following contributions: our dataset and code are available at https:// github.com/aadelucia/jhu-cut. ## related work several studies have examined specific events on social media. examples include tracking information related to public health, such as covid-19 @xcite , and riots, such as the london riots @xcite . several studies have specifically considered building datasets related to the task of civil unrest detection. @xcite label a sample of 5,000 tweets from the middle east from october to november 2015, however they were labelled for general events and not only disruptive events (e.g. weather). @xcite use a list of keywords to filter tweets, and then manually label a sample 10,500 tweets from 178 countries from november 26, 2017 to june 25, 2018 to verify their "informative" vs "uninformative" filtration model. de silva and riloff (2014) incorporated profile information (i.e. organization or user) to predict protest-related tweets from a collection of 6,000 english and spanish disease and civil-unrest keyword-filtered tweets. edouard (2018) created an event2012/fsd corpus with roughly 155k tweets from late 2011 and late 2012, and labelled the tweets as event/non-event related according to a list of events from wikipedia current event portal. korolov et al. (2016) other work uses external information from news articles and other sources to categorize groups of tweets, e.g. country and day, but not to identify individual tweets related to an event @xcite . a drawback of these efforts is that they focus on specific events or locations, rather than producing a more general dataset that can be used to identify civil unrest tweets from new or emerging events. our goal is to produce a more general dataset from a large number of countries (42) over several years (2014 to 2019). ## dataset creation we present a dataset to support the study of civil unrest on twitter. our data set contains english tweets with annotations related to civil unrest produced through annotations from amazon mechani-cal turk. twitter data we selected a sample of tweets from 2014 to 2019 that were collected from the twitter streaming api based on filters to collect geolocated data. our geolocation filters included african, middle eastern, and southeast asian countries (see we select tweets based on their inclusion of an english language keyword related to civil unrest. using an approach similar to that of @xcite and @xcite , we used a combination of manual and automated methods to create a large set of 709 keywords, which include terms such as "unemployment," "police," and "extremist." the full list of keywords appears with the released dataset. in total, we include 4,415 tweets in the dataset for annotation. foot_1 34 tweets are removed from the dataset after annotators report them as non-english, resulting in the final dataset of 4,381 tweets. annotations our goal was to collect a wide range of annotations that could potentially be helpful in the study of civil unrest on twitter. annotators were asked several questions about each tweet: questions 1(b)-1(e) were only answered if the answer to (1) was 'specific' or 'nonspecific', and question 1(a) was only answered if (1) was 'specific.' a screenshot of the survey is in the appendix (figure 2 ). ## civil unrest classification using our annotated dataset we created a baseline model for predicting if a tweet was related to civil unrest, i.e. predicting if the label was "yes, a specific event" or "yes, in a non-specific fashion" for question 1 (690 of 4,381 tweets or 16%). the resulting classifier can be used to identify large amounts of data around specific events for further study @xcite @xcite @xcite . we considered two logistic regression classifiers. 1) unigram counts of all tokens in a tweet. 2) counts of the civil unrest keywords only. both of these methods used the scikit learn implementation of logistic regression and countvectorizer @xcite . the unigram models were regularized with l2 loss and were evaluated with 5-fold cross validation (same folds across experiments). all methods preprocessed tweets with the littlebird implementation of the bertweet tokenizer (delucia, 2020). this tokenizer was used to allow easy extendability and future comparison with a bertweet-based model. features f1 precision recall keywords 0.782 0.894 0.697 unigrams 0.775 0.892 0.687 regression model outperformed the unigram model (0.782 f1 vs 0.775 f1). this out-performance is slight, but still worthwhile, especially since the keyword model is smaller (only 700 features instead of 15k) and converges faster (roughly 50 iterations versus 100). additionally, since the civil unrest keywords were used to sample tweets their inclusion as features adds further information to the keyword-based model. ## ethical considerations many studies of event detection on twitter have explored the implications for public health (e.g. spread of infectious diseases) @xcite and natural disasters @xcite , which offer benefits in combating harmful events. however, civil unrest presents a more complex cost benefit trade off as it can yield insights into what issues are most important to a population, but can also be used to monitor or track individuals who participate in these events. deciding what constitutes civil unrest versus unjustified violence requires a value judgement, which could easily degrade into weaponizing against dissenting opinion. additionally, non-government actors could use predictions of unrest to squash disapproving voices. moreover, frequently, marginalized voices have found solace and organization using social media @xcite and predicting civil unrest could unintentionally lead to actions such as further policing of overpoliced communities. with this in mind, we should consider twitter data not just as text data, but as people. several proposals for protecting people including avoiding reverse identification @xcite ) and data anonymization tools @xcite . we believe that the numerous studies of civil unrest that further our understanding of complex societal issues are convincing evidence that there is much to be gained from developing data resources in support of this topic. at the same time, we must remain vigilant in our evaluation of research efforts to ensure they remain supportive of these goals. ## conclusion we have presented the civil unrest on twitter (cut) dataset and a baseline classifier trained on the data for identifying tweets related to a civil unrest event. future work can build on our multifaceted annotations to expand the study of communities and how they express concern about complex societal issues through civil unrest.