Study of Manifestation of Civil Unrest on T witter (2021)

Abstract: Twitter is commonly used for civil unrest detection and forecasting tasks, but there is a lack of work in evaluating how civil unrest manifests on Twitter across countries and events. We present two in-depth case studies for two specific large-scale events, one in a country with high (English) Twitter usage (Johannesburg riots in South Africa) and one in a country with low Twitter usage (Burayu massacre protests in Ethiopia). We show that while there is event signal during the events, there is little signal leading up to the events. In addition to the case studies, we train Ngram-based models on a larger set of Twitter civil unrest data across time, events, and countries and use machine learning explainability tools (SHAP) to identify important features. The models were able to find words indicative of civil unrest that generalized across countries. The 42 countries span Africa, Middle East, and Southeast Asia and the events range occur between 2014 and 2019.

## introduction citizens utilize public demonstrations, protests, and in extreme cases, riots, to express dissatisfaction over the current political or social state in their country. some of these movements successfully achieve government reforms, many do not. these movements are often driven from the ground up, by grassroots advocacy that snowballs into social and political change. understanding the factors that drive sociopolitical change can help policy makers and advocates petition for and advance their causes. since these causes emerge from the public, studying them requires data on public attitudes, perceptions, and actions around previous movements. in particular, understanding the factors behind civil unrest based on social media posts can reveal important trends. however, while social media data reflects many public attitudes, it's unclear how sites such as twitter capture activities around civil unrest. previous work has considered the task of predicting protests or riots using social media, often with the aid of news sources (alsaedi et al., 2017; ramakrishnan et al., 2014; osborne and dredze, 2014; islam et al., 2020; edouard, 2018; korolov et al., 2016; ranganath et al., 2016; de silva and riloff, 2014; littman, 2018). in this paper we consider how these activities manifest themselves on twitter. specifically, we consider events across a range of countries to understand how twitter is used by different populations. our analysis has two facets: (1) qualitative case studies focusing on two specific events in countries with different civil unrest rates and twitter usage patterns and (2) a quantitative analysis to find indicators of civil unrest that generalize across countries and events. for the case studies, we look at the 2019 johannesburg riots in south africa and the 2018 burayu massacre and subsequent protests in ethiopia. we chose these events because they were significant within those countries and received global attention. their scale should mean they are reflected on social media. furthermore, these countries have high (south africa) and low (ethiopia) twitter usage, and the majority of their tweets are in english. for each event we ask: (1) is the event discussed on twitter? (2) is there any noticeable buildup to the event? (3) who is talking about the event? (4) do tweets reflect the issues that motivated the events? we then take a quantitative approach to find generalizable indicators of civil unrest across time, events, and countries. this focus on generalizability is motivated by previous work that detects/forecasts unrest across countries or regions (alsaedi et al., 2017; ramakrishnan et al., 2014; islam et al., 2020; korolov et al., 2016; ranganath et al., 2016), and the desire to have a single model perform well across countries. for this analysis we introduce n-gram models trained on a new dataset, global civil unrest on twitter, or g-cut, which spans 42 countries from africa, the middle east, and southeast asia, and riots and protests from 2014 to 2019. we evaluate model generalizability by checking not only the overall performance, but also the per-country performance. also, we use ai explainability tools (shap) to analyze the top indicators and ensure they are not country-or event-specific. all data (tweet ids), models, and code will be made publicly available at https://github .com/aadelucia/civil-unrest-case -study . ## data we analyze twitter since it is a widely used social media platform, with many public posts about major events in real-time. for ground-truth data on civil unrest we use the armed conflict location & event data project (acled) (raleigh et al., 2010), a manually curated database tracking civil unrest events. ## case studies we focus on two countries with differing levels of available english tweets and civil unrest levels according to acled: south africa and ethiopia. south africa is a high-data country (i.e., has an abundance of english tweets) with high rates of civil unrest (almost 2,000 events in the 6 year period), and ethiopia is a low data country with relatively lower rates of civil unrest (almost 600 events). these two countries are not representative of all countries, but do provide insight on issues that may arise when using twitter data from high and low data countries for civil unrest detection and forecasting. for each country, we select a single event and ask: (1) is the event discussed on twitter? (2) is there any noticeable buildup in twitter activity prior to the event? (3) who is talking about the event? we limit our study to events with a clear start and end date (e.g., a specific protest) as opposed to general sociopolitical movements with a buildup over years (e.g., a black lives matter (blm) protest in the us versus studying the blm movement). for (1) we look at the ratio of civil unrest-related tweets to all tweets (the relevance classifier discussed in §2) and then proxy the twitter discussion through popular hashtags. the analysis for (2) is important for systems that aim to forecast civil unrest. to approximate event "buildup" we expand our scope from the day of the event to a few days before and after. this analysis looks at the emergence of event-related hashtags and whether twitter is used in event buildup or is more reactionary. for the question of who is talking (3), we use user verification status and rates of likes and retweets as a heuristic for influential users. only users active in the one week time frame are included. after the case studies, we test how well postevent knowledge can help with data filtration. other work uses location, time, keyword, or hashtag information gathered through post-event analysis (littman, 2018; alsaedi et al., 2017). for this experiment, we use the provided acled descriptions of the events to query event-related tweets. ## general indicators of civil unrest in §3 we determined that for two specific events in different countries (south africa and ethiopia), there is signal in the country's twitter activity that a civil unrest event occurred. those case studies acled event descriptions foreign nationals demonstrated and barricaded roads in rosettenville (city of johannesburg, gauteng). some sources reported demonstrators held weapons, in case they were attacked. the demonstration was in response to nationals attacking foreign nationals. students, mainly female, demonstrated at the university of cape town, disrupting classes. the demonstration was in response to the rape and murder of student, calling for more safety for students. most similar tweets is the @saps investigation of the alleged murder of the pretoria taxi driver by undocumented immigrants over yet? glad something is happening but maybe they should have waited longer for investigations... but where are the police?! ## ethical considerations the bias in event information can stem from the language of study, user misinformation, and outside information from inaccurate geotags (tweet location). language can proxy status in certain countries, providing different viewpoints depending on the studied language. also, tweet content is not fact-checked and may contain misleading or incomplete information. this misinformation is not necessarily malicious, but could warp observers' perceptions of events. this is very important to note if twitter is being used to identify event information and details, as opposed to a reputable news source. claims made by twitter users should be attempted to be verified before being treated as fact. misleading information can also come from users outside of the country. inaccurate geotags can come from the use of a vpn, which masks a user's true location. vpns are commonly used to circumvent government-authorized social media shutdowns, especially in countries present in our analysis. @xcite a user could potentially tweet about the situation in their country, but it would be aggregated with the tweets for a different country based on the geotag information. ## conclusion there are ample real-time systems and research using twitter to forecast civil unrest, but not many studies step back and evaluate the manifestation of civil unrest on twitter, especially across countries. in this work we presented a mix of qualitative indepth case studies of events in two countries, and a quantitative large-scale automatic detection of civil unrest events across 42 countries. this evaluation of civil unrest on twitter is not comprehensive, but demonstrates some concerns of using twitter for civil unrest detection across countries. from the case studies of the johannesburg riots in south africa and the burayu massacre protests in ethiopia, we find there is presence of eventrelated discussion for both events. in south africa there was clear development of event-specific hashtags, but the hashtags in ethiopia were very generic, mostly using location words. gradual rise in civil unrest-related discussion before the large event was not able to be identified in either country, indicating users see twitter as a reactionary platform, i.e., share their opinions in real-time or after the event. this is an important finding for work that uses twitter data to forecast civil unrest events days or weeks in advance. while the case studies did not unearth immediately generalizable patterns indicative of civil unrest, our n-gram model trained on twitter data from 42 countries did find generalizable token-based patterns. despite the low f1 score (0.5 for the highest performing non-country debiased model), the top tokens were related to civil unrest and human rights crises. for future work we would expand the case studies to other languages, possibly even comparing event manifestation between languages. also, we would explore explainability techniques for more powerful embedding-based models. we also only focused on large-scale events and did not evaluate the model's ability to detect small-scale events. figure 8 : the language breakdown (in percentages) of twitter data from all 42 countries. all languages that covered less than 10% of the country's tweets grouped as "other". languages were identified with langid (lui and baldwin, 2012).