A Multi-instance Learning Approach to Civil Unrest Event Detection on T witter (2023)

Abstract: Social media has become an established platform for people to organize and take offline actions, often in the form of civil unrest. Understanding these events can help support pro-democratic movements. The primary method to detect these events on Twitter relies on aggregating many tweets, but this includes many that are not relevant to the task. We propose a multi-instance learning (MIL) approach, which jointly identifies relevant tweets and detects civil unrest events. We demonstrate that MIL improves civil unrest detection over methods based on simple aggregation. Our best model achieves a 0.73 F1 on the Global Civil Unrest on Twitter (G-CUT) dataset.

## introduction social media has become an established platform for people around the world to share opinions and react to socio-political events. platforms enable communication between like-minded individuals and can facilitate offline action. these actions can take the form of democratic expression, such as protests or other types of civil unrest. in the right situation, these events can lead to prodemocracy actions and lead to political changes towards more free and open governments and societies. researchers who study these political movements often turn to social media platforms, especially the globally used twitter, foot_0 to facilitate an understanding of how these events develop @xcite @xcite , which in turn may be used to study pro-democratic movements. as part of that research program, different efforts have considered how to detect or forecast the start of the spread of these movements, including answering what will happen when. detection and forecasting models identify civil unrest either at the macro-(global or country) @xcite or micro-(city) level @xcite using one or multiple data sources, like social media, news, or economic indicators. others study event extraction, whose goal is to extract information about an ongoing or recent event, such as where it happened and who was involved. one of the challenges of developing models for detecting civil unrest events -is there an ongoing event? -is developing models responsive to rapid on the ground changes. social media provides a mechanism for rapid detection; messages can be collected and analyzed as the event unfolds. several studies have examined how twitter can be utilized in a civil unrest detection model @xcite @xcite . while there are a wealth of tweets from all over the world at any given time, not all tweets from a given location are relevant to an event. filtering and identifying relevant tweets remains a challenging problem @xcite @xcite . 2 given the goal of detecting any event, it is important for a detection method to work even without knowing which tweets are necessarily relevant. we follow @xcite and propose a multi-instance learning (mil) approach to detecting civil unrest events at the country-level using twitter data. in mil, examples are grouped and labeled as a group instead of individually (i.e., weak supervision). instead of aggregating all tweets from a given country within a specified time period, we utilize the mil formulation where at least one tweet is relevant while most are not to predict an event. we learn a tweet-level representation using a bert-style model and then group these representations (i.e., instances) into a single bag (country/time period, in our case a single day). we apply this method to the global civil unrest on twitter (g-cut) dataset @xcite , foot_2 which contains 200 million english tweets from 2014-2019 from 42 countries in africa, the middle east, and southeast asia. we focus on english tweets to take advantage of this large dataset and for easier analysis without the need for translation. following @xcite , we use the riots and protests labels at the day level for a country from the armed conflict location & event data project (acled) @xcite as ground truth, where a bag is positive if at least one event occurred in that country on that day. we show that providing the model with all tweets (i.e., not filtering for civil unrest) and allowing it to choose relevant information leads to improved performance on detecting that an event occurred on a specific day in a country, as measured by f1. we support these results with an analysis showing the key tweets identified by the model during prediction. in summary, we contribute the following: ## related work work in civil unrest analysis, also referred to as socio-political event analysis, focuses on event characterization @xcite , protest detection and forecasting @xcite , and event extraction @xcite . prior work on protest detection on the macro level (global or country) is most relevant to our task of country-level civil unrest detection. existing work differs with regard to source data, event ground truth, and methods. most prior work uses news, social media, or economic indicators as features, or sometimes, a combination for a fuller picture. for news data, the goal is often to identify whether an article is discussing a protest or event, as opposed to identifying whether an event is occurring at a location of interest. @xcite use an mil framework to predict whether a news article is discussing a protest, with sentences considered as instances and news articles as bags. they use the sentences that were most informative for the article prediction for further analysis in event extraction. our approach is inspired by this work; we describe it in section 3.2. this setup of classifying the overall articles and the sentences within them was a part of the multilingual protest news detection case 2021 shared task @xcite . similar to sentence-level event identification, prior work has trained models for social media post-level identification of civil unrest discussion @xcite . while not their final goal, @xcite and @xcite incorporate tweet-level models as filters for whether to include a tweet in future steps. their tweet labels were gathered from manual annotation. we omit this filtration step since the proposed mil approach can handle irrelevant tweets. this is an important note since our dataset was not collected with eventspecific tweets as in @xcite . 4 alsaedi et al. ( 2017 ) leverage tweets discussing the london riots (2011) to predict micro (smallscale) events like fires, car accidents, and assaults identified by police records through tweet filtering, clustering, and then automatically selecting a tweet as an event summary. our model allows for a flexible number of posts to be provided as an explanation for each prediction. they also evaluate their system on larger-scale events across the middle east in 2015, clustering with a variety of features like hashtags, sentiment, time, and location, if provided. thus, predictions are on an event-level and not exactly a country-level as ours is (i.e., using only information from a single country). @xcite also extract event-related keywords from clustered tweets, but cluster tweets in an online active-learning mil setting. this mil formulation differs from ours because their "bags" are clusters of similar tweets that are hand-labeled, whereas our bags represent a day in a country. they use a strict formulation where a cluster is predicted as positive (i.e., event) if it has at least one positive tweet (i.e., discussing event/unrest), and negative if there are no positive tweets. this assumption does not work for our setting, where we expect unrest-related tweets even on days where no event occurred. similar to our goal of detecting unrest on the country-level while maintaining tweets individuality, @xcite learn weights on individual tweets for a location that are updated in a temporally streaming fashion. their tweet weights are based on a civil unrest dictionary of terms that correspond to different unrest "stages" (observe, agitation, mobilization, organization, occurrence), along with other temporal and spatial information. our method does not rely on keyword dictionaries and instead, we update embeddings to obtain tweet weights for a given country-day. they also include a spatial component that incorporates information of nearby events. for ground truth they used the global data on events, location, and tone (gdelt) database @xcite and restricted to events with high coverage. in this work we follow other work and use labels from acled @xcite . other systems combine multiple streams of information, such as news and twitter data @xcite @xcite and news and macro-socio political indicators like gdelt and worldwide governance indicators (wgi) @xcite . we focus on twitter because prior work has shown strong civil unrest indicators in past events such as the arab spring @xcite . there is also work on civil unrest prediction in the fields of social and political science. @xcite built a model that incorporates a variety of socioeconomic and political indicators (e.g., infant mortality rate, stability of neighboring countries, and type of government) to predict whether a country in a given year would experience a largescale event like a regime change or genocide. similarly, chenoweth and ulfelder (2015) use structural condition theories (e.g., political opportunity) to predict large-scale non-violent events in a countryyear. while @xcite and @xcite achieved impressive 80% accuracy and 0.75 auc, respectively, curating their rich political and socioeconomic country profiles requires a large amount of domain knowledge compared to our twitter-centric approach. ## multi-instance learning for detecting civil unrest our task is to identify days on which, in a given country, there is a civil unrest event based on twitter data. for each country and day, we acquire a large number of tweets potentially relevant to an event. a model examines this data and predicts whether or not an event is taking place. instead of aggregating the tweets, we propose a multiinstance learning approach that considers whether or not tweets are relevant. specifically, the model assumes that on a day in which an event occurs, only a subset of the provided tweets are relevant to the event. this framing supports explainable predictions, where the tweets deemed relevant by the model can be examined for further context. ## data we use existing datasets for general civil unrest: the armed conflict location & event data project (acled) @xcite , global civil unrest on twitter (g-cut) @xcite , and civil unrest on twitter (cut) @xcite . together g-cut and acled provide the tweets for each day in a country and the label of whether an event occurred on that day. g-cut contains 200 million english tweets from 2014-2019 covering 42 countries in africa, the middle east, and southeast asia. due to the large variety of amount of tweets for each day, we randomly sampled a maximum of 1000 tweets from each country from each day to represent the "bag". we also pruned the dataset further than @xcite to remove spam-like tweets; see appendix a for details. following @xcite , we use the riots and protests labels at the day label for a country from acled. we consider a day in a country as "positive" for a civil unrest event if acled identified a protest or riot on that day in that country. 6 even if multiple events are identified on the same day, that still only counts as one positive example. all other days are negative (i.e., no event). as mentioned in section 3.2, we used cut to train our instance model. ## experiments we evaluate the proposed mil models along with baselines for the civil unrest detection task. we follow prior work and evaluate model performance on f1, precision, and recall @xcite . the model's prediction is marked as correct if it predicts a civil unrest event occurred on a country-day (i.e., bag) that is also identified by the acled ground truth. we use the weighted f1 score due to the class imbalance (roughly 30% positive in the training set). ## results the notable findings are as follows: the mil models outperformed all the baselines. all variations of the mil models outperformed the other aggregation models and baselines. the strongest baseline was one of the more simple, the country-random model, as shown in ## discussion while performance is important, a strength of the mil approach is the identification of which tweets contribute the most to the final prediction, i.e. those with the highest probabilities. these top tweets can be used to explain the model's prediction. we examine the top tweets for a single event below. also, we revisit the mil collective assumption in an analysis of civil-unrest related tweet distributions on days with and without events. ## conclusion our goal was to evaluate how well a multi-instance learning (mil) approach to civil unrest detection on twitter performed to other, aggregated methods. we modeled tweets that occurred on the same day in a country as a bag where each tweet is an instance. we showed that this formulation worked well, achieving an f1 score of 0.73 on de-tecting events identified by acled. the number of instances that contributed to the final prediction for each bag had little effect, but incorporating instance-level supervision in the form of a loss penalty for misclassifying unrest-related tweets did negatively impact overall event prediction performance. since we only evaluated on the civil unrest task, it is unclear if these results are task or data-specific. the remaining challenges are to quantitatively test whether the identified key instances (1) contain event information from the acled event(s) (bag labels) and ( 2 ) can be used in a full event extraction or summarization pipeline, as in @xcite . we showed that this approach is promising for civil unrest detection, but it can easily be adapted for a new task by substituting new tweets and bag labels, such as the detection of other types of events or even stance classification. the common thread between these problems is that tweets are commonly analyzed in aggregate and all are assumed to be directly related to the topic in question, however since twitter data is noisy, this is a potentially incorrect assumption. award n00014-19-1-2316 issued by the office of naval research. the united states government has a royalty-free license throughout the world in all copyrightable material contained herein. any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the office of naval research.