Socio-political Events of Conflict and Unrest: A Survey of Available Datasets (2024)

Abstract: There is a large and growing body of literature on datasets created to facilitate the study of socio-political events of conflict and unrest. However, the datasets, and the approaches taken to create them, vary a lot depending on the type of research they are intended to support. For example, while scholars from natural language processing (NLP) tend to focus on annotating specific spans of text indicating various components of an event, scholars from the disciplines of political science and conflict studies tend to focus on creating databases that code an abstract but structured representation of the event, less tied to a specific source text.The survey presented in this paper aims to map out the current landscape of available event datasets within the domain of social and political conflict and unrest – both from the NLP and political science communities – offering a unified view of the work done across different disciplines.

## introduction and background like in most social sciences, political scientists started to rely more and more on quantitative data to empirically test their hypotheses during the course of the 20th century. @xcite observes a rapid increase in the use of quantitative data, from 11.6% of political science articles in @xcite to satisfy this demand for numerical data, researchers started manually collecting large databases of politically significant events from news journals @xcite . these databases contain structured abstract descriptions of real-world events, enabling researchers to perform large-scale analysis. from an nlp perspective, these sorts of databases can be viewed as the desired output of the event extraction task. event extraction models are trained on natural language texts, such as news or wikipedia articles, annotated with event information at the token-level. yet, while information extraction was originally motivated by practical endeavours @xcite , modern event extraction is more closely associated with linguistic formalisations of sentential semantics and natural language understanding @xcite . when we look at both modern socio-political event databases and annotated nlp datasets, we observe several discrepancies that make annotated datasets less suited for the evaluation of sociopolitical event extraction systems. a first discrepancy pertains to the link precision between text and events. while the events encoded by database approaches commonly reflect information scattered in entire documents (typically one or multiple new articles), nlp events tend to be defined by word or phrase-level annotations tied to specific spans of text in a given document. a second and closely related discrepancy is what we refer to as the abstraction gap. for political science, the text of news articles is but a clue to what happened. sociopolitical databases purpose to contain information about what actually happened in the real-world, which can only be elucidated through a combination of sources and expert knowledge. moreover, the recorded events are typically defined within the context of the phenomena, theories, or research goals that are explored. in nlp, events are often defined based on linguistic motivations, meaning they are defined and specified within the text based on linguistic structures, patterns, or features present. the events defined in the text annotations of nlp datasets are usually more atomic and granular compared to the more aggregated and high-level events typically found in database resources. a third discrepancy has to do with source text availability, which is in turn closely tied to the underlying purpose of the data resource. while the main point of socio-political event databases is simply the set of events themselves, i.e. the actual information that is recorded, the text annotations found within nlp, in contrast, are meant to enable training and/or testing of event extraction systems, i.e. systems that can map text into structured representations like those of the annotations. in nlp, therefore, it is generally seen as vital to make the annotated texts freely available, whereas it is significantly less common that the text sources used to build socio-political databases are shared. this has the unfortunate consequence of making many event databases not as directly applicable for nlp research as they might have been. a fourth discrepancy is related to the account of temporal dynamics. socio-political event databases describe an evolving world, while annotated event extraction datasets are typically comprised of independent and identically distributed samples. several surveys in nlp describe annotated event datasets together with methodologies and techniques approaching the task of event extraction @xcite @xcite . similarly, multiple articles describe socio-political databases, often with a focused comparison within the same domain, such as protest events @xcite or violent events (hammond and weidmann, 2014; @xcite . however, comprehensive studies linking the two fields together are notably lacking. considering the extensive data sources available, our survey does not aim to be exhaustive. our primary focus is on central databases used in the social sciences and prominent annotated datasets in nlp concerning conflict and unrest. in structuring this survey, we classify datasets according to their purported goal. we start in section 2 with datasets created for the main purpose of studying the recorded events themselves. we refer to these as socio-political event databases. the section will start by introducing manually annotated databases before we introduce databases created using automated methods. this naturally leads to section 3 on annotated event datasets from the field of nlp covering socio-political events. the key characteristic of the datasets in this section is that they contain text-span annotations with the purpose of training and evaluating machine learning models for the event extraction task. we then describe and analyse the gap between the two types of event data and discuss works that can be seen as early attempts to bridge this gap in section 4. finally, we give special attention to our ethics section, as biases in the selection and description of datasets are critical when political analyses are derived from them. as a note on terminology, while writing this survey, we opted to use the vocabulary of nlp, but also to make the parallel between the practices found between the two fields clearer. instead of speaking of annotation, political scientists prefer the term of coding, which usually refers to manual annotation performed by human experts, but can also include machine coding, which refers to the automatic annotation of text by algorithms. sociopolitical events usually involve one or more actors, those are entities, often states, armed groups, or other politically relevant organisations. finally, the process of extracting political events from text is described in a codebook, which can be seen as similar in purpose to an annotation guideline. ## socio-political event databases early on, @xcite noted the necessity of building databases of politically relevant events to better our understanding of international politics. in contrast to annotated datasets geared towards training and evaluating systems for information extraction, these types of databases are built solely for the knowledge they encode, without much importance given to an underlying source text. the source texts are typically only included in the form of a reference for checking the validity of the event or indicating its provenance. however, most events recorded in such databases could, in principle, be automatically extracted from published texts. foot_0 following this observation, there was an attempt to automatically extract these databases from news feeds in the late 1980s. these efforts resulted in the kansas event data system (keds; @xcite , extracting events from reuters. this initiated the advance of machine-coded databases, which parallels the development of event extraction systems on the nlp side. in this section, we describe important databases of socio-political conflict and unrest. while the focus of this survey is on data rather than modelling, we do briefly touch on methodology when we discuss the automatically extracted databases, where modelling and data are inherently intertwined. the main manually annotated databases included in this section are listed in ## text annotation for event extraction on the nlp side, annotated datasets are created for the purpose of training models, shaping their design and annotation to align with the event extraction task's approach. event extraction has been a central task in nlp, dating back to the message understanding conferences (muc) series in the 1990s. initially, annotating event participants was formulated to fit a template-filling task, where information from a document is to be structured into a predefined set of fields such as finding the victims, time and location from a terrorist attack report. following these early attempts, the highly influential automatic content extraction (ace) program released manual event annotations for text spans at the sentence level, performed jointly with annotation of rich information about entities, temporal expressions, and relations between entities. below we describe these in more detail and also compare them with more recent annotation efforts. the nlp datasets covered in this survey are summarised in while looking at the 1990s muc datasets, it is striking how closely they resonate with current socio-political event databases compared to modern nlp annotated datasets. the evolution of template filling into event extraction is not clearly defined, and similar models are used for the two tasks @xcite . indeed, both of them capture a semantic relationship between entities, as described by the template or event schema. two other closely related tasks are relation extraction -which usually focuses on binary templates, often in the context of knowledge bases -and semantic role labellingwhich usually focuses on the argument relations conveyed by specific predicates. even though all of these tasks can be relevant to socio-political event databases, in this section, we only focus on annotated datasets for event extraction and templatefilling, describing them in chronological order. the message understanding conferences (muc; @xcite , held from 1987 to 1997 and funded by darpa, are regarded as pioneering efforts in generating annotated datasets for information extraction. the conferences operated as shared tasks, where each muc is associated with a designated dataset covering the corresponding information to be extracted and prepared by human annotators for training purposes along with a task definition. although muc maintains mainly a military theme, the various datasets focus on different types of events. the first two conferences centred on military messages from the tactical navy domain. in muc-1 the participants were provided with merely 10 paragraphs as data without any formal evaluation. building on muc-1, muc-2 introduced a dataset with 130 messages and 10 elements to be extracted, such as event type, agent, time, place, and the effect of the event @xcite . following the initial conferences, muc-3 and muc-4 introduced annotated datasets focused on terrorist events in central and south america, reported by the foreign broadcast information service. these iterations of muc marked a shift by increasing the complexity of the task, both by including several event and argument types, but also by moving from extraction of information from simple and short military messages to longer texts with more complex language. muc-4 includes 4 event types arson, attack, bombing, kidnapping, with the 4 arguments roles perpetrator, instrument, target, and victim, which are shared across event types. additionally, the datasets increased in size, with respectively 1 400 and 1 700 news articles for muc-3 and muc-4. the last two instalments, muc-6 and muc-7, shift the focus towards domain-independent annotations, targeting named entity recognition, coreference resolution, and relation identification. this transition also includes an expansion to more languages. interestingly, this shift was accompanied by a return to smaller training datasets, comprising only 100 documents. muc-6 consists of events involving high-level officers joining or departing from companies, while muc-7 targets satellite launch events, with event arguments such as date, country of launch, and payload information. these were followed by the automatic content extraction (ace) program. the event annotation in the ace tradition has become a de facto standard for the evaluation of event extraction systems in the field of nlp. @xcite provides manual annotation for entities, relations, and events for joint evaluation of multiple ie tasks and in multiple languages (ace05 in english, chinese, and arabic). the annotations distinguish specific text spans indicating the event trigger and associated arguments of an event at the sentence level. an event trigger is typically the word(s) in the text that most clearly describes an event, such as "bomb", which evokes an attack event in the example sentence "u.s. forces continued to bomb fallujah" where "u.s. forces" is the associated attacker argument. ace annotates 8 general event types, e.g. life, conflict, transaction with 33 subtypes (e.g. conflict.attack) and 22 argument roles, e.g. attacker, agent and recipient. of particular relevance in the current setting are the conflict event type (with subtypes attack and demonstration) as well as the life.die and life.injure event types. more recently, the entities, relations and events (ere) annotation effort @xcite has con-tributed both data and annotation guidelines for event extraction purposes. from the light ere to rich ere datasets, the ere effort has evolved from lightweight annotation automating the ace guidelines to more complex treatment of entities and events aimed at paving the way for event coreference at the document-level. the rich ere annotation scheme extends on that of ace, annotating 38 event subtypes under 9 main event types, including more fine-grained event subtypes in the movement, contact, and transaction event types. in light ere, an event trigger can be associated with only one event. still, in rich ere, an event trigger can be annotated for more than one event due to correlations of different event types. for instance, an attack event and an injure event can share the same event trigger; it is natural that when a person is attacked, the person is also injured. in light ere, only asserted events are annotated; in rich ere, apart from assorted events, events that did not actually occur are also annotated, hence annotating event modality. the massive event detection dataset (maven; @xcite is introduced to provide a largescale annotated event dataset in the general domain, covering 168 event types. maven follows the ace terminology, targeting events at the sentencelevel, and consists of event-related articles from english wikipedia. framenet frames @xcite are used to derive event types, with the lexical units serving as the corresponding triggers. automatic pos-tagging and heuristic methods are used to narrow down trigger candidates and the corresponding event type candidates to aid human annotators. in maven, the event types follow a hierarchical schema resembling a tree structure, prioritising the most detailed event types. if no finegrained event type aligns with the event, the annotators resort to more general event types. for example, the most coarse-grained event type action includes the event subtype violence, which again contains subsubtypes such as killing, attack, terrorism, and military operation. in the context of social and political conflict and unrest, the event types terrorism, kidnapping, violence, use firearm, military operation, and attack are especially relevant event types. @xcite presents wikievents, a document-level annotated dataset based on wikipedia articles and their referenced news articles. the annotations resemble ace, but expand the number of sub-events from 33 to 67 following the kairos ontology. additionally, it incorporates a more fine-grained event-type hierarchy. for instance, whereas ace identifies the event type and subtype such as conflict.attack, wikievents introduces event types at three levels, such as conflict.attack.detonateexplode. furthermore, @xcite expand their annotations to include events that extend beyond the sentence boundary, capturing event arguments occurring in sentences lacking an explicit event trigger. apart from the conflict.attack events, the dataset includes event types such as life.die and conflict.demonstrate, each with subtypes that are relevant in the socio-political domain context. human annotators label event types, event mentions (triggers and arguments), and event coreferences across sentences in the document. the docee dataset @xcite is the largest document-level annotated dataset containing 27 485 documents and covers a wide range of event types in the socio-political domain, including armed conflicts, riot and protest. it includes two types of events, historical events, defined as events with their own wikipedia page, and timeline events, which are news events organised in chronological order on wikipedia. the wikipedia article is annotated for the historical events, while the corresponding news article is used for the timeline events. each document is manually given an event type based on the title and then annotated with event arguments from the event type schema. for example, the event type protest is annotated with arguments date, location, protesters, cause, slogan, method, arrested, government reaction, casualties and losses, and damaged property. the recently released mee dataset @xcite provides event-annotated data for eight typologically diverse languages (english, spanish, portuguese, polish, turkish, hindi, korean and japanese). the data is based on wikipedia articles under the subcategory event from a number of different domains (e.g. economy, politics, crimes and military). the annotation scheme is based on the ace guidelines and its 8 event types, however, limit the set of annotated subtypes to 16. unlike ace, the articles are split into 5-sentence segments and argument relations may span across the full-text segment. for the most relevant category in the current context, the dataset only includes the conflict.attack, life.die and life.injure event types. ## bridging the gap in this section, we start by highlighting the main obstacles to transferring event extraction nlp expertise to the automatic extraction of socio-political event databases. one obstacle currently being addressed in the field of nlp is the restriction of events to single sentences. as we show in section 3, document-level event extraction datasets are now starting to reemerge. in the second part of this section, we describe datasets that establish bridges between political science databases and annotated datasets. token-level annotations to facilitate model training, nlp event extraction datasets include token-level annotations delineating which words correspond to specific event triggers or arguments. on the other hand, manually coded socio-political event databases do not usually include this information, with the exception of the ner-automated speed database. therefore, training machine learning models from socio-political databases requires either token-level annotation efforts or weakly supervised learning techniques. alternatively, and perhaps more interestingly, one could directly prioritise research on end-to-end learning of document-level event extraction. source availability regardless of the learning strategy used, a prerequisite is having available source texts, preferably in a free and open manner. the news articles used to code socio-political event databases are usually unavailable, mostly due to copyright restrictions. this significantly limits the appeal of these datasets within the nlp commu-nity. abstraction gap furthermore, all the annotated datasets described in section 3 solely capture the mapping between text and structured information, while the socio-political databases described in section 2.1 attempt to record whether the event actually occurred in the real world. in the first case, only linguistic knowledge is necessary, even when encoding event modality. in the second, sociopolitical event databases require expert knowledge to evaluate and corroborate what is conveyed in the text. this implies that future research on learning to automatically extract document-level events should also address how to incorporate domain knowledge. temporal dynamic the socio-political databases describe an ever-changing situation with new actors regularly appearing and engaging in new conflicts. on the other hand, annotated datasets tend to be more stationary, with little to no temporal variation in the distribution of events. in a way, the efforts to automatically create sociopolitical event databases overlook these issues because they tend to rely on older, rule-based models that do not necessitate data supervision. they fall into the abstraction gap by overcounting events, extracting from all uncorroborated news. moreover, as they are typically not disclosed to the nlp community, there is no requirement to publish their source data. 3 this comes at the cost of reliability. some previous work has made efforts to bridge this gap between socio-political event databases and annotated event datasets. the muc datasets, detailed in section 2, represent the initial strides in this effort. we will here describe some of the more recent approaches. the iraq body count corpus (ibc-c; žukov-gregorič et al., 2016) is introduced to automate the annotation process for the iraq body count project @xcite discussed in section 2.1. the corpus provides event annotations for whole documents, where each document contains references to one or multiple events. the annotations for the ibc-c are created through a form of distant supervision @xcite , using different pattern matching and semantic functions to create named entity labels corresponding to ten argument roles, such as fatality numbers, named individuals, and location. ibc-c provides token-level annotations, somewhat addresses the abstraction gap and can capture the temporal dynamic of the evolving war. unfortunately, the complete dataset is no longer available due to copyright restrictions (and potential privacy concerns). the global contentious politics database (glo-con; @xcite @xcite is a partly automated protest event database. part of the data used to train the event extraction model is referred to as glocon gold and is freely available upon request. 4 it includes manually annotated datasets for three sub-tasks: document classification, sentence classification, and event extraction. it encodes five specific event sub-types: demonstrations, industrial actions, group clashes, armed militancy, and electoral mobilisation. regarding the concerns we identified, the dataset includes token-level annotations, is associated with source texts, and preserves the temporal dynamic of the political system. however, although future or hypothetical events are not annotated, these types of events can be recognised from linguistic cues alone, leading to continued susceptibility to the abstraction gap. subsequently, the glo-con gold dataset was extended to define the case 2021 and 2022 shared task 1 on protest news detection @xcite . compared to glocon, the shared task datasets include more source articles and define an additional subtask: event sentence coreference identification. finally, shared task 2 in 2021 @xcite and 2023 @xcite attempt to bridge the gap more directly as they use data annotated following the acled codebook for evaluation. the 2023 task 2 tackles the prediction of battle events from social media messages in the russo-ukrainian war. on the prediction of whether a prio-grid cell contained a battle event, the two systems submitted for the task reached f 1 scores of 0.04 and 0.152, demonstrating the considerable amount of work that lies ahead. ## limitations due to space constraints, we needed to limit the number of datasets discussed. we strive to high-light datasets that are both central and relevant to the domain of political conflicts and unrest and showcase the evolution of practices in their respective fields. however, most of the datasets we selected are based on english-language news, even when used to analyse the political situation in non-english-speaking countries. some notable mentions that could not be included for relevancy or duplication concerns are poldem @xcite , mar @xcite , icbe @xcite , ucdp vpp (svensson et al., 2022), pitf's wad @xcite , rams @xcite , etc. additionally, there has been a rise in annotated datasets made from user-generated text not encompassed in this survey, such as the twitter-based datasets on civil unrest cut @xcite and g-cut @xcite . ## ethics working on event data concerning sensitive topics such as armed conflict, protest data, or other sociopolitical events necessitates a high degree of ethical consideration and responsibility. the fact that the main source for several of the socio-political event databases is news articles, one should raise awareness of the inherent bias when reporting on these topics in the news. in the context of creating databases for conflict events based on media reporting, @xcite highlights the importance of awareness towards both description bias, meaning errors in how conflicts are reported, and selection bias, meaning which conflicts are reported, and more importantly, those that remain unreported. regarding selection bias, @xcite suggests that researchers can solely make assumptions about the representativeness of the reported news, while for description bias, efforts should be made to mitigate and reduce potential bias in the extracted events. while similar biases can be present in manually annotated databases @xcite , both description and selection biases from media sources can be partly mitigated using human experts to assess the validity of the reported events and or seek out sources to confirm the information. however, these types of biases do not seem to be addressed for annotated datasets in nlp. another concern is that this paper describes a dataset (ibc-c) that has been retracted due to copyright restrictions and is no longer accessible be-cause of the mishandling of sensitive personal data. other datasets are still accessible but do not clarify the handling of personal data and/or licences for redistributing data. access to data while upholding copyright and privacy considerations is crucial to ethical research practice. including these datasets in this work does not represent endorsement but is necessary to discuss different approaches and challenges associated with socio-political event data. an important consideration when dealing with annotated datasets and databases involves the annotators' exposure to distressing or harmful content. constantly engaging with descriptions of conflict and violence can lead to desensitisation, emotional numbness, and potential emotional and psychological distress. recently, more attention has been directed toward the impact of secondary or vicarious trauma and the psychological well-being of annotators, content moderators, and others handling harmful content @xcite @xcite . however, strategies and specific actions to alleviate potential risks for annotators, such as providing psychological support, remain limited or inadequately addressed in the datasets described in this paper. we strongly advocate for a more focused approach on supporting annotators to mitigate the effects of exposure and encourage leveraging existing datasets in research on automatic event extraction instead of creating new event datasets in order to minimise exposure. we now address the concern of misuse and misinterpretations of socio-political event data. for instance, the glocon dataset strives to use neutral terms to describe different actors, e.g. using militant, instead of terrorist. other datasets vary in their approaches when dealing with language that might be insulting, marginalising, or criminalising. the extent of this handling often depends on factors such as the use of standardised actor lists and whether the datasets are manually annotated by experts. notably, in annotated datasets used for nlp, with a one-to-one mapping between text-span and label, this issue remains unaddressed. the vocabulary used to describe individuals and groups, particularly those from minority communities, holds the dual power to shape our perceptions of said groups and might impact the reliability of extracted events and subsequent analyses derived from event databases. finally, some datasets described in this work may contain fine-grained details about individuals, organisations, or groups, which can be used mali-ciously. some datasets, such as glocon enforce responsible data use and seek to mitigate unethical usage by assessing the declared research intentions before granting access to the dataset @xcite .