C onfli BERT : A Pre-trained Language Model for Political Conflict and Violence (2022)

Abstract: Analyzing conflicts and political violence around the world is a persistent challenge in the political science and policy communities due in large part to the vast volumes of specialized text needed to monitor conflict and violence on a global scale. To help advance research in political science, we introduce ConfliBERT, a domain-specific pre-trained language model for conflict and political violence. We first gather a large domain-specific text corpus for language modeling from various sources. We then build ConfliBERT using two approaches: pre-training from scratch and continual pre-training. To evaluate ConfliBERT, we collect 12 datasets and implement 18 tasks to assess the models’ practical application in conflict research. Finally, we evaluate several versions of ConfliBERT in multiple experiments. Results consistently show that ConfliBERT outperforms BERT when analyzing political violence and conflict.

## introduction the study of political violence is a central concern of conflict scholars and security analysts in the academic and policy communities. for decades, scholars and governments have devoted incalculable resources to monitoring, understanding, and predicting the dynamics of social unrest, political violence, and armed conflict worldwide. conflict research is a sub-field of political science that analyzes a broad scope of interactions between government agents, their challengers, and the civilian population, including material and verbal conflict and cooperation. conflict research covers protest, riots, repression, insurgency, civil war, terrorism, human rights, genocide, criminal violence, forced displacement, conventional and unconventional warfare, nuclear deterrence, peacekeeping, diplomatic disputes and cooperation, among others. traditionally, researchers used manual coding to track conflict processes worldwide @xcite . unfortunately, the high costs and slow pace of domain experts conducting these tasks make it extraordinarily difficult and costly to monitor highly complex and rapidly changing conflicts in an ever-growing volume of information available on a global scale. furthermore, these efforts tend to focus on quantifying particular types of conflict events between specific kinds of actors @xcite . initial efforts to address these challenges motivated political scientists to develop automated systems to classify or extract structured event data from news articles @xcite @xcite @xcite @xcite @xcite . these systems capture a broader range of event types, including different conflict and cooperation events, between a larger number of political actors. they can also extract volumes of data that are orders of magnitude greater than manual coding efforts. automated event data such as the integrated crisis early warning system have been used for conflict forecasting and other kinds of research in political science @xcite @xcite . however, these existing systems rely on dated pattern matching techniques and large dictionaries, which often yield low-accuracy results and are too costly to maintain. recent efforts by political scientists employ traditional machine learning @xcite and deep learning @xcite @xcite to analyze political conflict and violence. standard supervised learning requires labeled data, which are expensive to obtain due to the expertise required for quality annotation. this led conflict scholars to seek alternative solutions based on the latest developments in natural language processing (nlp). recent progress in nlp has been driven by pre-trained transformer language models @xcite @xcite . self-supervision using large-scale unlabeled text can significantly alleviate the annotation bottleneck using transfer learning. the training parallelization of transformers also improves their efficiency on large datasets. as a result, the use of powerful computational devices and the advantage of transformer structures make large-scale language models' pre-training possible. furthermore, the introduction of extensive benchmarks @xcite @xcite validates the significant improvement of pre-trained language models on various downstream tasks. while many language models are built on general domain corpora, such as wikipedia, @xcite , and webtext @xcite , recent works show that pre-training on domain-specific corpora can boost downstream performance on those domains @xcite . domain-specific work in bio-medicine focuses not only on developing pre-trained models @xcite @xcite @xcite but also on proposing domainrelevant evaluation benchmarks @xcite . pre-training models also have advanced research in other domains such as academic papers @xcite and legal studies @xcite . despite some efforts to apply transformers-based approaches in political science @xcite @xcite @xcite @xcite , we are unaware of any studies that develop and evaluate domain-specific pretrained language models for political science or conflict research. by combining the expertise of conflict scholars and computer scientists, we developed con-flibert, a pre-trained language model designed for conflict and political violence. conflibert improves downstream tasks for conflict research while significantly alleviating the annotation bottleneck. we expect it to support a broad community of academic and policy researchers, enabling the analysis of conflict processes using a domainspecific nlp tool that yields accurate and valid results at minimum operational cost. our paper provides the following key contributions: (1) we curate a large domain-specific corpus for language modeling in the domain of political violence, conflict, cooperation, and diplomacy. (2) based on our domain-specific corpora, we devise a pre-trained language model, conflibert, and make it available to the general public, which directly benefits the political science and policy communities. (3) to evaluate our model in practical applications, we collect 12 datasets and conduct 18 tasks relevant to conflict research. we are the first to carry out such a comprehensive evaluation of language models for conflict studies. (4) we evaluate different versions of conflibert and show it outperforms models trained on generic domains. we also perform indepth analyses of different tasks to investigate the factors affecting the performance. ## preliminaries recent pre-trained transformer language models, such as the bidirectional encoder representation from transformers (bert) @xcite , follow a two-steps framework: (1) pre-train on a large unlabeled corpus; and (2) fine-tune on taskspecific labeled data. these models learn semantics during the pre-training step and require smaller labeled data to significantly improve their performance on downstream tasks. the fine-tuning step requires minor network modifications to create state-of-the-art models for various tasks. technically, bert uses the multi-layer, multihead self-attention mechanism, which provides substantial advantages for language modeling, such as allowing parallel gpu computation and capturing long-range dependencies. this allows to efficiently pre-train large language models on large corpora using powerful devices. another key element of bert-like models is the design of self-supervision tasks. self-supervision refers to generating labels for unlabelled data and using them to train a model in a supervised manner. bert uses two self-supervision tasks during pre-training. on the one hand, masked language model (mlm) is a fill-in-the-blank task based on randomly masking a token and then using the surrounding words to predict the word hidden behind the mask. on the other hand, next sentence prediction (nsp) determines whether one sentence follows another one in the same document. recent works also propose variants of self-supervision tasks. for example, joshi et al. (2020) mask out contiguous sequences of tokens to improve span representations. clark et al. (2020) use replaced token detection, where the model distinguishes real input tokens from plausible but synthetically generated replacements. however, liu et al. (2019) prove that mlm is competitive with other recently proposed training objectives with more data and improved training strategies. finally, most bert-like models focus on a generic domain, such as wikipedia, bookcorpus @xcite and webtext @xcite . however, bert without domain adaptation tends to underperform in target domains with distinct characteristics such as specialized vocabulary, language style, and specific semantics. our domain includes political violence, armed conflict, international cooperation, and diplomacy-all of which have these characteristics. this performance gap is the primary motivation for developing domainspecific language models. specifically, the language of political actors involves strategic and complex semantics. policy positions that show support for one actor while also threatening another are sometimes embedded in simple statements. for example, "nato will not tolerate this aggression" mixes a negation, a conditional, and the action of potential interest. signals are highly context-dependent, adapted for a target audience, and vary in strength depending on the specific actor sending the signal @xcite . compared to a generic language model, we expect ours to incorporate important contextual information and learn more about the strategic ways that political actors convey information. this context and the related political biases in it are exactly a need that conflibert aims to fulfill. political conflict and violence text gains from this domain knowledge: one political actor's definition of "rebels" is another's "freedom fighters". ## approach as described in section 2, mlm-based bert achieves competitive performance among other transformer models with different self-supervision tasks. besides, bert has been validated in various domains @xcite @xcite @xcite shown in ## conclusion and future work this paper presents the development, application, evaluation, and further exploration of conflibert, a pre-trained language model for political conflict and violence. the development of conflibert rests on an unprecedented effort on three fronts. first, we collect and curate a large domain-specific corpus to support the pre-training process. second, we conduct a comprehensive evaluation across several datasets and various nlp tasks of distinct nature and varying degrees of complexity. the results show that conflibert consistently outperforms bert in the conflict and political violence domain. furthermore, the biggest improve-ments are with limited training data, which conflict researchers often have due to the high costs of data annotation. in this way, conflibert constitutes a valuable development that will contribute to a broad community of researchers in political science and policy sectors interested in tracking, analyzing, and predicting political violence and conflict on a global scale. due to limited time and computational resources, we did not conduct more experiments to explore various hyperparameters that could affect finetuning results, such as vocabulary size and pretraining epochs, to name a few. future work should analyze how to optimize conflibert, expand con-flibert to multi-lingual settings, and apply con-flibert to more challenging tasks such as understanding, inference, question answering, uncertainty qualification @xcite , and few-/ zero-shot tasks to speed up the study of nlp application for the political science community.