Team “ D a D e F r N i” at CASE 2021 Task 1: Document and Sentence Classification for Protest Event Detection (2021)

Abstract: This paper accompanies our top-performing submission to the CASE 2021 shared task, which is hosted at the workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text. Subtasks 1 and 2 of Task 1 concern the classification of newspaper articles and sentences into “conflict” versus “not conflict”-related in four different languages. Our model performs competitively in both subtasks (up to 0.8662 macro F1), obtaining the highest score of all contributions for subtask 1 on Hindi articles (0.7877 macro F1). We describe all experiments conducted with the XLM-RoBERTa (XLM-R) model and report results obtained in each binary classification task. We propose supplementing the original training data with additional data on political conflict events. In addition, we provide an analysis of unigram probability estimates and geospatial references contained within the original training corpus.

## introduction can natural language processing (nlp) be leveraged to extract information on socio-political events from text? this is an important question for conflict and peace studies, as events like protests or armed conflicts are frequently reported in textual format, yet are costly to extract. the workshop on challenges and applications of automated extraction of socio-political events from text (case 2021) aims at bringing together political scientists and nlp researchers to improve methods for automated event extraction 1 . as part of this workshop, a shared task is proposed to advance progress on various problems associated with reliable event detection @xcite . ## dataset in order to train our model, we leverage the data provided by the organizers as well as additional data on political conflict events. in this section, we describe both of these datasets. ## model selection informed model selection is crucial for competitively solving the task. we choose pre-trained transformer-based @xcite classification models due to their state-of-the-art performance in various tasks @xcite . given the fact that the provided dataset is multilingual, we face a crucial design decision: option (a): select a monolingual model e.g. bert @xcite , that is pre-trained on huge, unlabeled text corpora in english with the need to translate all the other languages in the dataset back to english, then finetune the model on that. or option (b): choose a multilingual model e.g. a multilingual version of bert(mbert), xlm((lample and conneau, 2019) or xlm-roberta (xlm-r) @xcite , that handles multiple languages simultaneously and fine-tune the model on the original languages. we ultimately choose the xlm-r model to experiment with. recent results suggest that multilingual models achieve better performance, especially for low-resource languages. ## experiments to conduct our experiments we rely on implementations provided by the huggingface library 56 . for experiment tracking we make use of wandb library foot_5 . after several rounds of hyperparameter search, we select a batch size of 16, learning rate of 2e-5, weight decay of 0.01 and train for 4 epochs. we train models for each of the subtasks separately (t 0 ), then we experiment with combinations of datasets, mixing subtasks and languages (t 0mix ). we achieve the best results when training on the combined dataset including all the languages. we try different combinations of extensions (t 1 -t 3 ), e.g. having a balanced dataset or keeping the original imbalance rate of the shared task data. finding protest events in hindi language is challenging. therefore, we translate protest events from english sources. additionally, we experiment with removing contextual information and basing our classification on linguistic patterns only. to this end, we remove all named entities from the dataset (t 0non er -t 0mix non er ). the results, surprisingly, reveal only a slight degradation compared to the original dataset and even a small increase in performance on subtask 2 on english text. ## results in this subsection we present the results achieved by our xlm-r models fine-tuned on different datasets. ## discussion in this section we present and analyse the conflict event data corpus, performing a descriptive analysis on the dataset using unigram probabilities and geospatial coordinates. ## conclusion in conclusion, the paper outlines two major contributions to the case 2021 shared task. firstly, our xlm-roberta model for classification task 1.1 and task 1.2 yields competitive results, especially for the hindi subtask, where no training data was available. secondly, we provide a descriptive analysis of idiosyncrasies contained with the provided text corpora. our analysis qualitatively investigates geographical connotations in the corpora and possible outliers using word probability estimation. figure 3 : this figure shows the training english set with the first three principal components. even if most of the data is concentrated in one dense cluster, there are a few points that can be very easily distinguished. they generally are either in a language different than english (id 108218), or have other very rare characteristics, (id 106495 having an extremely large character length).