Team “ N o C onflict” at CASE 2021 Task 1: Pretraining for Sentence-Level Protest Event Detection (2021)

Abstract: An ever-increasing amount of text, in the form of social media posts and news articles, gives rise to new challenges and opportunities for the automatic extraction of socio-political events. In this paper, we present our submission to the Shared Tasks on Socio-Political and Crisis Events Detection, Task 1, Multilingual Protest News Detection, Subtask 2, Event Sentence Classification, of CASE @ ACL-IJCNLP 2021. In our submission, we utilize the RoBERTa model with additional pretraining, and achieve the best F1 score of 0.8532 in event sentence classification in English and the second-best F1 score of 0.8700 in Portuguese via simple translation. We analyze the failure cases of our model. We also conduct an ablation study to show the effect of choosing the right pretrained language model, adding additional training data and data augmentation.

## introduction with the growing volume of online news from both traditional news media and social media, large amounts of texts are being created every day. these text data contain information about events happening around the world. for social science and policy making, the event information in these texts can be extremely valuable. due to the sheer volume of data available, there is a strong demand for tools to automatically extract and analyze socio-political events. automatic event extraction enables governments, non-governmental organizations and society as a whole to take more timely, proportional and appropriate actions in changing circumstances. event sentence classification is an important step in the event extraction pipeline @xcite . in this work, we present our submission to the case 2021 shared task, hosted jointly with the workshop on challenges and applications of automated extraction of socio-political events from text (case) @ acl-ijcnlp 2021 @xcite . the shared task consists of two main tasks: multilingual protest news detection and fine-grained classification of socio-political events. in the first shared task, there are four subtasks: event document classification, event sentence classification, event sentence coreference resolution and event extraction. in this paper, we focus on task 1, subtask 2, namely event sentence classification. for a detailed description of the shared task, please refer to @xcite . prior iterations of the workshop can be found in @xcite . within this subtask, we further narrow down our scope by focusing on the english event descriptions only. we train a classifier to solve the binary classification problem to identify whether a sentence contains a protest event, as defined in @xcite . given the huge success of pretrained language models such as bert @xcite , roberta @xcite , xlnet @xcite and electra @xcite , we adopt roberta as the backbone of our model. inspired by the good result achieved through additional pretraining @xcite , we harness the polusa dataset @xcite of political news articles to second-pretrain our model with a masked language modeling (mlm) objective. further, we conduct a series of ablation studies to justify our design choices. we first run experiments to choose a suitable base model. then, we conduct experiments on using additional training data from other subtasks. given the limited amount of training data available, we experiment with data augmentation techniques, including back translation, embedding augmentation, and checklist augmentation @xcite . the rest of this paper is organized as follows: section 2 describes the dataset of the subtask. section 3 discusses the method of our best-performing submission. section 4 presents quantitative results achieved by our model as well as a failure case analysis. in section 5, we present additional experiments as part of an ablation study. in section 6, we discuss observations of the dataset and models trained on this dataset from the perspective of named entities before concluding the paper in section 7. ## dataset we are provided with a dataset of labeled sentences which was introduced in @xcite . each sentence has a binary label indicating if the sentence contains a protest event. while the dataset comprises sentences in english, spanish and portuguese, we solely focus on english sentences. the english version of this dataset contains 22,825 sentences, out of which 18,602 (81.50%) have label 0 and 4223 (18.50%) have label 1. since no official train-validation split is provided, we divide the dataset into a training set (80%) and a validation set (20%). ## proposed method we utilize the roberta base model @xcite as the backbone of our model. throughout this work, we refer to the pretrained roberta model @xcite as "roberta default". we use the term language model to refer to transformer-based @xcite cloze language models. ## results we conduct second pretraining for only 42,000 steps, with a batch size of 16 and a maximum sequence length of 256, due to time and resource constraints. for the downstream task, we train for 25 epochs and take the best epoch based on validation f1 score. ## ablation study in this section, we explore different base models to finetune on, using additional training data from other subtasks and data augmentation techniques. all results reported in this section are on the validation set, without second pretraining. ## effect of named entities in this section, we analyse the effect of named entities (ne) on the results. we train models with two modifications of the subtask 2 data: 1. we remove all named entities in all sentences; 2. we remove all text tokens except for named entities in each sentence. for each data modification setting, we train two models, one model initialized with the roberta default weight, one initialized with the second-pretrained weight as mentioned in section 3. for comparison, we include a random guessing baseline model. it draws label from the same distribution of the ground truth labels in the training set, without considering the sentences at all. we report the average f1 score of 100 such random assignments. we also include the result of the model trained with the original subtask 2 training data using roberta default weight and second-pretrain weight for reference. the result is shown in ## conclusion in this paper, we present our submission to task 1, subtask 2 at case @ acl-ijcnlp 2021. our model is based on roberta with a second pretraining step done on the polusa dataset. we inspect the failure cases of our model on the validation set and provide some explanations. to justify our design choices, we conduct an ablation study. overall, we achieve the highest f1 score in the english version of this subtask and the second highest f1 score in portuguese on the evaluation server. in future work, we plan to incorporate knowledge from the annotation manual into the model and incorporate richer semantic context by means of topological graph structures @xcite .