CASE 2021 Task 2 Socio-political Fine-grained Event Classification using Fine-tuned R o BERT a Document Embeddings (2021)

Abstract: We present our submission to Task 2 of the Socio-political and Crisis Events Detection Shared Task at the CASE @ ACL-IJCNLP 2021 workshop. The task at hand aims at the fine-grained classification of socio-political events. Our best model was a fine-tuned RoBERTa transformer model using document embeddings. The corpus consisted of a balanced selection of sub-events extracted from the ACLED event dataset. We achieved a macro F-score of 0.923 and a micro F-score of 0.932 during our preliminary experiments on a held-out test set. The same model also performed best on the shared task test data (weighted F-score = 0.83). To analyze the results we calculated the topic compactness of the commonly misclassified events and conducted an error analysis.

## introduction event detection and classification as natural language processing (nlp) tasks can be used to analyze data gathered in the information space. the findings of this analysis can then be connected to events in the physical world and contribute to situational awareness, particularly when they are related to socio-political events. the sheer amount of data that is generated and stored in the information space every day, means that strategies need to be developed to be able to efficiently and effectively process this data. given the large amounts of data, deep learning strategies are often preferred. however, time and computational constraints may play a role in deciding how to extract and analyze data. task 2 in the socio-political and crisis events detection shared task at the case @ acl-ijcnlp 2021 workshop aims at the fine-grained classification of events @xcite . the task is based on data extracted from the armed conflict location & event data (acled) database @xcite . it consist of socio-political events that have been annotated based on the acled event taxonomy, and includes 6 event types and 25 event subtypes. the aim of this task is to label event snippets using a model trained on data from the acled dataset, in order to see how robust models are when presented with data that is not directly covered by acled or contains unseen event classes. the results presented in this paper pertain only to subtask 1, where the task is the classification of 25 different event subtypes with acled-compliant labels. in other words, all the classes are seen classes from the acled dataset. the second and third subtasks are zero-short learning tasks that contain unseen classes. this paper proceeds by first describing the data collection process in section 3. section 4 contains the system description and the following section contains the experimental results. section 6 provides an overview of the results based on the test data provided by the organizers. finally, in section 7 the error analysis provides an insight into the system results and the data. ## related work previous research in event detection and classification shows that there are numerous approaches to solve the problem of detecting events in texts. @xcite give a coherent overview of suitable strategies, starting with earlier approaches like pattern matching, and describing methods of machine learning as well as deep learning. there have been a number of shared tasks that have taken place in previous years that contribute to research conducted in this area. specifically, the shared tasks clef 2019 protest news @xcite , aespen 2020 @xcite , and case 2021 (task 1) @xcite focus on event detection at both the sentence and document level, as well as event co-reference resolution. currently, not much research has been conducted that further analyzes event data once the events have been identified. there are a handful of studies across different domains. @xcite achieve state of the art results detecting and classifying social event data with a pairwise popularity graph convolutional network (pp-gcn) with an external knowledge base. @xcite compare different supervised classification methods for detecting a range of different events, and achieve good results with support vector machines (svm) and convolutional neural networks (cnn). a benchmark corpus for fine-grained political event classification was created by the organizers of this task and an initial exploration and classification of the data is reported on in @xcite and @xcite . the findings reported that bert transformer models achieved a micro f1 of (0.943-0.949) and a macro f1 of (0.860-0.889). more simple tf-idf-weighted character n-gram models also achieved good results. a large dataset of 600,000 annotated acled event snippets was used as training data. ## data collection due to copyright reasons, the data used in this paper was collected directly from the acled website. 1 to create the corpus, all data from each available region was downloaded and then filtered using the following steps. firstly, all events with less than 25 tokens and more than 1000 tokens were removed. the next step was to balance the corpus based on the 25 different fine-grained event classes. originally, the largest class in the corpus consisted of 36.69% of the events, compared to the smallest with 0.001%. to create a more balanced version of the corpus, we extracted a sample of events per class, with the smallest classes being fully represented and extracting only a percentage of the largest classes. note that it was not possible to fully balance all of the classes as there was only a very small amount of data for classes such as chem weap. a random sample of this balanced corpus was then split into a train (@xmath0, development (@xmath1, and test (@xmath2, which also all contain the balanced class distribution. we observed that randomizing the order of events was crucial, to avoid 1 https://acleddata.com/data-export-tool/ introducing a bias based on the different acled regions. figure 2 illustrates the distribution of the corpus. a more detailed table can be found in appendix a. in a further step, we created three different versions of the original corpus. the first version, referred to as acled n, contains the original text from the acled download, an example of which can be found below. ## system description we submitted five system runs for evaluation. the systems differ slightly from each other, either in the model or the way the used data was pre-processed. the general approach for all submitted systems was to use fine-tuned pre-trained transformer document embeddings. all experiments were conducted using the flair framework @xcite . ## preliminary experiments preliminary model evaluations on 10 held-out test sets show that each of the systems performed comparatively well. the roberta model with the normal acled text as input performed slightly better than the other systems. an analysis of the results of the individual classes, shows that each of the 25 subtypes achieve f1-scores of over 0.800. ## results additionally, the second model that uses original acled text, system 4, achieves the second best result. as was the case in our preliminary experiments, we see that the inclusion of specific location and timestamps in the training data, does not greatly influence the ability of the system to predict the different classes correctly or incorrectly. ## error analysis to get a better insight into the workings of our systems, we conducted an error analysis on the test data provided by the organizers for all five submissions. in order to investigate misclassifications made by the models, we decided to look at the performance of the system with regard to the individual classes. ## conclusion in this study we proposed the use of fine-tuned roberta transformer document embeddings for the fine-grained classification of socio-political events. we balanced the corpus to ensure that the 25 subtypes were represented as equally as possible. compared to the results that were achieved during the preliminary experiments, we observed a drop in performance on the test set provided by the organizers. however, compared to the baseline figures provided by the organizers in @xcite , we achieved very similar results with less training data. this suggests that balancing the training data data had a positive effect on model performance. our analysis of the results of both different test sets, the set created for preliminary experiments and the set provided by the organizers for system evaluation, show that there is definitely a difference in performance in the various classes. it also highlighted the issue of events that could be classed as more than one different subtype, and the challenge that these events pose for fine-grained classification. depending on the given use case, parts of our system could already be implemented in a real world setting in order to analyze the flow of data in the information space and achieve situational awareness in the physical world, as clear cut classes like chem weap and grenade are identified reliably. in a military setting, for example, these classes are far more relevant than occurrences of property distruct. in future work, it would be interesting to evaluate if the use of more training data, while still trying to obtain a more even distribution of classes, would further increase performance. particularly, it raises the question if more training data would increase performance for the classes that currently do not perform as well. a more thorough class analysis, which would contribute to understanding why there seem to be systematic errors in specific classes, could provide insight into answering this question.