SPARTA at CASE 2021 Task 1: Evaluating Different Techniques to Improve Event Extraction (2022)

Abstract: We participated in the Shared Task 1 at CASE 2021, Subtask 4 on protest event extraction from news articles and examined different techniques aimed at improving the performance of the winning system from the last competition round. We evaluated in-domain pre-training, task-specific pre-fine-tuning, alternative loss function, translation of the English training dataset into other target languages (i.e., Portuguese, Spanish, and Hindi) for the token classification task, and a simple data augmentation technique by random sentence reordering. This paper summarizes the results, showing that random sentence reordering leads to a consistent improvement of the model performance.

## introduction the generation of protest event datasets over the last decades has allowed social movement scholars to study the dynamics and evolution of collective action in contemporary societies. the collection of relevant events is usually based on the systematic, manual analysis of news articles, which provide information about the variables of interest such as the location, date, and main protagonists of protest demonstrations @xcite . it has been noted, however, that the manual coding of news articles is time and labor-consuming, and, as a result, comparative and longitudinal studies that rely on multiple news sources may not be feasible @xcite . recent work on approaches that automatically retrieve protest information is promising and may address this challenge. case 2021 task 1: multilingual protest news detection @xcite constitutes a collaborative project that attempts to map the features of political contention through the automated analysis of news articles at different data levels. we participate in subtask 4, which focuses on identifying event triggers and their arguments and involves detecting protest events in three languages: english, portuguese, and spanish. the paper proceeds as follows: section 2 discusses related work in the field of computational social science, whereas section 3 defines the task of event extraction. section 4 describes the architecture of our approach. section 5 provides details about the experiments we conducted. finally, in section 6, we summarize and discuss the results. ## related work the use of automated tools for the identification and coding of political event data spans a period of more than 30 years @xcite , and, for this task, several methodological approaches have been developed and tested. initial attempts to automatically parse text and produce structured data were based on the kansas event data system (keds) @xcite , which, along with its successors programs such as tabari @xcite and petrarch @xcite , was designed to provide information about different types of political action and also their source and target actors. in the field of contentious politics, that is mainly interested in the activities of social movements and protest groups, the standard approach involved for a long time the manual coding of text. however, halfautomated techniques have also been introduced. for instance, @xcite have developed several filters (e.g., a location-based filter) and document and event-trigger classifiers to select newspaper articles that contain protest-related information. in the final step of their procedure, the authors create samples of relevant articles and manually extract the features of protest events. taking advantage of recent advances in machine learning methods, other scholars have turned their attention to approaches that automatically detect and classify protest information. however, unlike coding systems such as keds and its successors programs that make use of actor and verb dictionaries, the new techniques primarily rely on pretrained transformer-based language models @xcite , such as bert @xcite . case 2021 and 2022 task 1 @xcite are such research projects-organized as shared tasks-that focus on the generation of multilingual protest event data and involve four subtasks: 1. document classification; 2. sentence classification; 3. event sentence coreference identification; and 4. event extraction. in the following sections, we focus on subtask 4 and discuss techniques that improve over the baseline multilingual model xlm-roberta @xcite . ## event extraction task the event extraction task consists of identifying text spans in given news article sentences and classifying them into entity types such as trigger, participant, place etc. given @xmath0, .., w n ) a sentence and @xmath1, .., t m } a set of entity types, the task consists of identifying spans @xmath2, .., w e ) such that typeof (s) ∈ t . this task can be reformulated as the token classification task, where iob2 labels (sang and veenstra, 1999) are assigned to tokens in sentences to form spans. hereby, the first token w b within the span s is assigned the label b type and the rest of the tokens the label i type , where @xmath3. all tokens outside of any identified spans are assigned the token o. ## architecture the objective of the conducted evaluations was to show possible improvement compared to the winning system from last year's participation at case 2021 @xcite by the ibm team @xcite . the authors trained variants of the multilingual model xlm-roberta large @xcite on news article sentences to predict iob2 labels for event extraction. therefore, all experiments in our paper used the same base model and similar training settings. in contrast to ibm team's approach, we did not provide an ensemble variant of the model but relied only on a single multilingual model. another significant architectural difference was how the inputs were provided to the model; instead of splitting the news articles into single sentences, we used the maximum possible input length of 512 tokens and fed as many full sentences as possible to the model, providing as a result more context. if the news article exceeded the maximum input length, it was split into overlapping sentence sequences as shown in figure 1 . thus, some sentences were presented to the model multiple times during the fine-tuning procedure with different preceding or following contexts. however, the final predicted token labels during the test procedure were derived only from the reconstructed non-overlapping sequence of sentences, leading to unique predictions. in both procedures, we removed the concatenating separator token [sep] from the input. we should also note that the predicted token labels correspond to the iob2 labels. ## experiments starting from the base model, several techniques were evaluated after fine-tuning the model on the provided dataset for subtask 4 @xcite . similar to the ibm team, we used only 10% of the english dataset as a development set. thus, the influence of the employed techniques on other languages was mainly inferred from the testing results in the provided codalab page. the best models for submission were selected according to the highest conll f1 score and lowest mean validation loss on the development set. the best values of f1 achieved 80.06% and 80.86%. models were fine-tuned for 20 epochs using hyperparameters as shown in ## final results and discussion the final results on testing datasets for the approaches of pre-training and pre-fine-tuning are summarized in the numbers indicate that conducting an expensive pre-training procedure on additional protestrelated data does not have the expected boosting effect for the model performance. this suggests that the xlm-r models already integrate sufficient knowledge about the type of language used to describe protests. comparable results can be achieved using a pre-fine-tuned model on a similar task. furthermore, the usage of the dice loss does not lead either to very different results compared to the classical cross-entropy loss on this task. it is important to mention that models in finally, we evaluated the translation technique, which resulted in performance drops. ## conclusion in this paper, we presented the models developed for the shared task 1 subtask 4 at case 2021. we explored different techniques to improve the baseline multilingual model. the best result was achieved by improving on the spanish test data by 3.67 points of conll f1 score over the winner of the previous competition round. our submissions ranked 1st for portuguese and spanish and 2nd for english in the current competition round.