SU - NLP at CASE 2021 Task 1: Protest News Detection for E nglish (2021)

Abstract: This paper summarizes our group’s efforts in the multilingual protest news detection shared task, which is organized as a part of the Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE) Workshop. We participated in all four subtasks in English. Especially in the identification of event containing sentences task, our proposed ensemble approach using RoBERTa and multichannel CNN-LexStem model yields higher performance. Similarly in the event extraction task, our transformer-LSTM-CRF architecture outperforms regular transformers significantly.

## introduction identifying events and extracting event related information from text is an important language understanding task which has been studied for quite some time. this challenging task has been studied in several steps or divided into some sub-tasks. the first step is identifying whether a document or a sentence contains an event or not. if it contains then the event co-reference resolution task analyses whether the context around it (such as other sentences) refer to the same event or not. event related information such as the event trigger and its arguments are also extracted, which can be later on used to create event taxonomies. these steps either alone or together have been studied for english extensively, similar to many other natural language processing tasks. this year as part of the challenges and applications of automated extraction of socio-political events from text (case) workshop, a shared task covering some of these sub-tasks has been organized not only for english but also for portuguese, spanish and hindi @xcite . the common theme was the identification of protest events from news articles. the organizers specifically focus on the four subtasks. in the first and second sub-task, the aim is to predict whether a given document (subtask 1) or sentence (subtask 2) contains information about an event (either past or ongoing). the third subtask focuses on event sentence coreference and the participants are asked to predict whether the sentences containing an event are referring to the same event or not. in subtask 4, the goal is to identify event triggers and related arguments from sentences. it is hard to choose among these interesting subtasks, therefore we participate in all four of them. due to time constraints we only work on english and leave the rest of the languages as future work. the first and the second subtask focus on predicting whether a content contains an event or not. for these tasks in addition to trying standard transformer based models, we explore ensemble models which combine the strengths of different models. furthermore, the effect of stemming the context is also explored in these subtasks. the third subtask is related to the event coreference task. for this task, we explore the rescoring and clustering approach proposed by @xcite . finally, the goal of subtask 4 is to extract event information from context. for this task, we exploit the transformer-lstm-crf architecture which has shown success in several ner tasks. the rest of the paper is organized as following: section 2 describes our proposed approach for identifying whether a content contains an event or not, and details our submissions for subtasks 1 and 2. section 3 explains our submission to the event coreference resolution subtask. section 4 presents the experimental results for event extraction subtask and finally section 5 concludes the paper with future work. ## subtask 1 & 2: event or not the goal of the first two subtasks is to predict whether the provided input context contains an event (either past or ongoing) or not. therefore, the task is a binary classification task. in these two subtasks the only difference is the input context. in subtask 1 the input is the whole news article while in subtask 2, it is only a sentence. the main difference between these two tasks is the length of the input. in subtask 1's dataset, even though most documents contain around 3 sentences, the maximum length in the data is almost 10 times larger than the maximum length in subtask 2 data. this makes subtask 1 slightly more challenging. one expects documents as longer input, to contain more clues about an event if there is; therefore more useful. however, there is also the risk of unrelated content causing mixed signals. even though this difference between the tasks, we mostly apply same approaches to both. for this binary classification problem, we use some simple neural network architectures as baselines and also investigate fine-tuning several pretrained transformer based models. the models applied are listed as follows: word2vec foot_0 @xcite , nnlm @xcite model trained on google news dataset foot_1 and glove @xcite ) 6b wikipedia embeddings foot_2 , have been tried. since the ratio of out-of-vocabulary words were very small, character-based embeddings have not been explored. we have seen that using different embeddings resulted in minor changes, and rather finetuning the embedding layer or not, does not have any significant effect on the performance of models in terms of overfitting resistance or achieved scores. nnlm and glove return slightly better performance compared to word2vec, when used in standalone cnn or rnn models. however, as we try ensembling approaches (to be described in the upcoming sections), nnlm outperforms glove with its high precision score. therefore, nnlm embedding is used in all reported experiments in this section. ## subtask 3: event sentence coreference identification in event sentence coreference task, event containing sentences in a document are analyzed to see whether they refer to the same event or not. this task is slightly different than other ones as it does not only consist of a classification step, but also requires clustering afterwards. this two step procedure is known as the mention-pair model @xcite in coreference resolution tasks. the first step includes a binary classification model to classify pairs of mentions and the second step uses these predictions to determine the coreference relations by clustering them @xcite . in this paper, we also use the two step approach, and first perform pairwise classification of sentences and then cluster them. ## subtask 4: event extraction the goal of the final subtask is to identify the event triggers and its arguments from the sentence. the training dataset consists of 808 sentences which contain iob type token-based labels of 7 different labels. similar to previous tasks, 20% of this data is used for validation and the rest for training purposes. in many sequence modeling tasks, the bidirectional transformer models outperform other machine learning architectures; therefore, bert and roberta are used as strong baselines in this task. as a further development, the transformer model is connected with a bilstm and a crf layer as our second architecture. connecting bilstm and crf to a transformer has shown success in several named entity recognition tasks @xcite . the performance of these models over both validation and test sets are presented in according to even though we achieved good performance, due to a minor format issue at our test submission file, our submissions were not correctly evaluated. based on our scores at analyzing the individual tag performances revealed that model is doing a better job at identifying the triggers compared to its arguments. this is expected as trigger tag is the second most popular tag at the data after the o tag. trigger is closely followed by event time, which is easier to predict due to its smaller vocabulary variance and common language patterns, even though its lower presence in the training data. in order to analyze the weak points of the models, the confusion table of the top performing roberta-bilstm-crf model over the validation data is shown in figure 2 . the confusion matrix specifically focuses on the event trigger and arguments tags. table for event trigger and arguments tags based on figure 2 , the etime (event time) is the tag which has not been mistaken with any other event specific tags. on the other hand, the highest confusion is between the organizer and participant tags. that is followed by place and fname (facility name) which is expected due to use of similar wordings and context around. ## conclusion in this paper, we mainly focus on english, and try to improve the current state-of-the-art on event specific nlp tasks. source codes of all of our models are available online 9 . additional details of our models, like hyper-parameters, are also summarized in the github. as future work, we will focus on other languages and see whether the trends observed with english, exist in those other languages as well.