C am P ros at CASE 2022 Task 1: Transformer-based Multilingual Protest News Detection (2022)

Abstract: Socio-political protests often lead to grave consequences when they occur. The early detection of such protests is very important for taking early precautionary measures. However, the main shortcoming of protest event detection is the scarcity of sufficient training data for specific language categories, which makes it difficult to train data-hungry deep learning models effectively. Therefore, cross-lingual and zero-shot learning models are needed to detect events in various low-resource languages. This paper proposes a multi-lingual cross-document level event detection approach using pre-trained transformer models developed for Shared Task 1 at CASE 2022. The shared task constituted four subtasks for event detection at different granularity levels, i.e., document level to token level, spread over multiple languages (English, Spanish, Portuguese, Turkish, Urdu, and Mandarin). Our system achieves an average F1 score of 0.73 for document-level event detection tasks. Our approach secured 2nd position for the Hindi language in subtask 1 with an F1 score of 0.80. While for Spanish, we secure 4th position with an F1 score of 0.69. Our code is available at https://github.com/nehapspathak/campros/ .

## introduction the recent technological advancement has led to a continuous flow of information among users in online and offline ecosystems. users' information may cover various social and political factors, often constituting information related to political violence, crisis, and protests, among others. the automatic detection of such socio-political protests/crisis events from news and social media has become crucial from a peaceful society perspective @xcite . not only does the early detection of such event helps in the deployment of early interventions, but it also helps understand people's perception of a socio-political event. event detection aims to identify and extract pertinent data from a text about specific categories of events. it is a crucial information extraction task that unearths and collects information about current and historical occurrences concealed in vast amounts of textual data. the case 2022 workshop focuses on detecting socio-political and crisis events in a multi-lingual setting at different granularity levels. this paper focuses on developing models and systems for "multilingual protest news detection -shared task 1". in shared task 1, there are 4 subtasks. the aim of subtask 1 is to detect whether a news article contains event information. the news articles are in the form of documents. hence the subtask looks at whether a given document contains event information. the second subtask focuses on detecting a sentence containing information about a past or ongoing event. the third task focuses on event sentence coreference identification, such as which event sentences in subtask 2 belong to the same event. the fourth and final subtask focuses on event extraction and aims to identify the event triggers and their arguments. we present our proposed system for subtask 1 in this paper. researchers have focused on event extraction from various aspects in the past @xcite . the task presented by @xcite focused on event sentence co-reference identification. in the case 2021 socio-political and crisis event detection, the training dataset consisted of english, spanish and portuguese, while the test data were from english, spanish, portuguese, and hindi @xcite . in case 2022, however, new languages are introduced for multilingual document-level event detection. the workshop allows participants to create models for various subtasks and contrast related approaches. subtask 1 consists of documents from english, spanish, and portuguese for training. for testing, the documents are available in a zero-shot setting, including languages from low-resource languages; hindi, turkish, urdu, and mandarin. identifying crisis and socio-political protest detection in a multi-lingual setting makes the task very complex. our work mainly focuses on document-level (subtask 1) event detection in a multilingual setting. our approach is based on pretrained transformer models and different learning strategies for making predictions. since the tasks are designed for protests in a multilingual setting, we do not perform language-level pre-processing on our dataset. our submission for subtask 1 achieved 2 nd position in zero-shot hindi document-level event detection. the rest of the paper is organized as follows. section 2 describes the related literature. the details of the task and dataset are presented in section 3. the proposed approach and experimental setup are described in section 4. results are described in section 5, followed by conclusion in section 6. we intend to make our code public for further use by the community. ## related work in natural language processing (nlp), event detection is a task that detects event triggers/mentions (i.e., the key terms that drive or express an event) and categorizes them into predefined event types @xcite . the early detection of ongoing and past events exploited featurebased approaches to detect events @xcite . however, the early data-driven @xcite , knowledge-driven, and rulebased approaches missed the semantic relationship in the data @xcite . other early approaches for event detection include machine learning models such as svm and decision trees @xcite . the recent deep learning approaches proposed in the literature @xcite improve event detection; nonetheless, they are not generalizable for low-resource languages. to address the data scarcity problem for low-resource languages, researchers have recently used the pre-trained language model gpt-2 to generate training samples @xcite . another less-discovered approach in the event detection task is cross-lingual event detection which proposes model creation for effective performance over different languages @xcite . the work presented in @xcite utilizes knowledge from open-domain word ## methodology the transformer-based models have recently gained success in various multilingual nlp tasks such as offensive content detection @xcite and various zero-shot cross-lingual tasks @xcite . we experiment with different multilingual models and analyze how the different models perform on the downstream task of document classification in subtask 1. we design the document classification problem as a sequence classification problem @xcite . in our approach, we use different transformer models including xlm-roberta @xcite , mbert @xcite and encoderdecoder based laser @xcite to generate embedding from the documents. we experiment with different layers on top of the multi-lingual sentence embedding. our preliminary analysis found that transformer-based xlm-roberta with a sigmoid layer outperformed other models in the macro-f1 score. therefore, in our approach, we propose the xlm-roberta model with a sigmoid classification layer for event pre-diction. xlm-roberta is pre-trained on unlabeled wikipedia text and commoncrawl corpus of 100 languages. the xlm-roberta has a vocabulary size of 25, 000 and uses sentencepiece tokenizer @xcite . we finetuned the model for our task with the training data provided. the training data was highly imbalanced. however, oversampling and under-sampling methods did not provide any marginal improvement in the model's output as per our experiments. ## conclusion this paper describes our approaches for case@emnlp 2022: shared task on socio-political and crisis events detection in multilingual settings. we explored various multilingual and zero-shot approaches and showed results across the languages in subtask 1. we propose xlm-roberta with a sigmoid layer for classifying crisis events in zero-shot and low-resource language settings. our system achieved an average f1 score of 0.73. among the given languages, our proposed approach was able to secure 2 nd place in the hindi document event classification task. while comparing with our approach, the multilingual bert with softmax layer obtained better results for spanish and mandarin, with the result for spanish securing the 4 th spot in the shared task.