NSUT - NLP at CASE 2022 Task 1: Multilingual Protest Event Detection using Transformer-based Models (2022)

Abstract: Event detection, specifically in the socio-political domain, has posed a long-standing challenge to researchers in the NLP domain. Therefore, the creation of automated techniques that perform classification of the large amounts of accessible data on the Internet becomes imperative. This paper is a summary of the efforts we made in participating in Task 1 of CASE 2022. We use state-of-art multilingual BERT (mBERT) with further fine-tuning to perform document classification in English, Portuguese, Spanish, Urdu, Hindi, Turkish and Mandarin. In the document classification subtask, we were able to achieve F1 scores of 0.8062, 0.6445, 0.7302, 0.5671, 0.6555, 0.7545 and 0.6702 in English, Spanish, Portuguese, Hindi, Urdu, Mandarin and Turkish respectively achieving a rank of 5 in English and 7 on the remaining language tasks.

## introduction protests exist as a natural way for citizens of a nation to show their dissatisfaction with decisions taken by the respective governments or authorities @xcite . the sentiment prevalent in such events and the reaction by various parties to these events provide the basis for carrying out many studies in the sociopolitical field, such as the public opinion about the event that was the cause for the protest, how much freedom the protesters were afforded as a measure of the democracy in the nation, and so on. with the advancement of technology, there has also been an exponential rise in the use of social networks as a medium for exchanging information across the globe, with global events and their inner nuances being available to the public at large. however, extracting valuable insights from such events on a national or global scale is a daunting task if done manually @xcite . even if we leverage automated techniques for the process, there are numerous challenges faced while working on multilingual data @xcite . hence, there exists an incentive to automate the task of processing protest news from multiple locations and in multiple languages and to create an nlp system that could be generalized for the task of detecting protest news. task 1 of case 2022 @xcite aims at working on multilingual protest news corpora, with subtask 1 working towards the binary classification of news reports, where if a document reports on an event that has happened or is ongoing, it is marked as relevant, otherwise it is considered irrelevant. our approach revolves around the use of stateof-art pre-trained language models (plms) and finetuning them to perform the task we require. we leverage the bert-base-multilingual-cased @xcite that was trained in over 104 languages to tackle the multilingual task. we finetune it for protest news detection. since most of the training datasets had a bias toward the negative class, we augmented the datasets by translating positive samples from other language datasets and hence improving the balance between the positive and negative class to prevent our model from being biased towards the negative class. furthermore, the lack of samples in portuguese and spanish presents us with a few-shot learning scenario, which we tackle by augmenting these datasets with samples from the english dataset translated into the respective languages. for languages with no training datasets (urdu, turkish, mandarin and hindi), we created training datasets by translating the english corpus. the rest of the paper is organised as follows: we begin by laying out the past literature and work done in the field of protest event detection in section 2 followed by the description of the task at hand and the data given to us in section 3. in section 4, we describe the techniques we employed, namely data augmentation and the model we used, multilingual bert (mbert). the experimental setup for our system is described in section 5 and the results on the test set are mentioned and analysed in section 6. finally, in section 7, we draw a conclusion to our work and go over prospective directions for additional research. ## related work protest detection and allied fields have drawn a lot of attention from researchers in the nlp domain. maven @xcite and @xcite are annotated datasets in the english language created for the purposes of event detection. ace 2005 @xcite and tempeval-2 @xcite are multilingual datasets where ace 2005 covers english, arabic, and chinese and tempeval-2 covers chinese, english, french, italian, korean and spanish. @xcite is another multilingual ed dataset covering 8 different languages (english, spanish, portuguese, polish, turkish, hindi, japanese and korean). mm-chived @xcite ) is another dataset containing multimodal data like text and images compiled from social media regarding chile and venezuela protests. there have also been region-specific case studies, such as detection of protest events in turkey 2013 @xcite and protest analysis in greece over the last twenty years through the scope of computational social science (papanikolaou and papageorgiou, 2020). previously event detection has also been researched upon by researchers participating in the task 1 of case 2021 @xcite . teams which participated in the task earlier have used multilingual pre-trained language models @xcite @xcite which is similar to the approach used by our system. ## experimental set-up the models were developed on keras 1 @xcite , and implemented using the transformers library by huggingface 2 @xcite . the model used is 1 https://keras.io/ 2 https://huggingface.co/docs/transformers/ index bert-base-multilingual-cased foot_0 . we use the autotokenizer foot_1 offered by huggingface's transformers library to tokenize our inputs. we experimented with learning rates of 1e-5, 3e-5 and 5e-5 for all models, finding the best results at 3e-5. for all the models, we fixed the max length parameter at 512 tokens and the batch size parameter to 6. the finetuning for the models was performed on google colab gpu. we trained each model for 3-4 epochs and found the best results at 4 epochs. the dropout rate during fine-tuning is 0.2. we used the adam @xcite optimizer from keras. the loss function used is binary cross-entropy. the translation was performed using the google translation library in python googletrans(v3.1.0a0) foot_2 . ## results and discussion our model seems to have performed better on the english dataset indicating that the multilingual bert has a better contextualizing ability for the lingua franca, english. the preprocessing process involves removal of accents which might be detrimental to performance of many languages which heavily rely on accents such as hindi, urdu, turkish and spanish. for example, in turkish ı and i (non -dotted and dotted) are very different vowels with the phonetic sounds ( as in cycle -sīkl ) and ē (as in easy -ēzē). ## conclusion and future work the amounts of publicly available data on the internet, especially social networks, desire for skillful analysis for the purposes of protest detection. this becomes especially imperative because of the significance of protests in the social, political and economic domains. our submission in task 1 of case 2022 demonstrated the effective use of pretrained language models (plms), specifically multilingual bert (mbert) in the binary classification of documents into events or not events. we were also successful in tackling the dearth in training data and class imbalance using data augmentation. we have been able to achieve f1 scores of 0.8062, 0.6445, 0.7302, 0.5671, 0.6555, 0.7545 and 0.6702 in english, spanish, portuguese, hindi, urdu, mandarin, and turkish respectively. in the future, we can deal with class imbalance using class weighing @xcite . we would also like to experiment with cross-lingual finetuning on a multilingual model by training in one language and testing in another language. we would like to extend this work by using language specific plms rather than a multilingual model.