IIITT at CASE 2021 Task 1: Leveraging Pretrained Language Models for Multilingual Protest Detection (2021)

Abstract: In a world abounding in constant protests resulting from events like a global pandemic, climate change, religious or political conflicts, there has always been a need to detect events/protests before getting amplified by news media or social media. This paper demonstrates our work on the sentence classification subtask of multilingual protest detection in CASE@ACL-IJCNLP 2021. We approached this task by employing various multilingual pre-trained transformer models to classify if any sentence contains information about an event that has transpired or not. We performed soft voting over the models, achieving the best results among the models, accomplishing a macro F1-Score of 0.8291, 0.7578, and 0.7951 in English, Spanish, and Portuguese, respectively.

## introduction the recent surge in social media users has led many people to express their opinions on various global issues. these opinions travel far and wide within a matter of seconds @xcite . this can influence many people and may engage public movements @xcite . therefore, there is a definite need to detect these protests and analyse them to know the significant areas of disinterest. being a free and easy to use platform, social media has become a part of our day to day life. it incorporates people of different ages, gender, location, religions, background, and so on. the enormous number of rich and diversified users results in an enormous amount of information being generated, which is helpful in many ways @xcite . some of this even contains private information about the users, which others could misuse. cases were also found where certain users were being targeted and harassed by people using this platform, a common scenario in cyberbullying @xcite . social media plays a crucial role in amplifying these protests and movements @xcite . it enables political groups and protesters to organise protest movements and share information. it acts as a platform for the people who are underrepresented by giving a voice to them. it also offers new opportunities for people to engage in activism, political resistance, and protest outside the political groups and civic institutions. thus, it has a social impact on everyone @xcite . it is to be noted that social media, similar to news media, plays a vital role in its social and political events worldwide @xcite . for the above reasons, we can state that social media plays a crucial role in most worldwide events. the english language is widely regarded as the first lingua franca. statistically, it is one of the most widely spoken languages globally, having official status in over 53 countries @xcite . over 400 million people speak english as their primary language and widely spoken in the united states and the united kingdom. blacklivesmatter (dave et al., 2020), earthday @xcite are some of the major protests that have occurred in these countries. español commonly referred to as spanish, is spoken by over 360 million people worldwide, with most of its speakers residing in mexico, argentina, spain. 15-m movement @xcite and yosoy132 (garcía and treré, 2014) are some of the recent protests where people have been vocal about in the spanish language. portuguese has over 220 million native speakers. brazil, portugal, angola are some of the major countries where this language is spoken. protests like racism kills, may 68 @xcite are the recent ones that occurred in the portuguese language. the recent upheavals of protests are due to so-sentence language label fabius ran against royal for the presidential nomination in 2007. english event he planned to start a race war. english event metro police intervened and the fire was put out. english not-event pero no es ése el mayor problema. spanish event la argentina retrocedería un paso todos los días. spanish event carrió no objetó que se trató de un secuestro. spanish not-event os servidores do piauí estão em greve há 17 dias. portuguese not-event é uma nova experiência mobilizatória. portuguese event é decidiram ir às aulas e passar o dia de saia. ## related work the need to detect events that could lead to protests is of prime interest to sociologists and governments @xcite . there are several active ongoing projects for socio-political event systems such as keds (kansas event data system) @xcite , cameo (conflict and mediation event observation) @xcite , and several other databases for protest de-tection systems @xcite . these methods have focused on news data as they have traditionally been the most reliant source of events. protest detection has been one of the major issues in the context of social and political @xcite . @xcite presented a computational social science methodology to analyse protests in greece. @xcite constructed a corpus of protest events comprising various language sources from various countries. several systems were submitted to the clef protestnews track that consisted of three shared tasks, primarily aimed at identifying and extracting event information spanning to multiple countries @xcite . ## dataset this dataset comprises 26,208 sentences in three languages, namely english, spanish, and portuguese. the dataset consists of two classes: the volume of sequences indicating not-event is higher in contrast to that of the event label. therefore, the dataset distribution is quite imbalanced. we can also notice that the number of english samples exceeds that of spanish and portuguese ones. refer to ## methodology we used pretrained transformer-based models for identifying if a sentence talks about an event or not. the models that were used are bert @xcite , roberta @xcite and distilbert @xcite . even though there are 3 different languages, we used a single model for all three due to memory constraints and reduced training time. we fine-tuned these models for sequence classification. soft voting is done on all these models to produce the respective final outputs for the languages. in soft voting, each classifier predicts that a specific data point belongs to the particular target class. a weighted sum of the predictions is done based on the importance of the classifier (all models have equal weights). the overall prediction is chosen as the target with the greatest sum of the weighted probability, thus winning the vote @xcite . ## results and analysis all pretrained language models are fine-tuned in google colab 2 for ten epochs. we use the tensorflow implementation of the models 3 on the huggingface transformers library 4 . we compare the macro f1-scores of our fine-tuned models on the validation set, which were created by splitting the given dataset. the remaining split is the training data. the validation set contains samples from all three languages. it has 4,387 not-event sequences and 963 event sequences making a combined total of 5,350. the results are shown in table3. we fine-tuned bert, distilbert, and roberta models on the training set. we have combined the language macro f1-score english 0.8291 spanish 0.7578 portuguese 0.7951 based on the performance of soft voting on the validation set, we have used the same for the test set. the results for the test set are shown in table5. the reason for relatively low scores of spanish and portuguese could be due to the inadequate support of the training set (2,741 and 1,182) instead of @xcite . we also believe that our approach of combining datasets could have influenced the performance of the low support datasets. ## conclusion the need to develop automated systems to detect any event is an active protest has constantly been increasing because of the escalation of social media users and several platforms to support them. in this paper, we have explored several multilingual language models to classify if a given sentence talks about an event that has happened (event) or not (not-event) in three languages. our work primarily focuses on fine-tuning language models and feeding them to an architecture we created. we also observe that the problem of class imbalance has had a significant impact on the performance of the models. the soft voting approach has achieved macro f1-scores of 0.8291, 0.7578, and 0.7951 for english, spanish, and portuguese, respectively. for future work, we intend to explore class weighting techniques and semi-supervised approaches to improve our performance.