Mavericks at BLP -2023 Task 1: Ensemble-based Approach Using Language Models for Violence Inciting Text Detection (2023)

Abstract: This paper presents our work for the Violence Inciting Text Detection shared task in the First Workshop on Bangla Language Processing. Social media has accelerated the propagation of hate and violence-inciting speech in society. It is essential to develop efficient mechanisms to detect and curb the propagation of such texts. The problem of detecting violence-inciting texts is further exacerbated in low-resource settings due to sparse research and less data. The data provided in the shared task consists of texts in the Bangla language, where each example is classified into one of the three categories defined based on the types of violence-inciting texts. We try and evaluate several BERT-based models, and then use an ensemble of the models as our final submission. Our submission is ranked 10th in the final leaderboard of the shared task with a macro F1 score of 0.737.

## introduction in today's digital age, numerous social platforms play an important role in connecting individuals around the world. however, certain malicious elements resort to using these platforms to instigate riots, protests, and disturbances that lead to violence. the online posts and comments involve direct threats pertaining to resocialization, vandalism, and deportation while indirect threats involve derogatory language and abusive remarks. these texts which are thought to be a potential reason for instigating violence are called violence-inciting texts. classifying them has become a major challenge and various techniques are used to implement it. these applications can be used to monitor social media websites and take precautions to avoid any mishaps. thus the task boils down to text classification wherein we need to label such texts into predefined categories. ## data we use the vio-lens dataset provided by @xcite for the task. the dataset consists of youtube comments related to nine violent incidents in the bengal region (bangladesh and west bengal) within the past ten years. the comments are in the bangla language with a length of up to 600 words. the dataset consists of two attributes: text, and label. the "text" column contains comments while the "label" column contains three values 0, 1, and 2 representing non-violence, passive violence, and direct violence respectively. the training dataset consists of 2700 samples out of which approximately 15% depict direct violence, 34% portray passive violence and the remaining 51% represent non-violent instances. the development dataset consists of 1330 samples out of which approximately 15% illustrate direct violence, 31% depict passive violence and the remaining 54% represent non-violent instances. the test dataset provided at the time of evaluation consists of 2016 samples as seen in ## system this shared task discusses the problem of violence inciting text detection. this issue falls under the category of classification, for which transformerbased models have seen extensive application and have demonstrated outstanding performance. as a result, we use and experiment with a variety of such models and ensembling techniques in our research. in the section below, the approaches have been briefly discussed. language in this task, muril is specifically relevant. it is trained on two learning objectives, first -masked language modeling, and second -translation language modeling. the model has 236m parameters and a vocabulary of 197285. @xcite states that even though multilingual bert models are suitable for very low-resource languages, models trained on a single language outperform it when sufficient resources for a language are available. based on this assertion, they propose several models for different languages. bengali-bert foot_1 is of specific interest to us. existing multilingual models are fine-tuned on the bangla language corpus to create this model. @xcite demonstrates how crosslingual understanding can be improved by pretraining multilingual models on a large scale. xlm-roberta foot_2 is pre-trained on 2.5tb of filtered com-moncrawl data, which included 100 different languages. it is trained with the multilingual masked language modeling objective. we use the basesized model in our experiments, xlm-robertabase which has 270m parameters. ## ensembling ensembling is a technique that combines the results of various models to generate the system's eventual result. statistical as well as non-statistical methods are used for this purpose. ensembling is useful as it helps generate results that are better than the results given by the individual models. amongst several methods leveraged for ensembling, we use the "hard voting" ensemble technique. in hard voting, the majority vote or the "mode" of all the predictions is selected as the final prediction. it helps improve the robustness of the system and minimizes the variance in the results. the ensembling mechanism is illustrated in figure 1 . in the post-evaluation phase, we experiment with the weighted ensemble keeping in mind the varied performances of the underlying models. we give higher weights to the models which perform model pre-training approach macro f1-score banglabert electra (rtd) 0.733 banglishbert electra (rtd) 0.662 muril mlm 0.720 xlm-roberta mlm 0.705 bengalibert mlm 0.690 ensemble -hard voting -0.737 weighted ensemble -0.745 ## results this section discusses the findings of our experiments. banglabert performs marginally better on the development dataset as seen in ## conclusion we present our approach for the shared task in the first workshop on bangla language processing through this paper. we experiment with several bert and electra-based models as a part of our efforts. we observed that the electra-based banglabert model has the best performance, followed by muril. we can see that the electra-bsased models have similar performances compared to their bert-based counterparts, despite being smaller in size. our final submission consists of predictions generated by ensembling the evaluated models and has a macro f1 score of 0.737, placing us tenth on the shared task leaderboard. our experiments have shed light on several further avenues for improvement. larger pre-training datasets are required for better low-resource models. more sophisticated ensembling techniques can better utilize the performance of individual models and need to be researched further.