T eam_ S yrax at BLP -2023 Task 1: Data Augmentation and Ensemble Based Approach for Violence Inciting Text Detection in B angla (2023)

Abstract: This paper describes our participation in Task1 (VITD) of BLP Workshop 1 at EMNLP 2023,focused on the detection and categorizationof threats linked to violence, which could po-tentially encourage more violent actions. Ourapproach involves fine-tuning of pre-trainedtransformer models and employing techniqueslike self-training with external data, data aug-mentation through back-translation, and en-semble learning (bagging and majority voting).Notably, self-training improves performancewhen applied to data from external source butnot when applied to the test-set. Our anal-ysis highlights the effectiveness of ensemblemethods and data augmentation techniques inBangla Text Classification. Our system ini-tially scored 0.70450 and ranked 19th amongthe participants but post-competition experi-ments boosted our score to 0.72740.

## introduction in today's social media-driven world, easy selfexpression has brought a downside: a surge in harmful, violent content harming people physically and mentally @xcite . this critical concern needs addressing. the emnlp-2023 organized the blp shared task 1 (vitd) @xcite , addressing a vital challenge: identifying violence-inciting text in bangla. the aim was to build models that identify violent content, especially content that might provoke more violence. yet, in bangla, this is tough due to limited language resources. the text from youtube comments often lacks clear context, making it even harder to understand. additionally, the dataset @xcite used in this task is relatively small, limiting the variety of language patterns. to overcome these issues, we used pretrained transformer models, fine-tuning them with vitd dataset. we also applied techniques like selftraining on external data, back-translation for data 1 https://blp-workshop.github.io/ augmentation, and ensemble learning (bagging and majority voting). these techniques, particularly when combined with self-training and backtranslation, as well as ensemble approach across multiple models, moderately improved our model's performance. post-competition experiments, including selftraining on external data and back-translation, raised our score to 0.72740. this paper details our approach, challenges, and methods for addressing violence-inciting text in bangla. ## related work hate speech, cyberbullying, harassment, and incitements to violence on social platforms can harm individuals and communities in online spaces. increasing studies have been undertaken to detect violent content on social media @xcite @xcite . people usually confront violence on social media through text, images, and videos. researchers use natural language processing @xcite to analyze text, visual, and audio content on social media sites. these excellent initiatives are happening worldwide in many languages. implementing the same method in languages with low resources, like bangla, is problematic @xcite . poorly annotated bangla-language violence detection datasets are a widespread issue @xcite . bangla has a large vocabulary and several sentence forms @xcite . bangla dialects vary by region, which might alter text interpretation. although bangla is a low-resource language @xcite with its own issues, numerous studies @xcite are being undertaken to identify social media breaches in this language context. modern models, such as bert @xcite , have been substantially altered and used in these studies. these evolving investigations are encouraging us to use these modern approaches for low-resource lan-guages like bangla @xcite . changing studies have provided new perspectives on violence recognition in bangla @xcite and expanded our knowledge of it. banglabert @xcite was a key tool in our study for addressing blp task 1. we wanted to get the most out of ensemble methods by using pre-trained transformers in our experiments @xcite . this is because the field of transformer model applications is still growing and changing @xcite . we were able to use the combined knowledge of several cutting-edge transformer models with this new method, which made our experiments more in-depth and varied. several well-thought-out tests with multiple models have yielded key results and refined our method to make it more accurate. ## task and dataset description the blp shared task 1, known as violence inciting text detection (vitd), offers an outstanding chance to address the significant problem of detecting violence-inciting text. the dataset being analyzed consists of youtube comments containing the most significant violent incidents. three distinct categories are established for the purpose of classification: direct violence, which includes explicit threats targeted towards individuals or communities; passive violence, which involves the utilization of derogatory language, abusive remarks, or justification of violence; and non-violence, which encompasses content that is unrelated to subjects involving violence. the task has a role in the identification and mitigation of potential threats that may lead to violent situations. the vitd dataset is divided into three subsets: the training set, development set, and testing set, all of which are formatted in csv structure. each entry within these csv files consists of two key columns: "text" and "label." the "text" column contains textual data collected from various social media sources, while the "label" column assigns a numerical value of 0, 1, or 2 to each entry, representing different categories of violence: non-violence, passive violence, and direct violence, respectively. in appendix a.1, as shown in figure 4 , we tried to visualize the category distribution within each set and noticed that, the datasets are highly skewed towards non-violence. occurrence of direct violence is very rare. the class distribution within the dataset is imbalanced, with non-violence being the dominant category. detecting and classifying the less frequent instances of passive violence and direct violence poses a significant challenge. we also tried to visualize the texts associated with the labels through wordclouds in figure 5 , 6, 7 in the a.1 appendix section. the distribution of words in the wordcloud provides some insights. we discovered some words that are uniquely associated with a given label. along with that, we also noticed, the datasets contain instances of ambiguous labeling, in which the categorization of text into the correct category of violence is difficult due to the complexity and ambiguity of the language. given the nature of text inciting violence, the dataset may contain instances of religious bias. during annotation, it is crucial to deal with this sensitivity and maintain an ethical perspective. ## limitations and error analysis error analysis is challenging in this task. a model may fail on certain datasets for many reasons. our top performing model and the test dataset indicate the model's inaccurate classification of certain texts as direct violent or passive violent, and vice versa. disparities in dataset labeling are a big issue. why certain texts are labeled "2" for direct violence and others "1" for passive-violent texts is unclear. for instance, "2" is placed next to longer sentences strengthen the model's contextual foundation, enabling more exact classification. after thoroughly studying the test set, we observed 472 label discrepancies between test set labels and best model predictions. our model identified 207 of these texts as non-violent (label 0), while the test set classified them as passive-violence (label 1). the second greatest label differences was 91 instances between the test set's identification of texts as non-violent (label 0) and our model's labeling as passive violence (label 1). more than 50% of the mistakenly predicted classifications are non-violence and passive violence. this gap may be due to subtle distinctions between indirect passive violence and non-violence sentences. besides, back-translation data augmentation improved model performance, but it might alter text meaning and structure, therefore nlp tasks should be used with caution. it is important to evaluate this potential impact on augmented data quality. 7 conclusion and future work the objective of this research was to classify texts into three groups and determine whether or not they promote violence in any way. we have experimented with some prominent transformer based models for text classification before trying out other approaches to make those models perform better. after the test set was made public, we were able to strengthen the performance of our model by running further tests. in order to accurately identify violent texts in social media comments, there is still work to be done in the future. it is necessary to conduct more and more experiments with low resource languages like bangla. we think that our efforts prepared the groundwork for this to happen. processing (blp-2023), singapore. association for computational linguistics. sourav saha, jahedul alam junaed, maryam saleki, arnab sen sharma, mohammad rashidujjaman rifat, mohamed rahout, syed ishtiaque ahmed, nabeel mohammad, and mohammad ruhul amin. 2023b. vio-lens: a novel dataset of annotated social network posts leading to different forms of communal violence and its evaluation. in proceedings of the 1st international workshop on bangla language processing (blp-2023), singapore. association for computational linguistics. rico sennrich, barry haddow, and alexandra birch. 2016. improving neural machine translation models with monolingual data. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 86-96, berlin, germany. association for computational linguistics. omar sharif, eftekhar hossain, and mohammed moshiul hoque. 2022. m-bad: a multilabel dataset for detecting aggressive texts and their targets. in proceedings of the workshop on combating online hostile posts in regional languages during emergency situations, pages 75-85. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, ł ukasz kaiser, and illia polosukhin. 2017. attention is all you need. in advances in neural information processing systems, volume 30. curran associates, inc. marcos zampieri, preslav nakov, sara rosenthal, pepa atanasova, georgi karadzhov, hamdy mubarak, leon derczynski, zeses pitenis, and çagrı çöltekin. 2020. semeval-2020 task 12: multilingual offensive language identification in social media (offenseval 2020). arxiv preprint arxiv:2006.07235.