SUST _ B lack Box at BLP -2023 Task 1: Detecting Communal Violence in Texts: An Exploration of MLM and Weighted Ensemble Techniques (2023)

Abstract: In this study, we address the shared task of classifying violence-inciting texts from YouTube comments related to violent incidents in the Bengal region. We seamlessly integrated domain adaptation techniques by meticulously fine-tuning pre-existing Masked Language Models on a diverse array of informal texts. We employed a multifaceted approach, leveraging Transfer Learning, Stacking, and Ensemble techniques to enhance our model’s performance. Our integrated system, amalgamating the refined BanglaBERT model through MLM and our Weighted Ensemble approach, showcased superior efficacy, achieving macro F1 scores of 71% and 72%, respectively, while the MLM approach secured the 18th position among participants. This underscores the robustness and precision of our proposed paradigm in the nuanced detection and categorization of violent narratives within digital realms.

## introduction while fostering connections and facilitating information dissemination, social media has inadvertently become a platform for propagating hostility. such hateful actions, encompassing communal violence, cyberbullying, and social platform attacks disrupt online communities and erode the foundational trust and safety intrinsic to such platforms @xcite . by utilizing the latest advancements in artificial intelligence and natural language processing (nlp), we can effectively identify and prevent potential violent incidents, thus creating a safer environment. in this context, we will examine the blp shared task 1: violence inciting text detection (vitd). recent advancements in the field have highlighted the potential of informal text embeddings in enhancing the accuracy of hate speech (hs) † these authors have equal contributions figure 1 : words after exclusion of words in neutral class and discarding most of the positive and neutral words detection, evidenced by the work of @xcite . furthermore, the advent of masked language model (mlm) pre-training, exemplified by models such as bert @xcite , has revolutionized text classification tasks. the landscape of nlp has been significantly shaped by the adoption of transfer learning in recent years. pioneering methodologies such as ulmfit @xcite have demonstrated the superiority of fine-tuning language models over traditional deep learning algorithms, especially when confronted with limited datasets and resources. this paradigm shift is further exemplified by models like banglabert @xcite , which builds upon the foundational bert architecture, benefiting from extensive pretraining on diverse datasets. the burgeoning interest in bangla text classification has catalyzed the development of several pivotal datasets and transformer-based approaches @xcite @xcite , further enriching the ecosystem and setting the stage for our research. our approach to the vitd task @xcite ) is informed by these advancements, leveraging transfer learning (tl) and mlm training to incorporate informal texts into our models. during training, we have used a large volume of similar informal data collected from various domains @xcite @xcite using domain adaptation along with the vitd dataset. utilizing our approaches, we have gotten better results than the benchmark models. ## dataset the class distribution of training, validation, and test set of the vitd dataset by @xcite , facilitating the main classification task, is shown in to get a deeper insight into the linguistic traits of the av and pv classes, all words of the neutral class were excluded from the av and pv classes. the resultant set of words of av class and pv class is represented by the closed circular curve on the left and right respectively of the venn diagram (figure 3 ). this av and pv set consists of 2702 and 8259 words respectively, while the intersection contains 245 words. from the word samples presented in the venn diagram, the words unique to av class(excluding the set of pv words from av set) encompass most of the words that indicate violence of some form, and the words unique to pv class(excluding the set of av words from pv set) hold most of the words related to dehumanization. while words common to both classes are predominantly linguistically dehumanizing, only a small portion of them consist of violence-inciting words. the ratio of dehumanizing-natured words within the intersection set significantly exceeds the ratio of such words in the exclusive pv class set. in these sets, neutral words also exist in a significant amount. 3 system description 3.1 dataset for mlm training and tl task related datasets have been divided into three groups based on their usage, as shown in for mlm training, 9,674 text samples labeled as 'negative' sentiment from the banglabook dataset by @xcite , 6,807 text samples labeled as 'aggressive' from the bad dataset by @xcite , 24,156 text samples labeled 'hate speech' from the bd-shs dataset by @xcite and negative emotion-sentiment dataset from @xcite . this group also includes the vitd (train and dev) dataset. only the negative class samples have been taken for mlm training. the bd-shs containing 50,281 samples labeled 'hate speech' or 'non hate' has been used for tl. used for dataset(s) mlm training banglabook, bad, bd-shs, emotion-sentiment transfer learning bd-shs (with labels) fine-tuning vitd (with labels) ## experimental setup we have presented our approach to strengthen a vitd model using a pre-trained mlm which is an electra model based on transformers network @xcite which we have referred to as mlm approach. to facilitate the vitd model, first we have used pre-trained mlm on texts to comprehend contextual representations. the final classification has been done by freezing the 6 encoder layers of the electra model and fine-tuning the hyperparameters of the model. during both processes, we utilized a learning rate of 2e-5 and ran the model for 10 and 50 epochs respectively in which the epoch with the highest macro f1-score is stored as the final result. for stacking, we have trained a deep neural network for 21 epochs with a learning rate of 0.03. for we, we have assigned the fine-tuned banglabert a weight of 3 and other tl-based models a weight of 1. we have used the mini-batch training paradigm for our experiments. corresponding all the codes are publicly available at this repository. foot_0 ## results we present our results using the macro f1 score for both the validation and test datasets, as detailed in ## discussion in this section, we present the results of our experiments with mlm and tl methodologies, which have outperformed the base models. the primary reason for this improvement is the inclusion of informal words that were previously absent in the pretraining datasets of the pre-trained models. to further optimize our results, we used ensemble techniques. we prioritized mlm within the weighted ensemble (we) framework by assigning it the highest weight, recognizing its superior accuracy. interestingly, we found that integer weights of we predominantly excelled in av class detection, despite our initial expectations of learned weights from stacking yielding superior outcomes. this also highlights the importance of the inclusion of models in we trained on both validation and training sets. to improve our outcomes further, we integrated upsampling, which, in certain instances, led to improved outcomes. during training with upsampled data, our approach of freezing the embedding layer throughout the training process and selectively freezing and unfreezing different layers at various stages of training lessens the chance of overfitting. lastly, being dominated by the majority neutral class, the micro f1 score is considerably higher compared to the macro f1 score. this indicated that, as backed up by individual class f1, the finetuned models were able to classify between neutral and non-neutral classes more rigorously. ## conclusion and future work our experiments aimed to explore various approaches to integrating informal words, and we found that the mlm and we methods performed the best. our mlm and tl approaches are still unexplored for all bert baseline models, including exploring based on the same models. discovering the effects of our approaches and their comparison will lead to promising future research directions and help improve our methods' robustness and scalability. the effect of freezing embedding layers and, selectively freezing and unfreezing other layers on overfitting due to upsampled data still needs in-depth study. as we worked to familiarize pretrained models with the nuances of informal words for the vitd task in bangla, we hope to contribute to safer online spaces for everyone and unlock new frontiers in nlp.