Semantics Squad at BLP -2023 Task 1: Violence Inciting B angla Text Detection with Fine-Tuned Transformer-Based Models (2023)

Abstract: This study investigates the application of Transformer-based models for violence threat identification. We participated in the BLP-2023 Shared Task 1 and in our initial submission, BanglaBERT large achieved 5th position on the leader-board with a macro F1 score of 0.7441, approaching the highest baseline of 0.7879 established for this task. In contrast, the top-performing system on the leaderboard achieved an F1 score of 0.7604. Subsequent experiments involving m-BERT, XLM-RoBERTa base, XLM-RoBERTa large, BanglishBERT, BanglaBERT, and BanglaBERT large models revealed that BanglaBERT achieved an F1 score of 0.7441, which closely approximated the baseline. Remarkably, m-BERT and XLM-RoBERTa base also approximated the baseline with macro F1 scores of 0.6584 and 0.6968, respectively. A notable finding from our study is the under-performance by larger models for the shared task dataset, which requires further investigation. Our findings underscore the potential of transformer-based models in identifying violence threats, offering valuable insights to enhance safety measures on online platforms.

## introduction the global use of social media platforms has significantly increased due to the massive use of the internet and over the past few decades, internet use has rapidly expanded. people are now more able to share information and their opinions online because of easier access to the internet. social media use has gained a curve that has been steadily increasing and shows no signs of stopping foot_0 . the study of @xcite includes how some actors use social media platforms to spread false information that covers offensive language, cyber-bullying, cyber-aggression, rumors, and hate speech. as a result, spreading hate speech is now easy and common. this change also gives social media significant influence over our society. on these social media platforms, people are free to express different points of view on religion, politics, education, and other issues. the use of bangla on social media platforms is growing because it is the seventh most spoken language foot_1 , and internet usage is rising. while there has been much research on detecting hate speech in english, there is a knowledge gap in identifying hostile content in low-resource languages like bangla. this type of information generated interest in identifying and flagging it due to its potentially misleading or dangerous nature, which might help stop its future spread. according to research by @xcite , machine learning algorithms are more accurate than keyword searches and textual content analysis for social media data. however, many of the proposed machine learning techniques are, in fact, topic-specific. even though there are some works on sentiment analysis for the bangla language, there has not been much research done recently to identify abusive bangla text on social networking sites. due to the constantly changing nature of social media and the wide range of language used, identifying abusive text is difficult. researchers tried to develop various methods to identify abusive or objectionable text @xcite to prevent abuse on online platforms. in this study, we participated in blp shared task 1: violence inciting text detection (vitd) @xcite and the dataset offered in the shared task has two columns (label, text), with the label in three categories (direct violence, passive violence, and non-violence) @xcite . we conducted a series of transformer-based experiments on the dataset provided by the organizers. in our blind submission with banglabert large, we achieved a macro f1 score of 0.7441, securing the 5th position on the official leaderboard. the highest baseline for the task was a macro f1 score of 0.7879, while the best-performing system, developed by deepblueai, achieved a macro f1 score of 0.7604. subsequently, we re-ran the experiments employing m-bert @xcite , xlm-roberta base @xcite , xlm-roberta large, banglishbert @xcite , banglabert @xcite , and banglabert large @xcite models. banglabert, m-bert, and xlm-roberta base came quite close to the respective official baseline performance in these evaluations. other models, including xlm-roberta large, banglishbert, and banglabert large, demonstrated noteworthy and commendable performance. the structure of this paper is as follows: section 2 summarizes the relevant works for this study. section 3 reports the methodology. a detailed discussion of the results of our study is provided in section 4. finally, we state limitations and future work in section 5. ## related works social media has integrated itself into everyone's daily lives. it makes it possible to communicate quickly, share information easily, and receive opinions across geographical boundaries. however, this manifestation of freedom has also contributed to the development of harsh language and hate speech on social media platforms. unfortunately, the detection of hate speech in bangla social media has received very little attention. the main issue is the unavailability of sufficient data. in addition, the terminology used in hate speech is extremely diverse. social media language frequently deviates significantly from that of traditional print media. numerous linguistic characteristics are present in it. therefore, it is difficult to recognize hate speech automatically. in heterogeneous language-speaking countries like ethiopia, data in amharic was gathered and annotated to detect hate speech by @xcite and proposed a method for automatic detection of hate speech directed towards vulnerable minority groups on social media. the authors reported that rnn-gru exhibits the best performance with an accuracy of 92.56% and an auc of 97.85%. the accuracy of all algorithms improved using word embeddings like word2vec. early research by @xcite created a method for identifying and categorizing hate speech using con-tent from self-identified hate communities on x (formally known as twitter) and suggested that the naive bayes classifier greatly outperformed the existing approaches with 67.47% accuracy. another study with gpt-3 to identify sexist and racist text passages, @xcite discovered that the model accuracy could reach as high as 85% with few-shot learning. @xcite created the hs-ban dataset on hate speech with their benchmark system, and the best outcome was obtained by combining bi-lstm with ft(sg) or bi-lstm+ft(sg), which achieved an f1 score of 86.85%. (ishmam and sharmin, 2019) built a dataset with 5,126 bangla comments from social media and got an accuracy of 70.1% using grubased models, which gave 18% higher accuracy than ml algorithms. a recent study by @xcite using several publicly accessible datasets for the experiments by fine-tuning multilingual transformer models for bangla text classification tasks in several areas to improve accuracy upon the prior results between 5%-29% across different tasks. the dataset for this study was obtained from x (previously known as twitter). according to @xcite , the xlm-roberta model has the highest accuracy on their developed annotated dataset which consists of 10k bangla posts where 5k is actual and 5k is romanized bangla tweets. by preparing only a multi-modal hate speech dataset, after experiments @xcite reported f1 scores of 78% and 82%, respectively, using conv-lstm and xlm-roberta models, which scored best for texts. resnet-152 and densenet-161 models produced f1 scores of 78% and 79% for memes, respectively. concerning multi-modal fusion, xlm-roberta + densenet-161 demonstrated the best performance, producing an f1 score of 83%. @xcite took data from some controversial pages of social media and after evaluation, the maximum accuracy of 88% was achieved by svm using the entire dataset. on the dataset of @xcite , the authors ran baseline experiments, applied several deep learning models, and extensively trained word2vec, fasttest, and bengfast-text models on bangla words to facilitate future research opportunities, and the experiment showed that svm had the best outcome with 87.5% accuracy. ## experimental methodology this section describes our experimental methodology. we start with a brief overview of the dataset, then talk about our pre-processing steps, and present in-depth explanations of the models used in this study. ## conclusion and future work in this study, we performed a comparative study on several transformer-based models to detect violent text. we used the dataset offered by shared task 1 of the blp workshop for this study. we used such as m-bert, xlm-roberta base, xlm-roberta large, banglishbert, banglabert, and banglabert large to compare their result. the result shows that banglabert outperformed other models in terms of performance measures. despite being larger models m-bert, xlm-roberta large, banglabert large could not outperform banglabert. one of the limitations of our work is that we were not able to reveal the specific reason why our large models are not performing as they are supposed to for this task. to extend this study, we plan to employ transfer learning, efficient model designs, or model compression to improve the performance of large models such as m-bert, xlm-roberta, and banglabert large. in the context of hate speech detection, we will investigate the ideal hyperparameters for transformer-based models, including learning rate schedules, model size, and optimization strategies. the development of more accurate, fair, and reliable hate speech detection algorithms may emerge from future research in these areas, thus, resolving the limitations of this study and enhancing the field of nlp.