V ac LM at BLP -2023 Task 1: Leveraging BERT models for Violence detection in B angla (2023)

Abstract: This study introduces the system submitted to the BLP Shared Task 1: Violence Inciting Text Detection (VITD) by the VacLM team. In this work, we analyzed the impact of various transformer-based models for detecting violence in texts. BanglaBERT outperforms all the other competing models. We also observed that the transformer-based models are not adept at classifying Passive Violence and Direct Violence class but can better detect violence in texts, which was the task’s primary objective. On the shared task, we secured a rank of 12 with macro F1-score of 72.656%.

## introduction in the age of digital empowerment, microblogging sites and social media have ushered in a new era of unfettered expression, providing a global stage for individual voices to be heard like never before. however, this newfound freedom of speech has a darker side, one characterized by the rampant spread of hate speech, cyberbullying, and the toxic dissemination of prejudice across various online platforms. as the digital landscape evolves, so too does the challenge of striking a balance between enabling free expression and curbing the rising tide of online hostility. in this digital dichotomy, the need for innovative solutions to detect and combat hate speech in multiple languages has never been more pressing. while significant progress has been made in identifying hate speech in languages with more resources, bangla, despite being spoken by nearly 230 million people across the globe and characterized by its linguistic richness and diversity, faces a substantial shortage of computational resources, language models, annotated datasets and efficient methodologies needed for effective natural language processing(nlp) tasks. transformerbased models that provide state-of-the-art results * these authors contributed equally to this work in various downstream tasks in european languages lag for bangla @xcite . in this paper, we tried to analyze the impact of transformer-based models on detecting violent inciting text @xcite , specifically aiming to categorize communal violence on social media platforms in the bangla language worldwide. banglabert outperforms all the other competing models with a macro f1score of 72.65% which helped us to secure a rank of 12 on the shared task. we observed that the transformer-based models misclassify passive violence as direct violence but there performance enhances in detecting violence in texts. ## related works numerous methods have been proposed to effectively detect offensive and hateful statements across various platforms, primarily relying on traditional machine learning (ml) techniques, which heavily depend on manual feature engineering. however, ml-based approaches exhibit lower accuracy and also need to improve on scalability issues @xcite . in contrast, methods based on neural networks, particularly deep neural networks (dnns), have the capability to learn more abstract features directly from raw text. prominent dnn architectures, including convolutional neural networks (cnn), long short-term memory (lstm) @xcite , and gated recurrent unit (gru) @xcite , have their advantages. some approaches have amalgamated cnn and lstm into a unified network known as convolutional lstm (convl-stm) @xcite . these hybrid models @xcite have demonstrated superior classification accuracy compared to only neural networks. additionally, pre-trained word embeddings, such as fasttext @xcite and word2vec @xcite , have been employed in conjunction with cnn, lstm, or gru in recent years @xcite ). it's important to note that the majority of these methods have primarily been designed for well-resourced languages like english. consequently, research in nlp for many underresourced languages, such as bangla, is still in its early days. in recent times, language models based on transformers, such as bidirectional encoder representations from transformers (bert) built on attention mechanism and the robustly optimized pretraining approach (roberta) @xcite , have achieved remarkable success in a multitude of natural language processing (nlp) tasks emerging as a natural and highly effective option for addressing the challenges in low-resource languages like bangla. other transformer-based language models such as gpt @xcite , distil-bert @xcite , albert @xcite , electra @xcite has also been proposed for bangla. for indian languages, including bangla, multilingual bert such as xlm-r @xcite , multilingual bert (mbert) @xcite , indicbert @xcite and muril @xcite are available. banglabert @xcite , banglishbert @xcite , sahajbert @xcite are bert models made specifically for bangla. banglabert outperforms all other transformerbased models. banglabert was trained on an extensive 40gb dataset derived from various internet sources, such as news articles, web discussions, blogs, government publications, ted talks, subtitles, newspapers, and articles by crawling data from the web. this naturally led us to choose bert over other techniques, and we anticipated that incorporating additional training data could further enhance our approach. ## dataset the dataset provided for blp shared task 1 @xcite comprises youtube comments primarily from social media discussions related to the nine most significant violent incidents in the bengal region (encompassing bangladesh and west bengal) within the past decade. this dataset is characterized by its content being in the bangla language, with individual comments extending up to 600 words. the dataset is categorized into three classes. they are: the training dataset comprises 2,700 samples, with an allocation of around 15% for direct violence, 34% for passive violence, and the remaining 51% for non-violence instances. in the development dataset, which includes 1,330 samples, 15% pertain to direct violence, 31% to passive violence, and 54% to non-violence occurrences. ## dataset preparation we used the pre-trained models to train on the given dataset and then evaluate the test data provided. since the training dataset only had around 2.7k sentences, we augmented the training dataset by integrating an additional dataset obtained from @xcite , consisting of 30,000 examples, with 10,000 categorised as violence. our approach involved annotating these 10,000 hate speech examples into direct and passive violence categories. this annotation was done manually based on our observation from the original dataset, where sentences containing slang were classified as direct violence. we first cleaned different unicode characters to prepare the dataset and removed punctuations from the sentences @xcite . our next task involved identifying the top 200 words that contributed significantly to the direct violence class (directlist) and the passive violence class (passivelist) from this original dataset.to form the top 200 word list, we first removed stop words from the original dataset and created a word dictionary consisting of each word and its count of occurrence,one word dictionary for each of the classes -direct and passive violence. we then selected the top 200 words that contributed to each of the classes. subsequently, we compiled a corpus comprising slang words in the bangla language. for each hate speech example within the new additional dataset, we assessed its likelihood of belonging to either the direct violence or passive violence class. if the sentence contained any word from the slang word corpus, it was immediately classified as direct violence. otherwise, we evaluated each word in the sentence against the lists directlist and passivelist. if a word was found in either of these lists, the corresponding score for direct violence or passive violence was incremented by 1. in cases where a word appeared in both directlist and passivelist sets, the scores for both classes were incremented by 1. the final classification was determined based on which class had the higher score. in instances of a tie, we labelled the example as passive violence. in this way, all of the 10,000 hate speech examples were categorised into direct and passive violence. to maintain the same class proportions as the original dataset, with non-violence at 51%, passive violence at 34%, and direct violence at 15%, we selected an appropriate number of samples from each class in the newly annotated dataset. ## baseline systems we have used several pretrained models, for the blp workshop task 1 @xcite . ## conclusion we tried to leverage transformer-based models for violence detection in bangla for the blp shared task 1. our analysis shows that the transformerbased models are not adept at segregating direct violence from passive violence but are good at detecting violence-inciting text. we would like to develop models that can accurately classify passive violence in the future.