BLP -2023 Task 1: Violence Inciting Text Detection ( VITD ) (2023)

Abstract: We present the comprehensive technical description of the outcome of the BLP shared task on Violence Inciting Text Detection (VITD).In recent years, social media has become a tool for groups of various religions and backgrounds to spread hatred, leading to physicalviolence with devastating consequences. To address this challenge, the VITD shared task was initiated, aiming to classify the level of violence incitement in various texts. The competition garnered significant interest with a total of 27 teams consisting of 88 participants successfully submitting their systems to the CodaLab leaderboard. During the post-workshop phase, we received 16 system papers on VITD from those participants. In this paper, we intend to discuss the VITD baseline performance, error analysis of the submitted models, and provide a comprehensive summary of the computational techniques applied by the participating teams

## introduction social media's growth over the past decade has reshaped the distribution of information to the broader public @xcite . however, it has also surfaced as a potential breeding ground for provoking violence among different groups, from religious to ethnic to gender-based distinctions. in fact, many of the violent incidents of the recent past era can directly or indirectly be attributed to incitement from social media @xcite . such platforms can act as catalysts for the incitement of violence and the radicalization of * † authors have equal contributions individuals or groups @xcite . extremist ideologies and hate speech can spread rapidly, leading to real-world acts of violence. acts of violence, triggered or fueled by content shared on social media, can inflict physical harm to individuals and communities with dire consequences that include physical injuries, destruction of properties, and even loss of human lives. in the recent past, numerous studies were conducted into areas like hate speech detection @xcite , abusive content identification @xcite , and misinformation detection @xcite , aiming to understand and prevent harmful social media activities. there have been several workshops that contributed datasets and organized shared tasks on online harmful content detection in different languages @xcite . however, to the best of our knowledge, there exists no research work on the violence incitement in the bengal region (bangladesh and west bengal in india), the residence of more than 272 million foot_0 , foot_1 people of many diverse background. therefore, this shared task seeks to bridge this gap by contributing a novel dataset on vitd for the development of new systems and methodologies with the objective to advance our collective understanding and capabilities in this crucial domain. in this paper, we discuss the following: ## participants system description in this section, we present a comprehensive summary of each submitted system for the shared task. aambela @xcite stood second in the competition with an overall macro-f1 score of 76.040 for the test set. they propose an instructionfinetuned csebuetnlp-banglabert @xcite with three classification heads. as banglabert's vocabulary does not fully cover the tokens in the data, the team added them as special tokens that were learned during the training phase. they also observe the significance of emojis in the dataset, and removing them often leads to a minor result. on the other hand, converting emojis to text and normalizing the text leads to a better result. they experimented with various approaches such as traditional classifiers (svm, random forest, xg-boost) with tf-idf embeddings, deep learning models (lstm), and transformer-based architectures (mbert-case, mdeberta-v3 base @xcite , xlm-roberta base, sagorsarker-banglabert @xcite , banglabert @xcite . finally, banglabert trained on three epochs with a batch size of 16 came out on the top. nlp_cuet @xcite achieved 3rd rank in this task with an overall macro-f1 score of 74.587. they preprocessed data by removing unwanted characters and employed feature extraction methods like tf-idf and word2vec. after investigating several machine learning, deep learning, and transformer-based models, they propose a hybrid method using gan @xcite and bangla-electra. here, they considered both labeled data and unlabeled data for model training. the generator and discriminator are both multilayer perceptrons with a single hidden layer of 512 neurons. the generator input is a randomly generated vector of 100 dimensions, and it outputs a fake transformer embedding vector for a single token. the transformer-based model processed the input text, generating a contextualized embedding vector for the cls token. these embedding vectors from the transformer and generator were then input into the discriminator. the output of the discriminator is extended to k+1 classes where k is the number of classes in this classification task, and the extra class is "real." in this approach, they focused on determining whether the embedding produced by the transformer-based architecture is real or fake. during the testing phase, they discarded the generator and used the bert and discriminator model to classify the input data. they masked the prediction output for the 'real' class during testing. @xcite received the fifth rank with an overall macro-f1 score of 74.413. they applied a preprocessing step of removing punctuation, lemmatization, and oversampling/undersampling. afterward, they used different transformer-based models such as xlm-roberta (base and large), banglabert (bhattacharjee et al., 2022) (base and large), and mbert. among the approaches, banglabertbase achieved the highest result. ## limitations quantitative limitations: the main limitation of the shared task arises from the dataset. first of all, the dataset is small in size, with only 4k data points for the training and validation sets, and around 2k data points for the test set. this often creates problems in terms of over-fitting on large models. additionally, the dataset is highly imbalanced with only a minor fraction of the data for direct violence creating a challenge for class detection which is also reflective of the participant's results. qualititive limitations: emojis play a crucial role in sentence classification, so removing any emoji during preprocessing leads to a loss of context. the dataset consists of data from bangladesh and west bengal, comprising only in bengali language. therefore, the nature of violence-inciting text's nature may differ based on culture and language. finally, the dataset requires a hectic process to annotate and validate thus expanding the correct data is much more difficult. ## conclusion in this paper, we have presented an overview of the shared task on the violence inciting text detection (vitd) dataset. the contest fostered submissions from 27 teams with 16 teams outperforming the highest baseline system banglabert @xcite , and 17 teams submitted the system paper. the system description and subsequent analysis and limitations discussion demonstrate the successful completion of the task. the task has some vast scope for improvement. as mentioned in @xcite , there is significant unlabelled data ready for further improvement of the systems to invoke larger systems without over-fitting the larger models. a potential scope for improvement is adding more data from huge unlabelled data. a future version of the task may be arranged with the challenge of more data from dif-ferent sources, languages, and regions. also, realtime violence detection models can be the next step of the task.