M - BAD : A Multilabel Dataset for Detecting Aggressive Texts and Their Targets (2022)

Abstract: Recently, detection and categorization of undesired (e. g., aggressive, abusive, offensive, hate) content from online platforms has grabbed the attention of researchers because of its detrimental impact on society. Several attempts have been made to mitigate the usage and propagation of such content. However, most past studies were conducted primarily for English, where low-resource languages like Bengali remained out of the focus. Therefore, to facilitate research in this arena, this paper introduces a novel multilabel Bengali dataset (named M-BAD) containing 15650 texts to detect aggressive texts and their targets. Each text of M-BAD went through rigorous two-level annotations. At the primary level, each text is labelled as either aggressive or non-aggressive. In the secondary level, the aggressive texts have been further annotated into five fine-grained target classes: religion, politics, verbal, gender and race. Baseline experiments are carried out with different machine learning (ML), deep learning (DL) and transformer models, where Bangla-BERT acquired the highest weighted f_1 -score in both detection (0.92) and target identification (0.83) tasks. Error analysis of the models exhibits the difficulty to identify context-dependent aggression, and this work argues that further research is required to address these issues.

## introduction social media platforms have become a powerful tool to spontaneously connect people and share information with effortless access to the internet. these platforms provide users with a cloak of anonymity that allows them to speak their opinions publicly. unfortunately, this power of anonymity is misused to disseminate aggressive, abusive, hatred and illegal content. in the recent past, these mediums have been used to incite religious, political and communal violence @xcite . a significant portion of such incidents has been com-municated through textual content @xcite . therefore, it has become crucial to develop automated systems to restrain the proliferation of such undesired or aggressive texts. this issue has been taken seriously in english, german, and other high-resource languages @xcite . however, minimal research effort has been made in low-resource languages, including bengali. systems developed in english or other languages can not detect detrimental texts written in bengali due to the significant variations in language constructs and morphological features. nevertheless, people use their regional language to communicate over social media. therefore, developing benchmark datasets and regional language tools is monumental to tackle the undesired text detection challenges. this work develops m-bad containing 15650 texts using a two-level hierarchical annotation schema. in level-1, texts are categorized into binary classes: aggressive or non-aggressive. in level-2, 8289 aggressive texts are further annotated with multilabel targets. these labels are used to identify aggression's target into five fine-grained classes, such as religion, gendered, race, verbal and politics (detailed taxonomy discussed in section 3). proper annotation guidelines and the detailed statistics of the dataset is described to ensure m-bad's quality. several experiments are performed using ml, dl and transformer models to assess the task. the experiments demonstrate that (i) transformer models are more effective in detecting aggressive texts and their targets than ml/dl counterparts, (ii) covert propagation of aggression using ambiguous, context-dependent and sarcastic words is difficult to identify. the significant contributions of this work can be summarized as follows, reproducibility: the resources to reproduce the results are available at https://github.com/omarsharif03/m-bad . the appendix contains details about data sources, annotators and a few samples of m-bad. ## related work this section briefly describes the past studies related to aggression and other undesired content detection concerning non-bengali and bengali languages. non-bengali aggressive text classification: kumar et al. (2018a) compiled a dataset of 15000 aggression annotated comments in english and hindi with three classes: overtly aggressive, covertly aggressive, non-aggressive. in their subsequent work @xcite , bengali aggressive comments were added in the corpus. early works with neural network techniques such as lstm @xcite , cnn @xcite , combination of shallow and deep network @xcite achieved good accuracy. however, with the arrival of bert based models, it acquired superior performance and outperformed all the models on these datasets @xcite @xcite . @xcite developed a multilabel dataset in hindi with five hostile classes: fake, defamation, offensive, hate, non-hostile. their baseline system was implemented with m-bert embedding and svm. leite et al. (2020) introduced a multilabel toxic language dataset. the dataset contains 21k tweets manually annotated into seven categories: insult, lgbtq+phobia, obscene, misogyny, racism, nontoxic and xenophobia. they also performed baseline evaluation with the variation of bert models. in a similar work, @xcite developed a corpus to detect toxic speech in korean online news comments. bengali aggressive text classification: no significant research has been conducted yet to detect multilabel aggression in bengali. the scarcity of benchmark corpora is the primary reason behind this. few works have been conducted to develop datasets and models in other correlated domains such as hate, abuse, fake and offence. @xcite developed a hate speech dataset of 3000 samples with four categories: political, personal, religious, geopolitical. emon et al. (2019) presented a dataset comprised of 4.7k abusive bengali texts collected from online platforms. they proposed lstm based classifier to categorize texts into seven classes. however, they did not investigate other dl models' performance, which might get similar accuracy with less computational cost. to detect the threat and abusive language, a dataset of 5.6k bengali comments is created by @xcite . in recent work, sharif and hoque (2021a) introduced a benchmark bengali aggressive text dataset. they employed a hierarchical annotation schema to divide the dataset into two coarse-grained (aggressive, nonaggressive) and four fine-grained (political, religious, verbal, gendered) aggression classes. in their later work @xcite , they extended the dataset from 7.5k texts to 14k texts. differences with existing studies: as far as we are concerned, very few works have been accomplished to detect aggressive texts and identify the target of aggression (e.g. religion, gender, race). existing works @xcite @xcite have framed it as a multi-class classification problem and ignored the overlapping phenomena of classes. however, a text can express aggression towards multiple targets simultaneously. suppose a text has an aggressive write up against political women, expressing political and gendered aggressions. the proposed work addresses the issues that are previously overlooked and differs from the existing research in the following ways, (i) develop a novel bengali aggressive text dataset annotated with the multiple targets of an aggressive text. as our knowledge goes, this is the first attempt to develop such a dataset in bengali, (ii) illustrate a detailed annotation guideline which can be followed to develop resources for the similar domains in bengali and other low-resource languages, (iii) perform experimentation with multilabel classes with various ml, dl and transformer-based models. gression dataset in bengali (m-bad). level-1 has two coarse-grained categories: aggressive and nonaggressive. in contrast, level-2 has five fine-grained multilabel target classes (religion, politics, verbal, gender, race). this work differs from previous work done by @xcite in two ways; (i) overlapping phenomena between aggression targets are considered, (ii) a new target class (i.e., racial aggression) is added into the m-bad. figure 1 illustrates the taxonomic structure of m-bad. because of the subjective nature of the dataset, it is crucial to have a clear understanding of the categories. it helps develop a quality dataset by mitigating annotation biases and reducing ambiguities. after analyzing past studies @xcite @xcite on textual aggression and other related phenomena, we differentiate between the coarse-grained and fine-grained categories. coarse-grained aggression classes : the system initially identifies an input text as aggressive (ag) or non-aggressive (noag) classes. fine-grained target classes: an ag text is further classified into five fine-grained categories: religious aggression (reag), political aggression (poag), verbal aggression (veag), gendered aggression (geag) and racial aggression (raag). each of the classes is defined in the following: ## m-bad: multilabel aggression dataset as far as we are concerned, no dataset is available to date for detecting or classifying multilabel aggressive texts and their targets in bengali. however, the availability of a benchmark dataset is the prerequisite to developing any deep learning-based intelligent text classification system. this drawback motivates us to construct m-bad: a novel multilabel bengali aggressive text dataset. this work follows the guidelines and directions given by @xcite to ensure the quality of the dataset. this section briefly describes the data collection and annotation steps with detailed statistics of m-bad. ## methodology several computational models are investigated to develop the target aware aggression identification system. at first, the investigation is carried out for classifying the aggressive texts, and then we develop models for categorizing the target of the aggression @xcite considering the multilabel scenario. machine learning and deep learning-based methods are employed to build the system. this section briefly discussed the techniques and methods used to develop the system. ## experiments the experiments were carried out in a google collaboratory platform with a gpu environment. the evaluation of the dataset is performed based on the weighted f 1 -score. due to the highly skewed distribution of the classes, we considered macro f 1 -score (mf1) as our primary metric in multilabel evaluation. besides, the individual class performance is measured through precision (p), recall (r), and f 1 -score (f1) matrices.