Hate Speech and Offensive Language Detection in B engali (2022)

Abstract: Social media often serves as a breeding ground for various hateful and offensive content. Identifying such content on social media is crucial due to its impact on the race, gender, or religion in an unprejudiced society. However, while there is extensive research in hate speech detection in English, there is a gap in hateful content detection in low-resource languages like Bengali. Besides, a current trend on social media is the use of Romanized Bengali for regular interactions. To overcome the existing research’s limitations, in this study, we develop an annotated dataset of 10K Bengali posts consisting of 5K actual and 5K Romanized Bengali tweets. We implement several baseline models for the classification of such hateful posts. We further explore the interlingual transfer mechanism to boost classification performance. Finally, we perform an in-depth error analysis by looking into the misclassified posts by the models. While training actual and Romanized datasets separately, we observe that XLM-Roberta performs the best. Further, we witness that on joint training and few-shot training, MuRIL outperforms other models by interpreting the semantic expressions better. We make our code and dataset public for others.

## introduction social media websites like twitter and facebook have brought billions of people together and given them the opportunity to share their thoughts and opinions rapidly. on the one hand, it has facilitated communication and the growth of social networks; on the other, it has been exploited to propagate misinformation, violence, and hate speech @xcite against users based on their gender, race, religion, or other characteristics. if such content is left unaddressed, it may result in widespread conflict and violence, raising concerns about the safety of human rights, the rule of law, and freedom of speech, all of which are crucial for the growth of an unprejudiced democratic society @xcite . organizations such as facebook have been blamed for being a forum for instigating anti-muslim violence in sri lanka that resulted in the deaths of three individuals foot_1 , and a un report accused them of disseminating hate speech in a way that contributed significantly to the plausible genocide of the rohingya population in myanmar 3 . in order to reduce the dissemination of such harmful content, these platforms have developed certain guidelines foot_3 that the users of these platforms ought to comply with. if these rules aren't followed, the post can get deleted, or the user's account might get suspended. even to diminish the harmful content from their forum, these platforms engage moderators @xcite to manually review the posts and preserve the platform as wholesome and people-friendly. however, this moderation strategy is confined by the moderators' speed, jargon, capability to understand the development of slang, and familiarity with multilingual content. moreover, due to the sheer magnitude of data streaming, it is also an ambitious endeavor to examine each post manually and filter out such harmful content. hence, an automated technique for detecting hate speech and offensive language is extremely necessary and inevitable. it has already been witnessed that facebook vigorously eliminated a considerable amount of malicious content from its platforms even before users reported it @xcite . however, the hindrance is that these platforms can detect harmful content in certain popular languages such as en-glish, spanish, etc. @xcite so far, several investigations have been conducted to identify hate speech automatically, focusing mainly on the english language; therefore, an effort is required to determine and diminish such hateful content in lowresource languages. with more than 210 million speakers, bengali is the seventh most widely spoken language 5 , with around 100 million bengali speakers in bangladesh and 85 million in india. apart from bangladesh and india, bengali is spoken in many countries, including the united kingdom, the united states, and the middle east foot_5 . also, a current trend on social media platforms is that apart from actual bengali, people tend to write bengali using latin scripts(english characters) and often use english phrases in the same conversation. this unique and informal communication dialect is called code-mixed bengali or roman bengali. code-mixing makes it easier for speakers to communicate with one another by providing a more comprehensive range of idioms and phrases. however, as emphasized by chittaranjan et al. @xcite , this has made the task of creating nlp tools more challenging. along with these challenges, the challenges specific to identifying hate speech in roman bengali contain the following: absence of a hate speech dataset, lack of benchmark models. thus, there is a need to develop open efficient datasets and models to detect hate speech in bengali. although few studies have been conducted in developing bengali hate speech datasets, most of these have been crawled with comments from facebook pages, and all of them are in actual bengali. hence, there is a need for developing more benchmarking datasets considering other popular platforms. to address these limitations, in this study, we make the following contributions. ## related work over the past few years, research around automated hate speech detection has been evolved tremendously. the earlier effort in developing resources for the hate speech detection was mainly focused around english language @xcite @xcite . recently, in an effort to create multilingual hate speech datasets, several shared task competitions have been organized (hasoc @xcite , offenseval @xcite )" trac @xcite , etc.), and multiple datasets such as hindi @xcite , danish (sigurbergsson and derczynski, 2020), greek @xcite , turkish @xcite , mexican spanish (aragón et al., 2019), etc. have been made public. there is also some work to detect hate speech in actual bengali. ismam et al. (ishmam and sharmin, 2019) collected and annotated 5k comments from facebook into six classes-inciteful, hate speech, religious hatred, communal attack, religious comments, and political comments. however,the dataset is not publicly available. karim et al. (karim et al., 2021) provided a dataset of 8k hateful posts collected from multiple sources such as facebook, news articles, blogs, etc. one of the problems with this dataset is that all comments are part of any hate class(personal, geopolitical, religious, and political), so we cannot build hate speech detection models using this dataset to screen out hate speech. romim et al. (2021) curated a dataset of 30k comments, making it one of the most extensive datasets for hateful statements. the author achieved 87.5% accuracy on their test dataset using the svm model. however, these datasets do not consider roman bengali posts, a prevalent communication method on social media nowadays. with regards to the detection systems, earlier methods examined simple linguistic features such as character and word n-grams, pos tags, tf-idf with a traditional classifier such as lr, svm, decision tree, etc @xcite . with the development of larger datasets, researchers have shifted to data-hungry complex models such as deep learning @xcite and graph embedding techniques to enrich actual roman total hateful 825 510 1,335 offensive 1,341 2,063 3,404 normal 2,905 2,534 5,439 total 5,071 5,107 10,178 the classifier performance. recently, transformer-based @xcite language models such as bert, xlm-roberta @xcite are becoming quite popular in several downstream tasks. it has already been observed that these transformer-based models outperform several earlier deep learning models @xcite . having observed these transformer-based models' superior performance, we focus on building these models for our classification task. further, researchers have begun to explore few shot classifications. one of the most popular techniques for few-shot classification is transfer learning -where a model (pre-trained in a similar domain) is further fine-tuned on a few labeled samples in the target domain @xcite . keeping these experiments in mind, we also examine the ability of transfer learning capabilities between actual and roman bengali data. ## dataset creation in this section, we provide the data collection procedure, annotation strategies we have followed and the statistics of the collected dataset. ## results in this section, we discuss the findings of our experiments. ## additional experiment in addition, we perform another experiment to further compare the quality of our dataset with the existing dataset of romim et al. @xcite . using their dataset, we train the xlm-roberta model 14 and test its performance on our dataset. likewise, we test the performance on their dataset when the model is trained on our dataset. we only conduct this experiment with the actual bengali tweets to have valid comparison with their dataset. we combine hate and offensive into a single class for this experiment, as the authors in @xcite have considered these two labels as same. in ## error analysis in order to deep dive into the models further, we conduct a manual error analysis on our models by using a sample of 50 posts where the model incorrectly categorizes some test instances. we analyze common errors and classify them into the following five categories. in ## discussion in this section we discuss the key insights from our results. we observe that depending on the availability of training data points, the performance of the model varies. when we have sufficient number of training instances xlm-roberta model performs the best. further we argue that when actual & roman bengali instances are merged together for joint training, models like muril performs the best by leveraging the semantic connection between actual and romanized instances. this is, to some extent, expected from muril due to the nature of its pretraining mechanism, where both actual language and its transliterated counterpart have been used. further exploring the performance of these models in zero-shot setting shows, although xlm-robera performs best while trained with standalone data, it performs very poorly for unseen data with similar semantic content but a different orthography. in such scenarios, models like m-bert, in-dicbert exhibit better performance. to improve the performance of these models, when some instances from the target language are used, muril shows an increase in performance at a rate higher than the other models. observing these results it may be safe to say that when there is a data scarcity for a particular language, it is better to reuse existing fine-tuned models in the same domain. also careful selection of model is needed. in our case, actual bengali and roman bengali use different characters for writing, but their semantic expressions are same, which is why muril performed best overall. while doing the in-depth error analysis, we also found that for some cases it can be even difficult for a model to find the actual label correctly. not only models, as hate speech is complex in nature, sometimes annotators make mistake while labelling them due to differing viewpoints. limitation: there are a few limitations of our work. first is the lack of external context. we have not considered any external context such as profile bio, history of user's posting pattern, gender etc., which might be helpful for the hate speech detection task. although the effectiveness of these transformer-based models are quite good, they have not been tested against adversarial examples. ## conclusion this paper presents a new benchmark dataset for bengali hate speech detection, consisting of 10k posts from twitter, covering both actual & roman scenarios. each tweet was annotated with one of the hate/offensive/normal labels. we assessed different transformer-based architectures for hate speech detection. we also experimented with several interlingual transfer mechanisms. our experiments show how few-shot techniques could be beneficial. besides, we saw how joint training performs better than training on standalone data. we further notice that joint transliterated training performs best in the case of the roman bengali dataset. our error analysis reveals some of the typical shortcomings of the transformer models. as part of the future work, we plan to evaluate the robustness of these models' under adversarial attack as hateful users keep contriving newer ways to deceive the standard hate speech detection models. another direction could be lessening the biases that can be present in the dataset/model.