Offensive Language Identification in Transliterated and Code-Mixed B angla (2023)

Abstract: Identifying offensive content in social media is vital to create safe online communities. Several recent studies have addressed this problem by creating datasets for various languages. In this paper, we explore offensive language identification in texts with transliterations and code-mixing, linguistic phenomena common in multilingual societies, and a known challenge for NLP systems. We introduce TB-OLID, a transliterated Bangla offensive language dataset containing 5,000 manually annotated comments. We train and fine-tune machine learning models on TB-OLID, and we evaluate their results on this dataset. Our results show that English pre-trained transformer-based models, such as fBERT and HateBERT achieve the best performance on this dataset.

## introduction as the popularity of social media continues to grow, the spread of offensive content in these platforms has increased substantially, motivating companies to invest heavily in content moderation strategies and robust models to detect offensive content. we have observed a growing interest in this topic, evidenced by popular shared tasks at semeval @xcite and other venues. apart from a few notable exceptions @xcite , most of the work on this topic has not addressed the question of transliteration and code-mixing, two common phenomena in social media. code-mixing is the phenomenon of embedding linguistic units such as phrases, words, or morphemes of one language into another language @xcite . code-mixed texts often feature transliterations where speakers use an alternative script to the language's official or standard script by mapping from one writing system (e.g., hindi and its original devanagari script) to another one (e.g., latin transliteration of hindi) based on phonetic similarity. transliterated texts are widely used in social media platforms as transliteration allows users to write in their native language using a script that may not be supported by the platform and/or using latin-based default keyboards. furthermore, the use of transliteration also allows users to easily switch between languages with otherwise different scripts (e.g., english and hindi). as discussed in a recent survey @xcite , however, processing code-mixing datasets is a challenge that hinders performance in a variety of nlp tasks, thus deserving special attention. code-mixing and transliteration are common in various languages, including bangla @xcite . related work on bangla offensive language identification @xcite , however, has mostly focused on standard bangla script. as such, the performance of offensive language identification models on codemixing and transliterated bangla remains largely unexplored. to address this shortcoming, we create tb-olid, a manually annotated transliterated bangla offensive language dataset. tb-olid was annotated following the popular olid hierarchical taxonomy @xcite , allowing crosslingual experiments. to the best of our knowledge, the dataset is the first of its kind for bangla, opening exciting new avenues for future research. the main contributions of this paper are as follows: comment c|t o|n i|g|u bn: tui to 1 ta rastar chele mother chod. c o i en: you are a motherfucking street vagabond bn: o to manush na sokun t o i en: he/she is not a person but a vulture bn: r kichudin por kanglu ra uth ar mut diye cha banabe t o g en: after some days, the barbarians will make tea with camel piss bn: dhoren r dog ar baccha gulo ke gono dholai diye pongu kore den c o g en: capture these son of bitches and beat them to their death bn: pagole kina bole chagole kina khai t o u en: a mad man and an animal have no difference bn: ami kintu parlam na hojom korte t n en: i cannot fathom it anymore ## baselines and models baselines we report the results of three baselines: ## results and discussion we use f1-score to evaluate the performance of all models. the training and test sets are obtained by the aforementioned 80-20 random split on the entire tb-olid dataset. we further subdivide the test set into transliterated code-mixed (c), transliterated (t), and all instances. we present results for offensive text classification (offensive vs. nonoffensive) in overall, target classification is a more challenging task than offensive language identification due to the presence of three classes instead of two. therefore, all results are substantially lower for this task. hatebert performs better than all other mod-els with an f1 score of 0.68. roberta achieved more competitive performance for target classification than for offensive language identification whereas bangla-bert did not perform well in both tasks. finally, similar to the previous task, the bestperforming models achieved higher f1 scores on the code-mixed data than on the transliterated data. one key observation is that the transformerbased models do not perform very well, since most of them are not pre-trained on transliterated bangla. among the models that we experiment with, only xlm-roberta is pre-trained with a comparatively small set of romanized bangla. however, the lack of any standard rules for spelling in transliterated bangla makes tb-olid very challenging. ## conclusion and future work in this work, we introduced tb-olid, a transliterated bangla offensive language dataset containing 5,000 instances retrieved from facebook. three native speakers of bangla have annotated the dataset with respect to the presence of code-mixing, the presence of offensive language, and its target according to the olid taxonomy. tb-olid opens exciting new avenues for research on offensive language identification in bangla. we performed experiments with multiple models such as general monolingual models like bert @xcite , roberta @xcite and bangla-bert @xcite ; crosslingual models like mbert @xcite and xlm-roberta @xcite ; and models fine-tuned for offensive language identification like hatebert @xcite , and fbert @xcite ). the best results were obtained by the task-specific models. in future work, we would like to extend the tb-olid dataset and annotate the offense type (e.g., religious offense, political offense, etc.). this would help us identify the common targets in various platforms. furthermore, we would like to pre-train and fine-tune a bangla transliterated bert model to see how it performs on tb-olid. finally, in future work, we would like to evaluate the performance of other recently released large language models (llms) (e.g., gpt 4.0, llama 2) on tb-olid. the first baseline results using gpt 3.5 indicate that general-purpose llms still struggle with the transliterated and code-mixed content presented in tb-olid.