Towards Offensive Language Identification for D ravidian Languages (2021)

Abstract: Offensive speech identification in countries like India poses several challenges due to the usage of code-mixed and romanized variants of multiple languages by the users in their posts on social media. The challenge of offensive language identification on social media for Dravidian languages is harder, considering the low resources available for the same. In this paper, we explored the zero-shot learning and few-shot learning paradigms based on multilingual language models for offensive speech detection in code-mixed and romanized variants of three Dravidian languages - Malayalam, Tamil, and Kannada. We propose a novel and flexible approach of selective translation and transliteration to reap better results from fine-tuning and ensembling multilingual transformer networks like XLMRoBERTa and mBERT. We implemented pretrained, fine-tuned, and ensembled versions of XLM-RoBERTa for offensive speech classification. Further, we experimented with interlanguage, inter-task, and multi-task transfer learning techniques to leverage the rich resources available for offensive speech identification in the English language and to enrich the models with knowledge transfer from related tasks. The proposed models yielded good results and are promising for effective offensive speech identification in low resource settings.

## introduction offensive speech is defined as speech that causes a person to feel upset, resentful, annoyed, or insulted. in recent years, social media such as twitter, facebook and reddit have been increasingly used for the propagation of offensive speech and the organization of hate and offense-based activities @xcite ). in a country like india with multiple native languages, users prefer to use their regional language in their social media interactions @xcite . it has also been identified that users tend to use roman characters for texting instead of the native script. this poses a severe challenge for the identification of offensive speech, considering the under-developed methodologies for handling code-mixed and romanized text @xcite . until a few years ago, hate and offensive speech were identified manually which is now an impossible task due to the enormous amounts of data being generated daily on social media platforms. the need for scalable, automated methods of hate speech detection has attracted significant research from the domains of natural language processing and machine learning. a variety of techniques and tools like bag of words models, n-grams, dictionary-based approaches, word sense disambiguation techniques are developed and experimented with by researchers. recent developments in multilingual text classification are led by transformer architectures like mbert @xcite and xlm-roberta @xcite ). an additional advantage of these architectures, particularly xlm-roberta, is that it yields good results even with lower resource languages and this particular aspect is beneficial to indian languages which do not have properly established datasets. in our work, we focused on using these architectures in multiple ways. however, there is a caveat in directly using the models on the romanized or codemixed text: the transformer models are trained on languages in their native script, not in the romanized script in which users prefer to write online. we solve this problem by using a novel way to convert the romanized sentences into their native language while preserving their semantic meaning -selective translation and transliteration. this work is an extension of @xcite . our important contributions are as follows 1) proposed selective translation and transliteration for text conversion in romanized and code-mixed settings which can be extended to other romanized and code-mixed contexts in any language. 2) experimented and analyzed the effectiveness of finetuning and ensembling of xlm-roberta models for offensive speech identification in code-mixed and romanized scripts. 3) investigated the efficacy of inter-language, inter-task, and multi-task transfer learning techniques and demonstrated the results with t-sne (maaten and hinton, 2008) visualizations. the rest of the paper is organized as follows: in section 2, we discuss related work followed by datasets' description in section 3. section 4 describes methodology and section 5 discusses about results. finally, we conclude the paper in section 6. ## related works two major shared tasks organized in offensive language identification are offenseval 2019 @xcite and germeval @xcite . other tasks related to offensive speech identification include hasoc-19 @xcite which dealt with hate speech and offensive content identification in indo-european languages, trac-2020 @xcite , which dealt with aggression identification in bangla, hindi and english. while hasoc-19 and trac 2020 dealt with offensive speech identification in indian languages of bangla and hindi, hasoc-dravidian-codemix -fire 2020 @xcite is the first shared task to conduct offensive speech identification task in dravidian languages. researchers used a wide variety of techniques for the identification of offensive language. recently, @xcite )(work published after the first version of this paper was submitted) used the xlm-roberta model for offensive language identification in bengali, hindi, and spanish. they show that the xlm-r model beats all other previous approaches for offensive language detection. @xcite , used the lgbm classifier on top of the combination of multilingual bert and laser pre-trained embeddings in hasoc-19. @xcite fine-tuned monolingual and multilingual bert based network models to achieve good results in identifying hate speech. @xcite used an ensemble of bert models with different random seeds for aggression detection on social media text, which is the inspiration behind our ensembling strategy with xlm-roberta models. there has been less research on text classification in dravidian languages and there is no research in offensive speech identification for dravidian languages so far. @xcite uses a simple lstm model for sentiment analysis in the malayalam language. our work adresses this gap of less research in offensive speech identification methods for dravidian languages and the systems proposed can be extended to other indian and foreign languages as well. ## datasets the datasets used in this work were taken from three competitions -hasoc-dravidian-codemix -fire 2020 @xcite (henceforth hdcm), sentiment analysis for dravidian languages in code-mixed text (chakravarthi et al., 2020a,d)(henceforth sadl) and offensive language identification in dravidian languages @xcite @xcite (henceforth odl). as a part of hdcm, there were two binary classification tasks, with the second task having two subtasks. the objective of all of the tasks was the same: given a youtube comment, classify it as offensive or not offensive. but the format and language of data provided to different tasks were different: code-mixed malayalam for task-1(henceforth referred to as mal-cm), tanglish for task-2a(henceforth referred to as tanglish), and manglish for task-2b(henceforth referred to as manglish). from odl, we used the kanglish dataset provided. originally, the task proposed in odl is a multi-class classification problem with multiple offensive categories. however, to maintain uniformity among the datasets, we divided the classes into two -offensive and not offensive. offensive-targeted-insult-individual, offensive-targeted-insult-group, offensive-untargeted, and offensive-targeted-insult-other classes in the dataset are renamed as offensive, and all the non-kannada posts are removed from the dataset. as far as sadl is concerned, we used the given sentiment analysis datasets as helper datasets in inter-task transfer learning technique(4.5). olid @xcite ## results and discussion to test the performance of our proposed models, we had evaluated them on validation data before the test set was made available by the concerned task organizers. when unlabeled test data was released, we used the above-mentioned validation data as dev set to develop the model, and final predictions are made using these models. ensemble of xlm-roberta models is not experimented on dev data due to computational limitations. nevertheless, we verified the better performance of ensemble models with few random seeds and directly used them for final training. ## conclusion in this paper, we have presented various techniques and neural network models to identify offensive language in social media text for code mixed and romanized dravidian languages. a novel techfigure 8: t-sne visualization for kanglish on test data nique of selective translation and transliteration is proposed to deal with code-mixed and romanized offensive speech classification in dravidian languages. this technique is flexible and can be extended to other languages as well. for classification, classical classifiers on top of pre-trained embeddings, fine-tuned xlm-roberta models, and an ensemble of xlm-roberta models are used. we experimented with different transfer learning techniques to leverage the offensive speech datasets from resource-rich languages. our work also points to the usefulness of transformer architectures, particularly xlm-roberta, for low resource languages like tamil and malayalam. our proposed models show an average performance of 85% f1weighted score across all datasets.