Hate-Alert@ D ravidian L ang T ech- EACL 2021: Ensembling strategies for Transformer-based Offensive language Detection (2021)

Abstract: Social media often acts as breeding grounds for different forms of offensive content. For low resource languages like Tamil, the situation is more complex due to the poor performance of multilingual or language-specific models and lack of proper benchmark datasets. Based on this shared task “Offensive Language Identification in Dravidian Languages” at EACL 2021; we present an exhaustive exploration of different transformer models, We also provide a genetic algorithm technique for ensembling different models. Our ensembled models trained separately for each language secured the first position in Tamil, the second position in Kannada, and the first position in Malayalam sub-tasks. The models and codes are provided.

## introduction social media platforms have become a prominent way of communication, be it for acquiring information or promotion of business foot_1 . while we cannot deny the positives, there are some ill consequences of social media as well @xcite . bad actors often use different social media platforms by posting tweets/comments that insult others by targeting their culture and beliefs. in social media, such posts are collectively known as offensive language @xcite . to reduce offensive content, different social media platforms like youtube have laid down moderation policies and employ moderators for maintaining civility in their platforms. recently, the moderators are finding it difficult to continue the moderation due to the ever-increasing volume of offensive data. hence, platforms are looking toward automatic moderation systems. for instance, facebook is proactively removing a large part of the harmful content from its platform, even before the users report them. there are concerns by different policy-makers that these automatic moderation systems may be erroneous foot_2 . situation for countries like india is more complex, as courts often face dilemma while interpreting harmful content and social platforms like facebook are often unable to take necessary actions foot_3 . hence, more effort is required to detect and mitigate offensive language in the indian social media. recently, different shared tasks like hasoc 2019 foot_4 have been launched to understand hateful and offensive language in indian context but it is limited to hindi and english mostly. a subtask in hasoc 2020 6 (chakravarthi et al., 2020c; mandl et al., 2020) aimed to detect offensive posts in a code-mixed dataset @xcite . extending that task further, the organisers of this shared task have put together a large dataset of 43919, 7772, 20010 posts in three dravidian languages -tamil, kannada, malayalam respectively, to further advance research on offensive posts in these languages @xcite @xcite . in this paper, we aim to build algorithmic systems that can detect offensive posts. contributions of our paper are two-fold. first, we investigate how the current state-of-the-art multilingual language models perform on these languages. second, we demonstrate how we can use ensembling techniques to improve our classification performance. ## related work offensive language has been studied in the research community for a long time, one of the earliest studies @xcite tried to detect offensive users by using lexical syntactic features generated from their posts. although, they provided an efficient framework for future research, their dataset was small for any conclusive evidence. davidson et al. curated one of the largest dataset containing both offensive and hate speech. the authors found that one of the issues with their best performing models was that they could not distinguish between hate and offensive posts. in order to mitigate this, subsequent research @xcite tried to use deep learning to identify offensive language in english and found that recurrent neural networks (rnns) are quite effective this task. recently, the research community has begun to focus on offensive language detection in other low resourced languages like danish (sigurbergsson and derczynski, 2019), greek @xcite and turkish (c ¸öltekin, 2020). in the indian context, the hasoc 2019 shared task @xcite was a significant effort in that direction, where the authors developed a dataset of hate and offensive posts in hindi and english. the best model in this competition used an ensemble of multilingual transformers, fine-tuned on the given dataset @xcite . in dravidian part of hasoc 2020, renjit and idicula used an ensemble of deep learning and simple neural networks to identify offensive posts in manglish (malayalam in roman font). transformer based language models are becoming quite popular in the past few years. recently, different multilingual models like xlm-roberta @xcite , multilingual-bert @xcite , muril foot_6 and indic-bert @xcite have been introduced to facilitate nlp ## dataset description the shared task on offensive language identification in dravidian languages-eacl 2021 @xcite ) is based on a post classification problem with an aim to moderate and minimise offensive content in social media. the objective of the shared task is to develop methodology and language models for code-mixed data in low-resource languages, as models trained on monolingual data fail to comprehend the semantic complexity of a code-mixed dataset. dataset: the dravidian offensive codemixed language dataset is available for tamil @xcite , kannada @xcite and @xcite . the data provided is scraped entirely from the youtube comments of a multilingual community where code-mixing is a prevalent phenomenon. the dataset contains rows of text and the corresponding labels from the list not-offensive, offensive-untargeted, offensive-targeted-individual, offensive-targeted-group, offensive-targeted-other, or not-in-indented-language. final evaluation score was calculated using weighted f1-score metric on a held-out test dataset. we present the dataset statistics in ## methodology in this section, we discuss the different parts of the pipeline that we followed to detect offensive posts in this dataset. ## results and conclusion we observe that among the individual transformer models, the best performance is obtained using xlm-roberta-large (xlmr-large) in the tamil dataset and custom xlm-roberta-base (xlmr-c) in the kannada dataset. for malayalam dataset, both the former models perform similarly. the higher performance of xlm-roberta (artetxe and schwenk, 2019) models can be attributed to the fact that they are pretrained using a parallel corpus (same corpus in different languages). further pretraining with our dataset helps in further improvement of the performance in the kannada dataset. we did not use the xlm-r large model further due to limited gpu space. next, we note the performance of the fusion models, which perform almost similarly across different combinations. when we use different random seeds, the performance of multilingual bert models varied around 2-3% across different languages. for xlm-roberta models the variation was more (around 15-20%). in this shared task, we evaluated different transformer based architectures and introduced different ensembling strategies. we found that xlm-roberta models usually perform better than other transformer models, although their performance is highly variable across different random seeds. ga based ensembling helps us in further improving the models. our immediate next step will be to investigate the reason behind lower performance of indicbert and muril which are specifically trained for indian context.