Bitions@ D ravidian L ang T ech- EACL 2021: Ensemble of Multilingual Language Models with Pseudo Labeling for offence Detection in D ravidian Languages (2021)

Abstract: With the advent of social media, we have seen a proliferation of data and public discourse. Unfortunately, this includes offensive content as well. The problem is exacerbated due to the sheer number of languages spoken on these platforms and the multiple other modalities used for sharing offensive content (images, gifs, videos and more). In this paper, we propose a multilingual ensemble-based model that can identify offensive content targeted against an individual (or group) in low resource Dravidian language. Our model is able to handle code-mixed data as well as instances where the script used is mixed (for instance, Tamil and Latin). Our solution ranked number one for the Malayalam dataset and ranked 4th and 5th for Tamil and Kannada, respectively.

## introduction online communication has helped break a lot of barriers in terms of time, distance and ease of communication. the number of active internet users has grown rapidly over the last few years. the ease of sharing content and the lack of automatic systems for monitoring them, has led to a great increase in the amount of offensive and hate speech in the open internet. hate speech is often targeted towards a group of people or individuals hurting their identity, beliefs or sentiments. owing to the ease of access and lack of monitoring, individuals tend to misuse this freedom to hurl abuses and cause disharmony in the community. it is therefore important to address this issue. social media is easily accessible by a larger domain of people and the scale of open internet * equal contribution. the style of data on open internet also plays a major role in the understanding of data. the language structure is often missing and people tend to make use of words from different languages, ultimately resulting in codeswitched data @xcite . the problem is exacerbated as people use words from different written scripts, mixing both latin script and native script (devanagri, dravidian, mandarin etc) from the language. a unified model which can understand a multitude of these scripts can play a major role in understanding the discourse in open internet data and conducive to creating a safer virtual environment. majority of the research work in nlp has been predominantly in english @xcite . and the multilingual models currently available are trained on a multitude of languages making it hard to fine-tune for downstream tasks like sentiment analysis, text classification etc. on low-resource languages. our work addresses this issue by employing pseudo labelling and ensemble based techniques. the importance of the issue and the challenges posed, calls for novel ideas for offensive language detection. owing to this many workshops @xcite and shared tasks @xcite @xcite , have been conducted to address the problem at hand. in this paper, we present our system for the task of offensive language identification in dravidian languages. we make use of multilingual bert based models with pseudo labelling and ensemble strategies to achieve 1st rank out out 30 participants on malayalam data. our models perform equally well on the kannada (5th rank) and tamil (4th rank) dataset as well. ## related works ever since social media platforms started gaining popularity, the problem of detecting offensive language has existed. many researchers have worked to develop different ways that automate the process to tackle the issue. authors in @xcite have discussed the intricacy hate speech concept and its conclusive potential for societal impact, specifically in online communities and digital media platforms. detection of profanity and hate speech in tweets and comments has been a part of many shared tasks @xcite @xcite . the semeval 2019 task 9 @xcite ), aimed at identification of offensive and non-offensive comments in english tweets. it used the olid dataset @xcite which has 14000 tweets annotated using a hierarchical annotation model. offen-seval 2020 @xcite ) was a profanity identification task presented in semeval 2020. it was conducted for 5 languages (multilingual) language, namely english, arabic, danish, greek, and turkish. many researchers have tried to solve hate speech, offense and aggression detection using deep learning techniques like cnns, lstms, etc. @xcite @xcite . some researchers have also tried using machine learning algorithms for the same @xcite . recently language models like bert @xcite have become very popular for this problem @xcite @xcite . there have been attempts at developing models for hate speech detection in english, hindi-german @xcite and italian @xcite have emerged, but not many works for dravidian code-mix languages. nevertheless, attempts are underway to accelerate advancements in nlp in dra-vidian languages @xcite , have emerged. methods like lstms @xcite , transformer @xcite etc. have been previously tried to detect offense in dravidian languages. ## data there are three datasets for the three languages that we consider: kannada @xcite @xcite . the six class labels in the kannada and tamil data are: all classes mentioned above except 'offensive other' are the classes in the malayalam data set. the distribution of the data is described in the majority class in all three languages is the 'not offensive' class. this accounts for 56.97% of the samples in kannada, 88.77% of the samples in malayalam and 72.25% of the samples in tamil data. another important fact to note is the skewness in the data. the dataset is extremely skewed toward the non-offensive class and in order to overcome this challenge we make use of class weighting by penalising more for the under-represented classes. this is discussed in detail in the next section. ## methods this section describes our solution for the offensive language detection task. it is divided into 4 sub-sections viz, models, classweighting, pseudo-labelling and ensemble. a look at the data provided for the task calls for a multi-lingual approach, as it has both latin script and text in the native language. ## experiment we describe the experiments that we perform. all the models are trained on google colab 2 . the code is available publicly 3 . ## results all the results for all experiments are reported in having different models which are pretrained on different types of data, we ensembled our model predictions from ulmfit and distilmbert which boosted our model f1score from 0.65 to 0.697 on kannada data. we observe that the distilmbert performs better than the ulmfit (with and without pseudo-labelling). its better performance can be attributed to the truly bidirectional nature of bert @xcite based models. secondly, bert based models use transformers at their heart and hence do not suffer from long dependency issues. the use of the class-weighting scheme and focal loss @xcite help to better represent the underrepresented classes. it can be seen that the focal loss approach performs better than the naive class-weighting. but empirical results show that using focal loss led to much lower precision and recall for the minority classes than class-weighting. for example, for the malayalam dataset, the precision and recall obtained using class-weighting were greater by 19% and 10% respectively than using focal loss. this is an interesting observation which we believe needs further experimentation for validation. overall our unified model which uses classweighting, pseudo labelling and ensemble methods was the best performing model on malayalam testset with an f1-score of 0.97. our model was also in the top 5 best performing models for both tamil and kannada with f1-scores of 0.75 (4th place) and 0.70 (5th place). ## conclusion and future work in this paper, we proposed an ensemble model utilising pseudo labelling to effectively detect offensive statements in dravidian languages, namely kannada, malayalam and tamil. we show competitive results on all three languages with first rank for malayalam and within top-5 for kannada and tamil. pre-trained multilingual model worked best for our use case as knowledge from similar language families was used across all the languages. in future research, we will consider synthetically creating new code-mixed data for each language and the usage of language specific tokenizers for the multi-lingual models.