IIITT @ D ravidian L ang T ech- EACL 2021: Transfer Learning for Offensive Language Detection in D ravidian Languages (2021)

Abstract: This paper demonstrates our work for the shared task on Offensive Language Identification in Dravidian Languages-EACL 2021. Offensive language detection in the various social media platforms was identified previously. But with the increase in diversity of users, there is a need to identify the offensive language in multilingual posts that are largely code-mixed or written in a non-native script. We approach this challenge with various transfer learning-based models to classify a given post or comment in Dravidian languages (Malayalam, Tamil, and Kannada) into 6 categories. The source codes for our systems are published.

## introduction over the past decade, there has been a tremendous increase in the user-generated content on social media platforms such as twitter, youtube, and instagram @xcite . they provide a common space for discussion and interactions, for users to connect with each other, express their opinions, and share their knowledge. users may use offensive posts/comments which may be directed towards an individual or community @xcite which is one of the common problems in the online social media platforms (nogueira dos @xcite . they act as catalysts for leaving offensive content which could have a harmful and detrimental effect on users' mental health. the automatic detection of such malevolent comments/posts has become a crucial field of research in natural language processing in recent years @xcite . tamil (iso 639-1: ta), malayalam (iso 639-1: ml), and kannada (iso 639-3:kan) belong to the dravidian languages, spoken mainly in india @xcite . the earliest inscription in india dated to 580 bce was the tamil inscription in pottery. a tamil prayer book in ancient tamil script called thambiran vanakkam, was written by portuguese christian missionaries in 1578, thereby rendering tamil the first indian language to be printed and published. one of the first dictionaries written in the indian language was the tamil lexicon, published by the university of madras. tamil, malayalam, and kannada has its own script however users in the social media use the latin script generating code-mixing @xcite . code-mixing refers to the coupling of two or more languages in a single sentence @xcite . it is a quite common phenomenon observed in multilingual societies throughout the world (chakravarthi, 2020; @xcite . it is widely considered as a default mode of communication in countries like india and mexico @xcite @xcite . code-mixed sentence maintains the fundamental grammar and script of the languages it is comprised of @xcite . this paper is a description of our submission to the shared task for offensive language detection @xcite . the task is to identify offensive content in the code-mixed comments/posts in the dravidian languages collected from social media and classify it into not offensive, offensive untargeted, offensive targeted insult individual, offensive targeted insult group, offensive targeted insult other and not in-indentedlanguage. the rest of the paper is organized as follows, section 2 represents previous work on offensive language detection in dravidian languages. section 3 entails a detailed analysis of the datasets for tamil, malayalam, and kannada. section 4 presents a description of the models used for our purpose, while section 5 explains the experiment setup for the models. section 6 analyzes our re-sults achieved, and section 7 presents the future direction for our work. ## related work the extensive use of offensive content on social media platforms is disastrous to an advancing society as it serves to promote violence, chaos, abuse, and verbal hostility and extremely affects individuals at distinct levels. research in offensive language detection has been evolving rapidly over the past few years. @xcite gives an outline of the current state-of-the-art in offensive language detection and related tasks like hate speech detection. @xcite introduced a publicly available dataset, notably for offensive language detection, by classifying tweets into hate speech, offensive but not hate speech, and neither. several attributes like tf-idf, n-grams, readability scores, and sentiment were used to build machine learning models such as logistic regression and support vector machine in their work. a system combination of svm and deep neural networks were developed by @xcite for detecting abusive language which achieved f1-score of 90.51% on the test set. various experiments have been performed on code-mixed data. @xcite developed numerous systems for detecting offensive language in hindi and english which used data from twitter and facebook. hindi-english offensive tweet (heot) dataset comprising of tweets in hindi-english code mixed language classified into three classes; non-offensive, abusive, and hate-speech was introduced by @xcite . their work utilized transfer learning wherein the model used convolutional neural networks which was pretrained on tweets in english followed by retraining on hinglish tweets. @xcite examined the problem of hate speech detection in codemixed texts and presented a dataset of code-mixed hindi-english comprising of tweets posted on twitter. @xcite presented a system, c-bigru, comprised of a convolutional neural network(cnn) along with a bidirectional recurrent neural network(rnn) to identify offensive speech on social media. an embedding model-based classifier to identify offensive language from manglish dataset was developed in @xcite . multimodal systems of tamil troll memes were developed to classify memes that were deemed offensive towards other people @xcite . ## dataset the organizers provided us with tamil-english @xcite , malayalam-english @xcite and kannada-english @xcite code-mixed text data derived from social media. the datasets comprised of all six types of code-mixed sentences : nocode-mixing, inter-sentential code-mixing, only tamil/kannada/malayalam (written in latin script), code-switching at morphological level (written in both latin and tamil/kannada/malayalam script), intra-sentential mix of english and tamil/kannada/malayalam (written in latin script only) and inter-sentential and intra-sentential mix @xcite . the training dataset consists of comments in six different classes: ## system description we use pre-trained transformer models for classifying offensive speech in tamil, kannada, and malayalam. we do not perform text preprocessing techniques such as lemmatization, stemming, removing sinp words, etc, to preserve context to the users' intent. since we use transformer models, it is observed that stop words receive a similar amount of attention as non-stop words, as transformer models are contextual models ( bert, xlm-roberta, etc). ## experiment setup we describe the experiment setup for our experiments performed. all of our systems were trained on google colab @xcite . all of our models' parameters are as stated in ## results and analysis we have experimented with various classifiers like multilingual bert, xlm-roberta, distilbert, ulmfit, cnn. the evaluation metric of this task is weighted average f1-score . this is done to account for the class imbalance in the dataset. the results of the experiments performed using different models on the test datasets of malayalam, tamil and kannada are shown in we have trained bert-bilstm, xlm-roberta, cnn-bilstm and ulmfit models on the training datasets of malayalam, tamil and kannada. among the mentioned models, cnn-bilstm gave a good f1-score of 0.8444 on malayalam development set. for tamil and kannada, this model showed rather poor performance with f1-scores of 0.6128 and 0.4827, respectively. ulmfit and xlm-roberta models gave almost similar f1-scores of 0.7034 and 0.7083 respectively on tamil. we submitted bert-bilstm model as it has obtained an f1-score of 0.7285 on tamil development set. ulmfit gave f1-scores of 0.9048 and 0.7077 on malayalam and kannada development set. for malayalam and kannada, xlm-roberta model was submitted with f1-scores of 0.9113 and 0.7156 as the model has marginally outclassed ulmfit and bert-bilstm models. models like multilingual bert, albert, and xlm-roberta gave similar and poor results on the three test datasets. one of the reasons for the poor performance of these models is the imbalance in the distribution of the classes. in the dataset, the majority of the texts belong to not-offensive while the other classes like not-in-indented language, offensive-targeted-insult-group, offensivetargeted-insult-other, offensive-untargeted have a small classification of texts. these models performed better on the majority class and poorly on the minority classes. xlm-roberta gave better results on the validation set, but due to the class imbalances and the use of code-mixed and writing in non-native languages, it could have underperformed on the test set. it is observed that the cnn-bilstm model also performed poorly. in the cnn-bilstm model, the convolution layer was not capturing the correlations and patterns within the input. moreover, the bilstm layer did not apprehend the dependencies within the attributes extracted by the cnn layer, which has led to the poor performance of the model. for the word embeddings, we used glove embedding which did not perform well on the cnn. multilingual bert-bilstm performed well on the test set, but did not perform well on the development set. fine-tuning the transformer model distilbert has resulted in a good performance. ulmfit model attained a better performance in predicting the minority classes as well. the major reasons for the better performance of ulmfit over other models are due to its superior fine-tuning methods and learning rate scheduler. ## conclusion in this paper, we have explored various transformer models for detecting offensive language in social media posts in malayalam, tamil and kannada. we observed a class imbalance problem in the provided datasets of the task, which has a consequential impact on system performance. different network architectures can show different results. our work manifests that fine-tuning transformer models result in better performance. the relatively high f1-scores of 0.9603, 0.7895 on malayalam, tamil were achieved by ulmfit and 0.7277 on kannada was achieved by distilmbert model. for future work, we intend to explore pseudo-labelling and class weighting for better performance of our models.