indicnlp@kgp at D ravidian L ang T ech- EACL 2021: Offensive Language Identification in D ravidian Languages (2021)

Abstract: The paper aims to classify different offensive content types in 3 code-mixed Dravidian language datasets. The work leverages existing state of the art approaches in text classification by incorporating additional data and transfer learning on pre-trained models. Our final submission is an ensemble of an AWD-LSTM based model along with 2 different transformer model architectures based on BERT and RoBERTa. We achieved weighted-average F1 scores of 0.97, 0.77, and 0.72 in the Malayalam-English, Tamil-English, and Kannada-English datasets ranking 1st, 2nd, and 3rd on the respective shared-task leaderboards.

## introduction offensive language identification is a natural language processing (nlp) text classification task where the goal is to moderate and reduce objectionable social media content. there has been a rapid growth in offensive content on social media and users from different ethnicities and cultures worldwide. a significant portion of offensive content is specifically targeted at various individuals and minority & ethnic groups. consequently, the identification and classification of these different kinds of foul language are receiving increased importance. dravidian languages like kannada, malayalam, and tamil (chakravarthi, 2020) are low-resourced, making this task challenging. training embeddings of words has previously been a common approach employed in text classification tasks. however, transfer learning approaches in deep learning @xcite have been shown unsuccessful or requiring extensive collections of in-domain documents to produce strong results @xcite . code-mixing is a prevalent practice in a multilingual culture, and code-mixed texts are often written in native scripts. due to code-switching complexity at multiple linguistic levels, systems trained on monolingual data can fail on code-mixed data. while multilingual versions of transformer models have been shown to perform remarkably well, even in zero-shot settings @xcite , a zeroshot transfer may perform poorly or fail altogether @xcite . this is when the target language, here code-mixed dravidian data, is different from the source language, mainly monolingual in english. in our work, we tackle these problems by exploiting additional datasets for fine-tuning our models and using effective transfer learning techniques. our code and experiments are available on github foot_0 for reproducing our models. ## task description and datasets this task aims to classify offensive language material gathered from social media from a set of code-mixed posts in dravidian languages. the systems have to classify each post into one of the six labels: ## methods a variety of methods are experimented on our datasets to provide a complete baseline. in section 3.1, we describe our implementation of three traditional machine learning classifiers; multinomial naive bayes, linear support vector machines (svm), and random forests. these approaches work well on small datasets and are more computationally efficient than deep neural networks. their performance is similar to the later sections described in the absence of pretraining and additional data. in section 3.2, our recurrent neural network (rnn) models are explained. we have compared an lstm model, using word-level embeddings trained from scratch, to an ulmfit model, an effective transfer learning approach for language models. finally, in section 3.3, we discuss transformer architectures using their cross-lingual pretrained models, which can be data-intensive during fine-tuning but provide the strongest results. ## experiments and results this task's assessment metric is weighted-f1, which is the f1 score weighted by the number of samples in all the classes. the datasets are divided into train, validation, and test sets in an approximately 8:1:1 ratio. the test labels are hidden and only available to us after the evaluation phase is over. we strictly train our models on the train set using the validation set scores for hyperparameter tuning. we have reported the results of our various models on the validation set. the averageensemble of our top three performing models is submitted finally, and we also report its scores on the validation and test set using the scores we are ranked with on the task leader board. the results on the validation set of our transferlearned xlm-r model are the best across all 3 datasets and are followed closely by the transfer learned multilingual bert model and the ulmfit model. we finally submit an average ensemble of these three models, and our results on the validation ## conclusion this paper describes various approaches for offensive language identification in three code-mixed english-dravidian language datasets. we also discuss the final system submitted by the indicnlp@kgp team, which ranks first, second, and third on the competition's three tasks. the benefit of pre-trained language models was shown by the significant improvement in results using a robust transfer learning framework (ulmfit) compared to a vanilla lstm model trained from scratch. transformer networks' performance also improved when all the dravidian language datasets were combined. this suggests that learning from one dravidian language may help in zero-shot or few-shot transfer to other new dravidian languages. in future works, we wish to explore these effects in more detail.