Hypers@ D ravidian L ang T ech- EACL 2021: Offensive language identification in D ravidian code-mixed Y ou T ube Comments and Posts (2021)

Abstract: Code-Mixed Offensive contents are used pervasively in social media posts in the last few years. Consequently, gained the significant attraction of the research community for identifying the different forms of such content (e.g., hate speech, and sentiments) and contributed to the creation of datasets. Most of the recent studies deal with high-resource languages (e.g., English) due to many publicly available datasets, and by the lack of dataset in low-resource anguages, those studies are slightly involved in these languages. Therefore, this study has the focus on offensive language identification on code-mixed low-resourced Dravidian languages such as Tamil, Kannada, and Malayalam using the bidirectional approach and fine-tuning strategies. According to the leaderboard, the proposed model got a 0.96 F1-score for Malayalam, 0.73 F1-score for Tamil, and 0.70 F1-score for Kannada in the bench-mark. Moreover, in the view of multilingual models, this modal ranked 3rd and achieved favorable results and confirmed the model as the best among all systems submitted to these shared tasks in these three languages.

## introduction the use of social media platforms (e.g., facebook, twitter, and youtube) has become an important activity in the everyday lives of most internet users. the users share their thoughts with multilingual societies in the form of comments and posts on social media. in the last few years, with a proliferation of social media users, posts and comments have become an information form that is simply called code-mixed text @xcite @xcite due to the lack of impressions in a single language when conveying their thoughts online with the multi-linguistic community @xcite . most of the social media platforms encourage users to convey and express their opinions in their preference with a small number of restrictions and intend to collect user comments and posts to provide a customized feed. in addition to this, these collected data are also used for advertising, marketing, and detecting the occurrence of bad activities. it is rather difficult to identify offensive code-mixed content written in a non-native (roman) script @xcite . this complexity in such execution of the natural language processing tasks cannot be solved by the technologies developed for a monolingual text @xcite but multilingual models can. offensive language identification of code-mixed data on social media platforms is important to identify the movie reviews, product reviews, enable advertisements, monitor political activities, and identify social trends. the code-mixed data always adopts the vocabulary and grammar of multiple languages and generates new user-based content structures @xcite . this is challenging for offensive language identification tasks as traditional and unidirectional approaches and lack of dataset result in not capturing the meaning and limits the accuracy of the task. according to the shared task, this study executed a comment or post level offensive language identification of the code-mixed contents which was collected from youtube comments and posts on dravidian language pairs (e.g. tamil-english, kannada-english, and malayalam-english). each of given sentence is annotated with labels as follows: not-offensive, offensive targeted insult other, offensive targeted insult individual, offensive targeted insult group, not tamil or not malayalam or not kannada, and offensive untargeted. this research work proposed a system that uses m-bert which stands for multilingual-bidirectional encoder representations with transformers @xcite , as a pre-trained model and applied fine-tuning strategies as a transfer learning approach to solve the given task. the rest of the sections in the paper are as follows. section 2, reviews related experiment works in offensive language identification in three views. section 3 describes the methodology with system architecture which includes the classification features and transfers learning strategies. the fourth section presents the conducted experiments using different approaches and features. section 5 analyses the results of the proposed system in the benchmark. benchmark results are discussed in section 6 and finally, the conclusion followed by future research directions. ## related work due to the code-mixed dataset limitation on the low-resourced languages such as tamil, sinhala, malayalam, and kannada, there were a few numbers experiments have been done in this area @xcite @xcite @xcite @xcite . @xcite developed a model to identify hate speech and offensive contents in dravidian languages (e.g. tamil-english and malayalam-english) from the code-mixed comments and posts collected from social media using pre-trained ulmfit on synthetically generated code-mixed data. their proposed method achieved 0.88 and 0.91 weight f1-score for codemixed tamil-english and malayalam-english, respectively. on the other hand, some tasks have been done based on dataset creations to provide support for low-resource language identification tasks. to encourage sentiment analysis tasks and overcome the non-availability of the code-mixed dataset in tamil-english chakravarthi et al. (2020) created a sentiment-annotated corpus containing 15744 youtube comments and posts. in addition to this, they examined the benchmark system with various machine learning algorithms such as logistic regression (lr), support vector machine (svm), k-nearest neighbour (k-nn), decision tree (dt), random forest (rf), multinominal naive bayes (mnb), 1dconv-lstm, bert multilingual, dme and cdme. in this task, the logistic regression, random forest classifiers, and decision trees were provided better results compared to other algorithms. @xcite build a standard corpus for malayalam-english to increase the sentiment analysis tasks in the codemixed contents. @xcite created a newly code-mixed dataset annotated in the sentence and word level collected from facebook comments, chat history, and from public posts for sinhala-english language pairs. hande et al. (2020) explored in kannada-english by the traditional learning approaches such as logistic regression (lr), support vector machine (svm), multi-nomial naive bayes (mnb), k-nearest neighbors (knn), decision trees (dt), and random forest (rf). in high-resource languages, priyadharshini et al. (2020) examined methods using transformer models to predict the named entities in corpus written both in the native script as well as in the roman script for the english-hindi language pairs. on the other hand, the explorations for english-spanish @xcite , chinese-english @xcite , and english-bengali @xcite language pairs were used multiple-classifier based automatic detection approach, unidirectional lstm, and common generic forms of deep learning and neural network approaches to perform their identification tasks. according to the social media explosion, offensive language identification has become an increasingly important task and a few studies have been published on single languages @xcite . @xcite performed an analysis to separate the general profanity from the hate speech in social media. they presented a supervised classification system that used character n-grams, word n-grams, and word skip grams and their model was able to achieve 80% accuracy on a huge dataset which contained english tweets annotated with three labels, namely, "hate speech (hate)", "offensive language but no hate speech (offensive)", and "no offensive content (ok)". pitenis et al. (2020) addressed the problem of offensive language identification for the greek language by created their annotated corpus using comments retrieved from twitter posts and experimented with seven different classification models: pooled gru @xcite , stacked lstm with attention @xcite , lstm and gru with attention @xcite , 2d convolution with pooling, gru with capsule, lstm with capsule and attention and bert @xcite . in this task, lstm and gru with attention performed well compared to all other models in-terms of macro-f1. however, the fine-tuning approach with the bert-base multilingual cased model did not provide good results. based on the above analysis, most of the approaches were unidirectional-based except the works done by @xcite ) and (pitenis et al., 2020). they used bert @xcite as their bidirectional algorithm but it did not reach the state-of-the-art on code-mixed contents at that time. after a lot of studies and observations, bert is chosen for this low-resource language experiment as a challenge. ## results and analysis teams are ranked by the weighted average f1 score of their classification system. according to the models which were submitted to all three languages, we have found ourselves as 3 rd in the order of models. this shows that the research model of this study achieved state-of-the-art results in all three languages. moreover, even though the research model of this study got the above rank, the f1-score difference between the first team and hypers is only 1%, 7%, and 6% respectively. therefore, it is assumed that the proposed model of this study provided better results compared to other models for monolingual languages and also multilingual languages. ## discussions even with the better f1-scores for all three models, there was some confusion according to the results. f1-score for malayalam is comparatively higher than the other two models. according to the dataset, the tamil training set has approximately 35139 posts and comments, in the meantime, kannada and malayalam are 6214, and 16011 respectively but given results behaves as opposed to the number of posts and comments in the training sets. it may have happened due to the lack of training of the bert multilingual based model in the tamil and kannada language. since bert multilingual model was trained on wikipedia and analyzed wiki contents foot_5 and got the total contents in kannada, malayalam, and tamil are 26,769, 71,762, and 134,011 respectively. the lower f1 score for kannada could be attributed to the less training data. on the contrary, despite having more training data, bert trained on tamil and malayalam performs poorly. in addition to the above wikipedia analysis, it was carried out a qualitative analysis to find patterns and observations on the given dataset. commonly, in the training dataset, more than 80% of the posts and comments were labeled as not-offensive and the remaining were labeled as other types of labels. the use of english words was minimal although there are many comments and posts which are in the native language but written in roman script. therefore, it is assumed that the lack of training in those informal forms of languages led to such results in the benchmark evaluation. ## conclusion offensive language identification in dravidian language code-mixed text is one of the most challenging tasks in natural language processing. we have done such a task in this study using bert with a fine-tuning strategy in the low-resourced languages. in this benchmark, our (hypers team) proposed system got 2 nd , 5 th , 6 th rankings for the malayalam, kannada, and tamil test set respectively. finally, it was proved that the bert system can be used on low-resource languages to the offensive language identification tasks with the best f1-score.