Offensive language identification in D ravidian code mixed social media text (2021)

Abstract: Hate speech and offensive language recognition in social media platforms have been an active field of research over recent years. In non-native English spoken countries, social media texts are mostly in code mixed or script mixed/switched form. The current study presents extensive experiments using multiple machine learning, deep learning, and transfer learning models to detect offensive content on Twitter. The data set used for this study are in Tanglish (Tamil and English), Manglish (Malayalam and English) code-mixed, and Malayalam script-mixed. The experimental results showed that 1 to 6-gram character TF-IDF features are better for the said task. The best performing models were naive bayes, logistic regression, and vanilla neural network for the dataset Tamil code-mix, Malayalam code-mixed, and Malayalam script-mixed, respectively instead of more popular transfer learning models such as BERT and ULMFiT and hybrid deep models.

## introduction the hate speech is generally defined as any communication which humiliates or denigrates an individual or a group based on the characteristics such as colour, ethnicity, sexual orientation, nationality, race and religion. due to huge volume of usergenerated content on the web, particularly social networks such as twitter, facebook, and so on, the problem of detecting and probably restricting hate speech on these platforms has become a very critical issue @xcite . hate speech lasts forever on these social platforms compared to physical abuse and terribly affects the individual on the mental status creating depression, sleeplessness and even suicide @xcite . owing to the high frequency of posts, detecting hate speech on social media manually is almost impossible. some recent researches have indicated that the automation of hate speech detection is a more reliable solution. @xcite extracted n-gram tf-idf features from tweets using logistical regression to classify each tweet in hate, offensive and non-offensive classes. another model for the detection of the cyberbullying instances was presented by @xcite with a genetic algorithm to optimize the distinguishing features of multimodal posts. @xcite used the linguistic, semantic and sentimental feature to detect racial content. the lstm and cnn based model for recognising hate speech in the social media posts were explored by @xcite . @xcite exploited the semantic word embedding to classify each tweet into racist, sexist and neither class. another deep learning model for the detection of hate speech was proposed by @xcite . however, most of the works for hate speech detection were validated with english datasets only. in a country such as india, the majority of people in social media use at least two languages, primarily english and hindi (or say hinglish). these texts are considered bilingual. in a bilingual setting, the script of the entire post may be same with words coming from both of these languages termed as mixed-code (or code mix) text. a few popular code mixed posts in india are english and hindi (or say hinglish), tanglish (tamil and english) @xcite , manglish (malayalam and english) @xcite , kanglish (kannada and english) @xcite , and so on. the tamil language is one of the world's longest-enduring traditional languages, with a set of experiences tracing all the way back to 600 bce. tamil writing is overwhelmed by verse, particularly sangam writing, which is made out of sonnets formed between 600 bce and 300 ce. the main tamil creator was the writer and thinker thiruvalluvar, who composed the tirukkua, a gathering of compositions on morals, legislative issues, love and ethical quality broadly thought to be the best work of tamil writing. tamil has the oldest extant literature among dravidian languages. all dravidian languages evolved from classical tamil language @xcite . even though they have their own scripts still in the internet code-mixing comments can be found in these languages @xcite . identifying hate content in such bilingual or code mixed language is a very challenging task @xcite @xcite ). an automatic model which is trained in a monolingual context to detect hate posts may not yield the same result when tested bilingually or with a code-mix @xcite @xcite . this is because each system learns and recognises words in the given vocabulary. when a new word is encountered, which is not in the vocabulary, it is marked as an undefined token that makes no difference in the estimation of the model. therefore, when checked with the language in other scripts the model's performance decreases. the current study identifies the hate content in tanglish, manglish and malayalam script mixed in tweets and validated with the dataset provided in hasoc-dravidian-codemix-fire2020 challenge @xcite . the dataset proposed in the challenge was collected from twitter. a variety of deep learning models have been examined in the current paper to distinguish offensive posts from script-mixed posts. along with that we also examined a few transfer learning models like bert @xcite and ulmfit @xcite for the classification task. the rest of the article is summarized as follows: section 2 presents the overview of articles proposed in the domain of hate or offensive speech. the task and dataset description is explained in section 3. this followed by the explanation of the proposed methodology in section 4. the experimental results and discussion are explained in section 5 and 6. the paper concludes by highlighting the main findings in section 7. ## related works the hate speech identification in social media texts suffers from many challenges; such as code-mixed social media content, script-mixed social media contents, and so on. this section sheds light on a few state-of-art techniques presented to handle such issues. most of the analysis proposed for the detection of hate contents were validated with monolingual datasets. it is relatively easy to build a monolingual model, since (i) it is readily accessible, (ii) it learns a single language dictionary, and (iii) unknown token frequency is lower in the test data. @xcite worked on 25000 tweets in english and reported that tweets that contained racism and homophobic contexts were hate speech and tweets that contained sexism contexts were offensive contents. other work, on english data, was proposed by @xcite where n-gram features were extracted for identifying sexiest, racism, and none class. apart from this, some works were reported on a multilingual dataset where scripts of two or more languages are mixed. @xcite proposed a model for multi-lingual datasets containing aggressive and non-aggressive comments in english as well as hindi from facebook and twitter. @xcite used ensemble learning based on various machine learning classifiers such as logistic regression, svm with word ngram, character n-gram, word embedding, and sentiment as a feature set. they found that combined words and character n-gram features performed better than an individual feature. @xcite identified online social aggression on the facebook comment in a multilingual scenario and wikipedia toxic comments using stacked lstm units followed by convolution layer and fasttext as word representation. they achieved 0.98 auc for wikipedia toxic comment classification and a weighted f1 score of 0.63 for the facebook test set and 0.59 for the twitter test set. @xcite presented several models and their results for english, hindi, and german datasets. they reported the best model as long short term memory-based network that could capture the multilingual context in a better way. @xcite extended the earlier research of hate speech detection for code mixed tweets of hindi and english. @xcite presented a convolutional neural network (cnn) and binary particle swarm optimization (bpso) based model to classify multimodal posts with images and text into non-aggressive, mediumaggressive and high-aggressive classes. another multilingual context could be code-mixed where two languages are written in a single script. for example, @xcite @xcite proposed a code mixed dravidian data in tamil, malayalam, and kannada. @xcite developed a hinglish dataset from twitter. they reported preliminary experiment results of support vector machine (svm) and random forest (rf) classifiers with n-grams and lexicon-based features with an accuracy of 0.71. research in hate and offensive language, as described above, is mainly conducted in a monolingual setting. the paper aims to propose a machine learning system for the code-mixed and scriptmixed dataset to identify hate contents. ## task and data description the current study performs two tasks; (i) task 1 includes the development of an offensive and nonoffensive classification system for distinguishing script-mixed malayalam comments, and (ii) task 2 requires to build a classifier to differentiate tanglish and manglish (tamil and malayalam have written using roman characters) into offensive and not-offensive classes. ## methodology three different models were developed to identify hate or offensive contents in dravidian posts; (i) conventional learning based models, (ii) neural network-based models, and (iii) transfer learningbased models. in this section, we explain the working of each model in detail. a detailed diagram for presented models is shown in figure 1 . the results of the models are explained in section 5. ## result this section presents the experimental results of all three aforementioned models explained in section 4. the results are presented in terms of precision, recall, and f 1 -score of class offensive and notoffensive. the weighted average of both classes is also presented. a particular model is identified as best if it has reported the highest weighted average of precision, recall, and f 1 -score. the value in bold represents the highest value for a particular dataset. the convectional learning experiments were performed using character n-gram (1 to 6-gram) tf-idf features. the results are shown in the the results of the proposed neural-based models for tamil code-mixed, malayalam code-mixed, and malayalam script-mixed text are listed in ## result comparison and discussion of all the experimental models, conventional learning models with character 1 to 6 gram tf-idf features showed the best output for the two datasets, tamil code-mixed and malayalam code-mixed. for tamil code-mixed, the best performance was reported by nb model with precision, recall, and f 1 -score of 0.90. similarly, for malayalam codemixed the lr model reported best with precision, recall, and f 1 -score of 0.78. however, for the malayalam script mixed, vanilla neural network (vnn) reported best results having precision, recall, and f 1 -score of 0.95, 0.95, and 0.95 respectively. the receiver operating characteristics (roc) area under curve for all three best models are shown in figures 2 , 3 , and 4 . the outcome of this comprehensive study was surprising, given that the performance of all complex models, such as the bilstm-cnn hybrid model and transfer models, was relatively low, but 0.0 0.2 0.4 0.6 0.8 1.0 false positive rate 0.0 0.2 0.4 0.6 0.8 1.0 true positive rate receiver operating characteristic curve micro-average roc curve (@xmath0.97) macro-average roc curve (@xmath1.93) not-offensive (@xmath2.92) offensive (@xmath3.94) figure 4: roc for vanilla neural network (malayalam script-mixed) for many nlp tasks, such as text classification or language modelling, it was proven better. the results indicate that the character n-gram tf-idf features play a very important role for code-mixed and code-script data. secondly, in the same sense, the performance of the transfer models is not encouraging. bert, which is trained in the english language, treats most of the token of code-mixed and code-script data as an unknown token which could affect the model performance. the bert-multilingual which is trained on 102 languages identifies the language of input text first and loads its vocabulary then. even, for code-mixed and code-script data bert-multilingual identified a single language and was subsequently processed. in effect, the overall model performance was reduced. moreover, it was found that the language identified by bert-multilingual for code-mixed and the code-script dataset was different for different runs. consequently, the results fluctuated even. ## conclusion hate speech identification in code-mixed and scriptmixed context is one of the most challenging tasks in nlp. the current study presented extensive experiments utilizing various conventional learning, deep learning, and transfer learning models. three datasets used in the study were tamil code-mixed, malayalam code-mixed, and malayalam scriptmixed. the results reported by all models clearly show that conventional learning models along with vanilla neural model outperformed other complex deep learning, and transfer learning models. the character n-gram tf-idf based naive bayes classifier performed best with the weighted precision, recall, and f 1 -score of 0.90 for tamil code-mixed text. the logistic regression classifier with character n-gram tf-idf features performed best with the weighted precision, recall, and f 1 -score of 0.78 for malayalam code-mixed text. the vanilla neural network with character n-gram tf-idf features performed best with the weighted precision of 0.95, recall of 0.95, and f 1 -score of 0.95 for malayalam script-mixed text.