NLP - CUET @ D ravidian L ang T ech- EACL 2021: Offensive Language Detection from Multilingual Code-Mixed Text using Transformers (2021)

Abstract: The increasing accessibility of the internet facilitated social media usage and encouraged individuals to express their opinions liberally. Nevertheless, it also creates a place for content polluters to disseminate offensive posts or contents. Most of such offensive posts are written in a cross-lingual manner and can easily evade the online surveillance systems. This paper presents an automated system that can identify offensive text from multilingual code-mixed data. In the task, datasets provided in three languages including Tamil, Malayalam and Kannada code-mixed with English where participants are asked to implement separate models for each language. To accomplish the tasks, we employed two machine learning techniques (LR, SVM), three deep learning (LSTM, LSTM+Attention) techniques and three transformers (m-BERT, Indic-BERT, XLM-R) based methods. Results show that XLM-R outperforms other techniques in Tamil and Malayalam languages while m-BERT achieves the highest score in the Kannada language. The proposed models gained weighted f_1 score of 0.76 (for Tamil), 0.93 (for Malayalam ), and 0.71 (for Kannada) with a rank of 3rd, 5th and 4th respectively.

## introduction the exponential increase of offensive contents in social media has become major concern to government organizations and tech companies. it is impossible to identify offensive texts manually from enormous amounts of online contents generated every moment in social media and other online platforms. therefore, academicians, policymakers and stakeholders are trying to develop robust computational systems to limit the spread of offensive contents using state of the art nlp techniques @xcite . in the last few years, several studies have been conducted regarding offensive languages such as hate speech, aggression, toxicity and abusive language. although most social media users used their regional languages to carry out communication; however, plenty of resources developed in english, arabic @xcite and other high-resources languages. lately, people started using code-mixed texts on social media @xcite . linguistic level understanding of the text, complex morphological structure, and lack of training corpora are crucial barriers to classifying code-mixed data. moreover, the system trained on monolingual data alone may fail to classify code-mixed data because of the texts linguistic level code-switching complexity. this work aims to address the above mentioned problems by contributing the following: ## related work offensive social media content might trigger objectionable consequences to its user like mental health problem, and suicide attempts @xcite . to keep the social media ecosystem, coherent researchers and stakeholders should try to develop computational models to identify and classify offensive contents within a short period. @xcite develop an offensive language identification dataset using hierarchical annotation schema. three layers of annotation are used in the dataset: offensive language detection, categorization of offensive language, and offensive language target identification. semeval task 6 is organized based on this dataset which had opened interesting research directions @xcite . in early stages, computational models created by using support vector machine, naive bayes and other traditional machine learning approach @xcite . these models performance is not up to the mark as they could not capture the semantic and contextual information in texts. this problem was mitigated with the arrival of word embeddings and recurrent neural networks (aroyehun and gelbukh, 2018). networks like bidirectional lstms and grus can hold contextual information from both past and future, creating more robust classification systems @xcite @xcite . in recent years, transformerbased model such as bert @xcite , xlm-r gained more attention to identify and classify offensive texts. these large pre-trained model can classify code-mixed texts of different languages with astonishing accuracy @xcite . ## dataset in order to detect offensive text from social media, task organizers developed a gold standard corpus. as many social media texts are code-mixed, so system trained on monolingual data fails to classify code-mixed data due to the complexity of codeswitching at different linguistic levels in the texts. to address this phenomenon, chakravarthi et al. the corpus consists of the text of three different languages, i.e. tamil, kannada and malayalam. system developed with the different number of train, validation and test examples for each language. the assigned task is a multi-class classification problem where the model has to classify a potential text into one of six predefined classes. these classes are not offensive (nf), offensive-targeted-insult-other (otio), offensive-targeted-insult-individual (otii), offensive-targeted-insult-group (otig), not-tamil (nt)/not-malayalam (nm)/not-kannada (nk), and offensive-untargetede (ou). ## methodology in this part, we briefly describe the methods and techniques employed to address the previous section's problem. first, features are extracted with different feature extraction techniques and various machine learning (ml) and deep learning algorithms are used for the baseline evaluation. furthermore, different transformer models, i.e. xlm-r, m-bert and indic-bert, are exploited as well. figure 1 shows the schematic process of the developed system. feature extraction: ml and dl algorithms are unable to learn from raw texts. therefore, feature extraction is required to train the classifier models. for ml methods, tf-idf technique @xcite is applied to extract the unigram features. on the other hand, word2vec @xcite and fasttext @xcite embeddings are used as feature extraction techniques for dl models. word2vec implemented using keras embedding layer with embedding dimension 100 for all languages while pre-trained embedding matrix of each language is utilized for fasttext embedding. machine learning models to develop the offensive text detection system, we begin our investigation with traditional ml approaches, including lr and svm. the ensemble of multiple ml classifiers is also employed to achieve better performance. we use 'lbfgs' optimizer with c value of 0.4, 0.7, and 5 for tamil, malayalam and kannada languages to implement lr. we selected 'linear' svm and settled c value to 3, 10, and 7. besides, decision tree (dt) and random forest (rf) classifiers are incorporated with lr and svm to construct the ensemble approach. for lr and svm, the same parameters were retained while 'n @xmath0. majority voting technique is applied to get the prediction from the ensemble method. deep learning models dl algorithms are proven superior over many traditional ml approaches in various classification tasks @xcite . we employ lstm and combination of lstm with attention-based approach to classify the offensive text to continue the investigations. lstm is well known for its ability to capture the semantic information as well as long term dependencies. bidirectional lstm (bilstm) with 100 cells are used to exploit the information from both past and future states. to mitigate the chance of overfitting, a dropout technique utilized with a rate of 0.1. finally, the output of the bilstm layer transferred to a softmax layer for the prediction. however, sometimes all the words in a text do not contribute equally for the classification. to emphasize the words that have a noticeable impact on the input text, we use attention @xcite mechanism. an attention layer of 20 neurons is constructed on the top of a bilstm layer. the attention operation applied upon the output of bilstm layer and the attention vector passed to a softmax layer for the prediction. we employ the same architectures for all the languages and use 'sparse categorical crossentropy' as the loss function. 'adam' optimizer with learning rate 1e -3 and batch size 32 is used to train the models for 20 epochs. the best intermediate model is stored by using keras callbacks. ## results and analysis meanwhile, transformer-based models showed outstanding performance for all the languages. for tamil, indic-bert obtained f 1 score of 0.74 while both m-bert and xlm-r exceeds all the previously mentioned models by achieving maximum f 1 score of 0.76. though two models give an identical f 1 score, xlm-r is opted as the best model by considering both the precision, recall scores. in malayalam language, m-bert and indic-bert obtained f 1 score of 0.90 and 0.92 respectively. while xlm-r shows a rise of 0.01 and thus beats all the other models. for the kannada language, both xlm-r and m-bert outperformed all the previous models obtaining f 1 score of 0.71. however, m-bert is selected as the best model as it outdoes xlm-r concerning precision and recall values. the results revealed that the transformerbased models performed exceptionally well to accomplish the assigned task for all the languages. compared to other techniques, the ml models performed moderately while the deep learning models lag than the other models. the possible reason for this weaker performance is the extensive appearance of cross-lingual words in the text. as a result, word2vec and fasttext embeddings have failed to create the appropriate feature mapping among the words. thus, lstm and attentionbased models may not find sufficient relational dependencies among the features and performed below the expectation. ## conclusion in this work, we have described and analyzed the system's performance implemented as a participation in the offensive language identification shared task at eacl-2021. initially, svm, lr, lstm, lstm+attention models have employed with tf-idf and word embedding features. results indicate that ml ensemble achieved higher accuracy than dl methods. however, the outcomes are not promising for the available datasets. code-mixing of multilingual texts might be a reason behind this. we applied transformer-based models to overcome this situation, which provides an astonishing rise in accuracy than ml and dl-based methods. weighted f 1 score increased from 0.73 to 0.76, 0.88 to 0.93 and 0.48 to 0.71 for tamil, malayalam and kannada language respectively. in future, the idea of ensemble technique could be adopted on transformer-based models to investigate the system's overall performance.