CUSATNLP @ D ravidian L ang T ech- EACL 2021:Language Agnostic Classification of Offensive Content in Tweets (2021)

Abstract: Identifying offensive information from tweets is a vital language processing task. This task concentrated more on English and other foreign languages these days. In this shared task on Offensive Language Identification in Dravidian Languages, in the First Workshop of Speech and Language Technologies for Dravidian Languages in EACL 2021, the aim is to identify offensive content from code mixed Dravidian Languages Kannada, Malayalam, and Tamil. Our team used language agnostic BERT (Bidirectional Encoder Representation from Transformers) for sentence embedding and a Softmax classifier. The language-agnostic representation based classification helped obtain good performance for all the three languages, out of which results for the Malayalam language are good enough to obtain a third position among the participating teams.

## introduction these days, social media platforms support almost all forms of languages for people to communicate. internet use has increased through the support of language technologies, and code-mixing is common nowadays @xcite . with more and more language support, free form communication is possible among people, and as such, comment filtering or evaluating comment content based on its offensiveness to an individual, group, or other forms is an essential task @xcite . these platforms also support languages from the dravidian family @xcite . as a result, it is necessary to identify offensive content from tweets in english and all other languages, including code mixed tweets used in these platforms @xcite . dravidian languages are spoken and used in written form by the people in south india and sri lanka. this language is also used by a small portion of people in many parts of the world. dravidian languages are not new, it is almost 4500 years old @xcite , but it remains an underresourced language (chakravarthi, 2020). the commonly used languages of this type include kannada, malayalam, and tamil. these are free word order languages and have high agglutinating nature. tweet classification based on its offensive content has been challenging over these years. initially, the focus was on the english language, which is the prime language spoken and used worldwide. nowadays, its focus is on the low resource languages. the tweets classification is based on different classes, and it can be a binary classification into offensive or not-offensive class. in an alternative way, we can classify tweets based on offensive content, their targeted audience (individual, group, or other), language, and form of insult. this working note paper discusses the methodology used to identify offensive content from tweets in kannada, malayalam, and tamil. this paper's subsequent sections are as follows: section 2 discusses the related works based on offensive content identification. details about the task and the dataset are in section 3. the methods used for this classification task is in section 4. we project the results and discuss the observations in section 5. the last section, section 6, concludes the working note. ## related works we discuss the works done so far for offensive content classification in this section. it is one of the very relevant topics of concern these days as communication is mostly through online platforms, especially in a pandemic situation like covid-19 that we face now. offensive language identifica-tion has started with english and other foreign languages, and it has gained its importance concerning other languages, including dravidian and low resource languages. apart from the different forms of offensive content, such as hate speech, abusive language, cyberbullying, cyber-aggression, there are different kinds of offensive content. @xcite discuss the different kinds of offensive content classes and a basic definition of the different classes. they proposed a hierarchical model of offensive content in different levels, namely, level a: offensive language detection, level b: categorization of offensive language, level c: offensive language target identification. the first level (level a), offensive language detection, consists of two major classes, namely not offensive, defined as posts/comments without any offense or profanity and offensive defined as posts with profanity (non-acceptable language) or offense, including insults, threats, and swear words. the second level (level b) consists of categorizing offensive language into targeted insult and untargeted class. the targeted insult class consists of posts that contain insult to an individual, group, or others. the untargeted class contains posts that are not targeted but have non-acceptable language or swearing. the third level (level c) classifies the targets of insults or threats. it is of three types: individual, group, and other. the individual (ind) class has posts that target an individual, may or may not be a named person, also called cyberbullying, whereas the group (grp) class targets people of the same religion, affiliation, or gender (also referred to as hate speech). the other (oth) category targets organizations, situations, events, or issues that do not fall in the above categories. zampieri et al. (2019) also classified the offensive language identification (olid) dataset based tweets based on the defined classes using svm, cnn, bilstm, out of which cnn performed better for the three levels of classification. offensive language identification has taken place in shared tasks these years. we discuss a few of the notable works here. apart from the english language, this problem is also being solved in other languages. they predicted offensive content in bengali, hindi, and spanish languages. pretrained bert models combined with cnn were used for arabic, greek, and turkish offensive language identification @xcite . the significant contributions of offensive language identification from different teams that participated in semeval, 2020 are discussed in @xcite . three sub-tasks corresponding to three levels of the olid dataset hierarchy were offered for arabic, danish, english, greek, and turkish languages. german bert model based on offensive language identification was experimented with for the german language by @xcite as part of germeval,2019. ## task details and dataset the shared task on offensive language identification in dravidian languages aims to identify offensive content from kannada-english, malayalam-english, and tamil-english code-mixed tweet datasets @xcite . it is an imbalanced multi-class classification at the comment level. the dataset for this task consists of classes, namely, the dataset statistics for the three languages are shown in ## methods our classification method uses a good sentence representation that is purely language-agnostic in nature. the system design consists of a representation module and a classifier module. representation module : efficient sentence embedding for the comments or tweets is derived using the language-agnostic bert (labse). the comments are in the form of code mixed language scripts written in the non-native script, english, and transliterated form. we have not used any preprocessing technique to make the method more generic and add to the language-agnostic nature. the language-agnostic bert uses a masked language model and a translation language model based pretrained encoder. the dual encoder architecture helps to encode source and target sentences separately and feed a combinational function. we obtain the individual embeddings from the [cls] token's embeddings using a shared bert encoder. the source and target sentence embeddings comparison using cosine similarity and additive margin softmax loss learn useful cross-lingual embeddings. the architecture of language-agnostic bert is in figure 1 . this approach performed equally well for all the resource-rich languages and resource-poor languages. in zero-shot transfer, it showed aboveaverage performance. this method was evaluated on different corpus like bucc shared task for parallel sentence extraction, united nations parallel corpus, tatoeba corpus for 109 languages, and zeroshot transfer for languages without training data @xcite . classification module : the embedded representation of sentences are passed through a perceptron based classifier. it is a two-layer classifier with dense(100) as an input layer with relu activation and dense(5,6) as an output layer with a softmax activation function for multiclass classification. the classifier is trained for 20 epochs with validation data used from the development dataset provided. the same approach is followed for all three languages. the overall system design is shown in figure 2 . ## results and observations this classification task's experimental results are measured in terms of weighted average precision, recall, and f1-score using scikit-learn classification report. weighted average averages the support weighted mean per label 1 and are used when the 1 https://scikit-learn.org/stable/modules/generated /sklearn.metrics.classification report.html datasets are highly imbalanced. ## conclusions in this paper, we used language-agnostic bert for embedding comments/tweets of any language and identified the different categories of offensive content using a softmax classifier. this model achieved comparatively good results without applying any pre-processing techniques. the quality of the embedding model helps to encode information more effectively. comparing labse for kannada, malayalam, and tamil language, this embedded representation is more efficient for the malayalam language and helped improve classification results. our team was ranked 3rd for the malayalam language, 11th for the kannada language, and 9th for the tamil language in dravidianlangtech 2021, part of eacl, 2021.