DLRG @ D ravidian L ang T ech- ACL 2022: Abusive Comment Detection in T amil using Multilingual Transformer Models (2022)

Abstract: Online Social Network has let people to connect and interact with each other. It does, however, also provide a platform for online abusers to propagate abusive content. The vast majority of abusive remarks are written in a multilingual style, which allows them to easily slip past internet inspection. This paper presents a system developed for the Shared Task on Abusive Comment Detection (Misogyny, Misandry, Homophobia, Transphobic, Xenophobia, CounterSpeech, Hope Speech) in Tamil DravidianLangTech@ACL 2022 to detect the abusive category of each comment. We approach the task with three methodologies - Machine Learning, Deep Learning and Transformer-based modeling, for two sets of data - Tamil and Tamil+English language dataset. The dataset used in our system can be accessed from the competition on CodaLab. For Machine Learning, eight algorithms were implemented, among which Random Forest gave the best result with Tamil+English dataset, with a weighted average F1-score of 0.78. For Deep Learning, Bi-Directional LSTM gave best result with pre-trained word embeddings. In Transformer-based modeling, we used IndicBERT and mBERT with fine-tuning, among which mBERT gave the best result for Tamil dataset with a weighted average F1-score of 0.7.

## introduction the usage of the internet and social media has increased exponentially over the previous two decades, allowing people to connect and interact with each other @xcite . this has resulted in a number of favourable outcomes such as monitoring pandemic trends, empowering patients and enhancing public communication through social media, amongst others @xcite @xcite . at the same time, it has also brought with it hazards and negative consequences, one of which is the use of abusive language on others @xcite . the rapid spread of abusive content on social networking has become a major source of concern for government organisations. it is very difficult to identify abuse over online social network due to the massive volume of content generated through social media in different online platforms @xcite @xcite . it becomes a bigger problem when most of the communication is in multilingual style @xcite . hence, there is increasing interest in the use of automated methods for detecting online social abuse @xcite . it is becoming a major area of research to find solutions with powerful algorithmic systems to curb the growth of abusive content online. one possible way of achieving such a system is by using state-of-the-art natural language processing (nlp) techniques, which can analyse, comprehend and interpret the meaning of the natural language data. in addition, the detection of abusive language online is harder for some languages like tamil due to the presence of code-mixed @xcite and code-switched @xcite data. codeswitching is when in a single discourse, a person switches between two or more languages or language varieties/dialects @xcite . it refers to using elements from more than one language in a way that is consistent with the syntax, morphology, and phonology of each language or dialect. code-mixing is the hybridization of two languages (for example, parkear, which uses an english root word and spanish morphology), which refers to the migration from one language to another. many such language pairs have a hybrid name. tamil is a member of the southern branch of the dravidian languages, a group of about 26 languages indigenous to the indian subcontinent. it is also classed as a member of the tamil language family, which contains the languages of around 35 ethno-linguistic groups, including the irula and yerukula languages @xcite @xcite . malayalam is tamil's closest significant cousin; the two began splitting during the 9th century ad. although several variations between tamil and malayalam indicate a pre-historic break of the western dialect, the process of separating into a different language, malayalam, did not occur until the 13th or 14th century @xcite @xcite @xcite . tanglish is an example which is tamil+english. in this task, we are given two datasets: one with a tamil meaning written in english but the content is a combination of tamil and english. the other is a tamil+english dataset (tanglish) which is written in tamil and english with content in tamil and english as well. there are also known challenges in the development of computational systems in tamil because of the lack of linguistic resources @xcite . in this paper, we present computational systems for the automated detection of abusive language using the two different data sets containing tamil and tamil+english. ## related work in this section, we review the various methodologies and systems previously implemented for similar tasks in under-resourced languages like tamil. hope speech is annotated equality, diversity and inclusion (hopeedi) (chakravarthi, 2020). they also created several baselines to standard the dataset. (chakravarthi and muralidaran, 2021) reports on the shared task of hope speech detection for tamil, english and malyalam languages. they presents the dataset used in the shared task and also surveys various competing approaches developed for the shared task and their corresponding results. (mandalam and sharma, 2021) presents the methodologies implemented while classifying dravidian tamil and malayalam code-mixed comments according to their polarity and uses lstm architecture. @xcite @xcite use xlm-roberta for offensive lan-guage identification. novel approach of selective translation and transliteration have been used to improve the performance of multilingual transformer networks such as xlmroberta and mbert by fine-tuning and ensembling. online messaging has become one of the most popular methods of communication with instances of online/digital bullying. the challenge of detecting objectionable language in youtube comments from the dravidian languages of tamil, malayalam, and kannada is viewed as a multi-class classification problem (andrew, 2021). several machine learning algorithms have been trained for the task at hand after being exposed to language-specific pre-processing. ## dataset the dataset for the current study is taken from the competition foot_0 which consists of youtube comments in tamil and tamil-english languages annotated for misogyny, homophobia, transphobic, xenophobia, counter-speech, hope-speech and misandry (and none-of-the-above) @xcite . ## proposed technique raw texts are inaccessible to machine learning (ml) and deep learning (dl) algorithms. to train the models for classification, feature extraction is necessary. to extract features in ml approaches, the tf-idf representation is used. for dl models, we use fasttext word embeddings feature extraction strategies @xcite . fasttext embedding uses a pre-trained embedding matrix for tamil language @xcite . to study the results and come up with the best model possible, we follow three approaches -machine learning, deep learning and transformer-based. as it can be clearly seen from the issue of class imbalance, we practice various approaches: changing the performance metric: since accuracy is not always the best metric to use on imbalanced datasets, we use f1-score instead to evaluate the models. using a penalized algorithm (cost-sensitive training): this algorithm also handles class imbalance which can be achieved by using 'balanced' as a parameter while computing class weights. changing the algorithms: this is why we have used a wide variety of algorithms to get a bigger picture of which models suit the dataset and the classification problem better. to study the results and come up with the best model possible, we follow three approaches -machine learning, deep learning and transformerbased, described in the sub-sections below. ## results and discussion we ran 8 machine learning algorithms, 2 deep learning and 1 transformer model on the tamil+english dataset. for the tamil dataset, we used 2 deep learning and 1 transformer model. for the tamil+english dataset, the best performance was of random forest with macro average f1-score of 0.32 and weighted average f1-score of 0.78. for the tamil dataset, the best model was bilstm-m2 with macro average f1-score of 0.39 and weighted average f1-score of 0.70. for tamil, performance improved from switching bilstm-m2 to mbert. and for tamil+english, the best performer was bilstm-m1, followed by bilstm-m2 and then indicbert and mbert. ## conclusions and future work in this paper, we presented approaches for the automated detection of abusive comments in tamil. we used various models to do a comparative study to see which model performed better with the dataset given in the shared task. we found that deep learning and transformer models outperformed machine learning models with tamil data whereas machine learning models achieved better results than deep learning and transformer-based for tamil+english data. we did not apply contextualized embeddings (such as elmo, flair) which may improve the performance of the system. implementation of contextualised embeddings using language modelling with deep learning is the future work to explore.