GJG @ T amil NLP - ACL 2022: Using Transformers for Abusive Comment Classification in T amil (2022)

Abstract: This paper presents transformer-based models for the “Abusive Comment Detection” shared task at the Second Workshop on Speech and Language Technologies for Dravidian Languages at ACL 2022. Our team participated in both the multi-class classification sub-tasks as a part of this shared task. The dataset for sub-task A was in Tamil text; while B was code-mixed Tamil-English text. Both the datasets contained 8 classes of abusive comments. We trained an XLM-RoBERTa and DeBERTA base model on the training splits for each sub-task. For sub-task A, the XLM-RoBERTa model achieved an accuracy of 0.66 and the DeBERTa model achieved an accuracy of 0.62. For sub-task B, both the models achieved a classification accuracy of 0.72; however, the DeBERTa model performed better in other classification metrics. Our team ranked 2nd in the code-mixed classification sub-task and 8th in Tamil-text sub-task.

## introduction the advent of social media and social networks have completely revolutionized the way people communicate with one another @xcite . there are many positive aspects of social mediaimproved connectivity, real-time conversation across multiple locations, a new type of social construct, etc. however, this surge of internetbased communication has also brought about an increase in the volume of negative comments @xcite @xcite @xcite . being able to detect and classify such negative and abusive comments is a fundamental and challenging problem to solve. * these authors contributed equally to this work it is fundamental because better hate & abusive comment detection leads to the improvement of spam detection systems, improves web-inclusivity, and ultimately makes the internet a better place for everybody @xcite . abusive comment detection and classification falls under the broader spectrum of text classification tasks in natural language processing (nlp). improvements in abusive comment classifiers can directly enhance content filtering systems, digital well-being software, spam detection, etc @xcite . it also leads to a better physical and mental experience for the end-user, as these classifiers can be used to reduce the various types of abusive comments in the digital world. just as with any other nlp task, building good abusive comment classifiers requires annotated datasets. there are plenty of such datasets available for high-resource languages like english, which has lead to a lot of published research in this space. however, for low-resource languages like tamil, there are very few publicly available datasets for downstream nlp tasks @xcite . tamil is a dravidian classical language used by the tamil people of south asia. tamil is an official language of tamil nadu, sri lanka, singapore, and the union territory of puducherry in india @xcite @xcite . significant minority speak tamil in the four other south indian states of kerala, karnataka, andhra pradesh, and telangana, as well as the union territory of the andaman and nicobar islands. it is also spoken by the tamil diaspora, which may be found in malaysia, myanmar, south africa, the united kingdom, the united states, canada, australia, and mauritius @xcite @xcite @xcite . tamil is also the native language of sri lankan moors. tamil, one of the 22 scheduled languages in the indian constitution, was the first to be designated as a classical language of india @xcite . this shared task is an attempt to promote research in abusive comment detection and classification in tamil. this novel dataset, generated from youtube comments, consists of 8 types of abusive comments. the publication of the tamil-text as well as the code-mixed tamil-english datasets also provides an opportunity to learn about the performance of models on different character sets. in this shared task, we participated in both the sub-tasks: the tamil-text classification sub-task a, and the code-mixed sub-task b. the goal of this paper is to demonstrate the performance of finetuning pre-trained transformer-based models for such a text-classification task in tamil. we train an xlm-roberta and deberta model for each subtask on the given train splits, optimize parameters, and evaluate the performance on the test split. the rest of our paper is organized as follows: we discuss related work in tamil emotion recognition, describe the datasets, our methodology, and conclude with the results and performance metrics. we provide a link to our models and evaluations to provide reproducability, and empower future research in this domain 1 . ## related work as mentioned above, a lot of published work exists in the domain of offensive language detection for high-resource languages. @xcite trained a binary classifer to detect racist tweets in english. @xcite presented off-theshelf nlp approaches to identify bullying in social media, in english. @xcite present their findings from a shared task for aggression identification in social media. @xcite demonstrated a machine learning approach to detect abusive language in english online content. the volume of published research in abusive comment detection for low-resource languages is much lower than that of english and other high-resource languages. @xcite provided an overview of a shared abusive comment detection task in german. kannan and mitrović chakravarthi (2020), chakravarthi and muralidaran (2021), and @xcite have published their findings from other text classification-related shared tasks in dravidian languages (hope speech detection). mandl et al. (2020) organized a workshop track for hate speech detection in tamil, malayalam, hindi, english and german. we believe this was the first big focus on developing abusive content identification and classification techniques for dravidian languages. this was followed up by @xcite , who organized a shared task for offensive language identification in tamil, malayalam, and kannada. there have also been multiple previous works that use xlm-roberta for text classification tasks. @xcite proposed a system using xlm-roberta and dpcnn for detecting offensive text in dravidian languages. @xcite used textcnn and xlm-roberta from emotion classification in spanish. ou and li (2020) also demonstrated using xlm-roberta for a hate speech identification classification task. the number of published works using deberta is fewer than that of xlm-roberta. there have been some studies that use deberta for entity extraction and text-classification tasks. @xcite 3 data the organizers of the shared task released the annotated training and development splits for both the sub-tasks. the testing dataset, without labels, was released a few days prior to the run submission deadline. once the results were announced, the organizers released the labeled test dataset for verification purposes. for sub-task a, the dataset consisted of tamil sentences annotated to one of eight english abusive categories: misandry, counter speech (sp.), misogyny, xenophobia, hope sp., homophobia, transphobia, or none of the above (n.o.t.a). the dataset for sub-task b was code-mixed tamil-english with the same eight english abusive comment classes as sub-task a. the average length of a sentence in the corpora was 1 (priyadharshini ## methodology for each classification sub-task, we fine-tune an xlm-roberta and deberta base model on sentences from the training splits to create a classification model. we do not remove any stop words, special characters, or emojis from the test splits, in order to preserve the context of the comment. special characters and emojis provide useful context, especially for a text classification task. for sub-task a, we also did not remove the 2 instances of "not tamil" from the training set. xlm-roberta is a multilingual version of roberta, which in itself was an improvement over bert to achieve state-of-the-art results in multiple nlp tasks. xlm-roberta is pre-trained on 2.5tb of filtered commoncrawl data containing 100 languages @xcite . deberta uses disentangled attention and enhanced mask decoder to enhance roberta and outperform it in a majority of nlp tasks @xcite . ## results we use accuracy and the weighted averages of precision, recall, and f1-score as performance metrics to evaluate our classification models. we calculate all four evaluation metrics to get a better sense of the classification performance. for sub-task a, we find that the xlm-roberta outperforms the deberta in all evaluation metrics. both models used the same training splits and parameters. the multi-lingual nature of xlm-roberta is evident from its better performance. the overall classification performance for sub-task 2 was higher than that for sub-task a. we believe this is because of the english charater set used by the code-mixed dataset. both the transformer-models are pre-trained on large english datasets. deberta outperforms xlm-roberta in all metrics, which justifies deberta's improvement over xlm-roberta for english-character-nlp tasks ( ## conclusion and future work we present xlm-roberta and deberta models for two multi-class text classification tasks in tamil. the objective of the shared task was to identify abusive content in tamil text. there were two subtasks: sub-task a in tamil text, and sub-task b which used tamil-english code-mixed text. the classes of abusive comments were the same for both the sub-tasks. the tamil dataset, for sub-task a, consisted of 2,240 rows in the training split. the we propose the fine-tuning of pre-trained xlm-roberta and deberta, which are transformerbased models, for classifying tamil text into abusive comment classes. we trained all the models using the respective training splits as-is. for sub-task a, the xlm-roberta achieved a classification accuracy of 66% with a weighted f-1 of 0.65, precision of 0.65, and a recall value of 0.66. the deberta model achieved an accuracy of 62% with weighted f-1 of 0.57, precision of 0.62, and 0.56 recall. for sub-task b, the xlm-roberta achieved a classification accuracy of 72% with a weighted f-1 of 0.70, precision of 0.70, and a recall value of 0.72. the deberta model achieved an accuracy of 72% with weighted f-1 of 0.72, precision of 0.72, and 0.72 recall. we show that the xlm-roberta model outperforms deberta for sub-task a, which used tamil text. for the code-mixed tamil-english text (sub-task b), the deberta model outperforms the xlm-roberta. this validates the strength of the deberta model on english character tasks, and the superiority of the xlm-roberta for non-english languages. by using the training split asis, we retain the information provided by special characters like emojis and extended punctuation symbols. the xlm-roberta model had the eight best classification performance in the shared task for sub-task a, and the deberta model ranked second best. we have open-sourced the code used in this study in a public github repository. for future-work, removing the "not tamil" rows from the training split for sub-task a would eliminate an extremely under-sampled class, which may lead to performance improvements. extracting emojis separately from the text and incorporating emoji-information in identifying abusive comments is also an area for potential study. using pretrained models is a good starting point in this domain, however, we feel that custom model architectures and systems have to be studied for such classification tasks in tamil.