HARMONY @ D ravidian L ang T ech: Transformer-based Ensemble Learning for Abusive Comment Detection (2023)

Abstract: Millions of posts and comments are created every minute as a result of the widespread use of social media and easy access to the internet.It is essential to create an inclusive environment and forbid the use of abusive language against any individual or group of individuals.This paper describes the approach of team HARMONY for the “Abusive Comment Detection” shared task at the Third Workshop on Speech and Language Technologies for Dravidian Languages.A Transformer-based ensemble learning approach is proposed for detecting abusive comments in code-mixed (Tamil-English) language and Tamil language. The proposed architecture achieved rank 2 in Tamil text classification sub task and rank 3 in code mixed text classification sub task with macro-F1 score of 0.41 for Tamil and 0.50 for code-mixed data.

## introduction online social networks (osns) have become increasingly significant in recent years and are now considered to be go-to resource for news, knowledge, and entertainment @xcite @xcite . however, despite the numerous benefits of using osns, mounting evidence suggests that an increasing number of illicit actors are taking advantage of these platforms to spread toxicity and harm other people. numerous risks are brought on by these illicit people, including online abuse, vulgarity, harassment and bullying. abusive comments are those that mock or disparage a person or a group based on traits like colour, ethnicity, sexual orientation, nationality, race, or religion @xcite . social media abuse can have a negative impact on users' lives in a number of ways. this will have a terrible impact on that person's mental health, leading to depression and insomnia @xcite @xcite . some of these remarks have the potential to stir up controversy on social media about a particular person or group of people. this demonstrates the need to prohibit the posting of these kinds of offensive comments on social media. both the union territory of puducherry (pondicherry) and the indian state of tamil nadu speak tamil as their official language. in addition, it is spoken widely in malaysia, mauritius, fiji, and south africa. it is also the official language of singapore and sri lanka. @xcite in the past, text classification was carried out on the text's sentence embedding using linear classifiers. this was followed by recurrent neural networks. transformers became prominent in the field of natural language processing following the publication @xcite . they have an attention layer mechanism that provides context to words in the text. the development of the transformer architecture has resulted in the development of numerous other transformer variations, such as bert @xcite , xlmroberta @xcite , muril @xcite , etc. in this paper, transformer-based models and recurrent neural network models are used for abusive comment detection in tamil and code-mixed data(tamil-english). the results from the performance of individual models and their ensemble are obtained to determine the best-performing model for this task. ## related works numerous published works have addressed the problem of identifying offensive content in highresource languages. @xcite analysed the impact of fasttext word embedding on text classification. compared to contextual word embeddings, fasttext has limited ability to capture complex se-mantic relationships. @xcite demonstrated an attentional multi-channel convolutional-bilstm network for the classification of hateful content. glove embedding used in this paper may struggle with out-of-vocabulary words. the pretrained bert model ensembled with deep learning (dl) models are the foundation of @xcite proposed multi-aspect hate speech detection approach.this approach will have difficulty in handling multi-lingual or non-english data. bas ¸arslan and kayaalp (2023) proposed a deep learning model with multiple bidirectional gated recurrent units and convolution layers for social media sentiment analysis. compared to english and other high-resource languages, low-resource languages have significantly fewer published studies on the detection of abusive comments. datasets have been created by @xcite to promote research in tamil, one of the indian dravidian languages. @xcite proposed a method to detect hate speech or offensive content in tamil. a detailed analysis was made to study the performance of stemming and pre-trained transformer models for word embedding. for the detection of offensive language in tamil youtube comments, @xcite proposed adapter-based transformer models. it was done using mbert, muril (base and large), and xlm-roberta (base and large).chakravarthi et al. (2023) proposed a novel approach of fusing mpnet with a deep neural network for detecting offensive language content in low-resource dravidian languages ## dataset the objective of this shared-task @xcite is to determine whether a given comment contains abusive language. the annotations in the corpus are done at the comment or post level. tamil and tamil-english comments were gathered from the youtube comment section for the abusive comment detection dataset @xcite . the dataset consists of a comment and its corresponding label from one of the nine labels in the dataset: misandry, counter-speech, misogyny, xenophobia, hope-speech, homophobia, transphobic, not-tamil, and none of-the-above. only eight classes are classified because the 'not-tamil' class has no test or development data instances. a few weeks before the deadline for run submission, the testing dataset, which lacked labels, was made available. the labelled test dataset was made available by the organisers for verification purposes after the results were declared. the number of samples in the training, validation, and testing datasets for each class are listed in tables 1 and 2 . ## methodology the overall architecture for abusive comment detection in tamil and tamil-english is given in figure 1. ## results and analysis the training set was used to train all the models, and the development set was used to validate them.the organisers indicated macro f1-score as their primary evaluation metric.apart from this,a few more performance metrics like accuracy and the macro averages of precision and recall are also used to assess the classification models. in order to determine the classification performance more accurately, all four evaluation metrics are calculated.the performance of the models on the tamil and tamil-english validation datasets is shown in tables 4 and 5 , respectively.on analysing the performance of all the models on validation datasets, it is clearly seen that the ensemble model performs better than the individual models. the weighted average ensemble model works better because it combines diverse representations, corrects errors, reduces bias, improves robustness to variability, and leverages the strengths of each model. by integrating multiple perspectives and leveraging their complementary abilities, the ensemble captures a broader range of patterns and linguistic nuances, leading to an improved performance. due to higher macro-f1 score, the ensemble model is used in both cases for the test dataset. ## conclusion and future work in this paper, a new approach has been proposed for abusive comment detection based on ensemble learning. the proposed model combined the pre-trained transformer models(muril, xlm-roberta, deberta) with a deep-learning model built by stacking bi-lstms and bi-grus on fast-text embeddings. this ensemble learning has actually reduced the number of misclassified instances and thus improved the precision of the abusive comment detection model. for tamil dataset, an ensemble of muril, bi-lstm-bi-gru and xlm-roberta provided the best results with a macroaveraged f1 score of 0.41. for tamil-english dataset, an ensemble of deberta, bi-lstm-bi-gru and xlm-roberta provided the best results with a macro-averaged f1 score of 0.50. further this work can be extended by exploring the performance of adapter-based transformer models. ensembling of other transformer models with rnns can also be explored in future.