Transformers@ LT - EDI - EACL 2024: Caste and Migration Hate Speech Detection in T amil Using Ensembling on Transformers (2024)

Abstract: In recent years, there has been a persistent focus on developing systems that can automatically identify the hate speech content circulating on diverse social media platforms. This paper describes the team “Transformers” submission to the Caste and Migration Hate Speech Detection in Tamil shared task by LT-EDI 2024 workshop at EACL 2024. We used an ensemble approach in the shared task, combining various transformer-based pre-trained models using majority voting. The best macro average F1-score achieved was 0.82. We secured the 1st rank in the Caste and Migration Hate Speech in Tamil shared task.

## introduction hate speech can be defined as the use of aggressive, abusive or threatening expressions or phrases. with the advancement of the technological age, everyone has access to the internet to voice their opinions to a large audience. however, some people may misuse this power to spread hate against a certain individual or a group of individuals based on certain distinguishing characteristics. this could be through posts on social media, blogs, videos, or comments on various platforms. hence, it has become crucial to regulate the comments on social media platforms to avoid hurting sentiments. the shared task 1 organized by lt-edi aimed to detect caste and migration hate speech in tamil text @xcite . social media platforms have taken the freedom of speech and expression beyond global borders. platforms like twitter, instagram, and youtube allow ideas shared from one corner of the world to reach millions of people across the world in just a few milliseconds @xcite . however, the increased anonymity provided by 1 https://codalab.lisn.upsaclay.fr/ competitions/16089 such platforms has lead users to exploit this power by sharing opinions and ideas targeted against an individual or a group. this makes it crucial to regulate the hateful content shared online automatically to attenuate the societal harm it can cause. in the targeted hate speech identification domain, natural language processing (nlp) has experienced major breakthroughs in the past few years. recent developments include long short term memory @xcite , and gated recurrent units @xcite . but with the introduction of transformers @xcite , the results have seen a paradigm shift. tamil is one of the twenty-two scheduled languages in the constitution of india. tamil is also a member of the dravidian languages' family @xcite , which dates back over 4,500 years. however, tamil continues to be under-resourced @xcite . multiple nlp approaches have also been devised with a special focus on the indian context, this includes, indicbert and muril @xcite . the aim of this shared task was to build an automatic classification system which could classify whether the given text in tamil contains caste and migration hate speech or not. in this context, the current work presents a novel approach based on transformers to classify whether a text has caste and migration hate speech in tamil. ## related work with the recent boom in the number of internet users, many researchers worldwide have directed their efforts towards finding whether text online contains hate speech. the methodologies have evolved from the traditional machine learning models to the recent transformer-based approaches. in the work done by @xcite , a machine learning-based approach was proposed for the detection of abusive comments in the tamil language after exploring various deep learning and transformer techniques. they achieved a macro-f1 score of 0.35. the work also showed how the traditional machine learning models can outperform certain deep learning and transformer-based techniques when the dataset is not large enough and complex where the deep learning approaches excel. similarly, @xcite experimented with multiple traditional machine learning models and transfer learning approaches. they found that even though machine learning models performed well, transfer learning approaches outperformed them. among the different approaches that were tested, xlm-roberta (large) gave the highest accuracy. they attributed the reason to the fact that xlm-roberta (large) has more layers. hence, the number of trainable parameters is approximately three times the other alternatives. bhawal et al. (2021) also observed that transformer based approaches consistently performed better on both tamil and malayalam text. various models, including logistic regression, support vector machine, etc., were implemented on the tamil dataset, and the highest f1-score achieved was 0.82. a simple deep neural network was implemented on the same dataset, and it achieved an f1-score of 0.89 on the tamil text. the muril model, however, outperformed both of these techniques and achieved an f1-score of 0.91. an ensemble approach was adopted by @xcite . initially, many traditional machine learning models and transformer based models were implemented individually. however, it was found that the individual models had a high misclassifictaion rate. hence, in order to improve the accuracy, a combination of any three of the high scoring models were used for ensembling. two different approaches were considered to ensemble the models. the first approach involved averaging the outcomes of the models and the second approach involved using custom weights which were determined by grid search method for each member of the ensemble model. ## dataset description the dataset was provided by the organizers of the competition @xcite @xcite . the train and dev dataset is comprised of three fields, namely, id, text, and label. the test set comprised of only id and text. the labels for the dataset were 0 or 1, where 0 denoted absence and 1 denoted presence of no caste and migration hate speech, respectively. the distributions of the dev and train datasets have been shown in xlmr is an unsupervised model which has been trained on 100 different languages. this model was based on facebook's 2019 roberta @xcite model. this is a large multi-lingual model which was trained on 2.5tb of filtered data from commoncrawl. muril cased temp model is an nlp model that has been trained in the transformers library implemented using python. it was observed that the performance of the selected models used in the ensemble model suffered after pre-processing the text, where the pre-processing included the removal of numbers, special characters and emojis. hence, no preprocessing was done before training the models. the performances of the various models that were implemented have been shown in for performing tokenization, different tokenizers were used, which were specific to each model. the xlmrobertatokenizer was used for xlmr, the berttokenizer was used for mbert, and the autotokenizer was used for muril. for evaluating the label, as shown in figure 1 , the data was first passed through the three models individually. then the predictions of these three models were combined by using majority voting. the label with the highest frequency was finally predicted as the output of the ensemble model. ## results and discussion the ensemble model was designed by selecting the top three models with the best performance on the training data. the performance of all the models that were implemented on the training data has been shown in the base models and transformer models were both trained and tested. the transformer models consistently performed better than the base machine learning models. each of the transformer models was trained for 5 to 50 epochs each. the highest f1 score and its corresponding accuracy have been mentioned in the proposed ensembling technique achieved the highest f1 score of 0.82. this also shows that combining the various transformer-based techniques can lead to improved performance. ## conclusion and future work in this work, an ensembling technique was proposed to automatically detect whether a given text in tamil contains caste and migration hate speech for the caste and migration hate speech detection in tamil shared task by the lt-edi 2024 workshop at eacl 2024. the technique was based on transformer models and utilized transfer learning. the performance of the ensemble model can be further improved by taking the predictions from more transformer models or other traditional machine learning and deep learning techniques. also, taking a weighted vote of the models according to their performance on the training data can help give better results than majority voting, where each model is given equal importance irrespective of their performance relative to the other models.