CUET ’s_ W hite_ W alkers@ LT - EDI 2025: Transformer-Based Model for the Detection of Caste and Migration Hate Speech (2025)

Abstract: Hate speech on social media is an evolving problem, particularly in low-resource languages like Tamil, where traditional hate speech detection approaches remain under developed. In this work, we provide a focused solution for cast and migration-based hate speech detection using Tamil-BERT, a Tamil-specialized pre-trained transformer model. One of the key challenges in hate speech detection is the severe class imbalance in the dataset, with hate speech being the minority class. We solve this using focal loss, a loss function that gives more importance to harder-to-classify examples, improving the performance of the model in detecting minority classes. We train our model on a publicly available labeled dataset of Tamil text as hate and non-hate speech. Under strict evaluation, our approach achieves impressive results, outperforming baseline models by a considerable margin. The model achieves an F1 score of 0.8634 and good precision, recall, and accuracy, making it a robust solution for hate speech detection in Tamil. The results show that fine-tuning transformer-based models like Tamil-BERT, coupled with techniques like focal loss, can substantially improve performance in hate speech detection for low-resource languages. This work is a contribution to this growing amount of research and provides insights on how to tackle class imbalance for NLP tasks.

## introduction the sudden rise in social networking websites has come with the ever-mounting responsibility of monitoring and regulating harmful content, primarily hate speech. hate speech is defined as any form of speech that incites violence or is directed against individuals on the basis of race, religion, gender, or any other quality and hence is an emerging menace to the cyber world. though most of the hate speech detection research has focused on high-resource languages like english, hate speech detection in low-resource languages is not yet explored. tamil, a dravidian language with millions of speakers, is a typical example of a low-resource language where hate speech detection tools are nonexistent or inefficient. recognizing hate speech in tamil is particularly challenging due to its complex syntax, local dialects, and lack of massive annotated datasets. latest advances in natural language processing (nlp) have revealed that transformer-based architectures such as bert (bidirectional encoder representations from transformers) represent an effective remedy for text classification issues, such as hate speech recognition. but one of the persistent problems in training such models for hate speech detection is class imbalance. in the majority of datasets, hate speech instances are significantly less than non-hate speech instances, which may lead to model bias and bad generalization. to address these issues, we propose a novel tamil hate speech detection method based on tamil-bert, the language-specific variant of bert. we enhance the model's performance with the help of focal loss, which is a technique that allows the model to focus more on the minority class and therefore mitigate the effects of class imbalance. our contributions are as follows: the implementation details have been provided in the following github repository:- https://github.com/mizan116/lt-edi-ldk2025/hate speech . this study builds on the earlier studies of @xcite , who employed transformer models for hate speech detection across multiple languages, and (zhao, 2020), who demonstrated the efficacy of bert across low-resource languages. our study goes one step further by applying these approaches to the special case of tamil, demonstrating how fine-tuning transformer-based models and focal loss can be used to overcome the specific challenges posed by language-specific characteristics and dataset skew. ## related work hate speech detection has attracted significant interest as a consequence of the rapidly rising amount of toxic material on social media platforms. early approaches to coping with this problem used typical machine learning methods like support vector machines (svms) and naive bayes (waseem and hovy, 2016), where manually designed features like n-grams were common. however, these approaches often struggled to accommodate the complex linguistic and contextual nature of hate speech. with the arrival of deep learning, models such as convolutional neural networks (cnns) and recurrent neural networks (rnns) emerged, with better performance in classification tasks, especially in recognizing sequential dependencies @xcite . yet these models were not yet able to deal with long-range dependencies and deeper context in text, especially in very morphologically complex languages such as tamil. the recent progress in natural language processing (nlp) has been transformer-based models, such as bert @xcite , which has been the key driver for text classification. models based on bert have particularly excelled at hate speech detection since they can learn contextual information through attention mechanisms @xcite . other than this, experimentation has also demonstrated that multilingual bert models can be used for hate speech detection in low-resource languages such as hindi and tamil @xcite , thereby demonstrating that transformer models can also be fine-tuned for lowresource languages. class imbalance is the greatest challenge in hate speech classification. it has been addressed by focal loss @xcite that focuses on hardto-class samples more, thus improving the identification of minority classes like hate speech. focal loss has shown great promise for its application in nlp such as sentiment analysis and hate speech classification. in addition, research involving adversarial training @xcite and data augmentation strategies like paraphrasing @xcite has indicated improvements in model robustness and performance in detecting aggressive language on social media. this research is grounded on these breakthroughs and uses a tamil-specific bert model (tamil-bert) and focal loss to address class imbalance in hate speech detection in tamil. in a related shared task on dravidian languages, the authors of rahman et al. (2025) employed transformer models like xlm-r and muril, demonstrating high performance in abusive language detection. ## dataset we have applied the given dataset for shared task on caste and migration hate speech detection in lt-edi@ldk 2025 @xcite , which is focused entirely on caste and migration hate speech detection in tamil language (ponnusamy et al., 2024). the dataset is of type two classes: caste/migration-related hate speech and non-caste/migration-related hate speech. the dataset is divided into training, development, and test datasets. the training dataset contains 2,790 samples, and the validation and test datasets contain 598 samples each. we used the dataset of rajiakodi et al. (2024) for training, testing and evaluation our model. the distribution of the dataset is as follows: the information was collected from tamil social media, with real-world forms of caste and migration-based hate speech. as there exists class imbalance, whereby non-hate speech instances are greater than hate speech samples, we used focal loss when training the model to give more importance to the minority class. text was tokenized using a tamil-specific tokenizer and padded to 128 tokens. the evaluation metric for the task is macro f1-score to ensure balanced evaluation of both classes. ## methodology in this section, we provide an overview of the methodology and approaches utilized to build the system using the previous tamil-bert transformer model. methodology of our work is shown in figure 2 . ## results and analysis in this section, we evaluate the tamil-bert model for hate speech binary classification through four components : parameter setting, comparative analysis, performance metrics and error analysis. ## conclusion in this paper, we proposed a deep learning-based approach for caste and migration hate speech detection in tamil based on the pre-trained transformer model tamil-bert. using focal loss to handle class imbalance, our method attained a high f1score of 0.8634, which testifies to its efficiency in classifying hate speech and non-hate speech. although the model achieved promising performance, there is still room for further improvement, specifically in reducing overfitting and enhancing generalization. future work will explore added features, better loss functions, and hyperparameter tuning to continue to advance performance. our study contributes another entry to the growing corpus of work in low-resource hate speech detection and calls for responsible ai innovation to create secure digital spaces. t. y. lin, p. goyal, r. girshick, k. he, and p. dollár. 2017. focal loss for dense object detection. proceedings of iccv 2017. rahul ponnusamy, bharathi raja chakravarthi, prasanna kumar kumaresan, saranya rajiakodi, sathiyaraj thangasamy, and charmathi rajkumar. 2024. overview of shared task on caste/immigration hate speech detection. in proceedings of the fourth workshop on language technology for equality, diversity and inclusion, malta. european chapter of the association for computational linguistics. md mizanur rahman, srijita dhar, md mehedi hasan, and hasan murad. 2025. msm_cuet@dravidianlangtech 2025: xlm-bert and muril based transformer models for detection of abusive tamil and malayalam text targeting women on social media. in proceedings of the fifth workshop on speech, vision, and language technologies for dravidian languages, pages 243-247, acoma, the albuquerque convention center, albuquerque, new mexico. association for computational linguistics. saranya rajiakodi, bharathi raja chakravarthi, rahul ponnusamy, prasanna kumar kumaresan, sathiyaraj thangasamy, bhuvaneswari sivagnanam, and charmathi rajkumar. 2024. overview of shared task on caste and migration hate speech detection. in proceedings of the fourth workshop on language technology for equality, diversity and inclusion, malta. european chapter of the association for computational linguistics. saranya rajiakodi, bharathi raja chakravarthi, rahul ponnusamy, shunmuga priya mc, prasanna kumar kumaresan, sathiyaraj thangasamy, bhuvaneswari sivagnanam, balasubramanian palani, kogilavani shanmugavadivel, abirami murugappan, and charmathi rajkumar. 2025. findings of the shared task on caste and migration hate speech detection. in proceedings of the fifth workshop on language technology for equality, diversity and inclusion, italy. fifth conference on language, data and knowledge (ldk2025). h. t. ta. 2022. gan-bert: adversarial learning for aggressive text detection on social media. proceedings of the 2022 ieee/acm international conference on computer-aided design. z. waseem and d. hovy. 2016. hateful symbols or hateful people? predictive features for hate speech detection on twitter. proceedings of naacl-hlt 2016. q. zhang, k. zhao, and x. li. 2020. bert for hate speech detection: a comparative study. proceedings of the 2020 conference on empirical methods in natural language processing (emnlp). z. zhang, y. zhao, and y. lecun. 2018. deep learning for hate speech detection. proceedings of the inter-national conference on learning representations (iclr). d. zhao. 2020. multilingual bert for hate speech detection. journal name.