kubapok@ LT - EDI 2024: Evaluating Transformer Models for Hate Speech Detection in T amil (2024)

Abstract: We describe the second-place submission for the shared task organized at the Fourth Workshop on Language Technology for Equality, Diversity, and Inclusion (LT-EDI-2024). The task focuses on detecting caste/migration hate speech in Tamil. The included texts involve the Tamil language in both Tamil script and transliterated into Latin script, with some texts also in English. Considering different scripts, we examined the performance of 12 transformer language models on the dev set. Our analysis revealed that for the whole dataset, the model google/muril-large-cased performs the best. We used an ensemble of several models for the final challenge submission, achieving 0.81 for the test dataset.

## introduction this paper deals with hate speech detection in the tamil language, which is an official language in sri lanka and singapore. it is also the official language of the indian state of tamil nadu and the union territory of puducherry. the language is spoken by groups of citizens of malaysia, mauritius, fiji, and south africa. the current number of tamil speakers is estimated at 75 million. the tamil language belongs to the family of 24 dravidian languages, spoken by approximately 250 million people. the tamil alphabet consists of 246 characters: 12 vowels, 18 consonants, and 216 vowel-consonant combinations. being spoken in india, a country with a caste-based social system, the tamil language may suffer from hate speech referring not only to religion, ethnicity, gender, sexual orientation, or political affiliation, but also to caste and migration. in this paper, we describe a submission to caste/migration hate speech detection task organized at lt-edi-2024 @xcite . our approach, which relied on an ensemble of several models, achieved second place in the competition, with a 0.81 f1-score. besides the research related strictly to the contest, we examine the performance of 12 up-to-date models that are most suitable for this task. we evaluate each model's performance separately on tamil, latin, and combined scripts, finding that the model's performance is different based on the script. ## related work a contest on hate speech detection in the dravidian languages, called hasoc 2021, was organized in 2021 @xcite . the data were collected from youtube comments and posts. the contest consisted of two tasks differing in the nature of the data: the first task was based on tamil only, while the second task was based on a data set combining tamil and malayalam. the winning solution for task 1 achieved a 0.86 f1-score. the winning solution in the tamil track for task 2 achieved a 0.68 f1-score. the hasoc 2021 shared task gave rise to a number of papers, such as @xcite ; @xcite and @xcite . the papers report submissions with f1-scores ranging from 0.66 to 0.84. ## caste/migration hate speech detection challenge the caste/migration hate speech detection task is a part of the fourth workshop on language technology for equality, diversity, and inclusion (lt-edi-2024) @xcite . the main objective of the challenge is to develop a text classifier in the tamil language that can determine whether a given social media text contains hate speech related to caste or migration. the competition's evaluation metric is the macro average f1-score, and the participants are provided with training (train) and development (dev) datasets. ## dataset analysis we examined the train and dev datasets and discovered that the texts could be classified into three primary categories: ## conclusions we conducted an evaluation of several english, multilingual, and hindi encoder language models for a classification task in the tamil language. this task was a part of the fourth workshop on language technology for equality, diversity, and inclusion. our post-competition study revealed that the most effective model was google/muril-largecased. all types of language models performed well on the latin script portion of the dataset, which may result from the fact that some of the texts were in the english language. our approach, which relied on an ensemble of selected models, achieved second place in the competition.