Overview of the Hate Speech Detection in T urkish and A rabic Tweets ( HSD -2 L ang) Shared Task at CASE 2024 (2024)

Abstract: This paper offers an overview of Hate Speech Detection in Turkish and Arabic Tweets (HSD-2Lang) Shared Task at CASE workshop to be held jointly with EACL 2024. The task was divided into two subtasks: Subtask A, targeting hate speech detection in various Turkish contexts, and Subtask B, addressing hate speech detection in Arabic with limited data. The shared task attracted significant attention with 33 teams that registered and 10 teams that participated in at least one task. In this paper, we provide the details of the tasks and the approaches adopted by the participant along with an analysis of the results obtained from this shared task.

## introduction hate speech, which targets groups based on characteristics such as ethnicity, nationality, religion, colour, gender, and sexual orientation, is a significant problem on social media platforms. the automated detection of such content is crucial for efficient content moderation and the mitigation of societal harm. moreover, it can also be instrumental in socio-political event analysis. the effectiveness of current hate speech detection models is often hampered by issues such as limited data and lack of generalizability. following the siu2023-nst competition @xcite , which was organized to benchmark progress in turkish hate speech detection and classification, we present a new shared task, hate speech detection in turkish and arabic tweets (hsd-2lang) shared task, in conjunction with the 7th workshop on challenges and applications of automated extraction of socio-political events from text @xcite . this shared task focuses on tackling the challenge of identifying hate speech in tweets in turkish and arabic languages. ## tasks the shared task involves the development of models for hate speech detection in social media, with a specific focus on turkish and arabic languages. the task is divided into two distinct subtasks: a) hate speech detection in turkish across various contexts (subtask a), b) hate speech detection with limited data in arabic (subtask b). both subtasks are formulated as binary classification problems where the objective is to determine whether individual tweets are hateful or nonhateful. ## system descriptions the hsd-2lang shared task attracted participation from 33 teams associated with various universities and organizations. this task involved developing systems for specific subtasks, detailed in the following subsections. ## competition results for both tasks, the performance of models is evaluated using the test samples of the corresponding dataset. the test samples are randomly divided into public and private samples. public samples make up 20% of the test samples and are used by partici-pants for validation during the test phase. private samples were used to evaluate the model's performance after the test phase has concluded and to generate the final leaderboard of the shared task. ## results for subtask a detectiveredasers took first place on the private leaderboard and second on the public leaderboard, with f1 scores of 0.69645 and 0.73362, respectively. this system, leveraging a soft-ensemble of convberturk models, outperformed the competing systems in recall and f1 scores on both leaderboards. rebert and vrllab, although employing a similar approach using turkishber-tweet with lora fine-tuning, showed a noticeable difference in their f1 scores (rebert: 0.73786 public, 0.68886 private; vrllab: 0.70642 public, 0.66432 private). notably, rebert achieved significantly higher precision scores compared to vrllab. this variation underscores the critical role of hyperparameter tuning in optimizing model performance. ## results for subtask b in subtask b, rebert and team curie showed very similar performances in the public leaderboard, while their private leaderboard scores varied. rebert took the first-place with an f1 score of 0.683532, while team curie came second place with an f1 score of 0.65854. interestingly, despite using the same arabic bert model trained on tweets as the top-ranking system, team uriel lagged behind the second place system (team curie) that used an arabic bert model not specifically pre-trained on tweets. this result highlights the importance of hyperparameter tuning, especially in scenarios with limited data. the winner of subtask a, detectiveredasers @xcite , took 4th place in subtask b, with an f1 score of 0.6 on the private dataset. they reported that they did not use cross-lingual augmentation in this task (i.e. translating turkish tweets into arabic) as it degraded the performance, even though this strategy had worked well in subtask a. this may be due to the relative numbers of the two datasets. ## conclusion this paper presented an overview of the hsd-2lang shared task that was organized to benchmark models for hate speech detection on social media platforms, in turkish and arabic languages. the task consisted of two distinct subtasks, each addressing unique challenges: subtask a focused on hate speech detection in various contexts in turkish, and subtask b addressed the challenge of hate speech detection in arabic under limited data conditions. the results from these subtasks provided valuable insights into the efficacy of different models and strategies. all participating systems used bert-based models, demonstrating their effectiveness. on the other hand, systems using the same model achieved noticeably different results due to hyperparameter choices. in subtask a, the top-performing system employed a soft-ensemble of convberturk models, achieving an f1 score of 0.69645 on the private test set. this shows the effectiveness of ensemble methods in tackling hate speech detection across various contexts. moreover, the noticeable performance differences among systems using the same method, specifically turkishbertweet with lora fine-tuning, underscore the importance of hyperparameter tuning in improving model performance. subtask b yielded comparable results due to similar model implementations with minor variations. the top system achieved an f1 score of 0.68354 on the private test set. the small performance difference between the two tasks are interesting, considering that the turkish dataset had three times more data compared to the arabic set, showing the effectiveness of pretrained models. on the other hand, the difference between the performances of rebert and team uriel, even though they used the same arabic bert model, further highlights the importance of hyperparameter tuning, especially in scenarios with limited data availability.