VRLL ab at HSD -2 L ang 2024: T urkish Hate Speech Detection Online with T urkish BERT weet (2024)

Abstract: Social media platforms like Twitter - recently rebranded as X - produce nearly half a billion tweets daily and host a significant number of users that can be affected by content that are not properly moderated. In this work, we present an approach that ranked third at the HSD-2Lang 2024 competition’s subtask-A along with additional methodology developed for this task and evaluation of different approaches. We utilize three different models and the best performing approach use publicly-available TurkishBERTweet model with low-rank adaptation (LoRA) for fine tuning. We also experiment with another publicly available model and a novel methodology to ensemble different hand-crafted features and outcomes of different models. Finally, we report the experimental results, competition scores, and discussion to improve this effort further.

## introduction despite the significant opportunities presented with the use of social media, these platforms are shifting towards more hostile environments, especially for marginalized groups. social networks have been used to access information efficiently @xcite , participate important societal events @xcite , and discuss political issues online @xcite @xcite . the increasing popularity of social networks and the opportunities presented to reach millions of individuals simultaneously made these platforms vulnerable to manipulation of discourse by bad actors who utilize automated accounts @xcite , spread disinformation @xcite , and coordinate targeted attacks @xcite . these targeted attacks can be coordinated or organic, and mostly, the target is minority and vulnerable groups. to prevent vulnerable groups and improve their experience in the online sphere, researchers develop systems to automatically identify these activities, and platforms build systems to moderate content and accounts. hate speech detection is a task to identify hateful content aimed towards groups such as refugees and individuals with certain beliefs or ethnicities @xcite @xcite . in this work, we demonstrate our approach as part of the hsd-2lang 2024 challenge to detect hate speech from textual information presented in social media posts. ## data this challenge is organized in collaboration with the hrant dink foundation for their ongoing project about "media watch on hate speech." collaborative efforts of computational and social scientists defined hate speech on social media and carried out a detailed procedure to annotate posts around specific topics and keywords. the provided dataset in this competition contains 9,140 tweets in the context of israel-palestine and turkish-greek conflicts and content produced against refugees and immigration @xcite . we preprocessed the dataset by removing samples with inconsistent ground truth information (exact text with different labels), and we applied deduplication, resulting in 8,805 tweets. figure 1 shows word and character length distributions. when the ground-truth labels are considered, we measure that 30.5% of the dataset contains hate speech, suggesting an imbalance between the two classes. since the dataset only contains the textual information presented in each tweet, we further processed them to take into account platform-specific features. removal of hyperlinks and mentions of other accounts in the tweets. this information could be valuable if we had a chance to process realtime data by scraping external web content or using profile information of accounts from twitter's api since these fields are omitted in the dataset. since we do not incorporate them into our analysis, we omit them from the dataset. preprocessing pipeline for turkishbertweet model. we consider different special tags for twitter-specific entities and translated the unicode characters of emojis to words describing the meaning using the preprocessor created for the turkish-bertweet project @xcite . ## methodologies in this challenge, we built different approaches. we considered not only the textual data to fine-tune models but also incorporated additional signals obtained from text and blacklisted word dictionaries. here, we present the language models used as the foundation and additional features we extracted to improve the model's performance. for the competition, we submitted the model with the best public leaderboard score; however, one of our approaches achieved an even higher score in the private evaluation. we presented all approaches and their respective performances in the results section. turkishbertweet foot_0 is a new language model that was specifically trained on nearly 894m turkish tweets and the model offers a special tokenizer that takes social media entities such as hashtags and emojis into account. this model utilized lora @xcite , which is a novel way of fine-tuning llms in an efficient way, and recent research reports state-of-the-art performance and generalizability capabilities @xcite . berturk foot_1 is a pre-trained model that utilizes large-scale corpus from various sources. it is a wellknown model among the turkish nlp community @xcite . ensemble of models (eom) approach combines outputs of aforementioned hate speech models along with custom features extracted for this task. these additional features consist of i) logits scores retrieved from an emotion classifier based on a bertbase model fine-tuned model for emotion analysis, foot_2 ii) logit scores of a sentiment classifier using turk-ishbertweet sentiment analysis model, iii) collection of turkish blacked-list words foot_3 used for token level features such as binary exact match feature, levenshtein distance, hashtag exact match, and hashtag levenshtein distance. these features are concatenated, resulting in 16 features for the ran-domforest classifier with 100 estimators trained to optimize gini-impurity. since the outputs of ensemble models for imbalanced datasets can be biased, we calibrated the outputs of the model using platt's scaling for interpreting output scores as probabilities @xcite . ## results this section presents the experimental evaluation of approaches we tested within the dataset using stratified 5-fold cross-validation. we also report the performance of models we submitted to challenge for comparison. as considering the performance differences between public and private leaderboards, the eom demonstrates less variability than the other two approaches. even though it is not our best-performing model in both settings, we may consider it for our research projects since both cross-validated scores point to better performance, and the leaderboard score differences are negligible and can be due to we also conduct an error analysis to identify misclassifications that our model is making. this effort can reveal additional features we can implement and issues observed in the labeled dataset. ## discussion in the provided dataset, we noticed tweets written in languages other than turkish, such as arabic and hebrew. this could be an artifact of the data collection process, and one can consider i) language-level features, ii) filtering them, or iii) obtaining representation from llms. furthermore, a study about the annotator's influence on the annotation quality for hatespeech datasets shows that the expertise of annotators positively influences the data quality @xcite . considering the annotators' influence, applying impurity analysis by randomly or strategically changing the annotations and monitoring the hate speech system's performance could be a good practice. moreover, in this competition, we are only considering the text data to detect the existence of hate speech. infusing the account information into these systems could help them be more accurate and reliable, such as the number of followers, number of followings, account creation date, etc. another approach for improving the performance of the systems is to expose pre-trained models with hateful content by further masked-language modeling on the hate speech dataset, like @xcite presented in their recent work and improved the system's performance. multilingual models could also be utilized for this challenge since turkish is a low-resource language, and the model can benefit from the other languages' hate speech datasets to infuse the broader knowledge of hate speech and then obtain a better performance @xcite . recently, commercial models like chatgpt have been used in various challenges. @xcite suggest that the chatgpt demonstrates high accuracy and can be considered an alternative to human annotators in detecting implicit hate speech @xcite . other work also investigated the performance of llms for hate-speech or offensive language detection tasks in english @xcite , portuguese @xcite , and turkish (çam and özgür, 2023). however, we want to raise a concern about the adversarial use of these models to attack vulnerable groups and bypass the detection systems. additional information about accounts, network structure, and temporal activities should be incorporated into detection systems to address the mentioned risk. ## conclusion in this challenge, the collective effort of research teams points to best practices and demonstrates the capabilities of the state-of-the-art models. here, we demonstrated different approaches and their respective performances in detecting online hate speech toward three different groups. we obtained the third rank in the final leaderboard of the competition with the turkishbert+lora model. we hope language models like turkishber-tweet will be used in different downstream tasks on turkish social media. research efforts especially need to assess the online participation of minority groups. there is a significant need for publicly available models since the quality of content moderation and use of automated accounts on platforms like x is questionable after the acquisition false positive model predicts as hs labeled no hs • #katil i̇srail [url] • hükümet cumhurbaşkanı erdogan şerefsiz suriyeliler yagma sizler şu an hem suç hem cinayet işliyorsunuz. i̇nsanlar twitter ı kullanmak için vpn kullanıyor ve vpn mobil cihazların şarj süresini oldukça azaltıyor. tarihe böyle geçeceksiniz. • onursuz ırkıcılar kökünüz kurusun lanet olsun size evet kürdüz türküz ermeniyiz afgan'ız arabız ırkcı itler geberin lan bu ülke hepimizin # #hepimizkürdüz • i̇nsanlık yapıp ülkeye alıyorsun hainlik,bu zor günde yagmacılık yapıyorlar.bazı şeref yoksunu suriyeliler yüzünden masum olan insanlar arada kaynıyor.açıkçası #ülkemdemülteciistemiyorum ! allah herkesin yardımcısı olsun yardıma ihtiyacı olana koşulsun ama ülkemi terketsinler. [url] false negative model predicts no hs labeled as hs • #uelkemdemueltec i̇i̇ stem i̇yorum [url] • heryerde bilim uzmanı ve yer bilimci prof hocalar. gerçegi açıklıyor. sonra unutulup , açgözlü, rantçı,yagmacı yöneticiler soyguna devam eder. 3 yıllık bina yıkılmış, 3 yıl. #deprem #earthquake #yagmacılar. • sayıları 8 milyon olan suriyeli, afgan, irak ne varsa çok acil ülkelerine geri gönderilmeli. *güvenlik tehdidi oluşturuyorlar. *işsizlik sorunu oluşturuyorlar. bill gates #billgates #sedatpeker10 open source models: turkishbertweet model used in this challenge is available online at the hug-gingface platform. https://huggingface . co/vrllab/turkishbertweet