HATECAT - TR : A Hate Speech Span Detection and Categorization Dataset for T urkish (2025)

Abstract: Hate speech on social media in Turkey remains a critical issue, frequently targeting minority groups. Effective moderation requires not only detecting hateful posts but also identifying the specific hateful expressions within them. To address this, we introduce HATECAT-TR, a span-annotated dataset of Turkish tweets, containing 4465 hateful spans across 2981 posts, each directed at one of eight minority groups. Annotations were created using a semi-automated approach, combining GPT-4o-generated spans with human expert review to ensure accuracy. Each hateful span is categorized into one of five discourse types, enabling a fine-grained analysis of the nature and intent behind hateful content. We frame span detection as binary and multi-class token classification tasks and utilize the state-of-the-art language models to establish a baseline performance for the new dataset. Our findings highlight the challenges of detecting and categorizing implicit hate speech, particularly when spans are subtle and highly contextual. The source code is available at github.com/boun-tabi/hatecat-tr and HATECAT-TR can be shared by complying with the terms of X upon contacting the authors.

## introduction identifying hateful content is crucial for effective content moderation. while many automated systems detect hate speech, they often classify entire posts as hateful, lacking the granularity needed for informed decision-making. a recent study found that structured explanations improve moderator efficiency, reducing decision-making time by 7.4% @xcite . while explanations can take many forms, identifying and categorizing hateful spans provides an effective approach, helping moderators quickly assess which parts of a post are problematic and why. consider the following example where the hateful spans are highlighted: "@username bak yunan tohumu, feodalite denen bir şey vardı. korkudan surların arkasına saklanmıştınız." (english translation: "@username look, greek seed. there was something called feudalism. you hid behind the walls out of fear."). here, "yunan tohumu" (greek seed) falls under symbolization, as it reduces a group to a derogatory term, reinforcing negative stereotypes. meanwhile, "korkudan surların arkasına saklanmıştınız" (you hid behind the walls out of fear) is categorized as threat of enmity/war, as it implies historical cowardice and weakness, potentially inciting hostility. instead of flagging the entire post, detecting and categorizing these spans allows moderators to distinguish the nature of hate speech, prioritize severe cases, ensure consistency in content moderation, and improve overall transparency in enforcing platform guidelines. beyond post-level classification, span detection has been explored to provide a more granular understanding of hate speech across various languages and domains. studies have investigated span detection in english @xcite @xcite @xcite , korean @xcite , and vietnamese @xcite , with applications in news, social media platforms, and extremist forums. these efforts demonstrate the importance of identifying specific hateful expressions to improve interpretability and moderation strategies. hate speech detection in turkish has gained increasing attention, with various datasets and models developed to tackle hateful content @xcite @xcite . shared tasks such as siu2023 and hsd-2lang have contributed benchmark datasets, advancing the field with transformer-based models and en-semble approaches @xcite . to our knowledge, the only study that addresses span-level detection in turkish is jl-hate @xcite , which provides span-level annotations but lacks categorization and suffers from a limited dataset size @xcite . these gaps underscore the need for a comprehensive turkish dataset with detailed span annotations and categorical labels, enabling more explainable hate speech detection, and improved content moderation. to address these gaps, we introduce hatecat-tr, a span-annotated dataset of turkish tweets, each targeting specific ethnic, religious, and other minority groups. our dataset is the first turkish dataset for hate speech span detection and categorization, offering a larger, more structured resource for hate speech research in turkish. we construct the dataset using a semi-automated annotation process, combining gpt-4o generated spans with human expert review. hateful spans are labeled with five distinct categories. we approach span detection as both a binary token classification problem (identifying hateful spans) and a multi-class token classification problem (distinguishing between hate speech types). we establish baseline results using state-of-the-art language models. the comprehensive dataset with fine-grained annotations for hate speech spans and their categories as well as the developed baseline models will foster further research in this challenging but important area. this research can be useful in several areas. for instance, in sociolinguistics, the detailed data on hate speech can help researchers understand how language and culture impact the way hate is expressed online. the span-level annotations also make it possible to build more transparent content moderation tools that not only detect hate speech but also explain why something is flagged. this can lead to more effective moderation and even help guide users toward avoiding hateful language in the future. ## hatecat-tr initial dataset identifying hateful content spans is essential for combating hate speech. while tweet-level detection determines if a tweet contains hate speech, span detection pinpoints the specific segments responsible, providing more granular insights. to support this task, we curated a dataset of tweets annotated with hate speech spans. the dataset was sourced from a nongovernmental organization that has been working on tracking and analysis of hateful discourse in turkish media. it consists of tweets labeled for hate speech presence and the specific target groups affected including the armenian, greek, jewish, arab, immigrant/refugee, lgbti+, alevi, and kurdish groups. part of this dataset has been previously used in the hsd2lang hate speech detection challenge @xcite . in this study, we provide an extended version of this data set annotated with hate speech spans and their categories. automated span annotation to automate span annotation, we employed the gpt-4o model (gpt-4o-2024-08-06) @xcite , extracting hate speech spans within the tweets. we used a structured system message, provided in appendix b, to guide gpt-4o in identifying and labeling hate speech spans. each tweet was pre-assigned a single target during the prompting stage and processed individually using openai's official api. the model was queried with the following input format: "tweet id: [tweet_id], target: [tar-get], text: [content]". it was instructed to extract specific spans and classify them under one of the following five predefined categories. our categorization aligns with prior work that differentiates hate speech based on the nature and form of discourse. a comparable five-level scheme was introduced by @xcite and later adopted in the siu2023-nst hate speech detection contest @xcite , with only minor differences in category definitions. while our taxonomy closely mirrors the siu2023 scheme, it includes an additional category that explicitly captures exclusionary or discriminatory discourse. similar discourse-based classification is also used by @xcite , which proposed a fourlevel taxonomy for italian-language content to capture escalating harmful discourse. the automated span annotation approach was prone to errors, including minor modifications to the original tweet text and hallucinated spans that introduced unrelated content. to mitigate these issues, hallucinated spans were discarded, while minor textual differences were resolved using the edit distance metric. if a gpt-4o-annotated span differed by fewer than four characters from an existing span in the original tweet, the closest matching span in the tweet was retained. manual review additional refinement was required to ensure high-quality span annotations. we conducted a two-stage manual review of the generated spans, the details of which are provided in appendix c and d. two annotators manually reviewed the gpt-4o-generated spans, deleting, creating, or modifying spans to capture hate speech and assign appropriate categories accurately. annotations with substantial disagreements regarding the hatefulness of spans or an intersection over union (iou) score below 0.5 between the two annotators were re-evaluated by a third annotator. statistics after the re-evaluation process, the final dataset consists of 2981 tweets and 4465 hateful spans. of the 2981 tweets annotated by gpt-4o, 1530 were accepted without modification, while 1451 were revised by human annotators. these revisions involved adjusting span boundaries, correcting category labels, or removing hallucinated content. tables 1 and 4 present the dataset statistics for categories and target groups, respectively. although the annotation framework supported multilabel spans, this was rarely observed; most spans were annotated with a single discourse type. on average hateful spans are 30.30 characters long with a 21.69 standard deviation. for all categories, there are tweets containing multiple instances of hateful spans within the same tweet. the average number of spans per tweet is 1.50. cohen's kappa penalizes disagreements in category labels even when annotators select the same span, which may underestimate agreement in subjective or ambiguous cases. to account for near matches in span annotations, we computed a relaxed version of cohen's kappa using a span-level evaluation protocol. in this approach, two annotated spans are considered a match if their intersection over union (iou) exceeds a specified threshold. this relaxed metric tolerates minor discrepancies in span boundaries, focusing on whether annotators identified roughly the same region. as a result, the agreement score increases from 0.57 (standard kappa) to 0.65 at 90% iou, and further to 0.76 at 50% iou, indicating that annotators often agree on the approximate region even if their exact span selections differ. ## experiments we frame span detection and categorization as binary and multi-class token classification tasks, where each token is assigned a label indicating whether it is part of a hateful span and category. due to the complexity of detecting hate speech and distinguishing between overlapping categories, we model categorization as a multiclass task by assigning the most salient category to each span. this reflects the dominant pattern of single-label spans in the data and simplifies the learning setup. in the hate speech span detection task, bio tagging is applied for encoder-based models (e.g., bert variants). the o label marks tokens outside of hateful spans, b-hate denotes the beginning of a hateful span, and i-hate is used for tokens continuing the span. for the hate speech categorization task, we adopt an io tagging scheme to simplify the process for encoder-based models. the label o is assigned to tokens outside any hateful span, while tokens within a span receive an iprefix corresponding to the specific hate speech category (e.g., i-threat). for decoder-only llms, we do not follow a token classification framework and therefore do not use bio or io tagging. instead, we used the same natural language prompt format as in our gpt-4o setup, shown in listing 1 in appendix b. ## results we establish baseline results using two state-of-theart turkish language models, berturk @xcite with finetuning and turkishbertweet @xcite , as well as multilingual models from the llama @xcite and gemma families @xcite . models were evaluated using span-level f1-scores. as shown in attack; murder; or harm" category, with berturk reaching 0.44 and turkishbertweet achieving 0.35. this suggests that explicit threats and violent language are easier to classify compared to more nuanced hate speech types. in contrast, both models struggle with "exclusive/discriminatory discourse" and "exaggeration; generalization; attribution; distortion". these results indicate that more implicit and context-dependent hate speech categories are difficult to detect and categorize. similar trends are observed across multilingual models. the confusion matrix in figure 1 highlights key misclassification patterns: (1) poor recognition of exclusive/discriminatory discourse, particularly in ethnic profiling statements such as "bütün türkiyeli siyasilere dna ırk testi yapılarak açıklanmalı, kim yahudi geni taşıyor, kim yunan geni taşıyor, kim ermeni geni taşıyor." ("a dna ancestry test should be conducted on all politicians in turkey, and the results should be publicly announced-who carries jewish, greek, or armenian genes."). (2) failures in detecting exaggeration and generalization, particularly in demographic-based fear-mongering, such as "aynen arapça ögren de, 5 sene içinde ülkenin yarısı arap olunca araplara da anlatırsın vatana sahip çıkmayı." ("learn arabic, so in five years, when half the country is arab, you can explain to them how to protect the homeland."). (3) false negatives in threat-related speech, where phrases like "yunan'ı denize döktük" ("we threw the greeks into the sea") were overlooked due to historical or patriotic framing. (4) missed swearing and dehumanization, particularly obfuscated insults ("pzvnk" for "pezevenk", "pimp") and indirect comparisons such as "bunların bir rum, bir ermeni kadar bile insanlıgı yok" ("they don't have even as much humanity as a greek or an armenian"). these findings suggest that the model requires improved handling of implicit biases, coded language, and contextual nuance for more accurate classification. ## conclusion we introduce hatecat-tr, a turkish tweet dataset annotated with hate speech spans and their categories. the dataset was created using automated gpt-4o annotations, followed by manual expert review, ensuring high-quality labeled spans across five hate speech categories. our evaluation demonstrates berturk's better performance over turkishbertweet for the span detection and categorization tasks. both models perform best in detecting explicit threats, particularly in "threat of enmity; war; attack; murder; or harm", but struggle with more implicit categories like "exclusive/discriminatory discourse" and "exaggeration; generalization; attribution; distortion." multilingual models, including variants of llama and gemma, perform noticeably worse than the turkish models across all categories, highlighting the importance of language-specific pretraining for nuanced hate speech detection. notably, the categorization task is challenging not only for the models but also for human annotators, who showed lower agreement in labeling specific hate speech types. error analysis reveals key challenges, including false negatives in historically or politically framed speech, and difficulties in detecting subtle dehumanization and coded language. these findings highlight the need for improved models capable of handling implicit bias and contextual nuances. we believe that the dataset and baseline models presented in this study will form the basis for further research on fine-grained hate speech detection.