HODIAT : A Dataset for Detecting Homotransphobic Hate Speech in I talian with Aggressiveness and Target Annotation (2025)

Abstract: The escalating spread of homophobic and transphobic rhetoric in both online and offline spaces has become a growing global concern, with Italy standing out as one of the countries where acts of violence against LGBTQIA+ individuals persist and increase year after year. This short paper study analyzes hateful language against LGBTQIA+ individuals in Italian using novel annotation labels for aggressiveness and target. We assess a range of multilingual and Italian language models on this newannotation layers across zero-shot, few-shot, and fine-tuning settings. The results reveal significant performance gaps across models and settings, highlighting the limitations of zero- and few-shot approaches and the importance of fine-tuning on labelled data, when available, to achieve high prediction performance.

## introduction the rise of homophobic and transphobic discourse in online and offline spaces has been recently escalating, posing a global concern. reports from organizations such as ilga-europe and human rights watch document a «worrying increase in anti-lgbtqia+ hate crimes», fueled by a strong growing opposition to the so-called "gender ideology". in several european countries, policies restricting lgbtqia+ rights have gained traction. italy is among the countries where the lgbtqia+ community faces institutional opposition, with the government taking steps against homoparentality further reinforcing queer discrimination. furthermore, italy still lacks a national law criminalizing homo-transphobic hate crimes. the proposed ddl zan law, which aimed to extend anti-discrimination 1 examples have been obfuscated with a python package for obfuscating profanities: prof by @xcite . the consequences are tangible both offline and online, where hate speech is often perpetrated, underscoring the urgent need for effective computational tools to detect and mitigate homotransphobic discourse. despite the critical need, computational research on homophobia and transphobia detection in italian is underdeveloped. in a previous research, we organized the hodi shared task @xcite , providing the first benchmark dataset for homotransphobia detection in italian, while other research, like queereotypes @xcite , focused on lgbtqia+ stereotypes, addressing different aspects of the same issue. this study builds on previous work that highlights the importance of aggressiveness and target identification in understanding hate speech dynamics and their potential escalation into real-world violence @xcite . our contributions are the following: ## related work despite growing interest in hate speech detection, research specifically addressing lgbtqia+ communities remains limited. developing a hate speech model that effectively covers different targets and languages has proven challenging @xcite . indeed, @xcite conduct a cross-lingual study on homotransphobia in twitter discourse, demonstrating that while homotransphobia is a global issue, it manifests through distinct cultural expressions, shaped by factors such as misinformation, cultural prejudices, demographic bias and religious beliefs. further exploring the multilingual nature of hate speech, @xcite introduce a dataset for homotransphobia detection in telugu, kannada, and gujarati. @xcite examine the challenges of multilingual lgbtqia+ hate speech detection, particularly how translation affects detection across english, italian, chinese, and english-tamil code-mixed text. their findings indicate that fine-tuning consistently improves model performance across languages, whereas translation has mixed effects on detection accuracy. moreover, shared tasks have played a crucial role in advancing hate speech detection for the lgbtqia+ community. the lt-edi@eacl series focuses on identifying homophobia, transphobia, and non-anti-lgbtqia+ content across tamil, english, and code-mixed english-tamil @xcite @xcite . homo-mex is dedicated to the detection of phobic messages towards the mexican spanish speaking lgbtqia+ community @xcite . for italian, the hodi shared task @xcite represents the first initiative focused on homotransphobia detection. while explicit forms of hate speech have been extensively studied, implicit and subtle forms, such as sarcasm and coded language, have often been overlooked. recent efforts have focused on detecting implicit forms of hate speech @xcite @xcite . in a follow-up study, damo et al. (2024a) also demonstrated that including explanations while detecting implicit hateful messages enhances models' performance. however, two critical components of hate speech, i.e. detecting aggressiveness and identifying targets of abusive content, are areas that remain under-explored, despite some attention in previous work @xcite @xcite . our work builds upon these prior studies by enhancing the hodi dataset for detecting homotransphobia in italian twitter discourse, with a particular focus on aggression and target identification. our study contributes to the growing body of research on online abuse and discrimination against the lgbtqia+ community and to the creation of resources for languages other than english. ## dataset hodiat builds on our previous work, the hodi dataset @xcite , which includes 6,000 italian tweets binary-labeled (0/1) for hate toward lgbtqia+ people and annotated with hateful text spans (rationales). 3 this work introduces two novel annotation layers addressing factors that are crucial yet often overlooked in existing research: aggressiveness and target type. three independent annotators, all members of the lgbtqia+ community, were recruited to enrich the data (for details, see positionality statement in appendix a). they annotated whether the homophobic content targets an individual or the lgbtqia+ community as a whole and they were asked to provide a binary label (aggressive vs. nonaggressive) to capture the intensity of hate speech. the annotation guidelines, including the working definitions of "hate speech", "aggressiveness" and "target", were adapted from established practices in related tasks @xcite . the annotation guidelines are available in the same repository of the dataset. 4 these additional layers allow us to (1) differentiate between hate speech aimed at discrediting or isolating individuals versus that which undermines or marginalizes the entire community and ( 2 ) gain more insights into the potential harm and urgency associated with the textual content. below are some examples of possible annotations: • type of target. it specifies whether the hateful comment addresses a generic group of lgbtqia+ people or whether it is directed towards a specific individual. the two possible labels are group and individual. x che paese di m*rda, è più importante dar la libertà ai fr*ci di sposarsi che dare il diritto al lavoro alla gente sia essa fr*cia etera o aliena. pazzesco what a sh*t country, it's more important to give freedom to f*ggots to marry than to give the right to work to people whether they are f*ggots, hetero, or alien. insane [group] x a [pers] c*lattone di merda ......ti va bene che non ti incontro per strada ,ti sputerei in faccia m*rdaaaa a [pers] you sh*t f*ggot......you're lucky i don't meet you on the street, i'd spit in your face, piece of sh*t. [individual] • aggressiveness. the comment contains a message spreading, inciting or promoting violence against lgbtqia+ people, or a message legitimizing an aggressive action or behaviour that intimidates them. this is a binary category, the possible labels are aggressive and non-aggressive. x il prossimo r*cchione che fa sta roba lo tiro sotto con l'auto @user the next f*ggot who does this, i'll run them over with the car. [aggressive] x sembra un tr*vione di quelli potenti looks like a tr*nny, a huge one ## experiments our experimental setup combines various large language models (llms), including gpt-4omini (openai, 2024), llama (meta, 2024), llamantino @xcite , minerva (sapienzanlp, 2024), and qwen @xcite , along with transformer-based models such as modern bert @xcite , xlm-t @xcite , and hate-ita @xcite . we apply the llms in zero-shot, few-shot (5 examples), and fine-tuned settings. finally, we predict the three labels all at once in zero-shot and fewshot settings. appendix b contains further details about the experimental setting. ## results our evaluation reveals several key trends in the performance of different models in the hateful, aggressive, and target classification tasks. fine-tuning consistently outperforms both zero-shot and few-shot approaches, particularly for hatefulness and aggressiveness tasks. xlm-t achieves the best overall performance and all other models also show improvements (e.g. llama3 demonstrates a 0.23 increase in hatefulness and a substantial 0.82 rise in aggressiveness). the improvement in target classification is less pronounced across all models. few-shot learning offers limited advantages over zero-shot methods. in some cases, few-shot performance is even lower than zero-shot. this suggests that few-shot prompting does not effectively leverage in-context learning and may introduce noise. minerva shows a strong but imbalanced performance, achieving the highest zero-shot f1 for aggressiveness (0.84), outperforming all other models (scoring below 0.30). however, minerva's performance suffers from severe class imbalance, particularly for aggressiveness (where scores vary widely between labels) and to a lesser extent for target classification. this makes its predictions less reliable and inconsistent, especially in hatefulness classification, where its zero-shot score is only 0.40. in contrast, xlm-t maintains a similar performance across labels. this contributes to its robust fine-tuning performance across all tasks, making it the top-performing model, particularly in target classification and aggressiveness. similarly, modernbert shows solid fine-tuning results, confirming that transformer-based models benefit greatly from fine-tuning, especially in tasks like target classification. the hate-ita results emphasize the challenges of transferring hate speech tasks across languages and targets. hate-ita is an xlm-t model trained on english and available italian datasets (focused on hate speech against immigrants and women). fine-tuning the model solely on hodi (i.e., xlm-t) outperforms training on these diverse datasets. on the other hand, llamantino, which shows competitive zero-shot and few-shot results, struggles when fine-tuned. its hatefulness score decreases from 0.57 (zero-shot) to 0.50 (fine-tuning), diverging from the general trend where fine-tuning typically leads to improved performance. this suggests that fine-tuning may not always be the most effective strategy for every model, especially when dealing with certain tasks. target classification proved to be the most fluctuating task, with varying performances between models. minerva achieves the highest llms finetuned score (0.77), and xlm-t outperforms all models with a score of 0.85. other models showed fluctuating results, suggesting that target classification may require additional optimization or taskspecific tuning to achieve consistent performance. a notable finding is that training the 3 labels jointly (hatefulness, aggressiveness, and target) tends to worsen performance compared to training them separately, as shown in ## conclusion this paper introduces an enhanced dataset, ho-diat, with additional aggressiveness and target annotations, designed to refine the granularity of homotransphobia detection in italian, extending the hodi dataset @xcite . our findings based on testing several state-of-the-art llms and encoder-based models reinforce the importance of fine-tuning when labelled data is available, as it consistently outperforms both zero-shot and fewshot learning. however, even with fine-tuned models, class imbalance remains a challenge, particularly in tasks like aggressiveness and target classification. this highlights the need for further research on addressing bias and improving the stability of model performance across different classes. moreover, minerva's strong but imbalanced performance, along with xlm-t's consistent performance across all tasks, highlights the importance of considering each model's strengths and weaknesses when selecting the best approach for a specific task.