Detecting Bias and Intersectional Bias in I talian Word Embeddings and Language Models (2025)

Abstract: Bias in Natural Language Processing (NLP) applications has become a critical issue, with many methods developed to measure and mitigate bias in word embeddings and language models. However, most approaches focus on single categories such as gender or ethnicity, neglecting the intersectionality of biases, particularly in non-English languages. This paper addresses these gaps by studying both single-category and intersectional biases in Italian word embeddings and language models. We extend existing bias metrics to Italian, introducing GG-FISE, a novel method for detecting intersectional bias while accounting for grammatical gender. We also adapt the CrowS-Pairs dataset and bias metric to Italian. Through a series of experiments using WEAT, SEAT, and LPBS tests, we identify significant biases along gender and ethnic lines, with particular attention to biases against Romanian and South Asian populations. Our results highlight the need for culturally adapted methods to detect and address biases in multilingual and intersectional contexts.

## introduction bias in natural language processing (nlp) applications has become a widespread problem. various methods have been developed to measure and partially mitigate bias in word embeddings, e.g., @xcite ; @xcite , and language models, e.g., @xcite ; @xcite . however, bias appears across many dimensions and contexts. therefore, the majority of existing approaches address only one type of bias at a time (e.g., gender or ethnicity). only and handful of studies, e.g., @xcite ; @xcite , explore intersectional bias, especially in languages other than english. additional challenges arise when adapting existing bias metrics to gendered languages @xcite . in this paper, we extend the state-of-the-art by providing insights into both single category and intersectional biases in italian word embeddings and language models. we leverage known metrics and culturally adapt them to the italian context, in close collaboration with an interdisciplinary team and native speakers. in particular, we introduce gg-fise, a method for studying intersectional bias based on @xcite that partially corrects for measurement errors resulting from grammatical gender. bias statement. this work focuses on diversity bias, defined as the unfair positive or negative treatment of individuals based on protected grounds foot_0 , with particular attention to intersectional categories. the technical methods presented here aim to quantify the extent to which potentially harmful social stereotypes are intrinsically encoded within word embeddings and language models. in other words, a model is understood to be biased if it encodes harmful social stereotypes. this connects to diversity bias because the use of biased models could lead to harmful outcomes in downstream tasks, where our main research concerns are gender, ethnic and/or intersectional bias in ai-assisted hiring decisions. for example, an nlp hiring system that computes the similarity between job ads and candidate applications using a model encoding stereotypical occupational associations could lead to unfair outcomes. ## discussion single category bias in the case of static word embeddings, weat tests provided ample evidence that italian fasttext embeddings encode stereotypical biases regarding gender roles and societal expectations. men are more associated with career, while women are more associated with family. although the wordlists containing the academic fields of study with the highest gender imbalances were compiled in a swiss context, indicators for the same biases were also detected, associating fields such as engineering and computer science to men, and pedagogy and psychology to women. in 2021, the italian national institute of statistics (istat) reported that romanians make up the largest immigrant group in italy, nearly a quarter of all foreign residents. together, indian and bangladeshi residents make up 6.5% of the immigrant population, making south asia the most represented region of the asian continent. of the members of each minority with work experience in a foreign country, the majority of that experience was in low-skilled work. 9 negative stereotypes against the romanian population in italy are documented in existing work, e.g., @xcite . prejudices are exacerbated by the association between romanians and the roma people, who face prejudice and marginalization across europe @xcite . our findings provide evidence that all of these biases are encoded within language models, demonstrating that fasttext embeddings associate low-skilled labor and unpleasantness to both groups. moreover, the results provide good evidence (@xmath0.07) that the embeddings also encode harmful prejudices against queer and transgender identities. weat testing also indicated gender and ethnic biases in contextual models. in the case of gender, the model appear to encode stereotypes regarding career and family, as well as the gendering of rational vs. emotional character. although our results do not indicate negative associations to romanian and south asian minorities, both bert and gpt-2 transformer models appear to associate the two groups to low-skilled labor. the biases encoded in these word embeddings and language models can have dire social consequences. in particular, our findings indicate significant gender and ethnic encoded bias in the occupational context. the use of technologies built on such models in the labor market could reinforce existing inequalities in hiring practices and prolong structural inequalities. intersectional bias according to a national report on romanian immigrants in italy foot_9 , most romanian men work in the construction sector, whereas romanian women are associated with domestic or care work, but are also often employed in shops, hotels and restaurants, health care, and social services. our findings indicate that similar biases are encoded in word embeddings, most notably through the low ingressivity of occupations in romanian quadrants (6) and the top intersection occupations identified in our findings also demonstrate particular intersectional biases with respect to women, most visible in the top intersectional traits corresponding to each intersectional group (tables 14 and 15 ). with regard to character traits, italian women are associated with femininity, romance, worldliness, and refinement, with proportionally more positive traits relative to women of other ethnicities. black women are associated with more sexualized traits as well as superstition and other negative words. the traits most associated romanian women are uniformly negative. not all of our results align with expectations regarding known stereotypes. for example, the fact that the majority of words landed in black quadrants ( there were also further unexpected observations regarding occupations. we draw particular attention to the presence of jailer, lawyer and paralegal in black quadrants. such observations do not necessarily imply that those jobs employ more black people, but could instead reflect more frequent encounters with a discriminatory justice system. the high level of ingressivity measured for italian women in fise-it1 were also surprising. while statistical error could contribute, a portion of the measured ingressivity could also come from a general higher attribution of ingressive traits to white italians in comparison to black italians. the relatively close levels of ingressivity between italian men and women in the same test could also be linked to the 'strong/fiery' stereotypes often associated with southern european women. technical methods our findings suggest that grammatical gender plays a significant role in bias measurements and should be carefully accounted for. in addition. seat methods did not measure any biases that were not already detected by weat methods. this is not surprising, as using the same sentence templates for every word would be expected to make the corresponding embeddings more similar. lpbs, however, computes word similarity in an entirely different manner. although not as many biases were detected using this method, lpbs proved to be an important complement be-cause it detected bias in bert that other methods overlooked. crows-pairs tests also indicate that the bert model encodes a significant number of harmful stereotypes. it would be interesting to expand the dataset to encompass a broader variety of stereotypes particular to the italian context. conclusion with respect to rq1, the bias measurement techniques carried out in this work demonstrate strong evidence that harmful gender and ethnic (intersectional) biases are encoded in both static and contextual italian word embeddings. of particular value are the tests particularly tailored to the italian cultural context, which detect encoded biases that would not be detected by simple translation of existing methods. in response to rq2, we find that many elements of existing bias detection methods are particular to an english-language and american context. careful cultural adaptation requires extensive investigation of local stereotypes and validation by nativespeakers. connecting to rq3, linguistic adapation is also essential, particularly regarding interference between grammatical and semantic gender. however, the averaging approach we employ to mitigate the contribution of grammatical gender may not be precise enough to preserve essential semantic information. future work could investigate more sophisticated methods for removal of the grammatical gender component of word embeddings @xcite . recently, large language models (llms) have largely superseded the language models studied in this work. the obsolescence of the models studied here is a significant limitation. llm bias detection is dominated by prompt-based methods, in part because many such models are proprietary and researchers do not have access to the models' inner workings. however, in cases where the necessary information is accessible, the methods described here by be adapted to state-of-the-art models as well. this could be the subject of future work. moreover, the datedness of the models studied in this work does not preclude their use; they may be better suited than llms to many nlp tasks in which social bias is relevant, even beyond saving on computational costs. for example, this work was undertaken in the context of studying fairness and bias in ai-assisted recruitment. in this context, understanding of which features about job candidates were used to render a decision is a neces-sity to ensure fairness. well-developed explainable ai methods make the models studied in this work more relevant in such situations.