Insights from a Disaggregated Analysis of Kinds of Biases in a Multicultural Dataset (2025)

Abstract: Warning: This paper contains explicit statements of offensive stereotypes which may be upsetting.Stereotypes vary across cultural contexts, making it essential to understand how language models encode social biases. MultiLingualCrowsPairs is a dataset of culturally adapted stereotypical and anti-stereotypical sentence pairs across nine languages. While prior work has primarily reported average fairness metrics on masked language models, this paper analyzes social biases in generative models by disaggregating results across specific bias types.We find that although most languages show an overall preference for stereotypical sentences, this masks substantial variation across different types of bias, such as gender, religion, and socioeconomic status. Our findings underscore that relying solely on aggregated metrics can obscure important patterns, and that fine-grained, bias-specific analysis is critical for meaningful fairness evaluation.

## introduction the prevalence of unintended social biases in language models is a major concern for the field, especially those involved in spreading hurtful and offensive stereotypes, as shown in @xcite , @xcite , @xcite . a number of papers have published evidence of uneven treatment of different demographics @xcite , @xcite , @xcite , @xcite , @xcite , @xcite , @xcite . the bulk of the work conducted on bias in language models has addressed transformer models, and more specifically masked language models (mlms), popularized with the bert family of models @xcite . recent work in nlp has largely focused on so-called large language models (llms), in particular autoregressive models such as bloom @xcite or llama @xcite . fort notes in @xcite that the question of adapting bias evaluation frameworks designed for masked language models to these new models is still open. biases in language models can vary when prompted with stereotypes of different cultures due to the inherent cultural contexts embedded in both the training data and the societal norms reflected in the prompt language. these biases may result from the cultural values, beliefs, and social practices prevalent in the data used to train the models, leading to inconsistencies between the cultural contexts underlying model development and the diverse cultural ecosystems they are intended to serve. in this paper we use the multicultural dataset of stereotypes published in @xcite to explore the different predictions of recent llms on different types of social biases represented in the dataset. in this paper, we extend prior work on bias evaluation from masked language models to generative models such as llama and mistral. our contributions are as follows: ## previous work stereotypes are commonly held beliefs and generalizations about individuals based on characteristics like their race, gender, or nationality. the categorization of people into groups with these associated social stereotypes is a recurring cognitive process in our daily interactions @xcite . with the advances in nlp, specifically with significant development of llms in recent years, a large body of work has focused on understanding and evaluating their potential risks and harms @xcite , @xcite , @xcite , @xcite . years of research in the social sciences have resulted in the creation of multiple frameworks for comprehending various aspects of social stereotyping. @xcite , @xcite , @xcite , @xcite . two benchmark datasets stereotypes have been developed in english: crows-pairs @xcite , with 1508 pairs of sentences: one that is more stereotyping and another that is less stereotyping. and stereoset @xcite , with 17,000 covering both intrasentence and intersentence variations. french crows-pairs @xcite and nlp-fairness-for-india @xcite added insight to stereotyping datasets by shedding light into the french and indian cultures respectively, extremely useful in such an english western-centric context. ## dataset multilingualcrowspairs @xcite is based on the popular bias identification dataset crow-spairs @xcite . it extended the base dataset to seven new languages, related to seven different socio-cultural contexts: arabic from maghreb and the arab world in general, catalan from spain, german from germany, spanish from argentina, italian from italy, maltese from malta and simplified chinese from china. these are added to the corrected english from the united states and french from france corpora released by @xcite . see ## experiment setup all pairs of stereotype and anti-stereotype sentences for all languages were used. we computed the joint-likelihood metric for every sentence and compared it to its pair. this is the metric used in multicrowspairs @xcite . if sentence a had a higher score than sentence b, we classified it as a preference of the model for sentence a. all computation was performed using one nvidia a30 gpu, resulting in a total vram of 24gb. we decided to leverage meta-llama-3-8b and mistral-7b-v0.1 since we needed openweights models to access internal values to calculate these metrics, api-based closed models don't give the necessary means to do this. both were quantized to 16-bit and used approximately 16gb of vram each. the joint-likelihood probability of a sentence, as described by @xcite , is the product of conditional probabilities of the a word given all the previous ones. this is a common metric in the area for model confidence and calibration @xcite . for example, this is the computation required to compute it for the example sentence "it is a great day": p(
,it,is,a,great,day) frequently, the probability of a certain token was exactly zero because the precision limit of floating point numbers was reached. this caused the entire product to become zero, even when only a single token had underflowed. to mitigate this, we applied the transformation recommended by @xcite , x ′ = x(n -1)+s n , where n is the sample size and @xmath0, 1). as they note, "from a bayesian standpoint, s acts as if we are taking a prior into account. a reasonable choice for s would be .5." ## results in figure 1 , we show violin plots of stereotypical sentence preference across languages. most languages lie above the 0.5 mark, indicating a general preference for stereotypical over anti-stereotypical sentences. this trend is especially strong in majority languages like us english and german. we speculate this is due to higher resources available for training. in figure 2 we show matrices for preferred sentence distribution across language and bias type. each cell represents the percentage of stereotypical sentences that had a higher joint-likelihood than its anti-stereotypical pair. we observe that several types of bias score differently in different cultures. surprisingly, the most studied biases in the area such as race, nationality, gender, are the ones that exhibit the lowest average biases in multicrows-pairs. most of the published work on biases exploration and mitigation has been produced by english speaking communities, focusing mostly in the english language and for gender biases @xcite @xcite . the strongest biases in figure 2 that persisted across cultures are those associated with sexual orientation and disability (including mental illnesses). for both models, french and italian scored the highest percentage of preference for stereotypical sentences for the former, and us english for the latter. argentinian spanish's most stereotypical score was for socioeconomic biases. this is in line with the findings of the "argentinian national map of discrimination" @xcite , where discrimination due to poverty was experienced by 21% of respondents and placed third among all discrimination types. for physical appearance biases, us english and french consistently showed the strongest stereotypical preferences across models-for example, "angie couldn't find good workout clothes in her large size" vs. "in her small size" (en_us). chinese was the top scorer across the board with respect to age-related biases. this is consistent with the "35-year-old phenomenon" in china @xcite , where age discrimination causes unemployment.