Profiling Bias in LLM s: Stereotype Dimensions in Contextual Word Embeddings (2025)

Abstract: Large language models (LLMs) are the foundation of the current successes of artificial intelligence (AI), however, they are unavoidably biased. To effectively communicate the risks and encourage mitigation efforts these models need adequate and intuitive descriptions of their discriminatory properties, appropriate for all audiences of AI. We suggest bias profiles with respect to stereotype dimensions based on dictionaries from social psychology research. Along these dimensions we investigate gender bias in contextual embeddings, across contexts and layers, and generate stereotype profiles for twelve different LLMs, demonstrating their intuition and use case for exposing and visualizing bias.

## introduction amongst many other semantic concepts, large language models (llms) pick up stereotypes from the data they are trained on. unbiased data is hard to come by, especially in the amounts needed for the ever-larger models, which are the foundation of the current successes of ai and the respective hype. thus bias in these models is basically unavoidable, making it necessary to understand its characteristics and extents to communicate the risks and find ways to mitigate adverse discriminatory effects on affected populations. past research on bias has often involved word embedding association tests @xcite , inspired by the implicit association tests (iat) @xcite of social psychology. by another inspiration from the social sciences, a newer direction of natural language processing (nlp) research transforms opaque embeddings into a space of meaningful dimensions *statistically significant differences (@xmath0.05). @xcite @xcite , enabling new ways to study concepts. similar to semantic differentials @xcite , this methodology relies on antonyms (e.g. fast vs. slow) or opposing concepts described by lexicons. in this work we study bias in llms by transforming their embeddings based on the stereotype content model (scm) by @xcite , enabling the study along theoretically and empirically grounded stereotype dimensions. the scm entails two primary dimensions originating from interactions, where people seek to understand the other party's intent (dimension of warmth) and their capabilities (dimension of competence). stereotypically, women are thereby associated with higher warmth, and men with higher competence. furthermore we employ the extended model @xcite @xcite , allowing us to provide detailed 7d bias profiles as shown in figure 1 . dimension direction n terms n add. additional terms sociability high 43 nice, friendliness, warmth 199 accomodating, witty low 42 unfriendly, unsociability, distant 162 acid, withdrawn morality high 51 humane, morality, benevolent 205 allegiance, true low 69 untrustworthiness, evil, insincere 635 abandon, wrongful ability high 40 intelligence, capable, graceful 302 accomplished, ace low 39 ignorant, stupid, inefficient 160 awkward, unadvised agency high 42 motivated, autonomous, resolute 256 action, worker low 39 vulnerable, submission, helpless 113 bowing, unsure status high 21 superior, wealth, important 187 advantage, win low 13 poor, insignificant, unsuccessful 117 bankrupt, welfare politics traditional 12 conventional, conservative 34 classical, capitalist progressive 16 modern, liberal, democrat 45 contemporary, feminist religion religious 18 believer, church, god-fearing 146 spirit, testament non-religious 10 atheist, skeptical, secular 6 unholy, impious contributions. in summary, we (i) show how the stereotype content model can be employed to expose and visualize bias in contextual embeddings foot_0 , (ii) generate bias profiles for twelve llms for gender-associated names and gendered terms, displaying overall stereotypical associations of warmth and competence, (iii) provide insights on stereotype dimensions and gender bias across context examples and network layers. ## related work inspired by the human implicit association test @xcite , @xcite developed the first word embedding association test (weat) to assess the association between two target concepts (e.g., scientist vs. librarian) and two attributes (e.g., male vs. female) in static word embeddings by cosine similarity and a permutation test. later tan and celis (2019) built a first approach to measure bias for llms using the contextual embeddings of the words within examples. the contextual embedding association test (ceat) @xcite employs a random effects model to quantify bias with sampled contexts from a corpus. a newer approach to interpreting the highdimensional embedding spaces is again inspired by a concept from the social sciences; seman-tic differentials @xcite . @xcite introduced polar, a transformation of static word embeddings to a new polar, interpretable space. the polar opposites are antonyms such as hot-cold or soft-hard, and their word vectors are employed to define the new dimensions, which were shown to align with human judgment in an evaluation study. similar frameworks are se-maxis @xcite , frameaxis @xcite and @xcite . more recently, the sensepolar framework was introduced by @xcite , extending the polar approach to contextual word embeddings. the poles are hereby defined not by the word alone but by their embedding within sense-specific example sentences from a dictionary. the authors showed that these more interpretable embeddings can achieve similar performance to regular ones on natural language understanding (nlu) tasks, and furthermore confirmed the approach by a human evaluation study. most similar to our work @xcite analyzed stereotype dimensions in static embeddings, combining the polar framework by @xcite with the warmth and competence dimensions of the stereotype content model @xcite . they demonstrated that static word embeddings can recreate the stereotype dimensions from literature by predicting the coldwarm and competent-incompetent associations for additional known words, and by further comparing the results to psychological surveys. for contextual embeddings ungless et al. (2022) measured bias with ceat @xcite based on the warmth and competence dimensions and in a generation-based approach @xcite elicited evaluation of different social groups on these dimensions, with multiple prompting strategies. the stereotype content model has also been used for de-biasing methods @xcite . the researchers suggest that this theory-driven approach has an advantage because it is socialgroup-agnostic and thus does not require iteration over discriminated groups or previous knowledge of specific bias. in another projection approach, omrani sabbaghi et al. (2023) used a maximum margin support vector classifier to learn the valence subspace (pleasantness vs. unpleasantness) and projected the word 'person' to this dimension, placing different words in its context. the bias between words is measured by their effect on the contextualized representation of 'person'. this work furthermore relates with a broader range of studies trying to understand the contents of contextual representations, most notably by knowledge probing (e.g. @xcite ; @xcite ). see @xcite for a recent survey. ## discussion our results provide substantial evidence of stereotype dimensions in the embedding space of llms and a gender bias that predominantly corresponds to the human bias found in studies of the stereotype content model @xcite . for all studied models, female names are relatively associated with higher warmth, and for most models, male names are associated with higher competence. there is less evidence of bias for the studied gendered terms, which in part is likely due to the small groups of only nine terms per gender. the direction of gender differences is, however, overwhelmingly consistent. we furthermore find stereotype dimensions and bias across layers, in line with prior work that semantics are spread throughout the network @xcite ). 645 1.0 0.5 0.0 0.5 1.0 projected values non-religious progressive low status low agency low ability low morality low sociability female names male names religious traditional high status * high agency * high ability * high morality * high sociability * 1.0 0.5 0.0 0.5 1.0 projected values non-religious progressive low status low agency low ability low morality low sociability female terms male terms religious traditional high status high agency high ability high morality high sociability * the projection of contextual embeddings based on the stereotype content model can deliver robust insights when analyzing large vocabulary groups. as the magnitude of the values depends on the properties of the original embedding space, statistical analysis is employed to assert the significance of gender differences. this is viable with the larger collection of gender-associated names, also providing context for the differences between binary gendered terms. for the analysis of small vocabulary groups or individual terms, e.g. for comparing terms of binary and nonbinary gender, increasing the number of context examples offers potential for statistical tests. while both 2d and 7d stereotype dimensions provide interesting results, a significant gender bias is most evident in the warmth and competence dimensions. these benefit from the larger numbers of low and high-rated words in the dictionaries, increasing the robustness of the concept representations. likely associations are also more stable when relating to broader concepts. therefore, the high-level projection is a suitable first level of analysis and starting point for bias mitigation. significant gender bias, however, may also occur on a more granular level. different dimensions can be relevant depending on domains and tasks, such as progressive-traditional in the realm of politics, and the mode of projection can be easily adapted with the presented methodology. a combined projection with other dimensions such as valence (unpleasantness vs. pleasantness) (see e.g. omrani sabbaghi et al. ( 2023 )), could provide even further insights. as we have shown, the term context can have a considerable influence on the behavior of the stereotype dimensions. thus, examples for pole and projected terms should be chosen deliberately. gender non-specific context is our default choice because no additional bias is introduced through the examples and we get a clearer picture of the bias already present within the pre-trained embeddings. even smaller open-source llms are now able to provide examples with this property at scale. however, measurement is certainly best conducted with domain-specific data when a specific use case exists. while we use simple templates as contexts for the gendered terms and names, these could as well be sampled from a target domain or be generated to test specific scenarios. for example, similar to @xcite , this could involve introducing success in a historically male-dominated field to the term/name context, to test if a penalty exists for females, as found in psychological studies @xcite . while embedding-based methods for bias measurement have been critiqued for their remoteness from downstream applications @xcite , and are certainly no substitute for taskspecific investigations, they have multiple advantages. first, the methodology does not depend on natural language datasets that can be leaked into training data and are therefore applicable to older and newer models alike. second, the same stereotype dimensions can easily be used for bias mitigation @xcite , alleviating representational harm and the risk that it influences downstream behavior. finally, our paper shows they can be exploited for intuitive visualizations exposing gender bias. ## conclusion large pre-trained language models reflect the biases in their training data, which in turn reflect the biases of their creators. as the foundation for ai applications, their biases are further propagated, warranting their study to uncover the risks and promote mitigation efforts. in this work, we profile gender stereotypes in twelve llms by means of the stereotype content model of social psychology @xcite , thereby theoretically grounding the analysis, which has in the past been described as the missing link for bias measurements @xcite . by a matrix transformation, the opaque contextual embeddings of the models reveal interpretable stereotype dimensions. along the two major dimensions of warmth and competence, we find significant bias for genderassociated names and some evidence of bias for gendered terms, widely aligned with stereotypes found in human studies. the shown presence of stereotype dimensions in llms is a comprehensible replication of semantics in human language, however, the differential associations of social groups along these dimensions constitute a representational harm. equal treatment starts with equal representation; stereotypes already statistically significant in embedding space come with the risk of being exploited in downstream tasks, which could lead to different and unfair treatment of social groups. while the first access point would be the training data itself, the embedding space allows a quantification of patterns that is useful for bias assessment and mitigation. the analysis of embedding spaces by interpretable dimensions provides a means to evaluate both functional semantics and harmful associations that should be 'unlearned' to prevent their propagation. awareness of bias in llms needs to be increased beyond the expert audience, as biased models are already deployed, and completely debiased models may hardly be attainable, as they are trained on vast amounts of biased humancreated data. the here presented bias profiles based on the stereotype content model employ an intuitive scoring along meaningful scales of opposing concepts (e.g. low vs. high warmth), as proven effective by semantic differentials in human surveys @xcite . the result is a highly visual solution for communicating bias to wider audiences and users of artificial intelligence.