Seeing Race, Feeling Bias: Emotion Stereotyping in Multimodal Language Models (2025)

Abstract: Large language models (LLMs) are increasingly used to predict human emotions, but previous studies show that these models reproduce gendered emotion stereotypes. Emotion stereotypes are also tightly tied to race and skin tone (consider for example the trope of the angry black woman), but previous work has thus far overlooked this dimension. In this paper, we address this gap by introducing the first large-scale multimodal study of racial, gender, and skin-tone bias in emotion attribution, revealing how modality (text, images) and their combination shape emotion stereotypes in Multimodal LLMs (MLLMs). We evaluate four open-source MLLMs using 2.1K emotion-related events paired with 400 neutral face images across three different prompt strategies. Our findings reveal varying biases in MLLMs representations of different racial groups: models reproduce racial stereotypes across modalities, with textual cues being particularly noticeable. Models also reproduce colourist trends, with darker skin tones showing more skew. Our research highlights the need for future rigorous evaluation and mitigation strategies that account for race, colorism, and gender in MLLMs.

## results we present our aggregated results across all four models in figures 3 , 4 , and 5. they show the eight most frequently attributed emotions categorized by (i) race, (ii) race and gender, and (iii) skin tone, respectively. since we evaluate four mllms across three distinct axes (race, gender, and skin tone) using three prompt templates, presenting all modelspecific results in the main paper would have been impractical. therefore, we chose to report aggregated patterns in the main text to illustrate the broader, recurring trends, while providing complete model-specific breakdowns in appendix f for transparency and completeness. we also analyze whether these results are significant using a chi-squared statistical significance test @xcite . for more detailed statistical significance test results, see appendix c. in the following sections, we analyze these results in detail. before moving to the detailed results, we note that our prescriptive stance for a fair model is that emotion attribution should remain invariant to race, skin tone, and gender cues when these factors are not semantically relevant to the emotional content of the scenario. in other words, a fair model should not disproportionately associate certain emotions (e.g., anger, shame) with specific demographic groups unless these associations are grounded in the content of the scenario rather than socially ingrained stereotypes. ## emotion attribution by stereotypes the results we observe in section 3 mostly reflect the underlying societal stereotypes about the emotional tendencies associated with different races and genders. models consistently attribute anger more frequently to black individuals than to others across all prompting templates. our results consistently indicate that models attribute the emotion of anger more frequently to black individuals across all prompting templates (text-only, imageonly, and text+image). this aligns with deeply ingrained societal stereotypes associating black individuals with heightened aggression or hostility. these stereotypes signal that there must be something about the inner lives of people who are black that is dangerous @xcite . such stereotypes have been extensively documented in psychological and social science literature, showing that black individuals are often unfairly perceived as more threatening or aggressive @xcite @xcite . the systematic assignment of anger to black individuals by mllms highlights how societal biases embedded in training data reinforce harmful racial stereotypes. models attribute guilt and disgust more frequently to white individuals. the consistent attribution of guilt and disgust to white individuals reflects complex societal associations that link white racial identity to historical and moral accountability, particularly in western societies. these associations may stem from socio-cultural narratives around privilege, colonialism, or racial injustice, which implicitly characterize white populations in a moral context @xcite . by disproportionately assigning these emotions, the models reflect societal stereotypes rooted in narratives of racial guilt and moral scrutiny tied to historical events involving white populations. we also note that this trend is evident in some models, rather than being a universal pattern. models show a clear pattern of assigning anger and frustration to males, while attributing sadness and fear more often to females. our findings reveal a consistent gendered pattern in emotion attribution, with models frequently assigning anger and frustration to males, while attributing sadness and fear predominantly to females. these outcomes are reflective of widely recognized gender stereotypes that associate males with emotions signaling dominance or assertive-ness, and females with emotions considered passive or submissive @xcite @xcite . such stereotypes persistently influence societal expectations and behaviors, thereby perpetuating biased emotional categorizations. models show differences in their responses based on skin tone: within the same race, they assign more anger to darker skin tones than to lighter ones. our results further expose biases based on skin tone, even within the same racial groups. darker-skinned individuals consistently received higher attributions of anger than their lighter-skinned counterparts. such bias, often described as colorism, is a pervasive societal issue globally, associating darker skin tones with negative traits such as aggression, criminality, or threat @xcite . this phenomenon is especially troubling as it highlights how fine-grained visual cues (skin tone) profoundly influence emotion attributions in mllms. to shed further light on how race shapes the emotion-related stereotypes reproduced by mllms, we analyze the sets of unique words the models generated. ## qualitative analysis: model-generated explanations to uncover the underlying rationale behind the model's attributions, we prompted them to provide explanations by adding the instruction in the prompt template. we use the same prompting template as we used in the ## discussion there are three key points worth highlighting: (1) different attributions of anger to black and asian people, (2) and of shame and guilt to asians and whites, and (3) the models' inconsistent refusals. ## conclusion our work presents the first large-scale, multimodal audit of racial, gender and skin-tone bias in emotion attribution by mllms. across all modalities, anger and frustration are over-assigned to black people (especially black men), whereas guilt and disgust attributed more for white people. males receive more externalising emotions (e.g., anger, frustration) and females more internalising ones (e.g., sadness, fear), reproducing well-documented gender tropes. within the same race, darker skin tones attract significantly more anger attributions than lighter tones, evidencing colourist bias. biases are strongest when race cues are provided only in text, attenuate when cues are visual-only, and sit in-between when cues are combined, indicating that textual racial labels act as a particularly potent stereotype trigger. collectively, these findings demonstrate that current mllms not only inherit but sometimes amplify long-standing social stereotypes. because emotion labels often underpin downstream applications, ranging from mental-health triage to social-robot interaction, these biases risk entrenching representational harms and unequal treatment of marginalised groups.