“You are Beautiful, Body Image Stereotypes are Ugly!” BIS tereo: A Benchmark to Measure Body Image Stereotypes in Language Models (2025)

Abstract: While a few high-quality bias benchmark datasets exist to address stereotypes in Language Models (LMs), a notable lack of focus remains on body image stereotypes. To bridge this gap, we propose \textbf{BIStereo} , a suite to uncover LMs’ biases towards people of certain physical appearance characteristics, namely, \textit{skin complexion, body shape, height, attire,} and a \textit{miscellaneous category} including \textit{hair texture, eye color, and more} . Our dataset comprises 40k sentence pairs designed to assess LMs’ biased preference for certain body types. We further include 60k premise-hypothesis pairs designed to comprehensively assess LMs’ preference for fair skin tone. Additionally, we curate 553 tuples consisting of a \textit{body image descriptor, gender, and a stereotypical attribute} , validated by a diverse pool of annotators for physical appearance stereotypes.We propose a metric, \textbf{TriSentBias} , that captures the biased preferences of LMs towards a certain body type over others. Using \textbf{BIStereo} , we assess the presence of body image biases in ten different language models, revealing significant biases in models Muril, XLMR, Llama3, and Gemma. We further evaluate the LMs through downstream NLI and Analogy tasks.Our NLI experiments highlight notable patterns in the LMs that align with the well-documented cognitive bias in humans known as \textbf{\textit{the Halo Effect}} .

## introduction the prevalence of biases and stereotypes based on physical appearance has long plagued society. as ai tools and language technologies expand globally, ensuring they are free from such biases is crucial. extensive research highlights the presence of social biases and stereotypes in nlp data and models @xcite . 1 data and code: https://github.com/narjisasad/ bistereo stereotypes, in social science, are generalized beliefs about people belonging to different social groups @xcite . bias is a prejudice towards or against an individual or community @xcite . actions, decisions, or opinions based on stereotypes lead to different biases, for instance, 'she was not selected for the sales job due to her dark complexion.' the underlying stereotype here is that 'dark-skinned people are less attractive.' however, biases can also stem from personal prejudice or favoritism, independent of stereotypes, such as, 'i denied her promotion because she rarely agrees with me.' our work focuses on body image or physical appearance stereotypes and the biased preferences or favoritism that lms and llms develop based on these stereotypes. lms learn statistical associations from their training data to associate concepts, and stereotypes are also often reflected in data as statistical associations. however, not all statistical associations learned from data are stereotypes. for instance, data might associate lighter skin tones with higher uv sensitivity and sunburn risk, as well as with attractiveness. while the former is a factual correlation based on medical research @xcite , the latter is a stereotype shaped by societal standards of beauty @xcite . several benchmark datasets have also been developed to evaluate the presence of harmful biases and stereotypes in lms @xcite . while existing datasets help detect biased preferences in lms, they lack a focus on body image stereotypes, limiting their ability to evaluate lms against such biases comprehensively. to bridge this gap, we present bistereo, a suite designed to uncover lms' biases based on physical appearance. motivation: body image stereotypes are deeply ingrained in human society, often manifesting as favoritism towards individuals with certain physical appearance characteristics. the entertainment industry and social media have also played a considerable role in overly glamorizing certain body types and perpetuating unrealistic beauty standards. if our cultural data-be it newspaper articles, magazines, social media posts, or movie dialoguesecho an obsession over certain body types, will not the language models (lms) trained on such data reflect these biased preferences too? while fairness and bias in language models have garnered significant attention, no comprehensive study has explored how these models reinforce harmful body image preferences, such as favoring plus-sized or size-zero bodies, dark-skinned or fairskinned individuals, or tall versus short stature. to address this gap, we introduce bistereo: a benchmark comprising a dataset, a metric, and downstream nli and analogy tasks, all meticulously designed to identify and quantify the stereotypical associations learned by lms. our dataset is in english language and includes both a global component and an india-specific component, enabling the analysis of body image biases across diverse cultural contexts. ## characterization of body image stereotypes: a brief overview body image stereotypes have a long-standing prevalence in human society. these biases and stereotypes can manifest in spoken or written text, audiovisual media, and memes. a few examples of body image bias and stereotypes are: s 1 : fat brides are a big turn off. s 2 : i will definitely not marry her, she is so fat. here sentence s 1 is an example of a stereotype, while s 2 is a bias based on physical appearance. moreover, stereotypes vary across global and geocultural contexts, and can reflect large variations among different states of the same country and between countries. for example, s 3 : women wearing burqa are seen as modest in arabian countries. s 4 : women wearing burqa are seen as conservative in asian countries. s 5 : full-figured women are seen as desirable in south india. s 6 : slim women are seen as desirable in north india. sentences s 3 and s 4 are examples of the globally varying nature of stereotypes, while s 5 and s 6 highlight regional variations within india. for a detailed review of the societal prevalence and impact of body-image stereotypes, and the risks associated with lms perpetuating them, see appendix b. with ai and nlp tools increasingly used across legal, medical, educational, and media sectors, ensuring fairness in llms is critical at both national and global levels. bistereo supports this goal by providing a high-quality, context-sensitive benchmark to evaluate physical-appearance stereotypes in lms and llms. ## related work bias and stereotype, often used interchangeably, refer to systematically favoring or opposing certain individuals or groups based on some attributes @xcite . ## bistereo: dataset creation bistereo is an agglomerate of three different components, each designed with a unique principle to address different ways in which physical appearance stereotypes can manifest in lms. the first component, bistereo-pairs, is designed to examine if lms associate certain physical appearance characteristics (eg. fair skin, tall, etc.) with positive attributes (eg. pretty, attractive, etc.) and if they associate certain characteristics (eg. dark skin, fat, etc.) with negative attributes (eg. ugly, unattractive, etc.). bistereo-pairs captures body image stereotypes that are globally prevalent. the second component, bistereo-nli, examines lms' association of fair-skinned and dark-skinned individuals with certain traits. it is designed to examine if the physical attractiveness stereotype foot_0 a subtype of the halo effect, which is a common cognitive bias in humans, is present in lms. the following subsections provide a detailed description of each component of the dataset. finally, the third component, bistereo-tuples, is designed to capture physical appearance stereotypes specific to the indian society. we also design an analogy task to demonstrate the utility of bistereo-tuples in uncovering harmful body image stereotypes in lms. body image stereotypes vary across geographical and sociocultural contexts, section 2 and appendix 2 describe this phenomenon in detail. ## uncovering body image stereotypes in lms with bistereo to comprehensively evaluate lms' stereotypical preferences for specific body image characteristics, we designed three experimental setups. each setup in its design uses one component of bistereo dataset. our experiments, their outcomes and implications are detailed below. ## conclusion and future work we introduce bistereo as a robust framework for evaluating physical appearance stereotypes in lms. the bistereo-pairs dataset, alongside the trisentbias metric, effectively probe lms, assessing their associations of positive and negative traits with physical appearance. bistereo-nli offers a comprehensive textual entailment dataset, ideal for assessing the presence of stereotypical associations pertaining to skin complexion, while bistereo-tuples provides valuable insights into body image stereotypes prevalent in indian society. our experiments on downstream nli and analogy tasks reveal strong alignment between lm outputs and existing societal stereotypes based on physical appearance, highlighting notable patterns that mirror the cognitive bias known as the halo effect. the use of pll scores allows us to precisely capture the influence of protected attributes on the remaining tokens of a sentence, although this method is limited to bidirectional models. existing methods for decoder-only models rely on sentence probability, but when protected attribute terms appear at the end of a sentence, this approach fails to accurately reflect their impact on the preceding tokens. developing an equivalent mechanism for decoderonly models is a promising direction for future research. we conclude with this thought: language models trained on human-generated data inevitably learn and reflect the biases and stereotypes embedded in the data.