What’s Not Said Still Hurts: A Description-Based Evaluation Framework for Measuring Social Bias in LLM s (2025)

Abstract: Large Language Models (LLMs) often exhibit social biases inherited from their training data. While existing benchmarks evaluate bias by term-based mode through direct term associations between demographic terms and bias terms, LLMs have become increasingly adept at avoiding biased responses, leading to seemingly low levels of bias. However, biases persist in subtler, contextually hidden forms that traditional benchmarks fail to capture. We introduce the Description-based Bias Benchmark (DBB), a novel dataset designed to assess bias at the semantic level that bias concepts are hidden within naturalistic, subtly framed contexts in real-world scenarios rather than superficial terms. We analyze six state-of-the-art LLMs, revealing that while models reduce bias in response at the term level, they continue to reinforce biases in nuanced settings. Data, code, and results are available at https://github.com/JP-25/Description-based-Bias-Benchmark .

## introduction the remarkable performance of large language models (llms) is frequently accompanied by the propagation of social bias inherent in their training data @xcite @xcite . these biases raise serious ethical concerns by perpetuating stereotypes and reinforcing discrimination. in many high-stakes domains, the deployment of llms may disproportionately harm marginalized individuals and communities @xcite @xcite @xcite @xcite . numerous studies @xcite @xcite evaluate bias from a termbased perspective, which assesses direct lexicallevel associations between demographic identities and bias-related terms (demographic-term associations). as illustrated in figure 1 (a), an example from bbq @xcite demonstrates bias when the model consistently associates "margaret" (female) with the term "bad at math" and "george" (male) with the term "good at math", or vice versa. we call such a bias measurement focused on associations to terms term-based evaluation. however, this evaluation method is too superficial, and the measurement method can be easily manipulated by breaking the direct association between demographic identities and bias-related terms @xcite . thus, state-ofthe-art (sota) llms usually show a low level of bias when evaluated by existing term-based evaluation benchmarks @xcite @xcite . in our experiments (details in section 4.2.2), gpt-4o achieves a score of -0.000807 on the bbqambiguous dataset, with 0 indicating no bias. does this suggest that llms are truly unbiased -or, current benchmarks measuring bias in the superficial term-based way are insufficient to capture the full spectrum of biases? indeed, real-world biases are rarely expressed in explicit terms. instead, they operate behind the scenes, manifesting through semantic-level associations between demographic identities and biasrelated concepts (demographic-concept associations). critically, these concepts are conveyed through depictions of behaviors, emotions, mental activities, and more. a single concept can be represented by different descriptions. we call this semantic-level measurement manner descriptionbased evaluation, distinguishing it from the termbased methods in prior work. within the same scenario as figure 1 (a), option a in figure 1(b ) portrays behaviors that subtly imply the concept of "bad at math", whereas option b reflects the notion of "good at math". bias is detected when females are consistently associated with depictions aligned with the concept of "bad at math", while males are linked to "good at math", or vice versa. existing works on social bias in llm focused on superficial demographic-term associations and overlooked the evaluation of demographic-concept associations. toward this, we propose the description-based bias benchmark (dbb) to systematically evaluate social bias. with this new benchmark, we find that llms often avoid showing bias at the term level (e.g., figure 1(a) ) but can unintentionally perpetuate the same instance of bias when expressed at the description level (figure 1(b) ). in our experiments (section 4.2.2), when we use dbb to examine the same set of biases tested by bbq, we observe a significant increase in bias metrics for gpt-4o, illustrating the necessity and significance of investigating bias in the proposed nuanced setting. data, code, and results are available at https://github.com/ jp-25/description-based-bias-benchmark . in summary, our contributions are: ## related work due to space limitations, more extensive discussions of related works are provided in appendix b. bias evaluation in llms has been extensively studied via term-based methods that assess lexicallevel associations between demographic terms and bias terms (term-based evaluation). multiple benchmarks @xcite @xcite @xcite @xcite @xcite quantify bias using term-based evaluation from diverse perspectives and form the foundation for evaluating how llms respond to superficially biased content @xcite @xcite @xcite . as llms evolve, their responses to such direct evaluations have become more neutral and socially aligned, often resulting in lower bias scores. however, this trend does not necessarily indicate genuine mitigation. subtler forms of bias may persist in ways that traditional benchmarks cannot detect @xcite . @xcite move beyond conventional benchmarks by prompting llms to associate specific words with demographics, then generate narratives. nonetheless, the method remains within a term-based mode, relying on lexical-level associations. in contrast, we introduce the description-based bias benchmark (dbb), which assesses bias by description-based evaluation. this approach captures semantic-level associations between demographic identities and bias-related concepts that are subtly hidden within naturalistic contexts. specifically, dbb evaluates bias by measuring response variations across parallel test instances with different demographic descriptors where bias-related concepts are hidden within naturalistic contexts rather than superficial term associations. ## description-based bias benchmark as llms show low bias in existing term-based bias benchmarks, we aim to develop a dataset designed to measure bias by the description-based method in llms that previous works do not measure. ## experiments in this section, we conduct comprehensive experiments on our benchmark to evaluate bias from three perspectives: analyze biases measured by our proposed dbb. compare biases measured by different benchmarks. compare instance to instance between dbb and bbq. ## semi-generation based dbb (dbb-sg) motivation. we introduce a semi-generationbased dbb (dbb-sg) alongside the original mcq-based dbb. dbb-sg is motivated by the growing application of llms in open-ended tasks, such as text generation, providing a more realistic assessment of social bias. mcq offers limited answer options, restricting the model's ability to fully reveal biases as they might appear in real-world scenarios. since free-text generation is challenging in this study, we adopt a semi-generation approach. specifically, for each bias concept, we generate ten sentence variations to approximate the probability of producing any sentence reflecting that concept. the core goal of dbb-sg is to measure the probability of llms generating the sentence that subtly hidden bias concept, rather than measuring the probability of llms picking one specific option that conveys the concept. bias measures. following the same bias measurement mechanism in section 3.3, the probability of selecting an answer option for question 1 option a, p 1 (a), is computed as the average across all generated variations. the same method applies to other answer options. bias score calculation also follows equation 1. details on the answer option calculations for dbb-sg are in appendix e.1. bias analysis. for bias analysis in dbb-sg, we have three observations: (1) dbb-sg reveals biases across models, (2) llms display similar bias patterns across categories in dbb-sg, with the race category showing the highest bias, and (3) the influences of bias descriptor demonstrate similarities across llms in dbb-sg. the complete experiment results are in appendix e.2. in summary, the findings suggest that bias patterns vary across models when evaluated using the semi-generation format, indicating that different models exhibit distinct biases under generative conditions. additionally, it is important to note that dbb-sg results cannot be directly compared to the dbb results due to fundamental methodological differences. as discussed in limitations, a direct comparison requires further investigation, which we plan to pursue in future work. moreover, the generative approach is expected to introduce greater bias, as it more closely resembles natural language usage in real-world scenarios. ## conclusion in this work, we introduce the description-based bias benchmark (dbb), a novel dataset for systematically evaluating bias by the description-based method in llms. unlike prior benchmarks that assess bias via explicit demographic-term associations to form term-based evaluation, dbb captures how biases persist in realistic depictions where stereotypes are subtly hidden. we detail dbb's construction, where demographic descriptors and bias concepts are hidden within naturalistic contexts, and evaluate model responses across parallel test instances. our analysis reveals that while llms show reduced bias in term-based evaluation, they continue to reinforce bias in subtle, descriptive settings. this highlights dbb's value as a complementary tool for bias measurement, addressing the limitations of previous benchmarks.