Social Bias Probing: Fairness Benchmarking for Language Models (2024)

Abstract: While the impact of social biases in language models has been recognized, prior methods for bias evaluation have been limited to binary association tests on small datasets, limiting our understanding of bias complexities. This paper proposes a novel framework for probing language models for social biases by assessing disparate treatment, which involves treating individuals differently according to their affiliation with a sensitive demographic group. We curate SoFa, a large-scale benchmark designed to address the limitations of existing fairness collections. SoFa expands the analysis beyond the binary comparison of stereotypical versus anti-stereotypical identities to include a diverse range of identities and stereotypes. Comparing our methodology with existing benchmarks, we reveal that biases within language models are more nuanced than acknowledged, indicating a broader scope of encoded biases than previously recognized. Benchmarking LMs on SoFa, we expose how identities expressing different religions lead to the most pronounced disparate treatments across all models. Finally, our findings indicate that real-life adversities faced by various groups such as women and people with disabilities are mirrored in the behavior of these models.

## introduction the unparalleled ability of language models (lms) to generalize from vast corpora is tinged by an inherent reinforcement of social biases. these biases are not merely encoded within lms' representations but are also perpetuated to downstream tasks @xcite , where they can manifest in an uneven treatment of different demographic groups @xcite @xcite . direct analysis of biases encoded within lms allows us to pinpoint the problem at its source, potentially obviating the need for addressing it for every application @xcite . therefore, a number of studies have attempted to evaluate social biases within lms @xcite @xcite . one approach to quantifying social biases involves adapting small-scale association tests with respect to the stereotypes they encode @xcite . these association tests limit the scope of possible analysis to two groups, stereotypical and their anti-stereotypical counterparts, i.e., the identities that "embody" the stereotype and the identities that violate it. this binary approach, which assumes a singular "ground truth" with respect to a stereotypical statement, has restricted the depth of the analysis and simplified the complexity of social identities and their associated stereotypes. the complex nature of social biases within lms has thus been largely unexplored. our social bias probing framework, as outlined in fig. 1 , is specifically designed to enable a nuanced understanding of biases inherent in language models. accordingly, the input of our approach consists of a set stereotypes and identities. to this end, we generate our probing dataset by com-bining stereotypes from the social bias infer-ence corpus (sbic; @xcite ) and identities from the lexicon by @xcite . in this paper we examine identities belonging to four social categories: gender, religion, disability, and nationality. secondly, we assess social biases across five state-of-the-art lms in english. we use perplexity @xcite , a measure of language model uncertainty, as a proxy for bias. by analyzing the variation in perplexity when probes feature different identities within the diverse social categories, we infer which identities are deemed most likely by a model. this approach facilitates a three-dimensional analysis -by social category, identity, and stereotype-across the evaluated lms. in summary, the contributions of this work are: ## related work social bias benchmarking prior work, such as crows-pairs @xcite and stere-oset @xcite , was pioneering in benchmarking models in terms of social biases and harmfulness. however, concerns have been raised regarding stereotype framing and data reliability of benchmark collections designed to analyze biases in lms @xcite . specifically, @xcite determine the extent to which a masked language model prefers stereotypical or anti-stereotypical responses, while the stereotype score developed by @xcite expands this approach to include both masked and autoregressive lms. a significant limitation of both benchmarks is their use of a 50% bias score threshold, where models are considered biased if they prefer stereotypical associations more than half the time, and unbiased otherwise @xcite . another approach, which does not rely on choosing one correct answer from two options, is the proposed by @xcite all unmasked likelihood (aul) method which predicts all tokens in a sentence, considering multiple correct candidate predictions for a masked token, which is shown to improve accuracy and avoid selection bias. @xcite instead leverage pseudo-perplexity @xcite in combination with a toxicity score to assess the tendency of lms' to generate statements distinguished between harmful vs. benevolent. our social bias probing framework (i) probes biases across multiple identities without assuming the existence of solely two groups and contests the need for a deterministic threshold for dividing these groups; (ii) is developed with benchmarking social bias in the autoregressive causal lms in mind. social bias datasets benchmarking social bias is highly reliant on the underlying dataset, i.e., the bias categories, stereotypes, and identities it includes @xcite . stereoset presents over 6k triplets (for a total of approximately 19k) crowdsourced instances measuring race, gender, religion, and profession stereotypes, while crows-pairs provides roughly 1.5k sentence pairs (for a total of 3k) to evaluate stereotypes of historically disadvantaged social groups. @xcite introduce a conversational dataset consisting of 11, 873 sentences generated from reddit conver-sations to assess stereotypes between dominant and minoritized groups along the dimensions of gender, race, religion, and queerness. these datasets cover a limited set of identities and stereotypes. therefore, bias measurements using these resources could lead to inaccurate fairness evaluations. in fact, @xcite show that they are able to measure previously undetectable biases with their large-scale dataset of over 450, 000 sentence prompts from two-person conversations. our sofa benchmark includes a total of 408 identities and 11, 349 stereotypes across four social bias dimensions, for a total amount of 1, 490, 120 probes, presenting an extensive resource for social bias probing of language models. ## social bias probing framework social bias foot_1 can be defined as the manifestation through language of "prejudices, stereotypes, and discriminatory attitudes against certain groups of people" @xcite . these biases are featured in training datasets and are carried over into downstream applications, resulting in, for instance, classification errors concerning specific minorities and the generation of harmful content when models are prompted with sensitive identities @xcite . to measure the extent to which social bias is present in language models, we propose a social bias probing framework (see fig. 1 ) which serves as a technique for fine-grained fairness benchmarking of lms. we first collect a set of stereotypes and identities (section 3.1-section 3.2), which results in the sofa (social fairness) dataset (section 3.3). the final phase of our workflow involves evaluating language models by employing our proposed perplexity-based fairness measures in response to the constructed probes (section 3.4), exploited in the designed evaluation setting (section 3.5). ## experiments and results in stereoset @xcite : to assess the bias in a language model, the model is scored using likelihood-based scoring of the stereotypical or anti-stereotypical association in each example. the percentage of examples where the model favors the stereotypical association over the anti-stereotypical one is calculated as the model's stereotype score. crows-pairs @xcite : the bias of a language model is assessed by evaluating how often it prefers the stereotypical sentence over the anti-stereotypical one in each pair using pseudolikelihood-based scoring. ## conclusion this study proposes a novel social bias probing framework to capture social biases by auditing lms on a novel large-scale fairness benchmark, sofa, which encompasses a coherent set of over 400 identities and a total of 1.49m probes across various 11k stereotypes. a comparative analysis with the popular benchmarks crows-pairs @xcite and stereoset @xcite reveals marked differences in the overall fairness ranking of the models, suggesting that the scope of biases lms encode is broader than previously understood. further, we expose how identities expressing religions lead to the most pronounced disparate treatments across all models, while the different nationalities appear to induce the least variation compared to the other examined categories, namely, gender and disability. we hypothesize that recent efforts to mitigate racial and gender biases in lms could be why disparities in religion are now more apparent. consequently, we stress the need for a broader holistic bias investigation. finally, we find that real-life harms experienced by various identities -women, people identified by their nations (potentially immigrants), and people with disabilities -are reflected in the behavior of the models.