In-Depth Look at Word Filling Societal Bias Measures (2023)

Abstract: Many measures of societal bias in language models have been proposed in recent years. A popular approach is to use a set of word filling prompts to evaluate the behavior of the language models. In this work, we analyze the validity of two such measures – StereoSet and CrowS-Pairs. We show that these measures produce unexpected and illogical results when appropriate control group samples are constructed. Based on this, we believe that they are problematic and using them in the future should be reconsidered. We propose a way forward with an improved testing protocol. Finally, we also introduce a new gender bias dataset for Slovak.

## introduction language models (lms) are ubiquitous in current nlp and have brought undeniable performance improvements for many tasks. concerns have been raised about the fairness of these models @xcite @xcite . since lms are usually trained with web-based text corpora generated by a general population, there is a risk that they will learn certain societal biases, such as sexist or racist stereotypes. with these models regularly being used as backbones for further fine-tuning, this unfairness might propagate further to downstream models and ultimately to user-facing applications. based on this assumption, many attempts were made to quantify the bias in lms. the measures usually observe lm outputs or inner workings to reveal problematic biased behavior. a popular method is to create a set of word filling prompts that test lm behavior in various situations, and then interpret the differences. for example, would an lm choose a negative stereotypical word for x in the sentence all women are x? tests like these are often proposed because neural lms are notoriously blackbox and it is otherwise difficult to interpret their inner working reliably. however, the observation process must be done in a methodologically sound manner and the correct assumptions must be used to ensure accurate results. in this work, we examine the validity of two widely used methodologies -stereoset @xcite and crows-pairs @xcite -for measuring societal bias in masked lms. we first identify several theoretical problems in their score calculations. then we show that these problems can be observed in the available data and we demonstrate that the lms exhibit unexpected behavior that violates key assumptions made by these methodologies. this leads us to question the validity of the reported results. we propose a way to improve the methodologies by introducing a new score definition. during experiments we introduce several new variants of the existing datasets and a completely new dataset in slovak. these new datasets are used to compare the expected behavior of the lms with their actual behavior. our results challenge the validity of previous studies. this is a significant issue as these measures are widely used foot_1 to demonstrate the level of bias in lms @xcite , as benchmarks for debiasing techniques @xcite , as inspiration for bias research in languages other than english @xcite , and for other bias-related research. if our assertions about their validity are accurate, all these efforts could be in danger. moreover, it is possible that other similar measures may face similar problems. ## related work measuring lm bias. in recent years, numerous methodologies and datasets have emerged for measuring societal bias in lms and other nlp mod-els @xcite . the techniques for masked lms are generally based on three types of analysis: critique. papers criticize and evaluate the proposed bias measuring techniques from various perspectives. @xcite identify several conceputalization and operationalization pitfalls in the existing benchmarks and estimate that a significant portion of samples have validity issues. the lack of robustness of the proposed metrics w.r.t. specific choices of templates, prompts, lexicon seeds, metrics, sampling strategies is also a concern @xcite @xcite . low correlations between individual scores raise questions about what exactly is being measured @xcite @xcite . other criticisms are more conceptual. @xcite point out that the motivation behind bias measuring techniques is often "vague, inconsistent and, lacking in normative reasoning" and that the techniques are often "poorly matched to the motivation". the lack of cultural sensitivity results in methods that are often anglo-centric or us-centric @xcite , do not correctly handle marginalized groups @xcite , or have other similar cultural issues. many of these problems could be addressed by improved training for data creators and by increasing data quantity and quality. however, we show that even with perfect data, some of the proposed methodologies may still not yield valid results. ## methodologies and datasets both stereoset (ss) and crows-pairs (cs) measure bias against certain groups using sets of word filling samples. there is usually a coupling between the dataset (the specific samples used for score calculations) and the methodology (how is the score calculated) @xcite . @e distinguish between these two concepts (e.g., by saying stereoset (ss) dataset and stereoset (ss) methodology) and we effectively decouple them when we stress-test the methodologies with new datasets. in this section, we briefly introduce the two existing methodologies and also our own dataset in slovak language that is compatible with both of them. we also created various new versions and extensions of the existing datasets for our experiments, which will be introduced as appropriate. all the datasets are documented in appendix a. nadeem et al. (2021) introduce the ss methodology and several datasets compatible with it that address different bias types -gender, race, profession, and religion. each dataset consists of pairs of sentences that differ in exactly one word. the dataset creators were instructed to first generate stereotypical and anti-stereotypical words associated with a specific group of people and then write a template in which these words could be used. for example, the male is strong / the male is weak is a pair that stereotypes males as strong. in this case, strong and weak are the keywords that differ. ## case study: stereoset some of the pitfalls previously identified in ss datasets @xcite could theoretically be addressed by improving data quality. however, we claim that the ss methodology is problematic by itself and does not provide a valid measurement of bias, regardless of how good the data is. first, we will identify several theoretical problems and then show how these problems manifest in the data by breaking several key assumptions the ss methodology makes. the problems are related to both how we calculate scores for individual samples and the way we aggregate them. ## case study: crows-pairs the cs methodology involves both marginalized and non-marginalized groups of people in its examples. however, it still faces problems similar to ss. as before, we will first outline theoretical problems and then demonstrate them experimentally. ## a way forward we have found several weaknesses in the existing measures, such as unexpected results for control samples, strong correlations between original pairs and control pairs and weak statistical power of results. to improve these measures and increase their validity, we propose a new score f based on the observation that, despite issues with existing datasets and methodologies, control pairs consistently have lower scores. we believe that this can be used to consistently measure bias. f is defined as the difference between the ss score for the original marginalized group (using a template t o ) and the ss score for the control group (using a template t c ): f (ws, wa, to, tc) = ss(ws, wa, to)ss(ws, wa, tc) (3) looking back at figure 1 , the f score measures the distance below the identity function for the sample. the lower the score, the greater the stereotypical difference between how the lms treat the original group versus the control group. to use this measure, the samples need to consist of quadruplets of sentences, as is the case with our slovak dataset and with the extended ss datasets. with clearly defined control groups and behavior that we expect, we believe that this measure has better normative reasoning compared to the other measures presented so far. conceptually similar approaches, using samples along two axesone for groups of people and the other for their attribute -are already used for measuring bias in word embeddings @xcite , sentence embeddings @xcite , or in lexicon-based approaches @xcite . on the other hand, f can still be influenced by spurious lm behavior. anecdotally, we noticed that the results for the slovak quadruplets ženy/muži nevedia/vedia x (in english women/men don't know/know how to x) will often flip if we use the past tense instead of the present. creating samples compatible with this methodology is also harder, as we now need a quadruplet of sentences, and it might be difficult to write natural sounding sentences for all four slots or even identify correct control groups and anti-stereotypes. the slovak dataset used throughout the experiments was the first attempt to use this methodology. ## discussion bias measures need to be validated. there is a growing body of evidence indicating that existing bias measures and datasets are often not reliable enough. issues such as data quality, robustness, statistical significance, weak correlation with one another, or even basic operationalization and conceptualization, exist in current work. each measure or dataset should be evaluated as thoroughly as possible, for instance by using contrastive examples, control groups, or stress tests. language models do not "understand" language like humans do. analyzing lm behavior can result in illogical conclusions from human perspective, for example, lms can exhibit both stereotypical and anti-stereotypical behavior towards certain groups. bias measures sometimes assume that lms have a worldview of their own. however, as demonstrated in this work, lms do not have consistent beliefs or thoughts, and their output often depends on minor input perturbations. this is an unintuitive behavior for humans, as we expect rational agents to be consistent in their opinions. when measuring whether an lm "prefers" certain statements, it's important to consider other reasons than bias. language models have limited language understanding capabilities. smaller lms struggle with negation @xcite ) and other simple linguistic phenomena. it is questionable whether they can accurately measure bias with more complicated sentences that contain negations or compound sentences, as these might be beyond the capabilities of some lms to process reliably. this should be taken into consideration during dataset collection or bias evaluation. word filling evaluations examine only a small fraction of the lexical space. comparing probabilities for only a few selected words ignores most possibilities. it is impossible to say anything about bias, when we have no information about what the rest of the lexical space looks like. many other words that have stereotypical or @xmath0. instead of ana- extrinsic downstream evaluation should be preferred. considering the case studies presented in this paper, we believe that word filling methodologies are currently not reliable enough for bias measurement. there is limited evidence so far that these measures correlate with how bias manifests in downstream applications. until these issues are resolved, evaluation of bias in downstream tasks should be the preferred method. inconclusive results for the slovak dataset. more slovak samples are required to thoroughly evaluate slovak lms. f is not statistically significant, and results for ss and cs scores also also inconclusive, although in general it seems that the models might be biased for slovak as well. in the future, it is crucial to expand the size and diversity of the samples and conduct a more in-depth analysis. ## conclusion this work provides an in-depth analysis of the limitations of word filling lm bias measures. despite their popularity, these measures have significant issues that call into question the validity of their results. our findings show that these measures can produce unexpected and contradictory results. for example, the stereoset methodology can generate stereotypical scores for both marginalized and non-marginalized groups, while crows-pairs methodology yields scores that strongly correlate for stereotypical and anti-stereotypical pairs. we propose a new dataset format, but it too can still be affected by various spurious correlations. based on these results, we do not recommend using existing word filling techniques to measure bias in lms. if they are to be used, we recommend setting up various sanity checks to distinguish true bias signals from model misbehavior or data annotation artifacts. the issues identified here might also be present in other datasets and methodologies.