The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks (2023)

Abstract: How reliably can we trust the scores obtained from social bias benchmarks as faithful indicators of problematic social biases in a given model? In this work, we study this question by contrasting social biases with non-social biases that stem from choices made during dataset construction (which might not even be discernible to the human eye). To do so, we empirically simulate various alternative constructions for a given benchmark based on seemingly innocuous modifications (such as paraphrasing or random-sampling) that maintain the essence of their social bias. On two well-known social bias benchmarks (Winogender and BiasNLI), we observe that these shallow modifications have a surprising effect on the resulting degree of bias across various models and consequently the relative ordering of these models when ranked by measured bias. We hope these troubling observations motivate more robust measures of social biases.

## introduction the omnipresence of large pre-trained language models @xcite @xcite has fueled concerns regarding their systematic biases carried over from underlying data into the applications they are used in, resulting in disparate treatment of people with different identities @xcite . in response to such concerns, various benchmarks have been proposed to quantify the amount of social biases in models @xcite @xcite . these measures are composed of textual datasets built for a specific nlp task (such as question answering) and are accompanied by a metric such as accuracy of prediction which is used as an approximation of the amount of social biases. these bias benchmarks are commonly used by machine learning practitioners to compare the degree of social biases (such as gender-occupation gender-occupation bias ❌ ## related work a large body of work investigates ways to evaluate biases carried inherently in language models @xcite @xcite and expressed in specific tasks @xcite @xcite @xcite @xcite @xcite @xcite . alongside, there is also growing concern about the measures not relating to experienced harms @xcite , not inclusive in framing @xcite , ambiguous about what bias is measured @xcite , not correlated in their findings of bias across intrinsic versus extrinsic techniques @xcite , and susceptible to adversarial perturbations @xcite and seed word selection @xcite . the concurrent work by @xcite ) discusses the unreliability of quantifying social biases using templates by varying templates in a se-mantic preserving manner. while their findings are consistent with ours, the two works provide complementary experimental observations. @xcite study a wider range of tasks, though we focus our experiments on a wider set of models and alternate dataset constructions (with a greater range of syntactic and semantic variability). as a result, we are able to illustrate the effect of the observed variability on ranking large language models according to measured bias for deployment in real world applications. ## social bias measurements and alternate constructions bias measures in nlp are often quantified through comparative prediction disparities on language datasets that follow existing tasks such as classification (de-arteaga et al., 2019) or coreference resolution @xcite . as a result, these datasets are central to what eventually gets measured as "bias". not only do they determine the "amount" of bias measured but also the "type" of bias or stereotype measured. datasets often vary combinations of gendered pronouns and occupations to evaluate stereotypical associations. it is important to note that these constructs of datasets and their templates, which determine what gets measured, are often arbitrary choices. the sentences could be differently structured, be generated from a different set of seed words, and more. however, we expect that for any faithful bias benchmark, such dataset alterations that are not relevant to social bias should not have a significant impact on the artifact (e.g. gender bias) being measured. thus, to evaluate the faithfulness of current benchmarks, we develop alternate dataset constructions through modifications that should not have any effect on the social bias being measured in a dataset. they are minor changes that should not influence models with true language understanding -the implicit assumption made by current bias benchmarks. any notable observed changes in a model's bias measure due to these modifications would highlight the incorrectness of this assumption. consequently, this would bring to light the unreliability of current benchmarks to faithfully measure the target bias and disentangle the measurement from measurement of other non-social biases. a non-exhaustive set of such alternate constructions considered in this work are listed below. negations: a basic function in language understanding is to understand the negations of word groups such as action verbs, or adjectives. altering verbs in particular, such as 'the doctor bought' to 'the doctor did not buy' should typically not affect the inferences made about occupation associations. synonym substitutions: another fundamental function of language understanding is the ability to parse the usage of similar words or synonyms used in identical contexts, to derive the same overall meaning of a sentence. for bias measuring datasets, synonymizing non-pivotal words (such as non-identity words like verbs) should not change the outcome of how much bias is measured. varying length of the text: in typical evaluation datasets, the number of clauses that each sentence is composed of and overall the sentence length are arbitrary experimental choices. fixing this length is common, especially when such datasets need to be created at scale. if language is understood, adding a neutral phrase without impacting the task-specific semantics should not alter the bias measured. adding descriptors: sentences used in real life are structured in complex ways and can have descriptors, such as adjectives about an action, person, or object, without changing the net message expressed by the text. for example, the sentences, "the doctor bought an apple.", and "the doctor bought a red apple." do not change any assumptions made about the doctor, or the action of buying an apple. random samples: since the sentence constructs of these datasets are not unique, a very simple alternate construction of a dataset is a different subsample of itself. this is because the dataset is scraped or generated with specific assumptions or parameters, such as seed word lists, templates of sentences, and word order. however, neither the sentence constructs or templates, nor the seed word lists typically used are exhaustive or representative of entire categories of words (such as gendered words, emotions, and occupations). see fig. 2 for example constructions on wino-gender (app. a, b for detailed descriptions). ## case studies we discuss here the impact of alternate constructions on two task-based measures of bias. foot_1 ## discussion and conclusion social bias measurements are very sensitive to evaluation methodology. our empirical evidence sheds light on how the model's non-social biases brought out or masked by alternate constructions can cause bias benchmarks to underestimate or overestimate the social bias in a model. more interestingly, it is important to note that different models respond differently to perturbations. in fact, the same perturbation can result in a higher or lower measured bias depending on the model (as seen in §4.1 and §4.2), which points to how models might parse information (and thus bias) differently. while current bias measures do play a role in exposing where model errors have a stereotypical connotation, a lack of sentence construction variability or even assumptions made when creating seed word lists can reduce the reliability of the benchmarks, as we see in this work ( §4.2). even with simple sentences, it is not apparent how to disentangle the biased association of the identity with the verb or the occupation amongst others. this is especially important to note as it highlights that measures can lack concrete definitions of what bi-ased associations they measure. consequently, the relation between measured bias and experienced harm becomes unclear. we hope that our troubling observations motivates future work that thoroughly investigates how to construct robust benchmarks that faithfully measure the target bias without being affected by model errors and other non-social biases. as suggested by our subsampling experiments (appendix f), it might be fruitful to encourage both syntactic and semantic diversity in these benchmarks. bias benchmarks that provide uncertainty measures (instead of a single number) might enable practitioners to better compare models before deploying them. furthermore, since the opaqueness of large language models makes it challenging to understand how and to what extent a linguistic change will affect the measured bias, explainable models might indeed facilitate better measurement of their social bias. assuming that we can generate faithful explanations for a model's predictions, an exciting future direction is to explore construction of bias benchmarks which operate on the explanations of the predictions rather than the predictions themselves. lastly, we also encourage discussions on the complexity of the sentences used in benchmarks and their implications on what gets measured in relation to un-templated, naturally-occurring text @xcite , as an attempt to ground our measurements in experienced harms.