BBQ : A hand-built bias benchmark for question answering (2022)

Abstract: It is well documented that NLP models learn social biases, but little work has been done on how these biases manifest in model outputs for applied tasks like question answering (QA). We introduce the Bias Benchmark for QA (BBQ), a dataset of question-sets constructed by the authors that highlight attested social biases against people belonging to protected classes along nine social dimensions relevant for U.S. English-speaking contexts. Our task evaluate model responses at two levels: (i) given an under-informative context, we test how strongly responses reflect social biases, and (ii) given an adequately informative context, we test whether the model’s biases override a correct answer choice. We find that models often rely on stereotypes when the context is under-informative, meaning the model’s outputs consistently reproduce harmful biases in this setting. Though models are more accurate when the context provides an informative answer, they still rely on stereotypes and average up to 3.4 percentage points higher accuracy when the correct answer aligns with a social bias than when it conflicts, with this difference widening to over 5 points on examples targeting gender for most models tested.

## the dataset bbq contains 58,492 unique examples. we create templates which consist of two questions, answer choices, a partial context missing information necessary to answer the questions, and a disambiguating context that provides the necessary information. each template is written from scratch by the authors and is inspired by an attested bias. each context isolates the targeted bias and has two slots that we fll with labels representing the given category (see bbq tests biases in model predictions, allowing us to study how social biases may manifest in downstream tasks. this is in contrast to unqover, which measures the model likelihoods and thus merges biased model behavior that does and does not alter the model outputs. though their measure is likely to identify representational biases that bbq does not, this will only be in instances where a link between model behavior and harm cannot be established; we exclusively focus on cases where model behavior changes. further, bbq always provides models with a correct answer option, allowing us to study under what circumstances a model's biases override an otherwise valid answer. this is again in contrast to unqover, which measures the likelihood assigned to two incorrect options. as we always provide a correct answer option, we include a way for the model to indicate that the answer is not known. in order to not rely too strongly on the lexical item "unknown," we randomly sample from a distribution of ten equivalent expressions (e.g., "cannot be determined", "not known"); we refer to these as unknown answers. ## validation we validate examples from each template on amazon mechanical turk. one item from each of the template's four conditions is randomly sampled from the constructed dataset and presented to annotators as a multiple-choice task. each item is rated by fve annotators, and we set a threshold of 4/5 annotators agreeing with our gold label for inclusion in the fnal dataset. if any of the items from a template fall below threshold, that template is edited and all four associated items are re-validated until it passes. additional details on the validation procedure are in appendix d. to estimate human accuracy on bbq, we repeat the validation procedure 5 with a random sample of 300 examples from the fnal dataset. we estimate that raw human (crowdworker annotator) accuracy on bbq is 95.7%, and aggregate human accuracy calculated via majority vote is 99.7%. agreement between raters is high, with a krippendorf's α of 0.883. ## results accuracy overall accuracy on bbq is highest for unifedqa with a race-style input format at 77.8% and lowest for roberta-base at 61.4% (chance is 33.3%). however, models are generally much more accurate in the disambiguated contexts than in the ambiguous contexts (see figure 5 in the appendix), showing that when a correct answer is in the context, models are fairly successful at selecting it, even when that answer goes against known social biases. however, accuracy in disambiguated contexts where the correct answer aligns with a social bias is still higher than examples in which -6.4 -2.8 3.7 -1.7 -5.5 -2.2 -5.5 -0.8 -4.8 -6.3 2.4 3.4 4.4 0.0 -19.2 -1.1 -3.8 -5.5 -6.9 -0.9 -4.6 -4.1 -7.9 -3.9 figure 2 shows how much the accuracy drops when the correct answer is nonaligned with a social bias, relative to when the correct answer aligns with the bias. 6 within each model, this difference is present in most of the categories, as shown in figure 2 . ## discussion interpretation of bias scores we note here a strong caveat about the interpretation of these results: bias scores near zero mean that, in the aggregate, the model tested tended to give an answer including a certain label as often in response to negative questions as it did in response to a nonnegative questions. the scores refect behavior on just 25 templates in each category and should not be taken as proof that the model is unbiased or less biased, but only that the model does have a directionally consistent bias on this small sample. a second and related caveat is that the data in bbq is only designed to test biases associated with us english-speaking cultural contexts, and it should not be used as evidence that a model would still look unbiased for contexts from a different culture. categories with high bias scores categories such as physical appearance have a high bias score in ambiguous contexts that persist even in disambiguated contexts on some templates, indicating that models are sometimes using social biases in place of recognizing a correct answer that is explicitly provided in the context. a closer look at this category's example sets reveals that biases re- while it's possible the biased answer is due to an association of words like "sloppy" and "fat," the potential impact of this issue extends beyond representational harm, as given the qa setting, the model is assigning the property of being sloppy to an actual individual because of this bias. though it can be useful to look at individual template results, it is important to keep in mind that each one represents a very small number of examples from the whole dataset and is susceptible to noise issues that come with having a small number of items ( ## conclusion we present bbq, a hand-built dataset for measuring how social biases targeting nine different categories manifest in qa model outputs given different kinds of contexts. bbq covers a broad range of categories and biases relevant in us contexts and allows researchers and model developers to (i) measure in which contexts model behavior is likely to lead to harm, and (ii) begin exploratory analyses of lms to understand which biases (both individual and intersectional) require mitigation or further study. we show that current models strongly rely on social biases in qa tasks when the contexts are underspecifed. models achieve low accuracy in these ambiguous contexts (no more than 67.5%), and their errors reinforce stereotypes up to 77% of the time. even when a short context provides a clear answer, both the model's accuracy and outputs are occasionally affected by these social biases, overriding the correct answer to instead select one that perpetuates harm against specifc populations.