This prompt is measuring <mask>: evaluating bias evaluation in language models (2023)

Abstract: Bias research in NLP seeks to analyse models for social biases, thus helping NLP practitioners uncover, measure, and mitigate social harms. We analyse the body of work that uses prompts and templates to assess bias in language models. We draw on a measurement modelling framework to create a taxonomy of attributes that capture what a bias test aims to measure and how that measurement is carried out. By applying this taxonomy to 90 bias tests, we illustrate qualitatively and quantitatively that core aspects of bias test conceptualisations and operationalisations are frequently unstated or ambiguous, carry implicit assumptions, or be mismatched. Our analysis illuminates the scope of possible bias types the field is able to measure, and reveals types that are as yet under-researched. We offer guidance to enable the community to explore a wider section of the possible bias space, and to better close the gap between desired outcomes and experimental design, both for bias and for evaluating language models more broadly.

## introduction concurrent with the shift in nlp research towards the use of pretrained and generative models, there has been a growth in interrogating the biases contained in language models via prompts or templates (henceforth bias tests). while recent work has empirically examined the robustness of these tests @xcite , it remains unclear what normative concerns these tests aim to, or ought to, assess; how the tests are constructed; and to what degree the tests successfully assess the concerns they are aimed at. for example, consider the prompt "people who came @xmath0, which is used for testing "ethnic bias." in the absence of common words like "piratopia" or "pirateland," it is not clear how we might want the * equal contribution. correspondence to whomever. model to behave. one possibility is to consider (as @xcite do) a model biased to the extent that it predicts particular countries, such as "somalia" over "austria," to replace the masked token; a model that is not biased might be one that does not vary the prior probabilities of country words when "pirate" is present, or else predicts all countries with equal likelihood. but such a bias definition would require the model to disregard the 'knowledge" that austria, unlike somalia, is landlocked. it is no more self-evidently appropriate a definition than one requiring a model to give equal country probabilities given some features (e.g., geographic, historical) or requiring the gap in probability between "somalia" and "austria" to be constant for all sea terms, positive or negative (e.g., "pirate," "seamen"). to be meaningful and useful, then, a bias test must articulate and connect: a) the normative concern it is meant to address, b) desirable and undesirable model outcomes given that concern, and c) the tests used to capture those outcomes. in this work, we critically analyse these bias tests by developing a taxonomy of attributes grounded in measurement modelling ( §3), a framework originating from the social sciences @xcite . our taxonomy captures both what a bias test aims to measure-its conceptualisation-and details of how that measurement is carried out-its operationalisation. by disentangling these aspects of bias tests, our taxonomy enables us to explore threats to bias tests' validity-when a given test may not be meaningful or useful @xcite . in an individual bias test, our taxonomy reveals threats to validity, and whether the test is trustworthy and measures what it purports to. in aggregate, our taxonomy outlines the broader landscape of the concerns identified by the current literature, and the approaches taken to measure them. we apply our taxonomy to annotate 77 papers proposing bias tests ( §4). we find that bias tests are often poorly reported, missing critical details about what the paper conceptualises as the bias or harm to be measured, and sometimes even details about how the test is constructed. this lack of detail makes it challenging (or impossible) to assess the measurement's validity. even where sufficient detail is provided, tests' validity are frequently threatened by mismatches between the test's construction and what papers state that they are trying to capture. finally, we find that many bias tests encode implicit assumptions, including about language and culture and what a language model ought (or ought not) to do. when left unstated, these assumptions challenge our ability both to evaluate the test and to explicitly discuss desired and undesired outcomes. therefore, despite the wealth of emerging approaches to bias testing that a practitioner might like to apply, it is not clear what harms and biases these tests capture, nor to what extent they help mitigate them. as a result of these issues, the space of possible biases captured by current bias tests underestimates the true extent of harm. this paper makes several contributions. by drawing out aspects of how bias tests are described and constructed, we hold a mirror to the literature to enable and encourage reflection about its assumptions and practices. our analysis illuminates where existing bias tests may not be appropriate, points to more appropriate design choices, and identifies potential harms not well-captured by current bias tests. additionally, we offer some guidance for practitioners ( §6), grounded in insights from our analysis, on how to better design and document bias tests. while this study focuses on bias, our taxonomy and analysis can be applied to prompt-based analysis of generative models more broadly. future work in other subfields of nlp may, in using our taxonomy as scaffolding, be able to see reflected back the assumptions that limit the scope and the predictive power of their research, and will have a roadmap for correcting them. foot_0 ## related work a number of recent meta-analyses use measurement modelling, either implicitly or explicitly. explicitly, blodgett et al. (2020) uses measurement modelling to survey bias papers in nlp, and to expose the often hazy links between normative mo-tivation and operationalisation in bias works, as well as lack of clarity and precision in the field overall. our work has a different focus, but is inspired by their analytical approach. blodgett et al. (2021) also explicitly uses measurement modelling to critique a variety of benchmarks, but focuses primarily on their design and quality, and less on either metrics used, or on generative models. recent work in nlp has empirically found some threats to convergent validity @xcite by finding disagreement in results across benchmarks that purport to all measure the same biases. this suggests that something in these benchmarks' experiment setup is incorrect or imprecise, or that they are in reality measuring different constructs. other work has found threats to predictive validity where embedding and language model based measures of bias do not correlate with bias in downstream applications @xcite . @xcite implicitly look at both predictive and convergent validity of a number of intrinsic and extrinsic classificationbased bias metrics, and have difficulty establishing either correlation betweeen the intrinsic ones (convergent) or between the intrinsice and extrinsic (predictive). @xcite examine template based tests of social bias for mlms and three downstream tasks (toxicity, sentiment analysis, and nli) for brittleness to semantically equivalent rephrasing. this work is topically related to ours (though it stops short of looking at generative systems), but does not engage with measurement modelling either implicitly or explicitly. @xcite do a meta-analysis of 146 different bias metrics and fit them into three generalised categories of bias metric. this is valuable groundwork for future tests of convergent validity, though they do not engage with the validity of these metrics. the combination of theoretical taxonomy and empirical results was conceptually influential to our work. ## findings we detail our observations here, beginning with those surrounding conceptualisations and operationalisations, and concluding with those about basic details and scope. figure 1 presents a selection of quantitative results of our 90 bias tests. ## discussion validity and reliability whereas validity asks, "is [the measurement] right?", construct reliability asks, "can it be repeated?" @xcite . sometimes design choices that aid in establishing validity can threaten reliability, and vice versa. for example, many papers that conceptualise bias in terms of toxic content generation use prompt continuation as a prompt task, and operationalise bias as differences in toxicity across generated output. this setting reflects good predictive validity in testing whether, over a broad set of outputs, the model generates toxic content. however, reliability may be threatened, as the test is brittle to choices such as decoding parameters @xcite . in the opposite direction, tests using generation from a fixed set of n words are easier to replicate than less constrained generation, but at the cost that the set of phenomena that can be captured is narrower. similarly, sentiment and toxicity have the advantage of having many available classifiers in different languages, and many tests use an ensemble of multiple such classifiers. despite this, because these classifiers may differ in subtle ways and be frequently updated, their use may threaten reliability, since tests relying on them may yield inconsistent results. by contrast, regard is operationalised via a classifier developed by @xcite al. (2019 the papers capture only a small subset of both the ways in which marginalised communities can be harmed, and the ways their identities are encoded in language. with the use of relatively few proxy types, bias tests are generally unable to address bias against speakers of marginalised language varieties (as opposed to direct targets), or the under-representation of marginalised groups (erasure bias). ## recommendations guided by our analysis, we formulate the following list of questions that future bias research can consult to inform experimental design. at minimum, the answers to these questions should be provided when reporting bias research. these questions can be easily adapted to guide reviewers when evaluating bias research, and practitioners in assessing whether and how to apply particular bias tests. ## conclusion we hope that via our taxonomy and analysis, practitioners are better-equipped to understand and take advantage of the wealth of emerging approaches to bias testing-in particular, to clearly conceptualise bias and desired model outcomes, design meaningful and useful measurements, and assess the validity and reliability of those measurements.