Stereotyping N orwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets (2021)

Abstract: Auditing NLP systems for computational harms like surfacing stereotypes is an elusive goal. Several recent efforts have focused on benchmark datasets consisting of pairs of contrastive sentences, which are often accompanied by metrics that aggregate an NLP system’s behavior on these pairs into measurements of harms. We examine four such benchmarks constructed for two NLP tasks: language modeling and coreference resolution. We apply a measurement modeling lens—originating from the social sciences—to inventory a range of pitfalls that threaten these benchmarks’ validity as measurement models for stereotyping. We find that these benchmarks frequently lack clear articulations of what is being measured, and we highlight a range of ambiguities and unstated assumptions that affect how these benchmarks conceptualize and operationalize stereotyping.

## introduction auditing nlp systems for computational harms like the reproduction of stereotypes or hate speech remains a persistent challenge, due in no small part to the deeply contextual and open nature of language use and tasks @xcite @xcite @xcite , and a lack of consensus about how to conceptualize or operationalize such harms @xcite . to identify potential computational harms such as the reproduction of stereotypes, recent efforts rely on benchmark datasets. these datasets consist of tests that can take a variety of formats, including sentence templates where terms pertaining to groups or their attributes are perturbed @xcite @xcite , prompts designed to elicit problematic responses @xcite @xcite example sentences context i really like norweigan salmon. ## background: benchmark datasets the four benchmark datasets we consider 1) are designed to test nlp systems on two tasks-language modeling and coreference resolution, 2) consist of pairs of contrastive sentences ( §2.1), and 3) are accompanied by aggregating metrics ( §2.2). the datasets also vary in how the sentence pairs were constructed (by subject matter experts, or by crowdworkers), and by what is changed or perturbed within pairs (e.g., target group, or group attributes). in addition, pairs were also constructed with different evaluation paradigms in mind: a) intra-sentence prediction -where a model is used to estimate which candidate terms are more likely to fill-in-the-blank in a given sentence (e.g., which underlined term is more likely in girls/boys are smart); b) inter-sentence prediction -where a model is used to estimate which candidate next sentences are more likely to follow a given context sentence (e.g., given he is arab, which continuation is more likely: he is likely a terrorist/pacifist); and c) pronoun resolution -where a model is used to determine which entity a given pronoun is likely to refer to (e.g., which entity is the pronoun he likely refers to in [the worker] told the nurse that [he] has completed the task). ## inventory of pitfalls: sets and metrics some pitfalls are only visible when examining the dataset as a whole or the proposed aggregating metrics. since what the benchmarks aim to measure is not well articulated, it can be difficult to distinguish whether and when the pitfalls we list below suggest a poor conceptualization of stereotyping or instead call into question the way it is operationalized. ## discussion evaluating constructs we do not evaluate how well different benchmarks adhere to any particular conceptualization of stereotyping. rather, by identifying inconsistencies within and between sentence pairs and known aspects of stereotyping, we highlight possible implicit decisions about what constitutes stereotyping and what the benchmarks should focus on, which are not explicitly discussed or justified. since nlp practitioners using a benchmark might assume that everything included therein is meaningful, harmful, or worth measuring, we raise these pitfalls to suggest that researchers constructing such benchmarks should carefully consider which groups and content are included and prioritized, and make those decisions and the reasoning behind them explicit. harm reduction since we do not evaluate benchmarks against any particular conceptualization of stereotyping, we also do not evaluate the effectiveness of their conceptualizations (and operationalizations) towards harm reduction. however, if we assume the goal of the benchmarks is to reduce harm, then the pitfalls we raise become more concerning. without clearly articulated conceptualizations of stereotyping, much less conceptualizations grounded in the realities of how stereotypes uphold social hierarchies, and without analyses of what groups and stereotype content are ultimately covered in the constructed pairs; it is impossible to know whether the resulting measurements capture material, harmful stereotypes. aggregating metrics can also cause harm by assigning all groups and stereotype content equal weight, or by encouraging models to produce stereotypes just as often as anti-stereotypes. crowdsourcing offers several advantages over generating tests from templates or having researchers write them manually: crowdsourced datasets may reflect better ecological validity-by capturing a wider range of text than templates or nlp researchers might come up with-and coverage-by likely getting many stereotypes that are salient to crowdworkers. one question is thus how to retain these advantages, while avoiding the pitfalls we describe. involving experts in related areas, especially participants with lived experiences of language-related harms, might aid decisions at all parts of this process like deciding what groups and content to include. drawing on work in social psychology and related fields on developing measurement instruments or better processes for designing crowdsourcing tasks might also be helpful. finally, it is possible that crowdworkers might be too removed from the end goal of creating such benchmarks, and it might be better to invest in the (admittedly longer) process of working with experts, including participants. ## conclusion in our analysis, we identify a lack of clarity in how stereotyping is conceptualized, as well as a range of pitfalls threatening the validity of subsequent operationalizations. many of these pitfalls are not limited to the settings we examine, and are likely to arise wherever contrastive pairs are constructed to measure computational harms. therefore, it is critical to uncover the explicit and implicit assumptions that these benchmarks carry and the incentives to which they may give rise @xcite . we have aimed to be as clear and constructive as possible, in the hopes that the measurement modeling framework can provide analytical clarity and a scaffold for future work in this direction. throughout the paper, we also show examples of harmful stereotypes and statements, including some with offensive language. while these examples are illuminating, readers may also find them upsetting.