On Measures of Biases and Harms in NLP (2022)

Abstract: Recent studies show that Natural Language Processing (NLP) technologies propagate societal biases about demographic groups associated with attributes such as gender, race, and nationality. To create interventions and mitigate these biases and associated harms, it is vital to be able to detect and measure such biases. While existing works propose bias evaluation and mitigation methods for various tasks, there remains a need to cohesively understand the biases and the specific harms they measure, and how different measures compare with each other. To address this gap, this work presents a practical framework of harms and a series of questions that practitioners can answer to guide the development of bias measures. As a validation of our framework and documentation questions, we also present several case studies of how existing bias measures in NLP—both intrinsic measures of bias in representations and extrinsic measures of bias of downstream applications—can be aligned with different harms and how our proposed documentation questions facilitates more holistic understanding of what bias measures are measuring.

## introduction as language technologies and their applications become more widely deployed in our society, there are also increasing concerns of the disparate impacts and harms these technologies have on different demographic groups @xcite . to address some of these concerns, a large body of work has emerged to discuss @xcite @xcite , detect @xcite , measure @xcite @xcite , and mitigate (dev and phillips, 2019; * equal contribution @xcite @xcite the social biases encoded by nlp models. several of these works include bias measures comprising of metrics and datasets to define and investigate social biases within the constructs of a specific nlp task, such as text classification or machine translation. though these works propose different approaches for measuring biases, there is often similarly a lack of explicit alignment to harms, as well as a lack of comparative understanding of the advantages and disadvantages between the different bias measures for various language tasks. as an example, for the task of coreference resolution, there are several measures investigating gender bias @xcite @xcite @xcite . however, each measure is unique in either the targeted demographic groups, metrics, dataset sentence structures, or the definition of bias, all of which ultimately affect what harms are measured. a better understanding of bias measures ultimately enables better adaptation and deployment for specific use cases. this paper is motivated by two main goals. the first goal is to define a practical framework for harms that is both theoretically-motivated and empirically useful for describing bias measures. we organize a framework that is motivated by concepts from social psychology and linguistics, and narrow down specific definitions and heuristics to tag normative notions of harm with which bias measures align. moreover, we illustrate the utility of this measure-harm alignment exercise with case studies that demonstrate how a measure might unknowingly conflate different harms, or how separate measures with nearly identical task definitions can actually measure very different harms. the second goal is to define a collection of documentation questions around bias measures that helps others capture measure limitations and align operationalizations of "biases" to harms. documenting various attributes (e.g., considerations for targeted demographic groups and tasks, dataset limitations, bias metric definitions and motivations) of a bias measure can help practitioners better articulate harms, appropriate use cases, and limitations. to achieve these goals, we organize a practical framework of harms, a tagged collection of 43 existing bias measures and the associated harms, a set of documentation questions, and a collection of case studies. ## background we clarify the definitions of several terms used throughout this paper. bias in nlp bias in language models is commonly defined as "skew that produces a type of harm" @xcite towards different social groups, though it is a complex notion that is often not well-defined in existing literature @xcite @xcite . in the existing nlp literature, "biases" are often operationalized via a measurement model (jacobs and wallach, 2021) through bias measures. while these bias measures are proxies for evaluating bias, they are often necessarily localized to measuring very specific skews and lack context of how a system would be used by real users. additionally, unstated assumptions and definitions often pervade these measures @xcite . it remains an open question whether these bias measures actually measure meaningful and useful distinctions of "biases"-this work provides initial explorations to answer this question for several measures. bias measures bias evaluations in nlp typically have been categorized broadly into intrinsic or extrinsic evaluations based on whether they measure biased associations within the word embedding spaces @xcite or biased decisions from models for specific tasks @xcite , respectively. we define a bias measure as an evaluation standard that includes a metric(s) applied to a dataset. here, we use the term dataset broadly, such that it could be applicable to datasets ranging in size and curation technique (e.g., manually crafted, generated). to show inequalities between demographic groups, existing works typically define bias metrics (e.g., specialized notions of group fairness) that they then apply to a dataset specially designed to reveal social inequalities or stereotypes. these measures span several nlp tasks such as question answering @xcite , relation extraction @xcite , textual entailment @xcite , toxicity prediction @xcite @xcite , coreference resolution @xcite , autocomplete generation @xcite , dialogue generation @xcite , machine translation @xcite , as well as intrinsic measurements of the embeddings themselves @xcite @xcite . ## a framework for harms conflating harms impedes accurate measurement; adequate and consistent delineation of harms enables ongoing appraisal of the effectiveness of mitigation strategies and the comparison of trade-offs. our practical framework of harms builds upon existing taxonomies of representational harms (e.g. blodgett (2021) and establishes specific heuristics (appendix a) to disentangle the characteristics of five non-mutually exclusive categories. while these harms have previously been taxonomized, we ground the definitions of harms into documentation questions and heuristics to help practitioners align nlp bias measures with specific harms. addressing a single phenomenon with different lenses can surface multiple harms; precisely which harm a method captures is sometimes solely dependent on the experimental framing, rather than some inherent taxonomic difference. using the harm heuristics we devise in appendix a, we tag and distinguish between types of harms targeted by popular nlp bias measures presented in tables 1 and appendix disparagement disparagement encapsulates any behavior by a model which reinforces the notion that certain groups are less valuable than others and less deserving of respect (or resources). commonly associated measures of disparagement include toxicity ratings and hate speech detection scores. dehumanization dehumanization actively casts disfavored groups as "others" and aims to erase signs of shared humanity (e.g. emotions, agency, intelligence), thus suppressing opportunities for empathy with said "out group" by characterizing them as sub-human @xcite . dehumanization can therefore be challenging to measure directly, as instances of dehumanizing language or sentiments are often closely intertwined with disparagement and stereotyping. erasure erasure refers to the lack of adequate representation of members of a particular social group @xcite , whether intentional or not. while the data used to represent the intricacies of reality will always be necessarily incomplete, erasure can arise from mismatches in reality and the data chosen to represent it. it can also serve to reinforce existing power structures via incautious mathematical averaging or aggregation of disparate groups. while relational group sizes from the real world can be reflected from the model in a quantitative sense, the challenge is designing systems which do not allow relative size to inappropriately affect prominence, i.e., attention needs to be paid to the potential effects these probabilities have on produced output. quality of service quality of service harms result from instances where a model fails to perform equitably for different groups @xcite . this harm can in turn potentially result in inequitable allocation of resources @xcite , though this harm can also exist independently. the potential 'quality' of service is operationalized and quantified via defined performance indicators, which can be systematically compared between commensurable groups. ## documenting bias measures while bias measures aimed at various tasks are widely developed across the nlp community, the measures are often underused or re-developed by researchers for the same task. this stems largely from a lack of usability since little to no documentation of motivation and various choices is available for these measures. documentation for datasets and models have proliferated over the last few years but the rapidly growing collection of bias measures lacks such organized efforts. existing works have stressed the importance of documenting models @xcite , datasets @xcite , measurement modeling validity and reliability (jacobs and wallach, 2021), and, more recently, ethical considerations @xcite . this paper adds a complimentary resource focusing on documentation considerations for bias measures into the existing collection. in this work, we build upon the existing guidelines from @xcite , which are more generally for datasets of any modality or purpose, and narrow the focus to bias measures for nlp tasks. we add questions related to the composition and collection process sections as proposed by @xcite . additionally, we propose new sections on motivation specifically for bias measures and bias metrics. the specificity of the questions addresses the intended usage of different bias measures more explicitly. ## case studies we present a series of case studies as examples of how our proposed framework of harms and documentation questions reveal unique insights into different bias measures. in ## conclusion bias measures in nlp are critical for estimating and mitigating potential harms towards different demographic groups. however, a lack of structured understanding of what harms exist, how they are operationalized through bias measures, and how they can be measured can diminish the usefulness of bias measures. in this work, we organize a framework to define and distinguish between different types of harms-presented through heuristics and documentation questions-to guide more intentional development of bias measures. our proposed documentation template also facilitates combining, comparing, and utilizing different bias measures, and continuously re-visiting them to update limitations and comparative understanding with other measures.