Bias Identification and Attribution in NLP Models With Regression and Effect Sizes (2022)

Abstract: In recent years, there has been an increasing awareness that many NLP systems incorporate biases of various types (e.g., regarding gender or race) which can have significant negative consequences. At the same time, the techniques used to statistically analyze such biases are still relatively simple. Typically, studies test for the presence of a significant difference between two levels of a single bias variable (e.g., male vs. female) without attention to potential confounders, and do not quantify the importance of the bias variable. This article proposes to analyze bias in the output of NLP systems using multivariate regression models. They provide a robust and more informative alternative which (a) generalizes to multiple bias variables, (b) can take covariates into account, (c) can be combined with measures of effect size to quantify the size of bias. Jointly, these effects contribute to a more robust statistical analysis of bias that can be used to diagnose system behavior and extract informative examples. We demonstrate the benefits of our method by analyzing a range of current NLP models on one regression and one classification tasks (emotion intensity prediction and coreference resolution, respectively).

## introduction machine learning has been a major driver of innovation in natural language processing since the 1990s, but only the last decade has seen the widespread deployment of nlp methods for use by non-experts: applications such as neural machine translation @xcite or voice assistants @xcite ) are now routinely available through end users' mobile phones, and nlp methods are increasingly used in domains outside computer science such as police work @xcite and recruiting @xcite . such systems are, from a user perspective, black boxes whose predictions are generally taken at face value. this makes the question pertinent to what extent the machine learning methods underlying these nlp models are fair, or, on the contrary, to what extent they are subject to biases which impact their predictions. more formally, @xcite defined biased computer systems as systems that "systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others"; (see @xcite for a very similar definition). clearly, such biases have the potential to cause concrete harm for the disadvantaged groups or individuals (bender @xcite and must be observed and controlled as far as possible. a practical aspect of bias analysis, which the above definition leaves open, is whether discrimination is measured "in vitro" (at the level of system performance) or "in vivo" (at the level of real world consequences). in line with the majority of nlp studies on bias, the present study focusses on bias measured "in vitro", i.e., in the form of systematic differences in system performance across groups. we acknowledge the need to better understand how such "in vitro" bias translates into "in vivo" real-world consequences, and argue below that the methods we propose offer a first step in this direction. a quickly growing body of studies has indeed found that biases are, unfortunately, pervasive in nlp systems @xcite . one of the first studies on bias, @xcite analyzed similarity relations in word embeddings and found a substantial gender bias, as a result of which, e.g., woman was more similar to nurse than doctor, while man was more similar to doctor than nurse. @xcite found systematic and substantial racial biases in five twitter datasets annotated for offensive language detection, where african american english tweets were overclassified as hateful compared with standard american english, and @xcite found a significant age bias in many sentiment northern european journal of language technology analysis algorithms, attributing less positive attitudes to older participants. see section 2 for more details. consequently, dealing with biases is rapidly becoming a major high-level consideration in the design and development of nlp systems. the three main bias-related tasks are (a) bias identification (is bias present?), (b) bias attribution (where does the bias come from?) and (c) bias mitigation (how to minimize the bias?). in this article, we focus on the first two tasks, bias identification and attribution. following the definition given above, the identification of "in vitro" bias involves the establishment of systematic differences in system performance between two parallel stimuli sets for different levels of a bias variable such as gender or race. put simply, the question is: does, e.g., the gender of an author have a systematic influence on the output of an nlp system (e.g., are texts written by women predicted to be less positive?), or on the quality of the nlp system? (e.g., are text written by women analyzed less reliably?) this question can be answered using statistical analysis techniques of increasing complexity, shown in although this procedure is conceptually simple and straightforward, it is problematic for two reasons. first, the pairwise hypothesis tests that are being employed in existing work assume that differences between the two sets of stimuli are due to the selected bias variable. they cannot ensure that the putative effect of bias is not due to a covariate that acts as a confounding variable @xcite . for instance, studies on gender bias often use sets of male and female names as part of their stimulus sets (cf. second, bias studies in nlp currently generally test for statistical significance, but very few consider model fit and effect sizes (with the notable exception of @xcite ). significance ensures that an identified effect is not a random fluke, but does not quantify how much of the variance in the predictions is due to the bias. given a sufficiently large dataset, even very small differences that are not practically relevant can reach significance. in contrast, the computation of effect sizes permits users to understand the practical impact of biases @xcite , and is therefore arguably a first step moving from bias "in vitro" towards bias "in vivo". in this article, we propose that these two limitations can be alleviated by adopting multivariate regression models such as linear and logistic regression for bias identification. this solution has already become standard in neighboring disciplines like linguistics and psychology. in regression models, bias variables and their covariates form the independent variables, and the predictions of nlp systems for corresponding instances constitute the dependent variable of the equation. as the last column in our concrete contributions are as follows: ## related work this section sketches the state of the art in bias analysis, more comprehensive reviews are provided by @xcite , @xcite and @xcite . ## bias identification with regression models: a workflow following the discussion in the previous sections, the task of ("in vitro") bias identification is to establish that a bias variable -in contrast to other covariates which act as confounders -is primarily responsible for systematic variance in an observed variable, namely the performance of some computer system. this is, of course, a very general task that arises in many empirical fields. a prominent family of techniques to address this task is matching @xcite , which aims at generating two datasets that differ in the bias variable, but are as close as possible in their distribution over the covariates, so that any difference between the two datasets can be attributed to the bias variable. matching is widely used in social sciences, economy, and medicine and many specific methods exist; see @xcite for an overview. 1 importantly, matching takes place a priori, before the experiment is carried out. this poses two challenges for applications in natural language processing: (a), dataset creation is dependent on the selection of covariates, so that it is not possible to assess the impact of new covariates on existing datasets without loss of comparability; (b), matching samples from the set of all datapoints, creating controlled rather than natural datasets, which may conflict with the desideratum of estimating model performance in broad-coverage scenarios. the alternative is to carry out a post-hoc analysis that assesses the effects of the various covariates. the intuition is to start from a simple pairwise comparison of two levels of a bias variable (cf. the first and second column in at this point, it can be whether the fundamentally linear regression models are the right tool for the job, in particular given the broad success of non-linear deep learning models in nlp over the last years. we believe that it makes sense to distinguish carefully between the task of output prediction (given language input, predict language output) on which non-linear models indeed excel and the task of performance prediction (given [meta data for an] input and a model, predict how well the model does on the input). the latter is a considerably simpler problem which permits the use of linear models, as evidenced by a number of successful studies taking this approach @xcite @xcite . this section provides a practical workflow to set up a regression model for bias analysis, shown in figure of predictors with the potential to systematically influence the predictions of the systems, (i.e., the putative bias variable and plausible confounders) and carry out a regression analysis. next, step 3, model validation, ensures that the regression model is well specified and interpretable. finally, step 4 utilizes effect size analysis methods to explore how much of the system predictions can be attributed to the influence of the predictors. running example. we will illustrate the steps of the workflow on an actual (non-nlp) example, namely the effect of smoking on mortality, a topic of long-running interest in public health that has been analyzed extensively with regression models. the most basic finding is that smoking, overall, causes a strong increase in mortality @xcite . why it is still reasonable to carry out a regression analysis in this case is that other lifestyle choices (alcohol consumption, diet, etc.) also presumably influence mortality, but exhibit correlations @xcite . these are sometimes surprisinge.g., @xcite found a correlation between wine and healthy diet. at the same time, approaches like matching are not applicable since the lifestyle properties of the participants cannot be influenced retroactively. ## experiment 1: emotion intensity prediction we now employ regression models to reanalyze model predictions on two experiments on standard datasets from the bias literature using the workflow defined in template 1. [per] feels [emo]. 2. the situation makes [person] feel [emo]. 3. i made [person] feel [emo]. 4. [per] made me feel [emo]. 5. [per] found herself in a [emo] situation. 6. [per] told us about the recent [emo] events. 7. the conversation with [person] was [emo]. 8. i saw [person] in the market. 9. i talked to [person] yesterday. 10. [per] goes to school in our neighborhood. 11. [per] has two children. african american european american female male female male ebony alonzo amanda adam jasmine alphonse betsy alan lakisha darnell courtney andrew latisha jamel ellen frank latoya jerome heather harry nichelle lamar katie jack shaniqua leroy kristin josh shereen malik melanie justin tanisha terrence nancy roger tia torrance stephanie ryan ## experiment 2: coreference resolution our second experiment analyzes several coreference resolvers in order to show how the logistic regression version of our approach can perform bias analysis on classification models. we choose coreference resolution as our task because of its established status in bias analysis; previous work has established that bias, in particular gender bias, is present in numerous coreference systems @xcite @xcite . at the same time, coreference resolution, as a discourse level task, is faced with more complex data than more local (i.e., sentence-level) tasks, with a correspondingly larger set of potential confounders. we re-analyze a well-known coreference resolution dataset to verify the presence of gender bias in a manner that is robust against possible covariates. ## conclusion in this article, we have argued that bias analysis, a task of major importance concerning the societal implications clearly, regression analysis is no panacea on its own: it presupposes a set of plausible covariates of bias, which can come from a wide variety of sources, including taskspecific annotation, task-unspecific input representations, or model architecture @xcite . such covariates are typically known through domain expertise or uncovered by exploratory data analysis. furthermore, the values of these bias variables must be available, or annotated, for all data points, which can represent a bottleneck. thus, regression analysis complements, but does not replace, traditional methods of bias analysis. we have demonstrated the usefulness of our approach by analyzing a range of model architectures on a regression task and a classification task, obtaining model-level results that are in line with the existing literature, e.g., bert-based systems appear to exhibit comparatively little bias @xcite . in addition, adding predictor-level analysis offers a richer understanding of the importance of the bias variables and their interactions with other textual properties. note that we only considered datasets specifically designed to exhibit the effects of a single bias variable. we believe that the benefits of our analysis framework would be even clearer on more naturalistic datasets where pairwise hypothesis tests become even more problematic (see, e.g., @xcite ). another methodological debate that we hope to contribute to is what constitutes a 'substantial' bias? we have argued that effect sizes offer a statistically sound approach to measuring the amount of variation in the output that can be attributes to a set of input properties. our study provides a starting point for the community to establish a magnitude for what it considers a 'substantial' bias, similar to the often-used thresholds for inter-annotator agreement @xcite or general effect sizes in psychology @xcite . regarding future work, one avenue concerns richer regression models that analyze interactions among predictors. such interactions, when properly motivated, can further improve our understanding of the performance data. in fact, our last example in exp. 2 essentially demonstrates an interaction: the degree of gender bias in the conference resolvers is affected by an interaction between the position of the incorrect and the correct antecedents. ideally, such observations might serve as motivation for assessing and potentially modifying model architectures or training regimens. another avenue of future research is widening our scope from the analysis of bias in nlp models (that is, "in vitro" bias according to our terminology in section 1) to real life "in vivo" bias in academic communities. recent studies have identified multiple such biases, e.g., gender bias in publications @xcite and hiring @xcite . we would hope that the application of robust regression analysis, a standard method in the social sciences, would help bolstering these studies and contribute towards redressing such social harms.