Stubborn Lexical Bias in Data and Models (2023)

Abstract: In NLP, recent work has seen increased focus on spurious correlations between various features and labels in training data, and how these influence model behavior. However, the presence and effect of such correlations are typically examined feature by feature. We investigate the cumulative impact on a model of many such intersecting features. Using a new statistical method, we examine whether such spurious patterns in data appear in models trained on the data. We select two tasks— natural language inference and duplicate-question detection— for which any unigram feature on its own should ideally be uninformative, which gives us a large pool of automatically extracted features with which to experiment. The large size of this pool allows us to investigate the intersection of features spuriously associated with (potentially different) labels. We then apply an optimization approach to *reweight* the training data, reducing thousands of spurious correlations, and examine how doing so affects models trained on the reweighted data. Surprisingly, though this method can successfully reduce lexical biases in the training data, we still find strong evidence of corresponding bias in the trained models, including worsened bias for slightly more complex features (bigrams). We close with discussion about the implications of our results on what it means to “debias” training data, and how issues of data quality can affect model bias.

## introduction machine learning research today, including within nlp, is dominated by large datasets and expressive models that are able to take advantage of them. at the same time, as the scale of training data has grown, this explosion of data has come at the expense of data curation; for many of the datasets currently in use today, human oversight of the full breadth of their contents has become unrealistic. this makes it more likely that training datasets con-tain undesirable associations or shortcuts to learning intended tasks. many cases are attested (e.g., @xcite @xcite @xcite @xcite , and we suspect a vast number of these so-called "spurious correlations" remain undetected. one question is whether these unintended biases in the training data propagate to models trained on that data. recent work has found mixed results on this point @xcite . we begin by introducing an approach to testing for undesirable model biases that can operate using existing held-out data, even though that data might itself have spurious correlations. in particular, we repurpose the classic permutation test to examine whether observed differences in model performance between instances exhibiting more common feature-label pairings and those exhibiting less common feature-label pairings are statistically significant. for our experiments, we focus on the simplest kind of feature-label association: correlations between lexical features and task labels. we select two tasks (natural language inference and duplicatequestion detection) for which any such lexical feature should be uninformative on its own. finding strong evidence that models finetuned on three different datasets have at least some of the same lexical biases that exist in their training data, we then examine the extent to which those biases are mitigated by lessening biases in the training data. to do this, we apply an optimization-based approach to reweighting the training instances. the approach brings uneven label distributions closer to uniform for thousands of different intersecting lexical features, many more than we use for our model bias evaluation, and still manages to have a strong effect on the most initially biased features despite our reweighting approach not focusing on those in particular. we then finetune new models on those (reweighted) datasets. we find that although model bias lessens somewhat when we do this, we still find strong evidence of bias. surprisingly, this holds even when we consider models that make use of no pretraining data. we close with a discussion of possible factors contributing to these results. we first note that perhaps the continued relative lack of variety of minority-class examples containing certain features hinders the reweighted models' ability to generalize their recognition of those less-common featureclass pairs, even though the combined weight given to those few instances in the loss function is increased. however, when we examine the effect of our reweighting on higher-order features (namely, bigrams), we see another problem: the same reweighting that mitigates associations between unigrams and any particular label actually strengthens associations between bigrams and certain labels in data. based on this observation, we arrive at two conclusions: (1) simultaneously reducing bias across features of different levels of granularity for natural-language data is likely not feasible, and (2) even if we aim to mitigate model bias only with respect to simple features, if we do so by reweighting the data, the high-capacity models used in modern nlp are still capable of learning the spurious correlations of the original unweighted data through associations that remain encoded in more complex features even after reweighting. we conclude that bias reduction in nlp cannot be cast purely as a "data problem," and solutions may need to focus elsewhere (e.g., on models). the term "bias" is polysemous, having been adopted by different communities to mean different things, from historically rooted social inequity to skewed model evaluations @xcite to techniques that help with supervised class imbalance in labels @xcite . in our work, we use "bias" to mean correlations between individual input features and task labels. this framework is fairly general, but our focus in this work is natural language data. therefore, as an example to illustrate our definition of bias, we will refer to correlations between the presence of individual word types in the input (unigrams) and a given label in a classification task. more formally, consider a task of mapping inputs in x to labels in y. we assume a training dataset @xmath0, y i )⟩ n @xmath1, each x @xmath2. we are particularly interested in a designated collection of d binary features on x , the jth of which is denoted f j : @xmath3, 1}. for example, f j might be the presence of the word "nobody" in an instance. let f j,i be shorthand for f j (x i ) (e.g., whether instance x i contains the word "nobody" (f j ( introducing random variable notation, we can characterize d by its empirical conditional distribution over labels given each feature, such that for all @xmath4, if the conditional distribution of output labels given the presence of a particular lexical feature is very different from the overall label distribution in the data, we consider that feature to be biased in the training data. ## measuring bias in model performance and data recall that when p(@xmath5, it means feature j is correlated with label y in a given dataset. let us denote the set of examples that contain feature j and have the label most strongly associated with feature j in d by u j , which we call the "usual-labels" set. then, denote the examples that contain j but have a different label by n j , which we call the "unusual-labels" set. to build intuition, the accuracy of the model on instances which contain feature j is the accuracy over the union u j ∪ n j . however, to measure if the model is picking up bias from the data, we will measure accuracy over u j and n j separately. to maximize accuracy on u j ∪ n j the model would be justified in disproportionately labeling instances containing f j with y, so we can't use accuracy by itself to measure model bias. instead, the key idea here will be to look for differences in error rates between instances whose labels align with features' training biases (the "usual-labels" set), and instances whose labels do not. if the model has learned a biased representation of the data, we expect it to have higher accuracy on the "usual-labels" set, u j . on the other hand, if the model hasn't learned that bias, we would expect the correct predictions to be uniformly distributed between u j and n j . we use this as the basis for a hypothesis test: the null hypothesis h 0 is that the accuracy of model is the same on both sets acc(u j ) = acc(n j ), and the alternative hypothesis h 1 is that acc(u j ) > acc(n j ). that is, if the errors are distributed uniformly at random, how likely is it that u j would have at least its observed number of correct instances? ## applying the test here we shift our focus to particular tasks and datasets, in order to apply our test in practice. ## effects of rebalancing on higher-order features we have found that rebalancing labeled data doesn't remove bias in a downstream model. another possible explanation is that rebalancing also affects higher-order features' effective correlations with labels, and such bias may carry over into models (whether it was originally present or not). we consider bigrams, as they represent only a slight additional level of complication. to get a sense of how bigrams overall are affected, we randomly sample 200 bigrams for each of the three successfully rebalanced datasets, selecting uniformly at random among the set of bigrams that appear in at least one instance of each label. we then examine the effect of our (unigram-based) rebalancing of data from this analysis provides a possible explanation for how rebalancing the data with respect to biased unigram features fails to prevent models from learning bias: the rebalancing didn't correct for biased bigram features, which mislead the model, effectively "bringing the unigram features" along with them so that unigram-bias gets learned anyway. this is a troubling sign for approaches to bias reduction that focus on data alone, pointing to the need for methods that focus on other aspects of model learning as well. ## methods from related work considerable research has posed similar questions of undesirable associations in data manifesting in models, whether through spurious correlations between lexical features and labels @xcite @xcite or through gender or racial bias @xcite @xcite @xcite . out of this large body of work, a few prevailing evaluation methods have emerged. foremost among these is assembling a single test set in which a particular bias of interest is lessened and evaluating models' aggregate performance on that test set, such as by excluding instances for which a model that should be too simple to perform the task is correct @xcite or by constructing such a dataset from scratch @xcite . similarly, @xcite assemble what is essentially a new, miniature test set (a "contrast set") for each human-identified possible category of mistake that a model might make. we now consider what existing work finds regarding bias in models using these different methods. overall, we see mixed results. @xcite determine that trained word vectors do pick up societal biases from their training corpora. likewise, @xcite find evidence of gender bias in coreference resolution systems, @xcite find gender bias in machine translation systems, and @xcite find racial bias in hate speech detection models. however, whether multiple attributes' biases in data transfer to models is less clear. for example, @xcite find that both pretraining data and finetuning data have an effect on biases having to do with gendered pronouns and identity terms that are learned by occupation and toxicity classifiers, but that certain forms of bias reduction in either pretraining or fine-tuning data don't necessarily overcome bias that the model might pick up from the other. this is possibly explained by the results of zhou and srikumar (2022), who find that data used for finetuning largely distances clusters of textual representations by label without significantly changing other properties of the underlying distribution of data. in a similar vein, @xcite find that counterfactually augmented training data can actually exacerbate other spurious correlations in models. for all the different results reported in this body of literature, there are some typical characteristics of the bias evaluation methodology they apply. as referenced earlier, it is common for this work to test for a single undesirable form of behavior (e.g., biased use of gendered pronouns). for example, @xcite focus on whether nli models ignore input instances' premise, an important problem, but this also simplifies their evaluation, as they doesn't need to consider the potentially disparate impact of their adjusted model on intersecting biases. another common characteristic is the creation of new and separate test data @xcite , on which decreased performance is taken to indicate bias @xcite . a concern regarding this strategy, though, is that such test sets very likely still contain (undetected) biases of their own. due to the complicated nature of natural language and the highly intertwined features that occur together in text, it is very likely that this will be true regardless of the test set created. results using our permutation testing framework indicate the difficulty of removing or mitigating bias from data in a way that corresponds to the mechanisms by which models absorb that bias in practice. this is reminiscent of results from, for example, @xcite or @xcite , who note that certain ways of seemingly covering up bias still leave traces of that bias in models, and is in line with arguments made by, for example, @xcite and @xcite . further development and testing of hypotheses about how models acquire bias will be important to ensuring that they truly perform the tasks that we intend, and not versions that rely on biased shortcuts in the data. ## conclusion we explored how lexical bias in labeled data affects bias in models trained on that data. our method-ological contribution is a procedure, based on the permutation test, for analyzing biased associations between given features and model predictions, in test data that might itself contain biases. our empirical finding is that, in cases where a dataset can be rebalanced to remove most lexical bias, the resulting models remain biased. this may be related to our observation that the correlations of higher-order (bigram) features with labels actually get worse after rebalancing. we conclude that reducing bias in nlp models may not be achievable by altering existing training data distributions.