Benchmarking Intersectional Biases in NLP (2022)

Abstract: There has been a recent wave of work assessing the fairness of machine learning models in general, and more specifically, on natural language processing (NLP) models built using machine learning techniques. While much work has highlighted biases embedded in state-of-the-art language models, and more recent efforts have focused on how to debias, research assessing the fairness and performance of biased/debiased models on downstream prediction tasks has been limited. Moreover, most prior work has emphasized bias along a single dimension such as gender or race. In this work, we benchmark multiple NLP models with regards to their fairness and predictive performance across a variety of NLP tasks. In particular, we assess intersectional bias - fairness across multiple demographic dimensions. The results show that while current debiasing strategies fare well in terms of the fairness-accuracy trade-off (generally preserving predictive power in debiased models), they are unable to effectively alleviate bias in downstream tasks. Furthermore, this bias is often amplified across dimensions (i.e., intersections). We conclude by highlighting possible causes and making recommendations for future NLP debiasing research.

## introduction as state-of-the-art natural language processing (nlp) language models become increasingly powerful and pervasive, recent progress in nlp has underscored the need for deeper analyses of how such models perform with respect to underrepresented groups. research on fairness in nlp has shown that distributed representations of words often encode stereotypes -particularly towards different demographic groups @xcite . there is a growing stream of research that looks at mitigating these biases, especially when it manifests in the learned embedding state @xcite @xcite . while prior work has undoubtedly moved the needle, recent surveys and research articles have identified several important gaps and issues @xcite . first, much of the current work on examining nlp bias (and proposing debiasing strategies) has focused on representational harm -how a model describes certain groups, including stereotyping and other misrepresentations @xcite . conversely, there has been far less work exploring allocational harm in downstream nlp prediction tasks -when a system distributes resources or opportunities differently @xcite . downstream tasks, such as sequence classification, also affect underrepresented groups, as these models show disparate impact on various demographic subsets, including women, african americans, and the elderly @xcite @xcite . second, there has been limited work that examines intersectional bias across a wide array of relevant charactersitics, including several demographic dimensions, for a variety of non-debiased and debiased embeddings, foot_0 on a multitude of downstream tasks. some work has studied demographic intersections such as young men and old women from a theoretical perspective (e.g., @xcite . other recent studies have empirically shown that the biases inherent in language models for gender and race intersections might exceed those observed for gender and race alone @xcite , and that only debiasing along a single dimension can be problematic @xcite . based on these two gaps, there is a need for a more systematic analysis of how current state-ofthe-art language models and mitigation strategies perform with regards to intersectional bias in down-figure 1 : overview of our fairness benchmarking analyses. we benchmark performance across datasets, models, and debiasing methods for tasks involving multiple demographic variables. stream tasks. accordingly, in this study we perform a broad benchmark analysis of intersectional bias (figure 1 ) encompassing the following key characteristics: having three or more dimensions on many of the tasks affords opportunities to examine bias for various demographic intersection subgroups in a more in-depth manner. on four of the datasets, these demographics are selfreported as opposed to being algorithmically or heuristically inferred -an important consideration for debiasing research. our results show that existing debiasing methods are generally very adept at preserving predictive power in downstream tasks. however, their ability to mitigate intersectional bias in such tasks is limited. in general, debiasing bert/roberta only incrementally alleviates disparate impact of model classifications. further, while gender bias alone has disparate impact rates of 5-10% or less on most tasks, the range of bias is amplified for intersections -with unfairness rates often being 20 to 50% higher. on tasks such as inferring personality traits, literacy, or numeracy of users, these debiased models are still outside the fairness ranges recommended by governing bodies @xcite . interestingly, these biases are more pronounced in models using glove, suggesting that debiased transformer-based models generally have better predictive power, and are fairer. our main contributions are two-fold. first, we perform a large-scale examination of intersectional bias across an array of downstream tasks. our benchmark evaluation offers empirical evidence that the concerns voiced in recent critical surveys about too much emphasis on representational debiasing devoid of explicit normative goals @xcite , relative to mitigation of downstream allocational harm, are well-founded. second, we quantify the size and scope of the intersectional bias problem, and the risks it can introduce for select underprivileged sub-groups when deploying nlp models for sequence prediction tasks. we are hopeful our work will spur future research that further sheds light on intersectional biases in down-stream tasks, as well as mitigation strategies for alleviating allocational harm. towards this goal, the code and data used in this work is publicly available via github. foot_1 2 related work ## results and discussion figure 2 shows the adi results for bert and glove using contexted and worded for debiasing, respectively. in most cases, particularly for bert, disparate impact scores for gender alone are in a reasonable range (within 10%). for glove, we do observe high gender adi on anxiety and thinking. however, as the number of demographics under consideration grows, the range of adi scores widens. while debiasing the word embeddings typically helps to reduce the unfairness for the target demographic (e.g., gender), in the intersectional cases the model still performs poorly. there are similar trends in fv scores as the number of demographics increases, with the extent of violations often increasing by a factor of 3x to 10x as intersections increase ( in some cases the intersectional disparities are extreme. on the bert models, the ratio of positive numeracy predictions for the protected class is three-to-one compared to the privileged class. in the other direction, for 3-demographics, hatespeech detection positive predictions are significantly less likely for the protected group than the privileged group. this is consistent with prior hatespeech detection work that has shown large (absolute value) fairness gaps between protected and privileged groups (e.g., @xcite . in most cases, trends are consistent between the bert and glove models (e.g., extraverted, numeracy, perceiving). some counterexamples are the trust and anxiety tasks. here model choice impacts the direction of bias. as more demographics are considered, the glove model skews more unfair against the protected group, while the bert model remains mostly fair, skewing slightly unfair against the privileged group. higher trust in physicians is associated with better well-being and lower anxiety when visiting a doctor (netemeyer et al., 3 ). these findings on the anxiety, literacy, numeracy, and trust tasks suggest that debiasing at the dataset, embedding pretraining, and post-tuning levels leads to similar increases in unfairness as the number of demographic intersections considered increases. collectively, the results underscore the allocational harm implications of nlp models on several downstream tasks -ones that even well-designed and well-intentioned debiasing strategies cannot overcome. this can be problematic in the era of personalized marketing and precision health, with nlp-based persona-generation playing a bigger role. for tasks like numeracy and literacy, this can affect how a patient is treated by a medical staff during a hospital visit (i.e., a false positive high literacy prediction for a person who has trouble understanding his or her medical record). for the personality indicators, inconsistent predictions may lead to biased decisions in the workplace (e.g., a manager looking to form a team of extroverts). ## conclusion in this work we present a comprehensive benchmarking analysis of fairness for sequence prediction models. we also look at known debiasing methods for these models and show that while the debiased versions maintain predictive performance (as expected), they do not help with mitigating biases. while most models are relatively fair when looking at a single demographic characteristic, accounting for intersectional groups leads to less fair models and wider ranges of bias because of the