Gender Bias in BERT - Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task (2022)

Abstract: Pretrained language models are publicly available and constantly finetuned for various real-life applications. As they become capable of grasping complex contextual information, harmful biases are likely increasingly intertwined with those models. This paper analyses gender bias in BERT models with two main contributions: First, a novel bias measure is introduced, defining biases as the difference in sentiment valuation of female and male sample versions. Second, we comprehensively analyse BERT?s biases on the example of a realistic IMDB movie classifier. By systematically varying elements of the training pipeline, we can conclude regarding their impact on the final model bias. Seven different public BERT models in nine training conditions, i.e. 63 models in total, are compared. Almost all conditions yield significant gender biases. Results indicate that reflected biases stem from public BERT models rather than task-specific data, emphasising the weight of responsible usage.

## introduction as complex machine learning (ml) based systems are nowadays naturally intertwined with media, technology and everyday life, it is increasingly important to understand their nature and be aware of unwanted behaviour. this also applies to the natural language processing (nlp) community, where several recent breakthroughs promoted the application of sophisticated data-driven models in various tasks and applications. only a decade ago, mlbased vector space word embeddings as word2vec @xcite or glove @xcite emerged and opened up new ways to extract information and correlations from large amounts of text data. in this context, it has widely been shown that embeddings tend to reflect human biases and stereotypes @xcite and that unintended imbalances in text-embeddings figure 1 : example sample masking. the original review contains both male (orange) and female (green) terms. in masked versions, all terms are homogeneously male or female. the present study presents a comprehensive analysis of gender bias in bert models in a downstream sentiment classification task with imdb data. this task is a realistic scenario, as ml-based recommendation systems are widely used, and reflected stereotypes could directly harm people, e.g. by underrating certain movies or impairing their visibility. the investigation comprises two main contributions: first, we propose a novel method to calculate biases in sentiment classification tasks. sentiment classifiers inherently possess valuation abilities. we exploited these to rate "female" and "male" sample versions (see fig. 1 ) and therefore need no additional association dimension, e.g. occupation. the classifier is biased if one gender is preferred over the other. second, we analyse the impact of different training factors on the final classifier bias to better understand the origin of biases in nlp tasks. seven different base models, three different training data conditions and three different bias implementations lead to a total num-ber of 63 compared classifiers. additional observations could be made regarding training hyperparameters and model accuracies. results reveal significant gender biases in almost all experimental conditions. compensating imbalances of gender terms in finetuning training data did not show any considerable effect. the size and architecture of pretrained models, in contrast, correlate with the biases. these observations indicate that classifier biases are more likely to stem from the public bert models than from task-specific data and emphasise the importance of selecting trustworthy resources. the present work contributes to understanding biases in language models and how they propagate through complex machine learning systems. ## methodology this investigation analyses to what extent bert gender biases are present in an imdb sentiment classification task. we aim to observe what portion of bias emerges in which experimental step by systematically varying conditions in each step. 63 different classifiers and their biases are finally reported in this paper. many more were trained to observe different training aspects. this section provides a detailed description of experimental steps and how different conditions are achieved. the experimental pipeline can be divided into four major steps, as illustrated in fig. 2 . the structure of this section roughly follows these steps. first, the preparation of training data is described in sec. 2.1. we compare seven training conditions where gender information in training data is removed or balanced. by that means, it can be measured how much bias is induced during the taskspecific finetuning. second, the sentiment classifiers were trained by finetuning seven different common bert models, as can be read in sec. 2.2. by observing whether the choice of model affects the bias magnitude, we can infer how much bias stems from the pretrained bert model. also, we compare different sizes of the same architecture. in the third step, the trained classifiers were applied to rate the manipulated test data, which is here referred to as experimental data. the setup is described in sec. 2.3. finally, these ratings are used to calculate the model bias that is defined contextually in sec. 1 and mathematically in sec. 2.4. three different sets of gender terms were considered in the experiments. ## results a condensed list of absolute and total biases is reported in tab. 1. out of 63 reported experimental models, 57 showed highly significant biases. exceptions are distbase mix-all, bertbase mix-pro, robertbase mix-weat, robertbase mix-all, and albertlarge mix-weat. 16 classifiers prefer female terms over male terms, and 41 prefer male terms over female terms. thus, even though more classifiers are in our definition discriminating against women than against men, biases are directed differently. the sizes and especially the directions of biases are visualised in fig. 3 for distilbert classifiers, in fig. 4 for all other architectures. is bias induced by model finetuning? in finetuning systems like this, biases in models can have different origins. we aimed to analyse how much bias was introduced during further training by the task-specific data. to this end, we removed (r) or balanced (mix) gender terms in the task-specific data to reduce the modification of their representations by finetuning. both conditions are represented in fig. 3 and fig. 4 by red and green bars, respectively. although biases are decreasing by removing gender information from imdb data in some cases, e.g. albertalarge pro , there are likewise examples where it seems to have the opposite effect, such as bertlarge weat mix. however, for most conditions, these preprocessing measures do not change the magnitude of biases considerably. especially, removing the gender terms from data does not significantly affect the biases. for some models, although the behaviour of the mix conditions is different from the other settings, there is no clear pattern observable. observed differences in that category might also be related to the doubled size of training sets (@xmath0, which is likely to reinforce effects. is bias induced by pretrained models? we applied models with different architectures and sizes to observe how measured biases depend on the underlying pretrained model. we compare three different sizes of bert models, which are distbase, bertbase and bertlarge. moreover, we consider models with roberta architecture in the sizes robertabase and robertalarge and albert in albertbase and albertlarge. this comparison leads to two major observations: first, biases differ steadily between considered architectures. as can be well observed in fig. 3 and fig. 4 , distilbert's biases are about half as big as bert's and roberta's biases. alberta's biases, again, are about twice as big as those of bert and roberta. this observation does not only hold among all training conditions but also for both base and large variants. thus, the architecture of a selected model has an essential impact on the biases of downstream systems. second, we observe increasing biases depending on model sizes within one architecture. distbase again yielded the smallest biases, followed by bertbase, and bertlarge. simultaneously, robertalarge yielded much bigger biases than robertabase, and albertalarge yielded much bigger biases than albertbase. thus, we observe a correlation between bias and model size, i.e. the number of layers. this indicates that larger models tend to encode greater gender biases. is bias dependent on applied term sets? as mentioned before, we defined three sets of target terms for the implementation of bias, of which the largest comprises more than a three hundred term pairs and the smallest only five. analogously to term set sizes, the absolute biases are the smallest for the pronoun set and the largest for the all set in almost all conditions. in other words, the more terms are included, the bigger the measured bias. the only exception is bertlarge r and some conditions on albertbase. despite the differences in bias magnitude, measured values in all categories were similarly significant. also, the patterns of effects of training data manipulation or base model comparison could similarly be observed in all three bias definitions. we conclude from these observations that all types of included vocabulary encode biases, i.e. pronouns, weat-terms and other nouns. the more terms are included, the higher the measured bias. for the presented results, the term set of training data manipulating and the term set for bias measure were always the same. for instance, if we applied the pro set to measure biases, we also only removed/ balanced terms of pro in the training data. we also tested whether biases vary when mixing term sets between different experimental steps. however, that did not reveal any considerable effect, as bias values differed marginally. do hyperparameter settings affect biases? due to computational capacity, some larger models needed to be trained with smaller batch sizes. to see if that affects the final biased, we performed additional experiments where we only varied the batch size while fixing all other parameters. for 21 different experimental conditions, models were retrained with batch sizes 32, 16 and 8. naturally, this affected the course of loss and accuracy during training, but only to a limited extent. all settings led to stable classifiers with convenient model accuracy. the biases of all tested classifiers did not show any indication to be different among the training batch sizes. these results reveal that the batch size does not immediately cause the measured correlation. tab. 2 reports correlations between further basic training details and biases to examine whether there are observable connections. the f-score naturally correlates with accuracy, which is the highest value in the table. f1 and accuracy yield a medium negative correlation with absolute biases. in contrast to absolute biases, total biases barely show significant correlations with training values. all considered classifiers showed good performance in the model evaluation. test accuracies lie between 77% and 84%, which is comparable to baseline values. the evaluation details of all classifiers are attached to the appendix. ## discussion we observed highly significant gender biases in almost all tested conditions. thus, the present results verify the hypothesis that downstream sentiment classification tasks reflect gender biases. although most considered classifiers prefer male samples over female ones, this direction is not consistent: about thirty per cent of classifiers prefer female over male samples. the high significance values are likely to be facilitated by the large sample number and do not necessarily correspond to the effect size. it might be insightful to analyse the contexts and types of individual samples to understand how these contrary directions occur. the rating of male and female presence likely depends on the scenario, rather than one gender being strictly advantaged. we could not observe any effect of removing gender information from task-specific training data. thus, in the present case, the biases associated with gender terms are most likely not learnt during finetuning. in contrast, results showed significant differences in downstream classifier biases depending on the selection of pretrained models. this is true for both the size and the architecture of the model. it is reasonable that the pretrained bert models, which comprise information from a training set much larger than the imdb set, are more capable of reflecting complex constructs such as gender stereotypes. it is, therefore, all the more important to develop these models carefully and responsibly and to respect risks and limitations in the application. as a small number of provided base models form the basis for a large portion of applications in nlp, it is especially critical to understand included risks and facilitate debiasing. although the results of the present investigation indicate the origin of biases in pretrained bert models, that does not preclude the risk to generate biases during finetuning. all elements of the development pipeline need to be audited adequately. we showed that all of the compared term sets are generally appropriate to measure gender bias. however, term sets yielded large differences in bias sizes, showing how crucial the experimental setup is for the validity of measured results. the fact that biases increase relative to the number of gender terms strengthens the conclusion that the majority of these terms reflect biases. it also needs to be further investigated whether the sentiment rating of individual gender terms might be affected by other factors than gender. nevertheless, the applied definition of male and female biases is a rudimentary implementation of real-world circumstances. first, there is a large number of facets that possibly encode gender @xcite , e.g. names or topics. second, gender is much more diverse in reality than this implementation can reflect. especially since modern language models are contextual, conceptual stereotypes and biases are likely to be deeply encoded in the embeddings. automatically learnt models likely cover a large variety of latent biases that contemporary research cannot grasp @xcite . this investigation underlines the complexity of bias formation in real-life multi-level systems. results verify the existence of gender biases in bert's downstream sentiment classification tasks. in order to further analyse how much of the final system bias stems from the pretrained model, similar experiments could be conducted on debiased bert models. this way, whether the bias can be further reduced could be tested. another exciting direction might be to examine how the suggested measurement approach could be transferred to non-binary classification tasks. as a next step, we plan to expand the present experiments to further downstream applications. ## related work language models as bert @xcite recently became the new standard in a wide variety of different tasks and superseded static embeddings, as word2vec @xcite or glove @xcite . for these older embeddings, there already is a huge body of empirical research on bias measuring and mitigation @xcite @xcite , which unfortunately seem to be not straightforwardly tailorable to the new setting @xcite . however, recent research finds that bert also encodes unwanted human biases, such as gender bias @xcite @xcite . downstream task analyses mostly consider shortcomings in dialogue-systems @xcite . in the context of sentiment analysis, @xcite introduced a data set that is designed to measure gender-occupation biases. although the reported results across 219 tested systems are ambiguous, the framework has been frequently applied ever since @xcite . @xcite measure biases in text-generation systems, i.e. gpt. while the general experimental setting is fundamentally different from the present investigation, they apply a similar idea of measuring biases via sentiment classification. to the best of our knowledge, we are the first to utilise sentiment classification to learn about the origin of biases in bert. we contribute to a growing body of exploratory literature regarding bias measure @xcite @xcite and bias mitigation @xcite in contextualised language models. ## conclusion contextualised language models such as bert form the backbone of many everyday applications. we introduced a novel approach to measuring bias sentiment classification systems and comprehensively analysed the reflection of gender bias in a realistic downstream sentiment classification task. we compared 63 classifier settings, covering multiple pretrained models and different training conditions. all trained classifiers showed highly significant gender biases. results indicate that biases are rather propagated from underlying pretrained bert models than learnt in task-specific training. pretrained models should not be applied blindly for downstream tasks as they indeed reflect harmful imbalances and stereotypes. just as gender-neutral language is important to mitigate everyday discrimination holistically, it is critical to avoid encoded biases in automated systems. we hope that the present work contributes to raising awareness of hidden biases and motivates further research on the propagation of unwanted biases through complex systems. to the best of our knowledge, there is no similar work so far that utilises the valuation capacity of sentiment classifiers to measure downstream biases. yukun zhu, ryan kiros, rich zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in the ieee international conference on computer vision (iccv).