Occupational Biases in N orwegian and Multilingual Language Models (2022)

Abstract: In this paper we explore how a demographic distribution of occupations, along gender dimensions, is reflected in pre-trained language models. We give a descriptive assessment of the distribution of occupations, and investigate to what extent these are reflected in four Norwegian and two multilingual models. To this end, we introduce a set of simple bias probes, and perform five different tasks combining gendered pronouns, first names, and a set of occupations from the Norwegian statistics bureau. We show that language specific models obtain more accurate results, and are much closer to the real-world distribution of clearly gendered occupations. However, we see that none of the models have correct representations of the occupations that are demographically balanced between genders. We also discuss the importance of the training data on which the models were trained on, and argue that template-based bias probes can sometimes be fragile, and a simple alteration in a template can change a model’s behavior.

## introduction measuring the presence of stereotypical representations of occupations in pre-trained language models has been an important effort in combating and reducing possible representational harms @xcite . however, and as pointed out by @xcite , most of the current work is motivated by an idealised vision of the world where occupations should not be correlated with genders, and where the expectations are that models should not be stereotypical when e.g., predicting female or male pronouns in relation to occupations. the idea that we are all equal is an important factor in our quest of reaching fair and less biased models, and reflect our normative judgments. while this is true for most stereotypes, it might not directly apply to occupations. with a descriptive and realistic view of the society, there clearly exists gender disparities in occupations. this is inherently tied to many societal constructs and cultural backgrounds, and are a reality for many occupations. also pointed out by @xcite , the importance of the connection between language and social hierarchies, has not been considered in most previous work on bias in nlp. it is a reality that most norwegian nurses are females. having a model reflecting this reality might not be problematic per se, but using this disparity to for example systematically reject male applicants to a nurse position is a very harmful effect. in this paper, we investigate how the real-world norwegian demographic distribution of occupations, along the two gender dimensions male versus female, is reflected in large transformer-based pre-trained language models. we give a descriptive assessment of the distribution of occupations, and investigate to what extent these are reflected in four pre-trained norwegian and two multilingual models. more precisely, we focus on the following research questions: to address these questions, we investigate the correlations of occupations with norwegian gendered pronouns and names. we analyse five template-based tasks, and compare the outputs of the models to real-world norwegian demographic distributions of occupations by genders. after first providing a bias statement in section 2, we give an overview of previous relevant work in section 3. section 4 describes our experimental setup, and outlines our template-based tasks. we present and discuss our main results and findings in section 5 and 6. we conclude with a summary of our work, and discuss our future plans in section 7. ## bias statement we follow the bias definition of @xcite , where bias is defined as the cases where automated systems exhibit a systematic discrimination against, and unfairly process, a certain group of individuals. in our case, we see this as reflected in large pre-trained language models and how they can contain skewed gendered representations that can be systematically unfair if this bias is not uncovered and properly taken into account in downstream applications. another definition of bias that we rely on is that of @xcite , where bias is defined as the discrepancy between the distribution of predicted and ideal outcomes of a model. we focus on the associations between gendered (female and male) pronouns/names and professional occupations. we investigate to what degree pre-trained language models systematically associate specific genders with given occupations. however, we explore this from the perspective of a descriptive assessment: instead of expecting the system to treat genders equally, we compare how these gender-occupation representations reflect the actual and current norwegian demographics. this will in no way reduce the representational harms of stereotypical female and male occupations, that could both be propagated and exaggerated by downstream tasks, but would rather shed light on which occupations are falsely represented by such models. moreover, our work will provide knowledge about the biases contained in these models that may be important to take into account when choosing a model for a specific application. arguably, a limitation of our work is that we are only able to evaluate correlations between occupations and the binary gender categories male/female, although we acknowledge the fact that gender as an identity spans a wider spectrum than this. ## background and related work training data in nlp tasks may contain various types of bias that can be inherited by the models we train @xcite , and that may potentially lead to unintended and undesired effects when deployed @xcite . the bias can stem from the unlabeled texts used for pretraining of language models (lms), or from the language or the label distribution used for tuning a downstream classifier. since lms are now the backbone of most nlp model architectures, the extent to which they reflect, amplify, and spread the biases existing in the input data is very important for the further development of such models, and the understanding of their possible harmful outcomes. efforts so far have shown a multitude of biases in pre-trained lms and contextualized embeddings. @xcite show that pre-training the lm bert @xcite on a medical corpus propagates harmful correlations between genders, ethnicity, and insurance groups. hutchinson et al. (2020) show that english lms contain biases against disabilities, where persons with disabilities are correlated with negative sentiment words, and mental illness too frequently co-occur with homelessness and drug addictions. both @xcite and @xcite show that elmo @xcite ) contains, and even amplifies gender bias. especially, @xcite discuss the differences of contextualized and noncontextualized embeddings, and which types of gender bias are mitigated and which ones are amplified. most work on detecting gender bias has focused on template-based approaches. these templates are simple sentences of the form "[pronoun] is a [description]", where a description could be anything from nouns referring to occupations, to adjectives referring to sentiment, emotions, or traits @xcite @xcite @xcite . @xcite investigate the propagation of gender biases of bert in five downstream tasks within emotion and sentiment prediction. they propose an approach to identify gender directions for each bert layer, and use the equity evaluation corpus @xcite as an evaluation of their approach. they show that their approach can reduce some of the biases in downstream tasks. @xcite also use a template-and lexicon-based approach, in this case for sentence completion. they introduce a dataset for the six languages english, french, italian, portuguese, romanian, and spanish, and show that lms both reproduce and amplify gender-related societal stereotypes. winograd schemas data @xcite . this dataset was developed for the task of coreference resolution, and contains a set of manually annotated templates that requires commonsense reasoning about coreference. it is used to explore the existence of biases in coreference resolution systems, by measuring the dependence of the system on gendered pronouns along stereotypical and nonstereotypical gender associations with occupations. similarly, the winobias @xcite dataset focuses on the relationship between gendered pronouns and stereotypical occupations, and is used to explore the existing stereotypes in models. the winogender dataset @xcite also contains sentences focusing on the relationship between pronouns, persons, and occupations. here, they also include gender-neutral pronouns. unlike winobias, winogender's sentences are built such that there is a coreference between pronouns and occupations, and between the same pronouns and persons. based on these datasets for coreference resolution, winomt @xcite has been developed for the task of machine translation. the dataset also contains stereotypical and nonstereotypical templates used to investigate gender bias in machine translation systems. moreover, @xcite point out the dangers of lms and how they can potentially amplify the already existing biases that occur in the data they were trained on. they highlight the importance of understanding the harmful consequences of carelessly using such models in language processing, and how they in particular can hurt minorities. they also discuss the difficulty of identifying such biases, and how complicated it can be to tackle them. this is partly due to poor framework definitions, i.e., how culturally specific they are, but also how unreliable current bias evaluation methods are. we focus therefore in this work on investigating how culturally specific norwegian demographics related to gender and occupations are reflected in four norwegian and two multilingual pre-trained lms. our work differs from previous work in that we ground our bias probes to real-world distributions of gender, and rather than expecting the models to always have a balanced representation of genders, we explore to which degree they reflect true demographics. ## experimental setup following the methodology of previous research on gender bias in pre-trained language models, and due to the corresponding lack of resources for norwegian, we generate our own set of templates that we use with the pre-trained language models to make use of their ability to compute the probabilities of words and sentences. we present an empirical analysis of gender biases towards occupational associations. by using the templates we hope to reduce variation by keeping the semantic structure of the sentence. we analyze the probability distributions of returned pronouns, occupations, and first names; and compare them to real-world gold data representing the demographic distribution in norway. investigating the differences between the models can also give us insights into the content of the various types of corpora they were trained on. data and codes will be made available foot_0 . below we discuss in turn (i) the gold reference distribution of occupations and genders, (ii) the templates, (iii) how the templates are used for probing pre-trained language models, and finally (iv) the models that we test. reference distribution we use a set of 418 occupations. these represent the demographic distribution of females and males in the respective occupations in norway 2 originating from the norwegian statistics bureau. the bureau releases yearly statistics covering various aspects of the norwegian society, and all data is made freely available. this list comprises a fine-grained level of occupations, where e.g., lege (doctor) and allmennlege (general practitioner) are considered two different occupations. the gender-to-occupation ratios in these statistics are used as 'gold standard' when probing the models. in templates our templates combine occupations, pronouns, and first names. we focus on five template-based tasks, and generate the following corresponding templates that we use as bias probes @xcite ): 1. task1: [pronoun] is a/an [occupation] (original: [pronoun] er [occupation]) 2. task2: [pronoun] works as a/an [occupation] (original: [pronoun] jobber som [occupation]) 3. task3: [name] is a/an [occupation] (original: [name] er [occupation]) 4. task4: [name] works as a/an [occupation] (original: [name] jobber som [occupation]) 5. task5: the [occupation] [name] (original: [occupation] [name]) as pronouns, our work mainly focuses on hun and han (she and he respectively). as demographic statistics are still made using a binary gender distribution, we could not include the gender neutral 2 https://utdanning.no/likestilling pronoun hen (they), which is, in addition, rarely used in norway. as first names, we also extract from the norwegian statistics bureau foot_1 the 10 most frequent female and male names in norway from 1880 to 2021, this results in 90 female names and 71 male names. for tasks 1-4 we use the full set of 418 occupations, while in task 5 we focus on those that either have a balanced distribution between genders or are clearly female-or male-dominated. this was decided after an analysis of the distribution of occupations across genders, and resulted in two thresholds. all occupations that had between 0 and 10% differences in distribution, were deemed balanced (e.g., 51% female and 49% male). all occupations that had more than 75% distribution of one gender against the other, were deemed unbalanced, and are referred to as either clearly female (≥75%) or clearly male (≥75%) occupations. this resulted in a set of 31 clearly female occupations, 106 clearly male occupations, and 49 balanced occupations. for tasks 1 and 2, we mask the pronouns and compute the probability distribution across the occupations for female and male pronouns. for tasks 3, 4, and 5, we mask the occupations and compute the probability distributions in each bias-probe. masking pronouns will allow us to uncover how likely a gendered pronoun is correlated with an occupation, and masking the occupation will allow us to uncover how likely occupations are correlated with female and male names. probing and evaluation for each task, we first generate the probability distributions of masked tokens in each bias probe. in order to have a comparable distribution to the gold standard (which is given as a percentage), we compute a simple percentage representation of the probability distributions by following the following formula: where f_pron% is the percentage of a female pronoun, and prob x_pron is the output probability of each model for each of the female and male pronouns. the same simple formula is used in all tasks. we are aware that this is a simplified representation of the output of each model, nevertheless, we believe that it will not change the overall distribution. once probability distributions are mapped to per-centages, we quantify the difference between female and male scores by simply subtracting the scores of males from the scores of female. positive values will represent occupations that are more strongly associated with females than males by the model, and negative values represent the opposite. this is also applied to the gold standard data. we use the demographic distribution of the occupations from the norwegian statistics bureau as gold data. based on this, values greater than 0 are deemed female-dominated occupations, and values lower that 0 are male-dominated occupation. this is used to compute the macro f1 values for each model. pre-trained language models we analyse the predictions of six pre-trained language models, four norwegian and two multilingual. note that norwegian has two official written standards; bokmål (literally 'book tongue') and nynorsk (literally 'new norwegian'). while bokmål is the main variety, roughly 15% of the norwegian population write in the nynorsk variant. all the norwegian models are trained on data comprising both bokmål and nynorsk. as can be seen above, each model has been trained on different types of corpora, and are all of various sizes. the ncc corpus, is a collection of ocr-scanned documents from the norwegian library's collection of newspapers and works of fiction (with publishing years ranging from early 1800s to present day), government reports, parliament collections, ocr public reports, legal resources such as laws, as well as norwegian wikipedia. in short, some models are trained on well structured texts, that follow a somewhat formal style, while other models also include less structured texts in the form of online content. ## results ## discussion one of our main observations is that models behave differently based on the template used as bias probe. the templates we have used, in e.g., task1 and task2, and task3 and task4, differ only by one token, and do not change the semantics of the template even if it changes its syntactic realization. this might both be due to the input data on which the models have been trained on, but can also be a manifestation of the fragility of the template-based approach. while these types of approaches do shed light on the inner representations of models, it is difficult to point out why exactly a subtle change in the expression of a template can seemingly alter a model's representation. another interesting observation, is that languagespecific models seem to be better at identifying the clearly unbalanced occupations, that demographically are clearly female or male occupations. while both language-specific and multilingual models are not able to correctly represent gender-balanced occupations. this in turn of course, indicates that these models do contain bias, and mostly map gender-balanced occupations as male-dominated ones. to give a simple example of this phenomenon, we show in figure 6 a couple of handpicked examples of demographically balanced and unbalanced occupations from our gold data for the first task, task1: [pronoun] is a/an @xcite . we compare these realworld representations to those of each of the four norwegian and two multilingual models. the occupations with positive values in gold (green bar, first to the left in each group) are female-dominated occupations, and occupations with negative values are male-dominated occupations. as previously mentioned, occupations with values [-10, +10] in gold are deemed to be genderbalanced occupations. in figure 6 , the occupations diplomat, doctor, associate professor, and judge are demographically gender-balanced occupations in norway. the occupations midwife, secretary, and nurse are female-dominated, and the occupations pilot, plumber, and bricklayer are maledominated. as can be seen from the figure, all four norwegian models are very good at representing the clearly female-and male-dominated occupations (with the exception of norbert2 for secretary). the same holds for the multilingual models, except for mbert for nurse, and xlm-roberta for bricklayer. when it comes to gender-balanced occupations, it is quite clear from figure 6 that all models fail to predict probabilities near the real demographic distribution. however, norbert gives the closest distribution for the two occupations diplomat and associate professor, while for doctor, it is the two multilingual models and mbert and xlm-roberta that give the closest distribution. ## conclusion we have presented in this paper an investigation into how a demographic distribution of occupations, along two gender dimensions, is reflected in pre-trained language models. the demographic distribution is a real-world representation from the norwegian statistics bureau. instead of giving a normative analysis of biases, we give a descriptive assessment of the distribution of occupations, and investigate how these are reflected in four norwegian and two multilingual language models. we have generated simple bias probes for five different tasks combining pronouns and occupations, and first names and occupations. our main observations are that norwegian language-specific models give closer results to the real-world distribution of clearly gendered occupations. moreover, all models, language-specific and multilingual, have a biased representation of gender-balanced occupations. our investigations also show the fragility of template-based approaches, and the importance of the models' training data. in future work, we plan to extend our investigations and include several demographic distributions from other countries, and compare them to their respective language-specific pre-trained language models to corroborate our findings.