Measuring Normative and Descriptive Biases in Language Models Using Census Data (2023)

Abstract: We investigate in this paper how distributions of occupations with respect to gender is reflected in pre-trained language models. Such distributions are not always aligned to normative ideals, nor do they necessarily reflect a descriptive assessment of reality. In this paper, we introduce an approach for measuring to what degree pre-trained language models are aligned to normative and descriptive occupational distributions. To this end, we use official demographic information about gender–occupation distributions provided by the national statistics agencies of France, Norway, United Kingdom, and the United States. We manually generate template-based sentences combining gendered pronouns and nouns with occupations, and subsequently probe a selection of ten language models covering the English, French, and Norwegian languages. The scoring system we introduce in this work is language independent, and can be used on any combination of template-based sentences, occupations, and languages. The approach could also be extended to other dimensions of national census data and other demographic variables.

## introduction pre-trained language models (lms) may contain various types of biases, and the field of nlp has seen a lot of work in recent years on attempting to identify, mitigate, and reduce these biases. biases can originate both from the unlabeled texts used for pre-training these lms, and from texts and annotations used for tuning downstream classifiers. lms have become a cornerstone in most nlp model architectures, and the extent to which they reflect, amplify, and spread the biases present in their training data is still a problematic issue to be solved. several efforts in this direction have focused on gender as a variable @xcite @xcite @xcite @xcite @xcite , also in correlation with occupations @xcite @xcite . while there have been several efforts on exploring the existing biases related to these demographic variables, most work approaches the task from a normative point of view @xcite , where equality between the demographic distributions is prioritized. although normativity in this aspect is crucial for certain applications, we argue that it is also interesting to explore the task from a descriptive perspective. this is especially interesting for occupations, since a descriptive and realistic view of society already contains gender disparities. we propose that national census data, in our case about gender-occupation distributions, can offer a reliable ground truth against which model predictions can be compared. moreover, we argue for taking both normative and descriptive assessments into account, in order to give a broader picture of the representations of demographics within lms. this has also been partly pointed out by @xcite , who stress the importance of the connection between language and social hierarchies, which has not been taken into consideration in most previous work on bias in nlp. in this paper, we introduce a new score for measuring how lms are aligned with normative and descriptive occupational demographic distributions. we use demographic distributions covering occupations in four countries, namely france, norway, the united kingdom, and the united states. we manually select gendered pronouns and nouns, as well as specific verb phrases, to construct template-based sentences, subsequently used to probe a selection of ten lms covering the relevant languages. our contributions include; (i) creating novel benchmark datasets for english, french, and norwegian based on manually crafted templates to mea-sure occupational gender biases, (ii) proposing a scoring system to measure normative and descriptive biases in lms, and (iii) releasing our code and data for reproducibility. in what follows, we give a detailed description of our new benchmark datasets in section 2. we then, in section 3, give a detailed description of the normative and descriptive bias scores, and present our analysis on ten lms as proof of concept. we discuss and summarize our findings in section 4, and conclude by discussing possible directions for future work in section 5. the limitations of our work are discussed in the limitations section. ## benchmark datasets in this work we develop a set of benchmark templates for english, french, and norwegian that cover occupations in france, norway, the united kingdom (uk), and the united states (us). these templates are then used for probing different lms. more details are given in what follows. occupations we retrieve country-specific lists of occupations and their associated (male/female) gender ratios from the national statistics bureaus of france, norway, uk, and the us. foot_0 this resulted in 235 occupations from france, 2 415 from norway, 3 325 from the uk, 4 and finally 314 occupations from the us. 5 all of these occupations were listed in either masculine singular or masculine plural form. as some of the languages we are focusing on inflect nouns for gender, we manually generate for each occupation in singular masculine form the corresponding forms in singular feminine, plural feminine, and plural masculine. this was performed by a native speaker of norwegian and french, and a proficient speaker of english. templates our work builds on the methodology of template-based probing. to measure a model's occupational biases we follow the same procedure for all languages. our templates are based on the gender-inflected occupations, preceded by a sequence of selected gendered pronouns and a set of gender-specific identifier terms in singular and plural forms, followed by a predicate generically denoting the act of having an occupation. as an example, a template could be: the woman gender-specific identifier worked as a predicate nurse occupation we select 28 gender-specific identifiers, and 6 predicates for all three languages. the full list of gender-specific identifiers can be found in the templates we created cover different grammatical tenses, such that each template is given in the past, present, and future tense. we have decided to include such a broad collection of variations to the templates to get a better representation of how occupations are correlated with genders, especially since research has shown that bias probes are sensitive to grammatical tense @xcite . ## method lms trained with a masked language modelling objective are trained such that random tokens in the input training data are replaced with a placeholder token, [mask], which will subsequently be predicted by the trained model. template-based approaches to probe biases take advantage of this feature of lms. for our purposes, we mask the gendered identifier in each template-generated probe (as introduced in section 2), and use the returned probability of each masked identifier to compute our bias scores. a masked version of the example template above would be: the @xcite gender-specific identifier worked as a predicate nurse occupation language models we select ten lms covering the three languages english, french, and norwegian. all models are available from the hugging-face library @xcite . we use four norwegian models, four english models, and two french models. these are: scoring system the scoring system we introduce is the same for both bias scores. we will give more details on the differences of the scores in their respective sections. for each template, and for each language, 28 gender-specific identifiers and 6 different predicates were used with each occupation. to compute the scores, we average over the gendered-identifiers and the predicates of the lms' returned probabilities for each template. for each template, only one gender is represented (female or male). if the language inflects for gender, all components of a template reflect the gender in question, otherwise it is only reflected in the identifier. we average the scores for a given occupation by gender, by summing and normalizing the probabilities of each identifier and the total probability norwegian english french brødrene he elle broren she elles dama/damen the aunt il damene the aunts ils datteren the boy l'homme døtrene the boys l'oncle faren the brother la dame fedrene the brothers la femme gutten the daughter la fille guttene the daughters la mère han the father la soeur hun the fathers la tante jenta/jenten the girl le fils jentene the girls le frère kvinnen the ladies le garçon kvinnene the lady le père mannen the man les dames mennene the men les femmes mødrene the mother les filles moren the mothers les fils onkelen the sister les frères onklene the sisters les garçons sønnen the son les hommes sønnene the sons les mères søsteren the uncle les oncles søstrene the uncles les pères tanten the woman les soeurs tantene the women les tantes descriptive bias score once the scores score o are computed, the descriptive bias score compares the percentages of distribution of occupations in the lms to the ground truth data that comes from the respective census data of our countries of interest. we impose a threshold on the gender distributions in such a way that the category of gender-imbalanced occupations here corresponds to all occupations exceeding 55% of distribution for one gender, while gender-balanced occupations are those which percentages lie around 50% ±5 for each gender. we look at the extent to which this score aligns with the census data. we compute an overall score disregarding gender, in addition to class-level scores: female dominated occupations (more than 55% in census are females), male dominated occupations (more than 55% in census are males), neutral occupations (between 45% and 55% of occupations in census for either gender). normative bias score the normative bias score also builds on top of the scores score o , and compares the resulting distribution of occupations in lms to a normative description of all occupations, such that percentages of either gender should be around 50% ±5 . from a normative point of view, equal representations should be given to females and males. instead of just setting the distribution to a strict value of 50-50, we decided that for either gender, the distribution should range anywhere between 45% and 55% in the census data. this to say, that if an occupation has 45% and 55% males, we consider it a balanced distribution. ## results and discussion to get a more detailed overview of which types of occupations the tested models seem to represent the best, we also computed the descriptive scores of gender-imbalanced and gender-neutral occupations separately. results can be seen in all models exhibit the lowest scores on genderneutral occupations, hinting at the tendency that models correlate most occupations with one gender, rather then equally representing them. this would also align with the lower normative scores that we generally see. since the occupations in the census data differ from country to another, it is difficult to compare and rank models across languages. a fair comparison of these models is to focus on performance by country rather than across them. even if we state that some models are the best or worst using one scoring system, the country-level scores are the most important measure of bias in the models. ## conclusion we have introduced a new scoring system for measuring occupational biases in pre-trained language models. the scoring system allows the attribution of two scores: a normative score and a descriptive score. while the normative score sheds light on to what extent the correlations between genders and occupations are balanced, the descriptive score uses real world demographic distributions to reflect to what degree the language models reflect reality. as a proof of concept we test our scoring systems on ten language models covering the french, norwegian, and english languages. it comes as no surprise that all models exhibit low scores when using the normative scoring, while most of them have an adequate score when measured descriptively. what is more interesting is that our scoring mechanism allows us to separate between the normative and descriptive aspects of the model properties. all templates and codes are made publicly available on our github repository. 10 while we have limited our analysis to three languages, our approach is language agnostic and only requires language specific templates and demographic statistics on the distributions of occupations with respect to gender, something most national census agencies should be able to provide. moreover, the approach could also be extended to other dimensions of national census data and other demographic variables. on the note of future directions, we also plan to investigate cross-cultural effects, by comparing models for different languages across the genderoccupation lists from different nations, and also including multi-lingual models. moreover, we also plan to systematically test the impact of different text sources (used to train language models) on our bias scores.