U n MASK ed: Quantifying Gender Biases in Masked Language Models through Linguistically Informed Job Market Prompts (2024)

Abstract: Language models (LMs) have become pivotal in the realm of technological advancements. While their capabilities are vast and transformative, they often include societal biases encoded in the human-produced datasets used for their training. This research delves into the inherent biases present in masked language models (MLMs), with a specific focus on gender biases. This study evaluated six prominent models: BERT, RoBERTa, DistilBERT, BERT- multilingual, XLM-RoBERTa, and DistilBERT- multilingual. The methodology employed a novel dataset, bifurcated into two subsets: one containing prompts that encouraged models to generate subject pronouns in English and the other requiring models to return the probabilities of verbs, adverbs, and adjectives linked to the prompts’ gender pronouns. The analysis reveals stereotypical gender alignment of all models, with multilingual variants showing comparatively reduced biases.

## introduction in recent years, large language models (llms) have emerged as a powerful tool in the field of natural language processing (nlp), demonstrating an unparalleled ability to capture hidden patterns from large datasets @xcite @xcite . these models owe their power to the extensive training on corpora of human-generated text, enabling them to mimic human-like linguistic capabilities with remarkable accuracy @xcite . while the ability to capture and reproduce these patterns often results in beneficial outcomes, it is not without its caveats. an increasing amount of studies @xcite @xcite have underscored the potential risks associated with language models, pointing out their role in inheriting the biases present in the training data, a reflection of human prejudices and societal norms. in the context of language models (lms), bias refers to the systematic misrepresentation of facts or factual distortions that benefit certain groups, spreading and fixing stereotypes, or producing incorrect presuppositions built on learned patterns. these prejudices can be intentionally or unintentionally introduced by (1) training data, (2) algorithms, or (3) human annotators @xcite . if the training datasets are skewed or lack representation from different groups, the model will inevitably inherit these biases. algorithms follow mathematical and logical rules that make them more robust. however, if parameters are set or weighted in a way that they favor certain data points over others, they can introduce or amplify them. lastly, human annotators bring their own perspectives and beliefs. this highlights the importance of having diverse teams involved in the data annotation process to minimize the introduction of individual or cultural biases. gender bias poses ethical concerns, particularly when found in models deployed in sensitive domains, such as the job market, where fairness and impartiality are paramount @xcite . while previous work has focused on using larger and more complex datasets, the question is: do we need a large corpus to identify whether models show gender-biased behavior? this study seeks to delve deeper into the gender biases exhibited by masked language models (mlms), especially in the context of the job market. to do so, this work uses different widely used mlms to evaluate biases from a quantitative and qualitative perspective. this study proposes the use of a small linguistically informed testing dataset targeting the prediction of gender pronouns, adverbs, adjectives, and verbs. the results show that (i) for pronoun resolution, all models show significant biases for gender-stereotypical roles, and (ii) multilingual models show more balanced completions, suggesting a reduced bias. ## previous work the exploration of bias in language models has gained significant attention in the ai research community. given the vastness of this topic, various sub-domains have emerged, each looking into different aspects or types of bias. the first studies focused on word embeddings. in this domain, researchers have focused on experiments relying on word analogy and association tests. it has been shown that word2vec @xcite or glove @xcite display strong biases when facing such experimental scenarios. @xcite identified these inequalities using the embedding similarity between male and female names and career terms. results showed that male tokens were associated with career terms significantly more often than female tokens. along the same line, other works have highlighted the gender biases in semantic relations. @xcite showed that certain professions established undesired logical propositions among male and female tokens (e.g. doctor is to man what nurse is to woman). in the realm of association tests, @xcite proposed the word embedding association test (weat). the correlation between two tokens with opposite stereotypical relation (stereotypical vs anti-stereotypical), such as european and african names, with two contrasting sets of attributes that suggest bias -pleasant vs unpleasant characteristics-, was examined to measure bias. @xcite followed the line of weat and extended it to masked language models with the sentence encoder association test (seat). @xcite presented stereoset, a collection of sentences found in natural environments to assess model biases. the authors proposed a methodology to go beyond the intrasentential bias identification and extend it to the text level. in the context of masked language models (mlms), @xcite presented crows-pairs, an alternative to stereoset. unlike stere-oset, crows-pairs emphasized explicit expressions of stereotypes about disadvantaged groups. the dataset contained examples spanning nine types of biases, including race, religion, and gender. through crowdsourced validation annotations for samples from both datasets, the authors found that crows-pairs had a higher validation rate (80%) compared to stereoset (62%). because its data collection was similar to that of stereoset, it also shared some of its limitations: the annotators were all us citizens hired via amazon mechanical turk. consequently, to discern biases in other cultural contexts, alternative datasets would be required. other works have put special emphasis on the socioeconomic dimension. @xcite showed that countries with lesser gdp also had less in-text representation. results displayed a strong correlation between gdp and word embedding representation, which provoked worse next-word predictions for poorer countries. to show this, the work used token masking such as the country producing most cocoa is @xcite , where the token expected was ghana @xcite . ## methodology this study tested monolingual and multilingual masked language models against two main linguistically informed tasks. first, models were asked to fill the masked tokens ([mask] @xmath0. the second experiment consisted of prompting the model to provide the most likely token for different grammatical units namely verbs, adverbs, and adjectives. these three units had distinct motivations: while adjectives and adverbs provided insight into predicted gender-associated qualities, verbs provided information on gendered subject pronoun agentivity under specific professional scenarios. ## discussion and future work this study offered several advancements. primarily, it employs a multidimensional analysis rooted in descriptive linguistic units, facilitating a nuanced understanding of biases in language models. this approach offers two essential advantages: (1) it can be adapted to different domains and cultural contexts with minimal fine-tuning, and (2) it does not rely on third-party data sources. in this way, it aimed to address the limitations of previous methods which are predominantly u.s.-centric. the method's foundation on linguistic principles allows for a more discerning bias analysis, especially with its emphasis on inter-category relations. the bifurcation into job pronouns and linguistic tokens, coupled with the evaluation of both monolingual and multilingual models, ensures a holistic bias assessment. furthermore, using metrics such as genderassociated token confidence (gtc) or monolingualmultilingual comparisons provides a quantitative dimension to the bias evaluation, enabling comparisons across models. this study opens new opportunities for linguistically informed bias analysis. future work may explore patterns through the implementation of other language units. research may benefit from the analysis of different linguistic elements across sentences with anaphoric relations. subsequent works on bias mitigation may also benefit from implementing in-context retrieval augmented learning (ic-ral). other promising outcomes include data selection techniques such as gradient information optimization (gio) for training data selection @xcite . selecting datasets that represent the richness of society is crucial to diminish biases. ## conclusion this study provides a systematic examination of gender biases within masked language models, particularly in the context of job-related prompts. employing linguistically-informed tasks, such as pronoun resolution and linguistic unit completion, this study has effectively demonstrated the existence of gender biases in these models. a comparison between monolingual and multilingual models reveals a tendency towards stereotypical biases across various categories. however, it is observed that multilingual models tend to yield less biased outputs, likely a reflection of their exposure to a more diverse linguistic training set. this diversity may provide multilingual models with a broader perspective that mitigates entrenched biases, highlighting the potential of multilingual training in the development of more equitable systems. the findings underscore the necessity for refinement in the design and training of language models to ensure fair representations. moreover, the scope of language diversity considered here is limited. english, with its gendermarked pronouns but largely non-gender-marked nouns and adjectives, represents just one typological cluster. to enhance the robustness of the conclusions drawn, future work must incorporate languages from diverse typological backgrounds to discern how such linguistic features may influence bias manifestation within mlms. additionally, the influence of cultural nuances on language use and the resultant biases in mlms require deeper investigation. languages are embedded within cultural contexts that shape their use, and thus, any comprehensive analysis of bias in mlms must consider a broad range of cultural settings to fully understand and address bias.