Using Artificial F rench Data to Understand the Emergence of Gender Bias in Transformer Language Models (2023)

Abstract: Numerous studies have demonstrated the ability of neural language models to learn various linguistic properties without direct supervision. This work takes an initial step towards exploring the less researched topic of how neural models discover linguistic properties of words, such as gender, as well as the rules governing their usage. We propose to use an artificial corpus generated by a PCFG based on French to precisely control the gender distribution in the training data and determine under which conditions a model correctly captures gender information or, on the contrary, appears gender-biased.

## introduction numerous studies have demonstrated nlms' ability to capture linguistic properties, such as grammatical categories and syntactic rules like number and gender agreement, without any direct supervision @xcite . however, they leave unanswered the fundamental question of how nlms uncover these properties from raw textual data alone. this study aims to address this question by focusing on the specific case of gender in french. our motivation for working on gender is twofold. firstly, the presence of gender bias in nlms raises significant societal concerns @xcite and understanding how these models capture and use gender information is a crucial step towards addressing such biases. secondly, the grammatical expression of gender in languages such as french makes it possible to construct testbeds to explore how this linguistic information can emerge in an unsupervised way in lms. this work addresses two questions: i) whether the gender that should be assigned to a word is ingrained in its representation or dynamically computed based on the linguistic context and ii) which characteristics of the training set, such as word frequency or gender imbalance, affect an nlm's ability to capture gender information. by answering these two questions, we hope to enhance our comprehension of how an nlm determines a word's gender (or other linguistic properties) and gain insights into the conditions that give rise to a biased representation of gender in these models. to achieve this, we propose to use artificial corpora generated by pcfgs ( §2). 1 this approach allows us to have complete control over the gender of nouns and the contexts in which they are presented. following the methodology from kim and linzen (2020), we evaluate the nlm's capacities on a test set specifically designed to contain examples that differ from those of the training set based on well-defined criteria. by precisely manipulating the contexts in which words appear when training and testing, we can establish causal relations between changes in these contexts and the nlm's ability to assign gender. while the scope of this study is limited to examining the impact of certain training data characteristics on nlms' gender learning, our aspiration is that it serves as an important step in a larger process towards understanding and rectifying biased behaviour in nlms. the subsequent phase entails evaluating how alterations in the training corpus and therefore on how gender is learnt manifest in the text generated by these models. once a causal relationship is established between potential factors influencing gender learning and the behaviour of nlms, the objective is to use this knowledge to mitigate bias in "real" language models trained on natural data by specifically targeting the factors identified as playing an important role in our controlled experiments. contrary to prior research, our results ( §3) show that transformer language models do not necessarily amplify gender imbalances present in the data. moreover, we observe the presence of gender bias even in cases where the training data is perfectly balanced. ## observing the emergence of gender information in nlms to investigate how nlms discover gender information, we propose to train models on controlled sets where gender expression is precisely regulated. we then use a linguistic probe @xcite , to assess the model's ability to learn this property in various scenarios. following a current line of research (kim and linzen, 2020; white and cotterell, 2021), we use artificial language corpora generated by pcfgs to have full control over the expression of gender. this control encompasses, for example, the frequency of each gender and the informativeness of word occurrences' context with regard to gender. our pcfgs aim to mimic a subset of the french language focusing on the expression of gender within noun phrases (nps). while some prior studies in the literature (kim and linzen, 2020) aim to maintain semantic coherence in their artificial data, others @xcite leverage the absence of such coherence to disentangle syntax from semantics. in this work, we chose to focus solely on the grammatical aspect of gender to eliminate potential confounding factors introduced by semantics, such as stereotypes propagated by the distributional environment of words. this deliberate decision narrows down the potential sources of bias under investigation, at least for the present. when constructing the syntax of our language, we also prioritise simplicity over realism, particularly concerning elements unrelated to the specific aspects we investigate in relation to gender learning. our main motivation for selecting french as the basis for our pcfgs stems from the opportunity it provides to construct contexts that either mark gender or not. french has two grammatical genders: feminine and masculine. 2 nouns, determiners, adjectives and pronouns can have fixed genders or be epicene, meaning they can denote individuals of any gender. in this case, the gender must be inferred from the context. for instance, when the epicene noun "artiste" is accompanied by the feminine determiner "une", a feminine gender can be inferred. however, when used with an epicene determiner as in "chaque artiste," the gender becomes ambiguous since the context provides no information about it. the noun phrase generation in the pcfgs involves three levels of rules. the first level determines whether the noun occurs in a context where gender information is revealed by determiners and adjectives or in a gender-ambiguous context: foot_2 @xmath0, following a similar structure, describes the constituents of each of these contexts. meanwhile, the third level establishes the lexical rules that associate terminal symbols (words) to respective word categories. for example: through the manipulation of rules and their probabilities, we can determine the gender of nouns and exert precise control over the gender-related information accessible to the model during training (e.g., by ensuring that some nouns only appear in genderneutral contexts). we can then use the following experimental pipeline to assess the lm's capacity to assign a gender to nouns in different scenarios: 1. we generate two train sets (lm_train and probe_train) and a test set (probe_test) using slightly different pcfgs; 2. we train a lm on lm_train; 3. using probe_train, we train a linguistic probe @xcite to predict the gender (masculine or feminine) to be assigned to noun occurrences based on their contextualised representation at the final layer of the lm; 4. we evaluate the probe's performance on probe_test. @xcite highlight that by relying solely on probe accuracy we cannot tell whether a high score means the information is encoded in the representations or whether the probe has just learnt the task. in order to avoid this problem and ensure that a high probe accuracy indicates the encoding of gender information within the language model's representations, nouns present in probe_train are excluded from probe_test. in all our experiments, we used the transformer lm from @xcite (see §b). we generated 10,000 sentences for lm_train, 1,000 for probe_train and 1,000 × n for probe_test where n represents the number of distinct groups considered in our analyses. all aspects of french syntax relevant to establishing our pcfgs, and those regarding gender in particular, are very similar to other romance languages, such as spanish, portuguese and italian. changing the lexical rules would be enough to make the pcfgs generate sentences emulating another one of those languages. since such a pcfg would be equivalent from the point of view of the nlm, the results of our experiments can be generalised to these other languages. however, different pcfgs would be necessary to study how nlms learn gender in languages with vastly different gender systems, such as languages with more than two genders or languages that have no gender system @xcite . the exploration of this topic is deferred to future research. ## experimental results decoupling contextual gender information from static gender associations our first experiment aims to assess whether nlms can use contextual information during inference to find the gender of words that occur only in ambiguous contexts during training, thus assessing whether the model's representation of gender relies on information memorised from training data or is dynamically constructed during inference based on linguistic cues. ## conclusions in this study, we use artificial language corpora to gain deeper insights into how nlms acquire linguistic knowledge from unannotated texts. our experiments shed light on the complex factors influencing the emergence of gender (as an illustrative linguistic property) in nlms. it is crucial to underscore that, being derived from artificial data, our observations offer insights into the potential mechanisms involved in learning gender, but they do not provide definitive proof that these mechanisms are the core factors driving nlms' ability to acquire and use linguistic knowledge when working with natural language data. however, our findings on nlms' learning process can suggest future directions for bias mitigation. for instance, our results challenge the assumption that training data imbalance is the sole cause of gender bias in nlms and thus shed doubt on the idea that training or fine-tuning on gender-balanced corpora would be enough to avoid gender bias. they also show that an imbalanced gender distribution in the training data does not necessarily lead a transformer-based lm to over-assign the majority class gender. this is cause for optimism as it suggests that transformers' learning process is not inherently biased or irremediably predisposed to reproducing the most frequent patterns. our future work will concentrate on generalising our observations to models trained on natural language data. in future research, it would be valuable to explore more intricate pcfgs that could support the coreference resolution challenges usually used to highlight gender bias in nlms, allowing us to compare gender assignment measurements obtained through this method with the results obtained using probes. in addition to our previous considerations, the extent to which our findings can be extrapolated to "real" models trained on natural language also warrants further investigation. current state-of-the-art lms are significantly larger than the models used in our study and exhibit emergent capacities resulting from their increased size and training data @xcite . therefore, it would be relevant to examine how model and dataset sizes impact our experiments. additionally, comparing the variability of gender expression and the gender distributions in our data to those observed in natural language would offer valuable insights. by further exploring these points, we could gain a better understanding of the robustness and applicability of our findings to real-world language models. it should also be kept in mind that while all the results described in this work are based on the analysis of an artificial corpus of data, the design of our pcfgs draws inspiration from the expression of gender in french. the expression of gender varies significantly across languages and exploring the ability of systems to learn more complex gender systems would be an interesting avenue for future research. a final limitation of this work is the consideration of a binary gender system, which will be revisited in section 6. tatu ylonen. 2022. wiktextract: wiktionary as machine-readable structured data. in proceedings of the thirteenth language resources and evaluation conference, pages 1317-1325, marseille, france. european language resources association. jieyu zhao, tianlu wang, mark yatskar, vicente ordonez, and kai-wei chang. 2018. gender bias in coreference resolution: evaluation and debiasing methods. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 2 (short papers), pages 15-20, new orleans, louisiana. association for computational linguistics.