An Information-Theoretic Approach and Dataset for Probing Gender Stereotypes in Multilingual Masked Language Models (2022)

Abstract: Bias research in NLP is a rapidly growing and developing field. Similar to CrowS-Pairs (Nangia et al., 2020), we assess gender bias in masked-language models (MLMs) by studying pairs of sentences with gender swapped person references. Most bias research focuses on and often is specific to English.Using a novel methodology for creating sentence pairs that is applicable across languages, we create, based on CrowS-Pairs, a multilingual dataset for English, Finnish, German, Indonesian and Thai.Additionally, we propose S_{JSD} , a new bias measure based on Jensen–Shannon divergence, which we argue retains more information from the model output probabilities than other previously proposed bias measures for MLMs.Using multilingual MLMs, we find that S_{JSD} diagnoses the same systematic biased behavior for non-English that previous studies have found for monolingual English pre-trained MLMs. S_{JSD} outperforms the CrowS-Pairs measure, which struggles to find such biases for smaller non-English datasets.

## introduction pretrained language models (plms) have greatly benefited nlp @xcite @xcite . however, commonly used plms such as bert have been shown to encapsulate social biases, including those relating to gender and race @xcite @xcite . the general consensus is that these biases are learned from the statistical distributional cooccurrence of words relating to a group (such as terms relating to men or women) with a context in which that group is often mentioned in corpora @xcite . for * *now at apple s jsd that that s jsd doctor s jsd the s jsd is is woman man english the doctor s jsd here here diese dieser ärztin die s jsd ist ist frau mann german thai der arzt s jsd hier hier avg. score score ... cps: {0,1} avg. s jsd : [-1,1] 2020 ), we assess multilingual gender bias in mlms by matching genderspecific tokens (light blue) in the context of non-genderspecific tokens (dark blue) in sentence pairs. we develop a methodology for creating sentence pairs that we argue is applicable across languages in contrast to prior work. we mask unchanged tokens one at a time and calculate s jsd , a novel information-theoretic bias measure whose sentence-level average we show to be better behaved than competing measures. in this work we tackle this type of binary stereotypical representational gender bias (henceforth simply "gender bias") in mlms in a multilingual setting. we propose a multilingual approach to study gender bias in mlms, outlined in figure 1 , which, to the best of our knowledge, can in principle be extended to any language. 1 the importance of developing ai systems that are mindful of different societal groups, such as people of different genders, is a topic much discussed in the area of fairness research in nlp @xcite . however, a shortfall of this area is its almost exclusive focus on english. as far as we are aware, ours is the first study to attempt to create a truly multilingual approach to study gender bias in language models. previous multilingual approaches were largely limited to sentences with fixed templates and grammar structures, which heavily constrains the range of languages that may be studied with a given template @xcite . our approach builds on @xcite and attempts to study natural sentences by comparing a pair of sentences that differ only by the gender of persons mentioned, a process which we will refer to as gender swapping. to illustrate the problem of using templates, consider the following sentence pair and its german translation. in german the only parts that remain the same are "ist" and "hier" under gender swapping, as the german word for the profession "doctor" and its associated definite article change form depending on the gender of the person. thus, template structures developed for english of the form (3) @xcite is the [profession] here. the problem is exacerbated in multilingual studies, where appropriate templates need to be defined for each language. we take inspiration from crows-pairs (cps) @xcite , which studies pairs of crowdsourced sentences, for a range of social biases. it includes gender-swapped pairs for the diagnosis of gender bias. however, we found that we cannot simply translate cps into other languages. the main problem is that english pronouns are clear indicators of gender -at least of binary gender, which we focus on in this paper. but this clear indication gets lost in translation for languages that have gender-neutral pronouns like finnish and those that predominantly use null pronouns like thai. 2 we could mandate that only words with "gender-inherent" meaning like "mother", "wife" and "sister" are used, but that would exclude many topics that we need to cover in a good diagnostic dataset, e.g., work life and sports. the solution we propose is to simply use names to indicate gender. our assumption here is that all languages have words for names and that there are two subsets of names that can only have female and male referents. note that there are certainly "unisex" names, i.e., names that can refer to both men and women, even in english ("jess", "leslie"). but as far as we know there is no language that has no "monosex" names, i.e., names that can refer to only one gender. we rely on such monosex names to construct sentence pairs. in english, we select a few frequent male and female names; we only use them for english. before translating the sentence pairs into another language, we first identify corresponding frequent male and female names in the target language. the translators are then instructed to only use those names. this methodology should be applicable universally, so that we can construct a multilingual gender bias resource for any set of languages. in this paper, we translate the cps dataset into german, indonesian, thai and finnish. we edit the original cps dataset before translation to heed the recommendations of @xcite . a more detailed description of dataset creation will be given in §3.1. 3 the second contribution of this paper is s jsd , a novel measure based on the jensen-shannon divergence @xcite , to test mlms for social biases by using sentence pairs that capture a binary contrast between two groups. the measure used in cps (see §3.2) makes use of a binary decision process, which has the effect of removing information of the probability values from the mlm, which we show reduces the measure's predictive power. our motivation for introducing s jsd is to retain as much information from the mlm output probabilities as possible in our final reported score in order to make effective use of the limited amount of human-translated sentences that are available. thus, our contributions are (1) developing a method for creating multilingual datasets for diagnosing gender bias in language models that is ap-plicable across the diverse set of human languages, (2) applying this method, taking the cps dataset @xcite as a starting point, and creating a multilingual gender bias diagnosis dataset for english, german, thai, indonesian and finnish, (3) proposing the s jsd measure, which retains information regarding the numeric output probabilities of mlms. ## related work given this work focuses on multilingual methods to measure gender bias in mlms, this discussion will focus on evaluation measures and techniques; a thorough discussion of debiasing methods is beyond the scope of this paper. ## methodology 3.1 dataset a major obstacle in transferring existing techniques to measure gender bias in languages beyond english is that we need to adapt methods to the target language's gender agreement system. methods for measuring gender bias in mlms often rely on fixed sentence templates, where predefined words are inserted that test some aspect of bias, such as occupational gender bias (e.g., @xcite ). while these template structures can be modified and applied to a range of languages, once a template is chosen, the range of languages that can be studied is restricted @xcite . thus, to design a multilingual approach to gender bias, we want to move beyond the rigid artificial sentence structures that result from using templates. we also speculate that moving away from rigid sentence structures allows us to probe the language model more deeply for biases. it may be possible that superficially debiased language models can perform well on certain bias evaluation tasks that use templates, similar to the situation for linearly debiased word embeddings that perform well on some bias measures but still encapsulate significant distributional biases @xcite . two evaluation datasets that go beyond templates are stereoset @xcite and cps @xcite . one important difference between them is the masking pattern. while stereoset's context association test masks words that may be gendered in a different language (e.g., adjectives in spanish), cps consists of pairs of sentences and only masks tokens that are shared by the two sentences. here we will only consider the cps dataset, which also marks which of the two sentences is more stereotypical @xcite . for our dataset, we consider sentence pairs where people of the male and female gender are being contrasted, for example: for this example, we assume each word is a separate token. the unmodified tokens common to both sentences are: "is", "a", "pilot". for each sentence, the unmodified tokens form a set u and the remaining modified tokens a set m ("he" for (4)a, for example). thus, for each sentence, the set of all tokens is the union of u and m . we will make the assumption that, for sufficiently long and complex sentences, when swapping the gender of a person reference in a sentence there remain sections of the sentence that remain unchanged and that this is true for all languages. from this observation, we found the masking pattern cps implements to be appropriate for multiple languages and thus the sentences labeled with the "gender" tag in the cps dataset were selected as the basis for subsequent translations. the cps dataset was recently criticized for lacking clear explanations of what types of social biases are being measured @xcite . for this reason the selected cps sentences have been minimally modified to be mindful of the pitfalls outlined in @xcite . for example, some sentences were omitted because the contrasted groups were unrelated to the stated "gender" label, such as for sentences that contrasted two racial groups instead. we will now outline the modifications of the cps dataset for this study. first, we ensured each sentence only compares binary gender. non-binary gender adds a level of complexity in the multilingual context that we leave for future work. we also removed sentences that compare clothing items, most likely intended as a proxy for gender. clothing items and their significance differ across cultures, so such sentences are difficult to translate. second, for sentences that only used a pronoun to identify gender, we exchanged the pronoun with a common name that is stereotypically associated with one gender in the english dataset. subsequently, when translating the english dataset into other languages, the names were exchanged for others that are common gendered names in the target language. we limited the number of names in the english dataset to four to simplify the subsequent translation process. names were introduced because many languages do not have gendered pronouns, and thus information relating to gender may be lost in translation. for example, a typical translation of (4) into indonesian results in two identical sentences, which makes the sentence pair useless for indonesian. using names as a proxy for identifying a social group is discouraged in @xcite for race bias, but using stereotypically gendered names as a proxy for binary gender seems unproblematic to us. for example, whereas names only indirectly and ambiguously identify race (at least in english), we can easily find names that are "monosex", i.e., names that can only have either male or female referents. thus, we would modify example (4) as follows for our dataset: finally, we removed sentences that did not correctly isolate a stereotype, an issue noted in the original paper @xcite . in this work we investigate binary gender stereotypes as a representational harm across languages, to use the terminology of @xcite . the cps dataset was created by us crowdworkers @xcite . we make the assumption that most aspects of gender bias should be part of a diagnostic test across languages and cultures. for example, the associations of "doctor" with "male" or of "childcare" with "female" are biases that most cultures are at risk for. so we should test whether our language models exhibit these biases for all cultures. there probably are aspects of gender bias that are relevant to only a small subset of cultures (e.g., the association of "being eligible to drive a car" with "male"). we stress the importance of investigating gender bias multilingually. given that our study is the first to do this, we feel justified to leave the issue of how to comprehensively test for all aspects of bias in gender diagnosis to future work. note that we do not make the assumption that gender bias is the same across languages! if "childcare" is strongly associated with "female" in (the training corpus of) language a, but not in (the training corpus of) language b, then (assuming we use models that pick up bias from their training corpora) our methodology will find less gender bias for language b -and this would be the intended result of our work. ## experiments for our experiments we make use of the transformers library @xcite . we use two multilingual models, multilingual bert (mbert) @xcite , trained on wikipedia, and base xlm-roberta (xlmr) @xcite , trained on wikipedia and filtered commoncrawl data from the internet @xcite . we choose xlmr as it has been shown to significantly outperform mbert on numerous cross-lingual tasks @xcite . as of this writing, xlmr seems to be the best performing multilingual model in the transformers library @xcite . the two models differ in training data by the commoncrawl, which we assume to be more of a source of bias than wikipedia, based on the results of the cps study. @xcite found roberta, trained on wikipedia and the commoncrawl, among other datasets @xcite , to generally have higher bias scores, unperturbed perturbed model s jsd cps s jsd cps bert -6 ±1 60.5 ±1.3 -6 ±1 58.6 ±1.3 roberta -10 ±1 65.5 ±1.2 -10 ±1 63.5 ±1.2 albert -13 ±1 67.0 ±1.2 -11 ±1 64.5 ±1.2 mbert -4 ±1 53.6 ±1.3 -3 ±1 55.6 ±1.3 xlmr -4 ±1 57.1 ±1.3 -4 ±1 56.6 ±1.3 we run the two models on our translated datasets and calculate cps and s jsd scores. running a model on a single language using an intel xeon processor e5-2680 v2 takes roughly 15 minutes. we also test s jsd on the models and dataset used in the cps study @xcite . finally, we test the effect of model size on the scores by comparing the large and base xlm-roberta models. see the appendix for a list of all models used. ## results and analysis a first indication to suspect that we might be in this regime is that the cps standard errors are close in value to the estimated standard errors assuming a bernoulli process, as discussed in §3.2. thus we cannot make a reliable inference regarding model bias. we can also observe a clustering of cps sentence scores, before binarization, around the decision boundary in figure 2 , indicating that slight variations in bias scores can substantially change the cps score. furthermore, the effect of binarizing s jsd (i.e., following the cps method but replacing log p more (u g,i ) with the jsd distance to the gold token) is shown in to test our measure on a larger dataset, we applied s jsd to the original cps dataset (which is much larger since it covers many different types of bias) in ideally, future mlm bias datasets would have a statistical justification that the calculated sample error is adequate to make judgments about a model's bias as probed by a particular measure. finally, we also tested for model size by comparing the base and large xlmr models @xcite . while both models were found to be biased for the more stereotypical sentence, the scores between the two models were too similar and no pattern distinguishing them could be found using either bias measure. we suspect this is due to the small size of the dataset. more complex factors could be at play as well, such as multilingual models benefiting from more multilingual data, in terms of gender bias mitigation, a hypothesis supported by the more negative s jsd scores for monolingual models in ## summary of limitations our results indicate that s jsd is superior to the original crows-pairs measure. but like the crows-pairs measure, s jsd does not provide reliable measurements consistently. the most noticeable case of this is that for many models, "reverse bias" is well within the confidence interval of the bias measures, i.e., values below 50 are within the confidence interval for the crows-pairs measure and positive values for s jsd . we use reverse bias to refer to bias that is the opposite from the stereotype. examples would include that the model favors women to be doctors and men to cry easily. while we did not confirm this experimentally, it seems not possible that a language model would learn a (spurious) stereotype even though the reverse of the stereotype dominates in the training corpus. thus, this finding suggests that the measures must be interpreted with caution. one of our original goals was a cross-lingual quantitative comparison of subtypes of gender bias. for example, maybe the "doctors are men" subtype of gender bias is less prevalent in sweden than in germany. or the subtype "childcare is women's business" is stronger in russia than in canada. however, the two measures are not reliable on a sentence-pair by sentence-pair basis, so that one would need hundreds of examples of a subtype to make such inferences. this would require a dataset two orders of magnitude larger than the one we created. we hypothesize that the main reason for the unreliability of the measures for individual sentence pairs is that predicting subwords is easy and not strongly linked to the difficulty of predicting a word; see " qualitative analysis" in the last section. since most non-english languages will contain words broken into subwords in a given sentence pair, unrealistically high prediction accuracy and a lack of comparability of scores of a sentence pair across languages are the result.