Gender Bias in Masked Language Models for Multiple Languages
Abstract
Masked Language Models (MLMs) pre-trained by predicting masked tokens on large corpora have been used successfully in natural language processing tasks for a variety of languages.
Unfortunately, it was reported that MLMs also learn discriminative biases regarding attributes such as gender and race.
Because most studies have focused on MLMs in English, the bias of MLMs in other languages has rarely been investigated.
Manual annotation of evaluation data for languages other than English has been challenging due to the cost and difficulty in recruiting annotators.
Moreover, the existing bias evaluation methods require the stereotypical sentence pairs consisting of the same context with attribute words (e.g. He/
She is a nurse ).
We propose Multilingual Bias Evaluation (MBE) score, to evaluate bias in various languages using only English attribute word lists and parallel corpora between the target language and English without requiring manually annotated data.
We evaluated MLMs in eight languages using the MBE and confirmed that gender-related biases are encoded in MLMs for all those languages.
We manually created datasets for gender bias in Japanese and Russian to evaluate the validity of the MBE.
The results show that the bias scores reported by the MBE significantly correlates with that computed from the above manually created datasets and the existing English datasets for gender bias.
Introduction
Masked Language Models (MLMs) (Devlin et al., 2019), which are pre-trained on large corpora, have been used successfully in natural language processing tasks for various languages (Conneau and Lample, 2019; Martin et al., 2020; Conneau et al., 2020).
Unfortunately, it has been reported that MLMs also learn social biases regarding attributes such as gender, religion, and race (Kurita et al., 2019; Dev et al., 2020; Kaneko and Bollegala, 2021a; Bender et al., 2021).
The bias in MLMs is evaluated by the imbalance of the likelihood between pairs of sentences associated with an attribute that has a common context (e.g. He/
She is a nurse ).
Nadeem et al.
(2021) masked the modified tokens (e.g. He,
She ), and Nangia et al.
(2020) masked the unmodified tokens (e.g. is, a, nurse ) one word at a time and calculated the likelihood from their predictions to evaluate the bias.
Kaneko and Bollegala (2021c) evaluated the bias using the average of the likelihoods of all tokens without masking the MLM.
Despite the numerous studies of social bias in
MLMs covering English, social biases in MLMs for other languages remain understudied (Lewis and Lupyan, 2020;
Liang et al., 2020; Bartl et al., 2020; Zhao et al., 2020).
To realise the diverse and inclusive social and cultural impact of AI, we believe it is important to establish tools for detecting and mitigating unfair social biases in MLMs, not only for English but for all languages.
However, the significant manual annotation effort, the costs and difficulties in recruiting qualified annotators remain major challenges when creating bias evaluation benchmarks for target languages.
For example, existing bias evaluation benchmarks such as CrowS-Pairs ( CP ; Nangia et al., 2020) and StereoSet ( SS ; Nadeem et al., 2021) require humanwritten sentences (or pairs of sentences) eliciting different types of social biases expressed in the target language.
However, scaling up this approach to all languages is challenging because recruiting a sufficiently large pool of annotators to cover the different types of social biases in those languages is difficult.
Because of the above-mentioned challenges, bias evaluation datasets and studies outside
English remain under-developed.
To address this problem, we propose Multilin- gual Bias Evaluation (MBE) score 1 , a social bias evaluation method that can be used to evaluate biases in pre-trained MLMs for a target language without requiring significant manual annotations for that language.
MBE can perform equivalent bias evaluation using only existing parallel corpora and lists of English masculine (e.g. he, his, father, son etc.)
and feminine (e.g. she, her, mother, daughter , etc.)
words, without requiring any manually annotated sentences for social biases in the target language.
Although MBE require parallel corpora, such sources already exist for numerous language pairs 2 or can be automatically mined with less effort compared to annotating bias evaluation data (Artetxe and Schwenk, 2019b,a).
As a concrete example, we evaluate the proposed method for gender bias, which exists in many languages.
Extending the proposed method to other types of social biases beyond gender biases is deferred to future work.
As shown in Figure 1, MBE first (shown in Step 1) extracts target language sentences containing female words (female sentences) and sentences containing male words (male sentences) from a parallel corpus between English and the target language using a list of gender words in English.
Second, (shown in Step 2) MBE calculates the likelihoods for each of the extracted female and male sentences in the target language using the given MLM under evaluation.
Finally, (shown in Step 3)
MBE compares the likelihoods between each female sentence and all male sentences considering all pairwise combinations, and increment a count by 1 if the likelihood of the male sentence is greater than that of the female sentence, and 0 otherwise.
As for Step 1, we do not require any knowledge about the target language or manual annotations because we use only the existing English attribute word lists and parallel corpora between English and the target language.
This is attractive from a data availability point of view, which makes our proposed method easily extendable to different languages.
Kaneko and Bollegala (2021c) found that the frequency of the words associated with the male gender to be significantly higher than that for the female gender in the data used to train MLMs.
They showed that these frequency-related biases are encoded into MLMs, and independently of whether a sentence contains a stereotypical or antistereotypical context, an MLM that is biased towards the male gender, on average, assigns higher likelihood scores to sentences that contain masculine words than feminine words 3 .
Inspired by this finding, in
Step 2, we calculate the likelihood scores assigned by an MLM under evaluation to sentences that contain male and female related words in different contexts (e.g. He is a baseball player and She is a nurse ).
Step 3 of our proposed method performs the computation of the bias score considering the similarity between the contexts in sentence pairs that contain male and female related words.
The more similar the sentence pairs are, the more similar the estimates would be compared to the bias evaluation measures that require identical contexts.
Therefore, we weight the bias score estimates by the similarity of the sentence pairs using the sentence representations obtained from the MLM under evaluation.
We ignore dissimilar sentence pairs and compute the bias score from the similar sen- tence pairs, which is defined as the percentage of sentence pairs where the sentence containing the masculine words is assigned a higher likelihood than the sentence containing the feminine words.
Bias in MLM is thought to depend on both the
MLM and the evaluation data, so in this paper, we are investigating both using two corpora for English
MLMs.
Using the proposed method, we evaluated gender bias in MLMs in eight other languages:
German, Japanese, Arabic, Spanish, Portuguese,
Russian, Indonesian, and Chinese.
Prior work investigating social biases in MLMs for English have shown that different types and levels of biases are shown by different MLMs even for the same language (Kaneko and Bollegala, 2021a,c; Dev et al., 2020).
We defer covering different MLMs across multiple languages to future work and focus on establishing MBE as an evaluation measure that can be used for such a study.
Our evaluations show that all MLMs learn gender-related biases in all languages studied.
To further validate MBE, we conduct a metaevaluation where we use an existing manually annotated English bias evaluation dataset and two additional datasets we annotate in this work covering gender-related biases in Japanese and Russian languages.
The bias scores computed using
MBE show significantly high correlations with human bias annotations on both datasets (CP and SS), showing its validity for multiple languages as a gender bias evaluation method.
Furthermore, we show that MBE is superior to methods using machine translations to evaluate bias in non-English languages.
We also show that bias evaluation methods based on templates and word lists significantly overestimate the bias in MLMs due to the unnaturalness of the created templates.
Our analyses on the effects of English names on gender information and preservation of gender information in parallel corpora suggest that bias can be evaluated reasonably even with some loss of gender information.
Related Work
In the study of bias in English MLMs, May et al. (2019) and Kurita et al.
(2019) use a pair of artificial sentences created using manually written templates.
However, template-based evaluation is problematic because it uses an artificial context that does not reflect the natural usage and distribution of words in the target language.
To solve this problem, Nadeem et al.
(2021) and Nangia et al. (2020) manually created bias evaluation datasets,
SS and CP, respectively, with stereotypical and antistereotypical sentence-pairs with identical contexts, except the attribute words.
However, recent work has pointed out various issues in CP and SS datasets and has argued that they may not provide effective measurements of stereotyping (Blodgett et al., 2021).
In this study, (social) bias is defined as the tendency towards outputting sentences about a particular advantageous or disadvantageous group, such as males or females, given the same context by an MLM.
However, these benchmarks are currently the most commonly used benchmarks for bias evaluation in MLMs, so we also use them in this work.
We note that MBE is independent of any bias evaluation benchmark datasets.
Our focus in this paper is on evaluating gender bias in multiple languages and not on comparing or proposing novel debiasing methods.
However, for the completion of the discussion, we note that methods for debiasing MLMs using sentence vectors from MLMs (Bommasani et al., 2020) and lists of English male and female words has been studied (Sedoc and Ungar, 2019;
Kaneko and Bollegala, 2021a;
Dev et al., 2020; Chou et al., 2022).
In prior work on MLMs, social biases for languages other than English have rarely been investigated.
Ahn and Oh (2021) investigated ethnic bias in monolingual MLM in six languages by extending the templates to other languages using machine translation.
The biases of MLMs have been evaluated using templates for English and Chinese (Liang et al., 2020) and for English and German (Bartl et al., 2020).
Zhao et al.
(2020) investigated the gender bias of a classifier that predicts the occupation from resumes using multilingual word embeddings and multilingual MLM embedding in
Spanish, German and French.
They evaluated bias by using machine translation on the English data, when an MLM is used to create feature representations in a specific task.
However, this setting is different from that of our study, where we evaluate the bias of MLMs independently of a specific task.
Moreover, the above studies do not discuss or propose methods on how to create evaluation data that can be applied to many languages.
Following the pioneering work by Bolukbasi et al.
(2016) that proposed a bias evaluation and debiasing methods, various studies have investigated social biases in English (Caliskan et al., 2017; Zhao et al., 2018; Kaneko and Bollegala, 2019, 2021b;
Dev and Phillips, 2019).
Unlike the contextual word embeddings produced by MLMs, evaluating social biases in static word embeddings is relatively less complicated because it can often be done using word lists without requiring annotated sentences.
In static word embeddings, bias has been investigated in various languages besides English due to this ease of annotating evaluation data.
Lauscher and Glavaš (2019) translated the English word lists into six languages and evaluated the bias of the word embeddings.
Zhou et al.
(2019) proposed an evaluation metric for languages that require gender morphological agreement, such as in Spanish and French.
Friedman et al.
(2019) quantified the gender bias of word embeddings to understand cultural contexts with large-scale data, and used it to characterize the statistical gender gap in education, politics, economics, and health in US states and several countries.
Bansal et al.
(2021) proposed a debiasing method by constructing the same bias space for multiple languages, and adapted it to three Indian languages.
Other bias studies have been conducted for specific languages (Takeshita et al., 2020; Pujari et al., 2019; Sahlgren and Olsson, 2019; Chávez Mulsa and Spanakis, 2020), but they are not easily transferable to novel languages.
Bias Evaluation for Multiple Languages
Our proposed MBE score evaluates the gender bias of the target language under evaluation in three steps (see Figure 1).
In Step 1, we first define the set of English sentences E and the set of target language sentences T of the parallel corpus, where N is the data size, and ( e i , t i ) is a parallel sentence pair.
Let V f
(e.g. she, woman, female ) be the list of female words and V m (e.g. he, man, male ) be the list of male words in English.
We then extract sentences that contain a female or a male word from E .
Sentences that contain both male and female words are excluded.
Let us denote the set of sentences extracted for a female or a male word w by Φ( w ) .
Let E f = (cid:83) w ∈V f Φ( w ) and E m =
(cid:83) w ∈V m Φ( w ) be the sets of sentences containing respectively all of the male and female words.
The set of sentences in the target language of the source sentences included in E f and E m is denoted by T f and T m , respectively.
It is assumed that gender information is retained in the parallel corpus, and whether this is actually the case is verified later.
In Step 2, we compute the likelihood for the full sentences in T f and T m .
Let us consider a target sentence T = w 1 , w 2 , . .
, w | T | , containing length | T | sequence of tokens w i .
We calculate the likelihood with All Unmasked Likelihood with
Attention weights ( AULA ; Kaneko and Bollegala, 2021c) which evaluates the bias by considering the weight of MLM attention as the importance of tokens.
Given an MLM with pre-trained parameters θ , which we must evaluate it for its gender bias, let us denote the probability P MLM ( w i | T ; θ ) assigned by the MLM to a token w i conditioned on all the tokens of T .
AULA predicts all of the tokens in T using the attention weights to evaluate social biases considering the relative importance of words in a sentence, which is given by (1).
[Equation]
Here, α i is the average of all multi-head attentions associated with w i .
In Step 3, by comparing the likelihoods of female and male sentences returned by AULA, we calculate the bias score as the weighted average of the similarities of contexts using the sentence representations produced by the MLM under evaluation.
Specifically, We use the percentage of male ( T m ) sentences (e.g. He is a baseball player ) preferred by the MLM over female ( T f ) ones (e.g. She is a nurse ) to define the corresponding multilingual bias evaluation measure ( MBE bias score ) as follows: [Figure]
[Equation]
Here, I is the indicator function, which returns 1 if its argument is True and 0 otherwise.
C ( T m , T f ) uses the average of the last layer in MLM for all tokens except special tokens to compute the sentence embeddings of T m and T f respectively and computes the cosine similarity of these embeddings.
According to this evaluation measure, values close to 50 indicate that the MLM under evaluation is neither females nor males biased, hence, it can be regarded as unbiased.
On the other hand, values below 50 indicate a bias towards the male group and above 50 towards the female group.
We report a statistically significant difference comparing to the model with randomly assigned results of the indicator function I in Equation 2 with the McNemar’s test ( p < 0 . 05 ).
For each sentence, the presence or absence of bias is predicted by two methods, MLM
[Table]
and Random.
The McNemar ’ s test was used by classifying into four categories: only MLM was biased, only random was biased, both were unbiased, and both were biased.
We use the statistically significant difference to determine if there is a bias.
Gender Bias in Masked Language Models
We use two parallel corpora, the TED2020 v1 corpus in the spoken language domain ( TED ) 4 and the
GlobalVoices corpus in the news domain ( News ) 5 .
Table 1 shows the total number of extracted male and female sentences for each language.
Except for Spanish, the News corpus is smaller than the TED corpus for all languages.
In particular, the Indonesian news corpus is an extremely low resource.
For the list of female and male words in English, we use the list created by Bolukbasi et al. (2016) 6 in addition to the female and male names in CP (Nangia et al., 2020).
The extracted male and female sentences were downsampled to create sets of an equal number of sentences.
We experimented on the GeForce [-RTX
-]{+RTX 
+}
Ti using the transformers 7 implementation with default settings (Wolf et al., 2020).
All evaluations are completed within 10 minutes.
We used Masked Language Models ( MLMs ) in eight languages for our experiments: Japanese 8 ,
German 9 (Chan et al., 2020), Arabic 10 (Antoun
/wordlist 7 https://github.com/huggingface/ transformers 8 https://huggingface.co/cl-tohoku/ bert-base-japanese-whole-word-maskinghttps://huggingface.co/deepset/ gbert-base 10 https://huggingface.co/aubmindlab/ bert-base-arabertv
[Table]
et al., 2020), Spanish 11 (Cañete et al., 2020), Portuguese 12 (Souza et al., 2020), Russian 13 , Indonesian 14 and Chinese 15 (Cui et al., 2020).
As English MLMs, we use BERT 16 , multilingual BERT 17 (Devlin et al., 2019), RoBERTa 18 (Liu et al., 2019), ALBERT 19 (Lan et al., 2019), DistilBERT 20 , DistilRoBERTa 21 (Sanh et al., 2019), ConvBERT 22 (Jiang et al., 2020), XLM 23 (Conneau and Lample, 2019), and Deberta 24 (He et al., 2020). Table 2 shows the bias scores of the proposed
MBE method for the TED and News corpora for the MLMs considered.
Here, the significant difference is evaluated against the MBE score of a randomly assigned indicator function.
Overall, we see gender-related biases are reported in all cases.
[Table]
[Table]
In particular, significant biases are shown in the
News corpus for all languages except Japanese,
Russian and Indonesian.
Moreover, the different levels of biases reported for Russian and Japanese between TED and News corpora indicate that bias evaluations are affected not only by the MLMs but also the corpora used.
It is known that the bias tendency of MLMs changes depending on the training data (Babaeianjelodar et al., 2020), and similarly, the bias evaluation of MLMs is affected by the evaluation corpus.
Because MBE can evaluate bias in various domains as long as there are parallel corpora.
It can also capture corpus-dependent biases, unlike existing methods requiring manually created domain-specific sentence pairs.
Meta-Evaluation
We perform a meta-evaluation to validate MBE scores against human bias ratings.
In §5.1 we measure the correlation between MBE scores and existing measures on CP and SS, which are manually annotated bias evaluation benchmarks for English.
In §5.2 to compare the evaluation methods in the target languages using MBE and manually annotated data, we manually translate the CP into the Japanese and Russian, which demonstrate high corpus-specific biases according to Table 2.
5. Gender Bias Evaluation Using Manually Annotated Data in English
To validate MBE scores using human bias ratings, we use CP and SS datasets for English .
As baseline method we use Shf , which shuffles the sets of male and female sentences and randomly pair sentences from this set.
Shf is used to show the usefulness of comparing the likelihoods of male and female sets.
In the existing evaluation method using manually annotated sentence pairs, the bias score is calculated for stereotypical S s (e.g. He is a doctor ) and anti-stereotypical S a (e.g. She is a doctor ) sentences with identical contexts as follows:
[Equation]
where N is the total number of sentences.
We use this bias score as an upper bound score to compare against it the results for Shf and MBE using the rank correlations ( Spearman and Pearson ), the agreement of the direction of bias between female and male directions ( Direction ), where the bias scores above 50 indicate a bias towards the male direction and that below 50 towards the female direction, and the difference of the bias scores ( Diff ) from the results of the method using manual annotation.
In the proposed method and Shf, for the gender bias data of CP and SS, we extract sentences containing male and female words for each sentence, instead of sentence pairs, and use them for evaluation using Equation 3.
As English MLMs 25 , we use BERT, multilingual BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), ALBERT (Lan et al., 2019), DistilBERT, DistilRoBERTa (Sanh et al., 2019), ConvBERT (Jiang et al., 2020), XLM (Conneau and
Lample, 2019), and Deberta (He et al., 2020).
Since
BERT and RoBERTa each use two models of different sizes, we use a total of 11 models.
We report the averaged results over the above 11 models.
Table 3 shows that MBE has high performance in all evaluations.
Performance of Shf highlights the importance of comparing male against female sentences in sentence pairs.
5. Gender Bias Evaluation Using Manually Annotated Data in Japanese and Russian
To validate MBE scores, which does not require evaluation data with identical context, nor manual creation of evaluation data in the target languages other than English , we use the following methods:
HT:
We manually translate sentence pairs in CP into Japanese and Russian and apply Equation 3.
This human translated ( HT ) baseline can be seen as an upper bound for bias evaluation compared [Table] to MBE, which does not require translated examples.
Lower difference from these bias scores in this human-translated (HT) method would indicate a more reliable bias evaluation measure.
Note that, it is not appropriate to compare the bias score calculated using the English MLMs with the bias score calculated using the Japanese MLMs because we are evaluating different models.
Therefore, we calculate the bias score in Equation 3 using the data translated into Japanese and Russian.
MBE:
Here, we let MBE(Japanese) and
MBE(Russian) be the MBE scores computed using the Equation 2 and parallel data created above by manually translating original (English)
CP dataset into Japanese and Russian for Step 1, respectively.
MT: As an alternative to costly manual translations, we use Google Machine Translation method ( MT ) 26 to translate sentence pairs in CP sharing identical contexts into each target language and apply Equation 3.
Tmp: Although it requires some knowledge about the target language, one can create templates in the target language for both genders such as “[Gender] は [Occupation] です ” ( [Gender] is a/an [Oc- [Table] cupation] ) in Japanese, and fill in male and female word pairs, and occupation words as in “ 彼 / 彼 女 は 医 者 です ” ( He/
She is a doctor ) to create an equal number of sentences as the evaluation data for
Equation 3.
In the template-based method ( Tmp ), five word pairs were used for Japanese and Russian following prior work by Kurita et al.
The templates were “[Gender] は [Occupation] で す。 ” and “[Gender] は [Occupation] に 興 味 があ る。 ” in Japanese and “[Gender] - [Occupation].” and “[Gender] - [Occupation] поспециа льности .” were used for Russian.
We extracted respectively 644 and 154 occupation words for Japanese and Russian from Wikipedia 28 .
Following prior work by Kurita et al.
(2019), we generated respectively 6400 and 1500 template sentences for Japanese and Russian, and evaluated them using sentence pairs with identical contexts.
For Japanese MLMs, we evaluate four Japanese
BERT models ( base-subword 29 , large-subword 30 , base-char 31 , large-char 32 ), subword-based and character-based, with base and large sizes.
For Russian, we use two MLMs – one trained on Wikipedia
[Table] and news data ( wiki&news ) 33 and the other on OpenSubtitles (Lison and Tiedemann, 2016) and SNS data (Shavrina and Shapovalova, 2017).
For Japanese and Russian, we use the difference of the bias scores instead of the correlation coefficients with HT because the number of publicly available pre-trained MLMs is smaller than that of English.
Tables 4 and 5 show the bias scores of HT, MT,
Tmp and MBE and their differences measured against HT for Japanese and Russian MLMs, respectively.
We see that the difference between the bias scores of HT and MBE are smaller than that for MT, indicating that MBE closely approximates the human bias ratings in HT than other alternatives.
Moreover, we see that the direction of bias is reversed for base-char, large-char, and subtitle&sns compared to HT.
Note that we can not directly compare Tmp with other methods due to the difference in evaluation data.
However, as one of the previous bias evaluation methods, Tmp overestimates the biases of MLMs, especially for Japanese subwords.
This is because simple artificial templates often over-emphasize gender biases compared to natural sentences, Interestingly, MBE is more accurate than MT when evaluating gender biases.
Further investigations revealed that MT model itself could produce gender-biased translations, thereby adding noise to the translated sentences.
Bias in Personal Names
One of the most significant differences in the frequency of words used in each language that affects gender bias is the names of people.
In bias evaluation, male and female names are used to identify the gender (Caliskan et al., 2017; Romanov et al., 2019).
However, when names are transliterated [Figure] preserving gender information from English News data. from English to the target language, those transliterated names might be infrequent in the target language and might not be gender representative.
To study the effect of this issue on gender bias evaluation, we conduct the following experiment.
First, for the Japanese and Russian target languages, we replace the transliterated English names in the CP data with native Japanese and Russian names of the same gender.
Next, we compare the bias scores with those before the replacement in Tables 4 and 5.
We extracted the top 10 most popular names among Japanese 34 and Russians 35 for both genders, and randomly substituted them with the transliterated English names.
For example, we rewrite “ シェリーはナースです ” → “ 美 咲 はナースで す ” (“Shelly is a nurse” → “Misaki is a nurse”).
Table shows the
MBE score for
Japanese (HT name (Japanese)), and
Russian (HT name (Russian)) after the name replacement and the corresponding differences w.r.t.
original bias scores shown in Tables 4 and 5).
We can see that the bias scores of the Japanese base models and all the Russian models are almost the same compared to respective values in Tables 4 and 5.
The large models for Japanese differ by about -2.29, which is lower than the baseline in the table.
Moreover, the direction of the bias has not changed in both languages compared to respective directions in
Tables 4 and 5.
These results suggest that the bias can be evaluated reasonably even when English names are transliterated into a target language.
Preserving Gender in Parallel Corpora
Step 1 of the proposed method requires that gender information in English (source) sentence matches that with the target translation in the parallel data.
To test for this, we examine the proportion of sentences in which the corresponding translated words of English “she” and “he” appear to determine whether female or male gender information is retained.
We use the News corpus and select Japanese and Russian, which had no bias, and German and
Portuguese, which had significant biases (Table 2).
Figure 2 shows the percentage of sentences where gender was retained for male and female sentences in the target languages 36 .
For German,
Portuguese, and Russian, gender is retained in more than 80% of the sentences.
This suggests that when the the percentage of gender-preserved sentences is large, it does not affect the MBE score.
In Japanese, gender information is retained in only about 60% of sentences, which is much lower than in other languages.
This may be because Japanese is a nullsubject language that allows independent clauses to omit explicit subjects.
In fact, in some cases, gender words were omitted in the parallel corpus, for example “He owns a grocery store and runs a motorcycle rental business.” was translated to “ 自 分 の 食 料 品 店 を 持 ち、 レンタルバイクビジ ネスも 営 んでいる。
(Owns a grocery store and runs a rental motorcycle business.)”.
Contrarily, from the results in Table 4, MBE(Japanese) can detect the bias better than other methods.
The reason may be that even if the gender words are omitted if the context is composed of words that often cooccur with male and female words, it is possible that it complements the gender information.
In fact, Bolukbasi et al.
(2016) show that words that co-occur with male and female words retain gender information.
The results also show that gender preservation is not heavily biased in either the male or female direction, based on the small difference between percentages for male and female sentences for each language.
This suggests that the bias in the preservation of gender information may not affect the evaluation of the proposed method.
Conclusion
In this paper we showed that a bias evaluation data and evaluation of MLMs for discriminatory bias can be systematically created as long as there is a parallel corpus of English and the target language and a list of female and male words in English.
Our meta-evaluation proved that the proposed multilingual bias evaluation method could perform correct evaluation comparing against method using manually created data, at least for Russian, Japanese and
English.
The experimental results show that gender bias exists in all eight languages of our experiments.
We also showed that the proposed method is superior to the methods that use machine translation to translate the English bias evaluation data into the target language and the methods that use templates and word lists.
Acknowledgements
This paper is based on results obtained from a project, JPNP18002, commissioned by the New
Energy and Industrial Technology Development
Organization (NEDO).