Are Pretrained Multilingual Models Equally Fair across Languages? (2022)

Abstract: Pretrained multilingual language models can help bridge the digital language divide, enabling high-quality NLP models for lower-resourced languages. Studies of multilingual models have so far focused on performance, consistency, and cross-lingual generalisation. However, with their wide-spread application in the wild and downstream societal impact, it is important to put multilingual models under the same scrutiny as monolingual models. This work investigates the group fairness of multilingual models, asking whether these models are equally fair across languages. To this end, we create a new four-way multilingual dataset of parallel cloze test examples (MozArt), equipped with demographic information (balanced with regard to gender and native tongue) about the test participants. We evaluate three multilingual models on MozArt –mBERT, XLM-R, and mT5– and show that across the four target languages, the three models exhibit different levels of group disparity, e.g., exhibiting near-equal risk for Spanish, but high levels of disparity for German.

## introduction fill-in-the-gap cloze tests @xcite ask language learners to predict what words were removed from a text and it is a "procedure for measuring the effectiveness of communication". today, language models are trained to do the same @xcite . this has the advantage that we can now use fill-in-the-gap cloze tests to directly compare the linguistic preferences of humans and language models, e.g., to investigate task-independent sociolectal biases (group disparities) in language models @xcite . this paper presents a novel four-way parallel cloze dataset for english, french, german, and spanish that enables apples-to-apples comparison across languages of group disparities in multilingual language models. foot_0 ## dataset we introduce mozart, a four-way multilingual cloze test dataset with annotator demographics. we sampled 100 sentence quadruples from each of the four languages (english, french, german, spanish) in the corpus provided for the wmt 2006 shared task. 4 the data was extracted from the publicly available europarl corpus @xcite and enhanced with word-level bitext alignments @xcite . the word alignments are important for what follows. we manually verify that sentences make sense out of context and use the data to generate comparable cloze examples, e.g.: we only mask words which are (i) aligned by oneto-one alignments, and which are (ii) either nouns, verbs, adjectives or adverbs. 5 we mask one word in each sentence and verify that one-to-one alignments exist in all languages. following @xcite , we rely on part-of-speech information to avoid masking words that are too predictable, e.g., auxiliary verbs or constituents of multi-word expressions, or words that are un-predictable, e.g., proper names and technical terms. annotators were recruited using prolific. 6 we applied eligibility criteria to balance our annotators across demographics. participants were asked to report (on a voluntary basis) their demographic information regarding gender and languages spoken. each eligible participant was presented with 10 cloze examples. we collected answers from 240 annotators, 60 per language batch, divided in four balanced demographic groups (gender × native language). we made sure that each sentence 3 see @xcite ; @xcite for discussion of the native/non-native speaker dichotomy. participants were asked "what is your first language?" and "which of the following languages are you fluent in?". we use native (n) for people whose first language coincides with the example sentences, and non-native (nn) otherwise, without any sociocultural implications. the dataset is made publicly available at github.com/coastalcph/mozart under a cc-by-4.0 license. we include all the demographic attributes of our annotators as per agreement with the annotators. the full list of protected attributes is found in ## results following previous work on examining fairness of document classification @xcite @xcite , we focus on group-level performance differences (group disparity). we measure the group disparity as the variance in plm's performance (p@k) across demographics (gender and native language). xlm-r consistently exhibits better overall performance on average, but higher between-group and between-language differences in terms of precision (σ gd ). figure 1 complements results from xlm-r seem to be biased towards native speakers because answers from mn and fn generally rank highest. despite none of the models perform equally across groups, xlm-r shows a lower divergence across languages: between-group differences are more than 50% smaller than with mbert and mt5 when looking at the average mrr per language. it is worth mentioning that our study does not aim to compare models' performance, but rather to motivate a discussion about the between-group and between-language differences within each model. the general low precision of mt5 outputs compared to human answers is likely due to the nature of the task itself. because mt5 was trained with a span-mask denoising objective, it tends to complete the masked-out span with more than one token. when constraining generation to output one token, we are conditioning its default behaviour. better correlation could be achieved by fine-tuning the model on completing cloze tests. ## related work multilingual plms have been analyzed in many ways: researchers have, for example, looked at performance differences across languages @xcite , looked at their organization of language types @xcite , used similarity analysis to probe their representations @xcite , and investigated how learned self-attention in the transformer blocks affects different languages @xcite . previous work on fairness of multilingual models has, to the best of our knowledge, focused exclusively on task-specific models, rather than plms: huang et al. (2020) evaluate the fairness of multilingual hate speech detection models, and several researchers have explored gender bias in multilingual models @xcite . @xcite consider the effects of adversarial debiasing in multilingual models. avg. cloze tests were previously used in @xcite to evaluate the fairness of english (monolingual) language models. in psycholinguistics, cloze tests have been performed with different age groups @xcite and native language @xcite , but these datasets have, to the best of our knowledge, not been used to evaluate language models. ## conclusion in this paper, we present mozart, a new multilingual dataset of parallel cloze examples with annotations from balanced demographics. this dataset is, to the best of our knowledge, the first to enable apples-to-apples comparison of group disparity of multilingual plms across languages. the dataset includes several demographic attributes, but we present preliminary experiments with gender and native language. we show that mbert, xlm-r and mt5 are not equally fair across languages. for example, group disparities are much higher for german (and french) than for english and spanish. this shows the importance of evaluating fairness across languages instead of stipulating from results for a single language. we further show that cloze test answers of female native speakers tend to rank highest in both predictive plms. we followed best practices for mitigating the dangers of crowdsourcing @xcite (see §2) and hope mozart will be widely adopted and, over time, generate more results for other languages, plms and demographic attributes.