Multilingual BERT has an accent: Evaluating E nglish influences on fluency in multilingual models (2023)

Abstract: While multilingual language models can improve NLP performance on low-resource languages by leveraging higher-resource languages, they also reduce average performance on all languages (the ‘curse of multilinguality’). Here we show another problem with multilingual models: grammatical structures in higher-resource languages bleed into lower-resource languages, a phenomenon we call grammatical structure bias. We show this bias via a novel method for comparing the fluency of multilingual models to the fluency of monolingual Spanish and Greek models: testing their preference for two carefully-chosen variable grammatical structures (optional pronoun-drop in Spanish and optional Subject-Verb ordering in Greek). We find that multilingual BERT is biased toward the English-like setting (explicit pronouns and Subject-Verb-Object ordering) as compared to our monolingual control language model. With our case studies, we hope to bring to light the fine-grained ways in which multilingual models can be biased, and encourage more linguistically-aware fluency evaluation.

## introduction multilingual language models share a single set of parameters between many languages, opening new pathways for multilingual and low-resource nlp. however, not all training languages have an equal amount, or a comparable quality of training data in these models. in this paper, we investigate if the hegemonic status of english influences other languages in multilingual language models. we propose a novel method for evaluation, whereby we ask if model predictions for lower-resource languages exhibit structural features of english. this is similar to asking if the model has learned some languages with an "english accent", or an english grammatical structure bias. ## method our method relies on finding a variable construction in the target language which can take two structural surface forms: one which is parallel to english (s parallel ) and one which is not (s different ). surface forms parallel to english are those which mirror english structure. for example, english has strict subject-verb-object word order, and so a parallel structure in another language is one where the verb and its arguments appear in subject-verb-object order, while a different structure is one where the verb appears before the subject (see once we have identified such a construction in our target language, we can ask: are multilingual models biased towards s parallel ? for a native speaker of the target language, structural, semantic, and discourse features determine whether they will use s parallel or s different in a given context -with the alternative option usually being less fluent. we assume that a bert-sized monolingual model in the target language will have a sufficiently accurate representation of this fluent variation between s parallel and s different without being influenced by other languages. therefore, to understand if multilingual models have an english structural bias, we now just have to answer: do multilingual models prefer s parallel over s different more than the fluent distribution defined by a monolingual model? ## results results are shown in figures 2 and 3 , showing for both of our case studies that multilingual bert has a greater propensity for preferring englishlike sentences which exhibit s parallel . multilingual bert significantly prefers pronoun sentences over pro-drop compared with monolingual beto (bootstrap sampling, @xmath0.05), and significantly prefers subject-verb sentences over verb-subject sentences over greekbert (bootstrap sampling, @xmath1.05). ## discussion in this paper, we proposed fluency evaluation as a further way of understanding the curse of multilinguality: what can be lost when we train many languages together. the discrepancies that we point out in these experiments are not going to seriously affect multilingual lm performance, especially in the more coarse-grained classification tasks that are most commonly used for evaluation. but, as we demonstrate here, not all levels of language learning can be evaluated from such datasets. our experiments do not pinpoint the reasons behind the effects that we measure: there are different possible explanations for the english-like trends that we showcase. on the one hand, the effects we measure might stem from training with a language that's more dominant in the training data, like english is for many multilingual models. such training could lead to an english-biased representation space which the representations of other languages conform to. on the other hand, the effects we show might be down to the data: the non-english datasets used to train a multilingual model may be more limited in domain, may contain a high proportion of data that's actually been translated from english (multilingual wikipedia is often translated, @xcite , or might be more polluted with irrelevant or non-linguistic elements. domain limitations and translationese stemming from the data are separate, but related issues to fluency: fluency can be grammatical, but also involves proficiency in a range of registers or possibilities. it is also possible that the effects we show are due to a combination of both multilingual representation learning artifacts, and training data quality. further controlled fluency experimentation on the limits and abilities of multilingual models is needed to disentangle these effects. we hope the case studies in this paper can inspire more finegrained evaluation of multilingual models, so that we understand the "accent"-like effects of hegemonic languages more fully. ## limitations this study is meant to highlight the kinds of modeling flaws that have so far gone undetected and that can arise for lower-resource languages in multilingual models. however, our study does not focus on languages that are truly low-resource. in fact, as designed it could not do so: our methodology relies on having an available monolingual model, which of course requires a large amount of training data. this is because our method requires a control: we can only judge multilingual models against what we can believe to be a non-biased language model in the language. there are ways to test for fluency in low-resource languages that would not require a monolingual model as a control, but would require dataset collection in the target language for features that reflect fluency and linguistic acceptability (similar to what @xcite achieve with the cola dataset for english). we hope our study can create motivation for such work in linguistically-aware, fine-grained multilingual evaluation for languages of all resource levels. our experiments focus on bert-style models, since this is mostly the size of model available for monolingual, non-english models (in our case beto and greekbert). however, it is not necessary from these experiments that our findings extrapolate to larger models that are commonplace at the time of writing. lastly, both pro-drop and subject-verb order are largely discourse-dependent constructions. for example, pro-drop is more likely when the subject of the sentence is very clear from the discourse, while subject-verb order in greek is changed to achieve different discourse focus, similar to how intonation changes the focus of a sentence in english (e.g., stressing the verb in "mary helped john" puts the focus on the verb, which in greek can be done by putting the verb first). despite this, all of our ex-periments are done on isolated sentences from the ud treebanks and do not contain discourse content. though this means that the models do not have the full relevant context for each input, we do not expect that having more context should favor one model more than another for our evaluation. since this work compares models on the same inputs, we did not consider this a significant confounder.