Multilingual BERT has an Accent: Evaluating E nglish Influences on Fluency in Multilingual Models (2023)

Abstract: While multilingual language models can improve NLP performance on low-resource languages by leveraging higher-resource languages, they also reduce average performance on all languages (the ‘curse of multilinguality’). Here we show another problem with multilingual models: grammatical structures in higher-resource languages bleed into lower-resource languages, a phenomenon we call grammatical structure bias. We show this bias via a novel method for comparing the fluency of multilingual models to the fluency of monolingual Spanish and Greek models: testing their preference for two carefully-chosen variable grammatical structures (optional pronoun-drop in Spanish and optional Subject-Verb ordering in Greek). We find that multilingual BERT is biased toward the English-like setting (explicit pronouns and Subject-Verb-Object ordering) and against the default Spanish and Gerek settings, as compared to our monolingual control language model. With our case studies, we hope to bring to light the fine-grained ways in which multilingual models can be biased, and encourage more linguistically-aware fluency evaluation.

## introduction multilingual language models share a single set of parameters between many languages, opening new pathways for multilingual and low-resource nlp. however, not all training languages have an equal amount, or a comparable quality @xcite , of training data in these models. in this paper, we investigate if the hegemonic status of english influences other languages in multilingual language models. we propose a novel method for evaluation, whereby we ask if model predictions for lower-resource languages exhibit structural features of english. this is similar to asking if the model has learned some languages with an "english accent", or an english grammatical structure bias. we demonstrate this bias effect in spanish and greek, comparing the monolingual models beto @xcite and greekbert @xcite to multilingual bert (mbert), where english is the most frequent language in the training data. we show that mbert prefers english-like sentence structure in spanish and greek compared to the monolingual models. our case studies focus on spanish pronoun drop (prodrop) and greek subject-verb order, two structural grammatical features. we show that multilingual bert is structurally biased towards explicit pronouns rather than pro-drop in spanish, and subjectbefore-verb order in greek: the structural forms parallel to english. the effect we showcase here demonstrates the type of fluency that can be lost with multilingual training -something that current evaluation methods miss. our proposed method can be expanded, without the need for manual data collection, to any language with a syntactic treebank and a monolingual model. since our method focuses on finegrained linguistic features, some expert knowledge of the target language is necessary for evaluation. our work builds off of a long literature on multilingual evaluation which has until now mostly focused on downstream classification tasks @xcite @xcite @xcite @xcite . with the help of these evaluation methods, research has pointed out the problems for both high-and lowresource languages that come with adding many languages to a single model @xcite @xcite , and proposed methods for more equitable models @xcite @xcite @xcite . we hope that our work can add to these analyses and methodologies by pointing out issues beyond downstream classification performance that can arise with multilingual training, and aid towards building and evaluating more equitable multilingual models. ## method our method relies on finding a variable construction in the target language which can take two structural surface forms: one which is parallel to english (s parallel ) and one which is not (s different ). surface forms parallel to english are those which mirror english structure. once we have identified such a construction in our target language, we can ask: are multilingual models biased towards s parallel ? we can use syntactic treebank annotations to pick out sentences that exhibit the structures s parallel or s different , and put these extracted sentences into two corpora, c parallel and c different . we then calculate a ratio r model for each model: the average probability of a sentence in c parallel divided by the average probability of a sentence in c different according to the model. our experimental question then boils down to asking if r multi is significantly larger than r mono . to get an estimation of p model (x), we can extract the probability of one word w in each sentence that best represents the construction, and approximate the probability of x with p (w x |x). using a carefully chosen word as a proxy for the probability of a construction is a methodological choice also made in reading time psycholinguistics experiments @xcite . ## results results are shown in figures 1 and 2 , showing for both of our case studies that multilingual bert has a greater propensity for preferring englishlike sentences which exhibit s parallel . multilingual bert significantly prefers pronoun sentences over pro-drop compared with monolingual beto (bootstrap sampling, @xmath0.05), and significantly prefers subject-verb sentences over verb-subject sentences over greekbert (bootstrap sampling, @xmath1.05).