Are All Languages Created Equal in Multilingual BERT ? (2020)

Abstract: Multilingual BERT (mBERT) trained on 104 languages has shown surprisingly good cross-lingual performance on several NLP tasks, even without explicit cross-lingual signals. However, these evaluations have focused on cross-lingual transfer with high-resource languages, covering only a third of the languages covered by mBERT. We explore how mBERT performs on a much wider set of languages, focusing on the quality of representation for low-resource languages, measured by within-language performance. We consider three tasks: Named Entity Recognition (99 languages), Part-of-speech Tagging and Dependency Parsing (54 languages each). mBERT does better than or comparable to baselines on high resource languages but does much worse for low resource languages. Furthermore, monolingual BERT models for these languages do even worse. Paired with similar languages, the performance gap between monolingual BERT and mBERT can be narrowed. We find that better models for low resource languages require more efficient pretraining techniques or more data.

## introduction pretrained contextual representation models trained with language modeling @xcite or the cloze task objectives @xcite have quickly set a new standard for nlp tasks. these models have also been trained in multilingual settings. as the authors of bert say "[...] (they) do not plan to release more single-language models", they instead train a single bert model with wikipedia to serve 104 languages, without any explicit cross-lingual links, yielding a multilingual bert (mbert) @xcite . surprisingly, mbert learn high-quality cross-lingual representation and show strong zero-shot cross-lingual transfer performance @xcite . however, evaluations have focused on high resource languages, with cross-lingual transfer using english as a source language or within language performance. as @xcite evaluated mbert on 39 languages, this leaves the majority of mbert's 104 languages, most of which are low resource languages, untested. does mbert learn equally high-quality representation for its 104 languages? if not, which languages are hurt by its massively multilingual style pretraining? while it has been observed that for high resource languages like english, mbert performs worse than monolingual bert on english with the same capacity @xcite . it is unclear that for low resource languages (in terms of monolingual corpus size), how does mbert compare to a monolingual bert? and, does multilingual joint training help mbert learn better representation for low resource languages? we evaluate the representation quality of mbert on 99 languages for ner, and 54 for partof-speech tagging and dependency parsing. in this paper, we show mbert does not have equally highquality representation for all of the 104 languages, with the bottom 30% languages performing much worse than a non-bert model on ner. additionally, by training various monolingual bert for low-resource languages with the same data size, we show the low representation quality of lowresource languages is not the result of the hyperparameters of bert or sharing the model with a large number of languages, as monolingual bert performs worse than mbert. on the contrary, by pairing low-resource languages with linguisticallyrelated languages, we show low-resource languages benefit from multilingual joint training, as bilingual bert outperforms monolingual bert while still lacking behind mbert, our findings suggest, with small monolingual corpus, bert does not learn high-quality representation for low resource languages. to learn better representation for low resource languages, we suggest either collect more data to make low resource language high resource @xcite , or consider more data-efficient pretraining techniques like @xcite . we leave exploring more data-efficient pretraining techniques as future work. several factors need to be considered in understanding mbert. first, the 104 most common wikipedia languages vary considerably in size ( finally, @xcite show that in general larger pretraining data for english leads to better downstream performance, yet increasing the size of pretraining data exponentially only increases downstream performance linearly. for a low resource language with limited pretraining data, it is unclear whether contextual representations outperform previous methods. ## experimental setup we begin by defining high and low resource languages in mbert, a description of the models and downstream tasks we use for evaluation, followed by a description of the masked language model pretraining. ## are all languages created equal in mbert? fig. 1 shows the performance of mbert and the baseline averaged across all languages by wikipedia size (see tab. 1 for groupings). for wikisize over 6, mbert is comparable or better than baselines in all three tasks, with the exception of ner. for ner in very high resource languages (wikisize over 11, i.e. top 10%) mbert performs worse than baseline, suggesting high resource languages could benefit from monolingual pretraining. note mbert has strong uas on parsing but weak las compared to the baseline; wu and dredze (2019) finds adding pos to mbert improve las significantly. we expect multitask learning on pos and parsing could further improve las. while pos and parsing only cover half (54) of the languages, ner covers 99 of 104 languages, extending the curve to the lowest resource languages. mbert performance drops significantly for languages with wikisize less than 6 (bottom 30% languages). for the smallest size, mbert goes from being competitive with state-of-the-art to being over 10 points behind. readers may find this surprising since while these are very low resource languages, mbert training up-weighted these languages to counter this effect. for ner over languages with different resources, where we show how much task-specific supervised training data was available for each language. for languages with only 100 labeled sentences, the performance of mbert drops significantly as these languages also had less pretraining data. while we may expect that pretraining representations with mbert would be most beneficial for languages with only 100 labels, as @xcite show pretraining improve data-efficiency for english on text classification, our results show that on low resource languages this strategy performs much worse than a model trained directly on the available task data. clearly, mbert provides variable quality representations depending on the language. while we confirm the finding of others that mbert is excellent for high resource languages, it coefficient p-value ci univariate training size 0.035 <0.001 [0.029, 0.041] training vocab 0.021 <0.001 [0.017, 0.025] wikisize 0.015 <0.001 [0.007, 0.023] multivariate training size 0.029 <0.001 [0.023, 0.035] wikisize -0.014 <0.001 [-0.022, -0.006] ## discussion while mbert covers 104 languages, the 30% languages with least pretraining resources perform worse than using no pretrained language model at all. therefore, we caution against using mbert alone for low resource languages. furthermore, training a monolingual model on low resource languages does no better. training on pairs of closely related low resource languages helps but still lags behind mbert. on the other end of the spectrum, the highest resource languages (top 10%) are hurt by massively multilingual joint training. while mbert has access to numerous languages, the resulting model is worse than a monolingual model when sufficient training data exists. developing pretrained language models for lowresource languages remains an open challenge. future work should consider more efficient pretraining techniques, how to obtain more data for low resource languages, and how to best make use of multilingual corpora.