How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2021)

Abstract: In this work, we provide a systematic and comprehensive empirical comparison of pretrained multilingual language models versus their monolingual counterparts with regard to their monolingual task performance. We study a set of nine typologically diverse languages with readily available pretrained monolingual models on a set of five diverse monolingual downstream tasks. We first aim to establish, via fair and controlled comparisons, if a gap between the multilingual and the corresponding monolingual representation of that language exists, and subsequently investigate the reason for any performance difference. To disentangle conflating factors, we train new monolingual models on the same data, with monolingually and multilingually trained tokenizers. We find that while the pretraining data size is an important factor, a designated monolingual tokenizer plays an equally important role in the downstream performance. Our results show that languages that are adequately represented in the multilingual model’s vocabulary exhibit negligible performance decreases over their monolingual counterparts. We further find that replacing the original multilingual tokenizer with the specialized monolingual tokenizer improves the downstream performance of the multilingual model for almost every task and language.

## introduction following large transformer-based language models @xcite pretrained on large english corpora (e.g., bert, roberta, t5; @xcite @xcite , similar monolingual language models have been introduced for other languages @xcite our code is available at https://github.com/adapter-hub/hgiyt . @xcite , offering previously unmatched performance in all nlp tasks. concurrently, massively multilingual models with the same architectures and training procedures, covering more than 100 languages, have been proposed (e.g., mbert, xlm-r, mt5; @xcite @xcite . the "industry" of pretraining and releasing new monolingual bert models continues its operations despite the fact that the corresponding languages are already covered by multilingual models. the common argument justifying the need for monolingual variants is the assumption that multilingual models-due to suffering from the so-called curse of multilinguality @xcite ., the lack of capacity to represent all languages in an equitable way)-underperform monolingual models when applied to monolingual tasks @xcite @xcite . however, little to no compelling empirical evidence with rigorous experiments and fair comparisons have been presented so far to support or invalidate this strong claim. in this regard, much of the work proposing and releasing new monolingual models is grounded in anecdotal evidence, pointing to the positive results reported for other monolingual bert models @xcite @xcite . monolingual bert models are typically evaluated on downstream nlp tasks to demonstrate their effectiveness in comparison to previous monolingual models or mbert @xcite @xcite . while these results do show that certain monolingual models can outperform mbert in certain tasks, we hypothesize that this may substantially vary across different languages and language properties, tasks, pretrained models and their pretraining data, domain, and size. we further argue that conclusive evidence, either supporting or refuting the key hypothesis that monolingual models currently outperform multilingual models, necessitates an independent and controlled empirical comparison on a diverse set of languages and tasks. while recent work has argued and validated that mbert is under-trained @xcite , providing evidence of improved performance when training monolingual models on more data, it is unclear if this is the only factor relevant for the performance of monolingual models. another so far under-studied factor is the limited vocabulary size of multilingual models compared to the sum of tokens of all corresponding monolingual models. our analyses investigating dedicated (i.e., language-specific) tokenizers reveal the importance of high-quality tokenizers for the performance of both model variants. we also shed light on the interplay of tokenization with other factors such as pretraining data size. contributions. 1) we systematically compare monolingual with multilingual pretrained language models for 9 typologically diverse languages on 5 structurally different tasks. 2) we train new monolingual models on equally sized datasets with different tokenizers (i.e., shared multilingual versus dedicated language-specific tokenizers) to disentangle the impact of pretraining data size from the vocabulary of the tokenizer. 3) we isolate factors that contribute to a performance difference (e.g., tokenizers' "fertility", the number of unseen (sub)words, data size) and provide an in-depth analysis of the impact of these factors on task performance. 4) our results suggest that monolingually adapted tokenizers can robustly improve monolingual performance of multilingual models. the final shared mbert vocabulary comprises a total of 119,547 subword tokens. other multilingual models followed mbert, such as xlm-r @xcite . concurrently, many studies analyzed mbert's and xlm-r's capabilities and limitations, finding that the multilingual models work surprisingly well for cross-lingual tasks, despite the fact that they do not rely on direct cross-lingual supervision (e.g., parallel or comparable data, translation dictionaries; @xcite @xcite @xcite . however, recent work has also pointed to some fundamental limitations of multilingual lms. @xcite observe that, for a fixed model capacity, adding new languages increases crosslingual performance up to a certain point, after which adding more languages results in performance drops. this phenomenon, termed the curse of multilinguality, can be attenuated by increasing the model capacity @xcite @xcite or through additional training for particular language pairs @xcite . another observation concerns substantially reduced crosslingual and monolingual abilities of the models for resource-poor languages with smaller pretraining data @xcite @xcite . those languages remain underrepresented in the subword vocabulary and the model's shared representation space despite oversampling. despite recent efforts to mitigate this issue (e.g., @xcite propose to cluster and merge the vocabularies of similar languages, before defining a joint vocabulary across all languages), the multilingual lms still struggle with balancing their parameters across many languages. monolingual versus multilingual lms. new monolingual language-specific models also emerged for many languages, following bert's architecture and pretraining procedure. there are monolingual bert variants for arabic @xcite , french @xcite , finnish @xcite , to name only a few. @xcite released 44 monolingual wikibert models trained on wikipedia. however, only a few studies have thus far, either explicitly or implicitly, attempted to understand how monolingual and multilingual lms compare across languages. @xcite extracted task results from the respective papers on monolingual berts to facilitate an overview of monolingual models and their comparison to mbert. foot_0 however, they have not verified the scores, nor have they performed a controlled impartial comparison. @xcite probed mbert and monolingual bert models across six typologically diverse languages for lexical semantics. they show that pretrained monolingual bert models encode significantly more lexical information than mbert. zhang et al. ( 2020 ) investigated the role of pretraining data size with roberta, finding that the model learns most syntactic and semantic features on corpora spanning 10m-100m word tokens, but still requires massive datasets to learn higher-level semantic and commonsense knowledge. @xcite compared monolingual and bilingual elmo @xcite lms across three downstream tasks, finding that contextualized representations from the bilingual models can improve monolingual task performance relative to their monolingual counterparts. foot_1 however, it is unclear how their findings extend to massively multilingual lms potentially suffering from the curse of multilinguality. @xcite compared mbert to monolingual bert models for six languages (german, english, swedish, danish, norwegian, finnish) on three different tasks. they find that mbert lags behind its monolingual counterparts in terms of performance on cloze and generation tasks. they also identified clear differences among the six languages in terms of this performance gap. they speculate that mbert is under-trained with respect to individual languages. however, their set of tasks is limited, and their language sample is typologically narrow; it remains unclear whether these findings extend to different language families and to structurally different tasks. despite recent efforts, a careful, systematic study within a controlled experimental setup, a diverse language sample and set of tasks is still lacking. we aim to address this gap in this work. ## controlled experimental setup we compare multilingual bert with its monolingual counterparts in a spectrum of typologically diverse languages and across a variety of downstream tasks. by isolating and analyzing crucial factors contributing to downstream performance, such as tokenizers and pretraining data, we can conduct unbiased and fair comparisons. ## further analysis at first glance, our results displayed in with monolingual tokenizers with lower fertility and proportion-of-continued-words values than the mbert tokenizer (such as for ar, fi, id, ko, tr), consistent gains can be achieved, irrespective of whether the lms are monolingual (the monomodel-* comparison) or multilingual (a comparison of mbertmodel-* variants). whenever the differences between monolingual models and mbert with respect to the tokenizer properties and the pretraining corpus size are small (e.g., for en, ja, and zh), the performance gap is typically negligible. in qa, we even find mbert to be favorable for these languages. therefore, we conclude that monolingual models are not superior to multilingual ones per se, but gain advantage in direct comparisons by incorporating more pretraining data and using language-adapted tokenizers. figure 3: spearman's ρ correlation of a relative decrease in the proportion of continued words (cont. proportion), a relative decrease in fertility, and a relative increase in pretraining corpus size with a relative increase in downstream performance over fully finetuned mbert. for the proportion of continued words and the fertility, we consider fully fine-tuned mbert, the monomodel-* models, and the mbertmodel-* models. for the pretraining corpus size, we consider the original monolingual models and the monomodel-monotok models. we exclude the id models (see appendix b.2 for the clarification). ## conclusion we have conducted the first comprehensive empirical investigation concerning the monolingual performance of monolingual and multilingual language models (lms). while our results support the existence of a performance gap in most but not all languages and tasks, further analyses revealed that the gaps are often substantially smaller than what was previously assumed. the gaps exist in certain languages due to the discrepancies in 1) pretraining data size, and 2) chosen tokenizers, and the level of their adaptation to the target language. further, we have disentangled the impact of pretrained corpora size from the influence of the tokenizers on the downstream task performance. we have trained new monolingual lms on the same data, but with two different tokenizers; one being the dedicated tokenizer of the monolingual lm provided by native speakers; the other being the automatically generated multilingual mbert tokenizer. we have found that for (almost) every task and language, the use of monolingual tokenizers outperforms the mbert tokenizer. consequently, in line with recent work by chung et al. ( 2020 ), our results suggest that investing more effort into 1) improving the balance of individual languages' representations in the vocabulary of multilingual lms, and 2) providing languagespecific adaptations and extensions of multilingual tokenizers @xcite can reduce the gap between monolingual and multilingual lms. another promising future research direction is completely disposing of any (language-specific or multilingual) tokenizers during pretraining @xcite . our code, pretrained models, and adapters are available at https://github.com/adapter-hub/hgiyt .