How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training? (2022)

Abstract: A multilingual tokenizer is a fundamental component of multilingual neural machine translation. It is trained from a multilingual corpus. Since a skewed data distribution is considered to be harmful, a sampling strategy is usually used to balance languages in the corpus. However, few works have systematically answered how language imbalance in tokenizer training affects downstream performance. In this work, we analyze how translation performance changes as the data ratios among languages vary in the tokenizer training corpus. We find that while relatively better performance is often observed when languages are more equally sampled, the downstream performance is more robust to language imbalance than we usually expected. Two features, UNK rate and closeness to the character level, can warn of poor downstream performance before performing the task. We also distinguish language sampling for tokenizer training from sampling for model training and show that the model is more sensitive to the latter.

## introduction tokenization is an essential pre-processing step for most natural language processing (nlp) models. out of different tokenization methods, subword tokenization @xcite @xcite has become de facto. the creation of each subword is mainly based on frequency, i.e., if two characters often appear together, they will be merged into a subword. when more than one language is involved, instead of learning independent tokenizers for each language, people usually train a joint tokenizer from a multilingual training corpus @xcite . in this case, the data percentage of each language directly affects how it will be represented. if one language dominates the training corpus, its words will mostly stay intact and hardly be split into subwords. in contrast, if the language gets starved, it will be excessively tokenized into characters, thus, the sentence length will be dramatically longer, and some tokens will be considered as unknown (unk). moreover, neural machine translation (nmt) is known to be bad at dealing with long sentences and unks @xcite . recently, there is an increasing interest in building multilingual neural models that can process multiple languages @xcite @xcite . a challenge that comes with this important task is to balance languages with different amounts of training data to avoid low-resource languages being under-represented, e.g., being excessively tokenized and being less seen by the neural models. existing works usually adopt the temperature sampling strategy @xcite @xcite (see detailed descriptions in section 2.2). however, very few investigations of how language imbalance affects downstream performance have been conducted. additionally, whenever previous works apply a certain language balancing strategy, they apply it for both tokenizer training (balancing the data sizes of different languages in the tokenizer training corpus) and model training (balancing the frequencies of sampling training mini-batches from different languages). until now, it is unclear how each of them separately affects the downstream performance. in this work, we specifically investigate how robust nmt is to language imbalance in tokenizer training. we propose to vary the data ratio among languages in the tokenizer training corpus while keeping other settings (e.g., language sampling for model training, hyperparameters) fixed, and then check how translation results change (section 3.1). however, finding the best data ratio through performing the downstream task is highly expensive. to provide an easy indication of tokenizer quality (or early prediction of downstream performance), we examine two intermediate features (section 3.2): unk rate -the average percentage of unknown words (marked with the unk token) in each sentence, and closeness to the character level -the average sentence length in subwords divided by sentence length in characters. through comprehensive bilingual and multilingual experiments among 8 languages (english, tagalog, icelandic, danish, indonesian, tamil, greek, and chinese), we make the following five main observations: (1) nmt performance is more robust to language imbalance than we usually expected: especially when languages share scripts, performance drops only happen when the data ratio of two languages is as disparate as 1:10 5 . (2) better performance is often achieved when languages are more balanced: we observe moderate pearson correlations between translation performance and the degree of language balance. (3) english can "never" be starved because english tokens often appear in the "monolingual" data of other languages. (4) in most cases, the two features (unk rate and closeness to the character level) can hint at poor translation performances before performing the task. (5) nmt is more sensitive to language imbalance in model training than in tokenizer training. see more observations and discussions in section 3 and section 4. based on these observations, we provide the following two practical suggestions: (1) instead of using temperature sampling, we want to keep the involved languages as balanced as possible when training a new multilingual tokenizer; (2) before applying a pretrained tokenizer for new experiments or languages, we suggest evaluating it on a development set to make sure every language's unk rate is low (lower than around 3.7%, according to our experiments) and every language's closeness to the character level is also low (lower than around 0.87, according to our experiments). foot_0 2 related works ## bilingual experiments to examine how language imbalance in tokenizer training affects downstream translation performance, we first conduct english-centric bilingual experiments in which imbalance only happens for one single pair of languages (i.e., english and anther language). this gives us a more controlled setting compared to when multiple languages are involved. nonetheless, we conduct multilingual experiments in section 4. our main methodology is to keep the total tokenizer training data size fixed, gradually "starve" english, i.e., reduce english data percentage and increase the percentage of the other language, and then check the downstream translation performance. it is important to note that, to separate the influences of tokenizer and model, we use different data for tokenizer training and model training, and the model training data are always the same. ## multilingual experiments here, we move to a more complex multilingual setting. similarly, we want to understand how the data percentages of the involved languages affect their downstream translation performance. ## conclusion we systematically analyze how language imbalance in multilingual tokenizer training affects translation performances. overall, we find that nmt performance is quite robust to language imbalance especially when languages share scripts. better performance is often achieved when languages are more balanced. we suggest keeping the involved languages as balanced as possible in the tokenizer training corpus and evaluating pretrained tokenizers on an evaluation set to make sure no language's unk @xmath0.7% and no language's closeness to the character @xmath1.87. we hope our work can provide some guidance for future multilingual tokenizer training and usage.