Subword Evenness ( S u E ) as a Predictor of Cross-lingual Transfer to Low-resource Languages (2022)

Abstract: Pre-trained multilingual models, such as mBERT, XLM-R and mT5, are used to improve the performance on various tasks in low-resource languages via cross-lingual transfer. In this framework, English is usually seen as the most natural choice for a transfer language (for fine-tuning or continued training of a multilingual pre-trained model), but it has been revealed recently that this is often not the best choice. The success of cross-lingual transfer seems to depend on some properties of languages, which are currently hard to explain. Successful transfer often happens between unrelated languages and it often cannot be explained by data-dependent factors.In this study, we show that languages written in non-Latin and non-alphabetic scripts (mostly Asian languages) are the best choices for improving performance on the task of Masked Language Modelling (MLM) in a diverse set of 30 low-resource languages and that the success of the transfer is well predicted by our novel measure of Subword Evenness (SuE). Transferring language models over the languages that score low on our measure results in the lowest average perplexity over target low-resource languages. Our correlation coefficients obtained with three different pre-trained multilingual models are consistently higher than all the other predictors, including text-based measures (type-token ratio, entropy) and linguistically motivated choice (genealogical and typological proximity).

## introduction since pre-trained multilingual models became available, the most common approach to nlp tasks on low-resource languages has been cross-lingual transfer learning. after initial tests on cross-lingual ability within the languages covered by the multilingual models @xcite @xcite , a more recent line of research shows that new languages (not seen in figure 1 : best cross-lingual transfer results mentioned in previous work. the size of each circle is proportional to the number of mentions of a given transfer-target pair. english is the best transfer language for european target languages. for target languages outside of europe, the best transfer languages are hard to predict. the training set) are best processed by continued training or fine-tuning (depending on the task) of a pre-trained multilingual model on one transfer language, even if the transfer language is different from the target language. as a matter of fact, a single transfer language can help improve the performance on many target languages @xcite @xcite . this insight is especially important for low-resource languages, for which only test data is typically available. multilingual models are usually fine-tuned or (additionally) trained on a high-resource language and the resulting weights are applied as zero-or few-shot transfer to a target low-resource language @xcite . it seems that injecting a bias towards some particular linguistic traits, while slightly "forgetting" some others, makes the multilingual models adapt better to a new language. an evident but hard question arises here: which languages are especially well-suited to serve as transfer languages for many low-resource target languages? this is the first research question that we address in our study. most research on cross-lingual transfer regards english as the most natural choice for any target language due to the abundance of training data. however, more recent studies reveal that other languages are often better choices. for instance, russian turns out to be a good transfer language for thai, arabic, swahili, and chinese @xcite . it is hard to see why this happens as these languages are neither related nor similar by any of the known criteria. our own review of several studies on cross-lingual transfer, summarized in figure 1 , shows that english is the best transfer language when european languages are the target. for other target languages, the choice of the transfer languages is anything but clear. english is rarely the best choice, while russian, chinese, german, greek, or arabic seem to help in many cases. this observation leads us to the second question that we address in our study: which linguistic traits should be used to improve the performance of pre-trained multilingual models? in other words, what are the traits that make some languages more suitable for transfer? to answer these questions, we test language models on a highly diverse set of low-resource languages, whose geographical distribution is shown in figure 2 . we follow the current cross-lingual transfer learning framework in order to train the models. we consider three popular multilingual pre-trained models and 19 high-resource languages as potential transfer languages. in these experiments, we look for the best transfer languages, but also, more importantly, for a strong predictor of the transfer results across all target low-resource languages. as a predictor, we propose a novel text-based measure, subword evenness (sue), a parameter that describes the differences in the length of subword units (one value per language). the motivation for looking into the properties of subword units to assess the suitability of a given language to be a transfer language comes from the fact that language models are trained over the output of subword tokenization. we know that subword splits can depend on the properties of a language. for example, finnish is known for its regular subword structure (morphology), which is expected to give more even subword tokens, while other languages might have less subword regularity, leading to more uneven splits. our measure shows the preference for even splits in a given language. an example of a relatively even split would be co-work-@xmath0, 4, 3] compared to an uneven split co-@xmath1. a detailed explanation of the measure is in section 3. the main finding of our work is that the property of subword evenness matters for successful transfer to diverse non-indo-european languages. transfer languages that score low on our measure give the lowest average perplexity over target lowresource languages. compared to other text-based statistics and alternative linguistically motivated choices, our measure is the best predictor of transfer results. the code and data used for this work are publicly available in the repository https://github.com/olgapelloni/subword_evenness . ## related work the main idea behind cross-lingual transfer learning is that both similarities and differences between languages can be exploited for improving model performances. in early work on multilingual models, similarities between languages were exploited to increase training data, for instance in multilingual syntactic parsing @xcite and pivot-based statistical machine translation @xcite . crosslinguistic differences, as a form of distant supervision, have been shown to help the disambiguation of lexical meaning @xcite and part-of-speech (pos) tags @xcite . neural models made this idea even more attractive, leading to many proposals on how to share some parameters across languages, for instance through cascading and multi-task learning (p. @xcite . with transformer-based models @xcite , the transfer approach has proven to be widely applicable and successful, as shown by the 2018 model ulmfit @xcite . previous analyses of multilingual transformerbased models have shown that the representations produced during training are multilingual but the language-specific information is preserved @xcite @xcite . forgetting some of the language-specific information has been helpful for the task of question-answering @xcite . @xcite show a positive correlation between the number of shared tokens and transferability between languages, but this finding has not been confirmed in later studies. @xcite do not find a strong influence of lexical overlap on successful language transfer. @xcite add that the shared tokens help to create cross-lingual representations, but language transfer success depends more on the structural similarity between languages. studies on low-resource languages demonstrate how a lack of resources impacts performance @xcite @xcite . @xcite show that the performance of xlm-r is lower on low-resource languages than on high-resource ones. moreover, performance is lower on languages with non-latin scripts, such as hebrew, japanese, thai or chinese. the current processing workflows often include continued training of a multilingual model on a single high-resource transfer language and then applying the resulting weights in a few-or zero-shot manner to many low-resource languages. focusing on the question of which transfer languages give good results over multiple target languages, we count the mentions of the best transfer pairs in several previous studies @xcite @xcite @xcite @xcite . the counts are plotted in figure 1 , showing an interesting asymmetry between european and other languages. when european languages are the target of transfer, english seems to be the best transfer language. this is in line with the findings on the xtreme benchmark for evaluating crosslingual transfer @xcite , which led the authors to conclude that english is the most common and the most universal choice for a transfer language. @xcite found that english as a pivot language in statistical machine translation works well in approximately half of the observed language pairs (22 indo-european and asian languages). our review shows that english is not the best choice when non-european languages are the target. one approach to choosing a transfer language is to rely on structural similarity, which is measured using grammar features from the uriel database @xcite . @xcite find that transfer is better in language pairs that are closer in the uriel vector space regarding pos tagging and syntactic dependency parsing. other factors, such as data size, are better predictors on the tasks of question answering and inference. @xcite find that surface string similarity is the best predictor for pos tagging. @xcite develop their own linguistic typology based on negation constructions, which helps to choose a better transfer language for negation scope resolution. @xcite suggest aggregating various types of linguistic features, including geographic location. aggregated measures select better transfer candidates than single features. we depart from previous research in terms of both data and methods. we work with a much bigger and more diverse sample of languages than any previous studies. the main methodological novelty of our work is the proposed text-based parameter, which captures an interesting subword feature of good transfer languages. ## subword evenness (sue) as a language parameter defining formal properties of languages relevant to nlp is still a challenging task. @xcite , for instance, notice that language model sentence surprisal scores (aggregated at the level of a language) differ across languages, but do not manage to identify any properties of a language that would predict the surprisal. when it comes to transfer learning, previous research shows little agreement on which languages should be chosen for cross-lingual transfer and why. the full workflow for calculating the sue score (one value per language) is shown in figure 3 . we first apply a subword tokenization algorithm to split words in an unsupervised fashion. we then convert each segmented word @xmath2, w 2 , . . . w n in the data into a sequence of integers @xmath3, l 2 , . . . l n , where n is the number of subword segments in each word and each integer l i represents the length of the subword segment w i measured in unicode characters. we then map each sequence of integers to a single integer @xmath4, where u @xmath5. the values of u i are plotted against the values of word length measured in unicode characters (the x-axis in figure 3 ), resulting in a distribution, which then undergoes a kernel density estimation (kde) foot_0 analysis for identifying the shape of the density region. we aim to describe the whole distribution with a single parameter by approximating the shape of the density area with two lines and measuring the angle between them. the first line (intercepting the xaxis on the left side in figure 3 ) is a natural bound of the distribution f (x) = x -2 determined by the fact that the difference between any two subword segments cannot be greater than the length of the whole word. the second line is a linear function fitted to two points at the edge of the density area: max(x) and max(y), g(x) = kx + b. finally, the measure of subword evenness (equation 1 ) is the upper angle in the triangle formed by these two lines and the x-axis. for equation 1, we first calculate the inclination of each intersecting line using the formula @xmath6, where m is the slope of the line and θ is the inclination, i.e. the angle between the x-axis and the line @xcite . the slope of the first line f (x) equals 1, the slope of the second line g(x) equals k. in order to get the value of the inclination θ from tan(θ), we apply the inverse trigonometric function arctan(tan(θ)) = θ @xcite . we use absolute values in order to work with the angles measured in degrees. once we know the inclination of both lines (their angles to the x-axis), we can find the upper angle inside the triangle. since all angles in a triangle sum up to 180°, we subtract both inclinations from 180°to find the upper angle foot_1 . the upper angle shows the preference for subword evenness, thus called sue. the higher the value of this angle, the higher the preference towards even subword splits in longer words. there are many methods that can be used to perform unsupervised subword tokenization. the most widely used methods are byte-pair encoding (bpe, @xcite implemented by @xcite and also in the sentencepiece library @xcite , wordpiece @xcite and the sentencepiece unigram model. the output of all these algorithms is highly dependent on the hyperparameters that determine the size of the resulting subword vocabulary. however, there are currently no general criteria to determine the value of these hyperparameters @xcite . we follow the method of gutierrez-vasques et al. @xcite , which is motivated by information theoretic properties of segmented text in many languages: the bpe algorithm is run until the text redundancy value reaches its minimum. this criterion is independent of any particular nlp task and is crosslingually aligned, which is important for our study. cross-lingual alignment allows us to normalize, to a certain degree, the difference in writing systems: minimum redundancy is reached faster in alphabetic scripts and slower in syllabic and logographic scripts. for reference, we extract the sue measure using several other segmentation algorithms and hyperparameters. these results can be found in ## data and experiments we obtain our training and test data from the text data diversity sample (teddi, @xcite . the teddi corpus contains texts representing the languages included in the 100-language sample, published by the world atlas of language structures (wals, @xcite . this sample is compiled by experts in linguistic typology with the goal of representing overall linguistic diversity: language families, geographical areas and typological features. following this sample, the teddi corpus maximizes linguistic diversity and also covers different textual genres for languages with more available resources (gutenberg project foot_2 , the parallel bible corpus @xcite , the opensubtitles corpus @xcite and universal declaration of human rights 4 for the set of transfer languages (training set), we select 19 languages which contain at least one million tokens: basque (eus), english (eng), finnish (fin), french (fra), german (deu), greek (ell), hebrew (heb), hindi (hin), indonesian (ind), japanese (jpn), korean (kor), mandarin (cmn), persian (pes), russian (rus), spanish (spa), tagalog (tgl), thai (tha), turkish (tur), vietnamese (vie). we balance the genres when possible and cap the length of the sampled text to 1m tokens per language. we use the scikit-learn library @xcite to shuffle each dataset and split it into train (80%) and validation (20%) sets. for the set of target languages (test set), we define the threshold of at least 100k tokens available per language. there are 30 such languages: alamblak, amele, apurina, arabic (egyptian), arapesh (mountain), barasano, burmese, chamorro, daga, fijian, georgian, guarani, hausa, jakaltek, kewa, khalkha, khoekhoe, lango, malagasy, mapudungun, mixtec (chalcatongo), oromo (harar), quechua (imbabura), sango, sanuma, swahili, wichi, yagua, yaqui, yoruba. we fix the size of the test sets to be 100k tokens per language. the chosen languages belong to 23 different language families and are spoken in 5 geographical areas (figure 2 ). ## results and discussion the first general outcome of our experiments is the ranking of transfer languages according to the average perplexity score on the target languages shown in the second general outcome is the ranking of transfer languages according to the sue measure shown in ## conclusion in this study, we tackled the question of what languages are preferable for cross-lingual transfer when modelling diverse low-resource languages, and what motivates this selection. our experiments on the task of masked language modelling (mlm) with three multilingual pre-trained transformerbased models show that there is a small set of generally good transfer languages: japanese, greek, hebrew, thai, and russian. what is common to these languages is the fact that all of them are written in a non-latin script. however, the script alone is not the best general predictor of transfer performance. we show that the best predictor of the language model perplexity on a wide range of target low-resource languages is the subword evenness (sue) score of transfer languages, which we have presented in this paper. most of the languages that repeatedly come out as good transfer languages have low sue scores. our results are largely in line with the observations about good transfer languages made in previous work. in addition to providing new evidence confirming previous findings on a very diverse sample of languages (19 transfer and 30 target languages), we identify the strongest predictor of the observed performances up to now. the proposed sue measure is a better predictor for transfer languages than other text-based measures (mean word length, type-token ratio, unigram entropy) as well as genealogical and typological proximity. with this finding, we make a further step towards explaining why continued training on transfer languages is helpful for modelling low-resource languages.