Multilingual ELM o and the Effects of Corpus Sampling (2021)

Abstract: Multilingual pretrained language models are rapidly gaining popularity in NLP systems for non-English languages. Most of these models feature an important corpus sampling step in the process of accumulating training data in different languages, to ensure that the signal from better resourced languages does not drown out poorly resourced ones. In this study, we train multiple multilingual recurrent language models, based on the ELMo architecture, and analyse both the effect of varying corpus size ratios on downstream performance, as well as the performance difference between monolingual models for each language, and broader multilingual language models. As part of this effort, we also make these trained models available for public use.

## introduction as part of the recent emphasis on language model pretraining, there also has been considerable focus on multilingual language model pretraining; this is distinguished from merely training language models in multiple languages by the creation of a multilingual space. these have proved to be very useful in 'zero-shot learning'; i.e., training on a wellresourced language (typically english), and relying on the encoder's multilingual space to create reasonable priors across languages. the main motivation of this paper is to study the effect of corpus sampling strategy on downstream performance. further, we also examine the utility of multilingual models (when constrained to monolingual tasks), over individual monolingual models, one per language. this paper therefore has two main contributions: the first of these is a multilingual elmo model that we hope would see further use in probing studies as well as evaluative studies, downstream; we train these models over 13 languages, namely arabic, basque, chinese, english, finnish, hebrew, hindi, italian, japanese, korean, russian, swedish and turkish. the second contribution is an analysis of sampling mechanism on downstream performance; we elaborate on this later. in section 2 of this paper, we contextualise our work in the present literature. section 3 describes our experimental setup and section 4 our results. finally, we conclude with a discussion of our results in section 5. ## prior work multilingual embedding architectures (static or contextualised) are different from cross-lingual ones @xcite in that they are not products of aligning several monolingual models. instead, a deep neural model is trained end to end on texts in multiple languages, thus making the whole process more straightforward and yielding truly multilingual representations @xcite . following artetxe et al. (2020), we will use the term 'deep multilingual pretraining' for such approaches. one of the early examples of deep multilingual pretraining was bert, which featured a multilingual variant trained on the 104 largest languagespecific wikipedias @xcite . to counter the effects of some languages having overwhelmingly larger wikipedias than others, @xcite used exponentially smoothed data weighting; i.e., they exponentiated the probability of a token being in a certain language by a certain α, and re-normalised. this has the effect of 'squashing' the distribution of languages in their training data; larger languages become smaller, to avoid drowning out the signal from smaller languages. one can also look at this technique as a sort of sampling. other multilingual models, such as xlm @xcite and its larger variant, xlm-r @xcite , use different values of α for this sampling (0.5 and 0.3 respectively). the current paper is aimed at analysing the effects of different α choices; in spirit, this work is very similar to @xcite ; where it differs is our analysis on downstream tasks, as opposed to machine translation, where models are trained and evaluated on a very specific task. we also position our work as a resource, and we make our multilingual elmo models available for public use. ## results first, we examine the costs of multilingualism, as far as monolingual tasks are concerned. we present our results on our monolingual tasks in figure 1 . monolingual models appear to perform consistently better, particularly pos tagging; this appears to be especially true for our underresourced languages, strengthening the claim that compression is necessary to avoid drowning out signal. for pos tagging, the correlation between performance difference (monolingual vs. m0.4) and corpus size is highly significant (ρ = 0.74; @xmath0.006). we find that compression appears to result in visible improvements, when moving from α = 0.4 to α = 0.2. these improvements, while not dramatic, apply across the board (see figure 2b also shows the difference in performance between the truncated model, trunc, and m0.4; this is a lot less convincing than the difference to m0.2, indicating that no additional advantage is to be gained by downsampling data for better-resourced languages. we include full, detailed results in appendix c. ## discussion our results allow us to draw conclusions that come across as very 'safe': some compression helps, too much hurts; when compression does help, however, the margin appears rather moderate yet significant for most tasks, even given fewer training cycles. immediately visible differences along linguistic lines do not emerge when ratios differ, despite the relative linguistic diversity of our language choices; we defer analysis of this to a future work, that is less focused on downstream analysis, and more on carefully designed probes that might illuminate the difference between our models' internal spaces. note that a possible confounding factor in our results is also the complexity of the architectures we build on top of melmo: they also have significant learning capacity, and it is not implausible that whatever differences there are between our models, are drowned out by highly parameterised downstream decoders. to reiterate, this study is not (nor does it aim to be) a replication of models with far larger parameter spaces and more training data. this is something of a middle-of-the-road approach; future work could involve this sort of evaluation on downscaled transformer models, which we shy away from in order to provide a usable model release. we hope that the differences between these models provide some insight, and pave the way for further research, not only specifically addressing the question of sampling from a perspective of performance, but also analytically. there has already been considerable work in this direction on multilingual variants of bert @xcite , and we hope that this work motivates papers applying the same to recurrent melmo, as well as comparing and contrasting the two. the elmo models described in this paper are publicly released via nlpl vector repository.