Identifying Elements Essential for BERT ’s Multilinguality (2020)

Abstract: It has been shown that multilingual BERT (mBERT) yields high quality multilingual representations and enables effective zero-shot transfer. This is surprising given that mBERT does not use any crosslingual signal during training. While recent literature has studied this phenomenon, the reasons for the multilinguality are still somewhat obscure. We aim to identify architectural properties of BERT and linguistic properties of languages that are necessary for BERT to become multilingual. To allow for fast experimentation we propose an efficient setup with small BERT models trained on a mix of synthetic and natural data. Overall, we identify four architectural and two linguistic elements that influence multilinguality. Based on our insights, we experiment with a multilingual pretraining setup that modifies the masking strategy using VecMap, i.e., unsupervised embedding alignment. Experiments on XNLI with three languages indicate that our findings transfer from our small setup to larger scale settings.

## introduction multilingual models, i.e., models capable of processing more than one language with comparable performance, are central to natural language processing. they are useful as fewer models need to be maintained to serve many languages, resource requirements are reduced, and low-and mid-resource languages can benefit from crosslingual transfer. further, multilingual models are useful in machine translation, zero-shot task transfer and typological research. there is a clear need for multilingual models for the world's 7000+ languages. with the rise of static word embeddings, many multilingual embedding algorithms have been proposed @xcite @xcite ; for a survey together with overparameterization almost no multilinguality is left (17). pairing a language with its inversion (i.e., inverted word order) destroys multilinguality as well (3). having parallel training corpora is helpful for multilinguality (not shown). results are for embeddings from layer 8. see @xcite . pretrained language models @xcite @xcite have high performance across tasks, outperforming static word embeddings. a simple multilingual model is multilingual bert foot_0 (mbert). it is a bert-base model @xcite trained on the 104 largest wikipedias with a shared subword vocabulary. there is no additional crosslingual signal. still, mbert yields highquality multilingual representations @xcite @xcite . the exact reason for mbert's multilinguality is -to the best of our knowledge -still debated. @xcite provide an extensive study and conclude that a shared vocabulary is not necessary, but that the model needs to be deep and languages need to share a similar "structure". @xcite show that neither a shared vocabulary nor joint pretraining is required for bert to be multilingual. @xcite find that bert models across languages can be easily aligned and that a necessary requirement for achieving multilinguality are shared parameters in the top layers. this work continues this line of research. we find indications that six elements influence the multilinguality of bert. figure 1 summarizes our main findings. ## improving multilinguality so far we have tried to break bert's multilinguality. now we turn to exploiting our insights for improving it. mbert has shared position embeddings, shared special tokens and we cannot change linguistic properties of languages. our results on overparameterization suggest that smaller models become multilingual faster. however, mbert may already be considered underparameterized given that it is trained on 104 large wikipedias. one insight we can leverage for the masking procedure is no-random: replacing masked words with random tokens. we propose to introduce a fourth masking option: replacing masked tokens with semantically similar words from other languages. to this end we train static fasttext embeddings @xcite on the training set and then project them into a common space using vecmap @xcite . we use this crosslingual space to replace masked tokens with nearest neighbors from the other language. each masked word is then replaced with the probabilities (p @xcite , p @xcite , p @xcite , p @xcite ) = (0.5, 0.1, 0.1, 0.3), i.e., in 30% of the cases masked words get replaced with the nearest neighbor from the multilingual static embedding space. note that this procedure (including vecmap) is fully unsupervised (i.e., no parallel data or dictionary required). we call this method knn-replace. @xcite performed similar experiments by creating code switched data and adding it to the training data. however, we only replace masked words. figure 7 shows the multilinguality score and model fit over training time. compared to the original model in figure 6 , retrieval and translation have higher scores earlier. towards the end multilinguality scores become similar, with knn-replace outperforming the original model (see ## related work there is a range of prior work analyzing the reason for bert's multilinguality. @xcite show that bert stores language representations in different subspaces and investigate how subword tokenization influences multilinguality. @xcite show that neither a shared vocabulary nor joint pretraining is essential for multilinguality. k et al. (2020) extensively study reasons for multilinguality (e.g., researching depth, number of parameters and attention heads). they conclude that depth is essential. they also investigate language properties and conclude that structural similarity across languages is important, without further defining this term. last, @xcite find that a shared vocabulary is not required. they find that shared parameters in the top layers are required for multilinguality. further they show that different monolingual bert models exhibit a similar structure and thus conclude that mbert somehow aligns those isomorphic spaces. they investigate having separate embedding look-ups per language (including position embeddings and special tokens) and a variant of avoiding cross-language replacements. their method "extra anchors" yields a higher degree of multilinguality. in contrast to this prior work, we investigate multilinguality in a clean laboratory setting, investigate the interaction of architectural aspects and research new aspects such as overparameterization or inv-order. other work focuses on creating better multilingual models. @xcite proposed a method to learn multilingual contextual representations. @xcite introduce the translation modeling objective. @xcite propose xlm-r. they introduce the term "curse of multilinguality" and show that multilingual model quality degrades with an increased number of languages given a fixed number of parameters. this can be interpreted as the minimum number of parameters required whereas we find indications that models that are too large can be harmful for multilinguality as well. @xcite improve the multilinguality of mbert by introducing a regularization term in the objective, similar to the creation of static multilingual embedding spaces. @xcite extend mbert pretraining with three additional tasks and show an improved overall performance. more recently, better multilinguality is achieved by @xcite (adapters) and @xcite . we propose a simple extension to make mbert more multilingual; it does not require additional supervision, parallel data or a more complex loss function -in contrast to this prior work. finally, many papers find that mbert yields competitive zero-shot performance across a range of languages and tasks such as parsing and ner @xcite , word alignment and sentence retrieval @xcite and language generation @xcite ; @xcite show this for 40 languages and 9 tasks. wu and dredze (2020) consider the performance on up to 99 languages for ner. in contrast, @xcite show limitations of the zero-shot setting and @xcite observe poor performance of mbert in reference-free machine translation evaluation. prior work here focuses on investigating the degree of multilinguality, not the reasons for it. ## conclusion we investigated which architectural and linguistic properties are essential for bert to yield crosslingual representations. the main takeaways are: i) shared position embeddings, shared special tokens, replacing masked tokens with random tokens and a limited amount of parameters are necessary elements for multilinguality. ii) word order is relevant: bert is not multilingual with one language having an inverted word order. iii) the comparability of training corpora contributes to multilinguality. we show that our findings transfer to larger scale settings. we experimented with a simple modification to obtain stronger multilinguality in bert models and demonstrate its effectiveness on xnli. we considered a fully unsupervised setting without any crosslingual signals. in future work we plan to incorporate crosslingual signals as @xcite argue that a fully unsupervised setting is hard to motivate.