Emerging Cross-lingual Structure in Pretrained Language Models (2020)

Abstract: We study the problem of multilingual masked language modeling, i.e. the training of a single model on concatenated text from multiple languages, and present a detailed study of several factors that influence why these models are so effective for cross-lingual transfer. We show, contrary to what was previously hypothesized, that transfer is possible even when there is no shared vocabulary across the monolingual corpora and also when the text comes from very different domains. The only requirement is that there are some shared parameters in the top layers of the multi-lingual encoder. To better understand this result, we also show that representations from monolingual BERT models in different languages can be aligned post-hoc quite effectively, strongly suggesting that, much like for non-contextual word embeddings, there are universal latent symmetries in the learned embedding spaces. For multilingual masked language modeling, these symmetries are automatically discovered and aligned during the joint training process.

## introduction multilingual language models such as mbert @xcite and xlm @xcite enable effective cross-lingual transfer -it is possible to learn a model from supervised data in one language and apply it to another with no additional training. recent work has shown that transfer is effective for a wide range of tasks @xcite . these work speculates why multilingual pretraining works (e.g. shared vocabulary), but only experiment with a single reference mbert and is unable to systematically measure these effects. in this paper, we present the first detailed empirical study of the effects of different masked lan-guage modeling (mlm) pretraining regimes on cross-lingual transfer. our first set of experiments is a detailed ablation study on a range of zero-shot cross-lingual transfer tasks. much to our surprise, we discover that language universal representations emerge in pretrained models without the requirement of any shared vocabulary or domain similarity, and even when only a subset of the parameters in the joint encoder are shared. in particular, by systematically varying the amount of shared vocabulary between two languages during pretraining, we show that the amount of overlap only accounts for a few points of performance in transfer tasks, much less than might be expected. by sharing parameters alone, pretraining learns to map similar words and sentences to similar hidden representations. to better understand these effects, we also analyze multiple monolingual bert models trained independently. we find that monolingual models trained in different languages learn representations that align with each other surprisingly well, even though they have no shared parameters. this result closely mirrors the widely observed fact that word embeddings can be effectively aligned across languages @xcite . similar dynamics are at play in mlm pretraining, and at least in part explain why they aligned so well with relatively little parameter tying in our earlier experiments. this type of emergent language universality has interesting theoretical and practical implications. we gain insight into why the models transfer so well and open up new lines of inquiry into what properties emerge in common in these representations. they also suggest it should be possible to adapt pretrained models to new languages with little additional training and it may be possible to better align independently trained representations without having to jointly train on all of the (very large) unlabeled data that could be gathered. for example, concurrent work has shown that a pre-trained mlm model can be rapidly fine-tuned to another language @xcite . this paper offers the following contributions: with bilingual dictionary, we propose a simple technique to create more anchor points by creating synthetic code-switched corpus, benefiting especially distantly-related languages. ## background language model pretraining our work follows in the recent line of language model pretraining. elmo @xcite first popularized representation learning from a language model. the representations are used in a transfer learning setup to improve performance on a variety of downstream nlp tasks. follow-up work by @xcite ; @xcite further improves on this idea by fine-tuning the entire language model. bert @xcite significantly outperforms these methods by introducing a masked-language model and next-sentence prediction objectives combined with a bi-directional transformer model. the multilingual version of bert (dubbed mbert) trained on wikipedia data of over 100 languages obtains strong performance on zeroshot cross-lingual transfer without using any parallel data during training @xcite . this shows that multilingual representations can emerge from a shared transformer with a shared subword vocabulary. crosslingual language model (xlm) pretraining (lample and conneau, 2019) was introduced concurrently to mbert. on top of multilingual masked language models, they investigate an objective based on parallel sentences as an explicit crosslingual signal. xlm shows that cross-lingual language model pretraining leads to a new state of the art on xnli @xcite , supervised and unsupervised machine translation @xcite . other work has shown that mbert outperforms word embeddings on token-level nlp tasks @xcite , and that adding character-level information @xcite and using multi-task learning @xcite can improve cross-lingual performance. alignment of word embeddings researchers working on word embeddings noticed early that embedding spaces tend to be shaped similarly across different languages @xcite . this inspired work in aligning monolingual embeddings. the alignment was done by using a bilingual dictionary to project words that have the same meaning close to each other @xcite . this projection aligns the words outside of the dictionary as well due to the similar shapes of the word embedding spaces. follow-up efforts only required a very small seed dictionary (e.g., only numbers @xcite ) or even no dictionary at all @xcite . other work has pointed out that word embeddings may not be as isomorphic as thought @xcite especially for distantly related language pairs @xcite . @xcite show joint training can lead to more isomorphic word embeddings space. @xcite showed that elmo embeddings can be aligned by a linear projection as well. they demonstrate a strong zero-shot crosslingual transfer performance on dependency parsing. @xcite align mbert representations and evaluate on dependency parsing as well. ## cross-lingual pretraining we study a standard multilingual masked language modeling formulation and evaluate performance on several different cross-lingual transfer tasks, as described in this section. ## dissecting mbert/xlm models we hypothesize that the following factors play important roles in what makes multilingual bert multilingual: domain similarity, shared vocabulary (or anchor points), shared parameters, and language similarity. without loss of generality, we focus on bilingual mlm. we consider three pairs of languages: english-french, english-russian, and english-chinese. ## similarity of bert models to better understand the robust transfer effects of the last section, we show that independently trained monolingual bert models learn representations that are similar across languages, much like the widely observed similarities in word embedding spaces. in this section, we show that independent monolingual bert models produce highly similar representations when evaluated at the word level ( §5.1.1), contextual word-level ( §5.1.2), and sentence level ( §5.1.3) . we also plot the cross-lingual similarity of neural network activation with center kernel alignment ( §5.2) at each layer. we consider five languages: english, french, german, russian, and chinese. ## discussion in this paper, we show that multilingual representations can emerge from unsupervised multilingual masked language models with only parameter sharing of some transformer layers. even without any anchor points, the model can still learn to map representations coming from different languages in a single shared embedding space. we also show that isomorphic embedding spaces emerge from monolingual masked language models in different languages, similar to word2vec embedding spaces @xcite . by using a linear mapping, we are able to align the embedding layers and the contextual representations of transformers trained in different languages. we also use the cka neural network similarity index to probe the similarity between bert models and show that the early layers of the transformers are more similar across languages than the last layers. all of these effects were stronger for more closely related languages, suggesting there is room for significant improvements on more distant language pairs.