Combining Static and Contextualised Multilingual Embeddings (2022)

Abstract: Static and contextual multilingual embeddings have complementary strengths. Static embeddings, while less expressive than contextual language models, can be more straightforwardly aligned across multiple languages. We combine the strengths of static and contextual models to improve multilingual representations. We extract static embeddings for 40 languages from XLM-R, validate those embeddings with cross-lingual word retrieval, and then align them using VecMap. This results in high-quality, highly multilingual static embeddings. Then we apply a novel continued pre-training approach to XLM-R, leveraging the high quality alignment of our static embeddings to better align the representation space of XLM-R. We show positive results for multiple complex semantic tasks. We release the static embeddings and the continued pre-training code. Unlike most previous work, our continued pre-training approach does not require parallel text.

## introduction multilingual contextual encoders like xlm-r @xcite and mbert @xcite , despite being trained without parallel data, exhibit "surprising" cross-linguality @xcite and have demonstrated strong performance on multilingual and cross-lingual tasks (e.g., @xcite @xcite . however, their languageneutrality, meaning how well languages are aligned with each other, has clear limits @xcite . in particular, more typologically distant language pairs tend to be less well-aligned than more similar ones, affecting transfer performance. -m ). then, we use vecmap to align the embeddings further, giving us x2s-ma. the german example sentence reads 'the cat is striped'. b) taking the representations of words from x2s-ma and xlm-r, we train the contextual model representations to be more similar to the wellaligned static embeddings via an alignment loss (cca or mse). the french example 'chat' means 'cat'. by contrast, cross-lingual alignment is wellstudied for static embeddings (e.g., @xcite @xcite . they can be aligned using simple transformations, resulting in high quality multilingual embeddings. however, static embeddings are considerably less expressive than contextual models and have in many applications been superseded by them. this paper aims to combine the strengths of static and contextual models, and explore how they may benefit from each other. our methods require no parallel corpus. figure 1 shows a schematic of our two-part approach with an example: the words 'katze' in figure 1a and 'chat' in figure 1b both mean 'cat'. while creating x2s-m in figure 1a , static vectors are learned for both words in their respective language embeddings. we then align those embeddings with vecmap, obtaining x2s-ma. in fig 1b , we train the contextualised embeddings of 'katze' and 'chat' to be closer to their well-aligned x2s-ma vectors, improving the alignment of the contextualised embeddings. monolingual static embeddings have been extracted from bert by @xcite . we show that their approach can be applied to multilingual embeddings. we distill static embeddings for 40 languages from xlm-r, showing that the resulting embeddings are already somewhat crosslingually aligned, but that their alignment can be improved using established tools (figure 1a ; § 3). these vectors are of high monolingual and crosslingual quality despite being distilled using only 1m sentences per language. second, we present a novel continued pre-training approach for the contextual model, combining masked language modelling (mlm) with an alignment loss that leverages the aligned static embeddings (figure 1b ; § 4). this results in improved multilingual contextualised embeddings which work well for complex semantic tasks. recently, cross-lingual alignment objectives have been used to train multilingual contextual models from scratch @xcite , to align the outputs of monolingual models @xcite , or to apply a post-hoc alignment to a multilingual model after pre-training @xcite @xcite @xcite . these works typically use objectives that rely on translated or induced sentence pairs, such as translation language modelling (tlm; @xcite . @xcite and @xcite focus on word alignment as a task and fine-tune the models on word alignment gold data, though @xcite also use the approach for xnli. @xcite use translated task data to encourage a task-specific alignment of xlm-r. some use word-aligned corpora (e.g., @xcite , while others use parallel sentences plus unsupervised word alignment @xcite . @xcite introduce backtranslation to the alignment process, but still use some parallel data. @xcite instead create a synthetic parallel corpus, using this with tlm. by contrast, we propose an alternate objective that relies on aligned static embedding spaces instead of sentence pairs. our alignment approach is a post-hoc tuning of the contextual model using no parallel corpora at any point. this difference allows us to apply the alignment to many more languages than most related work. for example, @xcite use up to 18 languages, @xcite use 15 with parallel data though 94 in total, @xcite use 15, while other related work often uses 4-9 languages, with a significant focus on european languages. ## conclusions we have extracted high-quality, highly multilingual static embeddings from xlm-r using a modified version of x2static and only 1m sentences of data per language. our vectors have reasonable crosslingual quality immediately after extraction, but we are able to improve their performance using alignment with dictionaries induced from fasttext vectors using vecmap. no parallel corpus was needed for this process. our final models perform competitively with supervised vectors from muse, and outperform both muse and rcsls-or provide models at all-for a number of lower-and medium-resource languages. further, we proposed a continued pre-training approach that pairs a novel alignment loss with mlm. using the dcca loss, we can improve the language-neutrality of xlm-r, benefitting downstream performance on semantic tasks.