Decoupled Vocabulary Learning Enables Zero-Shot Translation from Unseen Languages (2024)

Abstract: Multilingual neural machine translation systems learn to map sentences of different languages into a common representation space. Intuitively, with a growing number of seen languages the encoder sentence representation grows more flexible and easily adaptable to new languages. In this work, we test this hypothesis by zero-shot translating from unseen languages. To deal with unknown vocabularies from unknown languages we propose a setup where we decouple learning of vocabulary and syntax, i.e. for each language we learn word representations in a separate step (using cross-lingual word embeddings), and then train to translate while keeping those word representations frozen. We demonstrate that this setup enables zero-shot translation from entirely unseen languages. Zero-shot translating with a model trained on Germanic and Romance languages we achieve scores of 42.6 BLEU for Portuguese-English and 20.7 BLEU for Russian-English on TED domain. We explore how this zero-shot translation capability develops with varying number of languages seen by the encoder. Lastly, we explore the effectiveness of our decoupled learning strategy for unsupervised machine translation. By exploiting our model’s zero-shot translation capability for iterative back-translation we attain near parity with a supervised setting.

## introduction in order to extend machine translation from a hundred languages to the 7,000 languages of the world new methods have to be developed with more economical use of data from individual languages. this requires better use of abstractions and representations of languages to facilitate transfer learning to low-resource languages. in machine translation, it had been attempted by developing linguistic structures explicitly in classical interlingua systems @xcite or imposing structure on statistical models @xcite . however, developing such structures was labor intensive and did not achieve acceptable performance or required domain restrictions. first attempts toward learning latent representations of language @xcite still required linguistic structures or specific design. successful performance on open-domain translation was only feasible later using multilingually trained end-to-end neural translation models @xcite @xcite . these models share hidden representations between languages and allow for multilingual translation that benefits low-resource languages by transfer learning from high-resource languages and enable zero-shot translation on new language directions. the multilingual encoder in such models will ideally learn to map semantically similar sentences onto similar latent representations -even across different languages. ongoing research in the field of zero-shot translation shows that multilingual nmt models exhibit this property to a certain extent @xcite and that enforcing the similarity between sentence representations across different languages in turn also improves the zeroshot capabilities @xcite @xcite @xcite . at the same time, cross-lingual transfer learning research shows that massively multilingual systems can be rapidly extended to new languages on very little data @xcite @xcite @xcite . this suggests that exposing the nmt model to a higher number of languages increases the plasticity of its sentence representation and enables easier adaptation to unseen languages. when extending a multilingual model by an unseen language, however, naturally the question arises of how to deal with the unknown vocabulary words. the field of incremental learning devised several strategies for adapting to new vocabularies. @xcite show that simply relearning the embedding layer can be done efficiently, compared to training the model from scratch. @xcite find that retraining the shared bpe codes once the new language data become available allows one to keep most of the pre-trained vocabulary, making even more efficient adaptation possible. in this work, we go one step further and show that by designing a system where word representations are learned in a separate step we can zero-shot translate from an unknown language without any adaptation whatsoever. we therefore employ crosslingual word embeddings @xcite ) -a technique that has extensively been used in unsupervised machine translation, but hasn't so far been given much consideration for incremental learning -to align monolingual word representations into a common space, and then use these as our regular word embedding layer in the nmt system. crucially, keeping these embeddings frozen in the nmt training allows us to align new language vocabularies into the common embedding space post-training. furthermore, we show that this zero-shot translation capability allows us to easily generate synthetic parallel data for unknown languages and enables simple and efficient unsupervised machine translation without the need for denoising autoencoding on massive amounts of monolingual data. finally, we also experiment on continuous output nmt (kumar and tsvetkov, 2019), which -while underperforming the softmax models in terms of bleu scores -perfectly fits our setup and attains vastly superior performance in terms of training times. we make our code and training configurations available under https://github.com/ cmullovisl/clwe-transfer and we release our pre-trained multilingual models and embedding alignments. ## multilingual neural machine translation in training the multilingual nmt system we aim to estimate the probability p(y ℓtgt = y | x ℓsrc = x) that the sentence y in the target language ℓ tgt is a suitable translation of the source sentence x in the source language ℓ src . we describe the distribution of the sentences in different natural languages ℓ through random variables x ℓ , y ℓ . a universal en-coder maps the sentences from the different input distributions onto a single, shared latent distribution h @xmath0. a decoder is then tasked to model the probability distribution from this latent variable: p(y ℓtgt = y | h @xmath1. identically to a regular bilingual nmt system, the multilingual translation system consists of a neural network, which, in a supervised setting, is trained to maximize the likelihood of a dataset of parallel sentences @xmath2, y 1 ), . . . , (x n , y n )}, where sentence pairs (x i , y i ) are translations of each other: note that -aside from the added target language selection mechanism -the only difference to bilingual translation is the nature of the input and output distribution. thus, the main practical difference in the training a multilingual system is that instead of using a bilingual training corpus, d consists of a concatenation of several parallel corpora. ## related work cross-lingual word embeddings @xcite look into the effectiveness of pre-trained embeddings for regular supervised nmt and find them to be helpful in a multilingual setting. @xcite conduct unsupervised mt by transferring the parameters learned on a high resource language pair to a new source language. similar to our work they therefore align the new language embeddings to the parent model source embeddings, but they focus on a bilingual setting. compared to their work we show that this setup not only allows for unsupervised mt but zero-shot translation from unseen languages, especially when combined with a multilingual setup. unsupervised mt our work is closely related to unsupervised nmt @xcite @xcite @xcite . initial works train unsupervised nmt systems by bootstrapping a common representation space between two languages through a combination of cross-lingual word embeddings and denoising autoencoding, and then exploit these common representations for iterative back-translation @xcite . through unsupervised means they thus induce a shared sentence representation, essentially bootstrapping a multilingual nmt system from monolingual data. later works do away with the cross-lingual word embeddings and create this shared representation space through large scale denoising autoencoding on massive amounts of monolingual data @xcite and multilinguality @xcite @xcite . in contrast to this development, we show that cross-lingual word embeddings remain valuable when combined with regular nmt training, allowing to bootstrap high-quality multilingual representations even on small-scale datasets. new language learning multiple works look into adding unseen languages to multilingual encoder-decoder systems. @xcite concerned about the possibility of rapidly extending a multilingual nmt model by an unseen lowresource language. they compare between bilingual training, multilingual training alongside a similar source language and multilingual pre-training with as many languages as possible. they come to the conclusion that pre-training in a highly multilingual setting -in their case 58 source languages -significantly improves transferability to low-resourced languages. in particular, they achieve significant performance (up to 15.5 bleu) on an entirely unseen language, provided parallel training data on a closely related language. in this work we use the same parallel data for nmt training and significantly improve upon their result through the employment of cross-lingual word embeddings. @xcite devise an approach to multilingual nmt with independent encoders and decoders that enables addition of new languages. @xcite fine-tune a model based on xlm-r and use it to zero-shot translate from languages seen in pre-training, but unseen in the supervised downstream task. tang et al. (2020) look into extending mbart25 @xcite ) by 25 new languages and show that given enough modelling capacity one can add new languages without loss in quality on existing ones. in their work, however, they only consider new languages sufficiently represented in the base model vocabulary, and they leave the original vocabulary unmodified. kim et al. (2019 @xcite show that rebuilding the subword vocabulary once the new language data is available results in most of the vocabulary matching the old one. retaining the embedding vectors for common subword units allows for quick adaptation and minimal forgetting of the existing languages. in our work we directly compare to this method and show that a vocabulary based on aligned pre-trained embeddings not only presents a viable alternative to a subword-based one, but also allows for more efficient adaptation to and zero-shot translation from unseen languages. ## generalizing to unseen languages to learn a multilingual nmt system that easily generalizes to unseen languages we propose a decoupled learning approach, divided into multiple steps: in this setup we may later extend the translation model by an unseen language ℓ new / ∈ l base by learning word representations for ℓ new and aligning them into the common word embedding space (see section 4.1). we show that this alignment alone enables us to start translating from ℓ new in a plug-and-play fashion, even without any additional updates to the nmt model's main body (i.e. the transformer layers). optionally, the learned word representations may also be employed on the decoder-side, which will further enable the back-translation-based adaptation detailed in section 5.3. figure 1 describes our decoupled learning nmt architecture. ## conclusion in this work, we have looked into the generalization ability of a multilingual nmt system on unseen languages. we therefore propose a decoupled learning setup for mt training, where we learn the word representations of each language in a separate step. this enables easier generalization to new languages as this approach enables extension of the vocabulary post-training. we show that through simple alignment of an unseen language word representations into the common vocabulary space, we can produce intelligible translations without any further adaptation. we further show that on languages similar to the ones the nmt system has already been exposed to, this zero-shot translation setup even approaches supervised translation performances. lastly we show that this capability to produce translations for unseen languages can be exploited for efficient unsupervised machine translation. future work in the future we would like to explore the efficacy of our presented setup for an incremental learning system, which would entail in-depth experiments on catastrophic forgetting.