Crosslingual Embeddings are Essential in UNMT for distant languages: An E nglish to I ndo A ryan Case Study (2021)

Abstract: Recent advances in Unsupervised Neural Machine Translation (UNMT) has minimized the gap between supervised and unsupervised machine translation performance for closely related language-pairs. However and the situation is very different for distant language pairs. Lack of overlap in lexicon and low syntactic similarity such as between English and IndoAryan languages leads to poor translation quality in existing UNMT systems. In this paper and we show that initialising the embedding layer of UNMT models with cross-lingual embeddings leads to significant BLEU score improvements over existing UNMT models where the embedding layer weights are randomly initialized. Further and freezing the embedding layer weights leads to better gains compared to updating the embedding layer weights during training. We experimented using Masked Sequence to Sequence (MASS) and Denoising Autoencoder (DAE) UNMT approaches for three distant language pairs. The proposed cross-lingual embedding initialization yields BLEU score improvement of as much as ten times over the baseline for English-Hindi and English-Bengali and English-Gujarati. Our analysis shows that initialising embedding layer with static cross-lingual embedding mapping is essential for training of UNMT models for distant language-pairs.

## introduction unsupervised approaches to training a neural machine translation (nmt) system typically involve two stages: (i) language model (lm) pre-training and (ii) finetuning of nmt model using back-translated (bt) sentences. training a shared encoder-decoder model on combined monolingual data of multiple languages helps the model learn better cross-lingual representations @xcite . fine-tuning the pre-trained model iteratively using back-translated sentences helps further align the two languages closer in latent space and also trains an nmt system in an unsupervised manner. unsupervised mt has been successful for closely related languages @xcite . on the other hand, very poor translation performance has been reported for distant language pairs @xcite . lack of vocabulary overlap and syntactic differences between the source and the target languages make the model fail to align the two language representations together. recently, few approaches @xcite ) take advantage of resources in the form of bilingual dictionary, parallel corpora, etc. to better align the language representations together during lm pre-training. in this paper, we explore the effect of initialising the embedding layer with cross-lingual embeddings for training unmt systems for distant languages. we also explore the effect of static cross-lingual embeddings (embedding are not updated during training) v/s non-static cross-lingual embeddings (embedding are updated during training). we experiment with two existing unmt approaches namely, masked sequence-to-sequence (mass) @xcite and a variation of denoising auto-encoder (dae) based unmt approach @xcite for english to indoaryan language pairs i.e. english-hindi, english-bengali, english-gujarati. the contribution of the paper is as follows: ## related work neural machine translation (nmt) @xcite @xcite typically needs a lot of parallel data to be trained on. however, parallel data is expensive and rare for many language pairs. to solve this problem, unsupervised approaches to train machine translation @xcite @xcite was proposed in the literature which uses only monolingual data to train a translation system. @xcite and @xcite introduced denoising auto-encoder-iterative (dae-iterative) unmt which utilizes cross-lingual embeddings and trains a rnn-based encoderdecoder model @xcite . architecture proposed by @xcite contains a shared encoder and two language-specific decoders while architecture proposed by @xcite contains a shared encoder and a shared decoder. in the approach by @xcite , the training starts with word-by-word translation followed by denoising and backtranslation (bt). here, noise in the input sentences in the form of shuffling of words and deletion of random words from sentences was performed. @xcite (xlm) proposed a two-stage approach for training a unmt system. the pre-training phase involves training of the model on the combined monolingual corpora of the two languages using masked language modelling (mlm) objective @xcite . the pre-trained model is later fine-tuned using denoising auto-encoding objective and backtranslated sentences. @xcite proposed a sequence to sequence pre-training strategy. unlike xlm, the pre-training is performed via masked sequence to sequence (mass) objective. here, random n-grams in the input are masked and the decoder is trained to generate the missing n-grams in the pre-training phase. the pre-trained model is later fine-tuned using backtranslated sentences. recently, @xcite demonstrated that the performance of current sota unmt systems is severely affected by language divergence and domain difference. the authors demonstrated that increasing the corpus size does not lead to improved translation performance. the authors hypothesized that existing unmt approaches fail for distant languages due to lack of mechanism to bootstrap out of a poor initialization. recently, @xcite trained unmt systems with 2 language pairs english-macedonian (en-mk) and english-albanian (en-sq) in low resource settings. these pairs achieved bleu scores ranging from 23 to 33 using unmt baseline xlm @xcite and re-lm @xcite systems. they showed further improvement up to 4.5 bleu score when initialised embedding layer with crosslingual embedding. however, they did not explore the effect of initialising embedding layers on mass, dae-pretrained, and dae-iterative approaches. moreover, they did not experiment with language-pairs for which unmt approaches with randomly initialised embedding layers fail completely even after training with a sufficient amount of monolingual data. additionally, there is some work on understanding multilingual language models and their effectiveness on zero-shot performance on downstream tasks @xcite @xcite @xcite . here, the pre-trained multilingual language model is fine-tuned for the downstream nlp task in one language and tested on an unseen language (unseen during fine-tuning stage). while multilingual models have shown promising results on zero-shot transfer, the gains are limited for distant languages unless additional resources in the form of dictionary and corpora are used @xcite . also, training a single model on unrelated languages might lead to negative interference @xcite . ## approaches in this section, we explain different approaches used in our experiments. we use mass @xcite and dae based iterative approach similar to @xcite as our baseline models. ## experimental setup we trained the models using 8 approaches for all language-pair out of which 3 approaches use dae as lm pretraining, 3 approaches use mass as lm pretraining, and the other two train dae and bt simultaneously. ## result and analysis in this section, we present the results from our experiments and present a detailed analysis of the same. ## conclusion we show that existing unmt methods such as dae-based and mass-based unmt models fail for distant languages such as english to indoaryan language pairs (i.e. en-hi, en-bn, en-gu). however, initialising the embedding layer with cross-lingual embeddings before language model (lm) pre-training helps the model train better unmt systems for distant language pairs. english source their hearts and my heart beat to the same rhythm . english source what a fine , purposeful message bengali reference কত সু র বাতর্ া । english transliteration kata sundara barta | word translation what a beautiful message . system translation কী একটা জিরমানা , purposeful বাতর্ া english transliteration ki ekata jarimana , purposeful barta word translation what a penalty , purposeful message english meaning what a penalty/fine , purposeful message english source they live in a parking shed with their family . bengali reference তাঁ রা সপিরবাের গািড় রাখার েশেডর মেধয্ থােকন । english transliteration ta.nra saparibare gadai rakhara shedera madhye thakena | word translation they with family parking shed inside lives system translation পািকর্ ং েশেডর সে বসবাস কের তােদর পিরবার । english transliteration parkim shedera sa nge basabasa kare tadera paribara | word translation parking shed with live their family english meaning their family live with parking shed figure 6: example of a english to bengali translation using dae static model