Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translation (2021)

Abstract: Successful methods for unsupervised neural machine translation (UNMT) employ cross-lingual pretraining via self-supervision, often in the form of a masked language modeling or a sequence generation task, which requires the model to align the lexical- and high-level representations of the two languages. While cross-lingual pretraining works for similar languages with abundant corpora, it performs poorly in low-resource and distant languages. Previous research has shown that this is because the representations are not sufficiently aligned. In this paper, we enhance the bilingual masked language model pretraining with lexical-level information by using type-level cross-lingual subword embeddings. Empirical results demonstrate improved performance both on UNMT (up to 4.5 BLEU) and bilingual lexicon induction using our method compared to a UNMT baseline.

## introduction unmt is an effective approach for translation without parallel data. early approaches transfer information from static pretrained cross-lingual embeddings to the encoder-decoder model to provide an implicit bilingual signal @xcite . @xcite suggest to instead pretrain a bilingual language model (xlm) and use it to initialize unmt, as it can successfully encode higher-level text representations. this approach largely improves translation scores for language pairs with plentiful monolingual data. however, while unmt is effective for high-resource languages, it yields poor results when one of the two languages is low-resource @xcite . @xcite show that there is a strong correlation between bilingual lexicon induction (bli) and final translation performance when using pretrained cross-lingual embeddings, converted to phrase-tables, as initialization of a unmt model @xcite . @xcite observe that static cross-lingual embeddings achieve higher bli scores compared to multilingual language models (lms), meaning that they obtain a better lexical-level alignment. since bilingual lm pretraining is an effective form of initializing a unmt model, improving the overall representation of the masked language model (mlm) is essential to obtaining a higher translation performance. in this paper, we propose a new method to enhance the embedding alignment of a bilingual language model, entitled lexically aligned mlm, that serves as initialization for unmt. specifically, we learn type-level embeddings separately for the two languages of interest. we map these monolingual embeddings to a common space and use them to initialize the embedding layer of an mlm. then, we train the mlm on both languages. finally, we transfer the trained model to the encoder and decoder of an nmt system. we train the nmt system in an unsupervised way. we outperform a unmt baseline and demonstrate the importance of cross-lingual mapping of token-level representations. we also conduct an analysis to investigate the correlation between bli, 1-gram precision and translation results. we finally investigate whether cross-lingual embeddings should be updated or not during the mlm training process, in order to preserve lexicallevel information useful for unmt. we make the code used for this paper publicly available foot_0 . ## proposed approach our approach has three distinct steps, which are described in the following subsections. ## experiments datasets. we conduct experiments on english-macedonian (en-mk) and english-albanian (en-sq), as mk, sq are low-resource languages, where lexical-level alignment can be most beneficial. we use 3k randomly sampled sentences of setimes @xcite as validation/test sets. we also use 68m en sentences from newscrawl. for sq and mk we use all the commoncrawl corpora from ortiz @xcite , which are 4m sq and 2.4m mk sentences. baseline. we use a method that relies on crosslingual language model pretraining, namely xlm @xcite . this approach trains a bilingual mlm separately for en-mk and en-sq, which is used to initialize the encoder-decoder of the corresponding nmt system. each system is then trained in an unsupervised way. ## results in the case of xlm, the effect of cross-lingual lexical alignment is more evident for en-mk, as mk is less similar to en, compared to sq. this is mainly the case because the two languages use a different alphabet (latin for en and cyrillic for mk). this is also true for re-lm when translating out of en, showing that enhancing the fine-tuning step of mlm with pretrained embeddings is helpful and improves the final unmt performance. in general, our method provides better alignment of the lexical-level representations of the mlm, thanks to the transferred vecmap embeddings. we hypothesize that static cross-lingual embeddings enhance the knowledge that a cross-lingual masked language model obtains during training. as a result, using them to bootstrap the pretraining procedure improves the ability of the model to map the distributions of the two languages and yields higher translation scores. overall, our approach consistently outperforms two pretraining models for unmt, providing for the highest bleu and chrf1 scores on all translation directions. ## analysis we conduct an analysis to assess the contribution of lexical-level alignment in the mlm training. we present bilingual lexicon induction (bli) and bleu 1-gram precision scores. we also investigate the best method to leverage pretrained cross-lingual embeddings during mlm training, in terms of final unmt performance. ## conclusion we propose a method to improve the lexical ability of a transformer encoder by initializing its embedding layer with pretrained cross-lingual embeddings. the transformer is trained for masked language modeling on the language pair of interest. after that, it is used to initialize an encoder/decoder model, which is trained for unmt and out-performs relevant baselines. results confirm our intuition that masked language modeling, which provides contextual representations, benefits from cross-lingual embeddings, which capture lexicallevel information. in the future, we would like to investigate whether lexical knowledge can be infused to multilingual mlms. we would also like to experiment with other schemes of training the mlm in terms of how the embedding layer is updated, such as regularizer annealing strategies, which would enable keeping the embeddings relatively fixed, but still allow for some limited training.