Efficient Unsupervised NMT for Related Languages with Cross-Lingual Language Models and Fidelity Objectives (2021)

Abstract: The most successful approach to Neural Machine Translation (NMT) when only monolingual training data is available, called unsupervised machine translation, is based on back-translation where noisy translations are generated to turn the task into a supervised one. However, back-translation is computationally very expensive and inefficient. This work explores a novel, efficient approach to unsupervised NMT. A transformer, initialized with cross-lingual language model weights, is fine-tuned exclusively on monolingual data of the target language by jointly learning on a paraphrasing and denoising autoencoder objective. Experiments are conducted on WMT datasets for German-English, French-English, and Romanian-English. Results are competitive to strong baseline unsupervised NMT models, especially for closely related source languages (German) compared to more distant ones (Romanian, French), while requiring about a magnitude less training time.

## introduction while traditional end-to-end neural machine translation (nmt) approaches have shown highly promising results when abundant parallel data is available @xcite , the task remains a considerable challenge when only monolingual training data is available, also called unsupervised mt @xcite . unsupervised nmt systems tend to combine backtranslation @xcite with crosslingual embeddings @xcite or, more recently, with weights of a pre-trained cross-lingual language model (xlm) @xcite . back-translation uses noisy translations, generated by a source-totarget model, as input for a target-to-source model @xcite . although shown to perform well if plenty of monolingual data is available, backtranslation is computationally very expensive. it is also highly inefficient as the inference performed to generate the noisy translations is of sequential nature, slowing down the training substantially. this paper presents a novel unsupervised nmt method that does not require back-translation. instead, it jointly fine-tunes a transformer @xcite , initialized with weights of crosslingual language models @xcite , on a denoising autoencoder @xcite and paraphrasing objective exclusively on data in the target language. the alignment of the languages in the transformer's encoder means we can learn similar hidden representations for sentences of similar meaning but from different languages. the decoder, fine-tuned to generate a sentence in the target language, can thus generate a translation based on the encoder's representation of a source-language input sentence. naturally, this method is more suitable for related languages, as the alignment of languages and hidden representations in the cross-lingual encoder is of particular importance to this approach. our experiments with the wmt datasets for @xmath0, @xmath1, and @xmath2, @xcite and @xcite -highlighting that the alignment quality achieved by the highquality cross-lingual language model as a translation signal is superior to aligned embeddings and back-translation. results for german are substantially higher than for french and romanian, highlighting that our approach works particularly well for more closely related languages. while achieving competitive results, the proposed approach is substantially more efficient. it converges much quicker while requiring less than 50% time per epoch during fine-tuning which results in around a magnitude less floating point operations for the proposed approach than for an equivalent setup when using back-translation. we further show that the paraphrasing objective improves translation quality considerably compared to using the autoencoder objective in isolation. ## method given an input sequence x s in source language s the objective is to generate a sequence y t in the target language t, which is semantically equivalent. a model nmt @xmath3 with v t being the set of all possible sequences in the target language. this paper focuses on the transformer model @xcite to solve equation 1. the transformer consists of an encoder and a decoder module: both are initialized with weights w s↔t of a cross-lingual language model and a shared subword vocabulary to align languages s and t ( § 2.3). the encoder transforms the input into a latent space while the decoder iteratively generates the output sequence ŷ t . ## results while previous attempts to using the denoising objective without back-translation resulted in unusable results @xcite , we observe that our model performs reasonably well already when 6 emb dim: 1024, #layers 6, #heads 8, dropout: 0.1, attention dropout: 0.1, tokens per batch: 2000, optimizer: adam inverse sqrt. denoising autoencoder parameters: word shuffle: 3, word dropout: 0.1, word blank: 0. furthermore, our approach (7) achieves particularly high results when a source language is related (german) to the training language, compared to a more distant one (french, romanian). while our model ( 7 ) outperforms (3) for @xmath6, it scores 2.1 points less for @xmath7. moreover, scores between (7) and (3) are only comparable for german, while (3) performs much better for both the less related source languages. this observation is even amplified when the source language is romanian. our model appears to be more susceptible to the relatedness of the source language than (5) which uses the same cross-lingual weights, scoring 2.1 points less on french than german, compared to only 1.0 for (5). while our approach solely relies on the alignments based on these weights, the backtranslation of (5) adds an important signal especially for more distant languages. this confirms the assumption made for our model: the bleu score of our model is particularly high for the closely related @xmath8, compared to the more distant language pairs. ## efficiency one major advantage of the proposed method over existing methods is its efficiency in terms of computational time. in our approach outperforms lample et al. (2018b) while being much more efficient, however, one might suggest that the shown efficiency of the proposed model can also be achieved with the better scoring @xcite by trading off some of its performance advantages. therefore, we explored to which extent either less training time or training data to achieve faster convergence improves its efficiency. regarding training times, our model for english is already openly accessible. moreover, the paraphrasing model uses a (bi-lstm) that was trained for 3 epochs on only 24,000 sentence pairs, which is less than 2% of the translation data used for the nmt models. proposed approach (7) scored consistently higher 8 than the other models until it's convergence, see figure 2. only after our model converged, the model of @xcite surpasses its scores. thus, stopping the training process earlier would not lead to better efficiency than our model. we then also analyzed the models' scores when modifying the amount of data used for fine-tuning. results using 10k, 25k, 50k, 100k, and 5m training sentences are shown in figure 3 . as seen, when using little training data, @xcite performs much worse than the proposed model since errors and noise from translating into one direction are propagated when translating back. the less monolingual data used, the stronger the effect of this error-propagation issue. the back-translation model of @xcite starts to outperform the proposed model starting from 100k 8 excluding the initial phase as our model computes the paraphrases first. since our model catches up at a bleu of around 1.7, we ignored this special case. training sentences. yet, even in this setting the proposed model remains much more efficient, as seen in the column cost (flops) @ 100k in ## model analysis & discussion alignment and translation quality: to investigate whether the alignment extends from a wordlevel to a phrase or even sentence level, a qualitative analysis was conducted. example reference texts and respective translations for @xmath10. it can be seen that the translation quality is high for shorter sentences but it declines with increasing sentence length. many words and phrases are translated correctly, which generally leads to preservation of sentence meaning for simple sentences in closely-related languages like english and german. while many simple phrases are grammatical, many longer more complex structures are not. furthermore, the model hallucinates content, especially when confronted with numbers and named entities. for example, in sentence 5 the model generates a made-up destination to northern croatia while new lloyd in sentence 6 also never occurs in the source sentence. in the translation for sentence 3, the name is simply omitted and the currency, as well as the amount, is wrong. this observation can be transferred to numbers as well: in sentence 3, the number of 7.5 million was changed to 67.5 million. in an extreme example, the model hallucinates an entire clause for sentence 7. we also observed some artifacts in the output: in sentence 7, oberflächlich is translated into oberpublic, instead of superficial, merging an german and english term. wrong translations frequently contain words that are still closely related to the correct translation. for instance, in sentence 2 the model generates the former prime minister of israel olmert, instead of netanyahu and translates meets instead of receives. or in sentence 5, the model translated into hurdles instead of obstacles. these very subtle differences are difficult for the model to capture. paraphrase quality although the paraphrasing objective improves the performance of the system substantially, the generated paraphrases are still far from optimal. figure 5 shows example paraphrases. x-1: die vorbereitung lief gut. y: the preparation went well. ŷ : the training season went well. x-2: obama empfängt netanyahu. y: obama receives netanyahu. ŷ : obama meets olmert. x-3: spaniens nationaltorhüter iker casillas hat für 7,5 millionen euro seine hände versichern lassen. y: spain's national goalkeeper iker casillas insured his hands for 7.5 million euro. ŷ : spain's king has asked for £ 67.5 million to reinsure his hands. x-4: wir wollten gewinnen und wir haben gewonnen, obwohl ich es bedaure, dass wir nicht noch ein oder zwei weitere tore erzielt haben". y: we wanted to win and we did, although i regret that we did not score one or two goals more. ŷ : we wanted to win and we have won, although i regret it not yet that we have scored one or two more goals." x-5: zagreb kündigt an, die durchreise über slowenien nach norden ohne hürden zu gewähren. y: zagreb has announced that they will allow passage over slovenia towards the north without obstacles. ŷ : zagreb has announced it will allow the migration through slovenia to northern croatia without hurdles. x-6: er könnte sich vorstellen, in startup-firmen der neuen werf zu investieren und später auch welche zu akquirieren. y: he can imagine investing in start-up business in neuen werft, and also later acquiring these. ŷ : he could also imagine investing in startup-owned companies like the new lloyd's to invest and later in companies to quiquire. x-7: dieser oberflächliche erfolg wird von dem dominiert, was psychologen extrinsische werte nennen; geld ,image, sozialer status, alles auf kosten intrinsischer werteunsere innere sehnsucht nach persönlicher entwicklung und freundschaft, die sich auf unsere geistige gesundheit viel tiefgehender auswirkt. y: this superficial success is dominated by what psychologists term extrinsic values; money, image, social status, all at the expense of intrinsic values -our inner yearning for personal growth and friendship -that more deeply impacts our mental health. ŷ : this oberpublic success is dominated by the things that economists call extraterrestrial values to characterize; money, image, social status all on the planet of intrinsatiable values -our inner desire for personal development and friendship, which can be felt on our mental health much deeper than it is in the physical world of human development and friendship, which is a real concern for our human health and friendship, which is in effect a greater word. they show clear noise and are not always grammatical either. the quality of the paraphrases degrades r: they have not been charged or formally arrested. p1: they were not charged or officially arrested. p2: they didn't have an arrest or official. r: the japanese-made tin robots have blocky heads and moveable arms and legs. p1: the japanese robots have blocky robots, and their arms and feet. p2: the japanese robots have blocky heads, their hands and feet. r: the a.p. said it hoped for a resolution so it could return to full coverage of the six-week tournament before the opening match friday between france and argentina. p1: the organisers said they hoped to find a resolution so he could return to full coverage of the six-week tournament before the opening -up friday between france and argentina. p2: the panasonic said that it hoped for a resolution for an order to return to full coverage of the six-week tournament to keep an eye on friday between france and argentina. figure 5 : example references (r) and respective paraphrases (p) of the employed paraphrasing system @xcite . ## conclusions & future work this work presented a simple fine-tuning method for unsupervised nmt that solely relies on the underlying alignment of a cross-lingual language model and monolingual data in the target language. joint learning on a denoising autoencoder and paraphrasing objective creates a competitive system to strong baselines, especially for related language pairs, while requiring much shorter training times. while this work has explored the proposed approach on commonly used language pairs for benchmarking unsupervised mt, future work includes testing the proposed method on other language pairs. this includes i) even more closely related languages (e.g. @xmath11, ii) language pairs without any parallel data, iii) translations between dialects. moreover, we aim to further explore to which extent the proposed method can be combined with back-translation, especially in the context of distant languages, e.g. by adding backtranslation iterations on top of the proposed approach, similar to pbsmt in @xcite . other training objectives should be explored, such as sentence simplification, translating between dialects, or even abstractive summarization when scaling this approach to document-level translation. besides higher efficiency, this approach appears promising when training data for a language pair origins from different domains (e.g. wikipedia versus news). since our approach only requires data in the target language, domain mismatches in the training data for the language pair do not affect the proposed method. we are further aiming for human evaluation of translation quality.