Low-Resource Unsupervised NMT : Diagnosing the Problem and Providing a Linguistically Motivated Solution (2020)

Abstract: Unsupervised Machine Translation has been advancing our ability to translate without parallel data, but state-of-the-art methods assume an abundance of monolingual data. This paper investigates the scenario where monolingual data is limited as well, finding that current unsupervised methods suffer in performance under this stricter setting. We find that the performance loss originates from the poor quality of the pretrained monolingual embeddings, and we offer a potential solution: dependency-based word embeddings. These embeddings result in a complementary word representation which offers a boost in performance of around 1.5 BLEU points compared to standard word2vec when monolingual data is limited to 1 million sentences per language. We also find that the inclusion of sub-word information is crucial to improving the quality of the embeddings.

## introduction machine translation (mt) is a rapidly advancing field of natural language processing, where there is an ever-increasing number of claims of mt systems reaching human parity @xcite . however, most of the focus has been on mt systems under the assumption c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. while the unsupervised approaches so far have done away with the assumption of parallel data, they still assume an abundance of monolingual data for the two languages, typically assuming at least 10 million sentences per language. this amount of data is not available for every language, notably languages without much of a digital presence. for example, fulah is a language spoken in west and central africa by over 20 million people, however there is a scarce amount of data freely available online. this motivates a new paradigm in unsupervised mt: low-resource unsupervised mt (lrumt). in this paper, we investigate the reasons why current unsupervised nmt methods fail in the low-resource setting, addressing the source of the issue, and we propose a potential solution to make unsupervised nmt more robust to the lack of availability of monolingual data. we start by giving a brief overview of the work so far in unsupervised mt in section 2, establishing the general pipeline used to train an unsupervised system. we then identify the source of the performance problem in lrumt in section 3, and propose potential improvements in section 4. lastly, in section 5, we present our conclusions and lines for future work. ## an unsupervised mt overview the typical unsupervised nmt pipeline can be broken down into 3 sequential steps: in the first step, monolingual embeddings (which we will also refer to as pretrained embeddings) are most often trained in the style of word2vec's skip-gram algorithm @xcite . to incorporate sub-word information, @xcite use fasttext @xcite , which formulates a word's embedding as the sum of its character n-gram embeddings. @xcite uses a word2vec extension phrase2vec @xcite , which learns embeddings of word n-grams up to trigrams, effectively creating embeddings for phrases. the second step involves the alignment of the two monolingual embeddings such that the embeddings of words with identical or similar meaning across language appear close in the shared embedding space. artetxe et al. achieve this using vecmap @xcite , which learns a linear transformation between the two embeddings into a shared space. if there is a large shared vocabulary between the two languages, it is also possible to concatenate the monolingual corpora and train a single embedding for both languages, effectively completing steps 1 and 2 simultaneously @xcite . the third and final step is to train the nmt model. the architecture can be any encoderdecoder model, with the condition that it can translate in both directions. models typically share an encoder and decoder for both languages, with a language token provided only to the decoder. two objectives are used to train the model: denoising and on-the-fly back-translation. denoising is monolingual; the model is given an altered sentence (e.g. with word order shuffling or word removal) and trained to reconstruct the original, un-altered sentence. on-the-fly back-translation involves first translating a sentence from the source language (s src ) to the target language (s tgt ). this creates a pseudo-parallel sentence pair (s tgt , s src ), so the output s tgt is translated back to the source language (creating s src ), and the model is trained to reconstruct the original source sentence, minimizing the difference between s src and s src . denoising and back-translation are carried out alternately during training. the unsupervised smt approach is fairly similar, with a replacement of step 3 (or in the hybrid approach, a step added between steps 2 and 3). in artetxe et al. (2019) for example, a phrase-based smt model is built by constructing a phrase table that is initialized using the aligned cross-lingual phrase embeddings, and tuning it using an unsupervised variant of the minimum error rate training @xcite ) method. for the hybrid model, the smt system can then create pseudo-parallel data used to train the nmt model, alongside denoising and back-translation. in the remainder of this paper, we focus on the purely nmt approach to unsupervised mt. ## the role of pretrained embeddings in unsupervised mt with the pipeline established, we now turn to the lrumt setting. in lrumt, the existing unsupervised approaches fail somewhere along the pipeline, but simply measuring mt performance does not make it clear where this failure occurs. we speculate that the failure is relative to the quality of the pretrained word embeddings, and subsequent quality of the cross-lingual alignment. we test this hypothesis in this section. the aligned pretrained embeddings of an unsupervised nmt system are what jump-starts the process of translation. from aligned pretrained embeddings alone, we can effectively do word-forword translation, which is commonly measured using bilingual lexicon induction (bli). without well-aligned pretrained embeddings, denoising and back-translation alone are not enough to produce meaningful translations. for our following experiments foot_0 , we train on english and german sentences from the wmt monolingual news crawl from years 2007 to 2017, use newstest 2015 for development and newstest 2016 for testing, following @xcite . the training data is filtered such that sentences that contain between 3-80 words are kept. we then truncate the corpora to sizes ranging from 0.1 to 10 million sentences per language, specified as necessary. we used udpipe @xcite for tokenization foot_1 , moses @xcite for truecasing, and we apply 60 thousand bpe joins (following lample et al. ( 2018 )) across both corpora using fastbpe. 3,4 we train the word embeddings using the word2vec skipgram model, with the same hyperparameters as used in @xcite , except using an embedding dimension size of 512. 5 for embedding alignment, we use the completely unsupervised version of vecmap with default parameters. we then train our unsupervised nmt models using lample et al. (2018)'s implementation, using the default parameters, with the exception of 10 backtranslation processors rather than 30 due to hardware limitations. we use the early stopping criterion from lample et al. (2018). 6 to demonstrate the importance of a large amount of training data, we vary the amount of monolingual data used for training the embeddings as well as the amount used for training the nmt system in figure 1 . 7 even if we then use 10 million sentences per language to train the nmt system, using only 100 thousand sentences per language to train the embeddings results in a bleu score below 1. conversely, the nmt system can achieve a bleu score of around 6 using embeddings trained on 10 million sentences, even when the nmt system is only trained on 100 thousand sentences per language. we also provide figure 2 , showing the bli scores of the aligned embeddings (using the @xmath0. (2017a) foot_7 ) as we vary the amount of training data used for the embeddings. we can see that the bli scores decrease dramatically as the amount of sentences decreases, matching the trend of the results from figure 1 . although bli has been criticized for not always correlating with downstream tasks @xcite , in this case, poor alignment corresponds to poor mt performance. in these experiments, we use vecmap for aligning embeddings. vecmap's algorithm begins by initializing a bilingual dictionary, which uses a word's relations to the other words in the same language, with the idea being that "apple" would be close to "pear" but far from "motorcycle" in every language, for example. however, if the quality of embeddings is poor, the random initialization of embeddings has a greater dampening effect. using embedding similarity tasks (shown in we measure the quality of the monolingual embeddings using 3 similarity datasets for english: men @xcite , ws353 @xcite , and simlex999 @xcite . we also use multilingual simlex999 @xcite for german and denote this as simlex_de. as we can see in ## getting more out of scarce data with the source of the problem established as the drop in quality of embeddings, we ask ourselves: how can we prevent this drop in a low-resource scenario, where considerably less monolingual data is available? we argue that the conventional word embedding methods (i.e. word2vec) do not use all of the information present within sentences during the training process. word embedding algorithms typically define a context-target pair as a word and its neighboring words in a sentence, respectively. while this method works with a large amount of data available, it relies on the fact that a word is seen in several different contexts in order to be represented in the embedding space with respect to its meaning. when data is limited, the contexts contain too much variability to allow for a meaningful representation to be learned. to test this, we use an embedding strategy that has a different definition of the context: dependency-based word embeddings @xcite . these embeddings model the syntactic similarity between words rather than semantic similarity, providing an embedding representation complementary to standard embeddings. this section presents our findings using dependency-based embeddings (4.1). we also consider the effect of using sub-word information via fasttext (4.2). with the previous two approaches, we find that ensembling models can be useful, and investigate this further (4.3). finally, we vary context window size and report on its effect (4.4). ## conclusion and future work unsupervised nmt has made great strides in making mt more accessible for language pairs that lack parallel corpora. we attempt to further this accessibility by introducing lrumt, where monolingual data is also limited. our results show that, in the current state-of-the-art pipeline, the quality of the pretrained word embeddings is the main issue, and that using syntactically-motivated dependency-based embeddings has the potential to improve performance when monolingual data is limited. we also see that the inclusion of sub-word information for training word embeddings provides a crucial performance increase, which provides further evidence that using the latent linguistic information in a sentence can improve embedding alignment quality. finally, on the topic of context size, we find that a larger context size is almost always better, most noticeably when more data is available. this helps explain the poorer performance of the dependencybased embeddings on larger amounts of data. to improve upon dependency-based embed-dings for unsupervised nmt, we consider two avenues to explore: including sub-word information and increasing the context size. to include sub-word information, it should be possible to combine the training methods of fasttext and dependency-based word2vec. to increase the context size, one might consider including a word's grandparent, grandchildren, and siblings (its parent's other children) as part of the context. we also note that we currently use a pretrained dependency parser, trained on labelled dependency data, which is often harder to come by than parallel data. we plan to switch to using unsupervised dependency parsing techniques to ensure this method is accessible for all languages. furthermore, there are several potential methods for incorporating more linguistic information into embeddings. one such possibility would be to use a morphological segmenter such as mor-fessor @xcite rather than bpe, which would likely provide better results for more morphologically-rich languages. as we only test on english-german, our future work will test this new paradigm on other language pairs, particularly those in which unsupervised nmt fails to perform such as english into morphologically-rich languages.