Flow-Adapter Architecture for Unsupervised Machine Translation (2022)

Abstract: In this work, we propose a flow-adapter architecture for unsupervised NMT. It leverages normalizing flows to explicitly model the distributions of sentence-level latent representations, which are subsequently used in conjunction with the attention mechanism for the translation task. The primary novelties of our model are: (a) capturing language-specific sentence representations separately for each language using normalizing flows and (b) using a simple transformation of these latent representations for translating from one language to another. This architecture allows for unsupervised training of each language independently. While there is prior work on latent variables for supervised MT, to the best of our knowledge, this is the first work that uses latent variables and normalizing flows for unsupervised MT. We obtain competitive results on several unsupervised MT benchmarks.

## introduction recent advances in deep learning have boosted the development of neural machine translation (nmt). typical nmt models leverage an encoder-decoder framework @xcite . however, nmt models have been shown to be datahungry, as the number of parallel sentences significantly influences the performance @xcite . unfortunately, large-scale bilingual corpora are limited to a relatively small subset of languages @xcite . in contrast to bilingual corpora, monolingual corpora are much easier to obtain. unsupervised nmt, compared with supervised nmt, aims to train a model without parallel data. some early works @xcite @xcite used monolingual corpora to boost performance when parallel data is not abundant. @xcite and @xcite explored the possibility of training a model relying only on monolingual corpora. they both leveraged a sharedencoder architecture in order to generate universal representations, trained with techniques such as initial word-by-word translation through bilingual dictionaries @xcite , denoising auto-encoding (dae) @xcite and iterative back-translation (bt) @xcite . however, @xcite argued that it is a bottleneck in such shared-encoder models to use a shared encoder that maps pairs of sentences of different languages to the same shared latent space. they proposed to use two independent encoders sharing part of their weights and achieved better results. but all of those aforementioned approaches trained the translation models almost from scratch (with only some prior knowledge in the pre-trained embeddings) and therefore it is hard to further advance their performance. more recently, with the advance in pre-trained models @xcite , researchers have begun to explore the possibility of using pre-trained models for unsupervised nmt. @xcite extended the pre-training from a single language to multiple languages, referred to as cross-lingual pre-training. by using pre-trained cross-language models (xlms) to initialize encoder and decoder, they achieved good unsupervised mt performance on multiple language pairs. in related work, @xcite proposed masked sequence to sequence pre-training (mass), which directly pre-trains a whole encoder-decoder model. @xcite proposed a language-specific denoising-adapter architecture to increase the multilingual modeling capacity of the pre-trained model mbart @xcite and used these adapters for multilingual unsupervised nmt. although these adapters are trained with monolingual data only, the finetuning step relies on parallel data. current nmt frameworks rely heavily on the attention mechanism @xcite to capture alignments. however, attention-based context vectors can fail to extract sufficiently accurate sentence-level semantics and thus result in incorrect translations or translation ambiguity @xcite . to tackle this issue, several variational frameworks for modeling the translation process have been proposed @xcite @xcite . these approaches incorporate sentence-level latent representations into nmt. a latent representation, in the context of this paper, is a fixed-size continuous vector from an unknown distribution that captures the semantics of a source sentence. the target sentence is then generated from this latent representation using a simple transformation along with the attention mechanism commonly found in transformer architectures. in this way, when the attention mechanism learns incorrect alignments, the latent representation plays a complementary role in guiding the translation. prior work in this vein has only been conducted in supervised nmt. in this paper, we propose a flow-adapter architecture for unsupervised nmt. similar to variational methods, we model the distribution of sentence-level representations. however, unlike variational methods, which model the distribution in an implicit way, we use a pair of normalizing flows to explicitly model the distributions of source and target languages. secondly, different from some previous unsupervised nmt models that assume that the representations of source and target sentences share a common semantic space, we assume the representations are different because of language-specific characteristics. hence they are modeled separately for each language. subsequently a simple transformation converts source representations into target representations. this makes it possible to better capture sentence semantics in a language-specific manner. lastly, instead of minimizing kl loss, the flows are directly trained by maximum likelihood estimation (mle) of sentence-level latent representations. this gives the latent representations more flexibility. our main contributions: (1) we propose a novel flow-adapter architecture. it uses normalizing flows to explicitly model the distributions of sentence-level representations and performs a latent representation transformation from source to target. to the best of our knowledge, this is the first work that uses latent variables and normalizing flows for unsupervised nmt. ## flow-adapter based framework in this work, we want to reap the benefits of introducing latent variables into unsupervised mt while at the same time avoiding the flaws of variational nmt we just discussed. therefore, we propose a flow-adapter based framework that uses two nfs to explicitly model the distribution of the sentence-level latent representations of the source and target sentences. in this way, we can take account of multilinguality in unsupervised mt and make use of language-specific sentence-level representations. during the translation process, a latent code transformation is performed to transform the source-language representation into the targetlanguage representation so that the decoder can leverage them to generate a better target-language sentence. we will first introduce the sentence-level representation as well as the latent code transformation in section 3.1, followed by the description of the flow-adapter based framework for unsupervised mt in section 3.2. ## conclusion in this work, we propose a novel flow-adapter architecture for unsupervised nmt. the flow-adapter employs a pair of nfs to explicitly model the distributions of the sentence-level representations. a latent code transformation is performed in translation, which enables the decoder to better capture the semantics of sentences in certain language domains. through extensive experiments, we show the flow-adapter can improve multilingual translation ability. moreover, it can alleviate the copying problem. by integrating the flow-adapter into pretrained xlm models, we achieve results competitive to state-of-the-art models on wmt datasets. in the future, we would like to explore the possibility of pre-training the flow-adapter simultaneously when pre-training the language models so that the flows can learn more information. moreover, we would like to extend normalizing flows to language generation. by using different flows for different languages, multilingual language generation of the same semantics can be performed.