Variational Neural Machine Translation with Normalizing Flows (2020)

Abstract: Variational Neural Machine Translation (VNMT) is an attractive framework for modeling the generation of target translations, conditioned not only on the source sentence but also on some latent random variables. The latent variable modeling may introduce useful statistical dependencies that can improve translation accuracy. Unfortunately, learning informative latent variables is non-trivial, as the latent space can be prohibitively large, and the latent codes are prone to be ignored by many translation models at training time. Previous works impose strong assumptions on the distribution of the latent code and limit the choice of the NMT architecture. In this paper, we propose to apply the VNMT framework to the state-of-the-art Transformer and introduce a more flexible approximate posterior based on normalizing flows. We demonstrate the efficacy of our proposal under both in-domain and out-of-domain conditions, significantly outperforming strong baselines.

## introduction translation is inherently ambiguous. for a given source sentence, there can be multiple plausible translations due to the author's stylistic preference, domain, and other factors. on the one hand, the introduction of neural machine translation (nmt) has significantly advanced the field @xcite , continually producing state-of-the-art translation accuracy. on the other hand, the existing framework provides no explicit mechanisms to account for translation ambiguity. recently, there has been a growing interest in latent-variable nmt (lv-nmt) that seeks to incorporate latent random variables into nmt to account for the ambiguities mentioned above. for instance, @xcite incorporated latent codes to capture underlying global semantics of source sentences into nmt, while @xcite proposed fine-grained latent codes at the word level. the learned codes, while not straightforward to analyze linguistically, are shown empirically to improve accuracy. nevertheless, the introduction of latent random variables complicates the parameter estimation of these models, as it now involves intractable inference. in practice, prior work resorted to imposing strong assumptions on the latent code distribution, potentially compromising accuracy. in this paper, we focus on improving variational nmt (vnmt) @xcite : a family of lv-nmt models that relies on the amortized variational method (kingma and welling, 2014) for inference. our contributions are twofold. (1) we employ variational distributions based on normalizing flows @xcite , instead of uni-modal gaussian. normalizing flows can yield complex distributions that may better match the latent code's true posterior. (2) we employ the transformer architecture @xcite , including transformer-big, as our vnmt's generator network. we observed that the generator networks of most vnmt models belong to the rnn family that are relatively less powerful as a translation model than the transformer. we demonstrate the efficacy of our proposal on the german-english iwslt'14 and english-german wmt'18 tasks, giving considerable improvements over strong non-latent transformer baselines, and moderate improvements over gaussian models. we further show that gains generalize to an out-of-domain condition and a simulated bimodal data condition. ## vnmt with normalizing flows background let x and y be a source sentence and its translation, drawn from a corpus d. our model seeks to find parameters θ that maximize the marginal of a latent-variable model p θ (y, z | x) ## discussions and conclusions we present a variational nmt model that outperforms a strong state-of-the-art non-latent nmt model. we show that the gain modestly comes from the introduction of a family of flexible distribution based on normalizing flows. we also demonstrate the robustness of our proposed model in an increased multimodality condition and on a simulated out-of-domain test set. we plan to conduct a more in-depth investigation into actual multimodality condition with highcoverage sets of plausible translations. we conjecture that conditioning the posterior on the target sentences would be more beneficial. also, we plan to consider more structured latent variables beyond modeling the sentence-level variation as well as to apply our vnmt model to more language pairs. of artificial intelligence (iaai-18), and the 8th aaai symposium on educational advances in artificial intelligence (eaai-18), new orleans, louisiana, usa, february 2-7, 2018, pages 5488-5495. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n. gomez, łukasz kaiser, and illia polosukhin. 2017. attention is all you need. in neural information processing systems conference (nips), pages 5998-6008, long beach, usa. felix wu, angela fan, alexei baevski, yann n. dauphin, and michael auli. 2019. pay less attention with lightweight and dynamic convolutions. in international conference on learning representations (iclr), new orleans, usa. biao zhang, deyi xiong, jinsong su, hong duan, and min zhang. 2016. variational neural machine translation. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 521-530, austin, texas. association for computational linguistics.