Almost Free Semantic Draft for Neural Machine Translation (2021)

Abstract: Translation quality can be improved by global information from the required target sentence because the decoder can understand both past and future information. However, the model needs additional cost to produce and consider such global information. In this work, to inject global information but also save cost, we present an efficient method to sample and consider a semantic draft as global information from semantic space for decoding with almost free of cost. Unlike other successful adaptations, we do not have to perform an EM-like process that repeatedly samples a possible semantic from the semantic space. Empirical experiments show that the presented method can achieve competitive performance in common language pairs with a clear advantage in inference efficiency. We will open all our source code on GitHub.

## introduction successful nmt (neural machine translation) @xcite @xcite can translate sentences through left to right or through right to left. however, there is one critical limitation in this diagram. that is, the decoder can only have access to directional information (left-to-right or rightto-left) when processing auto-regressive @xcite . to alleviate this pain, there have been three successful lines. 1) generative nmt: @xcite @xcite @xcite adapt vae (variational auto-encoder) @xcite @xcite for nmt that is trained in generative model settings, modeling the semantics of the source and target sentences in latent space. 2) deliberation: since the problem is caused by the one-pass process of decoding in the auto-regression process, @xcite present a framework to predict a guess target sentence in the first-pass and jointly considers the encoding and the guess target sentence in the second-pass. 3) soft-prototype: @xcite present a framework to generate a prototype on the encoder side and then the decoder can jointly use the encoding and the prototype. although empirical results show the previous methods can successfully inject global information into the decoder, these methods either introduce computational complexity to the encoder-decoder architecture or employ an em-like process in inferring, thus requiring even more than 100% additional time to produce and consider global information in inferring. in this work, we present an efficient method to sample and consider a semantic draft as global information for decoding with almost free of cost, following the line of generative nmt. concretely, we sample the semantic draft from semantic space that is a gaussian inference model with learnable parameters. in the classic utilization of the semantic space, e.g., generative nmt, inferring needs to work with the em-like process that could degrade the inference efficiency significantly. to mitigate the degradation but still use the semantic space, we train the encoder of nmt in multilingual settings and simultaneously train a cross-lingual generator to obtain an approximation of the targetsentence semantic, hence modeling the required semantic space from the approximation and the source-sentence semantic. in inferring, based on the source-sentence semantic and an approximation made by the cross-lingual generator, the semantic draft can be sampled from the semantic space in a one-shot style. once the semantic draft has been sampled, we aggregate the semantic draft and the encoding so that the variational decoder can simply decompose the aggregation. we train the model in generative settings with additional loss of kl-divergence that is used to optimize the semantic space, similar to generative nmt training @xcite @xcite @xcite and vae training @xcite @xcite . our work can build upon transformer @xcite , lstm/gru @xcite and convolutional sequence @xcite . in this work, we use transformer as an example to present our idea, evaluating our method on common translation tasks and 5 more comprehensive experiments. our empirical study shows that, compared to previously successful methods, our method can achieve competitive performance and has a clear advantage in inference efficiency. since we do not change the architecture of the nmt model, our model is compatible with common technics in nmt. nmt @xcite @xcite utilizes seq2seq learning @xcite and autoregressive @xcite to facilitate training and inferring. concretely, the current translation y j at time-step j is conditional on enc(x) and @xmath0, where @xmath1. the intrinsic problem is caused because the translation y j can only consider @xmath2. intuitively, a semantic draft or global information including @xmath3. some impressive methods have been proposed to produce and consider a draft providing global information for better translation quality. 1) gener-ative nmt (including variational nmt) @xcite @xcite @xcite ) study latent and continuous space of semantic @xcite for nmt, which can sample z. these methods inject z into nmt to provide global information for better translation. meanwhile, the encoder is encouraged to consider z. in this manner, generative nmt models the joint probability p nmt (x, y, z) = p(z)p(x|z)p nmt (y |x, z) in training. for inferring, the model utilizes the em-like process to maximize a lower bound on log(p(x, y )) by repeatedly guessing or predicting possible y and resampling z. however, compared to nmt without z, generative nmt costs over 100% additional time in inferring typically. 2) sharing the same idea of the reconsideration of the current translation, deliberation @xcite is proposed to deliberate the complete output of the first-pass decoding as the attention context of the second-pass decoding. with the deliberation, the final translation is based on the understanding of a possible translation in the target language. although deliberation is employed without the emlike process, which is more efficient than generative nmt in inferring, the doubled pass increases the time of auto-regressive in decoding that costs 80% additional time in inferring. 3) @xcite further consider the inference efficiency and the storage cost, proposing soft-prototype framework to use a prototype. the prototype is an approximation of the target sentence @xmath4, ..., y i ), produced by a probability generator r that accepts any x to generate a probability p(y ) over the target vocabulary to search y . these successful methods, although using different settings and frameworks, share the same idea to inject a draft of the required target sentence and introduce global information to the decoder. therefore, the decoder can understand the target globally. concretely, such an idea can be formulated into a framework as: however, these successful methods either introduce computational complexity to nmt @xcite or employ the em-like process, showing significant degradation in inference efficiency, e.g., gnmt @xcite needs 110% additional inferring time. intuitively, a high-quality draft should include two main aspects: ## nmt with semantic draft in this section, we present our framework and method. we then discuss how to train the model in generative settings and how to tackle optimization challenges in practice. ## conclusion and future work translation quality can be further improved by global information from the target sentence. although there have been three feasible solutions, successful methods do not consider inference efficiency carefully, which leads to high cost in inferring. in this work, we present a method/framework to improve the performance of nmt. we sample a semantic draft from semantic space that the decoder can consider the semantic draft to obtain the required global information with high efficiency in inferring. our empirical study shows that, compared to previously successful methods, our method can achieve competitive performance and has a clear advantage in inference efficiency. since we do not change the architecture of the nmt model, our model can be further improved by employing pre-training @xcite @xcite , back-translation @xcite and other finetuning methods with non-parallel data. and, our model can also be used in unsupervised nmt @xcite . we leave all these experiments for future work.