Train Once, and Decode As You Like (2020)

Abstract: In this paper we propose a unified approach for supporting different generation manners of machine translation, including autoregressive, semi-autoregressive, and refinement-based non-autoregressive models. Our approach works by repeatedly selecting positions and generating tokens at these selected positions. After being trained once, our approach achieves better or competitive translation performance compared with some strong task-specific baseline models in all the settings. This generalization ability benefits mainly from the new training objective that we propose. We validate our approach on the WMT’14 English-German and IWSLT’14 German-English translation tasks. The experimental results are encouraging.

## introduction neural sequence models @xcite @xcite have been successfully applied to machine translation. previous work in this area is mostly based on the encoder-decoder framework, where an encoder first makes a source sentence into hidden states and then a decoder generates a corresponding target sentence from these hidden states. the generation process usually relies on the chain-rule factorization and is performed in an autoregressive manner, i.e., tokens by tokens from left to right. although this manner brings some advantages in training and inference, high latency in inference is a grave disadvantage due to a linear number of generation steps @xcite . in order to reduce the latency during inference, @xcite proposed the non-autoregressive transformer (nat) that can generate an entire translation in parallel given a source sentence. @xcite modified the initial non-autoregressive translation repeatedly through a denoising autoencoder. although these non-autoregressive models can generate a corresponding translation at a constant number of generation steps, they are obviously worse than their autoregressive counterparts. @xcite devised the semi-autoregressive transformer (sat) that keeps the autoregressive property in global but relieves it in local to generate multiple successive tokens in parallel. by generating k tokens in parallel each time, sat can reduce the number of generation steps to 1/k of the original quantity. a proper k can achieve a good trade-off between speed and latency. in addition to parallel generation, some previous work @xcite @xcite pays attention to non-monotonic sequence generation, e.g., a binary tree traversal. these models are mainly based on the insertion operation. we can abstract the aforementioned different generation manners of machine translation into a more general process. one first determines a stopping criterion, then repeatedly selects positions and generates tokens at these selected positions until the stopping criterion is met. different position selection strategies can simulate different generation manners. for example, if a model always predicts the token at the next position and stops translation when a specific terminator (eos) is generated or the pre-specified length is met, this can be considered as an autoregressive model. based on this idea, we propose a unified framework of different generation manners of machine translation, taking the models mentioned above as special cases. in other words, our framework supports different popular generation manners such as autoregressive, semi-autoregressive, non-autoregressive based on iterative refinement and non-monotonic generation with one set of well-trained parameters. the benefit is obvious that deploying a flexible unified model is more space-efficient than deploying multiple task-specific models, assuming that users have different needs for the trade-off between translation speed and performance in different situations. to implement this framework, our approach should have the ability to generate corresponding tokens after pre-assigned or adaptive position selection. a vanilla neural sequence model, which always predicts the next token given a source sentence and all previous generated tokens, cannot complete this task. we organically incorporate the two-stream self-attention mechanism @xcite for target-aware representations into traditional machine translation models to solve this problem. in addition to a welldesigned model structure, a matching training objective is also indispensable. inspired by the permutation language modeling @xcite , we maximize the expectation of log conditional likelihood of all possible factorization orders of a pair of source and target sentences, rather than a traditional left-to-right objective. we experimentally validate that the new training objective significantly improves the robustness and performance of models. we evaluate our approach on the wmt'14 english-german and iwslt'14 german-english translation tasks. compared with some strong task-specific baseline models, our approach achieves better or comparable results in all generation manners with shared parameters. we also investigate the impact of different generation orders on translation performance. it is surprising to find that the left-to-right order achieves the best results among seven generation orders when generating tokens one by one, although our approach has no special preference for the generation order during training. this shows that the left-to-right order is still a simple but powerful way of language modeling, which may explain why most (conditional) language models are based on autoregressive decomposition. to this end, we summarize our contributions as follows: ## related work neural sequence models based on autoregressive decomposition @xcite @xcite have become the benchmark for machine translation. recently, some work @xcite @xcite has found that it is unnecessary to generate tokens in an autoregressive manner. in other words, the generation process can be parallel or non-monotonic. however, the model structure and training strategy of these methods are task-specific, and cannot flexibly support different generation requirements for a better trade-off between speed and translation performance. our model is inspired by xlnet @xcite , a pre-trained language model. it achieved the state-of-the-art results on 18 different language understanding tasks, partly due to the novel permutation language modeling. we find that the permutation language modeling also has special benefits for conditional language models, which makes the generation process more robust to the order and manner. we must also mention the work of @xcite . they similarly defined a generalized framework of sequence generation, but there are some differences between these two frameworks. the main difference is that our model is based on the permutation language modeling @xcite and selects the position by pre-specifying or adaptively according to the probability, while their model is based on the masked language modeling @xcite and selects the position by more complex adaptive gibbs sampling. apart from this, they used a pre-trained model trained on a monolingual dataset and finetuned it on the translation task while we only use the parallel corpus and the training process does not require two stages. ## methodology given a pair of source sentence @xmath0, x 2 , • • • , x t } and corresponding target sentence @xmath1, y 2 , • • • , y t }, vanilla neural sequence models, such as the transformer @xcite , factorize the conditional probability in an autoregressive manner, ## special cases thanks to our model's flexibility and robustness, different popular generation manners can be simulated by a specific position selection strategy given a predicted target length. ## experimental settings data and preprocessing we evaluate our framework on the wmt'14 english-german (around 4.5m sentences pairs) and iwslt'14 german-english (around 153k sentences pairs) translation tasks. the implementation is based on pytorch @xcite . we use scripts from fairseq @xcite to preprocess the dataset, which first tokenize each sentence using a script from moses @xcite and then segment each word into subword units using bpe @xcite with a joint vocabulary. the sizes of vocabulary for wmt dataset and iwslt dataset are 37k and 10k, respectively. ## results and analysis it is necessary to mention that different from other task-specific models, our model shares a set of welltrained parameters in all settings, namely the results of ## conclusion we have proposed a unified approach for different generation manners of machine translation. after being trained with the conditional permutation language modeling objective, our model is able to support different generation manners. we have validated our model on the wmt'14 english-german and iwslt'14 german-english translation tasks. compared with some strong baseline models, our model achieves better or competitive translation performance in all the settings without task-specific training. although we have only verified our ideas on translation tasks, it is easy to find that our approach can be naturally applied to other conditional sequence tasks. we would like to investigate the application of our approach to other tasks in the future.