Non-Autoregressive Machine Translation with Latent Alignments (2020)

Abstract: This paper presents two strong methods, CTC and Imputer, for non-autoregressive machine translation that model latent alignments with dynamic programming. We revisit CTC for machine translation and demonstrate that a simple CTC model can achieve state-of-the-art for single-step non-autoregressive machine translation, contrary to what prior work indicates. In addition, we adapt the Imputer model for non-autoregressive machine translation and demonstrate that Imputer with just 4 generation steps can match the performance of an autoregressive Transformer baseline. Our latent alignment models are simpler than many existing non-autoregressive translation baselines; for example, we do not require target length prediction or re-scoring with an autoregressive model. On the competitive WMT’14 En \rightarrow De task, our CTC model achieves 25.7 BLEU with a single generation step, while Imputer achieves 27.5 BLEU with 2 generation steps, and 28.0 BLEU with 4 generation steps. This compares favourably to the autoregressive Transformer baseline at 27.8 BLEU.

## introduction non-autoregressive neural machine translation @xcite aims to enable the parallel generation of output tokens without sacrificing translation quality. there has been a surge of recent interest in this family of efficient decoding models, resulting in the development of iterative refinement @xcite , ctc models @xcite , insertion-based methods @xcite , editbased methods @xcite , masked language models @xcite , and normalizing flow models @xcite . some of these methods generate the output tokens in a constant number of steps @xcite @xcite @xcite , while others require a logarithmic number of generation steps @xcite @xcite . recent progress has decreased the gap between autoregressive and non-autoregressive models' translation scores. however, non-autoregressive models often suffer from two main limitations: this paper addresses the limitations of existing non-autoregressive machine translation models by using latent alignment models. latent alignment models utilize a sequence of discrete latent alignment variables to monotonically align the nonautoregressive predictions of the model and output tokens. such models use dynamic programming to marginalize out the alignment variables during training. this paper studies two instances of latent alignment models including connectionist temporal classification (ctc) @xcite @xcite and imputer @xcite . @xcite have previously applied ctc to non-autoregressive machine translation. however, we report a significant improvement over the work of @xcite and demonstrate that ctc can achieve the state-of-the-art in single-step nonautoregressive machine translation. we attribute this performance difference primarily to our use of distillation during training, similar to @xcite . we adapt latent alignment models to machine translation and demonstrate their effectiveness on non-autoregressive machine translation, advancing state-of-the-art on wmt'14 en↔de and wmt'16 en↔ro. the main contributions of this paper include: 1. we adapt latent alignment models to nonautoregressive machine translation. ## latent alignment models we begin by describing the notion of alignment, which in the context of this paper is defined as in the ctc literature @xcite @xcite and should not be confused with word alignments in machine translation @xcite . alignment is a mapping between a sequence of predicted tokens and a sequence of target tokens. alignment can be constructed by inserting special "blank tokens" into the target sequence to match a pre-specified length. our alignments have the same length as the source sequences, and collapsing the alignment's blank tokens will recover the target sequence. let x denote a source sequence and let y denote a target sequence, where y @xmath0. we make two assumptions: 1) there exists a monotonic mapping between the model's predictions and the target sequence, and 2) the source sequence is at least as long as the target sequence, i.e. |x| ≥ |y|. we define an alignment a between x and y as a discrete sequence in which a @xmath1, |a| = |x|, and " " is a special "blank" token that is removed to convert a to the target sequence y. we define a function β(y) that returns all possible alignments for a sequence y of a particular length |x|. we also define the collapsing function β -1 (a) such that β -1 (a) = y if @xmath2. to avoid token repetitions, it is useful to define the collapsing function β -1 (a) as first collapsing all consecutive repeated tokens, and then removing all blank tokens. this formulation follows ctc precisely @xcite . for instance, given a source sequence x of length 10, and a target sequence @xmath3, a, b, c, d), then a possible alignment a is ( , a , a , , a , b , b , c , , d ). the log-likelihood of the target sequence is recovered by marginalizing the latent alignments: the summation in (1) is typically intractable, since there are a combinatorial number of alignments. in the next two sub-sections, we will briefly describe two variants of latent alignment models that leverage dynamic programming to tractably compute the log-likelihood, connectionist temporal classification (ctc) @xcite and imputer @xcite . ## latent alignment models for machine translation in this section, we will discuss how latent alignment models can be adapted to machine translation, and then describe key advantages offered by these models. section 2 identified two assumptions made by latent alignment models: 1) there exists a monotonic mapping between the model alignment predictions and the target sequence, and 2) the length of the target sequence is less than or equal to the length of source sequence, i.e. |y| ≤ |x|. we will now address these issues to adapt our models for machine translation. ## related work there has been significant prior work on nonautoregressive iterative methods for machine translation @xcite , some of which are: iterative refinement @xcite , insertionbased methods @xcite @xcite , and conditional masked language models @xcite . like insertion-based models @xcite , our work does not commit to a fixed target length; insertionbased models can dynamically grow the canvas size, whereas our work which relies on a latent alignment can only generate a target sequence up to a fixed maximum predetermined length. compared to conditional masked languages models @xcite , key differences are: 1) our models do not require target length prediction, and 2) we eschew the encoderdecoder neural architecture formulation, but rather rely on the single simple decoder architecture. kermit @xcite also has a similar neural architecture as us; they also eschew the conventional encoder-decoder architecture and have a unified architecture. our work relies on the superpositioning of the input and output sequences via the latent alignment, whereas kermit relies on concatenation to process the input and output sequences. their work is also more focused on input: ein weiterer, besonders wichtiger faktor sei die vernetzung von hochschulen und unternehmen. output: another particularly important factor is the networking of universities and businesses. ## experiments hyperparameters. we follow the base transformer @xcite for our experiments. however, since our architecture does not contain an encoder, we double the number of layers in our decoder to maintain the same number of parameters. our models consist of 12 self-attention layers, with 512 hidden size, 2048 filter size, and 8 attention heads per layer. we use 0.1 dropout for regularization. we batch sequences of approximately same lengths together, with approximately 2048 tokens per batch. we use adam optimizer @xcite with β = (0.9, 0.997) @xmath4. the learning rate warms up to 10 -3 in the first 10k steps and then decays with the inverse square root schedule following the tensor2tensor implementation @xcite . we train all our models for 2m steps. we train the imputer using ctc loss (all masked prior alignment) for 1m steps, followed by bernoulli masking policy @xcite for next 1m steps. we average the 5 checkpoints with the best performance on the development set to get the final model. for imputer, we use top-k decoding during inference. we use canvas scale @xmath5, meaning we upsample the source sequence by a factor of 2. dataset. we perform experiments on wmt'14 en↔de, using newstest2013 as the development set, and report newstest2014 as the test set. we also report our performance on wmt'16 en-ro. we use sentencepiece @xcite to generate a shared subword vocabulary. we evaluate the performance of our models with bleu @xcite . distillation. we follow prior work @xcite @xcite and use data distilled from an autoregressive teacher for training our models. we use autoregressive base transformers for generating distilled data. for iterative generation, we also report the performance of imputer model trained on data distilled from autoregressive big transformers to be comparable with @xcite which distilled from a big transformer. for wmt'16 en-ro, we use the distilled dataset provided by @xcite 1 . we analyze the impact of distillation on the performance of our models in section 6.3. ## analysis in this section, we present further analysis of our latent alignment models. we analyze the (1) impact on token repetitions in generated translations, (2) impact of the number of decoding iterations on imputer, (3) impact of distillation on our models, and (4) impact of target length on imputer. ## conclusion in this paper, we investigated two latent alignments models, ctc and imputer, for nonautoregressive machine translation. ctc is a single step generation model, while imputer is an iterative generative model requiring only a constant number of generation steps. our models rely on dynamic programming to marginalize out the latent alignments. unlike many prior works, our models do not need to perform target length prediction, or re-scoring of candidates and our models use a simplified neural architecture without the need of cross-attention mechanism found in many prior encoder-decoder architectures. we demonstrate the ease and effectiveness of the application of these simple latent alignment models primarily used in speech recognition to the task of machine translation. applying these latent alignment models for parallel translation of long documents can be an interesting research direction.