A lig NART : Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate (2021)

Abstract: Non-autoregressive neural machine translation (NART) models suffer from the multi-modality problem which causes translation inconsistency such as token repetition. Most recent approaches have attempted to solve this problem by implicitly modeling dependencies between outputs. In this paper, we introduce AligNART, which leverages full alignment information to explicitly reduce the modality of the target distribution. AligNART divides the machine translation task into (i) alignment estimation and (ii) translation with aligned decoder inputs, guiding the decoder to focus on simplified one-to-one translation. To alleviate the alignment estimation problem, we further propose a novel alignment decomposition method. Our experiments show that AligNART outperforms previous non-iterative NART models that focus on explicit modality reduction on WMT14 En↔De and WMT16 Ro→En. Furthermore, AligNART achieves BLEU scores comparable to those of the state-of-the-art connectionist temporal classification based models on WMT14 En↔De. We also observe that AligNART effectively addresses the token repetition problem even without sequence-level knowledge distillation.

## introduction in the neural machine translation (nmt) domain, non-autoregressive nmt (nart) models @xcite have been proposed to alleviate the low translation speeds of autoregressive nmt (art) models. however, these models suffer from degenerated translation quality @xcite . to improve the translation quality of nart, several studies on nart iteratively refine decoded outputs with minimal iterations @xcite @xcite @xcite ; other recent works target to improve nart without iteration @xcite . one of the significant limitations of non-iterative nart models is the multi-modality problem. this problem originates from the fact that the models should maximize the probabilities of multiple targets without considering conditional dependencies between target tokens. for example, in english-to-german translation, a source sentence "thank you very much." can be translated to "danke schön." or "vielen dank.". under the conditional independence assumption, the non-iterative nart models are likely to generate improper translations such as "danke dank." or "vielen schön." @xcite . for the same reason, other inconsistency problems such as token repetition or omission occur frequently in non-iterative nart @xcite . there are two main methods for non-iterative nart to address the multi-modality problem. some works focus on an implicit modeling of the dependencies between the target tokens @xcite . for example, @xcite , @xcite , and @xcite modify the objective function based on dynamic programming, whereas @xcite provide target tokens to the decoder during training. on the other hand, other works focus on an explicit reduction of the modality of the target distribution by utilizing external source or target sentence information rather than modifying the objective function. for example, @xcite and @xcite use syntactic or semantic information; @xcite , @xcite use the alignment information between source and target tokens. however, previous explicit modality reduction methods show suboptimal performance. @xcite and @xcite extract fertility @xcite and ordering information in word alignments, which enables the modeling of several types of mappings except for many-to-one and many-to-many cases. we hypothesize that leveraging entire mappings significantly reduces the modality and is the key to performance improvement. in this work, we propose alignart, a noniterative nart model that mitigates the multimodality problem by utilizing complete information in word alignments. alignart divides the machine translation task into (i) alignment estimation and (ii) non-autoregressive translation under the given alignments. modeling all the type of mapping guides (ii) more close to one-to-one translation. in alignart, a module called aligner is simply augmented to nat @xcite which estimates alignments to generate aligned decoder inputs. however, it is challenging to estimate the complex alignment information using only source sentence during inference. specifically, aligner should simultaneously predict the number of target tokens corresponding to each source token and their mapping. to overcome this problem, we further propose alignment decomposition which factorizes the alignment process into three subprocesses: duplication, permutation, and grouping. each sub-process corresponds to much feasible sub-problems: one-to-many mapping, ordering, and many-to-one mapping, respectively. our experimental results show that alignart outperforms previous non-iterative nart models of explicit modality reduction on wmt14 en↔de and wmt16 @xmath0. alignart achieves performance comparable to that of the recent stateof-the-art non-iterative nart model on wmt14 en↔de. we observe that the modality reduction in alignart addresses the token repetition issue even without sequence-level knowledge distillation @xcite . we also conduct quantitative and qualitative analyses on the effectiveness of alignment decomposition. ## background given a source sentence @xmath1, x 2 , ..., x m } and its translation @xmath2, y 2 , ..., y n }, art models with encoder-decoder architecture are trained with chained target distributions and infer the target sentence autoregressively: at each decoding position n, the decoder of the model is conditioned with previous target tokens @xmath3, ..., y n-1 }, which is the key factor of performance in art models. previous target tokens reduce the target distribution modality and provide information about the target sentence. however, the autoregressive decoding scheme enforces the decoder to iterate n times to complete the translation and increases the translation time linearly with respect to the length of the target sentence. non-iterative nart models @xcite @xcite ) assume conditional independence between the target tokens to improve the translation speed: ## conclusion and future work in this study, we leverage full alignment information to directly reduce the degree of the multimodality in non-iterative nart and propose an alignment decomposition method for alignment estimation. alignart with giza++ shows performance comparable to that of the recent ctcbased implicit dependency modeling approach on wmt14 en-de and modality reduction capability. however, we observe that alignart depends on the quality of the ground truth word alignments, which can be studied in the future work. furthermore, we can study on the combination of alig-nart and implicit dependency modeling methods. we derive the source subword to target subword matrix a using the alignment tool. a ws is achieved by clipping the maximum value of a•s to 1. a ws reduces the search space because of the assumption that source tokens duplicate, permute, and group at the word-level. however, there is a trade-off between the simplicity and resolution of information. the recovered source subword to target subword matrix a ws • s loses the subword-level information as shown in the rightmost matrix in figure 3 .