Multi-Granularity Optimization for Non-Autoregressive Translation (2022)

Abstract: Despite low latency, non-autoregressive machine translation (NAT) suffers severe performance deterioration due to the naive independence assumption. This assumption is further strengthened by cross-entropy loss, which encourages a strict match between the hypothesis and the reference token by token. To alleviate this issue, we propose multi-granularity optimization for NAT, which collects model behaviours on translation segments of various granularities and integrates feedback for backpropagation. Experiments on four WMT benchmarks show that the proposed method significantly outperforms the baseline models trained with cross-entropy loss, and achieves the best performance on WMT’16 En⇔Ro and highly competitive results on WMT’14 En⇔De for fully non-autoregressive translation.

## introduction neural machine translation (nmt) systems have shown superior performance on various benchmark datasets @xcite . in the training stage, nmt systems minimize the token-level cross-entropy loss between the reference sequence and the model hypothesis. during inference, nmt models adopt autoregressive decoding, where the decoder generates the target sentence token by token (o(n )). to reduce the latency of nmt systems, @xcite propose non-autoregressive neural machine translation (nat), which improves the decoding speed by generating the entire target sequence in parallel (o(1)). despite low latency, without modeling the target sequence history, nat models tend to generate translations of low quality @xcite @xcite . nat ignores inter-token dependency and naively factorizes the sequence-level probability as a product of independent token probability. however, vanilla † corresponding authors. nat adopts the same training optimization method as autoregressive (at) models, i.e., cross-entropy loss (xe loss), which forces the model to learn a strict position-to-position mapping, heavily penalizing hypotheses that suffer position shifts but share large similarity with the references. given a reference "she left her keys yesterday .", an inappropriate hypothesis "she left her her her ." can yield a lower cross-entropy than one reasonable hypothesis "yesterday she left her keys .". autoregressive models suffer less from the issue by considering previous generated tokens during inference, which is however infeasible for parallel decoding under the independence assumption. as a result, nat models trained using cross-entropy loss are weak at handling multi-modality issues and prone to token repetition mistakes @xcite @xcite . intuitively, generating adequate and fluent translations involves resolving dependencies of various ranges @xcite . for example, to generate a translation "each of the students has a book", the model needs to consider the local n-gram pattern "a -book", the subject-verb agreement across the non-continuous span "each -has", and the global context. to capture the token dependency without the language model, feedback on model's behavior on text spans of multiple granularities can be incorporated. to this end, we propose a multigranularity optimization method to provide nat models with rich feedback on various text spans involving multi-level dependencies. as shown in figure 1 , instead of exerting a strict token-level supervision, we evaluate model behavior on various granularities before integrating scores of each granularity to optimize the model. in this way, for each sample we highlight different parts of the translation, e.g., "a book" or "each of the students has". during training, instead of searching for a single output for each source sequence, we explore the search space by sampling a set of hypotheses. for each hypothesis, we jointly mask part of the tokens and those of the gold reference at the same positions. to directly evaluate each partially masked hypothesis, we adopt metric-based optimization @xcite which rewards the model with a metric function measuring hypothesis-reference text similarity. since both the hypothesis and the reference share the same masked positions, the metric score of each sample is mainly determined by those exposed segments. finally, we weigh each sample score by the model confidence to integrate the metric feedback on segments of various granularities. an illustrative representation is shown in figure 2 , where a set of masked hypothesis-reference pairs are sampled and scored respectively before being merged by segment probabilities. in this way, the model is optimized based on its behavior on text spans of multiple granularities for each training instance within a single forward-backward pass. we evaluate the proposed method across four machine translation benchmarks: wmt14 en⇔de and wmt16 en⇔ro. results show that the proposed method outperforms baseline nat models trained with xe loss by a large margin, while maintaining the same inference latency. the proposed method achieves two best performances for fully non-autoregressive models among four benchmarks, and obtains highly competitive results compared with the at model. to the best of our knowledge, we are the first to leverage multi-granularity metric feedback for training nat models. our code is released at https://github.com/yafuly/mgmonat . ## method we first briefly introduce some preliminaries including non-autoregressive machine translation (section 2.1) and cross-entropy (section 2.2), and then we elaborate our proposed method where the model learns segments of different granularities for each instance (section 2.3). ## conclusion we proposed multi-granularity optimization for nat, which considers metric feedback on hypothesis segments of multiple granularities. through integrating multi-granularity feedback, the model is optimized by focusing on different parts of the sequence within a single forward-backward pass, obtaining more detailed and informative training signals. empirical results demonstrated that our method achieved highly competitive performance compared with other representative baselines for fully nat. analysis further showed that mgmo maintained strong performance on long sequences that vanilla nat models suffer, and obtained high prediction confidence. beyond non-autoregressive translation, our proposed method can be used in other text generation tasks.