D e PA : Improving Non-autoregressive Translation with Dependency-Aware Decoder (2023)

Abstract: Non-autoregressive machine translation (NAT) models have lower translation quality than autoregressive translation (AT) models because NAT decoders do not depend on previous target tokens in the decoder input. We propose a novel and general Dependency-Aware Decoder (DePA) to enhance target dependency modeling in the decoder of fully NAT models from two perspectives: decoder self-attention and decoder input. First, we propose an autoregressive forward-backward pre-training phase before NAT training, which enables the NAT decoder to gradually learn bidirectional target dependencies for the final NAT training. Second, we transform the decoder input from the source language representation space to the target language representation space through a novel attentive transformation process, which enables the decoder to better capture target dependencies. DePA can be applied to any fully NAT models. Extensive experiments show that DePA consistently improves highly competitive and state-of-the-art fully NAT models on widely used WMT and IWSLT benchmarks by up to 1.88 BLEU gain, while maintaining the inference latency comparable to other fully NAT models.

## introduction autoregressive translation (at) systems achieve state-of-the-art (sota) performance for neural machine translation (nmt) and transformer @xcite encoder-decoder is the prevalent architecture. in at systems, each generation step depends on previously generated tokens, resulting in high inference latency when output is long. nonautoregressive translation (nat) models @xcite significantly accelerate inference by generating all target tokens independently and simultaneously. however, this independence assumption leads to degradation in accuracy compared to at models, as nat models cannot properly learn target dependencies. dependency in prior works and our work takes its standard definition in nlp, i.e., syntactic relations between words in a sentence. the mainstream nat models fall into two categories: iterative nat models and fully nat models. iterative nat models @xcite @xcite improve translation accuracy by iteratively refining translations at the expense of slower decoding speed. in contrast, fully nat models @xcite have great latency advantage over at models by making parallel predictions with a single decoding round, but they suffer from lower translation accuracy. in this paper, we aim at improving the translation accuracy of fully nat models while preserving their latency advantage. previous research @xcite argues that reducing dependencies is crucial for training a fully nat model effectively, as it allows the model to more easily capture target dependencies. however, dependency reduction limits the performance upper bound of fully nat models, since models may struggle to generate complex sentences. previous studies show that multi-modality @xcite is the main problem that nat models suffer from @xcite , i.e., the target tokens may be generated based on different possible translations, often causing over-translation (token repetitions), under-translation (source words not translated), and wrong lexical choice for polysemous words. ## conclusion we propose a novel and general dependency-aware decoder (depa) to enhance target dependency modeling for fully nat models, with forward-backward dependency modeling and decoder input transformation. extensive experiments show that depa improves the translation accuracy of highly competitive and sota fully nat models while preserving their inference latency. in future work, we will evaluate depa on iterative nat models such as imputer, cmlm, and levenshtein transformer and incorporate ranking approaches into depa.