Enriching Non-Autoregressive Transformer with Syntactic and Semantic Structures for Neural Machine Translation (2021)

Abstract: The non-autoregressive models have boosted the efficiency of neural machine translation through parallelized decoding at the cost of effectiveness, when comparing with the autoregressive counterparts. In this paper, we claim that the syntactic and semantic structures among natural language are critical for non-autoregressive machine translation and can further improve the performance. However, these structures are rarely considered in the existing non-autoregressive models. Inspired by this intuition, we propose to incorporate the explicit syntactic and semantic structure of languages into a non-autoregressive Transformer, for the task of neural machine translation. Moreover, we also consider the intermediate latent alignment within target sentences to better learn the long-term token dependencies. Experimental results on two real-world datasets (i.e., WMT14 En-De and WMT16 En- Ro) show that our model achieves a significantly faster speed, as well as keeps the translation quality when compared with several state-of-the-art non-autoregressive models.

## introduction recently, non-autoregressive models @xcite , which aim to enable the parallel generation of output tokens without sacrificing translation quality, have attracted much attention. although the non-autoregressive models have considerably sped up the inference process for real-time neural machine translation (nmt) @xcite , their performance is considerably worse than that of autoregressive counterparts. most previous works attribute the poor performance to the inevitable conditional independence issue when predicting target tokens, and many variants have been proposed to solve it. for example, several techniques in nonautoregressive models are investigated to mitigate the trade-off between speedup and performance, including iterative refinement @xcite , insertion-based models @xcite , latent variable based models @xcite , ctc models @xcite , alternative loss function based models @xcite @xcite , and masked language models @xcite . although these works have tried to narrow the performance gap between autoregressive and nonautoregressive models, and have achieved some improvements on machine translation, the nonautoregressive models still suffer from syntactic and semantic limitations. that is, the translations of non-autoregressive models tend to contain incoherent phrases (e.g., repetitive words), and some informative tokens on the source side are absent. it is because in non-autoregressive models, each token in the target sentence is generated independently. consequently, it will cause the multimodality issue, i.e., the non-autoregressive models cannot model the multimodal distribution of target sequences properly @xcite . one key observation to mitigate the syntactic and semantic error is that source and target translated sentences follow the same structure, which can be reflected from part-of-speech (pos) tags and named entity recognition (ner) labels. briefly, pos, which aims to assign labels to words to indicate their categories by considering the longdistance structure of sentences, can help the model learn the syntactic structure to avoid generating the repetitive words. likewise, ner, which discovers the proper nouns and verbs of sentences, naturally helps the model recognize some meaningful semantic tokens that may improve translation quality. this observation motivates us to leverage the syntactic as well as semantic structures of natural language to improve the performance of non- ## background regardless of its convenience and effectiveness, the autoregressive decoding methods suffer two major drawbacks. one is that they cannot generate multiple tokens simultaneously, leading to ineffi-cient use of parallel hardware such as gpus. the other is that beam search has been found to output low-quality translation with large beam size and deteriorates when applied to larger search spaces. however, non-autoregressive transformer (nat) could potentially address these issues. particularly, they aim at speeding up decoding through removing the sequential dependencies within the target sentence and generating multiple target tokens in one pass, as indicated by the following equation: ## methodology in this section, we present our model snat to incorporate the syntactic and semantic structure information into a nat model as well as an intermediate latent space alignment within the target. figure 1 gives an overview of the network structure of our proposed snat. in snat, the input sequence is segmented into sub-words by byte-pair one of the most important properties of snat is that it naturally introduces syntactic and semantic information when taking the structure-aware information as inputs and generating both words and labels. more precisely, given a source sentence x, as well as its label sequence l x , the conditional probability of a target translation y and its label sequence l y is: ## experiment in this section, we conduct experiments to evaluate the effectiveness and efficiency of our proposed model, with comprehensive analysis.