Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation (2020)

Abstract: Non-autoregressive neural machine translation (NAT) predicts the entire target sequence simultaneously and significantly accelerates inference process. However, NAT discards the dependency information in a sentence, and thus inevitably suffers from the multi-modality problem: the target tokens may be provided by different possible translations, often causing token repetitions or missing. To alleviate this problem, we propose a novel semi-autoregressive model RecoverSAT in this work, which generates a translation as a sequence of segments. The segments are generated simultaneously while each segment is predicted token-by-token. By dynamically determining segment length and deleting repetitive segments, RecoverSAT is capable of recovering from repetitive and missing token errors. Experimental results on three widely-used benchmark datasets show that our proposed model achieves more than 4 times speedup while maintaining comparable performance compared with the corresponding autoregressive model.

## introduction although neural machine translation (nmt) has achieved state-of-the-art performance in recent years @xcite @xcite , most nmt models still suffer from the slow decoding speed problem due to their autoregressive property: the generation of a target token depends on all the previously generated target tokens, making the decoding process intrinsically nonparallelizable. recently, non-autoregressive neural machine translation (nat) models @xcite @xcite @xcite have been investigated to mitigate the src. ## related work there has been various work investigating to accelerate the decoding process of sequence generation models @xcite . in the field of neural machine translation, which is the focus of this work, @xcite first propose non-autoregressive machine translation (nat), which generates all target tokens simultaneously. although accelerating the decoding process significantly, nat suffers from the multimodality problem @xcite which generally manifests as repetitive or missing tokens in translation. therefore, intensive efforts have been devoted to alleviate the multi-modality problem in nat. transformer method, which generates segments autoregressively and predicts the tokens in a segment non-autoregressively. however, none of the above methods explicitly consider recovering from multi-modality related errors. recently, multi-step nat models have also been investigated to address this issue. @xcite and @xcite adopt an iterative decoding methods which have the potential to re-cover from generation errors. besides, @xcite also propose to use dynamic insertion/deletion to alleviate the generation repetition/missing. different from these work, our model changes one-step nat to a semi-autoregressive form, which maintains considerable speedup and enables the model to see the local history and future to avoid repetitive/missing words in decoding. our work can further replace the one-step nat to improve its performance. ## conclusion in this work, we propose a novel semiautoregressive model recoversat to alleviate the multi-modality problem, which performs translation by generating segments non-autoregressively and predicts the tokens in a segment autoregressively. by determining segment length dynamically, recoversat is capable of recovering from missing token errors and reducing repetitive token errors. by explicitly detecting and deleting repetitive segments, recoversat is able to recover from repetitive token errors. experiments on three widely-used benchmark datasets show that our recoversat model maintains comparable performance with more than 4× decoding speedup compared with the at model.