Learning to Rewrite for Non-Autoregressive Neural Machine Translation (2021)

Abstract: Non-autoregressive neural machine translation, which decomposes the dependence on previous target tokens from the inputs of the decoder, has achieved impressive inference speedup but at the cost of inferior accuracy. Previous works employ iterative decoding to improve the translation by applying multiple refinement iterations. However, a serious drawback is that these approaches expose the serious weakness in recognizing the erroneous translation pieces. In this paper, we propose an architecture named RewriteNAT to explicitly learn to rewrite the erroneous translation pieces. Specifically, RewriteNAT utilizes a locator module to locate the erroneous ones, which are then revised into the correct ones by a revisor module. Towards keeping the consistency of data distribution with iterative decoding, an iterative training strategy is employed to further improve the capacity of rewriting. Extensive experiments conducted on several widely-used benchmarks show that RewriteNAT can achieve better performance while significantly reducing decoding time, compared with previous iterative decoding strategies. In particular, RewriteNAT can obtain competitive results with autoregressive translation on WMT14 En-De, En-Fr and WMT16 Ro-En translation benchmarks.

## introduction however, existing iterative nat models expose the weakness in distinguishing the erroneous words. the dominant approach to identify the mistakes is mask-predict algorithm @xcite , which employs inefficient heuristic rules to roughly choose the least confident words as the erroneous. in some case, mask-predict may mistake to rewrite correct words while maintain erroneous ones, acting as noises to make a negative impact on subsequent iterations. without explicitly classifying translated words into wrong or right, the translations decode in constant number of iterations, hindering the further improvement of inference speed. besides, decoder inputs of prevailing iterative nat models @xcite almost come from the ground-truth during training, while target sentences generated at different refinement steps are taken as decoder inputs in inference, creating a discrepancy that can hurt performance. in this paper, we propose an architecture named rewritenat, which explicitly learns to rewrite erroneous translation pieces. specifically, we introduce a locator module to locate incorrect words within previously generated translation. the located words will be masked out and revised by the revisor module in subsequent refinement. we frame learning to rewrite, comprised of two steps: locate and revise, as an iterative training procedure, where locate and revise operations are supervised by comparing the generated translation with the groundtruth. towards keeping the consistency with iterative decoding, iterative training is utilized to further improve the training procedure. experimental results on several typical machine translation datasets demonstrate that rewritenat achieves consistent improvement over iterative decoding baselines, but with substantially less decoding time. further analysis show that rewritenat prefers to generate the "easy" words at the early decoding iteration, and leaves the more complicated choice later. ## architecture as depicted in figure 2 , our proposed rewrite-nat literally consists of three major components: an encoder, a revisor and a locator. the encoder utilizes transformer encoder, comprisedof n e transformer blocks @xcite , to convert source sentence into the contextual representations, similar to previous work @xcite . the you thank . emb emb emb multi-head self-attention feed forward ×n e h e 1 h e 2 h e 3 [m ask ] [m ask ] [m ask ] emb emb emb multi-head self-attention ×n r multi-head encoder-decoder attention feed forward softmax softmax softmax dank dank . dank dank . emb emb emb multi-head self-attention ×n l multi-head encoder-decoder attention feed forward softmax softmax softmax dank [m ask ] . emb emb emb multi-head self-attention ×n r multi-head encoder-decoder attention feed forward softmax vielen × refinements m encoder revisor locator revisor keep revise keep dank vielen . emb emb emb multi-head self-attention ×n l multi-head encoder-decoder attention feed forward softmax softmax softmax locator keep keep keep dank . figure 2: architecture of our proposed rewritenat model, which consists of three major components: an encoder, a revisor and a locator. the encoder is utilized to convert the source sentence into contextual representations. during the decoding, the revisor converts the erroneous words annotated as "[mask]" into the correct ones, while the incorrect words within previously generated hypothesis are distinguished, by classifying the words into two classes: revise and keep. given previously located hypothesis, m refinements, each of which utilizes a revisor and a following locator refine the hypothesis, are applied to obtain the final translation. we take an instance from @xmath0, where source sentence is "thank you .". rewritenat applies two refinements into the initial hypothesis, merely comprised of "[mask]". subsequently, the decoding terminates since the locator categorizes the entire sequence into keep, meaning that any word is not required to be revised. ## analysis 6.1 word repetitions with decoupling the sequential dependencies among target sentence, nat shows the serious weakness in modeling highly multimodal distributions @xcite , often manifest as word repetitions @xcite in generated translations. towards evaluating the multi-modality, we follow @xcite to measure the percentage of consecutive repetitive words as a proxy metric. as shown in ## conclusion in this work, we propose an architecture named rewritenat, which explicitly learns to rewrite the erroneous translation pieces, and iterative training is utilized to train this architecture. extensive experimental results demonstrate rewritenat can achieve remarkable improvement over previous iterative nat models, but with significantly less decoding iterations. the further analysis reveals that the generation orders of rewritenat measured by the percentage of decoding iterations are consistent with easy-first hypothesis.