Synchronous Refinement for Neural Machine Translation (2022)

Abstract: Machine translation typically adopts an encoder-to-decoder framework, in which the decoder generates the target sentence word-by-word in an auto-regressive manner. However, the auto-regressive decoder faces a deep-rooted one - pass issue whereby each generated word is considered as one element of the final output regardless of whether it is correct or not. These generated wrong words further constitute the target historical context to affect the generation of subsequent target words. This paper proposes a novel synchronous refinement method to revise potential errors in the generated words by considering part of the target future context. Particularly, the proposed approach allows the auto-regressive decoder to refine the previously generated target words and generate the next target word synchronously. The experimental results on three widely-used machine translation tasks demonstrated the effectiveness of the proposed approach.

## introduction recently, the encoder-decoder framework has obtained impressive results over various machine translation tasks @xcite . typically, decoder first represents those generated target words as a dependent-time target representation, and then uses an attention mechanism to summarize a dependent-time context from the source input for generating the next target word. since this generated target word is conditioned on previously generated target words at each time step, the decoding process is often called auto-regressive decoding. finally, decoder generates a target language sentence word-by-word in the auto-regressive decoding manner @xcite . however, the auto-regressive decoder often encounters an inherent one-pass issue whereby * corresponding author each generated target word is one element of the final output of the machine translation model regardless of whether it is correct or not. these generated wrong target words are further added to the target historical context to affect the generation of subsequent target words, which hinders the performance of machine translation. take a chinese-to-english translation case in figure 1 generated by a trained neural machine translation (nmt) model @xcite , we illustrate the once-generation issue. in the generated target translation "tgt", there is first an inappropriate translation "clean up" compared with "monitor" in the reference "ref ". the "clean up" is regarded as the final translation to confuse the understanding of the meaning of the source sentence. when these inappropriate or incorrect target words constitute part of the target historical context, the oncegeneration issue further affects the generation of subsequent target words, for example, "drivers' mobile phone" is far away from the meaning of source sentence "the driver plays mobile phone while driving". to verify this issue, we artificially revised the inappropriate translation "clean up" as "monitor" during the decoding, and observed that the subsequent translation "driver plays mobile phone while driving" in "revised" almost completely expresses the corresponding chinese meaning. we believe that correcting the potential errors in generated translations will improve the quality of the translations. many efforts have been initiated on revising potential errors in the generated target translation for machine translation, for example, automatic post-editing @xcite @xcite and two-pass decoding @xcite @xcite . despite their success, most of these approaches asynchronously simulated the generation of the next target word and the revision additionally, the proposed approach is easily introduced into the autoregressive decoder without complex modification. we extensively evaluated it on three widelyused machine translation benchmarks, including wmt14 english-to-german, wmt14 english-to-french, and wmt17 chinese-to-english, and the experimental results demonstrated the effectiveness of the proposed approach. ## background in this paper, we use the advanced encoderdecoder framework, transformer @xcite , to introduce the language generation models. to simplify the process, we simply format the main self-attention network (san) module and do not involve other modules (e.g., positional encoding, multiple stacked layers, and so on). encoder represents the source input @xmath0, • • • , x j } as the source representation @xmath1, • • • , h j } using sans. decoder then generates the target sentence word-by-word based on h with attention mechanism and the generated target fragment. specifically, given a sequence of word vectors in the generated target fragment {e[y 1 ], • • • , e[y i-1 ]} (e is the embedding matrix of the target language vocabulary), they are packed into keyvalue matrices k i-1 and v i-1 at the i-th time-step: ## methodology intuitively, when there is a target sentence with several incorrect target words, a native target language speaker often detects incorrect words according to the contextual information and tries to revise them. we infer that there are two key aspects in the artificial refinement process: i) how to identify those incorrect target words based on the context information; ii) how to revise the identified incorrect target words. to this end, we propose to simulate the above artificial refinement process, to refine the previously generated target words and generate the next target word synchronously. ## conclusion this paper explored part of the target future context to revise fixed potential errors in the generated target fragment caused by the "onepass" issue of the auto-regressive decoder. we proposed a novel synrefinement approach to the machine translation, where the refinement of generated target words is synchronized with the generation of the next target word at each time step. meanwhile, the proposed synrefinement can be easily introduced into the encoder-decoder framework without complex modifications. we evaluated the effectiveness of the proposed approach on three classical machine translation tasks. in the future, we will try to explore how to intelligently identify and correct generation errors.