Rewriter-Evaluator Architecture for Neural Machine Translation (2021)

Abstract: A few approaches have been developed to improve neural machine translation (NMT) models with multiple passes of decoding. However, their performance gains are limited because of lacking proper policies to terminate the multi-pass process. To address this issue, we introduce a novel architecture of Rewriter-Evaluator. Translating a source sentence involves multiple rewriting passes. In every pass, a rewriter generates a new translation to improve the past translation. Termination of this multi-pass process is determined by a score of translation quality estimated by an evaluator. We also propose prioritized gradient descent (PGD) to jointly and efficiently train the rewriter and the evaluator. Extensive experiments on three machine translation tasks show that our architecture notably improves the performances of NMT models and significantly outperforms prior methods. An oracle experiment reveals that it can largely reduce performance gaps to the oracle policy. Experiments confirm that the evaluator trained with PGD is more accurate than prior methods in determining proper numbers of rewriting.

## introduction encoder-decoder architecture @xcite has been widely used in natural language generation, especially neural machine translation (nmt) @xcite @xcite @xcite . given a source sentence, an encoder firstly converts it into hidden representations, which are then conditioned by a decoder to produce a target sentence. in analogy to the development of statistical machine translation (smt) @xcite @xcite , some recent methods in nmt attempt to improve the encoder-decoder architecture with multipass decoding @xcite @xcite . in these models, more than one translation is generated for a source sentence. except for the first translation, each of the later translations is conditioned on the previous one. while these methods have achieved promising results, they lack a proper termination poqlicy for this multi-turn process. for instance, @xcite ; @xcite adopt a fixed number of decoding passes, which is inflexible and can be sub-optimal. @xcite utilize reinforcement learning (rl) @xcite to automatically decide the number of decoding passes. however, rl is known to be unstable due to the high variance in gradient estimation @xcite . to address this problem, we introduce a novel architecture, rewriter-evaluator. this architecture contains a rewriter and an evaluator. the translation process involves multiple passes. given a source sentence, at every turn, the rewriter generates a new target sequence to improve the translation from the prior pass, and the evaluator measures the translation quality to determine whether to end the iterative rewriting process. hence, the translation process is continued until a certain condition is met, such as no significant improvement in the measured translation quality. in implementations, the rewriter is a conditional language model @xcite and the evaluator is a text matching model @xcite . we also propose prioritized gradient descent (pgd) that facilitates training the rewriter and the evaluator both jointly and efficiently. pgd uses a priority queue to store previous translation cases. the queue stores translations with descending order of their scores, computed from the evaluator. the capacity of the queue is limited to be a few times of batch size. due to its limited size, the queue pops those translations with high scores and only keeps the translations with lower scores. the samples in source sentence 𝒙 target sentence 𝒛 (') quality score 𝑞 ('()) the queue are combined together with new cases from the training data to train the rewriter. rewriter-evaluator has been applied to improve two mainstream nmt models, rnnsearch @xcite and transformer @xcite . we have conducted extensive experiments on three translation tasks, nist chineseto-english, wmt'18 chinese-to-english, and wmt'14 english-to-german. the results show that our architecture notably improves the performance of nmt models and significantly outperforms related approaches. we conduct oracle experiment to understand the source of improvements. the oracle can pick the best translation from all the rewrites. results indicate that the evaluator helps our models achieve the performances close to the oracle, outperforming the methods of fixing the number of rewriting turns. compared against averaged performances using a fixed number of rewriting iterations, performance gaps to the oracle can be reduced by 80.7% in the case of rnnsearch and 75.8% in the case of transformer. quantitatively, we find the evaluator trained with pgd is significantly more accurate in determining the optimal number of rewriting turns. for example, whereas the method in @xcite has 50.2% accuracy in wmt'14, the evaluator achieves 72.5% accuracy on transformer. ## rewriter-evaluator rewriter-evaluator consists of iterative processes involving a rewriting process ψ and an evaluation process φ. the process of translating an n-length source sentence @xmath0, x 2 , • • • , x n ] is an application of the above processes. assume we are at the k-th iteration (@xmath1. the rewriter ψ gener-ates a target sequence z initially, z (0) and q (0) are respectively set as an empty string and -∞. the above procedure is repeatedly carried out until not much improvement in the estimated quality score can be achieved, i.e., in the former case, we adopt z (k-1) as the final translation. in the latter case, the last translation z (k) is accepted. ## prioritized gradient descent we present prioritized gradient descent (pgd) to train the proposed architecture. instead of the random sampling used in stochastic gradient descent ## applications following sec. 2.1, we use rewriter-evaluator to improve rnnsearch and transformer. rnnsearch w/ rewriter-evaluator. the improved rnnsearch is illustrated in fig. 2 . the two encoders (i.e., f se and f t e ) and the decoder g dec are gru @xcite . we omit computation details of these modules and follow their settings in @xcite . note that, at every decoding step, the hidden state of decoder is attended to not only we apply co-attention mechanism @xcite to model the estimator f est . firstly, we capture the semantic alignment between the source sentence x and the translation z (k-1) as then, we use average pooling to extract features and compute the quality score: transformer w/ rewriter-evaluator. the transformer @xcite is modified to an architecture in fig. 3 . the input to the encoder contains a source sentence x, a special symbol "align", and the past translation z (k-1) : transformer encoder transformer decoder dot product where operation denotes the concatenation of two sequences. the following mask matrix is applied to every layer in the encoder: in this way, the words in x can't attend to those in z (kfoot_0 ) and vice versa. "align" can attend to the words both in x and z (k-1) . this design is to avoid cross-sentence attention in encoder layers. in earlier studies, we find it slightly improves the performances of models. we denote the representation for "align" in the final encoder layer as h align . the estimator f est obtains the quality score as ## experiments we have conducted extensive experiments on three machine translation tasks: nist chinese-to-english (@xmath2, wmt'18 chinese-to-english, and wmt'14 english-to-german (@xmath3. the results show that rewriter-evaluator significantly improves the performances of nmt models and notably outperforms prior post-editing methods. oracle experiment verifies the effectiveness of the evaluator. termination accuracy analysis shows our evaluator is much more accurate than prior methods in determining the optimal number of rewriting turns. we also perform ablation studies to explore the effects of some components. ## related work multi-pass decoding has been well studied in statistical machine translation @xcite @xcite @xcite . @xcite ; @xcite propose training models with minimum error rate criterion on lattices from first-pass decoder. @xcite introduce an iterative method to refine search space generated from simple feature with additional information from more complex feature. @xcite investigate reranking of hypothesis using neural models trained with discriminative criterion. @xcite propose to reconfirm effectiveness of reranking. @xcite present a regeneration of search space from techniques such as n-gram expansion. these approaches are however applied to shallow models such as log-linear models @xcite . our work is closely related to recent efforts in multi-pass decoding on nmt. in these recent works @xcite @xcite , the models generate multiple target sentences for a source sentence and, except for the first one, each of them is based on the sentence generated in the previous turn. for example, @xcite propose deliberation networks that uses a second decoder to polish the raw sequence produced by the first-pass decoder. while these methods have achieved promising results, they lack a proper termination policy for the multi-pass translation process. @xcite adopt a predefined number of decoding passes, which is not flexible. @xcite incorporate post-editing mechanism into nmt model via rl. however, rl can be unstable for training because of the high variance in gradient estimation. the lack of a proper termination policy results in premature terminations or over-translated sentences, which can largely limit the performance gains of these methods. ## conclusion this paper has introduced a novel architecture, rewriter-evaluator, that achieves a proper termination policy for multi-pass decoding in nmt. at every translation pass, given the source sentence and its past translation, a rewriter generates a new translation, aiming at making further performance improvements over the past translations. an evaluator estimates the translation quality to determine whether to complete this iterative rewriting process. we also propose pgd that facilitates training the rewriter and the evaluator both jointly and efficiently. we have applied rewriter-evaluator to improve mainstream nmt models. extensive experiments have been conducted on three translation tasks, nist @xmath4, wmt'18 @xmath5, and wmt'14 @xmath6, showing that our architecture notably improves the results of nmt models and significantly outperforms other related methods. an oracle experiment and a termination accuracy analysis show that the performance gains can be attributed to the improvements in completing the rewriting process at proper iterations.