Regularized Context Gates on Transformer for Machine Translation (2020)

Abstract: Context gates are effective to control the contributions from the source and target contexts in the recurrent neural network (RNN) based neural machine translation (NMT). However, it is challenging to extend them into the advanced Transformer architecture, which is more complicated than RNN. This paper first provides a method to identify source and target contexts and then introduce a gate mechanism to control the source and target contributions in Transformer. In addition, to further reduce the bias problem in the gate mechanism, this paper proposes a regularization method to guide the learning of the gates with supervision automatically generated using pointwise mutual information. Extensive experiments on 4 translation datasets demonstrate that the proposed model obtains an averaged gain of 1.0 BLEU score over a strong Transformer baseline.

## introduction an essence to modeling translation is how to learn an effective context from a sentence pair. statistical machine translation (smt) models the source context from the source-side of a translation model and models the target context from a target-side language model @xcite @xcite . these two models are trained independently. on the contrary, neural machine translation (nmt) advocates a unified manner to jointly learn source and target context using an encoderdecoder framework with an attention mechanism, leading to substantial gains over smt in translation quality @xcite @xcite . prior work on attention mechanism @xcite @xcite @xcite @xcite have shown a better context representation is helpful to translation performance. figure 1: a running example to raise the context control problem. both original and context gated transformer obtain an unfaithful translation by wrongly translate "t ī qíu" into "play golf" because referring too much target context. by regularizing the context gates, the purposed method corrects the translation of "t ī qíu" into "play soccer". the light font denotes the target words to be translated in the future. for original transformer, the source and target context are added directly without any rebalancing. however, a standard nmt system is incapable of effectively controlling the contributions from source and target contexts @xcite to deliver highly adequate translations as shown in figure 1. as a result, @xcite carefully designed context gates to dynamically control the influence from source and target contexts and observed significant improvements in the recurrent neural network (rnn) based nmt. although transformer @xcite delivers significant gains over rnn for translation, there are still one third translation errors related to context control problem as described in section 3.3. obviously, it is feasible to extend the context gates in rnn based nmt into transformer, but an obstacle to accomplishing this goal is the complicated archi-tecture in transformer, where the source and target words are tightly coupled. thus, it is challenging to put context gates into practice in transformer. in this paper, under the transformer architecture, we firstly provide a way to define the source and target contexts and then obtain our model by combining both source and target contexts with context gates, which actually induces a probabilistic model indicating whether the next generated word is contributed from the source or target sentence @xcite . in our preliminary experiments, this model only achieves modest gains over transformer because the context selection error reduction is very limited as described in section 3.3. to further address this issue, we propose a probabilistic model whose loss function is derived from external supervision as regularization for the context gates. this probabilistic model is jointly trained with the context gates in nmt. as it is too costly to annotate this supervision for a large-scale training corpus manually, we instead propose a simple yet effective method to automatically generate supervision using pointwise mutual information, inspired by word collocation @xcite . in this way, the resulting nmt model is capable of controlling the contributions from source and target contexts effectively. we conduct extensive experiments on 4 benchmark datasets, and experimental results demonstrate that the proposed gated model obtains an averaged improvement of 1.0 bleu point over corresponding strong transformer baselines. in addition, we design a novel analysis to show that the improvement of translation performance is indeed caused by relieving the problem of wrongly focusing on the source or target context. ## methodology given a source sentence @xmath0, • • • , x |x| and a target sentence @xmath1, • • • , y |y| , our proposed model is defined by the following conditional probability under the transformer architecture: foot_0 ## experiments the proposed methods are evaluated on nist zh⇒en foot_2 , wmt14 en⇒de foot_3 , iwslt14 de⇒en foot_4 and iwslt17 fr⇒en foot_5 tasks. to make our nmt models capable of open-vocabulary translation, all datasets are preprocessed with byte pair encoding @xcite . all proposed methods are implemented on top of transformer @xcite which is the state-of-the-art nmt system. case-insensitive bleu score @xcite is used to evaluate translation quality of zh⇒en, de⇒en and fr⇒en. for the fair comparison with the related work, en⇒de is evaluated with case-sensitive bleu score. setup details are described in appendix a. ## conclusions this paper transplants context gates from the rnn based nmt to the transformer to control the source and target context for translation. we find @xcite [30 , 40) [40 , 50) [50 , 60) [60 , 130 that context gates only modestly improve the translation quality of the transformer, because learning context gates freely from scratch is more challenging for the transformer with the complicated structure than for rnn. based on this observation, we propose a regularization method to guide the learning of context gates with an effective way to generate supervision from training data. experimental results show the regularized context gates can significantly improve translation performances over different translation tasks even though the context control problem is only slightly relieved. in the future, we believe more work on alleviating context control problem has the potential to improve translation performance as quantified in