An Analysis of Source Context Dependency in Neural Machine Translation (2018)

Abstract: The encoder-decoder with attention model has become the state of the art for machine translation. However, more investigations are still needed to understand the internal mechanism of this end-to-end model. In this paper, we focus on how neural machine translation (NMT) models consider source information while decoding. We propose a numerical measurement of source context dependency in the NMT models and analyze the behaviors of the NMT decoder with this measurement under several circumstances. Experimental results show that this measurement is an appropriate estimate for source context dependency and consistent over different domains.

## introduction neural machine translation (nmt) with encoderdecoder structure and attention mechanism @xcite has achieved great success on several machine translation tasks. different from phrase based systems, neural machine translation is trained end-to-end and learns the alignment and translation jointly. at each decoding step, the alignment is predicted in the attention layer and represented as a distribution over words in a source sequence. then the source context information, which is an attention-weighted sum over encoder hidden states, is fed into the decoder for the prediction of the next word. c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. the decoder in a nmt model is similar to a recurrent neural network language model (rnnlm) @xcite , with additional input from the source side. it takes the previous hidden state, the previous predicted target word embedding, and source context information as inputs and produces a distribution over the next target words. this end-to-end approach can achieve state-ofthe-art performance on several machine translation tasks. joint training of the translation model and alignment gives a soft alignment between source side and target side. however, some of its flaws are observed under certain settings @xcite . one of the most common and important issue of neural machine translation is that it often generates fluent but inadequate translations especially under domain mismatch conditions. an example 1 is shown in figure 1 . here, the translation generated by the nmt models -while being fluent english -has no semantic connection to the source sentence. moreover, out-ofdomain models cause even more severe inadequacy. an intuitive explanation for this observation is that the nmt decoder lacks effective attention to the source information. because of the similarity between the nmt decoder and an rnnlm, it is possible that nmt models generate sentence based on its internal language model without properly taking advantage of the source information. while several researchers have explored the atten1 in this example, we choose a sentence that has been processed by byte pair encoding (bpe) @xcite and "@@" is used as a splitter token. the reason for this is that bpe has become a standard pre-processing step, which helps reducing the vocabulary size. long words in original text will be split into sub-word "phrases". it is also very interesting to investigate how sub-word prediction related to the source information. pérez-ortiz, sánchez-martínez, esplà-gomis, popović, rico, martins, van den bogaert, forcada (eds.) proceedings of the 21st annual conference of the european association for machine translation, p. 189-197 alacant, spain, may 2018. ## experimental setup we use the toolkit nematus @xcite for training and decoding. we use the gated recurrent unit (gru) @xcite in both encoder and decoder with a dimension of 1024. the dimension of embedding layer is 500. for optimizer, adadelta @xcite with learning rate 0.0001 is used. dropout @xcite with 0.2 probability was used to prevent overfitting. for decoding, we use beam search with a beam width 12. byte pair encoding (bpe) @xcite ) is used for processing training data to fit a 50,000 subwords vocabulary limit. we use bpe since it has been a very popular preprocessing procedure for machine translation, so that our evaluation method can be used in more general cases. in part-of-speech (pos) analysis, we use stanford pos tagger @xcite with a universal pos tagset. we first convert the translated subwords to complete words and tag the sequences with the stanford pos tagger we find that the amount of subwords is significantly smaller than complete words. so we let each subword inherit the tag from the corresponding complete word foot_0 . we carried out our experiments on german-english translation tasks. we used five corpora in five domains from opus @xcite , which is briefly described in ## conclusion and future work in this paper, we proposed a measurement of source context dependency in neural machine translation models. with our measurement, we analyzed source context dependency with different pos tags, domains and sentence lengths. from the analysis, we can see our measurement is a good estimation of source context dependency. in the future, we plan to extend our research in two directions. one is to investigate the relationship between source context dependency and word level translation quality, so that we can immediately detect when the system goes off track. the other is to improve the performance of nmt models. since our measurement is differentiable, we can use it as an auxiliary term of the training objective function.