In neural machine translation , a source sequence of words is encoded into a vector from which a target sequence is generated in the decoding phase . Differently from statistical machine translation , the associations between source words and their possible target counterparts are not explicitly stored . Source and target words are at the two ends of a long information processing procedure , mediated by hidden states at both the source encoding and the target decoding phases . This makes it possible that a source word is incorrectly translated into a target word that is not any of its admissible equivalent counterparts in the target language . Neural machine translation ( NMT ) is an endto-end approach to machine translation that has achieved competitive results vis-a-vis statistical machine translation ( SMT ) on various language pairs ( Bahdanau et al . , 2015 ; Cho et al . , 2014 ; Sutskever et al . , 2014 ; Luong and Manning , 2015 ) . In NMT , the sequence-to-sequence ( seq2seq ) model learns word embeddings for both source and target words synchronously . However , as illustrated in Figure 1 , source and target word embeddings are at the two ends of a long information processing procedure . The individual associations between them will gradually become loose due to the separation of source-side hidden states ( represented by h 1 , . . . , h T in Fig . 1 ) and a target- side hidden state ( represented by s t in Fig . 1 ) . As a result , in the absence of a more tight interaction between source and target word pairs , the seq2seq model in NMT produces tentative translations that contain incorrect alignments of source words with target counterparts that are non-admissible equivalents in any possible translation context . Differently from SMT , in NMT an attention model is adopted to help align output with input words . The attention model is based on the estimation of a probability distribution over all input words for each target word . Word alignments with attention weights can then be easily deduced from such distributions and support the translation . Nevertheless , sometimes one finds translations by NMT that contain surprisingly wrong word alignments , that would unlikely occur in SMT . For instance , Figure 2 shows two Chineseto-English translation examples by NMT . In the top example , the NMT seq2seq model incorrectly aligns the target side end of sentence mark eos to 下旬 / late with a high attention weight ( 0.80 in this example ) due to the failure of appropriately capturing the similarity , or the lack of it , between the source word 下旬 / late and the target eos . It is also worth noting that , as 本 / this and 月 / month end up not being translated in this example , inappropriate alignment of target side eos is likely the responsible factor for under translation in NMT as the decoding process ends once a target eos is generated . Statistics on our development data show that as much as 50 % of target side eos do not properly align to source side eos . The second example in Figure 2 shows another case where source words are translated into target items that are not their possible translations in that or in any other context . In particular , 冬奥 会 / winter olympics is incorrectly translated into a target comma " , " and 载誉 / honors into have . In this paper , to address the problem illustrated above , we seek to shorten the distance within the seq2seq NMT information processing procedure between source and target word embeddings . This is a method we term as bridging , and can be conceived as strengthening the focus of the attention mechanism into more translation-plausible source and target word alignments . In doing so , we hope that the seq2seq model is able to learn more appropriate word alignments between source and target words . We propose three simple yet effective strategies to bridge between word embeddings . The inspiring insight in all these three models is to move source word embeddings closer to target word embeddings along the seq2seq NMT information processing procedure . We categorize these strategies in terms of how close the source and target word embeddings are along that procedure , schematically depicted in Fig . 1 . ( 1 ) Source-side bridging model : Our first strategy for bridging , which we call source-side bridging , is to move source word embeddings just one step closer to the target end . Each source word embedding is concatenated with the respective source hidden state at the same position so that the attention model can more closely benefit from source word embeddings to produce word alignments . ( 2 ) Target-side bridging model : In a second more bold strategy , we seek to incorporate relevant source word embeddings more closely into the prediction of the next target hidden state . In particular , the most appropriate source words are selected according to their attention weights and they are made to more closely interact with target hidden states . ( 3 ) Direct bridging model : The third model consists of directly bridging between source and target word embeddings . The training objective is optimized towards minimizing the distance between target word embeddings and their most relevant source word embeddings , selected according to the attention model . Experiments on Chinese-English translation with extensive analysis demonstrate that directly bridging word embeddings at the two ends can produce better word alignments and thus achieve better translation . We have presented three models to bridge source and target word embeddings for NMT . The three models seek to shorten the distance between source and target word embeddings along the extensive information procedure in the encoderdecoder neural network . Experiments on Chinese to English translation shows that the proposed models can significantly improve the translation quality . Further in-depth analysis demonstrate that our models are able ( 1 ) to learn better word alignments than the baseline NMT , ( 2 ) to alleviate the notorious problems of over and under translation in NMT , and ( 3 ) to learn direct mappings between source and target words . In future work , we will explore further strategies to bridge the source and target side for sequence-to-sequence and tree-based NMT . Additionally , we also intend to apply these methods to other sequence-to-sequence tasks , including natural language conversation .

Challenge: Neural machine translation models do not explicitly keep associations between source and possible target counterparts but implicitly through latent representations causing wrong word alignments.
Approach: They propose simple methods to bridge the source and target word embeddings for neural machine translation models in three different levels
Outcome: The proposed methods that bridge source and target word embeddings significantly improve translation quality and also mitigate over and under translation.