Learning Source Phrase Representations for Neural Machine Translation (2020)

Abstract: The Transformer translation model (Vaswani et al., 2017) based on a multi-head attention mechanism can be computed effectively in parallel and has significantly pushed forward the performance of Neural Machine Translation (NMT). Though intuitively the attentional network can connect distant words via shorter network paths than RNNs, empirical analysis demonstrates that it still has difficulty in fully capturing long-distance dependencies (Tang et al., 2018). Considering that modeling phrases instead of words has significantly improved the Statistical Machine Translation (SMT) approach through the use of larger translation blocks (“phrases”) and its reordering ability, modeling NMT at phrase level is an intuitive proposal to help the model capture long-distance relationships. In this paper, we first propose an attentive phrase representation generation mechanism which is able to generate phrase representations from corresponding token representations. In addition, we incorporate the generated phrase representations into the Transformer translation model to enhance its ability to capture long-distance relationships. In our experiments, we obtain significant improvements on the WMT 14 English-German and English-French tasks on top of the strong Transformer baseline, which shows the effectiveness of our approach. Our approach helps Transformer Base models perform at the level of Transformer Big models, and even significantly better for long sentences, but with substantially fewer parameters and training steps. The fact that phrase representations help even in the big setting further supports our conjecture that they make a valuable contribution to long-distance relations.

## introduction nmt is a new approach to machine translation that has achieved great success in the last a few * corresponding author. years @xcite @xcite . compared to plain smt @xcite @xcite , a neural language model decoder @xcite is better at long-distance re-ordering, and attention mechanisms @xcite have been proven effective in modeling longdistance dependencies, while these two issues were both challenging for smt. the transformer @xcite , which has outperformed previous rnn/cnn based translation models @xcite , is based on multi-layer multi-head attention networks and can be trained in parallel very efficiently. though attentional networks can connect distant words via shorter network paths than rnns, empirical results show that its ability in capturing long-range dependencies does not significantly outperform rnns, and it is still a problem for the transformer to fully model long-distance dependencies @xcite . using phrases instead of words enables conventional smt to condition on a wider range of context, and results in better performance in reordering and modeling long-distance dependencies. it is intuitive to let the nmt model additionally condition on phrase level representations to capture long-distance dependencies better, but there are two main issues which prevent nmt from directly using phrases: instead of using phrases directly in nmt, in this work, we address the issues above with the following contributions: our approach empirically brings about significant and consistent improvements over the strong transformer model (both base and big settings). we conducted experiments on the wmt 14 english-german and english-french news translation task, and obtained +1.29 and +1.37 bleu improvements respectively on top of the strong transformer base baseline, which demonstrates the effectiveness of our approach. our approach helps transformer base models perform at the level of transformer big models, and even significantly better for long sentences, but with substantially fewer parameters and training steps. it also shows effectiveness with the transformer big setting. we also conducted length analysis with our approach, and the results show how our approach improves long-distance dependency capturing, which supports our conjecture that phrase representation sequences can help the model capture long-distance relations better. ## background and related work in this section, we first review previous work which utilizes phrases in recurrent sequence-to-sequence models, then give a brief introduction to the stronger transformer translation model that our work is based on. ## transformer with phrase representation for the segmentation of phrases, given that n-gram phrases are effective for tensor libraries, we first try to cut a token sequence into a phrase sequence with a fixed phrase length which varies with the sequence length. 2 we pad the last phrase in case it does not have sufficient tokens, thus we can transform the whole sequence into a tensor. the n-gram phrase segmentation is efficient and simple, and we suggest the drawbacks of such "casual" segmentation boundaries can be alleviated with self-attention computation across the whole sequence and the attention mechanism applied in the generation of phrase representation which values tokens differently to a large extent, given that 2 we implement this as: @xmath0, seql/6), 3), where ntok and seql stand for the number of tokens in each phrase and the length of a sentence respectively. algorithm 1 extracting phrases from a parse tree. input: a parse tree t , maximum tokens allowed in a phrase n; output: extracted phrase sequence s. append p to s; 17: end while 18: reverse s; 19: return s neural models have been proven good at learning competitively effective representations with gate or attention mechanism even without modeling linguistic structures @xcite @xcite . in our experiments we also explore phrases extracted from the stanford parser @xcite as as an alternative to our simple segmentation strategy. the maximum number of tokens allowed is consistent with the simple segmentation approach, and we try to use the tokens from the largest sub-tree that complies with the maximum token limitation or from several adjacent sub-trees of the same depth as a phrase for efficiency. our algorithm to extract phrases from parse trees is shown in algorithm 1. to efficiently parallelize parser-based phrases of various length in a batch of data, we pad short phrases to the same length of the longest phrases in the batch of sentences, thus a batch of sequences of phrases can be saved into a tensor. but significantly more "<@xmath1, and the model is slightly slower than the simple approach. ## experiments to compare with @xcite , we conducted our experiments on the wmt 14 english to german and english to french news translation tasks. ## conclusion considering that the strong transformer translation model still has difficulty in fully capturing long-distance dependencies @xcite , and that using a shorter phrase sequence (in addition to the original token sequence) is an intuitive approach to help the model capture long-distance features, in this paper, we first propose an attention mechanism to generate phrase representations by merging corresponding token representations. in addition, we incorporate the generated phrase representations into the transformer translation model to help it capture long-distance relationships. we obtained statistically significant improvements on the wmt 14 english-german and english-french tasks over the strong transformer baseline, which demonstrates the effectiveness of our approach. our further analysis shows that the transformer with phrase representation empirically improves its performance especially in long-distance dependency learning.