Improving Neural Machine Translation with Soft Template Prediction (2020)

Abstract: Although neural machine translation (NMT) has achieved significant progress in recent years, most previous NMT models only depend on the source text to generate translation. Inspired by the success of template-based and syntax-based approaches in other fields, we propose to use extracted templates from tree structures as soft target templates to guide the translation procedure. In order to learn the syntactic structure of the target sentences, we adopt constituency-based parse tree to generate candidate templates. We incorporate the template information into the encoder-decoder framework to jointly utilize the templates and source text. Experiments show that our model significantly outperforms the baseline models on four benchmarks and demonstrates the effectiveness of soft target templates.

## introduction recently, neural machine translation (nmt) @xcite @xcite has achieved significant progress. some advanced models @xcite @xcite @xcite predict the ultimate translation by multi-pass generation conditioned on the previous text such as cmlms @xcite , abd-nmt @xcite , synst @xcite , and deliberation network @xcite . inspired by these works and the successful application of templates for other intriguing tasks, including semantic parsing @xcite , summarization @xcite , question answering @xcite ; s denotes subject and vp denotes verb phrase. @xcite , and other text generation tasks @xcite , we assume the candidate templates of the target sentences can guide the sentence translation process. we denote these templates extracted from the constituencybased parse tree as soft templates, which consist of tags and target words. the templates are soft because no explicit paradigms are inaugurated to build new translation from them, and the target tokens could be modified. in order to effectively use the templates, we introduce soft template-based neural machine translation (st-nmt), which can use source text and soft templates to predict the final translation. our approach can be split into two phases. in the first phase, a standard transformer model is trained to predict soft target templates by using source text and templates extracted from the constituencybased parse tree. secondly, we use two encoders, including a soft target template encoder and a source language encoder to encode source text and templates and generate the final translation. as shown in figure 1 , given the source text "我喜欢 打篮球" and the target template "s like to vp", the final translation "i like to play basketball" is generated by two encoders. in this work, the templates play a part in guiding, and some target tokens in self attention add & norm add & norm cross attention add & norm feed forward self attention add & norm add & norm feed forward self attention add & norm add & norm feed forward position encoding position encoding position encoding linear & softmax template encoder source encoder source embedding template embedding target embedding x 1 x 4 x 2 x 3 x 5 𝑡 1 𝑡 4 𝑦 2 ′ t 3 𝑡 5 𝑦 1 𝑦 2 𝑦 3 𝑦 5 𝑦 4 target decoder n× n× n× figure 2: overview of our st-nmt. given the source text and the soft target template predicted by the p θ @xmath0, the source language transformer encoder and target template transformer encoder maps two sequences @xmath1, x 2 , x 3 , x 4 , x 5 ) and @xmath2, y 2 , t 3 , t 4 , t 5 ) into hidden states z x and z t . x i denotes the source word, t i denotes the template tag and y i denotes the target word. y i also denotes the target word but it can be modified to the other target words. the ultimate translation y is generated by a transformer decoder which incorporates the context z x and z y in the second phase. in order to prove the effectiveness of our approach, we conduct main experiments on the popular benchmarks, including iwslt14 german-english translation task, wmt14 english-german translation task, ldc chinese-english translation task, and aspec japanese-chinese translation task. experiments show that our approach achieves significant improvement compared to the baselines, which demonstrates the soft target templates can provide a positive influence for guiding translation procedure effectively. our approach can be used for diverse scale data sets, different styles, and multiple language pairs. ## our approach our model first reads the source language sequence @xmath3, x 2 , x 3 , . . . , x n ) in the conventional way by a source language transformer encoder and generates the template sequence @xmath4, t 2 , t 3 , . . . , t m ) by a template transformer decoder. as shown in figure 2 , our model uses a source language transformer encoder and a template transformer encoder, which encodes the source language sequence x and the template sequence t separately. we deploy the target language decoder to generate the final translation. in this section, we present the details of the proposed template-based approach. our method mainly in-cludes two phases: (1) the training data is constructed by the constituency-based parse tree. then, we adopt a standard transformer to convert the source text to the soft target template for the next generation. (2) based on the source text and the predicted soft target template, we utilize two encoders to encode two sequences into hidden states separately and a target language decoder to generate the ultimate translation. ## experiments we conducted experiments on four benchmarks, including ldc chinese-english, wmt14 english-german, iwslt14 german-english, and aspec japanese-chinese translation tasks. by conducting experiments on these four benchmarks, these settings prove that our approach is suitable for diverse situations: (1) these four benchmarks provide a wide coverage of both scale and genres. they vary from small scale to large scale (2) we use the different domains, which include news, science, and talk domain. (3) we also conduct the experiments on different language pairs, including the german-english translation task, the english-german translation task, the chinese-english translation task, and the japanese-chinese translation task. ## related work many types of encoder-decoder architecture @xcite @xcite @xcite have been proposed in the past few years. furthermore, transformer enhances the capability of nmt in capturing long-distance dependencies based on these backbone models, including cnn-based, rnn-based, and transformer based architecture. to improve the quality of the translation, many authors have endeavored to adopt multi-pass generation decoding method, their models first predict the rough translation and then generate the final translation based on the previous draft @xcite @xcite @xcite . besides, some works @xcite @xcite use the right-toleft (r2l) and left-to-right (l2r) to improve the quality of machine translation. non-autoregressive decoding @xcite first predicts the target tokens and masked tokens, which will be filled in the next iterations. then, the model predicts the unmasked tokens on top of the source text and a mixed translation consisting of the masked and unmasked tokens. semi-autoregressive also @xcite predicts chunked fragments or the unmasked tokens based on the tree structure before the final translation. in addition, there are many existing works @xcite @xcite @xcite @xcite which incorporate syntax information or the tree structure into nmt to improve the quality of translation results. ## conclusion in this work, we propose a novel approach that utilizes source text and additional soft templates. more specifically, our approach can extract the templates from the sub-tree, which derives from the specific depth of the constituency-based parse tree. then, we use a transformer model to predict the soft target templates conditioned on the source text. on top of soft templates and source text, we incorporate the template information to guide the translation procedure. we compare our soft-template neural machine translation (st-nmt) with other baselines on four benchmarks and multiple language pairs. experimental results show that our st-nmt significantly improves performance on these datasets.