Guided Alignment Training for Topic-Aware Neural Machine Translation (2016)

Abstract: In this paper, we propose an effective way for biasing the attention mechanism of a sequence-to-sequence neural machine translation (NMT) model towards the well-studied statistical word alignment models. We show that our novel guided alignment training approach improves translation quality on real-life e-commerce texts consisting of product titles and descriptions, overcoming the problems posed by many unknown words and a large type/token ratio. We also show that meta-data associated with input texts such as topic or category information can significantly improve translation quality when used as an additional signal to the decoder part of the network. With both novel features, the BLEU score of the NMT system on a product title set improves from 18.6 to 21.3%. Even larger MT quality gains are obtained through domain adaptation of a general domain NMT system to e-commerce data. The developed NMT system also performs well on the IWSLT speech translation task, where an ensemble of four variant systems outperforms the phrase-based baseline by 2.1% BLEU absolute.

## introduction nmt systems were shown to reach state-of-the-art translation quality on tasks established in mt research community such as iwslt speech translation task @xcite . in this paper, we also apply nmt approach to e-commerce data: user-generated product titles and descriptions for items put on sale. such data are very different from newswire and other texts typically considered in the mt research community. titles in particular are short (usually fewer than 15 words), contain many brand names which often do not have to be translated, but also product feature values and specific abbreviations and jargon. also, the vocabulary size is very large due to the large variety of product types, and many words are observed in the training data only once. at the same time, these data are provided with additional meta-information about the item (e.g. product category such as clothing or electronics), which can be used as context to perform topic/domain adaptation for improved translation quality. at first glance, established phrase-based statistical mt approaches are well-suited for ecommerce data translation. in a phrase-based approach, singleton, but unambiguous words and phrases are usually translated correctly. also, since the alignment between source and target words is available, it is possible to transfer certain entities from the source sentence to the generated target sentence "in-context" without translating them. such entities can include numbers, product specifications such as "5s" or brand names such as "samsung" or "lenovo". in training, these entities can be replaced with placeholders to reduce the vocabulary size. however, nmt approaches are more powerful at capturing context beyond phrase boundaries and were shown to better exploit available training data. they also successfully adapt themselves to a domain, for which only a limited amount of parallel training data is available @xcite . also, previous research @xcite has shown that it is difficult to obtain translation quality improvements with topic adaptation in phrase-based smt because of data sparseness and a large number of topics (e. g. corresponding to product categories), which may or may not be relevant for disambiguating between alternative translations or solving other known mt problems. in contrast, we expected nmt to better solve the topic adaptation problem by using the additional meta-information as an extra signal in the neural network. to the best of our knowledge, this is the first work where the additional information about the text topic is embedded into the vector space and used to directly influence nmt decisions. in an nmt system, the attention mechanism introduced in @xcite is important both for decoding as well as for restoration of placeholder content and insertion of unknown words in the right positions in the target sentence. to improve the estimation of the soft alignment, we propose to use the viterbi alignments of the ibm model 4 @xcite as an additional source of knowledge during nmt training. the additional alignment information helps the current system to bias the attention mechanism towards the viterbi alignment. this paper is structured as follows. after an overview of related nmt work in section 2, we propose a novel approach in section 3 on using statistical word alignemt to bias the training of neural mt attention mechanism, we call it guided alignment training. in section 4, we describe in more detail how topic information can benefit nmt. section 5 and section 6 describes our domain adaptation approach. experimental results are presented in section 7. the paper is concluded with a discussion and outlook in section 8. ## related work neural machine translation is mainly based on using recurrent neural networks to grasp long term dependencies in natural language. an nmt system is trained on end-to-end basis to maximize the conditional probability of a correct translation given a source sentence @xcite , @xcite , @xcite . when using attention mechanism, large vocabularies @xcite , and some other techniques, nmt is reported to achieve comparable translation quality to state-of-art phrase-based translation systems. most nmt approaches are based on the encoder-decoder architecture @xcite , in which the input sentence is first encoded into a fixed-length representation, from which the recurrent neural network decoder generates the sequence of target words. since fixed-length representation cannot give enough information for decoding, a more sophisticated approach using attention mechanism is proposed by @xcite . in this approach, the neural network learns to attend to different parts of source sentence to improve translation quality. since the source and target language vocabularies for a neural network have to be limited, the rare words problem deteriorates translation quality significantly. the rare word replacement technique using soft alignment proposed by @xcite gives a promising solution for the problem. both encoder-decoder architecture and insertion of unknown words into nmt output highly rely on the quality of the attention mechanism, thus it becomes the crucial part of nmt. some research has been done to refine it by @xcite , who proposed global and local attention-based models, and @xcite , who used biases, fertility and symmetric bilingual structure to improve the attention model mechanism. the most recent work done by @xcite is highly parallel with our guided alignment training, section 3. they use statistical alignment to supervise the nmt in a similar fashion as we do, the difference is that they smooth the statistical alignment and apply euclidean distance directly to the objective function, while we try with different divergence function and also re-weight it before adding to the overall objective function. research on topic adaptation most closely related to our work was performed by @xcite , but the features proposed there were added to the log-linear model of a phrasebased system. here, we use the topic information as part of the input to the nmt system. another difference is that we primarily work with human-labeled topics, whereas in @xcite the topic distribution is inferred automatically from data. when translating e-commerce content, we are faced with a situation when only a few product titles and descriptions were manually translated, resulting in a small in-domain parallel corpus, but a large general-domain parallel corpus is available. in such situations, domain adaption techniques have been used both in phrase-based systems @xcite and @xcite . in addition, while diverse nmt models using different features and techniques are trained, an ensemble decoder can be used to combine them together to make a more robust model. this approach was used by @xcite to outperform the state-of-art phrase-based system with their nmt approach in the wmt 2015 evaluation. ## guided alignment training when using the attention-based @xcite , we observed that the attention mechanism sometimes fails to yield appropriate soft alignments, especially with increasing length of the input sentence and many out-of-vocabulary words or placeholders. in translation, this can lead to disordered output and word repetition. in contrast to a statistical phrase-based system, the nmt decoder does not have explicit information about the candidates of the current word, so at each recurrent step, the attention weights only rely on the previously generated word and decoder/encoder state, as depicted in figure 1 . the target word itself is not used to compute its attention weights. if the previous word is an out-of-vocabulary (oov) or a placeholder, then the information it provides for calculating the attention weights for the current word is neither sufficient nor reliable anymore. this leads to incorrect target word prediction, and the error propagates to the future steps due to feedback loop. the problem is even larger in the case of e-commerce data where the number of oovs and placeholders is considerably higher. to improve the estimation of the soft alignment, we propose to use the viterbi alignments of the ibm model 4 as an additional source of knowledge during the nmt training. therefore, we first extract viterbi alignments using giza++ toolkit @xcite , then we use them to bias the attention mechanism. our approach is to optimize on both the decoder cost and the divergence between the attention weights and the alignment connections generated by statistical alignments. the multi-objective optimization task is then expressed as a single-objective function, which is a linear combination of two loss functions: original and new guided-alignment. ## topic-aware machine translation in the e-commerce domain, the information on the product category (e.g., "mens' clothing", "mobile phones", "kitchen appliances") often accompanies the product title and description and can be used as an additional source of information both in the training of a mt system and during translation. in particular, such meta-information can help to disambiguate between alternative translations of the same word that have different meaning. the choice of the right translation often depends on the category. for example, the word "skin" has to be translated differently in the categories "mobile phone accessories" and "make-up". outside of the ecommerce world, similar topic information is available in the form of e.g. tags and keywords for a given document (on-line article, blog post, patent, etc.) and can also be used for word sense disambiguation and topic adaptation. in general, a document may belong to multiple topics. here, we propose to feed such meta-information into the recurrent neural network to help generate words which are appropriate given a particular category or topic. ## bootstrapping when trained on small amounts of data, the attention-based neural network approach does not always produce reliable soft alignment. the problem gets worse when the sentence pairs available for training are getting longer. to solve this problem, we extracted bilingual sub-sentence units from existing sentence pairs to be used as additional training data. these units are exclusively aligned to each other, i. e. all words within the source sub-sentence are aligned only to the words within the corresponding target sub-sentence and vice versa. the alignment is determined with the standard approach (ibm model 4 alignment trained with the giza++ toolkit @xcite ). as boundaries for sub-sentence units, we used punctuation marks, including period, comma, semicolon, colon, dash, etc. to simplify bilingual sentence splitting, we used the standard phrase pair extraction algorithm for phrase-based smt, but set the minimum/maximum source phrase length to 8 and 30 tokens, respectively. from all such long phrase pairs extracted by the algorithm, we only kept those which are started or ended with a punctuation mark or started/ended a sentence; both on the source and on the target side. for the bootstrapped training, we merged the original training data with the extracted subsentence units and ran the neural training algorithm on this extended training set. since the extracted bilingual sub-sentence units generally showed good correspondence between source and target due to the constraints described above, the expectation was that having such units repeated in the training data as stand-alone training instances would guide the attention mechanism to become more robust and make it easier for the neural training algorithm to find better correspondences between more difficult source/target sentence parts. also, having both short and long training instances was expected to make neural translation quality less dependent on the input length. ## e-commerce domain adaptation for the e-commerce english-to-french translation task, we only have a limited amount of indomain parallel training data (item titles and descriptions). to benefit from large amounts of general-domain training data, we follow the method described by @xcite . we first train a baseline nmt model on english-french wmt data (common-crawl, europarl v7, and news commentary corpora) for two epochs to get the best result on a development set, and then we continue training the same model on the in-domain training set for a few more epochs. in contrast to @xcite , however, we use the vocabularies of the most frequent 52k source/target words in the in-domain data (instead of the out-of-domain data vocabularies). this causes nmt to focus on translation of the most relevant in-domain words. ## conclusion we have presented a novel guided alignment training for a nmt model that utilizes ibm model 4 viterbi alignments to guide the attention mechanism. this approach was shown experimentally to bring consistent improvements of translation quality on e-commerce and spoken language translation tasks. also on both tasks, the proposed novel way of utilizing topic metainformation in nmt was shown to improve bleu and ter scores. we also showed improvements when using domain adaptation by continuing training of an out-of-domain nmt system on in-domain parallel data. in the future, we would like to investigate how to effectively make use of the abundant monolingual data with human-labeled product category information that we have available for the envisioned e-commerce application.