Improving Neural Machine Translation on resource-limited pairs using auxiliary data of a third language (2016)

Abstract: In the recent years interest in Deep Neural Networks (DNN) has grown in the field of Natural Language Processing, as new training methods have been proposed. The usage of DNN has achieved state-of-the-art performance in various areas. Neural Machine Translation (NMT) described by Bahdanau et al. (2014) and its successive variations have shown promising results. DNN, however, tend to over-fit on small data-sets, which makes this method impracticable for resource-limited language pairs. This article combines three different ideas (splitting words into smaller units, using an extra dataset of a related language pair and using monolingual data) for improving the performance of NMT models on language pairs with limited data. Our experiments show that, in some cases, our proposed approach to subword-units performs better than BPE (Byte pair encoding) and that auxiliary language-pairs and monolingual data can help improve the performance of languages with limited resources.

## introduction in the recent years interest in deep neural networks (dnn) has grown in the field of natural language processing (nlp), as new training methods @xcite have been proposed. the encoder-decoder approach for neural machine translation (nmt) consists in encoding the source sentence into an intermediate vector representation and then generating (decoding) the target sentence from this representation. @xcite is an example of this approach. the nmt approach of jointly training alignment and translation models described by @xcite and its successive variations have shown promising results. its attention mechanism deals with the problem of having a fixed length vector for sentences of varying length by encoding the source sentence into a set of vectors, one vector for each of the tokens in the source sentence. nmt doesn't need complex feature engineering, which is convenient when dealing with resource-limited languages. however, a large parallel corpus is still needed in order to get competitive performance and avoid overfitting. as an example, @xcite and @xcite use a dataset of about 12 million parallel sentences. two main problems arise when using a small dataset for training a mt model. one of those problems is that the vocabulary exposed by a small dataset is inherently small. also, even if a word shows up in the data it may occur too few times for learning a reliable representation. one strategy for minimizing this problem is subdividing words into subword units, like syllables. doing so reduces the total vocabulary size and increases the hit-rate of each symbol in the dataset. this has been explored in @xcite . another problem concerns large nmt models. when the number of parameters is too large compared to the data size, the model may optimize for memorizing all or a large part of the dataset instead of modeling the translation; overfitting in practice. a solution to this problem is to reduce the model size to better match the amount of data. however, many relevant features may not be modeled with a smaller number of parameters. another approach is to artificially increase the number of samples by counterfeiting or using data of a third language. this research explores how much of an improvement using auxiliary parallel sentences from a third language to the target language (@xmath0. we also explore the effect of using an auxiliary language on the decoder side. for the case of phrase-based statistical machine translation, similar ideas have been explored before with varied results. for example, for closely related pairs @xcite or through lexical triangulation @xcite . for nmt, a couple of authors have also explored this possibility. @xcite modeled translation to different targets from a common source with shared representation. @xcite also explored the case of a common target language for different source languages. both papers claimed to get higher translation quality over individually trained models. a comparison to these papers follows in section 2. in order to assist the learning of the target language pair, mt for the auxiliary pair is trained jointly. we argue that doing so prevents the language pair with the small dataset from overfitting and leads to more robust models. this problem could also be addressed through transfer learning, as explored in @xcite and @xcite , but that approach falls outside the scope of this article. a third solution to parallel data scarcity is using monolingual data in addition in order to make the language model (lm) at the target side stronger. a strong lm at the decoder can increase performance, as already tried by @xcite . we perform experiments that are analogous to the ones described in @xcite with focus on the more resource-limited pairs and use their results as a baseline for comparison. this article has the following sections: section 2 summarizes previous related work; in section 3, the proposed approach is described; section 4 presents some experiments with their results and analysis; finally, in section 5, we draw some conclusions. ## proposed solution the method proposed here consists in jointly training various models with shared parameters; either the encoder or the decoder parameters. by jointly training the models they will co-adapt and benefit from each other. in the experiments we only explore the possibility of using one extra auxiliary language, either as the source language or as the target language, in addition to the intended source and target languages. more than one extra language could be used in a similar fashion. in addition to using parallel data from an auxiliary language we also experiment with using monolingual data to obtain improved results. we used subword units instead of words as the translation unit as a method to deal with large vocabulary sizes and differences in morpheme-per-word ratios (synthetic vs isolating). ## conclusion we evaluated three different ideas that could help improve nmt for language pairs with limited resources. we trained three models applying from one to the three of the ideas on four different datasets and assessed them by measuring their bleu scores on the same test-set. the first idea was to use subword-units instead of words. the subword-units we applied were intended to map better to single morphemes when compared to other methods as bpe. our experiments showed that these kind of subword-units can help with smaller datasets, getting a +1.24 bleu score improvement when compared to bpe for german-english translation when trained on a 210k sentence pair dataset. using data form an auxiliary language-pair helped improve the performance for small datasets (about 200k parallel sentences) but when the dataset was too small (100k parallel sentences) and the auxiliary language was in the decoder side the model ended memorizing the small dataset despite the extra data. even though the extra data improved the performance the solution by @xcite got better results for german-english translation. the monolingual data helped prevent overfitting in the cases when the dataset in the decoder side was too small. the monolingual data didn't help very much with german-english translation. our proposed solution increased considerably the time needed for each iteration and thus the time for convergence. in the future, we would like to rethink the subword-unit approach to represent better the consonant-gradation and other small sound changes related to morpheme combination. also, faster gpus may make our solution more feasible in the future. the prime minister of india and japan in tokyo in tokyo . +aux 210k the prime ministers of india and japan met came in tokyo . +aux+mono 210k the prime minister of india and japan came to tokyo . subword 420k prime minister india and japan met in tokio . +aux 420k the prime minister of india and japan joined in tokyo . +aux+mono 420k the prime minister , india and japan , met in tokyo .