Feature Decay Algorithms for Neural Machine Translation (2018)

Abstract: Neural Machine Translation (NMT) systems require a lot of data to be competitive. For this reason, data selection techniques are used only for finetuning systems that have been trained with larger amounts of data. In this work we aim to use Feature Decay Algorithms (FDA) data selection techniques not only to fine-tune a system but also to build a complete system with less data. Our findings reveal that it is possible to find a subset of sentence pairs, that outperforms by 1.11 BLEU points the full training corpus, when used for training a German-English NMT system .

## introduction in statistical machine translation (smt) it has been shown that having more data does not always lead to better results @xcite . in fact, performance can increase by limiting the training data to a smaller but more relevant set @xcite . neural machine translation (nmt) c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. models in contrast are data-hungry, and perform better only with large amounts of available training data, in some cases underperforming smt when low amounts of data are available @xcite . however, the amount of training data required to make nmt work really well depends a lot on the domain of the training data and test set, and possibly also how similar they are. for certain training domains such as ted talks @xcite it has already been shown that even with fairly limited training sizes nmt can already outperform smt by a large margin. larger training sets also introduce noise and require models to cover a larger number of words, whereas for practical reasons the vocabulary cannot be arbitrarily increased to facilitate these extra words. consequently, training material that is not relevant for the test set risks wasting limited entries in the vocabulary on source words that are not relevant to the test domain. this is why domain adaptation has proven to be useful in nmt @xcite by tailoring a model towards in-domain data. while traditional machine translation (mt) approaches perform an inductive learning (i.e. learn a model from translated sentences in order to predict unseen examples), transductive learning approaches aim to identify the best training instance to predict the test set @xcite . models trained with sentences retrieved by transductive learning methods are tailored towards the test set. this is similar to the way many domain-adaptation methods adapt the training material to be suitable for a specific domain. in this work, we use feature decay algorithms (fda), a transductive data selection method that has achieved good results in smt and apply it to nmt. our purpose is twofold: i) to question the widely held assumption that in neural machine translation it is always better to use more data; and ii) to explore how a transductive data-selection technique like fda should be applied in order to build models that outperform those built with all training data. ## related work feature decay algorithms in our work, in order to extract a subset of the data, we use feature decay algorithms @xcite @xcite . this is a method that uses the source side of the test set to select sentences that provide translation examples that are most relevant for this set. furthermore, fda aims to maximize the variability of these selected relevant n-grams in the training set by decreasing their value as they are being selected. in order to do that, the features (n-grams extracted from the test set) are assigned an initial value, and each sentence of the training data is scored as the normalized (by dividing by the number of words) sum of the values of its ngrams. then, the method iteratively selects the sentence with the highest score and adds it to the set of selected data (which initially is empty). after selecting a sentence, the values of the features contained in it are decreased ac-cording to the decay function. by default, the value of a feature f is defined as in (1): the score of a sentence s at a particular iteration is the sum of the values of c l (f ) of the features present in s, normalized by the length of s. the score of a sentence, using the default configuration in equation (1), computed as in (2): fda has proven to be useful in statistical machine translation (smt) @xcite ; @xcite . selecting a small subset of sentences from a parallel corpus using fda is enough to train smt systems that perform better than systems trained using the whole parallel corpus. ## data selection using the source-side of the test set using the source side of test examples is central to machine translation. for example, smt effectively uses only those phrase-pairs that match the source side of a test sentence. matching can be done implicitly, inside the decoder and during translation, or explicitly, by filtering the phrase-table with the source-side of the test set before passing it to the decoder. the usage of the test set source side by fda is conceptually not different from wellestablished lazy supervised learning meth-ods such as k-nearest neighbors, and is also not fundamentally different from the usage of source information for matching phrase-pair selection by smt grammar extractors/decoders @xcite . ## research questions due to the good performance achieved by fda in smt, we want to explore whether the improvements also maintain in nmt. accordingly, the first question we want to answer is: in nmt there are several possible configurations for applying a data-selection techniques. one method is to build a complete model from scratch using just a subset of the data. another way is to use fine-tuning to specialize an existing model. on the top of that, there are several possibilities of how to tune a model: (i) performing fine-tuning (and even in this option, there are several possibilities as we can choose different epochs of the model to tune); and (ii) perform a gradual tuning, where at each epoch the model is trained in using gradually smaller in-domain subsets due to the different configurations available, our second research question is: the test set may not always be accessible when building the nmt model. however, a system tuned for a document of one domain using fda may be be useful for translating a different one if they share the domain. in this work we have constructed a german-to-english nmt system using the pytorch port foot_0 of opennmt @xcite to train the models. according to the creators of open-nmt foot_1 a good baseline for german-to-english wmt 2015 data is the one built with default parameters (2-layer lstm with 500 hidden units, vocabulary size of 50002 and 50004 for source and target language, respectively) executed for 13 epochs. the words in the output that are not in the vocabulary are replaced with the word in the source with the highest attention. the data sets used in the experiments are based on the ones used in the work of @xcite : we extract subset of different sizes (100k, 200k, 500k, 1m and 2m sentences) from the training data with fda using the test set from the wmt 2015 translation task. we use the default configuration of fda (i.e. 3-grams as features, 0.5 as decay factor and 0 as decay exponent of 0). we perform several experiments building different nmt models using the training data and the data extracted with fda. in order to answer the research questions in section 4, models are built following different configurations: in addition, we are interested in exploring whether the model trained on data retrieved by fda using one document could also be useful for translating different documents in the same domain. we use the same models (trained with data using the test set of wmt 2015 as seed) for translating a different test set, the namely wmt 2014 @xcite news test set, which is in the same domain. ## results in we have seen that models trained with a subset of data perform better than those trained with all the data. as models built from scratch are not required to extract the words from the whole training data but only from the subset of sentences pairs relevant to the test set source, these are able to focus the limited vocabulary space more on those words that are relevant for the test set source. tuning approaches in contrast preserve the initial vocabulary, which means they do not benefit from the more focused vocabulary training from scratch using fda allows, which is one of the principles behind the working of fda. ## conclusion and future work in this work we have discovered that using fda, it is possible to find a subset of data that can be used to train an nmt model that achieves better results than a model trained with all data. in particular, our best model, trained on 44.6% of the data improves over the baseline trained on the full training set, while also giving significant improvements on other metrics. besides the significant improvement in translation quality, this also implies (in the chosen training regime, with 13 epochs and fda after 8 epochs) a linear reduction in training time compared to the baseline system. for example, by reducing the training data by half for the last 8 epochs we use only 81% of the original training time 4 . in future work, we want to study the impact of the differences in vocabulary in each experiment. we also want to compare these results to different data-selection techniques or different variants of fda (either using different values in the parameters, or different variants of the algorithm such us the one proposed in @xcite ). marie skłodowska-curie grant agreement no 713567.