Data selection for NMT using Infrequent n-gram Recovery (2018)

Abstract: Neural Machine Translation (NMT) has achieved promising results comparable with Phrase-Based Statistical Machine Translation (PBSMT). However, to train a neural translation engine, much more powerful machines are required than those required to develop translation engines based on PBSMT. One solution to reduce the training cost of NMT systems is the reduction of the training corpus through data selection (DS) techniques. There are many DS techniques applied in PBSMT which bring good results. In this work, we show that the data selection technique based on infrequent n-gram occurrence described in (Gasco ́ et al., 2012) commonly used for PBSMT systems also works well for NMT systems. We focus our work on selecting data according to specific corpora using the previously mentioned technique. The specific-domain corpora used for our experiments are IT domain and medical domain. The DS technique significantly reduces the execution time required to train the model between 87% and 93%. Also, it improves translation quality by up to 2.8 BLEU points. The improvements are obtained with just a small fraction of the data that accounts for between 6% and 20% of the total data.

## introduction until recently, machine translation (mt) systems were based mostly on pbsmt. today, the state of the art of mt is nmt. it has been shown that neural networks can improve the quality of translations by up to several bleu points and also make them more fluid @xcite . however, nmt is computationally much more expensive. to train an nmt engine, much more powerful machines are required than would be used for building translation engines based on pbsmt. for example, nmt engines require more ram memory, one or several gpus and storing the models requires more storage capacity. also, the training time of an nmt system is significantly longer than that of the systems based on pbsmt @xcite . one solution to reduce the training cost of nmt systems is the reduction of the training corpus through ds techniques. bilingual sentence selection (bss) is a type of ds where the best subset of bilingual sentences from the available parallel corpora is selected and leveraged to train a translation system. to date, many ds techniques are known that are applied to pbsmt systems, bringing very promising results. some of them not only reduce the training time but also outperform a system where all the bilingual data available is used, given that the selected sentences are better suited to the domain being dealt with. in this work, we demonstrate that a ds technique commonly used for pbsmt can also yield satisfactory results when applied in nmt systems. to prove a good performance of ds in nmt we select sentences from a large amount of data from different domains with the purpose of enlarging a small size, in-domain training corpus. the selection of more suitable sentences achieves improvements in translation quality. ## related work when creating a machine translation system, it is important to select high-quality bilingual data with a domain similar to the one in which the translation system will be used. there are multiple techniques of ds for pbsmt based on perplexity as @xcite , where the authors use maximum-likelihood based methods to select the lexicon, segment words, filter and adapt the training data, and reduce language model size. in @xcite , data selection is done comparing the cross-entropy according to domain-specific and non-domain specific. in @xcite , sentences are selected with a bilingual cross-entropy based method. the selected subset is used to train a small domain-adapted pbsmt system. this domain-adapted system is combined with the real in-domain pbsmt system. also, there are techniques based on distributed representations of words. in @xcite and @xcite , sentences are selected using a convolutional neural network. in @xcite , a continuous vector-space representation of word sequences is used for selecting the best subset of a bilingual corpus. in @xcite , a new data selection method is developed, based on a neural network classifier. other data selection techniques rely on information retrieval based methods. in @xcite , training data is adapted by redistributing the weight of each training sentence pair. there are also ds techniques which select sentences relying on information from the development and test set. in @xcite two data selection techniques are presented: 1) probabilistic sampling, that introduces new sentences into the in-domain corpus without distorting the original distribution. first, the sentences are selected according to length, then according to probability. the second technique presented in that work is infrequent n-gram recovery. this technique relies on the idea of enforcing model coverage for those n-grams that are present in the (source) test set. in (bic ¸ici and yuret, 2011), the authors explore the use of a data selection in a transductive scenario. feature decay algorithms increase the diversity of the training set by devaluing features that are already included. all commented techniques were initially implemented for pbsmt systems. there are also some techniques designed explicitly for nmt systems. in @xcite , the authors present an instance-based adaptive nmt approach that effectively handles translation requests from multiple domains in an unsupervised manner, that is without knowing the domain labels. in @xcite , the method developed consists in selecting, from a large monolingual pool of sentences in the source language, those instances that are more related to a given test set. next, this selection is automatically translated and the general neural machine translation system is fine-tuned with this data. also, there are some works that compare the effectiveness of data selection techniques in pbsmt and nmt. in @xcite , the authors compare the effects of a commonly used data selection approach (bilingual cross entropy) on pbmt and nmt using four different domains. they also introduce dynamic data selection as a way to make data selection profitable for nmt. ## infrequent n-gram recovery the data selection technique used in this work is called infrequent n-gram recovery @xcite . the main use of this technique is when the in-domain corpus provided is too small to train properly the translation engine. this technique consists on enlarging the in-domain training set by selecting sentences from a non domain-specific pool of sentences to maximise the coverage of n-grams which appear in the test and development set. for this, it is necessary to establish the minimum number of occurrences (t) required for a certain n-gram to be considered as infrequent, and also the order n of the n-grams (unigrams, bigrams, 3-grams etc.) that will be considered. the selected sentences will contain n-grams considered infrequent. with that we ensure that the training set will contain all n-grams from test and development set t times, as long as this is possible with the available out of domain dataset. the pool of sentences will be oppositely denoted as the out-of-domain corpus. sentences in the out-of-domain pool are sorted by their infrequency score in order to select first the sentences which most improve the coverage of n-grams belonging to the in-domain dataset which might be considered infrequent. let χ be the set of n-grams that appear in the sentences to be translated and w one of them; c(w) the counts of w in the source language training set; t the threshold of counts when an n-gram is considered infrequent, and n (w) the counts of w in the source sentence f to be scored. the infrequency score of f is: it already was demonstrated that the infrequent n-gram recovery technique works very well in pbsmt systems improving up to 1 point of bleu when compared to training with all the data available (in-domain + out-of-domain), while using only 0.5% of total data. the fact, that the infrequent n-gram recovery technique works well in pbsmt system does not mean that it will work fine for nmt, since pbsmt and nmt build the translation model in very different ways. pbsmt splits sentences into smaller chunks and looks for similar occurrences in other languages according to a statistical model. the alignment matrix can not be well estimated if words and n-grams appear rarely in the training corpus. also, the out-of-vocabulary words can not be translated by pbsmt model. the behaviour of nmt systems is different to pbsmt. nmt generates sequence of words in the target language given an input sequence of words in the source language. the translation is done following an encoder-decoder architecture. the encoder represents the input sequence using a word embedding model @xcite , and the decoder generates the sentence in the target language word by word @xcite . in nmt, it is necessary to adjust hyper-parameters as learning rate, number of hidden layers, and number of epochs. nmt needs to deal with millions of parameters coming from each neural network unit (weights and biases) to adjust the translation model. the best model is then selected according to translation quality on the development set. up until now there is no study about the efficiency of infrequent n-gram recovery in nmt. ## experiments the experiments were conducted using the opennmt @xcite deep learning framework based in torch. this toolkit is mainly specialised in sequence-to-sequence models covering a variety of tasks such as machine translation, image to text, and speech recognition. all experiments were conducted using an nvidia gtx 1080 gpu with 8gb of ram. to select domain-specific sentences, we need a small size in-domain dataset and an out-of-domain dataset which contains sentences from different domains. then, we select sentences from the out-of-domain corpus to enlarge the in-domain corpus. ## conclusions pbmt and nmt estimate the translation model in a different way. pbmt estimates the parameters using statistical models and use word alignments to generate the translation. instead, nmt features an encoder-decoder architecture. the encoder represents a sequence of input words mapping them to vectors of real numbers and then the decoder generates the output sequence in a word-by-word basis. in our work, we show that infrequent n-gram recovery brings very satisfactory results when applied to nmt. we demonstrate that, by selecting a subset of data more suitable to a specific in-domain corpus, we can get a model whose quality can improve the quality of a model trained with all the data available (in-domain and out-of-domain data jointly). such was the case with the it corpus. in contrast, a less usual case is when the model trained with all data performs worse than one trained with only in-domain data. this was the case with the medical domain dataset. it can be due to very specific vocabulary appearing in the in-domain corpus, and such vocabulary not being frequent in the out-of-domain data. this entails that including sentences from different domains lead to worse translation quality. despite this fact, the technique described manages to select only sentences that lead to improvements over the translation quality achieved by a system trained only with in-domain data. in our experiments, we achieve improvements of up to 1.7 bleu points over a model trained with a random selection of data. in the case of the it corpus, we improved translation quality by about 2.8 points of bleu when compared to a model trained on all the data available. another important issue is the reduction of execution time. by reducing the amount of training data, we achieved a reduction in execution time between 87% and 93%. we understand that this reduction is very important in the case of nmt, since training an nmt system can take up to several weeks. we demonstrate that with adequate ds, we can reduce execution time from 11 days to 17 hours, while simultaneously improving the translation quality achieved by a model trained with all the data available.