Bilingual Methods for Adaptive Training Data Selection for Machine Translation (2016)

Abstract: In this paper, we propose a new data selection method which uses semi-supervised convolutional neural networks based on bitokens (Bi-SSCNNs) for training machine translation systems from a large bilingual corpus. In earlier work, we devised a data selection method based on semi-supervised convolutional neural networks (SSCNNs). The new method, Bi-SSCNN, is based on bitokens, which use bilingual information. When the new methods are tested on two translation tasks (Chinese-to-English and Arabic-to-English), they significantly outperform the other three data selection methods in the experiments. We also show that the BiSSCNN method is much more effective than other methods in preventing noisy sentence pairs from being chosen for training. More interestingly, this method only needs a tiny amount of in-domain data to train the selection model, which makes fine-grained topic-dependent translation adaptation possible. In the follow-up experiments, we find that neural machine translation (NMT) is more sensitive to noisy data than statistical machine translation (SMT). Therefore, Bi-SSCNN which can effectively screen out noisy sentence pairs, can benefit NMT much more than SMT.We observed a BLEU improvement over 3 points on an English-to-French WMT task when Bi-SSCNNs were used.

## introduction when building a statistical machine translation (smt) system, it is important to choose bilingual training data that are of high quality foot_0 and that are typical of the domain in which the smt system will operate. in previous work, these two goals of data selection, i.e., picking high-quality data and picking data that ensure the smt system is well-adapted to a given domain, have often been achieved separately. for instance, the papers @xcite @xcite @xcite focus on reducing the noise in the data. they use different scoring functions, such as language model perplexity, word alignment score, or ibm model 1 score, to score each sentence pair, top scored sentence pairs are selected. while the papers @xcite @xcite @xcite @xcite focus on domain adaptation. they all select monolingual or bilingual data that are similar to the in-domain data according to some criterion. these state-of-the-art adaptive data selection approaches @xcite @xcite search for bilingual parallel sentences using the difference in language model perplexity between two language models trained on in-domain and out-domain data, respectively. furthermore, @xcite extends these approaches from n-gram models to recurrent neural network language models @xcite . while some previous work considers achieving the two goals simultaneously, such as @xcite which uses ibm model 1 and a language model to do data selection, @xcite uses a neural network joint model to select the in-domain data. in a recent paper @xcite , we describe one type of neural network for carrying out data selection: a semi-supervised convolutional neural network (sscnn) that is trained on the in-domain set to score one side of each sentence in a general-domain bilingual corpus (either the source side or the target side) for its suitability as training material for an smt system. the highest-scoring sentence pairs are chosen to train the smt system. experiments described in that paper, covering three different types of test domain and four language directions, show that this sscnn method yields significantly higher bleu scores for the resulting smt system than for three state-of-the-art data selection methods when the amount of training data selected is held constant. the advantage of the sscnn over the earlier methods is especially dramatic when the amount of in-domain data used to train the selection model is small (less than 800 sentence pairs): the in-domain set for the sscnn can be as few as 100 sentences, which makes fine-grained topic-dependent translation adaptation possible. in some cases, the sscnn is so effective at selecting good training data that it is possible to greatly reduce the amount of training data for the smt system without negative impact on translation quality: this reduces the footprint of the system, which can be advantageous for many practical applications. in the experiments for (chen and huang, 2016), we found that the best variant of the ss-cnn method was one in which we trained two cnn models -one that scores source-language sentences and one that scores target-language sentences -and sum the scores of these two models to get the overall score of each sentence pair in the bilingual corpus. thus, the sscnn variant we used assigns high scores to sentence pairs where both the source-language sentence and its target-language partner resemble sentences in the in-domain corpus. note what this method does not do: it does not check that the target sentence is a good translation of the source sentence. essentially, it scores the extent to which both the source and target sentence are in-domain, but does not in any way penalize bad translations. we say that such a method is "symmetric": it incorporates equal amounts of information from the source and the target language, but it is not "bilingual": it does not incorporate information about the quality of translations. the main motivation for this paper is to explore cnn-based data selection techniques that are bilingual. it is based on semi-supervised cnns that use bitokens as units instead of source or target words @xcite . for the bitoken semi-supervised cnn, we should use the abbreviation "bi-sscnn". we also experiment with the bilingual method that combines ibm model 1 and language model (lm) scores and neural network joint model. in this paper, we carried out experiments reported on two language pairs: chinese-to-english and arabic-to-english. we fix the number of training sentences to be chosen for the data selection techniques so that they can be fairly compared, and measure the bleu score on test data from the resulting mt systems. it turns out that three techniques have roughly the same performance in terms of bleu: the symmetric but non-bilingual word-based sscnn method, and two symmetric, bilingual techniques -the simple ibm-lm method, and the nnjm method. the bi-sscnn method, on the other hand, outperforms all other methods: by +0.5 bleu for chinese-to-english task, and by +0.3 bleu for arabic-to-english task. because the main motivation for the paper is exploration of bilingual methods for sscnnbased data selection, we perform another set of experiments to see how good each method is at rejecting sentence pairs whose target side is not a translation of the source side. we do this by permuting 50% of the pairs in the bilingual corpus. we then count the proportion of pairs chosen by each method that are mismatched. in these experiments, all three bilingual methods -ibm-lm, nnjm, and bi-sscnn outperform the other methods (they chose a smaller proportion of mismatched sentence pairs) and bi-sscnn is even more effective than the other two bilingual methods. ## four data selection methods in this section, we focus on four methods for data selection: the ibm-lm method, the nnjm method, the original word-based sscnn method, and the bi-sscnn method. the ibm-lm method is similar to @xcite , which is a simple bilingual method that is a good baseline for other bilingual methods. the nnjm-based data selection @xcite is the first bilingual nn method. the word-based sscnn method is described in @xcite . the sscnn method is symmetrical but not bilingual. the bi-sscnn method is a newly proposed method, which is symmetrical as well as bilingual. ## experiments on smt our goal is to adapt the mt system when only a small amount of in-domain data is available. so in most of our experiments, we ignored domain information about the training data, such as the source of each corpus. what we have is a small development set (dev) and one or more test sets (test) which are in the same domain. ## follow-up experiments on nmt after we submitted this paper, we did some experiments with a neural machine translation (nmt) system @xcite using bi-sscnn. 6 the experiments were carried out with an open source system called nematus @xcite , which is an attention-based nmt system @xcite . we carried out experiments on english-to-french (en2fr) wmt task foot_6 . the training data contain 12 million sentence pairs; the dev set is a concatenation of newstest2012 and 2013, which contains 6,003 sentence pairs; the test set is newstest2014, which contains 3,003 sentence pairs. ## conclusions we proposed a new method for data selection from a large bilingual corpus for the purpose of training an smt system. the new method is based on a bitoken semi-supervised convolutional id data size data description neural networks. it outperformed its nearest competitor, a method that uses a word-based ss-cnn, by +0.5 bleu on a chinese-to-english task and by +0.3 bleu on an arabic-to-english task. since one of the motivations underlying the creation of the bi-sscnn method (and two other data selection methods, those based on ibm-lm and nnjm) was the ability to screen out noisy sentence pairs, we carried out another type of experiment in which the methods were explicitly tested for the ability to do this when half the pairs in the bilingual corpus have been deliberately corrupted. according to this criterion, too, the bi-sscnn outperformed all other methods. in the follow-up experiments, we find that neural machine translation is more sensitive to noisy data than statistical machine translation. therefore, bi-sscnn, which can effectively screen out noisy sentence pairs, can benefit nmt much more than smt. for instance, given a potential training corpus with 30% sentence alignment error, data selected with bi-sscnn yields a system with a performance gain of over 3 bleu points above the baseline.