A Comparison of Sentence-Weighting Techniques for NMT (2021)

Abstract: Sentence weighting is a simple and powerful domain adaptation technique. We carry out domain classification for computing sentence weights with 1) language model cross entropy difference 2) a convolutional neural network 3) a Recursive Neural Tensor Network. We compare these approaches with regard to domain classification accuracy and and study the posterior probability distributions. Then we carry out NMT experiments in the scenario where we have no in-domain parallel corpora and and only very limited in-domain monolingual corpora. Here and we use the domain classifier to reweight the sentences of our out-of-domain training corpus. This leads to improvements of up to 2.1 BLEU for German to English translation.

## introduction neural machine translation (nmt) outperforms phrase based smt for settings with large amounts of parallel data. however, in general adding out-of-domain data during training does not particularly improve nmt translation quality and is sometimes even harmful. for smt domain adaptation is well understood and can be classified into two main approaches: 1) model centric techniques adapt the training objective on instance level (e.g., sentence weighting or regularization) or model level (e.g., ensembling or language models), and 2) data centric techniques perform a sentence selection based on a score indicating the similarity between the sentence to be translated and in-domain data. we combine ideas from model centric and data centric approaches. we apply cnns and recursive neural tensor networks (rntns) to compute domain scores for sentence weighting in nmt. we compare with a cross-entropy classifier (xenc) as a well established baseline. our approach modifies the training objective so that every sentence pair is scaled by its individual weight, with sentences most similar to the in-domain data having most impact during training. our classifier is trained on small amounts of in-domain and out-of-domain monolingual data. we then use the classifier to find useful sentences within the out-of-domain data, i.e., sentences which are similar to the in-domain data. we carry out intrinsic (classification) and extrinsic (mt) experiments applying sentence classification for domain adaptation. the scores obtained by the cnn and rntn are strongly peaked in comparison to the cross-entropy classifier, which is important for the nmt sentence weighting. as the neural classifiers showed rather extreme probability score distributions in the intrinsic experiments, we studied various transformations of the scores which we use to find less peaked distributions. the resulting distributions showed less extreme behavior while preserving the strong classification ability. applying our transformed scores to the task of sentence weighting for domain adaptation outperformed cross-entropy classifiers. in summary, the contributions of this paper are as follows: 1) neural classifiers show high confidence separating in-and out-of-domain data, higher than a cross-entropy classifier, hence posterior probabilities are distributed closely around the extremes 0 and 1. 2) the cnn and rntn classifiers don't differ much from each other with respect to their score distributions, both are strongly peaked. 3) the extreme scores need to be transformed in order to be applied as weights in nmt, and we show how to do this effectively. 4) we show that using transformed cnn scores as weights during nmt training is better than a cross-entropy based classifier, which was the previous state-of-the-art solution. ## sentence-weighting techniques in order to apply sentence weighting to the translation process, one first needs to come up with a method for scoring sentences with respect to how similar they are to in-domain data. here we carry out a comparison between an established baseline (cross entropy) to the two different techniques based on neural networks that we have discussed (cnn and rntn). ## extrinsic evaluation: neural machine translation in this section we first present our score transformations, and then we present the experiments and results. in initial experiments (which we present in detail later), we found that without applying score transformations instance weighting training of nmt models does not converge. during sentence weighting, the probability score from the classifiers is multiplied with the learning rate. as mentioned previously, the high classification confidence in neural classifiers lead to a vast majority of sentences scored very close to 0, setting the learning rate during training very low. this restricts the transformer to only learn fully on a small subset of its original training data. we suppose the rather extreme original probability scores let the nmt starve for data. for the purpose of sentence weighting, the data distributions from the classifier outputs are problematic in a sense that they put most of the mass to the borders of the distribution, i.e., almost all of the scores are very close to 0 or 1. this impacts the sentence weighting techniques significantly, since a score that is almost 0 effectively excludes these sentences from the data set. we therefore applied several score transformations to obtain a normalized score distribution, as we describe next. ## related work domain adaptation strategies can be separated into four categories: data selection, data generation, instance weighting and model interpolation @xcite . we focus our discussion on data selection and instance weighting, as these are closely related to our approach. data-centric methods. models are trained using in-domain and out-of-domain data to evaluate out-of-domain data and compute a similarity score. using a cut-off threshold on these scores the training data can be selected. language models moore and lewis ( 2010 in settings where the amount of parallel training corpora is not sufficient, generating pseudo-parallel sentences by information retrieval @xcite , self-enhancing @xcite or parallel word embeddings @xcite . aside from generating sentences, other approaches generate monolingual n-grams @xcite or parallel phrase pairs @xcite . in general, data-centric methods (data selection and data generation) are not smt specific and can be directly applied to nmt. however, because these methods are not directly related to nmt's training criterion, they only lead to minor improvements @xcite . model-centric methods. instance weighting is a technique from smt and was introduced to nmt as well @xcite . an in-domain language model was trained to measure the similarity between sentences and the in-domain data via cross-entropy. the weights are then integrated into the training objective. we improve on their work by using state-of-the-art neural classifiers and showing that they are more effective than cross-entropy. two works that are closer to our work are @xcite and @xcite . in wang et al. (2018) they generate sentence embeddings for all in-domain sentences and then measure the distance between every sentence and the in-domain core. the underlying assumption is that the core of all in-domain sentence embeddings is a typical representative and proximity in their sentence embeddings indicates being part of the same domain. this approach is appropriate when we have in-domain parallel text, but we study a different scenario, with no access to in-domain parallel text, which means the encoder has no access to in-domain training examples. in chen et al. (2017) a domain classifier is incorporated into the nmt system, using features from the encoder to distinguish between in-domain and out-of-domain data. the classifier probabilities are used to weight sentences with regard to their similarity to in-domain data, when training the neural network. scaling the loss function is similar to multiplying the learning rate with the instance weight. the classifier and nmt are trained at the same time, whereas we chose an approach with pretrained neural classifiers which are trained on a small amount of monolingual data (the scenario we study) with no access to parallel in-domain data. finally, while some previous work we have mentioned did look at various ways to use domain classification, such previous work has not focused on how to weight the classifier probabilities for effective use in nmt, which we showed is important for obtaining translation quality improvements, particularly when using neural classifiers which can be overconfident. ## conclusion neural classifiers have high confidence when separating in-domain from out-of-domain data, leading to a strong decision boundary. classification results are good, but the boundary was too drastic, resulting in a poor score distribution with most mass near 0 and 1. this can be fixed by adding +1, keeping sentences with a low score as they are and giving a bonus to sentences with a higher score. the scores from, e.g., a cnn, can be transformed by a sigmoid function, making the score distribution more natural while keeping its strong decision boundary. cross-entropy approaches lead to a poor score distribution. sigmoid cnn scores performed best. our mt experiments showed that neural classifiers can be used to score out-of-domain data effectively. our work showed that simple transformations of classifier outputs are necessary. the use of the transformed scores by applying sentence weighting on the nmt training data improves translation quality. our research shows that results from cnns trained on domain classification achieve significant domain adaptation effects in nmt. it was important to carry out light-weight score transformations. we outperformed baseline experiments by up to 2.1 bleu points.