Learning Feature Weights using Reward Modeling for Denoising Parallel Corpora (2021)

Abstract: Large web-crawled corpora represent an excellent resource for improving the performance of Neural Machine Translation (NMT) systems across several language pairs. However, since these corpora are typically extremely noisy, their use is fairly limited. Current approaches to deal with this problem mainly focus on filtering using heuristics or single features such as language model scores or bi-lingual similarity. This work presents an alternative approach which learns weights for multiple sentence-level features. These feature weights which are optimized directly for the task of improving translation performance, are used to score and filter sentences in the noisy corpora more effectively. We provide results of applying this technique to building NMT systems using the Paracrawl corpus for Estonian-English and show that it beats strong single feature baselines and hand designed combinations. Additionally, we analyze the sensitivity of this method to different types of noise and explore if the learned weights generalize to other language pairs using the Maltese-English Paracrawl corpus.

## introduction large parallel corpora such as paracrawl @xcite which have been crawled from online resources hold the potential to drastically improve performance of neural machine translation systems across both low and high resource language pairs. however, since these extraction efforts mostly rely on automatic language identification and document/sentence alignment methods, the resulting corpora are extremely noisy. the most frequent noise types encountered are sentence alignment errors, wrong language in source or target, and untranslated sentences. as outlined by @xcite , training algorithms for neural machine translation systems are particularly vulnerable to these noise types. as such, these web-crawled corpora have seen limited use in training large nmt systems. this paper proposes a method for denoising and filtering noisy corpora which explores and searches over weighted combinations of features. during nmt training, we score sentences and create batches using random weight vectors. these batches are use to train the system and measure improvement over the validation set (reward). finally, by modeling the weight-reward function, we learn the set of weights which maximize reward and are used to score and filter the noisy dataset. at a high level, this method (i) allows the use of multiple sentence level features, (ii) learns a set of interpolation weights for the features which directly maximize translation performance, (iii) requires no prior knowledge about which features are informative or even if they are mutually redundant, and (iv) trains within the nmt pipeline and does not require any special infrastructure. we include experiments which apply this method to building nmt systems for the noisy estonian-english paracrawl dataset and show that it beats strong single feature filtering-baselines and hand-designed feature interpolation. additionally, we analyze the robustness of this method in the presence of specific kinds of noise @xcite via a controlled experiment on the europarl datasets. finally, we look at the impact of transferring the learned weights from one language pair (estonian-english) to a noisy dataset of another language pair (maltese-english paracrawl). we present related work in section 2. section 3 describes the procedure we use to model and search over the weight-feature-reward space to estimate feature weights which maximize translation performance. our experiment design, datasets and features, appear in section 4. section 5 includes our primary results where we compare the performance of the proposed method to strong single feature filtering baselines and hand-design feature weights. we conclude in section 6 with an analysis of this method's performance at filtering specific kinds of noise and the application of learned weights to a different language pair. ## related work existing efforts towards filtering and denoising noisy corpora focus on pre-filtering using handcrafted rules and by using sentence pair scoring and filtering methods. deterministic hand-crafted rules @xcite remove sentence pairs with extreme lengths, unusual sentence length ratios and exact sourcetarget copies, and are extremely effective in removing most of the obvious automatic extraction errors. automatic sentence pair scoring functions have been used successfully to filter noisy corpora as well. this includes the use of language models @xcite , neural language models trained on trusted data (junczys-dowmunt, 2018) and lexical translation scores (gonzález-rubio, 2019). @xcite propose the use of cross-lingual sentence embeddings for determining sentence pair quality while several efforts @xcite @xcite have focused on the use of monolingual word embeddings. @xcite use a machine translation system trained on clean data to translate the source sentences of the noisy corpus and evaluate the translation against the original target sentences using bleu scores. @xcite and @xcite propose similar methods using meteor scores and levenshtein distance respectively. @xcite and @xcite present techniques for detecting machine translated sentence pairs in corpora. tools such as laser @xcite , bicleaner @xcite and zipporah @xcite have been used @xcite for noisy corpus filtering. curriculum learning has been used to obtain policies for data selection which can expose the model to noisy samples less often during training @xcite . more recently, elnokrashy et al. (2020) and esplà gomis et al. (2020) have used classifier based approaches to filtering noisy parallel data. ## method the proposed method centres around finding weights for combining sentence-level features, which are then used to compute sentence-level scores and filter the noisy corpus. while the choice of features can be arbitrary, this method's performance will eventually depend on their quality, and we would ideally want them to be informative and decorrelated. figure 1 provides an overview of the proposed method. we first train a number of candidate neural machine translation (nmt) systems. during training for each candidate system, we repeatedly (i) generate a random weight vector, (ii) sample a batch of sentences from the noisy corpus based on sentence-level scores computed using this weight vector, (iii) update nmt system parameters using this batch, and (iv) measure the improvement in translation quality on a validation set following this update. the weight vector w, the average feature vector φ of the batch, and the improvement r on the validation set (reward) are recorded for each batch t during the training of each candidate nmt system i, and w i,t , φ i,t , r i,t becomes a sample in new data set d, called the tuning data set 1 , for learning feature weights to maximize reward. hence, even though the parameters of the candidate systems are not used directly, they are used to gather noisy candidate evaluations of the latent weightfeature-reward function. once we have d, we use a feed-forward network to learn the weight vector that maximizes the reward. the learned weight vector w * is then used to compute sentence-level scores and filter the noisy data set. the final nmt system is trained using this clean data set. some subtleties in normalizing the observed rewards and learning weights are explained below. ## experiment setup we use fairseq @xcite for our neural machine translation systems configured to be identical to the systems described in @xcite . the feed-forward network used to tune weights has two 512-dimensional layers and is trained using standard sgd using a learning rate of 0.1. the grid search for the weights was done on the range [-2.5, 2.5] with 5000 points uniformly distributed per dimension. the number of samples used for reward normalization was 3 and the window for computing the delta-perplexity reward was set to 3. ## results for our experiments, we scored all sentences in the noisy corpus, sorted and sampled the top parallel sentences to form subsets with 10, 15 and 20 million english words. these filtered data sets were used to train standard nmt systems and performance was evaluated on the test set described in the previous section. the results of these filtering experiments appear in next, we look at interpolation of features using weights learned using the proposed method. as a baseline, we also include an experiment which filters based on a uniform interpolation of the five features we use. this baseline performs worse than the strongest single feature filtering experiments by 0.5 to 1 bleu points. for both the weight-based and feature-based methods of learning interpolation weights for the features, a significant number of candidate runs are required before adequate performance is achieved. this is not surprising, since we are searching for an optimal weight vector in a fairly large weight space and we need a large number of samples before a good representation of the weight-reward function can be learned. figure 2 shows the improvement in bleu scores for the weight-based approach as data from more candidate runs in added to the tuning stage for learning weights and filtering the data set. the performance of the final nmt system steadily improves as more data from more systems is added and eventually converges. our strongest result was achieved with 14 candidate runs for the weight-based approach for the 10, 15 and 20m setting respectively. this beat the uniform weight baseline by 1.5 to 2 bleu points and the strongest single feature (laser) baseline by 1 bleu point. the feature based approach performed slightly better with 15 candidate runs and beat the strongest single feature baseline (laser) by 1.3 bleu points. ## analysis the following sections examine the learned weights, the effect of transferring them to noisy corpora of a different language pair and the method's performance when exposed to specific kinds of noise. ## future work the validation set based delta-perplexity is expensive to compute per update and replacing it with a more stable or time invariant reward @xcite may help improve the performance of this method. additionally, we plan to replace grid search with a more granular search procedure over the weight space with respect to the weight-featurereward function. the tuning network can also be modified to include sentence-quality modulated loss functions (via feature values). an alternative to searching for feature weights is to instead search for the prototypical feature vector which maximizes translation performance and then use it to filter the closest sentence pairs from the noisy dataset. finally, as discussed in section 6.2, transferring learned weights has the potential to dramatically reduce the cost of applying to this method to new language pairs and may help with performance on low-resource language pairs where good feature weights cannot be learned. 9 we note that the performance of this analysis depends on the chosen features. as an extreme example, if we perturb the source sentences and only consider a target-side feature (such as target language model scores), we will have no way of discriminating bad noisy samples from the clean ones. ## conclusion we present a method for denoising and filtering noisy parallel data for improving the performance of neural machine translation systems. we learn interpolation weights for sentence-level features by modeling and searching over the weight-reward space. these are used to score and filter sentences in the noisy corpora. our experiments with estonian-english paracrawl show gains of over a bleu point over the strongest single feature filtering and uniform weight baselines. analysis also shows that this method is effective at addressing the most common noise types in web-crawled corpora.