Norm-based Noisy Corpora Filtering and Refurbishing in Neural Machine Translation (2022)

Abstract: Recent advances in neural machine translation depend on massive parallel corpora, which are collected from any open source without much guarantee of quality. It stresses the need for noisy corpora filtering, but existing methods are insufficient to solve this issue. They spend much time ensembling multiple scorers trained on clean bitexts, unavailable for low-resource languages in practice. In this paper, we propose a norm-based noisy corpora filtering and refurbishing method with no external data and costly scorers. The noisy and clean samples are separated based on how much information from the source and target sides the model requires to fit the given translation. For the unparallel sentence, the target-side history translation is much more important than the source context, contrary to the parallel ones. The amount of these two information flows can be measured by norms of source-/target-side context vectors. Moreover, we propose to reuse the discovered noisy data by generating pseudo labels via online knowledge distillation. Extensive experiments show that our proposed filtering method performs comparably with state-of-the-art noisy corpora filtering techniques but is more efficient and easier to operate. Noisy sample refurbishing further enhances the performance by making the most of the given data.

## introduction neural machine translation (nmt) has achieved significant progress with help from large parallel corpora for training @xcite . these data are typically extracted from the web without much control over the quality, which presents misalignment, wrong languages, too many numbers or urls, etc. in this case, noisy corpora filtering holds a critical research area to prevent noisy bitexts from degrading the generalization performance of nmt @xcite . this paper introduces a norm-based noisy sample filtering and refurbishing method, which avoids extra clean bitexts and heavy scorers. we distinguish unparallel sentence pairs from others based on observed model behaviors during the training of nmt. generally, the model captures two aspects of information to predict the given translation, the source-side context information from the encoder and the target-side one from history translations in the decoder. for unparallel sentence pairs, provided translations are partially or entirely unrelated to the source sentence. in this case, the nmt model behaves as a language model, which requires excessive target-side information to fit noises. thus, we use the information ratio of the source to the target side as the criterion to filter noises. specifically, we observe that a greater vector norm implies richer context information captured by the model. thus, we calculate the amount of each information flow by norms of corresponding context vectors. this metric is easy to obtain in the training process and sufficient to model how much information is encoded on each side. we take figure 1 as an example. when the model predicts the content word "china", the norm of the source-side context information is more significant than that of the target side. the opposite situation is in generating the function word "in". however, the quantity of target-side information appears to be exceptionally great for the noisy fraction, which presents a deeper color than others, leading to a lower score than correct translations under our estimation. we further propose to refurbish discovered noisy samples by generating pseudo labels via online knowledge distillation. by doing this, the source sentence in figure 1 is regarded as the monolingual data to complement limited clean bitexts. throughout the whole, we incorporate filtering and refurbishing into the training of nmt rather than separating data filtering and training, thus considerably improving computational efficiency. we validate the effectiveness of our approaches on transformer-based nmt @xcite , including the wmt2020 shared task for parallel corpus filtering (km⇒en and ps⇒en) and our inhouse web-crawled datasets (he, id, pt, ko, and es⇒zh). empirical results show that our proposed method performs comparably with sota noisy corpora filtering approaches. refurbishing noisy samples further substantially boosts the performance. detailed analyses show that our metric can reflect the alignment extent at word and sentence levels. the contributions of this paper are three-fold: ## background in this section, we first briefly introduce a mainstream nmt framework, transformer, with a focus on how to capture source-and target-side contexts. we then present how the vector norm serves as an indicator of diverse features, which motivates us to count how much information is encoded in context vectors of each side based on vector norms. ## methodology we aim to detect and refurbish noisy sentence pairs by observing how the model predicts each token. a sentence pair is potentially misaligned if the model depends heavily on history predictions rather than the source sentence to fit given translations. to this end, we first introduce a norm-based measurement to count the amount of information extracted from the source and target side (section 3.1). then, we show how to use this metric to filter noisy samples (section 3.2), which are further refurbished by producing pseudo labels via online knowledge distillation (section 3.3). ## experiments we conduct experiments on two types of datasets: ## analysis we conduct extensive analyses to evaluate the ability of r(x, y) to pinpoint unparallel sentence pairs. we first examine whether r(x, y) can reflect the overall degree of misalignment. from a more finegrained view, we explore the correlation between the score γ j at step j and linguistic properties. ## conclusion this paper presents a novel norm-based noisy corpora filtering and refurbishing method. we propose to use the information ratio from the source to the target side to distinguish unparallel sentence pairs. the amounts of these two information flows are calculated by norms of context vectors of each side. unlike parallel sentence pairs, the excessive targetside information is needed for the model to fit unparallel ones, which present relatively lower scores. we incorporate the noisy corpora filtering into the training of nmt without any extra clean data or costly pre-trained scorers. extensive experiments show that our method performs comparably with sota results with significant advantages in time and computational costs. we further refurbish the discovered noisy data by producing pseudo labels via online knowledge distillation, which obtains further performance gains.