A Comparison of Data Filtering Methods for Neural Machine Translation (2022)

Abstract: With the increasing availability of large-scale parallel corpora derived from web crawling and bilingual text mining, data filtering is becoming an increasingly important step in neural machine translation (NMT) pipelines. This paper applies several available tools to the task of data filtration, and compares their performance in filtering out different types of noisy data. We also study the effect of filtration with each tool on model performance in the downstream task of NMT by creating a dataset containing a combination of clean and noisy data, filtering the data with each tool, and training NMT engines using the resulting filtered corpora. We evaluate the performance of each engine with a combination of direct assessment (DA) and automated metrics. Our best results are obtained by training for a short time on all available data then filtering the corpus with cross-entropy filtering and training until convergence.

## introduction large-scale, publicly available bilingual corpora are an excellent resource for training neural machine translation (nmt) models. performance in the nmt task improves as the size of the training data increases @xcite , and with datasets like cc matrix @xcite , tens or even hundreds of millions of sentence pairs are freely available for many language pairs. however, these corpora are known to be noisy @xcite , and nmt models are quite sensitive to noisy training data @xcite . thus, tools to filter noisy data are becoming an important step in nmt training pipelines. in this paper, we compare the performance of several available tools in the task of data filtering, breaking down the results by different types of noise. we then train mt engines with different filtered versions of the same corpus to compare the effects of data filtering on the downstream task of translation. ## related research cleaning noisy data with the purpose of using them for mt training has been a major topic in research. since neural mt performance has shown to be highly dependent on the size of the training data @xcite as well as their quality @xcite , several large-scale initiatives for crawling and cleaning data from the web appeared, such as paracrawl @xcite and ccmatrix @xcite . for this reason, most works in this area focus on filtering this type of data, i.e. noisy data collected from the web. one of the earlier works proposed an unsupervised method, in particular using an outlier detection algorithm to filter a parallel corpus @xcite , which led to an increased performance of the smt system trained on these cleaned data. another unsupervised method consisted of a graph-based random walk algorithm and extracted phrasepair scores to weigh the phrase translation probabilities to bias towards more trustworthy ones @xcite . the method is based on the observation that better sentence pairs often lead to better phrase extraction and vice versa. several subsequent works treated the data filtering task as a classification problem. an example of this is the method proposed in @xcite , which is based on generating synthetic noisy data (inadequate and non-fluent translations) and using these data to train a classifier to identify good sentence pairs in a noisy corpus. another classification approach was proposed within the 2020 task on parallel data filtering @xcite . in this approach, the authors used an end-to-end classifier that learns to distinguish clean parallel data from misaligned sentence pairs. the system first uses a transformer model to obtain sentence representations, followed either by a classifier (siamese network) or additional layers that are fine-tuned (ac ¸arc ¸ic @xcite . another popular approach is based on utilizing cross-entropy. in the 2018 edition of the shared task on data filtering, the winning system used neural mt models in both directions trained on clean data to score sentence pairs with dual cross-entropy (junczys-dowmunt, 2018). the divergent cross-entropies are penalized and the penalty is weighed by the average crossentropy of the two nmt models. another winning system in the 2020 shared task enhanced this approach by combining a dual cross-entropy from two nmt models with a number of other features: a bilingual gpt-2 model trained on source-target language pairs as well as a monolingual gpt-2 model for each of the languages, and statistical word translation model scores @xcite . recently, there has been a new direction in parallel data filtering research consisting of using multilingual language models, which create sentence representations in a multilingual vector space. then, two parallel sentences are identified by taking the nearest neighbor of each source sentence in the target side according to cosine similarity, and filtering those below a fixed threshold @xcite . another work improves on these results suggesting an alternative scoring method that uses the margin between the similarity of a given candidate and that of its k nearest neighbors @xcite . as demonstrated in a recent work @xcite , the performance of a given parallel data cleaning method can vary significantly depending on the data conditions and the task definitions. in one attempt to clean mostly well-aligned bilingual data @xcite , the authors investigate the problem of filtering out semantically divergent sentences from a parallel corpus. some sentence pairs considered "parallel" present source and target sentence that do not convey exactly the same meaning, which is quite a common phenomenon in curated parallel corpora originating from translation memories. in our experiment, we use several multilingual language models, a method based on cross-entropy and a pre-trained model for mt evaluation with the goal of identifying the methods that can be most successfully applied to our use case of filtering corpora to train mt systems. ## materials and methods for this study, we selected two language pairs: @xmath0. these language pairs were chosen with consideration to their linguistic properties (diverse source languages with different scripts, differing levels of linguistic distance from english, and quite different linguistic characteristics), the demand for these language pairs in translation, and the availability of data and tools for the experiment. ## results below we present the results of the two parts of our study. the results of part i show that cross-entropy filtering is significantly better for removing the types of noise we studied. the automated metrics from engine training reinforce this conclusion. the results from part ii are less clear cut, with filtering having a comparatively stronger beneficial effect for the @xmath1. ## discussion in this paper we explored the relative performance of different methods of filtering noise from natural language training data, and the effect of filtering on the downstream task of machine translation. we found that cross-entropy filtering using models trained for the translation task performed better than multilingual alternatives such as laser or comet at identifying the types of noise we introduced across almost all noise types in both the @xmath2. language agnostic models have another disadvantage, which is that they cannot be used to identify wrong language data, a common source of noise in bilingual corpora. however, the clear superiority of cross-entropy filtering did not unambiguously extend to the downstream translation task, where a model trained on the unfiltered dataset performed the best in @xmath3, and no model achieved a statistically significant improvement over the baseline in the human evaluation. this suggests that in the regime of a few million sentences, the advantages of having more data volume or more diverse data can outweigh the costs incurred by significant noise present in the dataset. our results suggest that in situations where the quality of training data is uncertain, fair results can be obtained by training for a short time on all the available data, filtering the training data with laser or cross-entropy scores, and then continuing to train on a cleaner subset of the data. given that laser is language-agnostic, an additional filtering step based on languageidentification may be required when using this tool. in this study we generally followed the noise taxonomy found in @xcite , but other ways of categorizing noise also exist. we are also interested to investigate how these tools perform with noisy data categorized in linguistic terms, such as problems of fluency vs. adequacy. does data filtration with these tools introduce systemic bias of some sort, such as by preferentially removing sentences with numerous acronyms, shorter sentences, or sentences with lots of punctuation marks? would the same results be obtained with lower resource languages? we hope to pursue these questions in future research.