Filtering Matters: Experiments in Filtering Training Sets for Machine Translation (2023)

Abstract: We explore different approaches for filtering parallel data for MT training, whether the same filtering approaches suit different datasets, and if separate filters should be applied to a dataset depending on the translation direction. We evaluate the results of different approaches, both manually and on a downstream NMT task. We find that, first, it is beneficial to inspect how well different filtering approaches suit different datasets and, second, that while MT systems trained on data prepared using different filters do not differ substantially in quality, there is indeed a statistically significant difference. Finally, we find that the same training sets do not seem to suit different translation directions.

## introduction in recent years, machine learning (ml) research has generally focused on creating better models rather than better datasets. the focus on benchmarking model performance spurs researchers into adapting the largest existing datasets without fully considering fidelity to the underlying problem the model should solve @xcite . the effectiveness of ml models, however, depends on both algorithms and data. @xcite argue that as the datasets define the world within which models exist and operate, more work is needed on how the data can be optimized for more effective use. filtering parallel data for machine translation (mt) is the task of removing possible detrimental segments from data used for training mt models. detrimental segments, in this context, are sentence pairs in the training data that may degrade the performance of an mt system trained on the data. filtering is usually carried out using a set of rules, scoring mechanisms and/or classifiers, to remove sentence pairs with the lowest perceived quality. in this work, we experiment with filtering the raw data from two parallel corpora, paracrawl @xcite and parice @xcite , for the english-icelandic language pair. our goal is to minimize detrimental data while losing little or no useful data from the texts, thus building a more accurate training set. we investigate how shallow filters, four different scoring mechanisms and different classifiers based on them are suited to score and filter english-icelandic sentence pairs. we compare these to bicleaner @xcite and to how the two corpora (paracrawl and parice) were filtered for publishing. recent literature on parallel corpus filtering has largely focused on filtering noisy data collected from the web, as discussed in section 2. we want to investigate whether the same approaches are suitable to filter noisy web-crawled corpora and cleaner parallel corpora compiled from document pairs that are known to be mutual translations. furthermore, although the same training data is usually used for training both translation directions, @xmath0, for a given dataset, we investigate whether that is optimal or whether filtering separately for each translation direction is likely to bring improvements to a downstream mt task. our primary goal is to find out how to filter parallel corpora so as to compile a training set that potentially gives the best results when used to train an mt system. we seek an answer to the following research questions: 1) should the same filtering approaches be used for a given language pair, regardless of the datasets being filtered? 2) should the intended translation di-rection of an mt system effect how the data, used to train the system, is filtered? in order to answer these questions, we build mt models for both translation directions and multiple different filtering approaches for each one, and evaluate the results, both manually and automatically. we find for best results, specific filtering approaches should be chosen based on the dataset and translation direction being filtered. ## related work in their paper, @xcite show that incorrect translations, untranslated target text, misalignments, and other noisy segments in a parallel corpus have a detrimental effect on the output quality of nmt systems trained on that corpus, as measured using bleu @xcite . they specify five general classes of noise commonly found in the german-english paracrawl corpus: misaligned sentences, disfluent text, wrong language, short segments, and untranslated sentences. as this classification is rather coarse, some variation can be expected within each class; a misalignment in one sentence pair does not have to be equivalent to a misalignment in another sentence pair. @xcite focus on finegrained semantic divergences within mostly equivalent pairs (pairs of words, phrases or sentences that have similar meanings and connotations). an example given in the paper is fr: "votre père est français" → en: "your parent is french", where the correct translation should be: "your father is french". these fine-grained divergences can even be found in high-quality parallel corpora. they find that the divergences cause degradation on the mt output of a system trained on the data, as measured by bleu and meteor @xcite , and that divergences impact model confidence in their predictions. lexical substitution causes the largest degradation and subtree deletion the least. nevertheless, the impact on divergences seem to be smaller than that of noise. they argue that this suggests that noisefiltering techniques are subobtimal to deal with fine-grained divergences. in early work on filtering web-scraped parallel corpora, @xcite filter out machinetranslated content and show that removing large amounts of training data can improve performance of an mt system, challenging conventional wis-dom at the time that more data is better. cross-lingual word embeddings have been used to calculate distance between equivalences in different languages @xcite . @xcite treat filtering as a supervised regression problem and show that @xcite between the target and mt-translated source, as well as cosine distance between sentence embeddings of the source and target, are important features. the conference on machine translation, wmt, hosted three annual shared tasks on parallel corpus filtering @xcite @xcite , focusing on filtering noisy web-crawled corpora. chaudhary et al. (2019) and artetxe and schwenk (2019a ) introduced approaches based on crosslingual sentence embeddings trained from parallel sentences. when using cosine similarity to find the nearest neighbours in an embedding space, cosine similarity is not necessarily globally consistent and different scales of target candidates for a given source sentence may affect their relative ranking, causing the hubness problem, described by @xcite . the problem is caused by a few vectors in the embedding space that tend to be "universal" neighbours, i.e., neighbours of a large number of different mapped vectors, pushing the correct ones down the neighbour list. both papers tackle the scale inconsistencies of cosine similarity by considering the margin between a given sentence pair and its closest candidates to normalize the similarity scores. bicleaner uses a set of handcrafted rules to detect flawed sentences and then proceeds to use a random forest classifier based on lexical translations and several shallow features such as respective length, matching numbers and punctuation. it also scores sentences based on fluency using 5gram language models. bicleaner ai @xcite is a fork of bicleaner using a neural classifier. it has been shown to give significant improvements in translation quality as measured by bleu when used for filtering training data for multiple language pairs, as compared to the previous version of the tool. in contrast to tools that implement a single method for parallel corpus filtering, @xcite implement multiple different filters in the opusfilter toolbox. these include heuristic based filters, language identification, character-based language models and word alignment tools. @xcite revisit the noise classes specified by @xcite to investigate how accurately two of the strongest filtering approaches to date (according to them) cross entropy @xcite and laser @xcite can filter out different noise classes. they find that for a common language pair, @xmath1, most types of noise can be detected with over 90% accuracy, although misalignments and poor synthetic translation can only be detected with an accuracy of less than 70%. for a less common language pair, khmer-english, the performance of the filtering system degraded significantly and the accuracy of identifying noise was lowered by 8-19%, depending on the type of noise. ## experimental setup we compare a number of approaches and scoring mechanisms and apply them to a web-crawled corpus, on the one hand, and a parallel corpus compiled from parallel documents, on the other. we manually evaluate samples of the results using the taxonomy developed by @xcite to gain an understanding of what sort of data each approach and scoring mechanism filters out. we then train mt systems on datasets filtered using the different approaches, as well as on previously published, filtered versions of the corpora, and compare the quality of the resulting systems in terms of bleu scores. we measure bleu scores on the test set provided for the english-icelandic language pair in the wmt 2021 shared task @xcite , using sacrebleu @xcite . ## conclusions and future work in regards to our research questions, our results indicate that different filtering approaches suit different datasets and translation directions, even though we are working within the same language pair. manual inspection of filtering results and scoring mechanisms seem to be helpful for making informed decisions on how best to filter a dataset. for best results, filtering approaches should be chosen for each translation direction. a limitation of our work is that it does not show which data are detrimental and which are beneficial. in future work, we want to investigate if the differences between datasets used for training can give us an idea of which sentence pairs are most important to filter out. we intend to do this by investigating the pairs discarded by our filters, to compare the data that leads to rising bleu scores and that which lowers them. this could lead to insights that help constructing filters that work on a more fine-grained level when that is needed. our manual evaluation shows that the scores, generated by the automatic scoring systems we employ, have different interpretations depending on the dataset. if scores are used for filtering parallel data, the optimal score should lead to a dataset that produces the best mt model. @xcite suggest a threshold of 0.6 for labse when mining parallel text from commoncrawl, stating that the threshold was selected by manually inspecting sampled data, but do not specify the language pair used when inspecting the data. in order for the scoring mechanism to be most effective, the user should inspect the results for their dataset before setting a threshold. while all our scoring mechanisms seem to be useful, none of the methods are very good at identifying mutual translations in particular, labelled cc in our taxonomy. we trained two bicleaner models for our experiments and our lemmatized model gave the best results for filtering parice for the @xmath2. the bicleaner models could perhaps be improved. bicleaner uses n-gram models and we only used a part of our parallel corpora to train these. if we would use larger corpora the n-gram models would likely give us more accurate scores. the bilingual probability dictionary we used only contained lemmas. by producing all wordforms for the lemmas and trying to estimate the prevalence of each wordform using a monolingual corpus, we could perhaps provide an unlemmatized model with more accurate information leading to better results. furthermore, we only use 10k sentences to train our gpt-2 perplexity model for icelandic. a larger dataset may increase its accuracy. two systems participating in the wmt 2021 news translation task, evaluated on the same data, obtain scores only slightly higher than ours, but while we only train a transformer base model, they train larger models using more resources and much longer training time. in our experiments, models that have been filtered more tend to converge faster. we can deduce from this that training data that is better filtered, not only improves mt output quality, but is also in line with a call to greener and more sustainable models of ai, see e.g. @xcite and @xcite .