Selecting the best data filtering method for NMT training (2021)

Abstract: Performance of NMT systems has been proven to depend on the quality of the training data. In this paper we explore different open-source tools that can be used to score the quality of translation pairs, with the goal of obtaining clean corpora for training NMT models. We measure the performance of these tools by correlating their scores with human scores, as well as rank models trained on the resulting filtered datasets in terms of their performance on different test sets and MT performance metrics.

## introduction more and more parallel corpora are available today for mt training @xcite . however, when using data from public sources we can never be certain of the data quality, which is extremely important for an mt system's performance @xcite . in a commercial setting like ours, we typically face several data-related challenges. first, we want to be able to use publicly available parallel corpora which are already aligned, such as the opus corpus @xcite . second, we want to align our customers' translated documents on a sentence level and reliably filter out misaligned or poor quality sentence pairs. and finally, we want to use our customers' translation memories (tms) and be able to automatically select only the sentences that are relevant for nmt training. a large part of the data we use for mt training comes from tms where human translations are stored and are already aligned on a sentence level, which means that our data are generally better in terms of alignment and translation quality than the typical data collected from the web. however, there are other challenges that this type of corpora present for mt engine training. one example of this is that tms can contain expanded acronyms (the source segment contains an acronym and the target segment contains this acronym together with its expanded version), which can cause hallucinations. that is why in this experiment we focus on the task of cleaning specifically tm data. we explored different open-source tools that can be used for bilingual data cleaning. our goal was to choose the one that yields the best results when it comes to mt performance in order to incorporate it into our mt engine training pipeline. as a first step, we randomly selected 5 million sentence pairs from a corpus that contains all our potential training data in five different language directions: • english-chinese; as a next step, we selected approximately 100 sentence pairs from each language direction to be scored by professional linguists according to their translation quality. we then correlated the scores produced by each of the tools with the human scores. in addition, we used the human scores to establish a threshold for filtering the data for the mt training, and proceeded to create separate corpora for each language direction using only the sentences with scores above the threshold for that tool. next, we trained an nmt model with each data set for each language and compared the model performance. based on these results we make conclusions on whether they are in line with the results we achieved based on the correlation with human scores and which of the tools will be our preferred option for data cleaning. the remainder of the paper is structured as follows. section 2 includes an overview of previous related research, section 3 describes the experimental setup, and in sections 4 and 5 we discuss the results and the conclusions respectively. ## related research collecting and filtering parallel data has been a major topic in mt research. now it is more relevant than ever since neural mt performance is highly dependent on the size of the training data @xcite as well as its quality @xcite . most works in this area focus on filtering noisy data collected from the web. one of the earlier methods used an outlier detection algorithm to filter a parallel corpus @xcite . the method proposed by @xcite is based on generating synthetic noisy data (inadequate and non-fluent translations) and using these data to train a classifier to identify good sentence pairs from a noisy corpus. @xcite propose an unsupervised method to clean bilingual data, which uses a graph-based random walk algorithm and extracts phrasepair scores to weight the phrase translation probabilities to bias towards more trustworthy ones. the method is based on the observation that better sentence pairs often lead to better phrase extraction and vice versa. another method proposed by @xcite aims to identify semantic differences in translation pairs using cross-lingual textual entailment and additional length-based features. more recently, a number of new methods were proposed within the shared task on parallel corpus filtering and alignment, which has existed since 2016, although initially it aimed only at collecting parallel document pairs and did not cover the task of sentence alignment @xcite . in the 2018 edition, the winning system proposed to use neural mt in both directions to score sentence pairs with dual cross-entropy (junczys-dowmunt, 2018). one of the winning systems of the 2020 task @xcite ) also used dual cross entropy from neural mt models trained in both directions but combined it with a number of other features: a bilingual gpt-2 model trained on source-target language pairs as well as monolingual gpt-2 model for each of the languages, and statistical word translation model scores @xcite . another winner of the 2020 task uses an end-to-end classifier that learns to distinguish clean parallel data from misaligned sentence pairs. the model first uses a transformer model to obtain sentence representations, followed either by a classifier (siamese network) or additional layers that are fine-tuned (ac ¸arc ¸ic @xcite . several other recent works use multilingual language models similarly to @xcite , such as the 2019 shared task winner laser @xcite , as well as @xcite . our task of cleaning tm data is, however, different in nature from the task of cleaning noisy data collected from the web. the specific task of cleaning tms was addressed in the automatic translation memory cleaning shared task organized in 2016 @xcite . the methods used at the time mostly treated the task as a machine learning classification problem and differ mainly in the sets of features used by the classifier @xcite @xcite @xcite . our goal is to find out if using multilingual models, which are the basis of many tools used for cleaning noisy corpora, can successfully be applied to our use case of filtering corpora consisting mostly of tm content. ## conclusions and future work the two phases of this study suggest that using the right method to filter training data can result in similar or improved engine performance despite reducing the total amount of data the engine is exposed to. while training on an unfiltered (larger) dataset typically produced better results in terms of automated metrics, in practice we have observed more hallucinations and unacceptable translations from models trained without any form of data filtering. this is particularly pronounced when there is noise in the target, such as [sic] tags or expanded acronyms that do not exist in the source. among the data filtering methods we tested, our results show that marian-scorer and muse produce the best results. however, the limited scope and scale of the study mean that the results are far from generalizable. future work is still required to confirm or deny the validity of the results on a larger scale. for example, repeating the second phase of this experiment training three models per tool instead of one and taking the average score would help mitigate potential effects resulting from random weight initializations; human review of the model output would help ensure the automated evaluations in the second stage correspond with human judgment; and obtaining evaluations from more reviewers and calculating inter-rater reliability would help mitigate potential bias resulting from the use of a single reviewer on such a limited sample. there are also additional practical considerations that call for further investigation. how can an appropriate score threshold be identified in an automated way? do the appropriate threshold values vary across domains as well as languages? as the models trained on the full data set show some advantages over the models trained on filtered data, could using a twostep training process (training first on all available data, then fine-tuning on a subset of the cleanest data) produce superior models that demonstrate both robustness to input noise and high translation quality? beyond the topics enumerated above, our team plans to address several more analytical questions relevant to this line of inquiry in future research. multiple factors contribute to translation quality, and several different types of errors affecting translation quality exist; are these tools more likely to identify certain error types than others? do they identify problems with fluency equally as well as adequacy? are the conclusions drawn in this paper as applicable to the life sciences domain as the leisure and hospitality domain? and what biases are introduced by filtering data in this way? despite the limitations described here, we hope our work will provide a useful reference for other mt practitioners hoping to identify the best quality sentence pairs for use in their engine training.