Detecting Various Types of Noise for Neural Machine Translation (2022)

Abstract: The filtering and/or selection of training data is one of the core aspects to be considered when building a strong machine translation system. In their influential work, Khayrallah and Koehn (2018) investigated the impact of different types of noise on the performance of machine translation systems. In the same year the WMT introduced a shared task on parallel corpus filtering, which went on to be repeated in the following years, and resulted in many different filtering approaches being proposed. In this work we aim to combine the recent achievements in data filtering with the original analysis of Khayrallah and Koehn (2018) and investigate whether state-of-the-art filtering systems are capable of removing all the suggested noise types. We observe that most of these types of noise can be detected with an accuracy of over 90% by modern filtering systems when operating in a well studied high resource setting. However, we also find that when confronted with more refined noise categories or when working with a less common language pair, the performance of the filtering systems is far from optimal, showing that there is still room for improvement in this area of research.

## introduction the phenomenon of noisy data in the training of machine translation (mt) systems has been studied from various angles over recent years. to outline the impact of noise, @xcite specified ten common noise categories and synthetically generated noisy data samples for each of them. by adding the noisy samples to an otherwise clean training corpus they measured the effect on the resulting translation system. their conclusion was that neural machine translation (nmt) is less robust towards noisy data than statistical machine translation and that some noise types can prove very detrimental to nmt performance. as nmt had surpassed the statistical approaches just a few years prior, this work paved the way for a spiked interest in data filtering research for machine translation. in the same year, the conference on machine translation (wmt) started to host an annual shared task on parallel corpus filtering @xcite @xcite featuring a broad mix of academic and industrial submissions. this shared task highlights the general need for well working data filtering systems and resulted in the publication of a variety of new filtering approaches @xcite @xcite . the wmt evaluations simulate a real-world data filtering task on web crawled data. each participating system is required to extract a fixed amount of parallel data and is ranked according to the performance of the translation system that was trained on the selected data. while this form of evaluation is very relevant from a practical point of view, it does not show how well a certain approach is performing on detecting specific types of noise -information that is very important when working on improving data filtering approaches. in this work we aim to unite both viewpoints in regards to data filtering. from the work of khayrallah and @xcite we already know how detrimental certain categories of noise are to an nmt system, so we ask the question: how well can stateof-the-art filtering systems distinguish the synthetic noise classes proposed by @xcite from clean data? while downstream performance might be the ultimate objective for data filtering systems, this setup allows us to investigate the strengths and weaknesses of current data filtering systems. to further investigate the challenges for a filtering system we introduce several, more refined, synthetic noise categories and use them to benchmark the performance of the aforementioned filtering systems. ## related work the task of data filtering for machine translation has attracted increasing attention in recent years, mainly for two reasons: to get an overview of existing approaches to data filtering for machine translation, a good place to start is the wmt shared task for parallel corpus filtering, which was held in 2018 @xcite , 2019 @xcite and 2020 @xcite . in these tasks, the participants are asked to select a fixed amount of data from a noisy parallel corpus using automatic methods. the examined language pairs were german-english (2018), nepali-english (2019), sinhala-english (2019), khmer-english (2020) and pashto-english (2020). the winning systems used a combination of language identification and language model and translation model scoring @xcite , similarities in the cross-lingual sentence embedding space @xcite and even gpt-2 models @xcite . the only way in which data filtering systems are typically evaluated is by training a machine translation system on the selected data @xcite @xcite . while this may be an intuitive evaluation criterion, it does not give many insights into the system performance regarding the detection of specific types of noise. while much effort has been put into the building of powerful filtering systems, the same can not be said for analyzing their performance on specific types of noise. @xcite and @xcite both examine the impact of various noise types on the performance of nmt systems but do not ask the question which types of noise can be handled reliably by data filtering systems. @xcite create synthetic noisy data to train a data filtering system with a classifier while michel and neubig (2018) create a 'noise translation benchmark' for nmt systems. in this work we aim to fill this gap and systematically compare the performance of data filtering systems on specific categories of noise. we start from the categories defined in @xcite and expand them further. ## types of noise we aim to investigate which noise categories can be reliably detected by state-of-the-art data filtering systems. manual annotation of noisy corpora is expensive and tends to be very corpus and language specific, depending on the original data sources and the extraction techniques. therefore we decide to investigate filtering systems on an array of noisy datasets mostly created synthetically from clean parallel data like it was done by @xcite . most of the categories were introduced in the work of @xcite (in the following marked with an asterisk (*)) but we propose two additional categories which we found to commonly occur in practice. here we list all types of noise that we are investigating as well as our automatic and language agnostic methods of creating such noisy data. if a certain type of noise is specific to the source or target side of the data, we note the noisy side in brackets. misaligned sentences * are created by shuffling the target side of a clean corpus. hence, every source sentence get assigned a random target sentence from the same domain but (most probably) without any overlap in meaning. misordered words (src|trg) * are obtained by arranging the words of either source or target sentence in a random order. wrong language (src|trg) * samples are selected from a parallel corpus from a different language pair. we specify which side of the data is not fitting the intended task. untranslated (src|trg) * sentence pairs are created by converting a src-trg corpus into a src-src respectively trg-trg corpus via copying. short segments (max. length) * are from a corpus with very short average sentence length. a segment being short does not imply that it is noisy or hurtful to the training. however we keep these categories for completeness sake in our analysis but do not emphasize on them in the experimental results. raw crawled data * is a mix of different types of noise and probably the most realistic noise category. we use data from an unfiltered web crawling corpus. note that some sentence pairs from this category might be valid in practice and we address this in section 5.1. over-/undertranslation often times happen as a result of poor sentence splitting and alignment. to create sentence pairs in this category we remove the second half of the source sentence respectively target sentence. synthetic translations can be found on an increasing number of websites and the simpler structure of synthetic translations @xcite can make them easier to be extracted and aligned by crawling scripts. to analyze if the the quality of the synthetic translations has an effect on detection accuracy, we extract human annotated data from the wmt shared task on news translation and group sentence pairs according to the human evaluation score. it should be noted that most categories either distort the source or the target sentence of a pair. we keep this separation for our analysis even though many data filtering systems do not distinguish the languages direction, i.e. an @xmath0. hence, differences between the src-and trg-version of a noise category should best be seen as an indication of experimental variance or a dependency on the resources available for the two sides of a language pair. ## effect of noise on nmt performance in this work we solely focus on the question how well certain types of noise can be detected by modern data filtering systems. however, an equally important question is, how detrimental a certain noise type is to the performance of an nmt system. in this section we briefly recapitulate the findings of @xcite , ranging the different types of noise according to their effect on nmt performance. according to @xcite , the most severe noise type is the untranslated (trg) category. mixing just 20% of this type of noisy data into our clean training data results in an nmt performance drop to less than 10% bleu, completely destroying translation performance. the authors explain this with the system learning to copy sentences rather than translating them into the correct language. the second worst type of noise is raw crawled data, followed by misaligned sentences, misordered words (src|trg) and wrong language (trg) all leading to a significant performance degradation of the nmt system. on the other hand, @xcite find that adding wrong language (src), untranslated (src) and short segments (max. length) leads to only minor performance degradation. we additionally examine two types of noise which were not present in the work of @xcite , namely synthetic translations and over-/undertranslation. the latter mostly occurs as a result of bad segmentation and/or sentence splitting. we argue that in the most extreme case, this type of noise would coincide with the misaligned sentences so the impact of this category can be seen as an upper bound. in principle synthetic translations can be beneficial for nmt performance @xcite @xcite . however, this depends heavily on the quality of the system used to generate this synthetic data and we argue that the purpose of web crawling is not to extract synthetic translation from possibly older machine translation models. hence, we typically want to remove such samples. therefore in this work we examine both the ability of the filtering systems to differentiate between good and bad synthetic data as well as to differentiate between synthetic and real parallel data. we also point out that removing noise from the training data will have other benefits, aside from improved performance, such as faster convergence and less storage needs. ## conclusion the aim of this work is to determine how well stateof-the-art data filtering systems can detect different types of noise common in parallel machine translation datasets. we create synthetic noisy datasets for all noise categories defined by @xcite as well as for additional noise types that we define in this work. we find that modern data filtering systems can detect most types of noise with an accuracy of well over 90% on a @xmath1, that features a medium sized, rather clean training corpus for the filtering systems. however, well-formed but misaligned sentence pairs and over-/undertranslation can only be detected with an accuracy of less than 70%. when it comes to detecting more subtle errors like distinguishing between a good and a poor synthetic translation, the systems exhibit even worse performance. furthermore, when switching to a less common language pair, namely @xmath2, the performance of the filtering systems degrades significantly compared to @xmath3. in conclusion we find that the task of data filtering as defined by @xcite is not yet solved. there is still much room for improvement, especially when going to more subtle types of noise or to less common language pairs. for future research or when applying data filtering for a downstream task, we want to emphasize the following points: