Data Filtering using Cross-Lingual Word Embeddings (2021)

Abstract: Data filtering for machine translation (MT) describes the task of selecting a subset of a given, possibly noisy corpus with the aim to maximize the performance of an MT system trained on this selected data. Over the years, many different filtering approaches have been proposed. However, varying task definitions and data conditions make it difficult to draw a meaningful comparison. In the present work, we aim for a more systematic approach to the task at hand. First, we analyze the performance of language identification, a tool commonly used for data filtering in the MT community and identify specific weaknesses. Based on our findings, we then propose several novel methods for data filtering, based on cross-lingual word embeddings. We compare our approaches to one of the winning methods from the WMT 2018 shared task on parallel corpus filtering on three real-life, high resource MT tasks. We find that said method, which was performing very strong in the WMT shared task, does not perform well within our more realistic task conditions. While we find that our approaches come out at the top on all three tasks, different variants perform best on different tasks. Further experiments on the WMT 2020 shared task for parallel corpus filtering show that our methods achieve comparable results to the strongest submissions of this campaign.

## introduction in recent years, neural machine translation (nmt) systems have greatly improved the quality of automatically generated translations, some argue even to the point of human parity @xcite . while there most definitely have been advancements in designing the nmt system architectures @xcite , arguably the best (and easiest) way to improve an nmt system is to use more training data. with an ever increasing amount of parallel data for nmt training, which often comes from web-crawling foot_0 and is quite 'noisy', the task of data filtering becomes increasingly important @xcite . data filtering in the context of machine translation (mt) describes a collection of approaches which select a subset of a given, possibly noisy corpus with the aim to maximize the performance of an mt system trained on this data. there exist very simple approaches, the most prominent being based on language identification tools, to detect certain types of noise, e.g. sentences that are from a wrong language. however, other types of noise are much harder to detect, for example when both source and target sentence are well formulated and in the correct language but are not translations of one another. in some formulations of the data filtering task, for example in the wmt shared task for parallel corpus filtering @xcite @xcite , the assumption is that there already exists a large amount of 'clean' data which can be used to detect bad training samples in a separated 'noisy' corpus. however, such an assumption does typically not hold true in real-life scenarios. therefore, in this work, we make no such distinction between 'known-to-be-clean' and 'noisy' data. we present novel approaches that use all the available data to filter that very same data in order to improve translation performance. in the proposed methods, we use the structure of cross-lingual word embeddings to compare the words in a given source-target sentence pair to determine if the pair is of 'good' quality. this is done in a variety of ways, including nearest neighbor search in the embedding space and an explicit calculation of alignment scores. all proposed methods are specifically designed to detect the types of noise which cannot be detected by language identification tools. furthermore, we design our approaches to not rely on the quality of the sentence pair alignments between the source and the target side of the data, since this information might be highly unreliable in a 'noisy' corpus. the main contributions of this paper are summarized below: ## related work recently, a number of shared tasks for data filtering have been held, giving a good overview of current state-of-the-art methods. best known is the wmt shared task for parallel corpus filtering, which was held in 2018 @xcite , 2019 @xcite and 2020 @xcite ) respectively. in these tasks, the participants are asked to provide scores for every sentence pair in a noisy corpus. afterwards, a fixed amount of sentence pairs is selected according to that score. the best performing submissions from past years use language identification tools as the first part of their setup @xcite @xcite , removing sentence pairs where the language of either source or target sentence does not match the expectation. @xcite and junczys-dowmunt (2018) use a combination of language model and translation model scores to sort the sentence pairs by quality. @xcite use the cosine distance between cross-lingual sentence embeddings of source and target sentence as score. @xcite estimate the quality of a sentence pair using the euclidean distance between each sentence vector and two vectors representing in-domain and out-domain data. @xcite score the similarity between source and target sentence by averaging the word-pair similarity, which is calculated from cross-lingual word embeddings. since the above mentioned methods are evaluated on different tasks with very different data conditions, one can not easily make a statement about which approach works best. however, all approaches have in common that they use 'known-tobe-clean' parallel data in order to train the models of their filtering pipeline. creating cross-lingual word embeddings from parallel and/or monolingual data is an active field of research @xcite . in addition to capturing semantic relationships within each language, these representations should be aligned in such a way that the embeddings of the same word in different languages are close together in the embedding space. the standard approach for creating such embeddings is to first train embeddings for each language pair separately @xcite and then projecting them into the same vector space @xcite , which is possible with or without the help of parallel data. word alignments between a source and a target sentence were an integral part in count-based statistical machine translation systems @xcite and it has been shown that they can be used to help certain aspects of nmt systems as well @xcite . for a long time, ibm-model-based frameworks like giza++ @xcite or fastalign @xcite produced the best word alignments. however, recently sabet et al. (2020) report equally good results by using a word similarity matrix calculated from cross-lingual word embeddings. ## detecting different types of noise applying language identification (language id) is a well established first step in most high performing data filtering approaches. during this step, all sentence pairs for which either the source or target sentence is not mapped to the correct language are discarded. it can be argued that this step does not only remove sentence pairs in the wrong language, but also that language-agnostic noise, e.g. sequences of numbers, is almost completely removed. in order to evaluate the effectiveness of the filtering by language id approach, we decide to test the method on the popular @xmath0. by manually checking the noisy corpus (see section 5.1 for details) we find different types of 'noise patterns'. for each of these 'noise patterns', we create a synthetic corpus (50k lines each), only consisting of sentence pairs with this specific noise. we find/create the following 'noise patterns': trg to src: the source and target side of a valid sentence pair are swapped. for the unrelated third language (other) we choose french. next, we use the langid.py toolkit @xcite to filter each of these synthetic corpora and check which percentage of noise (ideally 100.0%) gets removed. the results are shown in we find that the language identification filtering approach does an outstanding job in detecting noise that comes from wrong language alignment. furthermore it also removes basically all of the random noise, represented by the random digits corpus. however, we also see where this approach noise type percentage removed trg to src 100.0% trg to trg 100.0% src to src 100.0% src to other 99.5% other to trg 99.8% other to other 100.0% sentence misalign 0.0% overtranslation 7.8% undertranslation 6.7% random digits 100.0% ## data filtering methods intuitively a bilingual sentence pair is appropriate for training if a) both the source and the target sentence belong to the corresponding languages and b) they are translations of each other. we rely on established language identification methods (see section 5.1) to verify the first condition. following state of the art filtering systems @xcite we predict the language for source and target sentence and keep the sentence only if both match the requirements of the task. to check whether the sentences of a training pair (f j 1 , e i 1 ) are indeed translations of each other we propose several approaches based on cross-lingual word embeddings. for the details of how the cross-lingual word embeddings are constructed we refer to section 5.1. here we assume that we are given a cross-lingual word embedding e : v src ∪ v @xmath1. for convenience we use e w := e(w). in practice all embedding vectors are length normalized, i.e. ||e w || = 1. ## conclusion in this work we focus on data filtering for machine translation. we define this task as the selection of a subset of a given, possibly noisy corpus, without the help of additional large-scale 'clean' corpora. in order to develop a helpful filtering method, we first analyze the commonly used 'filtering by lan-guage identification' approach by applying it to synthetically generated noisy data. we find that while 'filtering by language identification' does an outstanding job in detecting noise that comes from wrong language alignment, it fails to detect noise resulting from a semantic mismatch between source and target sentence. building on these findings, we develop several approaches -based on cross-lingual word embeddings -specifically targeting the word alignments between source and target sentence. furthermore, we conduct a systematic comparison on data selection methods in an effort to uncouple the scoring and selection parts of any data filtering pipeline. we compare our approaches to one of the winning methods from the wmt 2018 shared task on parallel corpus filtering on three real-life, high resource tasks as well as on the recent wmt 2020 shared task on parallel corpus filtering. we find that the existing approach does not perform well in our more realistic scenario, leading to a degradation in performance in most cases. our methods result in improvements over the baseline on all three three tasks. however, different variants of our methods perform best on different tasks and we can not identify a single best approach. finally, we compare our methods to state-ofthe-art data-filtering systems on the wmt 2020 shared task on parallel corpus filtering. here, our proposed approaches yield comparable results to aforementioned state-of-the-art methods while not relying on any parallel training data.