There’s No Data like Better Data: Using QE Metrics for MT Data Filtering (2023)

Abstract: Quality Estimation (QE), the evaluation of machine translation output without the need of explicit references, has seen big improvements in the last years with the use of neural metrics. In this paper we analyze the viability of using QE metrics for filtering out bad quality sentence pairs in the training data of neural machine translation systems (NMT). While most corpus filtering methods are focused on detecting noisy examples in collections of texts, usually huge amounts of web crawled data, QE models are trained to discriminate more fine-grained quality differences. We show that by selecting the highest quality sentence pairs in the training data, we can improve translation quality while reducing the training size by half. We also provide a detailed analysis of the filtering results, which highlights the differences between both approaches.

## introduction in the times of statistical machine translation, a well-known motto was "there's no data like more data". experimental results seemed to confirm this, with the performance of the systems steadily improving as more data was made available. web crawling has proven to be a valuable source of data for training translation systems, with projects like common crawl 1 or paracrawl @xcite providing numerous parallel sentences with which to train mt systems. inevitably, when crawling huge amounts of data, noise will be present. taking the web as an example, the quality of available texts varies greatly between websites. there are sources which reliably produce high-quality text, e.g. large circulations newspaper websites usually contain text written and proof-read by professional journalists. but by its open nature, the web also contains texts of dubious quality (both in style and in content) which may pollute the collected texts. when considering bilingual data collection, an additional difficulty comes into play, namely the alignment of segments between two or more languages. sentence alignment algorithms @xcite @xcite are bound to make mistakes, resulting in pairing of sentences that are not necessarily translations of each other. even if the correspondence between sentences may be correct, the quality of sentences may differ greatly between languages. while one source may provide high quality text in its original language, the available translations may be of sub-par quality due to a variety of reasons @xcite . in addition, given the increased availability (and quality) of machine translation engines, mt output is expected to be part of the crawled data, thus contaminating the training material. statistical systems were robust against such type of noise @xcite . the maximum likelihood estimators of phrase probabilities (and related models) were based on relative frequencies, with the consequence that noisy translation units, while being available to the system at translation time, had a low chance of being used. in fact, works on filtering data dealing with statistical systems, e.g. @xcite were more concerned with the efficiency of the systems, rather than with quality. with the advent of neural machine translation, the situation has changed, and the quality of the data has a major impact in the resulting quality of the translation system. neural networks have a great ability of memorizing (parts of) the training data @xcite . whereas for phrase-based models the noise was diluted in the abundance of better-quality data, in neural models such outliers may have a critical effect on the output of the system. therefore data filtering has become increasingly important, even spawning dedicated shared tasks @xcite @xcite . research in this area has allowed nmt systems to take advantage of big amounts of training data. as an example, initial versions of paracrawl degraded translation quality when adding them unfiltered to the training data of mt systems @xcite , whereas nowadays it is one of the main data sources used in wmt for the languages where it is available. thanks to these filtering techniques, huge amounts of parallel sentences are available for several languages, in some cases reaching up to hundreds of millions of sentences. but as noted above, not all texts are of the same quality. it is only natural to ask the question, once the training data reaches a size which is "big enough", if all the available text is useful for training nmt systems, or if the lower quality sentences are hindering the system. note that most data filtering systems are focused in detecting noise or problems in the translation (e.g. under-or overtranslation). in very wide strokes, most systems answer the question "is sentence a a translation of sentence b?", without looking (too much) into the quality. judging the quality of translations is the focus of the quality estimation (qe) field of machine translation. it can be considered an extension of machine translation evaluation, where references are not available. in the last years, the use of neural models has improved the results in this area dramatically. would it then be possible to use quality estimation methods for filtering data and improve the quality of a neural machine translation system, which has been trained on already cleaned data? in other words, can we do a more fine-grained data selection beyond discarding "obvious" errors, focusing on selecting the best data that can be found in the training corpus? we explore these questions in this paper. our scientific contributions are: ## related work already in the early days of the popularization of statistical methods for machine translation, the potential of mining data from the web was recognized by @xcite . in contrast to other data sources at the time (e.g. canadian hansards, european parliament proceedings) which consisted largely of clean data, the necessity of including additional "quality assurance" steps were recognized in this work. as pointed out above, statistical systems were robust against noisy input data @xcite , and as such, the topic of "corpus filtering" was mainly focused on selecting subsets of data closer to a given domain @xcite . nevertheless @xcite shows that statistical systems may also benefit from careful curation of the training data. the situation changed dramatically with the advent of neural machine translation, as such systems are much more sensitive to noisy input data @xcite . a clear reflection of this fact was the creation of a new dedicated shared task in the wmt yearly conference @xcite @xcite in the years 2018 to 2020. 2 junczys-dowmunt (2018a) was the best performing system in the first edition of the filtering shared task, using a cross-entropy approach between two translation systems trained on clean data. in the next edition, the focus was moved towards low resource conditions. that year @xcite presented the best performing system, using a system based on laser embeddings @xcite . the 2020 edition continued the focus on lowresource languages. at that evaluation, three were the best performing systems: @xcite and @xcite both use pre-trained multilingual models as a key component of their filtering systems. @xcite used an improved version of bicleaner, their submission for the 2018 campaign @xcite . the authors further improved their system @xcite , including an extension using neural models @xcite . this latest version is considered state-of-the art and it is that which we take as baseline for comparing our method. of course, this is just but a very rough overview of the best performing systems in each evaluation campaign. we refer to the reports of each campaign for a more detailed overview of the methods explored each year. @xcite provide one more recent overview of data filtering methods. in this work, the authors sample 5m sentences from original training data and added 1m noise samples manually. they show that a two-stage approach can be beneficial for improving the quality of a translation system. all of these methods have a common focus on detecting the type of noise that may originate from crawled data. this type of noise has been analyzed in @xcite , and @xcite build on their work and carry out a comparison of the efficiency of different filtering methods on various types of noise. @xcite also provide an extensive analysis on the noise present in several widely used corpora. most similar to our work @xcite start with already clean data and analyze the effect of semantic divergence on translation quality. they are able to effectively select a subset of the training data and improve translation quality measured in bleu. bernier-colborne and @xcite use yisi-2, also a quality estimation metric as a component in their corpus filtering system for the wmt 2019 shared task. @xcite extend this idea by including bert (word) alignments in the yisi pipeline. we follow a conceptionally similar approach to these papers, using state-of-the-art qe metrics and provide a more in-depth comparison to other corpus filtering methods more oriented towards noise detection. quality estimation is again its own area of research, with dedicated shared tasks, e.g. @xcite , that measure how well metrics can predict word-and sentence-level quality scores. in contrast to traditional mt evaluation, qe aims to assess the quality of the output texts without the use of a reference translations. the most successful qe metrics learn to jointly predict word-and sentencelevel scores, like cometkiwi @xcite . another possibility is to modify the input to a learned reference-based metrics like bleurt @xcite or comet @xcite to use the source segment instead of a reference translation to predict sentence-level quality scores @xcite . we follow the latter approach and train a qe version of bleurt that predicts sentence-level quality scores (see section 3) that are used for data filtering. ## from bleurt to bleurtqe the qe metric that we propose for data filtering is a learned mt evaluation metric that is based on a bleurt-style architecture @xcite . bleurt is a reference-based regression metric that is trained to predict a quality score for a hypothesis translation given a reference. the hypothesis and reference are concatenated together with a special token in between, then fed as input to the metric, which predicts a floating point quality score. our qe metric is a modification of the original model. to make it a qe metric, we pass the source segment as input to the metric instead of the reference. then, we follow the winning submission to the wmt'22 metrics shared task @xcite , metricx, and use a modified version of the mt5 encoder-decoder language model @xcite as our network architecture. not that these is a multilingual model, so the same system can be used for a variety of languages. the source and hypothesis are passed as input to the encoder, and an arbitrary logit from the first step of the decoder is trained to predict the hypothesis quality score. the qe metric is trained on the direct assessment quality judgments that were collected as part of the wmt metrics shared task from 2015-2020 @xcite @xcite @xcite for all available language pairs. to (meta-)evaluate the metric we measure its correlation with ground-truth translation quality ratings using the benchmark mqm dataset from wmt'22 @xcite that includes 3 language pairs: en-de, zh-en, and en-ru. since our metric is used to score individual segments and not systems, we report the segment-level correlation between our metrics' scores and the gold mqm scores using pearson's r and kendall's τ , shown in a (more refined) version of this metric has been submitted to this year's qe shared task @xcite , and has been open sourced. we refer the reader to the system description for a more fine-grained discussion of the details of the metric. @xcite , unite-src @xcite . ## experiments we report experiments on three language pairs: english ↔ german, japanese ↔ english and chinese ↔ english. our starting point is the full training data as provided by the wmt evaluation campaign. corpus sizes can be found in for each language pair we will consider different ways to reduce the size to 50% of their original size. this value was chosen in preliminary experiments on the english to german data, and it is comparable to previous work @xcite . fixing the target size beforehand also allows a fair comparison between all the methods. ## analysis in this section we provide an in-depth analysis of the differences between the bleurtqe-based and the bicleaner filtering methods. ## conclusions in this paper we have shown that filtering data using qe metrics is an effective way of improving translation quality. in contrast to "traditional" data filtering methods that focus on detecting noise in the data, qe methods focus on selecting the best translation examples. analyzing the differences between the two different methods, we see that qe metrics are not as effective at detecting certain types of noise, e.g. untranslated sentences, but are much better at identifying more fine grained problems in the data, like small translation errors or grammatical mistakes. therefore, when starting with already cleaned data, we can obtain a boost in performance by focusing the nmt system training on the best sentences. our results show that the improvements obtained generalize across different domains, as measured by a variety of metrics. even for more distant domains, like the acl talks of of the iwslt'23 corpus, the performance of the systems remains largely constant. qe estimation is a very active field of research. using this approach, the improvements obtained in this area can have a direct impact on improving the quality of nmt systems.