Refining an Almost Clean Translation Memory Helps Machine Translation (2022)

Abstract: While recent studies have been dedicated to cleaning very noisy parallel corpora to improve Machine Translation training, we focus in this work on filtering a large and mostly clean Translation Memory. This problem of practical interest has not received much consideration from the community, in contrast with, for example, filtering large web-mined parallel corpora. We experiment with an extensive, multi-domain proprietary Translation Memory and compare five approaches involving deep-, feature-, and heuristic-based solutions. We propose two ways of evaluating this task, manual annotation and resulting Machine Translation quality. We report significant gains over a state-of-the-art, off-the-shelf cleaning system, using two MT engines.

## introduction major language service providers that handle huge numbers of translation requests in various domains typically call upon a large pool of translators (internal and freelance) whose translations are fed into one or several internal translation memories (tm). translators access these tms through a dedicated interface to match requested translations with previously completed ones, speeding up the translation process. given their provenance and purpose, it is normally assumed that tms are mostly exempt of incorrect translations (henceforth "noise"). unfortunately, because of the high number of translators involved, with varying levels of expertise, tight deadlines, and other technological issues, noise inevitably accumulates over the years. tm noise falls under two broad categories. first, mechanical noise, which occurs because of the pipeline used to populate the tm, whereby text is extracted from source and target documents, segmented into sentences and then aligned -all three processes producing occasional errors. second, human-induced noise, which can arise for a variety of reasons, including spelling or morphosyntax that do not meet established norms, missing translation units on the target side, as well as typical translation errors such as calques, the use of false friends, etc. these errors reduce the usefulness of a tm, and motivate our investigation into whether parallel corpus filtering methods may be useful to increase the quality of a large tm. this contrasts with the typical use of these methods, which are more commonly applied to large, very noisy bilingual corpora automatically extracted from the web (or other uncontrolled sources). in the following section (sec. 2), we discuss related work in more details. we then describe the large tm on which our experiments are based in section 3, and the corpus filtering methods we implemented in section 4. we evaluate the performance of these methods by measuring their impact on neural machine translation (nmt) in section 5, and report our results and analysis in section 6. we show that, surprisingly, significant translation quality gains can be obtained by cleaning an "already clean" translation memory. ## datasets for our experiments, we obtained access to a large english-french corporate tm from a major language service provider (lsp). from this, we extracted various datasets (see also tab. 1): tm-xl: from over 1.8m tmx files across more than 200 broad domains (e.g. health, environment), we extracted a total 139 454 913 sentence pairs (sps). in the tm system used by the lsp, translators may flag a problem with a sentence pair, which marks the entire tmx file containing it as problematic. flagged material represents 7.7% of all sps, but we don't know which sp from a flagged tmx was the cause of the problem. ## noisy sentence pair detection we compare five approaches to identify and filter out noisy sentence pairs. ## neural machine translation models we evaluate the quality of a tm using the performance of a translation engine trained on it as a task-based proxy. to reach conclusions that are independent of a specific system, we experiment with two very different neural translation models: xlm, a deep transformer model, and convs2s, a convolutional seq2seq model. typical training time on 4 tesla v100-sxm2 was 22-30 hours for xlm and 72-96 hours for convs2s, depending on the dataset. ## experimental results we now report results on noisy sp detection and its impact on machine translation. ## conclusions we explored several ways to filter a mostly clean translation memory, used daily by professional translators. we showed that (pre-trained) laser and (trained in-house) bi-lstm are able to discriminate noisy sentence pairs from clean ones with high accuracy. the former is unsupervised, delivering the best results on our small scale manual evaluation. both methods outperform heuristics devised specifically for this task, feature-based classifiers trained with supervision, as well as the tmop system which turned out to be very challenging to deploy. we also showed that filtering the noisy sps results in machine translation gains. deep learning methods allow significant gains in bleu: the bi-lstm classifier filters out over half the training material, and yields a gain of over one bleu point. future work includes better characterizing, modeling and generating subtle noise in misaligned segments, in order to build better detectors.