Boosting Neural Machine Translation with Similar Translations (2022)

Abstract: This presentation demonstrates data augmentation methods for Neural Machine Translation to make use of similar translations, in a comparable way a human translator employs fuzzy matches. We show how we simply feed the neural model with information on both source and target sides of the fuzzy matches, and we also extend the similarity to include semantically related translations retrieved using distributed sentence representations. We show that translations based on fuzzy matching provide the model with “copy” information while translations based on embedding similarities tend to extend the translation “context”. Results indicate that the effect from both similar sentences are adding up to further boost accuracy, are combining naturally with model fine-tuning and are providing dynamic adaptation for unseen translation pairs. Tests on multiple data sets and domains show consistent accuracy improvements.

## introduction for decades, the localization industry has been proposing fuzzy matching technology in cat tools allowing the human translator to visualize one or several fuzzy matches from translation memory when translating a sentence leading to higher productivity and consistency @xcite . hence, even though the concept of fuzzy match scores is not standardized and differs between cat tools @xcite , translators generally accept discounted translation rate for sentences with "high" fuzzy matches 1 . with improving machine translation technology 1 https://signsandsymptomsoftranslation . com/2015/03/06/fuzzy-matches/. and training of models on translation memories, machine translated output has been progressively introduced as a substitute for fuzzy matches when no sufficiently "good" fuzzy match is found and proved to also increase translator productivity given appropriate post-editing environment @xcite . these two technologies are entirely different in their finality -indeed, for a given source sentence, fuzzy matching is just a database retrieval and scoring technique always returning a pair of source and target segments, while machine translation is actually building an original translation. however, with statistical machine translation, the two technologies are sharing the same simple idea about managing and retrieving optimal combination of longest translated n-grams and this property led to the development of several techniques like use of fuzzy matches in smt decoding @xcite , adaptive machine translation @xcite or "fuzzy match repairing" @xcite . with neural machine translation (nmt), the integration of fuzzy matching is less obvious since nmt does not keep nor build a database of aligned sequences and does not explicitly use n-gram language models for decoding. the only obvious and important use of translation memory is to use them to train an nmt model from scratch or to adapt a generic translation model to a specific domain (fine-tuning) @xcite . while some works propose architecture changes @xcite or decoding constraints @xcite ; a recent work @xcite has proposed a simple and elegant framework where, like for human translation, translation of fuzzy matches are presented simultaneously with source sentence and the network learns to use this additional information. even though this method has showed huge gains in quality, it also opens many questions. in this work, we are pushing the concept further a) by proposing and evaluating new integration methods, b) by extending the notion of similarity and showing that fuzzy matches can be extended to embedding-based similarities, c) by analyzing how online fuzzy matching compares and combines with offline fine-tuning. finally, our results also show that introducing similar sentence translation is helping nmt by providing sequences to copy (copy effect), but also providing additional context for the translation (context effect). ## translation memories and nmt a translation memory (tm) is a database that stores translated segments composed of a source and its corresponding translations. it is mostly used to match up previous translations to new content that is similar to content translated in the past. assuming that we translated the following english sentence into french: [how long does the flight last?] ↝ [combien de temps dure le vol?]. both the english sentence and the corresponding french translation are saved to the tm. this way, if the same sentence appears in a future document (an exact match) the tm will suggest to reuse the translation that has just been saved. in addition to exact matches, tms are also useful with fuzzy matches. these are useful when a new sentence is similar to a previously translated sentence, but not identical. for example, when translating the input sentence: [how long does a cold last?], the tm may also suggest to reuse the previous translation since only two replacements (a cold by the flight) are needed to achieve a correct translation. tms are used to reduce translation effort and to increase consistency over time. ## results we compare our baseline model, without augmenting input sentences, to different augmentation formats and retrieval methods. our base model is built using the concatenation of all the original corpora. all other models extend the original corpora with sentences retrieved following various retrieval methods. it is worth to notice that extended bitexts share the target side with the original data. ## conclusions and further work this paper explores augmentation methods for boosting neural machine translation performance by using similar translations. based on "neural fuzzy repair" technique, we introduce tighter integration of fuzzy matches informing neural network of source and target and propose extension to similar translations retrieved from their distributed representations. we show that the different types of similar translations and model fine-tuning provide complementary information to the neural model outperforming consistently and significantly previous work. we perform data augmentation at inference time with negligible speed overhead and release an open-source toolkit with an efficient and flexible fuzzy-match implementation. in our future work, we plan to optimise the thresholds used with the retrieval algorithms in order to more intelligently select those translations providing richest information to the nmt model and generalize the use of edit distance on the target side. we would also like to explore better techniques to inject information of small-size n-grams with possible convergence with terminology injection techniques, unifying framework where target clues are mixed with source sentence during translation. as regards distributed representations, we plan to study alternative networks to more accurately model the identification and incorporation of additional context.