Dynamic Adaptation of Neural Machine-Translation Systems Through Translation Exemplars (2022)

Abstract: This project aims to study the impact of adapting neural machine translation (NMT) systems through translation exemplars, determine the optimal similarity metric(s) for retrieving informative exemplars, and, verify the usefulness of this approach for domain adaptation of NMT systems.

## introduction translation exemplars, i.e. previously observed or generated translations, whose source is similar to new input, play an important role in the translation process by providing explicit information on context, exceptions, and irregularities, which are difficult to generalise. in the context of machine translation (mt), the information provided by similar translations can also be considered complementary to the neural models, which are good at making generalisations. "dynamic adaptation of neural machine-translation systems through translation exemplars" is a three-year, post-doctoral research project (october 2020 -september 2023), funded by the research foundation -flanders (fwo) with the following scientific objectives: (i) study the impact of adapting nmt systems through similar translations retrieved from tms, (ii) determine the optimal metrics for measuring translation similarity, and (iii) verify the usefulness of this approach for domain adaptation. all research activities are carried out at the department of translation, interpreting and communication, ghent university, belgium. the project has been built upon the methodology used in the translation memory (tm) -nmt integration approach, neural fuzzy repair (nfr) (bulté and tezcan, 2019). @xcite in nfr, each source sentence in the tm is augmented (concatenated), with the translation of the most similar fuzzy match (fm), retrieved from the same tm using edit distance, when at least one fm is found above the minimum similarity threshold of λ. the augmented tm is merged with the original tm to train an nmt model. during inference, the same technique is applied to augment source sentences prior to obtaining translations from the trained model. if no fms above the minimum similarity threshold can be found, the translations are obtained for the original source sentence, using the same model. tests on multiple language pairs showed that this method results in substantial gains in translation quality compared to (i) baseline mt systems, (ii) the fms themselves when used as final output, even when they correspond to nearperfect translations in high similarity ranges, and (iii) a 'fuzzy match repair' approach, which relies on editing highly similar fms to arrive at the final translation (bulté et al., 2018). these experiments also demonstrated that using the nfr system starts being advantageous with a minimum fm score of λ = 0.5, and augmenting source sentences with fms of higher similarity scores leads to higher translation quality. while this study showed the usefulness of adapting nmt systems through similar translations, it also led to new research questions and formed the scientific objectives of this project. the following sections provide an overview of the progress made on the first two scientific objectives and summarise the plans for future work. ## optimal metrics for measuring translation similarity text similarity can be measured from different perspectives, such as using string-, semantic-and syntactic-level information. in the context of nfr, it is yet to be determined if, and to what extent, different similarity levels are important in retrieving informative fms and producing better translations. to seek optimal similarity metrics in the context of nfr, string-and semantic-similarity metrics have been studied in combination with different sub-word segmentation methods (tezcan et al., 2021). retrieving fms by measuring cosine similarity between sentence embeddings resulted in translations with higher quality in comparison to using edit distance. moreover, applying sub-word segmentation prior to measuring semantic similarity improved translation quality further. to utilise sub-sentence-level similarities between two sentences more explicitly, two additional approaches have been tested: (i) marking relevant tokens in retrieved fms, and (ii) augmenting source sentences with multiple fms that lead to a maximum coverage of source tokens (tezcan et al., 2021). when combined, these methods led to improvements in estimated translation quality for 8 language directions (english ↔ dutch, french, hungarian, polish), compared to a baseline transformer nmt model and the original nfr approach (bulté and tezcan, 2019). ## the impact of adapting nmt systems through similar translations to analyse the impact of adapting nmt systems through similar translations on translation quality, evaluations were carried out both automatically and manually. @xcite the fine-grained, human error analysis showed that both the adapted and non-adapted nmt systems made a comparable number of errors. on the other hand, while the adapted nmt system produced more fluent translations, with a significant reduction in lexicon and coherence errors, it also diverged from the source content and meaning (i.e. reduced accuracy) more often than the base-line nmt system, making more errors of addition and mistranslation. the automatic evaluation analysed translation quality by relying on the reference translations while using metrics that targeted lexical, sublexical, semantic, and syntactic aspects of translation quality. according to all evaluation metrics, the quality of the adapted nmt system was estimated to be higher than that of those produced by the nmt system. the difference in quality was always significant and large confirming that the improvements were obtained for all the different aspects of quality that were analysed. by analysing the mt output with different resources (source-mt vs. mt-reference), the two evaluation methods yielded a more nuanced picture of the differences in translation quality between the two systems. they also revealed an interesting property of the nfr system: by using the similar translations, it was able to learn systematic deviations from the source text (e.g. related to the use of cohesive devices or translation decisions affecting sentence boundaries) and produce translations that are similar to the reference translations, even though such deviations were marked as accuracy errors during human error analysis.