M 3 TRA : integrating TM and MT for professional translators (2018)

Abstract: Translation memories (TM) and machine translation (MT) both are potentially useful resources for professional translators, but they are often still used independently in translation workflows. As translators tend to have a higher confidence in fuzzy matches than in MT, we investigate how to combine the benefits of TM retrieval with those of MT, by integrating the results of both. We develop a flexible TM-MT integration approach based on various techniques combining the use of TM and MT, such as fuzzy repair, span pretranslation and exploiting multiple matches. Results for ten language pairs using the DGT-TM dataset indicate almost consistently better BLEU, METEOR and TER scores compared to the MT, TM and NMT baselines.

## introduction while software for professional translators has included translation memories (tms) since several decades, especially in the context of specialized documents, the use of machine translation (mt) in such software is more recent. even though certain commercial translation tools now offer functionalities such as automatic fuzzy match repair, tm and mt technologies are often still used independently, i.e. either a match for a query sentence or an mt output is provided. this is not ideal, as translators tend to have a higher confidence in 'human' tm than in mt. it has to be kept in mind, however, that only exact matches provide a transc 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. lation of the query sentence; 'fuzzy' matches offer a translation of a similar sentence. in contrast, mt systems provide a translation for any sentence, but they have problems with a number of, often linguistic, issues, such as complex morphological phenomena, long distance dependencies and word order @xcite . we investigate how to combine the confidence in fuzzy match retrieval with full sentence translation by integrating tm and mt output. we develop m3tra, 1 a method which performs a tm match preprocessing step before running a standard phrase-based statistical mt (pb-smt) system trained on the tm. m3tra combines different approaches, and is flexible in several respects: it applies various fuzzy match score thresholds, allows for more than one match to be used per query sentence, and can use several fuzzy metrics. it comprises two main components: (a) fuzzy repair, automatically editing high-scoring fuzzy matches, and (b) span pretranslation, constraining mt output by including certain consistently aligned spans of one or more tm matches. we perform tests on ten language pairs which involve multiple language families, using the dgt-tm dataset @xcite . we apply pbsmt without span pretranslation as a baseline, as well as 'pure' tm and a standard nmt system, and evaluate the translations using several metrics. m3tra is integrated in a prototype translation interface providing translators with more 'informed' mt output @xcite . the following sections describe the research context, system architecture, experimental design and results. the final sections contain a discussion, overview of work in progress and conclusions. the baseline approach to tm-mt integration uses mt to translate a query sentence in case no sufficiently similar translation unit is found in the tm @xcite . this can be augmented by using an estimation of the usefulness of mt and tm output @xcite . other studies focus on correcting close matches from a tm using pbsmt, based on a set of learned edit operations @xcite . @xcite propose a patching approach to correct tm matches with any kind of @xcite a more translatororiented method that offers word keeping recommendations based on information coming from an mt system. example-based mt systems have also been used to leverage sub-segmental tm data @xcite . of particular relevance are approaches that constrain a pbsmt system to use relevant parts of a fuzzy match @xcite , for example by adding xml markup to moses input @xcite @xcite or by using a constrained word lattice @xcite . related to these are methods that augment the translation table of a pb-smt system with aligned spans from a retrieved tm match, yet without forcing the smt system to incorporate (parts of) these aligned spans @xcite . alternatively, information from the fuzzy matches can also be integrated in the smt system itself @xcite , for example using sparse features @xcite . recent studies focus on how to leverage tm information for nmt systems. these approaches work, for example, by imposing lexical constraints on the search algorithms used by nmt @xcite , by augmenting nmt systems with an additional lexical memory @xcite , or by explicitly providing the nmt system with access to retrieved tm matches @xcite . m3tra combines different elements from these approaches, which is its main novelty. in this paper we focus on (a) repairing close fuzzy matches, and (b) augmenting the mt input with information derived from the parallel corpus (the tm) used to train the mt system, thus constraining the translation of certain (parts of) sentences. we use a pbsmt system as basis for tm-mt integration because smt allows a straightforward ap-plication of pretranslation (e.g. explicit alignment information is used in the process). ## system architecture m3tra consists of four components: (a) a tm system, (b) a pbsmt engine, (c) a system for fuzzy repair (fr) and (d) a system for pretranslation span search (pss). we elaborate on each of these components in the following sections. the sentence to translate can follow a number of routes, depending on the fuzzy match score of the best retrieved match and the success or failure of certain attempted operations (see figure 1 ). first, fr is attempted for sentences that have at least one match which meets the relevant threshold (θ f r ). if fr is performed, it may modify the translation of the fuzzy match by deleting, inserting or substituting words. in case fr is not performed or fails, there are three options: (a) if the score of the highest match satisfies the tm threshold (θ t m ), the translation of the tm match becomes the final output, (b) if the score is between the tm and mt thresholds, pss is attempted, and (c) if the score is below the mt threshold (θ m t ), or pss fails (i.e. the query sentence as such becomes input to mt), the 'pure' mt output is used as final output. each of the four m3tra components is described in detail below, followed by an overview of the parameter tuning process. ## experimental design this section describes the empirical tests that were carried out. we first describe the dataset and evaluation procedures, before turning to the results. ## discussion the main novelty of m3tra is in its adaptable parameters, threshold values and safeguards, as well as in its combination of various features that are present in a number of approaches described in section 2. most notably, the use of xml markup to add pretranslation spans to input sentences is also used by @xcite , @xcite and @xcite . in m3tra, moses is constrained to include these pretranslated spans in the final output (the so-called exclusive mode is used). the fuzzy repair feature is closely related to the work of @xcite . also the option to simply use tm target matches above a certain match score threshold has been implemented before @xcite . moreover, by making use of the information obtained during the alignment process, m3tra can be adapted easily to provide translators with information on the origin of parts of the proposed translations, possibly indicating which sentences should most likely be post-edited @xcite . finally, the combination of information from different fuzzy matches is also present in previous research @xcite . the test results show that integrating tm with mt can lead to better mt output, provided that sufficient high-scoring matches are retrieved from the tm. we argue that m3tra is especially beneficial in a context with enough repetition and where the focus is (at least to a certain extent) on consistency and formulaic language use. looking at the results for the different language pairs, the potential for improvement is highest for en-hu and hu-en, foot_12 which is most likely due to the (morphological) structure of the hungarian language and its associated problems for (s)mt. the significant improvements for almost all language combinations indicate that m3tra potentially works with different language families (germanic, romance, finno-ugric). the smallest improvement was found for the only slavic language we tested (polish). with regard to the relatively low scores obtained by our nmt baseline, a number of comments are in order. first, we only tested certain standard/recommended settings in opennmt. it is likely that higher scores can be reached by tuning other nmt hyperparameters to better fit the dataset used. second, smt uses bleu scores as tuning criterion, whereas in nmt perplexity is ## conclusions we designed and tested a system for the integration of mt and tm, m3tra, with a view to increasing the quality of mt output. m3tra contains two main components, fuzzy repair and span pretranslation, which both make use of a tm with fuzzy matching techniques and an smt system with related alignment information. the system uses the option to add xml markup to sentences sent to a moses smt system. tests on ten language combinations using the dgt-tm dataset showed that it is clear that this approach has potential. significantly higher bleu scores for 9 of the 10 language combinations were observed, and meteor and ter scores showed comparable patterns. in a next step, m3tra has to be evaluated in an actual translation environment involving professional translators.