Integrating Domain Terminology into Neural Machine Translation (2020)

Abstract: This paper extends existing work on terminology integration into Neural Machine Translation, a common industrial practice to dynamically adapt translation to a specific domain. Our method, based on the use of placeholders complemented with morphosyntactic annotation, efficiently taps into the ability of the neural network to deal with symbolic knowledge to surpass the surface generalization shown by alternative techniques. We compare our approach to state-of-the-art systems and benchmark them through a well-defined evaluation framework, focusing on actual application of terminology and not just on the overall performance. Results indicate the suitability of our method in the use-case where terminology is used in a system trained on generic data only.

## introduction high out-of-the-box quality for neural machine translation @xcite has boosted the adoption of automatic translation by the industry and invigorated the research and development on domain adaption and integration of technology in human translation workflows. for instance, combination with translation memories @xcite , terminology handling @xcite , interactive translation @xcite , post-editing modelling @xcite or dynamic adaptation @xcite are all different techniques to make machine translation part of real-life localization workflow. in this work, we focus on integrating terminology as a quick way to dynamically specialize a translation to a specific domain. terminology is a key high quality asset maintained by language specialists as part of a translation project: it is a way to guarantee language consistency, certify translation accuracy and define constraints to human translation. terminologists are putting a lot of effort to describe terms, including their morphology, their syntax, the semantic context in which these terms apply, etc. from a human perspective, even though presentation and usage of dictionaries have evolved from ontology (as found in paper dictionary) to corpus-based presentation, looking up terms in a dictionary is the ultimate point of reference for validating the correct term for a specific domain in a specific context. terminology resources with all their sophistication have been the core building bricks and a continuous challenge to acquire in volume @xcite for rule-based engines. at the other extreme, they have been reduced to corpus or aligned "phrases" @xcite for statistical machine translation approaches, missing most of their intrinsic linguistic properties. in contrast, neural machine translation operates on word and sentence representations in a continuous space so, on one hand, has access to deep actual linguistic knowledge @xcite and demonstrates a huge ability to generalize. but on the other hand, results are more difficult to interpret @xcite , and subsequently the translation process is far more complicated to control. therefore, as for several other linguistic annotations, the challenge is how terminological information can be "passed" to the model. in this work, we extend existing work on terminology adaptation, show similarity with translation memory, and propose a new approach and new benchmark through a well-defined evaluation framework focusing on actual application of terminology and not just on the overall performance. ## related work in recent years there has been significant work proposing methods to integrate such external specialized terminologies into nmt models. mainstream techniques to tackle this challenge can be divided into three broad approaches, each showing different levels of performance when facing terminology injection issues, mainly inference overhead and generalization power. we illustrate their particularities on a common scenario using two english-spanish terminology entries: [precedent ; precedente] (noun) and [to extend ; ampliar] (verb), in the following translation: these precedents can be extended. se pueden ampliar esos precedentes. placeholders incorporate non-terminal tokens into nmt systems, which requires modifying the preand post-processing of the data, and training the system with data that contains the same placeholders which occur in the test sets @xcite . following our example, source and translation terms appearing in the sentence pair are replaced by placeholders foot_0 : @xmath0. se @xmath1. the previous sentence pair is then used to feed the translation network, which learns to produce the target sentence with the corresponding placeholders. a similar workflow is applied in inference. firstly, pre-processing work replaces found source terms and their morphological variants by placeholders (precedents ; <term#1> and extended ; <term#2>). secondly, post-processing work is applied over the nmt output, where in turn translation terms replace placeholders (<term#1> ; precedentes @xmath2; ampliar) foot_1 . note that the network loses any possibility to model the tokens in the terminology, since it has only access to placeholders. the method also lacks flexibility, as the model will always replace the placeholder with the same phrase irrespective of grammatical context foot_2 . in contrast, no computational overhead is applied at inference time by pre-and post-processing. the approach inherits from @xcite where words translated as out-of-vocabulary by the nmt network are post-processed using a dictionary. learning to apply constraints tackles the same problem by learning a copy behaviour of terminology at training time @xcite . the nmt model is trained to incorporate terminology translations when they are provided as additional input in the source sentence. terminology translations are inserted as inline annotations, expecting the model to learn that additional words must be copied in the target hypothesis. the authors insert terminology translations in the source sentence either by appending the target term to its source version, or by directly replacing the original term with the target one. both alternatives obtain similar translation accuracy results. an additional input stream is also used to signal the switch between source text and target terminology to be copied. additional factors contain three values: 0 for source words, 1 for source terms, and 2 for target terms foot_3 : se pueden ampliar esos precedentes . this approach uses a generic nmt architecture which learns to use an external terminology provided at run-time, thus, showing no inference overhead. however, similarly to the preceding approach, it lacks generalization power as it simply "copies" the term found in the terminology base in the source sentence, irrespective of the target hypothesis context. @xcite argue that in some cases the approach exhibits the ability to inflect translation terms. recently, @xcite and @xcite followed a similar methodology where source sentences are augmented with entire translation sentences retrieved from translation memories, using fuzzy matching. results show that the model acquires the ability to reuse the appended translations when producing its own hypotheses. the authors show impressive translation accuracy improvements when sufficiently large fuzzy matches exist in translation memories. constrained decoding enforces translation terms as decoding constraints applied at inference. among others, @xcite introduced grid beam search (gbs), an algorithm which employs a separate beam for each lexical constraint (translation term) aiming at ensuring the apparition of each given constraint in the translation hypothesis. the algorithm explores all possible constraints at each time-step, making sure not to generate a constraint that has already been generated in previous timesteps. the approach generates all the constraints in the final output. other works @xcite @xcite attempt to reduce the computational problem caused by using multiple beams in the inference, a well known weakness of this approach. similar to the previous approach, constrained decoding does not consider target context when inserting translation terms, as it sets the target form and then produces a target context that fits this constraint. however, in a more realistic scenario, a source term may have multiple translation term inflections among which the mt engine should on-the-fly select the best one depending on the source and target context. previously, @xcite proposed a guide mechanism to enhance an nmt network with the ability to prioritize translation options presented in the form of xml annotations of source words. the mechanism is applied at every inference time-step, where the beam search is influenced with external suggestions coming from the attention model. similarly, @xcite exploit a search engine to retrieve sentence pairs whose source sides are similar with the input sentence, from which they collect translation pieces. then, the nmt model is modified to give an additional bonus to output sentences that contain the collected translation pieces. ## terminology injection this work builds on the placeholder method presented above. we extend the approach and adapt it to cover a wider variety of cases, and to control morphology to allow generalization power. to represent terminology we use several placeholders indicating part-of-speech (pos) and morphological information, both in source and target sides. 5 for each source-target term pair, we encode all possible inflections of the source and target word labelled with inflection type. not only does this analysis enable to lexically match any inflected form of the source term, but it can also produce any inflected form of the translation term, ensuring full flexibility in the inflection choice made by the neural network. the model can then learn to translate a sequence of dedicated placeholders in source by a corresponding sequence of placeholders in target, this way providing the post-process with enough information to choose the right form among the multiple ones available for each translation term, thus ensuring the correct grammatical inflection in inference. consider the previous example with extended placeholder annotations: @xmath3. se @xmath4. a challenging case concerns homographs like the word precedents above. source-side annotations indicate the homograph that can occur as a noun or an adjective, inflected in plural form. we also find it useful to convey in the source some information about the target word, namely that it is masculine, for the model to better integrate it in translation (article, agreement,...). the second term extended is unambiguously a verb in past participle. target-side annotations indicate that in the context of this example, the homograph precedents translates into spanish as a noun in plural while the second term extended translates into a verb in infinitive tense. annotations vary according to the language pair. for example, to control inflections in english-spanish, we annotate the following properties of each pos category: in english (source) in spanish (target) noun -number: s/p -number: s/p -gender of the spanish noun: m/f -gender for some nouns like careers : m/f adj -whether the spanish adjective -number: s/p precedes the noun (+leftadj) foot_5 -gender: m/f verb -tense: infinitive (w), present (p), ... -tense: infinitive (w), present (p), past part. (k), ... -person: 3 -person: 1/2/3 -number: s foot_6 -number: s/p homograph terms require morphological information for all possible pos categories. this is further illustrated here with the homograph close, for which the model, after seeing enough examples, will learn to disambiguate between [close ; final] (noun), [close ; próximo] (adjective) and [close ; cerrar] (verb). the table below illustrates three english-spanish translation examples with the word close assigned to a different pos category. each example shows: the english sentence note that our approach does not require performing any linguistic annotation in inference. all annotations are already compiled in the terminology base acquired from specialized data. following with the example, the word close triggers the use of the terminology placeholders: close ; <nnp a v#1> <s @xmath5, indicating that close is considered in our specialized terminology either as a noun, a verb or an adjective (b). the nmt network then produces the target hypothesis solving the ambiguity in translation (c), and post-processing converts remaining placeholders foot_7 into word forms by means of a set of rules (d). a potential disadvantage of this approach is that actual instances of injected terminology are completely hidden to the neural network, that only handles placeholders, whereas this information can be valuable, with the exceptions of rare words or oovs. we thus propose a second alternative where the source term is left in the source sentence surrounded by placeholders: se @xmath6. ## results results in terms of bleu score @xcite computed by multi-bleu.perl foot_13 are reported in in the many condition, when we inject a high number of terms, the app score makes a significant jump in specialized domains only, while scores of the models based on morphological marking (mrk, mrk+) suffer a substantial decrease in both generic and specialized domains, of higher importance for mrk and specialized domains. when we inject a "reasonable" quantity of terms, results highly depend on the nature of the injected terms. in the already condition, when terms are already well produced by the baseline, terminology injection creates a small drop for all models compared to the baseline, a drop that gets more important for mrk and specialized domains. these results indicate that models using morphological marking suffer from not having access to lexical instances, in particular when too many terms are injected, reflecting the limits of these models. however, in the improve condition, when injected terms where not already present in the baseline, terminology injection induces a considerable gain, in particular for specialized domains: for app (+0.38 in generic domains, +1.25 in specialized domains in average), and of larger magnitude for mrk (+0.50 generic, +2.15 specialized) and mrk+ (+0.71 generic, +2.08 specialized). ## conclusion our major finding is that, in a context where the terminology introduces specialized terms that were not already well translated by the baseline, the app model -appending terminology as inline annotations in the source text -fails to inject terms at 34% and therefore does not guarantee the presence of expected terms in translations. this can be highly critical in a real setting when the user wants terminology to enforce the use of certified brands, product names, acronyms, but also business concepts, such as noun phrases and verbs. with the constraints that ones need to curate highly detailed linguistic resources and that the quantity of injected terms needs to be limited, the mrk models -representing expected terms by their morphological analysis -offer further guarantee of term injection with an absence rate of only 2%: when the exact term cannot be injected, the model usually injects a case or inflection variation that fits the translation. additionally, the model can handle intricate patterns that are part of a vast majority of languages such as irregular forms, complex noun or verbal phrases, as well as multi-part and contextual entries. in contrast with the app model, that simply learns a copy behaviour from source to target agnostic to the context, the mrk models leverage the inner language knowledge of the neural network to perform morphological and syntactic analysis of the source, and more seamlessly generate the target. mantenga la jeringa en el cartón exterior para proteger de la luz . mrk+ mantenga la jeringa en el cartón exterior para proteger de la luz .