Terminology-Constrained Neural Machine Translation at SAP (2020)

Abstract: This paper examines approaches to bias a neural machine translation model to adhere to terminology constraints in an industrial setup. In particular, we investigate variations of the approach by Dinu et al. (2019), which uses inline annotation of the target terms in the source segment plus source factor embeddings during training and inference, and compare them to constrained decoding. We describe the challenges with respect to terminology in our usage scenario at SAP and show how far the investigated methods can help to overcome them. We extend the original study to a new language pair and provide an in-depth evaluation including an error classification and a human evaluation.

## introduction with over one billion words per year, sap deals with a huge translation volume; covering product localization and translation of documentation, training materials or support instructions for up to 85 languages. with a wide range of product lines in different industries, translation settings are diverse. there are over 100 active translation domains for which we maintain translation resources such as translation memories and terminologies. at sap we usually train multi-domain neural machine translation (nmt) engines, whose input consists of a multitude of data sources including the contents of the company-internal translation memories from various domains. the result-ing nmt system produces high-quality technical translations, but has difficulties generating appropriate and coherent terminology in specific contexts. given the great importance of correct and consistent terminology in technical translation, this is a nuisance for the translators that work in a postediting scenario as well as for users consuming machine translation (mt) in a self-service scenario. in our translation environment, translators are assigned projects along with the relevant translation domain's terminology. to achieve term consistency, sap maintains sapterm foot_0 , a large terminology database which also specifies viable term translations. translators can easily select target terms from sapterm in a computer-assisted translation (cat) environment, but applying terminology constraints in nmt is a challenge. as we do not have reliable term recognition or morphological inflection generation tools for all our productive languages at our disposal, we require an approach that not only enforces the correct terminology but also learns its contextually appropriate inflections. to that end, we investigate the approach presented in @xcite , which combines inline annotation with source factors @xcite , that provide an additional input stream with terminology annotation, to show how domain-specific terminology can be enforced in a multi-domain nmt model. the approach should be capable of handling unseen terminology while retaining nmt's ability to produce fluent output sequences without the need for additional resources such as morphological generators and without drastically reducing decoding speed. we will present results for variations of this approach which were not investigated in @xcite , but could be of interest to users of nmt who plan to implement that approach in a productive system. while the wmt news translation task that @xcite evaluate on is a viable test bed for new methods, we aim to validate that the method is also applicable to other scenarios, such as the translation of texts from the business and it context of sap, when constraining it with entries from sapterm. we furthermore extend the original study to a new language pair (english-russian) and provide an in-depth evaluation including a human assessment. our study yields very promising results, amongst others improvements of up to 11 bleu points on terminology data, and paves the way to the customization of nmt at sap: a selected sapterm glossary can be applied directly when producing mt proposals for a translation project. this yields better translation quality, helps to reduce post-editing costs and eases translators' frustration with correcting terms. ## related work several approaches to make nmt adapt to a domain-specific terminology have been proposed in the literature. fine-tuning on in-domain training data on-the-fly @xcite is shown to improve translation quality and term accuracy but creates additional technological challenges for model management and increases infrastructure costs. additionally, terminology constraints cannot be specified on a sentence or document level, but instead need to be distinctly present in the available training data, which often is not the case in a productive scenario. the latter argument also holds for domain-aware mt @xcite , where a multi-domain model distinguishes the translation domains using a domain tag, which is prepended to the source segment. since terminology databases are available in most translation environments, integrating them into nmt at run-time to enable domain-specific translation is an ongoing research topic. early approaches use placeholder tokens for source and target (for example @xcite ). placeholder approaches often suffer from disfluency as the nmt model does not have access to the term and therefore has difficulties creating a fluent and morphologically sound translation. constrained decoding is one of the most prominent approaches to enforcing terminology in nmt. the decoder is subject to a set of constraints that are strictly enforced during decoding @xcite . some issues with constrained decoding have already been addressed, such as better positioning of target terms by exploiting the correspondence between source and target terms @xcite , and improving performance for the base approach @xcite . nevertheless, the increase in decoding time compared to unconstrained decoding is still considerable (cf. section 5). also the output surface form is enforced exactly as provided by the constraint and no morphological adaptation is applied by the decoder. this leads to misplaced constraints and broken sentences @xcite as well as special cases where surface form variants of an enforced term are being produced by the decoder but not picked up by the constraint, leading to a duplication as the constraint produces the terminology again @xcite . @xcite offer a different approach to applying terminology constraints in nmt. the target terms are inserted into the source string during training and decoding, and thus the model learns a copying behavior. an indication of which words are source terms, target terms or no terms is provided to the model via an additional input stream. this input is encoded as source factors, in the same way that linguistic features can be encoded @xcite . for the english-german wmt 2018 news translation task, moderate improvements in bleu and term @xmath0. the zero-shot nature of this approach enables the application of unseen terminology at test time. furthermore, @xcite report cases of generating morphological variants of terminology entries in the output, while decoding times are not increased compared to the base model. as the ability to apply terminology constraints is trained into the nmt model by either appending the target term to the source term or by replacing it, @xcite refer to their models as train-by models, and we will continue doing so. many commercial providers of mt offer an option to upload a user dictionary in order to customize the nmt output to enforce a certain terminology. 2 this is a feature that users became ac-customed to in rule-based and statistical mt, and consequently they expect a similar functionality for nmt as well. naturally, the commercial providers usually leave us in the dark about the technology that is used for the implementation of that feature. such custom terminology features are described more for marketing purposes rather than from an objective technical viewpoint. usually, no transparent evaluation results are available. some product descriptions are nevertheless fair enough to describe the limitations of the feature and best practices. ## methodology we experiment with variants of the train-by approach introduced by @xcite , which is a form of inline term annotation. target terms t t are inserted into a source sentence by either appending them to the source term t s (append) or by replacing t s completely (replace). an additional signal is provided by a term annotation for each input token, where 1 means part of a source term, 2 means part of a target term and 0 is the default. an example for the input is provided in the term annotations are presented as source factors and have their own embedding vectors, which are combined with the respective (sub-) word embeddings to represent the input of the encoder in an encoder-decoder nmt architecture @xcite . the two embedding vectors can be combined by either concatenating (concat) or summing (sum) them. this makes the dimensionality of the source factor embedding either a variable-sized (concat) or a fixed sized (sum) vector. while @xcite only report results for the concatenation strategy with an embedding size of 16, we investigate an embedding size of 8 as well as the vector summarization combination. we are also interested in the impact of the source factors themselves, and thus investigate whether the additionally provided annotation is actually necessary by using only the inline annotation and no term factor annotation. the source sentences are annotated as described for all terminology entries (t s , t t ), when t s is present in the source and t t occurs in the reference. to ## experimental setup we evaluate the application of terminology constraints in the usage scenario of mt at sap, for two language pairs english-german (en-de) and english-russian (en-ru). we use target languages that are relatively morphologically rich because we want to investigate whether the approach is able to produce the target terms in an appropriate morphological form. ## automatic evaluation in this section we present the results of our experiments using automatic evaluation. ## translators' assessment as we apply mt in post-editing scenarios, it is of importance that our translators approve of our proposed solution of enforcing sap-specific terminology. taking the shortcomings of automatic metrics for mt into account, we therefore also conducted a human evaluation. ## examples & discussion in this section, we present examples of correct term translations as well as an in-depth human analysis of the terms that were not produced according to the automatic evaluation. examples for en-de and en-ru are displayed in ## conclusion we have investigated a new approach for terminology integration into nmt, originally proposed by @xcite , in an real-world setup. our experimental setting was it-related corporate data from sap with terminology from sap's terminology database, for two language pairs with rather morphologically rich target languages. our study yields positive results, namely term @xmath1, with neither impacting the translation speed nor the translation quality on data without terminology. the improvements in term accuracy were furthermore confirmed in a human evaluation for both language pairs. in an additional manual investigation, we inspected the problematic cases and found that almost half of them are false negatives, meaning that term rates are in fact even higher. we have furthermore confirmed that with this approach the term translations are used flexibly in the surface form required by the sentential context. overall, it seems to be a promising approach for applying terminology constraints.