Evaluation of Terminology Translation in Instance-Based Neural MT Adaptation (2018)

Abstract: We address the issues arising when a neural machine translation engine trained on generic data receives requests from a new domain that contains many specific technical terms. Given training data of the new domain, we consider two alternative methods to adapt the generic system: corpus-based and instance-based adaptation. While the first approach is computationally more intensive in generating a domain-customized network, the latter operates more efficiently at translation time and can handle on-the-fly adaptation to multiple domains. Besides evaluating the generic and the adapted networks with conventional translation quality metrics, in this paper we focus on their ability to properly handle domain-specific terms. We show that instance-based adaptation, by fine-tuning the model on-the-fly, is capable to significantly boost the accuracy of translated terms, producing translations of quality comparable to the expensive corpusbased method.

## introduction when deployed in production lines, machine translation (mt) systems need to serve requests from various domains (e.g. legal, medical, finance, sports, etc.) with a variety of structural and lexical differences. considering that technical translation (e.g. user guides, medical reports, etc.) represents the largest share in the translation industry @xcite and that a significant part c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. ideal solutions for this real-world multi-domain translation scenario should be scalable enough to enable the industrial deployment at a reasonable cost, while guaranteeing a high level of flexibility in delivering good-quality translations for all (or most of) the domains. this is of higher importance for the neural approach, where building the systems usually requires expensive gpu machines trained for several days to weeks on large amounts of parallel data. in this paper we analyze the ability of instancebased adaptation strategy in handling domain terminology (technical terms) and compare its performance with a non-adaptive generic neural mt (nmt) system trained on a large pool of parallel data, and a corpus-based adaptive nmt system as our results show that, in contrast to the generic and corpus-based adaptive solutions which compromise either the translation quality or the architectural cost, recently proposed instance-based adaptation methods @xcite provide a flexible solution at reasonable costs. this adaptive system is based on a retrieval mechanism that, given a test sentence to be translated, extracts from the pool of parallel data the top (source, target) pairs in terms of similarity between the source and the test sentence. using this small set of retrieved pairs, it then fine-tunes the model, and applies it to translate the input sentence. as shown in @xcite , by applying local adaptation to few training instances, not only the system is able to improve the performance of the generic nmt but, in some domains, it can also outperform strong specialized corpus-based nmt systems trained for several epochs on the corresponding domain-specific data. in this paper, we further explore the effectiveness of the instance-based adaptation method reporting, in addition to global corpus-level bleu scores, empirical results on how they perform in translating domain terminology. to this aim, we divide the terms into two categories of single-and multi-word phrases, and study the systems' behaviour in each class separately. unsurprisingly, in both cases corpus-based adaptation improves the performance of the generic model by a large margin. such improvements, however, come at the cost of computationally intensive adaptation on all the in-domain data. in contrast, instance-based adaptation achieves comparable results with a less demanding strategy based on adapting the model to few training instances retrieved from the pool of data on the fly. this empirical proof, focused on the proper treatment of domain terms in nmt adaptation, is the main contribution of this paper. ## related works when exposed to new domains @xcite or applied in multi-domain scenarios @xcite , machine translation systems in general and neural mt in particular, experience performance degradations due to the distance between the target domain and the domain(s) on which they were trained. previous studies on multi-domain mt provide solutions for this issue, making it possible to cover more than one domain while reducing performance degradations in the target domains @xcite @xcite @xcite @xcite . these solutions can be categorized into static and adaptive approaches. to train one single model using heterogeneous data from many domains, static approaches assume to have simultaneous access to all the training data and their corresponding domain/topic information. this information, which is either manually assigned or automatically inferred from the data, is passed as additional signal to the mt system, helping it to produce higher quality translations for the desired target domain. existing solutions in the field of nmt propose to incorporate this topic/domain information only on the source side (i.e. to support the encoder) @xcite , only on the target side (i.e. to support the decoder) @xcite , or on both sides @xcite . although consistent and significant improvements have been reported by this approach, its application to new domains is not trivial, mostly due to the fact that it requires performing expensive nmt and topic model adaptations using the original multi-domain data and the training set for the new domain. adaptive approaches, on the other hand, propose to fine tune an existing mt system, trained either on another domain or pool of parallel data, to the new domain or task. while luong and manning (2015) report significant improvements by this approach on the new target domain, freitag and al-onaizan (2016) observe a significant drop in system's performance on the original domain, which is due to the severe overfitting of the model to the new domain. to solve this issue, they propose a slightly different approach, which performs ensemble decoding using both the adapted and the generic model. the mixed fine tuning method proposed by @xcite is another approach for keeping under control the performance degradation on the original out-domain data while adapting the model to the new domain. given the outdomain and in-domain training sets and a model pre-trained only on the out-domain data, this approach continues the training on a parallel corpus that is a mix of the two training corpora, in which the smaller in-domain corpus is oversampled to have the same size as the larger out-domain corpus. the specialized models obtained by these adaptation techniques are empirically shown to be effective, improving the translation quality of a generic nmt system on the target domains. however, the practical adoption of this approach results in developing and maintaining multiple specialized nmt engines (one model per domain), which increases the infrastructure's costs and limits its scalability in real-world application scenarios. to combine the advantages of the two worlds, (i.e. to get close to the high quality of corpusbased adaptation still keeping the scalability of one single model), @xcite introduce an instance-based adaptation method for nmt inspired by @xcite . instead of adapting the original generic model to the whole in-domain training corpus, the instancebased method retrieves from the pool of parallel data a small set of sentence pairs in which the source side is similar to the test sentence. then, it fine-tunes the generic model on-the-fly by using the set of retrieved samples. this makes the instance-based adaptive approach a reasonable solution for real-world production lines, in which the mt system needs to cover a wide range of application domains while keeping under control the architecture's cost. in addition to the architectural costs of nmt deployment in multi-domain application scenarios, there is another important factor that has to be considered, that is their ability in translating domain-specific words and phrases (i.e. terms). based on their nature, these expressions can be fre-quently observed in their corresponding domains, being at the same time infrequent or even out of vocabulary (oov) in the other domains. nevertheless, data-driven mt systems need to be trained on large amounts of training data, which is generally collected from different sources. this further reduces the relative frequency of these words, making them less probable to be translated correctly by the system. this makes it even more challenging for nmt approach where rare and oov words are either segmented into their corresponding sub-word units @xcite or mapped to a special "unk" token @xcite . however, in the relatively recent history of nmt, there are few works that analyze its behavior focusing on domain terminology. @xcite achieve significant improvements with a guide mechanism that helps the nmt decoder to prioritize and adequately handle translation options obtained from terminology lists. @xcite empirically show that offline adaptation of a generic nmt system to the new domain improves its performance in translating domain-specific terms. however, they discuss only corpus-based adaptation techniques that, compared to instance-based methods, are less suitable for real-world application. moreover, they mostly work in a setting in which domain terminology has to be translated in isolation without any context, while in our working scenario we work with full sentences. ## neural machine translation adaptation in this section we first briefly review the stateof-the-art sequence-to-sequence neural machine translation and then describe the two corpus-based and instance-based adaptation approaches. ## analysis and discussion in this section we present a detailed analysis of the results obtained by different systems and compare the systems in translating the technical terms in gnome and kde4. ## further analysis in addition to the automatic evaluations we performed further manual analysis on the outputs of the instance-based adaptive system. the results of this analysis indicate that, compared to the generic system, its behavior differs in two main aspects: i) learning to translate the terms that are missed or wrongly translated by the generic system, ii) adapting to different style of the translation. when run on new domains, for which it has not seen any in-domain training data, it is highly probable that the generic system receives translation requests containing terms which are oov or infrequently observed in the training data. in such cases, even after applying bpe, it might not be able to produce proper translations. as an example, the english word dolphin, which is rarely observed in the generic training data, is always translated in the italian word delfino which refers to the animal. however, in the kde4 domain it corresponds to a proper noun that indicates a file manager application. as we see in the english terms control center and mouse cursor are two interesting examples of learning domain-specific translation styles. while in the generic training data these terms are usually translated into control center and cursore del mouse, in the kde4 domain the human translators prefer them to be translated into centro di controllo and puntatore del mouse. as we see in the examples of we also observed a few cases in which the instance-based approach learns to properly generate italian terms in the translation while there is no corresponding source english term in the given test sentence. the italian word pulsante in the fourth example provided in finally, we noticed that inconsistent translations of the terms can affect the instance-based adaptive system, resulting in translations which are different than the manually produced references. the last example provided in ## conclusions we investigated the application of instance-based adaptive nmt in a real-world scenario where translation requests come from new domains that contain many technical terms. in particular, we analyzed its ability to properly handle domain terminology, comparing its output against the translations produced by a generic (unadapted) nmt system and a corpus-based specialized nmt system. overall, our experiments with gnome and kde4 data reveal that the two adaptation methods significantly improve the performance of the generic system both in terms of global bleu score and term translation accuracy. unsurprisingly, by performing a computationally intensive fine tuning on the full in-domain training data, corpus-based adaptation produces specialized nmt systems that achieve better results at the cost of reduced scalability. however, the less demanding instancebased adaptation (performed on one parallel sentence pair retrieved from a pool of data based on its similarity to the test sentence), is able to effectively learn domain terms' translations, even for expressions that were never observed by the generic model. such capability allows instancebased adaptation to significantly reduce the gap between generic and corpus-based specialized nmt models at manageable costs. source [...]files may be dragged and dropped onto & kwrite; from the desktop, the filemanager & dolphin;[...] reference [...]i file possono essere trascinati e rilasciati su & kwrite; dal desktop, & dolphin;[...] ret. src. [...]files may be dragged and dropped onto & kate; from the desktop, the filemanager & dolphin;[...] ret. trg. [...]i file possono essere trascinati e rilasciati in & kate; dal desktop, dal gestore di file & dolphin;[...] generic [...]i file possono essere trascinati e rilasciati su & kwrite; dal desktop, dal file manager e dal delphin;[...] instance-based [...]i file possono essere trascinati e rilasciati in & kwrite; dal desktop, dal gestore di file & dolphin;[...] corpus-based [...]i file possono essere trascinati e caduti su kwrite; dal desktop, dal gestore file & dolphin;[...] source [...]the mouse cursor is identified in the status bar.[...] reference [...]al puntatore del mouse è identificato nella barra di stato.[...] ret. src. [...]when you hold the mouse cursor still for a moment[...] ret. trg. [...]mantenendo fermo per qualche istante il puntatore del mouse[...] generic [...]il cursore del mouse viene identificato nella barra di stato.[...] instance-based [...]il puntatore del mouse viene identificato nella barra di stato.[...] corpus-based [...]il cursore del mouse è identificato nella barra di stato.[...] source exiting the kde control center reference uscire dal centro di controllo di kde ret. src. the kde control center screen ret. trg. lo schermo del centro di controllo di kde; generic uscita da kde control center instance-based uscita dal centro di controllo di kde corpus-based uscita dal centro di controllo di kde source this saves the settings and closes the configuration dialog. reference questo pulsante salva le impostazioni e chiude la finestra di configurazione. ret. src. this saves the settings without closing the configuration dialog. ret. trg. questo pulsante salva le impostazioni senza chiudere la finestra di configurazione. generic pulsante salva le impostazioni e chiude la finestra di configurazione. instance-based questo pulsante salva le impostazioni e chiude la finestra di configurazione. corpus-based questo pulsante salva le impostazioni e chiude la finestra di configurazione. source automatically scan project's packages reference analizzare automaticamente i pacchetti del progetto ret. src. view or modify the uml system's packages. ret. trg. consente di visualizzare o modificare i package di sistema uml. generic scansione automatica dei pacchetti del progetto instance-based scansione automatica dei package di progetto corpus-based scansiona automaticamente i pacchetti del progetto