Exploring Inline Lexicon Injection for Cross-Domain Transfer in Neural Machine Translation (2024)

Abstract: Domain transfer remains a challenge in machine translation (MT), particularly concerning rare or unseen words. Amongst the strategies proposed to address the issue, one of the simplest and most promising in terms of generalisation capacity is coupling the MT system with external resources such as bilingual lexicons and appending inline annotations within source sentences. This method has been shown to work well for controlled language settings, but its usability for general language (and ambiguous) MT is less certain. In this article we explore this question further, testing the strategy in a multi-domain transfer setting for German-to-English MT, using the mT5 language model fine-tuned on parallel data. We analyse the MT outputs and design evaluation strategies to understand the behaviour of such models. Our analysis using distractor annotations suggests that although improvements are not systematic according to automatic metrics, the model does learn to select appropriate translation candidates and ignore irrelevant ones, thereby exhibiting more than a systematic copying behaviour. However, we also find that the method is less successful in a higher-resource setting with a larger lexicon, suggesting that it is not a magic solution, especially when the baseline model is already exposed to a wide range of vocabulary.

## introduction data-driven machine translation (mt) models, and in particular neural mt models, have led to signifi-cant progress in the quality of automatic translation, particularly in settings where large amounts of data are available @xcite @xcite . however, a scenario in which mt typically struggles to perform as well is cross-domain transfer @xcite @xcite @xcite , where a model trained on one domain is adapted to a second domain, for which there typically exists less data. a major challenge is ensuring that the model is capable of handling the domain-specific vocabulary of the new domain, which may be rare or even unseen in the initial training corpus @xcite . domain adaptation for mt has benefited from pretraining via language models @xcite @xcite trained on large quantities of monolingual text, therefore exposing the model to a wider vocabulary and improving cross-domain transfer @xcite . however, the model's capacity to exploit this underlying vocabulary is limited by the problem of catastrophic forgetting @xcite after fine-tuning @xcite ; the model becomes overly specific to the new data and loses the capacity to generalise to new domains. a line of research with the aim of tackling this problem is the use of external resources such as bilingual lexicons and dictionaries @xcite . these resources, comprising words or phrases and their translations (or words and their definitions in the case of dictionaries) provide a wider (and complementary) lexical coverage than the parallel training data. one aim is for the trained model to be able to exploit the external resource whenever a domain-specific or rare word appears. different integration strategies have been proposed, including interpolation of translation probabilities and external lexicon probabilities @xcite , the use of memory networks @xcite , constrained decoding @xcite , and the inclusion of inline information such as translation candidates @xcite @xcite and definitions @xcite . in this work, we explore the last of these strategies: attaching additional information inline within the source sentence as a way of incorporating domain-specific translation knowledge. it is a simple and commonly used method in the literature and one that has been shown to work well in controlled language settings (i.e. where terms are known in advance and can be translated without ambiguity) @xcite . our aim is to explore how this strategy could work in a practical setting for crossdomain adaptation in the general translation setting, particularly when using pretrained language models that have seen a wider variety of vocabulary than those trained just on parallel data (the previous studies concentrate on vanilla mt). we try to gain some insights into how inline information is used, whether models are able to generalise, disambiguate between multiple candidate translations and how this can ultimately help cross-domain transfer. our experiments on german-to-english (@xmath0. our results are largely negative; we can see small (although not systematic) improvements when applying a model to a new domain. however, we also analyse how the approach works; through a systematic analysis, we show that the approach is more than just a copy mechanism, as we see evidence for the inline translation candidates being used effectively by the model, even when distractor candidates are introduced. we also show that in a higher-resource setting with a more diverse training vocabulary and a larger lexicon, the method is less effective and therefore it is not a go-to method in all settings. our code and outputs will be made publicly available. ## related work different strategies exploiting bilingual lexicons and dictionaries have been developed in the past to handle rare words, the majority focusing on integrating bilingual lexicons containing word (or phrase) translation pairs @xcite @xcite . they differ from normal parallel data in that entries are shorter and they often cover domain-specific and rare vocabulary. these strategies include but are not limited to adding lexicons to the parallel training data @xcite , combining translation and external lexicon probabilities @xcite , using memory networks @xcite , constrained decoding @xcite and infixing of translation candidates within the source sentence @xcite @xcite . in this inline approach, the idea is to either add translations inline within the source sentences or to replace the terms with their translations. it has been shown to work well with controlled and non-ambiguous settings @xcite and when using a mechanism to encourage annotation copying @xcite . a similar code-switching-inspired method was introduced by @xcite , whereby terms are replaced by their translations from bilingual lexicons, and the generated examples used as extra training data. @xcite also look at code-switched data, replacing terms with their translation equivalents. similar strategies have been used elsewhere, for example @xcite integrate code-switching-style replacements using the bilingual lexicon in the back-translation step of an unsupervised mt model, and junczys-dowmunt and grundkiewicz (2016) and @xcite augment sentences with fuzzy translation matches. a few studies have looked into the use of dictionary definitions in mt, as opposed to bilingual lexicons. zhong and chiang (2020) use a method similar to @xcite , involving appending unknown words' definitions to source sentences and indicating through positional embeddings to which words the definitions are attached. beyond mt, the use of dictionary definitions has also been investigated for word embedding creation: bosc and vincent (2018) by auto-encoding and reconstructing definitions to improve word embeddings and @xcite by using definitions as a bridge between translations. theoretically, there is not a clear distinction between bilingual lexicons and bilingual dictionaries in that dictionary definitions often contain synonyms (corresponding to translations in the bilingual case). however, we would expect dictionary definitions to be descriptive rather than translations. 1 in this work, we use bilingual lexicons (containing possible translate candidates) rather than dictionaries, but where several possible candidates are present for each source word. ## integrating lexicon entries we concentrate on the use of bilingual lexicons with word-candidate pairs to improve domain transfer in mt. some examples of the bilingual lexicon entries are given in ## results to test inline's ability to transfer to new domains, we train one model per training dataset and evaluate on all test sets. we compare to concat approaches and to a baseline that does not use the lexicon, trained and evaluated in the same way. given that inline only sees the lexicon words seen in the training data, we also test a hybrid approach involving training on the concatenation of both data sources and then fine-tuning using the inline approach. we compare a total of five models: results are shown in compared to the baseline, both concat approaches improve the emea and ecb models' performance by at least 1 bleu on a majority of the test sets from different domains. however, they do not provide any gains to the globalvoices model's performance on other domains. this may be because of the small size of the globalvoices training data (the bilingual lexicon contains 30k more examples and so possibly outweighs it). the inline model trained on globalvoices does not show improved performance on most of the test sets either. however, similar to the concat models' results, there was at least +0.5 bleu when transferring from @xmath1,news18,news20,ecomm22} and @xmath2,news20,news22,soc22}. these results indicate that there is some evidence for cross-domain transfer for both approaches, which show small improvements for the ecb and emea models when evaluated on a different domain (although globalvoices models show little improvement, possibly due to the small dataset size). however, there is little improvement when these models are tested on the data from the same source as the training data (e.g. @xmath3. the hybrid approaches show some benefits over the individual methods in several cases especially for inline+concat-diff. ## going further: when are inline definitions used? these results show that the inline approach leads to slight improvements in translation performance in some cases and does not improve in others. examples 2-4 from the emea test sets (using the ecb-trained model) illustrate how attaching the candidates inline can sometimes be effectively used in the generated hypothesis and sometimes not. the models fail to use the annotations in example 2, will include these results in the appendix. we report bleu instead of comet since the conclusions are the same for the two metrics. comet is better correlated with human judgments and is recommended by @xcite for evaluation terminology translation, but bleu is more tangible, so readers familiar with mt can get a better appreciation of absolute quality. we did some initial experimentation with the inline models by manually sampling examples from the test sets and creating hypothetical test examples (either with manually created correct translations or invented (incorrect) translations). a few such examples are shown in ## conclusions our study focuses on a simple method of incorporating lexical knowledge from bilingual lexicons into nmt models for cross-domain transfer: infixing translation candidates to rare terms within source sentences. we compare to using lexicon entries as additional parallel training data. we show that lexicons can sometimes help cross-domain transfer, but the gains seen (according to automatic metrics) are limited and appear to diminish in higher-resource scenarios. this is in contrast to its previous successful use in controlled language settings, showing that it is not such a promising approach in the general translation setting. our analysis of the model outputs using distractor term translations showed that, despite the small difference in scores, the models make use of these definitions and they importantly can learn to ignore irrelevant definitions rather than blindly copying entries. however, the method is far from being as successful for this cross-domain setup as in the controlled language settings in which the method was developed, and experiments on a higher-resource language setting show that the approach does not have a huge effect to performance compared to a strong baseline.