Learning Kernel-Smoothed Machine Translation with Retrieved Examples (2021)

Abstract: How to effectively adapt neural machine translation (NMT) models according to emerging cases without retraining? Despite the great success of neural machine translation, updating the deployed models online remains a challenge. Existing non-parametric approaches that retrieve similar examples from a database to guide the translation process are promising but are prone to overfit the retrieved examples. However, non-parametric methods are prone to overfit the retrieved examples. In this work, we propose to learn Kernel-Smoothed Translation with Example Retrieval (KSTER), an effective approach to adapt neural machine translation models online. Experiments on domain adaptation and multi-domain machine translation datasets show that even without expensive retraining, KSTER is able to achieve improvement of 1.1 to 1.5 BLEU scores over the best existing online adaptation methods. The code and trained models are released at https://github.com/jiangqn/KSTER .

## introduction over the past years, end-to-end neural machine translation (nmt) models have achieved great success @xcite @xcite . how to effectively update a deployed nmt model and adapt to emerging cases? for example, after a generic nmt model trained on wmt data, a customer wants to use service to translate financial documents. the costomer may have a handful of translation pairs for the finance domain, but do not have the capacity to perform a full retraining. non-parametric adaptation methods enable incorporating individual examples on-the-fly, by retrieving similar source-target pairs from an external database to guide the translation process @xcite @xcite ; @xcite . the external database can be easily updated online. most of these methods rely on effective sentence-level retrieval. different from sentence retrieval, k-nearest-neighbour machine translation introduces token level retrieval to improve translation @xcite . it shows promising results for online domain adaptation. there are still limitations for existing nonparametric methods for online adaptation. first, since it is not easy for sentence-level retrieval to find examples that are similar enough to the test example, this low overlap between test examples and retrieved examples brings noise to translation @xcite . second, completely non-parametric methods are prone to overfit the retrieved examples. for example, although knn-mt improves domain-specific translation, it overfits severely and can not generalize to the general domain, as is shown in figure 1 . an ideal online adaptation method should introduce less noise to the translation process and generalize to the changeful test examples with the incrementally changing database. in this paper, we propose to learn kernel-smoothed translation with example retrieval (kster), to effectively learn and adapt neural machine translation online. kster retains the online adaptation advantage of non-parametric methods and avoids the drawback of easy overfitting. more specifically, kster improves the generalization ability of non-parametric methods in three aspects. first, we introduce a learnable kernel to dynamically measure the relevance of the retrieved examples based on the current context. then, the exampled-based distribution is combined with the model-based distribution computed by nmt with adaptive mixing weight for next word prediction. further, to make the learning of kster stable, we introduce a retrieval dropout strategy. the intuition is that similar examples can constantly be retrieved during training, but not the same situation during inference. we therefore drop the most similar examples during training to reduce this discrepancy. with above improvements, kster shows the following advantages: ## related work this work is mostly related to two research areas in machine translation (mt), i.e., domain adaptation for machine translation and online adaptation of mt models by non-parametric retrieval. ## methodology in this section, we first formulate the kernelsmoothed machine translation (kster), which smooths neural machine translation (nmt) output with retrieved token level examples. then we introduce the modeling and training of the learnable kernel and adaptive mixing weight. the overview of kster is shown in figure 2 . ## experiments we evaluate the proposed methods on two machine translation tasks: domain adaptation for machine translation (damt) and multi-domain machine translation (mdmt). in damt, in-domain translation model is built for each specific domain, since the domain labels of examples are available in test time. in mdmt, the domain labels of examples are not available in test time, so examples from all domains are translated with one model, which is a more practical setting. ## analysis in this section, we first conduct ablation studies to verify the effectiveness of each part of the proposed method. then we conduct detailed analysis on how kernel-smoothing with retrieved examples helps translation. in damt, we report the averaged performance on 5 specific domains. in mdmt, we report the averaged performance on general domain and 5 specific domains. with all k selections, kster outperforms knn-mt consistently. ## conclusion in this work, we propose kernel-smoothed machine translation with retrieved examples. it improves the generalization ability over existing non-parametric methods, while keeps the advantage of online adaptation.