Non-Parametric Unsupervised Domain Adaptation for Neural Machine Translation (2021)

Abstract: Recently, kNN-MT (Khandelwal et al., 2020) has shown the promising capability of directly incorporating the pre-trained neural machine translation (NMT) model with domain-specific token-level k-nearest-neighbor (kNN) retrieval to achieve domain adaptation without retraining. Despite being conceptually attractive, it heavily relies on high-quality in-domain parallel corpora, limiting its capability on unsupervised domain adaptation, where in-domain parallel corpora are scarce or nonexistent. In this paper, we propose a novel framework that directly uses in-domain monolingual sentences in the target language to construct an effective datastore for k-nearest-neighbor retrieval. To this end, we first introduce an autoencoder task based on the target language, and then insert lightweight adapters into the original NMT model to map the token-level representation of this task to the ideal representation of the translation task. Experiments on multi-domain datasets demonstrate that our proposed approach significantly improves the translation accuracy with target-side monolingual data, while achieving comparable performance with back-translation. Our implementation is open-sourced at https://github.com/zhengxxn/UDA-KNN .

## introduction non-parametric methods @xcite @xcite @xcite have recently been successfully applied to neural machine translation (nmt). these approaches complement advanced nmt models @xcite @xcite with external memory to alleviate the performance degradation when translating out-ofdomain sentences, rare words @xcite , etc. among them, knn-mt @xcite is a simple yet effective non-parametric method using nearest neighbor retrieval. more specifically, knn-mt equips a pre-trained nmt model with a knn classifier over a provided datastore of cached context representations and corresponding target tokens to improve translation accuracy without retraining. this promising ability to access any provided datastore or external knowledge during inference makes it expressive, adaptable, and interpretable. despite the potential benefits, knn-mt requires large-scale in-domain parallel corpora to achieve domain adaptation. however, in practice, it is not realistic to collect large amounts of high-quality parallel data in every domain we are interested in. since monolingual in-domain data is usually abundant and easy to obtain, it is essential to explore the capability of knn-mt on unsupervised domain adaptation scenario that utilizes large amounts of monolingual in-domain data. one straightforward and effective solution for unsupervised domain adaptation is to build in-domain synthetic parallel data via back-translation of monolingual target sentences @xcite @xcite . although this approach has proven superior effectiveness in exploiting monolingual data, applying it in knn-mt requires an additional reverse model and brings the extra cost of generating back-translation, making the adaptation of knn-mt more complicated and time-consuming in practice. in this paper, we propose a novel unsupervised domain adaptation framework based on knn-mt (uda-knn). the uda-knn aims at directly leveraging the monolingual target-side data to generate the corresponding datastore, and encouraging it to play a similar role with the real bilingual in-domain data, through the carefully designed architecture and loss function. specifically, we introduce an autoencoder task based on target language to enable datastore construction with monolingual data. then we incorporate lightweight adapters into the encoder part of pre-trained nmt model to make the decoder's representation in autoencoder task close to the corresponding representation in translation task. in this way, the adapter module implicitly learns the semantic mapping from the target language to source language in feature space to construct an effective in-domain datastore, while saving the extra cost of generating synthetic data via back-translation. we evaluate the proposed approach on multidomain datasets, including it, medical, koran and law domains. experimental results show that when using target-side monolingual data, our proposed approach obtains 4.9 bleu improvements on average and even achieves similar performance compared with back-translation. ## background in this section, we give a brief introduction to the domain adaptation of knn-mt. in general, the process includes two steps: creating an in-domain datastore and decoding with retrieval on it. in-domain datastore creation. given a pretrained general domain nmt model and an indomain parallel corpus, knn-mt utilizes the model to forward pass the corpus to create a datastore. formally, for each bilingual sentence pair in the corpus (x, y) ∈ (x , y), the nmt model will generate a context representation h(x, @xmath0. then, the datastore is constructed by collecting the representations and corresponding tokens as keys and values respectively: ## unsupervised domain adaptation with knn-mt although @xcite has shown the capability of knn-mt on domain adaptation, the datastore creation heavily relies on highquality in-domain parallel data, which cannot be always satisfied in practice. as in-domain monolingual data is usually abundant and easy to obtain, it is essential to extend the capability of knn-mt on unsupervised domain adaptation that merely uses large amounts of in-domain target sentences. in this paper, we design a novel non-parametric unsupervised domain adaptation framework based on knn-mt (uda-knn) to fully leverage in-domain target-side monolingual data. the overview framework of uda-knn is illustrated in figure 1 . the uda-knn starts with the autoencoder task based on target language y, where the target-side is simply copied to the source-side to generate pair (y, y). based on that, the uda-knn aims to make the decoder's representation in autoencoder task close to the ideal representation in translation task. in this way, we can directly leverage autoencoder structure and in-domain target sentences to construct the corresponding datastore for k-nearest-neighbor retrieval, which is similar to that from real in-domain bilingual data. next, we will introduce the architecture and training objective of our proposed method in detail. architecture. we insert lightweight adapter layers @xcite @xcite into the source embedding layer and each encoder layer of the pre-trained nmt model to perform the autoencoder task, by which we only increase a few parameters for our method. specifically, we simply construct the adapter layer with layer normalization as well as two feed-forward networks with non-linearity between them: training. the uda-knn is designed to leverage monolingual target-side data to generate the corresponding datastore, which plays a similar role with real in-domain bilingual data. we achieve this by leveraging out-of-domain bilingual data (x , y). more specifically, given a bilingual sentence pair in the corpus (x, y) ∈ (x , y), the original nmt model generates decoder representation h (x; @xmath1. meanwhile, with the targetcopied pair (y, y), the nmt model with adapters generates another representation for each y t , which can be denoted as h (y; @xmath2. we take the end-to-end paradigm to directly optimize the adapter layers by minimizing the squared euclidean distance of the two sets of decoder representations: prediction. for unsupervised domain adaptation, given the domain-specific target-side monolingual data, we first copy the target sentences to the source side to generate synthetic bilingual pairs. then the pre-trained nmt model with adapter layers forward passes these pairs to create an in-domain datastore. when translating in-domain sentences, we utilize the original nmt model and knn retrieval on the in-domain datastore to perform online domain adaptation as equation ( 3 ). ## conclusion in this paper, we present uda-knn, a simple yet effective framework that directly utilizes monolingual data to construct in-domain datastore for unsupervised domain adaptation of knn-mt. experimental results verify that our method obtains significant improvement with target-side monolingual data. our approach also achieves comparable performance with the bt-based method, while saving the extra cost of generating back-translation.