Iterative Nearest Neighbour Machine Translation for Unsupervised Domain Adaptation (2023)

Abstract: Unsupervised domain adaptation of machine translation, which adapts a pre-trained translation model to a specific domain without in-domain parallel data, has drawn extensive attention in recent years. However, most existing methods focus on the fine-tuning based techniques, which is non-extensible. In this paper, we propose a new method to perform unsupervised domain adaptation in a non-parametric manner. Our method only resorts to in-domain monolingual data, and we jointly perform nearest neighbour inference on both forward and backward translation directions. The forward translation model creates nearest neighbour datastore for the backward direction, and vice versa, strengthening each other in an iterative style. Experiments on multi-domain datasets demonstrate that our method significantly improves the in-domain translation performance and achieves state-of-the-art results among non-parametric methods.

## introduction neural machine translation (nmt) has demonstrated impressive performance when trained on large-scale corpora. however, despite the abundance of general-domain parallel data, domainspecific parallel data is not readily available @xcite . therefore, how to adapt a general-domain nmt model via in-domain monolingual data has become the research focus in the community. since no annotated data is involved, this effort is also generally known as the unsupervised domain adaptation of nmt. existing approaches are mostly focused on the data selection and finetuning techniques (pourdamghani et al., 2019; aharoni and goldberg, † contribution during internship at bytedance inc. ‡ corresponding authors. 2020a; @xcite . for example, @xcite proposed to induce indomain lexicon pairs as synthetic data for finetuning. @xcite proposed to use constrained back-translation model to generate synthetic in-domain data for fine-tuning. despite the progress they have made, the cumbersome finetuning process would lead to catastrophic forgetting @xcite and decrease the performance on general domain. besides, a copy of parameters is required for each domain, which is not flexible facing multi-domain scenario. therefore, @xcite propose to perform adaptation based on non-parametric nearest neighbour inference, and they introduce an autoencoder task based on target language to enable in-domain data construction with monolingual data. however, adapter layers are still required to be fine-tuned in their method, which is not fully non-parametric. in this work, we propose an iterative nearest neighbour approach named iter-knnmt to achieve fully non-parametric unsupervised domain adaptation. our framework is built based on the recently proposed knn-mt @xcite . we employ two pre-trained generaldomain nmt models in both forward and backward directions, and the datastores are constructed by the model in the reverse direction for each other. the forward model performs nearest neighbour inference to the source language sentences, and the results serve as the datastore for the backward model. then, the backward model performs nearest neighbour inference to the target sentences and generate better datastore for the forward model. this process is iteratively performed, making the most of monolingual data for nonparametric inference. we evaluate the proposed iter-knnmt on multidatasets, including it, medical, law and koran domains. experimental results show that, without introducing any extra parameters, we are able to achieve 6 bleu improvement on in-domain translation, bringing a new state-of-the-art results among non-parametric adaptation methods. ## conclusion this paper proposes a simple yet effective method to perform unsupervised domain adaptation of ma-13297 chine translation in a non-parametric manner. we perform nearest neighbour inference on both forward and backward directions, strengthening each other in an iterative manner. while accumulated datastore is more robust and effective than datastore generated in a single pass, the accumulated datastore introduce extra retrieval overhead. in the future, we would investigate how to compress the datastore and improve the decoding efficiency.