Efficient Domain Adaptation for Non-Autoregressive Machine Translation (2024)

Abstract: Domain adaptation remains a challenge in the realm of Neural Machine Translation (NMT), even in the era of large language models (LLMs). Existing non-parametric approaches like nearest neighbor machine translation have made small Autoregressive Translation (AT) models achieve efficient domain generalization and adaptation without updating parameters, but leaving the Non-Autoregressive Translation (NAT) counterparts under-explored. To fill this blank, we introduce Bi - k NN, an innovative and efficient domain adaptation approach for NAT models that tailors a k-nearest-neighbor algorithm for NAT. Specifically, we introduce an effective datastore construction and correlated updating strategies to conform the parallel nature of NAT. Additionally, we train a meta-network that seamlessly integrates the NN distribution with the NMT distribution robustly during the iterative decoding process of NAT. Our experimental results across four benchmark datasets demonstrate that our Bi - k NN not only achieves significant improvements over the Base-NAT model (7.8 BLEU on average) but also exhibits enhanced efficiency.

## introduction llms have dramatically shifted the paradigm of various language processing tasks, and there have also been extensive discussions recently to weigh up the pros and cons of llms to nmt @xcite @xcite . llms can achieve appealing generalization and task performance by pretraining on vast and varied corpora across languages and domains, surpassing the task expert model trained on supervised translation pairs under resource-rich scenarios, but might suffer from the prohibitive to explore this problem further, we first briefly compare different lines of possible solutions for nmt tasks, including closed-sourced llm (chat-gpt), open-sourced llm (llama-2), and two types of transformer-based expert models (i.e., the auto-regressive (at) and non-auto-regressive (nat) fashion). 2 llms are introduced to calibrate the domain-specific translation performance without any further task training due to possible prohibitive costs. expert models are presented to learn the domain adaptation capabilities of small capacity models with supervised task training, e.g., initially converged on the wmt training dataset but inference on specific domains like it, medical, law, and koran. there are already some effective strategies to enhance the domain adaptation performance of small models, particularly the non-parametric paradigm. for instance, @xcite present a k-nearest-neighbor machine translation method, which utilizes a trained nmt model to construct a datastore, consisting of (query: context representations; value: the correlated target tokens) pairs in the training set, and then retrieves relevant tokens during inference to enhance the translation accuracy. this non-parametric approach equips the model with rapid domain adaptation and generalization abilities without the need for parameter adjustments. however, most of the existing methods are tailored for at models, leaving the domain adaptation problem of nat models under-explored. to fill this blank, we introduce an innovative domain adaptation approach for nat models, namely bidirectional-iterative-knn (bi-knn), which is an efficient method tailored for nat models. unlike k-nearest-neighbor nmt for at models, nat models struggle with producing accurate representations due to insufficient context with parallel decoding. to overcome this, we present a novel and effective framework for knn-mt with nat models, including (1) building a bidirectional datastore, (2) renewing the indecipherable datastore, (3) training a robust meta-network, and (4) iterative-knn decoding. we conducted experiments on multiple domain-specific nmt tasks. across four domains, our approach achieved an average of 7.8 bleu score improvement for the base-nat models and outperformed the specialized models which are trained on the corresponding datasets on most datasets. furthermore, without tuning the parameters of pre-trained models, our method proved more efficient and avoided catastrophic forgetting compared to the straightforward fine-tuning method. ## methodology in this section, we present the overall process of our proposed bi-knn, as illustrated in figure 1 . concretely, our method contains four specific steps, i.e., build bidirectional datastore, renew indecipherable datastore, train robust meta-network and iterative decoding with knn. ## analysis 6.1 catastrophic forgetting of nat a straightforward approach for domain adaptation involves fine-tuning pre-trained models on specific target domain data. however, studies @xcite @xcite ## conclusion in this paper, we highlight the ongoing challenge of domain adaptation for nmt and point out the deficiency of nat models for domain adaptation tasks. subsequently, we introduce bi-knn, an innovative domain adaptation approach, tailoring knn for nat models, which creates a robust bidirectional datastore and integrates iterative decoding with knn retrievals. extensive evaluation results and in-depth analysis consistently demonstrate the overall effectiveness and efficiency of our method.