k NN - TL : k-Nearest-Neighbor Transfer Learning for Low-Resource Neural Machine Translation (2023)

Abstract: Transfer learning has been shown to be an effective technique for enhancing the performance of low-resource neural machine translation (NMT). This is typically achieved through either fine-tuning a child model with a pre-trained parent model, or by utilizing the out- put of the parent model during the training of the child model. However, these methods do not make use of the parent knowledge during the child inference, which may limit the translation performance. In this paper, we propose a k-Nearest-Neighbor Transfer Learning (kNN-TL) approach for low-resource NMT, which leverages the parent knowledge throughout the entire developing process of the child model. Our approach includes a parent-child representation alignment method, which ensures consistency in the output representations between the two models, and a child-aware datastore construction method that improves inference efficiency by selectively distilling the parent datastore based on relevance to the child model. Experimental results on four low-resource translation tasks show that kNN-TL outperforms strong baselines. Extensive analyses further demonstrate the effectiveness of our approach. Code and scripts are freely available at https://github.com/NLP2CT/kNN-TL .

## introduction although deep learning has significantly advanced the field of neural machine translation @xcite @xcite , the standard training procedure of nmt is not well-suited for languages with only a small amount of bilingual data, leading to challenges in developing nmt models for low-resource languages @xcite . to overcome this limitation, transfer learning has been proposed as an effective method to enhance low-resource nmt through the parent-child framework. this framework transfers knowledge from a * co-corresponding author ## knn-tl this section introduces the knn-tl method in detail. it begins by clarifying the motivation for the work by comparing knn-tl to previous methods. the training process of knn-tl is then presented with a specific focus on the parent-child representation alignment component for subsequent knn retrieval. after that, the steps for building a childaware datastore to improve inference speed are described. finally, the method of incorporating knowledge from the parent datastore to guide the child model during inference is presented. ## analysis in this section, we conduct extensive analyses to demonstrate the effectiveness of each component in knn-tl. by default, we choose ca-en and tr-en for the child model with the de-en parent model. ## conclusion and future works in this paper, we propose knn-tl to transfer knowledge from the parent throughout the entire developing process of child models. knn-tl aligns the output representations of parent and child during training, allowing for efficient retrieval of useful knowledge from the parent datastore. in addition, knn-mt builds a child-aware datastore by selectively distilling relevant entries of the largescale parent datastore, thereby improving the inference efficiency. experimental results on four low-resource nmt benchmarks show a continuous improvement over the other powerful transfer learning methods for nmt. further analysis reveals the effectiveness and importance to align the output representations for better model improvement. future works include:1) integrating parent datastores from different high-resource language pairs to improve the performance of the child model, and 2) analyzing the transferability of the parent model through the child-aware datastore construction.