A Novel Two-step Fine-tuning Framework for Transfer Learning in Low-Resource Neural Machine Translation (2024)

Abstract: Existing transfer learning methods for neural machine translation typically use a well-trained translation model (i.e., a parent model) of a high-resource language pair to directly initialize a translation model (i.e., a child model) of a low-resource language pair, and the child model is then fine-tuned with corresponding datasets. In this paper, we propose a novel two-step fine-tuning (TSFT) framework for transfer learning in low-resource neural machine translation. In the first step, we adjust the parameters of the parent model to fit the child language by using the child source data. In the second step, we transfer the adjusted parameters to the child model and fine-tune it with a proposed distillation loss for efficient optimization. Our experimental results on five low-resource translations demonstrate that our framework yields significant improvements over various strong transfer learning baselines. Further analysis demonstrated the effectiveness of different components in our framework.

## introduction neural machine translation (nmt) has achieved superior performance in terms of both fluency and adequacy for high-resource languages @xcite @xcite . with the introduction of the attention mechanism @xcite , nmt has been proven to be efficient and powerful in modeling long-distance dependencies. however, the performance of nmt systems deteriorates dramatically when insufficient parallel data are available for training @xcite @xcite . the scarcity of parallel corpora intensely limits the performance of an nmt system on low-resource languages. transfer learning is a learning paradigm for addressing the data scarcity problem @xcite 2016; @xcite . for nmt, transfer learning aims to transfer the knowledge from a well-trained high-resource translation model (i.e., a parent model, e.g., @xmath0.e., a child model, e.g., @xmath1. prior transfer learning methods in nmt @xcite primarily achieve knowledge transfer by initializing the parameters of the child model with the parent model and fine-tuning the child model on the corresponding data. such direct transfer of knowledge raises a vocabulary mismatch problem @xcite @xcite , and results in unsatisfied results for low-resource translations. some methods have been proposed to alleviate the vocabulary mismatch problem, such as constructing joint dictionaries or employing a crosslingual token mapping technique @xcite @xcite . @xcite proposed a token matching method that simply duplicates the embed-dings of overlapping tokens from the parent model to the child model. recently, based on the work of aji et al. (2020), @xcite proposed consisttl that uses the predictions of the parent model to continuously provide soft targets during the fine-tuning of the child model. however, given the differences between the source inputs of the parent and the child translation tasks, the parent model is not an optimal starting point for the single-step fine-tuning of the child model using limited parallel child data. therefore, it is necessary to pre-fine-tune the parent model to fit the child language before initializing the child model with it. building upon this insight, we propose a simple yet effective transfer learning framework, named two-step fine-tuning (tsft), for low-resource nmt. as shown in figure 1 , we introduce an intermediate (child) model initialized with the parent model to adjust the parent parameters to fit the child language. tsft involves two fine-tuning steps. in the first step, we feed child source sentences (i.e., monolingual data) and meaning-matched sentences in the parent source language into the intermediate and the parent models, respectively. then, the intermediate model is fine-tuned with the objective of aligning probability distributions from the parent and intermediate models, aiming to adjust the parameters transferred from the parent model to perform well with child source sentences. additionally, we propose a regularization-based strategy that can improve the translation performance of the intermediate model and benefit the child model. note that we apply the token matching method to alleviate the vocabulary mismatch problem in the first step. in the second step, we transfer the adjusted parameters from the intermediate model to the child model and fine-tune the entire child model on the pertinent parallel data, employing both a cross-entropy loss and a proposed distillation loss. extensive experiments on five low-resource translations show that tsft surpasses the strongest baseline method with up to 1.2 sacrebleu points. the ablation study demonstrates the effectiveness of different components within tsft. our contributions can be summarized as follows: ## related work existing studies have demonstrated the success of transfer learning for low-resource nmt @xcite @xcite . @xcite first introduced transfer learning into the field of nmt and proposed a parent-child framework, where parameters from a pre-trained parent model are directly transferred to a new child model with a shared target language. subsequent research largely builds upon the parent-child framework and tends to leverage highly related parent language to perform transfer learning @xcite . however, the languages closely related to low-resource languages are also low-resourced @xcite and offer only modest performance improvements. thus, researchers focused on identifying the critical factors for the effectiveness of the parent language. experimental results from @xcite emphasized that linguistic or geographical distance does not appear as important as the size of the parent data @xcite . this insight expands the range of parent languages available for transfer learning, and alleviates the limitations of highly related parent languages. consequently, later researchers shifted their attention to parent languages with low relatedness but highresourced. however, this exacerbates the vocabulary mismatch problem, posing a new challenge to transfer learning. one solution to the vocabulary mismatch problem is to build a joint dictionary before training a parent model @xcite . however, this restricts the applicability of a pre-trained parent model to a specific child model only. to overcome this limitation, @xcite proposed pre-training a language-agnostic cross-lingual word embedding independently from the parent model. concurrently, token matching methods also show their effectiveness in transfer learning without requiring additional training efforts @xcite . some other methods introduce highly related intermediate languages to gradually narrow the vocabulary disparity @xcite . these methods take advantage of both largescale data sources and syntactic similarity in the intermediate language. recently, @xcite incorporated the idea of consistency learning into transfer learning based on the work of @xcite and proposed a novel transfer learning method called consisttl. this method enables the child model to utilize the parent model during fine-tuning. subsequently, @xcite proposed knn-tl, which extends consisttl by integrating a k-nearest neighbor (knn) module, allowing the child model to utilize the parent model during inference. while our method also builds on consisttl, we focus on enhancing the child model's performance during fine-tuning. thus, our work is orthogonal to knn-tl. ## method in this section, we begin by providing an overview of the basic concepts behind transfer learning and then present our transfer learning framework, tsft, in detail. ## conclusion in this paper, we propose tsft: a novel twostep fine-tuning framework for low-resource nmt.