C onsist TL : Modeling Consistency in Transfer Learning for Low-Resource Neural Machine Translation (2022)

Abstract: Transfer learning is a simple and powerful method that can be used to boost model performance of low-resource neural machine translation (NMT). Existing transfer learning methods for NMT are static, which simply transfer knowledge from a parent model to a child model once via parameter initialization. In this paper, we propose a novel transfer learning method for NMT, namely ConsistTL, which can continuously transfer knowledge from the parent model during the training of the child model. Specifically, for each training instance of the child model, ConsistTL constructs the semantically-equivalent instance for the parent model and encourages prediction consistency between the parent and child for this instance, which is equivalent to the child model learning each instance under the guidance of the parent model. Experimental results on five low-resource NMT tasks demonstrate that ConsistTL results in significant improvements over strong transfer learning baselines, with a gain up to 1.7 BLEU over the existing back-translation model on the widely-used WMT17 Turkish-English benchmark. Further analysis reveals that ConsistTL can improve the inference calibration of the child model. Code and scripts are freely available at https://github.com/NLP2CT/ConsistTL.

## introduction neural machine translation (nmt) has achieved success on the high-resource language pairs @xcite . however, it achieves an inadequate performance on the low-resource language pairs with merely bilingual data due to the data sparsity @xcite . transfer learning is a simple and powerful method that can be used to boost the model performance of low-resource nmt, which can transfer knowledge from an off-the-shelf highresource parent model to the low-resource child model @xcite . the goal of transfer learning for low-resource nmt is to transfer knowledge sufficiently from the parent model. prior works attempt to transfer more information from the embedding layer of the parent model, using methods such as building a joint dictionary @xcite , creating cross-lingual token mapping @xcite , and transferring the word of same form @xcite . these strategies achieve success in parameter transfer. the methods used in these works is summarized in figure 1(a) : the parent model transfers its parameters to the child model once and lacks continual guidance on the child training. this method can not utilize the parent model sufficiently, which potentially limits the learning of the child model. this paper attempts to leverage a continual knowledge transfer from the parent model for the child training. the prediction distribution of the parent model is usually more informative since it has learned to predict various translations during training @xcite . therefore, utilizing the prediction of the parent model to continuously guide the child training might be a nice plus. to this end, we propose consistency-based transfer learning (consisttl), a novel framework that can be used to model the consistency between the parent model and the child model during the child training. for each instance from the child data, consisttl constructs a semanticallyequivalent pseudo source sentence for the parent by translating the child target sentence. then con-sisttl encourages the child prediction to be consistent with the parent prediction since these two predictions are conditioned on the same target sentence and semantically-equivalent source sentences. consisttl is summarized in figure 1(b) : unlike in vanilla tl, the parent model releases continual guidance for the child model during training by receiving the transformation of the training data of the child model. with consisttl, the child model can acquire more knowledge from the parent predictions beyond the simple parameter transfer. we conduct experiments on five low-resource nmt benchmarks. experimental results show that consisttl achieves significant improvements over strong baselines of transfer learning. combined with the representative data augmentation method of back-translation @xcite , our method can outperform the existing method up to 1.7 bleu on the widely-used wmt17 turkish-english benchmark. an extensive analysis reveals that our method, as an approach to model external consistency, is complementary to the method of modeling inner consistency. our contributions are as follows: there are also some works that consider multilingualism to enhance transfer learning. @xcite ; @xcite propose to transfer knowledge from a multilingual parent model. @xcite develop a universal architecture with shared lexical and sentence level representation to implement transfer learning. @xcite propose a meta-learning approach to learn the initialization process using several high-resource language pairs. in this paper, we mainly focus on the bilingual parent model, since off-the-shelf bilingual models are easier to obtain. consistency learning in nmt consistency learning aims to model consistency across different model predictions. the core idea is to craft the cross-view supervision and enhance cross-view consistency, which enriches the supervision signals. in the early studies, consistency learning methods focus on semi-supervised learning since consistency signals enable models to learn from data without manual labels @xcite @xcite . the semi-supervised consistency learning also enhances nmt models with the combination of forward-translation @xcite . recent studies of consistency learning gradually consider supervised learning areas @xcite . r-drop is one of the simple and generalized methods @xcite , which enhances the consistency between multiple model structures produced by dropout, figure 2 : overview of consisttl. consisttl first constructs semantically-equivalent pseudo parent source sentences from child target sentences using the reversed parent model. then, given the sentence group with the same semantic meaning, consisttl encourages the child prediction to be consistent with the parent prediction during child training. there are some consistency learning methods specially designed for supervised nmt based on data augmentation. shen et al. (2020) 2022 ) enhance the chinese representation by modeling the consistency between stroke representation and its corresponding encryption. these consistency modeling works in nmt mainly exploit the inner model consistency, i.e., modeling consistency within the same model. compared with these works, we exploit transferable information from the cross-model consistency in transfer learning for low-resource nmt. @xcite also exploit the consistency between two models and the differences from us are: 1) we construct the parent data instead of the child data; 2) we enhance the transfer learning framework for low-resource nmt while their method relies on a pivot language to improve zero-resource nmt. ## conclusion and future work we introduce consisttl for transfer learning in low-resource nmt, which can continuously transfer the knowledge from parent prediction during child training. in order to transfer knowledge from parent prediction, we propose to model crossmodel consistency between parent model and child model with the child source sentences and pseudo parent source sentences. experimental results on five translation benchmarks verify the effectiveness of consisttl, which can significantly improve the translation performance. consisttl is complementary to other powerful methods for low-resource nmt (e.g., back-translation). further analysis reveals that introducing parent prediction is helpful to shape child model prediction distribution, resulting in better inference calibration. in the future, we would like to apply curriculum learning @xcite to better organize the learning of the child model. it is also worthwhile to enhance the parent and child models by utilizing pre-trained knowledge learned from unlabeled data @xcite .