Taking Actions Separately: A Bidirectionally-Adaptive Transfer Learning Method for Low-Resource Neural Machine Translation (2022)

Abstract: Training Neural Machine Translation (NMT) models suffers from sparse parallel data, in the infrequent translation scenarios towards low-resource source languages. The existing solutions primarily concentrate on the utilization of Parent-Child (PC) transfer learning. It transfers well-trained NMT models on high-resource languages (namely Parent NMT) to low-resource languages, so as to produce Child NMT models by fine-tuning. It has been carefully demonstrated that a variety of PC variants yield significant improvements for low-resource NMT. In this paper, we intend to enhance PC-based NMT by a bidirectionally-adaptive learning strategy. Specifically, we divide inner constituents (6 transformers) of Parent encoder into two “teams”, i.e., T1 and T2. During representation learning, T1 learns to encode low-resource languages conditioned on bilingual shareable latent space. Generative adversarial network and masked language modeling are used for space-shareable encoding. On the other hand, T2 is straightforwardly transferred to low-resource languages, and fine-tuned together with T1 for low-resource translation. Briefly, T1 and T2 take actions separately for different goals. The former aims to adapt to characteristics of low-resource languages during encoding, while the latter adapts to translation experiences learned from high-resource languages. We experiment on benchmark corpora SETIMES, conducting low-resource NMT for Albanian (Sq), Macedonian (Mk), Croatian (Hr) and Romanian (Ro). Experimental results show that our method yields substantial improvements, which allows the NMT performance to reach BLEU4-scores of 62.24%, 56.93%, 50.53% and 54.65% for Sq, Mk, Hr and Ro, respectively.

## introduction nmt has achieved significant improvements @xcite in recent * corresponding author. years. nevertheless, it heavily relies on large-scale observable parallel corpora. as a result, nmt generally fails to perform perfectly in an infrequent translation scenario, where the available parallel data for training is sparse. for example, the size of training data for nmt between english (en) and macedonian (mk) is about 200k, which is significantly smaller than that (582m) between english (en) and german (de). the issue has been widely known as low-resource nmt. the existing studies attempt to overcome the issue primarily by 1) producing cross-language embeddings, 2) constructing bilingual shareable latent space for encoding, and 3) transferring well-trained models to low-resource languages. we overview the studies in section 2. within the aforementioned arts, nowadays, parent-child (pc) transfer learning @xcite represents a considerable advance in our knowledge. it allows a parent nmt model to be fully trained and developed over highresource languages (e.g., that for @xmath0, and transfers it to low-resource languages (e.g., that for @xmath1. this contributes to the construction of a child nmt model that inherits the translation experiences of parent model. the recent experimental results suggest that pc transfer learning suffers from the weak perception of semantics in low-resource languages, at the very beginning of encoding. in other words, fine-tuning parent nmt model over low-resource languages is unavoidably started from scratch. this results in less effective and inefficient representation learning. though, it is proven that duplicating embeddings @xcite of cross-language shareable tokens, synonyms and mutually-aligned tokens helps to alleviate the coldstart fine-tuning problem. this also implies that conventional methods of constructing shareable latent space @xcite may produce similar but general effects. in this paper, we intend to strengthen pc trans-fer learning by coupling it with space-shareable encoding. different from the previous work, we neither use an unabridged parent encoder for spaceshareable encoding, nor directly transfer it towards low-resource translation. instead, we divide the unabridged parent encoder into two parts. one part engages in space-shareable encoding for alleviating cold-start fine-tuning problems. the other is straight transferred without being "brainwashed" for the pre-existing translation experiences (i.e., the ones learned during high-resource nmt). the goal is to fulfill bidirectional adaptation, i.e., 1) establishing the encoding mode that adapts to linguistic characteristics of low-resource languages, conditioned on the shareable latent space; and 2) preserving the encoding mode that adapts to original translation experiences of the parent nmt model. in our experiments, we follow @xcite to build a transformer-based nmt model within the encoder-decoder architecture, where both encoder and decoder comprise 6 transformer layers. we intensively train it on large-scale highresource language pairs to produce a knowledgeable parent nmt model. on the basis, we take parent's encoder, and divide it into two teams: t1 (1 st transformer layer) and t2 (2 nd -6 th transformer layers). we train t1 to perform space-shareable encoding for low-resource languages. and we carry out monolingual unsupervised learning when training t1, where the generative adversarial network and masked language model are used. when moulding the child nmt model, we transfer parent to low-resource languages as usual. the difference lies in that t2 in parent performs hot-start encoding by absorbing "home-made" hidden states from t1, i.e., the ones fabricated by t1 in terms of both monolingual features of low-resource languages and distributions in shareable latent space. we conduct experiments on corpora setimes, where the low-resource mt scenarios of sq-en, mk-en, hr-en and ro-en are considered. experimental results show that our method yields substantial improvements, and achieves competitive performance compared to the state of the art. ## related work there are a variety of advanced methodologies proposed for tackling low-resource nmt. due to page limitation, we merely overview the closely-related arts reported in recent five years. the studies demonstrate the versatility of spaceshareable encoding for multilingual translation in simulated experiments, where high-resource language pairs (e.g., en-de) are used though the size of training data is reduced. soon after, @xcite prove that space-shareable encoding fails to obtain promising performance for authentic low-resource scenarios (e.g., en-nepali and en-sinhala). @xcite suggest that weak isomorphism between non-family languages results in the performance degradation. to strengthen space-shareable encoding, recently, multilingual bart @xcite is used in the unsupervised nmt framework, together with denoising autoencoder @xcite and multitask learning @xcite . the first study for alleviating weak isomorphism most probably derives from @xcite 's effort, where perplexity-based similarity computation is utilized to automatically select most relevant high-resource languages for space-shareable encoding. the obtained improvements in this study imply that common linguistic units (isomorphic constituents) between high-resource and low-resource languages serve as informative seeds for harvesting embeddings of heterogeneous constituents. it raises the interest in building shareable vocabulary. kim et al. (2018) construct a synthetic dictionary by iteratively updating linear mapping relationships between bilingual embeddings. aji et al. (2020) build a joint vocabulary where the matched tokens are assigned with the same embeddings, while the mismatched the randomly-initialized em-denotes the prediction network denotes the initialization denotes the transfer channel denotes the prediction network denotes the initialization denotes the transfer channel transformer layer 5-transformer layer embedding layer decoder body embedding layer decoder embedding layer transformer layer discriminator composite embedding layer 5-transformer layer transformer layer predictor adversarial training mask language modeling 5-transformer layer language a language a language c language b transformer layer composite embedding layer decoder body embedding layer encoder decoder language c language b bidirectionally-adaptive learning parent model child model denotes the transferable block denotes the prediction network denotes the initialization denotes the transfer channel t2 t1 interactive training denotes the modules initialized by the parent model's parameters denotes the transferable block denotes the prediction network denotes the initialization denotes the transfer channel beddings. @xcite retrain byte-pair-encoding (bpe) over bilingual hybrid corpus, and use the segmented tokens by bpe to build the shareable vocabulary. @xcite carry out word alignment between low-resource and highresource languages, and share embeddings among aligned sub-tokens. this effectively expands the existing shareable vocabularies. ## approach we show the architecture of our low-resource nmt model in figure 1 . the 12-layer transformer-based encoder-decoder network (at the left side in figure 1 ) serves as parent nmt model, which contains six layers of transformer encoder and six layers of transformer decoder. from here on, we refer them as encoder and decoer layers respectively. the parent nmt model has been intensively trained to perform translation for high-resource language pairs a and b (i.e., nmt for @xmath2. we divide parent encoder layers into two teams t1 and t2, where t1 is constituted merely with the 1 st encoder layer, while t2 the rest five encoder layers. on the basis, t1 is pushed into the bidirectionally-adaptive learning channel (which is marked by the dotted rectangular box with a purple background in figure 1 ). during the adaptation process, t1 engages in the bilingual encoding program launched by the generative adversarial network, so as to learn space-shareable encoding mode for both high-resource language a and low-resource language c. besides, t1 engages in the masked language modeling program for low-resource language c, with the aim to learn language-specific encoding mode in terms of distinct characteristics of c. t1 will be trained iteratively and alternately in the two programs. this results in a refined t1. we construct child nmt model by transferring parent decoder and t2 to low-resource language pairs c and b (nmt for @xmath3, and connect t2 with the refined t1. on the basis, we fine-tune child nmt model using small-scale low-resource parallel data. the child model is deployed at the right side in figure 1 . instead, the t1 and t2 we used come from the decoder of the parent model (@xmath4. in the rest of this section, we detail all the components of our model. ## conclusion we propose a bidirectionally-adaptive transfer learning (batl) approach to enhance lowresource nmt models. experimental results show that our approach yields substantial improvements, compared to the state of the art. in addition, it is demonstrated that batl is compatible with btbased data augmentation. combining batl and bt obtains additional performance gains. in a series of auxiliary experiments, we analyze the effects of various parent nmt models and multilayer batl on low-resource nmt, some of which are negative and therefore noteworthy for risks in real applications. the commonality of linguistics stands for the fundamental principle in prompting parent-child transfer for low-resource nmt. different family languages hold inconsistent commonalities with a specific low-resource language. considering this phenomenon, in the future, we will study on a multilingual parent-child transfer learning. a selective transfer will be developed, in terms of casespecific adhesion to different high-resource family languages. the adhesion will be perceived by modeling the relevance of topics, provenances and domains, as well as document-level structure information (e.g., monolingual discourse relationships and rhetorics).