In Neural Machine Translation, What Does Transfer Learning Transfer? (2020)

Abstract: Transfer learning improves quality for low-resource machine translation, but it is unclear what exactly it transfers. We perform several ablation studies that limit information transfer, then measure the quality impact across three language pairs to gain a black-box understanding of transfer learning. Word embeddings play an important role in transfer learning, particularly if they are properly aligned. Although transfer learning can be performed without embeddings, results are sub-optimal. In contrast, transferring only the embeddings but nothing else yields catastrophic results. We then investigate diagonal alignments with auto-encoders over real languages and randomly generated sequences, finding even randomly generated sequences as parents yield noticeable but smaller gains. Finally, transfer learning can eliminate the need for a warm-up phase when training transformer models in high resource language pairs.

## introduction transfer learning is a common method for lowresource neural machine translation (nmt) @xcite @xcite @xcite . however, it is unclear what settings make transfer learning successful and what knowledge is being transferred. understanding why transfer learning is successful can improve best practices while also opening the door to investigating ways to gain similar benefits without requiring parent models. in this paper, we perform several ablation studies on transfer learning in order to understand what information is being transferred. we apply a black box methodology by measuring the quality of end-to-end translation systems. typically, our experiments have a baseline that was trained from scratch, an off-the-shelf transfer learning baseline and simplified versions of the transfer learning scheme. if a simplified version recovers some of the quality gains of full transfer learning, it suggests that the simplified version has captured some of the information being transferred. since information may be transferred redundantly, our claims are limited to sufficiency rather than exclusivity. transferring word embeddings is not straightforward since languages have different vocabularies. @xcite claimed that vocabulary alignment is not necessary, while @xcite and @xcite suggest a joint vocabulary. we find that the vocabulary has to be aligned before transferring the embedding to achieve a substantial improvement. transfer learning without the embedding or with vocabulary mismatches is still possible, but with lower quality. conversely, transferring only the word embeddings can be worse than transferring nothing at all. a rudimentary model of machine translation consists of alignment and token mapping. we hypothesize that these capabilities are transferred across languages. to test this, we experiment with transferring from auto-encoders that learn purely diagonal alignment and possibly language modelling. to remove the effect of language modelling, we train auto-encoders on random strings sampled uniformly. however, all of these scenarios still have simple copying behaviour, especially with tied embeddings. therefore, we also attempt a bijective vocabulary mapping from source to target, forcing the model to learn the mapping as well. curiously, parents trained with bijectively-mapped vocabularies transfer slightly better to children. we then investigate transfer learning for highresource children, where the goal is reduced training time since they mainly attain the same quality. transfer learning primarily replaces the warm-up period, though only real language parents yielded faster training. ## related work transfer learning has been successfully used in lowresource scenarios for nmt. @xcite gain 5 bleu points in uzbek-english by transferring from french-english. their style of transfer learning copies the entire model, including word embeddings, ignoring the vocabulary mismatch between parent and child. they used separate embeddings for source and target language words, whereas tied embeddings @xcite have since become the de-facto standard in low-resource nmt. tied embeddings provide us with the opportunity to revisit some of their findings. in section 5, we find an english-english copy model does work as a parent with tied embeddings, whereas @xcite reported no gains from a copy model with untied embeddings. methods to cope with vocabulary mismatch have improved since @xcite . @xcite suggest that a shared vocabulary between the parent language and the child is beneficial, though this requires knowledge of the child languages when the parent is trained. addressing this issue, @xcite proposed a universal vocabulary for transfer learning. their universal vocabulary was obtained by jointly training the sub-word tokens across multiple languages at once, applying romanisation to languages in non-latin scripts. however, unseen languages may only be representable in this universal vocabulary with a very aggressive and potentially sub-optimal subword segmentation. orthogonally, @xcite ; @xcite ; @xcite ; @xcite use bilingual word embedding alignment to initialise the embedding layer to tackle low resource language pairs. in section 4.2, we compare a variety of vocabulary transfer methods. prior work @xcite stated that a related language is the best parent for transfer learning. @xcite explore options to choose the best parent and conclude that the best parent language might not necessarily be related but is instead based on external factors such as the corpus size. in section 3, we try two parent models in both directions to set baselines for the rest of the paper; an exhaustive search is not our main purpose. another approach to low-resource (or even zero-shot) nmt is through multilingual models @xcite , which is similar to training the parent and child simultaneously. a related idea creates meta-models with vocabulary residing in a shared semantic space @xcite . if there is more parallel data with a third language, often english, then pivoting through a third language can outperform direct translation @xcite . this approach requires enough sourcepivot and target-pivot parallel data, which is arguably hard in many low resource scenarios, such as burmese, indonesian, and turkish. orthogonal to transfer learning, @xcite and @xcite have proposed a fully zero-shot approach for low resource languages that relies on aligning separately-trained word embeddings to induce an initial bilingual dictionary. the dictionary is then used as the basis for a translation model. however, these methods do not generalise to arbitrary language pairs @xcite . moreover, our setting presumes a small amount of parallel data in the low-resource pair. ## baseline transfer learning we start with arguably the simplest form of transfer learning: train a parent model then switch to training with the child's dataset following @xcite . we attempt to initialise the embedding vectors of the same tokens from the parent to the child. we later investigate different approaches to transferring the embeddings. as transfer learning requires a parent model, we start by sweeping different high-resource languages for the parent model to set a baseline. choosing a parent language pair is one of the first issues to solve when performing a transferlearning experiment. however, this is not a simple task. prior work @xcite suggest that a related language is the best option, albeit related is not necessarily well defined. recently, @xcite performed a grid-search across various parent languages to determine the best criteria for selecting the optimal parent when performing transfer learning. their work showed that the best language parents might also be determined by external factors such as the corpus size, on top of the language relatedness. according to the bleu score, the difference between various parents is usually not that significant. we first explore four potential parents: german and russian from/to english. from each of them, we transfer the parameters to our low-resource language pair of {burmese, indonesian, turkish} to english. before presenting the results, we lay out the experimental setup used for the rest of the paper. ## transferring embedding information parent and child languages have a different vocabulary, so embeddings are not inherently transferable. we investigate what is transferred in the embeddings and evaluate several vocabulary combination methods. ## transferring structural information to understand what information is being transferred with transfer learning, we test the parent model's performance on the child language without any additional training. when a pre-trained model is transferred to another language pair, the model has not yet seen the child language vocabulary. when presented with an input in a new language, the model is unable to translate correctly. however, as we can see in based on this observation, we see that fallback copying behaviour, including monotonic alignment, is transferred. this can be useful for named entity translation @xcite . to test our claim, we prepare parents that implicitly learn to copy or transform input tokens diagonally. we can create a copy sequence model (or autoencoder) model by giving the model the same sentences for both source and target. we pick an english monolingual dataset. we also use a chinese monolingual corpus to explore whether the chosen parent shared example @xmath0: bank mandiri bisa masuk dari mikro hingga korporasi . to simulate the translation behaviour better, we also prepare a substitution parallel corpus. we transform every token into another based on a predetermined 1:1 mapping. we create a substitution corpus for both the english and the synthetic corpus. with tied embeddings, the substitution corpus should help the model translate one token into another, instead of just copying. we perform transfer learning experiments from every monolingual and synthetic parent to all three child languages, as summarised in in general, transferring from any monolingual or synthetic parent yields better bleu (or faster convergence for turkish) compared to training from scratch. although, the improvement is suboptimal when compared with transfer learning from a proper parent. however, we can use these gains to measure the information transferred in transfer learning. in general using monolingual english is better than using monolingual chinese. in monolingual english, we can transfer the embedding information correctly with token matching. therefore, con-sistent with our previous experiment, embedding information is transferred. using a chinese parent is better than using random sequences. our random sequence is uniformly sampled independently for each token. therefore, unlike a real monolingual corpus, learning language modelling from this random sequence is impossible. thus, we conclude that the model transfers some statistical properties of natural languages. transferring from a random sequence copy model yields better result compared to training the model from scratch. while the improvement is minimal, we can see that a naïve model that performs copying is better as a model initialisation. moreover, substitution sequence parent models perform better than their copying counterparts. we suspect that copy models with tied embeddings converge to a local optimum that is a poorer initialisation for other translation models, compared to the substitution models. transfer learning with an actual nmt system as a parent still outperforms the monolingual and synthetic parents, albeit they are initially a copy model. we argue that the monolingual parents perform nearly perfectly at the copying task, and have perfect diagonal alignment, and therefore overfit to this artificial setting when used as a parent. ## transfer learning for high-resource languages transfer learning can be used to initialise a model even if final quality does not change. compared to random initialisation, we argue that a pre-trained model functions as better initialisation. therefore, since we initialise the model better, it should converge faster. this behaviour was already presented in for this experiment, we take an english-to-russian model as a parent for an english-to-german model. we align the embedding with the same bpe tokens instead of using a joint vocabulary since this would require re-training the parent. we also attempt to exclude the embedding completely. these choices are practical in a realworld scenario, especially when we measure for efficiency. in in the early stage of training, the gradients produced are quite noisy, which is particularly harmful to the transformer model @xcite . therefore, training transformer models usually require a precise warm-up setup. however, transfer ## conclusion we demonstrate that the internal layers of the network are the most crucial for cross-lingual transfer learning. the embeddings contain transferable information, as long as the vectors are mapped correctly and the inner layers are also transferred. while not as optimal, we can still perform transfer learning by excluding the embedding. in transfer learning, we can also transfer the alignment. transferred parents without fine-tuning will align the input diagonally and copy most of the tokens. we further demonstrate that transfer learning still functions with a simple copy model, even with an artificial dataset-albeit with a reduced quality. from a theoretical perspective, our results indicate that while transfer learning is effective in our scenario, it performed less "transfer" than previously thought. therefore, a promising research direction to investigate would involve the development and assessment of improved initialisation methods that would more efficiently yield the benefits of the model transfer. from a practical perspective, our results indicate that we can initialise models with a pre-trained model regardless of the parent language or vocabulary handling. with this perspective in mind, we can use transfer learning as a better initialisation, resulting in the child model having more stable gradients from the onset of training. therefore, models can train and converge faster, which is useful in high-resource settings. with transfer learning, the model can be trained with more aggressive hyperparameters-such as removing the learning rate warm-up entirely-to further improve the convergence speed. this result further highlights the use of transfer learning as a better model initialisation.