Combining Sequence Distillation and Transfer Learning for Efficient Low-Resource Neural Machine Translation Models (2020)

Abstract: In neural machine translation (NMT), sequence distillation (SD) through creation of distilled corpora leads to efficient (compact and fast) models. However, its effectiveness in extremely low-resource (ELR) settings has not been well-studied. On the other hand, transfer learning (TL) by leveraging larger helping corpora greatly improves translation quality in general. This paper investigates a combination of SD and TL for training efficient NMT models for ELR settings, where we utilize TL with helping corpora twice: once for distilling the ELR corpora and then during compact model training. We experimented with two ELR settings: Vietnamese–English and Hindi–English from the Asian Language Treebank dataset with 18k training sentence pairs. Using the compact models with 40% smaller parameters trained on the distilled ELR corpora, greedy search achieved 3.6 BLEU points improvement in average while reducing 40% of decoding time. We also confirmed that using both the distilled ELR and helping corpora in the second round of TL further improves translation quality. Our work highlights the importance of stage-wise application of SD and TL for efficient NMT modeling for ELR settings.

## introduction neural machine translation (nmt) @xcite enables endto-end training of translation models and is known to give state-of-the-art results for a large variety of language pairs. nmt models with large hidden sizes or deep stacked layers tend to give better translations than those with small hidden sizes or fewer layers. large models inevitably need more storage space and computation, and are difficult to deploy on low-computation and low-memory devices. additionally, beam search decoding is known to improve translation quality but needs more computation and is unacceptable in a lowlatency real-time application where faster decoding is as valuable as if not more valuable than translation quality. consequently, neural models that are compact and fast are extremely important and a growing body of research known as neural model efficiency focuses on this issue. one of the most popular techniques to train efficient models is knowledge distillation @xcite which relies on transferring the knowledge learned by a large model (called teacher) into a smaller model (called student). sequence distillation (sd) @xcite is a special case of knowledge distillation for sequence-to-sequence models, such as those used for nmt. not only does it help in the training of compact and fast models with high translation quality, it sometimes helps in eliminating the need for beam search which further increases decoding speed. sd relies on the creation of distilled parallel corpora by translating the training source sentences into the target language by using a large model. the distilled corpora are simplified representations of how the large model sees the original corpora and their quality will have a direct impact on the translation quality of compact models trained with them. while sd is known to perform extremely well for high-resource settings, its direct application to extremely low-resource (elr) settings will not work due to over-fitting. in this paper, we investigate how to train efficient (compact and fast) nmt models for elr settings with helping corpora through domain adaptation or cross-lingual tl. we use tl twice: once for distilling elr corpora and then for training efficient models. we expect that training multi-domain or cross-lingual models by simply concatenating, without oversampling, the elr corpora with the helping corpora leads to nmt models that can help generate useful distilled corpora. to evaluate the effect of our proposed method, we experimented with two elr language pairs, vietnamese-english and hindi-english (4 translation directions), in the alt dataset. when we trained compact nmt models with 40% fewer parameters only on the distilled elr corpus, the resulting models showed improved translation quality with greedy search by 3.6 bleu points in average over the models trained on the original elr corpus, while reducing 40% of decoding time. furthermore, when we jointly used the distilled elr corpora with the helping corpora via tl, the quality of the resulting compact models was further improved by up to 3.7 bleu points over the best score achieved by using no distilled data. this highlights the importance of stage-wise application of sd and tl for efficient nmt models in elr settings with high translation quality. although the individual techniques utilized in this work are not novel, their combination and our empirical observations pertaining to the development of efficient models for elr settings are novel. the contributions of our paper are as follows: our work is at the intersection of knowledge distillation @xcite and transfer learning for training compact nmt models. ## our approach: transfer-generate-transfer refer to figures 1 and 2 for a visual overview of our approaches. figure 1 depicts the application of tl to generate distilled corpora for the elr settings. figure 2 depicts how the distilled elr corpora can be used with the distilled or non-distilled helping corpora to train compact models. our method for training compact nmt models for elr settings can be summarized as follows: there are four possible ways of combining low-resource and helping corpora as each of them can be either distilled or non-distilled. standard tl takes place when both the elr and helping corpora are non-distilled. in this case, tl is not used to distill data, and the elr corpus should be oversampled to match the size of the helping corpus to ensure the best translation quality. however, for the purposes of distillation, unlike previous work, we do not oversample the l 1 -l 2 corpus before concatenating it with the l 3 -l 4 corpus. we did so because our preliminary explorations revealed that oversampling causes the model to memorize the l 1 -l 2 corpus, thereby preventing the generation of useful distilled corpora. naturally, the lack of oversampling might negatively impact on the quality of distilled l 1 -l 2 corpus. one can empirically determine an optimal oversampling rate, but we decided to not search for it in order to make our method simple. we address this point in section 5.1.3 with empirical evidence justifying our choice. note that one can pre-train compact nmt models on helping corpora and then fine-tune them on elr corpora, avoiding sd altogether. however, the quality of tl is proportional to the quality of the pre-trained model, which tends to be high when using larger models. furthermore, distilled data is prone to be simpler than the original data and thus has higher potential for leading to compact models. we hypothesize that distilling elr corpora might help in better model compression. we test this hypothesis through experiments. ## experimental settings to determine the feasibility of the proposed method, we trained and evaluated nmt models in the following two groups of settings. ## results we show results using only distilled elr corpora and then using it with helping corpora. ## conclusion in this paper, we have explored the combination of transfer learning (tl) and sequence distillation for obtaining compact and fast models in extremely low-resource (elr) settings. our experiments on four translation directions revealed that leveraging helping corpora help in distilling elr corpora that help train compact models with 3.6 average bleu points improvement in translation quality. compact models trained on distilled elr corpora are not only fast but also give better translations than larger models trained on non-distilled elr corpora. we showed the effects of choosing appropriate training methods, using domain indicator tags, and managing corpora sizes on translation quality. our cost-benefit analysis of model size, decoding speed, and translation quality showed that we can achieve translation quality comparable to baselines trained on the original elr corpora with models that are approximately 3.0 times smaller and 3.2 times faster than said baselines. we also showed that combining distilled elr corpora with the distilled or non-distilled helping corpora, using simple tl methods, can further boost the performance of compact and hence fast nmt models. we strongly recommend to leverage distilled elr corpora through stage-wise tl for compact and high-quality nmt for elr settings. in our future work, we will extend our approach for a single compact multilingual nmt model, for instance, focusing on multi-parallel alt dataset.