Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation (2020)

Abstract: Scarcity of parallel sentence-pairs poses a significant hurdle for training high-quality Neural Machine Translation (NMT) models in bilingually low-resource scenarios. A standard approach is transfer learning, which involves taking a model trained on a high-resource language-pair and fine-tuning it on the data of the low-resource MT condition of interest. However, it is not clear generally which high-resource language-pair offers the best transfer learning for the target MT setting. Furthermore, different transferred models may have complementary semantic and/or syntactic strengths, hence using only one model may be sub-optimal. In this paper, we tackle this problem using knowledge distillation, where we propose to distill the knowledge of ensemble of teacher models to a single student model. As the quality of these teacher models varies, we propose an effective adaptive knowledge distillation approach to dynamically adjust the contribution of the teacher models during the distillation process. Experiments on transferring from a collection of six language pairs from IWSLT to five low-resource language-pairs from TED Talks demonstrate the effectiveness of our approach, achieving up to +0.9 BLEU score improvements compared to strong baselines.

## introduction neural models have been revolutionising machine translation (mt), and have achieved state-of-the-art for many high-resource language pairs @xcite @xcite . however, the scarcity of bilingual parallel corpora is still a major challenge for training high-quality nmt models @xcite . transfer learning by fine-tuning, from a model trained for a highresource language-pair, is a standard approach to tackle the scarcity of the data in the target low-resource language-pair @xcite @xcite . however, this is a one-to-one approach, which is not able to exploit models trained for multiple high-resource language-pairs for the target language-pair of interest. furthermore, models transferred from different high-resource language-pairs may have complementary syntactic and/or semantic strengths, hence using a single model may be sub-optimal. another appealing approach is multilingual nmt, whereby a single nmt model is trained by combining data from multiple high-resource and low-resource language-pairs @xcite @xcite . however, the performance of a multilingual nmt model is highly dependent on the types of languages used to train the model. indeed, if languages are from very distant language families, they lead to negative transfer, causing low translation quality in the multilingual system compared to the counterparts trained on the individual language-pairs @xcite . to address this problem, @xcite has proposed a knowledge distillation approach to effectively train a multilingual model, by selectively distilling the knowledge from individual teacher models to the multilingual student model. however, still all the language pairs are trained in a single model with a blind contribution during training. algorithm 1: input : d lr := {(x1, y1), .., (xn, yn)}, low resource dataset, individual models {✓ l } l @xmath0, total training epochs: n output : ✓ lr : low-resource model randomly initialize low-resource model ✓ lr ; ## adaptive knowledge distillation we address the problem of low-resource nmt, assuming that we have access to models for high resource languages, and data for low resource model. our approach relies on two main steps, (i) transferring from high-resource to low-resource language-pairs by fine tuning the high-resource models using the small amount of bilingual data, and (ii) adaptive distillation of knowledge from the teacher models to the student model. more specifically, given a training dataset for a low-resource language-pair, d lr := {(x 1 , y 1 ), .., (x n , y n )} and multiple individual high resource nmt models {✓ l } l @xmath1, we are interested in training a single nmt model (student) by adaptively distilling knowledge from all teachers based on their effectiveness to improve the accuracy of the student. knowledge distillation (kd) is a process of improving the performance of a simple student model by using a distribution over soft labels obtained from an expert teacher model instead of hard ground-truth labels @xcite . the training objective to distill the knowledge from a single teacher to the student involves, this is in contrast to the negative log-likelihood training objective, which only provides supervision signal based on the correct target words according to the bilingual training data, given a collection of teacher models {✓ l } l @xmath2, we pose the following training objective, ## conclusion in this paper, we present an adaptive knowledge distillation approach to improve nmt for low-resource languages. we address the inefficiency of the original transfer learning and multilingual learning by making wiser use of all high-resource languages and models in an effective collaborative learning manner. our approach shows its effectiveness in translation of low-resource languages especially when there are complementary knowledge in multiple high-resource languages from the same linguistic family and it is not explicitly clear which language has more impact in every mini-batch of low-resource training data. experiments on the translation of five extremely low-resource languages to english show improvements compared to the strong baselines. ref and great creativity is needed to do what it does so well : to provoke us to think differently with dramatic creative statements . individual kepler great mission mission to do it as well : to grow us to think with dramatic creativity . teacher (ru-en) and the first creativity needed to do what it does : to promote us to think about the dramatic creativity . teacher (de-en) now , the future creativity needs to do it as it does : to provoke us to think differently with dramatic creative expression . teacher (it-en) now , the future creativity is needed to do what it does so well : to provocate us to think differently about dramatic reactive . teacher (es-en) the future of creativity to do that as it's doing so good : to provocate us to think differently about dramatic creativity . teacher (pl-en) the future of creativity to do what it does so good : to promise others with dramatic creativity . teacher (nl-en) now , the frequent creativity is to make it that it makes so good : to provoke us with dramatic creative . proposed adapt. kd now , they need great creativity to do what it does so well : provoke us to think differently with dramatic creativity.