Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation (2020)

Abstract: We explore best practices for training small, memory efficient machine translation models with sequence-level knowledge distillation in the domain adaptation setting. While both domain adaptation and knowledge distillation are widely-used, their interaction remains little understood. Our large-scale empirical results in machine translation (on three language pairs with three domains each) suggest distilling twice for best performance: once using general-domain data and again using in-domain data with an adapted teacher.

## introduction machine translation systems rely on large amounts of data to deduce the rules underlying translation from one language to another. this presents challenges in some important niche domains, such as patent and medical literature translation, due to the high cost of hiring experts to generate suitable training data. a cost-effective alternative is domain adaptation, which leverages large amounts of parallel documents from less difficult and more readily-available domains, such as movie subtitles and news articles. domain adaptation works well in practice. however, these large datasets, which we call general domain datasets, introduce some scalability problems. large datasets require large models; neural machine translation systems can take days or weeks to train. some models require gigabytes of disk space, making deployment to edge computing devices challenging. they can also require excessive compute during inference, making them slow and costly to scale up in production environments @xcite . to alleviate these issues, knowledge distillation (aka teacher-student) @xcite is used to compress models into a manageable form. but although knowledge distillation is the most commonly used form of model compression in practice, it is also one of the least understood. in this work, we perform a large-scale empirical analysis to attempt to discover best practices when using knowledge distillation in combination with domain adaptation. out of several common-sense configurations, we find that two stages of knowledge distillation give the best performance: one using general-domain data and another using in-domain data with an adapted teacher. we perform experiments on multiple language pairs (russian-english, german-english, chinese-english), domains (patents, subtitles, news, ted talks), and student sizes. ## conclusion in this work, we conducted a large-scale empirical investigation to determine best practices when using sequence-level knowledge distillation and domain adaptation in combination. we found that adapting models from the general-domain makes them better teachers and that distilling using general-domain data does not impact a model's adaptability. this leads us to recommend distilling twice for best results: once in the general-domain to possibly improve student performance, and again using an adapted in-domain teacher. the results are robust among multiple language pairs, student sizes, in-domain settings.