An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Models (2023)

Abstract: Knowledge distillation (KD) is a well-known method for compressing neural models. However, works focusing on distilling knowledge from large multilingual neural machine translation (MNMT) models into smaller ones are practically nonexistent, despite the popularity and superiority of MNMT. This paper bridges this gap by presenting an empirical investigation of knowledge distillation for compressing MNMT models. We take Indic to English translation as a case study and demonstrate that commonly used language-agnostic and language-aware KD approaches yield models that are 4-5x smaller but also suffer from performance drops of up to 3.5 BLEU. To mitigate this, we then experiment with design considerations such as shallower versus deeper models, heavy parameter sharing, multistage training, and adapters. We observe that deeper compact models tend to be as good as shallower non-compact ones and that fine-tuning a distilled model on a high-quality subset slightly boosts translation quality. Overall, we conclude that compressing MNMT models via KD is challenging, indicating immense scope for further research.

## experiments we now focus on indic-to-english translation as a case study and describe experiments we conducted to compress indictrans, a 474m parameter model. ## results this section presents the results of applying knowledge distillation (kd) approaches to compress the indictrans indic-to-english teacher model. ## conclusion and future work in this paper we have empirically studied the compression of mnmt models, taking indic to english translation as a case study, and explored the effectiveness of prominent knowledge distillation approaches. we have also studied the impact of model size, parameter sharing, multi-stage training, and quality of training data. we confirm the high difficulty of this task but make several recommendations that we expect will benefit practitioners. having noted the positive impact of high-quality data, we will explore this aspect in further detail in the future. we will also expand to mnmt models focusing on other language groups. finally, the impact of post-training quantization approaches and low-precision decoding will also be investigated.