Limitations of Knowledge Distillation for Zero-shot Transfer Learning (2021)

Abstract: Pretrained transformer-based encoders such as BERT have been demonstrated to achieve state-of-the-art performance on numerous NLP tasks. Despite their success, BERT style encoders are large in size and have high latency during inference (especially on CPU machines) which make them unappealing for many online applications. Recently introduced compression and distillation methods have provided effective ways to alleviate this shortcoming. However, the focus of these works has been mainly on monolingual encoders. Motivated by recent successes in zero-shot cross-lingual transfer learning using multilingual pretrained encoders such as mBERT, we evaluate the effectiveness of Knowledge Distillation (KD) both during pretraining stage and during fine-tuning stage on multilingual BERT models. We demonstrate that in contradiction to the previous observation in the case of monolingual distillation, in multilingual settings, distillation during pretraining is more effective than distillation during fine-tuning for zero-shot transfer learning. Moreover, we observe that distillation during fine-tuning may hurt zero-shot cross-lingual performance. Finally, we demonstrate that distilling a larger model (BERT Large) results in the strongest distilled model that performs best both on the source language as well as target languages in zero-shot settings.

## introduction since the introduction of pretrained word embeddings @xcite , unsupervised representation learning has become an essential part of natural language processing (nlp). the idea of exploiting unlabeled corpuses to improve performance on downstream nlp tasks has been excelled in recent years by the introduction of contextualized representation learning techniques @xcite using the language modeling (lm) objective. follow-up works utilized transformer-based architecture @xcite and subword tokenization techniques @xcite to further improve these contextualized representations and achieve state-of-the-art performance on various nlp tasks when fine-tuned on labeled data @xcite . bidirectional encoder representations from transformers (bert) @xcite has also been successfully pretrained on corpuses from 100+ languages to learn a shared representation among different languages (a.k.a mbert). mbert has been demonstrated to be capable of performing cross-lingual zero-shot transfer learning given the labeled data in a single language (e.g., english) without ever seeing a parallel corpus during pretraining @xcite . moreover, it has been demonstrated that by increasing the size of the encoder and the pretaining corpus, such a multilingual encoder can achieve state-of-the-art cross-lingual zero-shot performance across many tasks @xcite . despite their success in many nlp task, bert encoders are not very attractive for practical applications due to their huge size (often a few gbs) and their latency (especially on cpu machines). hence, there has been a great effort in recent years to compress these encoders without a big drop in performance using knowledge distillation @xcite , network pruning @xcite , and quantization @xcite . however, there has been little work on compressing multilingual bert encoders and evaluating their effectiveness in maintaining their cross-lingual learning capabilities after compression, which is the main focus of our work. in this work, we focus on knowledge distillation (kd) which is an effective approach for reducing both size and latency of a bert style encoder. in particular, we first pretrain a multilingual bert base and a multilingual bert large encoders on wikipedia data of 8 languages (namely mbert8 and mbert8 large ) and follow @xcite to distill them into 4-layer encoders, namely mlean-bert and mxleanbert, respectively (see finally, we study the effectiveness of distillation during fine-tuning and demonstrate that in contrast to previous observations in monolingual settings @xcite , it can hurt zero-shot crosslingual performance. in particular, extra distillation during fine-tuning degrades zero-shot crosslingual performance of mleanbert by up to 2.5% despite improving or not changing its performance on english test set. the main contributions of this work are: ## conclusion in this work, we applied state-of-the-art knowledge distillation methods on multilingual bert encoders and demonstrated that distillation during pretraining is as effective in multilingual setting. however, in contradiction to previous observations in monolingual settings, we demonstrated that distillation during fine-tuning may degrade zero-shot cross-lingual performance. when it comes to multilingual distillation, we showed that pretraining a larger model and then distilling it results in the best performing distilled model.