Exploring the Boundaries of Low-Resource BERT Distillation (2020)

Abstract: In recent years, large pre-trained models have demonstrated state-of-the-art performance in many of NLP tasks. However, the deployment of these models on devices with limited resources is challenging due to the models’ large computational consumption and memory requirements. Moreover, the need for a considerable amount of labeled training data also hinders real-world deployment scenarios. Model distillation has shown promising results for reducing model size, computational load and data efficiency. In this paper we test the boundaries of BERT model distillation in terms of model compression, inference efficiency and data scarcity. We show that classification tasks that require the capturing of general lexical semantics can be successfully distilled by very simple and efficient models and require relatively small amount of labeled training data. We also show that the distillation of large pre-trained models is more effective in real-life scenarios where limited amounts of labeled training are available.

## introduction in recent years, large pre-trained models such as bert @xcite , gpt-2 @xcite and xlnet @xcite have demonstrated state-of-the-art performance in many nlp tasks and have become standard. however, the deployment of these models on devices with limited resources is challenging due to the models' large computational consumption and memory requirements. for example, the two variants of bert, named bert base and bert large consist of approximately 110m and 340m parameters, respectively. another deployment hurdle in real-world scenarios is the scarcity of labeled data resources. model distillation @xcite has shown promising results for reducing model size and computational load while preserving much of the original model's performance. a typical model distillation setup includes two stages; in the first stage, a large, cumbersome and accurate teacher neural network is trained for a specific downstream task. in the second stage a smaller and simpler student model, that is more practical for deployment in environments with limited resources, is trained to mimic the behavior of the teacher model. prior work related to transformer-based model distillation, focused on reducing the number of layers of the original model, obtaining shallower and more efficient student models @xcite @xcite . @xcite proposed a bert distillation method for single sentence classification tasks and sentence matching tasks using a bilstm @xcite student model. our work is closely related to the work of @xcite , however, in our work we push the boundaries of bert model distillation in terms of model size and complexity reduction, computational load and data scarcity for single-sentence classification tasks. the contribution of this paper is twofold; first, we show that classification tasks that require the capturing of general lexical semantics can be successfully distilled by simple and efficient models while preserving results comparable to those achieved by bert. second, building on previous work @xcite , we show that the distillation of large pre-trained models is more effective in real-life scenarios, where a limited amount of labeled training is available. moreover, we show that in low data resource scenarios, the distillation model size and complexity can be substantially reduced. specifically, we show that results produced by using a very simple and efficient model such as continuous bag of words (cbow) with a feed forward network(ffn) are comparable to results produced by using a more complex model such as bilstm. ## approach the aim of a model distillation process is to use a large pre-trained teacher model to train a small and computationally efficient student model so it achieves accuracy comparable to that of the teacher model. in this section we describe the teacher and student model architectures (sections 2.1) and the distillation process (section 2.2). ## conclusion we showed that in low resource scenarios, it is feasible to distil bert using very efficient models while preserving comparable results. however, the success of the distillation depends on the dataset and task at hand. classification tasks that require capturing of general lexical semantics can be successfully distilled by very simple and efficient models; however, classification tasks that require the detection of linguistic structure and contextual relations are more challenging for distillation using simple student models. for future work, we aim to explore the impact of the datasets' linguistic structures on the distillation success and to develop dataset-related measurements @xcite for predicting the success of the distillation in relation to different student models.