F ast BERT : a Self-distilling BERT with Adaptive Inference Time (2020)

Abstract: Pre-trained language models like BERT have proven to be highly performant. However, they are often computationally expensive in many practical scenarios, for such heavy models can hardly be readily implemented with limited resources. To improve their efficiency with an assured model performance, we propose a novel speed-tunable FastBERT with adaptive inference time. The speed at inference can be flexibly adjusted under varying demands, while redundant calculation of samples is avoided. Moreover, this model adopts a unique self-distillation mechanism at fine-tuning, further enabling a greater computational efficacy with minimal loss in performance. Our model achieves promising results in twelve English and Chinese datasets. It is able to speed up by a wide range from 1 to 12 times than BERT if given different speedup thresholds to make a speed-performance tradeoff.

## introduction last two years have witnessed significant improvements brought by language pre-training, such as bert @xcite , gpt @xcite , and xlnet @xcite . by pretraining on unlabeled corpus and fine-tuning on labeled ones, bert-like models achieved huge gains on many natural language processing tasks. despite this gain in accuracy, these models have greater costs in computation and slower speed at inference, which severely impairs their practicalities. actual settings, especially with limited time and resources in the industry, can hardly enable such models into operation. for example, in tasks like sentence matching and text classification, one often requires to process billions of requests per second. what's more, the number of requests varies with time. in the case of an online shopping site, the number of requests during the holidays is five to ten times more than that of the workdays. a large number of servers need to be deployed to enable bert in industrial settings, and many spare servers need to be reserved to cope with the peak period of requests, demanding huge costs. to improve their usability, many attempts in model acceleration have been made, such as quantinization @xcite , weights pruning @xcite , and knowledge distillation (kd) @xcite . as one of the most popular methods, kd requires additional smaller student models that depend entirely on the bigger teacher model and trade task accuracy for ease in computation @xcite . reducing model sizes to achieve acceptable speed-accuracy balances, however, can only solve the problem halfway, for the model is still set as fixated, rendering them unable to cope with drastic changes in request amount. by inspecting many nlp datasets @xcite , we discerned that the samples have different levels of difficulty. heavy models may overcalculate the simple inputs, while lighter ones are prone to fail in complex samples. as recent studies @xcite have shown redundancy in pre-training models, it is useful to design a onesize-fits-all model that caters to samples with varying complexity and gains computational efficacy with the least loss of accuracy. based on this appeal, we propose fastbert, a pre-trained model with a sample-wise adaptive mechanism. it can adjust the number of executed layers dynamically to reduce computational steps. this model also has a unique self-distillation process that requires minimal changes to the structure, achieving faster yet as accurate outcomes within a single framework. our model not only reaches a comparable speedup (by 2 to 11 times) to the bert model, but also attains competitive accuracy in comparison to heavier pre-training models. experimental results on six chinese and six english nlp tasks have demonstrated that fastbert achieves a huge retrench in computation with very little loss in accuracy. the main contributions of this paper can be summarized as follows: knowledge distillation: many attempts have been made to distill heavy models (teachers) into their lighter counterparts (students). pkd-bert @xcite adopts an incremental extraction process that learns generalizations from intermediate layers of the teacher model. tinybert @xcite performs a two-stage learning involving both general-domain pre-training and taskspecific fine-tuning. distilbert @xcite big model further leveraged the inductive bias within large models by introducing a triple loss. as shown in figure 1 , student model often require a separated structure, whose effect however, depends mainly on the gains of the teacher. they are as indiscriminate to individual cases as their teachers, and only get faster in the cost of degraded performance. adaptive inference: conventional approaches in adaptive computations are performed token-wise or patch-wise, who either adds recurrent steps to individual tokens @xcite or dynamically adjusts the number of executed layers inside discrete regions of images @xcite . to the best of our knowledge, there has been no work in applying adaptive mechanisms to nlp pre-training language models for efficiency improvements so far. ## methodology distinct to the above efforts, our approach fusions the adaptation and distillation into a novel speed-up approach, shown in figure 2 , achieving competitive results in both accuracy and efficiency. ## experimental results in this section, we will verify the effectiveness of fastbert on twelve nlp datasets (six in english and six in chinese) with detailed explanations. ## conclusion in this paper, we propose a fast version of bert, namely fastbert. specifically, fastbert adopts a self-distillation mechanism during the training phase and an adaptive mechanism in the inference phase, achieving the goal of gaining more efficiency with less accuracy loss. self-distillation and adaptive inference are first introduced to nlp model in this paper. in addition, fastbert has a very practical feature in industrial scenarios, i.e., its inference speed is tunable. our experiments demonstrate promising results on twelve nlp datasets. empirical results have shown that fastbert can be 2 to 3 times faster than bert without performance degradation. if we slack the tolerated loss in accuracy, the model is free to tune its speedup between 1 and 12 times. besides, fastbert remains compatible to the parameter settings of other bert-like models (e.g., bert-wwm, ernie, and roberta), which means these public available models can be readily loaded for fastbert initialization.