Efficient Transformer Knowledge Distillation: A Performance Review (2023)

Abstract: As pretrained transformer language models continue to achieve state-of-the-art performance, the Natural Language Processing community has pushed for advances in model compression and efficient attention mechanisms to address high computational requirements and limited input sequence length. Despite these separate efforts, no investigation has been done into the intersection of these two fields. In this work, we provide an evaluation of model compression via knowledge distillation on efficient attention transformers. We provide cost-performance trade-offs for the compression of state-of-the-art efficient attention architectures and the gains made in performance in comparison to their full attention counterparts. Furthermore, we introduce a new long-context Named Entity Recognition dataset, GONERD, to train and test the performance of NER models on long sequences. We find that distilled efficient attention transformers can preserve a significant amount of original model performance, preserving up to 98.6% across short-context tasks (GLUE, SQUAD, CoNLL-2003), up to 94.6% across long-context Question-and-Answering tasks (HotpotQA, TriviaQA), and up to 98.8% on long-context Named Entity Recognition (GONERD), while decreasing inference times by up to 57.8% . We find that, for most models on most tasks, performing knowledge distillation is an effective method to yield high-performing efficient attention models with low costs.

## introduction the rise of transformer-based models @xcite has driven significant advancements in the field of natural language processing (nlp). of these models, bert @xcite produced landmark performance in a variety of nlp tasks such as question answering (qa), named entity recognition (ner), and glue @xcite . bert-based models @xcite continue to dominate the field @xcite with variations such as roberta @xcite dramatically improving performance on downstream tasks. however, bert-based models often have a fairly short maximum input length of 512 tokens, severely limiting their capabilities in long-context situations. attempting to increase this limit to allow for longer sequences often results in significantly greater computational requirements. this has given rise to the creation of efficient attention transformer models @xcite such as longformer @xcite , big bird @xcite , nyströmformer @xcite , and lsg @xcite , which can accept as input much longer sequences with reduced computational overhead by modifying and approximating bert's original attention mechanism. while efficient attention models require less computational resources on long-context tasks when compared to their non-efficient counterparts, they are still often computationally expensive to train and deploy @xcite . thus, organizations and individuals are required to grapple with increased operational costs, difficulty deploying these models on resource-limited hardware such as mobile devices, and often must rely on cloudbased solutions which restricts model availability in scenarios with limited internet access. in response to computational challenges associated with transformer models, the nlp community has invested considerable efforts into creating cheaper yet performant models. this has been particularly the case in the study of knowledge distillation (kd) @xcite . however, despite the rapid progress of kd and its effectiveness in model compression, little work has been done toward the investigation of the intersection of kd and efficient attention architectures. as such, we focus on combining these two methodologies. we believe this is an essential effort for creating models that can cheaply and effectively operate on a production scale on long-context tasks. furthermore, despite its significance in practical nlp usage, named entity recognition (ner) still does not have a well-accepted long-context benchmark. our work attempts to address these two needs directly. the main contributions of this work are twofold: in particular, we find that distilling longformer-roberta @xcite yields the best results during our experiments, producing substantial improvements in cost performance over state-ofthe-art models. in short, it retains considerable performance on glue (92.3%), squad (93.0%), hotpotqa (88.4%), conll-2003 (99.8%), and gonerd (95.9%) while decreasing inference times by 49.3% on long sequences. in the context of gonerd, this is effectively 95.9% of the original model's performance for 50.7% of the cost. ## related work considerable success has been made in the compression of bert @xcite which, at the time of its release, was one of the largest models in nlp. bert itself has been expanded to fit many different use cases including, but not limited to, roberta @xcite , a model built to improve bert performance on a variety of tasks through clever choices in training data and hyperparameters, xlm-r @xcite , which was built using similar methods on extremely multilingual data (100 languages), and distilbert @xcite , which sought to greatly reduce * https://huggingface.co/giant-oak the computational costs of bert through knowledge distillation. albert @xcite factorizes the embedding matrices of bert and shares weights between layers to significantly decrease the parameter size, thereby decreasing training and inference costs. bert-based distillation methods, such as distil-bert @xcite , tinybert @xcite , and mobilebert @xcite have gained prominence due to their utilization of distillation techniques and can be applied to a wide variety of bert-based architectures. these models have significantly reduced the computational requirements and resource consumption associated with bert-based nlp models, making them more accessible and easily deployable on resourceconstrained hardware. however, bert's attention mechanism still results in a quadratic dependency on sequence length, resulting in greater computational requirements at higher sequence lengths. to solve this problem with bert-based architectures, methods have been developed to create efficient attention transformer models @xcite which can operate on sequences many times longer than their bert counterparts. two popular methods in this area are longformer @xcite and big bird @xcite , which use dilated sliding window and a combination of global, sliding, and random activations in their attention matrices, respectively, to increase the maximum input sequence length from 512 to 4096. more recently, local sparse global (lsg) @xcite attention uses a locality sensitive hashing algorithm @xcite with the local, sparse, and global patterns used in longformer and big bird, whereas nyströmformer @xcite uses a nyström matrix approximation to the regular softmax attention, reducing self-attention complexity to linear time. the long-range arena (lra) @xcite , a comprehensive suite of benchmarking tasks toward systematically evaluating long-context transformer architectures, is novel in that its tasks largely decouple the effect of masked language modeling (mlm) pretraining from efficient model performance. while useful for developing new transformer architectures, we are primarily focused on the comparative performance between student and teacher models on downstream tasks after having been pretrained/distilled on mlm. as such, lra is not utilized in this paper. ## conclusion in this work, we performed an investigation into the convert-then-distill paradigm, the process of (1) converting a teacher model to utilize an efficient attention mechanism, (2) pretraining the converted teacher model, (3) distilling into a smaller student model, then (4) fine-tuning the student on a downstream task. we saw an average decrease in inference times of up to 58%. the efficient attention students preserved up to 98.6% of performance across short-context (glue, squad, conll-2003) tasks, 94.1% of hotpotqa performance, 95.0% of triviaqa performance, and 97.4% percent of gonerd performance when compared to their teacher models. we saw distilled nyströmformer retained the most performance when compared to its teacher, while distilled longformer had the best base performance across most tasks. we introduced gonerd, a long-context ner dataset consisting of large amounts of hand-labeled web text data. finally, we release all models on the hugging face hub for general use. our research demonstrates that, for most models on most tasks, employing knowledge distillation on efficient attention architectures can be a highly effective approach. this technique yields models with a high level of performance on both short and long-context tasks at a fraction of the cost.