Towards Efficient NLP : A Standard Evaluation and A Strong Baseline (2022)

Abstract: Supersized pre-trained language models have pushed the accuracy of various natural language processing (NLP) tasks to a new state-of-the-art (SOTA). Rather than pursuing the reachless SOTA accuracy, more and more researchers start paying attention to model efficiency and usability. Different from accuracy, the metric for efficiency varies across different studies, making them hard to be fairly compared. To that end, this work presents ELUE (Efficient Language Understanding Evaluation), a standard evaluation, and a public leaderboard for efficient NLP models. ELUE is dedicated to depicting the Pareto Frontier for various language understanding tasks, such that it can tell whether and how much a method achieves Pareto improvement. Along with the benchmark, we also release a strong baseline, ElasticBERT, which allows BERT to exit at any layer in both static and dynamic ways. We demonstrate the ElasticBERT, despite its simplicity, outperforms or performs on par with SOTA compressed and early exiting models. With ElasticBERT, the proposed ELUE has a strong Pareto Frontier and makes a better evaluation for efficient NLP models.

## introduction driven by the large-scale pre-training, today's nlp models have become much more powerful @xcite @xcite @xcite @xcite . as a consequence of this drastic increase in performance, these pre-trained language models (plms) are notorious for becoming more and more computationally expensive due to the increasing number of parameters. therefore, rather than pre-training a larger model to achieve a new state-of-the-art (sota) accuracy, most studies are pursuing improvement on other dimensions such as the number of parameters or flops @xcite @xcite @xcite . for these works, the goal has shifted from simple sota to "pareto sota". a pareto sota model means that there is no other model is currently better than it on all the dimensions of interest. for example, a model may claim to be pareto sota as long as it achieves the best accuracy under the same number of parameters or flops. for these efficient models with fewer parameters or flops, it is unfair to get them evaluated on the accuracy-centric benchmarks such as glue @xcite , and ranked among many large-scale models. the shifted goal has outpaced the existing benchmarks, which cannot provide a comprehensive and intuitive comparison for efficient methods. in the absence of a proper benchmark, measures of efficiency in different studies cannot be standardized, and different methods cannot be fairly compared. as a result, it is difficult to say whether and how much a method achieves pareto improvement. to that end, we aim to build the pareto frontier for various tasks with standard evaluation for both performance and efficiency. our motivation can be briefly illustrated by figure 1 . need for a standard evaluation as the goal has shifted, a new benchmark is urgently needed to comprehensively compare the nlp models in multiple dimensions. currently, this multi-dimensional comparison is done in the individual papers, resulting in the following issues: (a) incomprehensive comparison. the comparison is usually point-topoint, e.g. comparing model performance under the same flops. the comparison in a broader range is usually missed, especially for works in conditional computation where the model performance varies with flops. (b) unaccessible results. even if the comprehensive line-to-line comparison is conducted, the results are usually presented in form of figure, in which the data points are not accessible for the following work. as a result, the following work has to reproduce or estimate the results (e.g. @xcite estimate values from the figures of @xcite ). (c) non-standard measurements. different works may adopt different metrics such as physical elapsed time, flops, and executed model layers, making them hard to directly compare. even if the adopted metrics are the same, there is no guarantee that they will be calculated in the same way (e.g. the hardware infrastructure, or the software to calculate flops can be very different foot_0 ). (d) inconvenience. recent studies usually choose glue @xcite as the main benchmark, which, however, is not suitable for dynamic methods due to its submission limitation that is designed to avoid overfitting on test sets. need for a strong baseline currently, there are roughly two branches of efficient methods in nlp: static methods (e.g. distillation, pruning, quantization, etc.) and dynamic methods (e.g. early exiting). (a) static models are obtained given an expected number of parameters or inference latency. these methods often use the first few layers (to keep the same number of parameters or flops) of some pre-trained model followed by a classification head as their baseline, which, however, is too weak to serve as a baseline. (b) dynamic models usually add multiple internal classifiers to the pre-trained lms, and therefore allow flexible inference conditioned on the input. nevertheless, the injected internal classifiers introduce a gap between pre-training and fine-tuning. training the internal classifiers on downstream tasks often degenerates the perfor-mance of the entire model @xcite . thus, static models need a strong baseline, and dynamic models need a strong backbone. contributions in this work, we address the above needs by contributing the following: another line of work is multi-dimensional evaluations. efficientqa @xcite ) is an opendomain question answering challenge that evaluates both accuracy and system size. the system size is measured as the number of bytes required to store a docker image that contains the submitted system. dynabench @xcite , an open-source benchmark for dynamic dataset creation and model evaluation, also supports multidimensional evaluation. in particular, dynabench measures model performance, throughput, memory use, fairness, and robustness. both efficientqa and dynabench require the user to upload the model along with the required environment to the server, which is costly for users to upload and also for the server to evaluate. in contrast, elue adopts a cheaper way to evaluate performance and efficiency of the model. recently, long-range arena (lra) @xcite is proposed to evaluate models under the long-context scenario. different from elue, lra mainly focuses on xformers @xcite . besides, some tasks included in lra are not nlp tasks, or even not real-world tasks, while elue consists of common language understanding tasks. in addition, elue is also inspired by other well-known benchmarks, such as squad @xcite , multinli @xcite , decanlp @xcite , clue @xcite , hot-potqa @xcite , gem @xcite , etc. efficient nlp models current efficient nlp models can be roughly categorized as two streams: model compression (static methods) and conditional computation (dynamic methods). model compression is to reduce the number or precision of model parameters to achieve faster training and inference. currently, there are several ways to achieve model compression: (1) knowledge distillation, which is to learn a compact student model that learns from the output distribution of a largescale teacher model @xcite (2) model pruning, which is to remove parts of parameters that are less important @xcite , (3) weight sharing across different parts of the model @xcite is also a common technique to significantly reduce parameters, (4) quantization, which is to use low bit precision to store parameter and accelerate inference with low bit hardware operations @xcite , and (5) module replacing, which is to replace the modules of a big model with more compact substitutes @xcite . in contrast, conditional computation is to selectively execute only parts of the model conditioned on a given input @xcite . as a representative, an end-to-end halting approach, adaptive computation time (act) @xcite , is developed to perform input-adaptive computation for recurrent networks. the idea of act is later adopted in universal transformer @xcite . recently, as the rising of deep models for natural language processing, early exiting is widely used to speedup inference of transformer models @xcite @xcite @xcite @xcite @xcite . ## elue: a standard benchmark for efficient nlp models elue aims to offer a standard evaluation for various efficient nlp models, such that they can be fairly and comprehensively compared. in section 3.1, we list the design considerations to achieve this motivation. in section 3.2, we describe the tasks and datasets included in elue. in section a.1, we illustrate how to make a submission on elue, and how the submission is evaluated. in section 3.3, we discuss the design of our leaderboard. ## elasticbert: a strong baseline for efficient inference despite the encouraging results achieved by existing efficient models, we argue that a strong baseline (backbone) is needed for both static methods and dynamic methods. static methods often choose the first few layers of some pre-trained models as their baseline (e.g. sun et al. (2019); jiao et al. (2020) ), which can be weak. dynamic methods that enable early exiting by training multiple internal classifiers usually introduce a gap between pre-training and fine-tuning, and therefore hurt the performance of the entire model @xcite . thus, as illustrated in figure 3 , we present the elasticbert that bridges the gap between static and dynamic methods, and therefore can serve as a strong baseline for static methods and also a strong backbone for dynamic methods. elasticbert is a multi-exit pre-trained language model with the following training objective: ## conclusion and future work in this work, we present elue, which is a public benchmark and platform for efficient models, and elasticbert, which is a strong baseline (backbone) for efficient static (dynamic) models. both of the two main contributions are aimed to build the pareto frontier for nlu tasks, such that the position of existing work can be clearly recognized, and future work can be easily and fairly measured. our future work is mainly in four aspects: (1) including more baselines in elue, (2) supporting the evaluation for more frameworks such as tensor-flow @xcite , (3) supporting diagnostics for submissions, (4) supporting the evaluation of more different types of tasks. performance metrics since the classes in mrpc are imbalanced, we report the unweighted average of accuracy and f1 score. for sts-b, we evaluate and report the pearson and spearman correlation coefficients. for other datasets, we simply adopt accuracy as the metric.