Operator Selection and Ordering in a Pipeline Approach to Efficiency Optimizations for Transformers (2023)

Abstract: There exists a wide variety of efficiency methods for natural language processing (NLP) tasks, such as pruning, distillation, dynamic inference, quantization, etc. From a different perspective, we can consider an efficiency method as an operator applied on a model. Naturally, we may construct a pipeline of operators, i.e., to apply multiple efficiency methods on the model sequentially. In this paper, we study the plausibility of this idea, and more importantly, the commutativity and cumulativeness of efficiency operators. We make two interesting observations from our experiments: (1) The operators are commutative—the order of efficiency methods within the pipeline has little impact on the final results; (2) The operators are also cumulative—the final results of combining several efficiency methods can be estimated by combining the results of individual methods. These observations deepen our understanding of efficiency operators and provide useful guidelines for building them in real-world applications.

## introduction natural language processing (nlp) tasks nowadays heavily rely on complex neural models, especially large-scale pre-trained language models based on the transformer architecture @xcite , such as bert @xcite , roberta @xcite , and gpt @xcite . despite being more accurate than previous models, transformer-based models are typically slow to execute, making it a non-trivial challenge to apply them in real-world applications. for example, it takes a bert-base model about 200 ms per query to perform a simple sequence classification task on a commercial cpu, which can be too slow in many scenarios. therefore, model efficiency has become an increasingly important research direction in the transformer era. a wide variety of efficiency methods have been individually studied for transformers, like pruning @xcite , distillation @xcite , dynamic inference @xcite , and quantization @xcite , just to name a few. there has also been work on applying multiple efficiency methods together as a pipeline @xcite @xcite , but the construction of such pipelines has not been methodically studied. for a desired accuracy-efficiency tradeoff, it remains unclear how to choose components for the pipeline among numerous possibilities. furthermore, even with a chosen set of efficiency methods, it is unclear whether we need to exhaustively examine all possible orders to find the best one. in this paper, we study how to effectively construct a pipeline of efficiency methods, and we do this by exploring their properties. conceptually, we consider each efficiency method as an operator applied on a model. we conduct experiments with the roberta model @xcite and the following components in our efficiency pipelines: distillation, structured pruning, quantization, early exiting, and dynamic length inference. we empirically study two important properties of efficiency operators: (1) commutativity: does swapping the order of operators affect the final accuracy-efficiency tradeoff of the model? (2) cumulativeness: how do the two core metrics of efficiency methods, time savings and accuracy drops, compound across multiple operators? under the condition of our experiments, we show that, for commutativity, the difference between various orderings of the same set of components is usually small and negligible in practice. for cumulativeness, we show that time saving and accuracy drop are both cumulative to the extent that we can estimate the performance of a new pipeline by combining the results of individual components. the observation of these properties provides the foundation for us to build new pipelines and estimate their performance without having to carry out time-consuming experiments. our main contributions in this paper include: (1) in section 3, we propose a conceptual framework to treat efficiency methods as operators and efficiency optimization processes as pipelines; (2) in sections 4 to 5, we demonstrate the properties of operators by experiments. finally we will conclude the paper and discuss its limitations. ## related work and background in this section, we first introduce related work, background, and modeling choices for individual efficiency methods chosen for our experiments. we then discuss related work for applying multiple efficiency methods. applying transformers for nlp tasks typically involves three stages: pre-training, fine-tuning, and inference @xcite . in this paper, we assume the availability of a pretrained roberta model and study different ways of fine-tuning it to achieve better tradeoffs between inference accuracy and efficiency. training henceforth refers to fine-tuning in this paper. ## experimental design in this section, we introduce the detailed design and setups for our experiments. since the experiments are exploratory rather than sota-chasing, we focus on providing a fair comparison. ## operator commutativity and order given a set of operators, we naturally wonder about the best order to apply them. although this question seems formidable due to the exponentially large number of possible orderings, we show that the question is actually simpler than expected: on the one hand, we have eliminated a number of invalid orderings as described in section 3; on the other, we show that operators are commutative in the remaining ordering candidates. ## operator cumulativeness and predictability of pipelines in order to choose components for an efficiency pipeline, an important question is whether time savings and accuracy drops of individual operators are cumulative. in this subsection, we show that they are indeed cumulative to the degree that accuracy-efficiency tradeoffs of a new pipeline can be estimated, simply by combining the results of individual operators. we first discuss operators from group i. in figure 4 , we show how we can estimate the tradeoff curve of a new pipeline based on the results of its constituents, using the two larger and more stable datasets, qqp and qnli. for example, in the top-right subfigure, we show the estimation for the tradeoff curves of pipelines comprising e, d, and p, based on the results of individually applying each of these operators. the idea for estimating accuracy drops is based on the following cumulativeness assumption. suppose r is a pipeline and a * is the accuracy for a pipeline *, the assumption is: in other words, our assumption is that adding d or p to any pipeline should result in similar relative accuracy drops. we can therefore estimate the accuracy of ed, ep, and edp (and other orders of the same set of operators) as follows: (1) calculate accuracy drops of d and p relative to o; empirically, the observation that operators are cumulative facilitates future experiments on efficiency pipelines: for pipelines that are computationally expensive to train and evaluate, simply measuring the performance of their components can provide us with a reliable estimation of the pipeline's behavior. therefore, choosing efficiency methods for a pipeline according to desired accuracyefficiency tradeoffs becomes easy calculation once the measurement of individual operators is finished. on the theoretical side, the cumulativeness observation also makes it easier to analyze the contribution of each component, i.e., how much time each operator saves and how much accuracy each sacrifices. the shapley value @xcite of each component, for instance, can be approximated by simply using the standalone estimation @xcite . ## conclusion in this paper, we propose a conceptual framework to consider efficiency methods as operators applied on transformer models and study the properties of these operators when used as pipelines. we ob-serve that, under the condition of our experiments, (1) efficiency operators are commutative: changing their order has little practical impact on the final efficiency-accuracy tradeoff; (2) efficiency operators are cumulative: a new pipeline's performance can be estimated by aggregating time savings and accuracy drops of each component. these observations facilitate future construction, evaluation, and application of efficiency pipelines, and also provide an interesting direction to better understand efficiency pipelines.