Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules (2023)

Abstract: Large language models (LLMs) have achieved remarkable results on NLP tasks but at the expense of huge parameter sizes and the consequent computational costs. In this paper, we propose Variator, a parameter-efficient acceleration method that enhances computational efficiency through plug-and-play compression plugins. Compression plugins are designed to reduce the sequence length via compressing multiple hidden vectors into one and trained with original LLMs frozen. Different from traditional model acceleration methods, which compress LLMs to smaller sizes, Variator offers two distinct advantages: (1) In real-world applications, the plug-and-play nature of our compression plugins enables dynamic selection of different compression plugins with varying acceleration ratios based on the current workload. (2) The compression plugin comprises a few compact neural network layers with minimal parameters, significantly saving storage and memory overhead, particularly in scenarios with a growing number of tasks. We validate the effectiveness of Variator on seven datasets. Experimental results show that Variator can save 53% computational costs using only 0.9% additional parameters with a performance drop of less than 2%. Moreover, when the model scales to billions of parameters, Variator matches the strong performance of uncompressed LLMs. Our code and checkpoints will be released to facilitate future work.

## introduction large pre-trained language models (plms) have made significant advancements in natural language processing tasks @xcite @xcite @xcite . it is widely observed that amplifying the model scale correlates positively with many efforts have been devoted to improving the training and inference efficiency of plms @xcite @xcite @xcite . these methods compress plms into fixed smaller sizes, and cannot fulfill the following requirements: (1) dynamic workload. in real-world scenarios, the system workload varies dynamically over time, while the computational resources are fixed. this implies that we can use more resources for higher performance when the workload is low, and ensure response efficiency when the workload is high. to address these issues, we propose a novel plugand-play acceleration framework named variator. as shown in figure 1 , different from compressing plms into smaller sizes, variator enables plms acceleration via devising compression plugins, which can be inserted into plms to enhance the inference speed. various plugins entail different acceleration ratios, and the system can dynamically choose the appropriate one to trade off response speed and model performance depending on the workload. moreover, variator only necessitates plugins with minimal parameters and freezes the original parameters of plms, which substantially lowers the memory and storage requirements. to achieve plug-and-play acceleration, there are two main challenges: (1) plugin architecture: compression plugins do not modify plms scales, and how to devise plugins to reduce the inference time is a challenge. (2) plugin training: compression plugins only contain limited parameters, and how to effectively train plugins so that they can enhance the model speed while preserving downstream performance. as for plugin architecture, inspired by previous findings about redundancy in hidden vectors @xcite , we design compression plugins for data compression rather than parameter compression. specifically, compression plugins consist of hidden compression layers and hidden decompression layers. the goal of the hidden compression layers is to compress multiple hidden vectors into one, thereby diminishing the sequence length for plms and enabling model acceleration. simultaneously, to preserve token-level information, we also devise decompression layers that recover the processed shorter sequence to the original length. compression plugins can be applied in any layer of plms, enabling various levels of acceleration. as for plugin training, we adopt a two-step training strategy. firstly, we train compression plugins on pre-trained plms with pre-training corpus. then the compression plugins trained in the first step are used as initialization for task-specific models. in both steps, we apply knowledge distillation objectives to train the compression plugins not to alter the hidden vectors produced by plms. to verify the effectiveness of variator, we conduct experiments with a widely-used pre-trained backbone, t5 @xcite , on seven widelyused language understanding benchmarks. the experimental results show that variator can save 53% computational costs using only 0.9% parameters with absolute average performance drops @xmath0. when the model scales to billions of parameters, variator can achieve nearly no performance drop. we also examine the effectiveness of variator on a decoder-only llm, llama @xcite . in addition, we conduct neuron-level analysis for compression plugins, and find that compression plugins can effectively store important information in the compressed vectors to achieve satisfactory performance with limited computational costs. ## methodology in this section, we first describe the paradigm and basic annotations for our plug-and-play model acceleration. then we present the framework and training recipes of variator to accelerate model inference with minimal additional parameters. to showcase the efficiency of variator, we also conduct an analysis of the computational and storage complexity. ## conclusion in this paper, we explore the parameter-efficient acceleration setting and propose variator, which reduces the computational costs with compression plugins. the extensive experiments on seven datasets show that we can reduce 53% computational costs with only 0.9% additional parameters. in the future, we will explore more effective tokenmerging frameworks to improve compression plugins. besides, we will further decouple compression plugins from specific tasks, thus we can construct compression plugins once and for all with transferability across multiple tasks. dataset train validation input template mnli-m 393k 9.8k sentence 1: premise sentence 2: hypothesis does sentence 1 entails sentence 2? <@xmath1.5k question: question sentence: sentence does the sentence contains the answer to the question? <@xmath2.4k question 1: question1 question 2: question2 are the two questions paraphrase of each other? <@xmath3.5k 277 sentence 1: sentence1 sentence 2: sentence2 does sentence 1 entails sentence 2? <@xmath4.3k 872 sentence: sentence does this sentence express positive or negative emotions? <@xmath5.7k 408 sentence 1: sentence1 sentence 2: sentence2 are the two sentences paraphrase of each other? <@xmath6.k 10.6k question: question context: context answer: <@xmath7: the performance with compression plugins in bert. as shown in