Quantized Side Tuning: Fast and Memory-Efficient Tuning of Quantized Large Language Models (2024)

Abstract: Finetuning large language models (LLMs) has been empirically effective on a variety of downstream tasks. Existing approaches to finetuning an LLM either focus on parameter-efficient finetuning, which only updates a small number of trainable parameters, or attempt to reduce the memory footprint during the training phase of the finetuning. Typically, the memory footprint during finetuning stems from three contributors: model weights, optimizer states, and intermediate activations. However, existing works still require considerable memory, and none can simultaneously mitigate the memory footprint of all three sources. In this paper, we present quantized side tuing (QST), which enables memory-efficient and fast finetuning of LLMs by operating through a dual-stage process. First, QST quantizes an LLM’s model weights into 4-bit to reduce the memory footprint of the LLM’s original weights. Second, QST introduces a side network separated from the LLM, which utilizes the hidden states of the LLM to make task-specific predictions. Using a separate side network avoids performing back-propagation through the LLM, thus reducing the memory requirement of the intermediate activations. Finally, QST leverages several low-rank adaptors and gradient-free downsample modules to significantly reduce the trainable parameters, so as to save the memory footprint of the optimizer states. Experiments show that QST can reduce the total memory footprint by up to 2.3× and speed up the finetuning process by up to 3 \times while achieving competent performance compared with the state-of-the-art. When it comes to full finetuning, QST can reduce the total memory footprint up to 7 \times .

## introduction recent advancements in large language models (llms), including gpt @xcite @xcite , palm @xcite , opt @xcite , and llama @xcite , have showcased remarkable taskgeneralization capabilities across diverse applications @xcite . the ongoing evolution of llms' capabilities is accompanied by exponential increases in llms' sizes, with some models encompassing 100 billion parameters @xcite . finetuning pre-trained llms @xcite @xcite for customized downstream tasks provides an effective approach to introducing desired behaviors, mitigating undesired ones, and thus boosting the llms' performance @xcite @xcite . nevertheless, the process of llm finetuning is characterized by its substantial memory demands. for instance, finetuning a 16-bit llama model with 65 billion parameters requires more than 780gb of memory @xcite . to reduce the computational requirement of llm finetuning, recent work introduces parameter-efficient finetuning (peft), which updates a subset of trainable parameters from an llm or introduces a small number of new parameters into the llm while keeping the vast majority of the original llm parameters frozen @xcite @xcite @xcite . peft methods achieve comparable performance as full finetuning while enabling fast adaption to new tasks without suffering from catastrophic forgetting @xcite . however, peft methods necessitate caching intermediate activations during forward processing, since these activations are needed to update trainable parameters during backward propagation. as a result, peft methods require saving more than 70% of activations and almost the same training time compared to full finetuning @xcite . concisely, existing peft techniques cannot effectively reduce the memory footprint of llm finetuning, restricting their applications in numerous real-world memory-constrained scenarios. recent work has also introduced approaches to combining peft and quantization. for example, qlora @xcite quantizes an llm's weights to 4-bit and leverages low-rank adaption (lora) @xcite to finetune the quantized llm. qlora reduces the memory footprint of an llm's weights and optimizer states, and as a result, finetuning a 65b llm requires less than 48 gb of memory. however, qlora does not consider the memory footprint of intermediate 1 figure 1 : figure 1a shows the memory footprint of different methods of fintuning llama-2-70b. figure 1b shows the mmlu 5-shot accuracy of different methods when tuning llama-2-7b, llama-2-13b, and llama-2-70b. note that we set the batch size to 16 and the sequence length to 384. larger markers represent larger models. even with the adoption of a side network, the inherent model size of the llm remains a challenge. meanwhile, these approaches focus on small models (i.e., less than 3 billion parameters), and their applicability and efficacy for larger models remain unexplored. in this paper, we propose a fast, memory-efficient llm finetuning framework, called quantized side-tuning (qst), which operates through a dual-stage process as shown in figure 2 . first, qst quantizes an llm into 4-bit to reduce the memory footprint of its model weights. second, qst introduces a side network separating from the quantized llm to avoid performing backward propagation for the quantized llm, thus saving the memory footprint of intermediate activations. during the training phase of qst, the input to each layer of the side network is formed by combining (1) the downsampled output of the corresponding quantized llm layer and (2) the output of the previous layer of the side network. a larger llm usually has a larger model depth (i.e., the number of layers) and width (the hidden size of each layer), which in turn requires more trainable parameters for the downsampling layers. unlike @xcite that leverages linear layer to perform downsampling, qst uses several low-rank adapter methods @xcite such as maxpooling @xcite and avgpooling, significantly reducing the required trainable parameters and the memory footprint for the optimizer states. after that, we use a learnable parameter to assign weights and subsequently aggregate the hidden states of the quantized llm and the side network. finally, we reuse the llm head or classifier to predict. combined with 4-bit quantization and side tuning, qst significantly reduces all three main contributors of the memory footprint and training time during the training phase. besides, qst does not increase inference latency since the llm and side network can be computed in parallel. figure 1 compares the memory footprint of qst and existing parameter-efficient fine-tuning methods, including qlora and lst. to validate the effectiveness of our qst, we conduct extensive evaluations for different types of llms (e.g., opt, llama 2), with 1.3b to 70b parameters, on various benchmarks. experiment results show that qst can reduce the total memory footprint by up to 2.3 × and speed up the finetuning process by up to 3 × while achieving competent performance compared with the state-of-the-art. our codes are released to the github foot_0 . ## quantized side tuning in this section, we first describe the process of quantizing an llm into 4-bit, and then introduce our design of the side network for side tuning. ## evaluation in this section, we empirically validate the effectiveness of our qst method by examining its performance for llms with different types (e.g., opt and llama 2), sizes (from 1.3b to 70b), and benchmarks. ## conclusion in this paper, we propose quantized side tuing (qst), a novel fast and memory-efficient finetuning framework. qst operates through a dual-stage process: first, qst quantizes the llm into 4-bit to reduce the memory footprint of the weights in llm; then qst introduces a side network separated from the llm, which utilizes the hidden states of the llm to make task-specific predictions. qst can significantly reduce the memory footprint of llm finetuning compared to existing approaches. in particular, experiments show that qst can reduce the total memory footprint by up to 2.3 × and speed up the finetuning process by up to 3 × while achieving comparable performance compared with the state-of-the-art.