A pi Q : Finetuning of 2-Bit Quantized Large Language Model (2024)

Abstract: Memory-efficient finetuning of large language models (LLMs) has recently attracted huge attention with the increasing size of LLMs, primarily due to the constraints posed by GPU memory limitations and the effectiveness of these methods compared to full finetuning. Despite the advancements, current strategies for memory-efficient finetuning, such as QLoRA, exhibit inconsistent performance across diverse bit-width quantizations and multifaceted tasks. This inconsistency largely stems from the detrimental impact of the quantization process on preserved knowledge, leading to catastrophic forgetting and undermining the utilization of pretrained models for finetuning purposes. In this work, we introduce a novel quantization framework named ApiQ, designed to restore the lost information from quantization by concurrently initializing the LoRA components and quantizing the weights of LLMs. This approach ensures the maintenance of the original LLM’s activation precision while mitigating the error propagation from shallower into deeper layers. Through comprehensive evaluations conducted on a spectrum of language tasks with various LLMs, ApiQ demonstrably minimizes activation error during quantization. Consequently, it consistently achieves superior finetuning results across various bit-widths. Notably, one can even finetune a 2-bit Llama-2-70b with ApiQ on a single NVIDIA A100-80GB GPU without any memory-saving techniques, and achieve promising results.

## introduction large language models (llms) have garnered significant acclaim and success across a wide range of domains and applications @xcite @xcite . with ongoing advancements, the scope and complexity of released llms have witnessed exponential growth, with some llms encompassing more than 50b parameters @xcite @xcite . this remarkable upscaling introduces considerable challenges, particularly when effectively adapting these models for downstream tasks. historically, a prevalent method for adapting pretrained models to specific downstream tasks is full finetuning, a process that updates all pretrained parameters. although this approach has led to many state-of-the-art achievements, its practicality is somewhat hindered in scenarios where gpu memory is limited. this limitation stems from the necessity to store the model's weights and optimizer's states in the gpu's memory and intensifies with the escalating sizes of llms. to mitigate the extensive memory requirement for finetuning llms, an alternative method is parameter-efficient finetuning (peft) @xcite @xcite . this technique involves selectively introducing and updating a limited set of parameters while leaving the majority of the llm's parameters unchanged. a key advantage of this approach is the substantial reduction in gpu memory required for the optimizer's states since the size of the optimizer states is proportional to the amount of trainable parameters. to further reduce the gpu memory required by loading the llm's weights, multiple model compression methods have been proposed @xcite @xcite @xcite , converting high-precision weight values into a discrete set of values. initially, quantization techniques were primarily developed for deploying llms in memory-limited environments for inference purposes. qlora @xcite innovatively combines peft, specifically lora @xcite , with quantization methods to remarkably reduce the gpu memory requirement for finetuning. however, qlora @xcite in-herits the same challenge as llm's quantization, namely the distortion of the learned knowledge from the full-precision llm due to the quantization error, which exacerbates at lower-bit quantizations, leading to catastrophic forgetting. recently, @xcite and @xcite proposed a new method to reduce the quantization error through a strategic initialization of the lora components to maintain the original weight states. this technique has demonstrated considerable success in the finetuning of lower-bit quantized llms (qllms). nonetheless, they focus on preserving the weight states on a per-linear-layer basis, resulting in accumulative error propagation through layers. in this paper, we introduce a novel and efficient quantization framework, termed apiq, which consists of two steps to adapt an llm, similar to qlora. during the quantization step, apiq preserves the activation instead of the weight of fullprecision llm by jointly optimizing the lora's components and quantizing the llm's weights. this approach ensures that the output of the qllm remains consistent with that of the full-precision llm, effectively mitigating quantization error by aligning the activations across corresponding layers. as a result, the quantization errors introduced in earlier layers are ameliorated. subsequently, we finetune the lora modules with the fixed qllm on downstream tasks, thereby significantly reducing the demands on gpu memory. our primary contributions are as follows: memory-efficient finetuning. in response to the gpu memory constraints and the increasing size of llms, various strategies have been developed to optimize memory efficiency during finetuning. to mitigate activation memory demands, techniques such as selective activation storage and ondemand recomputation are employed @xcite @xcite . ad-ditionally, to curtail the memory required for optimizer state storage, the pretrained llm is kept fixed while a limited amount of trainable parameters are introduced @xcite . a prime example is lora @xcite , which adapts the pretrained weight, @xmath0, of a linear layer as w ′ = w + α r ab ⊤ , where and α is a scalar. introducing a smaller bottleneck dimension r substantially reduces the memory demand for the optimizer, illustrated by a reduction from 26.4gb to 5.3gb as shown in figure 1 . to further diminish lora's memory usage, @xcite introduced a quantized version of w , such as a 4-bit representation in contrast to 16 bits. this technique significantly decreases the memory requirement for storing the model's weights, from 12.6gb to 4.6gb. quantization involves converting highprecision values into discrete levels. in this research, we focus on uniform affine quantization @xcite , known for its enhanced hardware support and efficiency. this process quantizes the pretrained weight as follows: ## challenges of finetuning qllm qlora @xcite employs a strategy wherein the fixed pretrained weights are loaded onto the gpu in a lower-bit format, while finetuning is confined to a minimal number of parameters from the adapters. this approach significantly reduces the memory allocation required from both the model's weights and optimizer states, decreasing it from 39gb to 10gb, as depicted in figure 1 . this reduction in memory demand facilitates the finetuning of llms on more modest computational resources. nevertheless, this method introduces certain challenges associated with quantization. ## method: apiq in this section, we introduce a novel quantization framework, activation-preserved initialization of qllm termed as apiq, that addresses all abovementioned challenges when finetuning a qllm. the core objective of apiq is to maintain the integrity of the starting point, while effectively minimizing the cumulative impact of the quantization errors as they traverse through the network. ## experiments in this section, we evaluate apiq on the language understanding, language modeling, arithmetic reasoning and commonsense reasoning tasks by quantizing deberta-v3 @xcite , roberta @xcite , llama-2 @xcite and mistral @xcite . like qlora, loftq and lq-lora, apiq consists of two steps: the quantization step and the finetuning step. during the quantization step, we initialize q, a and b in a way to preserve the starting point and mitigate the propagation of quantization error. for the finetuning step, we freeze q in a lower bit and train a and b in half-precision (bfloat16). implementation details. in algorithm 1, the quantization process of apiq requires a calibration dataset. we randomly sample 128 sentences from the training set of @xcite . following our baselines @xcite , lora modules are integrated into all linear layers. by default, the group/block size for quantization is 64 for all methods. we employ adamw @xcite as an optimizer to update a, b and θ. more implementation details for the quantization and finetuning steps are detailed in appendix §c for reproduction. baselines include full finetuning (full ft), lora @xcite , qlora @xcite , gptq-lora @xcite , loftq @xcite , and lq-lora @xcite . full ft and lora are considered the upper bound for finetuning. qlora and gptq-lora employ nf-quantization and uniform quantization, respectively, on the pretrained weights with the default lora initialization. these methods are memoryefficient but distort the starting point. in contrast, loftq and lq-lora initialize the matrices q, a, and b to preserve the initial weight state, thus serving as a strong baseline to apiq. ## conclusion in this work, we propose apiq, a novel framework that aims to reduce the activation error during quantization by jointly quantizing the llm's weights and initializing the lora's components. extensive experiments, on five tasks across various encoder-only and decoder-only models, demonstrate apiq's effectiveness in adapting qllm. it works extremely well with lower-bit quantization and larger models than the strong baselines. further experiments also demonstrate apiq's capability as a pure ptq method (appendix §b.1).