R o L o RA : Fine-tuning Rotated Outlier-free LLM s for Effective Weight-Activation Quantization (2024)

Abstract: Low-Rank Adaptation (LoRA), as a representative Parameter-Efficient Fine-Tuning (PEFT) method, significantly enhances the training efficiency by updating only a small portion of the weights in Large Language Models (LLMs). Recently, weight-only quantization techniques have also been applied to LoRA methods to reduce the memory footprint of fine-tuning. However, applying weight-activation quantization to the LoRA pipeline is under-explored, and we observe substantial performance degradation primarily due to the presence of activation outliers. In this work, we propose RoLoRA, the first LoRA-based scheme to apply rotation for outlier elimination, and then fine-tune rotated outlier-free LLMs for effective weight-activation quantization. Different from previous work tackling the outlier challenges from a post-training perspective, we propose rotation-aware fine-tuning to eliminate and preserve the outlier-free characteristics brought by rotation operations. RoLoRA can improve low-bit LoRA convergence and post-training quantization robustness in weight-activation settings. RoLoRA is evaluated across various LLM series (LLaMA2, LLaMA3, LLaVA-1.5), tasks, and quantization settings, achieving up to 29.5% absolute accuracy gain of 4-bit weight-activation quantized LLaMA2-13B on commonsense reasoning tasks compared to LoRA baseline. We further demonstrate its effectiveness on Large Multimodal Models (LMMs) and prove the compatibility with advanced LoRA variants.

## introduction while we have witnessed the success of large language models (llms) such as gpt-4 @xcite and llama @xcite across various tasks in recent years, the massive model size and expanding training cost for llms have necessitated the design of model compression and parameter-efficient fine-tuning (peft) methods. low-rank adaption (lora) @xcite , * all the work was done within hkust and zechun liu served an advisory role. recently, quantization techniques, which convert high-precision parameters into lower-bit formats such as int4, have been integrated with lora methods @xcite @xcite . existing quantization-lora schemes can save memory costs during fine-tuning, and some schemes @xcite can also reduce inference costs by producing quantized llms directly. however, these methods only perform weight-only quantization, while lora weight-activation quantization is under-explored. quantizing both weights and activations in low-bit further saves run-time gpu memory and accelerates compute-intensive matrix-multiplication operations. we observe that 4-bit or 6-bit weight-activation quantization with lora finetuning still incurs a high accuracy degradation in llms, attributing to the outliers in weight and activation distribution, which stretch the quantization range and increase the quantization error. existing methods in the post-training quantization research community have endeavored to tackle the outlier challenge by mixed-precision subgrouping @xcite or shifting outliers from activation to weight @xcite . more recently, applying rotation @xcite to the weight matrices of llms has demonstrated effectiveness in eliminating activation outliers and keeping computational invariance @xcite . however, all these methods solve the problems from a post-training perspective, ignoring that outliers will emerge and change distribution during pre-training and fine-tuning @xcite . in this work, we take a step further to utilize the rotation for outliers-removal in lora finetuning setting and investigate the optimal solution for dynamically integrating rotation with lora to preserve the outlier-free characteristics and im-prove weight-activation quantization. motivated by this target, we propose rotated outlier-free low-rank adaptation (rolora), which initially apply in-block and between-block rotation to the pretrained llms, and then utilize rotation-aware finetuning to produce outlier-free fine-tuned llms as shown in figure 1 . we explore the optimal rotationaware fine-tuning scheme based on approximation error analysis. extensive experimental results prove the effectiveness of rolora across diverse llms, tasks, and quantization settings. rolora improves the 4-bit quantization for weights and activations (w4a4) performance up to 14.6 points on the mmlu benchmark compared to lora. compared with existing low-bit lora methods, rolora outperforms previous sota ir- @xcite with up to 6.0 points on the mmlu benchmark. the proposed rolora is highly efficient with negligible fine-tuning overhead compared to lora in the same setting. rolora can also improve the quantization robustness of large multimodal models (lmms) such as llava @xcite , and we observe the multimodal understanding is largely retained even after w4a4 quantization as shown in in summary, our work contributes as follows: ## method motivated by existing challenges of activation outliers and the success of rotation-based solutions @xcite , we introduce rotated outlier-free low-rank adaptation (rolora). rolora initially apply inblock and between-block rotation to the pre-trained llms, and rotation-aware fine-tuning on the rotated llms will retain the optimal outlier-free characteristic, producing fine-tuned llms highly robust to weight-activation quantization. ## conclusion this paper presents rolora, the first work to explore the feasibility of weight-activation quantization in lora. rolora applies rotation for eliminating outliers in activation distribution and performs rotation-aware fine-tuning to preserve the outlier-free characteristics. we theoretically and empirically investigate how to integrate rotation into lora. rolora improves the performance of w4a4 and w6a6 llms by a great margin across various tasks with the same training cost. moreover, rolora can also help visual instruction tuning.