Outlier-weighed Layerwise Sampling for LLM Fine-tuning (2025)

Abstract: The rapid advancements in Large Language Models (LLMs) have revolutionized various natural language processing tasks. However, the substantial size of LLMs presents significant challenges in training or fine-tuning. While parameter-efficient approaches such as low-rank adaptation (LoRA) have gained popularity, they often compromise performance compared to full-rank fine-tuning. In this paper, we propose Outlier-weighed Layerwise Sampling (OWS), a new memory-efficient fine-tuning approach, inspired by the layerwise outlier distribution of LLMs. Unlike LoRA, which adds extra adapters to all layers, OWS strategically assigns higher sampling probabilities to layers with more outliers, selectively sampling only a few layers and fine-tuning their pre-trained weights. To further increase the number of fine-tuned layers without a proportional rise in memory costs, we incorporate gradient low-rank projection, further boosting the approach’s performance. Our extensive experiments across various architectures, including LLaMa2 and Mistral, demonstrate that OWS consistently outperforms baseline approaches, including full fine-tuning. Specifically, it achieves up to a 1.1% average accuracy gain on the Commonsense Reasoning benchmark, a 3.0% improvement on MMLU, and a notable 10% boost on MT-Bench, while being more memory efficient. OWS allows us to fine-tune 7B LLMs with only 21GB of memory. Our code is available at https://github.com/pixeli99/OWS.

## introduction the rapid advancements in ai driven by large language models (llms) have fundamentally transformed how people work and communicate. the impressive language capabilities of llms enable a single model to handle various tasks simultaneously, including but not limited to natural language * equal contribution. † corresponding author. to address these challenges, various parameterefficient fine-tuning (peft) approaches have been proposed, including prompt tuning @xcite , adaptors @xcite , and low-rank adaptation (lora) @xcite . these approaches enable the fine-tuning of pretrained llms with substantially fewer trainable parameters, making llm fine-tuning more feasible in practice. among these, lora @xcite stands out for its re-parameterization technique of the pre-trained weight matrix @xmath0, expressed as w 0 + ab, where @xmath1, @xmath2, and r ≪ min(m, n). by fine-tuning only the low-rank adaptor ab while keeping the pretrained weight w 0 frozen, lora significantly reduces the memory usage and computational costs associated with fine-tuning llms, rapidly becoming the preferred method for such tasks. despite its efficiency, recent research has highlighted the inferior performance of low-rank reparameterization compared to full-rank updates in both fine-tuning scenarios @xcite and pre-training contexts @xcite . these findings underscore the need for further exploration into balancing training efficiency with model performance, particularly in the context of large-scale language models. recently, layerwise importance sampled adamw (lisa) @xcite has emerged as a promising alternative for llm fine-tuning, integrating the concept of importance sampling @xcite into the fine-tuning process. unlike methods that add adapters to all layers, lisa selectively samples a small subset of layers and directly fine-tunes their pre-trained weights, demonstrating notable performance improvements over lora. however, our investigation reveals two limitations of lisa: ## background and motivation in this section, we first introduce lisa's algorithm and then present our findings of two key limitations of lisa: the shortcomings of its sampling approach and the significant memory overhead associated with the sampled layers. ## outlier-weighed layerwise sampling (ows) in this section, we introduce our approach, outlierweighed layerwise low-rank projection (ows). we will discuss the underlying rationales, present preliminary results, and detail the algorithm design. the above findings shed light on a principle for designing non-uniform layerwise sampling for llm fine-tuning: layers with higher outlier ratios should be prioritized during the fine-tuning process. this forms the foundation of our proposed method, outlier-weighed layerwise low-rank projection (ows), which we will present in detail. ## experiments in this section, we conduct extensive experiments foot_2 to evaluate the effectiveness of ows on multiple fine-tuning tasks. details are provided below. ## related work parameter-effieient fine-tuning (peft). peft is proposed to reduce the prohibitive cost of llm fine-tuning. various techniques have been proposed in this dynamic field. for instance, prompt tuning only optimizes input tokens or embeddings while keeping the rest of the model frozen, as demonstrated in studies @xcite @xcite . layer-freezing techniques @xcite @xcite enhance training and fine-tuning efficiency by freezing parts of the layers. adapter methods @xcite @xcite , incorporate a small auxiliary module within the model's architecture, which becomes the exclusive focus of updates during training, thus minimizing the number of trainable parameters and optimizer states. among these techniques, low-rank adaptation (lora) @xcite gains massive attention by applying low-rank matrices to approximate weight changes during fine-tuning, which can be merged into the pre-trained weights, leading to no inference overhead. lora has been enhanced through various modifications @xcite @xcite @xcite @xcite aimed at improving performance and efficiency. recently, low-rank has also been explored to pre-train llm from scratch @xcite . @xcite projects the gradient into a low-rank subspace for the update to enable full-parameter learning while significantly reduc-ing memory usage during optimization. badam @xcite partitions the entire model into distinct blocks and utilizes a block coordinate descent framework to update each block individually, either in a deterministic or random sequence. layerwise sampling for llm fine-tuning. importance sampling is a powerful statistical technique used in machine learning to estimate properties of a particular distribution by sampling from a different, more convenient distribution. recently, @xcite explored the idea of importance sampling to llm fine-tuning, with the key idea of sampling only γ layers at each step to finetuning while keeping the rest of layers frozen. the proposed method, layerwise importance sampled adamw (lisa), outperforms lora by a large margin on various benchmarks and even outperforms full parameters training under certain settings. inspired by lisa, our paper advances the performance of layerwise sampling for llm fine-tuning, by addressing a couple of shortfalls of lisa. ## conclusion in this paper, we study the sampling-based llm fine-tuning, where at each iteration, only a few layers are sampled and fine-tuned, instead of the whole model. specifically, we delve into recently-proposed lisa @xcite and unveil two shortcomings that constrain its memoryperformance trade-off: (1) the middle layers of lisa are sampled uniformly, which can result in suboptimal performance. (2) the sampled layers of lisa are fine-tuned in a full-rank manner, causing a significant memory increase as the number of sampled layers increases. to address these challenges, we introduced ows, which assigns higher sampling probabilities to outlier-rich layers and incorporates low-rank gradient projection for improved memory efficiency. our experiments on llama2 and mistral demonstrate that ows significantly boosts performance while reducing memory usage compared to full-rank fine-tuning.