Sparsity May Be All You Need: Sparse Random Parameter Adaptation (2025)

Abstract: Full fine-tuning of large language models for alignment and task adaptation has become prohibitively expensive as models have grown in size. Parameter-Efficient Fine-Tuning (PEFT) methods aim at significantly reducing the computational and memory resources needed for fine-tuning these models by only training on a small number of parameters instead of all model parameters. Currently, the most popular PEFT method is the Low-Rank Adaptation (LoRA), which freezes the parameters of the model and introduces a small set of trainable parameters in the form of low-rank matrices. We propose simply reducing the number of trainable parameters by randomly selecting a small proportion of the model parameters to train on, while fixing all other parameters, without any additional prior assumptions such as low-rank structures. In this paper, we compare the efficiency and performance of our proposed approach to other PEFT methods as well as full parameter fine-tuning. We find our method to be competitive with LoRA when using a similar number of trainable parameters. Our findings suggest that what truly matters for a PEFT technique to perform well is not necessarily the specific adapter structure, but rather the number of trainable parameters being used.

## introduction it has become common practice to train applicationready language models in two phases @xcite : first, the model is pre-trained on a very large and general corpus of (unlabeled) text; then further trained (or fine-tuned) on a smaller specific set of examples demonstrating the intended behavior for a particular application, such as instruction following @xcite . overall, supervised fine-tuning (sft) requires less computational resources than pre-training (pt) due to the significantly smaller size of the training set as well as the typical use of early stopping to deal with overfitting. this means orders-ofmagnitude less gradient computations and parameter updates are needed during sft compared to pt. however, a major drawback is that memory requirements remain the same, unless a parameter-efficient fine-tuning (peft) technique is used. the main memory bottleneck during training is the number of trainable parameters, since additional memory must be allocated for their gradients and other perparameter statistics needed by the optimizer. the idea behind peft @xcite is to significantly reduce the number of trainable parameters during fine-tuning while maintaining performance. low rank adaptation (lora), first introduced by @xcite , currently remains the most popular peft technique. lora freezes all the pretrained model parameters θ pt and introduces trainable low-rank matrices (e.g., b, a) to represent the changes (∆ = ba) needed for adapting the model to a new task. the adapted model parameters are given by θ pt + ∆. memory and computational efficiency are achieved by optimizing only over the parameters of these newly added, but significantly smaller, matrices. the success of lora begs one to ask what properties make this method perform well. is the low-rank structure critical, i.e., does ∆ need to be low-rank? is it sufficient to constrain ∆ to be low dimensional? a main goal of this paper is to investigate these questions. an abundance of research is going into new methods for structured ∆ (see section 3 below) and novel directions into unstructured methods for fine-tuning could open avenues in areas such as model merging @xcite or pluralistic alignment @xcite . in this work, we propose a different approach where ∆ is not factorized into low rank matrices but rather chosen to be a random subset of the model parameters. this sparse random parameter adaptation (sparta) method imposes a sparsity constraint on the adapter that can be easily controlled. by changing the desired sparsity, one can change the number of adaptation parameters. regardless of how the selected parameters are sampled from the model parameters, subsequent updates only affect these parameters. this sparsity constraint and randomness of selected parameters is in contrast to techniques such as lora that effectively affect all parameters in θ pt . see figure 1 for an illustration of the method. generally, one samples m parameters from the pre-trained model θ pt , stores their indices in ϕ, and uses adapter ∆ ϕ to fine-tune θ pt . to investigate the performance of sparta, we build adapters with different sparsity levels and evaluate them on a wide range of natural language understanding benchmarks. sparta is compared to other peft approaches including lora and dora @xcite , and found to be quite competitive compared to these methods given that it only modifies a small sparse number of model parameters. this motivates our approach, sparta, which produces a performant fine-tuned model by directly adapting only a small percentage of the pre-trained model parameters. sparta randomly selects the (scalar) parameters to train and freezes the remain-ing parameters (i.e., setting the corresponding ∆ values to zero). this ∆-sparsity also helps in reducing overfitting, as pre-trained models typically have more than enough capacity to learn the often limited amount of (labeled) data used for fine-tuning. there is no guarantee of ∆-sparsity in lora, but this is a desired property since it reduces parameter interference @xcite when merging fine-tuned models. sparta, like lora, reduces the number of gradient computations during training (when compared with full fine-tuning), and ultimately has the same inference cost as the original model (after merging back the sparse ∆ into the model). for an adaptation technique, having lower memory and computation needs during training as well as identical inference time to the original model are all quite desirable properties. sparta has all these properties plus the unique added benefit of producing only sparse changes in a small number of the parameters of the original model that can be beneficial for merging multiple sparta adapters. ## related work the last few years has seen many advances in peft methods. perhaps the most well-known and used method in practice is @xcite , which has spurred many variants including: dora, which adapts only the directions of pre-trained weights @xcite ; vera, which shares low-rank matrices across layers @xcite ; adalora, which adaptively allocates parameter budgets among weight matrices according to their importance @xcite ; and sora, which dynamically adjusts the intrinsic rank dur-ing adaptation @xcite . each of these methods has a different structure to the ∆ being optimized, with the commonality being the training of some function of low-rank matrices. adalora and sora are also dynamic methods, which adjust the function during adaptation. beyond adding structured parameters is the concept of fine-tuning a small subset of the total parameters of the model, i.e., sparse fine-tuning, where one must first decide which subset of parameters to fine-tune (similar to deciding which parameters in each layer to add adapters to in lora). @xcite start with an initial set of parameters and offer procedures to drop (according to parameter magnitude) and grow (according to gradient) the set, i.e., they learn the set of parameters to train. alternatively, @xcite focus on the sparsity of neuron activations during training by precomputing neuron importance scores and only including important neurons in computations during training. @xcite first fine-tune on all parameters, select the parameters that change the most, and then fine-tune again from scratch on the selected parameters. @xcite proposes improvements via a modified pruning strategy to the random dropping and rescaling method of @xcite which motivates this work. in contrast to these more complex, sometimes dynamic approaches, our proposed sparta method produces a performant fine-tuned model by directly adapting only a small percentage of the pre-trained model parameters chosen completely at random. yet another related direction is that of compression which results in sparse models; algorithms in this genre take a dense fine-tuned model with the goal of compressing it while maintaining performance. compression (see @xcite for a survey) could be accomplished by quantization, low-rank approximation, pruning (i.e., removing neurons, attention heads, or even layers), or distillation. the focus of this paper, however, is on finetuning dense models rather than learning sparse models as in @xcite . ## sparta: adapting a random subset of model parameters suppose the parameters of a pre-trained language model are θ @xmath0, and full parameter fine-tuning (ft) is performed with a labeled dataset characterizing a task. ft typically updates θ pt using a stochastic first-order gradient-based optimization algorithm, e.g., adam from @xcite , to maximize the conditional probability of the labels in the training dataset under the model. the ft model is then given by θ @xmath1. this has two drawbacks in terms of memory efficiency. first, the optimizer state can be large, e.g., the state of the adam optimizer is 4 times as large as the parameter space as it includes current parameter values, their gradients as well as per parameter statistics of those gradients. second, storing a new ft model takes as much memory as the pt model which could be an issue when many task-specific models are requested. sparta proposes to randomly select a small subset of the model parameters to optimize while freezing the rest. the model parameters θ ∈ r n are partitioned into trainable θ @xmath2, with m ≪ n being the number of selected parameters. this allows our approach to have a drastically lower memory footprint than ft by reducing the size of the optimizer state as well as faster training by reducing the number of gradients to compute. this is similar with respect to lora in that both reduce the memory footprint by optimizing a small set of parameters (the adapter) while freezing most (sparta) or all (lora) of the model parameters. in our sparta implementation foot_0 , we introduce: (i) non-trainable indices ϕ ∈ r m containing the indices of the randomly selected elements in θ pt , and (ii) trainable parameters ∆ ϕ ∈ r m representing the subset of ∆ that sparta learns at indices ϕ. the pseudocode for sparta is given in algorithm 1. whereas optimizing lora requires computing gradients with respect to the a and b's used to compute ∆ (which constitute additional parameters, independent from θ pt ), sparta requires computing gradients with respect to ∆ ϕ , the changes over a subset of parameters chosen from θ pt and indexed by ϕ. generating the index set ϕ in sparta is done by sampling from a bernoulli independently and including each scalar parameter in ϕ with probability m/n. hence, sparta uses m trainable parameters out of n in expectation. finally, inference is performed, similarly to lora, by merging the finetuned ∆ into θ pt and then making the necessary forward passes. thus, sparta does not introduce any additional inference latency. ## experimental setup we next detail our experimental framework. motivation is first given for the tasks followed by a description of models to be used and how they are adapted (with manipulations) to the tasks. ## experimental results efficacy of sparta is demonstrated empirically by comparing it to two baselines, lora (only the additional matrices representing low-rank adapters are optimized) and dora (magnitude is optimized in addition to low-rank adapters). we also explore the performance of sparta on a range of sparsity levels, varying the number of trainable parameters. we consider several nlu benchmarks, including imdb @xcite , glue @xcite , boolq @xcite and mmlu @xcite . see appendix b for detailed descriptions. ## conclusion as pt language models have grown in size, peft has become crucial for enabling fine-tuning large pt language models on limited hardware and financial resources. we have introduced sparta, an approach that sharply decreases the set of trainable parameters, reducing the gpu memory used by the optimizer and speeding up training. we have demonstrated on a variety of task adaptation scenarios that our fine-tuning approach is parameterefficient and competitive with lora, the current peft standard. experiments with 2b and 7b parameter pre-trained models demonstrate good performance, and as per @xcite , we expect larger models to allow for higher levels of sparsity in training, meaning that efficiency of sparta should get better with larger model sizes as also suggested in regarding future directions, while sparta has been applied to supervised learning, it is also amenable to reinforcement learning often used for model alignment @xcite . we also plan to explore the merging of sparta adapters due to their potential for little interference.