Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models (2025)

Abstract: Transformer-based large-scale pre-trained models achieve great success. Fine-tuning is the standard practice for leveraging these models in downstream tasks. Among the fine-tuning methods, adapter-tuning provides a parameter-efficient fine-tuning by introducing lightweight trainable modules while keeping most pre-trained parameters frozen. However, existing adapter-tuning methods still impose substantial resource usage. Through our investigation, we show that each adapter unequally contributes to both task performance and resource usage. Motivated by this insight, we propose Selective Adapter FrEezing (SAFE), which gradually freezes less important adapters early to reduce unnecessary resource usage while maintaining performance. In our experiments, SAFE reduces memory usage, computation amount, and training time by 42.85%, 34.59%, and 11.82%, respectively, while achieving comparable or better task performance compared to the baseline. We also demonstrate that SAFE induces regularization effect, thereby smoothing the loss landscape, which enables the model to generalize better by avoiding sharp minima.

## introduction large-scale pre-trained language models (plms) have manifested superior performance in various tasks @xcite @xcite . however, training plms from the scratch is timeconsuming and resource-intensive. common practice has been hence to fine-tune the large-scale pre-trained models by adapting all the parameters with the downstream tasks, i.e., full parameter finetuning (full-tuning). recently, parameter-efficient fine-tuning (peft), which focuses on optimizing a small fraction of parameters for downstream tasks, is receiving much attention @xcite @xcite @xcite . among various peft strategies, adapter-tuning has emerged as a prevalent method. it integrates lightweight modules, termed adapters, into each layer of plms and only tunes the adapters with the downstream tasks. as shown in figure 1 (a), the adapter-tuning methods @xcite @xcite @xcite , significantly reduce the number of trainable parameters compared to the full-tuning while exhibiting better performance on a downstream task. as adapter-tuning reduces the number of trainable parameters, it is also expected to reduce the resource (i.e., memory) usage accordingly. unfortunately, parameter-efficiency does not always translate into resource-efficiency. as shown in figure 1 (b), although adapter-tuning significantly reduces the number of trainable parameters (by 99.37%, on average) compared to the full-tuning, the memory usage is not much reduced (only by 22.19%, on average). this is because adaptertuning does not reduce activation memory (i.e., intermediate values for reuse during backpropagation) which account for 76.00% of memory usage -it only reduces optimizer memory (e.g., gradients and momentum vectors). considering the remarkable increase in model size compared to the modest increase in gpu memory capacity, adaptertuning methods still face challenges in terms of memory efficiency. for example, fine-tuning of a llama-65b @xcite requires more than 780gb of gpu memory. as shown in figure 1 (b), enabling resource-efficient fine-tuning may enhance accessibility of fine-tuning to researchers and end-users, by reducing memory requirements below the capacity of commodity gpu memory. according to previous work, the activation memory mostly depends on the backpropagation length @xcite , which is determined by the number of adapters trained during the backward pass. hence, to reduce the activation memory, it is crucial to reduce the number of training adapters. however, merely reducing the number of training adapters degrades accuracy. here, a pivotal research problem arises: can we reduce the number of training adapters without sacrificing accuracy? to answer the question, we analyze the impact of individual adapters on the accuracy and resource usage of training (figure 2 in section 3). we observe that some adapters are being trained, even after they finish contributing to the accuracy improvement, occupying memory. thus, it is possible to stop training (i.e., freezing) such adapters early if they do not contribute to the adaptation for a downstream task, de-allocating their activation memory. we also observe that such early freezing can even lead to the regularization effect on the model @xcite , improving the accuracy. in this paper, we propose safe (selective adapter freezing), which adaptively freezes adapters in the early epochs of training. in each epoch, safe identifies adapters that contribute relatively less to the accuracy improvement by using an importance score @xcite . it then freezes the adapters whose importance score is lower than a pre-defined threshold, reducing the memory usage and accelerating training time. by early freezing less important adapters, safe induces regularization effect on the model being trained, leading to a flatter loss surface. this is beneficial for finding an optimal point with higher generalization performance while optimizing neural network. in our evaluation, safe significantly reduces the average memory usage and tflops by 46.89% and 51.73%, respectively, across various models and downstream tasks without compromising accuracy compared to the baseline, lora @xcite . safe even improves the accuracy for some tasks, compared to lora, by up to 4.33% while reducing memory usage by 53.60%, by inducing the regularization effect. in summary, our key contributions include: ## related work parameter-efficient fine-tuning: to efficiently adapt large-scale plms to downstream tasks, many adapter-tuning methods @xcite @xcite @xcite have been proposed. in general, adapter-tuning methods inject small, trainable, and task-specific adapter modules into each transformer layer of a pre-trained model. given a pre-trained weight matrix w 0 ∈ r d×k and input @xmath0, the weight update of adapter-tuning is expressed as w 0 +∆w . during training, w 0 is frozen and does not receive gradient updates, while ∆w contains trainable parameters. for @xmath1, the modified forward pass in adapter-tuning yields: to further improve parameter efficiency of adapter-tuning, adalora @xcite adaptively adjusts the number of trainable parameters among adapters according to their importance score -it reduces the number of trainable parameters for less important adapters. however, the adapter-tuning methods still use a large amount of memory, as shown in figure 1 , since they do not reduce the activation memory which accounts for a large portion of memory usage. pruning llm model parameters: to reduce the model memory of fine-tuning, two categories of pruning methods have been proposed @xcite : structured pruning and unstructured pruning. structured pruning methods remove grouped parameters (e.g., channels, layers) from the llm. however, they usually degrade the accuracy. furthermore, they have a limitation in terms of the compression ratio because of the low flexibility. llm-pruner @xcite compensates the accuracy drop coming from pruning, by employing post-training. to overcome the limitation of structured pruning, unstructured pruning methods remove partial values of weight matrices regardless of their structures @xcite . however, unstructured pruning also degrades the accuracy. resource efficient fine-tuning: several works have tried to target resource efficient fine-tuning. adapterdrop @xcite , randomly excludes partial adapters from each training step. however, it cannot de-allocate the activation memory for the adapters, because of the random selections -an adapter excluded from training in a step can be included in training in the following steps. sparseadapter @xcite applies unstructured pruning to the adapters. however, it also does not reduce the actual memory usagethis is because the weight matrices pruned with zero values still need to be fully allocated in the memory. loraprune @xcite employs structured pruning for lora. unfortunately, the aforementioned methods usually have an adverse impact on the accuracy. meft @xcite applies a reversible model to peft. by using the reversible network, meft calculates activations with accumulated outputs of layers, without saving the intermediate activations reducing the activation memory. however, calculations of the activations severely degrades the training time performance. different from the previous works, this work freezes less important adapters in early steps of training. since the frozen adapters can only be used for the forward pass, early freezing of less important adapters can effectively reduce the backpropagation length as well as the activation memory. moreover, it induces regularization effect on the model, improving its accuracy. ## motivation in this section, we present a pivotal research question for resource-efficient fine-tuning. rq: do all adapters contribute equally to the process of adaptation? to answer this question, we analyze the impact of adapters injected into each transformer layer on accuracy and resource efficiency. we measure the accuracy and memory usage of bert base model on mnli and qnli dataset from glue @xcite , by attaching an adapter to each transformer layer one-by-one. figure 2(a ) and (b) show the measured accuracy and memory usage respectively -the x-axis indicates the index of transformer layer that the adapter is injected into. as shown in figure 2(a) , each adapter has different impact on the accuracy, and the importance of each adapter varies depending on the downstream task. in addition, despite uniform counts of trainable parameters, resource usage decreases for adapters closer to the output layer, as depicted in figure 2 @xcite . these observations point to the possibility that adapters in early layers contribute less to task adaptation, even though they require considerable resources. in other words, if we selectively deactivate less impactful adapters, it is possible to co-optimize the resource efficiency and accuracy. to further analyze changes of the feature representations for each adapter throughout the training process, we quantify the representation similarity between adapters in each training step and those in the final model (which we obtained after the convergence of fine-tuning). we quantify the representational similarity using centered kernel alignment (cka) by referring to previous works @xcite . figure 3 visualizes the representational similarity measured throughout the training process for each adapter for bert base model on mnli and qnli dataset from glue @xcite -lighter the color becomes, higher the feature representation similarity is. as shown in figure 3 , even in the early training steps, the feature representations of several adapters are almost the same as those of the final model -similar patterns are observed in other models and datasets. this means that those adapters are already representing the features that should be represented by the final model, and thus they may not further be adapted for the downstream task in the rest of the training steps. this is why such adapters are less contributing to the accuracy improvement of the model on the downstream tasks in figure 2 . one intuition is that lower adapters generally learn basic understanding of the input data, such as data bias and structural characteristics of the data, while adapters closer to the output build features unique to different tasks @xcite . motivated by the observation where not all adapters consistently contribute to adaptation, in the next section, we propose a selective adapter freezing method which preemptively freezes adapters that are relatively less important for each task. ## selective adapter freezing (safe) in this section, we propose a selective adapter freezing method, safe. safe adaptively freezes less important adapters in the early training steps, in order to reduce unnecessary computation and mem-ory usage without compromising the accuracy. figure 4 shows the overview of safe. safe consists of two stages: warm-up stage and freezing stage. in the warm-up stage, safe performs several epochs of fine-tuning while monitoring the feature representation changes (i.e., importance score) of the adapters (section 4.1). if the important score of all adapters is not much changed for consecutive epochs, safe enters the freezing stage. in the freezing stage, safe gradually freezes adapters that contribute less to the adaptation, based on the importance score (section 4.2). by early freezing less important adapters, safe induces regularization effect on model (section 4.3), leading to better performance. ## conclusion in this paper, we propose safe, which selectively freezes adapters for enabling resource efficient fine-tuning of plms. we observe that not all adapters contribute equally to adaptation. motivated by the observation, safe gradually freezes less-important adapters, which do not contribute to adaptation during the early training steps. in our evaluation on various models and datasets, safe significantly saves memory usage and computation and accelerating training time, with comparable (or even better) accuracy. we also demonstrate that safe induces regularization effect, thereby improving generalization performance and accuracy compared to the state-of-the-art peft methods. we believe that safe can enable resource-efficient fine-tuning of large-scale plms, and further pave the path forward to personalized fine-tuning on resource-constrained edge devices.