A Semantic-Aware Layer-Freezing Approach to Computation-Efficient Fine-Tuning of Language Models (2025)

Abstract: Finetuning language models (LMs) is crucial for adapting the models to downstream data and tasks. However, full finetuning is usually costly. Existing work, such as parameter-efficient finetuning (PEFT), often focuses on how to finetune but neglects the issue of where to finetune . As a pioneering work on reducing the cost of backpropagation (at the layer level) by answering where to finetune, we conduct a semantic analysis of the LM inference process. We first propose using transition traces of the latent representation to compute deviations (or loss). Then, using a derived formula of scaling law, we estimate the gain of each layer in reducing deviation (or loss). Further, we narrow down the scope for finetuning, and also, study the cost-benefit balance of LM finetuning. We perform extensive experiments across well-known LMs and datasets. The results show that our approach is effective and efficient, and outperforms the existing baselines. Our approach is orthogonal to other techniques for improving finetuning efficiency, such as PEFT methods, offering practical values on LM finetuning.

## introduction with the rapid advancements and notable performance of language models, their application has extended to numerous downstream tasks @xcite . fine-tuning techniques are pivotal in augmenting the capabilities of language models @xcite . for example, code llama is a code-specialized lm and is finetuned on 100b tokens of python code for a language-specialized variant @xcite . the python variant provides better capabilities in code understanding and generation, since python is most popular in programming @xcite . compared to their smaller pretrained predecessors, finetuning large lms offers both advantages and disadvantages. on one hand, the vast number of model parameters triggers the emergent abilities of large lms @xcite , leading to superior performance across a variety of tasks, which serves as an excellent foundation for domain-specific finetuning. on the other hand, the extensive parameter size presents challenges for downstream finetuning. for instance, large lms require greater memory costs and higher computational costs in finetuning. the challenge is on finding the correlation between performance and efficiency of lm finetuning. there have been developed techniques such as model quantization and peft methods to improve efficiency @xcite . model quantization reduces the precision of the model and data to reduce the burden of storage and computation. however, the performance of lm finetuning may be damaged to some extent. most peft methods introduce additional parameters to learn the updates in lm finetuning, and then merge the updates into the lm. their focus is to reduce the memory cost instead of the computational cost @xcite . overall, there has been relatively little work exploring the correlation between model performance and computational efficiency, that is, whether the performance of lm finetuning can be improved while saving computation cost. to mitigate the gap, we propose utilizing the semantics in lm latent space to specify the layers that are more in need of finetuning being trainable, and freeze other layers. our intuition is that, by interpreting the lm's functionality as a transition of semantics and comparing it with a set of special latent representations, we can estimate the gains of each layer in reducing deviations. the deviations can be used to evaluate the convergence degree of model layers, and further, as the evidence to decide which layers shall be trainable. based on empirical experience and theoretical analysis, the deviations in semantic transitions greatly decide the effects of lm finetuning. by freezing model layers with the maximum gains in reducing deviation and shortening the process of backpropagation, the computation cost may be reduced and meanwhile, the finetuning effects can be improved. computational-efficient finetuning via layer-freezing is orthogonal with existing techniques, including model quantization and peft methods, so can combine with these techniques to achieve more efficient performance. in this paper, we realize computation-efficient model finetuning by proposing an effective and reliable layer-freezing approach, referred to as semantic-aware layer-freezing (salf). first, on the shoulder of lm semantics @xcite , we study the phenomenon of semantic transition in lms. by deriving the scaling law of lm pretraining, we estimate the gains of reducing deviations in each model layer; next, our layer-freezing approach finds the model layer whose gain is the maximum and only finetunes the deeper layers; last, to support a flexible cost-benefit tradeoff in lm finetuning, we propose a shallow-to-deep policy for layer-freezing under a given budget. we also propose better budget plans for the tradeoff. we evaluate our approach in fine-tuning diverse datasets on a wide range of modern lms. based on the results, our semantic-based layer-freezing approach performs better than baselines. combined with budget plans, our approach can further reduce the computation cost and improve lm performance. we discuss the insights of efficient finetuning from the perspective of semantics and conclude the findings in lm finetuning. the replication artifact is available online for open science 1 . our contributions are as follows: ## computation-efficient fine-tuning figure 2 : our salf approach is demonstrated with a lm with 4 layers (so there are 5 latent spaces). the rectangles with dark bars are the deviations, and the rectangles with cross-hatching are the gains in reducing deviations. in lm finetuning, salf uses a semanticbased analysis to compute the deviations in each latent space, and then uses a derived formula of the scaling law to estimate the gains of each model layer. salf will find the layer with the maximum gain and only finetune the deeper layers. in the illustration, layer 2 is chosen and the first two layers are frozen, so only layer 3 and layer 4 will be finetuned. the layers and spaces marked in gray color mean their gains and deviations will remain unchanged in lm finetuning. semantic-aware layer-freezing, shortened as salf, is a novel technique to reduce the computation cost in lm finetuning, by freezing certain layers. the core idea is dropping the unnecessary computation in lm backward-pass. due to the chain rule in loss backpropagation, the computation on shallower layers requires the computation in deeper layers. that is, salf realizes a computation-efficient lm finetuning by freezing the first few layers. to guarantee that layerfreezing will not damage the finetuning effects, and even improve the finetuning effects, we proposed a semantic-based analysis on lm inference and a derived formula of scaling law to estimate the convergence of layers. an illustration of our salf approach is shown in figure 2 . we also introduce strategies of assigning data samples under a given budget, to obtain a good cost-benefit balance for layer-freezing. figure 3 : the transition of semantics is illustrated with a 4-layer lm, whose vocabulary is a collection of colorful labels. the green dot is the medium token (input-side semantic basis) and the blue dot is the ground truth (outputside semantic basis). the solid/dashed black curves (transition trace) represent the semantic transition of the medium token, defined by the dark dots (hidden states) in each latent space. the solid/dashed green lines (semantic deviation) indicate the differences between the hidden states and the ground truth, and they differ before and after lm finetuning: comparing solid green lines (before finetuning) with dashed green lines (after finetuning), the semantic deviation in each layer is reduced. through lm finetuning, the hidden state in the last-layer semantically approaches to the ground truth, and the argmax label becomes from red to blue. ## analysis on cost-benefit tradeoff we study the performance and cost-benefit tradeoff of budget plans and infilling practices to layerfreezing. for example, a geometric-growth budget with breadth-first infilling is denoted as geom @xcite . as shown in as shown in considering the efficiency and cost-benefit tradeoff, the budget of arithmetic-growth shows equivalent performance but saves more computation costs. also, the practice of depth-first infilling is better than breadth-first infilling. based on the results, an arithmetic-growth with depth-first infilling saves around 1/3 more computation cost and has a slightly better performance. the reason explaining why the combination is performant is the same as ## related work leveraging the layered structure of neural models, the concept of layer-freezing was proposed decades ago, but mainly for deep belief networks (dbn) @xcite . dbn is a stack of directed sigmoid belief network (sbn) @xcite and an undirected restricted boltzmann machine @xcite . the backpropagation is only applied to finetune the restricted boltzmann machine, while the dependencies between other layers are not bidirectional. therefore, progressively training each layer is proposed as a greedy strategy for training dbn @xcite . in the era of language models, there has been little significant work studying layer-freezing for efficient finetuning, while the focus often lies on parameter-efficient, namely reducing the amount of trainable parameters, instead of computationefficicent @xcite . one reason is the complexity and interpretability of language models. besides, the correlation between model layers is not intuitive, and the effects of bidirectional dependencies on layer-wise finetuning have not been studied. another reason is that, the prior work on peft shows similar effects on reducing the number of trainable parameters, or even making the trainable parameters detachable. ## conclusion in this paper, we have proposed the novel concept of semantic transition. by defining transition trace to describe the change of semantic meaning of the next token, we explain lm finetuning as the process of letting the representation gradually steer to the corresponding ground truth in latent space. meanwhile, based on a derived law of scaling law, we can reasonably estimate and compare the capability of model layers, so to better allocate the computation resources in lm finetuning. further, we propose layer-freezing to accelerate lm finetuning, by finding the layer with the maximum gains of reducing deviation and finetuning deeper layers. based on our results on diverse datasets and multiple models, semantic-aware layer-freezing provides better performance than the state-of-the-art as well as common practices. moreover, our work explores the effects of budget plans on the costbenefit tradeoff for layer-freezing. in return, the effectiveness of our lay-finetuning approach validates the usefulness of semantic transition.