On Surgical Fine-tuning for Language Encoders (2023)

Abstract: Fine-tuning all the layers of a pre-trained neural language encoder (either using all the parameters or using parameter-efficient methods) is often the de-facto way of adapting it to a new task. We show evidence that for different downstream language tasks, fine-tuning only a subset of layers is sufficient to obtain performance that is close to and often better than fine-tuning all the layers in the language encoder. We propose an efficient metric based on the diagonal of the Fisher information matrix (FIM score), to select the candidate layers for selective fine-tuning. We show, empirically on GLUE and SuperGLUE tasks and across distinct language encoders, that this metric can effectively select layers leading to a strong downstream performance. Our work highlights that task-specific information corresponding to a given downstream task is often localized within a few layers, and tuning only those is sufficient for strong performance. Additionally, we demonstrate the robustness of the FIM score to rank layers in a manner that remains constant during the optimization process.

## introduction fine-tuning of language encoders is a crucial step towards applying natural language processing solutions to real-world challenges. it allows adaptation of knowledge on target distribution after generally training on source data, but requires curation of adequately sized labelled dataset to gain 'new' knowledge while retaining information obtained during the pre-training phase. although preserving knowledge of target distribution by tuning the entire model can yield impressive results, it can be expensive and may increase data volume requirements. additionally, fine-tuning all layers arbitrarily might risk overfitting or adversely affecting the generalization ability of the model during the transfer learning process. while, recently, the focus has shifted to development of parameter efficient approaches of finetuning large-language and language models @xcite , @xcite , @xcite ), these techniques still require development of an 'adapter' architecture relative to a target dataset. we therefore focus on developing a data-driven criteria to automatically identify and tune a smaller sub-set of layers using only ≈ 100 target data samples. in this paper, we propose a simple strategy to select layers for fine-tuning for real-world nlp tasks, leveraging the fisher information matrix (fim) score, which quantifies the impact of parameter changes on a model's prediction. we further demonstrate the effectiveness of fim score on language encoder model(s) on practical nlp tasks from glue and superglue benchmark, by identifying a subset of layers that are most informative to adapt to the target data distribution. we find that fine-tuning parameters in identified subset of layers by fim score, outperforms full model fine-tuning in some, and results in comparable performance to full fine-tuning in almost all the glue and super-glue tasks. in niche scenarios, where fim scores leads to selection of layers that contribute to suboptimal performance in comparison with full model fine-tuning, we investigate the nuanced characteristics of the glue and superglue tasks through the lens of linguistic features learned during transfer learning process, and potential categories of target data distribution shifts that could influence the performance while we surgically fine-tune. interestingly, we find that glue and super-glue tasks that are dependent on a simpler understanding of linguistic features such as syntax and semantics as well as discourse understanding, can be surgically fine-tuned using our proposed fim score criteria. however, we find that for tasks that rely on learning more complex knowledge of both high and low level linguistic features such as textual entailment, common sense and world knowledge fim score criteria unperformed to select the relevant layers. on investigating categories of target distribution shifts that could surface in various glue/superglue tasks, we also find that fim score signals at efficient tuning of parameters for the group of tasks that align closely with concepts of domain, environmental, and demographic shifts, but fails to perform optimally on tasks that require learning of temporal drifts in language. several attempts have been made to conduct parameter efficient fine tuning (peft) in natural language. @xcite and @xcite explores the landscape of utilizing adapter layers and soft prompts, including additive methods, selective methods, and reparameterization-based techniques, whereas @xcite applies network pruning to develop a pruned adapter (sparse dapter). in another body of work, @xcite constructed a fish (fisher-induced sparse unchanging) mask to choose parameters with the largest fisher information. additionally, @xcite attempted to efficiently fine-tune by proposing a low rank adapter that reduces trainable parameters by freezing the weights of the pre-trained model and injecting trainable rank decomposition matrices into each layer of the architecture. some fine tuning techniques involve fine-tuning a small subset of the model parameters. @xcite propose a way of reducing memory requirement by introducing ladder side tuning (lst). a small 'ladder side' network connected to each of the layers of the pre-trained model is trained to make predictions by taking intermediate activations as input from the layers via shortcut connections called ladders. @xcite demonstrated the advantages of few-shot parameter-efficient finetuning over in-context learning in terms of effectiveness and cost-efficiency. additionally, techniques like prompt tuning are also considered as parameter-efficient fine-tuning methods. a handful of studies have investigated the knowledge gain in fine-tuning process for language encoders, particularly bert. @xcite , @xcite investigated the impact of shuffling the order of input tokens on the performance of the bert model for several language understanding tasks. @xcite further investigates the effectiveness of masked language modeling (mlm) pre-training and suggests that mlms achieve high accuracy on downstream tasks primarily due to their ability to model distributional information. however, our approach of efficient fine-tuning using the proposed fim score criteria (that is able to capture signals from ≈ 100 target data samples), differs from all existing methods, as it focuses on helping nlp practitioners with small size target datasets to efficiently rank and select important layers for optimizing the fine-tuning process. ## conclusion this paper contributes to the growing body of work that demonstrate that selective fine-tuning of language models is not only efficient but also effective in many downstream tasks. summarizing our contributions, we strongly prove that selecting layers for finetuning based on ranking according to the fim scores gives optimal results on a majority of glue and superglue tasks and could thus help nlp practitioners with small datasets to efficiently select a sub-set of relevant layers for optimized fine-tuning for many real-world natural language tasks. in future work, we plan to investigate the linguistic correlates of different layers in large-language models (llms) and the value of fim in surfacing them.