Advancing Parameter Efficiency in Fine-tuning via Representation Editing (2024)

Abstract: Parameter Efficient Fine-Tuning (PEFT) has gained significant attention for its ability to achieve competitive results while updating only a small subset of trainable parameters. Despite the promising performance of current PEFT methods, they present challenges in hyperparameter selection, such as determining the rank of LoRA or Adapter, or specifying the length of soft prompts. In addressing these challenges, we propose a novel approach to fine-tuning neural models, termed Representation EDiting (RED), which scales and biases the representation produced at each layer. RED substantially reduces the number of trainable parameters by a factor of 25,700 compared to full parameter fine-tuning, and by a factor of 32 compared to LoRA. Remarkably, RED achieves comparable or superior results to full parameter fine-tuning and other PEFT methods. Extensive experiments were conducted across models of varying architectures and scales, including RoBERTa, GPT-2, T5, and Llama-2, and the results demonstrate the efficiency and efficacy of RED, positioning it as a promising PEFT approach for large neural models.

## introduction pre-training on large-scale unlabeled datasets followed by fine-tuning on task-specific dataset has demonstrated remarkable efficacy across various natural language processing (nlp) tasks, establishing itself as the prevailing training paradigm @xcite @xcite . however, conducting full parameter fine-tuning for each task can be exceedingly costly and increasingly daunting as model scales continue to grow @xcite . for instance, bert comprises up to 220 million parameters, t5 scales up to 11 billion parameters, and gpt-3 boasts an astounding 175 billion parameters. consequently, the efficient and effective adaptation of large models to specific downstream tasks presents an intriguing research challenge @xcite . in response to this challenge, researchers have put forward three main lines of parameter efficient fine-tuning (peft) techniques @xcite . firstly, addition-based methods involve the introduction of additional trainable neural modules or parameters that were not present in the original model @xcite @xcite . specification-based methods, on the other hand, identify certain parameters in the original model to be trainable, while the rest are kept frozen @xcite . lastly, reparameterization-based methods reconfigure trainable parameters into a more parameterefficient form through certain transformations @xcite @xcite . among these peft methods, low-rank adaptation (lora) stands out as one of the most efficient techniques with its effectiveness empirically validated across various models of diverse scales. despite its impressive performance, lora still demands a significant number of trainable parameters. recent studies by @xcite and @xcite indicate that the upper bound for intrinsic dimensions is substantially smaller than what is typically used in such methods. for example, the d 90 value (the minimum number of trainable parameters required to reach 90% of the performance of the fully-parameter fine-tuned model) for the roberta base is reported to be 896. however, when lora is applied to fine-tune this model, the number of trainable parameters escalates to 0.3 million, suggesting the potential for further reduction in parameter count. in addition to the issue of requiring too many adjustable parameters, existing peft methods @xcite @xcite primarily focus on the design of lightweight modules and their integration (or placement) within base models. nonetheless, the implementation of these peft techniques introduces additional complexities in hyperparameter selection, such as choosing the rank of lora and adapter, or deciding on the length of soft prompt and prefix. inspired by the concept of representation engineering @xcite , we shift our focus away from the weights of models and turn our attention to their representations. in the neural architecture, network weights govern neural activities (or representations), which in turn determine the networks' output, and the networks' output ultimately shapes the networks' behavior. rather than concentrating on neurons and their interconnections (or weights), we explored to achieve control over network behavior by manipulating its internal representations. specifically, we fine-tune neural network models by directly editing the representation generated at each layer while maintaining the model parameters frozen, as illustrated in figure 1 @xcite . it is worth noting that the number of parameters required to edit representations is substantially fewer than that of weights within neural networks. taking llama-2 (7b) as an example, the proposed representation editing (red) method achieves competitive performance by adjusting only 0.26m parameters. this is approximately 25, 700 times less than what is required for full-parameter fine-tuning, rendering the method both storage and computation efficient. the contributions of this study are summarized as follows: ## related work existing peft methods can be broadly classified into three categories @xcite , each characterized by the particular parts of parameters that they tune efficiently: addition-based, specificationbased, and reparameterization-based methods. addition-based methods perform fine-tuning by integrating additional lightweight learnable components into foundational models. more specifically, @xcite ; stickland and @xcite ; karimi @xcite and @xcite proposed the integration of learnable bottleneck neural modules into the transformer layers. @xcite and @xcite found that the models' performance could be enhanced by appending discrete tokens (or prompts) at the beginning of the input texts without modifying the models' parameters. however, the manual creation of such prompts demands substantial effort, and optimization in a discrete space is comparatively more challenging. consequently, later studies substituted these discrete tokens with continuous vectors, also referred to as soft prompts, which can be optimized using the gradient descent algorithm @xcite @xcite . specification-based methods achieve parameterefficient fine-tuning by designating a subset of parameters for modification while leaving the remaining parameters untouched. within this category, @xcite suggested a method to exclusively update the parameters in certain top layers of bert and roberta. bitfit @xcite , on the other hand, fine-tunes a model by only optimizing the bias terms used in the model. contrary to these methods that pre-determine which parameters should be altered, @xcite and @xcite implemented a learnable masking strategy to automatically choose the parameters for tuning. reparameterization-based methods aim to optimize some parameters within a model, typically in their low-dimensional subspace. @xcite introduces the use of low-rank matrices, termed lora, to approximate the weight modifications during the fine-tuning process. qlora @xcite combines low-rank adaptation with model quantization to further reduce storage usage during the fine-tuning phase. adalora @xcite proposes to use svd decomposition to approximate the changes in weights, which allocates more trainable parameters to more important weight matrices, leading to a better performance. @xcite proposed an approach to analyzing and manipulating the behavior of neural networks through representation engineering, thereby shifting the focus from neurons and their circuits to representations and the transformations among them. @xcite extended this work to align large language models with human preferences, achieving results comparable or superior to rlhf @xcite , but at a lower computational cost. @xcite investigated the extraction of "steer vectors" from the hidden layers and performed unsupervised text style transfer by altering the hidden representations using these vectors. to control the style and sentiment expressed in a model's outputs, turner et al. (2023) integrated a similar "steer vector" into the representation of each hidden layer at the inference time. inspired by the recent advance in representation engineering @xcite , we suggested a novel perspective for fine-tuning models by directly editing their representations. this new perspective was embodied in the red training method, which demonstrated both effectiveness and efficiency while requiring only a minimal number of trainable parameters. this study is closely related to two recent works: ia3 @xcite and ssf @xcite . to fulfill the in-context learning capability with lower computational costs, @xcite proposed to modify the key and value vectors of the multi-head attentions and those generated by feedforward networks (ffns) by scaling operations. in the domain of computer vision, @xcite presented a similar method, called ssf, which was motivated by batch and layer normalization techniques. to ensure features fall in a discriminative space for better classification, ssf also necessitates the adjustment of feature vectors across nearly all transformer layers, including multi-head attentions, ffns, and normalization layers. as a result, the size of parameters adjusted in their method approximates those manipulated in other peft techniques, such as adapter and vpt @xcite . in contrast, our red method requires only the modification of activation patterns generated by ffn sub-layers, constituting a significantly smaller portion of the entire neural network's parameters for model fine-tuning. while ssf achieves a reduction in the number of fine-tuning parameters by a factor of 300 compared to original models, red accomplishes a reduction of approximately 25, 700 times. moreover, ssf's evaluation was restricted to a limited range of simple image classification tasks using relatively small models (under 100m parameters), whereas red has demonstrated its effectiveness across a variety of natural language understanding and generation tasks, with signifi-cantly larger models of up to 7b parameters. ## method we in this section give a concise overview of previous peft methods and introduce a new, parameterefficient fine-tuning method known as representation editing (red) by highlighting its distinctive features compared to existing peft methods. red facilitates the adaptation of pre-trained neural network models to downstream tasks by directly editing the model's representations. ## experiments extensive experimentation was conducted to assess the efficacy of our representation editing (red) method across a diverse array of natural language understanding and generation tasks, employing a range of foundational models spanning different scales. these models include roberta @xcite , t5 @xcite , gpt-2 @xcite , and llama-2 @xcite . specifically, we evaluated red and the baseline methods on the glue benchmark @xcite with roberta and t5, as previously conducted in @xcite and @xcite . to ensure consistency with previous studies, we adhered to the experimental setup outlined in @xcite and @xcite for comparative analysis with gpt-2. moreover, we conducted instruction tuning experiments on llama-2 using the ultrafeedback dataset @xcite to further assess the applicability of our proposed red on generative large language models. for further details on the datasets and evaluation metrics used, please refer to appendix a. ## ablation study in this section, we conducted a series of ablation studies to examine the impact of different editing operations used in red and to explore potential application points within transformer-based networks. we also evaluated the parameter efficiency of red by comparing it with the most parameter-efficient variants of other peft methods. ## conclusion in this study, we proposed a novel perspective on fine-tuning by editing model representations rather than adjusting the model's weights. through extensive experimentation across various nlp tasks and neural models of varying structures and scales, we demonstrated that red can deliver performance comparable or superior to existing peft methods while employing a minimal number of trainable parameters. we are hopeful that this study could inspire novel methods for fine-tuning llms, thereby contributing to the advancement of both effective and computationally efficient peft techniques.