Different Tunes Played with Equal Skill: Exploring a Unified Optimization Subspace for Parameter-Efficient Tuning (2022)

Abstract: Delta tuning (DET, also known as parameter-efficient tuning) is deemed as the new paradigm for using pre-trained language models (PLMs). Up to now, various DETs with distinct design elements have been proposed, achieving performance on par with fine-tuning. However, the mechanisms behind the above success are still under-explored, especially the connections among various DETs. To fathom the mystery, we hypothesize that the adaptations of different DETs could all be reparameterized as low-dimensional optimizations in a unified optimization subspace, which could be found by jointly decomposing independent solutions of different DETs. Then we explore the connections among different DETs by conducting optimization within the subspace. In experiments, we find that, for a certain DET, conducting optimization simply in the subspace could achieve comparable performance to its original space, and the found solution in the subspace could be transferred to another DET and achieve non-trivial performance. We also visualize the performance landscape of the subspace, and find that, there exists a substantial region where different DETs all perform well. Finally, we extend our analysis and show the strong connections between fine-tuning and DETs. The codes are publicly available at https://github.com/thunlp/Unified-DeltaTuning.

## introduction serving as the critical backbone for nlp, pretrained language models (plms) achieve superior performance when adapted to downstream tasks @xcite . conventionally, the dominant way for such an adaptation is fine-tuning, which requires updating and storing all the parameters in plms. consequently, with ever-larger plms continually being proposed @xcite , fine-tuning becomes extremely computationally expensive. as an alternative, various delta tuning algorithms (dets) spring up, which freeze most of the parameters and only optimize minimal adaptive parameters @xcite . up to now, various dets have been proposed, including introducing extra tunable neuron modules @xcite , specifying partial parameters to be tunable @xcite and re-parameterizing part of existing modules in plms @xcite , etc. dets extensively reduce the number of tunable parameters, and still achieves comparable downstream performance to fine-tuning. despite the success of dets, the mechanism behind it remains unclear. an essential question is: how could the plm adaptation using different dets relate to each other? to answer this question, a direct exploration of the connections among different dets is needed, but this would run into a problem: due to the versatile designs of dets, the parameter space of various dets is inherently different. to address the issue and investigate the above research question, we hypothesize that the adaptations of different dets could be reparameterized as low-dimensional optimizations in a unified optimization subspace. in this sense, optimizing various dets can all be viewed as finding optimal solutions within the same subspace. our hypothesis is inspired by recent findings that despite owning huge amounts of parameters, plms have an extremely low intrinsic dimension @xcite . in this regard, optimizing a certain pet, which is typically a high-dimensional optimization problem, could be equivalently re-parameterized as a low-dimensional optimization problem, while achieving non-trivial performance. to find evidence for our hypothesis, we design an analysis pipeline as follows: we first independently obtain solutions for different dets on a set of tasks. then we learn to project these solutions to a desired subspace. meanwhile, we also define a mapping from the subspace to each det's original space. we contend that if the found subspace is indeed shared among various dets, then two conditions should be satisfied: (1) the optimizations of different dets could be equivalently conducted in the found subspace and achieve nontrivial performance, and (2) the local optima of various dets have a substantial intersection in the subspace, which means the solution obtained in the subspace using a certain det could be directly transferred to other dets. if both conditions are well-established for the found subspace, then we could validate the existence of the unified optimization subspace for dets. we conduct experiments on a series of representative nlp tasks, and demonstrate that in the found subspace: when visualizing the performance landscape, we find that there exists a substantial region where different dets all perform well, indicating the close connections among different dets. in general, our study is the first work to reveal the connections among different dets and fine-tuning from the perspective of subspace optimization, and uncovers the underlying mechanism of plms' downstream adaptation. we believe many applications such as the ensemble and transfer among various dets can be well empowered by the unified optimization subspace. our findings can be of interest to researchers who are working on designing better dets, and may provide some guidance for using dets in many real-world scenarios. to fathom the mechanisms behind det, @xcite pioneered to explore the connections among different dets. they formalize various dets as different ways to compute the modifications on the hidden states and unify different dets in terms of formulas. however, the unification in the formula does not reveal the essence of dets' success, and does not indicate that their internal mechanisms are unified. our paper differs from theirs in that we explore whether dets can be unified in terms of internal mechanisms through the lens of optimization. specifically, we investigate whether the optimization of different dets can be unified in a certain subspace. intrinsic dimension. intrinsic dimension @xcite estimates the minimum number of tunable parameters needed to reach a satisfying performance for neural networks. instead of training networks in their native parameter space, they linearly re-parameterize all the tunable parameters θ 0 in a randomly oriented subspace: θ ← θ 0 + proj(θ i ), where proj : r 2021 ) reduce the computational complexity using fastfood transformation @xcite . in experiments, they find that for plms, a lowdimensional (e.g., |θ i | ∼ 10 3 ) re-parameterization could achieve over 85% performance of fine-tuning (|θ 0 | exceeds millions or even billions). further, @xcite extend the tuning method from fine-tuning to prompt tuning @xcite . they demonstrate that the projection proj can be trained in order to approximate a better optimization subspace. based on previous explorations of intrinsic subspace, we aim to validate the existence of a unified subspace for various tuning methods. ## preliminary following @xcite , we investigate three representative det algorithms to validate our hypothesis, including adapter @xcite , prefix-tuning @xcite , and lora @xcite . we will first recap the transformer layer @xcite , and then give a brief review of the three dets. transformer layer. plms generally have multiple transformer layers, each consisting of a multihead attention (mha) and a feed-forward network (ffn). mha is composed of n h attention heads, each containing a query / key / value weight matrix each (q (i) , k (i) , v (i) ) triple is then fed into a self-attention function to obtain the i-th head's representation h i . all head representations are then concatenated and combined using an output weight matrix w @xmath0: the ffn module is a two-layer mlp: adapter. adapter @xcite plugs in light-weight feed-forward networks in transformer layers (after the mha module and the ffn module). every adapter layer typically consists of a down-projection matrix w @xmath1, a nonlinear activation function f (•), and an up-projection matrix w @xmath2, where r a denotes the bottleneck dimension. denote the input as @xmath3, adapter applies a residual connection as follows: x ← x + f (xw down )w up . prefix-tuning. prefix-tuning (li and liang, 2021a) extends the queries k (i) / the values v (i) in every mha module by prepending learnable prefix vectors p (i) k / p (i) @xmath4, where m denotes the number of virtual tokens. the output of an attention head h i can be re-formulated as: lora. lora @xcite re-parameterizes the weight updates ∆w of the weight matrix w in the mha module with low-rank decompositions, i.e., ∆@xmath5, where w @xmath6, with r l being typically a small integer. for an input @xmath7, lora is formulated as: ## analysis pipeline as mentioned before, we consider three representative dets: adapter (t a ), prefix-tuning (t p ), and lora (t l ). each det t * defines a set of tunable parameters θ t * . to adapt a plm to a specific downstream task t i , we optimize θ i t * to minimize the loss function l i task (θ i t * |θ 0 ) defined by t i , where θ 0 denotes the pre-trained weights. to verify our hypothesis that there exists a unified optimization subspace where all dets can achieve non-trivial performance, we propose a three-stage analysis pipeline (visualized in figure 1 ), where the first stage is designed to approximate the desired subspace, so that in the second stage, the optimizations for different dets could all be conducted in this subspace. this makes it possible to explore the connections of different dets in the third stage. following @xcite , to validate the generality of the found subspace and avoid information leakage, we approximate the subspace with a series of training tasks t train , and conduct subsequent subspace optimization on unseen tasks t test . ## experiment we conduct experiments on representative nlp tasks. we first introduce the experimental setups in §5.1, next we approximate the subspace and present the analysis in §5.2. lastly, we explore the connection between dets and fine-tuning in §5.3. ## conclusion in this work, we explore the hypothesis that the adaptations of different delta tuning methods could all be re-parameterized as low-dimensional optimizations in a unified optimization subspace. the empirical results provide strong evidence for our hypothesis. we also extend our analysis to find the connection between fine-tuning and delta tuning. we hope our findings could provide insights for future research in designing better tuning methods and understanding the mechanisms behind plm adaptation.