Stochastic Bridges as Effective Regularizers for Parameter-Efficient Tuning (2023)

Abstract: Parameter-efficient tuning methods (PETs) have achieved promising results in tuning large pre-trained language models (PLMs). By formalizing frozen PLMs and additional tunable parameters as systems and controls respectively, PETs can be theoretically grounded to optimal control and further viewed as optimizing the terminal cost and running cost in the optimal control literature. Despite the elegance of this theoretical grounding, in practice, existing PETs often ignore the running cost and only optimize the terminal cost, i.e., focus on optimizing the loss function of the output state, regardless of the running cost that depends on the intermediate states. Since it is non-trivial to directly model the intermediate states and design a running cost function, we propose to use latent stochastic bridges to regularize the intermediate states and use the regularization as the running cost of PETs. As the first work to propose regularized PETs that use stochastic bridges as the regularizers (running costs) for the intermediate states, we show the effectiveness and generality of this regularization across different tasks, PLMs and PETs. In view of the great potential and capacity, we believe more sophisticated regularizers can be designed for PETs and better performance can be achieved in the future.

## introduction recent years have witnessed the dramatic growth of pre-trained language models (plms) in various fields @xcite . as the size of plms continues to increase, the number of parameters has now even reached hundreds of billions @xcite , making fine-tuning the whole plm both computationally impractical and environmentally unfriendly. in view of this, a variety of parameter-efficient tuning methods (pets) are proposed @xcite @xcite . by only tuning a small number of additional parameters, pets can be comparable to full-parameter fine-tuning. despite the success of pets, their underlying mechanism remains an open problem. recently, several works have proposed to interpret pets with optimal control theory. @xcite first show that the optimization in prefix tuning @xcite (a typical method of pets) can be considered as the search for optimal control variables in the context of optimal control, i.e., the trainable prefixes can be seen as the control variables that drive the plm (the system) to the desired output. @xcite further show that the optimal control perspective can be applied to almost all pets. the optimization of pets' parameters can be seen as minimizing the two cost functions in the optimal control literature: (1) terminal cost l t , which measures the quality of the terminal state, and (2) running cost l r , which measures the feasibility of the controlled intermediate states and the control variables. although l t can well correspond to the loss function of the model output, l r is only vaguely described as the regularizers on the parameters of pets (control variables) in yang and liu (2022) and @xcite , ignoring the dependency of l r on the intermediate states. in this work, we show that designing a running cost to regularize intermediate states not only makes the optimal control perspective of pets more theoretically sound but also empirically leads to better pets. we begin by assuming that in plms, the intermediate hidden states for generating different tokens in a sentence have different dynamics (or trajectories), and the dynamics can be approximated with stochastic processes in a latent space. specifically, we first freeze the plm and learn a mapping from the original hidden state space of the plm to a latent space. in the latent space, the dynamics of the intermediate hidden states for generating different target tokens can be approximated with different target-specific diffusion bridges. the obtained mapping can then be plugged to the model to regularize the intermediate hidden states when training pets. besides, since a diffusion bridge is (1) a markov process and (2) a solution to a stochastic differential equation (sde), we correspondingly propose two methods to learn the mapping: (1) fitting the markov transition probability density function (pdf) and (2) fitting the sde directly. the two methods act as a tradeoff between efficiency and effectiveness: the first method incurs only negligible computational cost and has satisfactory results, while the second one is slower but yields better regularizers. we conduct experiments on different plms of different sizes, and the experimental results on glue @xcite under both full-set and few-shot settings demonstrate the effectiveness of our proposal across four different pets. further analyses show that the learned regularizer helps pull apart the hidden states of different label words. we also observe that when we project the intermediate hidden states of pets without our regularizer into our latent space, the better the pets perform, the closer the latent states are to our latent bridges. this spontaneous approaching behavior may indicate that stochastic-bridge-like latent dynamics naturally exist in well-trained pets. in summary, our work has the following contributions: (1) guided by the perspective of optimal control for pets, we design latent stochastic bridge regularizers on the intermediate states during the training of pets. (2) we propose two methods to construct the latent space according to the two representations of stochastic bridges, offering a trade-off between efficiency and effectiveness. (3) our regularizers are shown to be effective and general across different plms, different pets, and different tasks. (4) we show that well-trained pets without any regularization spontaneously exhibit stochastic-bridge-like latent dynamics. ## experiments to verify the effectiveness and generality of the regularizers built on stochastic bridges, we conduct experiments on (1) different plms: bert large (340m) @xcite and deberta xlarge (750m) @xcite ; (2) different pets: prompt tuning, lora, bitfit and adapter; ## analyses ## related works recent years have witnessed the success of plms @xcite . however, as the sizes of plms continue to grow, it becomes increasingly impractical to perform finetuning on downstream tasks. many efforts have been devoted to pets, aiming to tune only a few parameters rather than the whole plm on downstream tasks. for example, prompt tuning @xcite prepends tunable embeddings to the input, adapter @xcite inserts small modules into each layer, bitfit @xcite tunes only the bias terms, and lora @xcite decomposes the weight updates into lowrank matrices. in this paper, based on the theoretical grounding of pets on optimal control @xcite , we propose stochastic bridges as the regularizer on intermediate hidden states and introduce regularized pets. our work also closely relates to continuous-time neural differential equations (ndes) @xcite @xcite . continuous-time ndes model the dynamics of the hidden states with odes or sdes parameterized by neural networks. inspired by these works, we use sdes to represent the latent dynamics of pets in the latent space. our work differs from them in that we focus on using neural sdes as regularizers for intermediate hidden states, rather than feature extractors. we also notice that @xcite explore the use of brownian bridge in regularizing the model dynamics across time. our work differs from theirs in that we regularize the dynamics of intermediate hidden states across model layers. we additionally show that other diffusion bridges can be easily applied as the regularizer. as far as we know, we are the first to show the diffusion-bridge-like dynamics for hidden states across plm layers and use diffusion bridges as regularizers on intermediate hidden states. ## conclusion starting from the optimal control perspective of pets, we notice that the existing pets lack a running cost that regularizes the intermediate hidden states. we thus propose to use stochastic bridges in a latent space as the regularizers for pets. experimental results on different models, tasks and pets show that the proposed regularizers effectively improve the pets' performance. our analyses further show that the hidden states of the vanilla pets spontaneously approach our diffusion bridges, indicating that there may exists intrinsically diffusionbridge-like dynamics in pets. as the first work using stochastic bridges as regularizers, we show its effectiveness and generality. we believe it will be a promising direction.