Original: S TOCHASTIC B RIDGES AS E FFECTIVE R EGULARIZERS FOR P ARAMETER -E FFICIENT T UNING
Revised: S TOCHASTIC B RIDGES AS E FFECTIVE R EGULARIZERS FOR P ARAMETER -E FFICIENT T UNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Parameter-efficient tuning methods (PETs) have achieved promising results in tuning large pre-trained language models (PLMs).
Revised: Parameter-efficient tuning methods (PETs) have achieved promising results in tuning large pre-trained language models (PLMs).

Original: By formalizing frozen PLMs and additional tunable parameters as systems and controls respectively, PETs can be theoretically grounded to optimal control and further viewed as optimizing  terminal cost and running cost in the optimal control literature.
Revised: By formalizing frozen PLMs and additional tunable parameters as systems and controls respectively, PETs can be theoretically grounded to optimal control and further viewed as optimizing the terminal cost and running cost in the optimal control literature.

Original: Despite the elegance of this theoretical grounding, in practice, existing PETs often ignore the running cost and only optimize the terminal cost, i.e., focus on optimizing the loss function of the output state, regardless of the running cost that depends on the intermediate states.
Revised: Despite the elegance of this theoretical grounding, in practice, existing PETs often ignore the running cost and only optimize the terminal cost, i.e., focus on optimizing the loss function of the output state, regardless of the running cost that depends on the intermediate states.

Original: Since it is non-trivial to directly model the intermediate states and design a running cost function, we propose to use latent stochastic bridges to regularize the intermediate states and serve as the running cost of PETs.
Revised: Since it is non-trivial to directly model the intermediate states and design a running cost function, we propose to use latent stochastic bridges to regularize the intermediate states and use the regularization as the running cost of PETs.

Original: As the first work to propose regularized PETs that use stochastic bridges as the regularizers (running costs) for  intermediate states, we show the effectiveness and generality of this regularization across different tasks, PLMs and PETs.
Revised: As the first work to propose regularized PETs that use stochastic bridges as the regularizers (running costs) for the intermediate states, we show the effectiveness and generality of this regularization across different tasks, PLMs and PETs.

Original: In view of the great potential and capacity, we believe more sophisticated regularizers can be designed for PETs and better performance can be achieved in the future.
Revised: In view of the great potential and capacity, we believe more sophisticated regularizers can be designed for PETs and better performance can be achieved in the future.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent years have witnessed the dramatic growth of pre-trained language models (PLMs) in various fields (Devlin et al., 2019; Dosovitskiy et al., 2021).
Revised: Recent years have witnessed the dramatic growth of pre-trained language models (PLMs) in various fields (Devlin et al., 2019; Dosovitskiy et al., 2021).

Original: As the size of PLM continues to increase, the number of parameters has now even reached hundreds of billions (Brown et al., 2020; Smith et al., 2022), making fine-tuning the whole model both computationally impractical and environmentally unfriendly.
Revised: As the size of PLMs continues to increase, the number of parameters has now even reached hundreds of billions (Brown et al., 2020; Smith et al., 2022), making fine-tuning the whole PLM both computationally impractical and environmentally unfriendly.

Original: In view of this, a variety of Parameter-Efficient Tuning methods (PETs) are proposed (Houlsby et al., 2019;
Revised: In view of this, a variety of Parameter-Efficient Tuning methods (PETs) are proposed (Houlsby et al., 2019;

Original: Hu et al., 2022; Zaken et al., 2022; Lester et al., 2021).
Revised: Hu et al., 2022; Zaken et al., 2022; Lester et al., 2021).

Original: By simply tuning a small number of additional parameters, PETs can be comparable to full-parameter.
Revised: By only tuning a small number of additional parameters, PETs can be comparable to full-parameter fine-tuning.

Original: Despite the success of PETs, its underlying mechanism remains an open problem.
Revised: Despite the success of PETs, their underlying mechanism remains an open problem.

Original: Recently, several works have proposed to interpret PETs from the perspective of optimal control.
Revised: Recently, several works have proposed to interpret PETs with optimal control theory.

Original: Yang & Liu (2022) first show that the optimizationprocess in Prefix Tuning (Li & Liang, 2021) (a typical method of PETs) can be considered as the search for optimal control variables in the context of optimal control, i.e., the  prefixes can be seen as the control variables that drive the PLM  to the desired output.
Revised: Yang & Liu (2022) first show that the optimization in Prefix Tuning (Li & Liang, 2021) (a typical method of PETs) can be considered as the search for optimal control variables in the context of optimal control, i.e., the trainable prefixes can be seen as the control variables that drive the PLM (the system) to the desired output.

Original: Ding et al.
Revised: Ding et al.

Original: (2022)go one step further and derive theconnection between optimal control and different PETs including Prefix Tuning, Adapter (Houlsby et al., 2019), LoRA (Hu et al., 2022) and BitFit (Zaken et al., 2022).
Revised: (2022) further show that the optimal control perspective can be applied to almost all PETs.

Original: The optimization of PETs’ parameters can be considered as minimizing a combination of two cost functions in the optimal control literature: (1) Terminal cost L T , which measures the quality of the terminal state, and (2) Running cost L R , which measures the feasibility of the controlled intermediatehidden states and the control variables.
Revised: The optimization of PETs’ parameters can be seen as minimizing the two cost functions in the optimal control literature: (1) terminal cost L T , which measures the quality of the terminal state, and (2) running cost L R , which measures the feasibility of the controlled intermediate states and the control variables.

Original: Although the terminal cost

Original:  L T can well correspond to the loss function of the model output,the running cost L Rreceives little attention in the previous works and is only vaguely described as the regularizers on the parameters of PETs (control variables) , ignoring the dependency of L R on the intermediate states.
Revised: Although L T can well correspond to the loss function of the model output, L R is only vaguely described as the regularizers on the parameters of PETs (control variables) in Yang & Liu (2022) and Ding et al. (2022), ignoring the dependency of L R on the intermediate states .

Original: In this work, we re-emphasize the importance of the running cost L R and demonstrate that PETs can benefit from a well-designed L R that regularizes the intermediate hidden states.
Revised: In this work, we show that designing a running cost to regularize intermediate states not only makes the optimal control perspective of PETs more theoretically sound, but also empirically leads to better

Original:  We begin by assuming that in PLMs, the intermediate hidden states for generating different tokens in a sentence have different dynamics (or trajectories), and the dynamics can be approximated with stochastic processes in a latent space.
Revised: PETs. We begin by assuming that in PLMs, the intermediate hidden states for generating different tokens in a sentence have different dynamics (or trajectories), and the dynamics can be approximated with stochastic processes in a latent space.

Original: More specifically, we first freeze the PLM and learn a mapping from the 
Revised: Specifically, we first freeze the PLM and learn a mapping from the original hidden state space of the PLM to a latent space.

Revised: In the latent space, the dynamics of the intermediate hidden states for generating different target tokens can be approximated with

Original: [Table]
Revised: W V W K Q

Original: 1. Endpoint Determination 2.
Revised: V K Attention Layer Adapter

Original: [Table]
Revised: W Q FFN Layer Adapter LoRA

Original: GLUE
Revised: LoRA

Revised: Word Embedding is cute

Original: [Table]
Revised: Endpoints for Bridges Tail of [is] Tail of [cute] PCA

Original:  [Figure] 
Revised: PCA [Table] [Figure] [Table]

Revised: h ( i ) o

Revised: Li

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] [Table]
Revised: +jUY+Sh0sScQhZybhh4IK4JMEERxXBiEHAcQTeEYm5J60MXcitIosXYilZHm+EweT2ip+nsjgi7nM9eWk0lKvugl4n+eGQrn3IqIF4QCe2j+kBNSTfhaUo42IgwjQWeSQMSIzKqhCWQQCVlhQZZgLH55mXSrFeOsUrurF5ulrI48OALHoAQM0ABNcANaoAMQeATP4BW8KU/Ki/KufMxHc0q2cwj+QPn8Ac90mzI=</latexit

Original: [Figure]
Revised: [Figure]

Original: original hidden state space to a latent space in which the intermediate hidden states for generating different tokens can be approximated with different  diffusion bridges .
Revised: different target-specific diffusion bridges .

Original: The obtained mapping can then be plugged to theside of the model to regularize the intermediate hidden states when using PETs on downstream tasks.
Revised: The obtained mapping can then be plugged to the model to regularize the intermediate hidden states when training PETs parameters.

Original: Besides, since a diffusion bridge isa (1)  Markov process and (2) a solution to a stochastic differential equation (SDE), we correspondingly propose two methods to learn the latent space: (1) fitting the Markov transition probability density function  and (2) fitting the SDE directly.
Revised: Besides, since a diffusion bridge is (1) a Markov process and (2) a solution to a stochastic differential equation (SDE), we correspondingly propose two methods to learn the mapping: (1) fitting the Markov transition probability density function (PDF) and (2) fitting the SDE directly.

Original: It turns out that the two methods act as a trade-off between efficiency and effectiveness: the first method incurs only negligible computational cost and has satisfactory results, while the second one is slower, but the learned regularizer has greater capability.
Revised: These two methods act as a trade-off between efficiency and effectiveness: the first method incurs only negligible computational cost and has satisfactory results, while the second one is slower, but yield better regularizers.

Original: We conduct experiments on different PLMs of different sizes, and the experimental results on
Revised: We conduct experiments on different PLMs of different sizes, and the experimental results on

Original: GLUE (Wang et al., 2019) under both full-set and few-shot settings demonstrate the effectiveness of our proposal across four different PETs.
Revised: GLUE (Wang et al., 2019) under both full-set and few-shot settings demonstrate the effectiveness of our proposal across four different PETs.

Original: Further analyses show that the  regularizer helps pull apart the hidden states of different label words.
Revised: Further analyses show that the learned regularizer helps pull apart the hidden states of different label words.

Original: We also observe that when we project the intermediate hidden states of PETs without our regularizer into our latent space, the better the PETs perform, the closer the latent states is to our latent bridges.
Revised: We also observe that when we project the intermediate hidden states of PETs without our regularizer into our latent space, the better the PETs perform, the closer the latent states are to our latent bridges.

Original: The spontaneous approaching behaviorindicates there may exists stochastic-bridge-like latent dynamics for well-trained PETs.
Revised: This spontaneous approaching behavior may indicate that stochastic-bridge-like latent dynamics naturally exists in well-trained PETs.

Original: In summary, our work has the following contributions: (1) Inspecting PETs from the perspective of optimal control, we re-emphasize the importance of the previously neglected running cost, and propose to use latent stochastic bridges as the regularizer to build the running cost function.
Revised: In summary, our work has the following contributions: (1) Guided by the perspective of optimal control for PETs, we design latent stochastic bridge regularizers on the intermediate states during the training of PETs.

Original: (2) We propose two methods to construct the latent space according to the two representations of stochastic bridges, offering the trade-off between efficiency and effectiveness.
Revised: (2) We propose two methods to construct the latent space according to the two representations of stochastic bridges, offering a trade-off between efficiency and effectiveness.

Original: The latent stochastic process regularizer is shown to be effective  across different PLMs, different PETs and different tasks.
Revised: (3) Our regularizers are shown to be effective and general across different PLMs, different PETs, and different tasks.

Original:  We show that there may be stochastic-bridge-like latent dynamics for well-trained PETs.
Revised: (4) We show that well-trained PETs without any regularization spontaneously exhibit stochastic-bridge-like latent dynamics.

Original:  B ACKGROUND 2. O PTIMAL C ONTROL P ERSPECTIVE OF PET S
Revised:  B ACKGROUND 2. D EFINITION AND M ATHEMATICAL N OTATIONS

Original: Consider a dataset D of text-to-text task, given a L -layer PLM with  vocabulary V and a  sample ( x , y ) ∼ D , x ∈ V N , y ∈ V  , where N is the length of x .[-
-]
Revised: Consider using a L -layer PLM with the vocabulary V to handle a text-to-text task D . For each sample ( x , y ) ∈ D , y ∈ V is the output token and x ∈ V N is the input token sequence 1 , where N is the length of x .[-
-]

Original: We denote the hidden states of the i -th PLM layer as h ( i ) =
Revised: With x as the input, each layer of the PLM will output a sequence of hidden states, and we denote the hidden states of the i -th PLM layer as h ( i ) =

Original: { h ( i ) j } Nj =1 , where h ( i ) j  is the state at the position j  . 
Revised: { h ( i ) j } Nj =1 , where h ( i ) j ∈ R d is the state at the position j of the i -th layer, and d is the model dimension. We denote the position where the model outputs the target y as o , i.e., the model should predict y with the hidden states h ( L ) o . 2. O PTIMAL C ONTROL P ERSPECTIVE OF PET S

Original: Conventionally, adapting  PLM to the dataset requires full-parameter fine-tuning, which is given as:
Revised: Conventionally, adapting the PLM to D requires full-parameter fine-tuning, which is given as:

Original: [Equation]
Revised: [Equation]

Original:  where θ is the PLM parameter, ∆ θ is the full-parameter update, L ( · , · ) is the loss function, R ( · ) is the regularization function, G ( i ) θ
Revised: , i = 1 , . . . , L, Embed ( x ) , i = 0 , (1) where θ is the PLM parameters, ∆ θ is the full-parameter update, L ( · , · ) is the loss function, R ( · ) is the regularization function, G ( i ) θ

Original: +∆ θ ( · ) is the forward propagation of the i -th PLM layer with the parameter θ + ∆ θ .
Revised: +∆ θ ( · ) is the forward propagation of the i -th PLM layer after updating the parameters, Embed transforms the input token sequence into input embeddings.

Original: As the size of θ  continues to increase, full-parameter fine-tuninggradually becomes impractical, and different PETs are proposed to mitigate the problem.
Revised: As | θ | continues to increase, full-parameter fine-tuning becomes impractical, and various PETs are proposed to mitigate this problem.

Original: Let ϕ =
Revised: Let ϕ =

Original: { ϕ ( i ) } Li = be PETs’ parameters.
Revised: { ϕ ( i ) } Li =0 be PETs’ parameters.

Original: Since | ϕ | ≪ | θ | , PETs can greatly reduce the tuning cost.

Original: More details of specific PETs can be found in Appendix A.

Original: Ding et al.
Revised: Ding et al.

Original: (2022) give a unified view of PETs from the perspective of optimal control, and Equation 1 can be re-written as
Revised: (2022) give a unified view of PETs from the perspective of optimal control, and Eq. 1 can be re-written as

Original: [Equation]
Revised: [Equation]

Original: where ˜ G ( i ) θ ( · , ϕ ( i ) ) represents the forward propagation of the i -th PLM layer intervened by PETs,  L T  isterminal the  cost and L R  is the running cost.
Revised: where ˜ G ( i ) θ ( · , ϕ ( i ) ) represents the forward propagation of the i -th PLM layer intervened by PETs, [ · ; · ] is the concatenation operation, L T ( · , · ) is the terminal cost and L R ( · ) is the running cost.

Revised: Since | ϕ | ≪ | θ | , PETs can greatly reduce the tuning cost (more details in Appendix A).

Original: Typically in classification and generation tasks, L T corresponds to the cross-entropy loss of the prediction, and L R can be seen as a regularizer on PETs’ parameters ϕ .
Revised: Typically for classification and generation tasks, L T corresponds to the cross-entropy loss of the prediction, and L R can be seen as a regularizer on PETs’ parameters ϕ .

Original: However, in the optimal control literature, L R depends on not only the control variables ϕ , but also the controlled intermediatehidden states { h ( i )  } Li =1 .
Revised: However, in the optimal control literature, L R depends on not only the control variables ϕ , but also the controlled intermediate states { h ( i ) o } L i =1 .

Original: In this paper, we re-emphasize the importance of considering intermediatehidden states to build L R (i.e. rewrite (cid:80)[-Li =
-]
Revised: In this paper, we show that including intermediate states to build L R (i.e., using (cid:80)[-Li =
-]

Original: L R{+Li =
+}
Revised: {+Li =
+}

Original: [Equation] ϕ [Equation] i )  (cid:1) to (cid:80) Li =

Revised: L R (cid:0) ϕ ( i ) , h ( i ) o (cid:1) instead of (cid:80) Li =

Original: L R (cid:0) ϕ ( i ), h ( i ) (cid:1) ), and show that exploiting latent stochastic bridges as L R improves all PETs we try in Section 4.
Revised: L R (cid:0) ϕ ( i ) (cid:1) ) not only makes the optimal control perspective of PETs more theoretically sound, but also empirically leads to better PETs.

Original: 2. D IFFUSION B RIDGES
Revised: 2. D IFFUSION B RIDGES

Original: A diffusion process X =
Revised: A diffusion process X =

Original: ( X t ) t ∈ [0 ,T ] is a continuous-time Markov process whose sample paths are almost surely continuous.
Revised: ( X t ) t ∈ [0 ,T ] is a continuous-time Markov process.

Original: For any t x < t y , the  process is equipped with a transition Probability Density Function (PDF) p ( t y , y | t x , x ) , which gives the probability density of reaching y atthe time t y given the history of reaching x atthe time t x .
Revised: For any t a < t b , the diffusion process is equipped with a transition Probability Density Function (PDF) p ( t b , b | t a , a ) , which gives the probability density of reaching b at time t b given the history of reaching a at time t a .

Original: A diffusion process is also the solution to an It ˆ o SDE d ˜ X t = µ ( t, ˜ X t ) dt + σ ( t, ˜ X t ) dB t , where B t is a standard Brownian motion, µ ( · , · ) is a drift function and σ ( · , · ) is a diffusion function.
Revised: A diffusion process is also the solution to an It ˆ o SDE d ˜ X t = µ ( t, ˜ X t ) dt + σ ( t, ˜ X t ) dB t , where B t is a standard Brownian motion, µ ( · , · ) is called drift function and σ ( · , · ) is called diffusion function.

Original: A diffusion bridge X T ; β is a diffusion process conditioning on the path observations of the two endpoints (0 , 0) and ( T, β ) , i.e., X T ; β= 0 and X T ; β T = β .
Revised: A diffusion bridge X T ; α,β is a diffusion process conditioning on the path observations of the two endpoints (0 , α ) and ( T, β ) , i.e., X T ; α,β= α and X T ; α,β T = β .

Original: By applying Bayes’ rule, we obtain the transition PDF for X T ; β as p T ; β ( t y , y | t x , x )
Revised: For simplicity, we assume α

Original: =
Revised: =

Original: p ( t y , y | t x , x, T, β ) = p ( T, β | t y , y ) p ( t y , y | t x , x ) /p ( T, β | t x , x ) .
Revised: in this work, and omit the superscript α .

Original: In this work, we consider two typical diffusion bridges, the Brownian bridge and the Ornstein-Uhlenbeck bridge (OU bridge).
Revised: We consider two typical diffusion bridges, the Brownian bridge and the Ornstein-Uhlenbeck bridge (OU bridge).

Original: We present hereonly the properties of the Brownian bridge and leave the properties of OU bridge to Appendix B.
Revised: We present here the properties of the Brownian bridge and leave the properties of OU bridge to Appendix B.

Original: Proposition 2.1 (The properties of the Brownian Bridge) .
Revised: Proposition 2.1 (The properties of the Brownian Bridge) .

Original: A Brownian bridge X T ; β with X T ; β=and X T ; β T = β is the solution to the following SDE:
Revised: A Brownian bridge X T ; β with X T ; β=and X T ; β T = β is the solution to the following SDE:

Original: [Equation]
Revised: [Equation]

Original: The transition PDF from X T ; β= 0 to X T ; β t y = y is given as
Revised: The transition PDF from X T ; β= 0 to X T ; β t b = b is given as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: Diffusion bridges and SDEs are battle-tested tools to describe the stochastic dynamics of complex systems in engineering (Sobczyk, 2013), finance (Wang & Sloan, 2011), biology (Horsthemke &

Revised: Lefever, 1984), etc.

Revised: Considering the dynamics of PLMs’ hidden states are necessarily complex, diffusion bridges and SDEs serve as ideal tools for us to model the dynamics.

Original:  L ATENT S TOCHASTIC B RIDGES R EGULARIZER 3. T HE O VERALL F RAMEWORK
Revised:  L ATENT S TOCHASTIC B RIDGES R EGULARIZER 3. T HE O VERALL F RAMEWORK

Original:  Since directly regularizing the intermediate states and constructing the running cost are non-trivial, we introduce a projection from  intermediate state space to a latent space, and leverage diffusion bridges as regularizers to calculate running cost.
Revised: Building latent dynamics in the latent space. Since directly regularizing the intermediate states and constructing the running cost are non-trivial, we introduce a projection from the intermediate state space to a latent space, and leverage diffusion bridges as regularizers to construct the running cost.

Original: Specifically,given a input sequence x and the position o where the model should output the target y , we define a r -dimensional latent space U ⊆ 
Revised: Specifically, we define a r -dimensional latent space U ⊆ R r

Original: R r ( r < d ) as well as a learnable mapping g γ : R d × R d → U , where  g γ projects the hidden state h ( i ) o and its context state ¯ h ( i )o into the latent space U at each layer of the PLM.
Revised: ( r < d ) and a learnable mapping g γ : R d × R d → U , where γ denotes the parameters. g γ projects the hidden state h ( i ) o and its context state ¯ h ( i ) into the latent space U at each layer of the PLM.

Revised: Since h ( i ) o is contextualized while latent bridges are not, introducing the dependency on ¯ h ( i ) can inform g γ about the context at the i -th layer and allow g γ to decontextualize the hidden states.

Original: We simply take  ¯ h ( i ) o as the average of all hidden states at the i -th layer.
Revised: We simply take the averaged states at the i -th layer ¯ h ( i ) = 1 N (cid:80) Nj =1 h ( i ) j as the context.

Original: We assume that the  states  in the latent space u
Revised: We define the latent states with discrete time in U as

Original: = { g γ ( h ( i ) o , ¯ h
Revised: [Equation] where t i +1 is the normalized layer index.

Original: ( i ) o ) }
Revised: We include the 0 -th layer (input layer) because some PETs (e.g., prompt tuning) act on the 0 -th layer.

Original:  Li =0 form a trajectory that is a sample path from X y with high probability.
Revised: We use t 0 = 0 , t L +2 = 1 represent the two endpoints. By using natural cubic spline knotted at { h ( i ) o } Li =0 to interpolate over [ − 1 , L + 1] , we further give a continuous representation of the states in the latent space U as [Equation]

Original: X y is the  latent diffusion bridge of the label word y .
Revised: Learning the mapping from hidden state space to latent space.

Original: Since adapting PLMs to downstream tasks can be seen as transferring  knowledge obtained from pretraining to downstream tasks, we argue that the latent dynamics of intermediate  states for generating the same token y  in  pre-training and downstream tasks should be similar.
Revised: Since adapting PLMs to downstream tasks can be seen as transferring the knowledge obtained from pre-training tasks to downstream tasks, we argue that the latent dynamics of intermediate hidden states for generating the same token y should be similar in both the pre-training and downstream tasks.

Original: We therefore train the mapping g γ on pre-training corpus where the backbone PLM was pre-trained, and then apply the  mapping tothe downstream tasks to encourage the  dynamics to be similar to that in pre-training.
Revised: Therefore, we train the mapping g γ on the corpus that is used to pre-train the backbone PLM, and then apply the learned mapping to downstream tasks to encourage the latent dynamics to be similar to that in pre-training.

Revised: Specifically, we assume that the states to generate the token y in the latent space U form a trajectory that is a path sampled from X 1; β y with high probability, where X 1; β y is the pre-determined diffusion bridge describing the latent dynamics to generate y , and β y is the tail endpoint of the diffusion bridge.

Revised: More details of X 1; β y will be discussed in Section 3.2.

Original: Specifically, on the corpus where the PLM is pre-trained, we freeze the PLM and use its hidden states  h  o  to learn g γ by maximizing the goodness of approximation for  u under the bridge X  y :
Revised: On the corpus where the PLM is pre-trained, we fix the PLM and use its hidden states { h ( i ) o } Li =1 to learn g γ by maximizing the goodness of approximation for latent states u under the bridge X 1; β y :

Original: [Equation]
Revised: [Equation]

Original: where  goodness-of-approx ( · , · ) is  a function that measures how likely { t i , g γ ( h ( i ) o ,
Revised: where u can be u D (Eq. 5) or u C (Eq. 6) depending on the fitting method, goodness-of-approx ( · , · ) is also a function depends on the choice of the fitting method, measuring how likely u is a sample trajectory of X 1; β y . In Section 3.3, we will define this function alongside the fitting methods.

Original: ¯ h ( i ) o ) } Li =0 is a sample trajectory of X y .
Revised: Regularizing PETs with latent dynamics.

Original: After learning g γ , by freezing γ , goodness-of-approx ( · , · ) is used as the running cost in Eq. 2 for PETs on downstream tasks.
Revised: After learning g γ with Eq. 7, we freeze γ and use the goodness-of-approx function as the running cost in Eq. 2 for PETs on downstream tasks.

Original: Finally, the overall objective for PETs becomes
Revised: The final objective for PETs becomes

Original: [Equation]
Revised: [Equation]

Original: h [Equation] (6) where the second term is the running cost and α is a hyper-parameter controlling the regularization strength.
Revised: where the second term is the running cost and α is a hyper-parameter controlling the regularization strength.

Original: We include the 0 -th layer (input layer), considering that some PETs (e.g., prompt tuning) only act on the 0 -th layer.

Original: By optimizing Equation 6, PETs need to ensure the final hidden state is capable of predicting y , and keep the latentintermediate hidden states at  position o conform to the diffusion bridge X  y .
Revised: By optimizing Eq. 8, PETs need to ensure the final hidden state is capable of predicting y , while keeping the latent states at the position o conform to the diffusion bridge X 1; β y .

Original: Note that introducing g γ as the regularizer does not increase the number of trainable parameters for PETs during  training  since γ is fixed.
Revised: Note that introducing g γ as the regularizer does not increase the number of trainable parameters for PETs during the training stage since γ is fixed.

Original: Also, g γ does not involve in inference.
Revised: Also, g γ does not involve in the inference stage.

Original: The determination of the two endpoints for the diffusion bridges { X y } y ∈ V will be introduced in Sec- tion 3.2.

Original: In Equation 5, different designs for the goodness-of-approx function leads to different learning method, and we will discuss in Section 3.3.

Original: 3. D ETERMINING E NDPOINTS FOR D IFFUSION B RIDGES
Revised: 3. D ETERMINING E NDPOINTS FOR D IFFUSION B RIDGES

Original: An intuitive approach to determine the endpoints for the diffusion bridges for each  token is to optimize the endpoints together with the mapping g γ .
Revised: An intuitive approach to determine the endpoints for the diffusion bridges for each target token is to optimize the endpoints together with the mapping g γ .

Original: However, optimizing endpoints and g γ jointly admits a trivial solution: endpointsfor diffusion bridges of all the tokens are set to be 0 ∈ R r, and g γ always outputs 0∈ R r .
Revised: However, optimizing endpoints and g γ jointly may admit a trivial solution: endpoints are both 0 ∈ R r and g γ always outputs 0 .

Original: Since 0 is always a point in the sample path of  a degenerated diffusion bridge with 0 being both ends, the goodness-of-approx can be high, but the diffusion bridges are meaningless.
Revised: Sinceis always a point in the sample path of such a degenerated diffusion bridge, the value of goodnessof-approximation function can be meaninglessly high.

Original: Although sophisticated constraints can be imposed, as the first work using diffusion bridges as regularizers for PETs, we simply pre-determine the endpoints and keep them fixed. We will show the effectiveness of the regularizer even when the endpoints are pre-determined and fixed.
Revised: Although sophisticated constraints can be imposed here, as the first work that uses diffusion bridges as regularizers, we simply pre-determine the endpoints and keep them fixed, and remain introducing constraints as our future work.

Original: Specifically, we apply PCA to the output word embedding matrix V ∈ R | V |× d of  PLM, obtaining a r -dimensional embedding matrix, and re-normalize each of the embedding to have  norm η .
Revised: Specifically, we apply principal component analysis (PCA) to the output token embedding matrix V ∈ R | V |× d of the PLM, obtaining a r -dimensional embedding matrix, and re-normalize each row to have a norm η .

Original: Let the resulting embedding matrix be ˜ V ∈ R | V |× r .
Revised: Let the resulting embedding matrix be β ∈ R | V |× r .

Original: We then use˜ V as the tails of the diffusion bridges, and 0 ∈ R r as the heads for all the bridges.
Revised: We then use 0 ∈ R r as the heads for all the bridges, and β as the tails of the diffusion bridges, i.e., the r -dimensional embedding of y in β is used as β y in X 1; β y .

Original: The intuition for selecting ˜ V as the tails is that the trajectories of the intermediatehidden states for similar words should be close.
Revised: The intuition for using β as the tails is that the trajectories of the intermediate states for similar target tokens should be close.

Original: In V , similar tokens are close, and ˜ V obtained by PCA can well preserve the token similarity after dimension reduction.
Revised: In V , similar tokens are close, and β obtained by PCA can well preserve the token similarity after reducing dimensions.

Original: 3. F ITTING THE M APPING g γ
Revised: 3. F ITTING THE M APPING g γ

Original: Without loss of generality, we use the Brownian bridges as the diffusion bridges to illustrate the fitting of g γ .
Revised: We use the Brownian bridge to illustrate the fitting of g γ .

Original: The following statements can be analogous tothe OU bridge.
Revised: It can be analogous to OU bridge easily.

Original: 3.3. M ETHOD 1: A PPROXIMATING T RANSITION P ROBABILITY D ENSITY F UNCTION
Revised: Method 1: Approximating the Transition PDF. Generalizing Eq.

Original: From Equation 4, the transition PDF of the Brownian bridge is Gaussian, which indicates X T ; β t ∼ N (µ t  , σ 2 t )  , where µ t = E (cid:104) X T ; β t (cid:105) =
Revised: 4 to high dimension, we can derive the transition PDF from (0 , 0 ) to ( t i +1 , g γ ( h ( i ) o , ¯ h ( i ) )) for X 1; β y : p 1; [Equation]

Original: t
Revised: y

Original: T β, σ 2 t =
Revised: [Equation]

Original: Var (cid:104) X T ; β t (cid:105) =
Revised: i

Original: t ( T − t ) .
Revised: [Equation] [Equation]

Original:  To make g γ approximate the transition PDF, we maximize the sum of log-probability of g γ ( h ( i ) o , ¯ h ( i ) o ) under the Gaussian distribution at time t i for all the layers:
Revised: where t i has the same definition as that in u D (Eq. 5). To make g γ approximate the transition PDF, we maximize the sum of log-probability of u D under the Brownian bridge X 1; β y : goodness-of-

Original: [Equation]
Revised: [Equation]

Original: Here, g γ can be seen as a mapping from the hidden state space to  latent space by predicting the expectation of the Brownian bridge X  y at  t i given the hidden states and context information.
Revised: Here, g γ can be seen as a mapping from the hidden state space to the latent space by predicting the expectation of the Brownian bridge X 1; β y at { t i +1 } Li =0 .

Original: 3.3. M ETHOD 2: A PPROXIMATING THE SDE
Revised: Method 2: Approximating the SDE.

Original: Since the Brownian bridge is the solution to the SDE in Equation 3, we can let g γ approximate the SDE.
Revised: Since the Brownian bridge is the solution to the SDE in Eq. 3, we can let g γ approximate the SDE.

Original: However, Brownian bridge is a continuous-time stochastic process, while we only have L + 1 discrete observations on the path, we thus use natural cubic splines knotted at { h ( i ) o } Li =0 to interpolate h o at any given time t , and denote it as h to .
Revised: Solving the SDE requires continuous latent states, while we only have L + 1 discrete observations, we thus use the continuous representation u C introduced in Eq. 6. Generalizing Eq.

Original: We define the SDE approximated by g γ  as:
Revised: 3 to high dimension, the SDE approximated by g γ can be defined as:

Original: [Equation]
Revised: [Equation]

Revised: ) where x is the same as that in Eq. 6, B : [0 , 1] → R r is a standard r -dimensional Brownian motion.

Original: Here, we additionally introduce the dependency on t for g γ , since time information is shown to be important in previous neural differential equation works (Zhang et al., 2020; Dupont et al., 2019).
Revised: Here, we additionally introduce the dependency on t for g γ , since time information is shown to be important in previous neural differential equation works (Zhang et al., 2020; Dupont et al., 2019).

Original: Following Li et al.
Revised: Following Li et al.

Original: Xu et al.

Original: (2022), when two SDEs share the same diffusion function, the KL divergence between the probability measures induced by the two SDEs is finite.
Revised: (2020), when two SDEs share the same diffusion function, the KL divergence between the probability measures induced by the two SDEs is finite.

Original: This is the case here since the diffusion function σ ≡ 1 for Equation 8 and Equation 3.
Revised: Since the diffusion function σ ≡ I for Eq. 11 and the multi-dimensional generalization of Eq.

Original: The KL divergence between the probability measures µ
Revised: 3, the KL divergence between the probability measures µ

Original: Y of Equation 8 and µ X of Equation 3 canthus be estimated by:
Revised: Y of Eq. 11 and µ X of generalized Eq. 3 can be estimated by:

Original: [Equation]
Revised: [Equation]

Original: , where [Equation]
Revised: where µ ( · , · ) is the drift function of the pre-determined Brownian bridge X 1; β y .

Original: Wecan thus use the KL divergence as the goodness-of-approx to optimize the mapping g γ .
Revised: We use the KL divergence as the goodness-of-approximation function to optimize the mapping g γ .

Original: Here, g γ can be seen as a mapping from the hidden state space to  latent  space by approximating the drift vector field of the underlying Brownian bridge given the hidden states, context and time.
Revised: Here, g γ can be seen as a mapping from the hidden state space to the latent state space by approximating the drift vector field of the underlying Brownian bridge X 1; β y .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To verify the effectiveness and generality of the latent stochastic bridges regularizer, we conduct experiments on (1)PLMs of different sizes: BERT large (340M) (Devlin et al., 2019) and Deberta xlarge (750M) (He et al., 2021); (2) different PETs: Prompt tuning, LoRA, BitFit and Adapter; (3) different diffusion bridges: Brownian bridge and OU bridge.
Revised: To verify the effectiveness and generality of the regularizers built on stochastic bridges, we conduct experiments on (1) different PLMs: BERT large (340M) (Devlin et al., 2019) and Deberta xlarge (750M) (He et al., 2021); (2) different PETs: Prompt tuning, LoRA, BitFit and Adapter; (3) different diffusion bridges: Brownian bridge and OU bridge.

Original: We show that the regularizers effectively improve the performance on GLUE (Wang et al., 2019) under both full-set and few-shot setting.
Revised: We show that the regularizers effectively improve the performance on GLUE (Wang et al., 2019) under both full-set and few-shot settings.

Original: 4. E XPERIMENTAL S ETUPS
Revised: 4. E XPERIMENTAL S ETUPS

Original: Datasets.
Revised: Datasets.

Original: Since both BERT large and Deberta xlarge include Wikipedia and BookCorpus (Zhu et al., 2015) in their pre-training corpus, we  use these two corpora to train g γ .
Revised: Since both BERT large and Deberta xlarge use Wikipedia and BookCorpus (Zhu et al., 2015) for pre-training, we thus use these two corpora to train g γ .

Original: We reportthe average performance and the standard deviation on development set over 3 different runs. We report accuracy for MNLI, SST-2, QNLI, RTE, F1 for MRPC and QQP, and Matthews correlation for CoLA. 
Revised: We report accuracy for MNLI, SST-2, QNLI, and RTE; F1 for MRPC and QQP; and Matthews correlation for CoLA. We report the average performance and the standard deviation on the development set over 3 different runs.

Original: All the tasks are re-casted to text-to-text format, the PLMs are required to output the label word for each example (e.g., negative or positive for SST-2).
Revised: We append a special token [MASK] to each sequence, and require the PLM to output the label word at [MASK] (e.g., negative or positive for SST-2).

Original: We exclude STS-B since it is a regression task.
Revised: We exclude STS-B for it is a regression task.

Original: Models and PETs.
Revised: Models and PETs.

Original: We use the checkpoint release by Megatron (Shoeybi et al., 2019) for BERT large , and the official v1 checkpoint for Deberta xlarge .
Revised: We use the checkpoint released by Megatron (Shoeybi et al., 2019) for BERT large , and the official v1 checkpoint for Deberta xlarge .

Original: We use a simple three-layer MLP as the g γ , and we set T = 1 in all the experiments.
Revised: We use a simple three-layer MLP to build g γ .

Original: For Prompt tuning, we use a soft prompt of length 20, and append it to the end of each sequence.
Revised: For Prompt tuning, we use a soft prompt of length 20, and append it to the end of each [Table] +B ROWN PDF [Table]

Original: For LoRA, we apply it to query and value layer in attention.
Revised: +B [Table] A DAPTER [Table]

Revised: +B ROWN PDF

Original: For
Revised: [Table] +B [Table] sequence.

Revised: For LoRA, we apply it to the query and value of attention modules.

Original:  Adapter, we apply it to the output of attention and feed-forward modules.
Revised: For Adapter, we apply it to the output of attention and feed-forward modules.

Original: For BitFit, we tune all the bias terms in linear layers and layer normalization modules.
Revised: For BitFit, we tune all the bias terms in linear layers and layer normalization modules.

Original: For all PETs, we append a special token [MASK] to the end of each sample, and use its output as the prediction.

Original: Hereafter, we use PDF regularizer to refer to using g γ fitted by approximating  transition PDF, and SDE regularizer to refers to using g γ fitted by approximating the SDE, vanilla x to refer to  PET x without regularizer. [Table] + [Table]
Revised: Hereafter, we use PDF regularizer to refer to using g γ fitted by approximating the transition PDF, and SDE regularizer to refer to using g γ fitted by approximating the SDE, vanilla x to refer to the PET x without using regularizers.

Original: Few-shot Experiments.
Revised: Few-shot Experiments.

Original: We randomly sample 2 × k examples from the original training set D train for each class.
Revised: We randomly sample 2 × k examples from the original training set D train for each class.

Original: The sampling is performed 5 times with different seeds to form 5 training sets  { ˜ D ( i ) train } 5 i = and development sets { ˜ D ( i ) dev } 5 i =1 with each being k -shot.
Revised: The sampling is performed 5 times with different seeds to form 5 training sets and development sets { ˜ D ( i ) train , ˜ D ( i ) dev } 5 i =1 with each being k -shot.

Original: Each time we train  onone of the training set ˜ D ( i ) train ,  select the best model on ˜ D
Revised: Each time we train PETs on ˜ D ( i ) train , we select the best model on ˜ D

Original: ( i ) dev , and report its performance on  original development
Revised: ( i ) dev , and report its performance on the original development

Original: set D dev .
Revised: set D dev .

Original: Hyper-parameters.
Revised: Hyper-parameters.

Original: We list hyper-parameters in
Revised: We list hyper-parameters in

Original: Appendix E. We mainly focus on the difference in performance between vanilla PETs and regularized PETs.
Revised: Appendix E. We mainly focus on the difference in performance between vanilla PETs and regularized PETs.

Original: Therefore we directly set the hyperparameters to reasonable values and do not perform much hyper-parameter search.
Revised: Therefore, we directly set the hyperparameters to reasonable values and do not perform much hyper-parameter search.

Original: But we ensure the hyper-parameters for vanilla PETs and regularized PETs are the same for a fair comparison.
Revised: But we ensure the hyper-parameters for vanilla PETs and regularized PETs are the same for a fair comparison.

Original: 4. F ULL - SET R ESULTS
Revised: 4. F ULL - SET R ESULTS

Original: The experimental results for BERT large are reported in Table 1.
Revised: The experimental results for BERT large are reported in Table 1.

Original: Due to space limitation, see Tablefor the complete results including OU bridge regularizers.
Revised: Due to space limitation, see Tablefor the complete results including OU bridge regularizers.

Original: The first line of each block in the table is the performance of vanilla PETs, and the rest of the lines are the performances of the regularized
Revised: The first line of each block in the table is the performance of vanilla PETs, and the rest of the lines are the performances of the regularized

Original: PETs.
Revised: PETs.

Original: We also conduct the same experiments for Deberta xlarge and place the results in Appendix C.
Revised: We also conduct the same experiments for Deberta xlarge and place the results in Appendix C.

Original: In general, both Brownian and OU bridge, and both PDF and SDE regularizers are able to effectively improve the performance of PETs, showing the effectiveness of our proposed regularizer.
Revised: In general, both Brownian and OU bridges, and both PDF and SDE regularizers are able to effectively improve the performance of PETs, showing the effectiveness of our proposed regularizers.

Original: In particular, for Prompt tuning, the SDE regularizer with both diffusion bridges yield an increase on the average performance of more than 2 points.
Revised: Particularly, for Prompt tuning, the SDE regularizer with both diffusion bridges yield an increase on the average performance of more than 2 points.

Original: We assume that it is because Prompt tuning has far less trainable parameters than other PETs, and it only acts at the input layer, which is far from the supervision signals from the terminal cost L T .
Revised: We assume that it is because Prompt tuning has far less trainable parameters than other PETs, and it only acts at the input layer, which is far from the supervision signals of the terminal cost L T .

Original: Therefore when provided with the regularization on the hidden states, the prompts receive more guidance and eventually reaching a better local optimal.
Revised: Therefore, when provided with the regularization on the hidden states, the prompts receive more guidance and eventually reaching a better local optimal.

Original: Overall, the two diffusion bridges in our experiments do not show much difference.
Revised: Overall, the two diffusion bridges in our experiments do not show much difference.

Original: As for thefitting methods, SDE regularizer is generally more effective, especially for Prompt tuning where the number of trainable parameters is restricted.
Revised: As for the two fitting methods, SDE regularizer is generally more effective, especially for Prompt tuning where the number of trainable parameters is restricted.

Original: However, we also observe that SDE regularizer is about 3 times slower than PDF regularizer, which brings the trade-off between performance and efficiency.
Revised: However, we also observe that SDE regularizer is about 3 times slower than PDF regularizer, which brings the trade-off between performance and efficiency.

Original: One can expect a better performance by leveraging more sophisticated underlying stochastic bridges, exploring more reasonable endpoints for bridges and designing better mapping g γ .
Revised: One can expect a better performance by leveraging more sophisticated underlying stochastic bridges, exploring more reasonable endpoints for bridges and designing better mapping g γ .

Original: As the first work using latent stochastic bridges as regularizers, we  consider the most straightforward cases and aim to show the great potential of the approach and the importance of regularizing hidden states.
Revised: As the first work using latent stochastic bridges as regularizers, we mainly consider the most straightforward cases and aim to show the great potential of the approach and the importance of regularizing hidden states.

Original: 4. F EW - SHOT R ESULTS
Revised: 4. F EW - SHOT R ESULTS

Original: We observe that in Table 1, the improvements brought by our regularizer are more substantial on small datasets MRPC, CoLA and RTE.
Revised: We observe that in Table 1, the improvements brought by our regularizers are more substantial on small datasets MRPC, CoLA and RTE.

Original: We assume that this is because in rich-resource datasets, the abundant data itself has provided enough information for the model to learn high quality PETs,
Revised: We assume that this is because in rich-resource datasets, [Table] DAPTER

Original: [Table]
Revised: [Table]

Original:  while in low-resource datasets, the data is insufficient and the regularizer offers helpful additional  supervision.
Revised: the abundant data itself has provided enough information to learn high quality PETs, while in lowresource datasets, the data is insufficient and the regularizer can offer additional helpful supervision.

Original: To validate this, we conduct the experiments under the few-shot setting on GLUE.
Revised: To validate this, we conduct the experiments under the few-shot setting on GLUE.

Original: The 16-shot results are shown in Table 2, and the results for  OU bridge, results for 4-, 8- and 32shot and results for Deberta xlarge are placed in Appendix C. On allthe PETs, the SDE regularizer yields an improvement of more than 3 points.
Revised: The 16-shot results are shown in Table 2, and the results for the OU bridge, results for 4-, 8- and 32-shot and results for Deberta xlarge are placed in Appendix C. For all PETs, the SDE regularizer yields an improvement of more than 3 points.

Original: Particularly,  SDE regularizer on LoRA brings an improvement of 5.2.
Revised: Particularly, the SDE regularizer on LoRA brings an improvement of 5.2 points.

Original: By applying the regularizer in the few-shot setting, there is now a substantial boost on what was originally a rich-resource dataset, such as MNLI, QQP and QNLI.
Revised: By applying regularizers under the few-shot setting, there is now a substantial boost on what was originally a rich-resource dataset, such as MNLI, QQP and QNLI.

Original: The PDF regularizer also gives modest improvements.
Revised: The PDF regularizer also gives modest improvements.

Original: Although slightly inferior to the SDE regularizer, it is still satisfying considering that  PDF regularizer brings such a performance improvement with little computational cost introduced.
Revised: Although slightly inferior to the SDE regularizer, it is still satisfying, considering that the PDF regularizer brings such a performance improvement with little computational cost introduced.

Original: We additionally observe that the improvement is more significant on Deberta xlarge in Table 9, demonstrating the potential of our regularizer on larger models.
Revised: We additionally observe that the improvement is more significant on Deberta xlarge in Table 9, demonstrating the potential of our regularizers on larger models.

Original:  A NALYSES
Revised:  A NALYSES

Original: In order to have a better understanding of the role played by our regularizers, we analyze the hidden states of the PETs with and without the regularizer in this section.
Revised: In order to have a better understanding of the role played by our regularizers, we analyze the hidden states of the PETs with and without regularizers in this section.

Original: We choose Prompt tuning as a representative in the analyses.
Revised: We choose Prompt tuning as a representative in the analyses.

Original: By varying the hyper-parameter α in Equation 6, we show that as the strength of the regularization gets stronger, the clusters of hidden states corresponding to different label word becomes compacter and more distinguishable.
Revised: By varying the hyper-parameter α in Eq. 8, we show that as the strength of the regularization gets stronger, the clusters of hidden states corresponding to different label tokens become compacter and more distinguishable.

Original: Also, we show that the hidden states of vanilla PETs spontaneously approach the latent bridges in  latent space without knowing the bridges, which may indicate that there exists intrinsically diffusion-bridge-like latent dynamics for PETs.
Revised: Also, we show that the hidden states of vanilla PETs spontaneously approach the latent bridges in the latent space without knowing the bridges, indicating that there may exist intrinsically diffusion-bridge-like latent dynamics for PETs.

Original: 5. T HE R EGULARIZER W IDENS THE D ISTANCES BETWEEN L ABEL C LUSTERS
Revised: 5. T HE R EGULARIZER W IDENS THE D ISTANCES BETWEEN L ABEL C LUSTERS

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  We use the different prompts obtained with or without the regularizer on  full-set GLUE, and record the intermediate hidden states at the position of prediction as { h ( i ) [MASK] } Li =1 .
Revised: [Table] [Table] [Table] We use the different prompts obtained with or without regularizers on the full-set GLUE, and record the intermediate hidden states at the position of prediction as { h ( i ) [MASK] } Li =1 .

Original: We vary the regularization strength by adjusting the coefficient α in Equation 6 to inspect the impact of  regularization strength on the hidden states.
Revised: We vary the regularization strength by adjusting the coefficient α in Eq. 8 to inspect the impact of the regularization strength on the hidden states.

Original: Note that when α = 0 , it degenerates to the vanilla PET.
Revised: Note that when α = 0 , it degenerates to the vanilla PET.

Original: We randomly sample 100 samples for each label in MNLI, and use UMAP (McInnes et al., 2018) to reduce the dimension of the last layer’s hidden states of different prompts, and plot them in Figure 2.
Revised: We randomly sample 100 samples for each label in MNLI, and use UMAP (McInnes et al., 2018) to reduce the dimension of the last layer’s hidden states of different prompts, and plot them in Figure 2.

Original: It shows clearly that for both regularizers, as thestrength of the regularization  becomes stronger, the hidden states of the last layer becomes more distinguishable among different labels.
Revised: It shows clearly that for both regularizers, as the regularization strength becomes stronger, the hidden states of the last layer becomes more distinguishable among different labels.

Original: By looking at the axes of these plots, we find that the distance between the clusters generally increases when 
Revised: By looking at the axes of these plots, we find that the distance between the clusters generally increases when the reg-

Original: Labelnoyesmaybe (a) w/o regularizer      

Revised: Labelnoyesmaybe (a) w/o regularizer

Original: Labelnoyesmaybe (b) SDE α = 0 .
Revised: Labelnoyesmaybe (b) SDE α = 0 .

Original: Labelnoyesmaybe (c) SDE α = 0 . Labelnoyesmaybe (d) SDE α = 0 .
Revised: Labelnoyesmaybe (c) SDE α = 0 . Labelnoyesmaybe (d) SDE α = 0 .

Original: Labelnoyesmaybe (e) PDF α = 0 .
Revised: Labelnoyesmaybe (e) PDF α = 0 .

Original: Labelnoyesmaybe (f) PDF α = 0 .4 2 0 2 4 6 8 

Revised: Labelnoyesmaybe (f) PDF α = 0 .4 2 0 2 4 6 8

Original: Labelnoyesmaybe (g) PDF α = 1 .
Revised: Labelnoyesmaybe (g) PDF α = 1 .

Original: the regularization strength is increased.

Original: We also notice that SDE regularizer better helps separate the last layer’s hidden states by substantially enlarging the distance between the centroids of different labels, which could be one of the reasons why SDE regularizer has a better effectiveness in almost all our experiments.

Original: We also calculate the Pearson’s correlation between the regularization strength α and the average distance between the centroids of different clusters.
Revised: We also calculate the Pearson’s correlation between the regularization strength α and the average distance between the centroids of different clusters.

Original: The results are shown in Table 3.
Revised: The results are shown in Table 3.

Original: On all the datasets, the regularization strength α has a positive correlation to the average centroid distance, and on most of the datasets, the correlations are significant in the sense that the p -value < .
Revised: On all the datasets, the regularization strength α has a positive correlation to the average centroid distance, and on most of the datasets, the correlations are significant in the sense that the p -value < .

Original: This indicates that as the regularization strength becomes stronger, the centroids of different label clusters become more distant, which is a desired effect because the regularizer encourages the hidden states for different label words to conform to different latent diffusion bridges.
Revised: This indicates that as the regularization strength becomes stronger, the centroids of different label clusters become more distant, which is a desired effect because the regularizer encourages the hidden states for different label tokens to conform to different latent diffusion bridges.

Original: 5. H IDDEN S TATES A PPROACH S PONTANEOUSLY TO THE L ATENT B RIDGES
Revised: 5. H IDDEN S TATES A PPROACH S PONTANEOUSLY TO THE L ATENT B RIDGES

Original: [Table]
Revised: [Table]

Original: An interesting phenomenon we observe is that the vanilla PETs’ intermediate hidden states spontaneously approach our latent bridges when they are projected by our mapping g γ .
Revised: An interesting phenomenon we observe is that the vanilla PETs’ intermediate hidden states spontaneously approach our latent bridges when they are projected by our mapping g γ .

Original: That is, applying our mapping g γ to the hidden states of vanilla PETs we find that when the performance of vanilla PETs becomes better, the average distance from{ g γ ( { h ( i ) o , ¯ h ( i ) } )} to our latent bridge gets closer.
Revised: That is, applying our mapping g γ to the hidden states of vanilla PETs, we find that when the performance of vanilla PETs becomes better, the average distance from g γ ( { h ( · ) o , ¯ h ( · ) } ) to our latent bridge gets closer.

Original: Here, similar to Wang et al.
Revised: Here, similar to Wang et al.

Original: (2022), we define the distance from g γ ( h ( i ) o , ¯ h ( i ) ) to its corresponding latent bridge X y using Equation 7 without the constant.
Revised: (2022), we define the distance from g γ ( h ( · ) o , ¯ h ( · ) ) to its corresponding latent bridge X y using Eq. 10 without the constant.

Original: Note that the vanilla PETs have no access tothe g γ and the latent bridges during the training, and g γ also has no access to the PETs during its fitting process.
Revised: Note that the vanilla PETs have no access to g γ and the latent bridges during the training process, and g γ also has no access to the PETs during its fitting process.

Original: We demonstrate the above observation by conducting analyses in few-shot scenarios and reporting the correlation between (1) the number of shots and the average distance from latent hidden states to latent bridges in Table 4 (2) the performance and the average distance from latent hidden states to latent bridges in Table 5.
Revised: We demonstrate the above observation by conducting analyses in few-shot scenarios and reporting the correlation between (1) the number of shots and the average distance from latent hidden states to latent bridges in Table 4 (2) the performance and the average distance from latent hidden states to latent bridges in Table 5.

Original: Specifically, we report Kendall’s rank correlation 2 for (1), and Pearson’s correlation for (2).
Revised: Specifically, we report Kendall’s rank correlation 2 for (1), and Pearson’s correlation for (2).

Original: See Appendix D for the detailed setup for the calculation of the correlations.
Revised: See Appendix D for the detailed setup for the calculation of the correlations.

Original: [Table]
Revised: [Table]

Original: From Table 4, the number of shots has a negative correlation to the distance, and the correlation is significant on 4 out of 6 datasets.
Revised: From Table 4, the number of shots has a negative correlation to the distance, and the correlation is significant on 4 out of 6 datasets.

Original: This indicates that as the amount of available data increases for vanilla
Revised: This indicates that as the amount of available data increases for vanilla

Original: PETs, its intermediate hidden states in latent space spontaneously approach latent bridges even without knowing the mapping g γ and the bridges.
Revised: PETs, its intermediate hidden states in latent space spontaneously approach latent bridges even without knowing the mapping g γ and the bridges.

Original: Additionally, the results in Table 5 show the negative correlation between the performance of vanilla PETs and the distance to the latent bridges, and it is significant on 3 out of 6 datasets.
Revised: Additionally, the results in Table 5 show the negative correlation between the performance of vanilla PETs and the distance to the latent bridges, and it is significant on 3 out of 6 datasets.

Original: Altogether, the two findings on correlation show that as the PETs learn to solve the problem, their intermediate hidden states spontaneously approach our latent bridge in the latent space defined by g γ .
Revised: Altogether, the two findings on correlation show that as the PETs learn to solve the problem, their intermediate hidden states spontaneously approach our latent bridges in the latent space projected by g γ .

Original: This implies that there exists intrinsically diffusion-bridge-like latent dynamics for PETs, and also justifies our use of latent diffusion bridges as regularizers.
Revised: This implies that there exists intrinsically diffusion-bridge-like latent dynamics for PETs, and also justifies our use of latent diffusion bridges as regularizers.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Recent years have witnessed the success of PLMs (Raffel et al., 2020; Brown et al., 2020), which acquires rich knowledge from unlabeled data in a self-supervised manner and can be adapted to specific tasks by tuning their parameters.
Revised: Recent years have witnessed the success of PLMs (Raffel et al., 2020; Brown et al., 2020), which acquires rich knowledge from unlabeled data in a self-supervised manner and can be adapted to specific tasks by tuning their parameters.

Original: Despite the success of PLMs, as the size of PLMs continues to grow, it becomes increasingly impractical to perform full-parameter fine-tuning for downstream tasks.
Revised: Despite the success of PLMs, as the size of PLMs continues to grow, it becomes increasingly impractical to perform full-parameter fine-tuning for downstream tasks.

Original: Therefore, various recent efforts have been devoted to PETs, aiming to freeze PLMs and only tune a few additional parameters for task adaptation, such as Prompt tuning (Lester et al., 2021) that prepends tunable tokens per task to the input text, Adapter (Houlsby et al., 2019) that inserts small modules into each layer, BitFit (Zaken et al., 2022) that tunes only the bias terms, and LoRA (Hu et al., 2022) that decomposes the weight updates into low-rank matrices.
Revised: Therefore, various recent efforts have been devoted to PETs, aiming to freeze PLMs and only tune a few additional parameters for task adaptation, such as Prompt tuning (Lester et al., 2021) that prepends tunable tokens per task to the input text, Adapter (Houlsby et al., 2019) that inserts small modules into each layer, BitFit (Zaken et al., 2022) that tunes only the bias terms, and LoRA (Hu et al., 2022) that decomposes the weight updates into low-rank matrices.

Original: In this paper, based on the theoretical grounding of PETs on optimal control (Yang & Liu, 2022; Ding et al., 2022), we develop stochastic bridges as the regularizer for intermediate hidden states and introduce regularized PETs, showing effectiveness and generality on different PLMs and tasks.
Revised: In this paper, based on the theoretical grounding of PETs on optimal control (Yang & Liu, 2022; Ding et al., 2022), we develop stochastic bridges as the regularizer for intermediate hidden states and introduce regularized PETs, showing effectiveness and generality on different PLMs and tasks.

Original: Since we adopt latent stochastic bridges as the regularizer, our work closely relates to continuoustime neural ODEs (Chen et al., 2018; Rubanova et al., 2019) and neural SDEs (Li et al., 2019;
Revised: Since we adopt latent stochastic bridges as the regularizer, our work closely relates to continuoustime neural ODEs (Chen et al., 2018; Rubanova et al., 2019) and neural SDEs (Li et al., 2019;

Original: Kidger et al., 2021).
Revised: Kidger et al., 2021).

Original: Continuous-time neural ODEs and SDEs model the dynamics of the hidden states with ODEs and SDEs parameterized by neural network respectively
Revised: Continuous-time neural ODEs and SDEs model the dynamics of the hidden states with ODEs and SDEs parameterized by neural network respectively

Original: For example Chen et al.
Revised: For example Chen et al.

Original: (2018) show the resemblance between ODE and ResNet (He et al., 2016), and propose to learn the neural ODEs as the generalization of ResNet in continuous time.
Revised: (2018) show the resemblance between ODE and ResNet (He et al., 2016), and propose to learn the neural ODEs as the generalization of ResNet in continuous time.

Original: Inspired by these works, we use
Revised: Inspired by these works, we use

Original: SDEs to represent the latent dynamics of PETs in the latent space.
Revised: SDEs to represent the latent dynamics of PETs in the latent space.

Original: Our work differs from these work in that we focus on using neural SDEs as regularizers for intermediate hidden states, rather than feature extractors on downstream tasks.
Revised: Our work differs from these work in that we focus on using neural SDEs as regularizers for intermediate hidden states, rather than feature extractors on downstream tasks.

Original: We also notice that Wang et al.
Revised: We also notice that Wang et al.

Original: (2022) explore the use of Brownian bridge in PLMs.
Revised: (2022) explore the use of Brownian bridge in PLMs.

Original: However, they use Brownian bridge to regularize the text dynamics across time, while we use Brownian bridge to regularize the dynamics of intermediate hidden states across model’s layers.
Revised: However, they use Brownian bridge to regularize the text dynamics across time, while we use Brownian bridge to regularize the dynamics of intermediate hidden states across model’s layers.

Original: We additionally show that diffusion bridges other than Brownian bridge can be easily applied in our regularizer.
Revised: We additionally show that diffusion bridges other than Brownian bridge can be easily applied in our regularizer.

Original: As far as we know, our work is the first to show the diffusion-bridge-like dynamics for hidden states across layers and use diffusion bridges as the regularizer for intermediate hidden states.
Revised: As far as we know, our work is the first to show the diffusion-bridge-like dynamics for hidden states across layers and use diffusion bridges as the regularizer for intermediate hidden states.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we start from the optimal control perspective of PETs and we notice that the existing
Revised: In this work, we start from the optimal control perspective of PETs and we notice that the existing

Original: PETs lack a running cost that regularizes the intermediate hidden states.
Revised: PETs lack a running cost that regularizes the intermediate hidden states.

Original: Therefore, we propose to use stochastic bridges in a latent space as the regularizers for
Revised: Therefore, we propose to use stochastic bridges in a latent space as the regularizers for

Original: PETs.
Revised: PETs.

Original: Experimental results on different models, different tasks and different PETs show that the proposed regularizer effectively improve the PETs’ performance.
Revised: Experimental results on different models, different tasks and different PETs show that the proposed regularizer effectively improve the PETs’ performance.

Original: Our analyses further show that when the PETs are trained without the regularizer, the hidden states spontaneously approach our diffusion bridges, indicating that there exists intrinsically diffusion-bridge-like dynamics for PETs.
Revised: Our analyses further show that when the PETs are trained without the regularizer, the hidden states spontaneously approach our diffusion bridges, indicating that there exists intrinsically diffusion-bridge-like dynamics for PETs.

Original: As the first work using stochastic bridges as regularizers, we show its effectiveness and generality even with simple diffusion bridges.
Revised: As the first work using stochastic bridges as regularizers, we show its effectiveness and generality even with simple diffusion bridges.

Original: We believe it is a promising direction and we are excited to see more future work.
Revised: We believe it is a promising direction and we are excited to see more future work.

Original: A B ACKGROUND FOR P ARAMETER -E FFICIENT T UNING M ETHODS
Revised: A B ACKGROUND FOR P ARAMETER -E FFICIENT T UNING M ETHODS

Original: The large number of parameters in the PLMs makes fine-tuning impractical, therefore different
Revised: The large number of parameters in the PLMs makes fine-tuning impractical, therefore different

Original: PETs are proposed to mitigate the problem.
Revised: PETs are proposed to mitigate the problem.

Original: The current PETs can be categorized into three groups: addition-based, specification-based and reparameterization-based (Ding et al., 2022).
Revised: The current PETs can be categorized into three groups: addition-based, specification-based and reparameterization-based (Ding et al., 2022).

Original: To verify the generality of our method, we include one or two PETs from each category in this work, and we give a brief review to these PETs.
Revised: To verify the generality of our method, we include one or two PETs from each category in this work, and we give a brief review to these PETs.

Original: Prompt Tuning is an addition-based PET.
Revised: Prompt Tuning is an addition-based PET.

Original: It prepends or appends trainable virtual tokens P ∈ R m × d to each sequence x ∈ R n × d to form a new input sequence [ P ; x ] or [ x ; P ] ∈ R ( n + m ) × d , where n, m are length of original sequence and virtual tokens respectively, d is the embedding dimension.
Revised: It prepends or appends trainable virtual tokens P ∈ R m × d to each sequence x ∈ R n × d to form a new input sequence [ P ; x ] or [ x ; P ] ∈ R ( n + m ) × d , where n, m are length of original sequence and virtual tokens respectively, d is the embedding dimension.

Original: The virtual tokens P can be either continuous Lester et al.
Revised: The virtual tokens P can be either continuous Lester et al.

Original: or be restricted to be embeddings of discrete tokens in vocabulary (Gao et al., 2021).
Revised: or be restricted to be embeddings of discrete tokens in vocabulary (Gao et al., 2021).

Original: Adapter  is an addition-based PET.
Revised: Adapter (Houlsby et al., 2019) is an addition-based PET.

Original: It inserts two-layer MLPs after the attention module and feedforward module at each layer.
Revised: It inserts two-layer MLPs after the attention module and feed-forward module at each layer.

Original: Denote h ∈ R d as the input of Adapter, r as the intermediate dimension of Adapter’s
Revised: Denote h ∈ R d as the input of Adapter, r as the intermediate dimension of Adapter’s

Original: MLP, W d ∈ R r × d , W u ∈ R d × r as the down-projection and up-projection of Adapter, and σ as the activation function.
Revised: MLP, W d ∈ R r × d , W u ∈ R d × r as the down-projection and up-projection of Adapter, and σ as the activation function.

Original: Then the computation of Adapter can be formulated as
Revised: Then the computation of Adapter can be formulated as

Original: [Equation]
Revised: [Equation]

Original: BitFit  is a specification-based PET.
Revised: BitFit (Zaken et al., 2022) is a specification-based PET.

Original: It specifies the bias terms in layer normalization modules and linear transformation modules as trainable.
Revised: It specifies the bias terms in layer normalization modules and linear transformation modules as trainable.

Original: LoRA  is a reparameterization-based PET.
Revised: LoRA (Hu et al., 2022) is a reparameterization-based PET.

Original: It assumes that when training the model, the updates ∆ W for model’s pre-trained parameters
Revised: It assumes that when training the model, the updates ∆ W for model’s pre-trained parameters

Original: W ∈ R d × k are low-rank, and thus reparameterize the ∆ W of each matrix in attention module with a low-rank decomposition ∆ W = BA , where B ∈ R d × r , A ∈ R r × k .
Revised: W ∈ R d × k are low-rank, and thus reparameterize the ∆ W of each matrix in attention module with a low-rank decomposition ∆ W = BA , where B ∈ R d × r , A ∈ R r × k .

Original: For a forward pass h = W x , the computation of LoRA can be written as
Revised: For a forward pass h = W x , the computation of LoRA can be written as

Original: [Equation]
Revised: [Equation]

Original: B P ROPERTIES FOR O RNSTEIN -U HLENBECK B RIDGE
Revised: B P ROPERTIES FOR O RNSTEIN -U HLENBECK B RIDGE

Original: Proposition B.1 (Properties of Ornstein-Uhlenbeck Bridge) .
Revised: Proposition B.1 (Properties of Ornstein-Uhlenbeck Bridge) .

Original: A Ornstein-Uhlenbeck X T ; β pinned at X T ; β= 0 and X T ; β T = β is the solution to the following SDE:
Revised: A Ornstein-Uhlenbeck X T ; β pinned at X T ; β= 0 and X T ; β T = β is the solution to the following SDE:

Original: [Equation]
Revised: [Equation]

Original: where q is the diffusion coefficient and σ is the diffusion for the OU process.
Revised: where q is the diffusion coefficient and σ is the diffusion for the OU process.

Original: The transition probability density function reads as:
Revised: The transition probability density function reads as:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: C O THER R ESULTS FOR GLUE E XPERIMENTS
Revised: C O THER R ESULTS FOR GLUE E XPERIMENTS

Original: In this section, we present the complete results including OU bridge regularizer for Table 1 and Table 2.
Revised: In this section, we present the complete results including OU bridge regularizer for Table 1 and Table 2.

Original: We also report the results for Deberta xlarge , and the results on few-shot GLUE for both
Revised: We also report the results for Deberta xlarge , and the results on few-shot GLUE for both

Revised: [Figure]

Original: BERT large and Deberta xlarge under 4-, 8-, 16-, and 32-shot.
Revised: BERT large and Deberta xlarge under 4-, 8-, 16-, and 32-shot.

Original: We observe that BERT large cannot give reasonable answers on CoLA and the Matthews correlations are around 0 for all the PETs and all the shots we have experienced with.
Revised: We observe that BERT large cannot give reasonable answers on CoLA and the Matthews correlations are around 0 for all the PETs and all the shots we have experienced with.

Original: However, the situation gets better for the larger model Deberta xlarge .
Revised: However, the situation gets better for the larger model Deberta xlarge .

Original: Therefore, we exclude CoLA for BERT large and keep it for Deberta xlarge .
Revised: Therefore, we exclude CoLA for BERT large and keep it for Deberta xlarge .

Original: We only select the Brownian bridge as the representative in this section, since the Brownian bridge and Ornstein-Uhlenbeck bridge have no significant difference in Table 1.
Revised: We only select the Brownian bridge as the representative in this section, since the Brownian bridge and Ornstein-Uhlenbeck bridge have no significant difference in Table 1.

Original: [Figure] In Table 7 and Table [Figure] we report the performance of OU bridge regularizers.
Revised: In Table 7 and Table 6, we report the performance of OU bridge regularizers.

Original: The experimental setups are the same as Table 1 and Table 2 respectively.
Revised: The experimental setups are the same as Table 1 and Table 2 respectively.

Original: The performances between OU bridge and Brownian bridge do not have a significant difference.
Revised: The performances between OU bridge and Brownian bridge do not have a significant difference.

Original: In Table [Figure] we report the performance of
Revised: In Table 8, we report the performance of

Original: Deberta xlarge on full GLUE datasets.
Revised: Deberta xlarge on full GLUE datasets.

Original: On all four PETs, the SDE regularizer outperforms the PDF regularizer, this is consistent with the results we see in Table 1.
Revised: On all four PETs, the SDE regularizer outperforms the PDF regularizer, this is consistent with the results we see in Table 1.

Original: The results for 4-, 8-, and 32-shot for BERT large and Deberta xlarge are plotted respectively in Figure 3 and Figure 4.
Revised: The results for 4-, 8-, and 32-shot for BERT large and Deberta xlarge are plotted respectively in Figure 3 and Figure 4.

Original: For simplicity, we only plot the average performance for each PET.
Revised: For simplicity, we only plot the average performance for each PET.

Original: We report the results for 16-shot experiments for Deberta xlarge in Table 9.
Revised: We report the results for 16-shot experiments for Deberta xlarge in Table 9.

Original: The setup for the experiment is almost the same as the experiment in Section 4.3, and the hyper-parameters are listed in Appendix E.
Revised: The setup for the experiment is almost the same as the experiment in Section 4.3, and the hyper-parameters are listed in Appendix E.

Original: The SDE regularizer outperforms the PDF regularizer on most of the PETs except Prompt tuning.
Revised: The SDE regularizer outperforms the PDF regularizer on most of the PETs except Prompt tuning.

Original: We notice that the SDE regularizer helps Deberta xlarge substantially on CoLA for most of the PETs, indicating the SDE regularizer can effectively provide useful guidance when the data is scarce and the task is hard.
Revised: We notice that the SDE regularizer helps Deberta xlarge substantially on CoLA for most of the PETs, indicating the SDE regularizer can effectively provide useful guidance when the data is scarce and the task is hard.

Original: D C ALCULATION OF C ORRELATION IN S ECTION
Revised: D C ALCULATION OF C ORRELATION IN S ECTION

Original: In this section, we elaborate on how we calculate the correlations reported in Table 4 and Table 5. 
Revised: In this section, we elaborate on how we calculate the correlations reported in Table 4 and Table 5. [Figure]

Original: D. C ORRELATION BETWEEN N UMBER OF S HOTS AND D ISTANCE TO B RIDGE
Revised: D. C ORRELATION BETWEEN N UMBER OF S HOTS AND D ISTANCE TO B RIDGE

Original: Definition D.1 (Tie) .
Revised: Definition D.1 (Tie) .

Original: A pair of observation { ( x i , y i ) , ( x j , y j ) } is defined as tied if x i = x j or y i =
Revised: A pair of observation { ( x i , y i ) , ( x j , y j ) } is defined as tied if x i = x j or y i =

Original: y j .
Revised: y j .

Original: Since we generate the few-shot datasets using 5 random seeds for each shot, each PET has 5 results for each shot.
Revised: Since we generate the few-shot datasets using 5 random seeds for each shot [Figure] each PET has 5 results for each shot.

Original: This results in observations with ties, e.g., the two observations for distances on the first seed and second seed for 8-shot { (8 , d 1 ) , (8 , d 2 ) } are tied.
Revised: This results in observations with ties, e.g., the two observations for distances on the first seed and second seed for 8-shot { (8 , d 1 ) , (8 , d 2 ) } are tied.

Original: To calculate the correlation for data with ties, the Tau-b of Kendall’s rank correlation is more suitable than Pearson’s correlation.We
Revised: To calculate the correlation for data with ties, the Tau-b of Kendall’s rank correlation is more suitable than Pearson’s correlation.

Original: [Figure] therefore report the Kendall’s rank correlation for the correlation between the number of shots and the hidden states distance to the latent bridges.
Revised: We therefore report the Kendall’s rank correlation for the correlation between the number of shots and the hidden states distance to the latent bridges.

Original: D. C ORRELATION BETWEEN P ERFORMANCE AND D ISTANCE TO B RIDGE
Revised: D. C ORRELATION BETWEEN P ERFORMANCE AND D ISTANCE TO B RIDGE

Original: We mix all the few-shot results for different shots and different seeds to form observations of performance and the hidden states distances to bridges, and then calculate the Pearson’s correlation.
Revised: We mix all the few-shot results for different shots and different seeds to form observations of performance and the hidden states distances to bridges, and then calculate the Pearson’s correlation.

Original: E H YPER - PARAMETERS E. T RAINING g γ
Revised: E H YPER - PARAMETERS E. T RAINING g γ

Original: we use simple 3-layer MLP for g γ in all of our experiments.
Revised: we use simple 3-layer MLP for g γ in all of our experiments.

Original: For PDF regularizer, the output dimensions of each layer are 1024, 256, 128, and for SDE regularizer, the output dimensions of each layer are 1024, 256, 32.
Revised: For PDF regularizer, the output dimensions of each layer are 1024, 256, 128, and for SDE regularizer, the output dimensions of each layer are 1024, 256, 32.

Original: We observe that the running time increases noticeably with the final output dimension for SDE regularizer, we thus choose a smaller one for SDE regularizer.
Revised: We observe that the running time increases noticeably with the final output dimension for SDE regularizer, we thus choose a smaller one for SDE regularizer.

Original: The hyper-parameters for training g γ are listed in Table 10.
Revised: The hyper-parameters for training g γ are listed in Table 10.

Original: We run all the experiments for 50k steps, and evaluate on the development set every 1k steps. For BERT large , we use 32 as the batch size while for Deberta xlarge , we use 16 as the batch size.
Revised: [Table]

Original: We choose learning rate 1e-3 for Prompt tuning for both PLMs, and 1e-4 for other PETs for both PLMs. We use 0.01 weight decay, 1.0 maximum gradient norm and no learning rate warm-up for all the experiments. We search the best regularization strength α in { 0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0 } for PDF regularizer, and in { 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.2,
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: +OU PDF 51.
Revised: +OU PDF 51.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: +OU SDE
Revised: +OU SDE

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: B IT F IT
Revised: B IT F IT

Original: [Table] [Table]
Revised: [Table] [Table]

Original: - +B ROWN PDF
Revised: - +B ROWN PDF

Original: [Table]
Revised: [Table]

Original: +B ROWN SDE
Revised: +B ROWN SDE

Original: [Table]
Revised: [Table]

Original: +OU PDF
Revised: +OU PDF

Original: [Table] [Table]
Revised: [Table] [Table]

Original: +OU SDE
Revised: +OU SDE

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: A DAPTER
Revised: A DAPTER

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: - +B ROWN PDF 49.

Revised: - +B ROWN PDF 49.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: +B ROWN SDE
Revised: +B ROWN SDE

Original: [Table]
Revised: [Table]

Original: +OU PDF
Revised: +OU PDF

Original: [Table]
Revised: [Table]

Original: +OU SDE
Revised: +OU SDE

Original: [Table] [Table]
Revised: [Table] [Table]

Original: +OU SDE 84.7 0 .
Revised: +OU SDE 84.7 0 .

Original: L O RA 88.8 0 .
Revised: L O RA 88.8 0 .

Original: +B ROWN PDF 88.9 0 .
Revised: +B ROWN PDF 88.9 0 .

Original: +OU PDF 88.9 0 .
Revised: +OU PDF 88.9 0 .

Original: +B ROWN SDE 88.9 0 .
Revised: +B ROWN SDE 88.9 0 .

Original: +OU SDE 88.8 0 . 

Revised: +OU SDE 88.8 0 .

Original: B IT F IT 87.9 0 .
Revised: B IT F IT 87.9 0 .

Original: +B ROWN PDF 87.9 0 .
Revised: +B ROWN PDF 87.9 0 .

Original: +OU PDF 87.9 0 .
Revised: +OU PDF 87.9 0 .

Original: +B ROWN SDE 87.9 0 .
Revised: +B ROWN SDE 87.9 0 .

Original: +OU SDE 87.9 0 .
Revised: +OU SDE 87.9 0 .

Original: A DAPTER 88.8 0 .
Revised: A DAPTER 88.8 0 .

Original: +B ROWN PDF 89.0 0 .
Revised: +B ROWN PDF 89.0 0 .

Original: +OU PDF 88.9 0 .
Revised: +OU PDF 88.9 0 .

Original: +B ROWN SDE 88.9 0 .
Revised: +B ROWN SDE 88.9 0 .

Original: +OU SDE 88.9 0 .
Revised: +OU SDE 88.9 0 .

Original: 0.5, 1.0 } for SDE regularizer.

Original: The best α for PDF regularizer are listed in Table 11, and best α for

Original: SDE regularizer are listed in Table 12.

Original: [Table]
Revised: [Table]

Original: B

Original: [Table][Table]
Revised: [Table]

Original: +B ROWN PDF 52.1 3 .
Revised: +B ROWN PDF 52.1 3 .

Original: +B ROWN SDE 49.6 4 .
Revised: +B ROWN SDE 49.6 4 .

Original: B IT F IT 41.9 3 .
Revised: B IT F IT 41.9 3 .

Original: +B ROWN PDF 45.2 3 .
Revised: +B ROWN PDF 45.2 3 .

Original: +B ROWN SDE 45.7 3 .
Revised: +B ROWN SDE 45.7 3 .

Original: A DAPTER 43.1 2 . 9 67.7 2 . 7 55.9 5 .
Revised: A DAPTER 43.1 2 . 9 67.7 2 . 7 55.9 5 .

Original: +B ROWN PDF 50.7 3 . 

Revised: +B ROWN PDF 50.7 3 .

Original: +B ROWN SDE 47.1 1 . 6 72.0 1 .
Revised: +B ROWN SDE 47.1 1 . 6 72.0 1 .

Revised: [Table]

Revised: 1e

Original: [Table]
Revised: [Table]

Revised: E. T RAINING PET S ON FULL - SET GLUE

Revised: We run all the experiments for 50k steps, and evaluate on the development set every 1k steps.

Revised: For

Revised: BERT large , we use 32 as the batch size while for Deberta xlarge , we use 16 as the batch size.

Revised: We choose learning rate 1e-3 for Prompt tuning for both PLMs, and 1e-4 for other PETs for both PLMs.

Revised: We use 0.01 weight decay, 1.0 maximum gradient norm and no learning rate warm-up for all the experiments.

Revised: We search the best regularization strength α in { 0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0 } for PDF regularizer, and in { 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.2, 0.5, 1.0 } for SDE regularizer.

Revised: The best α for PDF regularizer are listed in Table 11, and best α for

Revised: SDE regularizer are listed in Table 12.

Revised: [Table]

Revised: [Table]

Original: E. T RAINING PET S ON FEW - SHOT GLUE
Revised: E. T RAINING PET S ON FEW - SHOT GLUE

Original: We run all the experiments for 1k steps, and evaluate on the development set every 50 steps.
Revised: We run all the experiments for 1k steps, and evaluate on the development set every 50 steps.

Original: For all the shots for both regularizers and both models, we use a batch size of 2.
Revised: For all the shots for both regularizers and both models, we use a batch size of 2.

Original: Other hyper-parameters are kept the same as in the experiments on full-set GLUE.
Revised: Other hyper-parameters are kept the same as in the experiments on full-set GLUE.

Revised: F P ERFORMANCE OF R EGULARIZERS T RAINED WITH T INY C ORPUS

Original: [Table]
Revised: Although we use the pre-training corpus to train our mapping g γ , the training is actually fast and data-efficient. We show that when using only 10,000 documents in the pre-training corpus (about 0.1% of the corpus), the obtained regularizers still perform great and are comparable to the regularizers trained on the whole pre-training corpus. We train the mapping g γ for 5,000 iterations withbatch size. On a single NVIDIA A100 GPU, the training can be done in 1 hour for PDF regularizer, and 3 hours for SDE regularizer.

Revised: The cost of training our regularizer is quite small compared to the resources required for pre-training.

Revised: We conduct the same experiments as Section 4.2 and Section 4.3 with the regularizers trained with tiny corpus.

Revised: The results are presented in Table 13 and Table 14 respectively.

Revised: On full-set GLUE, the PDF regularizer performs even better on three out of four PETs, and although its performance is affected on Adapter, it still outperforms the vanilla Adapter.

Revised: The SDE regularizer is slightly affected on three out of four PETs, but it still brings substantial improvements on all the

Revised: PETs.

Revised: On few-shot GLUE, the impact of the shrinkage of the corpus is relatively obvious.

Revised: But overall, the regularizers still performs great on all the PETs.

Revised: The drop in performances are relatively small compared to the boost they bring to vanilla PETs.

Revised: [Table]

Revised: [Table]

Revised: G S PEED OF THE R EGULARIZERS

Revised: As we have mentioned in Section 1, the PDF regularizer only incurs negligible computational cost.

Revised: In this section, we present the plot of step-metric curve on full-set GLUE in Figures 5, 6, 7 and 8.

Revised: On different PETs, the regularized PETs with PDF regularizer has similar running time to the vanilla

Revised: PETs.

Revised: On the two large datasets, QQP and MNLI, regularized PETs with SDE regularizer take about 2 to 3 times longer to achieve the best performance than vanilla PETs.

Revised: However, on medium-sized (QNLI, SST-2) and small datasets (CoLA, MRPC, RTE), the time to achieve the best results with

Revised: SDE regularizer is comparable to vanilla PETs.

Revised: Overall, the PDF regularizer can effectively improve the performance of PETs without introducing much computational cost.

Revised: In scenarios where there is relatively more focus on the inference performance of PETs and less concern about the slightly longer training time, or when the dataset is small,

Revised: SDE regularizer should be a good choice.

Original: [Table]
Revised: [Figure] [Figure] [Figure] [Figure]