Original: AdapterBias: Parameter-efﬁcient Token-dependent Representation Shift for Adapters in NLP Tasks
Revised: AdapterBias: Parameter-efﬁcient Token-dependent Representation Shift for Adapters in NLP Tasks

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: Transformer-based pre-trained models withmillions of parameters require large storage.
Revised: Transformer-based pre-trained models with millions of parameters require large storage.

Original: Recent approaches tackle this shortcoming bytraining adapters, but these approaches still require a relatively large number of parameters.
Revised: Recent approaches tackle this shortcoming by training adapters, but these approaches still require a relatively large number of parameters.

Original: In this study, AdapterBias, a surprisingly simple yet effective adapter architecture, is proposed.
Revised: In this study, AdapterBias, a surprisingly simple yet effective adapter architecture, is proposed.

Original: AdapterBias adds a token-dependentshift to the hidden output of transformer layersto adapt to downstream tasks with only a vector and a linear layer.
Revised: AdapterBias adds a token-dependent shift to the hidden output of transformer layers to adapt to downstream tasks with only a vector and a linear layer.

Original: Extensive experimentsare conducted to demonstrate the effectivenessof AdapterBias.
Revised: Extensive experiments are conducted to demonstrate the effectiveness of AdapterBias.

Original: The experiments show thatour proposed method can dramatically reducethe trainable parameters compared to the previous works with a minimal decrease in taskperformances compared with ﬁne-tuned pretrained models.
Revised: The experiments show that our proposed method can dramatically reduce the trainable parameters compared to the previous works with a minimal decrease in task performances compared with ﬁne-tuned pretrained models.

Original: We further ﬁnd that AdapterBias automatically learns to assign more signiﬁcant representation shifts to the tokens related to the task in consideration.
Revised: We further ﬁnd that AdapterBias automatically learns to assign more signiﬁcant representation shifts to the tokens related to the task in consideration.

Original:  Introduction
Revised:  Introduction

Original: While large pre-trained language models (PLMs)reached state-of-the-art results on natural languageprocessing (NLP) tasks, PLMs require updatingall parameters and storing the fully ﬁne-tunedmodel for each downstream task.
Revised: While large pre-trained language models (PLMs) reached state-of-the-art results on natural language processing (NLP) tasks, PLMs require updating all parameters and storing the fully ﬁne-tuned model for each downstream task.

Original: These requirements have led to difﬁculties in real-world applications.
Revised: These requirements have led to difﬁculties in real-world applications.

Original: Moreover, ﬁne-tuning PLMs on lowresource datasets is subject to instabilities.
Revised: Moreover, ﬁne-tuning PLMs on lowresource datasets is subject to instabilities.

Original: To tackle these shortcomings, Adapters (Houlsbyet al., 2019), a more parameter-efﬁcient alternativetraining strategy for the transformer architecture(Vaswani et al., 2017) has been proposed.
Revised: To tackle these shortcomings, Adapters (Houlsby et al., 2019), a more parameter-efﬁcient alternative training strategy for the transformer architecture (Vaswani et al., 2017) have been proposed.

Original: Insteadof full ﬁne-tuning the whole model, Adapters introduces extra tunable weights and freezes the originalparameters of PLM.
Revised: Instead of full ﬁne-tuning the whole model, Adapters introduce extra tunable weights and freeze the original parameters of PLM.

Original: Adapters demonstrated comparable performance with fully ﬁne-tuning the entiremodel.
Revised: Adapters demonstrated comparable performance with fully ﬁne-tuning the entire model.

Original: Although Adapters solve the problem ofthe PLM’s massive parameters, researchers are curious about how many more parameters are required to reach state-of-the-art performance on standard NLP tasks.
Revised: Although Adapters solve the problem of the PLM’s massive parameters, researchers are curious about how many more parameters are required to reach state-of-the-art performance on standard NLP tasks.

Original: The results in Houlsby et al.
Revised: The results in Houlsby et al.

Original: (2019) have shown that the performance on GLUE benchmark (Wang et al., 2018) is almost the same when removing the Adapters in the ﬁrst layers, which indicates that not every adapter is useful.
Revised: (2019) have shown that the performance on GLUE benchmark (Wang et al., 2018) is almost the same when removing the Adapters in the lower layers, which indicates that not every adapter is useful.

Original: It leaves the question of whether adapters can be even more parameter-efﬁcient.
Revised: It raises the question of whether adapters can be even more parameter-efﬁcient.

Original: To develop practical and memory-efﬁcient adapters, Diff pruning (Guo et al., 2020) enables parameter-efﬁcient transfer learning that scales well with new tasks.
Revised: To develop practical and memory-efﬁcient methods of utilizing PLMs, Diff pruning (Guo et al., 2020) enables parameter-efﬁcient transfer learning that scales well with new tasks.

Original: The approach learns a taskspeciﬁc “diff” vector that extends the original pretrained parameters and encourages the sparsity of the vector through L 0 -norm regularization.
Revised: The approach learns a task-speciﬁc “diff” vector that extends the original pre-trained parameters and encourages the sparsity of the vector through L 0 -norm regularization.

Original: Another approach is BitFit (Ben Zaken et al., 2021), which shows that with small-to-medium training data, ﬁne-tuning only a subset of the bias terms of pre-trained BERT models (Devlin et al., 2018) is competitive with ﬁne-tuning the entire model.
Revised: Another approach is BitFit (Ben Zaken et al., 2021), which shows that with small-to-medium training data, ﬁne-tuning only a subset of the bias terms of pre-trained BERT models (Devlin et al., 2018) is competitive with ﬁne-tuning the entire model.

Original: The central concept of these approaches is to addtask-speciﬁc shifts to each output representationof the PLM layers so as to adapt to different tasks.
Revised: The central concept of these approaches is to add task-speciﬁc shifts to each output representation of the PLM layers so as to adapt to different tasks.

Original: In the previous works, Ben Zaken et al.
Revised: In the previous works, Ben Zaken et al.

Original:  Guo et al.
Revised: (2021); Guo et al.

Original: (2020) both add the same shifts to theoutput representation regardless of which token ismore relevant to the task.
Revised: (2020) both add the same shifts to the output representation regardless of which token is more relevant to the task.

Original: However, consideringsome speciﬁc tokens might be more critical to aparticular task, the representation can better adaptto the downstream task under a limited amount ofparameters if these shifts are based on the inputtokens.
Revised: However, considering some speciﬁc tokens might be more critical to a particular task, the representation can better adapt to the downstream task under a limited amount of parameters if these shifts are based on the input tokens.

Original: Based on this concept, in this study, we addtoken-dependent biases to the shifts by proposing
Revised: Based on this concept, in this study, we add token-dependent biases to the shifts by proposing

Original: AdapterBias, which consists of a vector and a linearlayer ( L α ).
Revised: AdapterBias, which consists of a vector and a linear layer ( L α ).

Original: The vector represents the task-speciﬁcshift, and L α produces the weights for input tokens.
Revised: The vector represents the task-speciﬁc shift, and L α produces the weights for input tokens.

Original: Thus, with the vector and the weights, AdapterBiascan add a token-dependent shift to the transformerlayer.
Revised: Thus, with the vector and the weights, AdapterBias can add a token-dependent shift to the transformer layer.

Original: Since the concept of BitFit (Ben Zaken et al.,2021) is similar to AdapterBias by adding a shift tothe representation, we demonstrate the differencebetween BitFit and AdapterBias in Figure 1.
Revised: Since the concept of BitFit (Ben Zaken et al., 2021) is similar to AdapterBias by adding a shift to the representation, we demonstrate the difference between BitFit and AdapterBias in Figure 1.

Original: BitFit assigns identical shifts to all the tokens, while
Revised: BitFit assigns identical shifts to all the tokens, while

Original: AdapterBias adds more signiﬁcant shifts to the tokens related to the task.
Revised: AdapterBias adds more signiﬁcant shifts to the representations that are related to the task.

Original: With fewer trainable parameters required,
Revised: With fewer trainable parameters required,

Original: AdapterBias achieves comparable performance onthe GLUE benchmark with Houlsby et al.
Revised: AdapterBias achieves comparable performance on the GLUE benchmark with Houlsby et al.

Original: Pfeiffer et al.
Revised: Pfeiffer et al.

Original: (2020a)
Revised: (2020a)

Original: ; Guo et al.
Revised: ; Guo et al.

Original: Ben Zaken et al.
Revised: Ben Zaken et al.

Original:  We further decrease the parameters of AdapterBias in different ways, includingpartial weight-sharing in AdapterBias and adding L 0 -norm regularization.
Revised: ; Hu et al. We further decrease the parameters of AdapterBias in different ways, including partial weight-sharing in AdapterBias and adding L 0 -norm regularization.

Original: Finally, AdapterBias hasbetter interpretability due to its simplicity.
Revised: Finally, AdapterBias has better interpretability due to its simplicity.

Original: Weuse different tools, including word cloud and PCA(Jolliffe, 2002), to visualize what AdapterBias haslearned, and we found that the proposed approachautomatically learns to assign larger representationshifts to the task-related tokens.
Revised: We use different tools, including word cloud and PCA (Jolliffe, 2002), to visualize what AdapterBias has learned, and we found that the proposed approach automatically learns to assign larger representation shifts to the task-related tokens.

Original:  Related Work
Revised:  Related Work

Original: For NLP tasks, adapters are introduced for thetransformer architecture.
Revised: For NLP tasks, adapters are introduced for the transformer architecture.

Original: A set of adapter parameters was added at each transformer layer, whichis mostly bottleneck architectures Houlsby et al.(2019).
Revised: A set of adapter parameters was added at each transformer layer, which is mostly bottleneck architectures Houlsby et al.

Original: By keeping the output dimension identical,they cause no change to the structure or parametersof the original model.
Revised: By keeping the output dimension identical, they cause no change to the structure or parameters of the original model.

Original: Adapters quickly gained popularity in NLP withvarious applications.
Revised: Adapters quickly gained popularity in NLP with various applications.

Original: For multi-task learning (Caruana, 1997; Zhang and Yang, 2017; Liu et al., 2019b), a projected self-attention layer is proposed by Stickland and Murray (2019), while Bapna et al.
Revised: For multi-task learning (Caruana, 1997; Zhang and Yang, 2017; Liu et al., 2019b), a projected self-attention layer is proposed by Stickland and Murray (2019), while Bapna et al.

Original: (2019) proposed an additional layer norm suitable for machine translation.
Revised: (2019) proposed an additional layer norm suitable for machine translation.

Original: Besides the applications of adapters, researchers are also dedicated to improving their performance.
Revised: Besides the applications of adapters, researchers are also dedicated to improving their performance.

Original: Based on the architecture introduced by Houlsby et al.
Revised: Based on the architecture introduced by Houlsby et al.

Original: , AdapterFusion (Pfeiffer et al., 2020a) leveraged knowledge from multiple tasks with a new two-stage learning algorithm.
Revised: , AdapterFusion (Pfeiffer et al., 2020a) leveraged knowledge from multiple tasks with a new two-stage learning algorithm.

Original: Despite the recent popularity of these methods, they still train a relatively large number of training parameters.
Revised: Despite the recent popularity of these methods, they still train a relatively large number of training parameters.

Original: Recently, studies start to focus on improving the parameter-efﬁciency of adapters.
Revised: Recently, studies start to focus on improving the parameter-efﬁciency of adaptation to a new task.

Original: Diff-pruning (Guo et al., 2020) achieves parameter efﬁciency by adding a sparse, task-speciﬁc difference-vector to the ﬁxed original parameters.
Revised: Diff-pruning (Guo et al., 2020) achieves parameter efﬁciency by adding a sparse, task-speciﬁc difference-vector to the ﬁxed original parameters.

Original: The vector is adaptively pruned during training with a differentiable approximation to the L 0 -norm penalty to encourage sparsity.
Revised: The vector is adaptively pruned during training with a differentiable approximation to the L 0 -norm penalty to encourage sparsity.

Original: Rücklé et al.
Revised: Rücklé et al.

Original:  introduced AdapterDrop, which has been recently integrated into AdapterHub (Pfeiffer et al., 2020b) by removing adapters from lower transformer layers during training and inference, which can dynamically reduce the computational cost.
Revised: (2020) introduced AdapterDrop, which has been recently integrated into AdapterHub (Pfeiffer et al., 2020b). It removes adapters from lower transformer layers during training and inference, which can dynamically reduce the computational cost.

Original: Mahabadi et al.
Revised: Mahabadi et al.

Original: (2021) proposed Compacter, which improved the trade-off between performance and trainable parameters per task with low-rank optimization.
Revised: (2021) proposed Compacter, which improved the trade-off between performance and trainable parameters per task with low-rank optimization.

Original: On the other hand, without modifying the architecture of the PLM, BitFit (Ben Zaken et al., 2021) shows that ﬁne-tuning only the bias terms of a large
Revised: On the other hand, without modifying the architecture of the PLM, BitFit (Ben Zaken et al., 2021) shows that ﬁne-tuning only the bias terms of a large

Original: PLM is also competitive with ﬁne-tuning the entire model.
Revised: PLM is also competitive with ﬁne-tuning the entire model.

Original: Fine-tuning only the bias terms can be considered as adding a task-speciﬁc shift to the token representation.
Revised: Fine-tuning only the bias terms can be considered as adding a task-speciﬁc shift to the token representation.

Original: BitFit is most similar to our work.
Revised: BitFit is most similar to our work.

Original: While in BitFit, the shifts added to all the representations are exactly the same for all input tokens, in our work, the shifts are token-dependent.
Revised: While in BitFit, the shifts added to all the representations are exactly the same for all input tokens, in our work, the shifts are token-dependent.

Original:  Method
Revised:  Method

Original: In this section, we present AdapterBias, an efﬁcient way to adapt large-scale PLMs.
Revised: In this section, we present AdapterBias, an efﬁcient way to adapt large-scale PLMs.

Original: In order to better adapt to different downstream tasks, the adapter module should be token-speciﬁc.
Revised: In order to better adapt to different downstream tasks, the adapter module should be token-speciﬁc.

Original: AdapterBias produces a suitable weight of the bias based on the input tokens.
Revised: AdapterBias produces a suitable weight for the bias based on the input token.

Original: Problem Formulation We consider the general problem of ﬁne-tuning PLMs, where the training data D =
Revised: Problem Formulation We consider the general problem of ﬁne-tuning PLMs, where the training data D =

Original: ( x i , y i )
Revised: ( x i , y i )

Original: Nn =1 is given.
Revised: Nn =1 is given.

Original: Assume that given a PLM with parameters θ and AdapterBias withparameters θ (cid:48) .
Revised: Assume that given a PLM with parameters θ and AdapterBias with parameters θ (cid:48) .

Original: During the training stage, we freezeθ and tune θ (cid:48) only.
Revised: During the training stage, we freeze θ and tune θ (cid:48) only.

Original: 3. AdapterBias
Revised: 3. AdapterBias

Original: The architecture of AdapterBias is shown in theright part of Figure 2.
Revised: The architecture of AdapterBias is shown in the right part of Figure 2.

Original: AdapterBias consists of twomodules: a vector ( v ) and a linear layer ( L α ).
Revised: AdapterBias consists of two modules: a vector ( v ) and a linear layer ( L α ).

Original: vis a task-speciﬁc shift added to the output of eachtransformer layer.
Revised: v is a task-speciﬁc shift added to the output of each transformer layer.

Original: Since some tokens  are moreimportant to some tasks, these tokens should be assigned larger representation shifts than other tokens.
Revised: The tokens which are more related to the task should be assigned larger representation shifts than other tokens.

Original: The linear layer ( L α ) produces a token-dependentweight vector
Revised: The linear layer ( L α ) produces a token-dependent weight vector

Original: α =
Revised: α =

Original: [ α 1 , α 2 . . .
Revised: [ α 1 , α 2 . . .

Original: α m ] T , where α i isthe weight of the i th token’s representation shift.
Revised: α m ] T , where α i is the weight of the i th token’s representation shift.

Original: By applying the token-speciﬁc weight to the taskspeciﬁc representation shift ( v ), AdapterBias canfocus on the tokens that are more important to thetask and is able to adapt to different downstreamtasks efﬁciently.
Revised: By applying the token-speciﬁc weight to the task-speciﬁc representation shift ( v ), AdapterBias can focus on the tokens that are more important to the task and is able to adapt to different downstream tasks efﬁciently.

Original: We deﬁne the output of AdapterBias as the bias( B ), which is the outer product of v and the learnedweights vector α .
Revised: We deﬁne the output of AdapterBias as the bias ( B ), which is the outer product of v and the learned weights vector α .

Original: When the dimension of the token’s representation is r withwith m input tokens,the function can be deﬁned as follows:
Revised: When the dimension of the token’s representation is r with m input tokens, the function can be deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where v ∈ R r , α ∈ R m , and B ∈ R r × m .
Revised: where v ∈ R r , α ∈ R m , and B ∈ R r × m .

Original: To further elaborate on the details of AdapterBias, we give an example of how AdapterBias produces B and how B adds to the transformer layer.
Revised: To further elaborate on the details of AdapterBias, we give an example of how AdapterBias produces B and how B adds to the transformer layer.

Original: In Figure 3, we assume that there are three representation outputs ( r 1 , r 2 , r 3 ) after the ﬁrst layer normalization.
Revised: In Figure 3, we assume that there are three representation outputs ( r 1 , r 2 , r 3 ) after the ﬁrst layer normalization.

Original: The dimension of r 1 , r 2 and r 3 is the dimension of the 2nd feedforward layer, while the  dimension of the linear layer ( L α ) is the output dimension of the ﬁrst feed-forward layer with the token representation ( r 1 , r 2 , r 3 ) as its inputs.
Revised: The dimension of r 1 , r 2 and r 3 is the dimension of the 2 nd feedforward layer, while the input dimension of the linear layer ( L α ) is the output dimension of the ﬁrst feed-forward layer with the token representation ( r 1 , r 2 , r 3 ) as its inputs.

Original: The linear layer ( L α ) produces α , where α ∈ R 3 .
Revised: The linear layer ( L α ) produces α , where α ∈ R 3 .

Original: The blocks in different colors represent the difference of the weights ( α 1 , α 2 , α 3 ).
Revised: The blocks in different colors represent the difference of the weights ( α 1 , α 2 , α 3 ).

Original: Take BERT-base for example, after performing outer product with the weights vector α and the vector ( v ), the dimension of B becomes 768 × 3 .
Revised: Take BERT-base for example, after performing outer product with the weights vector α and the vector ( v ), the dimension of B becomes 768 × 3 .

Original: For example, b 1 , the ﬁrst column of B , is the shift for the ﬁrst token representation.
Revised: For example, b 1 , the ﬁrst column of B , is the shift for the ﬁrst token representation.

Original: 3. Further improvement on 
Revised: 3. Further improvement on parameter-efﬁciency of AdapterBias

Original: parameter-efﬁciency of AdapterBias

Original: In this section, we experiment on two ways to make AdapterBias more parameter efﬁcient.
Revised: In this section, we experiment on two different methods to make AdapterBias more parameter efﬁcient.

Original: One is partial weight-sharing of AdapterBias among transformer layers, another is enforcing the weights of the linear layer ( L α ) to be sparse by utilizing L 0norm penalty.
Revised: One is partial weight-sharing of AdapterBias among transformer layers, another is enforcing the weights of the linear layer ( L α ) to be sparse by utilizing L 0 -norm penalty.

Original:  Cross-layer parameters sharing in AdapterBias
Revised: 3.2. Cross-layer parameters sharing in AdapterBias

Original: Redundancies have been observed in the information captured by adapters, with adapters inlowerlayers being less important. In the work of Houlsbyet al. (2019), they observed that their Adapter modules in the lower layers are less important.
Revised: Redundancies have been observed in the information captured by adapters, with adapters in lower layers being less important (Houlsby et al., 2019).

Original: In addition, sharing parameters of the Adapter acrosslayers leads to a comparatively small drop in performance in some tasks.
Revised: In addition, sharing parameters of the Adapter across layers leads to a comparatively small drop in performance in some tasks.

Original: In light of the above information, we further reduce the number of parameters required for each task by partially sharingthe weights of the adapters across all transformerlayers.
Revised: In light of the above information, we further reduce the number of parameters required for each task by partially sharing the weights of the adapters across all transformer layers.

Original: The experimental results are discussed at
Revised: The experimental results are discussed at

Original: Section 4.6.1.
Revised: Section 4.6.1.

Original: 3.2. L 0 regularization in AdapterBias
Revised: L 0 regularization in AdapterBias

Original: Sparsity has been utilized in various parameterefﬁcient methods.
Revised: Sparsity has been utilized in various parameterefﬁcient methods.

Original: For applications in NLP tasks,
Revised: For applications in NLP tasks,

Original: Diff-pruning (Guo et al., 2020) learns a sparse vector added to the whole PLM with L 

Revised: Diff-pruning (Guo et al., 2020) learns a sparse vector added to the whole PLM with L

Original: -norm penalty.
Revised: -norm penalty.

Original: Inspired by their work, we further apply L 0 -normregularization to L α in the AdapterBias module,aiming to encourage the sparsity of L α .
Revised: Inspired by their work, we further apply L 0 -norm regularization to L α in the AdapterBias module, aiming to encourage the sparsity of L α .

Original: We chooseto drop L α because it contributes most of the parameters in AdapterBias.
Revised: We choose to drop L α because it contributes most of the parameters in AdapterBias.

Original: Encouraging its sparsitycan further increase the parameter efﬁciency.
Revised: Encouraging its sparsity can further increase the parameter efﬁciency.

Original: Notethat we speciﬁcally apply L 0 regularization in Section 4.6.2.
Revised: Note that we speciﬁcally apply L 0 regularization in Section 4.6.2.

Original: In AdapterBias, we add L 0 -norm penalty to thelinear layer ( L α ).
Revised: In AdapterBias, we add L 0 -norm penalty to the linear layer ( L α ).

Original: The optimization problem canbe expressed as,
Revised: The optimization problem can be expressed as,

Original: [Equation]
Revised: [Equation]

Original: where L ( D ; · ) represents the original loss withtraining data D .
Revised: where L ( D ; · ) represents the original loss with training data D .

Original: λ is the hyperparameter for L 0norm penalty.
Revised: λ is the hyperparameter for L 0norm penalty.

Original: Note that θ (cid:48) represents trainableparameters and θ (cid:48)
Revised: Note that θ (cid:48) represents trainable parameters and θ (cid:48)

Original: L α represents the parameters of
Revised: L α represents the parameters of

Original: L α in AdapterBias.
Revised: L α in AdapterBias.

Original: Following the work of Diffpruning, we utilize a relaxed mask vector (Louizoset al., 2017) with a stretched Hard-Concrete distribution (Jang et al., 2016; Maddison et al., 2016) toencourage L 0 sparsity.
Revised: Following the work of Diffpruning, we utilize a relaxed mask vector (Louizos et al., 2017) with a stretched Hard-Concrete distribution (Jang et al., 2016; Maddison et al., 2016) to encourage L 0 sparsity.

Original:  Experiments
Revised:  Experiments

Original: In this section, we evaluate the effectiveness of ourproposed adapter module in NLP training tasks,and provide the analysis of what AdapterBias haslearned in different tasks.
Revised: In this section, we evaluate the effectiveness of our proposed adapter module in NLP training tasks, and provide the analysis of what AdapterBias has learned in different tasks.

Original: 4. Experimental settings
Revised: 4. Experimental settings

Original: For the experiments, we base our experiments on HuggingFace PyTorch implementation (Wolf et al., 2019) of BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019c) models.
Revised: We base our experiments on HuggingFace PyTorch implementation (Wolf et al., 2019) of BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019c) models.

Original: The learning rate is set in the range [ 10 − 4 , 10 − 3 ], with AdamW (Loshchilov and Hutter, 2017) as the optimizer.
Revised: The learning rate is set in the range [ 10 − 4 , 10 − 3 ], with AdamW (Loshchilov and Hutter, 2017) as the optimizer.

Original: GLUE benchmark (Wang et al., 2018) and SQuAD v1.0 (Rajpurkar et al., 2016) are the training data in our settings.
Revised: GLUE benchmark (Wang et al., 2018) and SQuAD v1.0 (Rajpurkar et al., 2016) are the training data in our settings.

Original: The training details are shown in Appendix A.3.
Revised: The training details are shown in Appendix A.3.

Original: Note that the second layer normalization in each transformer layer is also tuned during the training stage, corresponding to the orange component in the right part of Figure 2.
Revised: Note that the second layer normalization in each transformer layer is also tuned during the training stage, corresponding to the orange component in the right part of Figure 2.

Original: We experiment with 3 random seeds and choose the seed with the best performance on the validation set to evaluate on the
Revised: We experiment with 3 random seeds and choose the seed with the best performance on the validation set to evaluate on the

Original: GLUE server.
Revised: GLUE server.

Original: We report the test metrics provided on the submission website 1 .
Revised: We report the test metrics provided on the submission website 2 .

Original: 4. Results on GLUE
Revised: 4. Results on GLUE

Original: In this section, we compare AdapterBias to other parameter-efﬁcient methods, including Adapters (Houlsby et al., 2019), Diff-pruning (Guo et al., 2020),and BitFit (Ben Zaken et al.,  2021).
Revised: In this section, we compare AdapterBias to other parameter-efﬁcient methods, including Adapters (Houlsby et al., 2019), Diff-pruning (Guo et al., 2020), BitFit (Ben Zaken et al., 2021), and LoRA (Hu et al., 2021).

Original: In Table 1, we report the test scores on the GLUE benchmark 
Revised: In Table 1, we report the test scores on the GLUE benchmark and the required new parameters per task. Here we use BERTlarge as the PLM. AdapterBias reaches 81.2 average score in GLUE benchmark, with the smallest amount of parameters (0.17M) added per task. AdapterBias shows competitive performance as its

Original: [Table]
Revised: [Table]

Revised: et al., 2019; Guo et al., 2020; Ben Zaken et al., 2021) are their reported performance and Pfeiffer et al.

Original:  experiments with 3 random seeds and report the best.
Revised: (2020a) ; Hu et al. (2021) performance is reproduced on our setting. Due to instability during training, we restart experiments with 3 random seeds and report the best.

Original: [Table]
Revised: [Table]

Original: and the required new parameters per task.

Original: Here weuse BERT-large as the PLM.

Original: AdapterBias reaches81.2 average score in GLUE benchmark, with thesmallest amount of parameters (0.23M) added pertask.

Original: AdapterBias shows competitive performanceas its parameters are 30 × less than the works of Houlsby et al.
Revised: parameters are 40 × less than the works of Houlsby et al.

Original: Although Diff-pruning (Guoet al., 2020) has the best average score among allparameter-efﬁcient methods, their work trains anadditional vector whose parameter count is equivalent to the parameters of the whole PLM.
Revised: Although Diff-pruning (Guo et al., 2020) achieves the best average score among all parameter-efﬁcient methods, their work trains an additional vector whose parameter count is equivalent to the parameters of the whole PLM.

Original: Thus,
Revised: Thus,

Original: Diff-pruning requires 340M trainable parametersof BERT-large during the training stage, while
Revised: Diff-pruning requires 340M trainable parameters of BERT-large during the training stage, while

Original: AdapterBias only trains 0.23M parameters.
Revised: AdapterBias only trains 0.17M parameters.

Original: Furthermore, AdapterBias achieves comparable performance with BitFit  with fewer parameters neededper task.
Revised: Furthermore, AdapterBias achieves comparable performance with BitFit and LoRA with fewer parameters needed per task.

Original: This shows that AdapterBias is a worthwhile targeted ﬁne-tuning method.
Revised: This shows that AdapterBias is a worthwhile targeted ﬁne-tuning method.

Original: 4. Different base models
Revised: 4. Different base models

Original: To analyze how well this approachgeneralizes to different PLMs on different models of AdapterBias, as shown in Table 2, we apply AdapterBias in different transformer-based PLMs, including BERT-base (BB), BERT-large (BL), RoBERTa-base (RoB), and RoBERTa-large (RoL), on the GLUE benchmark.
Revised: To analyze the generalization ability of this approach to different PLMs on different models of AdapterBias, as shown in Table 2, we apply AdapterBias in different transformer-based PLMs, including BERT-base (BB), BERT-large (BL), RoBERTa-base (RoB), and RoBERTa-large (RoL), on the GLUE benchmark.

Original: All results are scored by the GLUE evaluate server.
Revised: All results are scored by the GLUE evaluation server.

Original: Compared with BitFit, In Table 2, not only can AdapterBias perform well on BERT but also achieve competitive performance on larger PLMs such as RoBERTa.
Revised: Compared with BitFit, In Table 2, not only can AdapterBias perform well on BERT but also achieve competitive performance on larger PLMs such as RoBERTa.

Original: 4. Size of training data
Revised: 4. Size of training data

Original: In the previous experimental results, we observe that AdapterBias tends to have higher performance on tasks with a smaller amount of data (i.e. CoLA,
Revised: In the previous experimental results, we observe that AdapterBias tends to have higher performance on tasks with a smaller amount of data (i.e. CoLA,

Original: SST-2, and RTE).
Revised: SST-2, and RTE).

Original: To further validate this observation, we follow the work of BitFit (Ben Zaken et al., 2021) by training AdapterBias on subsets ofMethod [Table] SQuAD v1.0 (Rajpurkar et al., 2016) of increasingsize.
Revised: To further validate this observation, we follow the work of BitFit (Ben Zaken et al., 2021) by training AdapterBias on subsets of SQuAD v1.0 (Rajpurkar et al., 2016) of increasing size.

Original: The experiments are conducted with BERTbase model.
Revised: The experiments are conducted with BERT-base.

Original: The results on the validation set ofthe SQuAD dataset are listed in Figure 4, whichshows the tendency of AdapterBias outperforming full ﬁne-tuning when the size of the trainingdataset is smaller.
Revised: The results on the validation set of the SQuAD dataset are listed in Figure 4, which shows the tendency of AdapterBias outperforming full ﬁne-tuning when the size of the training dataset is smaller.

Original: However, with more trainingdata available, the trend is reversed.
Revised: However, with more training data available, the trend is reversed.

Original: The resultsshow that AdapterBias has the ability to outperformﬁne-tuning the whole PLM with small-to-mediumdata size, similarly to BitFit.
Revised: The results [Table] show that AdapterBias has the ability to outperform ﬁne-tuning the whole PLM with small-to-medium data size, similarly to BitFit.

Original:  Investigation on the effectiveness of tokendependent representation shift
Revised: 4. Investigation on the effectiveness of token dependent representation shift

Original: Different from BitFit (Ben Zaken et al. , 2021),where the bias terms in all transformer layers aretuned, we claim that the bias added to the representation should be token-dependent, and proposed
Revised: Different from BitFit (Ben Zaken et al., 2021), where the bias terms in all transformer layers are tuned, we claim that the bias added to the representation should be token-dependent, and proposed

Original: AdapterBias based on this concept.
Revised: AdapterBias based on this concept.

Original: We conductablation studies to verify this claim.
Revised: We conduct ablation studies to verify this claim.

Original: In this experiment, the linear layer ( L α ) in AdapterBias thatproduces the token-dependent weights vector ( α )is removed; that is, only the v is trained.
Revised: In this experiment, the linear layer ( L α ) in AdapterBias that produces the token-dependent weights vector ( α ) is removed; that is, only the v is trained.

Original: All shiftsadded to the representation outputs are identicalwithin the same transformer layer.
Revised: All shifts added to the representation outputs are identical within the same transformer layer.

Original: The experimentsare conducted with BERT-base model.
Revised: The experiments are conducted with BERT-base model.

Original: We report the test scores on the GLUE benchmark in Table 3.
Revised: We report the test scores on the GLUE benchmark in Table 3.

Original: The performance of AdapterBias without the linear layer ( L α ) dramatically decreases.
Revised: The performance of AdapterBias without the linear layer ( L α ) dramatically decreases.

Original: Without L α , it is hard for the vector ( v ) to adapt to different downstream tasks.
Revised: Without L α , it is hard for the vector ( v ) to adapt to different downstream tasks.

Original: This result demonstrates the importance of L α .
Revised: This result demonstrates the importance of L α .

Original: In other words, assigning different shifts to different token representations improves the performance of the method.
Revised: In other words, assigning different shifts to different token representations improves the performance of the method

Original:  4. Improving the parameter efﬁciency of 
Revised: . 4. Improving the parameter efﬁciency of AdapterBias

Original: AdapterBias

Original: We further apply two additional methods to
Revised: We further apply two additional methods to

Original: AdapterBias to enhance its parameter efﬁciency.
Revised: AdapterBias to enhance its parameter efﬁciency.

Original: Experiments are conducted to see whether AdapterBias can be more parameter-efﬁcient by sharing its components across all layers.
Revised: Experiments are conducted to examine whether AdapterBias can be more parameter-efﬁcient by sharing its components across all layers.

Original: Moreover, we experiment on adding L 0 -norm regularization during the training stage to encourage the sparsity of AdapterBias.
Revised: Moreover, we experiment on adding L 0 -norm regularization during the training stage to encourage the sparsity of AdapterBias.

Original: 4.6. Sharing components in AdapterBias
Revised: 4.6. Sharing components in AdapterBias

Original: In this experiment, we conduct an ablation study of partial weight-sharing in the AdapterBias module.
Revised: In this experiment, we conduct an ablation study of partial weight-sharing in the AdapterBias module.

Original: In Table 4, we share components of AdapterBias among different transformer layers.
Revised: In Table 4, we share components of AdapterBias among different transformer layers.

Original: Share v represents sharing v across all transformer layers, while Share L α means sharing the linear layer ( L α ).
Revised: Share v represents sharing v across all transformer layers, while Share L α means sharing the linear layer ( L α ).

Original: Share v+ L α denotes sharing one AdapterBias across all transformer layers.
Revised: Share v+ L α denotes sharing one AdapterBias across all transformer layers.

Original: As can be seen in Table 4, the performance of Share L α stands out among other partial weight-sharing methods, while
Revised: As can be seen in Table 4, the performance of Share L α stands out among other partial weight-sharing methods, while

Original: Share v leads to a poor performance.
Revised: Share v leads to a poor performance.

Original: From the experiments above, we conclude that the linear layer ( L α ) captures general task information by learning the weights of the bias for different tokens.
Revised: From the experiments above, we conclude that the linear layer ( L α ) captures general task information by learning the weights of the bias for different tokens.

Original: Thus, sharing L α across all layers results in better performance compared to other components.
Revised: Thus, sharing L α across all layers results in better performance compared to other components.

Original: The vector module ( v ) in AdapterBias aims to learn local information in each transformer layer.
Revised: The vector module ( v ) in AdapterBias aims to learn local information in each transformer layer.

Original: If v among different transformer layers are shared, the performance drops dramatically.
Revised: If v among different transformer layers are shared, the performance drops dramatically.

Original: This might be due to a failure of v to learn general information which can be adapted to each individual transformer layer.[Table] [Table]
Revised: This might be due to a failure of v to learn general information which can be adapted to each individual transformer layer.

Original: 4.6. L 0 -norm regularization in AdapterBias
Revised: 4.6. L 0 -norm regularization in AdapterBias

Original: We observed that many of the trained parametersin L α have values that are extremely close to zeroafter tuning on downstream tasks, which mightcause redundancy of the parameters.
Revised: We observed that many of the trained parameters in L α have values that are extremely close to zero after tuning on downstream tasks, which might cause redundancy of the parameters.

Original: To furtherencourage the sparsity of AdapterBias, we add L 0norm regularization to L α during the training stage.
Revised: To further encourage the sparsity of AdapterBias, we add L 0 - [Table] [Table]

Original: In Table 5, we use BERT-base (BB) and BERTlarge (BL) for the PLM.
Revised: In Table 5, we use BERT-base (BB) and BERTlarge (BL) as the PLMs.

Original: We compare the performance of ﬁne-tuning, the original AdapterBias, andthe one trained with L 0 -norm regularization.
Revised: We compare the performance of ﬁne-tuning, the original AdapterBias, and the one trained with L 0 -norm regularization.

Original: Theexperiment shows that adding L 0 -norm regularization during the training step improves the performance on 7 out of 9 tasks in BERT-base models.
Revised: The experiment shows that adding L 0 -norm regularization during the training step improves the performance on 7 out of 9 tasks in BERT-base models.

Original: However, the performance did not improve whenapplied to BERT-large models.
Revised: However, the performance did not improve when applied to BERT-large models.

Original: As for the parameter efﬁciency of applying L 0 -norm penalty, thelinear layer ( L α ) with L 0 -norm penalty saves about17% parameter on average compared to the original
Revised: As for the parameter efﬁciency of applying L 0 -norm penalty, the linear layer ( L α ) with L 0 -norm penalty saves about 17% parameter on average compared to the original

Original: AdapterBias.
Revised: AdapterBias.

Original: The details of the reduced parametersof each task are shown in Appendix A.3.
Revised: The details of the reduced parameters of each task are shown in Appendix A.3.

Original:  What AdapterBias learns
Revised: 4. What AdapterBias learns

Original: AdapterBias has good interpretability due to itssimplicity.
Revised: AdapterBias has good interpretability due to its simplicity.

Original: Compared to our similar work BitFit (Ben Zaken et al., 2021), where the shifts areidentical for all tokens, AdapterBias adds tokendependent shifts to the output representation.
Revised: Compared to the similar work BitFit (Ben Zaken et al., 2021), where the shifts are identical for all tokens, AdapterBias adds tokendependent shifts to the output representation.

Original: Byobserving these token-dependent shifts, we analyzewhat AdapterBias learns when adapting to downstream tasks.
Revised: By observing these token-dependent shifts, we analyze what AdapterBias learns when adapting to downstream tasks.

Original:  Average representation shifting intransformer layers
Revised: 4.7. Average representation shifting in transformer layers

Original: In light of the works of Liu et al. (2019a); Tenney et al.
Revised: In light of the works of Liu et al. (2019a); Tenney et al.

Original: Kovaleva et al.
Revised: Kovaleva et al.

Original: (2019), whichshow that different information is being encodedby different transformer layers of PLMs.
Revised: (2019), which show that different information is being encoded by different transformer layers of PLMs.

Original: We assume that AdapterBias provides different representation shifts to the transformer layers throughtask-speciﬁc ﬁne-tuning.
Revised: We assume that AdapterBias provides different repre- sentation shifts to the transformer layers through task-speciﬁc ﬁne-tuning.

Original: In AdapterBias, the linear layer ( L α ) produces aweights vector α for representation shifts, therefore,the average absolute value of vector α can give us alook at the shifting amount in the transformer layerswhen adapting to downstream tasks.
Revised: In AdapterBias, the linear layer ( L α ) produces a weights vector α for representation shifts, therefore, the average absolute value of vector α can give us a look at the shifting amount in the transformer layers when adapting to downstream tasks.

Original: In Figure 5,the layers are ordered from lower to upper.
Revised: In Figure 5, the layers are ordered from lower to upper.

Original: Fromthe experimental result, we ﬁnd that the weightin each layer is considerably different in differenttasks in general.
Revised: From the experimental result, we ﬁnd that the weight in each layer is considerably different in different tasks in general.

Original: CoLA (Warstadt et al., 2019) is a syntactic taskthat consists of English acceptability judgmentsin the GLUE benchmark.
Revised: CoLA (Warstadt et al., 2019) is a syntactic task that consists of English acceptability judgments in the GLUE benchmark.

Original: As shown in Figure 5,its average shift at the ninth layer is the highestamong all layers, which is quite different from theothers.
Revised: As shown in Figure 5, its average shift at the ninth layer is the highest among all layers, which is quite different from the others.

Original: We speculate that the ninth layer has theability to extract the syntactic information, leading
Revised: We speculate that the ninth layer has the ability to extract the syntactic information, leading

Original: AdapterBias to add the largest shift in this layer.
Revised: AdapterBias to add the largest shift in this layer.

Original: Our experiment has a similar observation with thework of Jawahar et al.Jawahar et al.
Revised: Our experiment has a similar observation with the work of Jawahar et al.

Original: (2019)also observe on a syntactic task with BShift (Conneau et al., 2018) that the ninth layer of BERTembeds a rich hierarchy of syntactic information.(Jawahar
Revised: They observe on a syntactic task with BShift (Conneau et al., 2018) that the ninth layer of BERT embeds a rich hierarchy of syntactic information.

Original:  et al., 2019)
Revised: (Jawahar et al., 2019)

Original: Moreover, we observe similar distributions between speciﬁc tasks.
Revised: Moreover, we observe similar distributions between speciﬁc tasks.

Original: For instance, RTE (Giampiccolo et al., 2007; Bentivogli et al., 2009) and
Revised: For instance, RTE (Giampiccolo et al., 2007; Bentivogli et al., 2009) and

Original: MNLI (Williams et al., 2017), where both tasksrecognize textual entailment, have higher values inthe upper layers thanthose in the lower ones.
Revised: MNLI (Williams et al., 2017), where both recognize textual entailment, have higher values in the upper layers than the lower ones.

Original: Based on these ﬁndings, we ﬁnd that AdapterBias assigns suitable representation shifts in different tasks.
Revised: Based on these ﬁndings, we ﬁnd that AdapterBias assigns suitable representation shifts in different tasks.

Original: For tasks with similar objectives,
Revised: For tasks with similar objectives,

Original: AdapterBias tends to add similar representationshifts.
Revised: AdapterBias tends to add similar representation shifts.

Original: Which kind of word does L α focus on
Revised: Which kind of word does L α focus on

Original: Since α i represents the weight of the representationshift for i th token in a transformer layer, we canobserve the signiﬁcance of i th token from the summation of α i in all the transformer layers.
Revised: Since α i represents the weight of the representation shift for i th token in a transformer layer, we can observe the signiﬁcance of i th token from the summation of α i in all the transformer layers.

Original: Specialtokens, including [CLS], [SEP], and [PAD], are notincluded for analysis.
Revised: Special tokens, including [CLS], [SEP], and [PAD], are not included for analysis.

Original: We use the validation sets of CoLA and SST-2, and word cloud is used for visualizations.
Revised: We use the validation sets of CoLA and SST-2, and word cloud is used for visualizations.

Original: In Figure 6, we visualize all words in the validation data of CoLA.
Revised: In Figure 6, we visualize all words in the validation data of CoLA.

Original: The result shows that AdapterBias focuses more on reﬂexive pronouns, such as yourself, himself, and myself.
Revised: The result shows that AdapterBias focuses more on reﬂexive pronouns, such as yourself, himself, and myself.

Original: This is because there are many incorrect sentences with misused reﬂexive pronouns, such as "He washed yourself."
Revised: This is because there are many incorrect sentences with misused reﬂexive pronouns, such as "He washed yourself."

Original: In Figure 7, we visualize all words in the validation data of SST-2.
Revised: In Figure 7, we visualize all words in the valida- tion data of SST-2.

Original: The result shows that AdapterBias focuses more on adjectives, such as "bad", "awful", and "worst".
Revised: The result shows that AdapterBias focuses more on adjectives, such as "bad", "awful", and "worst".

Original: SST-2 is a binary sentiment analysis dataset, which classiﬁes movie reviews into positive and negative classes.
Revised: SST-2 is a binary sentiment analysis dataset, which classiﬁes movie reviews into positive and negative classes.

Original: AdapterBias learns that adjectives often constitute a crucial factor in sentiment analysis during tuning, and adds larger shifts to these adjective tokens.
Revised: AdapterBias learns that adjectives often constitute a crucial factor in sentiment analysis during tuning, and adds larger shifts to these adjective tokens.

Original:  Conclusion
Revised:  Conclusion

Original: In this study, we present AdapterBias.
Revised: In this study, we present AdapterBias.

Original: By adding token-dependent representation shifts to the PLM,
Revised: By adding token-dependent representation shifts to the PLM,

Original: AdapterBias shows competitive results even though it uses far fewer parameters than the existing methods.
Revised: AdapterBias shows competitive results even though it uses far fewer parameters than the existing methods.

Original: Through extensive experiments, not only does AdapterBias reaches competitive results on the
Revised: Through extensive experiments, not only does AdapterBias reach competitive results on the

Original: GLUE benchmark, butit also obtains good performance on small-to-medium datasets.
Revised: GLUE benchmark, but also obtain good performance on small-to-medium datasets.

Original: In addition, we demonstrate the robustness of AdapterBias to different PLMs.
Revised: In addition, we demonstrate the robustness of AdapterBias to different PLMs.

Original: Finally, we provide analysis on what AdapterBias learns by comparing α , the weights of representation shift for different tokens, ﬁnding it has the ability to identify task-speciﬁc information.
Revised: Finally, we provide analysis on what AdapterBias learns by comparing α , the weights of representation shift for different tokens, ﬁnding AdapterBias has the ability to identify taskspeciﬁc information.

Original: Our study overturns previous architectures of adapters by proposing a simple adapter that can produce suitable representation shifts for different tokens.
Revised: Our study is different from the previous architectures of adapters by proposing a simple adapter that can produce suitable representation shifts for different tokens.

Original: A AppendixA. Training Details
Revised: A Appendix A. Training Details

Original: We train our model on Pytorch.
Revised: We train our model on Pytorch.

Original: The training detailsare shown in Table A.
Revised: The training details are shown in Table A.

Original: In addition, the bottleneckof Adapters (Houlsby et al., 2019) and is 32.
Revised: In addition, the bottleneck of Adapters (Houlsby et al., 2019) and is 32.

Original: A. L 0 -norm regularization in AdapterBias
Revised: A. L 0 -norm regularization in AdapterBias

Original: In Table B, we report the remain parameter of utilizing L 0 -norm regularization compared with theoriginal AdapterBias. 
Revised: In Table B, we report the remaining parameters of utilizing L 0 -norm regularization compared with the original AdapterBias. BERT-base (BB) and

Original: BERT-base (BB) and BERTlarge (BL) are used as PLMs.
Revised: BERT-large (BL) are used as PLMs.

Original: A. 
Revised: A.

Original: The direction of representation shifts indifferent tasks
Revised: The direction of representation shifts in different tasks

Original: Different from BitFit (Ben Zaken et al. , 2021),where all the representation shifts are identicalwithin one task, AdapterBias produces differentweights for the shift based on each token.
Revised: Different from BitFit (Ben Zaken et al., 2021), where all the representation shifts are identical within one task, AdapterBias produces different weights for the shift based on each token.

Original: In thissection, we compare the transformed tokens in
Revised: In this section, we compare the transformed tokens in

Original: AdapterBias and BitFit.
Revised: AdapterBias and BitFit.

Original: We utilize PCA (Jolliffe,2002) to reduce the dimension of the vectors.
Revised: We utilize PCA (Jolliffe, 2002) to reduce the dimension of the vectors.

Original: In
Revised: In

Original: Figure A, we input ﬁve sentences from the evaluation set of SST-2.
Revised: Figure A, we input ﬁve sentences from the evaluation set of SST-2.

Original: We experiment on the last transformer layer since it has the most obvious shiftscompared to the previous layers. ’
Revised: We experiment on the last transformer layer since it has the most obvious shifts compared to the previous layers. ’

Original: 0’ with lightercolor indicates the representation before shifting,which is the output of the ﬁrst layer normalization.’1’ with darker color is the shifted representation,which is the output of the second layer normalization.
Revised: 0’ with lighter color indicates the representation before shifting, which is the output of the ﬁrst layer normalization. ’1’ with darker color is the shifted representation, which is the output of the second layer normalization.

Original: The color red represents positive sentences,and blue are the negative ones.
Revised: The color red represents positive sentences, and blue are the negative ones.

Original: The result shows that BitFit shifts all tokens towards the same direction regardless of the groundtruth label.
Revised: The result shows that BitFit shifts all tokens towards the same direction regardless of the groundtruth label.

Original: On the other hand, AdapterBias discerns the label of the sentences and thus shifts thetokens of different sentences toward different directions.et al., 2021) and AdapterBias on SST- validation set. ’
Revised: On the other hand, AdapterBias discerns the label of the sentences and thus shifts the tokens of different sentences toward different directions.

Original: with light color means the embedding before shifting. ’

Original: 1’ with dark color means the embedding after shifting.

Original: The color red represents positive sentences, and blue represents negative sentences.

Original: [Table] [Table][Table]
Revised: [Table] [Table]