Original: Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively
Revised: Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively

Original: Abstract
Revised: Abstract

Original: Large-scale pre-trained language models have achieved impressive results on a wide range of downstream tasks recently.
Revised: Large-scale pre-trained language models have achieved impressive results on a wide range of downstream tasks recently.

Original: However, fine-tuning an extremely large-scale pre-trained language model on limited target datasets is often plagued by overfitting and representation degradation.
Revised: However, fine-tuning an extremely large-scale pre-trained language model on limited target datasets is often plagued by overfitting and representation degradation.

Original: In this paper, we propose a Dynamic Parameter Selection (DPS) algorithm for the large-scale pre-trained models during fine-tuning, which adaptively selects a more promising subnetwork to perform staging updates based on gradients of back-propagation.
Revised: In this paper, we propose a Dynamic Parameter Selection (DPS) algorithm for the large-scale pre-trained models during fine-tuning, which adaptively selects a more promising subnetwork to perform staging updates based on gradients of back-propagation.

Original: Experiments on the GLUE benchmark show that DPS outperforms previous fine-tuning methods in terms of overall performance and stability, and consistently achieves better results with variable pre-trained language models.
Revised: Experiments on the GLUE benchmark show that DPS outperforms previous fine-tuning methods in terms of overall performance and stability, and consistently achieves better results with variable pre-trained language models.

Original: In addition, DPS brings a large magnitude of improvement in out-of-domain transferring experiments and low-resource scenarios, which shows that it can maintain stable general contextual features and reduce the representation collapse.
Revised: In addition, DPS brings a large magnitude of improvement in out-of-domain transferring experiments and low-resource scenarios, which shows that it can maintain stable general contextual features and reduce the representation collapse.

Original: We release our code at https://github.com
Revised: We release our code at https://github.com

Original: /DPS .
Revised: /DPS .

Original:  Introduction
Revised:  Introduction

Original: The paradigm of pre-training on a large-scale unlabeled corpus and fine-tuning on task-specific datasets has led to a remarkable breakthrough in the field of Natural Language Processing (NLP)
Revised: The paradigm of pre-training on a large-scale unlabeled corpus and fine-tuning on task-specific datasets has led to a remarkable breakthrough in the field of Natural Language Processing (NLP)

Original: [Devlin et al., 2018, Raffel et al., 2019, Clark et al., 2020,
Revised: [Devlin et al., 2018, Raffel et al., 2019, Clark et al., 2020,

Original: He et al., 2021].
Revised: He et al., 2021].

Original: While the great success has been achieved on various downstream tasks, previous research has pointed out that such approaches are still facing two limitations: ❶ Fine-tuning instability.
Revised: While the great success has been achieved on various downstream tasks, previous research has pointed out that such approaches are still facing two limitations: ❶ Fine-tuning instability.

Original: With the same hyper-parameters, different restarts will lead to huge differences in the performance of pre-trained language models, especially on small datasets [Phang et al., 2018, Dodge et al., 2020, Lee et al., 2020, Zhang et al., 2021, Mosbach et al., 2021].
Revised: With the same hyper-parameters, different restarts will lead to huge differences in the performance of pre-trained language models, especially on small datasets [Phang et al., 2018, Dodge et al., 2020, Lee et al., 2020, Zhang et al., 2021, Mosbach et al., 2021].

Original: ❷ Poor generalizability.
Revised: ❷ Poor generalizability.

Original: Limited by the size of the downstream datasets, aggressive finetuning can lead to catastrophic forgetting and overfitting, which causes the model to perform poorly when transferring to out-of-domain data or other related tasks [Mahabadi et al., 2021, Aghajanyan et al., 2021, Xu et al., 2021].
Revised: Limited by the size of the downstream datasets, aggressive finetuning can lead to catastrophic forgetting and overfitting, which causes the model to perform poorly when transferring to out-of-domain data or other related tasks [Mahabadi et al., 2021, Aghajanyan et al., 2021, Xu et al., 2021].

Original: Therefore, it remains a challenging topic to adapt large-scale pre-trained models to various scenarios while maintaining their stability and generalizability to the fullest.
Revised: Therefore, it remains a challenging topic to adapt large-scale pre-trained models to various scenarios while maintaining their stability and generalizability to the fullest.

Original: Optimizing a subnetwork of pre-trained models has been proved to be an effective approach to improve stability or mitigate overfitting, such as Mixout [Lee et al., 2020] and CHILD-TUNING D
Revised: Optimizing a subnetwork of pre-trained models has been proved to be an effective approach to improve stability or mitigate overfitting, such as Mixout [Lee et al., 2020] and CHILD-TUNING D

Original: [Xu et al., 2021].
Revised: [Xu et al., 2021].

Original: Mixout utilizes pre-trained weights to randomly replace some parameters with a probability of p and updates only the subnetwork composed of the parameters without replacement.
Revised: Mixout utilizes pre-trained weights to randomly replace some parameters with a probability of p and updates only the subnetwork composed of the parameters without replacement.

Original: CHILD-TUNING D computes the importance of parameters based on all training samples’ gradients before training and chooses an unchanged subnetwork composed of important parameters for updating.
Revised: CHILD-TUNING D computes the importance of parameters based on all training samples’ gradients before training and chooses an unchanged subnetwork composed of important parameters for updating.

Original: Despite their great performance, we argue that (i) choosing subnetworks randomly (Mixout) ignores the importance of parameters; (ii) choosing an unchanged subnetwork and ignoring the update of some parameters (CHILD-TUNING D ) may affect the model’s perception of downstream tasks.
Revised: Despite their great performance, we argue that (i) choosing subnetworks randomly (Mixout) ignores the importance of parameters; (ii) choosing an unchanged subnetwork and ignoring the update of some parameters (CHILD-TUNING D ) may affect the model’s perception of downstream tasks.

Original: Moreover, accumulating all training samples’ gradients calculated by back-propagation outside training would induce a heavy additional computational overhead.
Revised: Moreover, accumulating all training samples’ gradients calculated by back-propagation outside training would induce a heavy additional computational overhead.

Original: To overcome the above two restrictions of existing subnetwork optimizations, we propose a Dynamic
Revised: To overcome the above two restrictions of existing subnetwork optimizations, we propose a Dynamic

Original: Parameter Selection (DPS) algorithm to fine-tune pre-trained models adaptively.
Revised: Parameter Selection (DPS) algorithm to fine-tune pre-trained models adaptively.

Original: Overall, DPS consists of a cyclic two-stage updating strategy: Stage I accumulates in-batch gradients of updated parameters during fine-tuning; Stage II utilizes accumulated values of Stage I to derive a stage-specific subnetwork composed of important parameters to update while keeping non-subnetwork constant.
Revised: Overall, DPS consists of a cyclic two-stage updating strategy: Stage I accumulates in-batch gradients of updated parameters during fine-tuning; Stage II utilizes accumulated values of Stage I to derive a stage-specific subnetwork composed of important parameters to update while keeping non-subnetwork constant.

Original: Note that both two stages consist of several iterations and the whole fine-tuning process will repeat several two-stage updating processes until the model convergence.
Revised: Note that both two stages consist of several iterations and the whole fine-tuning process will repeat several two-stage updating processes until the model convergence.

Original: Compared to previous subnetwork optimizations, we summarize the advantages of DPS as follows: (i) DPS can adaptively choose out more promising subnetworks based on the changing parameter importance while paying attention to more parameters possibilities, which enables better optimization of the model.
Revised: Compared to previous subnetwork optimizations, we summarize the advantages of DPS as follows: (i) DPS can adaptively choose out more promising subnetworks based on the changing parameter importance while paying attention to more parameters possibilities, which enables better optimization of the model.

Original: (ii) DPS can effectively combine two processes: fine-tuning and deriving subnetworks, which greatly reduces the additional heavy computational costs introduced by separating above two processes like CHILD-TUNING D .
Revised: (ii) DPS can effectively combine two processes: fine-tuning and deriving subnetworks, which greatly reduces the additional heavy computational costs introduced by separating above two processes like CHILD-TUNING D .

Original: In details, we propose two variants, DPS Dense and DPS Mix, to optimize the fine-tuning process.
Revised: In details, we propose two variants, DPS Dense and DPS Mix, to optimize the fine-tuning process.

Original: DPS Dense aims at optimizing the full network update process.
Revised: DPS Dense aims at optimizing the full network update process.

Original: In Stage I, DPS Dense updates the whole network while accumulating gradients’ values.
Revised: In Stage I, DPS Dense updates the whole network while accumulating gradients’ values.

Original: In Stage II, DPS Dense updates the stagespecific subnetwork composed of important parameters measured by ranking the accumulated values of Stage I.
Revised: In Stage II, DPS Dense updates the stagespecific subnetwork composed of important parameters measured by ranking the accumulated values of Stage I.

Original: In contrast, DPS Mix is designed to optimize the stochastic parameter update process.
Revised: In contrast, DPS Mix is designed to optimize the stochastic parameter update process.

Original: In
Revised: In

Original: Stage I, DPS Mix randomly replaces some outgoing parameters of the neural network with pre-trained parameters and accumulates in-batch gradients of the remaining updated parameters.
Revised: Stage I, DPS Mix randomly replaces some outgoing parameters of the neural network with pre-trained parameters and accumulates in-batch gradients of the remaining updated parameters.

Original: In Stage II,
Revised: In Stage II,

Original: DPS Mix applies a frequency-penalized selection strategy, which utilizes updated gradients’ average value and an exponential penalty factor negatively correlated with update frequency to choose out the stage-related important parameters for updating.
Revised: DPS Mix applies a frequency-penalized selection strategy, which utilizes updated gradients’ average value and an exponential penalty factor negatively correlated with update frequency to choose out the stage-related important parameters for updating.

Original: It adaptively chooses a more promising subnetwork in stages while avoiding deviations from the pre-trained weights due to aggressive optimization.
Revised: It adaptively chooses a more promising subnetwork in stages while avoiding deviations from the pre-trained weights due to aggressive optimization.

Original: We re-evaluate several state-of-the-art fine-tuning regularization approaches [Lee et al., 2020, Aghajanyan et al., 2021, Zhang et al., 2021, Xu et al., 2021, Wu et al., 2021] on GLUE Benchmark and find that DPS outperforms prior approaches by 0 .
Revised: We re-evaluate several state-of-the-art fine-tuning regularization approaches [Lee et al., 2020, Aghajanyan et al., 2021, Zhang et al., 2021, Xu et al., 2021, Wu et al., 2021] on GLUE Benchmark and find that DPS outperforms prior approaches by 0 .

Original: 44 ∼ 1 .
Revised: 44 ∼ 1 .

Original: 33 points gains and 0 .
Revised: 33 points gains and 0 .

Original: 29 ∼ 0 .
Revised: 29 ∼ 0 .

Original: 80 std (standard deviation) drops.
Revised: 80 std (standard deviation) drops.

Original: Moreover, DPS brings an improvement in out-of-domain transferring experiments/low-resource scenarios with up to 1.86/3.27 average score, indicating that DPS can obtain excellent generalization ability.
Revised: Moreover, DPS brings an improvement in out-of-domain transferring experiments/low-resource scenarios with up to 1.86/3.27 average score, indicating that DPS can obtain excellent generalization ability.

Original: In extensive experiments, we further explore DPS in the following four settings: (i) subnetwork size; (ii) various pre-trained models; (iii) a sufficient number of training examples; (iv) time usage.
Revised: In extensive experiments, we further explore DPS in the following four settings: (i) subnetwork size; (ii) various pre-trained models; (iii) a sufficient number of training examples; (iv) time usage.

Original: Based on the above extensive experiments, we observe that (i)
Revised: Based on the above extensive experiments, we observe that (i)

Original: DPS can better capture task-relevant parameters across a variety of subnetwork sizes than Mixout and CHILD-TUNING D ; (ii) DPS can consistently bring a stable improvement in various pre-trained models; (iii) our proposed DPS is helpful to model performance in fine-tuning with a sufficient number of training examples; (iv) DPS is a more time-efficient approach than several fine-tuning methods.
Revised: DPS can better capture task-relevant parameters across a variety of subnetwork sizes than Mixout and CHILD-TUNING D ; (ii) DPS can consistently bring a stable improvement in various pre-trained models; (iii) our proposed DPS is helpful to model performance in fine-tuning with a sufficient number of training examples; (iv) DPS is a more time-efficient approach than several fine-tuning methods.

Original:  Approach 2.1 Background and Related Work
Revised:  Approach 2.1 Background and Related Work

Original: We first start to introduce the paradigm of fine-tuning the subnetwork by giving formulations of forward and back propagation during standard fine-tuning, Mixout, and CHILD-TUNING D .
Revised: We first start to introduce the paradigm of fine-tuning the subnetwork by giving formulations of forward and back propagation during standard fine-tuning, Mixout, and CHILD-TUNING D .

Original: We will describe all formulations for Stochastic Gradient Descent (SGD) iterations.
Revised: We will describe all formulations for Stochastic Gradient Descent (SGD) iterations.

Original: We assume W ( t ) =
Revised: We assume W ( t ) =

Original: ( W ( t ) 1 , · · · , W ( t ) n ) , where W ( t ) i represents parameters of the i -th neuron at the t -th iteration.
Revised: ( W ( t ) 1 , · · · , W ( t ) n ) , where W ( t ) i represents parameters of the i -th neuron at the t -th iteration.

Original: Standard fine-tuning formulates model parameters as follows:
Revised: Standard fine-tuning formulates model parameters as follows:

Original: [Equation]
Revised: [Equation]

Original: where L represents in-batch loss; η represents learning rate.
Revised: where L represents in-batch loss; η represents learning rate.

Original: Differ from fine-tuning whole network,
Revised: Differ from fine-tuning whole network,

Original: Mixout first derives a mask matrix M ( t ) =
Revised: Mixout first derives a mask matrix M ( t ) =

Original: ( M ( t ) 1 , · · · , M ( t ) n ) , where M ( t ) i is same-sized as W ( t ) i and
Revised: ( M ( t ) 1 , · · · , M ( t ) n ) , where M ( t ) i is same-sized as W ( t ) i and

Original: M ( t )
Revised: M ( t )

Original: i ’s are mutually independent Bernoulli (1 − p ) for all i and t .
Revised: i ’s are mutually independent Bernoulli (1 − p ) for all i and t .

Original: Then Mixout replaces parameters corresponding to zero values in M ( t ) with pre-trained parameters and only update the subnetwork composed of remaining parameters.
Revised: Then Mixout replaces parameters corresponding to zero values in M ( t ) with pre-trained parameters and only update the subnetwork composed of remaining parameters.

Original: Its formulation is as follows:
Revised: Its formulation is as follows:

Original: [Equation]
Revised: [Equation]

Original: where I is full-one matrix and is same-sized as M ( t ) ; W (0) represents the parameter matrix of the pre-trained model; the larger p is, the smaller the subnetwork.
Revised: where I is full-one matrix and is same-sized as M ( t ) ; W (0) represents the parameter matrix of the pre-trained model; the larger p is, the smaller the subnetwork.

Original: Besides selecting a subnetwork randomly, CHILD-TUNING D has explored fine-tuning subnetwork composed of measured important parameters.
Revised: Besides selecting a subnetwork randomly, CHILD-TUNING D has explored fine-tuning subnetwork composed of measured important parameters.

Original: Considering Fisher Information [Fisher, 1922] serves as a good way of measuring the amount of information that a random variable carries about a parameter of the distribution
Revised: Considering Fisher Information [Fisher, 1922] serves as a good way of measuring the amount of information that a random variable carries about a parameter of the distribution

Original: [Tu et al., 2016] and has been widely used in modern machine learning and deep learning [Achille et al., 2019, Kirkpatrick et al., 2017, Amari, 1996, Pascanu and Bengio, 2013, Singh and Alistarh, 2020, Crowley et al., 2018, Martens, 2014, Theis et al., 2018], CHILD-TUNING D utilizes Fisher Information to compute the importance of parameters based on all training samples before training, then derives an unchanged subnetwork composed of important parameters to update.
Revised: [Tu et al., 2016] and has been widely used in modern machine learning and deep learning [Achille et al., 2019, Kirkpatrick et al., 2017, Amari, 1996, Pascanu and Bengio, 2013, Singh and Alistarh, 2020, Crowley et al., 2018, Martens, 2014, Theis et al., 2018], CHILD-TUNING D utilizes Fisher Information to compute the importance of parameters based on all training samples before training, then derives an unchanged subnetwork composed of important parameters to update.

Original: Its formulation is as follows:
Revised: Its formulation is as follows:

Original: [Equation]
Revised: [Equation]

Original: (4) where sort ( A ) p represents the value of p percentile in matrix A after sorting in ascending order;
Revised: (4) where sort ( A ) p represents the value of p percentile in matrix A after sorting in ascending order;

Original: M CT D is an unchanged mask matrix, which determines the subnetwork used for updates throughout the fine-tuning process.
Revised: M CT D is an unchanged mask matrix, which determines the subnetwork used for updates throughout the fine-tuning process.

Original: Since their introduction that aggressive fine-tuning will lead to instability and overfitting, especially on small datasets[Devlin et al., 2018, Phang et al., 2018], various fine-tuning methods have been proposed to solve the above problems, except for subnetwork optimizations.
Revised: Since their introduction that aggressive fine-tuning will lead to instability and overfitting, especially on small datasets[Devlin et al., 2018, Phang et al., 2018], various fine-tuning methods have been proposed to solve the above problems, except for subnetwork optimizations.

Original: Some of them utilize output representations or inject noise into input to regularize models [Mahabadi et al., 2021, Aghajanyan et al., 2021, Wu et al., 2021], while others adjust the weights of top layers of Transformer or
Revised: Some of them utilize output representations or inject noise into input to regularize models [Mahabadi et al., 2021, Aghajanyan et al., 2021, Wu et al., 2021], while others adjust the weights of top layers of Transformer or

Original: BERTAdam optimizer to make the pre-trained models more task-relevant.
Revised: BERTAdam optimizer to make the pre-trained models more task-relevant.

Original: [Zhang et al., 2021,
Revised: [Zhang et al., 2021,

Original: Mosbach et al., 2021].
Revised: Mosbach et al., 2021].

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 2.2 Overview of DPS
Revised: 2.2 Overview of DPS

Original: DPS consists of a cyclic two-stage updating strategy: Stage I accumulates in-batch gradients of updated parameters during fine-tuning; Stage II utilizes accumulated values of Stage I to derive a stage-specific subnetwork composed of important parameters to update while keeping non-subnetwork constant.
Revised: DPS consists of a cyclic two-stage updating strategy: Stage I accumulates in-batch gradients of updated parameters during fine-tuning; Stage II utilizes accumulated values of Stage I to derive a stage-specific subnetwork composed of important parameters to update while keeping non-subnetwork constant.

Original: We provide a pseudo-code of DPS in Algorithm 1.
Revised: We provide a pseudo-code of DPS in Algorithm 1.

Original: We argue that fine-tuning a dynamic promising subnetwork while paying attention to more parameters possibilities can align all parameters more relevant for a specific downstream task, which in turn better unlocks the potential of the model.
Revised: We argue that fine-tuning a dynamic promising subnetwork while paying attention to more parameters possibilities can align all parameters more relevant for a specific downstream task, which in turn better unlocks the potential of the model.

Original: To select a promising subnetwork adaptively, we propose two enhanced fine-tuning strategies: DPS
Revised: To select a promising subnetwork adaptively, we propose two enhanced fine-tuning strategies: DPS

Original: Dense (in Sec. 2.3) and DPS Mix (in Sec. 2.4), which respectively optimize the fine-tuning processes of full network update and stochastic network update.
Revised: Dense (in Sec. 2.3) and DPS Mix (in Sec. 2.4), which respectively optimize the fine-tuning processes of full network update and stochastic network update.

Original: 2.3 DPS Dense
Revised: 2.3 DPS Dense

Original: We firstly explore the DPS in full network update called DPS Dense.
Revised: We firstly explore the DPS in full network update called DPS Dense.

Original: Before fine-tuning, DPS Dense initializes a Gradient Accumulation Matrix ( GAM ), which is the same-sized as W (0) .
Revised: Before fine-tuning, DPS Dense initializes a Gradient Accumulation Matrix ( GAM ), which is the same-sized as W (0) .

Original: In Stage I,
Revised: In Stage I,

Original: DPS Dense fine-tunes the whole network as Equation 1 while accumulating the gradients’ square of all parameters at every iteration with the help of GAM ;
Revised: DPS Dense fine-tunes the whole network as Equation 1 while accumulating the gradients’ square of all parameters at every iteration with the help of GAM ;

Original: In Stage II, DPS Dense first ranks GAM and derives a stage-specific mask matrix M DPS filtered by selecting top (1 − p ) highest Fisher
Revised: In Stage II, DPS Dense first ranks GAM and derives a stage-specific mask matrix M DPS filtered by selecting top (1 − p ) highest Fisher

Original: Information of GAM , then utilizes M DPS to update the stage-specific subnetwork composed of important parameters.
Revised: Information of GAM , then utilizes M DPS to update the stage-specific subnetwork composed of important parameters.

Original: we denote the formulation of Stage II as follows:
Revised: we denote the formulation of Stage II as follows:

Original: [Equation]
Revised: [Equation]

Original: The specific update process can be seen in Figure 1.
Revised: The specific update process can be seen in Figure 1.

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: 2.4 DPS Mix
Revised: 2.4 DPS Mix

Original: Considering stochastic parameter update is common to deploy in fine-tuning pre-trained models, we also propose an alternative to optimize this fine-tuning process called DPS Mix.
Revised: Considering stochastic parameter update is common to deploy in fine-tuning pre-trained models, we also propose an alternative to optimize this fine-tuning process called DPS Mix.

Original: Before fine-tuning,
Revised: Before fine-tuning,

Original: DPS Mix needs to initialize another Frequency Accumulation Matrix ( FAM ) besides GAM , which is used to store the frequency of every parameter being updated in Stage I.
Revised: DPS Mix needs to initialize another Frequency Accumulation Matrix ( FAM ) besides GAM , which is used to store the frequency of every parameter being updated in Stage I.

Original: In Stage I, DPS Mix derives a mask matrix M ( t ) like Mixout, and then fine-tunes the subnetwork determined by M ( t ) while replacing non-subnetwork as pre-trained weights as Equation 2.
Revised: In Stage I, DPS Mix derives a mask matrix M ( t ) like Mixout, and then fine-tunes the subnetwork determined by M ( t ) while replacing non-subnetwork as pre-trained weights as Equation 2.

Original: In Stage II, DPS Mix derives
Revised: In Stage II, DPS Mix derives

Original: M DPS according to a frequency-penalized selection strategy as follows:
Revised: M DPS according to a frequency-penalized selection strategy as follows:

Original: [Equation]
Revised: [Equation]

Original: As Equation7 illustrates, DPS Mix follows the ratio of GAM and FAM as the basis for deriving the
Revised: As Equation7 illustrates, DPS Mix follows the ratio of GAM and FAM as the basis for deriving the

Original: M DPS .
Revised: M DPS .

Original: During the random selection of Stage I, considering that each parameter may be updated at an inconsistent frequency and parameters with more frequent updates may have smaller potential,
Revised: During the random selection of Stage I, considering that each parameter may be updated at an inconsistent frequency and parameters with more frequent updates may have smaller potential,

Original: DPS Mix also multiplies an exponential penalty factor negatively correlated with update frequency in addition to the ranking selection, to give more chances to parameters with fewer updates.
Revised: DPS Mix also multiplies an exponential penalty factor negatively correlated with update frequency in addition to the ranking selection, to give more chances to parameters with fewer updates.

Original: Since this penalty factor may introduce some fluctuations, we grid search the boundary and decide to exclude this factor when us is less than 50.
Revised: Since this penalty factor may introduce some fluctuations, we grid search the boundary and decide to exclude this factor when us is less than 50.

Original: After obtaining the M DPS , DPS Mix formulates model parameters as follows:
Revised: After obtaining the M DPS , DPS Mix formulates model parameters as follows:

Original: [Equation]
Revised: [Equation]

Original: where W ′ are same-valued as W ((2 k +1) us +1)
Revised: where W ′ are same-valued as W ((2 k +1) us +1)

Original: ; Note that the subnetwork size of Stage I is the same-sized as Stage II’s in DPS Mix.
Revised: ; Note that the subnetwork size of Stage I is the same-sized as Stage II’s in DPS Mix.

Original: The specific update can be seen in Figure 1.
Revised: The specific update can be seen in Figure 1.

Original: In terms of implementation details, we apply DPS Dense and DPS Mix to specific layers.
Revised: In terms of implementation details, we apply DPS Dense and DPS Mix to specific layers.

Original:  Experiments 3.1 Datasets
Revised:  Experiments 3.1 Datasets

Original: GLUE benchmark.
Revised: GLUE benchmark.

Original: Following previous studies [Lee et al., 2020, Dodge et al., 2020, Zhang et al., 2021], we conduct a series of extensive experiments on eight datasets from the GLUE benchmark [Wang et al., 2019].
Revised: Following previous studies [Lee et al., 2020, Dodge et al., 2020, Zhang et al., 2021], we conduct a series of extensive experiments on eight datasets from the GLUE benchmark [Wang et al., 2019].

Original: The datasets cover four tasks: natural language inference (RTE, QNLI, MNLI), paraphrase detection (MRPC, QQP), linguistic acceptability (CoLA), and sentiment classification (SST-2).
Revised: The datasets cover four tasks: natural language inference (RTE, QNLI, MNLI), paraphrase detection (MRPC, QQP), linguistic acceptability (CoLA), and sentiment classification (SST-2).

Original: Appendix B provides dataset statistics and a brief description of each dataset.
Revised: Appendix B provides dataset statistics and a brief description of each dataset.

Original: Since the test set can only be submitted online for a very limited times, we follow several previous studies [Phang et al., 2018, Lee et al., 2020, Dodge et al., 2020, Aghajanyan et al., 2021, Zhang et al., 2021, Xu et al., 2021] that fine-tune on the training sets and report the results on the development sets.
Revised: Since the test set can only be submitted online for a very limited times, we follow several previous studies [Phang et al., 2018, Lee et al., 2020, Dodge et al., 2020, Aghajanyan et al., 2021, Zhang et al., 2021, Xu et al., 2021] that fine-tune on the training sets and report the results on the development sets.

Original: NLI Datasets.
Revised: NLI Datasets.

Original: We evaluate and probe the generalization ability of DPS on several Natural Language
Revised: We evaluate and probe the generalization ability of DPS on several Natural Language

Original: Inference (NLI) tasks, including SNLI [Bowman et al., 2015], MNLI [Williams et al., 2018], MNLIM [Williams et al., 2018], RTE [Bentivogli et al., 2009], SICK [Marelli et al., 2014] and SciTail [Khot et al., 2018].
Revised: Inference (NLI) tasks, including SNLI [Bowman et al., 2015], MNLI [Williams et al., 2018], MNLIM [Williams et al., 2018], RTE [Bentivogli et al., 2009], SICK [Marelli et al., 2014] and SciTail [Khot et al., 2018].

Original: All tasks are reported by Accuracy.
Revised: All tasks are reported by Accuracy.

Original: Since the target datasets have different label spaces, during the evaluation, we map predictions to each target dataset’s space (in Appendix E).
Revised: Since the target datasets have different label spaces, during the evaluation, we map predictions to each target dataset’s space (in Appendix E).

Original: 3.2 Experimental Setup
Revised: 3.2 Experimental Setup

Original: We use the pre-trained models and codes provided by HuggingFace *
Revised: We use the pre-trained models and codes provided by HuggingFace *

Original: Wolf et al.
Revised: Wolf et al.

Original: Appendix C provides specific hyper-parameter details, and unless noted otherwise, we follow the default hyperparameter setup of HuggingFace.
Revised: Appendix C provides specific hyper-parameter details, and unless noted otherwise, we follow the default hyperparameter setup of HuggingFace.

Original: 3.3 Comparing Prior Methods for Few-Sample BERT Fine-tuning 3.3.1 Baseline
Revised: 3.3 Comparing Prior Methods for Few-Sample BERT Fine-tuning 3.3.1 Baseline

Original: Mixout [Lee et al., 2020] is a stochastic parameter update technique based on the Bernoulli distribution for preventing catastrophic forgetting during fine-tuning.
Revised: Mixout [Lee et al., 2020] is a stochastic parameter update technique based on the Bernoulli distribution for preventing catastrophic forgetting during fine-tuning.

Original: It randomly replaces the model parameters with pre-trained parameters with a probability of p .
Revised: It randomly replaces the model parameters with pre-trained parameters with a probability of p .

Original: R3F [Aghajanyan et al., 2021] is motivated by trust region theory and injects parametric noise to the embedding of the pre-trained representations.
Revised: R3F [Aghajanyan et al., 2021] is motivated by trust region theory and injects parametric noise to the embedding of the pre-trained representations.

Original: R-Dropout [Wu et al., 2021] minimizes the bidirectional KL-divergence between the output distributions of two sub models sampled by Dropout.
Revised: R-Dropout [Wu et al., 2021] minimizes the bidirectional KL-divergence between the output distributions of two sub models sampled by Dropout.

Original: CHILD-TUNING D
Revised: CHILD-TUNING D

Original: [Xu et al., 2021] utilizes the downstream task data to detect the most task-related parameters as the child network and freezes the parameters in the non-child network to their pre-trained weights during fine-tuning.
Revised: [Xu et al., 2021] utilizes the downstream task data to detect the most task-related parameters as the child network and freezes the parameters in the non-child network to their pre-trained weights during fine-tuning.

Original: Re-init
Revised: Re-init

Original: [Zhang et al., 2021] re-initializes the pooler layers and the top L BERT Transformer layers using the original BERT initialization, N (0,0.02 2 ).
Revised: [Zhang et al., 2021] re-initializes the pooler layers and the top L BERT Transformer layers using the original BERT initialization, N (0,0.02 2 ).

Original: 3.3.2 Results
Revised: 3.3.2 Results

Original: In this section, we compare DPS with a variety of fine-tuning regularization approaches on
Revised: In this section, we compare DPS with a variety of fine-tuning regularization approaches on

Original: BERT LARGE , following [Lee et al., 2020, Zhang et al., 2021, Xu et al., 2021].
Revised: BERT LARGE , following [Lee et al., 2020, Zhang et al., 2021, Xu et al., 2021].

Original: The hyper-parameter search spaces of different fine-tuning regularization methods are supplemented in Appendix D. Tablesummarizes the experimental results.
Revised: The hyper-parameter search spaces of different fine-tuning regularization methods are supplemented in Appendix D. Tablesummarizes the experimental results.

Original: Firstly, we can notice that Mixout and CHILD-TUNING D achieve better results than vanilla fine-tuning, and perform nearly the best baseline Re-init, which proves that fine-tuning a subnetwork while utilizing pre-trained weights is a competitive approach to regularize the model.
Revised: Firstly, we can notice that Mixout and CHILD-TUNING D achieve better results than vanilla fine-tuning, and perform nearly the best baseline Re-init, which proves that fine-tuning a subnetwork while utilizing pre-trained weights is a competitive approach to regularize the model.

Original: Secondly, our proposed straightforward DPS Dense outperforms all baselines on average, which demonstrates that fine-tuning subnetworks adaptively is more effective.
Revised: Secondly, our proposed straightforward DPS Dense outperforms all baselines on average, which demonstrates that fine-tuning subnetworks adaptively is more effective.

Original: Since DPS accumulates in-batch gradients to derive important sub-networks during fine-tuning, we also consider
Revised: Since DPS accumulates in-batch gradients to derive important sub-networks during fine-tuning, we also consider

Original: DPS to be a more time-efficient approach than CHILD-TUNING D , which utilizes Fisher Information to derive important sub-network based on all training samples outside fine-tuning.
Revised: DPS to be a more time-efficient approach than CHILD-TUNING D , which utilizes Fisher Information to derive important sub-network based on all training samples outside fine-tuning.

Original: Thirdly, it is easy to find that DPS Mix achieves better results while further improving stability, especially on CoLA with great std drop.
Revised: Thirdly, it is easy to find that DPS Mix achieves better results while further improving stability, especially on CoLA with great std drop.

Original: This reveals that DPS can be effectively compatible with various fine-tuning strategies, enabling better optimization of the model.
Revised: This reveals that DPS can be effectively compatible with various fine-tuning strategies, enabling better optimization of the model.

Original: [Table]
Revised: [Table]

Original: 3.4 Results on Out-of-Domain Generalization
Revised: 3.4 Results on Out-of-Domain Generalization

Original: Previous research has found that models trained on annotated datasets create superficial shortcut features and do not generalize well to out-of-domain datasets [Belinkov et al., 2019].
Revised: Previous research has found that models trained on annotated datasets create superficial shortcut features and do not generalize well to out-of-domain datasets [Belinkov et al., 2019].

Original: Since DPS possesses strong and stable performance, we also expect DPS can help to discover deeper semantic features and achieve better generalization in transferring to out-of-domain datasets or other related tasks.
Revised: Since DPS possesses strong and stable performance, we also expect DPS can help to discover deeper semantic features and achieve better generalization in transferring to out-of-domain datasets or other related tasks.

Original: In detail, to evaluate out-of-domain generalization, we fine-tune BERT LARGE on mediumsized 8K subsampled MNLI and SNLI datasets respectively, and directly evaluate the fine-tuned models on several NLI datasets, including MNLI, MNLI-M, SICK, SciTail, SNLI, and RTE.
Revised: In detail, to evaluate out-of-domain generalization, we fine-tune BERT LARGE on mediumsized 8K subsampled MNLI and SNLI datasets respectively, and directly evaluate the fine-tuned models on several NLI datasets, including MNLI, MNLI-M, SICK, SciTail, SNLI, and RTE.

Original: The experimental results are shown in Table 2.
Revised: The experimental results are shown in Table 2.

Original: On the models trained on MNLI, DPS Dense/DPS mix outperforms vanilla fine-tuning by 1.16/1.86 average score, and consistently outperforms CHILDTUNING D by 0.32/1.05 average score.
Revised: On the models trained on MNLI, DPS Dense/DPS mix outperforms vanilla fine-tuning by 1.16/1.86 average score, and consistently outperforms CHILDTUNING D by 0.32/1.05 average score.

Original: On the models trained on SNLI, although CHILD-TUNING D only brings a very slight improvement over vanilla fine-tuning, DPS Dense/DPS mix consistently delivers a large improvement of 1.23/0.63 average score.
Revised: On the models trained on SNLI, although CHILD-TUNING D only brings a very slight improvement over vanilla fine-tuning, DPS Dense/DPS mix consistently delivers a large improvement of 1.23/0.63 average score.

Original: All the above statistics can fully verify that DPS can better explore deeper semantics and reduce superficial biases unique to source task than vanilla fine-tuning and CHILD-TUNING D .
Revised: All the above statistics can fully verify that DPS can better explore deeper semantics and reduce superficial biases unique to source task than vanilla fine-tuning and CHILD-TUNING D .

Original: We also conclude that making the model closer to real distribution for the source task may facilitate generalization ability better than freezing some parameters as pre-trained weights like CHILD-TUNING D .
Revised: We also conclude that making the model closer to real distribution for the source task may facilitate generalization ability better than freezing some parameters as pre-trained weights like CHILD-TUNING D .

Original: [Table]
Revised: [Table]

Original: 3.5 Results on Low-Resource Scenarios
Revised: 3.5 Results on Low-Resource Scenarios

Original: To further explore the generalization ability of DPS on extremely small datasets
Revised: To further explore the generalization ability of DPS on extremely small datasets

Original: [Table]
Revised: [Table]

Original: we downsample eight datasets in GLUE to 0.5K and 1K, and fine-tune them all on BERT LARGE .
Revised: we downsample eight datasets in GLUE to 0.5K and 1K, and fine-tune them all on BERT LARGE .

Original: As illustrated in Table 3,
Revised: As illustrated in Table 3,

Original: DPS outperforms vanilla fine-tuning by a large margin almost on all tasks in variable low-resource scenarios, which demonstrates that DPS, as a novel fine-tuning technique, can effectively mitigate the risk of overfitting, exploit model potential, and remove irrelevant features.
Revised: DPS outperforms vanilla fine-tuning by a large margin almost on all tasks in variable low-resource scenarios, which demonstrates that DPS, as a novel fine-tuning technique, can effectively mitigate the risk of overfitting, exploit model potential, and remove irrelevant features.

Original: [Table]
Revised: [Table]

Original: seeds and the best result of each task is bolded.
Revised: seeds and the best result of each task is bolded.

Original:  Analysis and Study 4.1 Effect of Subnetwork Size
Revised:  Analysis and Study 4.1 Effect of Subnetwork Size

Original: [Lee et al., 2020] have explored the hyper-parameter p and verified that Mixout shows great performance at many different probabilities p
Revised: [Lee et al., 2020] have explored the hyper-parameter p and verified that Mixout shows great performance at many different probabilities p

Original: (Note that the larger p , the smaller the subnetwork).
Revised: (Note that the larger p , the smaller the subnetwork).

Original: We also wonder what will happen when DPS probabilities vary, so we plot the validation performance distribution of CoLA and RTE of 10 random seeds when fine-tuning BERT LARGE across different probabilities p ∈ { 0 . 1 , 0 . 2 , 0 .
Revised: We also wonder what will happen when DPS probabilities vary, so we plot the validation performance distribution of CoLA and RTE of 10 random seeds when fine-tuning BERT LARGE across different probabilities p ∈ { 0 . 1 , 0 . 2 , 0 .

Original: 3 , 0 . 4 , 0 . 5 } in Figure 2.
Revised: 3 , 0 . 4 , 0 . 5 } in Figure 2.

Original: Firstly, it is easy to find the overall performance of DPS Dense/DPS Mix is better than CHILD-TUNING D /Mixout across all probabilities, which demonstrates fine-tuning subnetworks of the pre-trained model adaptively can better optimize the model than fine-tuning an unchanged subnetwork or fine-tuning random subnetworks.
Revised: Firstly, it is easy to find the overall performance of DPS Dense/DPS Mix is better than CHILD-TUNING D /Mixout across all probabilities, which demonstrates fine-tuning subnetworks of the pre-trained model adaptively can better optimize the model than fine-tuning an unchanged subnetwork or fine-tuning random subnetworks.

Original: Secondly, compared to vanilla fine-tuning, there is a visible performance drop in CHILD-TUNING D and
Revised: Secondly, compared to vanilla fine-tuning, there is a visible performance drop in CHILD-TUNING D and

Original: Mixout with some probabilities.
Revised: Mixout with some probabilities.

Original: However, DPS Dense/DPS Mix consistently outperforms vanilla fine-tuning across all probabilities.
Revised: However, DPS Dense/DPS Mix consistently outperforms vanilla fine-tuning across all probabilities.

Original: These fully indicate that CHILD-TUNING D and Mixout may fail to capture important parameters in some sizes of the subnetwork, resulting in failed runs (we refer to the average scores lower than vanilla as failed runs).
Revised: These fully indicate that CHILD-TUNING D and Mixout may fail to capture important parameters in some sizes of the subnetwork, resulting in failed runs (we refer to the average scores lower than vanilla as failed runs).

Original: However, Our proposed DPS can better capture more task-relevant parameters, which in turn results in a consistent and stable improvement in all sizes of the subnetwork.
Revised: However, Our proposed DPS can better capture more task-relevant parameters, which in turn results in a consistent and stable improvement in all sizes of the subnetwork.

Original: To further justify the effectiveness of DPS from another perspective, we conduct a case study in Appendix A.
Revised: To further justify the effectiveness of DPS from another perspective, we conduct a case study in Appendix A.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: CC
Revised: CC

Original: [Figure]
Revised: [Figure]

Original: 4.2 Results on Different Pre-Trained Language Models
Revised: 4.2 Results on Different Pre-Trained Language Models

Original: [Figure]
Revised: [Figure]

Original: STS
Revised: STS

Original: [Figure]
Revised: [Figure]

Original: B
Revised: B

Original: Avg
Revised: Avg

Original: [Table]
Revised: [Table]

Original: In this subsection
Revised: In this subsection

Original: [Table]
Revised: [Table]

Original: we conduct experiments on the four tasks in GLUE on five different Pre-trained
Revised: we conduct experiments on the four tasks in GLUE on five different Pre-trained

Original: Language Models (PLMs): BERT LARGE [Devlin et al., 2018], RoBERTa LARGE [Liu et al., 2019],
Revised: Language Models (PLMs): BERT LARGE [Devlin et al., 2018], RoBERTa LARGE [Liu et al., 2019],

Original: BART LARGE [Lewis et al., 2019], ELECTRA LARGE
Revised: BART LARGE [Lewis et al., 2019], ELECTRA LARGE

Original: [Clark et al., 2020] and DeBERTa LARGE [He et al., 2021].
Revised: [Clark et al., 2020] and DeBERTa LARGE [He et al., 2021].

Original: The experimental results are shown in Table 4.
Revised: The experimental results are shown in Table 4.

Original: In addition to the most classical model
Revised: In addition to the most classical model

Original: BERT, DPS consistently achieves better results on both RoBERTa and ELECTRA trained on larger pre-trained corpus and better pre-training tasks.
Revised: BERT, DPS consistently achieves better results on both RoBERTa and ELECTRA trained on larger pre-trained corpus and better pre-training tasks.

Original: DPS also improves a large magnitude on models with various structures: DeBERTa model) and BART encoder decoder model).
Revised: DPS also improves a large magnitude on models with various structures: DeBERTa model) and BART encoder decoder model).

Original: These fully demonstrate that DPS can bring a stable improvement regardless of model structure and quality.
Revised: These fully demonstrate that DPS can bring a stable improvement regardless of model structure and quality.

Original: 4.3 DPS with a Sufficient Number of Training Examples
Revised: 4.3 DPS with a Sufficient Number of Training Examples

Original: In addition to few-sample datasets, we believe it is also worth exploring whether DPS can deliver a boost with larger training sets.
Revised: In addition to few-sample datasets, we believe it is also worth exploring whether DPS can deliver a boost with larger training sets.

Original: Therefore, we fine-tune BERT LARGE on two datasets in GLUE with training samples over 50K, including SST-2 and QNLI.
Revised: Therefore, we fine-tune BERT LARGE on two datasets in GLUE with training samples over 50K, including SST-2 and QNLI.

Original: Since it has been stable to fine-tune
Revised: Since it has been stable to fine-tune

Original: BERT LARGE with a sufficient number of training examples Devlin et al.
Revised: BERT LARGE with a sufficient number of training examples Devlin et al.

Original: [2018], Phang et al.
Revised: [2018], Phang et al.

Original: [2018], we focus on the expressiveness of the above two tasks.
Revised: [2018], we focus on the expressiveness of the above two tasks.

Original: As the experimental results in the Tablenoted, when the training samples are large enough, DPS not only doesn’t damage performance but also brings certain improvements compared to other subnetwork optimizations.
Revised: As the experimental results in the Tablenoted, when the training samples are large enough, DPS not only doesn’t damage performance but also brings certain improvements compared to other subnetwork optimizations.

Original: This indicates that our proposed dynamic parameter selection algorithm is suitable for a wider range of data scenarios, except for few-sample scenarios, and thus is superior than previous subnetwork optimizations.
Revised: This indicates that our proposed dynamic parameter selection algorithm is suitable for a wider range of data scenarios, except for few-sample scenarios, and thus is superior than previous subnetwork optimizations.

Original: [Table]
Revised: [Table]

Original: 4.4 Time Usage
Revised: 4.4 Time Usage

Original: [Table]
Revised: [Table]

Original: In this subsection, we investigate the computational efficiency of DPS.
Revised: In this subsection, we investigate the computational efficiency of DPS.

Original: Specifically, we count the time usage of DPS in fine-tuning the BERT LARGE on RTE and compare time usage with various fine-tuning regularization methods, following [Lee et al., 2020].
Revised: Specifically, we count the time usage of DPS in fine-tuning the BERT LARGE on RTE and compare time usage with various fine-tuning regularization methods, following [Lee et al., 2020].

Original: We can notice from Table 6 that although DPS introduces a slight additional computational costs than vanilla fine-tuning, DPS is about 25 % much faster than R3F and R-Dropout, and about 240 % much faster than CHILD-TUNING D .
Revised: We can notice from Table 6 that although DPS introduces a slight additional computational costs than vanilla fine-tuning, DPS is about 25 % much faster than R3F and R-Dropout, and about 240 % much faster than CHILD-TUNING D .

Original: We also discuss memory consumption for above methods in Appendix F.
Revised: We also discuss memory consumption for above methods in Appendix F.

Original:  Conclusion
Revised:  Conclusion

Original: In this work we propose a family of new fine-tuning techniques to fine-tune large-scale PLMs adaptively: DPS Dense and DPS Mix, which respectively optimize full network update and stochastic parameter update processes.
Revised: In this work, we propose a family of new fine-tuning techniques to fine-tune large-scale PLMs adaptively: DPS Dense and DPS Mix, which respectively optimize full network update and stochastic parameter update processes.

Original: Our methods outperform various fine-tuning approaches in terms of stability and overall performance while saving computational costs a lot, and consistently improve the generalization ability of fine-tuned models by a large margin.
Revised: Our methods outperform various fine-tuning approaches in terms of stability and overall performance while saving computational costs a lot, and consistently improve the generalization ability of fine-tuned models by a large margin.

Original: Extensive results and analysis show that DPS can enhance largely on five different pre-trained language models meanwhile possessing extremely stable performance across a wider range of subnetwork sizes.
Revised: Extensive results and analysis show that DPS can enhance largely on five different pre-trained language models meanwhile possessing extremely stable performance across a wider range of subnetwork sizes.

Original: We believe that training neural networks adaptively may have broader application scenarios.
Revised: We believe that training neural networks adaptively may have broader application scenarios.

Original: We leave this area of exploration for future work.
Revised: We leave this area of exploration for future work.

Revised:  Acknowledgements

Revised: We thank all reviewers for their constructive comments.

Revised: This research is supported by the National

Revised: Key RD Program under Grant No.2021ZD0110303, the National Natural Science Foundation of

Revised: China under Grant No. 62072007, 62192733, 61832009, 62192731, 62192730.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: Our limitations lie in inducing additional training cost and memory overhead.
Revised: Our limitations lie in inducing additional training cost and memory overhead.

Original: We have discuss this in Sec. 4.4 and Appendix F. (c) Did you discuss any potential negative societal impacts of your work?
Revised: We have discuss this in Sec. 4.4 and Appendix F. (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We think our methods will not lead to any negative societal impact.
Revised: [No] We think our methods will not lead to any negative societal impact.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We will upload the code in the supplemental material.
Revised: [Yes] We will upload the code in the supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] We provide the details in Sec.
Revised: [Yes] We provide the details in Sec.

Original: 3.4, Sec. 3.5, Appendix C and Appendix D. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: 3.4, Sec. 3.5, Appendix C and Appendix D. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We report the running times below experimental tables in main text.
Revised: [Yes] We report the running times below experimental tables in main text.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Please refer to Appendix C. 4.
Revised: [Yes] Please refer to Appendix C. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We provide the citations for each data and the url for open-souce code.
Revised: [Yes] We provide the citations for each data and the url for open-souce code.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No] We use the publicly popular datasets for each experiments.
Revised: [No] We use the publicly popular datasets for each experiments.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] We do not provide new datasets.
Revised: [No] We do not provide new datasets.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] We use public datasets.
Revised: [N/A] We use public datasets.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] We use public datasets.
Revised: [N/A] We use public datasets.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Appendix A. Case Study
Revised: A Appendix A. Case Study

Original: In Sec. 3. [Figure] we have experimentally verified that DPS outperforms various fine-tuning methods.
Revised: In Sec. 3.3, we have experimentally verified that DPS outperforms various fine-tuning methods.

Original: To understand what type of cases DPS predicts more accurately and justify the effectiveness of DPS from another perspective [Figure] we conduct case study.
Revised: To understand what type of cases DPS predicts more accurately and justify the effectiveness of DPS from another perspective, we conduct case study.

Original: Specifically, we fine-tune BERT LARGE on RTE with 10 random restarts and count the overall proportion of easy cases (cases with more thanaccurate predictions out of 10) and hard cases (cases with more than 5 predictions incorrect out of 10).
Revised: Specifically, we fine-tune BERT LARGE on RTE with 10 random restarts and count the overall proportion of easy cases (cases with more thanaccurate predictions out of 10) and hard cases (cases with more than 5 predictions incorrect out of 10).

Original: Figure 3 summarizes the statistics.
Revised: Figure 3 summarizes the statistics.

Original: Compared with various baselines, DPS has the largest proportion of easy cases and the smallest proportion of hard cases.
Revised: Compared with various baselines, DPS has the largest proportion of easy cases and the smallest proportion of hard cases.

Original: This demonstrates that compared with vanilla fine-tuning, Mixout

Original: [Figure] and CHILD-TUNING, DPS can better maintain general contextual representation, explore the potential value of data and models [Figure] and thus solve difficult cases without affecting the ability to identify easy cases, which is ultimately reflected in the improvement of metrics
Revised: This demonstrates that compared with vanilla fine-tuning, Mixout, and CHILD-TUNING, DPS can better maintain general contextual representation, explore the potential value of data and models, and thus solve difficult cases without affecting the ability to identify easy cases, which is ultimately reflected in the improvement of metrics.

Original: [Figure]
Revised: [Figure]

Original: B Appendix B. GLUE Benchmark Datasets
Revised: B Appendix B. GLUE Benchmark Datasets

Original: we conduct experiments on 8 datasets in GLUE benchmark Wang et al.
Revised: we conduct experiments on 8 datasets in GLUE benchmark Wang et al.

Original: [2019], the dataset statistics for each task are illustrated in Table 7.
Revised: [2019], the dataset statistics for each task are illustrated in Table 7.

Original: we also provide a brief description for each dataset:
Revised: we also provide a brief description for each dataset:

Original: • RTE : Binary entailment classification task Bentivogli et al.
Revised: • RTE : Binary entailment classification task Bentivogli et al.

Original: [2009] • MRPC : Semantic similarity Dolan and Brockett [2005] • STS-B : Semantic textual similarity Cer et al.
Revised: [2009] • MRPC : Semantic similarity Dolan and Brockett [2005] • STS-B : Semantic textual similarity Cer et al.

Original: [2017] • CoLA : Acceptability classification Warstadt et al.
Revised: [2017] • CoLA : Acceptability classification Warstadt et al.

Original: [2019] • SST-2 : Binary sentiment classification Socher et al.
Revised: [2019] • SST-2 : Binary sentiment classification Socher et al.

Original: [2013] • QNLI :
Revised: [2013] • QNLI :

Original: Binary question inference classification Rajpurkar et al.
Revised: Binary question inference classification Rajpurkar et al.

Original: [2016] • QQP :
Revised: [2016] • QQP :

Original: Binary semantically equivalent classification Shankar Iyer and Csernai.
Revised: Binary semantically equivalent classification Shankar Iyer and Csernai.

Original: [2017] • MNLI :
Revised: [2017] • MNLI :

Original: Textual entailment classification Williams et al.
Revised: Textual entailment classification Williams et al.

Original: [Table]
Revised: [Table]

Original: C Appendix C. Hyper-parameters and Experimental Details of Different Pre-trained Language Models
Revised: C Appendix C. Hyper-parameters and Experimental Details of Different Pre-trained Language Models

Original: In this paper, we investigate the performance of DPS on five distinctive and widely used largescale pre-trained language models, namely BERT Devlin et al.
Revised: In this paper, we investigate the performance of DPS on five distinctive and widely used largescale pre-trained language models, namely BERT Devlin et al.

Original: [2018], RoBERTa Liu et al.
Revised: [2018], RoBERTa Liu et al.

Original: ELECTRA Clark et al.
Revised: ELECTRA Clark et al.

Original: [2020], BART Lewis et al.
Revised: [2020], BART Lewis et al.

Original: [2019], and DeBERTa He
Revised: [2019], and DeBERTa He

Original: et al.
Revised: et al.

Original: BERT is the first Transformer Vaswani et al.
Revised: BERT is the first Transformer Vaswani et al.

Original: [2017] encoder based pre-trained language model with Mask
Revised: [2017] encoder based pre-trained language model with Mask

Original: Language Modeling and Next Sentence Prediction pre-training tasks.
Revised: Language Modeling and Next Sentence Prediction pre-training tasks.

Original: RoBERTa is similar to BERT in terms of model architecture but is only pre-trained on the Mask Language Modeling task only, but for longer and on more data.
Revised: RoBERTa is similar to BERT in terms of model architecture but is only pre-trained on the Mask Language Modeling task only, but for longer and on more data.

Original: ELECTRA is a BERT-like model trained to distinguish tokens generated by masked language model from tokens drawn from the natural distribution.
Revised: ELECTRA is a BERT-like model trained to distinguish tokens generated by masked language model from tokens drawn from the natural distribution.

Original: BART is a sequence-to-sequence model trained as a denoising autoencoder.
Revised: BART is a sequence-to-sequence model trained as a denoising autoencoder.

Original: DeBERTa improves Transformebased pre-trained model with disentangled attention mechanism and enhanced mask decoder.
Revised: DeBERTa improves Transformebased pre-trained model with disentangled attention mechanism and enhanced mask decoder.

Original: Tablesummarizes the hyper-parameters of each model for each dataset.
Revised: Tablesummarizes the hyper-parameters of each model for each dataset.

Original: We use AdamWLoshchilov and
Revised: We use AdamWLoshchilov and

Original: Hutter [2019] optimizer, clip the gradients with a maximum norm of [Table] and the maximum sequence length is set as 128.
Revised: Hutter [2019] optimizer, clip the gradients with a maximum norm of 1, and the maximum sequence length is set as 128.

Original: We use mixed precision training to speed up the experimental process.
Revised: We use mixed precision training to speed up the experimental process.

Original: We conduct all the experiments on a single Tesla-V100 GPU (32G)
Revised: We conduct all the experiments on a single Tesla-V100 GPU (32G).

Original: [Table]
Revised: [Table]

Original: D Appendix D. Experimental Details for Different Fine-tuning Methods
Revised: D Appendix D. Experimental Details for Different Fine-tuning Methods

Original: The following is our hyperparameter search space for different fine-tuning regularization methods:
Revised: The following is our hyperparameter search space for different fine-tuning regularization methods:

Original: • Mixout We grid search Mixout probability p ∈ {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8}.
Revised: • Mixout We grid search Mixout probability p ∈ {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8}.

Original: • R3F : We grid search Noise Types ∈ { N , U }, σ ∈ {1e-5}, λ ∈ {0.1, 0.5, 1.0, 5.0}.
Revised: • R3F : We grid search Noise Types ∈ { N , U }, σ ∈ {1e-5}, λ ∈ {0.1, 0.5, 1.0, 5.0}.

Original: • Re-init
Revised: • Re-init

Original: : We grid search L ∈ {1, 2, 3, 4, 5, 6, 7}.
Revised: : We grid search L ∈ {1, 2, 3, 4, 5, 6, 7}.

Original: • CHILD-TUNING D : We grid search CHILD-TUNING D p D ∈ {0.1, 0.2, 0.3}, and learning rate lr ∈ {2e-5, 4e-5, 6e-5 ,8e-5, 1e-4}.
Revised: • CHILD-TUNING D : We grid search CHILD-TUNING D p D ∈ {0.1, 0.2, 0.3}, and learning rate lr ∈ {2e-5, 4e-5, 6e-5 ,8e-5, 1e-4}.

Original: • R-Dropout : We grid search Dropout probability p ∈ {0.1} and α ∈ {0.1, 0.5, 1, 3, 5}.
Revised: • R-Dropout : We grid search Dropout probability p ∈ {0.1} and α ∈ {0.1, 0.5, 1, 3, 5}.

Original: • DPS Dense : We grid search reserved probability p ∈ {0.1, 0.2, 0.3, 0.4, 0.5} and update ratio ur ∈ {0.05, 0.1, 0.2}.
Revised: • DPS Dense : We grid search reserved probability p ∈ {0.1, 0.2, 0.3, 0.4, 0.5} and update ratio ur ∈ {0.05, 0.1, 0.2}.

Original: • DPS Mix : We grid search reserved probability p ∈ {0.1, 0.2, 0.3, 0.4, 0.5}, and update ratio ur ∈ {0.05, 0.1, 0.2}.
Revised: • DPS Mix : We grid search reserved probability p ∈ {0.1, 0.2, 0.3, 0.4, 0.5}, and update ratio ur ∈ {0.05, 0.1, 0.2}.

Original: E Appendix E. Mapping Strategy
Revised: E Appendix E. Mapping Strategy

Original: We train DPS on SNLI and MNLI datasets respectively and evaluate several different target NLI datasets.
Revised: We train DPS on SNLI and MNLI datasets respectively and evaluate several different target NLI datasets.

Original: The SNLI and MNLI datasets contain three labels: entailment, neutral, and contradiction, however, some datasets have only two labels (SciTial, RTE).
Revised: The SNLI and MNLI datasets contain three labels: entailment, neutral, and contradiction, however, some datasets have only two labels (SciTial, RTE).

Original: The SciTail dataset contains two labels: entailment, neutral, and we map the predicted labels neutral and contradiction to neutral, following
Revised: The SciTail dataset contains two labels: entailment, neutral, and we map the predicted labels neutral and contradiction to neutral, following

Original: Mahabadi et al.
Revised: Mahabadi et al.

Original: We conduct the same process for RTE.
Revised: We conduct the same process for RTE.

Original: F Appendix F. Memory Consumption
Revised: F Appendix F. Memory Consumption

Original: In addition to time usage, we further analyze memory consumption of various approaches when fine-tuning BERT LARGE .
Revised: In addition to time usage, we further analyze memory consumption of various approaches when fine-tuning BERT LARGE .

Original: The table below shows the results.
Revised: The table below shows the results.

Original: DPS Dense requires extra memory because it needs to store Gradient Accumulation Matrix ( GAM ), DPS Mix requires more memory than DPS Dense because it needs to store Frequency Accumulation Matrix ( FAM ) and GAM .
Revised: DPS Dense requires extra memory because it needs to store Gradient Accumulation Matrix ( GAM ), DPS Mix requires more memory than DPS Dense because it needs to store Frequency Accumulation Matrix ( FAM ) and GAM .

Original: It is worth noting that since the size of GAM and FAM is fixed, the additional memory consumption introduced by DPS does not increase as the batch size increases.
Revised: It is worth noting that since the size of GAM and FAM is fixed, the additional memory consumption introduced by DPS does not increase as the batch size increases.

Original: Therefore, as the batch size increases, the percentage of additional memory consumption decreases for DPS compared to vanilla fine-tuning.
Revised: Therefore, as the batch size increases, the percentage of additional memory consumption decreases for DPS compared to vanilla fine-tuning.

Original: Overall, DPS does introduce additional memory overhead, but we believe that the memory consumption for DPS is acceptable compared to other regularization methods.
Revised: Overall, DPS does introduce additional memory overhead, but we believe that the memory consumption for DPS is acceptable compared to other regularization methods.

Original: [Table]
Revised: [Table]