Original: P ARAMETER -E FFICIENT T RANSFER L EARNING WITH D IFF P RUNING
Revised: P ARAMETER -E FFICIENT T RANSFER L EARNING WITH D IFF P RUNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While task-speciﬁc ﬁnetuning of deep networks pretrained with self-supervision has led to signiﬁcant empirical advances in NLP, their large size makes the standard ﬁnetuning approach difﬁcult to apply to multi-task, memory-constrained settings, as storing the full model parameters for each task become prohibitively expensive.
Revised: While task-speciﬁc ﬁnetuning of deep networks pretrained with self-supervision has led to signiﬁcant empirical advances in NLP, their large size makes the standard ﬁnetuning approach difﬁcult to apply to multi-task, memory-constrained settings, as storing the full model parameters for each task become prohibitively expensive.

Original: We propose diff pruning as a simple approach to enable parameterefﬁcient transfer learning within the pretrain-ﬁnetune framework.
Revised: We propose diff pruning as a simple approach to enable parameterefﬁcient transfer learning within the pretrain-ﬁnetune framework.

Original: This approach views ﬁnetuning as learning a task-speciﬁc “diff” vector that is applied on top of the pretrained parameter vector, which remains ﬁxed and is shared across different tasks.
Revised: This approach views ﬁnetuning as learning a task-speciﬁc “diff” vector that is applied on top of the pretrained parameter vector, which remains ﬁxed and is shared across different tasks.

Original: The diff vector is adaptively pruned during training with a differentiable approximation to the L 0 -norm penalty to encourage sparsity.
Revised: The diff vector is adaptively pruned during training with a differentiable approximation to the L 0 -norm penalty to encourage sparsity.

Original: Diff pruning becomes parameter-efﬁcient as the number of tasks increases, as it requires storing only the nonzero positions and weights of the diff vector for each task, while the cost of storing the shared pretrained model remains constant.
Revised: Diff pruning becomes parameter-efﬁcient as the number of tasks increases, as it requires storing only the nonzero positions and weights of the diff vector for each task, while the cost of storing the shared pretrained model remains constant.

Original: We ﬁnd that models ﬁnetuned with diff pruning can match the performance of fully ﬁnetuned baselines on the GLUE benchmark while only modifying 0.5 % of the pretrained model’s parameters per task.
Revised: We ﬁnd that models ﬁnetuned with diff pruning can match the performance of fully ﬁnetuned baselines on the GLUE benchmark while only modifying 0.5 % of the pretrained model’s parameters per task.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Task-speciﬁc ﬁnetuning of pretrained deep networks has become the dominant paradigm in contemporary NLP, achieving state-of-the-art results across a suite of natural language understanding tasks (Devlin et al., 2019; Liu et al., 2019b; Yang et al., 2019; Lan et al., 2020).
Revised: Task-speciﬁc ﬁnetuning of pretrained deep networks has become the dominant paradigm in contemporary NLP, achieving state-of-the-art results across a suite of natural language understanding tasks (Devlin et al., 2019; Liu et al., 2019c; Yang et al., 2019; Lan et al., 2020).

Original: While straightforward and empirically effective, this approach is difﬁcult to scale to multi-task, memory-constrained settings (e.g. for on-device applications), as it requires shipping and storing a full set of model parameters for each task.
Revised: While straightforward and empirically effective, this approach is difﬁcult to scale to multi-task, memory-constrained settings (e.g. for on-device applications), as it requires shipping and storing a full set of model parameters for each task.

Original: Inasmuch as these models are learning generalizable, task-agnostic language representations through self-supervised pretraining, ﬁnetuning the entire model for each task is an especially inefﬁcient use of model parameters.
Revised: Inasmuch as these models are learning generalizable, task-agnostic language representations through self-supervised pretraining, ﬁnetuning the entire model for each task is an especially inefﬁcient use of model parameters.

Original: A popular approach to parameter-efﬁciency is to learn sparse models for each task where a subset of the ﬁnal model parameters are exactly zero (Gordon et al., 2020; Sajjad et al., 2020; Zhao et al., 2020; Sanh et al., 2020).
Revised: A popular approach to parameter-efﬁciency is to learn sparse models for each task where a subset of the ﬁnal model parameters are exactly zero (Gordon et al., 2020; Sajjad et al., 2020; Zhao et al., 2020; Sanh et al., 2020).

Original: Such approaches often face a steep sparsity/performance tradeoff, and a substantial portion of nonzero parameters (e.g. 10%-30%) are still typically required to match the performance of the dense counterparts.
Revised: Such approaches often face a steep sparsity/performance tradeoff, and a substantial portion of nonzero parameters (e.g. 10%-30%) are still typically required to match the performance of the dense counterparts.

Original: An alternative is to use multi-task learning or feature-based transfer for more parameter-efﬁcient transfer learning with pretrained models (Liu et al., 2019a;
Revised: An alternative is to use multi-task learning or feature-based transfer for more parameter-efﬁcient transfer learning with pretrained models (Liu et al., 2019b;

Original: Clark et al., 2019; Stickland & Murray, 2019; Reimers & Gurevych, 2019; Feng et al., 2020).
Revised: Clark et al., 2019; Stickland & Murray, 2019; Reimers & Gurevych, 2019; Feng et al., 2020).

Original: These methods learn only a small number of additional parameters (e.g. a linear layer) on top of a shared model.
Revised: These methods learn only a small number of additional parameters (e.g. a linear layer) on top of a shared model.

Original: However, multi-task learning generally requires access to all tasks during training to prevent catastrophic forgetting (French, 1999), while feature-based transfer learning (e.g. based on taskagnostic sentence representations) is typically outperformed by full ﬁnetuning (Howard & Ruder, 2018).
Revised: However, multi-task learning generally requires access to all tasks during training to prevent catastrophic forgetting (French, 1999), while feature-based transfer learning (e.g. based on taskagnostic sentence representations) is typically outperformed by full ﬁnetuning (Howard & Ruder, 2018).

Original: Adapters (Rebufﬁ et al., 2018) have recently emerged as a promising approach to parameterefﬁcient transfer learning within the pretrain-ﬁnetune paradigm (Houlsby et al., 2019; Pfeiffer et al., 2020a;b;c).
Revised: Adapters (Rebufﬁ et al., 2018) have recently emerged as a promising approach to parameterefﬁcient transfer learning within the pretrain-ﬁnetune paradigm (Houlsby et al., 2019; Pfeiffer et al., 2020a;b;c).

Original: Adapter layers are smaller, task-speciﬁc modules that are inserted between layers of a pretrained model, which remains ﬁxed and is shared across tasks.
Revised: Adapter layers are smaller, task-speciﬁc modules that are inserted between layers of a pretrained model, which remains ﬁxed and is shared across tasks.

Original: These approaches do not require access to all tasks during training making them attractive in settings where one hopes to obtain and share performant models as new tasks arrive in stream.
Revised: These approaches do not require access to all tasks during training making them attractive in settings where one hopes to obtain and share performant models as new tasks arrive in stream.

Original: Houlsby et al.
Revised: Houlsby et al.

Original: ﬁnd that adapter layers trained on BERT can match the performance of fully ﬁnetuned BERT on the GLUE benchmark (Wang et al., 2019a) while only requiring 3.6% additional parameters (on average) per task.
Revised: ﬁnd that adapter layers trained on BERT can match the performance of fully ﬁnetuned BERT on the GLUE benchmark (Wang et al., 2019a) while only requiring 3.6% additional parameters (on average) per task.

Original: In this work, we consider a similar setting as adapters but propose a new diff pruning approach with the goal of even more parameter-efﬁcient transfer learning.
Revised: In this work, we consider a similar setting as adapters but propose a new diff pruning approach with the goal of even more parameter-efﬁcient transfer learning.

Original: Diff pruning views ﬁnetuning as learning a task-speciﬁc difference vector that is applied on top of the pretrained parameter vector, which remains ﬁxed and is shared across different tasks.
Revised: Diff pruning views ﬁnetuning as learning a task-speciﬁc difference vector that is applied on top of the pretrained parameter vector, which remains ﬁxed and is shared across different tasks.

Original: In order to learn this vector, we reparameterize the task-speciﬁc model parameters as θ task =
Revised: In order to learn this vector, we reparameterize the task-speciﬁc model parameters as θ task =

Original: θ pretrained + δ task , where the pretrained parameter vector θ pretrained is ﬁxed and the task-speciﬁc diff vector δ task is ﬁnetuned.
Revised: θ pretrained + δ task , where the pretrained parameter vector θ pretrained is ﬁxed and the task-speciﬁc diff vector δ task is ﬁnetuned.

Original: The diff vector is regularized with a differentiable approximation to the L 0 -norm
Revised: The diff vector is regularized with a differentiable approximation to the L 0 -norm

Original: penalty
Revised: penalty

Original: (Louizos et al., 2018) to encourage sparsity.
Revised: (Louizos et al., 2018) to encourage sparsity.

Original: This approach can become parameter-efﬁcient as the number of tasks increases as it only requires storing the nonzero positions and weights of the diff vector for each task.
Revised: This approach can become parameter-efﬁcient as the number of tasks increases as it only requires storing the nonzero positions and weights of the diff vector for each task.

Original: The cost of storing the shared pretrained model remains constant and is amortized across multiple tasks.
Revised: The cost of storing the shared pretrained model remains constant and is amortized across multiple tasks.

Original: On the GLUE benchmark (Wang et al., 2019a), diff pruning can match the performance of the fully ﬁnetuned BERT baselines while ﬁnetuning only 0 .
Revised: On the GLUE benchmark (Wang et al., 2019a), diff pruning can match the performance of the fully ﬁnetuned BERT baselines while ﬁnetuning only 0 .

Original: 5% of the pretrained parameters per task, making it a potential alternative to adapters for parameter-efﬁcient transfer learning.
Revised: 5% of the pretrained parameters per task, making it a potential alternative to adapters for parameter-efﬁcient transfer learning.

Original:  B ACKGROUND :M ULTI - TASK T RANSFER L EARNING FOR NLP
Revised:  B ACKGROUND : T RANSFER L EARNING FOR NLP

Original: The ﬁeld of NLP has recently seen remarkable progress through transfer learning with a pretrainand-ﬁnetune paradigm, which initializes a subset of the model parameters for all tasks from a pretrained model and then ﬁnetunes on a task speciﬁc objective.
Revised: The ﬁeld of NLP has recently seen remarkable progress through transfer learning with a pretrainand-ﬁnetune paradigm, which initializes a subset of the model parameters for all tasks from a pretrained model and then ﬁnetunes on a task speciﬁc objective.

Original: Pretraining objectives include context prediction (Mikolov et al., 2013), autoencoding (Dai & Le, 2015), machine translation (McCann et al., 2017), and more recently, variants of language modeling (Peters et al., 2018;
Revised: Pretraining objectives include context prediction (Mikolov et al., 2013), autoencoding (Dai & Le, 2015), machine translation (McCann et al., 2017), and more recently, variants of language modeling (Peters et al., 2018;

Original: Radford et al., 2018; Devlin et al., 2019) objectives.
Revised: Radford et al., 2018; Devlin et al., 2019) objectives.

Original: Here we consider applying transfer learning to multiple tasks.
Revised: Here we consider applying transfer learning to multiple tasks.

Original: We consider a setting with a potentially unknown set of tasks, where each τ ∈ T has an associated training set { x ( n ) τ , y ( n ) τ } Nn =1 .
Revised: We consider a setting with a potentially unknown set of tasks, where each τ ∈ T has an associated training set { x ( n ) τ , y ( n ) τ } Nn =1 .

Original: For all tasks, the goal is to produce (possibly tied) model parameters θ τ to minimize the empirical risk,
Revised: For all tasks, the goal is to produce (possibly tied) model parameters θ τ to minimize the empirical risk,

Original: [Equation]
Revised: [Equation]

Original: where f ( · ; θ ) is a parameterized function over the input (e.g. a neural network), L ( · , · ) is a loss function (e.g. cross-entropy), and R ( · ) is an optional regularizer with hyperparameter λ .
Revised: where f ( · ; θ ) is a parameterized function over the input (e.g. a neural network), L ( · , · ) is a loss function (e.g. cross-entropy), and R ( · ) is an optional regularizer with hyperparameter λ .

Original: This multi-task setting can use the pretrain-then-ﬁnetune approach by simply learning independent parameters for each task; however the large size of pretrained models makes this approach exceedingly parameter inefﬁcient.
Revised: This multi-task setting can use the pretrain-then-ﬁnetune approach by simply learning independent parameters for each task; however the large size of pretrained models makes this approach exceedingly parameter inefﬁcient.

Original: For example, widely-adopted models such as BERT BASE and
Revised: For example, widely-adopted models such as BERT BASE and

Original: BERT LARGE have 110M and 340M parameters respectively, while their contemporaries such as
Revised: BERT LARGE have 110M and 340M parameters respectively, while their contemporaries such as

Original: T5 (Raffel et al., 2020), Megatron-LM (Shoeybi et al., 2019), and Turing-NLG (Rajbhandari et al., 2019) have parameter counts in the billions.
Revised: T5 (Raffel et al., 2020), Megatron-LM (Shoeybi et al., 2019), and Turing-NLG (Rajbhandari et al., 2019) have parameter counts in the billions.

Original: Storing the fully ﬁnetuned models becomes difﬁcult even for a moderate number of tasks.
Revised: Storing the fully ﬁnetuned models becomes difﬁcult even for a moderate number of tasks.

Original: 1 A classic approach to tackling this parameterinefﬁciency (Caruana, 1997) is to train a single shared model (along with a task-speciﬁc output layer) against multiple tasks through joint training.
Revised: 2 A classic approach to tackling this parameterinefﬁciency (Caruana, 1997) is to train a single shared model (along with a task-speciﬁc output layer) against multiple tasks through joint training.

Original: However, the usual formulation of multi-task learning requires the set of tasks T to be known in advance in order to prevent catastrophic forgetting (French, 1999), 2 making it unsuitable for applications in which the set of tasks is unknown (e.g. when tasks arrive in stream).
Revised: However, the usual formulation of multi-task learning requires the set of tasks T to be known in advance in order to prevent catastrophic forgetting (French, 1999), 3 making it unsuitable for applications in which the set of tasks is unknown (e.g. when tasks arrive in stream).

Original:  D IFF P RUNING
Revised:  D IFF P RUNING

Original: Diff pruning formulates task-speciﬁc ﬁnetuning as learning a diff vector δ τ that is added to the pretrained model parameters θ pretrained .
Revised: Diff pruning formulates task-speciﬁc ﬁnetuning as learning a diff vector δ τ that is added to the pretrained model parameters θ pretrained .

Original: We ﬁrst reparameterize the task-speciﬁc model parameters,
Revised: We ﬁrst reparameterize the task-speciﬁc model parameters,

Original: [Equation]
Revised: [Equation]

Original: which results in the following empirical risk minimization problem,
Revised: which results in the following empirical risk minimization problem,

Original: [Equation]
Revised: [Equation]

Original: This trivial reparameterization is equivalent to the original formulation.
Revised: This trivial reparameterization is equivalent to the original formulation.

Original: Its beneﬁt comes in the multi-task setting where the cost of storing the pretrained parameters θ pretrained is amortized across tasks, and the only marginal cost for new tasks is the diff vector.
Revised: Its beneﬁt comes in the multi-task setting where the cost of storing the pretrained parameters θ pretrained is amortized across tasks, and the only marginal cost for new tasks is the diff vector.

Original: If we can regularize δ τ to be sparse such that (cid:107) δ τ (cid:107) 0 (cid:28) (cid:107) θ pretrained (cid:107) 0 , then this approach can become more parameter-efﬁcient as the number of tasks increases.
Revised: If we can regularize δ τ to be sparse such that (cid:107) δ τ (cid:107) 0 (cid:28) (cid:107) θ pretrained (cid:107) 0 , then this approach can become more parameter-efﬁcient as the number of tasks increases.

Original: We can specify this goal with an L 0 -norm penalty on the diff vector,
Revised: We can specify this goal with an L 0 -norm penalty on the diff vector,

Original: [Equation]
Revised: [Equation]

Original: 3. D IFFERENTIABLE APPROXIMATION TO THE L 0 - NORM
Revised: 3. D IFFERENTIABLE APPROXIMATION TO THE L 0 - NORM

Original: This regularizer is difﬁcult to directly optimize as it is non-differentiable.
Revised: This regularizer is difﬁcult to directly optimize as it is non-differentiable.

Original: In order to approximate this L 0 objective, we follow the standard approach for gradient-based learning with L 0 sparsity using a relaxed mask vector (Louizos et al., 2018).
Revised: In order to approximate this L 0 objective, we follow the standard approach for gradient-based learning with L 0 sparsity using a relaxed mask vector (Louizos et al., 2018).

Original: This approach involves relaxing a binary vector into continuous space, and then multiplying it with a dense weight vector to determine how much of the weight vector is applied during training.
Revised: This approach involves relaxing a binary vector into continuous space, and then multiplying it with a dense weight vector to determine how much of the weight vector is applied during training.

Original: After training, the mask is deterministic and a large portion of the diff vector is true zero.
Revised: After training, the mask is deterministic and a large portion of the diff vector is true zero.

Original: To apply this method we ﬁrst decompose δ τ into a binary mask vector multiplied with a dense vector,
Revised: To apply this method we ﬁrst decompose δ τ into a binary mask vector multiplied with a dense vector,

Original: [Equation]
Revised: [Equation]

Original: We can now instead optimize an expectation with respect to z τ , whose distribution p ( z τ ; α τ ) is initially Bernoulli with parameters α τ ,
Revised: We can now instead optimize an expectation with respect to z τ , whose distribution p ( z τ ; α τ ) is initially Bernoulli with parameters α τ ,

Original: [Equation]
Revised: [Equation]

Original: This objective is still difﬁcult in practice due to z τ ’s being discrete (which requires the score function gradient estimator), but the expectation provides some guidance for empirically effective relaxations.
Revised: This objective is still difﬁcult in practice due to z τ ’s being discrete (which requires the score function gradient estimator), but the expectation provides some guidance for empirically effective relaxations.

Original: We follow prior work (Louizos et al., 2018; Wang et al., 2019b) and relax z τ into continuous space [0 , 1] d with a stretched Hard-Concrete distribution (Jang et al., 2017; Maddison et al., 2017), which allows for the use of pathwise gradient estimators.
Revised: We follow prior work (Louizos et al., 2018; Wang et al., 2019b) and relax z τ into continuous space [0 , 1] d with a stretched Hard-Concrete distribution (Jang et al., 2017; Maddison et al., 2017), which allows for the use of pathwise gradient estimators.

Original: Speciﬁcally, z τ is now deﬁned to be a deterministic and (sub)differentiable function of a sample u from a uniform distribution,
Revised: Speciﬁcally, z τ is now deﬁned to be a deterministic and (sub)differentiable function of a sample u from a uniform distribution,

Original: [Equation]
Revised: [Equation]

Original: Here l < 0 and r > 1 are two constants used to stretch s τ into the interval ( l, r )  before it is clamped to [0 , 1] d with the min( 1 , max( 0 , · )) operation.
Revised: Here l < 0 and r > 1 are two constants used to stretch s τ into the interval ( l, r ) d before it is clamped to [0 , 1] d with the min( 1 , max( 0 , · )) operation.

Original: In this case we have a differentiable closed-form expression for the expected L 0 -norm,
Revised: In this case we have a differentiable closed-form expression for the expected L 0 -norm,

Original: [Equation]
Revised: [Equation]

Original: Thus the ﬁnal optimization problem is given by,min α τ , w τ E u ∼ U
Revised: Thus the ﬁnal optimization problem is given by,

Original: [Equation]
Revised: [Equation]

Original: and we can now utilize pathwise gradient estimators to optimize the ﬁrst term with respect to α τ since the expectation no longer depends on it.
Revised: and we can now utilize pathwise gradient estimators to optimize the ﬁrst term with respect to α τ since the expectation no longer depends on it.

Original: After training we obtain the ﬁnal diff vector δ τ by sampling u once to obtain z τ (which is not necessarily a binary vector but has a signiﬁcant number of dimensions equal to exactly zero due to the clamping function), then setting δ τ =
Revised: After training we obtain the ﬁnal diff vector δ τ by sampling u once to obtain z τ (which is not necessarily a binary vector but has a signiﬁcant number of dimensions equal to exactly zero due to the clamping function), then setting δ τ =

Original: z τ (cid:12) w τ .
Revised: z τ (cid:12) w τ .

Revised: We found work

Original: 3. L 0 - BALL P ROJECTION WITH M AGNITUDE P RUNING FOR S PARSITY C ONTROL
Revised: 3. L 0 - BALL P ROJECTION WITH M AGNITUDE P RUNING FOR S PARSITY C ONTROL

Original: Differentiable L 0 regularization provides a strong way to achieve high sparsity rate.
Revised: Differentiable L 0 regularization provides a strong way to achieve high sparsity rate.

Original: However, it would be ideal to have more ﬁne-grained control into the exact sparsity rate in the diff vector, especially considering applications which require speciﬁc parameter budgets.
Revised: However, it would be ideal to have more ﬁne-grained control into the exact sparsity rate in the diff vector, especially considering applications which require speciﬁc parameter budgets.

Original: As λ is just the Lagrangian multiplier for the constraint E [ (cid:107)
Revised: As λ is just the Lagrangian multiplier for the constraint E [ (cid:107)

Original: δ τ (cid:107) 0 ] < η for some η , this could be achieved in principle by searching over different values of λ .
Revised: δ τ (cid:107) 0 ] < η for some η , this could be achieved in principle by searching over different values of λ .

Original: However we found it more efﬁcient and empirically effective to achieve an exact sparsity rate by simply projecting onto the L 0 -ball after training.
Revised: However we found it more efﬁcient and empirically effective to achieve an exact sparsity rate by simply projecting onto the L 0 -ball after training.

Original: Speciﬁcally we use magnitude pruning on the diff vector δ τ and target a sparsity rate t % by only keeping the top t % × d values in δ τ .
Revised: Speciﬁcally we use magnitude pruning on the diff vector δ τ and target a sparsity rate t % by only keeping the top t % × d values in δ τ .

Original: 5 Note that unlike standard magnitude pruning, this is based on the magnitude of the diff vector values and not the model parameters.
Revised: 6 Note that unlike standard magnitude pruning, this is based on the magnitude of the diff vector values and not the model parameters.

Original: As is usual in magnitude pruning, we found it important to further ﬁnetune δ τ with the nonzero masks ﬁxed to maintain good performance (Han et al., 2016).
Revised: As is usual in magnitude pruning, we found it important to further ﬁnetune δ τ with the nonzero masks ﬁxed to maintain good performance (Han et al., 2016).

Original: Since this type of parameter-efﬁciency through projection onto the
Revised: Since this type of parameter-efﬁciency through projection onto the

Original: L 0 ball can be applied without adaptive diff pruning, 6 such an approach will serve as one of our baselines in the empirical study.
Revised: L 0 -ball can be applied without adaptive diff pruning, 7 such an approach will serve as one of our baselines in the empirical study.

Original: 3. S TRUCTURED D IFF P RUNING
Revised: 3. S TRUCTURED D IFF P RUNING

Original: Diff pruning, as presented above, is architecture-agnostic and does not exploit the underlying model structure—each dimension of z τ is independent from one another.
Revised: Diff pruning, as presented above, is architecture-agnostic and does not exploit the underlying model structure—each dimension of z τ is independent from one another.

Original: While this makes the approach potentially more ﬂexible, we might expect to achieve better sparsity/performance tradeoff through a structured formulation which encourages active parameters to group together and other areas to be fully sparse.
Revised: While this makes the approach potentially more ﬂexible, we might expect to achieve better sparsity/performance tradeoff through a structured formulation which encourages active parameters to group together and other areas to be fully sparse.

Original: Motivated by this intuition, we ﬁrst partition the parameter indices into G groups { g (1) , . . .
Revised: Motivated by this intuition, we ﬁrst partition the parameter indices into G groups { g (1) , . . .

Original: , g ( G ) } where g ( j ) is a subset of parameter indices governed by group g ( j ) .
Revised: , g ( G ) } where g ( j ) is a subset of parameter indices governed by group g ( j ) .

Original: 7 We then introduce a scalar z jτ (with the associated parameter α jτ ) for each group g ( j ) , and decompose the task-speciﬁc parameter for index
Revised: We then introduce a scalar z jτ (with the associated parameter α jτ ) for each group g ( j ) , and decompose the task-speciﬁc parameter for index

Original: i ∈ g ( j ) as δ jτ,i =
Revised: i ∈ g ( j ) as δ jτ,i =

Original: z τ,i × z jτ × w τ,i .
Revised: z τ,i × z jτ × w τ,i .

Original: The expected L 0 -norm is then given by,
Revised: The expected L 0 -norm is then given by,

Original: [Equation]
Revised: [Equation]

Original: and we can train with gradient-based optimization as before.
Revised: and we can train with gradient-based optimization as before.

Original:  E XPERIMENTS 4. M ODEL AND D ATASETS
Revised:  E XPERIMENTS 4. M ODEL AND D ATASETS

Original: For evaluation we use the GLUE benchmark (Wang et al., 2019b), a popular ﬁnetuning dataset.
Revised: For evaluation we use the GLUE benchmark (Wang et al., 2019b), a popular ﬁnetuning dataset.

Original: Following adapters (Houlsby et al., 2019), we test our approach on the following subset of the GLUE tasks: Multi-Genre Natural Language Inference ( MNLI ), where the goal is two predict whether the relationship between two sentences is entailment, contradiction, or neutral (we test on both MNLI m and MNLI mm which respectively tests on matched/mismatched domains);
Revised: Following adapters (Houlsby et al., 2019), we test our approach on the following subset of the GLUE tasks: Multi-Genre Natural Language Inference ( MNLI ), where the goal is two predict whether the relationship between two sentences is entailment, contradiction, or neutral (we test on both MNLI m and MNLI mm which respectively tests on matched/mismatched domains);

Original: Quora Question Pairs ( QQP ), a classiﬁcation task to predict whether two question are semantically equivalent; Question
Revised: Quora Question Pairs ( QQP ), a classiﬁcation task to predict whether two question are semantically equivalent; Question

Original: Natural Language Inference ( QNLI ), which must predict whether a sentence is a correct answer to the question; Stanford Sentiment Treebank ( SST-2 ), a sentence classiﬁcation task to predict the sentiment of movie reviews; Corpus of Linguistic Acceptability ( CoLA ), where the goal is predict whether a sentence is linguistically acceptable or not; Semantic Textual Similarity Benchmark ( STSB ), which must predict a similarity rating between two sentences; Microsoft Research Paraphrase
Revised: Natural Language Inference ( QNLI ), which must predict whether a sentence is a correct answer to the question; Stanford Sentiment Treebank ( SST-2 ), a sentence classiﬁcation task to predict the sentiment of movie reviews; Corpus of Linguistic Acceptability ( CoLA ), where the goal is predict whether a sentence is linguistically acceptable or not; Semantic Textual Similarity Benchmark ( STSB ), which must predict a similarity rating between two sentences; Microsoft Research Paraphrase

Original: Corpus ( MRPC ), where the goal is to predict whether two sentences are semantically equivalent;
Revised: Corpus ( MRPC ), where the goal is to predict whether two sentences are semantically equivalent;

Original: Recognizing Textual Entailment ( RTE ), which must predict whether a second sentence is entailed by the ﬁrst.
Revised: Recognizing Textual Entailment ( RTE ), which must predict whether a second sentence is entailed by the ﬁrst.

Original: For  CoLA, 
Revised: For evaluation, the uses Matthew’s correlation for CoLA, Spearman for

Original: STS-B, F 1 score for MRPC/QQC, and accuracy for MNLI/QNLI/SST-2/RTE.
Revised: STS-B, F 1 score for MRPC/QQC, and accuracy for MNLI/QNLI/SST-2/RTE.

Original: [Table]
Revised: [Table]

Original: For all experiments, we use the BERT LARGE model from Devlin et al.
Revised: For all experiments, we use the BERT LARGE model from Devlin et al.

Original: (2019), which has 24 layers, 1024 hidden size, 16 attention heads, and 340M parameters.
Revised: (2019), which has 24 layers, 1024 hidden size, 16 attention heads, and 340M parameters.

Original: We use the Huggingface Transformer library (Wolf et al., 2019) to conduct our experiments.
Revised: We use the Huggingface Transformer library (Wolf et al., 2019) to conduct our experiments.

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: We compare both structured and non-structured variants of diff pruning against the following baselines:
Revised: We compare both structured and non-structured variants of diff pruning against the following baselines:

Original: Full ﬁnetuning , which fully ﬁnetunes BERT LARGE as usual; Last layer ﬁnetuning , which only ﬁnetunes the penultimate layer (along with the ﬁnal output layer) 8 ; Adapters from Houlsby et al.
Revised: Full ﬁnetuning , which fully ﬁnetunes BERT LARGE as usual; Last layer ﬁnetuning , which only ﬁnetunes the penultimate layer (along with the ﬁnal output layer) 9 ; Adapters from Houlsby et al.

Original: (2019), which train task-speciﬁc bottleneck layers between between each layer of a pretrained model, where parameter-efﬁciency can be controlled by varying the size of the bottleneck layers; and Non-adaptive diff pruning , which performs diff pruning just based on magnitude pruning (i.e., we obtain θ τ through usual ﬁnetuning, set δ τ = θ τ − θ pretrained , and then apply magnitude pruning followed by additional ﬁnetuning on δ τ ).
Revised: (2019), which train task-speciﬁc bottleneck layers between between each layer of a pretrained model, where parameter-efﬁciency can be controlled by varying the size of the bottleneck layers; and Non-adaptive diff pruning , which performs diff pruning just based on magnitude pruning (i.e., we obtain θ τ through usual ﬁnetuning, set δ τ = θ τ − θ pretrained , and then apply magnitude pruning followed by additional ﬁnetuning on δ τ ).

Original: For diff pruning we set our target sparsity rate to 0.5% and investigate the effect of different target sparsity rates in section 5.1.
Revised: For diff pruning we set our target sparsity rate to 0.5% and investigate the effect of different target sparsity rates in section 5.1.

Original: 4. I MPLEMENTATION D ETAILS AND H YPERPARAMETERS
Revised: 4. I MPLEMENTATION D ETAILS AND H YPERPARAMETERS

Original: Diff pruning introduces additional hyperparameters l, r (for stretching the Hard-Concrete distribution) and λ (for weighting the approximate L 0 -norm penalty).
Revised: Diff pruning introduces additional hyperparameters l, r (for stretching the Hard-Concrete distribution) and λ (for weighting the approximate L 0 -norm penalty).

Original: We found l = − 1 .
Revised: We found l = − 1 .

Original: 5 , r = 1 . 5 , λ = 1 .
Revised: 5 , r = 1 . 5 , λ = 1 .

Original: 25 × 10 − 7 to work well across all tasks.
Revised: 25 × 10 − 7 to work well across all tasks.

Original: We also initialize the weight vector w τ to 0 , and α τ to a positive vector (we use 5 ) to encourage z τ to be close to 1 at the start of training.
Revised: We also initialize the weight vector w τ to 0 , and α τ to a positive vector (we use 5 ) to encourage z τ to be close to 1 at the start of training.

Original: While we mainly experiment with BERT LARGE to compare against prior work with adapters (Houlsby et al., 2019), in preliminary experiments we found these hyperparameters to work for ﬁnetuning RoBERTa
Revised: While we mainly experiment with BERT LARGE to compare against prior work with adapters (Houlsby et al., 2019), in preliminary experiments we found these hyperparameters to work for ﬁnetuning RoBERTa

Original: (Liu et al., 2019b) and XLNet (Yang et al., 2019) models as well.
Revised: (Liu et al., 2019c) and XLNet (Yang et al., 2019) models as well.

Original: For all tasks we use a learning rate of 1 × 10 − 5 and perform a hyperparameter search over batch size ∈ { 4 , 6 , 8 , 10 } and the number of epochs ∈ { 2 , 3 , 4 , 5 } .
Revised: For all tasks we use a learning rate of 1 × 10 − 5 and perform a hyperparameter search over batch size ∈ { 4 , 6 , 8 , 10 } and the number of epochs ∈ { 2 , 3 , 4 , 5 } .

Original: However we found the default settings used for regular ﬁnetuning as suggested in the original BERT paper to work well for most tasks.
Revised: However we found the default settings used for regular ﬁnetuning as suggested in the original BERT paper to work well for most tasks.

Original: Finetuning with the ﬁxed mask after projecting onto the L 0 -ball with magnitude pruning is done for 3 epochs with a learning rate of 5 × 10 − 5 .
Revised: Finetuning with the ﬁxed mask after projecting onto the L 0 -ball with magnitude pruning is done for 3 epochs with a learning rate of 5 × 10 − 5 .

Original: Grouping for the structured version of diff pruning is based on the matrix/bias vectors (i.e. parameters that belong to the same matrix or bias vector are assumed to be in the same group), which results in 393 groups.
Revised: Grouping for the structured version of diff pruning is based on the matrix/bias vectors (i.e. parameters that belong to the same matrix or bias vector are assumed to be in the same group), which results in 393 groups.

Original:  R ESULTS AND A NALYSIS
Revised:  R ESULTS AND A NALYSIS

Original: Our main results on the GLUE benchmark are shown in Table 1.
Revised: Our main results on the GLUE benchmark are shown in Table 1.

Original: Structured diff pruning can match the performance of a fully ﬁnetuned BERT LARGE model while only requiring 0.5% additional parameters per task.
Revised: Structured diff pruning can match the performance of a fully ﬁnetuned BERT LARGE model while only requiring 0.5% additional parameters per task.

Original: Diff pruning without structured sparsity also performs well, though slightly worse than the structured approach.
Revised: Diff pruning without structured sparsity also performs well, though slightly worse than the structured approach.

Original: Non-adaptive diff pruning, which magnitude prunes the diff vector without learning the binary mask z τ , performs signiﬁcantly worse, indicating the importance oflearning the masking vector.
Revised: Non-adaptive diff pruning, which magnitude prunes the diff vector without learning the binary mask z τ , performs signiﬁcantly worse, indicating the importance of

Original: Compared to adapters, diff pruning obtains similar performance while Our grouping is based on the Transformer implemen- tation available at https://github.com/huggingface/transformers/blob/master/src/ transformers/modeling_bert.py .
Revised: / src/transformers/modeling_bert.py ).

Original: In preliminary experiments we found this simple deﬁnition to work well compared to alternative group deﬁnitions (e.g. based on individual neurons).
Revised: In preliminary experiments we found this simple deﬁnition to work well compared to alternative group deﬁnitions (e.g. based on individual neurons).

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original:  requiring less parameters per task, making it a potential alternative for parameter-efﬁcient transfer learning.
Revised: learning the masking vector. Compared to adapters, diff pruning obtains similar performance while requiring fewer parameters per task, making it a potential alternative for parameter-efﬁcient transfer learning.

Original: 10 We now perform a series of analysis experiments on the development set.
Revised: 11 We now perform a series of analysis experiments on the validation set.

Original: 5. V ARYING THE T ARGET S PARSITY
Revised: 5. V ARYING THE T ARGET S PARSITY

Original: In Figure 1 (left), we plot results on the GLUE validation set averaged across all tasks at target sparsity rates of 0 .
Revised: In Figure 1 (left), we plot results on the GLUE validation set averaged across all tasks at target sparsity rates of 0 .

Original: 5% , 1 . 0% for the different baselines.
Revised: 5% , 1 . 0% for the different baselines.

Original: Structured diff pruning consistently outperforms non-structured and and non-adaptive variants across different sparsity rates.
Revised: Structured diff pruning consistently outperforms non-structured and and non-adaptive variants across different sparsity rates.

Original: The advantage of adaptive methods becomes more pronounced at extreme sparsity rates.
Revised: The advantage of adaptive methods becomes more pronounced at extreme sparsity rates.

Original: In Table 2, we report the breakdown of accuracy of structured diff pruning across different tasks and sparsity rates, where we observe that different tasks have different sensitivity to target sparsity rates.
Revised: In Table 2, we report the breakdown of accuracy of structured diff pruning across different tasks and sparsity rates, where we observe that different tasks have different sensitivity to target sparsity rates.

Original: This suggests that we can obtain even greater parameter-efﬁciency through targeting task-speciﬁc sparsity rates in the diff vector.
Revised: This suggests that we can obtain even greater parameter-efﬁciency through targeting task-speciﬁc sparsity rates in the diff vector.

Original: 5. S TRUCTURED VS . N ON -S TRUCTURED D IFF P RUNING
Revised: 5. S TRUCTURED VS . N ON -S TRUCTURED D IFF P RUNING

Original: Structured diff pruning introduces an additional mask per group, which encourages pruning of entire groups.
Revised: Structured diff pruning introduces an additional mask per group, which encourages pruning of entire groups.

Original: This is less restrictive than traditional group sparsity techniques that have been used with
Revised: This is less restrictive than traditional group sparsity techniques that have been used with

Original: L 0 -norm relaxations which force all parameters in a group to share the same mask (Louizos et al., 2018; Wang et al., 2019b).
Revised: L 0 -norm relaxations which force all parameters in a group to share the same mask (Louizos et al., 2018; Wang et al., 2019b).

Original: However we still expect entire groups to be pruned out more often in the structured case, which might bias the learning process towards either eliminating completely or clustering together nonzero diffs.
Revised: However we still expect entire groups to be pruned out more often in the structured case, which might bias the learning process towards either eliminating completely or clustering together nonzero diffs.

Original: In Figure 1 (right), we indeed ﬁnd that structured diff pruning leads to ﬁnetuned models that are much more likely to leave entire groups unchanged from their pretrained values (zero diffs).
Revised: In Figure 1 (right), we indeed ﬁnd that structured diff pruning leads to ﬁnetuned models that are much more likely to leave entire groups unchanged from their pretrained values (zero diffs).

Original: 5. T ASK -S PECIFIC S PARSITY
Revised: 5. T ASK -S PECIFIC S PARSITY

Original: Different layers of pretrained model have argued to encode different information (Liu et al.
Revised: Different layers of pretrained models have argued to encode different information (Liu et al., 2019a;

Original: ; Tenney et al., 2019).
Revised: Tenney et al., 2019).

Original: Given that each task will likely recruit different kinds of language phenomena embedded in the hidden layers, we hypothesize that diff pruning will modify different parts of the pretrained model through task-speciﬁc ﬁnetuning.
Revised: Given that each task will likely recruit different kinds of language phenomena embedded in the hidden layers, we hypothesize that diff pruning will modify different parts of the pretrained model through task-speciﬁc ﬁnetuning.

Original: Figure 2 shows the percentage of nonzero diff parameters attributable to the different layers for each task.
Revised: Figure 2 shows the percentage of nonzero diff parameters attributable to the different layers for each task.

Original: We ﬁnd that different tasks indeed modify different parts of the network, although there are some qualitative similarities between sometasks,
Revised: We ﬁnd that different tasks indeed modify different parts of the network, although there are some qualitative similarities between some

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original:  for example between QNLI & QQP (both must encode questions) [Table] and MRPC & STS-B (both must predict similarity between sentences).
Revised: tasks, for example between QNLI & QQP (both must encode questions), and MRPC & STS-B (both must predict similarity between sentences).

Original: The embedding layer is very sparsely modiﬁed for all tasks.
Revised: The embedding layer is very sparsely modiﬁed for all tasks.

Original: While some of the variations in the sparsity distributions is due to simple randomness [Table] we do observe some level of consistency over multiple runs of the same task, as shown in Figure 3 of the appendix.
Revised: While some of the variations in the sparsity distributions is due to simple randomness, we do observe some level of consistency over multiple runs of the same task, as shown in Figure 3 of the appendix.

Original: The ability to modify different parts of the pretrained model for each task could explain the improved parameter-efﬁciency of our approach compared to Houlsby et al.
Revised: The ability to modify different parts of the pretrained model for each task could explain the improved parameter-efﬁciency of our approach compared to Houlsby et al.

Original: (2019)’s adapter layers, which can only read/write to the pretrained model at certain points of the computational graph.
Revised: (2019)’s adapter layers, which can only read/write to the pretrained model at certain points of the computational graph.

Original: This potentially suggests that adapter layers with more ﬁne-grained access into model internals (e [Table] g. adapters for key/value/query transformations) might result in even greater parameter-efﬁciency.
Revised: This potentially suggests that adapter layers with more ﬁne-grained access into model internals (e.g. adapters for key/value/query transformations) might result in even greater parameter-efﬁciency.

Original: While left as future work, we also note that diff pruning can be applied in conjunction with adapters, which might further improve results.
Revised: While left as future work, we also note that diff pruning can be applied in conjunction with adapters, which might further improve results.

Original: 5. E FFECT OF L 0 - BALL P ROJECTION VIA M AGNITUDE P RUNING
Revised: 5. E FFECT OF L 0 - BALL P ROJECTION VIA M AGNITUDE P RUNING

Original: Applying magnitude pruning to project onto the L 0 -ball was crucial in achieving exact sparsity targets.
Revised: Applying magnitude pruning to project onto the L 0 -ball was crucial in achieving exact sparsity targets.

Original: As shown in Table 3, we observed little loss in performance through magnitude pruning.
Revised: As shown in Table 3, we observed little loss in performance through magnitude pruning.

Original: We re-iterate that it was crucial to ﬁnetune with the ﬁxed mask in order to maintain good performance.
Revised: We re-iterate that it was crucial to ﬁnetune with the ﬁxed mask in order to maintain good performance.

Revised: 5. SQ U AD E XTRACTIVE Q UESTION A NSWERING

Revised: To demonstrate the effectiveness of our approach beyond classiﬁcation, we additionally experiment on the extractive question answering task SQuAD, which asks model to select the answer span to a question given a Wikipedia paragraph.

Revised: To make direct comparisons with Houlsby et al.

Original: D ISCUSSION 6. M EMORY R EQUIREMENTS
Revised: (2019), we run all experiments on SQuAD v1.1. For diff pruning, we use the same general hyper-parameters as our full ﬁnetuning baseline. 14 Results are shown in Table 4. Diff pruning is able achieve comparable or better performance with only 1% additional parameters with the both a pretrained BERT LARGE .

Revised: Notably, we see that our method can improve the F1 score of full ﬁnetuning baseline by a signiﬁcant margin (e.g. 90 .

Revised: 8% ⇒ 93 .

Revised: 2% ) while modifying many fewer parameters (e.g., 100% ⇒ 1% ), which potentially implies that diff pruning can have a useful regularization effect.

Revised: [Table]

Revised:  D ISCUSSION 6. M EMORY R EQUIREMENTS

Original: For training, our approach requires more memory than usual ﬁnetuning due to additionally optimizing α τ and w τ .
Revised: For training, our approach requires more memory than usual ﬁnetuning due to additionally optimizing α τ and w τ .

Original: This did not present a signiﬁcant challenge for pretrained models that we experimented with in this study, since majority of GPU memory was utilized by the minibatch’s activation layers.
Revised: This did not present a signiﬁcant challenge for pretrained models that we experimented with in this study, since majority of GPU memory was utilized by the minibatch’s activation layers.

Original: However, this could present an issue as model sizes get larger and larger.
Revised: However, this could present an issue as model sizes get larger and larger.

Original: After training, storing the task-speciﬁc diff vector requires storing a compressed version with both the nonzero positions and weights, which incurs additional storage requirements.
Revised: After training, storing the task-speciﬁc diff vector requires storing a compressed version with both the nonzero positions and weights, which incurs additional storage requirements.

Original: 6. I NFORMATION - EFFICIENT T RANSFER L EARNING
Revised: 6. I NFORMATION - EFFICIENT T RANSFER L EARNING

Original: Efﬁciently representing pretrained models adapted to new tasks is becoming an increasingly important problem in contemporary NLP.
Revised: Efﬁciently representing pretrained models adapted to new tasks is becoming an increasingly important problem in contemporary NLP.

Original: This paper focuses on a rather narrow deﬁnition of efﬁciency— parameter-efﬁciency.
Revised: This paper focuses on a rather narrow deﬁnition of efﬁciency— parameter-efﬁciency.

Original: An interesting direction might be to target generalizations of parameterefﬁciency, for example, information-efﬁciency, which aims to minimize the number of bits required to represent the task-speciﬁc model when given the pretrained model for free.
Revised: An interesting direction might be to target generalizations of parameterefﬁciency, for example, information-efﬁciency, which aims to minimize the number of bits required to represent the task-speciﬁc model when given the pretrained model for free.

Original: This view can suggest other avenues for achieving information-efﬁcient transfer learning: for example, “what is the minimum number of (potentially synthetic) datapoints that we can ﬁnetune BERT on to obtain a good task-speciﬁc model?”, 13 or “what is the shortest preﬁx string that we can condition GPT3 on for it to become a good task-speciﬁc model”?
Revised: This view can suggest other avenues for achieving information-efﬁcient transfer learning: for example, “what is the minimum number of (potentially synthetic) datapoints that we can ﬁnetune BERT on to obtain a good task-speciﬁc model?”, 15 or “what is the shortest preﬁx string that we can condition GPT3 on for it to become a good task-speciﬁc model”?

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Multi-task learning Multi-task learning (Caruana, 1997), broadly construed, aims to learn models and representations that can be utilized across a diverse range of tasks, and offers a natural approach to training parameter-efﬁcient deep models.
Revised: Multi-task learning Multi-task learning (Caruana, 1997), broadly construed, aims to learn models and representations that can be utilized across a diverse range of tasks, and offers a natural approach to training parameter-efﬁcient deep models.

Original: Several works have shown that a single BERT model can obtain good performance across multiple tasks when jointly trained (Liu et al., 2019a; Clark et al., 2019; Stickland & Murray, 2019).
Revised: Several works have shown that a single BERT model can obtain good performance across multiple tasks when jointly trained (Liu et al., 2019b; Clark et al., 2019; Stickland & Murray, 2019).

Original: Adapter layers, which are task-speciﬁc layers that read and write to layers of a shared model (Rebufﬁ et al., 2018), offer an alternative approach to multi-task learning that does not require access to all tasks during training, and have also been applied to obtain parameter-efﬁcient BERT models (Houlsby et al., 2019; Pfeiffer et al., 2020a;b;c).
Revised: Adapter layers, which are task-speciﬁc layers that read and write to layers of a shared model (Rebufﬁ et al., 2018), offer an alternative approach to multi-task learning that does not require access to all tasks during training, and have also been applied to obtain parameter-efﬁcient BERT models (Houlsby et al., 2019; Pfeiffer et al., 2020a;b;c).

Original: A related line of work targets extreme parameter-efﬁciency through task-agnostic sentence representations that can be used without ﬁnetuning for downstream tasks (Le & Mikolov, 2014; Kiros et al., 2015; Wieting et al., 2016; Hill et al., 2016; Arora et al., 2017; Conneau et al., 2017; Cer et al., 2018; Zhang et al., 2018;
Revised: A related line of work targets extreme parameter-efﬁciency through task-agnostic sentence representations that can be used without ﬁnetuning for downstream tasks (Le & Mikolov, 2014; Kiros et al., 2015; Wieting et al., 2016; Hill et al., 2016; Arora et al., 2017; Conneau et al., 2017; Cer et al., 2018; Zhang et al., 2018;

Original: Subramanian et al., 2018; Zhang et al., 2020).
Revised: Subramanian et al., 2018; Zhang et al., 2020).

Original: Reimers & Gurevych (2019), building on the earlier work of Conneau et al.
Revised: Reimers & Gurevych (2019), building on the earlier work of Conneau et al.

Original: (2017), show that BERT ﬁnetuned on natural language inference obtains sentence representations that perform well across multiple sentence-level tasks.
Revised: (2017), show that BERT ﬁnetuned on natural language inference obtains sentence representations that perform well across multiple sentence-level tasks.

Original: These feature-based transfer learning methods are however generally outperformed by fully ﬁnetuned models (Howard & Ruder, 2018).
Revised: These feature-based transfer learning methods are however generally outperformed by fully ﬁnetuned models (Howard & Ruder, 2018).

Original: Model compression There has been much recent work on compressing pretrained trained with self-supervision (see Ganesh et al.
Revised: Model compression There has been much recent work on compressing pretrained trained with self-supervision (see Ganesh et al.

Original: (2020) for a recent survey).
Revised: (2020) for a recent survey).

Original: A particularly promising line of work focuses on obtaining smaller pretrained models (for subsequent ﬁnetuning) through weight pruning (Gordon et al., 2020; Sajjad et al., 2020; Chen et al., 2020) and/or knowledge distillation (Sanh et al., 2019; Sun et al., 2019;
Revised: A particularly promising line of work focuses on obtaining smaller pretrained models (for subsequent ﬁnetuning) through weight pruning (Gordon et al., 2020; Sajjad et al., 2020; Chen et al., 2020) and/or knowledge distillation (Sanh et al., 2019; Sun et al., 2019;

Original: Turc et al., 2019; Jiao et al., 2019; Sun et al., 2020).
Revised: Turc et al., 2019; Jiao et al., 2019; Sun et al., 2020).

Original: It would be interesting to see whether our approach can be applied on top of these smaller pretrained models to for even greater parameter-efﬁciency.
Revised: It would be interesting to see whether our approach can be applied on top of these smaller pretrained models to for even greater parameter-efﬁciency.

Original: Learning to prune Our work is closely related to the line of work on learning to prune pretrained models with differentiable relaxations of binary masks (Wang et al., 2019b;
Revised: Learning to prune Our work is closely related to the line of work on learning to prune pretrained models with differentiable relaxations of binary masks (Wang et al., 2019b;

Original: Zhao et al., 2020; Sanh et al., 2020; Radiya-Dixit & Wang, 2020).
Revised: Zhao et al., 2020; Sanh et al., 2020; Radiya-Dixit & Wang, 2020).

Original: While these works also enable parameter-efﬁcient transfer learning, they generally apply the masks directly on the pretrained parameters instead of on the difference vector as in the present work.
Revised: While these works also enable parameter-efﬁcient transfer learning, they generally apply the masks directly on the pretrained parameters instead of on the difference vector as in the present work.

Original: Regularization towards pretrained models Finally, diff pruning is also related to works which regularize the learning process towards pretrained models for continual learning (Kirkpatrick et al., 2017; Schwarz et al., 2018), domain adaptation (Wiese et al., 2017; Miceli Barone et al., 2017), and stable ﬁnetuning (Lee et al., 2020).
Revised: Regularization towards pretrained models Finally, diff pruning is also related to works which regularize the learning process towards pretrained models for continual learning (Kirkpatrick et al., 2017; Schwarz et al., 2018), domain adaptation (Wiese et al., 2017; Miceli Barone et al., 2017), and stable ﬁnetuning (Lee et al., 2020).

Original: However these works typically do not utilize sparse regularizers, and target a different goal than parameter-efﬁciency.
Revised: These works typically do not utilize sparse regularizers and target a different goal than parameter-efﬁciency.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose diff pruning as a simple approach for parameter-efﬁcient transfer learning with pretrained models.
Revised: We propose diff pruning as a simple approach for parameter-efﬁcient transfer learning with pretrained models.

Original: Experiments on standard NLP benchmarks and models show that diff pruning can match the performance of fully ﬁnetuned baselines while requiring only a few additional parameters per task.
Revised: Experiments on standard NLP benchmarks and models show that diff pruning can match the performance of fully ﬁnetuned baselines while requiring only a few additional parameters per task.

Original: We also propose a structured variant of diff pruning which provides further improvements.
Revised: We also propose a structured variant of diff pruning which provides further improvements.

Original: Future work will consider (i) applying this approach to other architectures (e.g. ConvNets for vision applications), (ii) injecting parameter-efﬁciency objectives directly into the pretraining process (to pretrain models that are better suited towards sparse transfer learning), and (iii) combining diff pruning with other techniques (e.g. adapters) to achieve even greater parameter-efﬁciency.
Revised: Future work will consider (i) applying this approach to other architectures (e.g. ConvNets for vision applications), (ii) injecting parameter-efﬁciency objectives directly into the pretraining process (to pretrain models that are better suited towards sparse transfer learning), and (iii) combining diff pruning with other techniques (e.g. adapters) to achieve even greater parameter-efﬁciency.

Original: A A PPENDIX A. C ONSISTENCY OF N ONZERO P ARAMETERS
Revised: A A PPENDIX A. C ONSISTENCY OF N ONZERO P ARAMETERS

Original: Figure 3 shows the percentage of modiﬁed parameters attributable to each layer across 5 runs of SST2.
Revised: Figure 3 shows the percentage of modiﬁed parameters attributable to each layer across 5 runs of SST2.

Original: We ﬁnd that there is nonotrivial variation in sparsity across runs, but also a degree of consistency.
Revised: We ﬁnd that there is nonotrivial variation in sparsity across runs, but also a degree of consistency.

Original: For example, the ﬁrst layer is modiﬁed considerably more than other layers across all runs.
Revised: For example, the ﬁrst layer is modiﬁed considerably more than other layers across all runs.