DSEE: D UALLY S PARSITY - EMBEDDED E FFICIENT T UNING OF P RE - TRAINED L ANGUAGE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Gigantic pre-trained models have become central to natural language processing (NLP), serving as the starting point for ﬁne-tuning towards a range of downstream tasks.
However, two pain points persist for this paradigm: (a) as the pre-trained models grow bigger (e.g., 175 B parameters for GPT-3), even the ﬁne-tuning process can be time-consuming and computationally expensive; (b) the ﬁne-tuned model has the same size as its starting point by default, which is neither sensible due to its more specialized functionality, nor practical since many ﬁnetuned models will be deployed in resource-constrained environments.
To address these pain points, we propose a framework for resource- and parameter-efﬁcient ﬁne-tuning by leveraging the sparsity prior in both weight updates and the ﬁnal model weights.
Our proposed framework, dubbed D ually S parsity- E mbedded
E fﬁcient Tuning (DSEE), aims to achieve two key objectives: (i) parameter efﬁcient ﬁne-tuning - by enforcing sparsity-aware weight updates on top of the pretrained weights; and (ii) resource-efﬁcient inference - by encouraging a sparse weight structure towards the ﬁnal ﬁne-tuned model.
We leverage sparsity in these two directions by exploiting both unstructured and structural sparse patterns in pre-trained language models via magnitude-based pruning and (cid:96) 1 sparse regularization.
Extensive experiments and in-depth investigations, with diverse network backbones (i.e., BERT, GPT-2, and DeBERTa) on dozens of datasets, consistently demonstrate highly impressive parameter-/training-/inference-efﬁciency, while maintaining competitive downstream transfer performance.
For instance, our DSEE-BERT obtains about 35% inference FLOPs savings with < 1% trainable parameters and comparable performance to conventional ﬁne-tuning.
I NTRODUCTION
Most recent NLP applications have been following the pre-train then ﬁne-tune paradigm, where we start from a gigantic pre-trained model and ﬁne-tune it towards downstream tasks.
Conventional ﬁne-tuning works by updating all of the parameters of the pre-trained model.
As the size of pretrained models grows, updating all parameters becomes less feasible in most practical scenarios, due to the expensive memory and computational requirements.
For example, BERT BASE (Devlin et al., 2019) has 110 M trainable parameters, while GPT-2 (Radford et al., 2019) has up to 1 .
5 B and the largest version of GPT-3 (Radford et al., 2019) has an astonishing 175 B trainable parameters.
As such, conventional ﬁne-tuning of the larger models could require hundreds of GPU hours.
Another downside of this paradigm is that it requires storing as many parameters, as in the large-scale pretrained models, for each downstream task.
That poses impediments to the deployment in real-world resource-constrained environments.
One solution to address the extensive resource requirement of conventional ﬁne-tuning is model pruning (LeCun et al., 1990; Han et al., 2015a;
Ren et al., 2018;
He et al., 2017; Liu et al., 2017), where unnecessary weights are eliminated to shrink the model size.
For example, Chen et al.
(2021b) leverages (cid:96) 1 regularization to remove insigniﬁcant attention heads and gains 35% ∼ 45% training time with comparable performance.
Guo et al.
(2020) learns sparse task-speciﬁc “diff” vectors for various downstream ﬁne-tuning tasks, leading to great memory savings.
All these studies indicate the rise of sparsity naturally during ﬁne-tuning a general-purpose pre-trained model, to some specialized downstream functionality.
One potential interpretation, of why sparsity arises, is that different subsets of the parameters may be responsible for different downstream tasks and data domains (Sanh et al., 2020).
However, identifying appropriate sparse pruning masks requires burdensome training of models with full weights, which can still be unaffordable for many practitioners.
For example, ﬁne-tuning a large pre-trained language model like GPT-3 for just one step consumes at least 1 . 2 TB of VRAM and requires 96 pieces of NVIDIA Tesla (Hu et al., 2021).
One parallel alternative is designing parameter-efﬁcient ﬁne-tuning algorithms, which aims to only optimize a small portion of weights while ﬁxing most of the pre-trained weights during the downstream task training step.
Pioneering works along this line, which utilize adapters (Houlsby et al., 2019) or low-rank decomposition (Hu et al., 2021), can signiﬁcantly reduce the number of trainable parameters while preserving good ﬁne-tuning performance.
Introducing only a small number of task-speciﬁc parameters for each new downstream task can substantially improve the memory and deployment efﬁciency of models as it allows us to reuse the remaining unchanged/shared parameters.
However, there are two major hurdles of current parameter-efﬁcient ﬁne-tuning: ( i ) it does not yield any inference efﬁciency gains since the full pre-trained weights are still required to calculate outputs; and ( ii ) current methods assume the weight update to be either sparse (Guo et al., 2020) or low-rank (Hu et al., 2021), yet those assumptions might be oversimpliﬁed and overly restricted to allow for effective updates. For example, the low-rank assumptions on weight matrices might be overly strong since some weights cannot be ﬁtted in the low-rank space (Yu et al., 2017). Recently Chen et al.
(2021a)
also ﬁnd that using both sparse and low-rank components performs better than either of them.
These observations have inspired us to explore better parameter-efﬁciency methods. To tackle both resource- and parameter- efﬁciency issues of large model ﬁne-tuning, we explicitly draw on the prior of sparsity for both weight updates and the ﬁnal weights , and establish a dually sparsity-embedding efﬁcient tuning ( DSEE ) framework.
From a pre-trained model, DSEE ﬁrst adopts a sparsity-aware low-rank weight update to achieve parameter efﬁciency of the ﬁne-tuning process ; and then enforces a sparse weight structure by masking to achieve resource efﬁciency of the ﬁne- tuned model at inference time.
Our contributions can be summarized as follows:
• We propose the dually sparsity-embedding efﬁcient tuning (DSEE), which uniﬁes sparsityaware weight update and sparse pretrained weight in ﬁne-tuning gigantic pre-trained models.
It is the ﬁrst attempt towards jointly optimizing both parameter efﬁciency of the ﬁnetuning process, and the resource efﬁciency of the ﬁne-tuned model.
• We exploit both unstructured and structured sparse patterns in the DSEE framework.
For weight updates, we ﬁnd the injected sparsity prior to greatly enhance existing parameterefﬁcient update schemes, and to further trim down the needed amount of trainable parameters.
For the ﬁnal weights, we learn well-performed sparse masks from the pre-trained weights, leading to substantial computation and memory reductions at inference.
• Extensive experiments demonstrate the effectiveness of our proposal across various representative pre-trained languages models, such as BERT, GPT-2, and DeBERTa; and on diverse evaluation benchmarks, such as E2E, DART, WebNLG, and GLUE.
Speciﬁcally, on (E2E, Dart, WebNLG), our methods can achieve a BLUE score of (69 . 75 , 55 .
40 , 46 . 66) with less than 1% of total trainable parameters.
On BERT, our method can save about 35% FLOPs, compared to traditional downstream ﬁne-tuning.
R ELATED W ORK
Pre-trained language models.
Transformer (Vaswani et al., 2017) is a sequence-to-sequence model that heavily uses the concept of self-attention.
Later on, numerous transformer-based models have been proposed and show overwhelming performance on natural language processing (NLP) and on computer vision (CV) tasks.
Devlin et al.
(2019) designed BERT that pre-train deep bidirectional representations from unlabeled text and reached powerful performance.
Liu et al.
(2019) found that BERT were terribly undertrained and proposed RoBERTa, an enhanced training recipe for BERT which can greatly boost the performance.
He et al.
(2020) proposed decoding-enhanced
BERT with disentangled attention (DeBERTa) that incorporates the disentangled attention mechanism and an improved mask encoder to enhance BERT and RoBERTa.
More variants like XL-Net,
Albert, and Electra have also been proposed in recent years (Yang et al., 2019; Lan et al., 2019; Clark et al., 2019).
The series of GPT models (Radford et al., 2019; Brown et al., 2020) are later developed based on transformers decoder blocks rather than encoder blocks like BERT, which again have shown superior performance on different tasks.
These large models pretrained on a large amount of unlabelled texts would need to be further ﬁne-tuned on downstream tasks for better performance.
One of the accompanying disadvantages of these pre-training models with tremendous parameter counts (e.g., 175 B in GPT-3) is the unaffordable computational cost for further ﬁne-tuning.
Pruning and Low-rank decomposition.
Pruning is a widely-used model compression technique.
It can reduce the number of parameters inside models, which possibly brings training and inference efﬁciency.
Along with weight pruning method (Han et al., 2015b) being one of the most effective methods (Gordon et al., 2020), various criterion have been proposed to select insigniﬁcant weights for pruning, such as Taylor approximation (Molchanov et al., 2019), Hessian score approximation (Hassibi & Stork, 1993), and other saliency scores such as SNIP (Lee et al., 2018),
GraSP (Wang et al., 2019) and SynFlow (Tanaka et al., 2020).
Several pruning methods have been commonly adapted to compress language models (McCarley et al., 2019; Gordon et al., 2020; Sanh et al., 2020; Wang et al., 2020; Chen et al., 2021b).
Speciﬁcally, McCarley et al.
(2019) proposed to prune attention heads that had less contribution to the model.
Wang et al.
(2020) pruned BERT models by involving low-rank factorization and (cid:96) 0 regularization.
Sanh et al.
(2020) invented an improved version of magnitude pruning ( i.e. , pruning based on the weight change) that can better suit the transfer learning.
Chen et al.
(2021b) performed structured pruning on BERT via (cid:96) 1 sparse regularization, which reduced a large portion of parameters and decreased the training cost.
Low-rank approximation (Ye, 2005) is also vastly studied.
One classical scenario is robust principle component analysis (Cand ` es et al., 2011), which decomposes a matrix into a low-rank plus a sparse component.
Existing literature shows that in deep learning, the learned over-parameterized models often naturally bear approximate low-rank weight structures (Oymak et al., 2019; Yu et al., 2017).
Some (Jaderberg et al., 2014; Povey et al., 2018; Sainath et al., 2013; Zhang et al., 2014; Zhao et al., 2016) have explicitly imposed the low-rank constraint during training.
Wang et al.
Hu et al.
(2021) utilized low-rank decomposition to shrink the model size and trim down the trainable parameters during ﬁne-tuning.
However, to our best knowledge, integrating sparsity and low-rank structures has never been studied before for efﬁcient ﬁne-tuning of pre-trained language models.
Parameter-efﬁcient adaptation.
Parameter-efﬁcient adaptation aims at reducing the number of trainable parameters when ﬁne-tuning the models across different downstream domains.
Unlike pruning, it generates updates that can be represented by fewer parameters instead of building sparse models.
Various approaches are invented to achieve the goal.
Rebufﬁ et al.
Houlsby et al.
(2019) inserted and only trained adapters between existing layers, whose parameters are much less compared to the pretrained models.
Guo et al.
leveraged (cid:96)
regularization to limit the number of non-zero elements in the update vectors.
Lester et al.
; Li & Liang (2021) introduced efﬁcient prompt tuning which optimizes only a small continuous task-speciﬁc vector.
Hu et al.
(2021) proposed a low-rank decomposition-based method that can also signiﬁcantly reduce the number of trainable parameters.
However, ﬁne-tuned models yielded by these methods work have the same amount of weights as the pre-trained starting point; hence they contribute no resource efﬁciency of the ﬁnal model.
M ETHODOLOGY
In this section, we begin by describing our notations and deﬁnitions of sparsity generation and parameter-efﬁcient ﬁne-tuning in Section 3.1.
Then, we introduce the (dually) sparsity-embedded efﬁcient ﬁne-tuning algorithms in Sections 3.2 and 3.3.
3. P RELIMINARIES
Sparsity generation and resource-efﬁcient ﬁne-tuning.
We adopt both unstructured and structured pruning methods to produce sparsity.
They can lead to resource-efﬁciency including memory and computation savings.
Let W ∈ R m × n denote a weight matrix.
The goal of pruning is to ﬁnd a binary mask S ∈ { 0 , 1 } (cid:107)W(cid:107) 0 , where (cid:107)W(cid:107) 0 is the number of parameters in W .
The mask S is applied to W and results in a sparse weight W (cid:12) S .
For unstructured pruning, only memory cost is saved; but for structured pruning, it helps save computational cost since the sparse weights can be smaller in size by wiping out all-zero columns or rows. However, the performance of networks after structured pruning is often shown to be inferior compared with the unstructured pruning counterpart. [Figure]
Parameter-efﬁcient ﬁne-tuning.
To leverage the knowledge in pre-trained weights W , downstream models learn task-speciﬁc weight update ∆ W via ﬁne-tuning and generate predictions with weights W + ∆ W . The output of models is therefore calculated as ( W + ∆ W ) x where x is the input. Since ∆ W has the same size of W , learning the update matrices usually requires massive resources as the size of the pre-trained model increases.
Parameter-efﬁcient ﬁne-tuning try to solve this problem by using as few trainable parameters as possible to represent ∆ W , while maintaining competitive downstream ﬁne-tuning performance.
Previous literature reaches the goal via either sparsifying weight update matrices ∆ W (Guo et al., 2020) or leveraging low-rank decomposed matrices to compute ∆ W (Hu et al., 2021), while in our work we combine both of them.
Algorithm 1:
Sparsity-Embedded LowRank Decomposition
Input: Pretrained weights W , number of non-zero elements N
Output: Sparse matrices S1 Initialize S 2 to be set.
2 for each self-attention projection weights w i in W do / * Decomposition */
Perform matrix decomposition: w i ≈ UV + S (cid:48) by solving the optimization problem in Eqn.1.
/* Identify important elements to form S*/
Perform thresholding on S (cid:48) :
Keep N elements in S (cid:48) with top magnitudes, and set the rest 0.
Append S (cid:48) into S 2 .
6 end
Algorithm
3 Train U , V , S with respect to L under the constraint of P Ω C ( S ) = 0 .
model */ 4 Prune (1-s%) parameters in W globally by sorting the magnitude of W + UV + S , deriving the sparsity mask S 1 /
5 Tune U V for E epochs for recovering the performance.
3. S PARSITY -E MBEDDED P ARAMETER -E FFICIENT F INE - TUNING
A recent study (Hu et al., 2021) enforces low-rank constraint to weight update tensors ∆ W , and obtains a satisfactory trade-off between parameter-efﬁciency and model quality.
However, as revealed experimentally by (Yu et al., 2017), a part of the important information in the trained weights will also scatter outside the low-rank subspace, creating sparse “residuals”.
Inspired by these observations, we investigate a new sparsity-aware low-rank subspace of ∆ W , and introduce the ﬁrst component of our proposal in Figure 1, i.e., sparsity-embedded parameter-efﬁcient ﬁne-tuning.
Speciﬁcally, we identify a sparse subspace Ω , that we can project our update matrices ∆ W to.
The update matrix is then decomposed into two components: a low-rank part which is represented by the multiplication of two low-rank matrices U ∈ R m × r and V ∈ R r × n , and a sparse residual
S 2 = P Ω (∆ W ) , where P Ω (∆ W ) =
[Equation]
s i
[Equation]
As illustrated in Figure 1, the update matrix ∆ W can be approximated by UV + S 2 , in which the
U , V , and the non-zero element of S 2 are learnable parameters while Ω is ﬁxed once determined.
Compared to the full ﬁne-tuning schemes has m × n individual trainable parameters, our method only has ( m + n ) × r +card(Ω) .
If r is smaller than m × n − card(Ω) m + n (cid:47) 0 . 5 min { m, n } , which is a loose bound, our method is capable of substantially reducing trainable parameters for downstream ﬁnetuning. In practice, the value of r is very small compared to m and n so the savings are considerable.
The next question is how to ﬁnd appropriate Ω .
Motivated by (Cand`es et al., 2011), we formulate our sparsity-aware decomposition as the following optimization problem:
[Equation]
where rank( · ) indicates the rank and card( · ) indicates the cardinality of a matrix.
Here we derive Ω from pre-trained weights W , which is different from previous low-rank techniques for model compression since we perform prior-training decomposition and can not access ∆ W before ﬁnetuning.
In this way, we actually assume that ∆ W shares a similar crucial subspace Ω with W (Sun et al., 2018).
We use the GreBsmo algorithm (Zhou & Tao, 2013) to solve this optimization problem.
This method adopts a similar idea as random projection and can solve the optimization problem rapidly.
The details of the algorithm are in Appendix.
Algorithm 1 summarizes the detailed procedure of our proposal.
We ﬁrst decompose the pretrained weights W into three matrices: U , V and S 2 .
U and V are matrices of low-rank and S 2 is a sparse matrix.
After the decomposition, we re-initialize U and V to be of shape R m × r and R r × n .
The initial values for U are set to 0, and the initial values for V will follow a normal distribution as Hu et al.
(2021) did.
We do not use the decomposed value inside S 2 , but only the index of non-zero elements.
We collect the indexes of non-zero elements into Ω , and reset the values of S 2 back to 0.
During the training, we only update the elements of S 2 whose indexes are in Ω , as well as U and V .
3. D UALLY S PARSITY -E MBEDDED E FFICIENT T UNING (DSEE)
Besides parameter-efﬁciency, resource-efﬁciency is another major goal of our proposed framework.
Sparsity-embedded low-rank weight updates alone do not necessarily lead to computational cost reductions since it works together with the dense pre-trained weights and the massive computation during forwarding processes is inevitable.
To achieve both resource- and parameter-efﬁciency, we also promote a sparse weight structure to the ﬁnal ﬁne-tuned model, as demonstrated by the sparse mask S 1 in Figure 1.
Our DSEE explores both unstructured/structured sparsity patterns as follows.
(cid:66)
Pruning with unstructured sparse masks.
The unstructured sparse mask is the most widely used mask since it usually produces almost undamaged performance compared to its dense counterpart (Han et al., 2015a).
It applies ﬁne-grained manipulation on each individual model weight, while the generated irregular sparse masks bring limited hardware speedup (Wen et al., 2016).
Speciﬁcally, based on the weight matrix W , a sparse mask S 1 can be calculated with various approaches, either heuristic or optimization-based.
We adopt the one-shot magnitude pruning (Han et al., 2015a) in
DSEE, due to its simplicity and competitive performance.
In our context, we create the sparse pattern S 1 from W + UV : First, it sorts the magnitude (i.e., absolute value) of individual weights and removes parameters with bottom 1 − k % magnitude.
Second, we further tune the value of the low-rank update matrices for a few epochs, which is important to keep the performance as stated in Han et al.
(2015b).
Third, the sparse mask Sthat we derive will be applied on the pretrained weights only and do not affect the output from the update matrices, as W (cid:12) S 1 + UV .
For an input sample x , the output is calculated by ( W (cid:12) S 1 + UV + S 2 ) x = W (cid:12) S 1 x + ( UV + S 2 ) x , where the ﬁrst term W (cid:12) S 1 x brings signiﬁcant resource-efﬁciency, and the second term brings parameter-efﬁciency.
(cid:66)
Pruning with structured sparse masks.
The second method exploits structured sparse masks, whose regular patterns are more hardware friendly yet have worse performance than unstructured masks.
Our DSEE considers (cid:96) 1 sparse regularization (Liu et al., 2017; Chen et al., 2021b) to craft high-quality structurally pruned subnetworks.
More precisely, motivated by Chen et al.
(2021b), we introduce learnable coefﬁcients ξ before each attention head module, and append a (cid:96) 1 sparse penalty on ξ to the original loss.
Detailed formulations are depicted as below:
We add trainable coefﬁcients c before attention heads.
The parameters c are optimized together with the decomposed matrices, i.e. , U , V and S 2 .
An extra term λ (cid:107) c (cid:107) 1 will be added to the training loss for sparse regularization.
The value of λ is set to 1e-4.
After training, we prune the attention heads that have the lowest contribution to the model ( i.e., lowest c ).
We use a layer-wise pruning scheme that prunes the same proportion of heads in each attention layer.
After pruning, several epochs of tuning are run to recover the performance of the pruned model.
The size of update matrices will change after structured pruning, so we also need to change the dimension of U and V .
Speciﬁcally, we change the size of V from R r × n to R r × [ n × s ] where s is the pruning ratio of the corresponding self-attention layer and [ x ] is the biggest integer not greater than x .
The size of S 2 is shrunk accordingly.
E XPERIMENT R ESULTS
Datasets and models.
We use three classical pre-trained language models in our experiments:
BERT BASE (Devlin et al., 2019), GPT-2 (Radford et al., 2019), and DeBERTa-large (He et al., 2020), which have 12 / 24 / 24 layers with hidden size of 768 / 1024 / 1024 and 110 / 354 / 380 M trainable parameters, respectively.
For BERT and DeBERTa, we evaluate our method on the GLUE benchmarks (Wang et al., 2018).
For GPT-2, we use E2E (Novikova et al., 2017), WebNLG (Gardent et al., 2017), and DART (Nan et al., 2021) for evaluations.
Training and evaluation details.
For BERT and DeBERTa, we follow the default settings in Wolf et al.
; Devlin et al.
We use the AdamW (Loshchilov & Hutter, 2017) optimizer for downstream training.
The batch size for BERT and DeBERTa is 32, 8 per GPU.
We train three epochs to search the sparse mask S 1 , and continue to tune the model for three epochs to converge.
The initial learning rates are reported in A1, and we linearly decay them.
For GPT-2, we follow the same hyperparameters as in Hu et al.
We train the model for ﬁve epochs to search for the mask, and further tune the pruned model for two epochs.
Evaluation Metrics.
For tasks on BERT and DeBERTa, we use the accuracy, Matthew’s Correlation, and Pearson’s r in the evaluation by default, which is also the conventional setting in the NLP community.
On GPT-2, we use BLEU (Papineni et al., 2002), METEOR (Denkowski & Lavie, 2014), TER (Snover et al., 2006) and NIST (Doddington, 2002) as our evaluation metrics.
To evaluate the efﬁciency of models, we use the number of trainable parameters to measure the parameter efﬁciency, use Sparsity in Pretrained Weights to measure the resource efﬁciency, and FLOPs to evaluate the computational efﬁciency.
We add a star sign ( ∗ ) to indicate the structured sparsity ( i.e. , structurally prune the pretrained weights).
Baselines.
On BERT and DeBERTa, we conduct comparisons with the following baseline methods: (cid:182) Fine-tune: directly ﬁne- tunes the full model; (cid:183) EarlyBERT (Chen et al., 2021b); (cid:184) BERT
Tickets (Chen et al., 2020); (cid:185) OMP: prunes the ﬁne-tuned weights by magnitude, and ﬁne-tune afterwards; and (cid:186)
LoRA
(Hu et al., 2021).
We also report the Huggingface’s ﬁne-tuning results.
On GPT-2, we conduct comprehensive comparisons with multiple baseline methods: (cid:182)
Adapters (Houlsby et al., 2019): insert adapters after linear layers; (cid:183) FT-Top2: ﬁne-tune the top 2 layers only; (cid:184) :
Preﬁx: preﬁx tuning introduced by Li & Liang (2021); and (cid:185) LoRA: low-rank decomposition, which assumes ∆ W = UV ; most results are directly cited from Hu et al.
4. E FFICIENT T UNING WITH DSEE
Parameter-efﬁciency with sparse masks.
To verify that using simple low-rank adaptation ( i.e. , LoRA) has limitations, we compare its performance with the performance of our sparsity-embedded efﬁcient ﬁne-tuning.
Table 1 proves that on MNLI, SST-2,
CoLA, and STS-B, the simple decomposition form shows unsatisfactory results compared to sparsity-embedded decomposition.
Adding a small proportion of parameters (onlytrainable parameters) can bring a performance boost to the model.
Speciﬁcally, under rank 8, the metrics increase (0.69/0.13/0.008/0.003) on SST-2/MNLI/CoLA/STS-B, respectively.
Moreover, using only approximately half of the number of trainable parameters, our method can achieve comparable parameters compared to the state-of-the-art method LoRA. [Table]
Table 2 shows the performance on GPT-2.
For all three tasks, our method can achieve comparable results with only about half of the trainable parameters with LoRA with rank four and make substantial performance gain compared to LoRA with rank two. On
WebNLG, our method even achieves a higher BLEU score, 55.56 versus 55.29, with half of the trainable parameters.
Such a phenomenon indicates that adding a sparse matrix to the low-rank decomposition can make substantial improvement.
[Table]
Resource- and parameter-efﬁciency with sparse masks.
We report the performance of DSEE, including both the number of trainable parameters and the sparsity in pretrained weights.
We use an unstructured sparsity of 50%, and structured sparsity of 25% and 33%, which is equal to pruning 1/4 and 1/3 heads from each attention layer.
For BERT BASE and DeBERTa-large, we set the r (the low-rank dimension) to be 16, and N (the number of non-zero elements in S 2 ) to be 64. We also prune each of the intermediate layers using a structured sparsity of 40% as in (Chen et al., 2021b).
For GPT-2, we set r to be 2 in DSEE and 4 in LoRA.
The choice of N remains 64.
Table 3 summarizes the performance on BERT BASE .
On BERT BASE , our DSEE method can: (cid:182) achieve parameter-efﬁciency at high level and retain the performance on various downstream tasks.
On BERT BASE , with only about 600K trainable parameters ( 110 / 0 .
≈ 200 × smaller than the full model) , DSEE can achieve comparable performance on all GLUE tasks.
(cid:183) DSEE can achieve resource-efﬁciency in the ﬁnal models.
Using either unstructured or structured sparse masks can reach our goal of reducing the number of parameters in the pretrained weights without sacriﬁcing much performance.
At the sparsity level of 50%, unstructured DSEE can achieve an accuracy of 90.46% on QNLI, surpassing the baseline of ﬁne-tuning, whose result is 90.15%.
At the sparsity level of 25%, the structured DSEE can have performance gains ranging from 0.46% to 2.22%, except for QQP and QNLI.
At the sparsity level of 33%, the performance gains range from 0.11% to 2.04%, except for QQP and QNLI.
(cid:184)
Compared to other baselines, e.g., EarlyBERT and BERT Tickets, our method beneﬁts from parameter efﬁciency, which updates the weights more efﬁciently.
These observations validate the effectiveness of our DSEE method.
We also calculate the inference FLOPs of BERT BASE on STS-B dataset.
The original BERT BASE on STS-B takes FLOPs of 3 .
7835 × 10 14 , while LoRA takes FLOPs of 3 . {+
+} × 10 14 , which is 0 .
69% higher.
Conversely, the structured version of DSEE takes a FLOPs of 2 .
4921 × 10 14 at the sparsity of 25% , and 2 . 3867 × 10 14 at the sparsity of 33% , which is 34.61% and 37.38% lower than LoRA.
This indicates that a large proportion of computational cost can be saved at inference phase.
[Table]
Table 4 summarizes the performance on GPT-2, which shares a similar trend.
Unstructured DSEE can achieve 2000 × reduction in trainable parameters and 2 × reduction in the ﬁnal ﬁne-tuned model size with almost no loss in downstream task performance compared to conventional ﬁnetuning.
When compared to LoRA, the unstructured DSEE can retain the same level of the number of trainable parameters and downstream task performance, while showing a 2 × reduction in the ﬁnal ﬁnetuned model size.
The structured DSEE on GPT-2 seems to be less competitive than on BERT
[Table]
but it can still hold performance on E2E and WebNLG after pruning 25% of heads in attention modules
[Table]
[Table]
[Table]
Finally
[Table]
we validate our method on DeBERTa
[Table]
The results are displayed in Table 5.
We use four datasets, CoLA, MNLI, MRPC
[Table] and RTE. They have greatly varied sizes, which are representatives of the GLUE benchmark.
DeBERTa is a larger model, so applying low-rank decomposition with our hyperparameters cannot match the performance of ﬁne-tuning.
Compared to LoRA, our method reaches higher performance on downstream tasks, albeit it needs slightly more training epochs.
However, such a slight extra cost for searching the sparse mask S 1 can be amortized by the efﬁciency gained in the future inference, since the burden on resources such as storing the pretrained weights is relieved.
4. U NDERSTANDING DSEE
[Table]
The position of embedded sparsity masks plays a crucial role in our proposed DSEE.
For instance, applying sparse masks to the pre-trained weights or weight updates produces resourceand parameter-efﬁciency.
Precisely, we compare four different methods on BERT BASE : one-shot magnitude weight pruning ( W (cid:12) S 1 ), two DSEE’s variants ( W (cid:12) S 1 + UV ) and W + UV + S 2 ), DSEE ( W (cid:12) S 1 + UV + S 2 ).
The results are collected in Table 6.
We can see that: (cid:182) NO embedded sparsity in the pretrained weights yields the overall best performance.
This is intuitive since the valuable knowledge learned from massive pretraining data is intact and not removed.
(cid:183)
Embedding sparsity into the pretrained weights harms only little to no performance.
A similar conclusion is drawn by Chen et al.
(2020) which validated that a pruned model can have a similar performance as the dense model.
(cid:184)
Using sparsity-embed efﬁcient ﬁne-tuning with the sparse pre-trained weights can also preserve performance, as well as achieve parameter efﬁciency.
4. A BLATION AND V ISUALIZATION
Different methods to generate sparse matrix S
In this section, we compare the performance of
BERT BASE using different methods to generate the sparse matrix S 2 and the corresponding space Ω .
Except for the aforementioned matrix decomposition method, we try to (1) randomly sample indexes into Ω ; and (2) directly select the indexes of elements of highest magnitude of W into Ω .
In Figurewe can see that using the matrix decomposition method has the highest metric overall.
It is also noteworthy that sparse matrices generated by other methods will sometimes harm the performance.
Another factor is the number of non-zero elements inside S 2 .
More non-zero elements in S 2 reduce the parameter efﬁciency, while less non-zero elements increase the efﬁciency but may not be much beneﬁcial to the accuracy or other metrics.
Figure 2 also shows the relationship between number of non-zero elements in S 2 and the performance on ﬁne-tuned model on SST-2.
From the graph, we can see that using an N of 64 seems to have the most stable results compared to other choices.
Another important piece of information we can derive from the ﬁgure is that a higher number of non-zero elements in the S 2 does not guarantee better performance.
[Figure]
Ablation of # ranks.
The rank of lowrank decomposition r is crucial to the transfer learning performance.
A small r will result in lower representation ability, and a large r will bring more parameters and reduce the parameter efﬁciency.
To ﬁnd the best value, we conduct experiments with different ranks r on four datasets, SST-2, MNLI, CoLA, and STSB.
The results are displayed in Figure 3.
We add quadratic trend lines in the graph along with the discrete points to smooth the results.
We can draw two conclusions from the graphs: (cid:182)
Overall, the ﬁnal performance is positively correlated with the number of trainable parameters; however, on some tasks (MNLI, CoLA) higher number of parameters (namely, the r for U and V ) will lead to lower performance.
(cid:183)
With a sparse matrix embedded, the performance of the trained models can be improved within a range of trainable parameters.
On SST-2 and CoLA, the applicable range seems to be 10 4 .
5 ∼ 10 6 .
On
STS-B, our method can consistently outperform the performance of using low-rank decomposition only.
For MNLI, the two methods behave similarly while our method is slightly better.
[Figure]
[Figure]
Ablation of sparsity.
Although creating sparsity in pretrained weights W does not change the number of trainable parameters of models, the different levels of sparsity control the number of non-zero elements and thereby inﬂuence the representation ability.
Therefore, we conduct experiments on BERT with different sparsity, ranging from 10% to 60%.
The results are in Figure A5.
From the ﬁgures, we can draw conclusions: (cid:182) DSEE out-performs magnitude pruning at low sparsity ( < 50% ) with respect to the performance on downstream tasks.
(cid:183) DSEE surpasses vanilla magnitude pruning with respect to the number of parameters.
For each self-attention layer, using vanilla magnitude pruning needs to train all the weights, while DSEE only needs to update the three low-rank matrices.
Visualizations.
Figure 4 shows the distribution of weight change.
From the graph, we can see that most weights are located around 0.
Such a phenomenon indicates that a natural sparsity exists within the update matrices, motivating us to explore sparse structures along with matrix decomposition.
C ONCLUSION
This paper draws on the prior of sparsity and establishes the DSEE framework.
It is the ﬁrst attempt towards jointly optimizing both parameter efﬁciency of the ﬁne-tuning process, and the resource efﬁciency of the ﬁne-tuned model.
On state-of-the-art large-scale language models (e.g., BERT, GPT, and DeBERTa) and across several of datasets, DSEE consistently demonstrates highly impressive parameter, training, and inference efﬁciency, in addition to preserving a competitive downstream transfer performance.
Our future work targets extending DSEE to the ﬁnetuning of large-scale computer vision and/or multi-modal pre-trained models.
A M ORE I MPLEMENTATION D ETAILS
We report the setting of other hyperparameters of our experiments, such as learning rate, in Table A7.
The device we used for experiments are various, including NVIDIA GeForce GTX 1080 Ti, GeForce
RTX 2080 Ti, Titan RTX, and A6000.
[Table]
GreBsmo Algorithm GreBsmo (Zhou & Tao, 2013) is an algorithm for solving the Robust PCAlike methods.
The optimization of U , V , and S follows the following iterative rules:
[Equation]
where X is the original dense matrix, QR( · ) means the QR decomposition, S λ ( · ) indicates the soft-threshold operation, and the subscripts k indicates the optimization step
[Figure]
Ablation of Sparsity To study the relationship between sparsity of unstructured pruning and the behavior of our DSEE, we conduct an ablation study on various datasets in GLUE benchmarks. The results are shown in Figure A
[Figure]
Results with more runs. We conduct experiments with three runs. The mean and standard deviation of accuracy of LoRA and DSEE (without pruning) are shown in Table A8. We have also included the p-value of t-tests to show the signiﬁcance of the performance gaps.
On half of the A datasets, the p-value is around 0.1; and on three datasets the p-value is between 0.2 and 0.4. The reason why p-value is large on some datasets is probably because the number of experiments still has room to increase.
[Table]
We have recorded the inference time of BERT BASE on various GLUE benchmarks, QQP, STS-B, CoLA and RTE. The number of samples in their evaluation set range from 277 to 40430, so they can be considered representative.
The results are shown in Table A9.
From the table we can see that, the inference time are greatly saved after using the structured version of DSEE.
[Table]
Convergence Speed We have compared the convergence speed (i.e., how the test accuracy changes) of LoRA and DSEE.
From Figure A6 we can see that the convergence speeds of the two methods are not signiﬁcantly different. This means introducing a sparse component into the decomposition form does not affect the convergence dynamics.
[Figure]
A