Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training
Abstract
Recently, sparse training has emerged as a promising paradigm for efficient deep learning on edge devices.
The current research mainly devotes the efforts to reducing training costs by further increasing model sparsity.
However, increasing sparsity is not always ideal since it will inevitably introduce severe accuracy degradation at an extremely high sparsity level.
This paper intends to explore other possible directions to effectively and efficiently reduce sparse training costs while preserving accuracy.
To this end, we investigate two techniques, namely, layer freezing and data sieving.
First, the layer freezing approach has shown its success in dense model training and fine-tuning, yet it has never been adopted in the sparse training domain.
Nevertheless, the unique characteristics of sparse training may hinder the incorporation of layer freezing techniques.
Therefore, we analyze the feasibility and potentiality of using the layer freezing technique in sparse training and find it has the potential to save considerable training costs.
Second, we propose a data sieving method for dataset-efficient training, which further reduces training costs by ensuring only a partial dataset is used throughout the entire training process.
We show that both techniques can be well incorporated into the sparse training algorithm to form a generic framework, which we dub SpFDE.
Our extensive experiments demonstrate that SpFDE can significantly reduce training costs while preserving accuracy from three dimensions: weight sparsity, layer freezing, and dataset sieving 1 .
Introduction
Sparse training, as a promising solution for efficient training on edge devices, has drawn significant attention from both the industry and academia [1].
Recent studies have proposed various sparse training algorithms with computation and memory savings to achieve training acceleration.
These sparse training approaches can be divided into two main categories.
The first category is fixed-mask sparse training methods, aiming to find a better sparse structure in the initial phase and keep the sparse structure constant throughout the entire training process [2, 3].
These approaches have a straightforward sparse training process but suffer from a higher accuracy degradation.
Another category is Dynamic Sparse Training (DST), which usually starts the training from a randomly selected sparse structure [4, 5].
DST methods tend to continuously update the sparse structure during the sparse training process while maintaining an overall sparsity ratio for the model.
Compared with the fixed-mask sparse training, the state-of-the-art DST methods have shown their superiority in accuracy and recently become a more broadly adopted sparse training paradigm [6].
However, although the existing sparse training approaches can reduce meaningful training costs, most of them devote their efforts to studying how to reduce training costs by further increasing sparsity while mitigating accuracy drop.
As a result, the community tends to focus on the sparse training performance at an extremely high sparsity ratio, e.g. , 95% and 98% .
Nevertheless, even the most recent sparse training approaches still lead to severe performance drop at these high sparsity ratios.
For instance, on the CIFAR-10 dataset [7], MEST [1] has a 2 .
5% and 4% accuracy drop at 95% and 98% sparsity, respectively.
In fact, the network performance usually begins to drop dramatically at the extremely high sparsity, while the actual gains from weight sparsity, i.e. , savings of computation and memory, tend to saturate.
This indicates that reducing training costs by pushing sparsity towards extreme ratios at the cost of network performance may not always the desirable methodology when a certain sparsity level has been reached.
Towards this end, we raise a fundamental question that has seldom been asked:
Are there other ways that can be seamlessly combined with sparse training to further reduce training costs effectively while maintaining network performance?
[Table]
To answer the question, we first take a step back to understand whether all layers in sparse networks are equally important.
Recent studies reveal that not all the layers in dense Deep Neural Networks (DNNs) need to be trained equally [8, 9, 10].
Generally, the early layers in DNNs are responsible for extracting low-level features and usually have fewer parameters than the later layers.
These make the early layers have higher representational similarity and converge faster during training [8, 11].
Therefore, the layer freezing techniques are proposed, which stop the training (updating) of specific DNN layers early in the training process to save the training costs.
The early work at tempts the layer freezing technique in dense model training [8], while many following works focus on layer freezing in the fine-tuning process of the large transformer-based models [12, 13, 9, 10, 14, 15].
Even with the progress, the layer freezing technique has never been explored in the sparse training.
Layer freezing seems like a promising solution for sparse training
that
further reduces train-
ing costs
However,
the
conclusion
is still too early to draw
, given that sparse
training
has two critical characteristics that make it a unique domain compared with
dense
DNN
training
and fine-tuning.
This might impede the incorporation of the layer freezing technique in sparse training.
① The superiority of the DST method is attributed to its continuously changed sparse structure, which helps it end up with a better result [6].
This could inherently contradict the layer freezing that requires the layers to be unchangeable early in the training process.
② The impact of the sparsity for each layer is unknown in terms of the convergence
speed
These two characteristics directly affect the feasibility and potentiality of using the layer freezing techniques in sparse training.
Back
to
the question
we raise
, whether
there are other directions to effectively reduce sparse training costs
besides
increasing sparsity
To tackle
the question,
in this work we propose
two
techniques
that can be well incorporated with sparse training, namely layer freezing and data sieving.
• We explore the feasibility and potentiality of leveraging layer freezing in the sparse training domain by carefully analyzing the structural and representational similarity of sparse models during sparse training.
We find the layer freezing technique is suitable for sparse training and has the potential to save considerable training costs (Sec. 3).
Based on this, we propose a progressive layer freezing method, which is simple yet effective in saving training costs and preserving accuracy (Sec. 4.2).
Shrinking the size of the training dataset is another possible dimension to reduce training costs.
Studies have shown that the importance of each training sample is different during DNN training [16, 17].
Toneva et al.
[18] distinguish the importance of each training sample by counting the number of times each training sample is forgotten by the network during training.
Later, Yuan et al.
[1] propose dataset
efficiency in the sparse training domain with two phases.
The first phase uses the whole dataset to count the forgotten times of each sample, and the second phase removes the unimportant samples.
Though they prove the feasibility of dataset efficiency training in the sparse domain, it only obtains limited cost savings from its second phase.
To fully exploit the potential of dataset efficiency, we propose the data sieving–a circular sieving method to dynamically update the shrunken training dataset, ensuring high dataset efficiency throughout the entire training.
Putting it all together, we propose a generic and efficient sparse training framework SpFDE , that achieves a significant reduction in the training computational and memory costs through threedimension : weight Sp arsity, layer F reezing, and D ataset E fficiency.
The comparison of key features between SpFDE and other representative sparse training works, i.e. , SNIP [2], GraSP [3], RigL [5],
ITOP [6], SET [19], DSR [4], and MEST [1], is provided in Tab. 1.
Extensive evaluation results show that the SpFDE framework consistently achieves a significant reduction in training FLOPs and memory costs while preserving a higher or similar accuracy.
Specifically, SpFDE maintains the highest 71 .
35% accuracy at 90% sparsity and 76 .
03% accuracy at 80% sparsity on the CIFARand ImageNet dataset, respectively, and achieves 18% and 29% training FLOPs reduction compared to the most recent methods such as MEST [1] and ITOP [6].
Moreover, SpFDE can further save 20% ∼ 25 .
3% average training memory, and 42 .
2% ∼ 43 .
9% minimum required training memory compared to the prior sparse training methods.
Related Work
Background for Sparsity Training.
Sparsity scheme and training strategy are two important components for defining a sparse training pipeline from literature.
Three main sparsity schemes introduced in the area of network pruning consists of unstructured [20, 21, 22], structured [23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36], and fine-grained structured pruning [37, 38, 39, 40, 41, 42, 43, 44, 45, 46].
Though network pruning is initially proposed for inference acceleration, it is widely adopted in sparse training to achieve the satisfactory trade-offs between network performance, e.g. , classification accuracy, memory footprint, and training time.
Most of these works follow the training pipeline of pretraining-pruning-retraining.
Instead, we consider a generic sparsity training framework that works for edge devices by focusing on sparse networks trained from scratch, instead of training dense networks, which is not feasible on resource limited devices [1].
Research on the sparse training strategies [47, 48, 49] can be categorized into fixed-mask and sparsemask sparse training, where the former aims to make it feasible that the training of the pruned models can be implemented on edge devices [2, 3, 50, 51, 52] and the latter studies how to reduce memory footprint along with the computation during training [53, 19, 4, 54, 5].
The sparsity scheme and training strategy in this work follows MEST [1] as it does not involve dense computations, making it desirable for edge device scenarios.
Unlike previous works on sparse training that mainly reduces computation by increasing sparsity ratio at the cost of network performance, we investigate a new method–layer freezing–into the training, which can reduce training cost for an arbitrary sparsity ratio.
Layer Freezing during Training.
The study on accelerating the training of dense neural networks have shown that not all the layers need to be trained equally and the decreasing of training iterations for certain layers can reduce training time with only minimal performance drop observed [55, 8, 12, 13, 56, 57, 9, 14, 58].
Liu et al.
[10] and He et al.
[15] calculate network gradients for automatically layer freezing during training.
Wang et al.
[59] use knowledge distillation [60] to guide the layer freezing schedule.
However, these works require the network to be static or an extra dense network during the training, which is not eligible for the sparse training scheme.
In this work, we show layer freezing can be incorporated into sparse training with detailed analysis.
Additionally, we propose a flexible and hybrid layer freezing strategy that can be fitted into the sparse training.
Analysis of Layer Freezing in Sparse Training
Existing works have shown the success of the layer freezing technique in dense model training, especially for the model fine-tuning, which effectively saves training costs in terms of computation and memory and thus accelerates the training [8, 10, 14, 15].
However, sparse training has two critical characteristics , i.e. , dynamically changed architecture and different sparsity per layer, that make it a unique domain compared to dense DNN training and fine-tuning, which might impede incorporating the layer freezing technique into the sparse training methods.
To investigate the best way to leverage layer freezing in sparse training, we need first to answer the following two questions.
[Figure]
c t u r a l s i
[Figure]
a r i t y
[Figure]
3.1 Is Layer Freezing Compatible for Dynamically Changed Network Structure?
The Dynamic Sparse Training (DST) method shows its superior performance by continuously changing its sparse model structure during training to search for a better sparse architecture, making it a desirable sparse training paradigm [5].
However, the dynamically changed network structure seems essentially contradictory to the layer freezing technique, given the weights of frozen layers are fixed and not further trained.
Thus, these layers cannot keep searching for better sparse structures, which may compromise the quality of the sparse model trained, leading to lower model accuracy.
Assumption.
Inspired by the convergence speed for various layers is different in conventional dense model training, we conjecture that, in DST, the early layers may also find desired sparse structure faster than the later layers .
If true, we may be able to introduce the layer freezing technique in sparse training without compromising the sparse training accuracy.
Experimental Setting.
We investigate the assumption by tracking the structural similarity of the sparse model along the sparse training process.
Specifically, we select the well-trained sparse model as the reference model and compare the intermediate sparse model obtained at each epoch with the reference model.
We define structural similarity as the percentage of the common non-zero weight locations, i.e. , indices, in both the intermediate sparse model and the final sparse model.
For instance, the structural similarity of 70% indicates that 30% of the current intermediate sparse structure will be altered during the rest of the training process and will not be presented in the final model.
The 100% structural similarity shows the sparse model structure is fully stabilized.
Analysis Results.
Fig.
1 (a) shows the trend of structural similarity of different layers within a model along the same sparse training process.
We adopt the DST method from MEST [1] with 90% unstructured sparsity and evaluate the results using ResNet-32 on the CIFAR-100 dataset.
Note that we check the structural similarity by choosing the locations of the 50% most significant non-zero weights from the intermediate models.
The reason is that sparse training algorithms ( e.g. , MEST and
RigL) may force less important weights/locations to be changed during sparse training, regardless of whether the sparse structure has already been stabilized.
Additionally, the most significant weights play the most important role in the model’s generalizability.
Therefore, tracking the structural similarity using 50% most significant non-zero weights is reasonable and meaningful.
Observation.
From the results, we can observe that the structural similarity of the first layer converges at the very early stage of the training, i.e. , 40 epochs, and the early layers’ structural similarity converge faster than the later layers.
The structural similarity of sparse training follows a similar pattern as in the dense model training.
This indicates that the changing/searching of the early layers from the sparse models can be stopped earlier without compromising the quality of the final sparse model, providing the feasibility of applying the layer freezing technique in sparse training.
3.2 What Is the Impact of Model Sparsity on the Network Representation Learning Process?
With the above exploration giving evidence that layer freezing is compatible with sparse training (Sec. 3.1), another critical question is to understand the impact of model sparsity on the neural
[Figure]
network representation learning process.
In other words, what is the convergence speed for the same layer under different sparsity?
If the convergence speed is different under different sparsity, applying layer freezing would be much more complicated since deciding the stop criterion would be very challenging, and the potential gain we can have by using layer freezing could be limited.
Previous work shows that layer freezing can be used to effectively reduce training costs due to the ability of fast representation learning of the early layers in the network [8].
Another work observes that a wider network is easier to learn the representation to a saturated level [11].
Therefore, whether the sparsity would significantly slow down the representation learning or the convergence speed of the layers is unknown since the width of the layer can be changed during sparse training.
Experimental Setting.
To explore the impact of sparsity on network representation learning speed, we adopt the centered kernel alignment (CKA) [11] as an indicator of representational similarity.
We track the trend of the CKA between the final and intermediate model at each training epoch.
Analysis Results and Observation.
The results are shown in Fig.
1 (b).
We compare the CKA trends of the same layer, i.e. , 10 th layer in ResNet-32, in dense model training and sparse training under different sparsity.
Surprisingly, we find that, in sparse training, even under a high sparsity ratio ( i.e. , 90% ), there is no apparent slowdown in the network representation learning speed.
Similar observations can also be found in other layers (more analysis results in Appendix A).
This indicates that the layer freezing technique can potentially be adopted in the sparse training process as early as in the dense model training process, thereby effectively reducing the training costs.
Takeaway.
Considering the unique characteristics of sparse training, we first explore the feasibility and potentiality of using the layer freezing techniques in the sparse training domain.
By investigating both the structural similarity and representational similarity of the sparse training, we tentatively conclude that the layer freezing technique is also suitable for sparse training and has the potential to save considerable training costs.
SpFDE Framework
In this section, we introduce our generic sparse training framework, with training costs saved through three dimensions: weight sparsity, layer freezing, and data sieving.
Fig.
2 shows the overview of our
SpFDE framework, and we introduce more details as follows.
4.1 Overview of Sparse Training Framework
The overall end-to-end training process can be logically divided into three stages, including the initial stage, the active training stage, and the progressive layer freezing stage.
Initial Stage.
As the first stage in training, we initialize the sparse network and partial training dataset.
The structure of the sparse network is randomly selected.
The partial training dataset is obtained by randomly removing a given percentage of training samples from the whole training dataset, which differs from prior work that starts with a whole training dataset [1].
We ensure that only parts of the whole training dataset will be used during the entire training process.
Active Training Stage.
Following the initial stage, all layers are actively trained (non-frozen) using a sparse training algorithm.
We apply DST from MEST [1] as the sparse training method due to its superior performance, while other sparse training algorithms are compatible with our framework.
We use the proposed data sieving method to update the current partial training dataset during the training periodically (more details in Sec. 4.3).
Besides the computation and memory savings provided by the sparse training algorithm, our SpFDE can benefit from the data sieving method to further save computation and memory costs.
Specifically, the computation costs are reduced by decreasing the number of training iterations in each epoch, and the memory costs are reduced by loading the partial dataset.
Progressive Layer Freezing Stage.
At this stage, we begin to progressively freeze the layers in a sequential manner.
The sparse structure and weight values of the frozen layers will remain unchanged during the sparse training.
The computational and memory costs of all gradients of weights and gradients of activations in the frozen layers can be eliminated, which is especially crucial for resource-limited edge devices.
More details are provided in Sec.
4.2 Progressive Layer Freezing
[Table]
[Table] [Table]
train
= for
= loop do if then do
Collect sample status
Motivated by the observation that the structural and representational similarity of early layers converges faster than later layers in sparse training (Sec. 3), we propose the progressive layer freezing approach to gradually freeze layers sequentially.
Specifically, a layer can be frozen only if all the layers in early of this layer are frozen.
The progressive manner brings the benefits for maximizing the saving of training costs since the entire frozen part of the model does not require computing backpropagation.
4.2.1 Layer Freezing Algorithm
Alg. 1 shows the training flow of SpFDE and the algorithm of progressive layer freezing.
For a given DNN model with
L layers, we divide it into N blocks, with each block consisting of several consecutive DNN layers, such as a bottleneck block in the ResNet [61].
We denote T as the total training epoch, ∆ τ as the sparse structure changing interval of dynamic sparse training, and T frz ( 0 < T frz < T ) as the epoch that we start the progressive layer freezing stage and freeze the first block.
Then, for every ∆ τ epochs, we sequentially freeze the next block until the expected overall training FLOPs satisfy the target _ flops .
We consider the frozen blocks still need to conduct forward propagation during training.
Therefore, we compute the training FLOPs reduction of freezing a block as its sparse back-propagation computation FLOPs (calculated by BpFlops ( · ) in Alg. 1) multiplied by the total frozen epochs of the block.
For detailed implementation, given the target training FLOPs target _ flops and the total number of layers to freeze, we empirically choose to freeze 2/3 layers of the model and the T frz can be easily calculated.
[Figure]
To better combine with the DST and make sure the layers/blocks are appropriately trained before being frozen, we synchronize the progressive layer freezing interval to the structure changing interval, i.e. , ∆ τ , of the sparse training, and adopt a layer/block-wise cosine learning rate schedule according to the total active training epoch of each layer/block.
4.2.2 Design Principles for Layer Freezing
There are two key principles for deriving a layer freezing algorithm, the freezing scheme and the freezing criterion.
Here we discuss the reasons that the proposed progressive layer freezing is rational.
Freezing Scheme .
Since sparse training may target the resource-limited edge devices, it is desired to have the training method as simple as possible to reduce the system overhead and strictly meet the budget requirements.
Therefore, we follow a cost-saving-oriented freezing principle to guarantee the target training costs and derive the layer freezing scheme, which can include the single-shot , single-shot & resume , periodically freezing , and delayed periodically freezing (as illustrated in Fig.(a)).
We adopt the single-shot scheme since it achieves the highest accuracy under the same training
FLOPs saving (detailed results in Appendix B).
The possible reason is that the single-shot freezing scheme has the longest active training epochs at the beginning of the training, which helps layers converge to a better sparse structure before freezing.
Freezing Criterion .
With the freezing scheme decided, another important question is how to derive the freezing criterion, i.e. , choosing which iterations or epochs to freeze the layers.
Existing works have explored adaptive freezing methods by calculating and collecting the gradients during the fine-tuning of dense networks [10].
However, from our observations, the unique property of sparse training makes these approaches not applicable.
For example, as shown in Fig.
3 (b), the difference of gradients norms from different layers decreases very fast at the beginning of the sparse training while it keeps fluctuating after some epochs because of the prune-and-grow weights.
Abstracting the freezing criterion based on the gradients norm would inevitably introduce extra computation and system complexity since the changing patterns of the gradient norm difference are volatile.
Therefore, our strategy of combining the layer freezing interval with the DST interval is more favorable.
4.3 Circular Data Sieving
We propose the data sieving method to achieve true dataset-efficient training throughout the sparse training process.
As shown in Fig.
2, at the beginning of the training, we randomly remove p % of total training samples from the training dataset to create a partial training dataset and a removed dataset.
During the sparse training, for every ∆ τ epoch, we update the current partial training dataset by removing the easiest p % of the training sample from the partial training dataset and adding them to the removed dataset.
Then, we retrieve the same number of samples from the removed dataset and add them back to the partial training dataset to keep the total number of training samples unchanged.
[Table]
We adopt the number of forgetting times as the criteria to indicate the complexity of each training sample.
Specifically, for each training sample, we collect the number of forgetting times by counting the number of transitions from being correctly classified to being misclassified within each ∆ τ interval.
We re-collect this number for each interval to ensure the newly added samples can be treated equally.
Additionally, we use the structure changing frequency ∆ τ in sparse training as the dataset update frequency to minimize the impact of changed structure on the forgetting times.
We treat the removed dataset as a queue structure, retrieving samples from its head and adding the newly removed sample to its tail.
After all the initial removed samples are retrieved, we shuffle the removed dataset after each update, making all the training samples can be used at least once.
As a result, we can gradually sieve the relatively easier samples out and only use the important samples for dataset-efficient training.
More analysis results are in the Appendix C.
Experimental Results
In this section, we evaluate our proposed SpFDE framework on benchmark datasets, including CIFAR100 [7] and ImageNet [63], for the image classification task with ResNet-32 and ResNet-50.
Note that we follow the previous work [1, 3, 2] using the 2 × widen version of ResNet-32.
We compare the accuracy, training FLOPs, and memory costs of our framework with the most representative sparse training works [2, 3, 53, 54, 4, 19, 5, 1] at different sparsity ratios.
Models are trained by using PyTorch [64] on an 8 × A100 GPU server.
We adopt the standard data augmentation and the momentum SGD optimizer.
Layer-wise cosine annealing learning rate schedule is used according to the frozen epochs.
To make a fair comparison with the reference works, we also use 160 training epochs on the CIFAR-100 dataset and 150 training epochs on the ImageNet dataset.
We choose
MEST+EM&S [1] as our training algorithm for weight sparsity since it does not involve any dense computations, making it desirable for edge device scenarios.
We apply uniform unstructured sparsity across all the convolutional layers while only keeping the first layer dense.
More experiments on other datasets and the detailed hyper-parameter setting are provided in the Appendix D.
5.1 Comparison on Model Accuracy and Training FLOPs
Tab. 2 shows the comparison of accuracy and computation FLOPs results on CIFAR-100 dataset using ResNet-32.
Each accuracy result is averaged over 3 runs.
We denote the configuration of our
SpFDE using x %
+ y
% , where x indicates the target training FLOPs reduction during layer freezing and y is the percentage of removed training data.
Our SpFDE can consistently achieve higher or similar accuracy compared to the most recent sparse training methods while considerably reducing the training FLOPs.
Specifically, at 90% sparsity ratio, SpFDE 20%+20% maintains similar accuracy as MEST [1], while achieving 27% training FLOPs reduction.
When compared with DeepR [53],
SET [19], and DSR [4], SpFDE 25%+25% achieves 27% FLOPs reduction and +1 .
36% ∼ +4 .
24% higher accuracy.
More importantly, when comparing SpFDE 25%+25% at 90% sparsity with the
MEST at 95% sparsity, we can see the two methods have the same training FLOPs, i.e. , 0 .
96 , while
SpFDE 25%+25% has a clear higher accuracy, i.e. , +0 . 66% .
This further strengthens our motivation
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
that pushing sparsity towards extreme ratios is not the only desirable direction for reducing training costs.
SpFDE provides new dimensions to reduce the training costs while preserving accuracy.
Tab. 3 provides the comparison results on the ImageNet dataset using ResNet-50.
At each training
FLOPs level, SpFDE consistently achieves higher accuracy than existing works.
It is interesting to see that SpFDE outperforms the original MEST, in both accuracy and FLOPs saving.
The FLOPs saving is attributed to layer freezing and data sieving for end-to-end dataset-efficient training.
Moreover, compared to the one-time dataset shrinking used in MEST, our data sieving dynamically updates the training dataset, mitigating over-fitting and resulting in higher accuracy.
We also conduct ablation studies on the impact of only applying layer freezing technique or data sieving technique and the results are provided in Appendix E.
5.2 Reduction on Memory Cost
From the Fig.
4, we can see the superior memory saving of our SpFDE framework.
The memory costs indicate the memory footprint used during the sparse training process, including the weights, activations, and the gradient of weights and activations, using a 32 -bit floating-point representation with a batch size of 64 on ResNet-32 using CIFAR-100.
The “SpFDE Min.” stands for the training memory costs after all the target layers are frozen, while the “SpFDE Avg.”
is the average memory costs throughout the entire training process.
The baseline results of “DST methods Min.”
only consider the minimum memory costs requirement for DST methods [2, 3, 53, 54, 4, 19, 5, 1, 6], which ignores the memory overhead such as the periodic dense back-propagation in RigL [5], dense sparse structure searching at initialization in [2, 3], and the soft memory bound in MEST [1].
Even under this condition, our “SpFDE Avg.” can still outperform the “DST methods Min.”
with a large margin ( 20% ∼ 25 .
The “SpFDE Min.” results show our minimum memory costs can be reduced by 42 .
2% ∼ 43 .
9% compared to the “DST methods Min.”
at different sparsity ratios.
This significant reduction in memory costs is especially crucial to edge training.
5.3 Discussion and Limitation
The reduction in training FLOPs of our method comes from three sources: weight sparsity, frozen layers, and shrunken dataset.
However, the actual training acceleration depends on different factors, e.g. , the support of the sparse computation, layer type and size, and system overhead.
Generally, the same FLOPs reduction from the frozen layers and shrunken dataset can lead to higher actual training acceleration than weight sparsity (more details in Appendix F).
This makes our layer freezing and data sieving method more valuable in sparse training.
We use the overall computation FLOPs to measure the training acceleration, which may be considered a theoretical upper bound.
Conclusion
This work investigates the layer freezing and data sieving technique in the sparse training domain.
Based on the analysis of the feasibility and potentiality of using the layer freezing technique in sparse training, we introduce a progressive layer freezing method.
Then, we propose a data sieving technique, which ensures end-to-end dataset-efficient training.
We seamlessly incorporate layer freezing and data sieving methods into the sparse training algorithm to form a generic framework named SpFDE.
Our extensive experiments demonstrate that our SpFDE consistently outperforms the prior arts and can significantly reduce training FLOPs and memory costs while preserving high accuracy.
While this work mainly focuses on the classification task, a future direction is to further investigate the performance of our methods on other tasks and networks.
Another exciting topic is studying the best trade-off between these techniques when considering accuracy, FLOPs, memory costs and actual acceleration. Acknowledgment
This work is supported in part by National Science Foundation CNS-1909172.
Appendix A More Results for the Impact of Model Sparsity on the Network Representation Learning Process
Sec. 3.2 of the main paper discusses the impact of model sparsity on the network representation learning process.
Here we provide more experimental results.
Specifically, we evaluate the representational similarity using the CKA value of the same layer from the model (ResNet-32) with different sparsity at each epoch and compare them with the final model.
We choose the early (1 st and 3 rd ), middle (18 th )
[Figure]
and late layers (25 th and 32 nd ) to track their CKA trends.
We evaluate three sparsity ratios, including medium ( 50% ), medium-high ( 80% )
[Figure]
and high ( 90% ) sparsity.
The results are shown in Fig.
A
[Figure]
(1 st , 3 rd , 18 th , 25 th , and 32 nd ) with different sparsity at different epochs.
All results are collected using ResNet-32 on the CIFARdataset during the sparse training process.
We can observe that the representation learning speed of sparse training under different sparsity ratios is similar to the dense model training at each layer.
This indicates that layer sparsity does not slow down the layer representation learning speed.
Therefore, the layer freezing technique can potentially achieve considerable training FLOPs reduction in sparse training domains similar to the dense model training.
B Ablation Analysis on Freezing Schemes
In our work, we evaluate four different types of freezing schemes (Sec. 4.4.4 of the main paper), including the single-shot freezing , single-shot freezing & resume , periodically freezing , and delayed periodically freezing .
Single-Shot Freezing Scheme.
The single-shot freezing scheme is the default freezing scheme used in our progressive layer freezing method.
For this scheme, we progressively freeze the layers/blocks in a sequential manner, as shown in Alg. 1 and Fig.
3 (a) in the main paper.
Single-Shot Freezing & Resume Scheme.
This scheme follows the same way to decide the per layer/block freezing epoch as the single-shot scheme, except that we make the freezing epoch for all layers/blocks t epochs earlier and resume (defrost) the training for all layers/blocks at the last t epochs.
In this case, we can keep the single-shot & resume has the same FLOPs reduction as the single-shot scheme, and the entire network can be fine-tuned at the end of training with a small learning rate.
Periodically Freezing Scheme.
For the periodically freezing scheme, we let the selected layers freeze periodically with a given frequency so that all the layers/blocks are able to be updated at different stages of the training process.
The basic idea is to let the early layers/blocks updated (trained) less frequently than the later layers.
For example, we let the early layers/blocks only be updated for one epoch in every four epochs and let the middle layers/blocks only be updated for one epoch in every two epochs.
Therefore, we consider the update frequency of the early and middle layers/blocks are 1 / 4 and 1 / 2 , respectively.
To ensure that when a layer is frozen, all the layers in early of it are frozen, we need to let the freezing period be the numbers of power of two ( e.g. , 2 , 4 , and 8 ).
In our experiments, we divide the ResNet32 into three blocks and set the update frequency for the first and second blocks to 1 / 4 and 1 / 2 , respectively.
The last block will not be frozen during the training.
We control the number of layers in each block to satisfy the overall training FLOPs reduction requirement.
Delayed Periodically Freezing Scheme.
For this scheme, we first let all the layers/blocks be trained actively for certain epochs, then periodically freeze the layers used in the periodically freezing scheme.
To achieve the same training FLOPs reduction as the periodically freezing scheme, more layers are needed to be included in the first and second blocks.
[Table]
Accuracy Comparison.
Tab. A1 shows the accuracy comparison of different freezing schemes at medium ( 60% ) and high ( 90% ) sparsity ratio.
We use the ResNet32 on the CIFAR-100 dataset.
Our target training FLOPs reduction through layer freezing is set to 20% .
We do not use the data sieving technique in this experiment.
The results show that the single-shot scheme consistently achieves the highest accuracy at both 60% and 90% sparsity ratio.
The accuracy of the single-shot freezing & resume scheme is slightly lower than the single-shot scheme and the two periodically freezing schemes are the worst.
These demonstrate that the layer freezing technique in sparse training prefers to train the layers/blocks as good as possible at the beginning of the training, and the “last-minute” or periodic fine-tuning does not benefit the final accuracy.
C Data Sieving Analysis C.1 Basic Concepts of Dataset Efficient Training
We use the number of forgetting events [18, 1] as the criteria to measure the difficulty of the training examples.
A forgetting event can be defined as a training sample that goes from being correctly classified to being misclassified by a network in two consecutive training epochs.
The training examples that have a higher number of forgetting events throughout the training indicate the examples are more complex and are considered more informative to the training.
On the contrary, the training examples that have a lower number of forgetting events or have never been forgotten are relatively easier examples and are less informative to the training.
Removing the unforgettable examples from the training dataset does not harm the training accuracy [18].
Whole training dataset
Partial training dataset
Removeddataset
Initial removed examples queue
Exampleindex
Removed examples queue
Add to partial training dataset 3 213 21 8 41 35 14 8 41 35 1431 73 6 3015 226 316 307331 7319 446 317 54 3 19 4415 8115 811st update removed from partial training dataset and add to queue
Removed examples queue 2nd update until all initially removed exmaples are added to the partial training dataset
Shuffle the queue every time before update 835
C.2 More Details about the Proposed Data Sieving Method
Fig.
A2 shows the detailed update process of our data sieving method.
We use a queue data structure to contain the indices of the removed examples.
For each time the partial training dataset is updated, we retrieve the examples from the head of the removed examples queue and add the newly removed examples to the tail of the removed examples queue.
To ensure all the examples are at least added to the partial training dataset once, we do not shuffle the queue until all the initial removed examples are added back to the partial training dataset.
[Table]
Tab. A2 shows an ablation study on the data sieving update ratio.
The number of updated examples in each dataset update process is proportional to the number of examples in the removed dataset.
The update ratio in the table denotes the percentage of examples retrieved from the removed dataset and added to the partial training dataset.
We evaluate different update ratios ( i.e. , 10% , 20% , 30% , and 50% ) under different dataset removal ratios ( 15% , 20% , and 25% ).
From the results, we can find that a 30% update ratio is the most desirable setting for the data sieving, which achieves the highest accuracy under different dataset removal ratios.
[Table]
[Table] [Table]
D Hyper-Parameter and More Experimental Results
Detailed Experiment Setup.
Tab. A3 shows detailed hyper-parameters regarding the general training and dynamic sparse training.
In our work, we use the MEST-EM&S [1] as our base sparse training algorithm.
To make fair comparisons to the reference works, we also use the 2 × widened version
ResNet-32 in our work, which is the same as all the baseline works shown in Tab. 2 and Tab.
A6.
In our data sieving method, we remove the easiest p % training examples from the partial training dataset every time we update our training dataset.
In our experiments, we make the p % equals to the 30% of the number of examples in the removed dataset.
Tab. A4 and Tab. A5 show the epoch
T frz that starts the progressive layer freezing stage for different target training FLOPs reduction for
ResNet32 on CIFAR-10/100 and ResNet50 on ImageNet, respectively.
More Results on the CIFAR-10 Dataset.
Tab.
A6 shows the accuracy comparison of our SpFDE and the most representative sparse training works using ResNet32 on the CIFAR-10 dataset.
Our
SpFDE consistently achieves higher or similar accuracy on the CIFAR-10 dataset compared to the most recent sparse training methods while considerably reducing the training FLOPs.
More Results on the ImageNet Dataset.
Tab.
A7 shows the accuracy comparison using ResNet on the ImageNet dataset at the 90% sparsity ratio.
At the similar training FLOPs level ( 0 . 32 ∼ 0 .
36 × 10 18 ), our SpFDE achieves 73 .
81% on top-1 accuracy, outperforming the best baseline work
MEST by 1 . 45% .
[Table] [Table]
E Ablation Study on Layer Freezing and Data Sieving
We also conduct ablation studies for the impact of layer-freezing and data sieving on accuracy by themselves (Tab. A8 and Tab. A9).
The results are obtained using ResNet-32 on the CIFAR-100 with the sparsity of 60% and 90%.
The accuracy results are the average value of 3 runs using random seeds.
[Table] [Table]
From the experiments
[Table]
we can further find some interesting observations:
• Under both sparsity of 60% and 90%, saving up to 15% training costs (FLOPs) via either layer freezing or data sieving does not lead to any accuracy drop.
• When under a higher sparsity ratio (90% vs. 60%), sparse training can tolerate a higher FLOPs reduction for both layer freezing and data sieving.
For example, compared to the non-freezing case (i.e., None in the second column), the layer freezing with a 20% FLOPs reduction leads to a -0.01% and -0.21% accuracy drop for 90% sparsity and 60% sparsity, respectively.
As for the data sieving, compared to the non-freezing case, under a 20% FLOPs reduction, there is a -0.19% and -0.31% accuracy drop for 90% sparsity and 60% sparsity, respectively.
The possiable reason is that, under a higher sparsity ratio, the upper bound
for model accuracy/generalization capability is decreased, mitigating the sensitivity to the number of training data or layer freezing.
• With a relatively higher FLOPs reduction ratio (i.e., 30% 35%), data sieving preserves higher accuracy than layer freezing under the same FLOPs reduction ratio.
This inspires that if people intend to pursue a more aggressive FLOPs reduction at the cost of accuracy degradation, removing more data via the data sieving method is a more desirable choice than freezing more layers.
Furthermore
[Table]
in Tab.
A
[Table]
we show a comparison between only using layer-freezing or data sieving, or both of them to achieve similar FLOPs reductions.
accuracy results are obtained using
[Table]
It can be observed that to achieve similar FLOPs reduction, using layer-freezing and data sieving together achieves much higher accuracy than by only using one of them individually, showing the importance of combining the two techniques.
[Table]
[Table]
F Discussion on Acceleration
In our work, the reduction in training FLOPs comes from three sources: weight sparsity, frozen layers, and shrunken dataset.
It is well-known that the acceleration based on weight sparsity is heavily affected by many different factors, such as the sparse computation support from a sparse matrix multiplication library or the dedicated compiler optimizations [39].
Besides, the sparsity schemes play an important role in the sparse computation acceleration.
Currently, the actual acceleration by leveraging weight sparsity is still limited even at a very high sparsity ratio [1].
We also evaluate the acceleration achieved by using our layer freezing and data sieving methods.
We measure the training time over 50 consecutive training epochs for each configuration and calculate the average value.
Tab. A11 and Tab.
A12 show the acceleration results by using our layer freezing and data sieving methods, respectively.
We compare the per epoch training latency with different FLOPs saving configurations ( i.e. , 10% , 15% , 20% , and 25% ) with the baseline result ( i.e. , using whole dataset and without freezing).
We can see that both methods achieve almost the linear training acceleration according to the FLOPs reduction.
This indicates that both methods only introduce negligible overhead to the training process.
Compared to the weight sparsity, this demonstrates the superiority of layer freezing and data sieving methods in the acceleration efficiency when under the same FLOPs reduction.
Most importantly, the layer freezing and data sieving methods have a high degree of practicality since the acceleration can be easily achieved using native PyTorch/TensorFlow without additional support.