APP: Anytime Progressive Pruning
UT
Austin
VITA, UT-Austin
Irina Rish Mila, UdeM Editors: Emtiyaz Khan and Mehmet Gonen Abstract
With the latest advances in deep learning, several methods have been investigated for optimal learning settings in scenarios where the data stream is continuous over time.
However, training sparse networks in such settings has often been overlooked.
In this paper, we explore the problem of training a neural network with a target sparsity in a particular case of online learning: the anytime learning at macroscale paradigm (ALMA).
We propose a novel way of progressive pruning, referred to as Anytime Progressive Pruning (APP); the proposed approach significantly outperforms the baseline dense and Anytime OSP models across multiple architectures and datasets under short, moderate, and long-sequence training.
Our method, for example, shows an improvement in accuracy of ≈ 7% and a reduction in the generalization gap by ≈ 22% , while being ≈ 1 / 3 rd the size of the dense baseline model in few-shot restricted imagenet training.
1. Introduction
Supervised learning has been one of the most well-studied learning frameworks for deep neural networks, where the learner is provided with a dataset D x,y of samples( x ) and corresponding labels( y ); and the learner is expected to predict the label y by learning on x usually by estimating p ( y | x ) .
In an offline learning environment Ben-David et al.
(1997), the learner has access to the complete dataset D x,y , while in a standard online learning setting Sahoo et al.
; Bottou et al.
(1998) the data arrive in a stream over time, assuming that the rate at which samples arrive is the same as that of the learner’s processing time to learn from them.
In this work, we are interested in exploring the training of sparse neural networks (pruned) in the ALMA setting Caccia et al.
Pruning Blalock et al.
; Luo et al.
(2017); Wang et al.
(2021) of over-parameterized deep neural networks has been studied for a long time.
Pruning deep neural networks leads to a reduction in inference time and memory footprint.
Although early pruning work focused exclusively on pruning weights [Figure] after pre-training the dense model for a certain number of iterations, extensive research has recently been conducted on pruning the model at initialization, that is, finding the lottery ticket Frankle and Carbin (2018); Frankle et al.
(2019a,b); Malach et al.
from a dense model at the start without pre-training the dense model Lee et al.
; Wang et al.
However, few studies Chen et al.
(2020) have investigated the training of sparse neural networks (pruned) in online settings.
Thus, our objective is to answer the following question: “ Given a dense neural network and a target sparsity, what should be the optimal way of pruning the model in ALMA setting? ”
In summary, our contributions can be summarized by the following two points.
∗ We provide the first comprehensive study of deep neural network pruning in the ALMA setting; henceforth, to this extent, we propose a novel approach of progressive pruning that we term Anytime Progressive Pruning (APP).
∗ We further investigate the APP training dynamics compared to baselines in the ALMA setting with a varied number of megabatches using C-10, C-100 1 , and Restricted ImageNet datasets.
2. Anytime Progressive Pruning
In this section, we formally introduce our proposed method Anytime Progressive Pruning (APP) as shown in the Fig.
For each megabatch M t ∈ S B , we construct the replay inclusive megabatch M t by taking the union of all previous megabatches along with the current megabatch and then create a small sample set π t of size 0 .
2 ∗ |M t | 2 to be used to prune the model to 0 .
8 δ t × 100% sparsity.
Here, δ t is obtained from a predetermined list δ of uniformly spaced values that denote the target sparsity levels for each megabatch in the stream S B .
After pruning the model, we train it on the M t megabatch and evaluate it on a held-out test set.
To evaluate APP, we use primarily 2 baselines:
1. C-10 and C-100 denote CIFAR-10 and 100 respectively.
The | · | denotes the size of the set/stream.
Baseline :
This denotes the model at full parametric capacity trained and fine-tuned on all megabatches in the stream S B using stochastic gradient descent in an ALMA setting.
Anytime OSP :
This denotes one-shot pruning (OSP) to the target sparsity 0 .
8 τ × 100% at the initialization of f θ and then subsequently training on all mega-batches in the stream S B in an ALMA setting.
Thus, Anytime OSP models have the lowest parametric complexity since the start of training on the first megabatch in the stream S B .
We use the same pruner of choice (SNIP) by default for both APP and Anytime OSP.
Similarly to APP, we prune the model at initialization using a small randomly selected subset π 1 of the first megabatch M 1 of size 0 .
2 ∗ |M 1 | .
∗ Cumulative Error Rate (CER
Along with test accuracy, we use CER to evaluate the methods described above, which can be defined by the following equation.
[Equation]
Here, T x,y represents the held-out test set used for evaluation, F t represents the trained model at the t -th megabatch, and F t ( x j ) represents the prediction on the j -th index sample of the test set T x,y compared to the true label for that sample y j . CER provides strong information on whether the learner is a good anytime learner, as it is expected to minimize CER at each megabatch training in the stream S B .
In addition, we also note the generalization gap as the difference between the training and the validation accuracy.
This gives a notion of whether the model is over- or under-fitting.
3. Results 3.1. CIFAR-10/ 100 experiments ( | S B | = 8 )
We start by analyzing the results shown in Fig.
Each megabatch M t consists ofsamples and the target sparsity was set to τ = 4 . 5 .
For all models, we observed a strong performance improvement for APP compared to baseline and Anytime OSP in all metrics: test accuracy, CER, and generalization gap.
For example, with ResNet-50 (R50) in C-100, APP improved the test accuracy by 17.97% and 11.12%, reduced the CER by 9927 and 5533 and decreased the generalization gap by 20.49% and 14.79% compared to baseline and Anytime OSP.
For C-10, we use a noncyclic step decay learning rate policy which reduces the learning rate only for the first megabatch ( M 1 ) and subsequently stays constant for all remaining megabatches.
However, for C-100, we used a cyclic step decay learning rate policy, where the learning rate resets to it’s initial value when starting on a new megabatch.
In Fig.
2, we show the results for using magnitude and random pruning instead of SNIP for APP and, based on the observations, we make SNIP the default pruner of choice due to its stability and strong performance. [Figure] [Figure] T e s t
A cc u r a c y [Figure] C E R [Figure]
R
WRN VGG
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
G e n e
[Figure]
a li s a t
i o n G a p
[Figure] R
R WRN VGG
[Figure]
G e n e r a li s a t i o n G a p 3.2. Few shot experiments on Restricted ImageNet
[Table]
In this section
[Table]
we investigate the performance of APP compared to Anytime OSP and the baseline models on Restricted Balanced ImageNet Engstrom et al.
; Tsipras et al.
(2018) using various few-shot learning settings.
We primarily conduct experiments using the following two few-shot settings
[Table] As reported in Table [Table] we observe that APP significantly reduces the generalization gap for each model variant compared to the Anytime OSP and the baseline counterparts.
α represent the number of samples per class in the complete dataset.
Excluding the experiment of α = 270 , | S B | = 70 , we observed a decrease in CER compared to the baseline model. For
APP [Figure] all models, APP significantly reduces the generalization gap and also improves test accuracy, except in the case of the experiment | S B | = 10 .
The target sparsity was kept fixed at τ = 4 .and
the backbone used throughout was R-50.
3.3. Transitions in generalization gap
We visualize the generalization gap as a function of training iterations across the megabatches in the stream S B in Fig.3 for the experiments reported in Table 1.
We observe non-monotonic transition in the high number of megabatch | S B | = 30 , 70 settings where the model initially oscillates within the under-fitting phase and then continues into a critical over-fitting regime before undergoing a smooth continuous transition where the generalization gap steadily decreases.
In all subplots, it can be seen that APP consistently maintains a lower generalization gap compared to its Anytime OSP and baseline counterparts.
4. Conclusion
In this work, we introduced Anytime Progressive Pruning (APP), a novel way to progressively prune deep networks while training in an ALMA regime.
We improvise on existing pruning at initialization strategy to design APP and perform an extensive empirical evaluation to validate performance improvement in various architectures and datasets.
We found that progressively pruning deep networks with APP while training in an ALMA setting causes a significant drop in the generalization gap compared to one-shot pruning methods and the dense baseline model.