D EEP E NSEMBLING WITH N O O VERHEAD FOR EITHER T RAINING OR T ESTING : T HE A LL -R OUND B LESSINGS OF D YNAMIC S PARSITY
A BSTRACT
The success of deep ensembles on improving predictive performance, uncertainty estimation, and out-of-distribution robustness has been extensively studied in the machine learning literature.
Albeit the promising results, naively training multiple deep neural networks and combining their predictions at inference leads to prohibitive computational costs and memory requirements.
Recently proposed efﬁcient ensemble approaches reach the performance of the traditional deep ensembles with signiﬁcantly lower costs.
However, the training resources required by these approaches are still at least the same as training a single dense model.
In this work, we draw a unique connection between sparse neural network training and deep ensembles, yielding a novel efﬁcient ensemble learning framework called FreeTickets .
Instead of training multiple dense networks and averaging them, we directly train sparse subnetworks from scratch and extract diverse yet accurate subnetworks during this efﬁcient, sparse-to-sparse training.
Our framework, FreeTickets , is deﬁned as the ensemble of these relatively cheap sparse subnetworks.
Despite being an ensemble method, FreeTickets has even fewer parameters and training FLOPs than a single dense model.
This seemingly counterintuitive outcome is due to the ultra training/inference efﬁciency of dynamic sparse training.
FreeTickets surpasses the dense baseline in all the following criteria: prediction accuracy, uncertainty estimation, out-of-distribution (OoD) robustness, as well as efﬁciency for both training and inference .
Impressively, FreeTickets outperforms the naive deep ensemble with ResNet50 on ImageNet using around only 1 / 5 of the training FLOPs required by the latter.
We have released our source code at https://github.com/VITA-Group/FreeTickets .
I NTRODUCTION
Ensembles (Hansen & Salamon, 1990; Levin et al., 1990) of neural networks have received large success in terms of the predictive accuracy (Perrone & Cooper, 1992; Breiman, 1996; Dietterich, 2000;
Xie et al., 2013), uncertainty estimation (Fort et al., 2019;
Lakshminarayanan et al., 2017; Wen et al., 2020; Havasi et al., 2021), and out-of-distribution robustness (Ovadia et al., 2019a; Gustafsson et al., 2020).
Given the fact that there are a wide variety of local minima solutions located in the high dimensional optimization landscape of deep neural networks and various randomness (e.g., random initialization, random mini-batch shufﬂing) occurring during training, neural networks trained with different random seeds usually converge to different low-loss basins with similar error rates (Fort et al., 2019;
Ge et al., 2015; Kawaguchi, 2016; Wen et al., 2019).
Deep ensembles, combining the predictions of these low-loss networks, achieve large performance improvements over a single network (Huang et al., 2017; Garipov et al., 2018;
Evci et al., 2020b).
Despite the promising performance improvement, the traditional deep ensemble naively trains multiple deep neural networks independently and ensembles them, whose training and inference cost increases linearly with the number of the ensemble members.
Recent works on efﬁcient ensembles are able
[Figure]
to reach the performance of dense ensembles with negligible overhead compared to a single dense model (Wen et al., 2020;
Wenzel et al., 2020; Havasi et al., 2021).
However, the training resources required by these approaches are still at least the same as training a single dense model.
Since the size of advanced deep neural networks is inevitably exploding (Touvron et al., 2020; Dosovitskiy et al., 2021;
Brown et al., 2020; Touvron et al., 2021), the associated enormous training costs are potentially beyond the reach of most researchers and startups, leading to ﬁnancial and environmental concerns (Garc ´ ıa-Mart ´ ın et al., 2019; Schwartz et al., 2019;
Strubell et al., 2019).
On the other hand, researchers have recently explored the possibility of directly training sparse neural networks from scratch (Mocanu et al., 2016; Liu et al., 2020a;
Evci et al., 2019), while trying to maintain comparable performance.
Training a sparse network from scratch typically results in worse performance than the traditional network pruning (Kalchbrenner et al., 2018;
Evci et al., 2019), with the exception of Dynamic Sparse Training (DST) (Mocanu et al., 2018; Evci et al., 2020a; Liu et al., 2021c;a).
Instead of inheriting weights from dense networks, DST starts from a randomly-initialized sparse network and optimizes the model weights together with the sparse connectivity during training.
However, the current only way for DST to match the performance of its dense counterpart on the popular benchmark, e.g., ResNet-50 on ImageNet, is to extend the training time (Evci et al., 2020a), which might require thousands of training epochs for extremely sparse models (Liu et al., 2021c).
In this paper, we attempt to address the above-mentioned two challenges jointly by drawing a unique connection between sparse training and deep ensembles.
Speciﬁcally, we ask the following question:
Instead of allocating all resources to ﬁnd a strong winning ticket, can we ﬁnd many weak tickets with very low costs (free tickets), such that the combination of these free tickets can signiﬁcantly outperform the single dense network, even the dense ensemble?
Note that it is not trivial to obtain free tickets.
To guarantee superior ensemble performance, three key desiderata that the free tickets are expected to satisfy (1) high diversity : according to the ensemble theory (LeCun et al., 2015; Hansen & Salamon, 1990; Ovadia et al., 2019b), higher diversity among ensemble members leads to higher predictive performance; (2) high accessibility : free tickets should be cheap to obtain so that the overall training cost does not compromise too much; and (3) high expressibility : the performance of each free ticket should be comparable with the dense model.
Leveraging the insight from Liu et al.
(2020b)
that a full network contains a plenitude of performative subnetworks that are very different in the topological space, we introduce the concept of
FreeTickets , an efﬁcient ensemble framework that utilizes sparse training techniques to create cheap yet accurate subnetworks for ensemble.
Furthermore, we instantiate FreeTickets by proposing two efﬁcient ensemble methods – Dynamic Sparse Training Ensemble (DST Ensemble) and Efﬁcient
Dynamic Sparse Training Ensemble (EDST Ensemble).
Both methods yield diverse subnetworks that consummately satisfy the above-mentioned criteria.
We summarize our contributions below:
• Our ﬁrst method, DST Ensemble, independently trains multiple subnetworks from scratch with dynamic sparsity.
By averaging the predictions of these subnetworks, DST Ensemble improves the predictive accuracy, OoD robustness, uncertainty estimation, and efﬁciency over the traditional dense ensemble.
• Our second, light-weight method ( EDST Ensemble) yields many free tickets in one single run , which is more efﬁcient to train and test than a single dense model , while approaching the performance of the traditional dense ensemble .
• We analyze the diversity of the individual subnetworks generated by our methods and conﬁrm the effectiveness of our methods on inducing model diversity.
• Our results suggest that besides the training/inference efﬁciency, sparse neural networks also enjoy other favorable properties which are absent in dense networks (robustness, out-ofdistribution generalization, etc), opening the path for new research directions.
R ELATED W ORKS
Efﬁcient Ensembles.
One major limitation of ensembles is the expensive computational and memory costs for both training and testing.
To address this problem, various approaches have been proposed.
TreeNet (Lee et al., 2015) shares weights in earlier layers and splits the following model into several branches, improving accuracy over the dense ensemble.
Monte Carlo Dropout (Gal & Ghahramani, 2016) can be used to approximate model uncertainty in deep learning without sacriﬁcing either computational complexity or test accuracy.
BatchEnsemble (Wen et al., 2020) was proposed to improve parameter efﬁciency by decomposing the ensemble members into the product of a shared matrix and a rank-one matrix personalized for each member.
MIMO (Havasi et al., 2021) uses a multi-input multi-output conﬁguration to concurrently discover subnetworks that co-habit the dense network without explicit separation.
Snapshot (Huang et al., 2017) and FGE (Garipov et al., 2018) discover diverse models by using cyclical learning rate schedules.
Furlanello et al.
(2018) applied knowledge distillation (Hinton et al., 2015) to train several generations of dense students.
The ensemble of the dense students outperforms the teacher model signiﬁcantly.
Other related works include but are not limited to hyper-batch ensembles (Wenzel et al., 2020) and Late Phase (Oswald et al., 2021).
However, the training resources required by these methods are still at least the same as training a single dense model.
In contrast to the existing efﬁcient ensemble methods, our methods (EDST Ensemble) can match the performance of naive ensemble with only a fraction of the resources required by training a single dense network.
Dynamic Sparse Training.
Dynamic Sparse Training (DST) is a class of methods that enables training sparse neural networks from scratch by optimizing the sparse connectivity and the weight values simultaneously during training.
DST stems from Sparse Evolutionary Training (SET)
(Mocanu et al., 2018), a sparse training algorithm that outperforms training a static sparse model from scratch (Mocanu et al., 2016;
Evci et al., 2019).
Weight reallocation was further proposed in Mostafa & Wang (2019); Dettmers & Zettlemoyer (2019); Liu et al.
(2021b) to reallocate new weights across layers for better layer-wise sparsity.
Further, Dettmers & Zettlemoyer (2019); Evci et al.
(2020a) leverage the gradient information in the backward pass to guide the optimization of sparse connectivity and demonstrate substantial performance improvement.
Some recent works (Jayakumar et al., 2020; Raihan & Aamodt, 2020; Liu et al., 2021c) demonstrate that a large range of exploration in the parameter space is important for dynamic sparse training.
Price & Tanner (2021) improved the performance of DST by adding additional non-trainable parameters.
DST has also demonstrated its strength in feature detection (Atashgahi et al., 2021), lifelong learning (Sokar et al., 2021), federated learning (Zhu & Jin, 2019; Bibikar et al., 2022; Huang et al., 2022), and adversarial training ( ¨Ozdenizci & Legenstein, 2021).
Sparse MoE (Shazeer et al., 2017; Fedus et al., 2021) sparsely activates one of the few expert networks to increase the model capacity – but with a constant computational cost.
It usually requires specialized modules, such as gating and selector networks to perform the sparsiﬁcation.
F REE T ICKETS 3. P RELIMINARIES
Dynamic Sparse Training.
Let’s consider an i.i.d.
classiﬁcation setting with data { ( x i , y i ) } N i =1 , where x usually denotes input samples and y the corresponding labels.
For a network f ( x ; θ ) parameterized by θ ∈ R d , we train f
( x ; θ ) to solve the following optimization problem: θ = arg min θ (cid:80) N i = 1 L ( f ( x i ; θ ) , y i ) .
Dynamic sparse training (DST) starts with a randomly-initialized sparse neural network f ( x ; θ s ) , parameterized by a fraction of parameters θ s .
The sparsity level of the model is pre-deﬁned as
S = 1 − (cid:107) θ s (cid:107)
(cid:107) θ (cid:107) 0 , where (cid:107) · (cid:107) 0 is the (cid:96)
-norm.
The goal of DST is to yield a sparse network with the target sparsity S after training, while maintaining the overall computational and memory overheads close to training a static sparse model (ﬁxed sparse connectivity).
During training, DST continuously minimizes the loss (cid:80) N i =1 L ( f ( x i ; θ s ) , y i ) , while periodically exploring the parameter space for better sparse connectivity with non-differentiable heuristics.
A common exploration heuristics is prune-and-grow, that is, pruning a fraction p of the unimportant weights from θ s , followed by regrowing the same number of new weights.
By repeating this prune-and-grow cycle, DST keeps searching for better sparse connectivities while sticking to a ﬁxed parameter budget.
See Appendix A for the general pseudocode and a brief literature review of DST.
3. F REE T ICKETS E NSEMBLE
We propose the concept of FreeTickets here.
FreeTickets refers to efﬁcient ensemble methods that utilize DST to generate subnetworks for the ensemble.
A free ticket is a converged subnetwork created by sparse training methods.
These free tickets { θ 1 s , θ 2 s , . . .
θ M s } , observed and collected either within one training run (EDST Ensemble) or multiple training runs (DST Ensemble), are further used to construct the FreeTickets Ensemble.
Assuming that the probability of the k th output neuron in the classiﬁer of the j th free ticket is given by p ( a jk ) .
Then the corresponding output probability in the ensemble is given by taking the average across all the M subnetworks, i.e., 1M (cid:80) M j =1 p ( a jk ) .
Compared with the existing efﬁcient ensemble techniques (Huang et al., 2017; Wen et al., 2020),
FreeTickets induces diversity inspired by the observations that there exist many performant subnetworks with very different sparse topologies located in the full network (Liu et al., 2020b).
The efﬁciency of FreeTickets comes from the fact that each subnetwork is sparse from the beginning , so that the memory and ﬂoating-point operations (FLOPs) required by FreeTickets can be even fewer than training a single dense network.
To realize the concept of FreeTickets , we introduce two DST-based ensemble methods, DST Ensemble and EDST Ensemble, as described below.
3.2. DST E NSEMBLE
Dynamic Sparse Training Ensemble (DST Ensemble) is presented in Algorithm 2,
Appendix B. It takes advantage of the training efﬁciency from DST and independently trains M sparse networks with
DST from scratch.
By averaging the predictions of each sparse neural network, DST Ensemble can improve the predictive accuracy and uncertainty estimation signiﬁcantly.
Except for the common diversity producers, i.e., random initializations and random stochastic gradient descent (SGD) noise, each DST run converges to different sparse connectivities, promoting even higher diversity over the naive dense Ensemble.
We choose the advanced DST method the Rigged Lottery (RigL)
(Evci et al., 2020a) for DST
Ensemble.
RigL contains three main steps: sparse initialization, model weight optimization, and parameter exploration.
Sparse Initialization.
Each subnetwork is randomly initialized with the Erd˝os-R´enyi-Kernel (ERK) (Mocanu et al., 2018;
Evci et al., 2020a) distribution at sparsity of S .
The sparsity level of layer l is scaled with 1 − n l − 1 + n l + w l + h l n l − 1 × n l × w l × h l , where n l refers to the number of neurons/channels of layer l ; w l and h l are the width and the height of the convolutional kernel in layer l .
ERK allocates higher sparsities to the layers with more parameters.
Model Weight Optimization.
After initialization, the activated weights are optimized by the standard optimizer SGD with momentum (Sutskever et al., 2013; Polyak, 1964) and the non-activated weights are forced to zero.
Parameter Exploration.
After every ∆ T iterations of training, we perform parameter exploration once to adjust the sparse connectivity.
More concretely, we ﬁrst prune a fraction p of weights from θ s with the smallest magnitude:
[Equation]
where TopK( v, k ) returns the weight tensor retaining the top k -proportion of elements from v .
Immediately after pruning, we grow the same number of new weights back which have the highest gradient magnitude:
[Equation]
where g i/ ∈ θ (cid:48)
s are the gradients of the zero weights.
We follow the suggestions in Liu et al.
(2021c)
and choose p = 0 .
5 and a large update interval ∆T = 1000 for CIFAR-10/100 and ∆T = 4000 for
ImageNet to encourage an almost full exploration of all network parameters.
3.2. EDST E NSEMBLE
While efﬁcient, the number of training runs (complete training phases) required by DST Ensemble increases linearly with the number of subnetworks, leading to an increased device count and resource requirements.
To further reduce the training resources and the number of training runs, we propose
Efﬁcient Dynamic Sparse Training Ensemble (EDST Ensemble) to yield many diverse subnetworks in one training run.
The overall training procedure of EDST Ensemble is summarized in Algorithm 3.
The challenge of producing many free tickets serially in one training run is how to escape the current local basin during the typical DST training.
Here, we force the model to escape the current basin by signiﬁcantly changing a large fraction of the sparse connectivity, like adding signiﬁcant perturbations to the model topology.
The training procedure of EDST Ensemble is one end-to-end training run consisting of one exploration phase followed by M consecutive reﬁnement phases.
The M reﬁnement phases are performed sequentially one after another within one training run.
Exploration phase.
We ﬁrst train a sparse network with DST using a large learning rate of 0.1 for a time of t ex .
The goal of this phase is to explore a large range of the parameter space for a potentially good sparse connectivity.
Training with a large learning rate allows DST to validly search a larger range of the parameter space, as the newly activated weights receive large updates and become more competitive at the next pruning iteration.
Reﬁnement phase.
After the exploration phase, we equally split the rest of training time by M to collect M free tickets.
At each reﬁnement phase, the current subnetwork is reﬁned from the converged subnetwork in the previous phase (the ﬁrst subnetwork is reﬁned from the exploration phase) and then trained with learning rates 0.01 followed by 0.001 for a time of t re .
Once the current subnetwork is converged, we use a large global exploration rate q = 0 .
8 1 and a larger learning rate of 0.01 to force the converged subnetwork to escape the current basin.
We repeat this process several times until the target number of free tickets is reached.
The number of subnetworks M that we obtain at the end of training is given by M = t total − t ex t re .
See Appendices I and J for the effect of the global exploration rate q and the effect of different regrowth criteria on EDST Ensemble, respectively.
Different from DST Ensemble, the diversity of EDST Ensemble comes from the different sparse subnetworks the model converges to during each reﬁnement phase.
The number of training FLOPs required by EDST Ensemble is signiﬁcantly smaller than training an individual dense network, as
DST is efﬁcient and the exploration phase is only performed once for all ensemble learners.
E XPERIMENTAL R ESULTS
In this section, we demonstrate the improved predictive accuracy, robustness, uncertainty estimation, and efﬁciency achieved by FreeTickets .
We mainly follow the experimental setting of
MIMO (Havasi et al., 2021), shown below.
Baselines.
We compare our methods against the dense ensemble, and various state-of-the-art efﬁcient ensemble methods in the literature, including MIMO (Havasi et al., 2021), Monte Carlo Dropout (Gal & Ghahramani, 2016), BatchEnsemble (Wen et al., 2020), TreeNet (Lee et al., 2015), Snapshop (Huang et al., 2017), and FGE (Garipov et al., 2018).
Moreover, to highlight the fact that the free tickets are non-trivial to obtain, we further implement three sparse network ensemble methods: Static Sparse Ensemble (naively ensemble M static sparse networks), Lottery Ticket Rewinding Ensemble (LTR Ensemble; Frankle et al.
(ensemble M winning tickets 2 trained with the same mask but different random seeds), and pruning and ﬁne-tuning (PF Ensembles; Han et al.
While Static Sparse Ensemble can be diverse and efﬁcient, it does not satisfy the high expressibility property (Evci et al., 2019).
LTR Ensemble suffers from low diversity and prohibitive costs.
PF Ensemble requires at least the same training FLOPs as the dense ensemble.
See Appendix C for their implementation and hyperparameter details.
Architecture and Dataset.
We evaluate our methods mainly with Wide ResNet28-10 (Zagoruyko &
Komodakis, 2016) on CIFAR-10/100 and ResNet-50 (He et al., 2016) on ImageNet.
Metrics.
To measure the predictive accuracy, robustness, and efﬁciency, we follow the Uncertainty
Baseline 3 and report the overall training FLOPs required to obtain all the subnetworks including forward passes and backward passes.
See Appendix D for more details on the metrices used.
[Table]
[Table]
[Table]
Results.
The metrics on CIFAR-10/100 and ImageNet are reported in Table 1, Table 2, and Table 3, respectively.
For a fair comparison, we mainly set M of our methods the same as the one used in
MIMO.
See Appendix E for the comparison between Snapshot, FGE, and our methods.
With multiple training runs, DST Ensemble consistently outperforms other efﬁcient ensemble methods, even the dense ensemble on accuracy, robustness, and uncertainty estimation, while using only 1/4 of the training FLOPs compared to the latter.
When the number of training runs is limited to 1, EDST Ensemble consistently outperforms the single dense model by a large margin, especially in terms of accuracy, with only 61% training FLOPs.
Moreover, we observe that the performance of EDST Ensemble can further be improved by increasing the sparsity level.
For instance, with a high sparsity level S = 0 . 9 , EDST Ensemble can collect 7 subnetworks, more than twice as many as
S = 0 . 8 .
Combining the prediction of these sparser subnetworks boosts the performance of EDST
Ensemble towards the dense ensembles with only 57% training FLOPs, beyond the reach of any efﬁcient ensemble methods.
More impressively, DST Ensemble achieves the best performance on uncertainty estimation and OoD robustness among various ensemble methods.
As we expected, Static Sparse Ensemble and LTR Ensemble consistently have inferior ensemble performance compared with DST Ensemble.
While PF Ensemble achieves comparable performance with DST Ensemble, its costly procedure requires more than triple the FLOPs of DST Ensemble.
We argue that LTR is not suitable for FreeTickets since (1) it starts from a dense network, with the costly iterative train-prune-retrain process.
Hence it is more expensive than training a dense network, in contrary to our pursuit of efﬁcient training (our method sticks to training sparse networks end-to-end); (2) The series of sparse subnetworks yielded by iterative pruning is not diverse enough, since the latter sparse masks are always pruned from and thus nested in earlier masks (Evci et al., 2020b).
Evci et al.
(2020b) show that the different runs of winning tickets discovered by LTR are always located in the same basin as the pruning solution.
Consequently, the ensemble of LTR would suffer from poor diversity, leading to poor ensemble performance.
We observe a very similar pattern in Section 5.1 as well.
The diversity and performance of LTR Ensemble are lower than DST-based ensembles, highlighting the importance of dynamic sparsity in sparse network ensembles.
F REE T ICKETS A NALYSIS 5. D IVERSITY A NALYSIS
According to the ensembling theory (LeCun et al., 2015; Hansen & Salamon, 1990; Ovadia et al., 2019b), more diversity among ensemble learners leads to better predictive accuracy and robustness.
We follow the methods used in Fort et al.
and analyze the diversity of the subnetworks collected by our methods in function space.
Concretely, we measure the pairwise diversity of the subnetworks collected by our methods on the test data.
The diversity is measured by D d = E
[ d ( P 1 ( y | x 1 , · · · , x N ) , P 2 ( y | x 1 , · · · , x N ))] where d ( · , · ) is a metric between predictive distributions and ( x, y ) are the test set.
We use two metrics (1)
Prediction Disagreement, and (2) Kullback–Leibler divergence.
Prediction Disagreement.
Prediction Disagreement is deﬁned as the fraction of test data the prediction of models disagree on: d dis ( P 1 , P 2 ) =
1N (cid:80) N i =1 (arg max ˆ y i P 1 (ˆ y i ) (cid:54) =
arg max ˆ y i P 2 (ˆ y i )) .
Kullback–Leibler (KL) Divergence.
KL Divergence (Kullback & Leibler, 1951) is a metric that is widely used to describe the diversity of two ensemble learners (Fort et al., 2019; Havasi et al., 2021), deﬁned as: d KL ( P 1 , P 2 ) =
E P 1 [log P 1 ( y ) − log P 2 ( y )] .
[Table]
We report the diversity among various sparse network ensembles in Table 4.
We see that DST
Ensemble achieves the highest prediction disagreement among various sparse ensembles, especially on CIFAR-100 where its diversity even surpasses the deep ensemble.
Even though EDST Ensemble produces all subnetworks with one single run, its diversity approaches other multiple-run methods.
The diversity comparison with non-sparse ensemble methods is reported in Figure 2-Right.
Again, the subnetworks learned by DST Ensemble are more diverse than the non-sparse methods and the diversity of EDST Ensemble is very close to the diversity of the dense ensemble, higher than the other efﬁcient ensemble methods e [Figure] g., TreeNet and BatchEnsemble.
[Figure] To better understand the source of disagreement, we apply KL divergence across intermediate feature maps learned by different subnetworks in Figure 2-Left.
Each line is the averaged KL divergence among M = 3 subnetworks.
It is interesting to observe that the KL divergence is quite high at early layers for the DST Ensemble and the Static Ensemble, which implies that the source of their high diversity might locate in the early layers.
We further provide a disagreement breakdown for the ﬁrst ten classes in CIFAR-100 between different subnetworks in Appendix G.
The overall breakdown of disagreement is very similar to the diversity measurement results, i.e., DST Ensemble >
Static
Ensemble > EDST Ensemble > LTR Ensemble.
We share heatmaps of prediction disagreement of
DST/EDST ensemble in Appendix F.
5. T RAINING T RAJECTORY
In this section, we use t-SNE (Van der Maaten & Hinton, 2008) to visualize the diversity of sparse networks discovered by our methods in function space.
We save the checkpoints along the training trajectory of each subnetwork and take the softmax output on the test data of each checkpoint to represent the subnetwork’s prediction.
t-SNE maps the prediction to the 2D space.
From Figure 3 we can observe that subnetworks of DST Ensemble converge to completely different local optima, and the local optima converged by EDST Ensemble are relatively closer to each other.
Interestingly, even the last two subnetworks of EDST Ensemble (blue and green lines) start from similar regions, they end up in two different optima.
[Figure]
5. A BLATION S TUDY OF P ARAMETER E XPLORATION
We conduct an ablation study to analyze the effect of parameter exploration on promoting the model diversity.
For comparison, we train the same number of subnetworks without any parameter exploration.
Shown in Table 5, without parameter exploration, the diversity and the ensemble accuracy consistently decreases, highlighting the effectiveness of parameter exploration on inducing diversity [Figure] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
5. E FFECT OF S PARSITY
We further study the effect of subnetworks sparsity on the ensemble performance and diversity, shown in Figure 4.
While the individual networks learned by DST Ensemble and EDST Ensemble achieve similar test accuracy, the higher diversity of DST Ensemble boosts its ensemble performance over the EDST Ensemble signiﬁcantly, highlighting the importance of diversity for ensemble.
What’s more, it is clear to see that the pattern of the ensemble accuracy is highly correlated with the accuracy of the individual subnetwork.
This observation conﬁrms our hypothesis in the introduction, that is, high expressibility is also a crucial desideratum for free tickets to guarantee superior ensemble performance.
We further support our hypothesis with the Pearson correlation (Pearson, 1895) and the
Spearman correlation (Powers & Xie, 2008) between the accuracy of individual networks and their ensemble accuracy in Appendix H.
[Figure]
5. E FFECT OF E NSEMBLE S IZE
In this section
[Figure]
we analyze the effect of the ensemble size (the number of ensemble learners) on our methods.
For EDST Ensemble, we ﬁx the total training time and vary the total number of ensemble learners M .
The larger ensemble size M leads to shorter training time of each ensemble learner.
For
DST Ensemble [Figure] we simply train a different number of individual sparse models from scratch and report the ensemble performance
[Figure]
The results are shown in Figure 5.
“Individual Subnetworks” refers to the averaged accuracy of single subnetworks learned by DST and
EDST.
The prediction accuracy of DST Ensemble keeps increasing as the ensemble size increases, demonstrating the beneﬁts of ensemble.
As expected, the predictive accuracy of individual EDST subnetworks decreases as the ensemble size M increases likely due to the insufﬁcient training time of each subnetwork.
Nevertheless, the ensemble performance of EDST is still quite robust to the ensemble size.
Speciﬁcally, despite the poor individual performance with M = 7 , the ensemble prediction is still higher than a single dense model.
This behavior conﬁrms our hypothesis mentioned before, i.e., multiple free tickets work better than one high-quality winning ticket or even a dense network.
5. E XPERIMENTS W ITH O O D AND A DVERSARIAL R OBUSTNESS
We further test the performance of FreeTickets on out-of-distribution (OoD) detection and adversarial robustness, reporting the results in Appendices K and L, respectively.
Our proposed ensemble methods also bring beneﬁts to OoD performance and adversarial robustness.
C ONCLUSION AND F UTURE W ORKS
We introduce FreeTickets , an efﬁcient way to boost the performance of sparse training over the dense network, even the dense ensemble.
FreeTickets is built as a combination of the diverse subnetworks extracted during dynamic sparse training and achieves a co-win in terms of accuracy, robustness, uncertainty estimation, as well as training/inference efﬁciency.
We demonstrate for the ﬁrst time that DST methods may not match the performance of dense training when standalone, but can surpass the generalization of dense solutions (including the dense ensemble) as an ensemble, while still being more efﬁcient to train than a single dense network.
The aforementioned compelling efﬁciency has not been fully explored, due to the limited support of the commonly used hardware.
Fortunately, some prior works have successfully demonstrated the promising speedup of sparse networks on real mobile processors (Elsen et al., 2020), GPUs (Gale et al., 2020), and CPUs (Liu et al., 2020a).
In future work, we are interested to explore sparse ensembles on hardware platforms for real speedups in practice.
R EPRODUCIBILITY
The training conﬁgurations and hyperparameters used in this paper are shared in Appendix C.
The metrics used to enable comparisons among different methods are given in Appendix D.
We have released our source code at https://github.com/VITA-Group/FreeTickets .
A CKNOWLEDGEMENT
We would like to thank Bram Grooten for giving feedback on the camera-ready version; the professional and conscientious reviewers of ICLR for providing valuable comments.
Z. Wang is in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML)
A D YNAMIC S PARSE T RAINING
In this Appendix, we describe the full details of dynamic sparse training.
See the survey of Mocanu et al.
(2021) and (Hoeﬂer et al., 2021) for more details.
A. A LGORITHM
Dynamic Sparse Training (DST) (Mocanu et al., 2018; Liu et al., 2020a) is a class of methods that enable the end-to-end training of sparse neural networks.
DST starts from a sparse network and simultaneously optimizes the sparse connectivity and model weights during training.
Without loss of generality, we provide the general pseudocode of DST that can cover most of the existing DST methods in Algorithm 1.
While there is an upsurge in increasingly efﬁcient ways for sparse training (Bellec et al., 2018; Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019;
Evci et al., 2020a; Liu et al., 2021b;
Jayakumar et al., 2020; Liu et al., 2021c; Dietrich et al., 2021; Chen et al., 2021a), most of DST methods contain three key components: sparse initialization, model weight optimization, and parameter exploration.
We explain them below.
Algorithm
Dynamic Sparse Training
Require: Network f Θ , dataset { x i , y i } N i =1 , Layer-wise sparsity ratio: S , Update Interval ∆T , Exploration Rate p 1: # Sparse Initialization 2: f ( x ; θ s ) ← f ( x ; θ ; S ) 3: for each training step t do 4: # Model Weight Optimization 5: f
( x ; θ s ) ← SGD ( f ( x ; θ s )) 6: if ( t mod ∆T) =
then 7: # Parameter Exploration 8:
Pruning p percentage of parameters using Eq.9:
Growing p percentage of parameters using Eq.10:
Update exploration rate p 11: end if 12: end for
A.1. S PARSE I NITIALIZATION
Layer-wise Sparsity Ratio.
Layer-wise sparsity ratio plays an important role for sparse training.
Mocanu et al.
ﬁrst introduced Erd˝os-R´enyi (ER) (Erd ˝ os & R ´ enyi, 1959) from graph theory to the ﬁeld of neural networks, achieving better performance than the uniform sparsity ratio.
Evci et al.
(2020a) further extended ER to CNN and brings signiﬁcant gains to sparse CNN training with the Erd˝os-R´enyi-Kernel (ERK) ratio.
Speciﬁcally, the sparsity level of layer l is scaled with 1 − n l − 1 + n l + w l + h l n l − 1 × n l × w l × h l , where n l refers to the number of neurons/channels of layer l ; w l and h l are the width and the height of layer l .
Works with weight redistribution (Mostafa & Wang, 2019; Dettmers &
Zettlemoyer, 2019; Liu et al., 2021b) start with the uniform layer-wise ratio and dynamically change the layer-wise ratio according to heuristic criteria, ending up with non-uniform layer-wise ratios.
Weight Initialization.
Weight initialization also affects the performance of sparse training.
Unlike the dense network, sparse networks contain a partial of weights to be zero, breaking the appealing properties of dense initialization such as dynamical isometry (Lee et al., 2020), gradient ﬂow (Evci et al., 2020b; Tessera et al., 2021), etc.
A.1. M ODEL W EIGHT O PTIMIZATION
After initializing the sparse model, we need to optimize the model weights to minimize the loss function.
In general, dynamic sparse training is compatible with the most widely used optimizers, e.g., minibatch stochastic gradient descent (SGD) (Robbins & Monro, 1951), minibatch stochastic gradient descent with momentum (Sutskever et al., 2013; Polyak, 1964), and Adam (Kingma & Ba, 2014).
However, we need to pay extra attention to average-based optimizers, such as Averaged Stochastic
Gradient Descent (ASGD), which is widely used in the language modeling tasks with RNNs and
LSTMs.
Liu et al.
(2021b) points out that ASGD seriously destroys the sparse connectivity optimization since the average operation brings the newly-grown weights immediately to zero, leading to an over-pruning situation.
Moreover, Tessera et al.
(2021) shows that optimizers that use an exponentially weighted moving average (EWMA) are sensitive to high gradient ﬂow.
A.1. P ARAMETER E XPLORATION
Dynamic sparse training outperforms static sparse training mainly due to the parameter exploration (Liu et al., 2021c).
One simple yet effective heuristic used here is prune-and-grow.
Removing p percentage of the existing weights and regrowing the same number of new weights allows DST to gradual optimize the sparse connectivity in a heuristic way.
Weight Prune: Although there exists various pruning criteria in pruning literature, the most common way for DST is the simple magnitude pruning.
We also tried other criteria for pruning at initialization e.g., SNIP (Lee et al., 2019), GraSP (Wang et al., 2020), SynFlow (Tanaka et al., 2020).
All of them fall short of the accuracy achieved by magnitude pruning.
Weight Grow: The most common ways to grow new weights are random-based growth (proposed in
SET (Mocanu et al., 2018)) and gradient-based growth (proposed in RigL (Evci et al., 2020a) and
SNFS (Dettmers & Zettlemoyer, 2019)).
Random-based growth ensures purely sparse forward pass and backward pass, while it very likely needs more steps to ﬁnd the important connections, especially for the very extreme sparsities.
Gradient-based growth detects the weights that reduce the current loss the fastest, whereas it involves the dense gradient calculation and is likely to cause a collapse of the explored parameter space (Liu et al., 2021b;c; Dietrich et al., 2021).
It is worth noting that the choice of when-to-explore is also of importance for DST.
The number of training iterations between two parameter explorations is controlled by the update interval ∆T .
Since the newly-grown weights are initialized to zero, a large ∆T is necessary to guarantee that the new weights receive enough weight updates to survive at the next pruning iteration (Liu et al., 2021c).
B P SEUDOCODE OF DST E NSEMBLE AND EDST E NSEMBLE
Require: network f Θ , dataset { x i , y i } N i =1 , ensemble size M , sparsity distribution S , update interval ∆T , DST exploration rate p , training time of each free ticket T .
1: for j = 1 to M do 2: # Sparse Initialization 3: f ( x ; θ js )
← f ( x ; θ j ; S ) 4: for t = 1 to T do 5: # Model Weight Optimization 6: f
( x ; θ js ) ← SGD ( f ( x ; θ js ))
7: if ( t mod ∆T) =
then 8: # DST Parameter Exploration 9:
DST parameter exploration using Eq. 1 & 2 with p 10:
Update exploration rate p 11: end if 12: end for 13:
Save the converged sparse subnetwork f ( x ; θ js ) 14: end for
Require: network f Θ , dataset { x i , y i } N i =1 , ensemble size M , sparsity distribution: S , update interval ∆T , DST exploration rate p , global exploration rate q , training time of the exploration phase t ex , training time of each reﬁnement phase t re / M 1: # Sparse Initialization 2: f ( x ; θ s ) ← f ( x ; θ ; S ) 3: # One Exploration Phase 4: for t = 1 to t
ex do 5: f ( x ; θ s ) ← SGD ( f ( x ; θ s ))
6: if ( t mod ∆T) =
then 7:
DST parameter exploration using Eq. 1 & 2 with p 8: end if 9: end for 10: # M Sequential Reﬁnement Phases 11: for j = 1 to M do 12: for t = 1 to t
re / M do 13: f ( x ; θ js ) ← SGD ( f ( x ; θ js ))
14: if ( t mod ∆T) =
then 15:
DST parameter exploration using Eq. 1 & 2 with p 16: end if 17: end for 18:
Save f ( x ; θ js ) and escape by parameter exploration using Eq. 1 & 2 with q 19: end for
C I MPLEMENTATION AND H YPERPARAMETERS
In this Appendix, we provide the hyperparameters used in Section 4.
We mentioned in the main content of paper that these free tickets (subnetworks) are non-trivial to obtain due to the three key desiderata.
To conﬁrm this, we implement and test two sparsity-based efﬁcient ensemble methods, Static Sparse Ensemble and Lottery Ticket Hypothesis (LTR) Ensemble.
Static Sparse Ensemble Static Sparse Ensemble refers to directly training M sparse subnetworks from scratch and ensemble them for test.
Even though it naturally satisﬁes the low training cost and high diversity, its insufﬁcient accuracy leads to worse performance than the proposed DST/EDST
Ensemble.
LTR Ensemble
Following Evci et al.
(2020b), we use gradual magnitude pruning (GMP)
(Zhu &
Gupta, 2017; Gale et al., 2019) for LTR, which is a well studied and more efﬁcient pruning method than iterative magnitude pruning (IMP).
Concretely, we ﬁrst use gradual magnitude pruning to prune the dense network to the target sparsity.
After that, we retrain and rewind the subnetworks to the 5% epoch for M times.
The converged subnetworks are further used for ensemble.
Note that even GMP is much more efﬁcient than IMP, the overall training FLOPs required by this method is much higher than directly training a single dense model.
We expect that using IMP for Ensemble can be more accurate, but also leads to prohibitive training costs, in contrary to our pursuit of efﬁcient ensemble.
PF Ensemble For the pruning baseline, we choose the strong techniques – global magnitude pruning.
We ﬁrst prune three independently pre-trained dense networks (each is trained forepochs) to the target sparsity, and then ﬁne-tune them for another 250 epochs with different random seeds.
We believe this setting can provide enough diversity as all the training procedures are independent, with different random seeds.
Hyperparameters of DST Ensemble For a fair comparison, we follow the training conﬁguration of MIMO.
For Wide ResNet28-10 on CIFAR, we train the sparse model for 250 epochs with a learning rate of 0.1 decayed by 10.
We use a batch size of 128, weight decay of 5e-4.
To achieve a good trade-off between ensemble accuracy and sparsity, we set sparsity as S = 0 . 8 .
Regarding the hyperparameters of DST, we choose a large update interval ∆T = 1000 between two sparse connectivity updates and a constant DST exploration rate p = 0 .
For ResNet-50 on ImageNet, to the best of our knowledge, all the state-of-the-art DST approaches can not match the performance of the dense ResNet-50 on ImageNet within a standard training time.
To guarantee sufﬁcient predictive accuracy for the individual ensemble learners, we follow the training setups used in Liu et al.
(2021c)
and train each sparse ResNet-50 for 200 epochs with a batch size of 64.
The learning rate is linearly increased to 0.1 with a warm-up in the ﬁrst 5 epochs and decreased by a factor of 10 at epochs 60, 120, and 180.
Even with a longer training time, it takes much fewer
FLOPs to train sparse networks using DST compared to the dense network.
We choose ∆T =and a cosine annealing schedule for the exploration rate with an initial value p = 0 . 5 .
Hyperparameters of EDST Ensemble For Wide ResNet28-10 on CIFAR, we train the 80% sparse model for 450 epochs and 90% sparse model for 850 epochs with t ex = 150 , t re = 100 , so that we can get M = 3 subnetworks with sparsity of 0.8 and M = 7 subnetworks with sparsity of 0.9 for ensemble.
The numbers of training FLOPs of this two settings are similar due to the different sparsity levels.
The DST exploration rate is chosen as p = 0 .
5 which achieves the best performance as shown in (Evci et al., 2020a;
Liu et al., 2021c).
To encourage a large diversity between ensemble learners, we use a large global exploration rate q = 0 .
8 to force the subnetowrk escape the current basin.
We have also tested a larger exploration rate, i.e., q = 0 .
The results are similar with q = 0 . 8 .
For ResNet-50 on ImageNet, we set t ex = 30 and t re = 70 so that we can obtain M = 2 subnetworks with 170 epochs and M = 4 subnetworks with 310 epochs, respectively.
We set p = 0 .
5 and q = 0 . 8 .
The rest of hyperparameters are the same as DST Ensemble.
Due to the limited resources, we only test the 80% sparsity for ResNet-50.
We believe that the ensemble performance can further improve with lower sparsity.
D E XPERIMENTAL M ETRICS
To measure the predictive accuracy and robustness, we follow the Uncertainty Baseline 4 and focus on test accuracy (Acc), negative log-likelihood (NLL), and expected calibration error (ECE) on the i.i.d.
test set, corrupted test sets (i.e., CIFAR-10-C and ImageNet-C)
(Hendrycks & Dietterich, 2019) which contain 19 natural corruptions such as added blur, compression artifacts, frost effects, etc, as well as on the natural adversarial samples (i.e., ImageNet-A)
(Hendrycks et al., 2019).
We adopt { cAcc, cNLL, cECE } and { aAcc, aNLL, aECE } to denote the corresponding metrics on corrupted test sets and natural adversarial samples, respectively.
Negative log-likelihood (NLL) is a proper scoring rule and a popular metric for evaluating predictive uncertainty (Quinonero-Candela et al., 2005).
Expected Calibration Error (ECE) (Naeini et al., 2015), is a widely adopted metric to approximate the difference in expectation between conﬁdence and accuracy of machine learning models (i.e., miscalibration).
Speciﬁcally, it partitions predictions into M equally-spaced bins, and then calculates a weighted average of the accuracy/conﬁdence discrepancy in each of these bins.
Larger ECE values represent a worse match between conﬁdence and accuracy.
To compare the computational efﬁciency, we report the training FLOPs required to obtain the targeted number of subnetworks for all methods and normalize them with the FLOPs required by a dense model.
Following RigL 5 (Evci et al., 2020a), the FLOPs are calculated with the total number of multiplications and additions layer by layer.
Brieﬂy speaking, with ERK distribution, the training
FLOPs of a sparse Wide ResNet28-10 at sparsity S = 0 .
8 and S = 0 .
9 are 33.7% and 16.7% of the dense model, respectively.
For sparse ResNet-50 at S = 0 . 8 , the required training FLOPs is 42% of the dense model.
We omit the additional computational cost for the extra inputs and the extra outputs of MIMO, as it is negligible compared with the whole training FLOPs.
Moreover, we suppose the hardware can fully utilize large batch size so that BatchEnsemble incurs almost no additional computational overhead and memory cost.
E C OMPARISON WITH S NAPSHOT AND FGE
In this Appendix
[Table]
we compare our methods with the ensemble methods that use cyclical learning rate schedules to discover diverse dense networks,
i
[Table]
e., Snapshot (Huang et al., 2017) and
FGE (Garipov et al., 2018).
For a relatively fair comparison, we compare our methods with their ‘1B’ versions (Garipov et al., 2018) which have similar training FLOPs with our methods.
However, their test FLOPs would be much larger as their ensemble members are dense networks.
As shown in Table 6, while achieving relatively similar performance, Snapshot and FGE require signiﬁcantly more test FLOPs than the DST-based ensemble methods
[Table] [Table]
F P REDICTION D ISAGREEMENT ON CIFAR-10/
We compare the prediction disagreement among the naive dense ensemble, DST Ensemble with sparsity S = 0 .
8 , EDST Ensemble with sparsity S = 0 .
8 and S = 0 .
The comparison results on CIFAR-10 and CIFAR-100 are shown in Figure 6 and Figure 7, respectively.
We ﬁnd that the subnetworks discovered by DST Ensemble are even more diverse than the ones discovered by the traditional dense ensemble, conﬁrming our hypothesis that dynamic sparsity can provide extra diversity in addition to random initializations.
While the average functional diversity of the subnetworks discovered by EDST Ensemble is a bit lower compared with DST Ensemble, the diversity is still notable.
Moreover, as expected, at a higher sparsity S = 0 .
9 , the prediction diversity of EDST
Ensemble slowly rises as the ensemble size M increases
[Figure]
and ends up with a similar diversity as the
Dense Ensemble
[Figure] [Figure]
G D ISAGREEMENT B REAKDOWN BETWEEN V ARIOUS S PARSE N ETWORK E NSEMBLES ON CIFAR-
In this Appendix
[Table]
we try to provide a detailed illustration about various classes that cause the disagreement between various sparse network ensembles.
We choose the ﬁrst 10 classes in CIFAR100 and measure the disagreement between the subnetworks learned by the same method on these classes.
The results are shared in Table 7.
The overall breakdown of disagreement is very similar to the diversity measurement results, i.e., DST Ensemble > Static Ensemble > EDST Ensemble > LTR
Ensemble.
The prediction on classes like Fish, Baby, and Bear have higher diversity than Apple and
Beetle.
[Table]
H C ORRELATION M EASUREMENT B ETWEEN THE A CCURACY OF S UBNETWORKS AND T HEIR E NSEMBLE A CCURACY
In the main paper, we hypothesize that high expressibility is a crucial desideratum for FreeTickets to guarantee superior ensemble performance.
Here, we apply two widely used correlation measurements,
Pearson correlation (Pearson, 1895) and Spearman correlation (Powers & Xie, 2008), to support our hypothesis.
We use these two methods to measure the correlation between the mean accuracy of individual networks and the accuracy of their ensemble (higher values refer to higher correlation).
The results are shared in Table 8.
We ﬁnd that the individual accuracy and the ensemble accuracy are very highly correlated on CIFAR-100.
Their correlation is relatively low on CIFAR-10 likely due to the almost saturated accuracy of the individual subnetwork (96%).
Given such high accuracy, even the usually clear and strong correlation between the sparsity and test accuracy becomes relatively vague on CIFAR-10 in Figure 4-a and Figure 4-b.
[Table]
I E FFECT OF THE G LOBAL E XPLORATION R ATE ON EDST E NSEMBLE
Intuitively, a larger global exploration rate q leads to larger sparse connectivity diversity whereas a too large q would prune the important connections, hurting the ensemble accuracy.
We perform a sensitivity study with EDST Ensemble in Table 9, whose results are in line with our conjecture.
A large global exploration rate q = 0 .
8 promote higher diversity but degrades the ensemble accuracy a bit compared with q = 0 .
q = 0 .
5 seems to achieve the best ensemble performance
[Table] [Table]
J E FFECT OF R EGROWTH M ETHODS ON EDST E NSEMBLE
Random regrowth naturally considers a larger range of parameters to explore compared to the gradient regrowth as used in the main paper.
We added below a small set of experiments to study different growth criteria (random vs gradients) on the EDST Ensemble.
As shown in Table 10, it seems that random growth may provide more diversity and better performance for more complex datasets (
i
[Table]
e.,
CIFAR-100), but more studies are necessary to understand the phenomenon better.
[Table]
K E XPERIMENTS WITH O O D R ESULTS
In this section, we report the performance of OoD detection for single sparse networks and ensemble models.
Speciﬁcally, we use datasets that are not seen during training time for OoD evaluation.
Following the classic routines (Augustin et al., 2020; Meinke & Hein, 2019), SVHN (Netzer et al., 2011), CIFAR-100 (Krizhevsky & Hinton, 2009), and CIFAR-10 with random Gaussian noise (Hein et al., 2019) are adopted for models trained on CIFAR-10 (Krizhevsky & Hinton, 2009); SVHN,
CIFAR-10, and CIFAR-100 with random Gaussian noise are used for models trained on CIFAR-100.
The ROC-AUC performance (Augustin et al., 2020; Meinke & Hein, 2019) are reported over the respective out of distribution datasets, as shown in Table 11 and Table 12.
As we expected, our proposed ensemble methods provide gains to the OoD performance.
DST Ensemble and EDST
Ensemble can even surpass the dense models by a clear margin in most cases
[Table] [Table]
[Table]
L E XPERIMENTS WITH A DVERSARIAL R OBUSTNESS
Table 13 presents the adversarial robustness performance of Wide ResNet-28-10 trained on CIFAR010 and CIFAR-100, where the Fast Gradient Sign Method (FGSM) (Szegedy et al., 2013) with step size 8255 is adopted.
Following a similar routine as in Strauss et al.
(2017), we craft adversarial examples for each single model and report the { min, mean, max } robust accuracy over generated attacks from different models.
The results demonstrate that our proposed ensemble methods improve robustness accuracy as well, especially in terms of average robust accuracy.
More surprisingly, the ensemble of static sparse models can even outperform DST-based Ensembles in this setting.
Further analysis on this task serves as interesting future work
[Table] [Table]