LCS: L EARNING C OMPRESSIBLE
S
UBSPACES FOR A DAPTIVE N ETWORK C OMPRESSION AT I NFERENCE
T IME
Anonymous authors
Paper under double-blind review
A BSTRACT
When deploying deep learning models to a device, it is traditionally assumed that available computational resources (compute, memory, and power) remain static.
However, real-world computing systems do not always provide stable resource guarantees.
Computational resources need to be conserved when load from other processes is high or battery power is low.
Inspired by recent works on neural network subspaces, we propose a method for training a compressible subspace of neural networks that contains a ﬁne-grained spectrum of models that range from highly efﬁcient to highly accurate.
Our models require no retraining, thus our subspace of models can be deployed entirely on-device to allow adaptive network compression at inference time.
We present results for achieving arbitrarily ﬁnegrained accuracy-efﬁciency trade-offs at inference time for structured and unstructured sparsity.
We achieve accuracies on-par with standard models when testing our uncompressed models, and maintain high accuracy for sparsity rates above 90% when testing our compressed models.
We also demonstrate that our algorithm extends to quantization at variable bit widths, achieving accuracy on par with individually trained networks.
I NTRODUCTION
Deep neural networks are deployed to a variety of computing platforms, including phones, tablets, and watches (Dhar et al., 2019).
Networks are generally designed to consume a ﬁxed budget of resources, but the compute resources available on a device can vary while the device is in use.
Computational burden from other processes, as well as battery life, may inﬂuence the availability of resources to a neural network.
Adaptively adjusting inference-time load is beyond the capabilities of traditional neural networks, which are designed with a ﬁxed architecture and a ﬁxed resource usage.
A simple approach to the problem of providing an accuracy-efﬁciency trade-off is to train multiple networks of different sizes.
In this approach, multiple networks are stored on the device and loaded into memory when needed.
There is a breadth of research in the design of efﬁcient architectures that can be trained with different capacities, then deployed on a device (Howard et al., 2017; Sandler et al., 2018;
Howard et al., 2019).
However, there are a few drawbacks with using multiple networks to provide an accuracy-efﬁciency trade-off: (1) it requires training and deploying multiple networks (which induces training-time computational burden and on-device storage burden), (2) it requires all compression levels to be speciﬁed before deployment, and (3) it requires new networks to be loaded into memory when changing the compression level, which prohibits real-time adaptive compression.
Previous methods address the ﬁrst issue in the setting of structured sparsity by training a single network conditioned to perform well when varying numbers of channels are pruned (Yu et al., 2018;
Yu & Huang, 2019).
However, these methods require BatchNorm (Ioffe & Szegedy, 2015) statistics to be recalibrated for every accuracy-efﬁciency conﬁguration before deployment.
This requires users to know every compression level in advance and requires storage of extra parameters (which can be problematic for low-compute devices).
Another method trains a larger network that can be queried for efﬁcient subnetworks (Cai et al., 2020), but requires users to reload network weights, and to know every compression level before deployment.
Efficiency
A cc u r ac y
L i n e a r S
u b
s p a c e
Efficiency-Accuracy Trade-Off
[Equation]
(a)
Learning Compressible Subspaces
Input: Network subspace ω ∗ ( α ) , compression level calculator γ ( α ) , compression function f ( ω , γ ) , stochastic sampling function α n ∈ [0 , 1] n , dataset D , loss function L , regularization function r .
Replace BatchNorm layers with GroupNorm.
while ω ∗ has not converged do for batch B in D do
Sample a vector α ∼ α n l ←for α ∈ α do # compute loss on batch l += L ( f ( ω ∗ ( α ) , γ ( α )) , B ) end for # apply regularization l += r ( ω ∗ ) backpropagate loss l apply gradient update to ω ∗ end for end while return ω ∗ (b)
We address the issue of deploying a model that can be compressed in real-time without specifying the compression levels beforehand.
Inspired by Wortsman et al.
(2021), we formulate this problem as computing an adaptively compressible network subspace (Figure 1a).
We train a linear subspace using two sets of weights, conditioning them so that a convex combination of them produces an accurate network.
We expand on Wortsman et al.
(2021) by (1) specializing our linear subspace to contain an arbitrarily ﬁne-grained efﬁciency-accuracy trade-off (meaning, any compression level within predeﬁned limits can be chosen), and (2) replacing BatchNorm (Ioffe & Szegedy, 2015) with
GroupNorm (Wu & He, 2018) to avoid BatchNorm recalibration.
Contributions: Our contributions are as follows.
(1) We introduce our method, Learning Compressible Subspaces (LCS), for training models that can be compressed with arbitrarily ﬁne-grained compression levels in real-time after deployment.
To our knowledge this has not been done before.
(2) We demonstrate a new use for neural network subspaces, specializing at one end for high-accuracy and at the other end for high-efﬁciency.
(3) We provide an empirical evaluation of our method using structured sparsity, unstructured sparsity, and quantization.
(4) We open source our code at https://withheld.for.review .
R ELATED W ORK
Architectures Demonstrating Efﬁciency-Accuracy Trade-Offs:
Several recent network architectures include a hyperparameter to control the number of ﬁlters in each layer (Howard et al., 2017;
Sandler et al., 2018; Howard et al., 2019) or the number of blocks in the network (Tan & Le, 2019).
In Once For All (Cai et al., 2020), the need for individually training networks of different sizes is circumvented.
Instead, a single large network is trained, then queried for efﬁcient subnetworks.
We differ from these methods in two ways.
First, we compress on-device without specifying the compression levels before deployment.
Previous works require training separate networks (or in
[Table]
| ω | 2 | ω | n | ω | | ω | + n | b | | ω | + n | b | the case of Once For All, querying a larger model for a compressed network) before deployment.
Second, we don’t require deploying a set of weights for every compression level.
A method for post-training quantization to variable bit widths appears in Shkolnik et al.
but it relies on calibration data and cannot be run on low-compute edge devices.
Thus, compression levels must be speciﬁed before deployment.
They also present a method for quantization-aware training to variable bit widths, but most of their results keep activation bit widths ﬁxed, whereas we vary it.
Adaptive Compression: Recent works train a single neural network which can be adaptively conﬁgured at inference time to execute at different compression levels.
These methods are the closest to our work.
In Learning Efﬁcient Convolutions (LEC) (Liu et al., 2017), the authors train a single network, then ﬁne-tune it at different structured sparsity rates (though they also present results without ﬁne-tuning, which is closer to our use case).
Other methods train a single set of weights conditioned to perform well when channels are pruned, but require recalibration (or preemptive storage) of BatchNorm (Ioffe & Szegedy, 2015) statistics at each sparsity level.
These methods include Network Slimming (NS) (Yu et al., 2018) and Universal Slimming (US) (Yu & Huang, 2019).
Similar methods train a single network to perform well at various levels of quantization by storing extra copies of BatchNorm statistics (Guerra et al., 2020).
We differ from these methods by avoiding the need to recalibrate or store BatchNorm statistics (Section 3.3) and by allowing for ﬁne-grained selection of compression levels at inference time, neither of which have been done before (Table 1).
Additionally, our method is broadly applicable to a variety of compression methods, whereas previous works focus on a single compression method.
Other Post-Training Compression Methods: Other works have investigated post-training compression.
In Nagel et al.
(2019), a method is presented for compressing 32 bit models to 8 bits, though it has not been evaluated in the low-bit regime.
It involves running an equalization step and assuming a Conv-BatchNorm-ReLU network structure.
A related post-training compression method is shown in Horton et al.
(2020), which shows results for both quantization and sparsity.
However, their sparsity method requires a lightweight training phase.
We differ from these methods in providing real-time post-deployment compression for both sparsity and quantization without making assumptions about the network structure.
Sparse Networks: Sparsity is a well-studied approach to reduce neural network compute.
Recent works have achieved state-of-the-art accuracy using simple magnitude-based pruning (Han et al., 2015; Zhu & Gupta, 2017; See et al., 2016; Narang et al., 2017).
When experimenting with structured sparsity, we simply remove ﬁlters from the network, as described in Yu et al.
When investigating unstructured sparsity, we use TopK pruning (Zhu & Gupta, 2017).
Quantized Networks: Network quantization represents the weights and/or activations of neural networks with lower-bit integral representations, rather than 32 -bit ﬂoating point.
This reduces model size and memory bandwidth and can accelerate inference.
Our work follows the afﬁne quantization scheme presented in Jacob et al.
(2018), in which a scale and offset are applied to an integer tensor to approximate the original ﬂoating-point tensor.
However, our method is general enough to utilize any other quantization scheme.
Neural Network Subspaces: The idea of learning a neural network subspace is introduced in Wortsman et al.
(2021) (though another formulation was introduced concurrently in Benton et al.
Multiple sets of network weights are treated as the corners of a simplex, and an optimization procedure updates these corners to ﬁnd a region in weight space in which points inside the simplex correspond to accurate networks.
We note that Wortsman et al.
(2021) recalibrate BatchNorm statistics before evaluation, which we avoid.
C OMPRESSIBLE S UBSPACES 3. C OMPRESSIBLE L INES
In our method, we train a neural network subspace (Wortsman et al., 2021) that contains a spectrum of networks that each have a different efﬁciency-accuracy trade-off.
In Wortsman et al.
(2021), an architecture is chosen, and its collection of weights is denoted by ω .
A linear subspace is parameterized by two sets of network weights, ω
1 and ω 2 .
At training time, a network is sampled from the line deﬁned by this pair of weights, ω ∗ ( α ) =
α ω 1 + (1 − α ) ω 2 , where α ∈ [0 , 1] .
The forward and backward pass of gradient descent are computed using ω ∗ ( α ) , and gradient updates are backpropagated to ω 1 and ω 2 .
A cosine distance regularization term is used to ensure that ω 1 and ω 2 don’t converge to a single point.
We propose training a linear subspace of networks in a similar manner, but we also bias the subspace to contain high-accuracy solutions at one end and high-efﬁciency solutions at the other end.
To accomplish this, we introduce a parameter γ used to control the compression level and a compression function f ( ω , γ ) used to compress a model’s weights ω using a compression level γ .
The parameter γ induces different compression levels at different positions in the linear subspace, and therefore can be considered a function γ ≡ γ ( α ) .
When training, we ﬁrst sample a network from the subspace, then compress it, obtaining a network with weights f ( ω ∗ ( α ) , γ ( α )) .
We then perform a standard forward and backward pass of gradient descent with it, backpropagating gradients to ω 1 and ω 2 .
In summary, our training method adjusts the standard subspace training method to include the application of a compression function f at a compression level γ ( α ) .
Note that we apply the compression after sampling the model ω ∗ ( α ) from the subspace, since we want the sampled model to be compressed (rather than ω 1 and ω 2 ) for efﬁcient inference.
We experiment with formulating our compression function as structured sparsity, unstructured sparsity, and quantization (Section 4).
In all cases, the cost of computing f is small compared to the cost of a network forward pass (Section A.1).
3. C OMPRESSIBLE P OINTS
In Section 3.1, we discussed formulating our subspace as a line connected by two endpoints in weight space.
This formulation requires additional storage resources to deploy the subspace (Table 1), since an extra copy of network weights is stored.
For some cost-efﬁcient computing devices, this storage may be signiﬁcant.
To eliminate this need, we propose training a degenerate subspace with a single point in weight-space (rather than two endpoints).
We still use α ∈ [0 , 1] to control our compression ratio, but our subspace is parameterized by a single set of weights, ω ∗ ( α )
= ω .
The compressed weights are now expressed as f ( ω ∗ ( α ) , γ ( α )) ≡ f ( ω , γ ( α )) .
This corresponds to applying varying levels of compression during each forward pass.
The cosine distance regularization term included in the linear subspace is no longer needed because only one set of weights is trained.
This method still produces a subspace of models in the sense that, for each value of α , we obtain a different compressed network f ( ω , γ ( α )) .
However, we no longer use different endpoints of a linear subspace to specialize one end of the subspace for accuracy and the other for efﬁciency.
Instead, we condition one set of network weights to tolerate varying levels of compression.
3. C IRCUMVENTING B ATCH N ORM R ECALIBRATION
Previous works that train a compressible network require an additional training step to calibrate
BatchNorm (Ioffe & Szegedy, 2015) statistics, as in Universal Slimming (Yu & Huang, 2019).
Alternatively, they require storing separate BatchNorm statistics for each compression level, as in
Slimmable Networks (Yu et al., 2018).
This precludes both methods from evaluating at arbitrarily ﬁne-grained compression levels after deployment (Table 1).
We seek to eliminate the need for recalibration or storage. [Figure]
CIFAR-10) To understand the need for recalibration in previous works, recall that BatchNorm layers store the per-channel mean of the inputs µ and the per-channel variance of the inputs σ 2 .
The recalibration step is needed to correct µ and σ 2 , which are corrupted when a network is adjusted.
Adjustments that corrupt statistics include applying structured sparsity, unstructured sparsity, and quantization.
In Figure 2, we analyze the inaccuracies of BatchNorm statistics for two models trained with speciﬁc unstructured sparsity levels and tested with a variety of inference-time unstructured sparsity levels.
We calculate the differences between stored BatchNorm means µ and the true mean of a batch ˆ µ during the test epoch of a cPreResNet20 (He et al., 2016) model on CIFAR-10 (Krizhevsky, 2009).
In Figure 2a and Figure 2b, we show the distribution of mean absolute differences (MADs) | µ − ˆ µ | across all layers of the models.
Models have lower BatchNorm errors when evaluated near sparsity levels that match their training-time target sparsity.
Applying mismatched levels of sparsity shifts the distribution of these errors away from 0 .
In Figure 2c and Figure 2d, we show the average of | µ − ˆ µ | across the test set for each of the BatchNorm layers.
Across layers, the lowest error is achieved when the level of sparsity matches training.
In Figure 2e and Figure 2f, we show the average of | µ − ˆ µ | and the corresponding test set accuracy for various levels of inference-time sparsity.
We ﬁnd that the increased error in BatchNorm is correlated with decreased accuracy.
See Figure 7 and Figurein the Appendix for similar analyses with structured sparsity and quantization.
Thus, BatchNorm layers’ stored statistics can become inaccurate during inference-time compression, which can lead to accuracy degradation.
To circumvent the need for BatchNorm, we adjust our networks to use GroupNorm (Wu & He, 2018).
This computes an alternative normalization over g groups of channels rather than across a batch.
It doesn’t require maintaining a running average of the mean and variance across batches of input, so there are no stored statistics that can be corrupted if the network changes.
Formally, let i be an index into a 4-dimensional image tensor, with components i =
[ i N , i C , i H , i W ] indexing the batch, channel, height, and width, respectively.
The set S i of indices used to compute the normalization of batch element i is
[Equation]
Batch element x i is normalized by subtracting the mean of elements indexed by S i and dividing by the standard deviation of elements indexed by S i .
GroupNorm typically uses the hyperparameter g = 32 , but it also includes InstanceNorm (Ulyanov et al., 2016) (in which g = c , where c is the number of channels) as a special case.
We use g = c in structured sparsity experiments, since the number of channels is determined dynamically and is not always divisible by 32.
We use g =in all other experiments 1 .
After applying this normalization, we also apply a learned per-channel afﬁne transformation, as with BatchNorm.
3. B IASED S AMPLING
The endpoints of a learned linear subspace typically demonstrate slightly reduced accuracy compared to the center of the line, as observed in Wortsman et al.
Inspired by this observation, we use biased sampling to choose network endpoints more frequently when training a linear subspace, to ensure that endpoints are accurate.
A similar algorithmic choice is made in Yu & Huang (2019), which proposes using a “sandwich method” for training.
This method involves performing forward and backward passes at the maximum and minimum sparsity levels, as well as a few randomly chosen sparsity levels in between.
After this round of forward and backward passes, the gradient update is applied.
To implement our sampling of α , we use a stochastic function α n : Ω → [0 , 1] n , where Ω represents the state of the stochastic function.
For each batch, we sample α ∼ α n .
We perform n forward and backward passes using compressed networks f ( ω ∗ ( α i ) , γ ( α i )) for i ∈ { 1 , ..., n } , where α i is the i th element of α .
The stochastic function used varies based on the compression function chosen.
See
Section 4.1 (structured sparsity), Section 4.2 (unstructured sparsity), and Section 4.3 (quantization) for more details.
Our full method for training a compressible subspace is shown in Figure 1b.
E XPERIMENTS
We present results in the domains of structured sparsity, unstructured sparsity, and quantization.
We train using Pytorch (Paszke et al., 2019) on Nvidia Tesla V100 GPUs.
On CIFAR-10 (Krizhevsky, 2009), we experiment with the pre-activation version of ResNet20 (He et al., 2016) presented in the PyTorch version of the open-source code provided by Liu et al.
We abbreviate it as “cPreResNet20” to denote that it’s a pre-activation network designed for usage with CIFAR-10.
We use a pre-activation version because our Learning Efﬁcient Convolutions (LEC)
(Liu et al., 2017) baseline requires it.
We follow hyperparameter choices in Wortsman et al.
(2021) for our methods and baselines.
We warm up to our initial learning rate of 0 .
1 (or 0 .
025 in the case of quantization, for stability) over 5 epochs, then decay to 0 over 195 epochs using a cosine schedule.
We use a batch size of 128 on a single GPU.
We set the weight decay to 5 × 10 − 4 .
We run 3 trials for each experiment and report the mean and standard deviation (which is often too small to be visible).
Our baseline cPreResNet (with BatchNorm) achieves an accuracy of 91 .
69% on CIFAR-10.
For ImageNet (Deng et al., 2009) experiments, we use ResNet18 (He et al., 2016) and VGG19 (Simonyan & Zisserman, 2014).
We use the version of VGG19 provided by the same repository as our cPreResNet20 implementation.
This implementation modiﬁes VGG19 slightly by adding BatchNorm layers and removing the last two fully connected layers.
Again, we follow hyperparameter settings in Wortsman et al.
(2021) for our methods and baselines.
We warm up to our initial learning rate of 0 .
1 (or 0 .
025 in the case of quantization, for stability) over 5 epochs, then decay to 0 overepochs using a cosine schedule.
We set the weight decay to 5 × 10 − 5 .
We use a batch size of 128 for
0.5 Sparsity 0.
ResNet18 (ImageNet)
NS US
LEC LCS+P+IN
LCS+L+IN 0.
0.5 Sparsity 0. T e s t A cc u r ac y cPreResNet20 (CIFAR-10) ResNet18 with a single GPU, and a batch size of 256 for VGG19 with 4 GPUs.
Unlike when training with CIFAR-10, we run a single trial for each experiment due to the increased computational expense of training on ImageNet.
Our baseline ResNet18 (with BatchNorm) achieves an accuracy of 70 .
72% on ImageNet, and our VGG19 (with BatchNorm) achieves 62 .
4. S TRUCTURED S PARSITY
For structured sparsity, our method follows the algorithm in Figure 1b.
Our compression level calculator γ ( α ) : [0 , 1] → [0 . 25 , 1 . 0] is the unique afﬁne transformation over its domain and range.
Our compression function f ( ω , γ ) prunes away a fraction 1 − γ of the input and output channels in each layer (except the input to the ﬁrst layer and the output of the last layer).
This is the same compression scheme used in Network Slimming (NS) (Yu et al., 2018) and Universal Slimming (US) (Yu & Huang, 2019).
Our stochastic sampling function samples α as [0 . 25 , 1 . 0 , U (0 . 25 , 1) , U (0 . 25 , 1)] , where U ( a, b ) samples uniformly in the range [ a, b ] .
This choice mirrors the “sandwich rule” used in US.
As discussed in Section 3.3, we use a special case of GroupNorm (Wu & He, 2018) known as InstanceNorm (Ulyanov et al., 2016) since the number of channels in the network varies. See Section A.3 for additional details on our linear subspace setup.
When running baselines, we do not allow recalibration of BatchNorm (Ioffe & Szegedy, 2015) statistics (as used in US), or storage of a predeﬁned set of BatchNorm statistics (as used in NS), since it would contradict our goal of adaptive compression to arbitrarily ﬁne-grained sparsity levels at inference time.
We instead use a single BatchNorm layer, using only the channels not removed by the structured pruning step.
Similarly, we do not allow ﬁne-tuning (normally used in LEC).
The main difference between our point method and US is that we use InstanceNorm instead of BatchNorm.
The main difference between US and SN is that US includes the sandwich rule described above. Instead, NS samples at predeﬁned sparsity levels [0 . 25 , 0 .
75 , 1] at each iteration.
We present results for structured pruning on ResNets in Figure 3 (see Section A.5 for VGG19 results and runtime characteristics).
Our method produces a stronger efﬁciency-accuracy trade-off than competing methods.
The poor performance of LEC (Liu et al., 2017) occurs because this method expects weights to be ﬁne-tuned after sparsity is applied, which we do not allow.
The dips and spikes shown for NS occur because this method trains at discrete width factors of [0 . 25 , 0 . 50 , 0 . 75 , 1] , which roughly correspond to sparsities of [0 . 9375 , 0 . 75 , 0 . 4325 , 0] .
This method shows stronger accuracies at these sparsities, with the exception of 0 . 9375 , at which point, inaccuracies in BatchNorm statistics are more extreme (see Figure 7 in the Appendix).
Similarly, US peaks in accuracy at moderate sparsity rates, but decreases in accuracy at high sparsity rates.
We believe this strong decrease for higher sparsity rates occurs because the BatchNorm channels used in this case are also used at all other sparsity rates, thus the stored statistics don’t accurately reﬂect the statistics of the highly-sparse network.
[Figure]
Interestingly, our method with a linear subspace (LCS+L+IN) and our method with a point subspace (LCS+P+IN) achieve very similar performance. We believe that the extra weights in LCS+L+IN are not necessary because individual channels are able to specialize, since some of them are only used for larger subnetworks.
4. U NSTRUCTURED S PARSITY
For unstructured sparsity, our method follows the algorithm in Figure 1b.
Our compression level calculator is γ ( α ) =
1 − α .
Our compression function f ( ω , γ ) is TopK sparsity (Zhu & Gupta, 2017), which removes a fraction γ of the weights with the smallest absolute value from each layer (except the input and output layer).
Our stochastic sampling function samples one value for α in [0 . 005 , 0 . 05]
(see Section A.4 for additional details regarding sampling and TopK’s warmup phase).
We choose this interval because our networks are shown to tolerate high sparsity levels, losing almost no accuracy even at 90% sparsity.
See Section A.5 for experiments with broader ranges of sparsity, and for runtime characteristics of sparse models.
To our knowledge, ﬁne-grained adaptive compression has not been explored for unstructured sparsity.
Thus, we compare to standard networks (using BatchNorm) trained at speciﬁc sparsity levels and evaluated at a variety of sparsity levels.
See Figure 4 for results.
We ﬁnd that our baselines always achieve highest accuracy at the sparsity level that matches their training-time TopK value, as expected.
Accuracy quickly degrades at other sparsity levels.
By contrast, our method is able to achieve high accuracy for a wide variety of sparsity levels.
Interestingly, our line method usually achieves higher accuracy at lower sparsity rates in Figure 4, but this pattern does not hold over a wider sparsity range (Section A.5).
Further investigation into this phenomena is future work.
4. Q UANTIZATION
For quantization, our method follows Algorthm 1b.
Our compression level calculator is γ ( α ) =
+6 α .
Our compression function f ( ω , γ ) is afﬁne quantization as described in (Jacob et al., 2018).
Our stochastic sampling function samples one value for α uniformly over the set { 1 / 6 , 2 / 6 , ..., 6 / 6 } .
This corresponds to training with bit widths 3 through 8 (we avoid lower bit widths to circumvent training instabilities we encountered in baselines).
We train without quantizing the activations for the ﬁrst 80% of training, then add activation quantization for the remainder of training.
Weights are quantized throughout training.
We present results for our method in Figure 5, comparing to models trained at a ﬁxed bit width and evaluated at a variety of bit widths.
See Section A.5 for ResNet18 results, and for memory usage characteristics of models.
Generally, baselines achieve high accuracy at the bit width at which they were trained, and reduced accuracy at other bit widths.
Our method using a linear subspace (LCS+L+GN) achieves high accuracy at all bit widths, matching or exceeding accuracies of individual networks trained for target bit widths.
In the case of VGG19, we found that our accuracy
[Figure]
(CIFAR-
[Figure]
[Figure]
even exceeded the baselines.
We believe part of the increase is due to GroupNorm demonstrating improved results on this network compared to BatchNorm (which does not happen with ResNets, as reported in
4. F URTHER A NALYSIS
In Figure
[Figure]
we provide additional experimental evidence that our linear subspace method (LCS+L) trains a subspace specialized for high-accuracy at one end and high-efﬁciency at the other end.
We plot the validation accuracy along our subspace, as well as the validation accuracy along our subspace when compressing with ˜ f
( ω ∗ ( α ) , γ ( α )) ≡ f ( ω ∗ ( α ) , γ (1 − α )) .
In other words, the weights that were trained for low compression levels are evaluated with high compression levels, and vice versa.
We see that this leads to a large drop in accuracy, conﬁrming that our method has conditioned one side of the line to achieve high accuracy at high sparsities, and the other side of the line to achieve high accuracy at low sparsities.
C ONCLUSION
We present a method for learning a compressible subspace of neural networks.
Our subspace can be compressed in real-time without retraining and without specifying the compression levels before deployment.
We can produce a variety of models with a ﬁne-grained efﬁciency-accuracy tradeoff.
We demonstrate that our generic algorithm can be applied to structured sparsity, unstructured sparsity, and quantization.
R EPRODUCIBILITY S TATEMENT
We open source our code to reproduce our results (including baselines) at https://withheld.
A A PPENDIX A. C OMPUTATIONAL C OST OF C OMPRESSION
We justify our claim that our network compression method is real-time.
For a typical convolutional layer without downsampling, the computational cost is O ( d 1 d 2 H 1 W 1 k h k w ) , where d 1 is the number of input channels, d 2 is the number of output channels, H 1 is the input tensor height, H 2 is the input tensor width, k h is the kernel height, and k w is the kernel width.
For our line method, we must calculate ω ∗ ( α ) , which takes O ( d 1 d 2 k h k w ) for each convolution, which is negligible compared to the convolution.
For our point method, calculating ω ∗ ( α ) requires no computation.
We must also calculate the compressed network, f ( ω ∗ ( α ) , γ ( α )) .
Our structured sparsity method (described in Section 4.1) takes O (1) for each convolution, since we only need to mark each layer with the number of convolutional ﬁlters to ignore.
Our unstructured sparsity method (TopK, described in Section 4.2) takes O ( d 1 d 2 k h k w ) for each convolution (to calculate the threshold, then discard parameters below the threshold).
Our quantization method (Jacob et al., 2018) takes
O ( d 1 d 2 k h k w ) for each convolution (to calculate the afﬁne transform parameters and apply them).
In all cases, the complexity of computing f ( ω ∗ ( α ) , γ ( α )) is lower than the cost of the convolutional forward passes, meaning our method can be considered real-time.
A. F URTHER B ATCH N ORM A NALYSIS
In Figure 2, we analyzed the inaccuracies of BatchNorm statistics for models trained with unstructured sparsity.
We show a similar analysis for the case of Universal Slimming (US) (Yu & Huang, 2019) and Network Slimming (NS) (Yu et al., 2018) in Figure 7.
We show the case of quantization in Figure 8.
A. S TRUCTURED S PARSITY D ETAILS
When training our method with a line in the structured sparsity setting, we do not use two sets of weights (e.g. ω 1 and ω 2 , Section 3.1) for convolutional ﬁlters.
Instead, we only use two sets of weights for afﬁne transforms in GroupNorm (Wu & He, 2018) layers.
For the convolutional ﬁlters, we instead use a single set of weights, similar to our point formulation (and similar to US (Yu &
Huang, 2019) and NS (Yu et al., 2018)).
By contrast, we use two sets of weights for convolutional ﬁlters as well as for afﬁne transforms when experimenting with unstructured sparsity (Section 4.2) and quantization (Section 4.3).
The reason for only using one set of convolutional ﬁlters in the structured sparsity setting is that the ﬁlters themselves are able to specialize, even without an extra copy of network weights.
Some ﬁlters are only used in larger networks, so they can learn to identify different signals than the ﬁlters used in all subnetworks.
Note that this ﬁlter specialization argument does not apply to our unstructured or quantized settings.
In preliminary experiments, we found that using a single set of weights for convolutions in our structured sparsity experiments gave a slight improvement over using two sets of weights (roughly 2% for cPreResNet20 (He et al., 2016) on CIFAR-10 (Krizhevsky, 2009)).
We hypothesize that this slight difference may be attributed to the subspace training framework’s regularization term (which increases cosine distance between ω 1 and ω 2 ), which discourages identical ﬁlters from being used in both ω 1 and ω 2 even if they are optimal.
A. U NSTRUCTURED S PARSITY D ETAILS
It is typical to include a warmup phase when training models with TopK sparsity (Zhu & Gupta, 2017).
In our baselines in Section 4.2, we increase the sparsity level from 0% to its ﬁnal value over the ﬁrst 80% of training epochs.
For our method, sparsity values fall within a range, so there is no single target sparsity value to warm up to.
For our point method (LCS+P), we simply train for the ﬁrst 80% of training with the lowest sparsity value in our sparsity range.
We ﬁnish training by sampling uniformly between the lowest and highest sparsity levels.
For our line method (LCS+L), our choice of sparsity level is tied to our choice of weight-space parameters through α .
We implement our warmup by simply adjusting γ ( α ) to apply less sparsity early in training, warming up to our ﬁnal sparsity rates over the ﬁrst 80% of training.
In detail, let α min and α max correspond to our minimum and maximum alpha values (for example, in Section 4.2, α min = 0 .
005 , and α max = 0 . 05 ).
As motivated in Section 3.4, we bias sampling of α towards the endpoints of our line. We set α = [ α min ] with 25% probability, α = [ α max ]
[Figure]
MAD Mean (a)
D e n s it y
Target Bit Width =
Layer(c) 0.
M AD
M ea n 0.
MAD Mean (e) 0.
T e s t
A cc u r ac y 0.
MAD Mean (b)
Target Bit Width =
Bit Width
Layer(d) 0.
Bit Width    0.
MAD [Figure] (f)
[Figure]
[Figure]
[Figure]
cPreResNet20 (CIFAR-10)
Sparsity 0.
T e s t
A cc u r ac y
VGG19 (ImageNet)
US LEC NS LCS+L+IN LCS+P+IN with 25% probability, and α =
[ U ( α min , α max )] with 50% probability, where U ( a, b ) samples uniformly in the range [ a, b ] .
To warm up our sparsity rates, we choose
[Equation]
where c is the current iteration number, and t is the total number of iterations in the ﬁrst 80% of training.
At the beginning of training, d = 1 , and γ ( α ) = 0 , corresponding to a sparsity level offor all values of α .
Once 80% of training is ﬁnished, d = 0 , and γ ( α ) = 1 − α for the remainder of training.
This corresponds to our ﬁnal sparsity range.
A. A DDITIONAL R ESULTS
Structured Sparsity: We present results for VGG19 on ImageNet in the structured sparsity setting in Figure 9.
Our method produces a better efﬁciency-accuracy trade-off than baselines.
Note that VGG19 with GroupNorm (Wu & He, 2018) had a higher baseline accuracy than VGG19 with
BatchNorm (Ioffe & Szegedy, 2015) (as noted in Section 4.3).
We also provide a table demonstrating the memory, ﬂops, and runtime of structured sparsity models in Table 2.
Unstructured Sparsity: We present results for a wider range of sparsity values in our unstructured sparsity experiments in Figure 11.
We change our sparsity limits from [0 . 005 , 0 . 05]
(in Section 4.2) to [0 . 025 , 1 . 0] .
All other training details are unchanged.
We ﬁnd that our method is able to produce a higher accuracy over a wider range of sparsities than our baselines.
We also provide a table of unstructured sparsity results in the high sparsity regime (Table 3) and the wider sparsity regime (Table 4), showing memory usage and FLOPS.
Quantization: We present quantization results for ResNet18 in Figure 10.
We ﬁnd that our models approach the accuracy of models trained at a single bit width, and our models generalize better to other bit widths.
We also provide a table of quantization results in Table 5, showing memory usage of the models. Bit Width 0.
T e s t
A cc u r ac y
ResNet18 (ImageNet)
[Figure]
cPreResNet
[Table]
ResNet (ImageNet)
Sparsity (%) 0.
FLOPS ( × 10 6 ) 1814.
Memory (MB) 46.
Runtime (ms) 45.
Acc (LCS+P+IN) 63.
Acc (LCS+L+IN) 64.
Acc (US) 58.
Acc (NS) 50.
VGG (ImageNet)
Sparsity (%) 0.
FLOPS ( × 10 6 ) 19533.
Memory (MB) 82.
Runtime (ms) 388.
Acc (LCS+P+IN) 66.
Acc (LCS+L+IN) 67.
Acc (US) 36.
[Table]
[Figure]
T e s t
A cc u r ac y
[Figure]
cc
[Figure]
ac
[Figure]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
=4) 9.
Acc (Bit Width=3) 0.
VGG (ImageNet)
Memory (MB) 20.
Acc (LCS+P+GN) 57.
Acc (LCS+L+GN) 63.
Acc (Bit Width=8) 60.
Acc (Bit Width=6) 56.
Acc (Bit Width=4) 0.
Acc (Bit Width=3) 0.