FLOPs as a Direct Optimization Objective for Learning Sparse Neural Networks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
There exists a plethora of techniques for inducing structured sparsity in parametricmodels during the optimization process, with the ﬁnal goal of resource-efﬁcientinference.
However, to the best of our knowledge, none use the actual number ofﬂoating-point operations (FLOPs) incurred by the model as part of the optimizationobjective, despite reporting FLOPs as part of the results.
Furthermore, a one-sizeﬁts-all approach ignores realistic system constraints, which differ signiﬁcantlybetween, say, a GPU and a mobile phone—FLOPs on the former incur less latencythan on the latter; thus, it is important for practitioners to be able to specify a targetnumber of FLOPs during model compression.
In the present work, we extend astate-of-the-art technique to directly incorporate FLOPs as part of the optimizationobjective, and we show that, given a desired FLOPs requirement, different neuralnetworks are successfully trained.
Introduction
Neural networks are a class of parametric models that achieve state of the art across a broad rangeof tasks, but their heavy computational requirements hinder practical deployment on resourceconstrained devices, such as mobile phones, Internet-of-things (IoT) devices and ofﬂine embeddedsystems.
Many recent works focus on alleviating these computational burdens, mainly falling undertwo non-mutually exclusive categories: manually designing resource-efﬁcient models, and automatically compressing popular architectures.
In the latter, increasingly sophisticated techniques haveemerged, like (cid:96) 1 norm-based ﬁlter-level pruning (Li et al., 2017), network slimming (Liu et al.,
2017),and Bayesian compression (Louizos et al., 2017), to name a few.
They have achieved respectableaccuracy–efﬁciency operating points, some even Pareto-better than that of the original network; forexample, Liu et al.
(2017) reach an error rate of 6.20% on CIFAR-10 using VGGNet (Simonyan and
Zisserman, 2015) with a 51% FLOPs reduction—an error decrease of 0.14% over the original.
However, to the best of our knowledge, none of the methods use the number of FLOPs as part of theend-to-end optimization objective, despite many reporting it as an important ﬁgure.
Yang et al.
(2017)attempts to target device-speciﬁc metrics directly, but their pruning procedure does not explicitlyinclude the metrics of interest as part of the optimization objective, instead using them as heuristics.
Falling short of continuously deploying a model candidate and measuring actual inference time, as intime-consuming neural architectural search (Tan et al., 2018), we believe that the number of FLOPsis reasonable to use as a proxy measure for actual latency and energy usage; across variants of thesame architecture, it has been suggested to be a stronger predictor of energy usage and latency thanthe number of parameters is (Tang et al., 2018).
Indeed, there are compelling reasons to optimize for the number of FLOPs as part of the trainingobjective: ﬁrst, it would permit FLOPs-guided compression to occur in a more principled manner.
Second, practitioners can directly specify a wanted target of FLOPs, which is important in deployment.
Thus, our main contribution is to present a novel extension of prior state of the art (Louizos et al.
,2018) to incorporate the number of FLOPs as part of the optimization objective, furthermore allowingpractitioners to set and meet a desired compression target.
FLOPs Objective
Formally, we deﬁne the FLOPs objective L flops : f × R m (cid:55)→ N 0 as follows:
[Equation]
where L flops is the FLOPs associated with hypothesis h ( · ; θθθ ) :
= p ( ·| θθθ ) , g ( · ) is a functional with theexplicit dependencies, and I is the indicator function.
We assume L flops to depend only on whetherparameters are non-zero, such as the number of neurons in a neural network.
For a dataset D , ourempirical risk thus becomes
[Equation]
Hyperparameters λ f ∈ R +0 and T ∈ N 0 control the strength of the FLOPs objective and the target,respectively.
The second term is a black-box function, whose combinatorial nature prevents gradientbased optimization; Thus, using the same procedure in Louizos et al.
(2018), we relax the objectiveto a surrogate of the evidence lower bound with a fully-factorized spike-and-slab posterior as thevariational distribution, where the addition of the clipped FLOPs objective can be interpreted as asparsity-inducing prior p ( θθθ ) ∝ exp( − λ f max(0 , L flops ( h,θθθ ) − T )) .
Let z ∼ p ( z | πππ ) be Bernoullirandom variables parameterized by πππ :
[Equation]
where (cid:12) denotes the Hadamard product.
To allow for efﬁcient reparameterization and exact zeros,
Louizos et al.
(2018) propose to use a hard concrete distribution as the approximation, which isa stretched and clipped version of the binary Concrete distribution (Maddison et al., 2017): ifˆ z ∼ BinaryConcrete ( α, β ) , then ˜
z := max(0 , min(1 , ( ζ − γ )
ˆ z + γ )) is said to be a hard concreter.v.
, given ζ > 1 and γ < 0 .
The approximation is then
[Equation]
ψ ( · ) is the probability of a gate being non-zero 1 under the hard concrete distribution.
The ﬁrstterm now allows for efﬁcient optimization via the reparameterization trick (Kingma and Welling,2013); for the second, we apply the score function estimator (REINFORCE; Williams, 1992), sincethe FLOPs objective is, in general, non-differentiable and thus precludes the reparameterizationtrick.
High variance is a non-issue because the number of FLOPs is fast to compute, hence lettingmany samples to be drawn.
At inference time, Louizos et al.
(2018) use the deterministic estimator 63 ˆ θθθ :=
θθθ (cid:12) max(0 , min(1 , Sigmoid (log ααα )( ζ − γ ) + γ )) for the ﬁnal parameters ˆ θθθ .
2.1 Computing number of FLOPs under group sparsity
In practice, computational savings are achieved only if the model is sparse across “regular” groups ofparameters, e.g., each ﬁlter in a convolutional layer.
Thus, Louizos et al.
(2018) use one hard concreter.v.
per computational group—in fully-connected layers, one per input neuron; in 2D convolutionlayers, one per output ﬁlter.
Under convention in literature where one addition and one multiplicationeach count as a FLOP, the FLOPs for a 2D convolution layer h conv ( · ; θθθ ) given a random draw z is thendeﬁned as L flops ( h conv , z ) =
( K w K h C in +1)
( I w − K w + P w +1)
( I h − K h + P h +1) (cid:107) z (cid:107) 0 for kernelwidth and height ( K w , K h ) , input width and height ( I w , I h ) , padding width and height ( P w , P h ) ,and number of input channels C in .
The number of FLOPs for a fully-connected layer h fc ( · ; θθθ )is L flops ( h fc , z )
=
( I n + 1) (cid:107) z (cid:107) 0 , where I n is the number of input neurons.
Note that these areconventional deﬁnitions in neural network compression works—the objective can easily use insteada number of FLOPs incurred by device-speciﬁc algorithms, such as fast Fourier transform-basedconvolution implementations.
Experimental Results
We report results on MNIST, CIFAR-10, and CIFAR-100, training multiple models on each datasetcorresponding to different FLOPs targets.
We follow the same initialization and hyperparameters in
Louizos et al.
(2018), using Adam (Kingma and Ba, 2015) with temporal averaging for optimization,a weight decay of 5 ×
10 − 4 , and an initial α that corresponds to the original dropout rate of that layer.
We similarly choose β = 2 / 3 , γ = − 0 .
[Table]
and ζ = 1 . 1 .
For brevity, we direct the interested readerto their repository 2 for speciﬁcs.
In all of our experiments, we replace the original L 0 penalty withour FLOPs objective
[Table]
and we train all models to 200 epochs; at epoch 190, we prune the networkby weights associated with zeroed gates and replace the r
[Table]
v.’s with their deterministic estimators,then ﬁnetune for 10 more epochs.
For the score function estimator, we draw 1000 samples at eachoptimization step—this procedure is fast and has no visible effect on training time.
[Table]
, T = 400 K
[Table]
L flops , T = 200 K
[Table]
L flops , T = 100 K
[Table]
We choose λ f = 10 − 6 in all of the experiments for LeNet-5-Caffe, the Caffe variant of LeNet-5.
2 Weobserve that our methods (Table 1, bottom three rows) achieve accuracy comparable to those frompast approaches while using fewer FLOPs, with the added beneﬁt of providing a tunable “knob” foradjusting the FLOPs.
Note that the convolution layers are the most aggressively compressed, sincethey are responsible for most of the FLOPs in this model [Table] [Table]
[Table]
Orig.
in Table 2 denotes the original WRN-28-10 model from Zagoruyko and Komodakis (2016)
[Table]
and
L 0 -* refers to the models in Louizos et al.
(2018); likewise, we augment CIFAR-10 and CIFAR-100with standard random cropping and horizontal ﬂipping.
For each of our results (last two rows), wereport the median error rate of ﬁve different runs, executing a total of 20 runs across two models foreach of the two datasets; we use λ f = 3 × 10 − 9 in all of these experiments.
We also report both theexpected FLOPs and actual FLOPs, the former meaning the number of FLOPs, on average, at trainingtime under stochastic gates and the latter denoting the number of FLOPs at inference time.
We restrictthe FLOPs calculations to the penalized non-residual convolution layers only.
For CIFAR-10, ourapproaches result in Pareto-better models with decreases in both error rate and the actual number ofinference-time FLOPs.
For CIFAR-100, we do not achieve a Pareto-better model, since our approachtrades accuracy for improved efﬁciency.
The acceptability of tradeoff depends on the end application.