E NERGY -C ONSTRAINED C OMPRESSION FOR D EEP N EURAL N ETWORKS VIA W EIGHTED S PARSE P ROJECTION AND L AYER I NPUT M ASKING
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep Neural Networks (DNN) are increasingly deployed in highly energyconstrained environments such as autonomous drones and wearable devices while at the same time must operate in real-time.
Therefore, reducing the energy consumption has become a major design consideration in DNN training.
This paper proposes the ﬁrst end-to-end DNN training framework that provides quantitative energy consumption guarantees via weighted sparse projection and input masking.
The key idea is to formulate the DNN training as an optimization problem in which the energy budget imposes a previously unconsidered optimization constraint.
We integrate the quantitative DNN energy estimation into the DNN training process to assist the constrained optimization.
We prove that an approximate algorithm can be used to efﬁciently solve the optimization problem.
Compared to the best prior energy-saving methods, our framework trains DNNs that provide higher accuracies under same or lower energy budgets.
I NTRODUCTION
Deep neural networks (DNN) have become the fundamental building blocks of many emerging application domains such as computer vision (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014),
speech recognition (Hinton et al., 2012), and natural language processing (Goldberg, 2016).
Many of these applications have to operate in highly energy-constrained environments.
For instance, autonomous drones have to continuously perform computer vision tasks (e.g., object detection) without a constant power supply.
Designing DNNs that can meet severe energy budgets has increasingly become a major design objective.
The state-of-the-art model compression algorithms adopt indirect techniques to restrict the energy consumption, such as pruning (or sparsiﬁcation)
(He et al., 2018; Han et al., 2015a;
Liu et al., 2015;
Zhou et al., 2016; Li et al., 2016; Wen et al., 2016) and quantization (Gong et al., 2014; Wu et al., 2016; Han et al., 2015a; Courbariaux et al., 2015; Rastegari et al., 2016).
These techniques are agonistic to energy consumption; rather they are designed to reduce the amount of computations and the amount of model parameters in a DNN, which do not truly reﬂect the energy consumption of a DNN.
As a result, these indirect approaches only indirectly reduce the total energy consumption.
Recently, Energy-Aware Pruning (EAP) (Yang et al., 2017) proposes a more direct manner to reduce the energy consumption of DNN inferences by guiding weight pruning using DNN energy estimation, which achieves higher energy savings compared to the indirect techniques.
However, a fundamental limitation of all existing methods is that they do not provide quantitative energy guarantees , i.e., ensuring that the energy consumption is below a user-speciﬁed energy budget.
In this paper, we aspire to answer the following key question: how to design DNN models that satisfy a given energy budget while maximizing the accuracy?
This work provides a solution to this question through an end-to-end training framework.
By end-to-end, we refer to an approach that directly meets the energy budget without relying heuristics such as selectively restoring pruned weights and layer by layer ﬁne-tuning (Han et al., 2015b; Yang et al., 2017).
These heuristics are effective in practice but also have many hyper-parameters that must be carefully tuned.
Our learning algorithm directly trains a DNN model that meets a given energy budget while maximizing model energy without incremental hyper-parameter tuning.
The key idea is to formulate the
DNN training process as an optimization problem in which the energy budget imposes a previously unconsidered optimization constraint.
We integrate the quantitative DNN energy estimation into the
DNN training process to assist the constrained optimization.
In this way, a DNN model, once is trained, by design meets the energy budget while maximizing the accuracy.
Without losing generality, we model the DNN energy consumption after the popular systolic array hardware architecture (Kung, 1982) that is increasingly adopted in today’s DNN hardware chips such as Google’s Tensor Processing Unit (TPU) (Jouppi et al., 2017), NVidia’s Tensor Cores, and ARM’s
ML Processor.
The systolic array architecture embodies key design principles of DNN hardware that is already available in today’s consumer devices.
We speciﬁcally focus on pruning, i.e., controlling the DNN sparsity, as the main energy reduction technique.
Overall, the energy model models the
DNN inference energy as a function of the sparsity of the layer parameters and the layer input.
Given the DNN energy estimation, we formulate DNN training as an optimization problem that minimizes the accuracy loss under the constraint of a certain energy budget.
The key difference between our optimization formulation and the formulation in a conventional DNN training is two-fold.
First, our optimization problem considers the energy constraint, which is not present in conventional training.
Second, layer inputs are non-trainable parameters in conventional DNN training since they depend on the initial network input.
We introduce a new concept, called input mask, that enables the input sparsity to be controlled by a trainable parameter, and thus increases the energy reduction opportunities.
This lets us further reduce energy in scenarios with known input data pattern.
We propose an iterative algorithm to solve the above optimization problem.
A key step in optimization is the projection operation onto the energy constraint, i.e., ﬁnding a model which is closest to the given (dense) model and satisﬁes the energy constraint.
We prove that this projection can be casted into a 0 / 1 knapsack problem and show that it can be solved very efﬁciently.
Evaluation results show that our proposed training framework can achieve higher accuracy under the same or lower energy compared to the state-of-the-art energy-saving methods.
In summary, we make the following contributions in this paper:
• To the best of our knowledge, this is the ﬁrst end-to-end DNN training framework that provides quantitative energy guarantees; • We propose a quantitative model to estimate the energy consumption of DNN inference on TPU-like hardware.
The model can be extended to model other forms of DNN hardware;
• We formulate a new optimization problem for energy-constrained DNN training and present a general optimization algorithm that solves the problem.
R ELATED W ORK Energy-Agnostic Optimizations
Most existing DNN optimizations indirectly optimize DNN energy through reducing the model complexity.
They are agonistic to the energy consumption, and therefore cannot provide any quantitative energy guarantees.
Pruning, otherwise known as sparsiﬁcation, is perhaps the most widely used technique to reduce
DNN model complexity by reducing computation as well as hardware memory access.
It is based on the intuition that DNN model parameters that have low-magnitude have little impact on the ﬁnal prediction, and thus can be zeroed-out.
The classic magnitude-based pruning (Han et al., 2015b) removes weights whose magnitudes are lower than a threshold.
Subsequent work guides pruning using special structures (Liu et al., 2015;
Zhou et al., 2016; Li et al., 2016; Wen et al., 2016), such as removing an entire channel, to better retain accuracy after pruning.
Quantization reduces the number of bits used to encode model parameters, and thus reducing computation energy and data access energy (Gong et al., 2014; Wu et al., 2016; Han et al., 2015a).
The extreme case of quantization is using 1-bit to represent model parameters (Courbariaux et al., 2015; Rastegari et al., 2016).
Such binary quantization methods are usually trained from scratch instead of quantizing a pre-trained DNN.
Energy-Aware Optimizations
Recently, energy-aware pruning (EAP)
(Yang et al., 2017) proposes to use a quantitative energy model to guide model pruning.
Different from pure magnitude-based pruning methods, EAP selectively prunes the DNN layer that contributes the most to the total energy consumption.
It then applies a sequence of ﬁne-tuning techniques to retain model accuracy.
The pruning step and ﬁne-tuning step are alternated until the accuracy loss exceeds a given threshold.
Although EAP a promising ﬁrst-step toward energy-aware optimizations, its key limitation is that it does not provide quantitative energy guarantees because it does not explicitly consider energy budget as a constraint.
Our work integrates the energy budget as an optimization constraint in model training.
Latency-Guaranteed Compression Lately, model compression research has started providing guarantees in execution (inference) latency, which theoretically could be extended to providing energy guarantees as well.
However, these methods are primarily search-based through either reinforcement learning (He et al., 2018) or greedy-search (Yang et al., 2018).
They search the sparsity setting for every single layer to meet the given budget.
Thus, they may require a large number of trials to achieve a good performance, and may not ensure that the resulting model accuracy is maximized.
M ODELING DNN I NFERENCE E NERGY C ONSUMPTION
This section introduces the model of estimating energy consumption of a single DNN inference.
We consider the widely-used feed-forward DNNs.
Note that our proposed methodology can be easily extended to other network architectures as well.
In this section, we ﬁrst provide an overview of our energy modeling methodology (Section 3.1).
We then present the detailed per-layer energy modeling (Section 3.2 and Section 3.3), which allow us to then derive the overall DNN energy consumption (Section 3.4).
Our energy modeling results are validated against the industry-strength DNN hardware simulator ScaleSim (Samajdar et al., 2018).
DNN model sparsity (via pruning) is well recognized to signiﬁcantly affect the execution efﬁciency and thus affect the energy consumption of a DNN model (He et al., 2018;
Yang et al., 2017; Han et al., 2015a; Liu et al., 2015; Zhou et al., 2016).
We thus use pruning as the mechanism to reduce energy consumption 1 .
Note, however, that model sparsity is not the end goal of our paper; rather we focus on reducing the energy consumption directly.
Many dedicated DNN hardware chips (a.k.a., Neural
Processing Units, NPUs) (Jouppi et al., 2017; Chen et al., 2016; Han et al., 2016; Parashar et al., 2017) have been developed to directly beneﬁt from model sparsity, and are already widely available in today’s consumer devices such as Apple iPhoneX, Huawei Mate 10, and Microsoft HoloLens.
Our paper focuses on this general class of popular, widely-used DNN chips.
3. E NERGY M ODELING O VERVIEW
A DNN typically consists of a sequence of convolution (CONV) layers and fully connected (FC) layers interleaved with a few other layer types such as Rectiﬁed Linear Unit (ReLU) and batch normalization.
We focus mainly on modeling the energy consumption of the CONV and FC layers.
This is because CONV and FC layers comprise more than 90% of the total execution time during a DNN inference (Chen et al., 2016) and are the major energy consumers (Han et al., 2015a;
Yang et al., 2017).
Energy consumed by other layer types is insigniﬁcant and can be taken away from the energy budget as a constant factor.
[Figure]
A DNN inference’s energy consumption is tied to the underlying hardware that performs the inference.
In particular, we assume a systolic-array-based DNN hardware architecture.
Systolic array (Kung, 1982) has long been know as an effective approach for matrix multiplication.
Many DNN hardware architectures adopt the systolic array, most notably the Google Tensor Processing Unit (TPU) (Jouppi et al., 2017), Nvidia’s Tensor Cores in their most recent Volta GPUs, and ARM’s ML Processor.
Targeting systolic-array-based DNN hardware ensures that our approach has a wide applicability.
However, our modeling and training strategies can generally be applied to other DNN architectures.
Figure 1 shows the overall hardware architecture.
The systolic array comprises of several compute units that perform the Multiply-and-Accumulate (MAC) operation, which conducts the following computation: a ← a +( b × c ) , where b and c are the two scalar inputs and a is the scalar intermediate result called “partial sum.”
MAC operation is the building block for matrix multiplication.
The MAC units are organized in a 2-D fashion.
The data is fed from the edges, both horizontally and vertically, which then propagate to the MAC units within the same row and columns.
We decompose the energy cost into two parts: computation energy E comp and data access energy E data .
E comp denotes the energy consumed by computation units, and E data denotes the energy consumed when accessing data from the hardware memory.
Since we mainly use pruning as the energy reduction technique, we now model how E comp and E data are affected by DNN sparsity.
3. E NERGY C ONSUMPTION FOR C OMPUTATION
CONV layers perform convolution and FC layer perform matrix-vector multiplication.
Both operations can be generalized to matrix-matrix multiplication, which involves only the MAC operation (Chetlur et al., 2014; Jouppi et al., 2017).
Figure 1 illustrates how a matrix-matrix multiplication is carried out on the systolic array hardware.
Given X and W , the systolic array computes XW by passing each row of X to each row in the systolic array and passing each column of W to each column in the systolic array.
If the width of the systolic array, denoted by s w , is less than the width of
W , the hardware will fold W column-wise in strides of s w .
Similarly, if the height of X is greater than the height of the systolic array ( s h ), X is folded row-size in strides of s h .
Figure 1 illustrates a 2 × 2 systolic array multiplying two 4 × 4 matrices.
Both matrices are folded twice in strides of 2.
Critically, if either inputs of a MAC operation is zero, we can skip the MAC operation entirely and thus save the computation energy.
At a high-level, the total computation energy, E comp , can be modeled as e MAC N MAC , where e MAC denotes the energy consumption of one MAC operation whereas N
MAC denotes the total number of MAC operations that are actually performed.
The challenge is to identify N MAC for CONV and FC layers, which we discuss below.
Fully connected layer Let X ( v ) ∈ R 1 × c be the input vector and W ( v )
∈ R c
×
d be the weight matrix of the FC layer v .
The FC layer performs matrix-vector multiplication X ( v ) W ( v ) .
The number of
MAC operations N MAC is sum ( supp ( X ) supp ( W )) , where supp ( T ) returns a binary tensor indicating the nonzero positions of tensor T .
So the computation energy for a fully connected layer v :
[Equation]
where the equality is reached when the input is dense.
Convolution layer The CONV layer performs the convolution operation between a 4-D weight (also referred to as kernel or ﬁlter) tensor and a 3-D input tensor.
Let W ( u ) ∈
R
d × c × r × r be the weight tensor, where d , c , and r are tensor dimension parameters.
Let X ( u ) ∈
R
c
× h × w be the input tensor, where h and w are the input height and width.
The convolution operation in the CONV layer u generates a 3-dimensional tensor:
[Equation]
where x, y indicate the position of the output tensor, which has height h (cid:48)
= (cid:98) ( h + 2 p − r ) /s (cid:99) +and width w (cid:48)
= (cid:98) ( w + 2 p − r ) /s
(cid:99)
+ 1 ( p is the convolution padding and s is the convolution stride).
Tensor convolution (2) can be seen as a special matrix-matrix multiplication (Chellapilla et al., 2006;
Chetlur et al., 2014).
Speciﬁcally, we would unfold the tensor X ( u ) to a matrix
¯ X ( u ) ∈
R h (cid:48) w (cid:48)
× cr 2 , and unfold the tensor W ( u ) to a matrix ¯ W ( u ) ∈ R cr 2 × d .
¯ X ( u ) and ¯ W ( u ) are then multiplied together in the systolic array to compute the equivalent convolution result between X ( u ) and W ( u ) .
Nonzero elements in X ( u ) and W ( u ) incur actual MAC operations.
Thus, N MAC = sum ( supp ( X ( u ) ) ∗
[-(cid:107) W ( u ) (cid:107) 
-][Equation]
0 (the equality means the input is dense), resulting in the following computation energy of a CONV layer u :
[Equation]
3. E NERGY C ONSUMPTION FOR D ATA A CCESS
Accessing data happens in every layer.
The challenge in modeling the data access energy is that modern hardware is equipped with a multi-level memory hierarchy in order to improve speed and save energy (Hennessy & Patterson, 2011).
Speciﬁcally, the data is originally stored in a large memory, which is slow and energy-hungry.
When the data is needed to perform certain computation, the hardware will load it from the large memory into a smaller memory that is faster and consume less energy.
If the data is reused often, it will mostly live in the small memory.
Thus, such a multi-level memory hierarchy saves overall energy and improves overall speed by exploiting data reuse.
Without losing generality, we model a common, three-level memory hierarchy composed of a
Dynamic Random Access Memory (DRAM), a Cache, and a Register File (RF).
The cache is split into two halves: one for holding X (i.e., the feature map in a CONV layer and the feature vector in a
FC layer) and the other for holding W (i.e., the convolution kernel in a CONV layer and the weight matrix in an FC layer).
This is by far the most common memory hierarchy in DNN hardware such as Google’s TPU (Jouppi et al., 2017; Chen et al., 2016; Zhu et al., 2018; Han et al., 2016).
Data is always loaded from DRAM into cache, and then from cache to RFs.
Critically, if the value of the data that is being loaded is zero, the hardware can skip the data access and thereby save energy.
This is a widely-explored optimization in the architecture community such as Eyeriss (Chen et al., 2016), SCNN (Parashar et al., 2017), and Cambricon-X (Zhang et al., 2016) as well as in recent computer vision literature such as Energy-Aware Pruning (Yang et al., 2017).
To compute E data , we must calculate the number of data accesses at each memory level, i.e.,
N DRAM , N cache , N RF .
Let the unit energy costs of different memory hierarchies be e DRAM , e cache , and e
RF , respectively, the total data access energy consumption E data will be e DRAM N DRAM + e cache N cache + e
RF N RF .
We count the number of data accesses for both the weights and input, then combine them together.
The detailed derivation of data access energy is included in the Appendix.
3. T HE O VERALL E NERGY E STIMATION F ORMULATION
Let U and V be the sets of convolutional layers and fully connected layers in a DNN respectively.
The superscript ( u ) and ( v ) indicate the energy consumption of layer u ∈ U and v ∈ V , respectively.
Then the overall energy consumption of a DNN inference can be modeled by
[Equation]
where X stacks input vectors/tensors at all layers and W stacks weight matrices/tensors at all layers.
E NERGY C ONSTRAINED DNN M ODEL
Given the energy model presented in Section 3, we propose a new energy-constrained DNN model that bounds the energy consumption of a DNN’s inference.
Different from prior work on model pruning in which energy reduction is a byproduct of model sparsity, our goal is to directly bound the energy consumption of a DNN while sparsity is just used as a means to reduce energy.
This section formulates training an energy-constrained DNN as an optimization problem.
We ﬁrst formulate the optimization constraint by introducing a trainable mask variable into the energy modeling to enforce layer input sparsity.
We then deﬁne a new loss function by introducing the knowledge distillation regularizer that helps improve training convergence and reduce overﬁtting.
Controlling Input Sparsity Using Input Mask
The objective of training an energy-constrained
DNN is to minimize the accuracy loss while ensuring that the DNN inference energy is below a given budget, E budget .
Since the total energy consumption is a function of (cid:107) X ( u ) (cid:107)
and (cid:107) W ( u ) (cid:107) 0 , it is natural to think that the trainable parameters are X and W .
In reality, however, X depends on the input to the DNN (e.g., an input image to an object recognition DNN), and thus is unknown during training time.
Therefore, in conventional DNN training frameworks X is never trainable.
To include the sparsity of X in our training framework, we introduce a trainable binary mask M that is of the same shape of X , and is multiplied with X before X is fed into CONV or FC layers, or equivalently, at the end of the previous layer.
For example, if the input to a standard CONV layer is
X ( u ) , the input would now be X ( u ) (cid:12) M ( u ) , where (cid:12) denotes the element-wise multiplication.
In practice, we do not really do this multiplication but only read X ( u ) on the nonzero positions of M ( u ) .
With the trainable mask M , we can ensure that (cid:107) X ( u ) (cid:12) M ( u ) (cid:107) 0 ≤ (cid:107) M ( u ) (cid:107) 0 , and thereby bound the sparsity of the input at training time.
In this way, the optimization constraint during training becomes
E ( M, W ) ≤ E budget , where E ( M, W ) denotes the total DNN inference energy consumption, which is a function of X and W (as shown in Equation (4)), and thus a function of M and W .
Knowledge Distillation as a Regularizer Directly optimizing over the constraint would likely lead to a local optimum because the energy model is highly non-convex.
Recent works (Mishra &
Marr, 2017; Tschannen et al., 2017; Zhuang et al., 2018) notice that knowledge distillation is helpful in training compact DNN models.
To improve the training performance, we apply the knowledge distillation loss (Ba & Caruana, 2014) as a regularization to the conventional loss function.
Intuitively, the regularization uses a pre-trained dense model to guide the training of a sparse model.
Speciﬁcally, our regularized loss function is:
[Equation]
[Equation] where W dense is the original dense model, and L ( M, W ) is the original loss, e.g., cross-entropy loss for classiﬁcation task. φ ( X ; W ) is the network’s output (we use the output before the last activation layer as in Ba & Caruana (2014)), | φ ( · ; W ) | is the network output dimensionality and 0 ≤ λ ≤ 1 is a hyper parameter similar to other standard regularizations.
Thus, training an energy-constrained DNN model is formulated as an optimization problem:
[Equation]
O PTIMIZATION
This section introduces an algorithm to solve the optimization problem formulated in (6).
The overall algorithm is shown in Algorithm 1.
Speciﬁcally, the algorithm includes three key parts:
• Initialization by training a dense model.
That is,
[Equation]
• Fix M and optimize W via approximately solving (using W dense initialization):
[Equation]
• Fix W and optimize M by approximately solving :
[Equation]
After the initialization step (Line 1 in Algorithm 1), the training algorithm iteratively alternates between the second (Line 3-6 in Algorithm 1) and the third step (Line 8-13 in Algorithm 1) while gradually reducing the sparsity constraint q (Line 14 in Algorithm 1) until the training accuracy converges.
Note that Equation (7) is the classic DNN training process, and solving Equation (9) involves only the well-known L 0 norm projection P (cid:107) M (cid:107) 0 ≤ q ( Q ) :
= arg min (cid:107) M (cid:107) 0 ≤ q (cid:107) M − Q (cid:107) 2 .
We thus focus on how Equation (8) is solved.
Optimizing Weight Matrix W
To solve (8), one can use either projected gradient descent or projected stochastic gradient descent.
The key difﬁculty in optimization lies on the projection step
[Equation]
[Equation] where Z could be W − η ∇ W ¯ L ( W, M ) or replacing ∇ W ¯ L ( W, M ) by a stochastic gradient ˆ ∇ W ¯ L ( W, M ) .
To solve the projection step, let us take a closer look at the constraint Equation (4).
We rearrange the energy constraint Ω( E budget ) into the following form with respect to W :
[Equation]
where W stacks all the variable { W ( u ) } u ∈ U ∪ V , and α ( u ) 1 , α ( u ) 2 , α ( u ) 3 , α ( u ) 4 and k are properly deﬁned nonnegative constants.
Note that α ( u )≤ α ( u ) 2 and k is a positive integer.
Theorem 1 casts the energy constrained projection problem to a 0 / 1 knapsack problem.
The proof is included in the Appendix.
Theorem 1.
The projection problem in (10) is equivalent to the following 0 / 1 knapsack problem:
[Equation]
where Z stacks all the variables { Z ( u ) } u ∈ U ∪ V , A and ξ are of the same shape as Z , and the j -th element of A ( u ) for any u ∈ U ∪ V is deﬁned by
[Equation]
is among the top k elements of Z ( u ) in term of magnitude
[Equation]
The optimal solution of (10) is Z (cid:12) ξ ∗ , where ξ ∗ is the optimal solution to the knapsack problem (12) .
The knapsack problem is NP hard.
But it is possible to ﬁnd approximate solution efﬁciently.
There exists an approximate algorithm (Chan, 2018) that can ﬁnd an (cid:15) − accurate solution in O ( n log(1 /(cid:15) )
+ (cid:15) − 2 . 4 ) computational complexity.
However, due to some special structure in our problem, there exists an algorithm that can ﬁnd an an (cid:15) − accurate solution much faster.
In the Appendix, we show that an (1 + (cid:15) ) -approximate solution of problem (10) can be obtained in ˜ O
( n + 1 (cid:15) 2 ) time complexity ( ˜ O omits logarithm), though the implementation of the algorithm is complicated.
Here we propose an efﬁcient approximate algorithm based on the “proﬁt density.”
The proﬁt density of item j is deﬁned as Z 2 j /A j .
We sort all items based on the “proﬁt density” and iteratively select a group of largest items until the constraint boundary is reached.
The detailed algorithm description is shown in the Appendix (Algorithm 2).
This greedy approximation algorithm also admits nice property as shown in Theorem 2.
Theorem 2.
For the projection problem (10) , the approximate solution W (cid:48)(cid:48)
∈ Ω( E budget ) to the greedy approximation algorithm admits
[Equation]
where max( A ) is the maximal element of A , which is a nonnegative matrix deﬁned in (13) ; Top k ( · ) returns the k -th largest element of · ; (cid:11) denotes the element-wise division.
gcd( · ) is the largest positive rational number that divides every argument, e.g., gcd(0 , 1 / 3 , 2 / 3)
= 1 / 3 .
In (14) , gcd( A ) denotes the greatest common divisor of all elements in A 2 , and R ( W (cid:48)(cid:48) ) denotes the remaining budget
[Equation]
The formal proof is in the Appendix.
W (cid:48)(cid:48) is the optimal projection solution to (10) if either of the following conditions holds:
( The remaining budget is 0 . )
It means that the greedy Algorithm 2 runs out of budget; 2.
( The matrix A satisﬁes max( A )
=
gcd( A ) . )
It implies that all elements in A have the same value.
In other words, the weights for all items are identical.
E VALUATION
The evaluations are performed on ImageNet (Deng et al., 2009), MNIST, and MS-Celeb-1M (Guo et al., 2016) datasets.
For the MS-Celeb-1M, we follow the baseline setting reported in the original paper (Guo et al., 2016), which selects 500 people who have the most face images.
We randomly sample 20% images as the validation set.
We use both classic DNNs, including AlexNet (Krizhevsky et al., 2012) and LeNet-5 (LeCun et al., 1998), as well as recently proposed SqueezeNet (Iandola et al., 2016) and MobileNetV2 (Sandler et al., 2018).
We compare our method mainly with ﬁve state-of-art pruning methods: magnitude-based pruning (MP) (Han et al., 2015b;a), structured sparsity learning (SSL)
(Wen et al., 2016), structured bayesian pruning (SBP) (Neklyudov et al., 2017), bayesian compression (BC) (Louizos et al., 2017) and energy-aware pruning (EAP)
(Yang et al., 2017).
Filter pruning methods (Li et al., 2016; He et al., 2017) require a sparsity ratio to be set for each layer, and these sparsity hyper-parameters will determine the energy cost of the DNN. Considering manually setting all these hyper-parameters in energy constrained compression is not trivial, we directly compare against NetAdapt (Yang et al., 2018) which automatically searches such sparsity ratios and use ﬁlter pruning to compress DNN models.
We implement an energy-constrained version of NetAdapt, which is originally designed to restrict the inference latency.
Note that MobileNetv2 and SqueezeNet have special structures (e.g. residual block) that are not fully supported by NetAdapt.
Thus, we show the results of NetAdapt only for AlexNet and LeNet-5.
[Figure]
Hyper-parameters In the experiment, we observe that knowledge distillation can improve the performance of MP and SSL, so we apply knowledge distillation to all methods including the baseline for a fair comparison. The results of removing knowledge distillation on MP and SSL are included in the Appendix.
We choose the distillation weight λ = 0 .
EAP proposes an alternative way to solve the overﬁtting issue, so we directly use their results.
For all the DNNs, we turn off the dropout layers since we ﬁnd the knowledge distillation regularization will perform better.
In all the experiments, we choose ∆ q = 0 .
1 | M | where | M | is the number of all mask elements.
For optimizing W , we use a pre-trained dense initialization and update W by SGD with the learning rate η 1 = 0 .
001 and weight decay 10 − 4 .
For optimizing input mask parameters M , we use the Adam optimizer (Kingma & Ba, 2014) with η 2 = 0 . 0001 and weight decay 10 − 5 (MNIST)/ 10 − 6 (MS-Celeb-1M).
To stabilize the training process, we exponentially decay the energy budget to the target budget, and also use this trick in MP training (i.e. decaying the sparsity budget) for fair comparisons.
6. I MAGE N ET
We set an energy budget to be less than the minimal energy consumption among the three baseline methods.
We use the same performance metric (i.e. top-5 test accuracy) and hardware parameters, i.e., e MAC , e DRAM , e cache , e RF , s h , s w , k W , k X , as described in the EAP paper (Yang et al., 2017).
We initialize all the DNNs by a pre-trained dense model, which is also used to set up the knowledge distillation regularization.
The top-5 test accuracies on the dense models are 79 .
1% (AlexNet), 80 .
5% (SqueezeNet), and 90 .
5% (MobileNetV2).
We use batch size 128 and train all the methods with 30 epochs.
For SSL and NetAdapt, we apply 20 additional epochs to achieve comparable results.
We implement the projection operation P Ω( E budget ) on GPU, and it takes < 0 .
2 s to perform it in our experiments.
The detailed wall-clock result is included in the Appendix.
Table 1 shows the top-5 test accuracy drop and energy consumption of various methods compared to the dense model.
Our training framework consistently achieves a higher accuracy with a lower
[Table]
energy consumption under the same energy budget.
For instance on AlexNet, under a smaller energy budget ( 26% < 27% ), our method achieves lower accuracy drop over EAP (0.5% vs. 0.8%).
The advantage is also evident in SqueezeNet and MobileNetV2 that are already light-weight by design.
EAP does not report data on MobileNetV2.
We observe that weight sparsity is not a good proxy for energy consumption.
Our method achieves lower energy consumption despite having higher density.
Figure 2 comprehensively compares our method with prior work.
Solid markers represent DNNs trained from our framework under different energy budgets ( x -axis).
Empty markers represent DNNs produced from previous techniques.
DNNs trained by our method have lower energies with higher accuracies (i [Table] e., solid markers are closer to the bottom-left corner than empty markers).
For instance on SqueezeNet, our most energy-consuming DNN still reduces energy by 23% while improves accuracy by 0.2% compared to EAP.
6. MNIST AND MS-C ELEB -1M
[Figure]
[Figure] represent datasets where inputs have regular patterns that are amenable to input masking.
For instance, MS-Celeb-1M is a face image dataset and we use its aligned face images where most of the facial features are located in the center of an image.
In such scenarios, training input masks lets us control the sparsity of the layer inputs and thus further reduce energy than merely pruning model parameters as in conventional methods.
We do not claim that applying input mask is a general technique; rather, we demonstrate its effectiveness when applicable.
We compare our method with MP and SSL using
LeNet-5 and MobileNetV2 for these two datasets, respectively.
The pre-trained dense LeNet-5 has 99 .
3% top-1 test accuracy on MNIST, and the dense MobileNetV2 has 65 . 6% top-5 test accuracy on MS-Celeb-1M.
EAP does not report data on these two datasets.
Similar to the evaluation on
ImageNet, we set the energy budget to be lower than the energy consumptions of MP and SSL.
We use batch size 32 on MNIST and 128 on MS-Celeb-1M, and number of epochs is set the same as the
ImageNet experiments.
Table 2 compares the energy consumption and accuracy drop.
Our method consistently achieves higher accuracy with lower energy under the same or even smaller energy budget.
We visualize the sparsity of the learned input masks in Figure 3.
and MS-Celeb-1M.
[Table]
C ONCLUSION
This paper demonstrates that it is possible to train DNNs with quantitative energy guarantees in an end-to-end fashion.
The enabler is an energy model that relates the DNN inference energy to the
DNN parameters.
Leveraging the energy model, we augment the conventional DNN training with an energy-constrained optimization process, which minimizes the accuracy loss under the constraint of a given energy budget.
Using an efﬁcient algorithm, our training framework generates DNNs with higher accuracies under the same or lower energy budgets compared to prior art.
A PPENDICES D ETAIL OF E NERGY C ONSUMPTION FOR D ATA A CCESS F ULLY C ONNECTED L AYER
To multiply X ( v ) ∈ R c and W ( v ) ∈ R c × d , each nonzero element of W ( v ) is used once but loaded three times, once each from DRAM, cache and RF, respectively. Thus, the number of DRAM, cache, and RF accesses for weight matrix W ( v ) is: [Equation]
[Equation]
Input X ( v ) is fed into the systolic array (cid:100)
d/s w (cid:101) times, where s w denotes the the systolic array width.
Thus, the number of cache accesses for X ( v ) is:
[Equation]
Let k X be the cache size for input X ( v ) .
If k X is less than (cid:107) X ( v ) (cid:107) 0 , there are (cid:107) X
( v ) (cid:107)
− k X elements that must be reloaded from DRAM every time.
The rest k X elements need to load from only DRAM once as they will always reside in low-level memories.
Thus, there are (cid:100) d/s w (cid:101)
( (cid:107) X
( v ) (cid:107)
− k X )
+
k X
DRAM accesses for X ( v ) .
In addition, the output vector of the FC layer (result of X ( v ) W ( v ) ) needs to be written back to DRAM, which further incurs d DRAM accesses.
Thus, The total number of
DRAM accesses to retrieve X ( v ) is:
[Equation]
Each input element is loaded from RF once for each MAC operation, and there are two RF accesses incurred by accumulation for each MAC operation (one read and one write).
Thus, the total number of RF accesses related to X ( v ) is:
[Equation]
In summary, the data access energy of a fully connected layer v is expressed as follows, in which each component follows the derivations in Equation (15) through Equation (18):
[Equation]
C ONVOLUTION L AYER
Similar to a FC layer, the data access energy of a CONV layer u is modeled as:
[Equation]
The notations are the same as in FC layer.
We now show how the different components are modeled. To convolve W ( u ) ∈ R d × c × r × r with X ( u ) ∈ R c × h × w , each nonzero element in the weight tensor W ( u ) is fed into the systolic array (cid:100) h (cid:48) w (cid:48)
/s h (cid:101) times, where s h denotes the height of the systolic array and h (cid:48) and w (cid:48) are dimension parameters of X ( u ) .
Thus,
[Equation]
Similar to the FC layer, the number of RF accesses for W ( u ) during all the MAC operations is:
[Equation]
[Equation]
Let k W be the cache size for the weight matrix W ( u ) .
If (cid:107) W ( u ) (cid:107) 0 > k W , there are k W nonzero elements of W ( u ) that would be accessed from DRAM only once as they would reside in the cache, and the rest (cid:107) W ( u ) (cid:107)
− k W elements would be accessed from DRAM by (cid:100) h (cid:48) w (cid:48)
/s h (cid:101) times.
Thus,
[Equation]
Let k X be the cache size for input X ( u ) .
If every nonzero element in X ( u ) is loaded from DRAM to cache only once, N inputDRAM would simply be (cid:107) X ( u ) (cid:107) 0 .
In practice, however, the cache size k X is much smaller than (cid:107) X ( u ) (cid:107) 0 .
Therefore, some portion of X ( u ) would need to be re-loaded.
To calculate the amount of re-loaded DRAM access, we observe that in real hardware X ( u ) is loaded from DRAM to the cache at a row-granularity. When the input X ( u ) is dense, there are at least cw elements loaded at once.
In this way, the cache would ﬁrst load (cid:98) k
X / ( cw ) (cid:99) rows from DRAM, and after the convolutions related to these rows have ﬁnished, the cache would load the next (cid:98)
k X / ( cw ) (cid:99) rows in X ( u ) for further processing.
The rows loaded in the above two rounds have overlaps due to the natural of the convolution operation.
The number of overlaps R overlap is (cid:100) h/ ( (cid:98)
k
X /cw (cid:99)
− r + s ) (cid:101) − 1 , and each overlap has cw ( r − s ) elements.
Thus, R overlap × cw ( r − s ) elements would need to be reloaded from DRAM.
Finally, storing the outputs of the convolution incurs an additional dh (cid:48) w (cid:48) DRAM writes.
Summing the different parts together, the upper bound ( X ( u ) is dense) number of DRAM accesses for X ( u ) is:
[Equation]
When the input X ( u ) is not dense, we can still count the exact number of elements in the overlaps
N overlap of the consecutive loading rounds, so we have:
[Equation]
[Equation]
Every nonzero element in the unfolded input ¯ X ( u ) would be fed into the systolic array (cid:100)
d/s w (cid:101) times (for grouped convolution, this number is divided by the number of groups).
Each MAC operation introduces 2 RF accesses.
Thus,
[Equation]
P ROOF TO T HEOREM
Proof.
First, it is easy to see that (10) is equivalent to the following problem
[Equation]
Note that if the optimal solution to problem (27) is ¯ ξ , the solution to problem (10) can be obtained by
Z (cid:12) ¯ ξ ; given the solution to (10), the solution to (27) can be obtained similarly.
Therefore, we only need to prove that (27) is equivalent to (12).
Meeting the following two conditions guarantees that (27) and (12) are equivalent since they have identical objective functions:
Any optimal solution of problem (27) is in the constraint set of problem (12); 2.
Any optimal solution of problem (12) is in the constraint set of problem (27).
Let us prove the ﬁrst condition.
Let ˆ ξ be the optimal solution to (27).
Then for any u ∈ U ∪ V , the elements of Z ( u ) selected by ˆ ξ ( u ) are the largest (in terms of magnitude) (cid:107) ˆ ξ ( u ) (cid:107) 0 elements of Z ( u ) ; otherwise there would exist at least one element that can be replaced by another element with a larger magnitude, which would increase the objective value in (27).
Since ˆ ξ ∈ Ω( E budget ) , according to the deﬁnition of A in (13), ˆ ξ satisﬁes the constraint of (12).
Let us now prove the second condition.
The deﬁnition of A in (13) show that there could at most be two different A ( u ) values for each element u , and the largest k elements in Z ( u ) always have the smaller value, i.e., α ( u ) 1 + α ( u ) 3 .
Let ¯ ξ be the optimal solution to the knapsack problem (12).
For any u ∈ U ∪ V , the elements selected by ¯ ξ ( u ) are also the largest elements in Z ( u ) in terms of magnitude; otherwise there would exist an element Z ( u ) j that has a larger magnitude but corresponds to a smaller
A ( u ) j
((13) shows that A ( u ) i ≥ A ( u ) j when | Z ( u ) i | ≤ | Z ( u ) j | ).
This would contradict the fact that ¯ ξ is optimal.
In addition, ¯ ξ meets the constraint in problem (12).
Therefore, ¯ ξ ∈ Ω( E budget ) .
It completes the proof.
A N (1 + (cid:15) )
- APPROXIMATE SOLUTION FOR PROBLEM
Theorem 3. For the projection problem (10) , there exists an efﬁcient approximation algorithm that has a computational complexity of O (cid:16)(cid:16) n + ( | U | + | V | ) 3 (cid:15)(cid:17) log n max [Equation]
A ) min( A + ) (cid:17) and generates a solution
W (cid:48)
∈ Ω( E budget ) that admits
[Equation]
[Equation]
where min( A + ) is the minimum of the positive elements in A .
| U | and | V | denote the number of CONV and FC layers, respectively.
They are very small numbers that can be treated as constants here.
Thus, the computational complexity for our problem is reduced to ˜ O ( n + 1 (cid:15) 2 ) , where ˜ O omits the logarithm term.
In the following, we will prove this theorem by construction.
P ROBLEM F ORMULATION
Deﬁnition 1.
Inverted knapsack problem.
Given n objects I := { ( v i , w i ) } ni =1 each with weight w i > 0 , and value v i ≥ 0 , deﬁne h I ( x ) to be the smallest weight budget to have the total value x :
[Equation]
We are more interested in the case that the weights of n objects are in m clusters, i.e. there are only m distinct weights,
[Equation]
In our case, m is proportional to the number of layers in DNN, and n is the number of all the learnable weights in W , so m (cid:28) n .
Deﬁnition 2.
Inverse of step function.
The inverse of the step function f is deﬁned as the maximal x having the function value y :
[Equation]
Observation The inverse of the step function h − 1 I ( y ) is just the maximal value we can get given the weight budget, i.e. the original knapsack problem:
[Equation]
Observation Given a step function with l breakpoints, its inverse can be generated with O ( l ) time complexity, and vice versa.
Thus, given the step function of h I in (29) which has l breakpoints, we can get h − 1 I (i.e. the original knapsack problem) within O ( l ) time complexity.
Deﬁnition 3.
w -uniform.
Step function f is w -uniform if the ranges of f is from −∞ , 0 , w, 2 w, ..., lw .
Observation If all the objects in I have the same weight w , i.e. m = 1 , then the function h
I ( x ) is nondecreasing and w -uniform.
Moreover, its breakpoints are: (
[Equation]
if the objects’ indices follows the decreasing order in terms of the values, i.e. v 1 ≥ v 2 ≥ ... ≥ v n .
Thus we can get all possible function values of h
I ( x ) :
[Equation]
Deﬁnition 4.
(min, +)-convolution.
For functions f, g , the (min, +)-convolution is:
[Equation]
Observation If object sets I 1 ∩ I 2 = ∅ , then
[Equation]
Observation The inverse of (min, +)-convolution between w -uniform function f and w -uniform function g is the (max, +)-convolution between f − 1 and g − 1 :
[Equation]
Lemma 4.
For any f and g nonnegative step functions, given an arbitrary number b , we always have
[Equation]
Proof.
Given any x , let z ∈
Arg min
x (cid:48)
f ( x (cid:48) )
+ g ( x − x (cid:48) ) and ¯ z ∈ Arg min
x (cid:48) min( f ( x (cid:48) ) , b ) + min( g ( x − x (cid:48) ) , b ) , so we have ( f ⊕ g )( x ) =
f ( z ) + g ( x − z ) and (min { f, b } ⊕ min { g, b } )( x ) =
min( f (¯ z ) , b ) + min( g ( x − ¯ z ) , b ) .
Consider the following cases:
( f ⊕ g )
( x ) ≥ b .
In this case, we claim that (min { f, b } ⊕ min { g, b } )( x ) ≥ b .
We prove it by contradiction.
Suppose (min { f, b } ⊕ min { g, b } )( x ) < b which implies min( f (¯ z ) , b ) + min( g ( x − ¯ z ) , b ) < b .
Because both f and g are nonnegative, we have f (¯ z ) <
b and g ( x − ¯ z ) < b which imply min( f (¯ z ) , b ) + min( g ( x − ¯ z ) , b ) =
f (¯ z )
+ g ( x − ¯ z ) < b , However, this contradicts ( f ⊕ g )( x ) ≥ b .
Therefore, we have min(( f ⊕ g )( x ) , b ) =
min((min { f, b } ⊕ min { g, b } )( x ) , b ) = b .
( f ⊕ g )( x ) < b .
In this case, we have f ( z ) <
b and g ( x − z ) < b , so min( f (¯ z ) , b ) + min( g ( x − ¯ z ) , b ) ≤ min( f ( z ) , b )+min( g ( x − z ) , b ) =
f ( z )+ g ( x − z )
=
( f ⊕ g )( x ) < b .
Since both f and g are nonnegative, we have f (¯ z ) <
b and g ( x − ¯ z ) < b which imply min( f (¯ z ) , b ) + min( g ( x − ¯ z ) , b ) =
f (¯ z )
+ g ( x − ¯ z ) ≥ ( f ⊕ g )
( x ) .
Therefore, we have min( f (¯ z ) , b ) + min
( g ( x − ¯ z ) , b )
=
f ( z ) + g ( x − z ) ⇔ (min { f, b } ⊕ min { g, b } )( x ) =
( f ⊕ g )( x ) .
E FFICIENCY OF ( MIN , +)- CONVOLUTION
Lemma 5.
Let f and g be nondecreasing w -uniform functions with O ( l ) breakpoints, the (min, +)
-convolution
f ⊕ g (having O ( l ) breakpoints) can be generated with O ( l 2 ) time complexity.
Proof.
Firstly, we compute the inverse representation of f and g , i.e. compute f − 1 and g − 1 from
Equation (30).
The inverse representation can be computed in O ( l ) time (proportional to the number of breakpoints).
From Equation (32), we can compute the inverse of f ⊕ g .
For each y ∈ { 0 , 1 w, ..., 2 lw } , function ( f ⊕ g ) − 1 ( y ) can be computed in O ( l ) time by brute force.
Thus a total O ( l 2 ) is enough to get ( f ⊕ g ) − 1 which has O ( l ) breakpoints.
We can get f ⊕ g via ( f ⊕ g ) −by
the inverse deﬁnition (30) in O ( l ) time.
Lemma 6.
Let f and g be nondecreasing step functions with l breakpoints in total, min { f ⊕ g, b } can be approximated by a step function φ b with O ( l + 1 (cid:15) 2 ) complexity and 2 (cid:15)b additive error, i.e. min { f ⊕ g, b } ≤ φ b ≤ min { f ⊕ g, b } + 2 (cid:15)b .
The resultant function φ b has O (1 /(cid:15) ) breakpoints.
Proof.
We can construct ( (cid:15)b ) -uniform functions f (cid:48) b , g (cid:48)
b which have (cid:100) 1 /(cid:15) (cid:101) breakpoints:
[Equation]
(cid:15)b.
This needs O ( l ) computational complexity.
From Lemma 5, we can compute f (cid:48)
b ⊕ g (cid:48)
b with O ( 1 (cid:15) 2 ) time complexity and φ b = min { f (cid:48)
b ⊕ g (cid:48) b , b } has O (1 /(cid:15) ) breakpoints.
Because f (cid:48) b and g (cid:48) b are constructed by ceiling min { f, b } and min { g, b } , we have: min { f, b [Equation] min { g, b
[Equation]
, which implies min
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
min { g, b } ,
[Equation]
} + [Equation] From Lemma 4, we know that min { min { f, b } ⊕ min { g, b } , b } = min { f ⊕ g, b } , so it completes the proof.
Lemma 7.
Let f 1 , f 2 , ..., f m be nondecreasing step functions with l breakpoints in total, min { f 1 ⊕ f 2 ⊕ ... ⊕ f m , b } can be approximated by a step function ψ b with O ( l + m/(cid:15) 2 ) computational complexity and m(cid:15)b additive error.
The resultant function ψ b has O (1 /(cid:15) ) breakpoints.
Proof.
From Lemma 6, we have shown the case m = 2 .
For general m > 2 , we can construct a binary tree to approximate pairs of functions, e.g., if m = 4 , we can ﬁrstly approximate ψ (1)
≈ min { f 1 ⊕ f 2 , b } , and ψ (2)
≈ min { f 3 ⊕ f 4 , b } , then approximate ψ (3)
b ≈ min { ψ (1) ⊕ ψ (2) , b } .
By this way, we construct a binary tree which has O (log m ) depth and O ( m ) nodes.
In the beginning, we use ceil function to construct m new (cid:15)b -uniform functions:
[Equation]
Then we can use the binary tree to “merge” all the m functions in pairs, via O (log m ) iterations.
Without loss of generality, we assume m is a power of two.
We can recursively merge t functions into t/ 2 functions:
Initialize t
= m , g (cid:48)
i,b = f (cid:48)
i,b , ∀ i ∈ { 1 , ..., t } .
Reassign g (cid:48)
i,b = min { g (cid:48)
i − 1 ,b ⊕ g (cid:48) 2 i,b , b } , ∀ i ∈ { 1 , ..., t/ 2 } .
According to Lemma 6, the number of break points of min { g (cid:48) {+
+} i − 1 ,b ⊕ g (cid:48) 2 i,b , b } is still O (1 /(cid:15) ) .
t =
t/ 2 .
If t > 1 , go back to Step 2.
For this binary tree, functions of the bottom leaf nodes have (cid:15)b additive error, and every (min, +)- convolution f (cid:48)
⊕ g (cid:48) will accumulate the additive error from the two functions f (cid:48) and g (cid:48) .
The root node of the binary tree will accumulate the additive errors from all the m leaf nodes, thus the resultant function ψ b ≤ min { f 1 ⊕ ... ⊕ f m , b } + m(cid:15)b .
For the computational complexity, initializing f (cid:48)
i,b takes
O ( l ) , Step 1 takes O ( l ) , Step 2 and 3 take O ( m/(cid:15) 2 ) (since there are O ( m ) nodes in the binary tree), and Step 4 takes O ( m/(cid:15) ) .
Therefore, there is O ( l + m/(cid:15) 2 ) in total.
Lemma 8.
For the inverted knapsack problem deﬁned in Equation (29), if all the n objects can be separated into m groups I 1 , ..., I m which have m distinct weights, there exists an approximate algorithm with computational complexity O (( n + m 3 (cid:15) 2 ) log n max( w ) min( w ) ) which can approximate h
I by ˜ h
I :
[Equation]
Proof.
Firstly, the step function h
I i , ∀ i ∈ { 1 , 2 , ..., m } can be easily generated within O ( n log n ) by sorting the objects of each group according to their values (in descending order).
From the deﬁnition of (min, +)-convolution, we know that h
I =
h
I 1 ⊕ ...
⊕ h
I m .
Let us construct an algorithm to approximate h
I :
Construct a set B := { 2 i n max( w ) ∈ [min( w ) , n max( w )]; i ∈ Z ≤ 0 } , where min( w ) and max( w ) are the minimum and maximum weight of items respectively, and Z ≤ 0 is the nonpositive integer set.
We have |B| =
O (log n max( w ) min( w ) ) .
For every b ∈ B , construct ψ b to approximate min { h I 1 ⊕ ...
⊕ h
I m , b } based on Lemma 7. 3.
Construct function ˜ h − 1 I :
[Equation]
[Equation]
where min( B ) is the minimum element in B .
The resultant function ˜ h − 1 I (or ˜ h I ) has at most O ( 1 (cid:15) log n max( w ) min( w ) ) breakpoints.
Compute the original function ˜ h
I from ˜ h − 1 I .
According to the above procedure, for any h I ( x ) ∈ ( b/ 2 , b ] , ˜ h
I ( x ) approximate h I ( x ) with additive error O ( m(cid:15)b ) , so we have h I ( x ) ≤ ˜ h
I ( x ) ≤ (1 + O ( m(cid:15) ))
h I ( x ) .
The algorithm takes O (( n + m/(cid:15) 2 ) log n max( w ) min( w ) ) , if we require the approximation factor to be 1 + O ( (cid:15) ) , i.e., [Equation] I
[Equation]
we need
[Equation]
time complexity.
Theorem 9.
For the knapsack problem deﬁned in Equation (12), if all the n objects have m distinct weights, there exists an approximate algorithm with computational complexity O (( n + m 3 (cid:15) 2 ) log n max( w ) min( w ) ) to generate a function ˜ h − 1 I satisfying:
[Equation]
Proof.
From Lemma 8, we have ˜ h
I ( x ) ≤ (1 + O ( (cid:15) ))
h I ( x ) which implies
[Equation]
So
[Equation]
Similarly, we can get { x | ˜
h
I ( x ) ≤ y } ⊆ { x | h I ( x ) ≤ y } from Lemma 8, so we have
[Equation]
Let I be the set of objects whose weights are nonzero elements in A and values are the corresponding elements in Z (cid:12) Z , i.e. I + =
{ ( Z 2 i , A i ) | ∀ i ∈ { 1 , 2 , ..., | A |} and A i > 0 } , ˜ ξ
+ be the solution corresponding to ˜ h − 1 I ( E budget − (cid:80) u
∈ U ∪ V α ( u ) 4 ) .
Let ˜ ξ I c + = 1 and ˜ ξ I + = ˜ ξ + , where I c + = { ( Z 2 i , A i ) | ∀ i ∈ { 1 , 2 , ..., | A |} and A i = 0 } is the complement of I + .
Here we have m ≤ 2 | U | + | V | distinct values in A .
According to Theorem 9, we have (cid:104) Z (cid:12) Z, ˜ ξ (cid:105) ≥ max ξ (cid:104) Z (cid:12) Z, ξ (cid:105) , s.t
. (cid:104) A, ξ (cid:105) ≤
[Equation]
, which implies
[Equation]
From Theorem 9, we can directly get Theorem 3.
Input : Z, A, E budget , { α ( u ) } u ∈ U ∪ V as in (12).
Result : Greedy solution ˜ ξ for problem (12).
1 Initialize b = 0 , ξ = 0 .
2 Generate the proﬁt density δ : δ j = (cid:26) ( Z j ) 2 /A
j , if A j > 0; ∞ , if A j = 0 .
3 Sort δ , let I be the indices list of the sorted δ (in descending order).
4 foreach index j ∈ I do b
= b + A j ;
If b > E budget − (cid:80) u ∈
U ∪ V α ( u ) 4 , exit loop; ξ j = 1 ; 8 end 9 ˜ ξ = ξ .
P ROOF TO T HEOREM
Proof.
From Theorem 1, we know the original projection problem (10) is equivalent to the knapsack problem (12).
So proving the inequality (14) is equivalent to proving
[Equation]
(34) and
[Equation]
(35) where ˜ ξ is the greedy solution of knapsack problem corresponding to W (cid:48)(cid:48) , and ξ ∗ is the exact solution of knapsack problem corresponding to P Ω( E budget )
( Z ) , i.e.,
[Equation]
Firstly, let us prove the inequality (35).
If we relax the values of ξ to be in the range [0 , 1] instead of { 0 , 1 } , the discrete constraint is removed so that the constraint set becomes
[Equation]
So the 0 / 1 knapsack problem is relaxed as a linear programming.
This relaxed problem is called fractional knapsack problem, and there is a greedy algorithm (Dantzig, 1957) which can exactly solve the fractional knapsack problem.
Slightly different from our Algorithm 2, the greedy algorithm for the fractional knapsack can select a fraction of the item, so its remaining budget is always zero.
The optimal objective value of the fractional knapsack is
[Equation]
Since the constraint set of the fractional knapsack problem is a superset of the constraint of the original knapsack problem, we have (cid:104) Z (cid:12) Z, ξ ∗ (cid:105) ≤
[Equation]
Z (cid:12) Z, ξ (cid:105) , that leads to inequality (34).
Secondly, we show that the inequality (35) is also true.
Since all the coefﬁcients in A are multiples of gcd( A ) , we can relax the original 0 / 1 knapsack problem in this way: for each item, split them to several items whose coefﬁcients in the constraint are gcd( A ) , and the coefﬁcients in the objective function are split equally.
For the j -th item, the coefﬁcient in the constraint is A j and the coefﬁcient in the objective function is ( Z (cid:12) Z ) j .
It will be split into A j / gcd( A ) items, and the j -th item is associated with coefﬁcient ( Z 2 j /A j ) · gcd( A ) in the objective function.
This relaxation gives us a new 0 / 1 knapsack problem, where all the items have the same coefﬁcient in the constraint, so the optimal solution is just selecting the ones with the largest coefﬁcients in the objective function.
We can formulate this problem as a relaxed knapsack problem by replacing the constraint of ξ into ξ ∈ Γ , where
[Equation]
All the elements of the solution are either 0 or 1 except the last picked one which corresponds to
Top (cid:107) ˜
ξ (cid:107)
+1 (( Z (cid:12) Z ) (cid:11) A ) .
Let the ( (cid:107) ˜
ξ (cid:107)
+ 1) -th largest element in ( Z (cid:12) Z ) (cid:11)
A be indexed by t .
We have 0 ≤ ˜ ξ t ≤ 1 − gcd( A ) /A
t .
Therefore, comparing with the original 0 / 1 knapsack problem, we have
[Equation]
[Table] Since { ξ | ξ is binary } ⊆ Γ , we have (cid:104) Z (cid:12) Z, ξ ∗ (cid:105) ≤ max ξ ∈ Γ (cid:104) Z (cid:12) Z, ξ (cid:105) . So we have the inequality (35).
S UPPLEMENTARY E XPERIMENT R ESULTS R ESULTS OF B ASELINE WITHOUT K NOWLEDGE D ISTILLATION
Table 3 shows the energy and accuracy drop results of the baseline methods MP and SSL when the knowledge distillation is removed from their loss function.
By using knowledge distillation, the results in Table 1 are much better.
Therefore, we use knowledge distillation in all the experiments when it is applicable.
[Table]
The projection operation P Ω( E budget ) in Algorithm 1 can be implemented on GPU. We measured its wall-clock time on a GPU server (CPU: Xeon E3 1231-v3, GPU: GTX 1080 Ti), and the result is shown in Table 4 (the time is averaged over 100 iterations).
[Table]