TESLA: T ASK - WISE E ARLY S TOPPING AND L OSS A GGREGATION FOR D YNAMIC N EURAL N ETWORK I NFERENCE
Anonymous authors
Paper under double-blind review
A BSTRACT
For inference operations in deep neural networks on end devices, it is desirable to deploy a single pre-trained neural network model, which can dynamically scale across a computation range without comprising accuracy.
To achieve this goal,
Incomplete Dot Product (IDP) has been proposed to use only a subset of terms in dot products during forward propagation.
However, there are some limitations, including noticeable performance degradation in operating regions with low computational costs, and essential performance limitations since IDP uses hand-crafted proﬁle coefﬁcients.
In this paper, we extend IDP by proposing new training algorithms involving a single proﬁle, which may be trainable or pre-determined, to signiﬁcantly improve the overall performance, especially in operating regions with low computational costs.
Speciﬁcally, we propose the Task-wise Early Stopping and Loss Aggregation (TESLA) algorithm, which is showed in our 3-layer multilayer perceptron on MNIST that outperforms the original IDP by 32% when only 10% of dot products terms are used and achieves 94.7% accuracy on average.
By introducing trainable proﬁle coefﬁcients, TESLA further improves the accuracy to 95.5% without specifying coefﬁcients in advance.
Besides, TESLA is applied to the VGG-16 model, which achieves 80% accuracy using only 20% of dot product terms on CIFAR-10 and also keeps 60% accuracy using only 30% of dot product terms on CIFAR-100, but the original IDP performs like a random guess in these two datasets at such low computation costs.
Finally, we visualize the learned representations at different dot product percentages by class activation map and show that, by applying TESLA, the learned representations can adapt over a wide range of operation regions.
I NTRODUCTION
Inference operations in deep neural networks on end devices, such as mobile phones, embedded sensors, IoT devices, etc., have recently received increasing attention including McMahan et al.
(2016), Howard et al.
(2017), and Teerapittayanon et al.
In such applications, it is desirable to deploy a single pre-trained CNN model on end devices, while allowing multiple operating regions to meet different power consumption, latency, and accuracy requirements.
To achieve this goal,
McDanel et al.
(2017a) proposed the incomplete dot product (IDP) operation, where only a subset of terms is used in dot products of forward propagation.
From now on, x % dot product (DP), where≤ x ≤ 100 , means the x % of terms used in dot products.
As illustrated in Figure 1, 50% DP means half of ﬁlters are used during forward propagation, and thus only half of the output channels are retained.
To reduce the deviation induced by IDP, ﬁlters are prioritized from most important to the least important by pre-determined monotonically non-increasing proﬁle coefﬁcients (say, γ 1 , ..., γ N ) during training.
Therefore, IDP can be applied at inference time with dynamically-adjusted degrees of completeness (speciﬁed by the percentage of terms being used) to trade off accuracy slightly for lowered power consumption and reduced latency.
Speciﬁcally, VGG-16 model with 50% DP achieves 70% in accuracy on the CIFAR-10 dataset compared to the standard network achieves only 35% accuracy when using the reduced channel set.
While the original IDP design seems promising, there are two limitations.
First, since the training process aims at optimizing the loss function computed using all weights of the model (100% DP),
[Figure]
Unused filters
Approximation
[Figure]
Y
Z
[Figure] Y
[Figure]
Z [Figure]
Y
Z [Figure] k
N
[Figure]
Y [Figure] X [Figure]
Y output channels
Z [Figure] Z [Figure]
channels there will be a mismatch between training and testing.
It is no surprise that inference performance signiﬁcantly decreases in low DP percentages and thus narrow the dynamic computation range.
To mitigate this problem, the original IDP design utilizes the multiple-proﬁle training strategy, where different proﬁles can be speciﬁed to focus on different dot product ranges.
In such a multipleproﬁle training process, however, certain subset of weights will be freezed in each training stage corresponding to the proﬁle being focused, and hence the overall performance may not be fully optimized.
Besides, each proﬁle needs to maintain a separate ﬁrst and last layer for adjusting to its own dot product range, resulting in additional memory overhead.
The second limitation relates to the pre-determined nature of proﬁle coefﬁcients.
While there are multiple ways to set the proﬁle based on different dynamic range requirements, the original IDP design did not focus on ﬁnding a single ”best” proﬁle that leads to the best performance.
Instead, they use multiple hand-crafted proﬁle coefﬁcients, which make the system design less general among different applications, and hence may limit the overall performance of the system.
To reduce the mismatch between training and testing performances, we propose the Task-wise Early
Stopping and Loss Aggregation (TESLA) algorithm, in which multiple loss functions are computed in different DP percentages.
By gradually aggregating these loss functions in decreasing order of DP percentages as the objective function to be optimized, TESLA signiﬁcantly improves testing performances in low DP percentages without compromising accuracy in medium to high DP percentages.
The loss functions can also be aggregated in random order of DP percentages to make a variant of TESLA, called Randomized TESLA (R-TESLA), which enables better performances under prespeciﬁed operating regions of end devices.
Moreover, we relax the constraint of pre-determined proﬁle coefﬁcients and propose the alternate training procedure (ATP) to alternately train the proﬁle coefﬁcients along with weights of the model.
By introducing trainable proﬁle coefﬁcients, customization among different applications can be achieved in a more generalized way, and the overall performance can also be further improved.
This paper has made two major contributions: (1) We propose the Task-wise Early Stopping and Loss Aggregation (TESLA) algorithm and Randomized
TESLA that can achieve dynamic scaling over a computation range in neural network inference without compromising accuracy.
(2) We also propose the Alternate Training Procedure (ATP) that can learn the proﬁle coefﬁcients and the model weights simultaneously without the need of manual conﬁguration of the proﬁle coefﬁcients.
I NCOMPLETE N EURAL N ETWORKS
Incomplete dot product (IDP) is a novel mechanism proposed by McDanel et al.
(2017a) that can be applied to a hidden layer of MLPs or deep CNN models to dynamically lower the inference costs by computing only a subset of terms in dot products during forward propagation.
By introducing a set of non-increasing coefﬁcients γ i , referred to as a proﬁle, to the channels during training, the channels will be ordered implicitly in non-increasing order from the most important to the least important.
By simply dropping out less important channels at inference time, it sufﬁces to train and deploy a single network, while still supporting different levels of computation scaling without compromising accuracy signiﬁcantly.
In this section, we brieﬂy introduce the main concepts of IDP.
2. I NCOMPLETE D OT P RODUCT O PERATION
Mathematically, for an IDP fully-connected layer with input dimension N and output dimension M , the j -th output component y j is computed as
[Equation]
for j ∈ { 1 , 2 , ..., M } , where x i is the i -th input component, w ji is the weight corresponding to the j -th output component and the i -th input component, and γ i is the i -th proﬁle coefﬁcient.
Similar expression can be derived for the IDP operation applied to a convolutional layer of CNN, as illustrated in Figure 1.
For an IDP convolutional layer with number of input channels N and number of output channels M , the j -th output channel y j is computed as
[Equation]
for j ∈ { 1 , 2 , ..., M } , where f ji ∗ x i denotes the convolution operation of the i -th input channel x i and the i -th channel of the j -th ﬁlter f ji , and γ j is the proﬁle coefﬁcient for the j -th ﬁlter.
Note that, instead of applying proﬁle coefﬁcients depthwise on each ﬁlter before convolution as is the case in the original IDP design, we multiply each γ j to each output channel after a complete convolution to produce y j .
These two approaches, however, are equivalent with negligible difference induced by the ﬁrst hidden layer.
Since the output channels y j ’s become input channels x i ’s to the next layer, applying γ j ’s to y j ’s is equivalent to applying them into the convolution operation in the next layer.
To compute IDP with a target dot product percentage, a truncated version of Eq. 1 or Eq.
2 replaces the original computation to keep only a subset of the beginning terms.
As for the case with all terms are kept, we refer to such operations as complete dot product (CDP) or 100% DP, interchangeably.
Note that in the training process in the original IDP design, only CDP is used.
2. M ULTIPLE -P ROFILE I NCOMPLETE N EURAL N ETWORKS
In the work of McDanel et al. (2017a), several proﬁle coefﬁcients are proposed and applied in a pre-determined manner.
When only a single proﬁle is applied to the model, the trade-off between computation range and performance in high DP percentage regions is also demonstrated.
Generally, the faster the proﬁle coefﬁcients decrease, the larger computation range can be achieved, at the expense of a performance degradation in high DP percentage regions.
To cover a larger computation range while maintaining the performance in high DP percentage regions, McDanel et al.
(2017a) further introduced the multiple-proﬁle incomplete neural networks (MP-IDP), where different proﬁles can be speciﬁed to focus on different DP ranges.
During training, all the speciﬁed proﬁles are applied in increasing order of their operating DP ranges.
When a proﬁle is applied, only weights corresponding to its operating DP range will be updated, leaving weights corresponding to lower DP percentages freezed since they have been trained in previous stages, and weights corresponding to higher IDP percentages set to zeros since they will be trained in later stages.
In such a stage-by-stage training process, the overall performance may not be fully optimized.
T ASK - WISE E ARLY S TOPPING AND L OSS A GGREGATION
As discussed in Section 2, in the original IDP design, CDP is used during training but IDP is applied at inference time.
This mismatch leads to a noticeable degradation in inference performance, especially in low DP percentages.
To mitigate this problem, we propose the Task-wise Early Stopping and Loss Aggregation (TESLA) algorithm.
In this paper, a task is deﬁned as the learning process that uses only a subset of weights determined by a DP percentage to learn the optimal representations.
For example, a task of 50% DP implies that the ﬁrst half of network weights are used for dot product computations and thus only these 50% of weights will be updated while conducting backpropagation.
With TESLA, we can optimize a network by tasks with different DP percentages to support various levels of computation scaling and meanwhile reduce the mismatch between training and inference.
The design of TESLA is described as follows.
3. T ASK - WISE E ARLY S TOPPING
Since tasks with different DP percentages may have different learning difﬁculties and convergence rates, we apply an early stopping mechanism to automatically adjust the learning processes of tasks.
Speciﬁcally, we keep all hyper-parameters unchanged except the numbers of epoches, which are controlled by the early stopping mechanism that halts the training process as long as the task performance has not been improved for a certain number of iterations.
For example, considering two tasks, one using 70% DP (task 1) and the other using 40% DP (task 2), we ﬁrst optimize task 1 and then switch to optimize task 2 until the optimization process of task 1 reaches the early stopping criterion. With this task-wise early stopping, we are able to optimize all the tasks sequentially, and each task initializes its model using the weights that have been optimized for all previous tasks.
However, the weights used in task 2 is exactly a subset of weights used in task 1 such that the optimization process of task 2 may contaminate the well-trained weights for task 1.
To reduce this unexpected disturbance while learning multiple tasks, some kinds of loss aggregation are needed to learn a new task without sacriﬁcing the performance of all the past tasks too much.
size ( T ) do 4: optimize L obji until meeting early stopping criteria 5:
{+L obji +1 ← α × L i +
 + (1 − α ) × L obji 6: i+} ← i +7: end while 1: Input
3: while do 4: optimize L obji until meeting early stopping criteria, which takes n epochs 5:
Sample a task called L k 6:
L obji +1 ← α × L k + (1 − α )
× L obji 7: i ← i +8: n ← n + n epcohs 9: end while
3. T ASK - WISE L OSS A GGREGATION
Task-wise loss aggregation is therefore proposed to jointly learn the shared representation for all tasks. By considering one new task at a time, we add the loss of the new task into the current objective function and optimize the aggregated objective function such that tasks are optimized incrementally and jointly .
The aggregated objective function can be expressed as
[Equation]
where α is the aggregation coefﬁcient shared by all subsequent tasks and greater α implies that we care more about the optimization of the new task.
As a consequence, the objective function in the whole learning process is an afﬁne combination of the losses of currently considered tasks. By [Figure]
[Figure] task-wise loss aggregation , these losses are aggregated incrementally and can be jointly optimized to learn a shared representation to be relevant to all tasks [Figure]
3. TESLA AND R ANDOMIZED TESLA
Task-wise Early Stopping and Loss Aggregation, TESLA.
We integrate task-wise early stopping and task-wise loss aggregation as TESLA to learn dynamic representations in neural networks.
The entire training process optimizes all tasks in an arbitrary order.
It is obvious that we have several options to order tasks in (i) increasing, (ii) decreasing, or (iii) random DP percentages.
Recall that we add a non-increasingcoefﬁcients to prioritize terms in computing dot product
[Figure]
and thus the beginning terms, e
[Figure]
g. at 10% DP, are more important than the terms at last 10% terms.
Therefore, discarding the terms from the end is less harmful to the optimized parameters, so TESLA is designed to optimize tasks in decreasing order of DP percentages.
The TESLA algorithm is shown in Algorithm 1.
Randomized Task-wise Early Stopping and Loss Aggregation, R-TESLA.
Here Randomized means that tasks are optimized in random order.
The beneﬁts of R-TESLA is two fold.
First, RTESLA provides an opportunity to turn attention back to optimize a task which had been halted before, and allows to ﬁnetune the weights, which may have been contaminated by other tasks.
Second, unlike TESLA that optimizes each task only once, R-TESLA allows each task to be optimized for multiple times, which can be speciﬁed by a customized task distribution derived from the behavioral statistics of users or the speciﬁcation of hardware design.
The detailed procedures of R-TESLA are in Algorithm 2.
3. T RAINABLE P ROFILE C OEFFICIENTS
In this section we propose to learn proﬁle coefﬁcients along with weights of the model alternately.
We initialize all coefﬁcients as one and as long as any update of proﬁle coefﬁcients, we manually clip the coefﬁcients to keep the non-increasing property.
The alternate training procedure (ATP) relaxes the constraint of ﬁxed coefﬁcients and we demonstrate the feasibility of ATP in the experiment of the MLP model on MNIST dataset in Section 4.
E XPERIMENTS
In this section, we demonstrate the effectiveness of using TESLA and R-TESLA to learn dynamic representations in MLP and CNN models, with the widely-used datasets MNIST, CIFAR-10, and CIFAR-100.
Figure 2 shows the network architectures in study.
Note that while working on CIFAR100, the last fully connected layer of Figure 2(b) is replaced by a single 100-class classiﬁer.
Here we compare TESLA and R-TESLA with the original IDP design proposed by McDanel et al.
(2017a)
over a range of dynamic scaling during inference.
All hyper-parameters and experiment settings are summarized in Table 1.
4. M ULTILAYER P ERCEPTRONS
First, we consider a 3-layer MLP model, in which the IDP operation is applied to the ﬁrst hidden layer, as shown in Figure 2(a), and evaluate on the MNIST dataset.
In this experiment, we deﬁne four tasks that optimize the model at 10%, 40%, 70%, and 100% DP, respectively.
It is noteworthy that deﬁning too many tasks in our experiment would not beneﬁt much, since there must be a large amount of shared parameters among tasks which makes the model vulnerable to overﬁtting. [Figure] [Figure] [Figure]
[Figure]
[Figure]
C l a ss i f i c a t i on A cc u [Figure] TESLA versus original IDP.
We compare TESLA and the original IDP design under various proﬁles.
Figure 3(a) shows that at 20% DP, the original IDP achieves 80%, 63% and 55% accuracy for the harmonic, all-one, and linear proﬁles respectively but TESLA keeps at least 88% accuracy for all proﬁles at 20% DP and reaches average accuracy of 94.7% using the linear proﬁle.
Most importantly, compared to the original IDP, TESLA performs only about 1% worse in accuracy at 100% DP but gains a signiﬁcant improvement from 50% to 90% in accuracy at 10% DP, which is an acceptable trade-off under practical applications.
R-TESLA versus TESLA and original IDP.
Figure 3(b) shows that R-TESLA outperforms the original IDP by a large margin and R-TESLA has comparable performance with TESLA in most cases.
R-TESLA with the harmonic proﬁle leads to the best average accuracy of 95.2% in this experiment.
By observing the optimization progress, we ﬁnd that TESLA achieves its best result after completing the last task thanks to its ordinal optimization.
On the other hand, we cannot ensure that R-TESLA can make the ultimate model retains the best dynamic representations due to its random nature.
Learn proﬁle coefﬁcients by ATP.
Here we demonstrate the feasibility of learning proﬁle coefﬁcients along with weights.
From Figure 3(c), with the help of trainable proﬁle coefﬁcients, both
TESLA and R-TESLA further boost by 1% in average, and we also observe that the learned proﬁle coefﬁcients are similar to harmonic ones as shown in Figure 3(d).
This may support why performance of harmonic coefﬁcients is the best in the original IDP.
By allowing coefﬁcients to be trainable, it is no longer to require hand-crafted proﬁle coefﬁcients and determine the best proﬁle coefﬁcients by extensive experiments.
4. C ONVOLUTIONAL N EURAL N ETWORKS
We choose the known VGG-16 model pre-trained on ImageNet to evaluate over CIFAR-10 and
CIFAR-100 dataset so that the last few dense layers are replaced by a 10-class classiﬁer and a 100class classiﬁer respectively.
Here we use the linear proﬁle coefﬁcients to compare: (i) the original IDP design, (ii) multiple-proﬁle IDP design (MP-IDP) as proposed in McDanel et al.
(2017a), (iii) TESLA, and (iv) R-TESLA.
The experimental results are summarized below.
VGG-16 on CIFAR-10.
According to Figure 4(a), the performance of original IDP by all-one coefﬁcients drops much faster than that by linear coefﬁcients. Appling all-one coefﬁcients is equivalent to using the original VGG-16 network; however, linear proﬁle coefﬁcients implicitly encourages networks to learn channel importance in order, and also brings about that pruning away later channels at different DP percentages does not hurt the performance that much.
With the use of multiple proﬁles, MP-IDP does enlarge the computational range with an increase in accuracy to 75% at 50% DP.
Furthermore, the proposed algorithms, TESLA and R-TESLA, boost the accuracy to reach 85% at 50% DP, and an even higher accuracy at 100% DP.
Following the previous experiment, here we augment another new task of 20% DP and observe whether TESLA can leverage up the performance at low DP percentages by adding a task of a low
DP percentage.
Figure 4(b) shows that TESLA and R-TESLA greatly widens the computational ranges by making accuracy reaching 75% at 20% DP. We contribute this effect to applying TESLA and R-TESLA in decreasing order of dot product percentages so that the representation learned at 100% DP drives the training of representation at 50% DP, which also makes the representation
[Figure]
much easier to be learned at 20% DP.
Compared to TESLA and R-TESLA, MP-IDP trains models in increasing order of DP percentages and thus MP-IDP doesn’t see much improvement at lower IDP percentages although adding another task at 20% DP. VGG-16 on CIFAR-100.
To sufﬁciently illustrate the effectiveness of the proposed approaches, we evaluate over a larger dataset, CIFAR-100. Figure 4(c) shows the performance of TESLA and R-TESLA still keeps around 60% accuracy from 30% to 50% DP, which outperforms either original
IDP or MP-IDP by a signiﬁcant margin, which is consistent with the result of CIFAR-10. Specifically, both TESLA and R-TESLA sacriﬁce about 4% accuracy at 100% DP but gain a great improvements of 60% accuracy in low DP percentages.
CAM visualization.
We visualize what the model sees at different DP percentages by deploying the Class Activation Mapping (CAM) technique introduced by Zhou et al.
(2016a).
A resulting CAM indicates how much each location contribute to the ﬁnal class prediction.
In this stage, we replace the max-pooling layers with average-pooling layers and train the VGG-16 network with linear coefﬁcients optimized at 20%, 50%, 100% DP.
From CAMs at different DP percentages, we found that the network is easier to make wrong prediction at 10% and 30% DP but still makes correct prediction at 20% DP as shown in Figure 5 since the representations at 20% DP are optimized.
This ﬁnding implies that we can specify any DP percentages to be optimized for satisfying custom requirements.
Compared to Figure 6, we also notice that the CAMs at 10% DP are almost the same no matter the correctness of predictions, which indicates too limited capacity to capture meaningful patterns, and thus the network at 10% DP behaves like a random guess.
R ELATED WORK
Our work is rooted from IDP proposed by McDanel et al.
(2017a), which, in addition to MLPs and regular CNNs, can also be used in conjunction with other variants of convolutional layers, such as separable convolution layer Howard et al.
(2017) and binary convolutional layer McDanel et al.
(2017b).
As discussed throughout this paper, our work extends the original IDP design by proposing new training algorithms involving a single proﬁle, which may be trainable or pre-determined, to signiﬁcantly improve the overall performance, especially in low DP percentages.
Network pruning is a widely-studied area that also aims at compressing the CNN models.
Early works of network pruning construct a threshold for dropping weights by information obtained from
Hessian matrix or inverse Hessian matrix in LeCun et al.
; Hassibi & Stork (1993), which adds memory and computation costs.
In most of the recent works, magnitude-based pruning and recovering are incorporated to compensate the potential loss incurred by inadequate pruning.
For example, Guo et al.
(2016) introduces the splicing operation to enable connection recovery, and
Han et al.
(2016) directly makes the network dense again.
Li et al.
(2016) also prune ﬁlters in CNNs based on magnitude, but the number of ﬁlters pruned away in each layer is decided by layer-wise sensitivity.
Besides magnitude-based pruning, a Taylor expansion-based criterion is introduced in
Molchanov et al.
(2016) to approximate the change in the cost function induced by pruning.
In addition to network pruning, some works focus on low-rank decomposition for network compression.
For example, Denton et al.
(2014) and Jaderberg et al.
(2014) approximate the weight matrix into low-rank components by minimizing the reconstruction error.
Yu et al.
(2017) further decomposes the weight matrix into its low-rank and sparse component.
Other works focus on grouping similar weights, such as quantization by Han et al.
(2015), Gong et al.
(2014), and Zhou et al.
(2017) and weight sharing by Ullrich et al.
(2017), aiming at reducing the level of redundancy and the required storage.
Yet another approach introduces group sparsity regularizer to constrain the structure of the model in Wen et al.
(2016), Zhou et al.
(2016b),
and Alvarez & Salzmann (2016).
While all the above techniques are promising in reducing the size of the networks, none of them supports dynamic adjustment during inference as IDP does.
Furthermore, most of the above techniques involve retraining the model iteratively, resulting in computational overhead.
In our proposed work, the goal of efﬁcient inference with dynamic adjustment can be readily fulﬁlled by training a single model at once, and the effectiveness is expected to be further improved by incorporating with other techniques listed above.
C ONCLUSION
In this paper, we extend the idea of incomplete dot product (IDP) by proposing the Task-wise Early
Stopping and Loss Aggregation (TESLA) algorithm to signiﬁcantly improve the performance of neural networks with dynamically computation regions at inference time without signiﬁcantly compromising accuracy.
A task is deﬁned as the learning process that uses only a subset of weights speciﬁed by a DP percentage to learn the optimal representations of the network.
By introducing non-increasing proﬁle coefﬁcients to prioritize weights or ﬁlters during training, TESLA can be used to optimize multiple tasks in decreasing order of DP percentages by aggregating the their loss functions.
Additionally, we propose Randomized TESLA (R-TESLA) which optimizes tasks in random order, and show that both TESLA and R-TESLA outperform original IDP and multiple-proﬁle
IDP signiﬁcantly.
The visualization of the class activation maps (CAMs) provide a strong evidence that the representations learned by TESLA allow dynamically scaling across a computation range to meet various power consumption, latency and accuracy requirements on end devices.