Efﬁcient Inference on Deep Neural Networks by Dynamic Representations and Decision Gates
Abstract
The current trade-off between depth and computational cost makes it difﬁcultto adopt deep neural networks for many industrial applications, especially whencomputing power is limited.
Here, we are inspired by the idea that, while deeperembeddings are needed to discriminate difﬁcult samples, a large number of samplescan be well discriminated via much shallower embeddings.
In this study, weintroduce the concept of decision gates (d-gate), modules trained to decide whethera sample needs to be projected into a deeper embedding or if an early prediction canbe made at the d-gate, thus enabling the computation of dynamic representationsat different depths.
The proposed d-gate modules can be integrated with anydeep neural network and reduces the average computational cost of the deepneural networks while maintaining modeling accuracy.
Experimental results showthat leveraging the proposed d-gate modules led to a ∼ 40% speed-up and 44%
FLOPS reduction on ResNet-101 and 45% speed-up and 36% FLOPS reduction on
DenseNet-201 trained on the CIFAR10 dataset with only ∼ 2% drop in accuracy.
Introduction
Past studies such as [16] have shown that deeper architectures often lead to better modeling performance; however, deeper architectures also pose a number of issues.
Besides becoming more prone tooverﬁtting and becoming more difﬁcult to train, the trade-off between depth and computational costmakes it difﬁcult to adopt deeper architectures for many industrial applications.
He et al.
[7] tackled the former issue of degradation in learning deeper neural networks (e.g., vanishinggradient) by introducing the concept of residual learning, where learning is based on the residualmapping rather than directly on the unreferenced mapping.
Following that, Xie et al.
[19] tookadvantage of the inception idea (i.e, split-transform-merge strategy) within a residual block structureto provide better subspace modeling while resolving the degradation problem at the same time,resulting in a ResNext architecture with improved modeling accuracy.
To tackle the issue of computational cost, a wide variety of methods have been proposed, including:precision reduction [10], model compression [6], teacher-student strategies [8], and evolutionaryalgorithms [13, 14]. More recently, conditional computation [1, 4, 12, 18, 2] and early prediction [17] methods have beenproposed to tackle this issue, which involve the dynamic execution of different modules within anetwork.
Conditional computation methods have largely been motivated by the idea that residualnetworks can be considered as an ensemble of shallower networks. As such, these methods takeadvantage of skip connections to determine which residual modules are necessary to be executed,with most leveraging reinforcement learning.
In this study, we explore the idea of early prediction but instead draw inspiration from the soft-marginsupport vector [3] theory for decision-making. Speciﬁcally, we introduce the concept of decision gates(d-gate), modules trained to decide whether a sample needs to be projected into a deeper embeddingor if an early prediction can be made at the d-gate, thus enabling the conditional computation of
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
Predicted Label Predicted Label
P r e d i c t e d
L a b e l
D ec i s i on
G a t e
D ec i s i on
G a t e
B l o c k
R e
s i du a l
[Figure]
dynamic representations at different depths.
The proposed d-gate modules can be integrated with anydeep neural network without the need to train networks from scratch, and thus reduces the averagecomputational complexity of the deep neural networks while maintaining modeling accuracy.
Methodology
Deeper neural network architectures have been demonstrated to provide a better subspace embeddingof data when compared to shallower architectures, resulting in better discrimination of data space andbetter modeling accuracy.
Inspired by soft-margin support vector [3] theory, we make a hypothesisthat while deeper subspace embeddings are necessary to discriminate samples that lie close tothe decision boundaries in the lower embedding space, their effect on samples that already lie farfrom decision boundaries in the shallower embedding space may be insigniﬁcant and unnecessary.
Therefore, an effective yet efﬁcient mechanism for determining the distance between samples andthe decision boundaries in the lower layers of the network would make it possible to perform earlypredictions on these samples without projecting them into a deeper embedding space.
This approachwould reduce the average computational cost of prediction signiﬁcantly.
However, devising anefﬁcient way to determine whether a sample is a boundary sample is a challenging problem.
Here, we formulate the early prediction problem as a risk minimization problem, and introduce aset of single-layer feed-forward networks (which we will refer to as decision gates (d-gate)) that areintegrated directly into a deep neural network (see Fig. 1).
The goal of d-gate modules is to not onlydecide whether a sample requires projection into a deep embedding space, but also minimize the riskof early wrong classiﬁcations as well.
Speciﬁcally, we train d-gate modules that are integrated into adeep neural network via a hinge loss [5] that minimizes the risk of early misclassiﬁcation in lowerembeddings while deciding whether the sample is a boundary sample:
[Equation]
where y is the ground truth label for the input data x and ˆ y is the predicted class label via the d-gatemodule with the set of weights w and biases b .
The set of weights w has a dimensionality of f × c ,where f denotes the number of input features to the d-gate module and c denotes the number ofclass labels in the classiﬁcation task.
This d-gate module provides an important beneﬁt where theresult of w T x − b provides the distances of the sample to the corresponding decision boundary ofeach class label in the embedding space.
Training the d-gate module in this way provides a linearclassiﬁer where samples that do not require deeper embeddings for discrimination are those withlarger distances (i.e., with the positive sign) from the decision boundary.
It is important to note thatsingle layer nature of d-gate modules is designed to account for efﬁciency.
The d-gate module is trained via the training data utilized to train the deep neural network and theobjective for each d-gate module is to minimize the classiﬁcation error on the training data.
Therefore,the loss function on the training data can be formulated as:
[Equation]
where Y denotes the set of ground truth labels for all training data.
What is most interesting about
L ( Y, ˆ Y ; w, b ) is the fact that L ( · ) is a convex function of w and b , and as such can be optimized viagradient descent.
Therefore, traditional gradient descent can be adapted here, where a step is taken inthe direction of a vector selected from the function’s sub-gradient [15] to ﬁnd the optimized values.
As a result, the d-gate can be trained within a mini-batch training framework, which makes it is veryconvenient for utilizing in the training of deep neural networks with large datasets.
In essence, the proposed d-gate module can compute the distance of each sample to the decisionboundaries based on w T x − b ; the calculated distances are compared with the decision threshold t ofeach d-gate to determine whether early prediction on the sample can be performed at the d-gate orhave the sample moved to a deeper stage of the deep neural network to project into a better embeddingspace for improved prediction.
The samples that are far from the decision boundaries result to outputlarger values in w T x − b ; therefore, if the d-gate distance for a sample satisﬁes the d-gate decisionthreshold, the class corresponding to the largest distance is assigned as the predicted class label forthe sample in this early prediction step. [Figure] [Figure] A cc u r a c y ( % )
[Figure]
[Table]
Results and Discussion
The efﬁcacy of the proposed d-gate modules is examined with two different network architectures(ResNet101 [7] and DenseNet201 [9]) on the CIFAR10 dataset.
A key beneﬁt of the proposedd-gate modules is that it enables ﬁne control over the trade-off between modeling accuracy andcomputational cost by adjusting the d-gate decision thresholds.
By decreasing the d-gate decisionthresholds, the number of samples undergoing early prediction increases, thus reducing the averagecomputational cost of network predictions greatly.
For this study, we integrated two d-gate modulesin ResNet-101 (after the ﬁrst and second main blocks) and DenseNet-201 (after the ﬁrst and seconddense blocks), and explore different d-gate conﬁgurations.
The networks are implemented in the
Pytorch framework and the prediction speeds are reported based on single Nvidia Titan Xp GPU.
It can be observed from Table 3 that the computational cost of ResNet network can be reduced by {+
+} MFLOPS while maintaining the same level of accuracy as to the original ResNet-101 by integratingtwo d-gate modules with decision thresholds of ( t 1 , t 2) = (2 . 5 , 2 . 5) for both d-gates.
The integrationof d-gate modules can reduce the computational cost of ResNet-101 network by 1.8 × (i.e., lowerby 1.12 GFLOPS) with 1.7% drop in accuracy compared to the original ResNet-101 (with distancethresholds ( t 1 , t 2) =
(1 . 0 , 2 . 0) in d-gate1 and d-gate2), resulting in a ∼ 40% speed-up on the GPU.
The experiments for DenseNet-201 show that it is possible to reduce the number of FLOPs by {+
+} MFLOPs with only a 1% drop in accuracy, leading to a 35% speed-up on the GPU.
Furthermore,a 2 × speed-up can be achieved with d-gate modules compared to the original DenseNet-201 witha 3% accuracy margin.
Based on the experimental results, the proposed d-gate modules lead to asigniﬁcant increase in prediction speed, making it well-suited for industrial applications.
In addition to the d-gate modules being proposed, one of the key contributions of this paper is theintroduction of a hinge loss for training the d-gate modules.
Past studies [11] have argued that crossentropy results in a small margin between the decision boundaries and the training data.
As such, it isvery difﬁcult to trust the conﬁdence values of the Softmax layer to decide about the sample sincethere is no valuable information in the Softmax output.
To demonstrate the effectiveness of the hingeloss leveraged in the proposed d-gates compared to the cross-entropy loss, an additional comparativeexperiment was conducted.
More speciﬁcally, two decision gates were added to ResNet101 in thesame way as reported.
However, rather than train using the proposed hinge loss, the decision gateswere instead trained via a cross-entropy loss.
This enables us to compare the effect of hinge loss vs.cross-entropy loss on decision gate functionality.
Figure 2 demonstrates the accuracy vs. numberof FLOPs for the network where the decision gates were trained based on the proposed hinge lossapproach compared to trained using a regular cross-entropy training procedure.
It can be observedthat, with the same number of FLOPs in the network, the network where the decision gates weretrained based on the proposed hinge loss provides much higher modeling accuracy compared tothat trained via cross-entropy loss.
The accuracy gap increases exponentially when the decisiongates are conﬁgured such that the network uses fewer number of FLOPs.
What this illustrates is theaforementioned issue with the use of cross-entropy loss and decision boundaries.
[-FLOPs × 
-]