Efﬁcient Inference on Deep Neural Networks by Dynamic Representations and Decision Gates
Abstract
The current trade-off between depth and computational cost makes it difﬁcult to adopt deep neural networks for many industrial applications, especially when computing power is limited.
Here, we are inspired by the idea that, while deeper embeddings are needed to discriminate difﬁcult samples, a large number of samples can be well discriminated via much shallower embeddings.
In this study, we introduce the concept of decision gates (d-gate), modules trained to decide whether a sample needs to be projected into a deeper embedding or if an early prediction can be made at the d-gate, thus enabling the computation of dynamic representations at different depths.
The proposed d-gate modules can be integrated with any deep neural network and reduces the average computational cost of the deep neural networks while maintaining modeling accuracy.
Experimental results show that leveraging the proposed d-gate modules led to a ∼ 38% speed-up and ∼ 39% FLOPS reduction on ResNet-101 and ∼ 46% speed-up and ∼ 36% FLOPS reduction on
DenseNet-201 trained on the CIFAR10 dataset with only ∼ 2% drop in accuracy.
Introduction
Past studies such as [16] have shown that deeper architectures often lead to better modeling performance; however, deeper architectures also pose a number of issues.
Besides becoming more prone to overﬁtting and becoming more difﬁcult to train, the trade-off between depth and computational cost makes it difﬁcult to adopt deeper architectures for many industrial applications.
He et al.
[7] tackled the former issue of degradation in learning deeper neural networks (e g., vanishing gradient) by introducing the concept of residual learning where learning is based on the residual mapping rather than directly on the unreferenced mapping.
Following that, Xie et al.
[19] took advantage of the inception idea (i.e, split-transform-merge strategy) within a residual block structure to provide better subspace modeling while resolving the degradation problem at the same time, resulting in a ResNext architecture with improved modeling accuracy.
To tackle the issue of computational cost, a wide variety of methods have been proposed, including: precision reduction [10], model compression [6], teacher-student strategies [8], and evolutionary algorithms [13, 14].
More recently, conditional computation [1, 4, 12, 18, 2] and early prediction [17] methods have been proposed to tackle this issue, which involve the dynamic execution of different modules within a network.
Conditional computation methods have largely been motivated by the idea that residual networks can be considered as an ensemble of shallower networks.
As such, these methods take advantage of skip connections to determine which residual modules are necessary to be executed, with most leveraging reinforcement learning.
In this study, we explore the idea of early prediction but instead draw inspiration from the soft-margin support vector [3] theory for decision-making.
Speciﬁcally, we introduce the concept of decision gates (d-gate), modules trained to decide whether a sample needs to be projected into a deeper embedding
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
B l o c k
R e
s i du a l
Predicted Label Predicted Label
P r e d i c t e d
L a b e l
D ec i s i on
G a t e
D ec i s i on
G a t e
B l o c k
R e
s i du a l
[Figure]
or if an early prediction can be made at the d-gate, thus enabling the conditional computation of dynamic representations at different depths.
The proposed d-gate modules can be integrated with any deep neural network without the need to train networks from scratch, and thus reduces the average computational complexity of the deep neural networks while maintaining modeling accuracy.
Methodology
Deeper neural network architectures have been demonstrated to provide a better subspace embedding of data when compared to shallower architectures, resulting in better discrimination of data space and better modeling accuracy.
Inspired by soft-margin support vector [3] theory, we make a hypothesis that while deeper subspace embeddings are necessary to discriminate samples that lie close to the decision boundaries in the lower embedding space, their effect on samples that already lie far from decision boundaries in the shallower embedding space may be insigniﬁcant and unnecessary.
Therefore, an effective yet efﬁcient mechanism for determining the distance between samples and the decision boundaries in the lower layers of the network would make it possible to perform early predictions on these samples without projecting them into a deeper embedding space.
This approach would reduce the average computational cost of prediction signiﬁcantly.
However, devising an efﬁcient way to determine whether a sample is a boundary sample is a challenging problem.
Here, we formulate the early prediction problem as a risk minimization problem, and introduce a set of single-layer feed-forward networks (which we will refer to as decision gates (d-gate)) that are integrated directly into a deep neural network (see Fig. 1).
The goal of d-gate modules is to not only decide whether a sample requires projection into a deep embedding space, but also minimize the risk of early wrong classiﬁcations as well.
Speciﬁcally, we train d-gate modules that are integrated into a deep neural network via a hinge loss [5] that minimizes the risk of early misclassiﬁcation in lower embeddings while deciding whether the sample is a boundary sample:
[Equation]
where y is the ground truth label for the input data x and ˆ y is the predicted class label via the d-gate module with the set of weights w and biases b .
The set of weights w has a dimensionality of f × c , where f denotes the number of input features to the d-gate module and c denotes the number of class labels in the classiﬁcation task.
This d-gate module provides an important beneﬁt where the result of w T x − b provides the distances of the sample to the corresponding decision boundary of each class label in the embedding space.
Training the d-gate module in this way provides a linear classiﬁer where samples that do not require deeper embeddings for discrimination are those with larger distances (i.e., with the positive sign) from the decision boundary.
It is important to note that single layer nature of d-gate modules is designed to account for efﬁciency.
The d-gate module is trained via the training data utilized to train the deep neural network and the objective for each d-gate module is to minimize the classiﬁcation error on the training data.
Therefore, the loss function on the training data can be formulated as:
[Equation]
where Y denotes the set of ground truth labels for all training data.
What is most interesting about
L ( Y, ˆ Y ; w, b ) is the fact that L ( · ) is a convex function of w and b , and as such can be optimized via gradient descent.
Therefore, traditional gradient descent can be adapted here, where a step is taken in the direction of a vector selected from the function’s sub-gradient [15] to ﬁnd the optimized values.
As a result, the d-gate can be trained within a mini-batch training framework, which makes it is very convenient for utilizing in the training of deep neural networks with large datasets.
In essence, the proposed d-gate module can compute the distance of each sample to the decision boundaries based on w T x − b ; the calculated distances are compared with the decision threshold t of each d-gate to determine whether early prediction on the sample can be performed at the d-gate or have the sample moved to a deeper stage of the deep neural network to project into a better embedding space for improved prediction.
The samples that are far from the decision boundaries result to output larger values in w T x − b ; therefore, if the d-gate distance for a sample satisﬁes the d-gate decision threshold, the class corresponding to the largest distance is assigned as the predicted class label for the sample in this early prediction step.
[Figure] [Figure]
[Figure]
[Figure]
trained via a conventional cross-entropy approach.
It can be observed that trained hinge loss provide greater computational efﬁciency with higher accuracy than if cross-entropy loss is leveraged
[Table]
Results and Discussion
The efﬁcacy of the proposed d-gate modules is examined with two different network architectures (ResNet101 [7] and DenseNet201 [9]) on the CIFAR10 dataset.
A key beneﬁt of the proposed d-gate modules is that it enables ﬁne control over the trade-off between modeling accuracy and computational cost by adjusting the d-gate decision thresholds.
By decreasing the d-gate decision thresholds, the number of samples undergoing early prediction increases, thus reducing the average computational cost of network predictions greatly.
For this study, we integrated two d-gate modules in ResNet-101 (after the ﬁrst and second main blocks) and DenseNet-201 (after the ﬁrst and second dense blocks), and explore different d-gate conﬁgurations.
The networks are implemented in the
Pytorch framework and the prediction speeds are reported based on single Nvidia Titan Xp GPU.
It can be observed from Table 1 that the computational cost of ResNet network can be reduced by[-
-]
MFLOPS while maintaining the same level of accuracy as to the original ResNet-101 by integrating two d-gate modules with decision thresholds of ( t 1 , t 2) =
The integration of d-gate modules can reduce the computational cost of ResNet-101 network by ∼ 39% (i.e., lower by 1. GFLOPS) with 1.7% drop in accuracy compared to the original ResNet-101 (with distance thresholds ( t 1 , t 2) =
(1 . 0 , 2 . 0) in d-gate1 and d-gate2), resulting in a ∼ 38% speed-up.
The experiments for DenseNet-201 show that it is possible to reduce the number of FLOPs by [-
-]970 MFLOPs ( ∼ 36% reduction) with only a ∼ 2% drop in accuracy, leading to a ∼ 46% speed-up.
Furthermore, a 2.3 × speed-up can be achieved with d-gate modules compared to the original DenseNet-201 within a 3% accuracy margin.
Based on the experimental results, the proposed d-gate modules lead to a signiﬁcant increase in prediction speed, making it well-suited for industrial applications.
In addition to the d-gate modules being proposed, one of the key contributions of this paper is the introduction of a hinge loss for training the d-gate modules.
Past studies [11] have argued that crossentropy results in a small margin between the decision boundaries and the training data.
As such, it is very difﬁcult to trust the conﬁdence values of the Softmax layer to decide about the sample since there is no valuable information in the Softmax output.
To demonstrate the effectiveness of the hinge loss leveraged in the proposed d-gates compared to the cross-entropy loss, an additional comparative experiment was conducted.
More speciﬁcally, two decision gates were added to ResNet101 in the same way as reported.
However, rather than train using the proposed hinge loss, the decision gates were instead trained via a cross-entropy loss.
This enables us to compare the effect of hinge loss vs. cross-entropy loss on decision gate functionality.
Fig. 2 demonstrates the accuracy vs. number of FLOPs for the network where the decision gates were trained based on the proposed hinge loss approach compared to trained using a regular cross-entropy training procedure.
It can be observed that, with the same number of FLOPs in the network, the network where the decision gates were trained based on the proposed hinge loss provides much higher modeling accuracy compared to that trained via cross-entropy loss.
The accuracy gap increases exponentially when the decision gates are conﬁgured such that the network uses fewer number of FLOPs.
What this illustrates is the aforementioned issue with the use of cross-entropy loss and decision boundaries.