SPECIALIZATION OF SUB - PATHS FOR ADAPTIVE DEPTH NETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a novel approach to anytime networks that can control network depths instantly at runtime to provide various accuracy-efficiency trade-offs.
While controlling the depth of a network is an effective way to obtain actual inference speed-up, previous adaptive depth networks require either additional intermediate classifiers or decision networks, that are challenging to train properly.
Unlike previous approaches, our adaptive depth networks require virtually no architectural changes from baseline networks.
Instead, we introduce a novel training method that enforces some sub-paths of the baseline networks to have a special property, with which the sub-paths do not change the semantic level of input features, but only refine them to reduce prediction errors.
Those specialized sub-paths can be skipped at test time, if needed, to save computation at marginal loss of prediction accuracy.
We first formally present the rationale behind the sub-paths specialization, and based on that, we propose a simple and practical training method to specialize sub-paths for adaptive depth networks.
While minimal architectural changes and training efforts are required, we demonstrate that our approach significantly outperforms non-adaptive baselines in various tasks, including ImageNet classification,
COCO object detection and instance segmentation.
Further, we show that the smallest sub-networks of our adaptive depth networks achieve competitive model compression effect compared to recent state-of-the-art techniques.
I NTRODUCTION
Modern deep neural networks provide state-of-the-art performance at high computational costs, and, hence, lots of efforts have been made to leverage those inference capabilities in resource-constrained systems, such as autonomous vehicles.
Those efforts include compact architectures (Howard et al., 2017; Zhang et al., 2018; Han et al., 2020), network pruning (Han et al., 2016; Liu et al., 2019), weight/activation quantization (Jacob et al., 2018), knowledge distillation (Hinton et al., 2015), to name a few.
However, those approaches provide static accuracy-efficiency trade-offs that are often tailored for worst-case scenarios, and, hence, the lost accuracy cannot be recovered even if more resources become available.
Adaptive networks such as anytime networks (Huang et al., 2018; Yu et al., 2018; Wan et al., 2020) attempt to provide runtime adaptability to deep neural networks by exploiting the redundancy in either depths or widths, as shown in Figure 1, or resolutions (Yang et al., 2020).
Dynamic networks (Wu et al., 2018; Li et al., 2021; 2020; Zhu et al., 2021) add additional control logic to the backbone network for input-dependent adaptation.
However, these adatpive networks usually require auxiliary networks, such as intermediate classifiers or decision networks, which are challenging to train properly.
Further, since adaptive networks have multiple sub-networks, embedded in a single neural network, training them incurs potentially conflicting training objectives for the sub-networks, resulting in worse performance than non-adaptive networks (Li et al., 2019).
In this work, we introduce a novel approach to anytime networks that is executable in multiple depths to provide instant runtime accuracy-efficiency trade-offs.
Unlike previous adaptive depth networks, our approach does not require additional add-on networks or classifiers, and, hence, it can be applied to modern residual networks easily.
While maintaining the structure of original networks, we train several sub-paths, or a sequence of residual blocks, of the network to have a special property, that preserves the semantic level of input features, and only refines them to reduce prediction errors.
At test time, these specialized sub-paths can be skipped, if needed, for efficiency at marginal loss of accuracy as shown in Figure-1 (right).
The proposed sub-paths specialization is achieved by enforcing sub-networks with different depths to produce semantically equivalent features for every spatial dimension.
In Section 3, we formally discuss the rationale behind the sub-paths specialization and introduce a simple and practical training method for sub-paths specialization.
In most previous anytime networks, training multiple subnetworks requires training time that is linearly proportional to the number of supported sub-networks, and resolving potential conflicts between sub-networks is an important problem.
In contrast, our approach does not try to resolve potential conflicts while jointly training many sub-networks.
Instead, our training method exploits only two sub-networks for sub-paths specialization, and, at test time, those specialized sub-paths are exploited selectively to build many sub-networks of various depths.
Therefore, the total training time is no greater than training two individual networks.
Finally, in Section 4, we demonstrate that our adaptive depth networks with sub-paths specialization outperform counterpart static networks and achieve actual inference acceleration in multiple tasks including ImageNet classification, COCO object detection and instance segmentation.
Further, the smallest sub-networks of our adaptive depth networks achieve significantly better performance than recent state-of-the-art model compression methods.
R ELATED W ORK
Adaptive Networks:
Anytime networks (Larsson et al., 2017; Huang et al., 2018; Hu et al., 2019;
Wan et al., 2020) and dynamic networks (Wu et al., 2018; Li et al., 2021; Guo et al., 2019;
Li et al., 2020; Yang et al., 2020) have attracted lots of attention for their runtime adaptability.
Most anytime networks have multiple classifiers that are connected to intermediate layers (Huang et al., 2018;
Li et al., 2019; Fei et al., 2022).
Training multiple classifiers is a challenging task and many anytime networks (Li et al., 2019; Zhang et al., 2019; Wan et al., 2020; Huang et al., 2018;
Hu et al., 2019) exploit knowledge distillation to supervise intermediate classifiers using the last, or the best, classifier.
Slimmable neural networks can adjust channel widths for adaptability and they exploit switchable batch normalization to handle multiple sub-networks with a single shared classifier (Yu et al., 2018;
Yu & Huang, 2019b).
While some dynamic networks (Li et al., 2021) extend anytime networks simply by adding input-conditioned decision gates at branching paths, a few dynamic networks (Wu et al., 2018; Veit & Belongie, 2018; Wang et al., 2018a) extend residual networks by applying block-level decision gates that determine if the block can be skipped.
The latter approach is based on the thought that some blocks can be skipped on easy inputs.
However, in these dynamic networks with adaptive depths, no formal explanation has been given why some blocks can be skipped for a given input.
Therefore, users have no control over the depth of the sub-networks.
Residual Blocks with Shortcuts: Since the introduction of ResNets (He et al., 2016), residual blocks with shortcuts have received extensive attention because of their ability to train very deep networks, and have been chosen by many recent deep neural networks (Sandler et al., 2018; Tan & Le, 2019;
Vaswani et al., 2017).
Veit et al.
(2016) argue that identity shortcuts make exponential paths and results in an ensemble of shallower networks.
This thought is supported by the fact that removing individual residual blocks at test time does not significantly affect performance (Huang et al., 2016;
Xie et al., 2020).
Other works argue that identity shortcuts enable residual blocks to perform iterative feature refinement, where each block improves slightly but keeps the semantic of the representation of the previous layer (Greff et al., 2016; Jastrzebski et al., 2018).
Our work build upon those views on residual blocks with shortcuts.
We further extend them for adaptive depth networks by introducing a novel training method that exploits the properties of residual blocks more explicitly for sub-paths specialization.
S UB -P ATHS S PECIALIZATION FOR A DAPTIVE D EPTH N ETWORKS
In this section, we first formally discuss the rationale behind the sub-paths specialization.
Then, the details of training of sub-paths specialization is discussed.
3. M OTIVATION AND O VERVIEW
In typical residual networks such as ResNets, the s -th residual stage is consisted of L identical residual blocks, which transform input features h s − 1 additively and produce the output features h s , as shown in Equation 1.
A block with a residual function F is supposed to perform two primary functions (1) learning new higher level features and (2) refining already learned features at the same semantic level (Jastrzebski et al., 2018).
We call the first and the second functions, respectively, as feature learning and feature refinement .
If a residual block mostly perform feature refinement while not changing the semantic level of features, the performance of the residual network is not significantly affected by dropping the block at test time (Huang et al., 2016; Xie et al., 2020).
However, in typical residual networks, most residual blocks are trained to perform both functions, and, hence, random dropping of residual blocks at test time degrades the performance significantly.
If some residual blocks can be trained to focus on one function while the remaining blocks are encouraged to focus on the other function, then we can safely skip blocks specialized to feature refinement to save computation at marginal loss of prediction accuracy, if needed for efficiency.
To this end, in our approach, residual functions of a residual stage are partitioned into two sub-paths,
F base and F skippable , as in Equation 1, and, during training, they are encouraged to focus more on respective functions, i.e., learning new level features and refining the learned features.
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Since the last half residual functions, or F skippable , is supposed to preserve the semantic level of input features, F skippable can be skipped for efficiency and the intermediate features, or h sbase , can be used as the input to the next stage.
When the higher inference accuracy is required, F skippable can be applied to h sbase to produce more refined features h ssuper .
Since the feature representations h sbase and h ssuper are at the same semantic level, either h sbase or h ssuper can be provided as an input h s to the next ( s + 1) -th residual stage with little change in the feature distribution.
Figure 2 shows an residual stage, in which the last half blocks are specialized to preserve the semantic level of input features.
Since each such residual stage provides two alternative internal paths, if there are N s residual stages with specialized sub-paths, 2 N s sub-networks with different accuracy-efficiency trade-offs become available at test time (Section 4.3).
3. S UB -P ATHS S PECIALIZATION
In order to make the skippable sub-paths to preserve the semantic level of input features, we enforce both feature representations h sbase and h ssuper to converge to some common target representation using the same data-generating distribution X during training:
[Equation] [Equation]
[Equation] [Equation]
Equation 5 does not mean that residual functions in F skippable do nothing to the input features h sbase .
Instead, it implies that residual functions in F skippable are unbiased estimators of target residual functions that preserve input features’ semantic level (Greff et al., 2016).
[Figure]
Then, what do the residual functions in F skippable do?
The functionality of them can be further investigated through Taylor expansion.
Any loss function L used for training can be approximated with Taylor expansion as follows:
[Equation]
In Equations 6 -8, the loss function is iteratively expanded around h sj − 1 ( j = K + 1 , ..., L ) , which is the input features to F j residual function in the skippable sub-path.
Note that high order terms
O ( · ) can be ignored since F s in F skippable have small magnitude, as shown in Equation 5.
Thus, in Equation 8, the overall training with any loss function L optimizes F j ( j = K + 1 , ..., L ) in the half space of − ∂ L ( h sj − 1 ) ∂
h sj −to minimize the dot product between F j and ∂ L ( h sj − 1 ) ∂
h sj − 1 .
This means that F j reduces overall loss L ( h ssuper ) by moving iteratively in the negative direction of ∂ L ( h sj − 1 )
∂
h sj − 1 .
Considering these results in Equations 5 and 8 together, we can conjecture that residual functions in F skippable (1) preserve the level of input features h sbase , but (2) refine h sbase iteratively to minimize the loss and contribute to better inference accuracy.
In other words, the skippable sub-path is specialized for refining input features for better inference accuracy while preserving its semantic level .
Therefore,
F skippable , or the specialized residual blocks, can be skipped without causing much changes of the feature distribution to the next residual stages (Sections 4.3 and 4.4).
One theory in neuroscience is the existence of regions of the brain focused on particular tasks (Caramazza & Coltheart, 2006).
In parallels to neuroscience, we cautiously conjecture that if some parts of a neural network is specialized for one function by suppressing the other functions, the remaining parts of the network are encouraged to focus more on the suppressed functions.
Accordingly, we expect that the first half residual functions F j ( j = 1 , ..., K ) , that are not skippable, are conversely encouraged to learn new level features more actively.
Our empirical result in Section 4.
shows that the sub-network, that consists of non-skippable blocks only, outperforms state-of-the-art model compression methods, demonstrating effective feature learning in non-skippable sub-paths.
3. T RAINING A DAPTIVE D EPTH N ETWORKS
To enforce E x ∈ X [ h ssuper − h sbase ] = 0 , given in Equation 2, we propose a training method shown in Algorithm 1, in which two sub-networks, or the super-net and the base-net , are exploited in a single training loop for sub-paths specialization.
The super-net and the base-net are the largest and the smallest sub-networks, respectively, of the proposed adaptive depth network M .
For example, the base-net skips every specialized sub-paths in M .
In contrast, the super-net executes all residual blocks of M without skipping.
1: Initialize an adaptive depth network M 2: for i=1,..., n iters do 3:
Get next mini-batch of data x and label y 4: optimizer.zero _ grad () 5:
[ ˆ y super , h super ] =
M ( x , skip=[False, ... , False] )
▷ Forward pass for the super-net of M 6: [ ˆ y base , h base ] =
M ( x , skip=[True, ... , True] )
▷ Forward pass for the base-net of M 7: subpath _ reg =
(cid:80)
Nss =1 D KL ( gap ( h ssuper )
∥ gap ( h sbase )) + D KL ( ˆ y super ∥ ˆ y base ) 8: loss = ( α ) criterion ( y , ˆ y super ) + (1 − α ) subpath _ reg 9: loss.backward () 10: optimizer.step () 11: end for
In steps 5 and 6, the forward passes of the super-net and the base-net are executed for the same input x .
The second argument, skip , of the model M indicates in which residual stage its specialized sub-path is skipped.
In ResNets with 4 residual stages, 16(= 2 4 ) sub-networks become available by varying the skip argument.
For example, we choose the base-net of a ResNet by setting the second argument to skip=[True, True, True, True] .
During the training, a wrapper function is used to obtain intermediate features from every residual stage as well as the outputs of the sub-networks.
For instance, during the forward pass of the base-net, intermediate features from N s residual stages, or h base =
[ h 1 base , ..., h N s base ] , and output ˆ y base are obtained.
In steps 7 and 8, we modify the training criterion by adding Kullback Leibler (KL) divergence between h base and h super as an regularization term.
This term enforces E x ∈ X
[ h super − h base ] =for sub-paths specialization.
Global average pooling, denoted by gap ( · ) , is applied to h base and h super before computing their KL divergence since we want h super and h base to have semantically equivalent feature distributions, not exactly same values.
In the regularization term, the KL divergence between logits, ˆ y base and ˆ y super , is also included since they are last features from the sub-networks.
Hyperparameter α determines the strength of the regularization term.
Unlike previous anytime networks, our approach does not train all sub-networks jointly or sequentially. Therefore, we do not need to resolve potential conflicts among sub-networks during training.
Instead, in Algorithm 1, only two sub-networks are exploited during training for sub-paths specialization, and, hence, the total training time is no greater than training two sub-networks individually.
However, at test time, the specialized sub-paths can be exploited selectively to construct 2 N s sub-networks of various depths.
We demonstrate this result in Section 4.3.
3. S KIP -A WARE B ATCH N ORMALIZATION FOR S HARED S UB - PATHS
In our adaptive depth networks, each residual stage expects the distribution of input features is not affected by the choices of internal paths of its previous residual stages.
However, the shared sub-paths inside each residual stage still need adaptability to handle different sub-networks.
Originally, batch normalization (BN) (Ioffe & Szegedy, 2015) was proposed to handle internal covariate shift during training non-adaptive networks by normalizing features.
In our adaptive depth networks, however, internal covariate shifts occur in shared sub-paths when different sub-networks are used.
To handle this internal covariate shifts, switchable BN operators, called skip-aware BNs , are used in shared sub-paths.
For example, at each residual stage, different BN operators are used by the layers in the shared sub-path when specialized sub-path is skipped.
The effectiveness of switchable BNs has been demonstrated in neural networks with adaptive widths (Yu & Huang, 2019a) and adaptive resolutions (Zhu et al., 2021), and we apply them for our adaptive depth networks.
The amount of parameters for skip-aware BNs is negligible.
In ResNet50, skip-aware BNs increase the parameters by 0.07%.
E XPERIMENTS
To demonstrate the effectiveness of our approach, we conduct experiments on ImageNet classification and COCO object detection and instance segmentation benchmarks.
We use two modern residual networks, ResNets (He et al., 2016) and MobileNetV2 (Sandler et al., 2018) as base models.
The models with the suffix ‘-ADN’ denotes the proposed adaptive depth networks extending those base networks.
For adaptive depth networks, we add skip-aware BNs for shared sub-paths and every residual stage is split equally into two sub-paths.
During training, every skippable sub-path is specialized according to Algorithm 1 from scratch.
One exception is the 3rd stage of ResNet101 that has exceptional long 23 blocks.
In the 3rd stage of ResNet101-ADN, the first 5 blocks are used for shared sub-path and the last 18 blocks are specialized and become skippable 1 .
4. I MAGE N ET C LASSIFICATION
We evaluate our method on ILSVRC2012 dataset (Russakovsky et al., 2015) that has 1000 classes.
The dataset consists of 1.28M training and 50K validation images.
We follow most training settings in the original papers (He et al., 2016; Sandler et al., 2018), except that ResNet models are trained for 150 epochs.
The hyperparameter α is set to 0 .
5 and 0 .
7 for ResNets and MobileNetV2, respectively.
[Table]
The results in Table 1 show that our adaptive depth networks, both the largest super-net and the smallest base-net, outperform individual counterpart networks consistently.
In Table 1, ResNet50ADN is also compared with two representative anytime networks (Zhang et al., 2019; Yu & Huang, 2019a) that adapt the depths and the widths, respectively.
The result shows that our adaptive depth networks outperform these competitors significantly in every aspect.
In particular, unlike competitors, the reduction of FLOPs of ResNet-ADN is closely translated into actual acceleration.
For instance, when the FLOPs of ResNet50-ADN (base-net) is reduced by 37.3%, its inference latency is also similarly reduced by 36.2%.
In contrast, although S-ResNet50 (0.75x) requires 22% less FLOPs than ResNet50-ADN (base-net), it shows 22.4% longer inference latency in practice.
Since S-ResNet50’s network depth is not changed by reducing widths, it still needs to perform similar number of GPU kernel invocations in practice.
In early-exiting anytime networks, effective depths of network can be reduced.
But as shown in Zhang et al.
(2019)’s work, sub-networks of early-exiting networks manifest significantly low inference accuracy since they only exploit low-level features.
In contrast, our sub-networks exploit features of all levels while significantly reducing effective network depths.
In Table 2, ResNet50-ADN (base-net), which is the smallest and the most efficient sub-network of ResNet50-ADN, is compared with several state-of-the-art model compression methods.
Our
[Table]
ResNet50-ADN (base-net) achieves very competitive results compared to many state-of-the-art methods.
Some of these state-of-the-art methods require unusual training techniques such as pretraining or retraining of pretrained networks to provide static accuracy-efficiency trade-offs.
In contrast, our ResNet50-ADN (base-net) is one of many sub-network of ResNet50-ADN that is trained from scratch according to Algorithm
[Table]
and, hence, it can increase its depth instantly at runtime if the higher inference accuracy is required.
This result demonstrates that non-skippable sub-paths of our adaptive depth networks learn compact feature representations effectively while skippable sub-paths are enforced to preserve the level of input features during training.
4. A BLATION S TUDY
[Table]
We conduct ablation study on ImageNet classification to investigate the influence of two key components of the proposed adaptive depth networks: (1) sub-paths specialization and (2) skip-aware BNs.
The results are shown in Table 3.
When neither of them is applied, the inference accuracy of the super-net and the base-net is significantly lower than that of individual networks by 1.5% and 2.8%, respectively.
This result shows the difficulty of training multiple sub-networks in anytime networks.
During training, uncoordinated training signals from the super-net and the base-net decrease the effect of training, resulting in worse performance than individual networks.
When one of the two components is applied individually, the performance is still slightly worse than individual networks’.
Finally, when both sub-paths specialization and skip-aware BNs are applied together, ResNet50-ADN achieves significantly better performance than individual networks, both in the super-net and the base-net.
This result demonstrates that both sub-paths specialization and skip-aware BNs are essential for the proposed adaptive depth networks.
4. V ARYING N ETWORK D EPTHS AT T EST T IME
One important advantage of sub-paths specialization is that each residual stage’s choice of internal paths does not affect the following residual stages.
Therefore, each residual stage can independently make a decision about skipping its specialized sub-path.
This implies that our adaptive depth networks can support 2 N s sub-networks at test time, when N s residual stages are given.
Figure 3-(a) shows the performance of ResNet50-ADN when its depth is varied at test time.
Among them, only the super-net and the base-net are used for training in Algorithm 1.
The other sub-networks are built by applying different combinations of skipping decisions at test time.
Although these sub-networks are not trained explicitly, their performance does not drop significantly.
Instead, they show gradual degradation of performance as the depth of sub-networks becomes gradually shallower.
[Figure] [Figure]
4. A NALYSIS OF S UB - PATHS S PECIALIZATION
The premise of sub-paths specialization is that each residual stage produce semantically equivalent features regardless of whether its specialized sub-path is skipped or not; i.e, E x ∈ X [ h ssuper − h sbase ][-= 
-]
{+= 
+} ( s = 0 ..N s ) .
We can verify this by inspecting the input distribution of the next residual stages.
Figure 3-(a) shows the distances between the means of switchable BNs in the shared sub-paths of
ResNet50-ADN.
In Figure 3-(a), the distance at every residual stage’s first layer is close to zero.
This demonstrates that their previous residual stages produce feature distributions of the same means regardless of whether the specialized sub-path is skipped or not.
Figure 3-(a) also shows that the shared layers except the first layers of each residual stage have very different input distributions, demonstrating the necessity of skip-aware BNs in the shared sub-paths.
To further verify that the non-skippable sub-paths and the skippable sub-paths, respectively, focus more on learning new level features and refining learned features, we visualize the activation of 3rd residual stage of ResNet50-ADN using Grad-CAM (Selvaraju et al., 2017) in Figure 4.
The 3rd residual stage of ResNet50-ADN has six residual blocks and the last three blocks are skippable.
In Figure 4-(a), the activation regions of original ResNet50 changes gradually across all consecutive blocks.
This implies that all blocks are involved in gradual learning of new level features as well as the refinement of the learned features.
In contrast, in Figure 4-(b), ResNet50-ADN (super-net) manifests very different progressions of activation regions in two sub-paths.
In the first three residual blocks, we can observe significant changes of hot regions in wide areas, suggesting active learning of new level features.
However, in the skippable last three blocks, the activation regions have little movements and gradually become more concentrated around the target object, demonstrating the refinement of learned features.
Further, in Figure 4-(c), we can observe that the final activation map of the ResNet50 (base-net) is very similar to the super-net’s final activation map in (b), implying that they are at the same feature level as suggested in Equation 2.
However, the final activation map of the base-net has less hot activation regions around the target object than super-net’s, potentially resulting in lower inference accuracy than super-net’s more refined features.
Finally, it should be noted that even though the three blocks of the base-net share weights with the first three blocks of the super-net, they manifest very different activation maps. This demonstrates that the skip-aware BNs are effective in processing multi-modal inputs.
4. O BJECT D ETECTION AND I NSTANCE S EGMENTATION
[Table]
In order to investigate the generalization ability of our approach, we use MS COCO 2017 datasets on object detection and instance segmentation tasks using representative detectors. We compare individual ResNets and our adaptive depth ResNets-ADN as backbone networks of the detectors.
For training of detectors, we use Algorithm 1 with slight adaptation.
For object detection, the intermediate features h sbase and h ssuper ( s = 1 ..N s ) can be obtained directly from backbone network’s feature pyramid networks (FPN) (Lin et al., 2017), and, hence, a wrapper function is not required to extract intermediate features.
All networks are trained on train2017 for 12 epochs from ImageNet pretrained weights, following the training settings suggested in (Lin et al., 2017; Han et al., 2020).
Table 4 shows the results on val2017 containing 5000 images.
Our adaptive depth backbone networks significantly outperform individual static backbone networks in terms of COCO’s standard metric AP.
This result demonstrates that our adaptive depth networks with the sub-paths specialization learn better feature representations in every feature level.
C ONCLUSIONS
We presented a novel approach to adaptive depth networks that provide instant accuracy-efficiency trade-offs at runtime.
Unlike previous approaches, our adaptive depth networks do not require auxiliary add-on networks or classifiers to control network depths.
Instead, our approach train sub-paths of a single network to have a special property, with which the sub-paths can be skipped for efficiency at marginal loss of prediction accuracy.
We discussed the theoretic rationale behind this sub-paths specialization, and presented a simple and practical training method.
While our training method uses only two sub-networks, it makes various sub-networks with different depths available at test time.
Through extensive experiments, we demonstrated that the proposed adaptive depth networks outperform individual static networks and state-of-the-art model compression methods while achieving actual inference speed-up.
R EPRODUCIBILITY
For reproducibility, we provide source codes and the links to the pretrained weights as the supplementary material.
Code and models will be available in authors’ public repository.
All programs for training and evaluation run on PCs equipped with two or four Nvidia RTX 3090 GPUs and an Intel i9-10900X CPU @3.7GHz.
A A PPENDIX A. V ARYING THE L ENGTHS OF S UB - PATHS
In our adaptive depth networks, most residual stages have 4-6 residual blocks, and each stage is split into two sub-paths; the first sub-path is shared by all sub-networks while the second sub-path can be skipped, if needed for efficiency.
One exception is the 3rd stage of ResNet101 that has exceptional long 23 blocks.
We investigate how the lengths of the sub-paths affect the performance by varying the number of shared/skippable blocks in the 3rd stage of ResNet101-ADN during training.
The result in Table 5 shows that if the number of shared blocks is too small (e.g., 3 shared blocks), it results in lower performance, both for the super-net and the base-net.
This result shows that the performance of the super-net is limited by the base-net’s low performance since our adaptive depth networks enforce both the super-net and the base-net to learn same feature levels at every residual stage.
As the more shared blocks are used by the base-net, the performance of both the super-net and the base-net improve.
As the performance of the base-net is improved gradually from 75.4% to 77.9%, the super-net’s performance improves from 77.8% to 78.6%.
However, when the number of shared blocks become greater than 5, the performance of the super-net is bounded at about 78.6%.
This result implies that some number of residual blocks is essential for proper learning of new level features, but their effectiveness diminishes after some optimal points.
[Table]