Z EN D ET : R EVISITING E FFICIENT O BJECT D ETECTION B ACKBONES FROM Z ERO -S HOT N EURAL A RCHITECTURE S EARCH
Anonymous authors
Paper under double-blind review
A BSTRACT
In object detection models, the detection backbone consumes more than half of the overall inference cost.
Recent researches attempt to reduce this cost by optimizing the backbone architecture with the help of Neural Architecture Search (NAS).
However, existing NAS methods for object detection require hundreds to thousands of GPU hours of searching, making them impractical in fast-paced research and development.
In this work, we propose a novel zero-shot NAS method to address this issue.
The proposed method, named ZenDet, automatically designs efﬁcient detection backbones without training network parameters, reducing the architecture design cost to nearly zero yet delivering the state-of-the-art (SOTA) performance.
Under the hood, ZenDet maximizes the differential entropy of detection backbones, leading to a better feature extractor for object detection under the same computational budgets.
After merely one GPU day of fully automatic design, ZenDet innovates SOTA detection backbones on multiple detection benchmark datasets with little human intervention.
Comparing to ResNet50 backbone, ZenDet is +2 .
0% better in mAP when using the same amount of
FLOPs/parameters and is 1 .
54 times faster on NVIDIA V100 at the same mAP.
Code and pre-trained models will be released after publication.
I NTRODUCTION
Seeking better and faster deep models for object detection is never an outdated task in computer vision.
The performance of a deep object detection network heavily depends on the feature extraction backbone (Li et al., 2018;
Chen et al., 2019b).
Currently, most state-of-the-art (SOTA) detection backbones (He et al., 2016; Xie et al., 2017; Zhu et al., 2019) are designed manually by human experts which could take years to develop.
Since the detection backbone consumes more than half of the total inference cost in many detection frameworks, it is critical to optimize the backbone architecture for better speed-accuracy trade-off on different hardware platforms varying from server-side
GPUs to mobile-side chipsets.
To reduce the manual design, Neural Architecture Search (NAS) has emerged to facilitate the architecture design.
Various NAS methods have demonstrated their efﬁcacy in designing SOTA image classiﬁcation models (Zoph et al., 2018; Liu et al., 2018;
Cai et al., 2019;
Tan & Le, 2019).
These successful stories inspire recent researchers to use NAS to design detection backbones (Chen et al., 2019b;
Du et al., 2020; Jiang et al., 2020) in an end-to-end way.
The existing NAS methods for detection backbone design are all training-based, meaning that they need to train network parameters to evaluate the performance of candidates on the target dataset, taking a long time and consuming huge hardware resources.
Hardware consumption and long time searching make these training-based methods inefﬁcient in modern fast-paced research and development.
To reduce the training cost, training-free methods are proposed recently, also known as zero-shot NAS in previous literatures (Tanaka et al., 2020; Mellor et al., 2021; Chen et al., 2021b;
Lin et al., 2021).
The zero-shot NAS predicts network performance without training network parameters therefore is way faster than training-based NAS.
As a relatively new technique, existing zero-shot NAS methods are mostly validated on image classiﬁcation datasets.
Applying zero-shot NAS to object detection backbone design is still an intact challenge.
In this work, we present the ﬁrst effort of introducing zero-shot NAS technique to design efﬁcient object detection backbones.
We show that directly transferring existing zero-shot NAS methods from image classiﬁcation to detection backbone design will encounter fundamental difﬁculties.
While image classiﬁcation network only needs to predict the class probability, object detection network needs to additionally predict the bounding boxes of multiple objects, making the direct architecture transfer sub-optimal.
To this end, a novel zero-shot NAS method, termed ZenDet, is proposed for searching object detection backbones.
The key idea behind ZenDet is inspired by the Principle of Maximum Entropy (PME) (Reza, 1994; Kullback, 1997; Brillouin, 2013).
Informally speaking, when a network is formulated as an information processing system, its capacity is maximized when its differential entropy (Shannon, 1948) achieves maximum under budget constraints, leading to a better feature extractor for object detection.
Based on this observation, ZenDet maximizes the differential entropy of detection backbones by searching for the optimal conﬁguration of network depth and width without training network parameters.
The above strategy raises two technical challenges.
The ﬁrst challenge is how to estimate the differential entropy of a deep network .
The exact computation of differential entropy requires knowing the precise probability distribution of deep features in high dimensional space which is difﬁcult to estimate in practice.
To address this issue, ZenDet estimates the Gaussian upper bound of the differential entropy which only requires computing the variance of the feature maps.
The second challenge is how to efﬁciently capture objects of different sizes .
In object detection benchmark datasets such as MS COCO Lin et al.
(2014), the distribution of object size is data-dependent and non-uniform.
To bring in this prior knowledge in backbone design, we introduce the Multi-Scale
Entropy Prior (MSEP) in backbone entropy estimation to capture different-scale objects.
We ﬁnd that the MSEP improves the detection performance signiﬁcantly.
The overall computation of ZenDet only requires one forward inference of the detection backbone at initialization therefore is nearly zero-cost comparing to previous backbone NAS methods.
The main contributions of this work are summarized as follows:
• Based on the entropy theory, the multi-scale entropy prior is present to rank the expressivity of the backbone instead of training on the target datasets, speeding up searching.
• While using less than one GPU day and 2GB GPU memory, ZenDet achieves competitive performance over other NAS methods on COCO with at least 50x times faster.
• ZenDet is the ﬁrst zero-shot NAS method designed for object detection with SOTA performance in multiple benchmark datasets under multiple popular detection frameworks.
R ELATED WORK Backbone for Object Detection
Recently, object detectors composed of backbone, neck and head have become increasingly popular due to their effectiveness and high performance (Lin et al., 2017a;b; Tian et al., 2019; Li et al., 2020; 2021).
Prevailing detectors directly use the backbone designed for image classiﬁcation to extract multi-scale features from an image, such as ResNet (He et al., 2016), ResNeXt (Xie et al., 2017) and Deformable Convolutional Network (DCN) (Zhu et al., 2019).
Nevertheless, the backbone migrated from image classiﬁcation may be suboptimal in object detection (Ghiasi et al., 2019).
To tackle the gap, many architectures are end-to-end designed for object detection, including Stacked Hourglass Newell et al.
FishNet Sun et al.
DetNet Li et al.
HRNet Wang et al.
(2020a) and so on.
Albeit with improved performance, these hand-crafted detection architectures heavily rely on human labor and tedious trial-and-error processes.
Neural Architecture Search
Neural Architecture Search (NAS) is initially developed to automatically design network architectures for image classiﬁcation models (Zoph et al., 2018;
Liu et al., 2018; Real et al., 2019;
Cai et al., 2019; Tan & Le, 2019; Lin et al., 2021).
Using NAS to design object detection models has not been well studied.
Currently, existing detection NAS methods are all training-based methods.
Some methods focus on searching detection backbones, such as DetNAS (Chen et al., 2019b), SpineNet (Du et al., 2020) and SP-NAS (Jiang et al., 2020), while the others focus on searching FPN neck, such as NAS-FPN (Ghiasi et al., 2019), NAS-FCOS (Wang et al., 2020b) and OPANet (Liang et al., 2021).
These methods require training and evaluation on the target datasets which is intensive in computation.
ZenDet distinguishes itself as being the ﬁrst zero-shot NAS method for the backbone design of object detection.
P RELIMINARY
In this section, we ﬁrst formulate a deep network as a system with continuous state space.
Then we deﬁne the differential entropy of this system and show how to estimate this entropy via its Gaussian upper bound.
Finally we introduce the basic concept of vanilla network search space for designing our detection backbones.
Continuous State Space of Deep Networks
A deep network F ( · ) : R d −→ R maps an input image x ∈ R d to its label y ∈ R .
The topology of a network can be abstracted as a graph G =
( V , E ) where the vertex set V consists of neurons and the edge set E consists of spikes between neurons.
For any v ∈ V and e ∈ E , h ( v ) ∈ R and h ( e ) ∈ R present the values endowed with each vertex v and each edge e respectively.
The set S =
{ h ( v ) , h ( e ) : ∀ v ∈ V , e ∈ E} deﬁnes the continuous state space of the network F .
According to the Principle of Maximum Entropy, we want to maximize the differential entropy of network F , under some given computational budgets.
The entropy H ( S ) of set S measures the total information contained in the system (network) F , including the information contained in the latent features H ( S v ) =
{ h ( v ) : v ∈ V} and in the network parameters H ( S e )
= { h ( e ) : e ∈ E} .
As for object detection backbone design, we only care about the entropy of latent features H ( S v ) rather than the entropy of network parameters H ( S e ) .
Informally speaking, H ( S v ) measures the feature representation power of F while H ( S e ) measures the model complexity of F .
Therefore, in the remainder of this work, the differential entropy of F refers to the entropy H ( S v ) by default.
Entropy of Gaussian Distribution
The differential entropy of Gaussian distribution can be found in many textbooks such as (Norwich, 1993).
Suppose x is sampled from Gaussian distribution
N ( µ, σ 2 ) .
Then the differential entropy of x is given by
[Equation]
From Eq.
1, the entropy of Gaussian distribution only depends on the variance.
In the following, we will use H ( x ) instead of H ∗ ( x ) as constants do not matter in our discussion.
Gaussian Entropy Upper Bound Since the probability distribution P ( S v ) is a high dimensional function, it is difﬁcult to compute the precise value of its entropy directly.
Instead, we propose to estimate the upper bound of the entropy, given by the following well-known theorem (Cover &
Thomas, 2012):
Theorem 1.
For any continuous distribution P ( x ) of mean µ and variance σ 2 , its differential entropy is maximized when P ( x ) is a Gaussian distribution N ( µ, σ 2 ) .
Theorem 1 says that the differential entropy of a distribution is upper bounded by a Gaussian distribution with the same mean and variance.
Combining this with Eq.
(1), we can easily estimate the network entropy H ( S v ) by simply computing the feature map variance and then use Eq.
(1) to get the Gaussian entropy upper bound for the network.
Vanilla Network Search Space
Following previous works, we design our backbones in the vanilla convolutional network space (Li et al., 2018;
Chen et al., 2019b; Du et al., 2020; Lin et al., 2021) because this space is widely adopted in detection backbones and is used as a prototype in theoretical literature (Poole et al., 2016; Serra et al., 2018; Hanin & Rolnick, 2019).
A vanilla network is stacked by multiple convolutional layers followed by RELU activations.
Auxiliary components such as residual link (He et al., 2016), Batch Normalization (BN) (Ioffe & Szegedy, 2015) and
Squeeze-and-Excitation (SE) (Hu et al., 2018) are all removed during the search and only during the search (More details in Appendix E).
These removed auxiliary components are plugged in back after the search.
Therefore, the ﬁnal architecture for training still has these components.
Consider a vanilla convolutional network with D layers of weights W 1 , ... , W D whose output feature maps are x 1 , ... , x D .
The input image is x 0 .
Let φ ( · ) denote the RELU activation function.
Then the forward inference is given by
[Equation]
For sake of simplicity, we set the bias of the convolutional layer to zero.
[Figure]
𝒩(
[Figure] W ! ~𝒩(
[Figure] [Figure] +𝑥 # [Figure]
𝑥 # 𝛾 # +𝑥 #$% 𝛾 !
[Figure]
𝑥 ! Conv 𝒉 ! 𝑥 "
[Figure]
𝒩(0,1)
W ! ~𝒩(
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
↓
[Figure]
Z EN D ET -NAS FOR O BJECT D ETECTION
In this section, we ﬁrst describe how to compute the differential entropy of the single-scale feature for very deep vanilla networks with the re-scaling trick.
Then we introduce the Multi-Scale Entropy
Prior (MSEP) to better capture the prior distribution of object size in ZenDet.
Finally, we present the complete ZenDet-NAS under the Evolutionary Algorithm (EA) framework.
4. S INGLE -S CALE E NTROPY FOR V ERY D EEP B ACKBONES
To compute the entropy of the detection backbone, we initialize all backbone parameters by the standard Gaussian distribution N (0 , 1) .
Then we randomly generate an image ﬁlled with the standard
Gaussian noise and perform forward inference.
Based on the discussion in Section 3, the (Gaussian upper bound) entropy H ( F ) of the network F is then given by
[Equation]
Please note that the variance is computed on the last pre-activation feature map h D .
For very deep networks, directly using Eq.
(3) might cause a numerical overﬂow.
The same problem is also observed in Zen-NAS (Lin et al., 2021).
Inspired by the analysis in Zen-NAS where the authors show that the gradient norm is scale-invariant to BN layers after variance compensation, we propose a more straightforward solution.
We directly re-scale each feature map x l by some constants γ l during inference, that is x l = φ ( h l ) /γ l , and then compensate the entropy of the network by
[Equation]
The values of γ l can be arbitrarily given as long as the forward inference does not overﬂow or underﬂow.
In practice, we ﬁnd that simply setting γ l to the Euclidean norm of the feature map works well.
The overall process is illustrated in Figure 1 (a).
Comparison with Zen-NAS Although the deﬁnition of vanilla network and solution to overﬂow are similar, the principles of single-scale entropy and Zen-NAS are fundamentally different.
ZenNAS is a gradient-based method for classiﬁcation, while our method is an entropy-based method for detection.
In implementation, Zen-NAS needs to process two forward propagations once, while our method only needs one propagation, doubling the search speed.
4. M ULTI -S CALE E NTROPY P RIOR FOR OBJECT DETECTION
In real-world images, the distribution of object size is not uniform.
To bring in this prior knowledge, the detection backbone has 5 stages where each stage downsamples the feature resolution to half.
The MSEP collects feature maps from the ﬁnal output of each stage and weighted-sum the corresponding feature map entropies as a new measurement.
We name this new measurement as multi-scale entropy.
The overall process is illustrated in Figure 1 (b).
In this ﬁgure, the backbone extracts multi-scale features C = (C1 , C2 , ..., C5) at different resolutions.
Then the FPN neck fuses
C as input features P = (P1 , P2 , ..., P7) for the detection head.
The multi-scale entropy Z ( F ) of backbone F is then deﬁned by
[Equation]
where H ( Ci ) is the entropy of Ci for i = 1 , 2 , · · · , 5 .
The weights α =
( α 1 , α 2 , · · · , α 5 ) store the multi-scale entropy prior to balance the expressivity of different scale features.
Weights α selection strategy
As a concrete example in Fig. 1(b), the parts of P3 and P4 are generated by up-sampling of P5 , and P6 and P7 are directly generated by down-sampling of P (generated by C5 ). Meanwhile, based on the fact that C5 carries sufﬁcient context for detecting objects on various scales (Chen et al., 2021a), C5 is important in the backbone search, so it is good to set a larger value for the weight α 5 . Then, different combinations of α and correlation analysis are explored in Appendix D, indicating that α = (0 , 0 , 1 , 1 , 6) is good enough for the FPN structure.
4. Z EN D ET -NAS IN E VOLUTIONARY A LGORITHM F RAMEWORK Algorithm
Require: Search space S , inference budget B , maximal depth L , total number of iterations T , evolutionary population size N , initial structure F 0 , ﬁne-search ﬂag Flag .
Ensure: NAS-designed ZenDet backbone F ∗ .
1: Initialize population P = { F 0 } , Flag=False . 2: for t = 1 , 2 , · · · , T do 3: if t equals to T/ 2 then 4:
Keep top 10 networks of highest multi scale entropy P remove the others.
Set Flag =
True .
6: end if 7:
Randomly select F t ∈ P .
Mutate ˆ F t = MUTATE
( F t , S , Flag ) 9: if ˆ F t exceeds inference budget or has more than L layers then 10:
Do nothing.
11: else 12:
Get multi-scale entropy Z ( ˆ F t ) .
Append ˆ F t to P .
14: end if 15:
Remove networks of the smallest multi-scale entropy if the size of P exceeds B .
16: end for 17: Return F ∗ , the network of the highest multi-scale entropy in P .
Require:
Structure F t , search space S , ﬁnesearch ﬂag Flag .
Ensure: Randomly mutated structure ˆ F t .
Uniformly select a
: else
6: end if : Return
[Figure]
Combining all above, we present our ZenDet-NAS in Algorithm 1.
The ZenDet-NAS maximizes the multi-scale differential entropy of detection backbones using Evolutionary Algorithm (EA).
To improve the evolution efﬁciency, a coarse-to-ﬁne strategy is proposed to reduce the search space gradually.
First, we randomly generate N seed architectures to ﬁll the population P .
As shown in
Figure 2, a seed architecture F t consists of a sequence of building blocks such as ResNet block (He et al., 2016) or MobileNet block
(Sandler et al., 2018).
Then we randomly select one block and replace it with its mutated version.
We use coarse-mutation in the early stage of EA and switch to ﬁne-mutation after T/ 2 EA iterations.
In the coarse-mutation, the block type, kernel size, depth and width are mutated randomly; in the ﬁne-mutation, only kernel size and width are mutated.
After the mutation, if the inference cost of the new structure ˆ F t does not exceed the budget (e.g.,
FLOPs, parameters and latency) and its depth is smaller than budget L , ˆ F t is appended into the population P .
The maximal depth L prevents the algorithm from generating over-deep structures.
During EA iterations, the population is maintained to a certain size by discarding the worst candidate of the smallest multi-scale entropy.
At the end of ZenDet-NAS, the backbone with the highest multiscale entropy is returned.
E XPERIMENTS
In this section, we ﬁrst describe detail settings for searching and training with ZenDet-NAS.
Then in subsection 5.2, we apply ZenDet-NAS to design better ResNet-like backbones on COCO dataset (Lin et al., 2014).
We align the inference budget with ResNet-50/101.
The performance of
ZenDet and ResNet are compared under multiple detection frameworks including RetinaNet (Lin et al., 2017b), FCOS (Tian et al., 2019), and GFLV2 (Li et al., 2021).
For fairness, we use the same training setting in all experiments for all backbones.
In subsection 5.3, we compare the search cost of ZenDet to SOTA NAS methods for object detection.
Subsection 5.4 reports the ablation studies of different components in ZenDet-NAS.
Finally, subsection 5.5 veriﬁes the transferability of ZenDets on several detection datasets and segmentation tasks.
5. E XPERIMENT S ETTINGS Searching Details
In ZenDet-NAS, the evolutionary population N is set to 256.
The total EA iterations T = 96000 .
Following the previous designs (Chen et al., 2019b; Jiang et al., 2020; Du et al., 2020), ZenDet is optimized for FLOPs.
The resolution for computing entropy is 384 × 384 .
Dataset and Training Details
We evaluate detection performance on COCO (Lin et al., 2014) using the ofﬁcial training/testing splits.
The mAP is evaluated on val 2017 by default and GFLV is additionally evaluated on test-dev 2007 following common practice.
All models are trained from scratch (He et al., 2019) for 6X (73 epochs) on COCO.
Following the Spinenet (Du et al., 2020), we use multi-scale training and Synchronized Batch Normalization (SyncBN).
For VOC dataset, train-val 2007 and train-val 2012 are used for training, and test 2007 for evaluation.
For image classiﬁcation, all models are trained on ImageNet-1k (Deng et al., 2009) with a batch size of 256 for 120 epochs.
Other setting details can be found in Appendix A.
5. D ESIGN B ETTER R ES N ET - LIKE B ACKBONES
We search efﬁcient ZenDet backbones for object detection and align with ResNet-50/101 in Table 1.
ZenDet-S uses 60% less FLOPs than ResNet-50; ZenDet-M is aligned with ResNet-50 with similar FLOPs and number of parameters as [-ResNet-
 ;-]ResNet-50; ZenDet-L is aligned with ResNet-101.
The feature dimension in the FPN and heads is set to 256 for ZenDet-M and ZenDet-L but is set to 192 for
ZenDet-S.
The ﬁne-tuned results of models pre-trained on ImageNet-1k are reported in Appendix C.
In Table 1, ZenDet outperforms ResNet by a large margin.
The improvements are consistent across three detection frameworks.
Particularly, when using the newest framework GFLV2, ZenDet improves COCO mAP by +2% at the similar FLOPs of ResNet-50, and speeds up the inference by 1.54x times faster at the same accuracy as ResNet-50.
Figure 3 visualizes the comparison in Table 1.
Remark Please note that we did not copy the numbers of baseline methods reported in previous works in Table 1 because the mAP not only depends on the architecture but also depends on the training schedule, such as training epochs, learning rate, pre-training and so on.
Therefore, for a fair [Table] [Figure] comparison, all models in Table 1 are trained by the same training schedule.
For comparison with numbers reported in previous works, see subsection 5. [Figure]
5. C OMPARISON WITH SOTA NAS M ETHODS
[Table]
In Table 2, we compare ZenDet with SOTA NAS methods for the backbone design in object detection.
We directly use the numbers reported in the original papers.
Since each NAS method uses different design spaces and training settings, it is impossible to make an absolutely fair comparison for all methods that everyone agrees with.
Nevertheless, we list the total search cost, mAP and [Table]
FLOPs of the best models reported in each work.
This gives us an overall impression of how each
NAS method works in real-world practice.
From Table 2, ZenDet is the only zero-shot (trainingfree) method with 48 . 0% mAP on COCO, using 0 .
6 GPU days of search.
SpineNet (Du et al., 2020) achieves a slightly better mAP with 2x more FLOPs.
It uses 100 TPUv3 for searching but the total search cost is not reported in the original paper.
ZenDet achieves better mAP than DetNAS (Chen et al., 2019b) and SP-NAS(Jiang et al., 2020) while being 50 ∼ 100 times faster in search.
To further fairly compare different backbones under the same training settings, we train backbones designed by ZenDet-NAS and previous backbone NAS methods in Table 3.
Because the implementation of SP-NAS is not open-sourced, we re-train ZenDet, DetNAS and SpineNet on COCO from scratch.
Table 3 shows that ZenDet requires fewer parameters and has a faster inference speed on
V100 when achieving the competitive performance over DetNAS and SpineNet on COCO.
5. A BLATION S TUDY AND A NALYSIS
[Table]
Table 4 reports the ZenDet backbones searched by different evolutionary strategies and whether using multi-scale entropy prior.
The COCO mAPs of models trained in two detection frameworks (YOLOF and FCOS) are reported in the right big two columns.
YOLOF models are trained by 12 epochs with ImageNet pre-trained initialization, while FCOS models are trained with the 3X training epochs.
We also compare their image classiﬁcation ability on ImageNet-1k.
All models are constrained by the FLOPs less than 4.4 G while the number of parameters is not constrained.
More details about the searching process and architectures could be found in Appendix G, H
Single-scale score Compared to ResNet-50, the model searched by single-scale entropy score obtains +0 .
7% accuracy gain on ImageNet, +2% mAP gain with FPN-free YOLOF and +0 . 8% mAP gain with FPN-based FCOS. Meanwhile, the model searched by Zen-Score achieves +0 . 9% accuracy gain on ImageNet, +1 .
1% mAP gain with YOLOF and +0 . 1% mAP gain with FCOS.
When using multi-scale entropy, both single-scale model and multiscale model get similar accuracy on ImageNet.
The single-scale model uses 2X more parameters than the multi-scale model under the same FLOPs constraint.
In terms of mAP, multi-scale model outperforms single-scale model by +0 .
3% on COCO with YOLOF and 0 .
6% on COCO with FCOS.
From the last row of Table 4, the coarse-to-ﬁne mutation further enhances the performance of multiscale entropy prior, and the overall improvement over ResNet-50 is +1 .
1% on ImageNet-1k, +2 .
5% on COCO with YOLOF and +2 .
0% on COCO with FCOS.
To further study the correlations between mAP and scores, models during the search are trained and the results are exhibited in Figure 4. The right part of Figure 4 indicates that the mAP positively correlates with the multi-scale entropy score.
The left part of Figure 4 reveals that the single-scale entropy score cannot represent the mAP well, so multi-scale entropy is necessary for detection tasks.
By analysing the structures in Appendix G, the computation of single-scale models is concentrated in the last stage C5 , ignoring the C3 and C4 stages, and leading the worse multi-scale score.
Instead, multi-scale models allocate more computation to the previous stages to enhance the expressivity of C3 and C4 , which improve the multi-scale score.
[Figure]
5. T RANSFER L EARNING
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
VOC and Cityscapes
To evaluate the transferability of ZenDet in different datasets, we transfer the FCOS-based ZenDet-M to VOC and Cityscapes dataset, as shown in the upper half of Table 5.
The models are ﬁne-tuned after pre-trained on ImageNet.
Comparing to ResNet-50, ZenDet-M achieves +4 .
1% better mAP in VOC, +1 .
1% better mAP in Cityscape.
Instance Segmentation
The lower half of Table 5 reports results of Mask R-CNN (He et al., 2017) and SCNet (Vu et al., 2021) models for the COCO instance segmentation task with 6X training from scratch.
Comparing to ResNet-50, ZenDet-M achieves better AP and mask AP with similar model size and FLOPs on Mask RCNN and SCNet.
C ONCLUSION
In this paper, we propose a zero-shot NAS method termed ZenDet-NAS for designing high performance backbones for object detection.
While achieving better or competitive detection accuracy, the search speed of ZenDet-NAS is several orders of magnitude faster than previous training-based NAS methods.
Within one GPU day, the ZenDet automatically designed by ZenDet-NAS is signiﬁcantly more efﬁcient than popular SOTA backbones for object detection in terms of FLOPs and inference latency.
Extensive experiments and analyses on various datasets validate its excellent transferability.
In the future, we will generalize ZenDet-NAS to search for detection backbone and FPN neck a
A T RAINING D ETAILS Searching Details
In ZenDet-NAS, the evolutionary population N is set as 256 while total iterations T = 96000 .
Residual blocks and bottleneck blocks are utilized as searching space when comparing with ResNet series backbone (He et al., 2016).
Following the previous designs (Chen et al., 2019b; Jiang et al., 2020; Du et al., 2020), ZenDet is optimized for the budget of FLOPs according to the target networks, i.e., ResNet-50 and ResNet-101.
To balance the computational complexity and large resolution demand, the resolution in search is set as 384 × 384 for ZenDet.
When starting the search, the initial structure is composed of 5 downsampling stages with small and narrow blocks to meet the reasoning budget.
In the mutation, whether the coarse-mutation or ﬁnemutation, the width of the selected block is mutated in a given scale { 1 / 1 . 5 , 1 / 1 . 25 , 1 , 1 . 25 , 1 . 5 , 2 } , while the depth increases or decreases 1 or 2.
The kernel size is searched in set { 3 , 5 } .
Note that blocks deeper than 10 will be divided into 2 blocks equally to enhance the diversity.
Dataset and Training Details
For object detection, trainval35k with 115K images in the COCO dataset is mainly used for training.
With the single-scale testing of resolution 1333 × 800 , COCO mAP results are reported on the val 2017 for most experiments and the test-dev 2007 for GFLV results in Table 1.
When training on the COCO dataset, the initial learning rate is set to 0 .
02 , and decays two times with the ratio of 0 . 1 during training.
SGD is adopted as optimizer with momentum 0.9; weight decay of 10 − 4 ; batch size of 16 (on 8 Nvidia V100 GPUs); patch size of 1333 × 800 .
Additionally, multi-scale training and Synchronized Batch Normalization (SyncBN) are adopted to enhance the stability of the scratch training without increasing the complexity of inference.
Training from scratch is used to avoid the gap between ImageNet pre-trained model, to ensure a fair comparison with baselines.
3X learning schedule is applied for the ablation study with a multi-scale range between [0 . 8 , 1 . 0] (36 epochs, decays at 28 and 33 epochs), and 6X learning schedule for the
SOTA comparisons with the range between [0 . 6 , 1 . 2] (73 epochs, decays at 65 and 71 epochs).
All object detection training is produced under mmdetection (Chen et al., 2019a) for fair comparisons, and hyperparameters not mentioned in the paper are always set to default values in mmdection.
For image classiﬁcation, all models are trained on ImageNet-1K with a batch size of 256 forepochs.
When training on ImageNet-1K, We use SGD optimizer with momentum 0.9; cosine learning rate decay (Loshchilov & Hutter, 2017); initial learning rate 0.1; weight decay 4 × 10 − 5 .
B Z EN D ET -NAS FOR MOBILE DEVICE
[Table] For mobile-size object detection, we explore building ZenDet-MB with MobileNetV2 (Sandler et al., 2018) blocks, using the inverted bottleneck block with expansion ratio of 1/3/6. The weight ratio α is still set as 1:1:6 and other searching settings are the same as the Resnet-like searching.
In Table.
6, ZenDet-MB use less computation and parameters but outperform MobileNetV2 by 1%
AP with similar inference time on Google Pixel 2 phone.
Additionally, inserting SE modules to
ZenDet-MB-M could improve the mAP by 0.4%.
In the main body of the paper, training from scratch is used to avoid the gap between ImageNet pre-trained model, to ensure a fair comparison with baselines He et al.
Since 6X training from scratch inevitably consumes 3 times more time than 2X pre-trained training, we use the ImageNet pre-trained model to initialize the ZenDet-M in various heads, including RetinaNet, FCOS and GFLV2.
As present in Table.
1, 7, whether using training from scratch or ImageNet pre-training,
ZenDet could outperform ResNet-50 in the three popular FPN-based frameworks by large margins. [Table]
D W EIGHTS A RRANGEMENT
[Table]
R
None 83.6G 23.5M 38.
ZenDet 1:1: 84.4G 11.5M 37.
46.6 ZenDet 1:1: 84.8G 13.4M 37.
47.8 ZenDet 1:1: 85.9G 17.2M 38.
48.6 ZenDet 1:1: 88.7G 29.4M 39. [Table] [Table] [Table]
50.0 ZenDet 1:1: 89.9G 31.7M 39.
[Table] [Table]
ZenDet 1:4: 86.3G 10.9M 35. [Table] 44.9 ZenDet 4:1: 86.1G 11.1M 33. [Figure]
[Figure]
In Table 8, we tune the different arrangements of multi-scale weights in a wide range.
Seven multiscale weight ratios are used to search different models, and all models are trained on the COCO dataset with FCOS and 3X learning schedule.
Table 8 shows that if the same weights are arranged to C3 - C5 , the performance of ZenDet on COCO is worse than ResNet-50.
Considering the importance of C5 (discussed in Section 4.2), we increase the weight of C5 , and ZenDet’s performance continues to improve.
To further explore the correlations between mAP and scores, we use the seven weight ratios to calculate the different scores of each model, along with the single-scale weight ratio of 0:0:1.
The correlations between mAP and different scores are plotted in Figure 5. Taking the results in Table 8 and Figure 5, we conﬁrm the ratio of 1:1:6 may be good enough for the current FPN structure.
E D ISCUSSION ABOUT AUXILIARY COMPONENTS
[Figure]
In this section, we explain why auxiliary components are removed during the search and plugged in back for training.
Common used three auxiliary components are built in Figure 6 to verify effects on feature variance.
BN
BN is a widely used method to re-centre and re-scale the feature to make the network converge faster and more stable.
In Figure 6 (a), BN normalize entropy adaptively to the network width (which can be related to output variance).
When BN is presented, networks of different widths will have the same entropy value.
Hence, BN must be removed when calculating the entropy.
Residual link
Building the structure like Figure 6 (b), the variances with or without residual link are 767 or 751 respectively.
The residual link brings less than 2% difference in entropy score.
The results means that residual link has little effect on entropy, so it could be removed during the search.
SE
SE module is used to adaptively recalibrate channel-wise feature responses by explicitly modelling interdependencies between channels Hu et al.
Note that the input is a zero-mean distribution in our computation of the entropy.
If SE module is used in the network, the output after global pooling is equal to 0.
So, the ﬁnal output of SE module is equal to 0.5, which loses the ability to model interdependencies between channels.
To sum up, removing these auxiliary components makes our method stable and applicable to most single-branch feed-forward networks.
F C OMPARISON WITH Z ERO -S HOT P ROXIES FOR I MAGE C LASSIFICATION
We compare ZenDet with architectures designed by zero-shot proxies for image classiﬁcation in previous works, including SyncFlow (Tanaka et al., 2020), NASWOT (Mellor et al., 2021), ZenNAS (Lin et al., 2021).
For a fair comparison, All methods use the same search space, FLOPs budget 91 G , searching strategy and training schedule.
All searched backbones are trained on COCO with the FCOS head and 3X training from scratch.
The results are reported in Table.
Among these methods, SyncFlow and NASWOT perform worse than ResNet-50 on COCO albeit they show competitive performance in image classiﬁcation tasks.
Zen-NAS achieves competitive performance over ResNet-50.
The ZenDet outperforms Zen-NAS by +1 .
3% mAP with slightly fewer FLOPs and nearly one third of parameters.
[Table]
G V ISUALIZATION OF S EARCHING PROCESS
[Figure]
[Figure]
[Figure]
The visualizations of searching process on different entropy scores are shown in Figure 7, 8, 9.
We compare the change of layers, FLOPs and parameters during different iterations.
Visualizations prove our assumptions in the main body of the paper
[Figure]
H D ETAIL S TRUCTURE OF Z EN D ETS
We list detail structure in Table 1, 4.
The ’block’ column is the block type. ’
Conv’ is the standard convolution layer followed by BN and
RELU. ’
ResBlock’ is the residual bottleneck block used in ResNet-50 and is stacked by two Blocks in our design. ’kernel’ is the kernel size of kxk convolution layer in each block. ’
in’, ’out’ and ’bottleneck’ are numbers of input channels, output channels and bottleneck channels respectively.
’stride’ is the stride of current block. ’
# layers’ is the number of duplication of current block type.
[Table] [Table] [Table]
C ResBlock C ResBlock
C
{+ResBlock
+}
C
[Table] [Table]
[Table]