L EARNING V ERSATILE N EURAL A RCHITECTURES BY P ROPAGATING N ETWORK C ODES
A BSTRACT
This work explores how to design a single neural network capable of adapting to multiple heterogeneous vision tasks, such as image segmentation, 3D detection, and video recognition.
This goal is challenging because both network architecture search (NAS) spaces and methods in different tasks are inconsistent.
We solve this challenge from both sides.
We ﬁrst introduce a uniﬁed design space for multiple tasks and build a multitask NAS benchmark (NAS-Bench-MR) on many widely used datasets, including ImageNet, Cityscapes, KITTI, and HMDB51.
We further propose Network Coding Propagation (NCP), which back-propagates gradients of neural predictors to directly update architecture codes along the desired gradient directions to solve various tasks.
In this way, optimal architecture conﬁgurations can be found by NCP in our large search space in seconds.
Unlike prior arts of NAS that typically focus on a single task, NCP has several unique beneﬁts.
(1) NCP transforms architecture optimization from data-driven to architecture-driven, enabling joint search an architecture among multitasks with different data distributions.
(2) NCP learns from network codes but not original data, enabling it to update the architecture efﬁciently across datasets.
In addition to our NAS-Bench-MR, NCP performs well on other NAS benchmarks, such as NAS-Bench-201.
(4) Thorough studies of NCP on inter-, cross-, and intra-tasks highlight the importance of cross-task neural architecture design, i.e. , multitask neural architectures and architecture transferring between different tasks.
Code is available at github.com/dingmyu/NCP
I NTRODUCTION
Designing a single neural network architecture that adapts to multiple different tasks is challenging.
This is because different tasks, such as image segmentation in Cityscapes (Cordts et al., 2016) and video recognition in HMDB51 (Kuehne et al., 2011), have different data distributions and require different granularity of feature representations.
For example, although the manually designed networks ResNet (He et al., 2016) and HRNet (Wang et al., 2020a) work well on certain tasks such as image classiﬁcation on ImageNet, they deteriorate in the other tasks.
Intuitively, manually designing a single neural architecture that is applicable in all these tasks is difﬁcult.
Recently, neural architecture search (NAS) has achieved great success in searching network architectures automatically.
However, existing NAS methods (Wu et al., 2019; Liang et al., 2019; Liu et al., 2019b; Xie et al., 2018;
Cai et al., 2020; Yu et al., 2020; Shaw et al., 2019b;a) typically search on a single task.
Though works (Ding et al., 2021; Duan et al., 2021; Zamir et al., 2018) designed
NAS algorithms or datasets that can be used for multiple tasks, they still search different architectures for different tasks, indicating that the costly searching procedure needs to be repeated many times.
The problem of learning versatile neural architectures capable of adapting to multiple different tasks remains unsolved, i.e. , searching a multitask architecture or transferring architectures between different tasks.
In principle, it faces the challenge of task and dataset inconsistency.
Different tasks may require different granularity of feature representations, e.g. , the segmentation task requires more multi-scale features and low-level representations than classiﬁcation.
The key to solving task inconsistency is to design a uniﬁed architecture search space for multiple tasks.
In contrast to most previous works that simply extend search space designed for image classiﬁcation
[Figure]
for other tasks and build NAS benchmarks (Dong & Yang, 2020;
Ying et al., 2019; Siems et al., 2020) on small datasets ( e.g. , CIFAR10, ImageNet-16) with unrealistic settings, we design a multiresolution network space and build a multi-task practical NAS benchmark (NAS-Bench-MR) on four challenging datasets including ImageNet-224 (Deng et al., 2009), Cityscapes (Cordts et al., 2016), KITTI (Geiger et al., 2012), and HMDB51 (Kuehne et al., 2011).
Inspired by HRNet (Wang et al., 2020a), our network space is a multi-branch multi-resolution space that naturally contains various granularities of representations for different tasks, e.g. , high-resolution features (Wang et al., 2020a) for segmentation while low-resolution ones for classiﬁcation.
NAS-Bench-MR closes the gap between existing benchmarks and NAS in multi-task and real-world scenarios.
It serves as an important contribution of this work to facilitate future cross-task NAS research.
To solve the challenge of dataset inconsistency, in this work, we propose a novel predictorbased NAS algorithm, termed Network Coding Propagation (NCP), for ﬁnding versatile and tasktransferable architectures.
NCP transforms data-oriented optimization into architecture-oriented by learning to traverse the search space.
It works as follows.
We formulate all network hyperparameters into a coding space by representing each architecture hyper-parameter as a code, e.g. , ‘3,2,64’ denotes there are 3 blocks and each block contains 2 residual blocks with a channel width of 64.
We then learn neural predictors to build the mapping between the network coding and its evaluation metrics ( e.g. , Acc, mIoU, FLOPs) for each task.
By setting high-desired accuracy of each task and
FLOPs as the target, we back-propagate gradients of the learned predictor to directly update values of network codes to achieve the target.
In this way, good architectures can be found in several forward-backward iterations in seconds, as shown in Fig.
NCP has several appealing beneﬁts: (1) NCP addresses the data mismatch problem in multi-task learning by learning from network coding but not original data.
(2) NCP works in large spaces in seconds by back-propagating the neural predictor and traversing the search space along the gradient direction.
(3) NCP can use multiple neural predictors for architecture transferring across tasks, as shown in Fig. 1(a), it adapts an architecture to a new task with only a few iterations.
In NCP, the multi-task learning objective is transformed to gradient accumulation across multiple predictors, as shown in Fig. 1(b), making NCP naturally applicable to various even conﬂicting objectives, such as multi-task structure optimization, architecture transferring across tasks, and accuracy-efﬁciency trade-off for speciﬁc computational budgets.
Our main contributions are three-fold.
(1) We propose Network Coding Propagation (NCP), which back-propagates the gradients of neural predictors to directly update architecture codes along desired gradient directions for various objectives.
(2) We build NAS-Bench-MR on four challenging datasets under practical training settings for learning task-transferable architectures.
We believe it will facilitate future NAS research, especially for multi-task NAS and architecture transferring across tasks.
(3) Extensive studies on inter-, intra-, cross-task generalizability show the effectiveness of NCP in ﬁnding versatile and transferable architectures among different even conﬂicting objectives and tasks
R ELATED W ORK
Neural Architecture Search Spaces and Benchmarks.
Most existing NAS spaces (Jin et al., 2019; Xu et al., 2019; Wu et al., 2019;
Cai et al., 2019; Xie et al., 2018; Stamoulis et al., 2019;
[Table]
[Table]
Mei et al., 2020; Guo et al., 2020; Dai et al., 2020) are designed for image classiﬁcation using either a single-branch structure with a group of candidate operators in each layer or a repeated cell structure, e
[Table]
g. , Darts-based (Liu et al., 2019b) and MobileNet-based (Sandler et al., 2018) search spaces.
Based on these spaces, several NAS benchmarks (Ying et al., 2019; Dong & Yang, 2020;
Siems et al., 2020; Duan et al., 2021) have been proposed to pre-evaluate the architectures.
However, the above search spaces and benchmarks are built either on proxy settings or small datasets, such as Cifar-10 and ImageNet-16 ( 16 × 16 ), which is less suitable for other tasks that rely on multiscale information.
For those tasks, some search space are explored in segmentation (Shaw et al., 2019a;
Liu et al., 2019a; Nekrasov et al., 2019;
Lin et al., 2020; Chen et al., 2018) and object detection (Chen et al., 2019; Ghiasi et al., 2019; Du et al., 2020; Wang et al., 2020b) by introducing feature aggregation heads ( e.g. , ASPP (Chen et al., 2017)) for multi-scale information.
Nevertheless, the whole network is still organized in a chain-like single branch manner, resulting in sub-optimal performance.
Another relevant work to ours is NAS-Bench-NLP Klyuchnikov et al.
(2020), which constructs a benchmark with 14k trained architectures in search space of recurrent neural networks on two language modeling datasets.
Compared to previous spaces, our multi-resolution search space, including searchable numbers of resolutions/blocks/channels, is naturally designed for multiple vision tasks as it contains various granularities of feature representations.
Based on our search space, we build NAS-Bench-MR for various vision tasks, including classiﬁcation, segmentation, 3D detection, and video recognition.
Detailed comparisons of NAS benchmarks can be found in Tab. 1.
Neural Architecture Search Methods.
Generally, NAS trains numerous candidate architectures from a search space and evaluates their performance to ﬁnd the optimal architecture, which is costly.
To reduce training costs, weight-sharing NAS methods (Liu et al., 2019b; Jin et al., 2019; Xu et al., 2019; Cai et al., 2019; Guo et al., 2020; Li & Talwalkar, 2020) are proposed to jointly train a large number of candidate networks within a super-network.
Different searching strategies are employed within this framework such as reinforcement learning (Pham et al., 2018), importance factor learning (Liu et al., 2019b;
Cai et al., 2019; Stamoulis et al., 2019; Xu et al., 2019), path sampling
(You et al., 2020; Guo et al., 2020; Xie et al., 2018), and channel pruning (Mei et al., 2020; Yu & Huang, 2019).
However, recent analysis (Sciuto et al., 2020; Wang et al., 2021) shows that the magnitude of importance parameters in the weight-sharing NAS framework does not reﬂect the true ranking of the ﬁnal architectures.
Without weight-sharing, hyperparameter optimization methods (Tan & Le, 2019;
Radosavovic et al., 2020; Baker et al., 2017; Wen et al., 2020; Lu et al., 2019;
Luo et al., 2020;
Chau et al., 2020; Yan et al., 2020) has shown its effectiveness by learning the relationship between network hyperparameters and their performance.
For example, RegNet (Radosavovic et al., 2020) explains the widths and depths of good networks by a quantized linear function.
Predictor-based methods (Wen et al., 2020; Luo et al., 2020; Chau et al., 2020; Yan et al., 2020; Luo et al., 2018) learn predictors, such as Gaussian process (Dai et al., 2019) and graph convolution networks (Wen et al., 2020), to predict the performance of all candidate models in the search space.
A subset of models with high predicted accuracies is then trained for the ﬁnal selection.
Our Network Coding Propagation (NCP) belongs to the predictor-based method, but is different from existing methods in: (1) NCP searches in a large search space in seconds without evaluating all candidate models by back-propagating the neural predictor and traversing the search space along the gradient direction.
(2) Beneﬁting from the gradient back-propagation in network coding space,
NCP is naturally applicable to various objectives across different tasks, such as multi-task structure
Neural Predictor 27-dimensional Coding
Prediction
Target
Coding
Multi-resolutionArchitecture
Number of channels (i, c, o) Number of blocks (b)
Number of residual units (n)
ParallelModule
FusionModule
ParallelModuleFusionModule b 3 =2, n 3 =[1,1,2], c 3 =[32,64,32]
Example of the Stage4x 8x 16x 16x 32x
Decrease
Increase Forward t acc t flops t task-spec
Editing
L task-spec
L FLOPs
L Acc
Backward p flops p acc p task-spec
In-layers
Stage Stage
Stage Stage
Out-layer 8x 4x 2x
Out-layer
Stage Stage
Stage Stage
In-layer In-layer
Fixed optimization, architecture transferring, and accuracy-efﬁciency trade-offs.
(3) Different from Luo et al.
; Baker et al.
(2017) jointly train an encoder, a performance predictor, and a decoder to minimize the combination of performance prediction loss and structure reconstruction loss, we learn and inverse the neural predictor directly in our coding space to make the gradient updating and network editing explicit and transparent.
M ETHODOLOGY
NCP aims to customize speciﬁc network hyperparameters for different optimization objectives, such as single- and multi-task learning and accuracy-efﬁciency trade-offs, in an architecture coding space.
An overview of NCP is shown in Fig.
In this section, we ﬁrst discuss learning efﬁcient models on a single task with two strategies and then show that it can be easily extended to multi-task scenarios.
Lastly, we demonstrate our multi-resolution coding space and NAS-Bench-MR.
3. N ETWORK C ODING P ROPAGATION
For each task, we learn a neural predictor F ( · ) that projects an architecture code e to its ground truth evaluation metrics, such as accuracy g acc and FLOPs g ﬂops .
Given an initialized and normalized coding e , the predicted metrics and the loss to learn the neural predictor are represented by:
[Equation]
where W is the weight of the neural predictor, which is ﬁxed after learning.
p acc , p ﬂops , and p task-spec denote the predicted accuracy, FLOPs, and task-speciﬁc metrics (if any), L2 is the L2 norm.
Note that although the FLOPs can be directly calculated from network codes, we learn it to enable the backward gradient upon FLOPs constraints for accuracy-efﬁciency trade-offs.
NCP edits the network coding e by back-propagating the learned neural predictor W to maximize/minimize the evaluation metrics ( e.g. , high accuracy and low FLOPs) along the gradient directions.
According to the number of dimensions to be edited, two strategies are proposed to propagate the network coding: continuous propagation and winner-takes-all propagation, where the ﬁrst one uses all the dimensions of the gradient to update e , while the latter uses only the one dimension with the largest gradient.
Fig.
3 visualizes the network update process of our two strategies.
Continuous Propagation.
In continuous propagation, we set target metrics as an optimization goal and use gradient descending to automatically update the network coding e .
Taking accuracy and FLOPs as an example, we set the target accuracy t acc and target FLOPs t ﬂops for e and calculate the loss between the prediction and the target by:
[Equation]
where L1 denotes the smooth L1 loss, λ is the coefﬁcient to balance the trade-off between accuracy and efﬁciency.
The loss is propagated back through the ﬁxed neural predictor F W ( · ) , and the gradient on the coding e is calculated as ∂L ∂e (use ∇ e in following for simplicity).
We update the coding
[Figure]
e iteratively by e ← e − ∇ e .
In this way, NCP can search for the most efﬁcient model upon a lower-bound accuracy and the most accurate model upon upper-bound FLOPs.
In practice, we set t acc =
p acc + 1 and t ﬂops = p ﬂops − 1 to ﬁnd a model with a good efﬁciency-accuracy trade-off.
The propagation is stopped once the prediction reaches the target or after a certain number of iterations ( e.g. , 70).
Finally, we round (the number of channels is a multiple of 8) and decode the coding e to obtain the architecture optimized for the speciﬁc goal.
Winner-Takes-All Propagation.
Editing different dimensions of the coding often have different degrees of impact on the network.
For example, modifying the number of network blocks has a greater impact than the number of channels, i.e. , a block costs more computational resources than several channels.
In light of this, we select one dimension of the coding e with the best accuracyefﬁciency for propagation in each iteration, termed as winner-takes-all propagation.
We create a lookup table T ( · ) to obtain the corresponding FLOPs of any network coding.
We then calculate the relative increase ∆ r in FLOPs as each dimension of the coding grows:
[Equation]
where l denotes the index of the coding e ; k = 8 if e l represents the number of convolutional channels, otherwise k = 1 .
We then use ∆ r as a normalization term of backward gradients ∇ e and select the optimal dimension l of the normalized gradients ∇ e / ∆ r :
[Equation]
if max( ∇ e / ∆ r ) > 0 , modifying the value of e l may improve accuracy and decrease FLOPs simul- taneously.
While if max( ∇ e / ∆ r ) < 0 , we choose the dimension with the highest improvement in accuracy under the unit FLOPs consumption.
The coding e is then updated as follows:
[Equation]
Cross-task Learning.
Our NCP provides a natural way for cross-task learning by transforming different objectives across tasks into gradient accumulation operations directly in the architecture coding space, facilitating joint architecture search for multiple tasks and cross-task architecture transferring, as shown in Fig.
For example, NCP enables joint search for classiﬁcation and segmentation by accumulating the gradients ∇ e cls and ∇ e seg from the two predictors:
[Equation]
Note that different weights coefﬁcients can be used to learn multi-task architectures with task preferences.
In this work, we set equal weight coefﬁcients for these tasks.
3. NAS-B ENCH -MR
Multi-Resolution Search Space.
Inspired by HRNet (Wang et al., 2020a; Ding et al., 2021), we design a multi-branch multi-resolution search space that contains four branches and maintains both high-resolution and low-resolution representations throughout the network.
As shown in Fig.
2, after two convolutional layers decreasing the feature resolution to 1 / 4 (4 × ) of the image size, we start with this high-resolution feature branch and gradually add lower-resolution branches with feature fusions, and connect the multi-resolution branches in parallel.
Multi-resolution features are resized and concatenated for the ﬁnal classiﬁcation/regression without any additional heads.
The design space follows the following key characteristics.
(1) Parallel modules: extend each branch of the corresponding resolutions in parallel using residual units (He et al., 2016); (2) Fusion modules: repeatedly fuse the information across multiple resolutions and generate the new resolution (only between two stages).
For each output resolution, the corresponding input features of its neighboring resolutions are gathered by residual units (He et al., 2016) followed by element-wise addition (upsampling is used for the low-to-high resolution feature transformation), e.g. , the 8 × output feature contains information of 4 × , 8 × , and 16 × input features; (3) A modularized block is then formed by a fusion module and a parallel module.
Coding.
We leverage the hyperparameter space of our multi-resolution network by projecting the depth and width values of all branches of each stage into a continuous-valued coding space.
Formally, two 3 × 3 stride 2 convolutional layers with i 1 , i 2 number of channels are used to obtain the 4 × high-resolution features at the beginning of the network.
We then divide the network into four stages with s = { 1 , 2 , 3 , 4 } branches, respectively.
Each stage contains b s modularized blocks, where the fusion module of the ﬁrst block is used for transition between two stages ( e.g. , frombranches to 3 branches).
For the parallel module of the block, we set the number of residual units and the number of convolutional channels to n s =
[ n 1 s , . .
. , n ss ] and c s =
[ c 1 s , . .
, c ss ] respectively, where s denotes the s -th stage containing s branches.
Since the fusion module always exists between two parallel modules, once the hyperparameters of the parallel modules are determined, the corresponding number of channels of fusion modules will be set automatically to connect the two parallel modules.
The s -th stage comprising a plurality of same modular blocks is represented by:
[Equation]
where the four stages are represented by 3, 5, 7, and 9-dimensional codes, respectively.
By the end of the 4 -th stage of the network, we use a 1 × 1 stride 1 convolutional layer with a channel number of o 1 to fuse the four resized feature representations with different resolutions.
In general, the entire architecture can be represented by a 27-dimensional continuous-valued code:
[Equation]
We then build our coding space by sampling and assigning values to the codes e .
For each structure, we randomly choose b, n ∈ 1 , 2 , 3 , 4 , and i, c, o ∈ { 8 , 16 , 24 , . . .
, 128 } , resulting in a 27dimensional coding.
In this way, a ﬁne-grained multi-resolution space that enables searching structures for various tasks with different preferences on the granularity of features is constructed.
Dataset.
We build our NAS-Bench-MR on four carefully selected visual recognition tasks that require different granularity of features: image classiﬁcation on ImageNet (Deng et al., 2009), semantic segmentation on Cityscapes (Cordts et al., 2016), 3D object detection on KITTI (Geiger et al., 2012), and video recognition on HMDB51 (Kuehne et al., 2011).
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
p
[Figure]
[Equation]
A
[Figure]
[Equation]
B
[Figure]
[Equation]
C
Cls
[Equation]
10c
Seg
Seg-4x 3dDet
Video
Video-
[Figure]
For classiﬁcation , considering the diversity of the number of classes and data scales in practical applications, we construct three subsets: ImageNet-50- (Cls-A), ImageNet-50-100 (Cls-B), and ImageNet-101000 (Cls-C), where the ﬁrst number indicates the number of classes and the second one denotes the number of images per class.
For segmentation , we follow HRNet (Wang et al., 2020a) by upsampling all branches to the high-resolution one.
For 3D detection , we follow PointPillars (Lang et al., 2019) that ﬁrst converts 3D point points into bird’s-eye view (BEV) featuremaps and then unitizes a 2D network.
For video recognition , we replace the last 2D convolutional layer before the ﬁnal classiﬁcation layer of our network with a 3D convolutional layer.
[Table]
We also explore several proxy training settings, such as 10-epoch training used in RegNet (Radosavovic et al., 2020) (Cls-10c), reduced input resolution by four times (Seg-4x), and pretraining instead of training from scratch (Video-p).
In summary, we randomly sample 2,500 structures from our search space, train and evaluate these same architectures for those above nine different tasks/settings, resulting in 22,500 trained models.
We believe these well-designed and fully-trained models serve as an important contribution of this work to facilitate future NAS research.
E XPERIMENTS 4. I MPLEMENTATION D ETAILS
Taking a three-layer fully connected network as the neural predictor, we apply NCP to NAS-BenchMR and obtain NCP-Net.
To train the neural predictor, 2000 and 500 structures in the benchmark are used as the training and validation sets for each task.
Unless speciﬁed, we use continuous propagation with an initial code of { b, n = 2; c, i, o = 64 } and λ = 0 .
5 for 70 iterations in all experiments.
The optimization goal is set to higher performance and lower FLOPs ( t acc = p acc + 1 , t ﬂops = p ﬂops − 1 ).
More experiments and details can be found in Appendix .
4. B ASELINE E VALUATIONS
We make comparisons among different search strategies on our multi-resolution space and segmentation benchmark in Tab. 2.
For Random Search (Bergstra & Bengio, 2012) and Neural Predictor (Wen et al., 2020), we sample 100 and 10,000 network codes respectively and report the highest result in top-10 models.
For NetAdapt (Yang et al., 2018), we traverse each dimension of the initial codes (increasing or decreasing a unit value) and adapt the codes by greedy strategy.
Effectiveness of NCP.
(1) NCP outperforms manually-designed networks, e.g. , ResNet (He et al., 2016) and HRNet (Wang et al., 2020a), hyperparameter optimization methods (Bergstra & Bengio, 2012; Wen et al., 2020;
Yang et al., 2018), and even well-designed weight-sharing NAS (Liu et al., 2019a; Shaw et al., 2019a).
(2) Compared to other hyperparameter optimization methods, NCP is not sensitive to the randomness and noise in model selection by adopting an approximate gradient direction.
(3) NCP is much more efﬁcient as it evaluates all dimensions of the codes with only one back-propagation without top-K ranking.
(4) NCP is able to search for optimal architectures under different computing budgets by simply setting the coefﬁcient λ .
Effectiveness of our search space.
(1) Random Search (Bergstra & Bengio, 2012) works well in our coding space, showing the effectiveness of our multi-resolution space.
(2) Models in our search space consist of only 3 × 3 convolution in the basic residual unit (He et al., 2016) without complex operators, making it applicable to various computing platforms, such as CPU, GPU, and FPGA.
Search strategies of NCP.
Intuitively, the winner-take-all strategy is more like a greedy algorithm.
It selects one coding dimension with the best accuracy-efﬁciency trade-off for propagation, resulting
[Table]
in efﬁcient models.
The continuous strategy updates all dimensions of the coding purely based on gradients.
Thus, it reaches better accuracy, though the generated model is not the most efﬁcient.
The experiment veriﬁes the above intuition.
Unless speciﬁed, we use the continuous strategy for all experiments.
Fig.
3 shows the code editing process of two strategies.
4. I NTER - TASK A RCHITECTURE S EARCH
We conduct multi-task architecture learning experiments on four basic benchmarks, including ClsA, Seg, Video, and 3dDet.
We use NCP to search models on single ( e.g. , Cls) or multiple tasks ( e.g. ,
Cls + Seg)
and evaluate their performance on ﬁve datasets, including a new dataset ADE20K (Zhou et al., 2017) which is not included in NAS-Bench-MR.
Quantitative comparisons are shown in Tab. 3.
Optimization for various objectives.
(1) NCP can customize good architectures for every single task, e.g. , the model searched on Cls achieves a top-1 accuracy of 85.56%, better than the models searched on the other three tasks.
(2) NCP can learn architectures that achieve good performance on multiple tasks by inverting their predictors simultaneously, e.g. , the model searched on Cls +
Seg achieves a top1 accuracy of 86.05% and 77.16% mIoU.
The joint optimization of all four tasks achieved moderate results on each task.
(3) NCP achieves good accuracy-efﬁciency trade-offs and tunes a proper model size for each task automatically by using the FLOPs constraint.
Relationship among tasks.
Two tasks may help each other if they are highly related, and vice versa.
For example, jointly optimizing Seg + Cls (correlation is 0.639) improves the performance on both tasks with 17% fewer FLOPs than a single Cls model, while jointly optimizing Seg +
Video (correlation is -0.003) hinders the performance of both tasks.
When optimizing Seg + 3dDet (correlation is 0.409) simultaneously, the resulted network has better results in 3dDet but worse in
Seg, which means accurate semantic information is useful for 3D detection, while the object-level localization may harm the performance of per-pixel classiﬁcation.
The above observations can be veriﬁed by Spearman’s rank correlation coefﬁcients in Fig.
This way, we can decide which tasks are more suitable for joint optimization, such as CLs + Video, improving both tasks.
Generalizability.
The searched models work well on the new dataset ADE20K (Zhou et al., 2017).
An interesting observation is that the searched models under multi-task objectives, e.g. , Seg + Cls, even conﬂicting tasks, e.g. , Seg + Video, show better performance on ADE20K than those searching on a single task, demonstrating the generalizability of NCP.
We also adapt our segmentation model (NCP-Net-L-continuous) in Tab. 2 to the COCO instance segmentation dataset.
The model with only 360 GFLOPs (measured using 800x1280 resolution, Mask R-CNN head) achieves 47.2% bounding box AP and 42.3% mask AP on the detection and instance segmentation tasks, respectively.
With fewer computational costs, our results outperform HRNet-W48 (46.1% for bbox AP and 41.0% for mask AP) by a large margin.
[Table]
4. C ROSS - TASK A RCHITECTURE T RANSFERRING
We experiment on architecture transferring across four different tasks.
We ﬁrst ﬁnd optimal network codings for each task and then use it as the initial coding to search each of the other three tasks.
Comparison results can be found in Tab. 4.
We see that: (1) Compared to searching a network from scratch for one task ( e.g. , 77.15% on segmentation) or ﬁnetuning architectures trained on other tasks ( e.g. , 75.42% by ﬁnetuning a classiﬁcation model), our transferred architecture are always better ( e.g. , 78.27% by transferring a classiﬁcation architecture to segmentation).
The architecture searched on classiﬁcation achieves better transferable ability on segmentation and video recognition, while the segmentation model is more suitable for transferring to 3D detection.
It can be validated by Spearman’s rank correlation in Fig.
(3) NCP is not sensitive to the initialized architecture coding.
Models can be searched with different initial codings and achieve good performance.
4. I NTRA - TASK A RCHITECTURE S EARCH
To show NCP ’s ability to customize architectures according to different data scales and the number of classes
[Table]
we evaluate the searched architecture on the three subsets of the ImageNet dataset.
From
Tab.
[Table]
we observe that: (1) NCP can ﬁnd optimal architectures for each subset, which is essential for practical applications with different magnitudes of data.
(2) We perform a joint architecture search for all three subsets.
The searched architecture (NCP-Net-ABC) shows surprisingly great results on all three subsets with only 0.77G extra FLOPs and fewer parameters, demonstrating intra-task generalizability of NCP.
[Table]
[Table] [Table]
[Table] [Table]
C ONCLUSION
This work provides an initial study of learning task-transferable architectures in the network coding space.
We propose an efﬁcient NAS method, namely Network Coding Propagation (NCP), which optimizes the network code to achieve the target constraints with back-propagation on neural predictors.
In NCP, the multi-task learning objective is transformed to gradient accumulation across multiple predictors, making NCP naturally applicable to various objectives, such as multi-task structure optimization, architecture transferring across tasks, and accuracy-efﬁciency trade-offs.
To facilitate the research in designing versatile network architectures, we also build a comprehensive NAS benchmark (NAS-Bench-MR) upon a multi-resolution network space on many challenging datasets, enabling NAS methods to spot good architectures across multiple tasks, other than classiﬁcation as the main focus of prior works.
We hope this work and models can advance future NAS research.
A CKNOWLEDGEMENTS
We sincerely appreciate all reviewers’ efforts and constructive suggestions in improving our paper.
Ping Luo was supported by the General Research Fund of HK No.27208720 and HKU-TCL Joint
Research Center for Artiﬁcial Intelligence.
Zhiwu Lu was supported by National Natural Science
Foundation of China (61976220)
A D ETAILS OF NAS-B ENCH -MR
In this section, we provide details of the datasets and settings used to build our NAS-Bench-MR.
We follow the common practice and realistic settings for training and evaluation in each task, making the scientiﬁc insights generated by our benchmark easier to generalize to real-world scenarios.
A. D ATASETS AND S ETTINGS
We randomly sample 2,500 structures from our network coding space, and train and evaluate these same architectures for each task and setting.
Each architecture is represented by a 27-dimensional continuous-valued code.
Tab.
7 shows the representation of each dimension of the code.
Fordifferent datasets and proxy tasks, we train a total of 22,500 models.
See below for details.
ImageNet for Image Classiﬁcation.
The ILSVRC 2012 classiﬁcation dataset (Deng et al., 2009) consists of 1,000 classes, with a number of 1.2 million training images and 50,000 validation images.
Considering the diversity of the number of classes and data scales in practical applications, and saving training time, we construct three subsets: ImageNet-50-1000, ImageNet-50-100, and ImageNet10-1000, where the ﬁrst number indicates the number of classes and the second one denotes the number of images per class.
In this work, we adopt an SGD optimizer with momentum 0.9 and weight decay 1e-4.
The input size is 224 × 224 .
The initial learning rate is set to 0.1 with a total batch size of 160 on 2 Tesla
V100 GPUs for 100 epochs, and decays by cosine annealing with a minimum learning rate of 0.
We adopt the basic data augmentation scheme to train the classiﬁcation models, i.e., random resizing and cropping, and random horizontal ﬂipping (ﬂip ratio 0.5), and use single-crop for evaluation.
We report the top-1 and top-5 accuracies as the evaluation metric on all three benchmark datasets.
We print the training log (top-1, top-5, loss, lr) every 20 iterations and evaluate the model everyepochs on the ImageNet validation set.
We will release four checkpoints of 25, 50, 75, 100 epochs with the optimizer data and all the training logs for each model.
Cityscapes for Semantic Segmentation.
The Cityscapes dataset (Cordts et al., 2016) contains highquality pixel-level annotations of 5000 images with size 1024 × 2048 (2975, 500, and 1525 for the training, validation, and test sets respectively) and about 20000 coarsely annotated training images.
Following the evaluation protocol (Cordts et al., 2016), 19 semantic labels are used for evaluation without considering the void label. [Table]
To study the effect of different image resolutions in segmentation, we conduct two benchmarks with the input size of 512 × 1024 and 128 × 256 , respectively.
For the small-resolution setting, we pre-resize the Cityscapes dataset to 256 × 512 before the data augmentation.
In this work, we use an SGD optimizer with momentum 0.9 and weight decay 4e-5.
The initial learning rate is set to 0.1 with a total batch size of 64 on 8 Tesla V100 GPUs for 25000 iterations (about 537 epochs).
Follows the common practice in (Zhao et al., 2017; Wang et al., 2020a; Ding et al., 2020b), the learning rate and momentum follow the “poly” scheduler with power 0.9 and a minimum learning rate of 1e-4.
We use basic data augmentation, i.e. , random resizing and cropping, random horizontal ﬂipping, and photometric distortion for training and single-crop testing with the test size of 1024 × 2048 and 256 × 512 respectively for two settings.
We report the mean Intersection over Union (mIoU), mean (macro-averaged)
Accuracy (mAcc), and overall (micro-averaged) Accuracy (aAcc) as the evaluation metrics.
We print the training log (mAcc, loss, lr) every 50 iterations and evaluate the model every 5000 iterations on the Cityscapes validation set.
We will release ﬁve checkpoints of 5000, 10000, 15000, 20000, 25000 iterations with the optimizer data and all the training logs for each model.
KITTI for 3D Object Detection.
The KITTI 3D object detection dataset (Geiger et al., 2012) is widely used for monocular and LiDAR-based 3D detection.
It consists of 7,481 training images and 7,518 test images as well as the corresponding point clouds and the calibration parameters, comprising a total of 80,256 2D-3D labeled objects with three object classes: Car, Pedestrian, and
Cyclist.
Each 3D ground truth box is assigned to one out of three difﬁculty classes (easy, moderate, hard) according to the occlusion and truncation levels of objects.
In this work, we follow the train-val split (Chen et al., 2015; Ding et al., 2020a), which contains 3,712 training and 3,769 validation images.
The overall framework is based on Pointpillars (Lang et al., 2019).
The input point points are projected into bird’s-eye view (BEV) feature maps by a voxel feature encoder (VFE).
The projected BEV feature maps ( 496 × 432 ) are then used as the input of our 2D network for 3D/BEV detection.
Following (Lang et al., 2019), we set, pillar resolution: 0.16m, max number of pillars: 12000, and max number of points per pillar: 100.
We apply the same data augmentation, i.e. , random mirroring and ﬂipping, global rotation and scaling, and global translation for 3D point clouds as in
Pointpillar (Lang et al., 2019).
We use the one-cycle scheduler with an initial learning rate of 2e-3, a minimum learning rate of 2e-4, and batch size 16 on 8 Tesla V100 GPUs for 80 epochs.
We use an AdamW optimizer with momentum 0.9 and weight decay 1e-2.
At inference time, axis-aligned non-maximum suppression (NMS) with an overlap threshold of 0.5 IoU is used for ﬁnal selection.
We report standard average precision (AP) on each class as the evaluation metric.
We will release ﬁve checkpoints of 76, 77, 78, 79, 80 epochs with the optimizer data and detailed evaluation results (Car/Pedestrian/Cyclist easy/moderate/hard 3D/BEV/Orientation detection AP) of each checkpoint for each model.
Due to the unstable training of the KITTI dataset, we provide the last ﬁve checkpoints for researchers to tune hyperparameters from different criteria, such as the single model best performance, the average best performance, and the last epoch best performance.
HMDB51 for Video recognition.
We train video recognition models on the HMDB dataset (Kuehne et al., 2011), consisting of 6,766 videos with 51 categories.
We use the ﬁrst training and validation split composing of 3570 and 1530 videos for evaluation, respectively.
The video containing less than 64 frames is ﬁltered, resulting in 2649 samples for training.
Considering the difﬁculty of learning temporal information, in addition to training from scratch, we also conduct training using models of ImageNet-50-1000 as pretraining.
To model the temporal
[Figure]
information, we replace the last 2D convolutional layer before the ﬁnal classiﬁcation layer of HRNet and its counterparts with a 3D convolutional layer.
In this work [Figure] the input size is set to 112 × 112 .
We adopt an Adam optimizer with momentum 0.
and weight decay 1e-5.
The initial learning rate is set to 0.01 with a total batch size of 80 on 4 Tesla
V100 GPUs for 100 epochs, and decays by cosine annealing with a minimum learning rate of 0.
We adopt random resizing and cropping, random brightness 0.5, random contrast 0.5, random saturation 0.5, random hue 0.25, and random grayscale 0.3 as data augmentation.
We report the top-1 and top-5 accuracies as the evaluation metric on both two settings.
We print the training log (top-1, top-5, loss, lr) every 10 iterations and evaluate the model every 10 epochs on the validation set.
We will release the last checkpoint and all the training logs for each model.
A. A NALYSIS OF NAS-B ENCH -MR
We summarize detailed statistics of our NAS-Bench-MR in Tab. 6.
We also give the mean, std, and the validation L1 loss (%) of our neural predictors under the main evaluation metric of each setting.
From Spearman’s rank correlation of nine benchmarks in our main paper, we observe that:
The four main tasks have different preferences for the architecture, and their correlation coefﬁcients are between -0.003 (segmentation and video recognition) and 0.639 (segmentation and classiﬁcation). This may be
because segmentation can be seen as per pixel
image classiﬁcation.
(2) Different settings in the same task also have different preferences for the architecture, and their correlation coefﬁcients are between 0.3 (Cls-50-1000 and Cls-50-100) and 0.818 (Cls-50-100 and Cls-10c).
From Fig.
5 (a) we also see that different training epochs result in a signiﬁcant performance change.
In this way, the proxy 10-epoch training setting (Radosavovic et al., 2020) may not generalize well to real-world scenarios.
(3) The correlation coefﬁcient matrix is related to the model performance of multi-task joint optimization using NP.
The higher the correlation coefﬁcient of the two tasks, the greater the gain of joint optimization.
If the correlation coefﬁcient of the two tasks is too low, joint optimization may reduce the performance on both tasks.
(4) We show the architecture ranking under different metrics for different tasks in Fig.
5 (b) and (c).
We noticed that the three metrics ( i.e. , mIoU, mAcc, aAcc) in segmentation are positively correlated, which means only one ( e.g. , mIoU) needs to be optimized to obtain a good model under all three metrics.
However, in the 3D object detection task, different metrics are not necessarily related, which makes multi-objective optimization especially important.
[Table] [Table]
[Table] [Table]
B I NTRA - TASK G ENERALIZABILITY
In this section, we explore the effect of multiple proxy settings such as different input sizes ( 512 × 1024 and 128 × 256 ), and pretraining strategies (pretraining from classiﬁcation or training from the scratch for video recognition) in designing network architectures.
We search architectures under single ( e.g. , 512 × 1024 ) or multiple ( e.g. , both resolutions) to validate the effectiveness of NCP for different intra-task settings.
Tab. 8 and 9 show the intra-task cross-setting generalizability of our searched NCP-Net and some manually-designed networks, such as ResNet (He et al., 2016) and HRNet (Wang et al., 2020a), on the Cityscapes dataset and the HMDB51 dataset, respectively.
We see that: (1) Generally, the network searched on a speciﬁc setting performs the best under this setting but poor under other settings, which means that NCP can optimize and customize model structures for a speciﬁc setting.
For example, Tab. 8 shows
NCP customizing structures for different input resolutions on Cityscapes, which is essential for practical applications in real-world scenarios.
(2) Beneﬁting from the high correlation between different settings of the same task, joint optimization of multiple settings within the same task (for both segmentation and video recognition) usually has a positive effect, i.e. , improving the performance on each setting, at the cost of the increasing model size (larger FLOPs).
However, different tasks may not necessarily correlate, e.g. segmentation + video recognition.
This can be veriﬁed by Spearman’s correlation, as discussed in our main paper.
(3) In Tab. 8, we apply the searched segmentation network to the ADE20K dataset (Zhou et al., 2017) to show the generalizability of the searched architectures.
The network trained with a large input resolution ( 512 × 1024 ) has better performance than the network trained with a small resolution ( 128 × 256 ).
Moreover, the jointly searched architecture using both resolutions shows better generalizability than those two with a single objective.
It also demonstrated that the network searched on our NAS-BenchMR has stronger transfer capability to real-world scenarios, compared to the previous benchmarks such as (Dong & Yang, 2020;
Ying et al., 2019) that uses a small input size.
For both the segmentation and the video recognition tasks, our joint searched networks outperform manually-designed networks, such as HRNet (Wang et al., 2020a) and ResNet (He et al., 2016), and achieve better generalizability to other settings and datasets, e.g.
, ADE20K.
C NCP ON NAS- BENCH -
In this section, we apply our NCP to the NAS-Bench-201 benchmark (Dong & Yang, 2020) to show its effectiveness.
We ﬁrst brieﬂy introduce NAS-Bench-201 and then state the difference between our NAS-Bench-MR and NAS-Bench-201.
Lastly, we detail the experiment.
C. NAS-B ENCH -
[Figure]
NCP ﬁnds the optimal structure from a low-Acc and high-FLOPs starting point with only 6 steps by optimizing accuracy and FLOPs constraints simultaneously.
NAS-Bench-201 employs a DARTS-like (Liu et al., 2019b) search space including three stacks of cells, connected by a residual block (He et al., 2016).
Each cell is stacked
N = 5 times, with the number of output channels as 16, 32, and 64 for the ﬁrst, second, and third stages, respectively.
The intermediate residual block is the basic residual block with a stride of 2.
There are 6 searching paths in the space of NAS-Bench-201, where each path contains 5 options: (1) zeroize, (2) skip connection, (3) 1-by-1 convolution, (4) 3-by-3 convolution, and (5) 3-by-3 average pooling layer, resulting in 15,625 different models in total.
NAS-Bench-201 train these models on the
ImageNet-16 (with an input size of 16 × 16 ) and
Cifar10 (with an input size of 32 × 32 ) datasets.
Since the ImageNet dataset is closer to practical applications, we use the models in ImageNet16 as a comparison in the following section
. C. D IFFERENCES
There are some differences between the NAS-Bench-201 benchmark and our NAS-Bench-MR.
(1) NAS-Bench-201 and NAS-Bench-MR are of different magnitudes (15,625 v.s. 10 23 ).
The number of channels/blocks/resolutions is ﬁxed in NAS-Bench-201, while it is searchable in our search space.
In this way, our search space is more ﬁne-grained and suitable for customizing to tasks with different preferences (high- and low-level features, deeper and shallower networks, wider and narrower networks, etc.).
(3) The architecture in NAS-Bench-201 is represented as a one-hot encoding (choose one of ﬁve operators).
While in our NAS-Bench-MR, the continuous-valued code is more appealing to the learning of the neural predictor.
(4) NAS-Bench-201 is built based on a single setting while NAS-Bench-MR contains multiple settings.
We then conduct experiments and show that despite the above many differences, NCP works well on NAS-Bench-201, demonstrating the generalization ability of NCP to other benchmarks.
[Figure]
number of parameters.
(cid:70) represents the optimal model with both high performance and low FLOPs.
Our approach achieves superior performance compared to its counterparts on all four tasks.
C. E XPERIMENTS
In this work, we formulate each architecture in NAS-Bench-201 to a 6 ∗ 5 = 30 -dimensional one-hot encoding.
We then use our continuous network propagation strategy to traverse architectures in the space with higher-Acc and lower-FLOPs as optimization goals.
Given an initial one-hot encoding, our NCP treats it as a continuous-valued coding and utilizes the argmax operation to obtain an edited one-hot coding after every several iterations.
In the experiment, we use the argmax operation after every 10 iterations, termed as a step.
As shown in Fig.
6, our NCP ﬁnds the optimal structure from a low-Acc and high-FLOPs starting point with only 6 steps (60 iterations).
The entire search process takes less than 10 seconds.
Compared to other neural predictor-based methods such as (Wen et al., 2020; Luo et al., 2020) that need to predict the accuracy of a large number of architectures, NCP is more efﬁcient.
From Fig.
6 we observe that from the second to fourth steps, the performance of the searched model vibrates.
This may be because of the instability caused by one-hot encoding.
NCP ﬁnds an optimal model with an accuracy of 46.8 and FLOPs of only 90.36 (the highest accuracy in NAS-Benchis 47.33%), showing its effectiveness and generalizability.
D NCP ON DIFFERENT SINGLE TASKS
We also use NCP to ﬁnd the optimal architecture on the four basic vision tasks, i.e. , image classiﬁcation, semantic segmentation, 3D detection, and video recognition.
Quantitative comparisons in Fig.
7 show that the architectures found by NCP outperform well-designed networks, such as
HRNet (Wang et al., 2020a) and ResNet (He et al., 2016).
We visualize the searched architectures by NCP in Fig.
8 to show it can customize different representations for different tasks (objectives).
We see that:
In-layersStage1 Stage
Stage
Stage4 Out-layer 0326496128 C h a nn e l 01234 B l o c k & C o n v
Cls
ChannelBlockResUnit
In-layersStage1 Stage
Stage
Stage4 Out-layer 0326496128 C h a nn e l 01234 B l o c k & C o n v
Seg
ChannelBlockResUnit
In-layersStage1 Stage
Stage
Stage4 Out-layer 0326496128 C h a nn e l 01234 B l o c k & C o n v
Video
ChannelBlockResUnit
In-layersStage1 Stage
Stage
Stage4 Out-layer 0326496128 C h a nn e l 01234 B l o c k & C o n v 3dDet
ChannelBlockResUnit
Algorithm
The network propagation process.
Learn a neural predictor F W ( · ) and ﬁx it; 2.
Initialize an architecture code e ; 3.
Set the target metrics, such as t acc and t flops ;
Re-set new target metrics (optional); 5.
Forward F W ( e ) and calculate the loss; 6.
Back-propagate and calculate the gradient ∆ e ; 7.
Select an optimal dimension l (optional); 8.
update e based on the gradient ∆ e ;
end
Round and decode e to obtain the ﬁnal architecture.
(1) Segmentation requires the most high-level information in the last two branches of the last stage, and the video recognition model contains the least low-level semantics in the ﬁrst two stages.
(2) The classiﬁcation model contains more channels than other tasks, because the FLOPs constraint for classiﬁcation is the weakest, i.e. , classiﬁcation costs fewer FLOPs than segmentation under
the same architecture.
The 3D detection model mainly utilizes the ﬁrst two branches, which means it may rely more on high-resolution representations.
E D ETAILS OF NCP
In this section, we provide supplementary details and complexity analysis of our NCP.
The algorithm of Network Coding Propagation (NCP) is shown in Algorithm 1.
Code is available.
E. I MPLEMENTATION D ETAILS
In this work, for each task, 2000 and 500 structures in the benchmark are used as the training set and validation set to train the neural predictor.
The optimization goal is set to higher performance (main evaluation metric of each task) and lower FLOPs.
Intuitively, large models can achieve moderate performance on multiple tasks by stacking redundant structures.
On the contrary, the lightweight model tends to pay more attention to task-speciﬁc structural design due to limited computing resources, making it more suitable for generalizability evaluation.
To this end, we set λ = 0 .
5 to obtain lightweight models unless speciﬁed.
We employ a three-layer fully connected network with a dropout ratio of 0.5 as the neural predictor.
The ﬁrst layer is a shared layer for all metrics with a dimension of 256.
The second layer is a separated layer for each metric with a dimension of 128.
Then for each evaluation metric, a 128by-1 fully-connected layer is used for ﬁnal regression.
During training, we use an Adam optimizer with a weight decay of 1e-7.
The initial learning rate is set to 0.01 with batch size 128 on a single
GPU for 200 epochs, and decays by the one cycle scheduler.
The metric prediction is learned using the smoothL1 loss.
In the network propagation process, unless speciﬁed, we use continuous propagation with an initial code of { b, n = 2; c, i, o = 64 } and learning rate of 3 for 70 iterations in all experiments.
[Table]
E. T IME C OMPLEXITY
Tab. 10 shows the searching time of three predictor-based searching methods.
NCP is the fastest as it traverses all dimensions of the code with only one back-propagation without the top-K ranking.
Suffering from the random noise in random search and the neural predictor, existing methods (Wen et al., 2020; Luo et al., 2020) often need to train the top-K models for ﬁnal evaluation, which is costly ( e.g. , training a segmentation model on Cityscapes costs 7 hours on 8 Tesla V100 GPUs).
NCP uses gradient directions as guidance to alleviate the randomness issue.
F V ISUALIZATION AND A NALYSIS F. A NALYSIS OF T ASK T RANSFERRING
We visualize the network coding propagation process of our cross-task architecture transferring, e.g. , an architecture transferring from the classiﬁcation task to the segmentation task by using the optimal coding on classiﬁcation as initialization and using the neural predictor trained on segmentation for optimization.
The detailed transferring visualizations of every two of the four tasks are shown in
Fig.
9-12 with many interesting ﬁndings.
For example, all networks in segmentation try to increase the number of channels of the 4th branch of stage 4, while the networks in classiﬁcation try to decrease it; the networks in video recognition keep it at an intermediate value.
F. A NALYSIS OF S INGLE - AND M ULTI - TASK S EARCHING
We visualize the searched architectures (Fig. 13-14) and the network propagation process of each architecture (Fig.
19-24) in Tab.
Classiﬁcation.
Manually-designed classiﬁcation models often use gradually decreasing resolution and gradually increasing the number of channels, because intuitively the learning of classiﬁcation task requires low-resolution high-level semantic information.
However, under different data scales and number of classes, the situation is different, as shown in Fig.
For example, by reducing the training samples from 1000 to 100 (from Fig. 15 to Fig. 16), the number of convolutional channels and the number of residual units in the 1st branch of stage 4 are increased during propagation, showing the high-resolution low-level information is important when the training samples are insufﬁcient.
Segmentation.
From Fig 19, Fig 20, and Fig 21 we observe the differences of the network editing process of segmentation models.
For example, compared to the two models optimized at a single resolution, the model optimized at both input resolutions reduces the numbers of residual units of the ﬁrst two stages and increases the number of blocks of stage 2, resulting in more fusion modules and fewer parallel units.
Segmentation and Video Recognition.
From the visualization of found architecture codes in
Fig. 13 and 14, we can ﬁnd, it is not always that the larger the model, the better the performance.
Different objectives result in different customized architecture codes.
For example, segmentation tends to select architectures with fewer input channels, while video recognition architectures often have more output channels.
[Table]
[Figure] [Figure]
[Table]
: N_channels in branch2 Stage4: N_channels in branch3 Stage4: N_channels in branch
[Figure]
[Figure]
Stage1: N_units in branch1 Stage2:
N_units in branch1 Stage2:
N_units in branch 0 20 40 60 Iterations
Stage3: N_units in branch1 Stage3:
N_units in branch2 Stage3:
N_units in branch 0 20 40 60 Iterations
[Figure]
Seg 3dDet
Stage4:
N_units in branch1 Stage4: N_units in branch2 Stage4:
N_units in branch3 Stage4:
N_units in branch
[Figure]
[Table]
[Figure] [Figure]
[Table]
[Figure] [Figure]
In
[Figure]
: N_
[Figure]
In
[Figure]
layers: N_
[Figure]
Out
[Figure]
layer: N
[Figure] [Figure]
[Figure]
n
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]