Original: F AST N EURAL N ETWORK A DAPTATION VIA P ARAMETERS R EMAPPING
Revised: F AST N EURAL N ETWORK A DAPTATION VIA P ARAMETERS R EMAPPING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks achieve remarkable performance in many computer vision tasks.
Revised: Deep neural networks achieve remarkable performance in many computer vision tasks.

Original: However, for most  semantic segmentation (seg) and object detection (det) tasks, the backboneof the network directly reuses the network manually designed for classiﬁcation tasks.
Revised: For most networks in semantic segmentation (seg) and object detection (det) tasks, the backbone directly reuses the network manually designed for classiﬁcation tasks.

Original: Utilizing a network pre-trained on ImageNet as the backbone has been a popular practice for seg/det challenges.
Revised: Utilizing a network pre-trained on ImageNet as the backbone has been a popular practice for seg/det tasks.

Original: However, because of the gap between different tasks, adapting the networkdirectly to the target task could bring performance promotion.
Revised: However, because of the gaps between different tasks, adapting the network to the target task brings performance promotion.

Original: Some recent neural architecture search (NAS) methods search for the backbone of seg/det networks.
Revised: Some recent neural architecture search (NAS) methods search for the backbone of seg/det networks.

Original: ImageNet pre-training of the search space representation or the searched network bears huge computational cost.
Revised: ImageNet pre-training of the search space representation or the searched network bears huge computational cost.

Original: In this paper, we propose a fast neural network adaptation method FNA, which can adapt  the  manually designed  networkon ImageNet to the new seg/det tasks efﬁciently.We adopt differentiable NAS to adapt the architecture of the network.
Revised: In this paper, we propose a fast neural network adaptation (FNA) method, which can adapt both the architecture and parameters of a manually designed seed network to the new seg/det tasks efﬁciently.

Original: We ﬁrst expand the manually designed network to a super network which is the representation of the search space.

Original: Then we successively conduct the  adaptation on the architecture-level and parameter-level. Our designed parameters-remapping paradigm accelerates the adaptation process.
Revised: A parameter remapping mechanism is designed to accelerate the whole adaptation process which takes full advantage of the knowledge from the seed network. The NAS method is utilized for architecture adaptation.

Original: Our experiments include both seg and det tasks. We conduct  adaptation on the MobileNetV2 network.
Revised: In experiments, we conduct network adaptation on the MobileNetV2 network to both seg and det tasks.

Original: FNA demonstrates  performance promotion compared with both manually and NAS designed networks.
Revised: FNA demonstrates clear performance gains compared with both manually and NAS designed networks.

Original: The total computational cost of FNA is much less than many SOTA seg/det NAS methods, 1737 × less than DPC, 6.8 × less than Auto-DeepLab and 7.4 × less than DetNAS.
Revised: The total computational cost of FNA is signiﬁcantly less than many SOTA seg/det NAS methods: 1737 × less than DPC, 6.8 × less than Auto-DeepLab and 7.4 × less than DetNAS.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks (DNNs) have achieved great success in recent years.
Revised: Deep neural networks (DNNs) have achieved great successes in recent years.

Original: Especially, they are widely applied in computer vision tasks, e.g., image classiﬁcation (Krizhevsky et al., 2012;
Revised: Especially, they are widely applied in computer vision tasks, e.g., image classiﬁcation (Krizhevsky et al., 2012;

Original: He et al., 2016; Howard et al., 2017),  segmentation (Long et al., 2015; Ronneberger et al., 2015; Chen et al., 2017b) and object detection (Ren et al., 2015; Liu et al., 2016; Lin et al., 2017; He et al., 2017), etc.
Revised: He et al., 2016; Howard et al., 2017), semantic segmentation (Long et al., 2015; Ronneberger et al., 2015; Chen et al., 2017b) and object detection (Ren et al., 2015; Liu et al., 2016; Lin et al., 2017) etc.

Original: Image classiﬁcation always serves as a fundamental task for neural network designing.
Revised: Image classiﬁcation always serves as a fundamental task for neural network designing.

Original: In general, one network elaborately designed and pre-trained on the classiﬁcation task is directly used as the backbone of the segmentation or detection network.
Revised: In general, one network elaborately designed and pre-trained on the classiﬁcation task is directly used as the backbone of the segmentation or detection network.

Original: The ﬁne-tuning technique is usually utilized to adapt the parameters of the backbone network to the new task.
Revised: The ﬁne-tuning technique is usually utilized to transfer the backbone network to the new task.

Original: There is no doubt that the backbone plays an important role in segmentation or detection tasks.
Revised: There is no doubt that the backbone plays an important role in segmentation or detection tasks.

Original: However, there is a gap between different tasks which leads to the different principles of architecture designing.
Revised: However, there are gaps between various tasks which lead to the different principles of architecture designing.

Original: For example, segmentation tasks focus on denser featuresof images and object detection tasks need to take both localization and classiﬁcation information for each instance.
Revised: For example, segmentation tasks require high-resolution features and object detection tasks need to infer both localization and classiﬁcation information from each convolutional feature.

Original: Therefore, neural architectures designed for classiﬁcation tasks do not sufﬁceso well for segmentation or detection tasks.
Revised: Therefore, neural architectures designed for classiﬁcation tasks do not sufﬁce well for segmentation or detection tasks.

Original: To tackle this problem, some works (Li et al., 2018; Wang et al., 2019) propose to improve the backbone architecture designing for better performance on the target tasks.
Revised: To tackle this problem, some works (Li et al., 2018; Wang et al., 2019) propose to improve the backbone architecture designing for better performance on the target tasks.

Original: Handcrafted neural architecture design is inefﬁcient and may not ﬁnd the best networks.
Revised: Handcrafted neural architecture design is inefﬁcient and may not ﬁnd the best-performing networks.

Original: Recently, neural architecture search (NAS) methods (Zoph et al., 2017; Pham et al., 2018; Liu et al., 2018; 2019b; Cai et al., 2019) aim at designing the architectures of DNNs automatically, which can alleviate the burden of human experts.
Revised: Recently, neural architecture search (NAS) methods (Zoph et al., 2017; Pham et al., 2018; Liu et al., 2018; 2019b; Cai et al., 2019) aim at designing the architectures of DNNs automatically, which can alleviate the burden of human experts.

Original: Some works (Liu et al., 2019a; Zhang et al., 2019; Chen et al., 2019b) propose to use NAS methods to specializein architecture designing for the backbones of segmentation or detection networks.
Revised: Some works (Liu et al., 2019a; Zhang et al., 2019; Chen et al., 2019b) propose to use NAS methods to specialize architecture designing for the backbones of segmentation or detection networks.

Original: However, pre-training has always been a non-negligible but costly procedure.
Revised: Nevertheless, pre-training has always been a non-negligible but costly procedure.

Original: Though it is demonstrated that pre-training is not so necessary recently (He et al., 2018), training from scratch on the target task still takes more iterations than ﬁne-tuning from the pre-trained model.
Revised: Though it is demonstrated that pre-training is not so necessary recently (He et al., 2018), training from scratch on the target task still takes more iterations than ﬁne-tuning from the pre-trained model.

Original: In architecture search methods,  pre-training becomes more intractable because there are millions of architectures in the search space.It is impractical to pre-train every backbone of the architectures.
Revised: In architecture search methods, the pre-training cost is non-negligible for [Figure] evaluating the networks in the search space.

Original: One-shot search methods (Brock et al., 2017; Bender et al., 2018;
Revised: One-shot search methods (Brock et al., 2017; Bender et al., 2018;

Original: Chen et al., 2019b) integrate all sub-architectures in one super network while pre-training the super network and the searched network both bears large computational cost.
Revised: Chen et al., 2019b) integrate all sub-architectures in one super network while pretraining the super network and the searched network still bears large computational cost.

Original: As ImageNet (Deng et al., 2009) pre-training has been an incredibly common manner in many computer vision tasks, there are lots of models trained on ImageNet available.
Revised: As ImageNet (Deng et al., 2009) pre-training has been an incredibly common manner in many computer vision tasks, there are lots of models trained on ImageNet available in the community.

Original: To take full advantage of these models, we propose a fast neural network adaptationmethod (FNA)  based on a parametersremapping paradigm.
Revised: To take full advantages of these models, we propose a fast neural network adaptation (FNA) method based on a novel parameter-remapping paradigm.

Original: Our method can adapt  one network to a new task with  low computational cost.
Revised: Our method can adapt both the architecture and parameters of one network to a new task with very low computational cost.

Original: Fig.
Revised: Fig.

Original: 1 shows the whole framework.
Revised: 1 shows the whole framework.

Original: The adaptation actsboth on  the architecture- and parameterslevel.
Revised: The adaptation acts on both the architecture- and parameter-level.

Original: We adopt thedifferentiable NAS method (Liu et al., 2019b; Wu et al., 2018;
Revised: We adopt the NAS methods (Zoph et al., 2017; Real et al., 2018;

Original: Cai et al., 2019) to implement the architecture-level adaptation.
Revised: Liu et al., 2019b) to implement the architecture-level adaptation.

Original: We select a manually designed network (MobileNetV2 (Sandler et al., 2018) in our experiments) as the seed network , which is pre-trained on ImageNet.
Revised: We select a manually designed network (MobileNetV2 (Sandler et al., 2018) in our experiments) as the seed network , which is pre-trained on ImageNet.

Original: With the parameters-remapping paradigm, the seed network can be expanded into a super network which is thecontinuous representation of the search space.
Revised: Then, we expand the seed network to a super network which is the representation of the search space in FNA.

Original: The new parameters in the super network can be initialized as one mapping of that in the seed network.
Revised: New parameters in the super network are initialized as one mapping of that in the seed network.

Original: Beneﬁtting from the parameters-remapping paradigm, the differentiable architecture search process can be fast carried out directly on the target task.
Revised: Beneﬁting from the proposed parameter-remapping paradigm, the neural architecture search process can be performed efﬁciently on the target task.

Original: With the NAS based architecture adaptation, we obtain a new architecture targeted at the new task.
Revised: With the NAS based architecture adaptation, we obtain a new architecture targeted at the new task.

Original: Then we conduct adaptation on the parameter-level.

Original: Similarly, we remap the parameters of the seed network on the target network.
Revised: Similarly, we remap the parameters of the seed network to the target architecture.

Original: Then the target network can be directly ﬁne-tuned on the target task with no need for pre-training once more.
Revised: Then the target architecture is well-initialized and can be directly ﬁne-tuned on the target task with no need of pre-training on a large-scale dataset.

Original: Our proposed FNA method aims at adapting  the  neural network to new tasksfast and efﬁciently.
Revised: Our proposed FNA method aims at adapting both the architecture and parameters of neural network to new tasks efﬁciently.

Original: Based on the parameters-remapping mechanism, FNA takes full advantage of the pre-trained parameters of the seed network.
Revised: Based on the parameter-remapping mechanism, FNA takes full advantages of the pre-trained parameters of the seed network.

Original: The remapping acts on both the NAS process and parameters ﬁne-tuning, which greatly degrades the computational cost of  adaptation.
Revised: The remapping mechanism greatly degrades the computational cost of network adaptation.

Original: The effectiveness and efﬁciency of our method are demonstrated by experiments on both segmentation and detection tasks.
Revised: The effectiveness and efﬁciency of our method are demonstrated by experiments on both segmentation and detection tasks.

Original: We adapt the manually designed network MobileNetV2 (Sandler et al., 2018) to segmentation framework DeepLabv3 (Chen et al., 2017b), detection framework RetinaNet (Lin et al., 2017) and SSDLite (Liu et al., 2016; Sandler et al., 2018).
Revised: We adapt the manually designed network MobileNetV2 (Sandler et al., 2018) to segmentation framework DeepLabv3 (Chen et al., 2017b), detection framework RetinaNet (Lin et al., 2017) and SSDLite (Liu et al., 2016; Sandler et al., 2018).

Original: Networks adapted by FNA surpass both manually designed and NAS searched networks in terms of both performance and model FLOPs.
Revised: Networks adapted by FNA surpass both manually designed and NAS searched networks in terms of both performance and model MAdds.

Original: Compared to NAS methods, FNA takes much less computational cost, 1737 ×
Revised: Compared to NAS methods, FNA takes much less computational cost, 1737 ×

Original: less than DPC, 6.8 × less than Auto-DeepLab and 7.4 × less than DetNAS.
Revised: less than DPC (Chen et al., 2018a), 6.8 × less than AutoDeepLab (Liu et al., 2019a) and 7.4 × less than DetNAS (Chen et al., 2019b).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural Architecture Search With reinforcement learning (RL) and evolutionary algorithm (EA) being applied to NAS methods, many works (Zoph & Le, 2016; Zoph et al., 2017;Pham et al., 2018; Tan et al., 2018; Real et al., 2018) make great progress in promoting the performances of neural networks.
Revised: Neural Architecture Search With reinforcement learning (RL) and evolutionary algorithm (EA) being applied to NAS methods, many works (Zoph & Le, 2016; Zoph et al., 2017; Real et al., 2018) make great progress in promoting the performances of neural networks.

Original: Recently, NAS methods based on one-shot model (Brock et al., 2017; Bender et al., 2018) or differentiable representations (Liu et al., 2019b; Cai et al., 2019; Wu et al., 2018) achieve remarkable results with low search cost.
Revised: Recently, NAS methods based on one-shot model (Brock et al., 2017; Bender et al., 2018) or differentiable representa- [Figure] et al.

Original: We use the differentiable NAS method to implement architecture adaptation, which adjusts the architecture of the backbone network automatically.
Revised: 2019b Cai et al., 2019; Wu et al., cost use the differentiable NAS method to implement architecture adaptation, which adjusts the architecture of the backbone network automatically.

Original: Backbone Design As deep neural network  (Simonyan & Zisserman, 2014; Szegedy et al., 2016;
Revised: Backbone Design As deep neural network designing (Simonyan & Zisserman, 2014; Szegedy et al., 2016;

Original: He et al., 2016)designing develops, the backbone of segmentation or detection evolves accordingly.
Revised: He et al., 2016) develops, the backbones of segmentation or detection networks evolve accordingly.

Original: Some works improve the backbone architectures by modifying existing networks.
Revised: Some works improve the backbone architectures by modifying existing networks.

Original: PeleeNet (Wang et al., 2018) proposes a variant of DenseNet (Huang et al., 2017) for more efﬁcient detectionsystem on mobile devices.
Revised: PeleeNet (Wang et al., 2018) proposes a variant of DenseNet (Huang et al., 2017) for more realtime object detection on mobile devices.

Original: DetNet (Li et al., 2018) apply dilated convolution (Yu & Koltun, 2016) on the backbone to enlarge the receptive ﬁeld.
Revised: DetNet (Li et al., 2018) applies dilated convolution (Yu & Koltun, 2016) in the backbone to enlarge the receptive ﬁeld which helps to detect objects.

Original: BiSeNet (Yu et al., 2018) and HRNet (Wang et al., 2019) design multiple paths to learn both high- and low- resolution representations.
Revised: BiSeNet (Yu et al., 2018) and HRNet (Wang et al., 2019) design multiple paths to learn both highand low- resolution representations for better dense prediction.

Original: Some works (Liu et al., 2019a; Zhang et al., 2019; Chen et al., 2019b) employ NAS methods to design better backbone architectures.

Original: Parameters Remapping Net2Net (Chen et al., 2015) proposes the function-preserving transformations to remap the parameters of one network to a new deeper or wider network.
Revised: Parameters Remapping Net2Net (Chen et al., 2015) proposes the function-preserving transformations to remap the parameters of one network to a new deeper or wider network.

Original: This remapping mechanism accelerates the training of the new larger network and achieves great performances.
Revised: This remapping mechanism accelerates the training of the new larger network and achieves great performances.

Original: Following this manner, EAS (Cai et al., 2018) extends the parameters-remapping concept to neural architecture search.
Revised: Following this manner, EAS (Cai et al., 2018) extends the parameter-remapping concept to neural architecture search.

Original: Moreover, some NAS works (Pham et al., 2018; Fang et al., 2019; Elsken et al.,[Figure] 2019) apply parameters sharing on child models to accelerate the search process.
Revised: Moreover, some NAS works (Pham et al., 2018; Fang et al., 2019; Elsken et al., 2019) apply parameters sharing on child models to accelerate the search process.

Original: Our parametersremapping paradigm makes the network adaptation get rid of the cost of ImageNet pre-training and largely decreases the computational cost.
Revised: Our parameterremapping paradigm makes the network adaptation get rid of the cost of ImageNet pre-training and largely decreases the computational cost.

Original:  M ETHOD
Revised:  M ETHOD

Original: The artiﬁcially-designed network MobileNetV2 (Sandler et al., 2018) has been widely used in many computer vision tasks because of its high performance and low FLOPs.
Revised: As the most commonly used network for designing search space in NAS methods (Tan et al., 2018;

Original: In this paper, we select the MobileNetV2 (Sandler et al., 2018) network as the seed network for its generality.
Revised: Cai et al., 2019; Wu et al., 2018), MobileNetV2 (Sandler et al., 2018) is selected as the seed network to give the details of our method.

Original: To adapt the network to segmentation and detection tasks, we adjust the architecture elements on three levels, i.e. convolution kernel size, depth and width of the network.
Revised: To adapt the network to segmentation and detection tasks, we adjust the architecture elements on three levels, i.e., convolution kernel size, depth and width of the network.

Original: In this section, we ﬁrst describe the parameters-remapping paradigm.
Revised: In this section, we ﬁrst describe the parameter-remapping paradigm.

Original: Then we explain the whole procedure of the network adaptation.
Revised: Then we explain the whole procedure of the network adaptation.

Original: 3. P ARAMETERS R EMAPPING
Revised: 3. P ARAMETERS R EMAPPING

Original: We deﬁne parameters-remapping as one paradigm which aims at mapping the parameters of one seed network to another one.
Revised: We deﬁne parameter-remapping as one paradigm which aims at mapping the parameters of one seed network to another one.

Original: We denote the seed network as N s and the new network as N n , whose parameters are denoted as W s and W n respectively.
Revised: We denote the seed network as N s and the new network as N n , whose parameters are denoted as W s and W n respectively.

Original: The remapping paradigm is illustrated in the following three aspects.
Revised: The remapping paradigm is illustrated in the following three aspects.

Original: The remapping on the depth-level is ﬁrstly carried out and  the remapping on the kernel- and width- level is conducted simultaneously.
Revised: The remapping on the depth-level is ﬁrstly carried out and then the remapping on the kernel- and width- level is conducted simultaneously.

Original: Moreover, we study different remapping manners in the experiments.
Revised: Moreover, we study different remapping strategies in the experiments (Sec. 4.5).

Original: Remapping on Depth-level We extend the depth settings in our  adaptation process.
Revised: Remapping on Depth-level We introduce more depth settings in our architecture adaptation process.

Original: In other word, we adjust the number of inverted residual blocks  (Sandler et al.,2018)
Revised: In other word, we adjust the number of inverted residual blocks (MBConvs) (Sandler et al.,

Original: (MBConv) in every stage of the network.
Revised: 2018) in every stage of the network.

Original: We assume that one stage in the seed network N s has l layers.
Revised: We assume that one stage in the seed network N s has l layers.

Original: Its parameters of each layer can be denoted as { W (1) s , W (2) s , . . .
Revised: Its parameters of each layer can be denoted as { W (1) s , W (2) s , . . .

Original: , W ( l ) s } .
Revised: , W ( l ) s } .

Original: Similarly, we assume that the corresponding stage with m layers in the new network N n has parameters { W (1) n , W (2) n , . . .
Revised: Similarly, we assume that the corresponding stage with m layers in the new network N n has parameters { W (1) n , W (2) n , . . .

Original: , W ( m ) n } .
Revised: , W ( m ) n } .

Original: The remapping process on the depth-level is shown in Fig. 2(a).
Revised: The remapping process on the depth-level is shown in Fig. 2(a).

Original: The parameters of layers in N n which also exit in N s are just copied from N s .
Revised: The parameters of layers in N n which also exit in N s are just copied from N s .

Original: The parameters of new layers are all copied from the last layer in the stage of N s .
Revised: The parameters of new layers are all copied from the last layer in the stage of N s .

Original: It is formulated as
Revised: It is formulated as

Original: [Equation]
Revised: [Equation]

Original: Remapping on Width-level In the MBConv block of MobileNetV2 (Sandler et al., 2018) network, the ﬁrst point-wise convolution expands the low-dimensional features to a high dimension.
Revised: Remapping on Width-level In the MBConv block of MobileNetV2 (Sandler et al., 2018) network, the ﬁrst point-wise convolution expands the low-dimensional features to a high dimension.

Original: This practice can be utilized for expanding the width and capacity of one neural network.
Revised: This practice can be utilized for expanding the width and capacity of one neural network.

Original: We adapt [Figure] the width of the network by setting smaller expansion ratios.
Revised: We allow smaller expansion ratios for architecture adaptation.

Original: We denote the parameters of the convolution in N s as W s ∈ R p × q × h × w and that in N n as W n ∈ R r × s × h × w , where r ≤ p and s ≤ q .
Revised: We denote the parameters of the convolution in N s as W s ∈ R p × q × h × w and that in N n as W n ∈ R r × s × h × w , where r ≤ p and s ≤ q .

Original: Asis shown in Fig.
Revised: As shown in Fig.

Original: 2(b), on the width-level, we just map the ﬁrst r/s channels of parameters in N s to the narrower one in N n .
Revised: 2(b), on the width-level, we directly map the ﬁrst r or s channels of parameters in N s to the narrower one in N n .

Original: This remapping strategy is utilized for parameters sharing in

Original: EAT-NAS (Fang et al., 2019) as well.

Original: It can be formulated as
Revised: It can be formulated as

Original: [Equation]
Revised: [Equation]

Original: Remapping on Kernel-level The kernel size is commonly set as 3 × 3 in most artiﬁcially-designed networks.
Revised: Remapping on Kernel-level The kernel size is commonly set as 3 × 3 in most artiﬁcially-designed networks.

Original: To expand the receptive ﬁeld and capturemore abundant features in segmentation and detection tasks, we introduce larger kernel size settings in the adaptation process.
Revised: To expand the receptive ﬁeld and capture abundant features in segmentation or detection tasks, we introduce larger kernel size settings in the adaptation process.

Original: As Fig. 2(c) shows, to expand the 3 × 3 kernel to a larger one, we assign the parameters of the central 3 ×region in the large kernel with the values of the original 3 × 3 kernel.
Revised: As Fig. 2(c) shows, to expand the 3 × 3 kernel to a larger one, we assign the parameters of the central 3 × 3 region in the large kernel with the values of the original 3 × 3 kernel.

Original: The values of the other region surrounding the central part are assigned with 0.
Revised: The values of the other region surrounding the central part are assigned with 0.

Original: We denote the parameters of the original 3 ×kernel as W 3 × 3 and the larger k × k kernel as W k × k .
Revised: We denote the parameters of the original 3 × 3 kernel as W 3 ×and the larger k × k kernel as W k × k .

Original: The remapping process on kernel-level can be formulated as follows,
Revised: The remapping process on kernel-level can be formulated as follows,

Original: [Equation]
Revised: [Equation]

Original: where h, w denote the indices of the spatial dimension.
Revised: where h, w denote the indices of the spatial dimension.

Original: This design principle conforms to the function-preserving concept (Chen et al., 2015), which accelerates and stabilizes the optimization of the new network.
Revised: This design principle conforms to the function-preserving concept (Chen et al., 2015), which accelerates and stabilizes the optimization of the new network.

Original: 3. N EURAL N ETWORK A DAPTATION
Revised: 3. N EURAL N ETWORK A DAPTATION

Original: We divide our neural network adaptation into three steps.
Revised: We divide our neural network adaptation into three steps.

Original: Fig.
Revised: Fig.

Original: 1 demonstrates the whole adaptation procedure.
Revised: 1 demonstrates the whole adaptation procedure.

Original: We select MobileNetV2 as the seed network N s .

Original: Firstly, we expand the seed network to a super network which is the representation of the search space in the latter architecture adaptation process.
Revised: Firstly, we expand the seed network to a super network which is the representation of the search space in the latter architecture adaptation process.

Original: Secondly, we utilize the differentiable NAS method to achieve network adaptation on the architecture-level and obtainan inner network N i with the target architecture.
Revised: Secondly, we utilize the differentiable NAS method to implement network adaptation on the architecture-level and obtain the target architecture Arch t .

Original: Finally, we adapt the parameters of the inner network and obtain the target network N t .
Revised: Finally, we adapt the parameters of the target architecture and obtain the target network N t .

Original: The aforementioned parametersremapping method is applied to both network expansion and parameters adaptation.
Revised: The aforementioned parameter-remapping mechanism is deployed before the two stages, i.e. architecture adaptation and parameter adaptation.

Original: Network Expansion We expand the seed network to a super network by introducing more options for architecture elements.
Revised: Network Expansion We expand the seed network to a super network by introducing more options for architecture elements.

Original: For every MBConv layer, we allow for more kernel size settings { 3 , 5 , 7 } and more expansion ratios { 3 , 6 } .
Revised: For every MBConv layer, we allow for more kernel size settings { 3 , 5 , 7 } and more expansion ratios { 3 , 6 } .

Original: As most differentiable NAS methods (Liu et al., 2019b; Cai et al., 2019; Wu et al., 2018) do, we set every layer as a weighted sum of all candidate operations.[Equation]
Revised: As most differentiable NAS methods (Liu et al., 2019b; Cai et al., 2019; Wu et al., 2018) do, we relax every layer as a weighted sum of all candidate operations.

Original: [Equation]
Revised: [Equation]

Original:  where O denotes the operation set, α io denotes the architecture parameter of operation o in the i th layer, and x denotes the input tensor.
Revised: (4) where O denotes the operation set, α io denotes the architecture parameter of operation o in the i th layer, and x denotes the input tensor.

Original: We extend more layers in one stage in the  network and add the identity connection to the  operation set for depth adjustment.
Revised: We set more layers in one stage of the super network and add the identity connection to the candidate operation set for depth search.

Original: After expanding the architecture of the network, we remap the parameters of the seed network to the super network based on the paradigm illustrated in Sec. 3.1.
Revised: After expanding the seed network to a super network, we remap the parameters of the seed network to the super network based on the paradigm illustrated in Sec. 3.1.

Original: This remapping strategy prevents the huge cost of ImageNet pre-training of the all the sub-networks or the super network.
Revised: This remapping strategy prevents the huge cost of ImageNet pre-training involved in the search space, i.e. the super network in differentiable NAS. [Table]

Original:  Architecture Adaptation We start the differentiable NAS process with the expanded super network directly on the target task, e.g., semantic segmentation or object detection.
Revised: [Table] Architecture Adaptation We start the differentiable NAS process with the expanded super network directly on the target task, e.g., semantic segmentation or object detection.

Original: We ﬁrst ﬁne-tune operation weights of the super network for some epochs on the training dataset.
Revised: We ﬁrst ﬁne-tune operation weights of the super network for some epochs on the training dataset.

Original: After the weights are sufﬁciently trained, we start alternating the optimization of operation weights w with ∂ L ∂w and architecture parameters α with ∂ L ∂α .
Revised: After the weights are sufﬁciently trained, we start alternating the optimization of operation weights w with ∂ L ∂w and architecture parameters α with ∂ L ∂α .

Original: To accelerate the search process and decouple the parameters of different sub-networks, we only sample one path in each iterationof operation weight optimization. We sample the path according to the distribution of architecture parameters.
Revised: To accelerate the search process and decouple the parameters of different sub-networks, we only sample one path in each iteration according to the distribution of architecture parameters for operation weight updating.

Original: With the optimization ﬁnished, we use the architecture parameters α to derive the target architecture.
Revised: As the search process terminates, we use the architecture parameters α to derive the target architecture.

Original: Parameters Adaptation After architecture adaptation, we obtainan inner network N i with the  architecture targeted at the new task.
Revised: Parameters Adaptation We obtain the target architecture Arch t from architecture adaptation.

Original: To accommodate the new segmentation or detection tasks, the target architecture becomesquite different from that of the seed network N s (which is primitively designed for the image classiﬁcation task).
Revised: To accommodate the new segmentation or detection tasks, the target architecture becomes different from that of the seed network N s (which is primitively designed for the image classiﬁcation task).

Original: Unlike conventional training strategy, we discard the cumbersome pre-training process of N i on ImageNet.
Revised: Unlike conventional training strategy, we discard the cumbersome pre-training process of Arch t on ImageNet.

Original: We remap the parameters of N s to N i utilizing the method described in Sec. 3.1.
Revised: We remap the parameters of N s to Arch t utilizing the method described in Sec. 3.1.

Original: Then we directly ﬁne-tune N i on the target task and obtain the ﬁnal target network N t .
Revised: Finally, we directly ﬁne-tune Arch t on the target task and obtain the ﬁnal target network N t .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We select the ImageNet pre-trained MobileNetV2 model as the seed network and apply our FNA method on both semantic segmentation and object detection tasks.
Revised: We select the ImageNet pre-trained MobileNetV2 model as the seed network and apply our FNA method on both semantic segmentation and object detection tasks.

Original: In this section, ﬁrstly we give the implementation details of our experiments; then we report and analyze the network adaptation results; ﬁnally, we perform ablation studies to validate the effectiveness of the parameters-remapping paradigm and compare different parameters remapping implementations.
Revised: In this section, ﬁrstly we give the implementation details of our experiments; then we report and analyze the network adaptation results; ﬁnally, we perform ablation studies to validate the effectiveness of the parameter-remapping paradigm and compare different parameters remapping implementations.

Original: 4.  A DAPTATION ON S EMANTIC S EGMENTATION
Revised: 4. N ETWORK A DAPTATION ON S EMANTIC S EGMENTATION

Original: The semantic segmentation experiments are conducted on the Cityscapes (Cordts et al., 2016)dataset, which contains 2975 images for the training and 500 for the validation dataset.The image size is 1024 × 2048 .
Revised: The semantic segmentation experiments are conducted on the Cityscapes (Cordts et al., 2016) dataset.

Original: In the architecture adaptation process, we map the seed network to the super network, which is used as the backbone of DeepLabv3 (Chen et al., 2017b).
Revised: In the architecture adaptation process, we map the seed network to the super network, which is used as the backbone of DeepLabv3 (Chen et al., 2017b).

Original: The output feature map is down-sampled by the factor of 16 .

Original: Depthwise-separable convolution is used in the ASPP module (Chen et al., 2017a;b).

Original: We randomly sample 20% images from the training set as the validation set for architecture parameters updating.
Revised: We randomly sample 20% images from the training set as the validation set for architecture parameters updating.

Original: The original validation set is not used in the search process.
Revised: The original validation set is not used in the search process.

Original: The image is ﬁrst resized to 512 × 1024 and 321 × 321 patches are randomly cropped as the input data.

Original: The stages where the expansion ratio of MBConv is 6 in the original MobileNetV are searched and adjusted.

Original: We set the numbers of layers in each searched stage of the super network as { 4 , 4 , 6 , 6 , 4 , 1 } .

Original: To optimize the FLOPs of the searched network, we deﬁne the loss function in search as L =
Revised: To optimize the MAdds of the searched network, we deﬁne the loss function in search as L =

Original: L task + λ log τ (γ · cost ) .
Revised: L task + λ log τ ( cost ) .

Original: The ﬁrst term denotes the cross-entropy loss and the second term controls the FLOPs of the network.
Revised: The ﬁrst term denotes the crossentropy loss and the second term controls the MAdds of the network.

Original: We set λ as 9 × 10 − 3 , τ as 45and γ as 10 .
Revised: We set λ as 9 × 10 − 3 and τ as 45 .

Original: The search process takes 80 epochs in total.
Revised: The search process takes 80 epochs in total.

Original: We set a warm-up stage in the ﬁrst 5 epochs to linearly increase the learning rate from 1 × 10 − 4 to 0 .

Original: Then, the learning rate decays to 1 × 10 − 3 with the cosine annealing schedule (Loshchilov & Hutter, 2017).

Original: We use the SGD optimizer with 0 .

Original: 9 momentum and 5 × 10 − 4 weight decay for operation weights and the Adam optimizer (Kingma & Ba, 2015) with 4 × 10 − 5 weight decay and a ﬁxed learning rate 1 × 10 − 3 .

Original: The architecture optimization starts after 30 epochs.
Revised: The architecture optimization starts after 30 epochs.

Original: The batch size is set as 16 .

Original: The whole search process is conducted on a single V GPU and takes only 1.4 hours in total.
Revised: The whole search process is conducted on a single V100 GPU and takes only 1.4 hours in total.

Original: In the parameters adaptation process, we remap the parameters of original MobileNetV2 to the inner network obtained in the aforementioned architecture adaptation.
Revised: In the parameters adaptation process, we remap the parameters of original MobileNetV2 to the target architecture obtained in the aforementioned architecture adaptation.

Original: The training data is cropped as a 769 × 769 patch from the rescaled image. The scale is randomly selected from [0 . 75 , 1 . 0 , 1 . 25 , 1 . 5 , 1 . 75 , 2 . 0] . The random left-right ﬂipping is used. We update the statistics of
Revised: The whole parameters adaptation process is conducted on 4 TITAN-Xp GPUs and takes 100 K iterations, which cost only 8 . 5 hours in total.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: the batch normalization (BN) (Ioffe & Szegedy, 2015) for 2000 iterations before the ﬁne-tuning process.

Original: We use the same SGD optimizer as the search process.

Original: The learning rate linearly increases from 1 × 10 − 4 to 0 .

Original: and then decays to 0 with the polynomial schedule.

Original: The batch size is set as 16 .

Original: The whole parameters adaptation process is conducted on 4 TITAN-Xp GPUs and takes 100 K iterations, which cost only 8 . 5 hours in total.

Original: Our semantic segmentation results are shown in Tab. 1.
Revised: Our semantic segmentation results are shown in Tab. 1.

Original: The FNA network achieves 76 .
Revised: The FNA network achieves 76 .

Original: 6% mIOU on Cityscapes with  DeepLabv3 (Chen et al., 2017b) framework, 1 . 1%  better than the manually designed seed Network MobileNetV2 (Sandler et al., 2018) with fewer MAdds.
Revised: 6% mIOU on Cityscapes with the DeepLabv3 (Chen et al., 2017b) framework, 1 . 1% mIOU better than the manually designed seed Network MobileNetV2 (Sandler et al., 2018) with fewer MAdds.

Original: Compared to a NAS method DPC (Chen et al., 2018a) (with MobileNetV2 as the backbone) which searches the multiscale module for semantic segmentation tasks, FNA gets 0 .
Revised: Compared with a NAS method DPC (Chen et al., 2018a) (with MobileNetV2 as the backbone) which searches a multi-scale module for semantic segmentation tasks, FNA gets 0 .

Original: 9% mIOU promotion with 0 .
Revised: 9% mIOU promotion with 0 .

Original: 52 B fewer MAdds.
Revised: 52 B fewer MAdds.

Original: For fair comparison with Auto-DeepLab (Liu et al., 2019a) which searches the backbone architecture on DeepLabv3 and retrains the searched network on DeepLabv3+ (Chen et al., 2018b), we adapt the parameters of the inner network N i to DeepLabv3+ framework.
Revised: For fair comparison with Auto-DeepLab (Liu et al., 2019a) which searches the backbone architecture on DeepLabv3 and retrains the searched network on DeepLabv3+ (Chen et al., 2018b), we adapt the parameters of the target architecture Arch t to DeepLabv3+ framework.

Original: Comparing to AutoDeepLab-S, FNA achieves far better mIOU with fewer MAdds, Params and training iterations.
Revised: Comparing with Auto-DeepLab-S, FNA achieves far better mIOU with fewer MAdds, Params and training iterations.

Original: With the remapping mechanism, FNA only takes 35.8 GPU hours, 1737 × less than DPC(Chen et al., 2018a) and 6.8 × less than Auto-DeepLab (Liu et al., 2019a).
Revised: With the remapping mechanism, FNA only takes 35.8 GPU hours, 1737 × less than DPC and 6.8 × less than Auto-DeepLab.

Original: 4.  A DAPTATION ON O BJECT D ETECTION
Revised: 4. N ETWORK A DAPTATION ON O BJECT D ETECTION

Original: We further implement our FNA method to object detection tasks.
Revised: We further implement our FNA method on object detection tasks.

Original: We adapt the MobileNetV2 seed network to two commonly used detection systems, RetinaNet (Lin et al., 2017) and a lightweight one SSDLite (Liu et al., 2016; Sandler et al., 2018).
Revised: We adapt the MobileNetV2 seed network to two commonly used detection systems, RetinaNet (Lin et al., 2017) and a lightweight one SSDLite (Liu et al., 2016; Sandler et al., 2018).

Original: The experiments are conducted on  MS-COCO dataset (Lin et al., 2014).
Revised: The experiments are conducted on the MS-COCO dataset (Lin et al., 2014b).

Original: Our implementation is based on the MMDetection (Chen et al., 2019a) framework.
Revised: Our implementation is based on the MMDetection (Chen et al., 2019a) framework.

Original: In the search process of architecture adaptation, we randomly sample 50% data from the original training set as the validation set.
Revised: In the search process of architecture adaptation, we randomly sample 50% data from the original trainval35k set as the validation set.

Original: The depth settings in each searched stage are set as { 4 , 4 , 4 , 4 , 4 , 1 } .

Original: We describe more implementation details in the Appendix.

Original: We demonstrate the results on object detection tasks in Tab. 3.
Revised: We show the results on the COCO dataset in Tab. 3.

Original: In the RetinaNet(Lin et al., 2017) framework, we compare FNA with two manually designed networks, ShufﬂeNetV2-10 (Ma et al., 2018; Chen et al., 2019b) and MobileNetV2 (Sandler et al., 2018). FNA achieves higher mAP with similar MAdds.
Revised: In the RetinaNet framework, compared with two manually designed networks, ShufﬂeNetV2-10 (Ma et al., 2018; Chen et al., 2019b) and MobileNetV2 (Sandler et al., 2018), FNA achieves higher mAP with similar MAdds.

Original: We compare FNA with DetNAS (Chen et al., 2019b) which searches the backbone of detection network. FNA achieves 0 .
Revised: Compared with DetNAS (Chen et al., 2019b) which searches the backbone of detection network, FNA achieves 0 .

Original: 6% higher mAP with 1 .
Revised: 6% higher mAP with 1 .

Original: 64 M fewer Params and 0 .
Revised: 64 M fewer Params and 0 .

Original: 2 B fewer MAdds.
Revised: 2 B fewer MAdds.

Original: As shown in Tab. 4, our total computational cost is only 13.5% of DetNAS, while DetNAS requires a huge cost for ImageNet pre-training.
Revised: As shown in Tab. 4, our total computation cost is only 13.5% of DetNAS.

Original: We degrade the cost largely with the assistance of parameters remapping.

Original: Moreover, we implement our FNA method on  SSDLite(Liu et al., 2016; Sandler et al., 2018) framework.
Revised: Moreover, we implement our FNA method on the SSDLite framework.

Original: In Tab. 3, FNA surpasses both the manually designed network MobileNetV2 and the NAS-searched network MnasNet-92, where MnasNet (Tan et al., 2018) takes around 3.8K GPU days to search for the backbone network on ImageNet.
Revised: In Tab. 3, FNA surpasses both the manually designed network MobileNetV2 and the NAS-searched network MnasNet-92, where MnasNet (Tan et al., 2018) takes around 3.8K GPU days to search for the backbone network on ImageNet.

Original: The speciﬁc cost FNA takes on SSDLite is shown in Tab. 4.
Revised: The speciﬁc cost FNA takes on SSDLite is shown in Tab. 4.

Original: It is difﬁcult to train the small network due to the[Table] [Table] simpliﬁcation (Liu et al., 2019c).
Revised: It is difﬁcult to train the small network due to the simpliﬁcation (Liu et al., 2019c).

Original: Therefore, experiments on SSDLite need long training schedule and take larger computational cost.
Revised: Therefore, experiments on SSDLite need long training schedule and take larger computational cost.

Original: The experimental results further demonstrate the efﬁciency and effectiveness of direct adaptation on the target task with parameters remapping
Revised: The experimental results further demonstrate the efﬁciency and effectiveness of direct adaptation on the target task with parameters remapping.

Original: [Table]

Original: 4. E FFECTIVENESS OF P ARAMETERS R EMAPPING
Revised: 4. E FFECTIVENESS OF P ARAMETERS R EMAPPING

Original: To validate the effectiveness of the parameters remapping paradigm in our method [Table] we carry out several experiments for ablation study.
Revised: To evaluate the effectiveness of the parameter remapping paradigm in our method, we attempt to optionally remove the parameter remapping process before the two stages, i.e. architecture adaptation and parameter adaptation.

Original: The experiments are conducted with thesemantic segmentation framework DeepLabv3 (Chen et al., 2017b)  on  Cityscapes dataset (Cordts et al., 2016).
Revised: The experiments are conducted with the DeepLabv3 (Chen et al., 2017b) semantic segmentation framework on the Cityscapes dataset (Cordts et al., 2016).

Original: In our FNA method, the parameters remapping is conducted on both stages of network expansion and parameters adaptation. To prove the importance of parameters remapping in FNA, weattempt to remove the parameters remapping process at the two stages.
Revised: In Row (2) we remove the parameter remapping process before architecture adaptation. In other word, the search is performed from scratch without using the pre-trained network. The mIOU in

Original: The parameters of the network without remapping will be initialized with random values based on the standard normal distribution. With parameters remapping applied onboth stages, the adaptation achieves the best results in Tab. 5.
Revised: Row (2) drops by 0.6% compared to FNA in Row (1). Then we remove the parameter remapping before parameter adaptation in Row (3), i.e. training the target architecture from scratch on the target [Table]

Original: Especially, the remapping operation on the parameters adaptation stage tends to have greater importance for the ﬁnal performance.
Revised: task. The mIOU decreases by 3.6% compared the result of FNA.

Original: To further evaluate the performance of parameters remapping on the parameters adaptation stage, we pre-train the inner network N i obtained in architectureadaptation onImageNet. Then we ﬁnetune the ImageNet pre-trained network on the semantic segmentation task.
Revised: When we remove the parameter remapping before both stages in Row (4), it gets the worst performance. In Row (5), we ﬁrst pretrain the searched architecture on ImageNet and then ﬁne-tune it on the target task.

Original: All the other settings in the experiments are the same as FNA. It is worth noting that FNA even achieves  higher mIOU by a narrow margin (0.1%)  in Tab.
Revised: It is worth noting that FNA even achieves a higher mIOU by a narrow margin (0.1%) than the ImageNet pre-trained one in Row (5).

Original: We conjecture that this may beneﬁt from the regularization  of parameters remapping in the parameters adaptation stage.
Revised: We conjecture that this may beneﬁt from the regularization effect of parameter remapping before the parameter adaptation stage.

Revised: All the experiments are conducted using the same searching and training settings for fair comparisons.

Revised: With parameters remapping applied on both stages, the adaptation achieves the best results in Tab. 5.

Revised: Especially, the remapping process before parameter adaptation tends to provide greater performance gains than the remapping before architecture adaptation.

Revised: All the experimental results demonstrate the importance and effectiveness of the proposed parameter remapping scheme.

Revised: the same as Tab. 5.

Original: [Table]
Revised: [Table]

Revised: 4. R ANDOM S EARCH E XPERIMENTS

Revised: We carry out the Random Search (RandSearch) experiments with the RetinaNet (Lin et al., 2017) framework on the MS-COCO (Lin et al., 2014a) dataset.

Revised: All the results are shown in the Tab. 6.

Revised: We purely replace the original differentiable NAS (DiffSearch) method in FNA with the random search method in Row (3).

Revised: We observe that FNA with RandSearch achieves comparable results with our original method.

Revised: It further conﬁrms that FNA is a general framework for network adaptation and has great generalization.

Revised: NAS is only an implementation tool for architecture adaptation.

Revised: The whole framework of FNA can be treated as a NAS-method agnostic mechanism.

Revised: It is worth noting that even using random search, our FNA still outperforms DetNAS (Chen et al., 2019b) with 0.2% mAP better and 150M MAdds fewer.

Revised: We further conduct similar ablation studies with experiments in Sec.

Revised: 4.3 about the parameter remapping scheme in Row (4) - (6).

Revised: All the experiments further support the effectiveness of the parameter remapping scheme.

Original: 4. S TUDY ON P ARAMETERS R EMAPPING
Revised: 4. S TUDY ON P ARAMETERS R EMAPPING

Original: We explore more other methods of the parameters remapping paradigm.
Revised: We explore more strategies for the parameters remapping paradigm.

Original: Similar to Sec.
Revised: Similar to Sec.

Original: 4.3, all the experiments are conducted with DeepLabv3 (Chen et al., 2017b) framework on Cityscapes dataset (Cordts et al., 2016).
Revised: 4.3, all the experiments are conducted with DeepLabv3 (Chen et al., 2017b) framework on Cityscapes dataset (Cordts et al., 2016).

Original: We make exploration from the following respects.
Revised: We make exploration from the following respects.

Original: For simplicity, we denote the weights of the seed network and the new network on the remapping dimension (output / input channel),
Revised: For simplicity, we denote the weights of the seed network and the new network on the remapping dimension (output / input channel),

Original: W s =
Revised: W s =

Original: ( W (1) s 
Revised: ( W (1) s . . .

Original: [Equation] and W n = ( W (1) n . . .
Revised: W ( p ) s ) and W n = ( W (1) n . . .

Original: W ( q ) n ) where q ≤ p .
Revised: W ( q ) n ) where q ≤ p .

Original: Remapping with BN Statistics on Width-level We review the formulation of batch normalization (Ioffe & Szegedy, 2015) as follows,
Revised: Remapping with BN Statistics on Width-level We review the formulation of batch normalization (Ioffe & Szegedy, 2015) as follows,

Original: [Equation]
Revised: [Equation]

Original: Input: the seed weights W s and the new weights W n , the reference vector v 1 // get indices of topk values of the vector 2 a ← topk - indices ( v , k = q ) 3 // sort the indices 4 sort ( a ) 5 for i ∈ 1 , 2 , . . . , q do W ( i ) n = W ( a ( i ) ) s 7 endOutput: W n with remapped values where x i = ( x (1) i . . .
Revised: where x i = ( x (1) i . . .

Original: x ( p ) i ) denotes the p -dimensional input tensor of the i th layer, γ ∈ R p denotes the learnable parameter which scales the normalized data on the channel dimension.
Revised: x ( p ) i ) denotes the p -dimensional input tensor of the i th layer, γ ∈ R p denotes the learnable parameter which scales the normalized data on the channel dimension.

Original: We compute the absolute values of γ as | γ | = ( | γ (1) | . . .
Revised: We compute the absolute values of γ as | γ | = ( | γ (1) | . . .

Original: | γ ( p ) | ) .
Revised: | γ ( p ) | ) .

Original: When remapping the parameters on the width-level, we sort the values of | γ | and map the parameters with the sorted top- q indices.
Revised: When remapping the parameters on the width-level, we sort the values of | γ | and map the parameters with the sorted top- q indices.

Original: More speciﬁcally, we deﬁne a weights remapping function in Algo.
Revised: More speciﬁcally, we deﬁne a weights remapping function in Algo.

Original: 1, where the reference vector is | γ | .
Revised: 1, where the reference vector is | γ | .

Revised: Input: the seed weights W s and the new weights W n , the reference vector v 1 // get indices of topk values of the vector 2 a ← topk - indices ( v , k = q ) 3 // sort the indices 4 sort ( a ) 5 for i ∈ 1 , 2 , . . .

Revised: , q do

Revised: W ( i ) n = W ( a ( i ) ) s 7 end

Revised: Output: W n with remapped values

Original: Remapping with Weight Importance on Width-level We attempt to utilize a canonical form of convolution weights to measure the importance of parameters.
Revised: Remapping with Weight Importance on Width-level We attempt to utilize a canonical form of convolution weights to measure the importance of parameters.

Original: Then we remap the seed network parameters with great importance to the new network.
Revised: Then we remap the seed network parameters with great importance to the new network.

Original: The remapping operation is conducted based on Algo.
Revised: The remapping operation is conducted based on Algo.

Original: 1 as well.
Revised: 1 as well.

Original: We experiment with two canonical forms of weights to compute the reference vector, the standard deviation of W s as ( std ( W (1) s ) . . .
Revised: We experiment with two canonical forms of weights to compute the reference vector, the standard deviation of W s as ( std ( W (1) s ) . . .

Original: std ( W ( p ) s )) and the L 1 norm of W s as ( || W (1) s || 1 . . .
Revised: std ( W ( p ) s )) and the L 1 norm of W s as ( || W (1) s || 1 . . .

Original: || W ( p ) s || 1 ) .
Revised: || W ( p ) s || 1 ) .

Original: [Table]
Revised: [Table]

Original: Remapping with Dilation on Kernel-level We experiment with another method of parameters remapping on kernel-level.
Revised: Remapping with Dilation on Kernel-level We experiment with another method of parameters remapping on kernel-level.

Original: Different from the function-preserving method deﬁned in Sec.
Revised: Different from the function-preserving method deﬁned in Sec.

Original: 3.1, we remap the parameters with a dilation manner as shown in Fig.
Revised: 3.1, we remap the parameters with a dilation manner as shown in Fig.

Original: The values in the convolution kernel without remapping are all assigned as 0 .
Revised: The values in the convolution kernel without remapping are all assigned as 0 .

Original: It is formulated as
Revised: It is formulated as

Original: [Equation]
Revised: [Equation]

Original: where W k × k and W 3 × 3 denote the weights of the new network and the seed network, h, w denote the spatial indices.
Revised: where W k × k and W 3 × 3 denote the weights of the new network and the seed network, h, w denote the spatial indices.

Original: Tab. 6 shows the experimental results.
Revised: Tab. 7 shows the experimental results.

Original: The network adaptation with the parameters remapping paradigm deﬁne in FNA achieves the best results.
Revised: The network adaptation with the parameter remapping paradigm deﬁne in FNA achieves the best results.

Original: Furthermore, the remapping operation of FNA is easier to implement compared to the several aforementioned ones.
Revised: Furthermore, the remapping operation of FNA is easier to implement compared to the several aforementioned ones.

Original: However, we explore limited numbers of methods to implement the parameters-remapping paradigm.
Revised: However, we explore limited numbers of methods to implement the parameter remapping paradigm.

Original: How to conduct the remapping strategy remains to be a signiﬁcative future work.
Revised: How to conduct the remapping strategy remains to be a signiﬁcative future work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose a fast neural network adaptation method (FNA) with a parameters remapping paradigm.
Revised: In this paper, we propose a fast neural network adaptation method (FNA) with a parameter remapping paradigm.

Original: We adapt the manually designed network MobileNetV2 to semantic segmentation and detection tasks on both architecture- and parameter- level.
Revised: We adapt the manually designed network MobileNetV2 to semantic segmentation and detection tasks on both architecture- and parameter- level.

Original: The parameters-remapping strategy takes full advantage of the seed network parameters, which accelerates both the architecture search and parameters ﬁne-tuning process greatly.
Revised: The parameter-remapping strategy takes full advantages of the seed network parameters, which accelerates both the architecture search and parameter ﬁne-tuning process greatly.

Original: With our FNA method, researchers and engineers could fast adapt more manually designed networks tothe backbone of various frameworks on different tasks.
Revised: With our FNA method, researchers and engineers could fast adapt more manually designed networks to various frameworks on different tasks.

Original: As there are lots of ImageNet pre-trained models available in the community, we could conduct adaptation with little cost and do more applications, e.g., face recognition, pose estimation, depth estimation, etc.
Revised: As there are lots of ImageNet pre-trained models available in the community, we could conduct adaptation with low cost and do more applications, e.g., face recognition, pose estimation, depth estimation, etc.

Original: We leave more efﬁcient remapping strategies and more applications for future work.
Revised: We leave more efﬁcient remapping strategies and more applications for future work.

Original: A A PPENDIX A. I MPLEMENTATION D ETAILS ON O BJECT D ETECTION
Revised: A A PPENDIX A. I MPLEMENTATION D ETAILS ON S EMANTIC S EGMENTATION

Revised: For architecture adaptation, the image is ﬁrst resized to 512 × 1024 and 321 × 321 patches are randomly cropped as the input data.

Revised: The output feature maps are down-sampled by the factor of 16 .

Revised: Depthwise-separable convolution is used in the ASPP module (Chen et al., 2017a;b).

Revised: The stages where the expansion ratio of MBConv is 6 in the original MobileNetV2 are searched and adjusted.

Revised: We set the numbers of layers in each searched stage of the super network as { 4 , 4 , 6 , 6 , 4 , 1 } .

Revised: We set a warm-up stage in the ﬁrst 5 epochs to linearly increase the learning rate from 1 × 10 − 4 to 0 .

Revised: Then, the learning rate decays to 1 × 10 − 3 with the cosine annealing schedule (Loshchilov & Hutter, 2017).

Revised: The batch size is set as 16 .

Revised: We use the SGD optimizer with 0 .

Revised: 9 momentum and 5 × 10 −weight decay for operation weights and the Adam optimizer (Kingma & Ba, 2015) with 4 × 10 −weight decay and a ﬁxed learning rate 1 × 10 − 3 .

Revised: For parameter adaptation, the training data is cropped as a 769 × 769 patch from the rescaled image.

Revised: The scale is randomly selected from [0 . 75 , 1 . 0 , 1 . 25 , 1 . 5 , 1 . 75 , 2 . 0] .

Revised: The random left-right ﬂipping is used.

Revised: We update the statistics of the batch normalization (BN) (Ioffe & Szegedy, 2015) foriterations before the ﬁne-tuning process.

Revised: We use the same SGD optimizer as the search process.

Revised: The learning rate linearly increases from 1 × 10 − 4 to 0 .

Revised: and then decays to 0 with the polynomial schedule.

Revised: The batch size is set as 16 .

Revised: A. I MPLEMENTATION D ETAILS ON O BJECT D ETECTION

Original: RetinaNet We describe the details in the search process of architecture adaptation.
Revised: RetinaNet We describe the details in the search process of architecture adaptation.

Revised: The depth settings in each searched stage are set as { 4 , 4 , 4 , 4 , 4 , 1 } .

Original: For the input image size, the short side is resized to 800 while the maximum long side is set as 1333.
Revised: For the input image size, the short side is resized to 800 while the maximum long side is set as 1333.

Original: For operation weights, we use the SGD optimizer with 1 × 10 − 4 weight decay and 0 .
Revised: For operation weights, we use the SGD optimizer with 1 × 10 − 4 weight decay and 0 .

Original: 9 momentum.
Revised: 9 momentum.

Original: We set a warmup stage in the ﬁrst 500 iterations to linearly increase the learning rate from 0 to 0 .
Revised: We set a warm-up stage in the ﬁrst 500 iterations to linearly increase the learning rate from 0 to 0 .

Original: Then we decay the learning rate by a factor of 0 .
Revised: Then we decay the learning rate by a factor of 0 .

Original: 1 at 8 and 11 epochs.
Revised: 1 at 8 and 11 epochs.

Original: For the architecture parameters, we use the Adam optimizer (Kingma & Ba, 2015) with 1 × 10 − 3 weight decay and a ﬁxed learning rate 3 × 10 − 4 .
Revised: For the architecture parameters, we use the Adam optimizer (Kingma & Ba, 2015) with 1 × 10 − 3 weight decay and a ﬁxed learning rate 3 × 10 − 4 .

Original: For the multi-objective loss function, we set λ as 0 .
Revised: For the multi-objective loss function, we set λ as 0 .

Original: 02 , τ as 10and γ as 5 .
Revised: 02 and τ as 10 .

Original: We begin optimizing the architecture parameters at 8 epochs.
Revised: We begin optimizing the architecture parameters at 8 epochs.

Original: We remove the random ﬂipping operation on input images in the search process.
Revised: We remove the random ﬂipping operation on input images in the search process.

Original: All the other training settings are the same as MMDetection (Chen et al., 2019a) implementation.
Revised: All the other training settings are the same as MMDetection (Chen et al., 2019a) implementation.

Original: For ﬁne-tuning of the parameters adaptation, we use the SGD optimizer with 5 × 10 − 5 weight decay and 0.9 momentum.
Revised: For ﬁne-tuning of the parameter adaptation, we use the SGD optimizer with 5 × 10 − 5 weight decay and 0.9 momentum.

Original: A similar warm-up procedure is set in the ﬁrstiterations to increase the learning rate from 0 to 0 .
Revised: A similar warm-up procedure is set in the ﬁrst 500 iterations to increase the learning rate from 0 to 0 .

Original: Then we decay the learning rate by 0 .
Revised: Then we decay the learning rate by 0 .

Original: 1 at the 8 and 11 epochs.
Revised: 1 at the 8 and 11 epochs.

Original: The whole architecture search process takes 14 epochs, 18 hours on 8 TITAN-Xp GPUs and the whole parameters ﬁne-tuning takes 10 hours on 8 TITAN-Xp GPUs.
Revised: The whole architecture search process takes 14 epochs, 18 hours on 8 TITAN-Xp GPUs and the whole parameters ﬁne-tuning takes 10 hours on 8 TITAN-Xp GPUs.

Original: SSDLite We resize the input images to 320 × 320 .
Revised: SSDLite We resize the input images to 320 × 320 .

Original: For operation weights in the search process, we use the standard RMSProp optimizer with 4 × 10 − 5 weight decay.
Revised: For operation weights in the search process, we use the standard RMSProp optimizer with 4 × 10 − 5 weight decay.

Original: The warm-up stage in the ﬁrstiterations increases learning rate from 0 to 0 .
Revised: The warm-up stage in the ﬁrst 500 iterations increases learning rate from 0 to 0 .

Original: Then we decay the learning rate by 0 .
Revised: Then we decay the learning rate by 0 .

Original: 1 at 16 andepochs.
Revised: 1 at 16 and 22 epochs.

Original: The architecture optimization starts at 12 epochs.
Revised: The architecture optimization starts at 12 epochs.

Original: We set λ as 0 . 2, τ as 10 and γ as 10 for the loss function.
Revised: We set λ as 0 . 2 and τ as 10 for the loss function.

Original: The other search settings are the same as the RetinaNet experiment.
Revised: The other search settings are the same as the RetinaNet experiment.

Original: For parameters adaptation, the initial learning rate is 0 . 2 and decays at 36, 50 and 56 epochs.
Revised: For parameters adaptation, the initial learning rate is 0 . 2 and decays at 36, 50 and 56 epochs.

Original: The training settings follow the SSD (Liu et al., 2016) implementation in MMDetection (Chen et al., 2019a).
Revised: The training settings follow the SSD (Liu et al., 2016) implementation in MMDetection (Chen et al., 2019a).

Original: The search process takes 24 epochs in total, 20 hours on 8 TITAN-Xp GPUs.
Revised: The search process takes 24 epochs in total, 20 hours on 8 TITAN-Xp GPUs.

Original: The parameters adaptation takes 60 epochs, 46 hours on 8 TITAN-Xp GPUs. 
Revised: The parameters adaptation takes 60 epochs, 46 hours on 8 TITAN-Xp GPUs. M B K 3 _ E M B K 7 _ E M B K 7 _ E

Original: [Table]
Revised: M B K 5 _ E M B K 7 _ E M B K 7 _ E M B K 7 _ E

Revised: M B K 7 _ E

Revised: M B K 7 _ E

Revised: M B K 5 _ E

Revised: M B K 7 _ E

Revised: M B K 3 _ E

Revised: C o n v 3 ×

Revised: M B K 3 _ E

Revised: M B K 3 _ E

Revised: M B K 7 _ E

Revised: M B K 3 _ E

Revised: M B K 3 _ E

Revised: M B K 5 _ E

Revised: M B K 7 _ E

Revised: M B K 7 _ E         (

Original: [Table]
Revised: [Table]

Revised: ) FNA on DeepLabv

Revised: M B K_ E

Revised: C o n v 3 ×

Revised: M B K_ E

Revised: M B K_ E

Revised: M B K_ E

Revised: M B K_ E

Revised: M B K_ E

Revised: M B K_ E

Revised: M B K_ E

Revised: M B K_ E

Revised: M B K_ E

Revised: M B K_ E

Revised: M B K_ E

Revised: M B K_ E

Revised: M B K_ E

Revised: M B K_ E

Revised: M B K_ E        (b) FNA on RetinaNet

Revised: M B K 3 _

Original: E
Revised: E

Revised: C o n v 3 ×

Revised: M B K 5 _

Revised: E

Original: M B K 7 _
Revised: M B K 5 _

Original: E
Revised: E

Original: M B K 7 _
Revised: M B K 5 _

Original: E
Revised: E

Revised: M B K 3 _

Revised: E

Original: M B K 5 _
Revised: M B K 5 _

Original: E
Revised: E

Revised: M B K 5 _

Revised: E

Original: M B K 7 _
Revised: M B K 7 _

Original: E
Revised: E

Revised: M B K 3 _

Revised: E

Original: M B K 7 _
Revised: M B K 7 _

Original: E
Revised: E

Original: M B K 7 _
Revised: M B K 7 _

Revised: E

Revised: M B K 7 _

Revised: E

Revised: M B K 7 _

Revised: E

Revised: M B K 5 _

Original: E
Revised: E

Original: [Figure]
Revised: M B K 7 _ E M B K 7 _ E

Revised: M B K 7 _

Revised: E

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]