Original: R ESIZABLE N EURAL N ETWORKS
Revised: R ESIZABLE N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we present a deep convolutional neural network (CNN) which performs arbitrary resize operation on intermediate feature map resolution at stagelevel.
Revised: In this paper, we present a deep convolutional neural network (CNN) that performs arbitrary resize operation on intermediate feature map resolution at stage-level.

Original: Motivated by weight sharing mechanism in neural architecture search, where a super-network is trained and sub-networks inherit the weights from the super-network, we present a novel CNN approach.
Revised: Motivated by weight sharing mechanism in neural architecture search, where a super-network is trained, and sub-networks inherit the weights from the supernetwork, we introduce a novel CNN approach.

Original: We construct a spatial supernetwork which consists of multiple sub-networks, where each sub-network is  asingle scale network that obtain a unique spatial conﬁguration, and the  convolutional layers are shared across all sub-networks. Such network, named as Resizable Neural Networks , are equivalent to training inﬁnite single scale networks, but has no extra computational cost.
Revised: We present Resizable Neural Networks (ResizableNet), which resize feature map with arbitrary scaling factor at stage level. The inference cost of ResizableNet is equivalent to a network with a single scale because the weights of a convolutional layer are shared across diversiﬁed scales.

Original: Moreover, we present a training algorithm such that all sub-networks achieve better performance than individually trained counterparts.
Revised: Additionally, we provide essential techniques to build and train our propose ResizableNet.

Original: On large-scale ImageNet classiﬁcation, we demonstrate its effectiveness  on  various modern network architectures such as MobileNet, ShufﬂeNet, and ResNet.
Revised: We demonstrate the effectiveness of ResizableNet on large-scale ImageNet classiﬁcation with various modern network architectures such as MobileNet, ShufﬂeNet, and ResNet.

Original: To go even further, we present three variants of resizable networks: 1) Resizable as
Revised: To go even further, we present three applications of resizable networks: 1) Resizable Augmentation ( Resizable-Aug ).

Original: Architecture Search ( Resizable-NAS ). On ImageNet, Resizable-NAS ResNetattain 0.4% higher on accuracy and 44% smaller than the baseline model.
Revised: We present a data augmentation technique based on ResizableNet.

Original: 2) Resizable as Data Augmentation ( Resizable-Aug ). While we use resizable networks as a data augmentation technique, it obtains superior performance on ImageNet classiﬁcation, outperform AutoAugment by 1.2% with ResNet-50. 3) Adaptive
Revised: It obtains superior performance on ImageNet classiﬁcation, outperform AutoAugment by 1.2% with ResNet-50. 2) Resizable Network for Neural Architecture Search ( Resizable-NAS ). We introduce ResizableNet as a single network that can deploy on many different hardware to meet various efﬁciency constraints.

Original:  Resizable  Network ( Resizable-Adapt ).
Revised: Adaptive Resizable Neural Network ( Resizable-Adapt ).

Original: We introduce the adaptive resizable networks as dynamic networks, which further improve the performance with less computational cost via data-dependent inference.
Revised: We introduce the adaptive resizable networks as dynamic networks, which further improve the performance with less testing cost via data-dependent inference.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Scale is the fundamental component of the physical world, and it has been an active research topic in the ﬁeld of computer vision.
Revised: Scale is a fundamental component of the physical world, and it has been an active research topic in the ﬁeld of computer vision.

Original: The latest design of scales for the Convolutional Neural Networks (CNN) fall into two categories: single scale networks and multi-scale networks.
Revised: The latest design of scales for the Convolutional Neural Networks (CNN) fall into two categories: single scale networks and multi-scale networks.

Original: In single scale networks, such as ResNet (He et al.
Revised: In single scale networks, such as ResNet (He et al.

Original: , only a ﬁxed scale representation is learned from the feature maps.
Revised: , only a ﬁxed scale representation is learned from the feature maps.

Original: Alternatively, in the multi-scale networks, e.g., image pyramid (Sermanet et al., 2013; Dalal & Triggs, 2005)/feature pyramid (Ghiasi et al., 2019; 
Revised: Alternatively, in the multi-scale networks, e.g., image pyramid (Sermanet et al., 2013; Dalal & Triggs, 2005) /feature pyramid (Ghiasi et al., 2019; Chen et al., 2017a) /ﬁlter

Original: Chen et al., 2017a)/ﬁlter pyramid structure (Szegedy et al., 2015), features from different resolutions are fused in a network; thus the models learn various type of scales from the objects.
Revised: pyramid structure (Szegedy et al., 2015), features from different resolutions are fused in a network; thus the models learn various type of scales from the objects.

Original: As a result, these architectures obtain state-of-the-art performance on scale critical tasks, such as object detection (Lin et al.
Revised: As a result, these architectures obtain state-ofthe-art performance on scale critical tasks, such as object detection (Lin et al.

Original: and semantic segmentation (Chen et al.
Revised: and semantic segmentation (Chen et al.

Original: (2017a)).
Revised: (2017a)).

Original: It can be observed that both single scale networks and multi-scale networks essentially perform representation learning on a predeﬁned feature map scales.
Revised: It can be observed that both single scale networks and multiscale networks essentially perform representation learning on a predeﬁned feature map scales.

Original: This natural and intuitive choice, despite their success on various computer vision tasks, limits the CNNs to a constrained predeﬁned scaling mechanism for dealing with scale variation problem ((He et al., 2019)).
Revised: This natural and intuitive choice, despite their success on various computer vision tasks, limits the CNNs to a constrained predeﬁned scaling mechanism for dealing with scale variation problem (He et al., 2019b).

Original: Our approach is motivated by the recent emergence of weight sharing approach on neural architecture search (NAS) (Pham et al.
Revised: Our approach is motivated by the recent emergence of weight sharing approach on neural architecture search (NAS) (Pham et al.

Original: ; Bender et al.
Revised: ; Bender et al.

Original: ; Guo et al.
Revised: ; Guo et al.

Original: (2019)), where an over-parameterized super-network is trained and sub-networks inherit the weights from the supernetwork.
Revised: (2019)), where an over-parameterized super-network is trained, and sub-networks inherit the weights from the supernetwork.

Original: Generally, there are a large number of architectural conﬁgurations within a super-network.
Revised: Generally, there are a large number of architectural conﬁgurations within a super-network.

Original: In this work, we proposea convolutional neural network with inﬁnite single scale networks, named as Resizable Neural Networks . It can be viewed as aspatial super-network that includes vast subnetworks which varies in spatial conﬁgurations.
Revised: In this work, we propose Resizable Neural Networks (ResizableNet), which can be viewed as a super-network from a scale perspective. A ResizableNet resizes a feature map with an arbitrary scaling factor at stage level. With a sequence of scaling factors at different stages, we obtain a scaling conﬁguration, which can be used to deﬁne the feature map resolution of a network.

Original: Speciﬁcally, a resizable network contains multiple 
Revised: In other words, a ResizableNet has multiple scaling conﬁgurations. It is a generalization of modern

Original: [Figure]
Revised: [Figure]

Original: sub-networks: each sub-network obtains a unique feature map scale.
Revised: (cid:13)(cid:4)(cid:14)(cid:7)(cid:17)(cid:4)(cid:3)(cid:1)(cid:5)(cid:4)(cid:2)(cid:15)(cid:16)(cid:13)(cid:4)(cid:1)(cid:9)(cid:2)(cid:12)(cid:1)(cid:13)(cid:4)(cid:14)(cid:11)(cid:8)(cid:16)(cid:15)(cid:7)(cid:11)(cid:10)

Original:  The resize operation for feature map resolution is performed on the stage-level by a non-integer scaling ratio. These sub-networks, called spatial sub-networks, are trained jointly in the resizable networks, and the weights of all convolution layers are shared.
Revised: CNN architecture in terms of scale, where one ﬁxed scaling conﬁguration is adopted. The major advantage of ResizableNet is that the weights of a convolution layer with diversiﬁed scale factors are shared, thus substantially reduce the computational costs and memory footprints during inference.

Original: Figure 1 gives an illustration of the resizable networks compared with the baseline network and input resize only network.
Revised: Figure 1 gives an illustration of the resizable net compared with the baseline network and input resize only network.

Original: Note that the network with resized input resolution is a special form of spatial sub-networks.
Revised: Note that the network with resized input resolution is a special form of spatial sub-networks.

Original: Using theapproach of resizable networks, the network capacity can be signiﬁcantly increased as an exponential number of  scales for input has been trained by the networks implicitly.
Revised: Training the resizable neural network is a non-trivial task because of two characteristics of ResizableNet. First, it requires to optimize a network with diversiﬁed feature map scales. It is challenging since there are many scaling conﬁgurations, which means it is computationally impractical to enumerate all of them, while random sampling cannot guarantee the performance of the network. Secondly, the interference in a layer between different scales leads to accuracy drops because the weights of a layer with various scales are shared in ResizableNets.

Original: Additionally, we present a training algorithm totrain the proposed resizable networks, which  resolve the unfair sampling issue as well as the discrepancy of batch normalization statistics between different spatial sub-networks.
Revised: We conjecture that the accuracy drop is due to the discrepancy of feature mean and variance across different scales, which leads to inaccurate batch normalization statistics. To resolve these two problems, we introduce fair sampling and scale-aware batch normalization for training the resizable network.

Original: With our proposed training algorithm, we can omit the process ofretraining, which is a standard procedure in weight sharing NAS, and obtain higher accuracy for all spatial sub-networks than individually trained counterparts.
Revised: The fair sampling samples the scaling factors multiple times of a layer in an update step without reusing the same scaling factor, thus maintain the performance of the network with different scaling conﬁguration to some extent. The scale-aware batch normalization introduces to use a group of BN layers to replace the shared BN layer, and each scaling factor obtains an independent BN layer. Thus it solves the inconsistent

Original:  In this paper, we ﬁrst validate the effectiveness of resizable neural networks compared with standalone networks.
Revised: BN statistics issue. In this paper, we ﬁrst validate the effectiveness of resizable neural networks compared with standalone networks on ImageNet.

Original: We demonstrate superior performances of Resizable Neural Network on efﬁcient networks such as MobileNet V1, ShufﬂeNet V2 as well as deep networks, i.e., ResNet-50.
Revised: We demonstrate superior performances of Resizable Neural Network on various network architectures, such as ResNet-50, ShufﬂeNetV2 and MobileNetV1.

Original: The experiments cover the range of modern neural architecture setting such as non-residual block, residual bottleneck, depth-wise convolution, and deep networks.

Original: Variants of Resizable Neural Network.
Revised: Applications of Resizable Neural Network.

Original: We further demonstrate three resizable network variants: resizable as architecture search ( Resizable-NAS ), resizable as data augmentation ( Resizable-Aug ) and resizable for data-dependent inference ( Resizable-Adapt ).Resizable-NAS.
Revised: We further demonstrate three applications of ResizableNet: ResizableNet as architecture search ( Resizable-NAS ), ResizableNet for data augmentation ( Resizable-Aug ) and ResizableNet for data-dependent inference ( Resizable-Adapt ).

Original: We demonstrate that our resizable networks can adaptively switch between subnetworks to fulﬁl the efﬁciency constraints on the target platform, thus achieve the run-time model compression objective.

Original: We show that there consistently exists smaller sub-network with similar (even higher) accuracy than the baseline.

Original: For example, Resizable-NAS ResNet-50 is 44% smaller on model size and 0.4% higher on accuracy compare with the baseline.

Original: This result is signiﬁcantly better than state-of-the-art compression methods, i.e., the accuracy of NISP (Yu et al.

Original: (2018b)) is dropped by 0.9% with the same computational complexity.

Original: Resizable-Aug.
Revised: Resizable-Aug.

Original: We introduce Resizable-Aug, a data augmentation technique using resizable network training scheme.
Revised: We introduce Resizable Augmentation (Resizable-Aug), a data augmentation technique based on ResizableNet.

Original: We are able to improve ResNet-50 by 2.4% compare with baseline. This result is also higher than state-of-the-art data augmentation methods, for instance, it is1.2% higher than AutoAugment (Cubuk et al.on ImageNet classiﬁcation task with no search cost.
Revised: We are able to improve ResNet-50 by 2.4% compare with baseline, which is higher than AutoAugment (Cubuk et al.

Original: Additionally, combine our augmentation technique on a new baseline model, we obtain a model achieve 77.8% with 365M FLOPs.
Revised: Additionally, combine our augmentation technique on a new baseline model, we obtain a model achieve 77.8% with 365M FLOPs.

Original: This is signiﬁcantly better than previous state-of-the-art model EfﬁcientNet-B0 (Tan & Le (2019)) which trained with AutoAugment (Cubuk et al.
Revised: This is signiﬁcantly better than previous state-of-the-art model EfﬁcientNet-B0 (Tan & Le (2019)) which trained with AutoAugment (Cubuk et al.

Original: (2018)), 1.5% higher on accuracy with 25M less FLOPs.
Revised: (2018)), 1.5% higher on accuracy with 25M less FLOPs.

Original: Resizable-Adapt.
Revised: Resizable-NAS.

Original: We introduce  an auxiliary module called ResizeLearner . By integrating into resizable networks, it allows data-dependent inference.
Revised: We introduce ResizableNet for NAS (Resizable-NAS) that can deploy on many different hardware platforms to fulﬁll the efﬁciency constraints with one trained model. Given an efﬁciency constraint, we can search the specialized scaling conﬁguration of ResizableNet for the target platform.

Original: This is achieved by adopting ResizeLearner to assign the optimal spatial sub-networkto each sample. Compared with  the naive resizable networks, this approach not only lowers the computational cost during inference, but improve representation learning via switch the spatial policy from implicit to explicit.
Revised: Thus it achieves the run-time model compression objective. Additionally, we observed that there consistently exists smaller sub-network with similar (even higher) accuracy than the baseline network. For example, Resizable-NAS ResNet-50 is 44% smaller on model size and 0.4% higher on accuracy compare with the baseline.

Original: In particular, Resizable-Adapt
Revised: Resizable-Adapt.

Revised: We investigate that if there exists an optimal scaling conﬁguration for every input.

Revised: We introduce a data-dependent inference network, named as Adaptive ResizableNet (ResizableAdapt).

Original:  ResNet-50 improve the performance of Resizable-NAS ResNet-50 up to 0.5% , depends on the target spatial sub-networks.
Revised: In our experiment, Resizable-Adapt ResNet-50 improve the performance of Resizable-NAS ResNet-50 up to 0.5% , depends on the target spatial sub-networks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Multi-scale Training.
Revised: Multi-scale Training.

Original: Our work is most related to multi-scale learning, which is important in image recognition (Li et al.
Revised: Our work is most related to multi-scale learning, which is important in image recognition (Li et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: , object detection(Lin et al.
Revised: , object detection(Lin et al.

Original: (2017); Singh et al.
Revised: (2017); Singh et al.

Original: ; Singh & Davis (2018)) and instance segmentation (Zhao et al.
Revised: ; Singh & Davis (2018)) and instance segmentation (Zhao et al.

Original: Chen et al.
Revised: Chen et al.

Original: (2017b)).
Revised: (2017b)).

Original: There are several approaches to fusing information of images at different visual resolutions.
Revised: There are several approaches to fusing information of images at different visual resolutions.

Original: The naive approach is image pyramid (Sermanet et al.
Revised: The naive approach is image pyramid (Sermanet et al.

Original: ; Dalal & Triggs (2005);
Revised: ; Dalal & Triggs (2005);

Original: Felzenszwalb et al.
Revised: Felzenszwalb et al.

Original: ; Cai et al.
Revised: ; Cai et al.

Original: (2016)), where an input image is passed through a model multiple times at different resolutions.
Revised: (2016)), where an input image is passed through a model multiple times at different resolutions.

Original: Our work introduce the resizable networks which comprise inﬁnite single scale network, that acted as a implicit multi-scale networks.
Revised: Our work introduce the resizable networks which comprise inﬁnite single scale network, that acted as a implicit multi-scale networks.

Original: Dynamic Neural Networks.
Revised: Dynamic Neural Networks.

Original: Our work is related to dynamic neural networks.
Revised: Our work is related to dynamic neural networks.

Original: Dynamic neural networks aim to train a single network to support different architectural conﬁguration, especially adjust networks according to input samples.
Revised: Dynamic neural networks aim to train a single network to support different architectural conﬁgurations, especially adjust networks according to input samples.

Original: Slimmable networks(Yu et al.
Revised: Slimmable networks (Yu et al.

Original: (2018a)) train a single model to support multiple width conﬁguration by switching width at runtime.
Revised: (2018a)) train a single model to support multiple width conﬁgurations by switching width at runtime.

Original: These dynamic networks are limited by two factors: 1) Performance.

Original: Most dynamic networks methods sacriﬁce accuracy in exchange of adaption in inference.

Original: 2) Flexibility. The ﬂexibility previous methods are limited, and the degree of ﬂexibility are limited to width or depth, and the number of sub-networks choice is . For example, slimmablenet (Yu et al. (2018a)) have only four to eight switches to adapt in inference.
Revised: Universal Slimmable (Yu & Huang (2019)) is proposed to expand the number of width that is supported by training a single network.

Original: Our proposed model is highly ﬂexible on spatial-level, while obtain an excellent performance gain over all individual network.
Revised: In our paper, we explore the dynamic network on the spatial level.

Original: Model Compression.
Revised: Model Compression.

Original: Compressing Over-parametrized model network with minimal performance drop is appealing on resource-constrained devices.
Revised: Compressing Over-parametrized model network with minimal performance drop is appealing on resource-constrained devices.

Original: Modern compression methods including channel pruning (Liu et al.
Revised: Modern compression methods including channel pruning (Liu et al.

Original: ; He et al.
Revised: ; He et al.

Original: , knowledge distillation(Hinton et al.
Revised: knowledge distillation (Hinton et al.

Original: ; Furlanello et al.
Revised: ; Furlanello et al.

Original: ; Romero et al.
Revised: ; Romero et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: ; Chen et al.

Original: (2015a))

Original: , hashing (Chen et al.
Revised: , hashing (Chen et al.

Original: (2015b)) , network binarization(Courbariaux et al. ; Rastegari et al. 
Revised: network binarization (Courbariaux et al. (2016); Rastegari et al. (2016)).

Original: Our work explore network compression at the runtime on spatial-level, which is complementary to existing compression methods.
Revised: Our work explore network compression at the runtime on spatial-level, which is complementary to existing compression methods.

Original: Neural Architecture Search.
Revised: Neural Architecture Search.

Original: Recently the design of efﬁcient neural networks has largely shifted from leveraging human knowledge to automatic methods, which is known as neural architecture search(NAS).
Revised: Recently the design of efﬁcient neural networks has largely shifted from leveraging human knowledge to automatic methods, which is known as neural architecture search (NAS).

Original: Designing search space is of the most important in NAS on various vision tasks such as image recognition (Radosavovic et al.
Revised: Designing search space is of the most important in NAS on various vision tasks such as image recognition (Radosavovic et al.

Original: ; Cai et al.
Revised: ; Cai et al.

Original: ; Howard et al.
Revised: ; Howard et al.

Original: ; Guo et al.
Revised: ; Guo et al.

Original: (2019); Tan et al.
Revised: (2019); Tan et al.

Original: , instance segmentation (Liu et al.
Revised: , instance segmentation (Liu et al.

Original: and object detection(Ghiasi et al.
Revised: and object detection(Ghiasi et al.

Original: ; Chen et al.

Original: Resizable networks can be consider as a designed search space, where each spatial sub-networks is a optional architecture.
Revised: Resizable networks can be consider as a designed search space, where each spatial sub-networks is a optional architecture.

Original: Data Augmentation.
Revised: Data Augmentation.

Original: Data augmentation is effective for improving the accuracy of modern neural network in various computer vision tasks (Szegedy et al.
Revised: Data augmentation is effective for improving the accuracy of modern neural network in various computer vision tasks (Szegedy et al.

Original: Many new augmentation techniques such as Cutout (DeVries & Taylor (2017)) and Mixup (Zhang et al.
Revised: Many new augmentation techniques such as Cutout (DeVries & Taylor (2017)) and Mixup (Zhang et al.

Original: on input image have shown promising results.
Revised: on input image have shown promising results.

Original: Recently, a new trend is raised that automatically learn the data transformation (Ho et al.
Revised: Recently, a new trend is raised that automatically learn the data transformation (Ho et al.

Original: Cubuk et al.
Revised: Cubuk et al.

Original: (2018) leverage RNN controller to automatically search the best data augmentation combination for training in a large augmentation sub-policy space.
Revised: (2018) leverage RNN controller to automatically search the best data augmentation combination for training in a large augmentation sub-policy space.

Original: We explore the data augmentation technique on feature map spatial-level by letting the network learns different feature map scales. 
Revised: We explore the data augmentation technique on feature map spatial-level by letting the network learns different feature map scales. Random Sampling

Original: R ESIZABLE N EURAL N ETWORK
Revised: Fair Sampling sample without replacement step t+1: ! " = 1. # = 0. $ = 0.

Original: In this section, we present the resizable networks.
Revised: % = 0. stage_ stage_ stage_ stage_ first sample !

Original: Section 3.1 gives a description of resizable networks.
Revised: $ third sample second sample fourth sample stage_ ! " ! # ! $ ! " ! # ! $ ! "

Original: Section 3.2 describe the training algorithm. Section 3.2 to 3.4 describe three variants of resizable networks respectively.
Revised: step t: … (cid:9)(cid:5)(cid:8)(cid:5)(cid:2)(cid:11)(cid:5)(cid:4)(cid:1)(cid:10)(cid:2)(cid:7)(cid:8)(cid:6)(cid:5)(cid:4)(cid:1)(cid:10)(cid:3)(cid:2)(cid:6)(cid:5)(cid:10)

Original: 3. D EFINITION OF  R ESIZABLE N  ETWORKS
Revised:  R ESIZABLE N EURAL N ETWORKS

Original: Let s ij ∈ [0 , 1] be a scale factor , each s ij is a j th scale ratio at i th stage. We use a set of scale factors to describe a network. We denote the S
Revised: Our CNN architecture has N stages.

Original: = { s ij } ni =1 ,j ∈ N to be the spatial conﬁguration , where N denotes the number oftotal stages in a network.
Revised: Each stage consists of a sequence of layers that have the same resolution of the feature map.

Original: Let x in denotes input tensor as C × H in × W in , where C denotes the number of channels, H in denotes the height and W in denotes the width ofthe input tensor. We can formalize the feature map resize operation as:
Revised: The feature map resolution is gradually reduced by resizing (denoted as G ) with a scaling factor (denoted as s ), at the beginning of a stage. From the scale perspective, we denote a network by a sequence of resize operation with scaling factors, named as scaling conﬁguration (denoted as S ), denoted formally as

Original: [Equation] 
Revised: [Equation] [Equation]

Original: where G ( · ; · ) denotes the  resize operation. x out is the output tensor as C × H out × W out after resize operation. 
Revised: where i denotes the i th stage that performs resize where i denotes the i th stage that performs resize operation. In the modern CNN architectures (see Figure 1(a)), G is convolution with stride 2 (He et al. ; Huang et al.

Original: The typically choice for the scale is s ij = 0 .
Revised: ; Xie et al. (2017)) or max/average pooling (He et al. (2019a)

Original: (He et al.
Revised: ; Li et al.

Original: and G ( · ; · ) use the max/average pooling or convolutional layer with stride equal to 2. Thus we can write the conventional single scale network as S =
Revised: to reduce the feature map size by half ( s = 0 . 5 ). These CNN architectures learn the input image with a ﬁxed scale. Recent works have indicated the importance of learning the images with various scales.

Original: 5 j } j ∈ N .
Revised: Several works design CNN architecture towards adding parallel branches to deal with different input scales (Li et al. ; Wang et al. The design of these architectures improve generalization. However, they are inefﬁcient since extra branches require extra parameters and increase FLOPs.

Original: Unlike the conventional single scale network, in the spatial sub-networks , the s ij is a arbitrary number and G ( · ; · ) represent the  bilinear interpolation.
Revised: In order to learn image features at different scales without increasing FLOPs, we propose the resizable neural networks. The ResizableNets (see Figure 1c) is a generalization of the CNN architectures in terms of scale. In ResizableNets, G is bilinear interpolation.

Original: This leads to a ﬂexible scaling policy that enable the feature mapresolution change to an arbitrary size.
Revised: It allows the feature map to resize to any resolution.

Original: Thus we can formalized the resizable networks as the ensemble of an inﬁnite number of single scale networks:
Revised: As such, modern CNN architecture can be viewed as a special case of ResizableNets.

Original: [Equation] where RS denotes the resizable networks, N is the total number of single scale networks.
Revised: During training, the ResizableNets are trained with different feature map scale dynamically.

Original: S k is some single scale network with ﬂexible scaling policy as we described in equation 1. From theequation 2, resizable networks can be viewed as a set of inﬁnite single scale network, where each single scale network S k obtain a unique spatial conﬁguration while all networks share the same convolutional layers.
Revised: Moreover, the ResizableNets share the weights of a convolutional layer across different scaling factors. In this way, we allow the network to learn multiple scales of input without extra inference cost. Training the resizable networks is a non-trivial task due to the ﬂexible resize operation and weight sharing mechanism.

Original: We  present a training algorithm for resizable networks in Section 3.2. 3. T RAINING D ETAILS
Revised: We now discuss the inherent problems of ResizableNets and present solutions to resolve these problems.

Original: Fair Sampling.
Revised: Fair Sampling.

Original: Thenaive approach to train a resizable networks is to uniformly select a spatial conﬁguration at each optimization step and update the weights accordingly.
Revised: The resizable neural network focuses on train one network to support various scaling conﬁgurations in inference without retraining.

Original: However, such sampling method brings out training bias.
Revised: From this perspective, the naive approach is to randomly sample a scaling factor from a predeﬁned scaling factor list.

Original: Each convolutional layer at different spatial conﬁgurationis trained with different mini-batch data and learning rate, which causes the issue of unfair training between different sub-networks.
Revised: With a sequence of scaling factors at each stage, we obtain a scaling conﬁguration. We update the weights of ResizableNet use this sampled scaling conﬁguration and repeat this process. However, random sampling brings instability into the ResizableNet training process, because different scaling factor list in a layer are sampled with different number of times by random sampling.

Original: To be concrete, some convolutional layers can go through more training iterations than the others.
Revised: As a result, some scaling factors in a convolution have more training opportunities than others, lead to imbalance performance between different scaling conﬁgurations.

Original: To reduce the training bias, we introduce a fair sampling method.
Revised: To tackle this problem, we introduce the fair sampling.

Original: The ideology behind our proposed sampling is simple: to fairly train every scale factors. Speciﬁcally, we propose to train multiple conﬁgurations in each iteration and accumulate the gradients of shared convolutional layers.
Revised: In each update step, the fair sampling samples a scaling factor from the scaling factor list multiple times.

Original: For each iteration, we uniformly sample scale ratios in a non-repetitive manner: if aconvolutional layer with the particular scale factor has already been chosen, then this particular layers will not be picked again in the same iteration.
Revised: Each time it selects a scaling factor in a layer if and only if this particular scaling factor in this particular layer has not been selected before in the same update step. In this regard, we ensure that all scaling factors in a layer are selected and trained to undergo the same training process. Figure 2 gives an illustration of fair sampling.

Original: Thus we ensure all scale ratios for convolution is trained fairly.
Revised: Moreover, we show in Appendix C that training of ResizableNet with fair sampling is faster than random sampling.

Original: Scale-Aware Batch Normalization.
Revised: Scale-Aware Batch Normalization.

Original: One issue with training  a network layer with different feature map size  is  the discrepancy of both feature  mean and variance between different sampled scales.
Revised: Batch normalization (BN) layer is a standard normalization method to accelerate training and improve the generalization of CNN. It is common to use a ” ConvBN ” couple as a building block. Because the feature map size of convolution is constantly changing during the training of ResizableNets, we say that a BN layer is shared by a convolution layer with multiple sizes. The shared BN layers are not applicable in ResizableNets since different resolutions of the same feature map are aggregated according to Algorithm 1, which results in different mean and variance.

Original: Thus shared batch normalization (BN) (Ioffe & Szegedy (2015)) layer brings instability into the training process, degrade the model performance.
Revised: It leads to inaccurate batch normalization statistics across different scaling factors in shared layers, based on our experiment in Table 9, it impairs the test accuracy.

Original: To address this issue, we  use scale-aware batch normalization in the network.
Revised: To tack this issue, we propose to use an independent BN layer to respond to a scale factor in a layer.

Original: Speciﬁcally, the shared convolutional layer is followed by a group of BN layers, where  the number of BN layers in this group is equal to the number of scale factors.
Revised: In other words, we use a ” Conv- { BN 1 , ..., BN k } ” as a building block, where k denotes as the number of scaling factors in a layer. As such, each BN layer speciﬁcally calculate BN statistics and learns the parameters of corresponding feature map resolution in the stage.

Original: Thus each scale ratio is corresponding to an independent BN layer which is scale-aware. Note that the extra parameters and expense of memory introduced by scale-aware batch normalization is negligible.
Revised: The number of extra parameter of scale-aware BN layers is negligible since the number of parameters in BN layers usually take less than one percent of the total model size. During testing, only the BN layer corresponding to the activated scaling factor is used. Thus its behavior is the same as a regular building block.

Original: Training Algorithm.
Revised: Scaling Factor List L. To help to understand the training process of ResizableNets, we describe scaling factor list L, which we used in Algorithm 1. The scaling factor s for a layer is discrete. It is chosen from a pre-deﬁned scaling factor list L .

Original: Algorithm 1 illustrates the training framework to efﬁciently and effectively train aresizable network combine with fair sampling and BN calibration techniques.{+we use L = (0 . 45 , 0 . 65 , 0 . 85 , 1 . 0) , then there is 4 4 = 
+}
Revised: An typical of scaling factor list we used in the experiments is L = (0 . 45 , 0 . The number of scaling conﬁgurations of ResizableNets is related to the number of stages and the length of L (number of scaling factors). For example, if a network contains four stages, and {+we use L = (0 . 45 , 0 . 65 , 0 . 85 , 1 . 0) , then there is 4 4 = 
+}

Original: During training, we adopt fair sampling, which gradients are accumulated and fairly sample spatial conﬁgurations in each iteration and then update the weights.
Revised: scaling conﬁgurations. We will discuss the impact of different choice of s and number of s on ResizableNets in Appendix A. Algorithm 1 Training resizable neural networks RS.

Original: Require:
Revised: Require:

Original: Initialize resizable network RS.
Revised: Initialize resizable network.

Original: Predeﬁned spatial list L for i
Revised: Deﬁne scaling factor list L .

Original:  = 1, ..., niters do Get data x and target y of current mini-batch.
Revised: for i = 1, ..., n do Get data x and target y of current mini-batch.

Original: Clear gradients for  for j = 0, ..., len( L ) doend end
Revised: Clear gradients for all parameters, optimizer.zero grad() for j = 0, ..., len( L ) do

Original: 3. R ESIZABLE N EURAL N ETWORK A S M ODEL S EARCH
Revised: Follows the fair sampling, sample scaling conﬁguration S j , Select the scale-aware batch normalization for each stage. Execute current scaling conﬁguration. ˆ y

Revised: = S j

Revised: ( x ) .

Revised: Compute loss for S j , loss s j = criterion

Revised: ( ˆ y, y )

Revised: Compute gradients for S j , loss s j .backward

Revised: () end

Revised: Update weights, optimizer.step().

Revised: end

Revised:  A PPLICATIONS 4. F OR D ATA A UGMENTATION

Revised: Previous data augmentation techniques transform the input image directly (Zhang et al.

Revised: Cubuk et al.

Revised: ; DeVries & Taylor (2017)).

Revised: In contrast, our method proposes to transform the feature map by scale at stage level, named as Resizable Augmentation (denoted as Resizable-Aug).

Revised: The network learns various scale combinations of feature map by switching scaling conﬁguration during training, thus potentially improve generalization.

Revised: The goal of Resizable-Aug is to increase the accuracy of the target network.

Revised: The challenge is that a convolutional layer shares the weights across all scaling factors.

Revised: As a result, training a convolution with a scale could affect the accuracy of the same convolution with other scales.

Revised: To address this challenge, we decouple the model training stage and the model ﬁne-tuning stage.

Revised: Precisely, in the model training stage, we follow the training Algorithm 1.

Original: Using the  training algorithm in Section 3.2, the resizable networks now contain multiple spatial sub-networks, where all sub-networks attain good performance.
Revised: As such, the model learns the input with various scales. In the model ﬁne-tuning stage, we speciﬁcally ﬁne-tune the target network. In other words, we ﬁx the target scaling conﬁguration and optimize the network with target scaling conﬁguration only. We empirically use two-third of total training epochs at the model training stage and the rest training epochs at the model ﬁne-tuning stage.

Original: Thus, with a trained super-network, the primary objective is to select the best-performing spatial sub-network given different efﬁciency constraints (FLOPs, latency, energy, etc.).
Revised: There is a rising demand for designing specialized neural architecture with a short response time on different hardware (e.g., mobile phones). Recently, neural architecture search methods are proposed to search for the specialized network architecture on a speciﬁc hardware. However, searching network architectures for every hardware is computationally expensive, since different hardwares or even the same hardware with different equipment conditions can have different efﬁciency constraints (e.g., FLOPs and latency).

Original: Generally, various search methods can be combined with the resizable networks forsearching the best-ﬁt sub-networks.
Revised: We present resizable neural networks for neural architecture search (denoted as Resizable-NAS). The ResizableNets can be intuitively viewed as a search space, where scale conﬁgurations are candidates in the search space. Every scale conﬁguration of ResizableNets can be instantiated as a subnetwork. As such, we can search for the specialized sub-network that fulﬁll the hardware efﬁciency constraints.

Revised: More importantly, with one trained ResizableNets, it can be deployed to diversiﬁed hardware simultaneously, thus cut down the computational cost for retraining networks.

Revised: To achieve this, we ﬁrst train a ResizableNet with Algorithm 1.

Revised: With a trained ResizableNet, we can search for the sub-network for the given efﬁciency constraints.

Revised: Speciﬁcally, we apply a search method on the ResizableNet to ﬁnd the specialized sub-networks.

Original: For simplicity, we present a naive approach.
Revised: For simplicity, we present a naive approach.

Original: We enumerate allspatial sub-networks and build a lookup table to include information such as accuracy, FLOPs and latency.
Revised: We enumerate all sub-networks and build a lookup table to include information such as accuracy, FLOPs, and latency.

Original: Therefore, given a target hardware and efﬁciency constraints, we can query the lookup table to get the best-ﬁt sub-network in the resizable networks.
Revised: Therefore, given the efﬁciency constraints, we can query the lookup table to get the specialized sub-network.

Original: There is no computational cost of querying the lookup table. Additionally,when the best-ﬁt spatial sub-network is chosen from the lookup table, we canﬁnetune this sub-network to increase the accuracy on target dataset further. It allows the  sub-network to recover its representation learning ability when it is under-trained in the resizable networks. It also mitigates the co-adaptation issue which we will discuss in Section 3.3.
Revised: Additionally, we can increase the accuracy of the specialized sub-network by ﬁne-tuning.

Original: Empirically, we ﬁnd that ﬁne-tune 15 epochs is sufﬁcient, and the ﬁne-tuning process increases the accuracy considerably.
Revised: Empirically, we ﬁnd that ﬁne-tune one epoch is sufﬁcient to increase the accuracy considerably, while ﬁne-tune ﬁfteen epochs can obtain the highest accuracy.

Original: 3. R ESIZABLE N EURAL N ETWORK AS D ATA A UGMENTATION
Revised: 4. F OR D YNAMIC I NFERENCE

Original: In Section 3.2, we present a training algorithm to effectively train resizable networks, which make sure thatall thesub-networks obtain good accuracy. Such trainingalgorithm compromises the accuracy between the target network and the others. In Section 3.3, we further show that signiﬁcant performance enhancement can be achieved by ﬁne-tuning the target sub-network.
Revised: In our work, we observed that the training accuracy on ImageNet classiﬁcation for ResizableNets with smaller scaling conﬁgurations (smaller FLOPs) sometimes perform better than those with larger scaling conﬁgurations . We conjecture that categories with complex image patterns beneﬁt from predicting with larger scaling conﬁgurations, and categories with simpler image patterns beneﬁt from predicting with smaller scaling conﬁgurations.

Original: This performance gain is not only coming from more training on the target network but to alleviate the co-adaptation issue, which we will discuss below.
Revised: (Wang et al. ; Li et al. Based on this observation, we present dynamic inference for the ResizableNets, named as Adaptive

Original: Co-adaptation.
Revised: Resizable Networks (Resizable-Adapt).

Original: Co-adaptation is a common phenomenon when network operations in a single model are shared (Bender et al., 2018).
Revised: Resizable-Adapt aims to ﬁnd the optimal scaling conﬁguration for each sample based on their inherent characteristics (e.g., object size and image complexity) learned by the network.

Original: Because of the co-adaptation, when a spatial sub-network is trained, the ability of other sub-networks will be impaired if they shared the same convolutional layers with a particular scale factor. When we adopt resizable networks as a data augmentation technique, we aim to achieve the best performance on the target network.
Revised: For the Resizable-Adapt, we ﬁrst randomly sample 50k images from the training data.

Original: If we train a resizable network naively, the shared convolutional layers can co-adapt.
Revised: We test all scaling conﬁgurations for each sample on an already trained ResizableNet.

Original: Thus we introduce a simple technique to alleviate the issue of co-adaptation, named as scheduled switching.
Revised: We take the scale conﬁguration with the highest accuracy for the sample as the label. We then introduce a lightweight

Original: [Table]
Revised: [Table]

Original: Scheduled Switching. Instead oftraining the  resizable networks follow the Section 3.2, we decompose the entire training procedure into two-stage.
Revised: scaling conﬁguration classiﬁer that is attached at the last stage of the original resizable network.

Original: At the ﬁrst stage, we follow the same training algorithm as we described in the Section 3.2. At the second stage, we stop the current training algorithm at some scheduled point, and switch to the normal optimization pipeline, where only the target network is sampled and optimized. This simple technique leverage the advantages of implicit multi-scale learning from the resizable networks, as well as resolve the layer co-adaption problem. 3. A DAPTIVE R ESIZABLE N EURAL N ETWORK
Revised: It is composed of three convolutional layers along with a pooling layer and followed by the softmax.

Original: There are two drawbacks in the original resizable networks.
Revised: It learns the image speciﬁc scaling conﬁguration.

Original: First, it is inefﬁcient to process all images to one sub-network, as the algorithm spends equal energy at every sample.
Revised: To be speciﬁc, this classiﬁer takes an image as input and predicts an optimal scaling conﬁguration for this input.

Original: Secondly, the  scaling policy in the original resizable network is implicit, which may be sub-optimal. To tackle these issues, we introduce a ResizeLearner, such that resizable networks can make data-dependent inference by assigning samples to different spatial sub-networks.
Revised: Then we switch the ResizableNet to this scaling conﬁguration and predict its labels using the input.

Original: This explicit scaling policy, while less computational power is used, enhance the ability of thenetworks. The pipeline of adaptive resizable network is simple yet effective: After a resizable network is trained, we initialize the ResizeLearner. It is composed of three convolutional layers along with a pooling layer and followed by the softmax, attached at the last stage of the original network. Note that we freeze the parameters of the original network, such that only the ResizeLearner update the weights.
Revised: Our extensive experiments verify our conjecture that the network can indeed achieve better performance if the sample is predicted by a suitable scaling conﬁguration.

Original: For the training, we randomly sample 50k images from the training data, validate on all spatial sub-networks for each sample, and take one with the highest accuracy as the target. During inference, we assign each sample with the optimal spatial sub-networks generated by ResizeLearner.
Revised: Additionally, the previous resizable network process all images to one scaling conﬁguration, as the network spends equal energy (e.g., FLOPs) at every sample. The adaptive resizable network can spend equal or less amount of computational cost during testing since some samples are assigned to smaller-scaling conﬁgurations. Thus, Resizable-Adapt is more efﬁcient in inference.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we validate the resizable networks on ImageNet classiﬁcation from several applications.
Revised: In this section, we validate the ResizableNets on ImageNet classiﬁcation from several perspectives.

Original: For consistency, we use RS- to indicate resizable network, ft to indicate resizable-nas networks by ﬁne-tuning, RS-NAS to denote the ﬁne-tuned target spatial sub-network, RS-Aug to represent resizable augmentation.

Original: RS-Adapt to denote adaptive resizable networks.

Original: Implementation details can be found in the Appendix A.1.
Revised: Implementation details can be found in the Appendix A.1.

Original: Section 4.1 give experiments for resizable networks compared with individual networks.
Revised: Section 5.1 give experiments for resizable networks compared with individual networks.

Original: Section 4.2-4.4 gives the result of resizable-nas, resizable- [Figure] data-dependent inference
Revised: Section 5.2, Section 5.3, and Section 5.4 gives the result of Resizable-NAS, Resizable-Aug and Resizable-Adapt network respectively.

Original: [Table]
Revised: Notation.

Original: [Table]
Revised: For consistency, we use RS- to indicate ResizableNets from Section 3, RS-NAS indicates Resizable-NAS from Section 4.2, RS-Aug to represent resizable augmentation from Section 4.1. RS-Adapt to denote adaptive resizable networks from Section 4.3.

Original: 4. C OMPARISON BETWEEN R ESIZABLE N ETWORKS WITH S TAND -A LONE N ETWORK .
Revised: 5. C OMPARISON BETWEEN R ESIZABLE N ETWORKS WITH S TAND -A LONE N ETWORK .

Original: We show in Table 1 the top-1 classiﬁcation accuracy for both individually trained networks and resizable networks give the same spatial conﬁgurations.
Revised: We show in Table 1 the top-1 classiﬁcation accuracy for both individually trained networks and resizable networks give the same scaling conﬁgurations .

Original: Due to the space limit, we only take several spatial conﬁgurations for comparison.
Revised: Due to the space limit, we only take several scaling conﬁgurations for comparison.

Original: We present top-1 test accuarcy and FLOPs [Table] which is the primary concerns in inference time [Table]
Revised: We present top-1 test accuracy and FLOPs, which is the primary concerns in inference time, of all networks for reference.

Original: of all networks for reference. Compared to independently trained networks, the spatial sub-networks in resizable networks achieved signiﬁcantly better topaccuracy [Table] across various network architectures and computational complexity.
Revised: Compared to independently trained networks, the spatial sub-networks in resizable networks achieved signiﬁcantly better topaccuracy, across various network architectures and computational complexity.

Original: This result conﬁrm our hypothesis that the implicit multi-scale training improve the model capacity
Revised: This result conﬁrm our hypothesis that the implicit multi-scale training improve the model capacity.

Original: [Table]
Revised: [Table]

Original: 4. T HE RESULT OF R ESIZABLE -NAS N ETWORKS .

Original: Comparison with Naive Resizable Networks.
Revised: Comparison with Naive Resizable Networks.

Original: The Figure 2(a) and 2(b) demonstrate the results of resizable networks and resizable-nas network with ResNet-50 and ShufﬂeNetV2.
Revised: The Figure 5(a) and 5(b) demonstrate the results of ResizableNet and Resizable-NAS network with ResNet-50 and ShufﬂeNetV2.

Original: The only difference between these two networks is that the resizable-nas network can 15 epochs  ﬁne-tune on target sub-network.
Revised: The only difference between these two networks is that the Resizable-NAS network take 15 epochs to ﬁne-tune on target scaling conﬁguration.

Original: It shows the notable performance improvement on individual sub-network when it is selected and ﬁne-tuned compare with the baseline method.
Revised: The Resizable-NAS shows the notable performance improvement over individual network and ResizableNet with the same scaling conﬁguration.

Original: Comparison with State-of-the-art Model Compression Methods.
Revised: Comparison with State-of-the-art Model Compression Methods.

Original: We compare with the following state-of-the-art model compression methods: NISP (Yu et al.
Revised: We compare with the following state-of-the-art model compression methods: NISP (Yu et al.

Original: (2018b))
Revised: (2018b))

Original: , ThiNet (Luo et al.
Revised: , ThiNet (Luo et al.

Original: ChannelPruning (He et al.
Revised: ChannelPruning (He et al.

Original: , Sparse Structure Selection ResNet50 (Huang & Wang (2018)) and SlimmableNets (Yu et al.
Revised: , Sparse Structure Selection ResNet50 (Huang & Wang (2018)) and SlimmableNets (Yu et al.

Original: (2018a))
Revised: (2018a))

Original: , AMC (He et al., 2018) and Netadapt (Yang et al., 2018).
Revised: , AMC (He et al., 2018) and Netadapt (Yang et al., 2018).

Original: We  present the ResNet-50 based results at Figure 2(c), and the detailed results of Resizable-NAS
Revised: We [Table] present the ResNet-50 based results at Figure5(c) [Table]

Original:  MobileNetV1 as well as Resizable-NAS ResNet-50 can be found inthe Table 4. Resizable-NAS ResNet-50 demonstrate strong performance, especially when the compression ratio is large than 50%.
Revised: and the detailed results of Resizable-NAS MobileNetV1 as well as Resizable-NAS ResNet-50 can be found in Table B. Resizable-NAS ResNetdemonstrate strong performance, especially when the compression ratio is large than 50%.

Original: It outperforms listed state-of-the-arts compression methods by a large margin over diverse computational complexity constraints.
Revised: It outperforms listed state-of-the-arts compression methods by a large margin over diverse computational complexity constraints.

Original: We emphasize that our Resizable-NAS network only trained 
Revised: We emphasize that our Resizable-NAS network only trained once

Original: [Table]
Revised: [Table]

Original: once, and ﬁne-tune 15 epochs on target sub-networks [Table] where the other compression methods ﬁnd the optimal structure and retrain accordingly.
Revised: and ﬁne-tune 15 epochs on target sub-networks, where the other compression methods ﬁnd the optimal structure and retrain accordingly.

Original: Comparing to the other methods [Table] our proposed model is more efﬁcient while performing better.
Revised: Comparing to the other methods, our proposed model is more efﬁcient while performing better.

Original: Notice even the version without the ﬁne-tuning process can achieve similar (even better) performance than other model compression methods.
Revised: Notice even the version without the ﬁne-tuning process can achieve similar (even better) performance than other model compression methods.

Original: 4. C OMPARISON WITH S TATE - OF - THE - ART A UGMENTATIONMETHODS
Revised: 5. R ESIZABLE A UGMENTATION

Original: On ResNet-50.
Revised: On ResNet-50.

Original: We compares the Resizable-Aug ResNet-50 with various state-of-the-art data augmentation methods (i [Table] e., Cutout (DeVries & Taylor (2017)) [Table]
Revised: We compares the Resizable-Aug ResNet-50 with various state-of-the-art data augmentation methods (i.e., Cutout (DeVries & Taylor (2017)), Mixup (Zhang et al.

Original: Mixup (Zhang et al.

Original: and AutoAugment (Cubuk et al.
Revised: and AutoAugment (Cubuk et al.

Original: The results are summarized in Table 2.
Revised: The results are summarized in Table 2.

Original: It shows that ResizableAug improve the ResNet-50 baseline by 2.4%, outperform AutoAugment by 1.2% using the same the baseline.
Revised: It shows that ResizableAug improve the ResNet-50 baseline by 2.4%, outperform AutoAugment by 1.2% using the same the baseline.

Revised: It is worthy to note that our Resizable-Aug takes the same number of epochs as the baseline to train the network.

Revised: In our experiment, we use scaling factor list of (0 . 45 , 0 . 65 , 0 . 85 , 1 . 0) .

Original: On Efﬁcient Models.
Revised: On Efﬁcient Models.

Original: We present the resizable augmentation results based on network that are search by AutoML.
Revised: We present the resizable augmentation results based on network that are search by AutoML.

Original: The details of the training strategy and network architecture are presented in the Appendix A.3.
Revised: The details of the training strategy and network architecture are presented in the Appendix B.3.

Original: Table 3 shows the results.
Revised: Table 3 shows the results.

Original: By adopting the resizable augmentation strategy, we improve the already high accuracy network from 77.4% to 77.8%, achieve a new state-of-the-art result.
Revised: By adopting the resizable augmentation strategy, we improve the already high accuracy network from 77.4% to 77.8%, achieve a new state-of-the-art result.

Original: The new models is 1.5% better than EfﬁcientNet-B0 (Tan & Le (2019)), which is trained
Revised: The new models is 1.5% better than EfﬁcientNet-B0 (Tan & Le (2019)), which is trained

Original: AutoAugment (Cubuk et al., 2018) with 25M less FLOPs.
Revised: AutoAugment (Cubuk et al., 2018) with 25M less FLOPs.

Revised: In our experiment, we use scaling factor list of (0 . 45 , 0 . 65 , 0 . 85 , 1 . 0) .

Original: 4. R ESIZABLE-A DAPT N ETWORKS R ESULTS
Revised: 5. A DAPTIVE R ESIZABLE N EURAL N ETWORK

Original: Figure 3(d) shows the results of Resizable-Adapt ResNet-50.
Revised: Figure 5(d) shows the results of Resizable-Adapt ResNet-50.

Original: The dynamic inference is performed on multiple computation complexity threshold.
Revised: The dynamic inference is performed on multiple computation complexity threshold.

Original: During inference, we ensure that only sub-networks that are smaller than the target network is used for inference.
Revised: During inference, we ensure that only sub-networks that are smaller than the target network is used for inference.

Original: The performance gain is range from 0% to 0.5%.
Revised: The performance gain is range from 0% to 0.5%.

Original: It can be observed that the adaptive resizable ResNet-50 consistently obtain better performance than the baseline.
Revised: It can be observed that the adaptive resizable ResNet-50 consistently obtain better performance than the baseline.

Original: This is a surprising result since most dynamic network struggle at the trade-off between efﬁciency and accuracy.
Revised: This is a surprising result since most dynamic network struggle at the trade-off between efﬁciency and accuracy.

Original: Less computational complexity in dynamic network always lowers the accuracy compared with the baseline.
Revised: Less computational complexity in dynamic network always lowers the accuracy compared with the baseline.

Original: Moreover, when the target network is large, sometimes the samples are still better predicted by the small sub-network.
Revised: Moreover, when the target network is large, sometimes the samples are still better predicted by the small sub-network.

Original: This gives the evidence that the explicit way to deal with the input scale is indeed beneﬁcial.
Revised: This gives the evidence that the explicit way to deal with the input scale is indeed beneﬁcial.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced a type of convolutional neural network, named as Resizable Neural Networks.
Revised: We introduced a type of convolutional neural network, named as Resizable Neural Networks.

Original: The resizable networks consist of inﬁnite single scale networks, where each single scale networks obtain a unique spatial conﬁguration.
Revised: The resizable networks can do arbitrary resize operation on feature map at stage level.

Original: Based on this, we present a training algorithm to  train this network, such that all spatial sub-networks in the resizable networks attain good accuracy.
Revised: We further provide essential tools to build and train our proposed network.

Original: Moreover, we apply three variants of resizable networks.
Revised: Moreover, we give three applications of ResizableNet.

Original: Speciﬁcally, we present Resizable-NAS, Resizable-Aug, and Resizable-Adapt.
Revised: Speciﬁcally, we present Resizable-NAS, Resizable-Aug, and Resizable-Adapt.

Original: The resizable-NAS network permits run-time model compression by searching for the best-ﬁt sub-network under efﬁciency constraints.
Revised: Resizable-Aug is a data augmentation techniques which do feature map transform by scaling and achieve state-of-the-art results on ImageNet classiﬁcation. Resizable-NAS can deploy on many hardwares with various efﬁciency constraints by on trained network.

Original: Moreover, we present adaptive resizable networks, which perform data-dependent dynamic inference.
Revised: Resizable-Adapt performs data-dependent dynamic inference.

Original: Overall, we believe that the resizable network’s approach provides an interesting and practical new perspective on designing a convolution neural network on the spatial dimension.
Revised: Overall, we believe that the resizable network’s approach provides an interesting and practical new perspective on designing a convolution neural network on the spatial dimension.

Original: A A PPENDIX A. I MPLEMENTATION D ETAILS .
Revised: A C HOOSING T HE P REDEFINED SCALING FACTOR L IST .

Revised: We will discuss the choice of predeﬁned scaling factor list to ResizableNet and ResizableAug separately.

Revised: A. R ESIZABLE N ET

Revised: We ﬁrst report the scaling conﬁguration list we used in the previous experiments.

Revised: For all Resizable

Revised: ResNet-50 and Resizable ShufﬂeNetV2, we use scaling conﬁguration S = (0 . 45 , 0 . 65 , 0 . 85 , 1 . 0) .

Revised: For all Resizable MobileNetV1 we use S =

Revised: We use only three scaling factors because the number of stage of MobileNetV1 is one larger than ResNet and ShufﬂeNetV2.

Revised: Choice of Scaling Conﬁguration S .

Revised: The choice of scaling conﬁguration depends on the range of efﬁciency constraints that need to be covered.

Revised: There is no speciﬁc guideline for choosing scaling conﬁguration for ResizableNet.

Revised: Number of scaling factors n .

Revised: With more scaling factors, the number of scaling conﬁguration are increased exponentially, thus more ﬂexible during testing.

Revised: We investigate how the number of scaling factors impact accuracy on Resizable-NAS.

Revised: In Table 4, we train ResNet-50 with two different scaling factor list.

Revised: We compare a four scaling factors list with a six scaling factors list.

Revised: We display some results.

Revised: The results of two different number of scaling factors are similar, showing the scalability of our proposed method.

Revised: [Table]

Revised: A. R ESIZABLE A UG

Revised: For all ResizableAug experiment, we use scaling conﬁgurations of S = { 0 . 45 , 0 . 65 , 0 . 85 , 1 . 0 } .

Revised: Choices of scaling conﬁguration S .

Revised: We study how to choose the scaling factors set S such that the best performance via ResizableAug can be achieved.

Revised: The performance gain in ResizableAug is due to the implicit multi-scale training.

Revised: Thus the predeﬁned scaling factor list can be viewed as the choice for multi-scale.

Revised: We empirically ﬁnd that, as long as we use the diversiﬁed scaling factor, the performance of ResizableAug is robust to the speciﬁc choice of scaling factors.

Revised: The left part of

Revised: Table 6 shows the results using different combinations of scaling factors.

Revised: The ﬁrst three experiments,

Revised: S = (0 . 45 , 0 . 65 , 0 . 85 , 1 . 0) , S = (0 . 4 , 0 . 6 , 0 . 8 , 1 . 0) or S = (0 . 25 , 0 . 5 , 0 .

Revised: 75 , 1 . 0) achieve the same or similar performance, which indicate the speciﬁc choice of scaling factor is a critical to increase the accuracy of the model.

Revised: In the fourth set, we use S =

Revised: 9 , 1 . 0) , where every scaling factor is close to each other, and the accuracy falls as we expected.

Revised: By default, in all our experiment we use S =

Revised: Number of scaling factors n .

Revised: We study how the number of scaling factor candidates affect the performance of ResizableAug.

Revised: We train three ResizableAug ResNet-50 with number of scaling factors equal to 3, 4 or 5, following the scaling factor diversity rule.

Revised: Right part of Table 6 shows that the model trained with n = 4 or n = 5 achieve the same performance, while n = 3 perform worse than n =

Revised: [Table]

Revised: B I MPLEMENTATION D ETAILS B. N ETWORK I MPLEMENTATION D ETAILS .

Original: Resize operation is place at the ﬁrst non-stride block at each stage in network.
Revised: Resize operation is place at the ﬁrst non-stride block at each stage in network.

Original: We apply bilinear interpolation to resize the feature map to target size.
Revised: We apply bilinear interpolation to resize the feature map to target size.

Original: Resize factor are uniformly taken from a predeﬁned list of possible resize choice, except when it contradicted to fair sampling rule.
Revised: Resize factor are uniformly taken from a predeﬁned list of possible resize choice, except when it contradicted to fair sampling rule.

Original: Our implementation is based on the Pytorch (Paszke et al.
Revised: Our implementation is based on the Pytorch (Paszke et al.

Original: The input resolution for all train and test images for ImageNet classiﬁcation are 224 x 224.
Revised: The input resolution for all train and test images for ImageNet classiﬁcation are 224 x 224.

Original: Other than ResNet-50, which we report baseline higher than the original paper, we have trained
Revised: Other than ResNet-50, which we report baseline higher than the original paper, we have trained

Original: Mobilenet v1, Mobilenet v2 and Shufﬂenet v2 baseline to match the performance as reported in original paper.
Revised: Mobilenet v1, Mobilenet v2 and Shufﬂenet v2 baseline to match the performance as reported in original paper.

Original: And based on the re-implemented settings, we perform both baseline and resizable networks experiments.
Revised: And based on the re-implemented settings, we perform both baseline and resizable networks experiments.

Original: For ResNet-50, we train for 100 epochs using weight decay of 1e-4.
Revised: For ResNet-50, we train for 100 epochs using weight decay of 1e-4.

Original: We use batch size of 512 and do learning rate decay by 10 at epochs 30, 60, 80, 90.
Revised: We use batch size of 512 and do learning rate decay by 10 at epochs 30, 60, 80, 90.

Original: The other resizable network train forepochs and double the epochs of learning rate decay.
Revised: The other resizable network train forepochs and double the epochs of learning rate decay.

Revised: For Mobilenet v1, Mobilenet v2 and Shufﬂenet v2, we use batch size of 1024 and weight decay of 4e-5.

Revised: We train both models for 240 epochs using linear learning rate decay where base learning rate is set to 0.5.

Original: SF 3 x
Revised: All models are train across synchronous SGD across 8 GPUs. The resizable networks train for 480 epochs The resizable augmentation network always use the same training setting as the baseline method. Extending the training epochs is unnecessary.

Revised: B. D ETAIL R ESULTS OF R ESIZABLE -NAS N ETWORKS

Revised: The detail results of Resizable-NAS ResNet-50 and Resizable-NAS MobileNetV1 can be found at

Revised: Table B.

Revised: B. D ETAILS OF E FFICIENT R ESIZABLE N ET

Revised: Training Details.

Revised: Efﬁcient ResizableNet is trained using resizable-aug technique as we discuss in the Section 4.1.

Revised: It is trained for 480 epochs with batch size 1024 on 8 GPUs.

Revised: The network parameters are optimized using an SGD optimizer with an initial learning rate of 0.5 (decayed linearly after each iteration), a momentum of 0.9 and a weight decay of 3 × 10 − 5 .

Revised: Additional enhancements including label smoothing (Szegedy et al.

Revised: and Dropout with 0.5 on last FC layer.

Revised: We follow the common practice to use inception augmentation.

Revised: This architecture is searched with Anonymous (2020) using search space proposed by Guo et al.

Revised: The Xception is proposed by Chollet (2017).

Revised: Detailed Network Architecture.

Revised: The network architecture along with feature map resolution and channels number are shown in Figure B.3.

Revised: [Table]

Revised: C T RAINING E FFICIENCY A NALYSIS .

Original: SF 3 x
Revised: We analysis the training time of ResizableNets. We evaluate the training times of ResizableNets with various number of scaling factors (1, 2, 4, and 6) are evaluated, as well as the total training time if the same amount of sub-networks in the resizable net are trained individually. The results are shown in Table 8. On the one hand, the total training time of resizable nets is remarkably less than the train the same amount of individual networks separately due to the weight sharing mechanism.

Revised: On the other hand, because we sample multiple scaling conﬁgurations in one step, we eliminate the need to provide mini-batch data and update gradients repeatedly.

Revised: As a result, the training time by fair sampling is less than simply extending the total training epochs by the number of predeﬁned scaling factors.

Revised: Particularly, as the number of scaling factors increases, the beneﬁt of fair sampling in terms of training time is more signiﬁcant.

Revised: For example, compare to random sampling, when len ( S ) = 4 , fair sampling is 10% faster, while len ( S ) = 8 , fair sampling is 52% faster.

Original: SF 3 x 
Revised: SF 3 x SF 3 x SF 3 x

Original: X c e p t i o n
Revised: X c e p t i o n

Original: SF 3 x
Revised: SF 3 x

Original: SF 5 x SF 5 x
Revised: SF 5 x SF 5 x

Original: SF 5 x
Revised: SF 5 x

Original: SF 3 x 3 / S E
Revised: SF 3 x 3 / S E

Original: SF 5 x 5 / S E
Revised: SF 5 x 5 / S E

Original: X c e p t i o n / S E
Revised: X c e p t i o n / S E

Original: SF 3 x 3 / S E
Revised: SF 3 x 3 / S E

Original: SF 5 x 5 / S E
Revised: SF 5 x 5 / S E

Original: SF 7 x 7 / S E SF 7 x 7 / S E
Revised: SF 7 x 7 / S E SF 7 x 7 / S E

Original: SF 5 x 5 / S E SF 5 x 5 / S E
Revised: SF 5 x 5 / S E SF 5 x 5 / S E

Original: SF 7 x 7 / S E
Revised: SF 7 x 7 / S E

Original: X c e p t i o n / S E
Revised: X c e p t i o n / S E

Original: SF 7 x 7 / S E 112 x 112 x 56 x 56 x56 x 56 x56 x
Revised: SF 7 x 7 / S E 112 x 112 x 56 x 56 x56 x 56 x56 x

Original: 56 x28 x 28 x 28 x 28 x 28 x 28 x 28 x 28 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 7 x 7 x7 x 7 x7 x 7 x7 x 7 x
Revised: 56 x28 x 28 x 28 x 28 x 28 x 28 x 28 x 28 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 14 x 7 x 7 x7 x 7 x7 x 7 x7 x 7 x

Original: C o n v 1 x
Revised: C o n v 1 x

Original: C o n v 3 x 3 224 x 224 x
Revised: C o n v 3 x 3 224 x 224 x

Original: P oo li n g F C 7 x 7 x7 x 7 x
Revised: P oo li n g F C 7 x 7 x7 x 7 x

Original: For Mobilenet v1, Mobilenet v2 and Shufﬂenet v2, we use batch size of 1024 and weight decay of 4e-5. We train both models for 240 epochs using linear learning rate decay where base learning rate is set to 0.5. All models are train across synchronous SGD across 8 GPUs.
Revised: Training time (gain.)

Original: The resizable networks train for 480 epochs
Revised: Length # of Networks

Original: The resizable augmentation network always use the same training setting as the baseline method. Extending the training epochs is unnecessary. A. D ETAIL R ESULTS OF R ESIZABLE -NAS N ETWORKS The detail results of Resizable-NAS ResNet-50 and Resizable-NAS MobileNetV1 can be found at
Revised: Individual

Original: Table 4.
Revised: Random

Original: Training Details.{+Fair  8 Gds 8 Gds (+0%)  4,896 Gds* 32 Gds* 29 Gds (+10%)  122,
+}
Revised: {+Fair  8 Gds 8 Gds (+0%)  4,896 Gds* 32 Gds* 29 Gds (+10%)  122,
+}

Original: ResizableNet is trained using resizable-aug technique as we discuss in the Section 3.4. It is trained for 480 epochs with batch size 1024 on 8 GPUs. The network parameters are optimized using an SGD optimizer with an initial learning rate of 0.5 (decayed linearly after each iteration), a momentum of 0.9 and a weight decay of 3 × 10 − 5 . Additional enhancements including label smoothing (Szegedy et al.
Revised: Gds* 48 Gds* 35 Gds (+26%) 390, 3,125,000 Gds* 64 Gds* 42 Gds

Original: and Dropout with 0.5 on last FC layer. We follow the common practice to use inception augmentation. Detailed Network Architecture. The network architecture along with feature map resolution and channels number are shown in Figure 4.
Revised: [Table]

Original: B A BLATION S TUDY B. I S NEURAL NETWORK NATURALLY RESIZABLE ?
Revised: D A BLATION S TUDY D. I S NEURAL NETWORK NATURALLY RESIZABLE ?

Original: It is intuitive to ask the question: is neural network naturally resizable?
Revised: It is intuitive to ask the question: is neural network naturally resizable?

Original: The answer is no.
Revised: The answer is no.

Original: We present the result of directly inference on a neural network with different spatial conﬁgurations and compare the result with resizable neural network.
Revised: We present the result of directly inference on a neural network with different scaling conﬁgurations and compare the result with resizable neural network.

Original: The naive network is trained with scale-aware batch normalization.
Revised: The naive network is trained with scale-aware batch normalization.

Original: Figure 5 shows that naively trained neural network can not adjust feature map size, which result in a extremely low test accuracy.
Revised: Figure D.1 shows that naively trained neural network can not adjust feature map size, which result in a extremely low test accuracy

Original: B. S CALE -A WARE BN A NALYSIS . Effect on Resizable Augmentation.
Revised: [Figure]

Original: Spatial BN calibration is especially important for resizable augmentation.
Revised: Scale aware batch normalization is especially important for resizable augmentation.

Original: The following table compare two network adopt resizable augmentation with and without BN calibration.
Revised: The following table compares two networks to adopt resizable augmentation with and without BN calibration.

Original: We can observed that without BN calibration, the resizable augmentation is only slightly better than the baseline model.
Revised: We can observe that without BN calibration, the resizable augmentation is only slightly better than the baseline model.

Original: The improvement is negligible.
Revised: The improvement is negligible.

Original: On the other hand, when BN calibrated for spatial conﬁgurations, is boost the performance compare with the baseline.
Revised: On the other hand, when BN calibrated for scaling conﬁgurations, is boost the performance compare with the baseline.

Original: The accuracy is increased by 1.1% on ShufﬂeNetV2 and 2.4% on ResNet-50.
Revised: The accuracy is increased by 1.1% on ShufﬂeNetV2 and 2.4% on ResNet-50.

Original: [Figure]
Revised: [Table]

Revised: D. F AIR S AMPLING A NALYSIS .

Revised: We compare the results of random sampling and fair sampling in this section.

Revised: We use the scaling factor list L = (0 . 5 , 0 .

Revised: on MobileNetV1.

Revised: The random sampling is trained with three times more epochs than fair sampling.

Revised: We showed some results in Table 11.

Revised: The results show that the performance of different scaling conﬁguration is not consistent.

Revised: By consistency, we mean that some scaling conﬁguration achieves better accuracy, while some scaling conﬁguration achieves lower accuracy

Original: [Table]
Revised: [Table]

Revised: using fair sampling we can get more stable results.

Revised: [Table]