Original: L OCALLY
Revised: L OCALLY

Original: F REE
Revised: F REE

Original: W EIGHT
Revised: W EIGHT

Original: SHARING
Revised: SHARING

Original: FOR
Revised: FOR

Original: N ETWORK
Revised: N ETWORK

Original: W IDTH S EARCH
Revised: W IDTH S EARCH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Searching for network width is an effective way to slim deep neural networks with hardware budgets.
Revised: Searching for network width is an effective way to slim deep neural networks with hardware budgets.

Original: With this aim, a one-shot supernet is usually leveraged as a performance evaluator to rank the performance w.r.t.
Revised: With this aim, a one-shot supernet is usually leveraged as a performance evaluator to rank the performance w.r.t.

Original: different width.
Revised: different width.

Original: Nevertheless, current methods mainly follow a manually ﬁxed weight sharing pattern, which is limited to distinguish the performance gap of different width.
Revised: Nevertheless, current methods mainly follow a manually ﬁxed weight sharing pattern, which is limited to distinguish the performance gap of different width.

Original: In this paper, to better evaluate each width, we propose a locally free weight sharing strategy (CafeNet) accordingly.
Revised: In this paper, to better evaluate each width, we propose a locally free weight sharing strategy (CafeNet) accordingly.

Original: In CafeNet, weights are more freely shared, and each width is jointly indicated by its base channels and free channels, where free channels are supposed to locate freely in a local zone to better represent each width.
Revised: In CafeNet, weights are more freely shared, and each width is jointly indicated by its base channels and free channels, where free channels are supposed to locate freely in a local zone to better represent each width.

Original: Besides, we propose to further reduce the search space by leveraging our introduced FLOPs-sensitive bins.
Revised: Besides, we propose to further reduce the search space by leveraging our introduced FLOPs-sensitive bins.

Original: As a result, our CafeNet can be trained stochastically and get optimized within a min-min strategy.
Revised: As a result, our CafeNet can be trained stochastically and get optimized within a min-min strategy.

Original: Extensive experiments on ImageNet, CIFAR-10, CelebA and
Revised: Extensive experiments on ImageNet, CIFAR-10, CelebA and

Original: MS COCO dataset have veriﬁed our superiority comparing to other state-of-the-art baselines.
Revised: MS COCO dataset have veriﬁed our superiority comparing to other state-of-the-art baselines.

Original: For example, our method can further boost the benchmark NAS network
Revised: For example, our method can further boost the benchmark NAS network

Original: EfﬁcientNet-B0 by 0.41% via searching its width more delicately.
Revised: EfﬁcientNet-B0 by 0.41% via searching its width more delicately.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks are easily constructed by stacking multiple layers of non-linearities on top of each other.
Revised: Deep neural networks are easily constructed by stacking multiple layers of non-linearities on top of each other.

Original: But it is a much more delicate job than it seems.
Revised: But it is a much more delicate job than it seems.

Original: Before going into the deep, we have to determine the width of each layer 

Revised: Before going into the deep, we have to determine the width of each layer

Original: (Chen et al., 2020) to be stacked, especially given the consideration of the computational budgets (He et al., 2016; Han et al., 2015; Ioffe & Szegedy, 2015;
Revised: (Chen et al., 2020) to be stacked, especially given the consideration of the computational budgets (He et al., 2016; Han et al., 2015; Ioffe & Szegedy, 2015;

Original: Huang et al., 2017)
Revised: Huang et al., 2017)

Original: ( e.g. , FLOPs, and latency) and the resulting network performance.
Revised: ( e.g. , FLOPs, and latency) and the resulting network performance.

Original: The width of a pretrained over-parameterized neural network can be further trimmed by classical channel pruning techniques (Liu et al., 2019b;
Revised: The width of a pretrained over-parameterized neural network can be further trimmed by classical channel pruning techniques (Liu et al., 2019b;

Original: He et al., 2017; Lin et al., 2018; Huang & Wang, 2018; Liu et al., 2017) to eliminate the redundancy.
Revised: He et al., 2017; Lin et al., 2018; Huang & Wang, 2018; Liu et al., 2017) to eliminate the redundancy.

Original: Inspired by the spark of neural architecture search (NAS) (Liu et al.
Revised: Inspired by the spark of neural architecture search (NAS) (Liu et al.

Original: ; Cai et al.; Guo et al., 2019), recent methods attempt to directly search for the optimal network width for a given network, and have achieved remarkable performance in various FLOPs levels or acceleration rates, such as AutoSlim (Yu & Huang, 2019), MetaPruning (Liu et al., 2019a) and TAS (Dong & Yang, 2019).
Revised: ; Cai et al.; Guo et al., 2019; Cai et al., 2019; Hu et al., 2020), recent methods attempt to directly search for the optimal network width for a given network, and have achieved remarkable performance in various FLOPs levels or acceleration rates, such as AutoSlim (Yu & Huang, 2019), MetaPruning (Liu et al., 2019a) and TAS (Dong & Yang, 2019).

Original: To determine the optimal network width, we need to evaluate and compare the performance of different network width.
Revised: To determine the optimal network width, we need to evaluate and compare the performance of different network width.

Original: One of the most straightforward yet authentic approaches is to examine their training-from-scratch performance and traverse all possible settings of network width (not exceeding a maximum allowable width).
Revised: One of the most straightforward yet authentic approaches is to examine their training-from-scratch performance and traverse all possible settings of network width (not exceeding a maximum allowable width).

Original: However, exhaustive training is computationally unaffordable due to the huge search space.
Revised: However, exhaustive training is computationally unaffordable due to the huge search space.

Original: Taking MobileNetV2 as an example, 100 channels in each of the 25 layers result in 100 25 possible network architectures.
Revised: Taking MobileNetV2 as an example, 100 channels in each of the 25 layers result in 100 25 possible network architectures.

Original: For sake of searching efﬁciency, current methods
Revised: For sake of searching efﬁciency, current methods

Original: Chen et al.
Revised: Chen et al.

Original: ; Yu & Huang (2019); Dong & Yang (2019) usually follow a weight sharing strategy by leveraging a one-shot supernet, with various sub-networks sharing the same weights with the supernet.
Revised: ; Yu & Huang (2019); Dong & Yang (2019) usually follow a weight sharing strategy by leveraging a one-shot supernet, with various sub-networks sharing the same weights with the supernet.

Original: Then each network width can be efﬁciently evaluated by querying the performance of its corresponding sub-network in the supernet.
Revised: Then each network width can be efﬁciently evaluated by querying the performance of its corresponding sub-network in the supernet.

Original: In this way, how to specify the sub-network for each network width indicates the search space and matters for the performance evaluation.
Revised: In this way, how to specify the sub-network for each network width indicates the search space and matters for the performance evaluation.

Original: Without loss of generality, we count channels in a layer from the left.
Revised: Without loss of generality, we count channels in a layer from the left.

Original: A popular weight sharing strategy for the supernet then follows a ﬁxed weight sharing
Revised: A popular weight sharing strategy for the supernet then follows a ﬁxed weight sharing

Original: [Figure]
Revised: [Figure]

Original: pattern, which simply assigns the left c channels as the sub-network for the width c (see the in Fig.1(b)).
Revised: pattern, which simply assigns the left c channels as the sub-network for the width c (see the in Fig.1(b)).

Original: In this manner, the search space of a layer with n channels is O ( n ) .
Revised: In this manner, the search space of a layer with n channels is O ( n ) .

Original: However, this ﬁxed pattern imposes an inherent constraint on the search space.
Revised: However, this ﬁxed pattern imposes an inherent constraint on the search space.

Original: Any two sets of width fully share their weights, limiting the accurate evaluation ability of supernet and affecting the searching performance accordingly.
Revised: Any two sets of width fully share their weights, limiting the accurate evaluation ability of supernet and affecting the searching performance accordingly.

Original: Ideally, each channel from the supernet shall enjoy the freedom to be selected, i.e. , any channel has a probability to occur in the subnetwork, no matter it locates at the leftmost or the rightmost of a layer.
Revised: Ideally, each channel from the supernet shall enjoy the freedom to be selected, i.e. , any channel has a probability to occur in the subnetwork, no matter it locates at the leftmost or the rightmost of a layer.

Original: But the price of this full channel freedom is the dramatic increase of the search space size to O (2 n ) , which affects the feasibility of width search.
Revised: But the price of this full channel freedom is the dramatic increase of the search space size to O (2 n ) , which affects the feasibility of width search.

Original: Instead of this unaffordable full freedom, we prefer restrictions and regularization over the search space to keep the channel freedom within an acceptable range, which beneﬁts the efﬁciency of width search.
Revised: Instead of this unaffordable full freedom, we prefer restrictions and regularization over the search space to keep the channel freedom within an acceptable range, which beneﬁts the efﬁciency of width search.

Original: In this paper, we propose a locally free weight sharing strategy of the supernet ( i.e. , CafeNet) for network width search.
Revised: In this paper, we propose a locally free weight sharing strategy of the supernet ( i.e. , CafeNet) for network width search.

Original: For each width c , our goal is to specify a sub-network within supernet by assigning c channels, indicated by an index set I ( c ) .
Revised: For each width c , our goal is to specify a sub-network within supernet by assigning c channels, indicated by an index set I ( c ) .

Original: To balance the weight sharing freedom and the searching efﬁciency, we split the index set I ( c ) into two parts, i.e. , c b base channels
Revised: To balance the weight sharing freedom and the searching efﬁciency, we split the index set I ( c ) into two parts, i.e. , c b base channels

Original: I b ( c ) and c f free channels I f ( c ) , with I ( c ) =
Revised: I b ( c ) and c f free channels I f ( c ) , with I ( c ) =

Original: I b ( c ) ∪ I f ( c ) and c =
Revised: I b ( c ) ∪ I f ( c ) and c =

Original: c b + c f .
Revised: c b + c f .

Original: (see Fig.1(b)).
Revised: (see Fig.1(b)).

Original: In detail, base channels follow the ﬁxed weight sharing pattern, while free channels are encouraged to select freely from a given zone, which is the neighborhood of the c -th channel.
Revised: In detail, base channels follow the ﬁxed weight sharing pattern, while free channels are encouraged to select freely from a given zone, which is the neighborhood of the c -th channel.

Original: In this way, each width c can be better evaluated by multiple locally free sub-subnetworks.
Revised: In this way, each width c can be better evaluated by multiple locally free sub-subnetworks.

Original: The supernet CafeNet can be trained by optimizing the sub-network with the minimum loss for each width and search for the one with the maximum accuracy.
Revised: The supernet CafeNet can be trained by optimizing the sub-network with the minimum loss for each width and search for the one with the maximum accuracy.

Original: To further ease the burden of searching cost, we take a group of channels ( i.e. , bin ) as the minimal searching unit.
Revised: To further ease the burden of searching cost, we take a group of channels ( i.e. , bin ) as the minimal searching unit.

Original: The bin sizes are carefully set by encouraging FLOPs to be more evenly allocated over bins.
Revised: The bin sizes are carefully set by encouraging FLOPs to be more evenly allocated over bins.

Original: We conduct extensive experiments on the benchmark ImageNet (Deng et al., 2009) and CIFAR-10 (Krizhevsky et al., 2014) datasets to validate the superiority of our proposed CafeNet.
Revised: We conduct extensive experiments on the benchmark ImageNet (Deng et al., 2009) and CIFAR-10 (Krizhevsky et al., 2014) datasets to validate the superiority of our proposed CafeNet.

Original: The effectiveness of our CafeNet is also examined on the face attribute recognition task on CelebA (Liu et al., 2015) dataset and the transferability of our searched backbone for object detection task on MS COCO dataset (Everingham et al., 2010).
Revised: The effectiveness of our CafeNet is also examined on the face attribute recognition task on CelebA (Liu et al., 2015) dataset and the transferability of our searched backbone for object detection task on MS COCO dataset (Everingham et al., 2010).

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: With a given network structure, the aim of the network width search is to optimize its width for each layer under certain budgets.
Revised: With a given network structure, the aim of the network width search is to optimize its width for each layer under certain budgets.

Original: Suppose the target network has L layers, then the network width can be represented as a L -length tuple of layer-wise width c
Revised: Suppose the target network has L layers, then the network width can be represented as a L -length tuple of layer-wise width c

Original: =
Revised: =

Original: ( c 1 , c 2 , ..., c L ) in which c i represents the width at the i -th layer.
Revised: ( c 1 , c 2 , ..., c L ) in which c i represents the width at the i -th layer.

Original: Without loss of clarity, we simply focus on searching for the optimal layer width of c i .
Revised: Without loss of clarity, we simply focus on searching for the optimal layer width of c i .

Original: Denote the maximum allowable width at the i -th layer as n i .
Revised: Denote the maximum allowable width at the i -th layer as n i .

Original: Then the number of all sub-networks in the supernet amounts to (cid:81)
Revised: Then the number of all sub-networks in the supernet amounts to (cid:81)

Original: Li
Revised: Li

Original: =1 (2 n i − 1) , which is absolutely huge.
Revised: =1 (2 n i − 1) , which is absolutely huge.

Original: Current methods mainly use weight sharing strategy to boost the searching efﬁciency.
Revised: Current methods mainly use weight sharing strategy to boost the searching efﬁciency.

Original: In the typically ﬁxed weight sharing pattern, each layer width c i ∈ [1 : n i ] = { 1 , 2 , ..., n i } is indicated by one and only one sub-network (see Fig.1(b)), and all layers within the sub-network adopt the same weight sharing pattern, then the size of search space C is reduced to (cid:81)
Revised: In the typically ﬁxed weight sharing pattern, each layer width c i ∈ [1 : n i ] = { 1 , 2 , ..., n i } is indicated by one and only one sub-network (see Fig.1(b)), and all layers within the sub-network adopt the same weight sharing pattern, then the size of search space C is reduced to (cid:81)

Original: Li =1 n i .
Revised: Li =1 n i .

Original: After specifying each width with its corresponding sub-network, we can evaluate it by examining the performance of its sub-network in the supernet with shared weights.
Revised: After specifying each width with its corresponding sub-network, we can evaluate it by examining the performance of its sub-network in the supernet with shared weights.

Original: Then the whole searching consists of two steps, i.e. , supernet training, and searching with supernet.
Revised: Then the whole searching consists of two steps, i.e. , supernet training and searching with supernet.

Original: Usually, the original training dataset is split into two datasets, i.e. , training dataset D tr and validation dataset D val .
Revised: Usually, the original training dataset is split into two datasets, i.e. , training dataset D tr and validation dataset D val .

Original: The one-shot supernet N with weights W is trained by randomly sampling a width c and optimizing its corresponding
Revised: The one-shot supernet N with weights W is trained by randomly sampling a width c and optimizing its corresponding

Original: [Equation]
Revised: [Equation]

Original: network with weights
Revised: network with weights

Original: [Equation]
Revised: [Equation]

Original: where U ( C ) is a uniform distribution of network width in search space, and E [[ · ]] is the expectation of random variables.
Revised: where U ( C ) is a uniform distribution of network width in search space, and E [[ · ]] is the expectation of random variables.

Original: After the supernet is trained, the performance ( e.g. , classiﬁcation accuracy) of various network width is indicated by evaluating its sub-network on the trained supernet N ∗ .
Revised: After the supernet is trained, the performance ( e.g. , classiﬁcation accuracy) of various network width is indicated by evaluating its sub-network on the trained supernet N ∗ .

Original: And the optimal width corresponds to that of the highest performance on the validation dataset, e.g. ,
Revised: And the optimal width corresponds to that of the highest performance on the validation dataset, e.g. ,

Original: [Equation]
Revised: [Equation]

Original: where F b is a constraint on the FLOPs. Here we consider FLOPs rather than latency as the hardware constraint since we are not targeting any speciﬁc hardware device like EfﬁcientNet Tan & Le (2019) and other channel pruning baselines Zhuang et al.
Revised: where F b is a constraint on the FLOPs, here we consider FLOPs rather than latency as the hardware constraint since we are not targeting any speciﬁc hardware device like EfﬁcientNet Tan & Le (2019) and other channel pruning baselines Zhuang et al.

Original: ; Ye et al.
Revised: ; Ye et al.

Original: ; Dong & Yang (2019);
Revised: ; Dong & Yang (2019);

Original: He et al.
Revised: He et al.

Original: (2018b).
Revised: (2018b).

Original: The searching of Eq.(2) can be fulﬁlled efﬁciently by various algorithms, such as random or evolutionary search (Hancock, 1992;
Revised: The searching of Eq.(2) can be fulﬁlled efﬁciently by various algorithms, such as random or evolutionary search (Hancock, 1992;

Original: Whitley et al., 1990; Liu et al., 2019a).
Revised: Whitley et al., 1990; Liu et al., 2019a).

Original:  C AFE N ET 3. L OCALLY FREE WEIGHT SHARING
Revised:  C AFE N ET 3. L OCALLY FREE WEIGHT SHARING

Original: As illustrated before, a one-shot supernet N is usually trained as an evaluator to reﬂect the performance of different settings of network width.
Revised: As illustrated before, a one-shot supernet N is usually trained as an evaluator to reﬂect the performance of different settings of network width.

Original: Assume we count channels in a layer from the left.
Revised: Assume we count channels in a layer from the left.

Original: To locate the sub-network in the supernet for width c at a certain layer 2 , conventional ﬁxed weight sharing pattern assigns the leftmost c channels as the sub-network as Fig.1(b).
Revised: To locate the sub-network in the supernet for width c at a certain layer 2 , conventional ﬁxed weight sharing pattern assigns the leftmost c channels as the sub-network as Fig.1(b).

Original: Formally, for width c , its corresponding sub-network is speciﬁed by a ﬁxed channel assignment with an index set
Revised: Formally, for width c , its corresponding sub-network is speciﬁed by a ﬁxed channel assignment with an index set

Original: I ( c ) of channels, i.e. , ﬁxed
Revised: I ( c ) of channels, i.e. , ﬁxed

Original: [Equation]
Revised: [Equation]

Original: However, this hard assignment imposes an inherent constraint on the search space.
Revised: However, this hard assignment imposes an inherent constraint on the search space.

Original: Channel assignment of smaller width will always be a subset of that of larger width.
Revised: Channel assignment of smaller width will always be a subset of that of larger width.

Original: Actually, this ﬁxed pattern forces the weight sharing of two different width to the greatest extent.
Revised: Actually, this ﬁxed pattern forces the weight sharing of two different width to the greatest extent.

Original: To quantify this property, we deﬁne a simple weight sharing degree d of two any width c and ˜ c by examining the overlap of their channel assignment, i.e. ,
Revised: To quantify this property, we deﬁne a simple weight sharing degree d of two any width c and ˜ c by examining the overlap of their channel assignment, i.e. ,

Original: [Equation]
Revised: [Equation]

Original: Intuitively, if the degree d ( c, ˜ c ) is larger, then sub-networks of width c and ˜ c will share more weights, and this implies that the supernet will enforce width c and ˜ c to have more similar performance.
Revised: Intuitively, if the degree d ( c, ˜ c ) is larger, then sub-networks of width c and ˜ c will share more weights, and this implies that the supernet will enforce width c and ˜ c to have more similar performance.

Original: And for the ﬁxed pattern, the weight sharing degree reaches its maximum as Theorem 1 3 .
Revised: And for the ﬁxed pattern, the weight sharing degree reaches its maximum as Theorem 1 3 .

Original: Theorem 1.
Revised: Theorem 1.

Original: The weight sharing degree of ﬁxed pattern will always be 1 for any two width.
Revised: The weight sharing degree of ﬁxed pattern will always be 1 for any two width.

Original: However, ideally, each width should have full freedom in the supernet to select its channel assignment.
Revised: However, ideally, each width should have full freedom in the supernet to select its channel assignment.

Original: In this situation, sub-networks of two different layer width can be fully shared, partially shared, or even have no overlap, thus the weight sharing degree freely varies from 0 to 1.
Revised: In this situation, sub-networks of two different layer width can be fully shared, partially shared, or even have no overlap, thus the weight sharing degree freely varies from 0 to 1.

Original: And each width can be better evaluated by some sub-networks, instead of a manually assigned sub-network.
Revised: And each width can be better evaluated by some sub-networks, instead of a manually assigned sub-network.

Original: Nevertheless, this induces the search space to proliferate from the ﬁxed O ( n ) to O (2 n ) , which is computationally unaffordable for practical search.
Revised: Nevertheless, this induces the search space to proliferate from the ﬁxed O ( n ) to O (2 n ) , which is computationally unaffordable for practical search.

Original: Therefore, we bridge these two extreme situations, and propose a locally free weight sharing strategy.
Revised: Therefore, we bridge these two extreme situations, and propose a locally free weight sharing strategy.

Original: Basically, to specify a sub-network of width c from a supernet for evaluation, we need to assign c channels from all n channels per layer, with the channel assignment index set I ( c ) .
Revised: Basically, to specify a sub-network of width c from a supernet for evaluation, we need to assign c channels from all n channels per layer, with the channel assignment index set I ( c ) .

Original: We split the index set I ( c ) into two parts, i.e. , c b base channels
Revised: We split the index set I ( c ) into two parts, i.e. , c b base channels

Original: I b ( c ) and c f free channels I f ( c ) , with
Revised: I b ( c ) and c f free channels I f ( c ) , with

Original: I ( c ) =
Revised: I ( c ) =

Original: I b ( c ) ∪I f
Revised: I b ( c ) ∪I f

Original: ( c ) and c =
Revised: ( c ) and c =

Original: c b + c f .
Revised: c b + c f .

Original: In detail, for free channels, we encourage to select c f channels from a given zone, which is the neighborhood of the c -th channel, i.e. , B ( c ; r ) =
Revised: In detail, for free channels, we encourage to select c f channels from a given zone, which is the neighborhood of the c -th channel, i.e. , B ( c ; r ) =

Original: [ c − r : c + r ] , and r is a preset allowed offset to control the search space with c f = r + 1 .
Revised: [ c − r : c + r ] , and r is a preset allowed offset to control the search space with c f = r + 1 .

Original: Besides, base channels follow the ﬁxed weight sharing pattern with I b ( c ) =
Revised: Besides, base channels follow the ﬁxed weight sharing pattern with I b ( c ) =

Original: [0 : c b ] ; let I b ( c ) ∩ I f ( c ) = ∅ , then we can have c b = max( c − r − 1 , 0) .
Revised: [0 : c b ] ; let I b ( c ) ∩ I f ( c ) = ∅ , then we can have c b = max( c − r − 1 , 0) .

Original: In this way, in our formulation, we can specify multiple channel assignment for width c to better evaluate its performance as Fig.1(b), i.e. , locally free weight sharing pattern: I ( c ) =
Revised: In this way, in our formulation, we can specify multiple channel assignment for width c to better evaluate its performance as Fig.1(b), i.e. , locally free weight sharing pattern: I ( c ) =

Original: [1 : c b ] ∪ I f ( c ) , where I f ( c ) ⊂
Revised: [1 : c b ] ∪ I f ( c ) , where I f ( c ) ⊂

Original: [ c − r : c + r ] (5)
Revised: [ c − r : c + r ] (5)

Original: According to Eq.(4), the weight sharing degree of our method is between c b /c ≤ d ( c, ˜ c ) ≤ 1 .
Revised: According to Eq.(4), the weight sharing degree of our method is between c b /c ≤ d ( c, ˜ c ) ≤ 1 .

Original: By properly setting offset r , we can achieve a tradeoff between the ﬁxed pattern and the fully free pattern.
Revised: By properly setting offset r , we can achieve a tradeoff between the ﬁxed pattern and the fully free pattern.

Original: Given a r , now our search space scales at O ( C r +12 r +1 n ) , which is only a constant times ( e.g. , O (3 n ) for r = 1 , see Fig.1(b)) of ﬁxed weight sharing search space.
Revised: Given a r , now our search space scales at O ( C r +12 r +1 n ) , which is only a constant times ( e.g. , O (3 n ) for r = 1 , see Fig.1(b)) of ﬁxed weight sharing search space.

Original: In this way, each width can be better represented, thus boosts the performance ranking ability of supernet.
Revised: In this way, each width can be better represented, thus boosts the performance ranking ability of supernet.

Original: Training with min-min optimization.
Revised: Training with min-min optimization.

Original: With the above search space, we can train our CafeNet in a stochastic setting as Eq.(1), which simply samples a network width c ﬁrst, and then optimizes its sub-network.
Revised: With the above search space, we can train our CafeNet in a stochastic setting as Eq.(1), which simply samples a network width c ﬁrst, and then optimizes its sub-network.

Original: Nevertheless, in CafeNet a width is speciﬁed more freely by several sub-networks, and the performance of these sub-networks can be different.
Revised: Nevertheless, in CafeNet a width is speciﬁed more freely by several sub-networks, and the performance of these sub-networks can be different.

Original: In this way, we propose to indicate the performance for each width by examining its sub-network with the best performance.
Revised: In this way, we propose to indicate the performance for each width by examining its sub-network with the best performance.

Original: In training
Revised: In training

Original: CafeNet, instead of randomly optimizing a sub-network, we optimize the sub-network with the smallest training loss.
Revised: CafeNet, instead of randomly optimizing a sub-network, we optimize the sub-network with the smallest training loss.

Original: To achieve this, we need to traverse all sub-networks for width c .
Revised: To achieve this, we need to traverse all sub-networks for width c .

Original: However, this can be efﬁciently fulﬁlled since our job is to identify a sub-network with the minimum loss, which can be quickly calculated in a feed-forward manner.
Revised: However, this can be efﬁciently fulﬁlled since our job is to identify a sub-network with the minimum loss, which can be quickly calculated in a feed-forward manner.

Original: Afterwards, we only need to implement  backward update for the target sub-network with minimum loss.
Revised: Afterward, we only need to implement a backward update for the target sub-network with minimum loss.

Original: A detailed explanation of this min-min optimization is elaborated in Appendix A.1.
Revised: A detailed explanation of this min-min optimization is elaborated in Appendix A.1.

Original: Searching with max-max selection.
Revised: Searching with max-max selection.

Original: After the CafeNet N is trained, we can evaluate each network width by examining its performance ( e.g. classiﬁcation accuracy) on the validation dataset D val .
Revised: After the CafeNet N is trained, we can evaluate each network width by examining its performance ( e.g. classiﬁcation accuracy) on the validation dataset D val .

Original: Similar to the training of CafeNet, for each width c , we use the sub-network with the highest performance to indicate the performance.
Revised: Similar to the training of CafeNet, for each width c , we use the sub-network with the highest performance to indicate the performance.

Original: Then we search for the optimal c ∗ as Eq.(2).
Revised: Then we search for the optimal c ∗ as Eq.(2).

Original: Note that since searching itself is much faster than the supernet training, the increased computational cost is subtle and acceptable in real practice.
Revised: Note that since searching itself is much faster than the supernet training, the increased computational cost is subtle and acceptable in real practice.

Original: A detailed description of this max-max selection is presented in Appendix A.1.
Revised: A detailed description of this max-max selection is presented in Appendix A.1.

Original: 3. R EDUCING SEARCH SPACE WITH FLOP S - SENSITIVE BINS
Revised: 3. R EDUCING SEARCH SPACE WITH FLOP S - SENSITIVE BINS

Original: Based on the locally free weight sharing pattern, each network width can be more ﬂexibly indicated and then evaluated by the CafeNet.
Revised: Based on the locally free weight sharing pattern, each network width can be more ﬂexibly indicated and then evaluated by the CafeNet.

Original: However, as previously illustrated, the size of the search space is considerably large and brings a tough challenge for the subsequent searching algorithms, such as random search and evolutionary search.
Revised: However, as previously illustrated, the size of the search space is considerably large and brings a tough challenge for the subsequent searching algorithms, such as random search and evolutionary search.

Original: In this way, current methods usually choose to reduce the search space by initializing the minimum searching unit as a group of channels (called bin ) instead of a single channel.
Revised: In this way, current methods usually choose to reduce the search space by initializing the minimum searching unit as a group of channels (called bin ) instead of a single channel.

Original: Concretely, all channels at each layer are partitioned into K groups, with ( n i /K ) channels in each group for the i -th layer.
Revised: Concretely, all channels at each layer are partitioned into K groups, with ( n i /K ) channels in each group for the i -th layer.

Original: However, this uniform partition neglects the extra information of channels within layers, i.e. , kernel size and feature map resolution, and all of these factors are involved in the calculation of FLOPs.
Revised: However, this uniform partition neglects the extra information of channels within layers, i.e. , kernel size and feature map resolution, and all of these factors are involved in the calculation of FLOPs.

Original: This inspires us that we should build bins based on the FLOPs.
Revised: This inspires us that we should build bins based on the FLOPs.

Original: For simplicity, we encourage all bins in our search space to have the same FLOPs.
Revised: For simplicity, we encourage all bins in our search space to have the same FLOPs.

Original: Note that FLOPs is up to the layer width between two adjacent layers; for example, for convolution with resolution H × W and kernel size K × K ,
Revised: Note that FLOPs is up to the layer width between two adjacent layers; for example, for convolution with resolution H × W and kernel size K × K ,

Original: its
Revised: its

Original: FLOPs can be calculated as
Revised: FLOPs can be calculated as

Original: [Equation]
Revised: [Equation]

Original: where c in and c out are the width for the input and output of this layer, respectively.
Revised: where c in and c out are the width for the input and output of this layer, respectively.

Original: In this way, channels at different layers contribute differently to the overall FLOPs of the supernet.
Revised: In this way, channels at different layers contribute differently to the overall FLOPs of the supernet.

Original: And if a channel at a layer contributes more to the FLOPs, then we can say it is more sensitive w.r.t.
Revised: And if a channel at a layer contributes more to the FLOPs, then we can say it is more sensitive w.r.t.

Original: FLOPs.
Revised: FLOPs.

Original: This FLOPs-sensitivity can be simply reﬂected by examining the real FLOPs of a single channel at a layer.
Revised: This FLOPs-sensitivity can be simply reﬂected by examining the real FLOPs of a single channel at a layer.

Original: Concretely, for each channel at the i -th layer, it serves as both the output channel of the ( i − 1) -th layer and input channel of the ( i + 1) -th layer.
Revised: Concretely, for each channel at the i -th layer, it serves as both the output channel of the ( i − 1) -th layer and input channel of the ( i + 1) -th layer.

Original: Thus the sensitivity ε i for the i -th layer can be represented as
Revised: Thus the sensitivity ε i for the i -th layer can be represented as

Original: [Equation]
Revised: [Equation]

Original: where n i − 1 and n i +1 are the width of the ( i − 1) -th and the ( i + 1) -th layer, respectively.
Revised: where n i − 1 and n i +1 are the width of the ( i − 1) -th and the ( i + 1) -th layer, respectively.

Original: As a result, we simply encourage all bins to have  same FLOPs-sensitivity.
Revised: As a result, we simply encourage all bins to have the same FLOPs-sensitivity.

Original: To do so, we propose to initialize the bin size ( i.e. , number of channels in a bin) more adaptively.
Revised: To do so, we propose to initialize the bin size ( i.e. , number of channels in a bin) more adaptively.

Original: The intuition is that the larger FLOPs sensitivity a layer has, the fewer channels should be allocated in a bin, and vice versa.
Revised: The intuition is that the larger FLOPs sensitivity a layer has, the fewer channels should be allocated in a bin, and vice versa.

Original: As a result, the bin size b i for i -th layer should satisfy
Revised: As a result, the bin size b i for i -th layer should satisfy

Original: [Equation]
Revised: [Equation]

Original: In practice, we usually specify a minimum bin size  as β , then the bin size for each layer is
Revised: In practice, we usually specify a minimum bin size ( i.e. , with maximum sensitivity ε ) as β , then the bin size for each layer is

Original: [Equation]
Revised: [Equation]

Original: Actually, the minimum bin size β in Eq.(9) controls the total number of bins, thus also determines the size of the search space.
Revised: Actually, the minimum bin size β in Eq.(9) controls the total number of bins, thus also determines the size of the search space.

Original: If β is set to be smaller, the search space will be much larger and ﬁne-grained since our search unit gets smaller and delicate accordingly.
Revised: If β is set to be smaller, the search space will be much larger and ﬁne-grained since our search unit gets smaller and delicate accordingly.

Original: We will discuss a multi-stage search strategy by controlling β actively in Section 3.3, and present empirical investigations in Appendix
Revised: We will discuss a multi-stage search strategy by controlling β actively in Section 3.3, and present empirical investigations in Appendix

Original: A.15.
Revised: A.16.

Original: 3. F URTHER BOOSTING PERFORMANCE WITH MULTI - STAGE SEARCH
Revised: 3. F URTHER BOOSTING PERFORMANCE WITH MULTI - STAGE SEARCH

Original: As previously illustrated, setting minimum bin size β in Eq.(9) to be smaller will bring in a smaller searching unit, and thus a more ﬁne-grained structure will be expected.
Revised: As previously illustrated, setting minimum bin size β in Eq.(9) to be smaller will bring in a smaller searching unit, and thus a more ﬁne-grained structure will be expected.

Original: However, this is accompanied by a larger search space and is challenging for both the ranking ability of supernet and searching performance.
Revised: However, this is accompanied by a larger search space and is challenging for both the ranking ability of supernet and searching performance.

Original: In this way, we can leverage a multi-stage searching strategy to avoid the search space being too large but still can search on a ﬁne-grained level.
Revised: In this way, we can leverage a multi-stage searching strategy to avoid the search space being too large but still can search on a ﬁne-grained level.

Original: In detail, we propose to search for the width under the multi-stage FLOPs budget, which is decayed linearly, i.e. ,
Revised: In detail, we propose to search for the width under the multi-stage FLOPs budget, which is decayed linearly, i.e. ,

Original: FLOPs
Revised: FLOPs

Original: [Equation]
Revised: [Equation]

Original: where FLOPs ( t ) represents the FLOPs budget in t -th stage, and FLOPs (0) is the FLOPs of the supernet.
Revised: where FLOPs ( t ) represents the FLOPs budget in t -th stage and FLOPs (0) is the FLOPs of the supernet.

Original: F b is the target FLOPs after all T stages as Eq.(13).
Revised: F b is the target FLOPs after all T stages as Eq.(13).

Original: Besides, at each stage, we shrink the minimum bin size β with evolving speed α > 0 .
Revised: Besides, at each stage, we shrink the minimum bin size β with evolving speed α > 0 .

Original: Thus the search space will evolve from the coarsegrained to the ﬁne-grained, but with controllable size.
Revised: Thus the search space will evolve from the coarsegrained to the ﬁne-grained, but with controllable size.

Original: More details are explored in Appendix A. and A.14.
Revised: More details are explored in Appendix A. and A.15.

Original: However, searching with the multi-stage strategy inevitably introduces more computation consumption.
Revised: However, searching with the multi-stage strategy inevitably introduces more computation consumption.

Original: Therefore, we only implement this method in the small dataset ( i.e. CIFAR-10).
Revised: Therefore, we only implement this method in the small dataset ( i.e. CIFAR-10).

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this section, we perform extensive experiments to validate the effectiveness of our CafeNet.
Revised: In this section, we perform extensive experiments to validate the effectiveness of our CafeNet.

Original: For all architectures, we use the SGD optimizer with momentum 0.9.
Revised: For all architectures, we use the SGD optimizer with momentum 0.9.

Original: Parameters β in Eq.(9) and α in
Revised: Parameters β in Eq.(9) and α in

Original: Eq.(10) are set to 1 and 2, respectively.
Revised: Eq.(10) are set to 1 and 2, respectively.

Original: Offset r for locally free channels is set to 1 for all experiments.
Revised: Offset r for locally free channels is set to 1 for all experiments.

Original: Detailed experimental settings are elaborated in Appendix A.2.
Revised: Detailed experimental settings are elaborated in Appendix A.2.

Original: We include various state-of-the-art width search methods for comparison, e.g. AutoSlim (Yu & Huang, 2019), MetaPruning (Liu et al., 2019a), TAS (Dong & Yang, 2019), GBN (You et al., 2019) and
Revised: We include various state-of-the-art width search methods for comparison, e.g. AutoSlim (Yu & Huang, 2019), MetaPruning (Liu et al., 2019a), TAS (Dong & Yang, 2019), GBN (You et al., 2019) and

Original: FPGM (He et al., 2019a).
Revised: FPGM (He et al., 2019a).

Original: Besides, for comprehensive comparison, we perform searching using both evolutionary search and random search, named as CafeNet-E and CafeNet-R , respectively.
Revised: Besides, for comprehensive comparison, we perform searching using both evolutionary search and random search, named as CafeNet-E and CafeNet-R , respectively.

Original: In addition, we also consider two vanilla baselines, i.e. , Uniform and Random.
Revised: In addition, we also consider two vanilla baselines, i.e. , Uniform and Random.

Original: Uniform: we shrink the width of each layer uniformly with a ﬁxed factor to meet the FLOPs budget.
Revised: Uniform: we shrink the width of each layer uniformly with a ﬁxed factor to meet the FLOPs budget.

Original: Random: we randomly sample 20 networks under FLOPs constraint, and train them by 50 epochs, then we continue training the one with the highest performance and report its ﬁnal result.
Revised: Random: we randomly sample 20 networks under FLOPs constraint, and train them by 50 epochs, then we continue training the one with the highest performance and report its ﬁnal result.

Original: 4. R ESULTS ON I MAGE N ET D ATASET
Revised: 4. R ESULTS ON I MAGE N ET D ATASET

Original: We perform experiments on ResNet50 (He et al., 2016), MobileNetV2 (Sandler et al., 2018) and
Revised: We perform experiments on ResNet50 (He et al., 2016), MobileNetV2 (Sandler et al., 2018) and

Original: EfﬁcientNet-B0 (Tan & Le, 2019) to examine the performance of CafeNet on heavy and light models, as summarized in Table 1 and 2.
Revised: EfﬁcientNet-B0 (Tan & Le, 2019) to examine the performance of CafeNet on heavy and light models, as summarized in Table 1 and 2.

Original: In detail, ResNet50 and MobileNetV2 are searched from the original size, and for EfﬁcientNet-B0, the 0.5 × FLOPs is also searched from the original model.
Revised: In detail, ResNet50 and MobileNetV2 are searched from the original size, and for EfﬁcientNet-B0, the 0.5 × FLOPs is also searched from the original model.

Original: But the 1.0 × FLOPs is from a supernet with 1.5 × FLOPs from uniform width scaling.
Revised: But the 1.0 × FLOPs is from a supernet with 1.5 × FLOPs from uniform width scaling.

Original: The original
Revised: The original

Original: EfﬁcientNet-B0, ResNet50 and MobileNetV2 has 5.3M, 25.5M and 3.5M parameters and 386M, 4.1G and 300M FLOPs with 76.4%, 77.8% and 73.7% Top-1 accuracy, respectively.
Revised: EfﬁcientNet-B0, ResNet50 and MobileNetV2 has 5.3M, 25.5M and 3.5M parameters and 386M, 4.1G and 300M FLOPs with 76.4%, 77.8% and 73.7% Top-1 accuracy, respectively.

Original: More detailed experiments and visualization results of our searched network are shown in Appendix A.8 and A.10.
Revised: More detailed experiments and visualization results of our searched network are shown in Appendix A.8 and A.11.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: As shown in Table
Revised: As shown in Table

Original: [Table]
Revised: [Table]

Original: our CafeNet achieves the highest performance on ResNet50 and MobileNetV w.r
Revised: our CafeNet achieves the highest performance on ResNet50 and MobileNetV w.r

Original: [Table]
Revised: [Table]

Original: t. different FLOPs, which proves the superiority of our method compared to other width search algorithms.
Revised: t. different FLOPs, which proves the superiority of our method compared to other width search algorithms.

Original: Besides, CafeNet obtains remarkable performance with tiny FLOPs budgets.
Revised: Besides, CafeNet obtains remarkable performance with tiny FLOPs budgets.

Original: For example, our 570M ResNet50 achieves 73.3% Top-1 accuracy, surpassing AutoSlim (Yu & Huang, 2019) by 1.1%.
Revised: For example, our 570M ResNet50 achieves 73.3% Top-1 accuracy, surpassing AutoSlim (Yu & Huang, 2019) by 1.1%.

Original: In addition, as shown in Table 2, for the NAS-based network EfﬁcientNet-B0, the performance can be further improved using CafeNet.
Revised: In addition, as shown in Table 2, for the NAS-based network EfﬁcientNet-B0, the performance can be further improved using CafeNet.

Original: It indicates that our CafeNet can optimize its width in a more ﬁne-grained manner.
Revised: It indicates that our CafeNet can optimize its width in a more ﬁne-grained manner.

Original: 4. R ESULTS ON CIFAR-10 D ATASET
Revised: 4. R ESULTS ON CIFAR-10 D ATASET

Original: We also investigate CafeNet with MobileNetV2 (Sandler et al., 2018) and VGGNet (Simonyan &
Revised: We also investigate CafeNet with MobileNetV2 (Sandler et al., 2018) and VGGNet (Simonyan &

Original: Zisserman, 2014) on the moderate CIFAR-10 dataset.
Revised: Zisserman, 2014) on the moderate CIFAR-10 dataset.

Original: Our original VGGNet (MobileNetV2) has 20M (2.2M) parameters and 399M (297M) FLOPs with accuracy of 93.99% (94.81%).
Revised: Our original VGGNet (MobileNetV2) has 20M (2.2M) parameters and 399M (297M) FLOPs with accuracy of 93.99% (94.81%).

Original: As shown in Table 3, our CafeNet enjoys signiﬁcant superiority to other algorithms.
Revised: As shown in Table 3, our CafeNet enjoys signiﬁcant superiority to other algorithms.

Original: In detail, our 44M MobileNetV achieves 95.31% accuracy and outperforms MuffNet (Chen et al., 2019) by 2.2%.
Revised: In detail, our 44M MobileNetV achieves 95.31% accuracy and outperforms MuffNet (Chen et al., 2019) by 2.2%.

Original: For VGGNet, our CafeNet achieves 94.36% accuracy with 189M FLOPs, which surpasses DCP (Zhuang et al., 2018) by 0.2%.
Revised: For VGGNet, our CafeNet achieves 94.36% accuracy with 189M FLOPs, which surpasses DCP (Zhuang et al., 2018) by 0.2%.

Original: Similarly, our CafeNet shows great advantages w.r.t.
Revised: Similarly, our CafeNet shows great advantages w.r.t.

Original: tiny models, e.g. with 2.11% accuracy improvement than AutoSlim (Yu & Huang, 2019) on MobileNetV2 with 28M FLOPs.
Revised: tiny models, e.g. with 2.11% accuracy improvement than AutoSlim (Yu & Huang, 2019) on MobileNetV2 with 28M FLOPs.

Original: More experiments on CIFAR-10 dataset are investigated in Appendix A.9.
Revised: More experiments on CIFAR-10 dataset are investigated in Appendix A.9.

Original: 4. E XPERIMENTS ON FACE ATTRIBUTE TASK WITH C ELEB A DATASET
Revised: 4. E XPERIMENTS ON FACE ATTRIBUTE TASK WITH C ELEB A DATASET

Original: To futher investigate the effectiveness of CafeNet, we perform experiments of multi-label classiﬁcation task on CelebA (Liu et al., 2015) dataset with MobileNetV2 (Sandler et al., 2018) and ResNet18 (He et al., 2016).
Revised: To futher investigate the effectiveness of CafeNet, we perform experiments of multi-label classiﬁcation task on CelebA (Liu et al., 2015) dataset with MobileNetV2 (Sandler et al., 2018) and ResNet18 (He et al., 2016).

Original: CelebA dataset has 162K training and 20K testing images with 40 attribute labels from 2 categories.
Revised: CelebA dataset has 162K training and 20K testing images with 40 attribute labels from 2 categories.

Original: We report the average accuracy of all 40 labels.
Revised: We report the average accuracy of all 40 labels.

Original: The original MobileNetV2 (ResNet18) has 213M (1.3G) FLOPs and 2.2M (11.2M) parameters with 92.07% (92.13%) average accuracy.
Revised: The original MobileNetV2 (ResNet18) has 213M (1.3G) FLOPs and 2.2M (11.2M) parameters with 92.07% (92.13%) average accuracy.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: As shown in Table
Revised: As shown in Table

Original: [Table]
Revised: [Table]

Original: our CafeNet can even achieve higher accuracy with FLOPs reducing a bit.
Revised: our CafeNet can even achieve higher accuracy with FLOPs reducing a bit.

Original: For example, our 162M-FLOPs MobileNetV2 has 92.19% average accuracy and outperforms the original model by 0.12%.
Revised: For example, our 162M-FLOPs MobileNetV2 has 92.19% average accuracy and outperforms the original model by 0.12%.

Original: Besides, our algorithm still enjoys great superiority in tiny FLOPs; for example, our 130M-ResNet18 achieves 91.92% average accuracy, which is 0.26% higher than the uniform baseline.
Revised: Besides, our algorithm still enjoys great superiority in tiny FLOPs; for example, our 130M-ResNet18 achieves 91.92% average accuracy, which is 0.26% higher than the uniform baseline.

Original: Besides, to comprehensively compare the classiﬁcation performances on all 40 labels, we present the comparison results for each label in Appendix A.9.
Revised: Besides, to comprehensively compare the classiﬁcation performances on all 40 labels, we present the comparison results for each label in Appendix A.9.

Original: 4. T RANSFERABILITY OF THE SEARCHED WIDTH TO OBJECT DETECTION TASK
Revised: 4. T RANSFERABILITY OF THE SEARCHED WIDTH TO OBJECT DETECTION TASK

Original: For object detection task, its backbone is usually initialized by a pretrained model on ImgeNet dataset.
Revised: For object detection tasks, its backbone is usually initialized by a pretrained model on ImgeNet dataset.

Original: In this way, we use MobileNetV2 and ResNet50 with 0.5 × FLOPs as the backbone feature extractors to examine the transferability of our searched network width.
Revised: In this way, we use MobileNetV2 and ResNet50 with 0.5 × FLOPs as the backbone feature extractors to examine the transferability of our searched network width.

Original: Besides, we leverage both the two-stage Faster R-CNN with Feature Pyramid Networks (FPN) (Lin et al., 2017a;
Revised: Besides, we leverage both the two-stage Faster R-CNN with Feature Pyramid Networks (FPN) (Lin et al., 2017a;

Original: Ren et al., 2015) and the one-stage RetinaNet (Lin et al., 2017b) frameworks for veriﬁcation.
Revised: Ren et al., 2015) and the one-stage RetinaNet (Lin et al., 2017b) frameworks for veriﬁcation.

Original: We train models using the trainval split of MS COCO (Everingham et al., 2010) as training data and report the results in mean Average Precision (Lin et al., 2017a;
Revised: We train models using the trainval split of MS COCO (Everingham et al., 2010) as training data and report the results in mean Average Precision (Lin et al., 2017a;

Original: Ren et al., 2015) on minival split.
Revised: Ren et al., 2015) on minival split.

Original: As shown in Table 5, the backbones obtained by our method (CafeNet 0.5 × ) can achieve higher performance compared to the uniform baseline (Uni 0.5 × ).
Revised: As shown in Table 5, the backbones obtained by our method (CafeNet 0.5 × ) can achieve higher performance compared to the uniform baseline (Uni 0.5 × ).

Original: 4. A BLATION S TUDIES
Revised: 4. A BLATION S TUDIES

Original: Effect of offset r for free channels.
Revised: Effect of offset r for free channels.

Original: As described before, free channels freely locate in the local zone indicated by an offset r .
Revised: As described before, free channels freely locate in the local zone indicated by an offset r .

Original: Since we search on bins, r controls the size of search space; larger r implies more accurate evaluation of width but induces larger search space, corresponding to more computational cost.
Revised: Since we search on bins, r controls the size of search space; larger r implies more accurate evaluation of width but induces larger search space, corresponding to more computational cost.

Original: To investigate its effect, we compare the performance and time cost between different r on 0.5 × -FLOPs MobileNetV2 and VGGNet on CIFAR-10 dataset.
Revised: To investigate its effect, we compare the performance and time cost between different r on 0.5 × -FLOPs MobileNetV2 and VGGNet on CIFAR-10 dataset.

Original: As shown in Fig.2(a), accuracy performance beneﬁts from the increase of r while the larger r also aggravates the burden of training.
Revised: As shown in Fig.2(a), accuracy performance beneﬁts from the increase of r while the larger r also aggravates the burden of training.

Original: For a trade-off between performance and time cost, we set r = 1 in all experiments.
Revised: For a trade-off between performance and time cost, we set r = 1 in all experiments.

Original: Note that in  training of CafeNet, larger r will have more sub-networks; however, identifying the one with  minimum loss only involves feed-forward calculation, and does not increase training cost dramatically.
Revised: Note that in the training of CafeNet, larger r will have more sub-networks; however, identifying the one with the minimum loss only involves feed-forward calculation and does not increase training cost dramatically.

Original: More results refer to Appendix A.5.
Revised: More results refer to Appendix A.5.

Original: [Figure]
Revised: [Figure]

Original: Effect of min-min optimization on supernet .
Revised: Effect of min-min optimization on supernet .

Original: During training CafeNet, we only optimize the sub-network with minimum training loss for each sampled width.
Revised: During training CafeNet, we only optimize the sub-network with minimum training loss for each sampled width.

Original: To investigate the effect of this optimization strategy, suppose we have all τ iterations, then we implement min-min optimization only on the last  λ  · τ iterations with λ ∈ [0 , 1] .
Revised: To investigate the effect of this optimization strategy, suppose we have all τ iterations, then we implement min-min optimization only on the last (1 − λ ) · τ iterations with λ ∈ [0 , 1] .

Original: For the ﬁrst(1 − λ) · τ iterations, we simply optimize one sub-network randomly for each sampled width.
Revised: For the ﬁrst λ · τ iterations, we simply optimize one sub-network randomly for each sampled width.

Original: As shown in Fig.2(b), our 0.5 × -FLOPs
Revised: As shown in Fig.2(b), our 0.5 × -FLOPs

Original: MobileNetV2 on CIFAR-10 improves 0.92% accuracy from λ = 0 to λ = 1 , which means that min-min optimization does help to better evaluate each width and boost the searching performance accordingly.
Revised: MobileNetV2 on CIFAR-10 improves 0.92% accuracy from λ = 1 to λ = 0 , which means that min-min optimization does help to better evaluate each width and boost the searching performance accordingly.

Original: We further record the performance of 1K sampled width;  to Appendix A
Revised: We further record the performance of 1K sampled width; details refer to Appendix A.13.

Original: Effect of performance improvement with more training and searching cost .
Revised: Effect of performance improvement with more training and searching cost .

Original: Since CafeNet needs extra training and searching cost than the ﬁxed weight sharing pattern, one natural question comes to whether  ﬁxed pattern can beneﬁt from more training and searching cost and even surpass
Revised: Since CafeNet needs extra training and searching cost than the ﬁxed weight sharing pattern, one natural question comes to whether a ﬁxed pattern can beneﬁt from more training and searching cost and even surpass

Original: CafeNet as a result .
Revised: CafeNet as a result .

Original: With this aim, we implement  ﬁxed pattern ( i.e. , r = 0 for CafeNet) with more times (1 ×∼ 3 × ) of training epochs and generations using evolutionary search to search for a 0.5 × -FLOPs MobileNetV2 and VGGNet on CIFAR-10 dataset.
Revised: With this aim, we implement a ﬁxed pattern ( i.e. , r = 0 for CafeNet) with more times (1 ×∼ 3 × ) of training epochs and generations using evolutionary search to search for a 0.5 × -FLOPs MobileNetV2 and VGGNet on CIFAR-10 dataset.

Original:  and searching cost on CIFAR-10 dataset.
Revised: more training and searching cost on CIFAR-10 dataset.

Original: Note CafeNet has 95.44% and 94.36% accuracy for  MobileNetV2 and , respectively.
Revised: Note CafeNet has 95.44% and 94.36% accuracy for 0.5 × -FLOPs MobileNetV2 and VGGNet, respectively.

Original: [Table]
Revised: [Table]

Original: As shown in Table 6, we can see that searching with more generations by evolutionary search can only improve the searching results slightly, while more times of training cost for supernet has almost no effect on the searching performance.
Revised: As shown in Table 6, we can see that searching with more generations by evolutionary search can only improve the searching results slightly, while more times of training cost for supernet has almost no effect on the searching performance.

Original: By comparing with Table 3 and Fig.2(a), our CafeNet does efﬁciently improve the performance of searching results with the same or a little extra training and searching cost.
Revised: By comparing with Table 3 and Fig.2(a), our CafeNet does efﬁciently improve the performance of searching results with the same or a little extra training and searching cost.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduce a new weight sharing pattern for network width search.
Revised: In this paper, we introduce a new weight sharing pattern for network width search.

Original: In detail, our locally free pattern enables each width is jointly indicated by its base channels and free channels, where free channels are used to better and more ﬂexibly to evaluate the performance.
Revised: In detail, our locally free pattern enables each width is jointly indicated by its base channels and free channels, where free channels are used to better and more ﬂexibly to evaluate the performance.

Original: Besides, we leverage FLOPs-sensitive bins to reduce the search space and allow FLOPs to distribute more evenly for different layers.
Revised: Besides, we leverage FLOPs-sensitive bins to reduce the search space and allow FLOPs to distribute more evenly for different layers.

Original: CafeNet can be trained stochastically using min-min optimization and searches the optimal width by max-max selection.
Revised: CafeNet can be trained stochastically using min-min optimization and searches the optimal width by max-max selection.

Original: Extensive experiments have been conducted on benchmark
Revised: Extensive experiments have been conducted on benchmark

Original: ImageNet, CIFAR-10, CelebA, and COCO datasets to show the superiority of our CafeNet to other state-of-the-art methods.
Revised: ImageNet, CIFAR-10, CelebA, and COCO datasets to show the superiority of our CafeNet to other state-of-the-art methods.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: We organize our appendix as follows.
Revised: We organize our appendix as follows.

Original: We explain the details of using CafeNet for training and searching in appendix A.1.
Revised: We explain the details of using CafeNet for training and searching in appendix A.1.

Original: In appendix A.2, we give details of our experiments w.r.t.
Revised: In appendix A.2, we give details of our experiments w.r.t.

Original: different datasets.
Revised: different datasets.

Original: We report the implementation details of our CafeNet with Algorithm 1 in appendix A.3.
Revised: We report the implementation details of our CafeNet with Algorithm 1 in appendix A.3.

Original: Then, we introduce the corresponding relationship between names and references in appendix A.4.
Revised: Then, we introduce the corresponding relationship between names and references in appendix A.4.

Original: And we explore the efﬁciency of CafeNet in training one-shot supernet N A.5.
Revised: And we explore the efﬁciency of CafeNet in training one-shot supernet N A.5.

Original: We show the performance gain of each part of CafeNet in appendix A.6.
Revised: We show the performance gain of each part of CafeNet in appendix A.6.

Original: We examine the effect of FLOPs-sensitive bins in appendix A.7.
Revised: We examine the effect of FLOPs-sensitive bins in appendix A.7.

Original: We explore more detailed experiments of ImageNet, CIFAR-10, and CelebA dataset in appendix A.8 and A.9.
Revised: We explore more detailed experiments of ImageNet, CIFAR-10, and CelebA dataset in appendix A.8 and A.9.

Original: Then we show the visualization of searched network width in appendix A.10.
Revised: Then we show the visualization of searched network width in appendix A.11.

Original: In appendix A.11, we visualize the conﬁguration of bins with EfﬁcientNet-B0 and
Revised: In appendix A.12, we visualize the conﬁguration of bins with EfﬁcientNet-B0 and

Original: MobileNetV2.
Revised: MobileNetV2.

Original: In addition, we investigate the effect of min-min optimization with MobileNetV2 and
Revised: In addition, we investigate the effect of min-min optimization with MobileNetV2 and

Original: ResNet50 in appendix A.12.
Revised: ResNet50 in appendix A.13.

Original: We study the impact of the multi-stage search with evolving bins on
Revised: We study the impact of the multi-stage search with evolving bins on

Original: CIFAR-10 dataset in appendix A.13.
Revised: CIFAR-10 dataset in appendix A.14.

Original: We provide the estimation of the size of the search space in the multi-stage search in appendix A.14.
Revised: We provide the estimation of the size of the search space in the multi-stage search in appendix A.15.

Original: Finally, the effect of the smallest bin size β in FLOPs-sensitive bins is shown in appendix A.15.
Revised: Finally, the effect of the smallest bin size β in FLOPs-sensitive bins is shown in appendix A.16.

Original: A. T RAINING AND SEARCHING WITH C AFE N ET
Revised: A. T RAINING AND SEARCHING WITH C AFE N ET

Original: Training with min-min optimization.
Revised: Training with min-min optimization.

Original: With the bin-based search space, we can train our CafeNet in a stochastic setting as Eq.(1), which simply samples a network width c ﬁrst, and then optimizes the corresponding sub-network.
Revised: With the bin-based search space, we can train our CafeNet in a stochastic setting as Eq.(1), which simply samples a network width c ﬁrst, and then optimizes the corresponding sub-network.

Original: Nevertheless, in CafeNet, a width is speciﬁed more freely by several sub-networks, and the performance of these sub-networks can be different to a great extent.
Revised: Nevertheless, in CafeNet, a width is speciﬁed more freely by several sub-networks, and the performance of these sub-networks can be different to a great extent.

Original: According to this, we propose to indicate the performance of width by examining its sub-network with the best performance.
Revised: According to this, we propose to indicate the performance of width by examining its sub-network with the best performance.

Original: Concretely, during the training of CafeNet, instead of randomly optimizing a sub-network, we optimize the sub-network with the smallest training loss.
Revised: Concretely, during the training of CafeNet, instead of randomly optimizing a sub-network, we optimize the sub-network with the smallest training loss.

Original: For a network width c , we denote its corresponding sub-network set as S c , thus for the CafeNet N with weights W , the training target of min-min optimization can be written as,
Revised: For a network width c , we denote its corresponding sub-network set as S c , thus for the CafeNet N with weights W , the training target of min-min optimization can be written as,

Original: [Equation]
Revised: [Equation]

Original: where w s ⊂ W , denotes the weights of subnet s .
Revised: where w s ⊂ W , denotes the weights of subnet s .

Original: To ﬁnd the optimal sub-network s ∗ c , it needs to traverse all sub-networks, and results to additional computational cost.
Revised: To ﬁnd the optimal sub-network s ∗ c , it needs to traverse all sub-networks and results in additional computational cost.

Original: However, this can be efﬁciently implemented by calculating the loss of all networks without back propagation ( i.e. , no_grad mode in
Revised: However, this can be efﬁciently implemented by calculating the loss of all networks without backpropagation ( i.e. , no_grad mode in

Original: PyTorch).
Revised: PyTorch).

Original: And then, we only need to backward once using the target sub-network s ∗ c .
Revised: And then, we only need to backward once using the target sub-network s ∗ c .

Original: Searching with max-max selection.
Revised: Searching with max-max selection.

Original: After the CafeNet N is trained, we can evaluate each network width by examining its performance ( e.g. classiﬁcation accuracy) on the validation dataset D val .
Revised: After the CafeNet N is trained, we can evaluate each network width by examining its performance ( e.g. classiﬁcation accuracy) on the validation dataset D val .

Original: Similar to the training of CafeNet, for a network width c , we use the sub-network with the highest performance to indicate its performance.
Revised: Similar to the training of CafeNet, for a network width c , we use the sub-network with the highest performance to indicate its performance.

Original: Then the searching amounts to a max-max selection problem
Revised: Then the searching amounts to a max-max selection problem

Original: [Equation]
Revised: [Equation]

Original: Note that the inner max also needs to calculate the validation accuracy of all sub-networks.
Revised: Note that the inner max also needs to calculate the validation accuracy of all sub-networks.

Original: Nevertheless, since searching itself is much faster than the supernet training, the increased computational cost is subtle and acceptable in real practice.
Revised: Nevertheless, since searching itself is much faster than the supernet training, the increased computational cost is subtle and acceptable in real practice.

Original: A. D ETAILS OF EXPERIMENTAL SETTINGS
Revised: A. D ETAILS OF EXPERIMENTAL SETTINGS

Original: In this section, we present the implementation details of our CafeNet w.r.t.
Revised: In this section, we present the implementation details of our CafeNet w.r.t.

Original: experiments on various datasets.
Revised: experiments on various datasets.

Original: In general, for most networks except EfﬁcientNet-B0, we use  SGD optimizer with momentum 0.9.
Revised: In general, for most networks except EfﬁcientNet-B0, we use a SGD optimizer with momentum 0.9.

Original: While for EfﬁcientNet-B0, we use RMSprop optimizer with epsilon set to 0.001.
Revised: While for EfﬁcientNet-B0, we use RMSprop optimizer with epsilon set to 0.001.

Original: The parameters of β and α are initialized to 1 and 2, respectively, to determine the search space.
Revised: The parameters of β and α are initialized to 1 and 2, respectively, to determine the search space.

Original: For evolutionary search, we set the population and iteration size to 40 and 50, respectively.
Revised: For evolutionary search, we set the population and iteration size to 40 and 50, respectively.

Original: Besides, 2k network width are sampled for a random search.
Revised: Besides, 2k network width are sampled for a random search.

Original: After searching, the optimized network width is trained from scratch for evaluation.
Revised: After searching, the optimized network width is trained from scratch for evaluation.

Original: ImageNet and CIFAR-10 dataset.
Revised: ImageNet and CIFAR-10 dataset.

Original: we set weight decay to 10 − 4 for ResNet50 (same setting is also adopted for ResNet34 and ResNet18) and VGGNet while 5 × 10 − 5 for MobileNetV2.
Revised: we set weight decay to 10 − 4 for ResNet50 (same setting is also adopted for ResNet34 and ResNet18) and VGGNet while 5 × 10 − 5 for MobileNetV2.

Original: The learning rate is decayed with cosine strategy from 0.1 to 10 − 5 for MobileNetV2 and ResNet50 withtraining epochs and a mini-batch size of 256.
Revised: The learning rate is decayed with cosine strategy from 0.1 to 10 − 5 for MobileNetV2 and ResNet50 withtraining epochs and a mini-batch size of 256.

Original: Besides, for ResNet50, we adopt the strategy of random erasing with a probability of 0.4.
Revised: Besides, for ResNet50, we adopt the strategy of random erasing with a probability of 0.4.

Original: For VGGNet, we train for 400 epochs with batch size 128, and the learning rate is decayed by step strategy from 0.1 and divided by 10 at 150-th, 225-th, 300-th epoch.
Revised: For VGGNet, we train for 400 epochs with batch size 128, and the learning rate is decayed by step strategy from 0.1 and divided by 10 at 150-th, 225-th, 300-th epoch.

Original: For EfﬁcientNet-B0, the learning rate is initialized to 0.13 and decayed by 0.96 for every 3 epochs.
Revised: For EfﬁcientNet-B0, the learning rate is initialized to 0.13 and decayed by 0.96 for every 3 epochs.

Original: We train EfﬁcientNet-B0 for 300 epochs, of which the ﬁrst 5 are warm-up epochs, and the weight decay is set to 1 × 10 − 5 .
Revised: We train EfﬁcientNet-B0 for 300 epochs, of which the ﬁrst 5 are warm-up epochs, and the weight decay is set to 1 × 10 − 5 .

Original: CelebA dataset.
Revised: CelebA dataset.

Original: We set the training epochs to 15 (10) for MobileNetV2 and ResNet18 with learning rate annealed from 0.1 to 10 − 5 by cosine strategy, and the batch size of these two networks are set to 64.
Revised: We set the training epochs to 15 (10) for MobileNetV2 and ResNet18 with learning rate annealed from 0.1 to 10 − 5 by cosine strategy, and the batch size of these two networks are set to 64.

Original: MS COCO dataset.
Revised: MS COCO dataset.

Original: We conduct object detection experiments using two popular frameworks Faster
Revised: We conduct object detection experiments using two popular frameworks Faster

Original: R-CNN with Feature Pyramid Networks (FPN) (Lin et al., 2017a) and RetinaNet (Lin et al., 2017b) on MS COCO dataset (Everingham et al., 2010).
Revised: R-CNN with Feature Pyramid Networks (FPN) (Lin et al., 2017a) and RetinaNet (Lin et al., 2017b) on MS COCO dataset (Everingham et al., 2010).

Original: The trainval35k split is used for training and we report the mean Average Precision (mAP) on minival split.
Revised: The trainval35k split is used for training and we report the mean Average Precision (mAP) on minival split.

Original: We train all the models using SGD forepochs from ImageNet pretrained weights, and the initial learning rate is set to 0.08 with batch size 256, which decays 0.1 at 8-th and 11-th epoch.
Revised: We train all the models using SGD forepochs from ImageNet pretrained weights, and the initial learning rate is set to 0.08 with batch size 256, which decays 0.1 at 8-th and 11-th epoch.

Original: A. A LGORITHM OF C AFE N ET
Revised: A. A LGORITHM OF C AFE N ET

Original: The details about CafeNet are presented in Algorithm 1.
Revised: The details about CafeNet are presented in Algorithm 1.

Original: In speciﬁc, we search network width on
Revised: In speciﬁc, we search network width on

Original: CIFAR-10 dataset with 3 stages while others with 1 stage.
Revised: CIFAR-10 dataset with 3 stages while others with 1 stage.

Original: Input
Revised: Input

Original: : The number of multi-stage M .
Revised: : The number of multi-stage M .

Original: The training epochs of
Revised: The training epochs of

Original: supernet
Revised: supernet

Original: N .
Revised: N .

Original: The smallest bin size β and the
Revised: The smallest bin size β and the

Original: bin evloving
Revised: bin evloving

Original: 1: while stage < M do 2: initialize the supernet N and bins groups.
Revised: 1: while stage < M do 2: initialize the supernet N and bins groups.

Original: 3: while epochs < N do 4: randomly assign the training and backwards path 5: updating the corresponding path of bin groups with dataset D tr .
Revised: 3: while epochs < E do 4: randomly assign the training and backwards path 5: updating the corresponding path of bin groups with dataset D tr .

Original: 6: end while 7: update β = β / α 8: end while 9: search the optimized network width within supernet N by searching methods A (as illustrated in section 3.3).
Revised: 6: end while 7: update β = β / α 8: end while 9: search the optimized network width within supernet N by searching methods A (as illustrated in section 3.3).

Original: 10: train the optimized network width from scratch for evaluation.
Revised: 10: train the optimized network width from scratch for evaluation.

Original: Output: the optimized network width with evaluation
Revised: Output: the optimized network width with evaluation

Original: [Table]
Revised: [Table]

Original: A. I NDEX AND LITERATURE
Revised: A. I NDEX AND LITERATURE

Original: In order to simplify the correspondence between names and references in all tables
Revised: In order to simplify the correspondence between names and references in all tables

Original: [Table]
Revised: [Table]

Original: we have listed the names used and their corresponding papers as follows:
Revised: we have listed the names used and their corresponding papers as follows:

Original: Table 7: The names used in tables and their corresponding papers.
Revised: Table 7: The names used in tables and their corresponding papers.

Original: [Table]
Revised: [Table]

Original: A. T HE EFFICIENCY OF C AFE N ET IN TRAINING ONE - SHOT SUPERNET N
Revised: A. T HE EFFICIENCY OF C AFE N ET IN TRAINING ONE - SHOT SUPERNET N

Original: To investigate the searching efﬁciency of CafeNet
Revised: To investigate the searching efﬁciency of CafeNet

Original: [Table]
Revised: [Table]

Original: we examine the time cost of training 1 epoch with different radius r .
Revised: we examine the time cost of training 1 epoch with different radius r .

Original: As shown in Table 8, the training time of CafeNet raised with an increase of r , which is because r represents the preset allowed offset for free channels, thus increasing the times of forwarding for each batch  information.
Revised: As shown in Table 8, the training time of CafeNet raised with an increase of r , which is because r represents the preset allowed offset for free channels, thus increasing the times of forwarding for each batch of information.

Original: However, since CafeNet involves multi-times forward and once backward and the time cost of forwarding is much shorter than the backward, the overall training efﬁciency will not be greatly affected.
Revised: However, since CafeNet involves multi-times forward and once backward and the time cost of forwarding is much shorter than the backward, the overall training efﬁciency will not be greatly affected.

Original: To balance the performance and search efﬁciency
Revised: To balance the performance and search efﬁciency

Original: [Table]
Revised: [Table]

Original: we set r = 1 for all experiments with CafeNet.
Revised: we set r = 1 for all experiments with CafeNet.

Original: [Table]
Revised: [Table]

Original: A. P ERFORMANCE GAIN OF EACH PART IN C AFE N ET
Revised: P ERFORMANCE GAIN OF EACH PART IN C AFE N ET

Original: To investigate the performance gain of each part in CafeNet
Revised: To investigate the performance gain of each part in CafeNet

Original: [Table]
Revised: [Table]

Original: we conduct experiments to search for 0.5 × FLOPs MobileNetV2 and 0.5 × FLOPs VGGNet on CIFAR-10 dataset and report their Topaccuracy.
Revised: we conduct experiments to search for 0.5 × FLOPs MobileNetV2 and 0.5 × FLOPs VGGNet on CIFAR-10 dataset and report their Topaccuracy.

Original: Since min-min optimization and max-max selection are always used simultaneously, for clarity, we abbreviate them as min-min optimization.
Revised: Since min-min optimization and max-max selection are always used simultaneously, for clarity, we abbreviate them as min-min optimization.

Original: As shown in Table 9, with evolutionary search,
Revised: As shown in Table 9, with evolutionary search,

Original: CafeNet can enjoy a performance gain of 0.91% (0.74%) accuracy on MobileNetV2 (VGGNet).
Revised: CafeNet can enjoy a performance gain of 0.91% (0.74%) accuracy on MobileNetV2 (VGGNet).

Original: Besides, similar results can also be obtained through the greedy search, which exactly shows that
Revised: Besides, similar results can also be obtained through the greedy search, which exactly shows that

Original: our
Revised: our

Original: CafeNet can boost to search for a decent network width.
Revised: CafeNet can boost to search for a decent network width.

Original: [Table]
Revised: [Table]

Original: A. E FFECT OF FLOP S - SENSITIVE BINS
Revised: A. E FFECT OF FLOP S - SENSITIVE BINS

Original: Our proposed FLOPs-sensitive bins aim to reduce the search space by allowing the FLOPs to distribute more evenly over layers for the search unit.
Revised: Our proposed FLOPs-sensitive bins aim to reduce the search space by allowing the FLOPs to distribute more evenly over layers for the search unit.

Original: To explore the effect of FLOPs-sensitive bins as a search unit compared with the uniform unit (uniform channel group)
Revised: To explore the effect of FLOPs-sensitive bins as a search unit compared with the uniform unit (uniform channel group)

Original: [Table]
Revised: [Table]

Original: we search for network width under these two search units on CIFAR-10 dataset with MobileNetV2 and VGGNet, as shown in Table 10.
Revised: we search for network width under these two search units on CIFAR-10 dataset with MobileNetV2 and VGGNet, as shown in Table 10.

Original: In detail
Revised: In detail

Original: [Table]
Revised: [Table]

Original: compared with the uniform unit, higher accuracy under various FLOPs budgets can be achieved by leveraging FLOPs-sensitive bins.
Revised: compared with the uniform unit, higher accuracy under various FLOPs budgets can be achieved by leveraging FLOPs-sensitive bins.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. M ORE DETAILED RESULTS OF I MAGE N ET DATASET FOR T ABLE
Revised: A. M ORE DETAILED RESULTS OF I MAGE N ET DATASET FOR T ABLE

Original: To explore the effectiveness of CafeNet
Revised: To explore the effectiveness of CafeNet

Original: [Table]
Revised: [Table]

Original: we also implement our algorithm on ResNet34 and ResNet with the same training strategy as ResNet50.
Revised: we also implement our algorithm on ResNet34 and ResNet with the same training strategy as ResNet50.

Original: The original ResNet34 and ResNet18 has 21.8M, 11.7M and 3.6G, 1.8G FLOPs with 74.9%, 71.5% Top-1 accuracy, respectively.
Revised: The original ResNet34 and ResNet18 has 21.8M, 11.7M and 3.6G, 1.8G FLOPs with 74.9%, 71.5% Top-1 accuracy, respectively.

Original: The detailed results are reported in Table 11, our 0.75 × FLOPs ResNet34 and ResNet18 can achieve close performance to the origin model.
Revised: The detailed results are reported in Table 11, our 0.75 × FLOPs ResNet34 and ResNet18 can achieve close performance to the origin model.

Original: Besides, under the tiny FLOPs budget ( i
Revised: Besides, under the tiny FLOPs budget ( i

Original: [Table]
Revised: [Table]

Original: e. , 10% FLOPs), CafeNet can outperform the unform baseline by a large margin
Revised: e. , 10% FLOPs), CafeNet can outperform the unform baseline by a large margin

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. M ORE DETAILED RESULTS OF CIFAR-10 AND C ELEB A DATASET FOR T ABLE 4 AND
Revised: A. M ORE DETAILED RESULTS OF CIFAR-10 AND C ELEB A DATASET FOR T ABLE 4 AND

Original: To examine the hyperparameters setting with CIFAR-10 and CelebA dataset
Revised: To examine the hyperparameters setting with CIFAR-10 and CelebA dataset

Original: [Table]
Revised: [Table]

Original: we also explore the searching results from two vanilla baselines named Random and Uniform, as described in Section 4.
Revised: we also explore the searching results from two vanilla baselines named Random and Uniform, as described in Section 4.

Original: The detailed results on CIFAR-10 dataset and CelebA dataset are reported in Table 12 and Table 13, respectively.
Revised: The detailed results on CIFAR-10 dataset and CelebA dataset are reported in Table 12 and Table 13, respectively.

Original: Besides, for CelebA dataset, the layer widths of fully-connection(FC) layers are not implemented to be searched and keep the same as (Sener & Koltun, 2018)
Revised: Besides, for CelebA dataset, the layer widths of fully-connection(FC) layers are not implemented to be searched and keep the same as (Sener & Koltun, 2018)

Original: [Table]
Revised: [Table]

Original: since they only takes a tiny amount of FLOPs to the network.
Revised: since they only takes a tiny amount of FLOPs to the network.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Classiﬁcation results for all 40 labels on CelebA dataset .
Revised: Classiﬁcation results for all 40 labels on CelebA dataset .

Original: As shown in Fig.3, our searched network width have better performance in comparing to the uniform baseline, especially for small FLOPs budget and labels that are difﬁcult to be classiﬁed.
Revised: As shown in Fig.3, our searched network width has better performance in comparing to the uniform baseline, especially for small FLOPs budget and labels that are difﬁcult to be classiﬁed.

Original: [Figure]
Revised: [Figure]

Original: A. V ISUALIZATION OF SEARCHED NETWORK WIDTH
Revised: A. M ORE EXPERIMENTAL RESULTS WITH ALIGNED TRAINING RECIPES OF A UTO S LIM Y U & H UANG (2019)

Revised: To examine the performance of our searched network width [Table] we retrain the searched width of

Revised: MobileNetV2 and ResNet50 with aligned training recipes of AutoSlim Yu & Huang (2019).

Revised: In detail, we report the retraining results of width with similar FLOPs to AutoSlim in Table

Revised: [Table]

Revised: [Table]

Revised: As Table 14 shows

Revised: [Table]

Revised: with similar FLOPs and the same training recipes

Revised: [Table]

Revised: CafeNet can search the width with higher accuracy, which proves the effectiveness of our method.

Revised: A. V ISUALIZATION OF SEARCHED NETWORK WIDTH

Original: For intuitively understanding, we visualize our searched networks with ImageNet dataset, as shown in Fig.4.
Revised: For intuitively understanding, we visualize our searched networks with ImageNet dataset, as shown in Fig.4.

Original: For clarity, we show the retained ratio of network width from the original model.
Revised: For clarity, we show the retained ratio of network width from the original model.

Original: Note that, EfﬁcientNet-B0, ResNet50, and MobileNetV2 have SE block, skipping or depthwise layers, we merged these layers which are required to have the same network width for visual clarity.
Revised: Note that, EfﬁcientNet-B0, ResNet50, and MobileNetV2 have SE block, skipping, or depthwise layers; we merged these layers, which are required to have the same network width for visual clarity.

Original: For
Revised: For

Original: EfﬁcientNet-B0, we also did not show the channel of the SE block, because these layers are ﬁxed during the network width search.
Revised: EfﬁcientNet-B0, we also did not show the channel of the SE block because these layers are ﬁxed during the network width search.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] [Table]
Revised: [Figure] [Figure]

Original: [Table] [Figure] [Table]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [-385M
 M-]
Revised: [-385M
 M-]385M192M (

Original: [Figure]
Revised: [Figure]

Original: B0 on

Original: 217M105M50M
Revised: 217M105M50M

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Table] 
Revised: [Table] [Figure]

Original: [Figure]
Revised: [Figure]

Original: For ImageNet dataset, from Fig.4, we can see that layer width in skipping layers are more inclined to be preserved, which means skipping layers are very useful for classiﬁcation.
Revised: For ImageNet dataset, from Fig.4, we can see that layer width in skipping layers are more inclined to be preserved, which means skipping layers are very useful for classiﬁcation.

Original: Besides, with a large FLOPs budget, width in the 1 × 1 convolution layers shrink more than in 3 × 3 convolution layers, which means 1 × 1 convolution may contribute less to classiﬁcation performance.
Revised: Besides, with a large FLOPs budget, width in the 1 × 1 convolution layers shrink more than in 3 × 3 convolution layers, which means 1 × 1 convolution may contribute less to classiﬁcation performance.

Original: However, we observe an opposite phenomenon for width pruned from small FLOPs budgets, which implies the network is forced to use more 1 × 1 convolution layers to extract information from feature maps instead of 3 × 3 convolution layers.
Revised: However, we observe an opposite phenomenon for width pruned from small FLOPs budgets, which implies the network is forced to use more 1 × 1 convolution layers to extract information from feature maps instead of 3 × 3 convolution layers.

Original: A. V ISUALIZATION OF BINS
Revised: A. V ISUALIZATION OF BINS

Original: For intuitively understanding of FLOPs-sensitive bins, as shown in Fig.5, we visualize the number of channels contained in each bin (bin size) and the number of bins in each layer (bin number) with MobileNetV2 and VGGNet.
Revised: For intuitively understanding of FLOPs-sensitive bins, as shown in Fig.5, we visualize the number of channels contained in each bin (bin size) and the number of bins in each layer (bin number) with MobileNetV2 and VGGNet.

Original: Similar to the visualization of the searched network width, we merged the SE blocks, skipping, and depthwise layers for clarity.
Revised: Similar to the visualization of the searched network width, we merged the SE blocks, skipping, and depthwise layers for clarity.

Original: To make the bins from all layers have the similar contribution to FLOPs, the number of bins varies signiﬁcantly between layers.
Revised: To make the bins from all layers have a similar contribution to FLOPs, the number of bins varies signiﬁcantly between layers.

Original: For
Revised: For

Original: EfﬁcientNet-B0, the size of bin changes according to the different selected blocks ( i.e. , blocks with 3 × 3 or 5 × 5 convolution kernel size), i.e. ,  convolution of 3 × 3 or 5 × 5, and the size of bin becomes larger at the middle and the end of the network.
Revised: EfﬁcientNet-B0, the size of bin changes according to the different selected blocks ( i.e. , blocks with 3 × 3 or 5 × 5 convolution kernel size), i.e. , the convolution of 3 × 3 or 5 × 5, and the size of bin becomes larger at the middle and the end of the network.

Original: For MobileNetV2, since only 3 × 3 convolution is involved, the bin size changes more regularly, and the number of bins gradually increases from the front to the end of the network.
Revised: For MobileNetV2, since only 3 × 3 convolution is involved, the bin size changes more regularly, and the number of bins gradually increases from the front to the end of the network.

Original: [Figure]
Revised: [Figure]

Original: A. E FFECT OF MIN - MIN OPTIMIZATION IN E Q .(11)~(13)
Revised: A. E FFECT OF MIN - MIN OPTIMIZATION IN E Q .(11)~(13)

Original: To train the supernet, we investigate random optimization and min-min optimization with 50%FLOPs
Revised: To train the supernet, we investigate random optimization and min-min optimization with 50%

Original:  MobileNetV2 and ResNet50 on ImageNet dataset for instance.
Revised: FLOPs MobileNetV2 and ResNet50 on ImageNet dataset, for instance.

Original: In detail, based on the trained supernet, we use the evolution search to obtain the optimized network width and record 1000 network width for both random optimization and min-min optimization, and report the results by a statistical distribution histogram, as shown in Fig.6.
Revised: In detail, based on the trained supernet, we use the evolution search to obtain the optimized network width and recordnetwork width for both random optimization and min-min optimization, and report the results by a statistical distribution histogram, as shown in Fig.6.

Original: During sampling, the average performance of network width with our min-min optimization achieved 62.4% (52.7%) Top-1 accuracy with ResNet50 (MobileNetV2), which is 1.4% (1.3%) better than results from random optimization.
Revised: During sampling, the average performance of network width with our min-min optimization achieved 62.4% (52.7%) Top-1 accuracy with ResNet50 (MobileNetV2), which is 1.4% (1.3%) better than results from random optimization.

Original: In addition, width with the best performance is trained from scratch for evolution, as shown in Table 14.
Revised: In addition, width with the best performance is trained from scratch for evolution, as shown in Table 15.

Original: [Table]
Revised: [Table]

Original: In Table 14, we record the accuracy of Top-1 performance network width in supernet as Sampling, and than we training from scratch the width to report its Top-1 accuracy as CafeNet.
Revised: In Table 15, we record the accuracy of Top-1 performance network width in supernet as Sampling, and than we training from scratch the width to report its Top-1 accuracy as CafeNet.

Original: [Figure]
Revised: [Figure]

Original: A. E FFECT OF MULTI - STAGE SEARCH WITH EVOLVING BINS
Revised: A. E FFECT OF MULTI - STAGE SEARCH WITH EVOLVING BINS

Original: To examine the effect of multi-stage search with evolving bins, we report the Top-1 accuracy of
Revised: To examine the effect of multi-stage search with evolving bins, we report the Top-1 accuracy of

Original: MobileNetV2 on CIFAR-10 dataset with different training stages and the bin evolving speed α in
Revised: MobileNetV2 on CIFAR-10 dataset with different training stages and the bin evolving speed α in

Original: Section 3.4, the results are summarized in Table 15.
Revised: Section 3.4, the results are summarized in Table 16.

Original: In detail, for CIFAR-10 dataset, our algorithm achieves superior performance with α set to 2 and searching in 3 stages, which indicates bin evolving strategy can lead to better results comparing with searching in one stage.
Revised: In detail, for CIFAR-10 dataset, our algorithm achieves superior performance with α set to 2 and searching in 3 stages, which indicates bin evolving strategy can lead to better results comparing with searching in one stage.

Original: However, when α is increased to 4, the performance of the searching results decreases a bit, which may be due to the larger search space that limits the ability of evolutionary search.
Revised: However, when α is increased to 4, the performance of the searching results decreases a bit, which may be due to the larger search space that limits the ability of evolutionary search.

Original: [Table]
Revised: [Table]

Original: A. E STIMATION OF THE SIZE OF SEARCH SPACE IN MULTI - STAGE SEARCH
Revised: A. E STIMATION OF THE SIZE OF SEARCH SPACE IN MULTI - STAGE SEARCH

Original: Note that by leveraging the multi-stage searching strategy, we can further reduce the search space by following the searching "from coarse-grained to ﬁne-grained" with evolving bins "from large to small".
Revised: Note that by leveraging the multi-stage searching strategy, we can further reduce the search space by following the searching "from coarse-grained to ﬁne-grained" with evolving bins "from large to small".

Original: In this way, we directly investigate how the search space is reduced by the multi-stage searching strategy.
Revised: In this way, we directly investigate how the search space is reduced by the multi-stage searching strategy.

Original: We compare the search space of supernet under different training stages, bin size β , and bin
Revised: We compare the search space of supernet under different training stages, bin size β , and bin

Original: evolving speed α in Table 16.
Revised: evolving speed α in Table 17.

Original: To fairly compare the results with different training stages, we set the size of the bin unit to be the same in Table 16.
Revised: To fairly compare the results with different training stages, we set the size of the bin unit to be the same in Table 17.

Original: And the search space is deﬁned as the product of the number of bins in all layers
Revised: And the search space is deﬁned as the product of the number of bins in all layers.

Original: [Equation]
Revised: [Equation]

Original: Where Channel ji represents the total channels at layer i in stage j , and b ji indicates the bin size of layer i in stage j .
Revised: Where Channel ji represents the total channels at layer i in stage j , and b ji indicates the bin size of layer i in stage j .

Original: We examine the search space of 0.5 × FLOPs MobileNetV2 on ImageNet dataset, as shown in Table 16.
Revised: We examine the search space of 0.5 × FLOPs MobileNetV2 on ImageNet dataset, as shown in Table 17.

Original: [Table]
Revised: [Table]

Original: In detail, our search space reduced by more than 1 / (1 . 3 × 10 − 8 ) times from the original size (with one stage) by searching more than or equal to 2 stages.
Revised: In detail, our search space reduced by more than 1 / (1 . 3 × 10 − 8 ) times from the original size (with one stage) by searching more than or equal to 2 stages.

Original: As a result, our proposed multi-stages search method can effectively reduce the search space.
Revised: As a result, our proposed multi-stages search method can effectively reduce the search space.

Original: A. E FFECT OF SMALLEST BIN SIZE β IN FLOP S - SENSITIVE BINS
Revised: A. E FFECT OF SMALLEST BIN SIZE β IN FLOP S - SENSITIVE BINS

Original: As formulated in Eq.(9)
Revised: As formulated in Eq.(9)

Original: [Table]
Revised: [Table]

Original: the total number of bins within each layer relies on the smallest bin size β .
Revised: the total number of bins within each layer relies on the smallest bin size β .

Original: To examine the effect of β w.r.t.
Revised: To examine the effect of β w.r.t.

Original: bin numbers and its corresponding performance
Revised: bin numbers and its corresponding performance

Original: [Table]
Revised: [Table]

Original: we perform experiments with MobileNetV2 on ImageNet dataset under 50% FLOPs budget, as shown in Table 17.
Revised: we perform experiments with MobileNetV2 on ImageNet dataset under 50% FLOPs budget, as shown in Table 18.

Original: In detail, the performance of searched network width ﬁrst increases when β goes large but a little decreased after β surpasses 0.5, which may result from that a very small β induces a too huge search space thus is harmful to searching algorithms to ﬁnd the optimized network width
Revised: In detail, the performance of searched network width ﬁrst increases when β goes large but a little decreased after β surpasses 0.5, which may result from that a very small β induces a too huge search space thus is harmful to searching algorithms to ﬁnd the optimized network width

Original: [Table] [Table]
Revised: [Table] [Table]