Paper:
We study how to set the number of channels in a neural network to achieve better accuracy under constrained resources (e.g., FLOPs, latency, memory footprint or model size). A simple and one-shot approach, named AutoSlim, is presented. Instead of training many network samples and searching with reinforcement learning, we train a single slimmable network to approximate the network accuracy of different channel configurations. We then iteratively evaluate the trained slimmable model and greedily slim the layer with minimal accuracy drop. By this single pass, we can obtain the optimized channel configurations under different resource constraints. We present experiments with MobileNet v1, MobileNet v2, ResNet-50 and RL-searched MNasNet on ImageNet classification. We show significant improvements over their default channel configurations. We also achieve better accuracy than recent channel pruning methods and neural architecture search methods with 100× lower search cost. Notably, by setting optimized channel numbers, our AutoSlim-MobileNet-v2 at 305M FLOPs achieves 74.2% top-1 accuracy, 2.4% better than default MobileNet-v2 (301M FLOPs), and even 0.2% better than RL-searched MNasNet (317M FLOPs). Our AutoSlim-ResNet-50 at 570M FLOPs, without depthwise convolutions, achieves 1.3% better accuracy than MobileNet-v1 (569M FLOPs).

Review:
In this paper, the authors propose a method to perform architecture search on the number of channels in convolutional layers. The proposed method, called AutoSlim, is a one-shot approach based on previous work of Slimmable Networks [2,3]. The authors have tested the proposed methods on a variety of architectures on ImageNet dataset. 
The paper is well-written and easy to follow. I really appreciate the authors for structuring this paper so well. I have the following questions:
Q1: In figure 4, the authors find that “Compared with default MobileNet v2, our optimized configuration has fewer channels in shallow layers and more channels in deep ones.” This is interesting. Because in network pruning methods, it is found that usually later stages get pruned more [1] (e.g. VGG), indicating that there is more redundancy for deep layers. However, in this case, actually deep layers get more channels than standard models. Is there any justification for this? Is it that more channels in deep layers benefit the accuracy?
Q2: In “Training Optimized Networks”, the authors mentioned that “By default we search for the network FLOPs at approximately 200M, 300M and 500M, and train a slimmable model.” Does this mean that the authors train the final optimized models from scratch as a slimmable network using “sandwich rule” and “in-place distillation” rule? Or are the authors just training the final model with standard training schedule? If it is the first case, can the authors justify why?
Q3: In Table 1, “Heavy Models”, what is the difference between “ResNet-50” and “He-ResNet-50”? Also, why the params, memory and CPU Latency of some networks are omitted?
Q4: In the last paragraph of section 4, the authors tried the transferability of networks learned from ImageNet to CIFAR-10 dataset. I am not sure how the authors transfer the networks from Imagenet to CIFAR-10? Is it the ratio of the number of channels? Can the authors provide the architecture details of MobileNet v2 on CIFAR-10 dataset?
Q5: What is the estimated time for a typical run of AutoSlim? How does it compare to network pruning methods or neural architecture search methods?
Q6: Can the methods be used to search for the number of neurons in fully connected layers? Are there any results?
[1] Rethinking the Value of Network Pruning. Zhuang et al. ICLR 2019
[2] Slimmable neural networks. Yu et al. ICLR 2019.
[3] Universally Slimmable Networks and Improved Training Techniques. Yu et al. Arxiv.