Paper:
We present a method that trains large capacity neural networks with significantly improved accuracy and lower dynamic computational cost. We achieve this by gating the deep-learning architecture on a fine-grained-level. Individual convolutional maps are turned on/off conditionally on features in the network. To achieve this, we introduce a new residual block architecture that gates convolutional channels in a fine-grained manner. We also introduce a generally applicable tool batch-shaping that matches the marginal aggregate posteriors of features in a neural network to a pre-specified prior distribution. We use this novel technique to force gates to be more conditional on the data. We present results on CIFAR-10 and ImageNet datasets for image classification, and Cityscapes for semantic segmentation. Our results show that our method can slim down large architectures conditionally, such that the average computational cost on the data is on par with a smaller architecture, but with higher accuracy. In particular, on ImageNet, our ResNet50 and ResNet34 gated networks obtain 74.60% and 72.55% top-1 accuracy compared to the 69.76% accuracy of the baseline ResNet18 model, for similar complexity. We also show that the resulting networks automatically learn to use more features for difficult examples and fewer features for simple examples.

Review:
Summary: This paper studies conditional channel gated networks. The network is designed to disable certain channels depending on the inputs. This can be used to save computation. The idea is built on top of “Convolutional Networks with Adaptive Inference Graphs.” The authors propose the technique of batch shaping to encourage the marginal statistics of the gating to be selective to different inputs. With similar inference time, the gated network can achieve better accuracy it can afford adding more layers in the network.
Detailed comments:
- The results show quite significance difference compared to ConvNet-AIG, which demonstrates that batch-shaping is very helpful. Table-1 show 3% increase in accuracy compared ResNet-18 by using similar inference time. It is good that the paper reports wall-clock time measurements.
- It’s good to see some visualizations in the paper, including the image samples and gate locations. I recommend to move Figure 7 to the main paper. A regular neural network can also be used to visualize the sensitivity of patterns of specific neurons. What would be the qualitative differences?
- 1-2x reduction in MAC is not super impressive, especially taking into consideration of the overhead for gathering the active channels for convolution. 
- Figure 3a) plot is cut off on the right. The baselines only have a single point in the plot, I guess it is also valid to simply add/remove layers in the baseline models to generate a curve in the plot.
- ResNet-50-L0 is missing in Figure 3b). It would be better if the plots can be grouped better. Currently there are too many lines and it is hard to understand the differences.
- It would be good to see comparisons to some other alternatives to batch shaping. For example, one can penalize so that the average value is around 0.5 by using a L1 loss |E(x) – 0.5|.
- The ImageNet experiment has a very complicated set-up, where L0 loss is applied in the middle of the training. Is this necessary? How important is this step? What would happen if L0 loss is not applied in ImageNet? And what would happen if L0 loss is applied from the beginning? Why is L0 loss not applied in other experiments (e.g. CIFAR or Cityscapes), will L0 loss be beneficial on these benchmarks as well?
- There are a number of related works on adaptive spatial attention for faster inference, which can be included in the related work section.
1) M.  Figurnov,  M.  D.  Collins,  Y.  Zhu,  L.  Zhang,  J.  Huang,D.  P.  Vetrov,  and  R.  Salakhutdinov. Spatially  adaptive computation  time  for  residual  networks. CVPR, 2017.
2) X. Li, Z. Liu, P. Luo, C. C. Loy, and X. Tang.  Not all pixelsare equal:  Difficulty-aware semantic segmentation via deeplayer cascade. CVPR, 2017.
3) M. Ren, A. Pokrovsky, B. Yang, R. Urtasun. SBNet: Sparse Blocks Network for Fast Inference. CVPR, 2018.
Conclusion: The batch shaping technique introduced in this paper has significant improvement on networks that exploit conditional inference. Further understanding of the effect of L0 loss and other alternative loss function is recommended. My overall rating is weak accept.