Paper:
Automatic neural network discovery methods face an enormous challenge caused by the size of the search space. A common practice is to split this space at different levels and to explore only a fraction of it. On one hand, neural architecture search methods look at how to combine a subset of layers to create an architecture while keeping a predefined number of filters in each layer. On the other hand, pruning techniques take a well known architecture and look for the appropriate number of filters per layer. In both cases, the exploration is made iteratively, training models several times during the search. Inspired by the constraints and advantages of these two approaches, we propose a straight-forward and fast option to find models with improved characteristics. We apply a small set of templates, that have been heuristically and experimentally evaluated, to make a one-shot redistribution of the number of filters in an already existing neural network. When compared to the initial base models we found that the resulting architectures, when trained from scratch, surpass the original accuracy even after been reduced to fit to the original amount of resources. Specifically, we show accuracy improvement for some network-task pairs of up to 5.5%, a reduction of up to 45% in parameters and 60% reduction in memory footprint.

Review:
This paper presents a simple methodological study on the effect of the distribution of  convolutional filters on the accuracy of deep convolutional networks on the CIFAR 10 and CIFAR 100 data sets. There are five different kind of distributions studied: constant number of filters, monotonically increasing and decreasing number of filters and convex/concave with a local extremum at the layer in the middle.  For these distributions, the total number of filters is varied to study the trade-off between running-time vs. accuracy, memory vs. accuracy and parameter count vs. accuracy.
Although the paper is purely experimental without any particular theoretical considerations, it presents a few surprising observations defying conventional wisdom:
- The standard method of increasing the number of filters as the number of convolutional nodes is increasing is not the most optimal strategy in most cases.
- The optimal distribution of channels is highly dependent on the network architecture.
- Some network architectures are highly stable with respect to the distribution of channels, while others are very sensitive.
Given that this paper is easy to read and presents interesting insights for the design of convolutional network architectures and challenges mainstream views, I would consider it to be a generally valuable contribution, at least I enjoyed reading it.
Despite the intriguing nature of this paper, there are several weaknesses which make me less enthusiastic about the quality of the paper:
- The experiments are done only on CIFAR-10 and CIFAR-100. These benchmarks are somewhat special. It would be useful to see whether the results also hold for more realistic vision benchmarks. Even if running all the experiments would be costly, I think that at least a small selection should be reproduced on OpenImages or MS-Coco or other more realistic benchmarks to validate the findings of this paper.
- It would be interesting to see whether starting from the best channel distributions, applying MorphNet would end up with different distributions. In general: whether MorphNet would end up with similar distributions automatically.
-  The paper does not clarify how the channel sizes for Inception were distributed, since proper balancing of the 1x1 and more spread out convolutions is a key part of that architecture. This is not clarified in this paper.
- The grammar of the paper is poor, even the abstract is hard to read and interpret.
- The paper presents itself as a methodology for automatically generating the optimal number of channels, while it is more of a one-off experiment and observation than a general purpose method.
Another small technical detail regarding the choice of colors in the diagrams: the baseline distribution and constant distribution are very hard to distinguish. This is especially critical because these are the two best distributions on average. Also the diagrams could benefit from more detailed captions.
The paper presents interesting, valuable experimental findings, but it is not extremely exciting theoretically. Also its practical execution is somewhat lacking. If it contained at least partial results on more realistic data sets, I would vote for strong accept, but in its current form, I find it borderline acceptance-worthy.