From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper58/Authors
comment: We thank the reviewers for reading our paper, and for the thoughtful comments.  See some responses to questions below.

“The modified network is again trained until convergence”
The network is not trained till convergence but trained only for an epoch before checking the difference in error to add more neurons.

“Because these networks have only one layer, it is not immediately clear to me than ANR would be more efficient than a grid search. ANR has two parameters gamma and dM, whereas in this case the grid search would only have one. Can the authors comment on this?”
Since these were smaller datasets, dM was set at 1 for all regression experiments, and for the regression datasets, grid search was performed only on gamma.  We will clarify in the text.  In general, though, we agree that larger models demonstrate the benefits of our approach better.

“Are the listed errors standard deviations?”
Yes.

“How many data points were collected for each experiment?”
5

Regarding questions on grid search:
Grid search on model architecture was performed with [512,1024,2048] units in each layer with 8 combinations for the total three layered network and for convolutional [24,48,96] filters were used.  While performing a grid search on model architecture, only discrete quantities of neurons in each layer can be tried and since validation accuracy was used to find the best model architecture 2048-2048-2048 was chosen during grid search.

“My impression from the text was that only the new neuron weights are trained during ANR, i.e. old weights are frozen; is this correct? Can the wording be clarified?”
The entire network is trained during ANR.  We will clarify in the text.

“Alg 1, line 3 implies all weights are re-initialized at every epoch, no?”
Line 3 indicates weight updates.

“I think this pseudocode does not quite match the algorithm decribed in the text. For instance, in Alg 1, new neurons are added after checking if the error improved; is this not backwards?”
Neurons are added first, the network is trained, and then and the error difference is computed against gamma to decide whether additional neurons are kept.  We will improve the presentation of Algorithm 1.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper58/Authors
comment: We thank the reviewers for reading our paper, and for the thoughtful comments.  See some responses to questions below.

The objective function (1) penalizes each hidden unit in the network by gamma.  If we add a new unit to the network, we know that the objective function decreases if the change in the loss after adding the new unit is greater than gamma.  Thus, our algorithm adds new units only if the change is greater than gamma.

We agree that it would be interesting to experiment with larger networks of 15+ layers, to see how effective our method is for large models.  We view this as important future work; in our preliminary study we have only tested the method on smaller networks.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper58/AnonReviewer6
title: An interesting exploration with issues to be justified & clarified.
review: 
The idea of augmenting the loss function with a penalty on the number of units is sound and has long been proposed before.  Taking a training strategy that grows the number of units, rather pruning from a large-sized networks, seems interesting. However, there are several questions to be clarified.  

First of all, the decision of adding new units is made when the change in the loss function is greater than gamma. It is unclear to me how this decision is related to the objective function (1).  

To optimize network architectures e.g. the set of nodes, is essentially a difficult combinatorial problem.  This paper proposed to reduce the complexity, by choosing uniformly at random a single layer at a time when adding new nodes.  This approach seems too much limited, in terms of the search space it explored.  That probably also explains the poor performances of CIFAR-10 experiments reported in Table 2 (left), which are all much worse than the state-of-the-art.  It would be more convincing to demonstrate the state-of-the-art performance, for the proposed growing strategy applied to deep networks e.g. with 16 or 19 layers. 

Minor: 
The acronym SVA should be mentioned in Line 27, before referring to it. 

rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper58/AnonReviewer2
title: The proposed algorithm, Adapative Network Regularization, appears to have significant advantages over grid search for finding compact deep neural network architectures that preserve good performance. The advantage for shallow networks is less clear to me.
review: The authors propose adaptive network regularization (ANR), an algorithm for learning network architectures during training. First, a relatively small network is trained to convergence. Then, in each iteration, a trial change is made to the network architecture: dM new neurons are added to one of the layers. The modified network is again trained until convergence; the trial change is accepted only if the new performance exceeds the old one by more than a value gamma. ANR converts architecture optimization to a grid search over two hyperparameters, gamma and dM, rather than a grid search over all the layers' widths. The authors provide numerical experiments to support two main claims: that ANR can produce compact networks with good performance more efficiently than grid search; and that training via ANR is better than training a fixed architecture from scratch, even when the fixed architecture is that produced by ANR. These are very exciting results. Although I am not convinced ANR is superior to grid search for shallow networks, the advantages of ANR are more clear in deeper networks.

The first 7 experiments consist of using three protocols to train shallow networks on regression tasks previously considered by Hernández-Lobato and Adam. The control used fixed un-optimized architectures. The ANR group was initialized with five neurons. The third protocol (which I will call retraining) copied the final architectures produced by ANR, but retrained from scratch.

Because these networks have only one layer, it is not immediately clear to me than ANR would be more efficient than a grid search. ANR has two parameters gamma and dM, whereas in this case the grid search would only have one. Can the authors comment on this?

The authors state that ANR performs comparably to control and better than the retraining group. However, I am not convinced that the differences between the three groups in Table 1 are statistically significant. Are the listed errors standard deviations? How many data points were collected for each experiment? Were these data normally distributed? In some cases, it appears they were not (see below). Throughout Table 1, I think the max and min performance for each group might provide important insights into the relative performances of the three protocols. To discuss these results, I have computed the errors on the differences between groups (which I will call sigma) by adding the respective errors in quadrature. Except where noted below, I believe the results for the three protocols are within 2 sigma of one another.

In exp 1, ANR performs 5 sigma worse than control, but retraining performed only 2.5 sigma worse than ANR. 
In exp 3, the ANR loss is four times worse than the control's. The result for the ANR group is listed as 0.004+/-0.0000; is the error truly zero? Conversely, the result for the retraining group is listed as 0.004+/-0.004; doesn't this imply that the retraining group performed better than the ANR group is most cases? 
In exp 4, the retraining result is worse than ANR to a statistically significant amount (~3.3 sigma), but the two results only differ by less than 1% of their mean values. 
In exp 6, the result for the retraining group is listed as 0.671+/-0.4421. Doesn't this imply that the data are far from normally distributed? 
In exp 7, no errors are listed, but the errors in the three groups differ by less than 1% of their mean values.

The remaining experiments were conducted with three-layer fully-connected and convolutional networks on image classification tasks; here, the advantage of ANR over grid search is more clear. For these, the control architectures were optimized by grid search. In Table 2, ANR is comparable to control and, in one case, 10x more compact. Retraining is only slightly worse than the other groups, though. How was the grid search performed? Can the authors comment on why the grid search didn't find something closer to the ANR result in exp 9? The results in Table 3 are most convincing: the ANR solution is over twice as compact as control but equally precise, and the retraining loss is ~3.6x larger than both.

Other notes:
- My impression from the text was that only the new neuron weights are trained during ANR, i.e. old weights are frozen; is this correct? Can the wording be clarified?
- The ANR only trials increases in layer widths. However, the effective objective (1) can also possibly be improved (by an amount greater than gamma) by decreasing excessively large layers. Can the authors comment on trialing decreases in layer widths?

Notes on Alg 1:
- E is used for both epochs and errors; U is used for layer widths and for the uniform distribution.
- Alg 1, line 3 implies all weights are re-initialized at every epoch, no?
- I think this pseudocode does not quite match the algorithm decribed in the text. For instance, in Alg 1, new neurons are added after checking if the error improved; is this not backwards?

rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct