Paper: Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimally (ICLR-2024)

Reviewer comment (Novelty):
How does the approach compare to NEAT based techniques? What are differences / communalities?

Author response:
Both approaches have the similar goals: the optimization of the architecture of a neural network, for a given machine learning task. However they proceed very differently. NEAT is based on evolutionary techniques, with a mix of exploration (random architecture changes), selection and cross-over. On the opposite we are driven by the gradient of the task loss, which enables us to directly know which part of the architecture we should change, and how (i.e. where to add neurons, and which ones). While NEAT needs to perform many random tries (and retrain each generated architecture) to find good moves, we exploit the information from the backpropagation to directly apply the optimal move. However our moves are optimal in a first order derivative meaning, and computing these moves is costly, while NEAT estimates the real impact of each tested move (but this comes at a computational cost as well, and arguably a higher one). We expect that in a near future, neural architecture optimization techniques involving both types of approaches will arise.

Reviewer comment (Experiments):
How well would your approach work on datasets with lower signal to noise ratio compared to CIFAR-10? Would you expect to see overfitting?

Author response:
Definitely, yes, if the user forgot to add a regularizer to the loss to optimize. We suppose that the user provides a loss that they actually want to optimize. This is not the case though in current practice of neural networks, as architectures are selected for known good training biases, and as batteries of indirect regularizers are used (early-stopping, drop-out, batch-norm, data augmentation, etc.). In case the user forgot to add a regularizer to the loss, we could still use these indirect regularizers to fight against overfit (except for early-stopping). We also have other ways to detect of prevent overfitting, as we can estimate the statistical significance of the candidate neurons to add. If the eigenvalue associated to the candidate neuron is too low, the performance gain provided by that neuron is likely to be due to spurious correlations, so we skip it. We detail this in Appendices D.1 and D.3. We actually plan to work further on the design of statistical tests.

Reviewer comment (Experiments):
Did you consider combination of standard gradient descent and your proposed method? How would they work out?

Author response:
Yes, this is what we do in practice, for instance with the new experiment on CIFAR-100: after each neuron addition we classically train the network for a certain time. This dramatically improves the results, compared to just adding neurons. However we noticed with a previous implementation that in some cases this would come with unstabilities due to the fact that different parts of the network might need different learning rates. These unstabilities have been tempered by better distributing the amplitude factor on added neurons: we now consider $(\sqrt{\varepsilon} \alpha, \sqrt{\varepsilon} \omega)$ instead of $(\alpha, \varepsilon \omega)$, which makes future gradients more homogeneous.

Reviewer comment (Theory):
Proposition 3.1 (Appendix): What happens in the step where after "$M^+$, we get:"? To me it looks like you substituted $\delta W_l$ with $\delta W_l^*$ in $V_\text{goal}^l B_{l-1}^T = \delta W_l B_{l-1} B_{l-1}^T$, as this is where the gradient of $g(\delta W_l)$ vanishes, and then multiplied $\tfrac{1}{n} (B_{l-1} B_{l-1}^T)^+$ from the right. However, I am missing the reasoning why $B_{l-1} B_{l-1}^T \tfrac{1}{n} (B_{l-1} B_{l-1}^T)^+ = I$.

Author response:
Indeed $\frac{1}{n} B_{l-1} B_{l-1}^T (\frac{1}{n} B_{l-1} B_{l-1}^T)^+$ is not the identity matrix, but a projection on a subspace, whose dimension is the rank $r$ of the matrix $\frac{1}{n} B_{l-1} B_{l-1}^T$. Vectors in the image of $\frac{1}{n} B_{l-1} B_{l-1}^T$ are kept identical while the other ones are sent to 0.  

Step by step:  
$$v_{goal}^{l}B_{l-1}^T = \mM B_{l-1}     B_{l-1}^T $$  
$$\frac{1}{n} v_{goal}^{l}B_{l-1}^T = \mM \left( \frac{1}{n} B_{l-1}     B_{l-1}^T \right)$$  
$$\frac{1}{n}v_{goal}^{l}B_{l-1}^T (\frac{1}{n}B_{l-1} B_{l-1}^T)^+ = \mM \left( \frac{1}{n} B_{l-1}     B_{l-1}^T \right)\left( \frac{1}{n} B_{l-1}     B_{l-1}^T \right)^+$$  

If $\left( \frac{1}{n} B_{l-1}     B_{l-1}^T \right)$ is full rank, then the projector $\left( \frac{1}{n} B_{l-1}     B_{l-1}^T \right)\left( \frac{1}{n} B_{l-1}     B_{l-1}^T \right)^+$ is the identity, which yields the solution.  

If it is not full rank, one can chose $\mM$ in the image of $\left( \frac{1}{n} B_{l-1}     B_{l-1}^T \right)$, and then we obtain the same formula (as the projection of $\mM$ is itself). Other solutions exist, by adding terms that are in the kernel of $\left( \frac{1}{n} B_{l-1}     B_{l-1}^T \right)$, i.e. whose image by the projection will be 0. The standard least square approach does not add such (useless) terms.

Reviewer comment (Theory):
Proposition 3.2: As $S$ is just positive semi-definite ($S := \tfrac{1}{n} B_{l-2} B_{l-2}^T$) and not necessarily positive definite (i.e. may not have full rank), how do we know that $S^{-1/2}$ exists in Proposition 3.2?

Author response:
In Proposition $3.2$, the matrix $S^{-1/2} := U\sqrt{\Sigma}^{-1}U$ is defined with the convention that  $0^{-1} = 0$ when taking the inverse of the diagonal of $\sqrt{\Sigma}$. We added that information to the Proposition, that we indeed had forgotten to specify.

Reviewer comment (Theory):
Proposition 3.2: Of which matrix are $\lambda_k$ the Eigenvalues? Currently I see that they are the singular values of $S^{-1/2} N$.

Author response:
The $\lambda_k$ are the singular values of $S^{-1/2}N$ and $\lambda_k^2$ are the eigenvalues of $S^{-1/2}NN^TS^{-1/2}$ and $NS^{-1}N^T$. We have updated the article by renaming everywhere the $\lambda_k$ "singular values", to avoid confusion.

Reviewer comment (Experiments):
Does your method scale to ImageNet?

Author response:
We have performed a complexity analysis in the common reply ["Computational complexity"](https://openreview.net/forum?id=Qp33jnRKda&noteId=wMiGwiU7Bs) that we will also add as an appendix. When considering large images and a convolutional network trained by classical gradient descent, the additional cost of adding neurons with our method is actually small compared to the one of the gradient descent, so, yes, our method would scale to large images.  

Regarding scaling to larger architectures, as a proof of concept we have trained a ResNet-18 on CIFAR-100, which we are adding at the end of the appendix.

Reviewer comment (Experiments):
Did you compare the three different initialization approaches? Random initialization, zero initialization and your in Proposition 3.2 proposed initialization?

Author response:
Definitely. For 0 initialization, one cannot set both incoming and output weights $\alpha$ and $\omega$ of the new neuron to 0, otherwise gradient descent will never update them (two 0 in a row make gradient 0). If one sets only $\omega$ (or only $\alpha$) to 0, then the network will be able to learn.  

Now, if one adds the new neuron with small weights, whether they are random or well-chosen, the difference in training will be small, as all initializations with small weights are close to each other in the space of weights.  

However if one performs line-search to multiply the neuron by potentially a big factor, the difference is dramatic, and our approach performs much better. Indeed, from the theory of random projections, one can expect any random optimization direction to be able to improve the loss, but only marginally. The best improvements will be realized for directions that align with the loss landscape, i.e., at first order, the gradient ("optimal move") that we compute.

Reviewer comment (Reproducibility):
Which hardware did you use to run your experiments?

Author response:
We usually use a standard GPU card (such as RTX2080), but some experiments are run on CPU only, when willing to estimate the full computational cost in practice.

Reviewer comment (Experiments):
A similar criticism also holds for the experiment section, which only compares with GradMax and not different types of approaches.

Author response:
We are still adding an appendix about experiments (we will submit a new version soon) but the main paper itself is already polished and in its final state. We ran supplementary experiments only because reviewers asked us to. We managed to run ResNet-18 on CIFAR-100, got a decent accuracy given the context (no drop-out, etc.) and compare very favorably to GradMax on all aspects (twice smaller network for same given accuracy, or 10 points of accuracy more for a same given size, while training faster). We hope this will be considered as sufficient validation for a theoretical paper.

Reviewer comment (Reproducibility):
It is impossible to deduce from Section 3 what the actual algorithmic proposal is. The links in the algorithm to the supplement are broken. (The actual description is on page 15.) The actual update equations are not provided in the main paper.

Author response:
> [About presentation]

As advised, we moved the Related Works paragraphs from Section 3 to the rest of the state of the art in the main introduction. We have put the appendix back to the end of the main paper, for links to work again. We have added a high-level description of the algorithm in the main paper, and we are adding the description of the sub-routines in the Appendix, with detailed complexity.

Reviewer comment (Experiments):
It appears that the number of neutrons that are added in each step is a hyper-parameter.

Author response:
> Limitation: It appears that the number of neutrons that are added in each step is a hyper-parameter.

As we discuss it in the common reply ["Neuron addition strategy"](https://openreview.net/forum?id=Qp33jnRKda&noteId=aJJH3Fi8jC), one can conceive neuron addition strategies that do really get rid of such hyper-parameters. In the paper we chose on purpose a very simple experimental setting (with fixed number of neuron additions) just to compare the quality of neurons added when using our approach or GradMax.

Reviewer comment (Experiments):
The update seems to involve a spectral decomposition to avoid neuron redundancy that is computationally costly.

Author response:
> What is the runtime complexity of an update step?

In practice, running our approach takes about the same time as training from scratch with standard gradient descent the final architecture we find.

We detail the theoretical complexity in the common reply ["Computational complexity"](https://openreview.net/forum?id=Qp33jnRKda&noteId=wMiGwiU7Bs). In particular, we show that the computational cost of the SVD is neglectible relatively to standard gradient descent. This is because the matrices we apply SVD to are very small compared to other computations done to run the network (their size is the number of channels in a layer, and not  minibatch size, neither image size).

Reviewer comment (Evaluation):
The computational complexity of the full training process (including the network growth) has not been evaluated, even though it forms an integral part of the claimed contributions.

Author response:
> What is the runtime complexity of an update step?

In practice, running our approach takes about the same time as training from scratch with standard gradient descent the final architecture we find.

We detail the theoretical complexity in the common reply ["Computational complexity"](https://openreview.net/forum?id=Qp33jnRKda&noteId=wMiGwiU7Bs). In particular, we show that the computational cost of the SVD is neglectible relatively to standard gradient descent. This is because the matrices we apply SVD to are very small compared to other computations done to run the network (their size is the number of channels in a layer, and not  minibatch size, neither image size).

Reviewer comment (Experiments):
The method is not very flexible in adding layers or different kinds of modules. It only grows the width of a chosen architecture.

Author response:
> The method is not very flexible in adding layers or different kind of modules. It only grows the width of a chosen architecture.

Our approach naturally extends to the addition of new layers (or nodes) in general computational graphs (any differentiable DAG), as it turns out that layer creation is the same as adding neurons to an empty layer. We discuss it briefly in the last paragraph of ["About experiments and architectures"](https://openreview.net/forum?id=Qp33jnRKda&noteId=bimw8pDivO).

Reviewer comment (Novelty):
The novelty of the method appears to be limited in comparison with GradMax.

Author response:
We ran supplementary experiments only because reviewers asked us to. We managed to run ResNet-18 on CIFAR-100, got a decent accuracy given the context (no drop-out, etc.) and compare very favorably to GradMax on all aspects (twice smaller network for same given accuracy, or 10 points of accuracy more for a same given size, while training faster).

Reviewer comment (Experiments):
Experiments are limited to CIFAR-10, a relatively small dataset of low complexity. (Note that GradMax was evaluated also on CIFAR-100 and ImageNet.)

Author response:
> How does the proposed method perform on CIFAR100 and ImageNet?

We explain in the common reply ["About experiments and architectures"](https://openreview.net/forum?id=Qp33jnRKda&noteId=bimw8pDivO) why we have not performed such tests, and that we are currently running them (on CIFAR-100 with a ResNet-style architecture), to show that our approach can scale to larger architectures and larger datasets. However we do not hope for competitive accuracy with the state-of-the-art as our current implementation does not include yet batch-norm, drop-out or data augmentation, which are needed to reach good scores.

Reviewer comment (Presentation):
Broken figure link in Section 3 on page 4.

Author response:
> [About presentation]

As advised, we moved the Related Works paragraphs from Section 3 to the rest of the state of the art in the main introduction. We have put the appendix back to the end of the main paper, for links to work again. We have added a high-level description of the algorithm in the main paper, and we are adding the description of the sub-routines in the Appendix, with detailed complexity.

Reviewer comment (Presentation):
The supplement is not included in the main paper so that important links are broken (see algorithm, for instance).

Author response:
> [About presentation]

As advised, we moved the Related Works paragraphs from Section 3 to the rest of the state of the art in the main introduction. We have put the appendix back to the end of the main paper, for links to work again. We have added a high-level description of the algorithm in the main paper, and we are adding the description of the sub-routines in the Appendix, with detailed complexity.

Reviewer comment (Experiments):
How does the proposed method perform on CIFAR100 and ImageNet?

Author response:
> How does the proposed method perform on CIFAR100 and ImageNet?

We explain in the common reply ["About experiments and architectures"](https://openreview.net/forum?id=Qp33jnRKda&noteId=bimw8pDivO) why we have not performed such tests, and that we are currently running them (on CIFAR-100 with a ResNet-style architecture), to show that our approach can scale to larger architectures and larger datasets. However we do not hope for competitive accuracy with the state-of-the-art as our current implementation does not include yet batch-norm, drop-out or data augmentation, which are needed to reach good scores.

Reviewer comment (Theory):
How is the matrix N on page 15 computed, since it depends on the (unknown?) $V_{goal}$?

Author response:
> How is the matrix N on page 15 computed, since it depends on the (unknown?) $V_{goal}$?

$V_{goal}$ is actually known and easy to compute. We have rewritten the paper to make this more explicit (beginning of Section 2.2). The quantity $v_{goal}(x)$ is actually what the first step of the usual backpropagation computes, when the network has been shown sample $x$. So we get this quantity for free. We have added a detailed explanation in Appendix A.1 to show this property.

Reviewer comment (Theory):
What is the runtime complexity of an update step?

Author response:
> What is the runtime complexity of an update step?

In practice, running our approach takes about the same time as training from scratch with standard gradient descent the final architecture we find.

We detail the theoretical complexity in the common reply ["Computational complexity"](https://openreview.net/forum?id=Qp33jnRKda&noteId=wMiGwiU7Bs). In particular, we show that the computational cost of the SVD is neglectible relatively to standard gradient descent. This is because the matrices we apply SVD to are very small compared to other computations done to run the network (their size is the number of channels in a layer, and not  minibatch size, neither image size).

Reviewer comment (Experiments):
How do the computational requirements compare with training just a wider model from scratch once?

Author response:
> How do the computational requirements compare with training just a wider model from scratch once?

This comparison is performed in Appendix E.1.2 COMPARISON WITH BIG MODEL. The left figure shows the accuracy as a function of time, while middle figure plots the accuracy as a function of gradient step. We easily see a persistent gap between the two curves on the left plot. While we achieve convergence around 90% accuracy on the test after only 15 seconds, the large models do not reach this performance even after 250 seconds of training.

Reviewer comment (Writing):
The paper is not easy to follow and there are a lot of typos in the paper, i.e., missing figure number in section 3, no caption for the algorithm.

Author response:
Thank you for your review and questions.

### About presentation
We are very sorry for the typos, we hope we have corrected all of them in the revision, which we introduce in the common reply ["New version of the paper, with improved clarity"](https://openreview.net/forum?id=Qp33jnRKda&noteId=7ZWDNGZNl9). We have added a description of the main algorithm and are adding details of subroutines in an appendix. More generally we have reworked the paper to make it more clear and easier to follow.

Reviewer comment (Reproducibility):
The paper should have a main algorithm that describes the whole training process, including how function gradients are calculated and how the optimization problem is solved according to which proposition.

Author response:
Thank you for your review and questions.

### About presentation
We have added a description of the main algorithm and are adding details of subroutines in an appendix. Calculating function gradients is actually very easy, it is part of what the standard backpropagation computes. We have added details in Section 2 and in Appendix A.1 to show this theoretically. We will also release the code (open-source). Technically one just has to call usual backpropagation and intercept an auxiliary computation, for instance with a ["hook"](https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html) in the desired layer.

Reviewer comment (Reproducibility):
How do we add neurons to the convolutional layers? Are we structurally adding kernels or adding neurons treating them as fully connected layers?

Author response:
> How do we add neurons to the convolutional layers? Are we structurally adding kernels or adding neurons treating them as fully connected layers?

Adding one neuron to a convolutional layer $l$ consists in adding one kernel to layer 𝑙 and consequently also one dimension to each kernel at layer $l+1$. We added a scheme, in dedicated Appendix A.6, to make this more clear.

Reviewer comment (Evaluation):
What is the computational cost of the proposed methods?

Author response:
> What is the computational cost of the proposed methods?

We have addressed computational complexity in the common reply ["Computational complexity"](https://openreview.net/forum?id=Qp33jnRKda&noteId=wMiGwiU7Bs).

Reviewer comment (Writing):
The presentation of the functional analysis viewpoint is hard to follow, probably due to notation overload.

Author response:
We apologize for the notation overload. We have corrected this, notably by distinguishing the sample-loss $\ell$ from the global loss $\mathcal{L}$, and by correcting a few typos. 
As stated in the common reply ["New version of the paper, with improved clarity"](https://openreview.net/forum?id=Qp33jnRKda&noteId=7ZWDNGZNl9), we have added Appendix A.1 to get the paper self-contained and to explain what precisely is the functional gradient, as well as how to obtain the property $${v_{goal}(x)} = -\left( \nabla_f \mathcal{L}(f) \right)(x) = -\nabla_{u} \ell(u, y(x))\big|_{u = f(x)}$$
which is not trivial indeed. We hope that with the new version of the paper we have answered all of your concerns regarding the mathematical soundness of the paper.

Reviewer comment (Experiments):
The submission seems to not directly address how to trade-off increasing the number of parameters versus updating the parameters already present.

Author response:
Indeed we do not provide this (we could, actually, by comparing the norm of the classical gradient with the performance gain associated to the candidate neurons, but we did not do it in the paper). We understand your point now.

> This issue might be addressed by making the contribution point more explicit.

Ok, we rephrase the last contribution as:
"We automatically adapt the architecture to the task at hand by making it grow where needed, and this in a single run, in competitive computational complexity with respect to classically training a large model just once. To remove any need for layer width hyper-optimization, one could define a target accuracy and stop adding neurons when it is reached."

Reviewer comment (Writing):
Given that a stated contribution of the submission is a definition of optimality, what is meant by that should be stated explicitly.

Author response:
Thank you for noticing this. Indeed we were using the same word "optimal" for different meanings. We have added a paragraph in Section 2 to make this more clear, and this was the opportunity to better introduce the criterion considered in the next sections. A supplementary appendix (A.3) details these concepts and their intertwinement.

Reviewer comment (Writing):
Some statements should be made more carefully to avoid overly general claims.

Author response:
Thank you for the minor points and noticing formulations leading to "overly broad claims", that we have addressed. 
Regarding  "[adding random neurons] would not yield any guarantee regarding the impact on the system loss" however, what we mean is that our method yields the best neuron to add to decrease the loss at first order (in a sense now well defined), while a random neuron does not have this property.

Reviewer comment (Theory):
In section 2.2, the high-level idea that \( v_{\text{goal}}(x) \) indicates the desired infinitesimal change in output of the network is reasonable, but it is unclear how this follows from the functional perspective. The notation seems overloaded, mixing functional and standard ML notation, making it hard to parse. It is also not obvious why \( \nabla_f \mathcal{L}(f) = \nabla_{u=f(x)} \mathcal{L}(u) \) holds, and clarification or references are needed. Furthermore, the lack of definition of the function space \( \mathcal{F} \) adds to the ambiguity.

Author response:
As stated in the common reply ["New version of the paper, with improved clarity"](https://openreview.net/forum?id=Qp33jnRKda&noteId=7ZWDNGZNl9), we have added Appendix A.1 to get the paper self-contained and to explain what precisely is the functional gradient, as well as how to obtain the property $${v_{goal}(x)} = -\left( \nabla_f \mathcal{L}(f) \right)(x) = -\nabla_{u} \ell(u, y(x))\big|_{u = f(x)}$$
which is not trivial indeed. We hope that with the new version of the paper we have answered all of your concerns regarding the mathematical soundness of the paper.

Reviewer comment (Writing):
The term "optimal" is used in many places with different apparent meanings, making it unclear what criteria establish optimality. Two possible definitions seem present: minimizing the distance between \( v_{\text{goal}} \) and the actual update (in §2.2) and layer-wise optimization to make the problem tractable (in §2.3), the latter being acknowledged as sub-optimal. Clarification is needed on the precise meaning in each context.

Author response:
Thank you for noticing this. Indeed we were using the same word "optimal" for different meanings. We have added a paragraph in Section 2 to make this more clear, and this was the opportunity to better introduce the criterion considered in the next sections. A supplementary appendix (A.3) details these concepts and their intertwinement.

Reviewer comment (Theory):
Certain claims are overly broad. For example:
- In the introduction, "removes the optimization issues" is too broad and should specify which issues are avoided and under what conditions.
- In §3.2, implying that the proposed method guarantees loss decrease when adding neurons is misleading, since no such results are shown.
- In §2.3, "recursive" may imply invariance of updates, but this is actually an approximation; this should be clarified at the start of §2.3.

Author response:
Thank you for the minor points and noticing formulations leading to "overly broad claims", that we have addressed. 
Regarding  "[adding random neurons] would not yield any guarantee regarding the impact on the system loss" however, what we mean is that our method yields the best neuron to add to decrease the loss at first order (in a sense now well defined), while a random neuron does not have this property.

Reviewer comment (Novelty):
The description of prior work is too brief to help readers unfamiliar with the literature. It is unclear how methods like Net2Net, AdaptNet, and MorphNet differ from the proposed approach. Citing classical works on neuron boosting/incremental neural network learning, such as Bengio et al. (2006), and related methods like those in GradMax would provide useful historical context. Additionally, a line of work in deep learning optimization with a constrained optimization/Lagrangian view that resembles the recursion argument in §2.3–§3.1 could be relevant to discuss.

Author response:
Thank you also for the references, which we did not know and are very interesting and indeed related to our work. We will discuss them, in an appendix section though, as space is too limited in the main paper, especially with the imposed bibliographic citation style. Speaking of which:

> I have heard the idea of "let's do gradient descent on the architecture" multiple time,

Are you referring to DARTS and to the Lottery ticket hypothesis, or to other papers? in which case, we would be delighted if you could provide references.

Reviewer comment (Theory):
Question on clarity of the functional view: Are the identified notation issues correct, or is there a misunderstanding? If so, what are the formal definitions used, and why is the functional gradient the same as the derivative with respect to the network output?

Author response:
We apologize for the notation overload. We have corrected this, notably by distinguishing the sample-loss $\ell$ from the global loss $\mathcal{L}$, and by correcting a few typos. 
As stated in the common reply ["New version of the paper, with improved clarity"](https://openreview.net/forum?id=Qp33jnRKda&noteId=7ZWDNGZNl9), we have added Appendix A.1 to get the paper self-contained and to explain what precisely is the functional gradient, as well as how to obtain the property $${v_{goal}(x)} = -\left( \nabla_f \mathcal{L}(f) \right)(x) = -\nabla_{u} \ell(u, y(x))\big|_{u = f(x)}$$
which is not trivial indeed. We hope that with the new version of the paper we have answered all of your concerns regarding the mathematical soundness of the paper.

Reviewer comment (Theory):
Question on complexity of the methods: The introduction claims computational competitiveness with standard training, but the methods seem to require SVD computations scaling at least with the square of the network width, which is more than gradient descent. Clarification is requested on this point.

Author response:
We have addressed this point in the common reply ["Computational complexity"](https://openreview.net/forum?id=Qp33jnRKda&noteId=wMiGwiU7Bs), which is being included in a dedicated appendix as well. TL;DR: SVD cost is actually low.