Paper: Implicit regularization of deep residual networks towards neural ODEs (ICLR-2024)

Reviewer comment (Theory):
It seems to me that both fully-connected and convolutional neural networks would be the particular cases where the ODE parameter matrices A, V, W are considered sparse (some or most of the entries are fixed to zero and the rest are optimised according to Equation (5)). The experiment, depicted in Figure 3, suggests that results in Theorem 4 might be expanded to the more generic sparse scenarios. Might be good if the authors could confirm this observation.

Author response:
We confirm that the reviewer is right. In Section 4.3, we describe a general setup where Theorem 4 holds, where it is only assumed that the residual connection $f$ is a Lipschitz-continuous $C^2$ function. This includes in particular the sparse version of the model considered in the paper. This is an interesting comment that we will incorporate in the next version of the paper.

Reviewer comment (Theory):
If it happens that the weights are initialised randomly, for every finite depth of the neural network (as there’s finite time and finite depth), there could be still a Lipschitz-continuous function (ii) over that set (as locally Lipschitz function on a compact set is Lipschitz as per, e.g. https://faculty.etsu.edu/gardnerr/5510/cspace.pdf ). However, what’s appearing in Figure 3 appears to go beyond literal application of Theorem 4: not only the function is compact but also it appears that the Lipshitz constant is preserved to be small during the optimisation.

Author response:
This is a very good point, thanks to the reviewer for raising it. It is true that for a fixed depth, it is always possible to construct a Lipschitz-continuous function that interpolates the weights, e.g., by piecewise-affine interpolation. Nevertheless, without further assumption, the Lipschitz constant of such a function depends on the depth $L$. On the contrary, as observed by the reviewer, in our results, the Lipschitz constant of the function that interpolates the weights is independent of the depth (see, e.g., Theorem 4 (ii) and Theorem 6 (ii)). Moreover, the proofs give a bound on this Lipschitz constant during training, even though we did not state it in the theorems to simplify the exposition. We will add a remark about this in the next version of the paper.

Reviewer comment (Theory):
Another interesting aspect is that it would be intuitive to see smaller Lipshitz constant could be beneficial for out-of-distribution recognition. I wonder if the authors could clarify on this aspect? To what extent would the proposed derivation generalise for higher-than-two layers of multilayer perceptrons?

Author response:
Yes, this is a correct intuition. This is confirmed by a recent NeurIPS paper (Marion 2023, ref. below), where it is shown that the generalization error of deep residual networks / neural ODEs grows with the Lipschitz constant.

Generalization bounds for neural ordinary differential equations and deep residual networks, Marion, NeurIPS 2023.

As mentioned above, Section 4.3 presents generalizations of our results to an arbitrary residual connection $f$, which we assume to be a Lipschitz-continuous $C^2$ function. This includes in particular the case where $f$ is a deeper MLP. Our finite training time results are fully proven in this general case. Regarding the limit $T \to \infty$, we prove that convergence to a neural ODE holds for a general residual connection $f$ if it satisfies a Polyak-Lojasiewicz (PL) condition. Proving the PL condition in the case where the residual connection is a deeper MLP is a challenging mathematical question, which deserves future work.

Reviewer comment (Evaluation):
For the experiments on cifar, the authors show the plot of a random coefficient and its value as one goes deeper into the layer. However, it would be more beneficial to see the difference in Frobenius norm of the weight matrices as well.

Author response:
Dear reviewer, 

We finished the figure according to your suggestions. It can be accessed at the following address:  https://anonymous.4open.science/r/double_blind_implicit_regularization_resnets_nodes-1768/fig_rebuttal.pdf

In this figure, we display the average (across layers) of the Frobenius norm of the difference between two successive weights in the convolutional ResNets after training on CIFAR 10, depending on the initialization strategy. The index i corresponds to the index of the convolution layer. Results are averaged over 5 runs. We see that a smooth initialization leads to weights that are in average an order of magnitude smoother than those obtained with an i.i.d. initialization. 

We will add this figure to the next version of the paper. 

Best regards, 

The authors

Reviewer comment (Theory):
One of the key drawbacks is requiring weight-tying or weight sharing. While results with weight-tied networks don't look bad, still this would have been nice to take care of. Can the authors elaborate on where all and where exactly does it interfere with the proof strategy?

Author response:
This is a good point, and the reviewer is right in pointing it out. We consider in the paper a weight-tied initialization in order for the network to satisfy an ODE-like structure at initialization. Such a structure is necessary for an ODE structure to hold after training. As stressed in Section 4.3 of the paper, the weight-tying assumption can be relaxed by taking a so-called “smooth initialization”, where the weights at initialization are a discretization of a smooth (potentially random) function.

It remains that this scheme does not encompass iid initializations (e.g., Gaussian or uniform), which is a classical scheme in initializing neural networks. In this case, the large-depth limit falls into a different regime, namely a stochastic different equation instead of an ordinary differential equation (see Marion et al., 2022, ref. below). It turns out that analyzing the convergence in this regime is challenging and deserves a paper by itself. We will make it clearer in the conclusion of the paper.

Scaling ResNets in the Large-depth Regime, Pierre Marion, Adeline Fermanian, Gérard Biau, Jean-Philippe Vert, arXiv:2206.06929, 2022.

Reviewer comment (Theory):
The other issue is that the linear overparameterization $m > c_1 n$ would, in practice, be somewhat unreasonable and is thus a bit of a stretch. Maybe this is a norm in the literature, but do the authors have some ideas how to go around this?

Author response:
We thank the reviewer for pointing out this important assumption. The assumption $m > c_1 n$ is required to show that our loss landscape satisfies a Polyak-Lojasiewicz (PL) condition, which is one of the main modern frameworks to show convergence of neural networks. The PL condition can be seen as an alternative to convexity, which is not satisfied in neural networks. Alleviating the assumption $m > c_1 n$ would require working under a weaker condition than PL to show global convergence. Unfortunately, we are not aware of such a weaker condition in the neural network literature at the moment.

Note however, as highlighted in Section 2, that the linear overparameterization condition is mild with respect to usual assumptions in the literature (polynomial overparameterization).