Paper: Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape (ICLR-2024)

Reviewer comment (Theory):
I find it very difficult to follow the logic around Corollary 8: before Corollary 8, the authors claim that under general position assumption of the dataset and d larger than n, one can show that most activation regions are non-empty. However, Corollary 8 still focuses on the "non-empty activation regions". If most activation regions are indeed non-empty, why not drop the term "non-empty"?

Author response:
Indeed, if $d$ is large relative to $n$, then most activation regions will be non-empty. We include the term non-empty here to emphasize that our result is not just being applied to empty activation regions which trivially do not contain any critical points.

Reviewer comment (Novelty):
Among the related works being provided in the first section, can you compare your work with Liu (2021)? They seem to provide a stronger result.

Author response:
Liu (2021) shows that in a given activation region, all local minima are global *within the given region*. This means that for a critical point $\theta$ within an activation region $\Omega$, $\theta$ is a global minimizer of the loss restricted to $\Omega$, but there could still exist $\theta' \notin \Omega$ which attains a smaller loss than $\theta$. In contrast, we show that in most activation regions, all critical points are global minima *over the entire loss landscape*. We are proving a proving stronger statements about the critical points themselves by incorporating information about the activation patterns.

Reviewer comment (Theory):
When $d_0=1$, Theorem 5 already provides some strong results, why do we bother deriving Theorem 10?

Author response:
In the one-dimensional case, most activation regions are empty (in contrast to the high-dimensional case where every activation pattern is realized). We provide a more refined analysis by focusing specifically on the non-empty activation regions, characterizing them, and showing that most of them have a full rank Jacobian.

Reviewer comment (Theory):
Unfortunately, the scope of this paper looks rather limited. The paper studies a single-hidden-layer model and only the first layer $W$ is trained. Hence, given an activation region, the prediction $F$ is linear in $W$, which makes the Jacobian matrix constant in the region and it makes proofs much easier. For Sections 5-6, the authors show results specialized to one-dimensional input space. I question if these proof techniques could be extended to deeper networks and higher-dimensional input space.

Author response:
We have added appendices to address non-differentiable critical points and trainable output weights (Appendix G) as well as the deep case (Appendix H). Although our focus has been on the two-layer case, our techniques can be used to understand more general settings. Even when looking at non-differentiable points of activation regions, we can show in the case of one-dimensional input data that most activation regions contain no spurious minima. 

To generalize to the deep case, one can compute gradients with respect to individual layers of the network and find conditions on the activation patterns for which the gradients are full rank. We demonstrate this technique in Appendix H.

Reviewer comment (Theory):
The paper only considers differentiable local minima, which misses the possible existence of non-differentiable local minima. In fact, Laurent & von Brecht (2018) show that in the hinge loss case, local minima can only occur at the non-differentiable boundaries except for flat regions. Is there any hope of including non-differentiable points into the analysis?

Author response:
We have added appendices to address non-differentiable critical points and trainable output weights (Appendix G) as well as the deep case (Appendix H). Although our focus has been on the two-layer case, our techniques can be used to understand more general settings. Even when looking at non-differentiable points of activation regions, we can show in the case of one-dimensional input data that most activation regions contain no spurious minima. 

To generalize to the deep case, one can compute gradients with respect to individual layers of the network and find conditions on the activation patterns for which the gradients are full rank. We demonstrate this technique in Appendix H. 

We would also like to note that we consider a different setting from Laurent and von Brecht by considering the square loss rather than the hinge loss. When we restrict the hinge loss to a given activation region, the loss is locally given by an affine function, so there are no critical points on the interior of the region. In contrast, when using the square loss, the loss restricted to an activation region is given by a quadratic function, so a more careful analysis is needed.

Reviewer comment (Theory):
Most results in the paper only prove that for most activation regions, "all critical points are global minima". If we take a closer look at it, the theorems do not talk about the existence of critical points in these regions; they may or may not exist. Therefore, the statements cannot rule out the following pathological scenario: critical points do not exist at all in the $1-\epsilon$ fraction of the linear regions and the $\epsilon$ fraction of the regions contain bad local minima. In order to make the results stronger, a more complete characterization of the existence of critical points should be helpful.

Author response:
We address the issue of the existence of global minimizers in Theorem 12, where we show that with one-dimensional inputs, most activation regions contain global minimizers. In the one-dimensional case, we are able to show that $d_1 \sim n \log n$ is sufficient parameterization for most regions to have a full rank Jacobian and for the existence of global minima, with the existence of global minima requiring a constant factor of additional parameterization. For the higher-dimensional case, we provide experiments (in Appendix E.2) which confirm that approximately linear scaling of $d_1$ with $n$ is sufficient for most regions to contain global minima.

Reviewer comment (Novelty):
Some directly relevant citations are missing. [A, B, C] show the existence of bad local minima in ReLU networks and thus are directly relevant. [D] and some other papers by the same authors also consider the partition of parameter space with respect to the sign of pre-activations. I feel there should be more missing relevant papers, so please consider reviewing the literature again and updating the paper accordingly. Lastly, the citation to Safran & Shamir (2016) in page 1 should be corrected to Safran & Shamir (2018).  

[A] Small nonlinearities in activation functions create bad local minima in neural networks  

[B] Piecewise linear activations substantially shape the loss surfaces of neural networks  

[C] Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory  

[D] THE HIDDEN CONVEX OPTIMIZATION LANDSCAPE OF REGULARIZED TWO-LAYER RELU NETWORKS: AN EXACT CHARACTERIZATION OF OPTIMAL SOLUTIONS

Author response:
Thanks for highlighting these references. We have added pointers to these works in the introduction.

Reviewer comment (Theory):
The setting described at the beginning of section 3 is different from a typical 2-layer ReLU network in that only one weight matrix is considered as parameters. It would be helpful to clarify whether including $v$ in the parameter space changes the conclusions in Theorem 5.

Author response:
We focused on computing the gradients with respect to the input weight matrix rather than the output head to facilitate our analysis. However, we would like to emphasize that our results showing that $\nabla_W F$ is full rank also imply that $\nabla_{(W, v)}F$ is full rank, since the former is a submatrix of the latter. In particular, our conclusions still hold when we include $v$. We have added a new Appendix G in which we analyze the effect of parameterizing $v$ as well as an extension to non-differentiable critical points.

Reviewer comment (Theory):
While smooth critical points are global minima in most of the activation regions, it is not clear what the volume of these regions are. In particular, results in this paper do not rule out the possibility that the small number of regions with bad local minima make up most of the parameter space. Quantifying the size of the activation regions appears difficult since the setting considered is independent of the distribution of dataset.

Author response:
Some of the methods we used to enumerate the activation regions with full rank Jacobian can be adapted to compute the volumes of such regions. We have added a new Appendix F illustrating this. For the one-dimensional case, we compute the volume of each activation region as a function of the dataset. We also bound the amount of parameterization needed for most of the parameter space (by volume) to have full rank Jacobian in terms of the separation between the data points. For the higher-dimensional case, we introduce an anticoncentration property of the data. Under this property, we prove that the portion of the loss landscape with full rank Jacobian is large by volume.

Reviewer comment (Theory):
The beginning of the second paragraph of section 6 states that a ReLU with one input with bias is equivalent to a ReLU with two input dimensions and no bias. Could the authors add a bit more explanation, as this equivalence does not seem straightforward?

Author response:
For a neural network on one-dimensional data with bias, each neuron is a function of the form $x \mapsto b + wx$. Instead, we could think of the data point $x$ as an element of 2-dimensional space, writing it as $(1, x)$. Then the mapping is of the form $(x_1, x_2) \mapsto bx_1 + wx_2 = (b, w)^T(x_1, x_2)$. The same trick applies in higher dimensions as well.

Reviewer comment (Theory):
Are the constants in the bound known? Can they be quantified in experiments?

Author response:
For the one-dimensional case, we include the constants in our bounds for Theorems 10 and 12. For the high-dimensional case, we rely on a bound from Bourgain et al. [1] which states that the probability of a random $d \times d$ matrix being singular is at most $(1 / \sqrt{2} + o(1))^d$. Recovering the exact constant from this bound is challenging. However, the exact scaling can be checked experimentally, as we demonstrate in our experiments in Section 7. For example, Figure 3b shows that if we use the scaling $\frac{n}{d_0} = 2$, then $d_1 \approx 7$ is sufficient for at least 80 percent of activation regions to have no bad critical points. Note also that for fixed $\epsilon$, if $\frac{n}{d_0} \gg 1$, then Theorem 5 implies that $d_1 = \frac{n}{d_0}$ is sufficient for most regions to have a full rank Jacobian.

Reviewer comment (Theory):
This paper shows that most activation regions do not have bad local minima in their interior. Do there exist non-differentiable points that are local minimizers? If so, will these points cause problems for gradient descent?

Author response:
Indeed, there can exist non-differentiable critical points which are local minimizers of the loss. We have added Appendix G to address this question, where we establish that in most activation regions, the set of spurious non-differentiable critical points forms a low-dimensional subset of the region.

Reviewer comment (Theory):
The assumption of 2-layer network with fixed last layer weights \( v \) is restrictive and impractical.

Author response:
We have added appendices to address non-differentiable critical points and trainable output weights (Appendix G) as well as the deep case (Appendix H). Although our focus has been on the two-layer case, our techniques can be used to understand more general settings. Even when looking at non-differentiable points of activation regions, we can show in the one-dimensional case that most regions have no spurious local minima. To generalize to the deep case, one can compute gradients with respect to individual layers of the network and find conditions on the activation patterns for which the gradients are full rank. We demonstrate this technique in Appendix H.  In Appendix F we now also discuss the volume of activation regions, providing further insight in regard to the comment "most of the landscape".

Reviewer comment (Theory):
The paper only discusses differentiable critical points, but there can be a lot of indifferentiable critical points depending on network settings. In fact generic deep ReLU networks with cross-entropy loss will have non-differentiable sub-optimal local minima [1]. This means that GD can still fall into bad non-differentiable local minima, even if the differentiable local minima are good. So it's still not clear to me that 'most of the landscape is favorable to optimization'.

Author response:
We have added appendices to address non-differentiable critical points and trainable output weights (Appendix G) as well as the deep case (Appendix H). Although our focus has been on the two-layer case, our techniques can be used to understand more general settings. Even when looking at non-differentiable points of activation regions, we can show in the one-dimensional case that most regions have no spurious local minima. To generalize to the deep case, one can compute gradients with respect to individual layers of the network and find conditions on the activation patterns for which the gradients are full rank. We demonstrate this technique in Appendix H.  In Appendix F we now also discuss the volume of activation regions, providing further insight in regard to the comment "most of the landscape".

Reviewer comment (Theory):
Continued on bad local minima. It seems to me that in figure 9(b) GD does not converge to good local minima when network is only mildly overparameterized, despite at initialization the Jacobian has full rank. Can you explain this? Does this contradict your theoretical findings?

Author response:
This experiment agrees with our theoretical results in the sense that when the network width $d_1$ scales linearly in the sample size $n$, gradient descent converges to a global minimizer with high probability. The amount of parameterization necessary for convergence to a global minimizer appears to be a constant multiple of the amount necessary for a full-rank Jacobian at initialization.

Reviewer comment (Theory):
How do you determine whether a Jacobian matrix is full rank?

Author response:
We compute the Jacobian and its rank with PyTorch. The method ```torch.linalg.matrix_rank``` computes the SVD of the Jacobian and counts the number of singular values which are larger than a fixed machine tolerance.

Reviewer comment (Experiments):
For figure 9(a) is the network overparameterized at all? The experiment setting for figure 9 is not clear to me. Can you elaborate on this, just like what you did for other figures?

Author response:
Yes, the network is overparameterized in this experiment because the number of parameters is given by $m = d_0d_1$, where $d_0$ is the input dimension and $d_1$ is the hidden dimension. Here the dataset, MNIST, consists of images represented by 784-dimensional vectors. Hence the hidden dimension (network width) does not need to be large for the network to be overparameterized. We have updated the paper to clarify this by adding the comment "The number of network parameters matches the training set size $n$ when the width satisfies $d_1=n/d_0$, where for MNIST the input dimension is $d_0=784$."