Paper: Symmetric Single Index Learning (ICLR-2024)

Reviewer comment (Theory):
There is no finite-sample analysis which is in contrast to the majority of recent works on learning single index models which had a particular focus on the sample complexity of the learning problem.

Author response:
We agree that a finite-sample analysis of full-batch gradient descent would be helpful, see our response to reviewer qFtS on that topic.

Reviewer comment (Theory):
Certain assumptions of the work are a bit restrictive. For example, the assumption on the activation is only verified for a particular choice that is not commonly used in practice, and the student link function $g$ needs to be carefully constructed using full information on the teacher link function $f$. However, these restrictions are understandable given that this work is a first attempt of studying the training dynamics of symmetric neural networks under a single index model.

Author response:
We agree the restrictions are non-negligible, although we note that some are seemingly necessary: for instance, using a different link function demonstrably improves the rate of convergence, and the theoretical issue of trying to optimize a polynomial landscape if one used the correlational loss with the same link function for student and teacher necessitate the restriction.

Reviewer comment (Writing):
The squared Vandermonde density is never explicitly defined in the manuscript. Perhaps providing an explicit definition and recalling some of its relevant properties can help the readers better understand the problem setting.

Author response:
We agree that the results on learning anisotropic Gaussian data are related in the sense of needing to handle norms of individual single index terms, and will mention these connections in a later draft.  We also agree that giving a proper characterization of the Vandermonde density, and the suggestions to the Lemma and Theorem statements, are all worthwhile for clarity.

Reviewer comment (Writing):
Is the probability in Lemma 4.5 over the randomness of $(a_m)$? In that case, it might be helpful to point this out in the statement of the lemma.

Author response:
We agree that the results on learning anisotropic Gaussian data are related in the sense of needing to handle norms of individual single index terms, and will mention these connections in a later draft.  We also agree that giving a proper characterization of the Vandermonde density, and the suggestions to the Lemma and Theorem statements, are all worthwhile for clarity.

Reviewer comment (Theory):
Theorem 4.2 point (iii) asks for $v_0 = 1 - r_0^2$. Could it be more intuitive to introduce this condition as $\Vert Aw_0 \Vert = 1$?

Author response:
We agree that the results on learning anisotropic Gaussian data are related in the sense of needing to handle norms of individual single index terms, and will mention these connections in a later draft.  We also agree that giving a proper characterization of the Vandermonde density, and the suggestions to the Lemma and Theorem statements, are all worthwhile for clarity.

Reviewer comment (Theory):
"These dynamics naturally motivate the question of learning efficiency, measured in convergence rates in time in the case of gradient flow". Does this say anything about sample complexity when discretized? Because gradient flow time can be rescaled, so it doesn't appear to be a well-defined complexity measure?

Author response:
Although gradient flow can be rescaled, we observe evidence from the literature that using constant rescaling is comparable to the sample complexity from comparable settings in gradient descent or stochastic gradient.  For example, in Theorem 5.3 in [1] the rates for all three methods are essentially equivalent.

Reviewer comment (Theory):
Can you prove a converse to Theorem 4.2 with corresponding lower bounds on the time? This seems doable and like it would strengthen the result to make it more of a characterization.

Author response:
In terms of lower bounds, it would be straightforward to have a lower bound of N^s.  Indeed, the convergence can only be faster if we initialize v = 0 and cos s theta = 1, in which case the ODE governing r is nearly equivalent to the one studied in [2].  However, unlike [2], it's not straightforward to get nearly tight upper and lower bounds on the problem: for one thing, the initialization guarantees are based on the condition number of the frozen matrix A, which is essentially equivalent to the condition number of a rectangular Vandermonde with nodes on the unit complex circle.  As far as we're aware this is still an active area of research to get tight bounds here, one representative paper in [3].

Reviewer comment (Novelty):
It is unclear to me what is conceptually new in this work that does not appear in previous analyses of single-index learning?  
(a) the teacher model / student model are single-index models, but on a different data distribution than usual. But this has a similar inner product to what allows analyzing the Gaussian case.  
(b) the analysis of the single-index model appears to follow a now-standardized template. Would be interesting for the authors to highlight what are the new elements, or how this generalizes the currently-known techniques.

Author response:
In terms of novelty, although other papers that use correlation loss [4], they can still track learning via a single parameter.  We necessarily have three parameters, since our correlation is complex and we can't project without solving the optimization problem onto an ellipsoid [5] at every step.  Therefore the technical novelty is treating the learning with a system of three ODEs, and in particular judiciously choosing the regularization so that the orthogonal part of the learned direction (v) shrinks faster than the correlation with the true direction (r) to guarantee eventual convergence.

Reviewer comment (Evaluation):
It is unclear to me what I am supposed to be learning from the experiments in Figure 1. How does the initial alignment of the model correspond to the gradient flow time in these cases?

Author response:
The experiments are to confirm the expected behavior still occurs under full gradient descent, and to observe that the technical details of controlling the descent of r are not a proof artifact but a necessary detail, as the regularization pushes r down in the initlal parts of training.  As in the vanilla single index setting, higher initial alignment usually corresponds to faster convergence.

Reviewer comment (Theory):
There are several assumptions that limit the applicability of the framework, though they are properly discussed. Proposition 2.3 is the key technique that governs the whole technical assumptions including the assumption on the link function (Assumption 2.4). It is unclear for me as a newbie reader to this field how much it is restrictive and whether this can be relaxed. Any further discussion would be appreciated.

Author response:
The assumptions on the link function can be slightly relaxed, in the sense that allowing for very small terms of higher degree would have minimal effect on the gradient overall.  But the analytic assumption is likely required due to the fragile nature of the hermite-like identity.  We will include more discussion of this in an updated draft.

Reviewer comment (Theory):
Though the paper starts with the single-index learning, it seems that the framework deviates from the framework by considering a different link function in the final form of the modified correlation loss in eq. (14). I found it quite confusing at first sight. After this modification, is this technically qualified as a single-index learning?

Author response:
This is certainly still a single-index model in the sense of trying to recover a single hidden direction that can only be accessed through non-linear measurements.  And there is precedent for using a different link function in the student architecture than the teacher in the vanilla single index setting, see [1].

Reviewer comment (Writing):
After eq. (6), "In other words" sounds confusing. It might be better to explicitly say that the first-layer weights ($\{a_1,\ldots,a_M\}$) and the third-layer weight (the function $g$) will be fixed (frozen), and the second-layer weight ($w$) will be the only trainable parameter.

Author response:
The suggestions given for clarity are appreciated.