Paper: Spectral Neural Networks: Approximation Theory and Optimization Landscape (ICLR-2024)

Reviewer comment (Experiments):
Q1 is motivated as being of practical importance - how many neurons are needed to achieve an accuracy of X%. While the theoretical results provide this, they are difficult to parse into practical considerations. It would be helpful to have predictions of the theory on how many neurons are needed (and how many layers) for a given accuracy plotted (for at least some choice of parameters described in Theorem 2.1). A comparison with an actual implementation of SNN would make the theoretical results especially convincing (if they match up).

Author response:
Regarding the simulation of neuron numbers for accuracy approximation in eigenvector calculations, we acknowledge this as an intriguing area for future research. This aspect, closely tied to the Laplace-Beltrami operator as highlighted in Corollary 1, offers a promising avenue for exploring the relationship between theoretical results and empirical findings. However, our current focus is on the theoretical foundations of Spectral Neural Networks (SNNs). Consequently, our current simulations were confined to the parameterized and ambient optimization landscapes in SNNs.

Future studies could also fruitfully compare SNNs with other neural network models for spectral embedding tasks. While incorporating a detailed discussion on approximations to Partial Differential Equation (PDE) operators would indeed be valuable, we believe it might overburden the scope of this paper. Such topics merit a thorough examination in subsequent research.

Reviewer comment (Novelty):
Q1, as phrased in the Introduction, suggests that it is unknown whether it is "possible to approximate eigenvectors of large adjacency matrices with a neural network". However, as noted by the authors, there has been work showing success in this direction already. Perhaps it would be better to phrase Q1 as "are there theoretical guarantees that a neural network can approximate the eigenvectors of large adjacency matrices".

Author response:
The updated manuscript emphasizes the theoretical nature of our inquiries when describing Q1-Q3. The existing empirical results in the literature were indeed important motivators for our work.

Reviewer comment (Theory):
Sec. 2.2 is said to be aimed at "constructive ways to approximate $\textbf{Y}^*$, but it is unclear how Theorem 2.2 achieves this. There is not mention of optimization in the theorem, and the number of neurons is set to $N = \infty$. While I understand that the number of neurons can be reduced (in Remark 2.3), this mismatch between aim and result disrupted the flow of the paper.

Author response:
We want to highlight the differences in nature of Theorems 2.1 and 2.2: it is very different from stating that an approximating NN can be found by solving an optimization problem with a **clearly defined objective** (as in Theorem 2.2) than to simply state the existence of an approximating NN with no indication on how even to begin the search for it (as in Theorem 2.1.). While we agree that Theorem 2.2. does not describe an explicit algorithm to find this approximating NN, it is implicit that once the clearly defined target objective has been introduced, one can use any optimization algorithm to search for its minimizers. Because of this, and given our desire to highlight the conceptual differences between Theorems 2.1 and 2.2, we used the word constructive to describe the latter. We would be happy to modify this term if the reviewer believes that there is a more suitable alternative.

Reviewer comment (Presentation):
The figures (Fig. 3-5) were mentioned only briefly in the introduction (before many of the details of the paper were introduced and I was under the impression that they would be referenced later in more detail. As it stands, I did not get anything from the figures. Including them later in the text, as the experiments are motivated (e.g., why it is reasonable to consider "Near optimal", "Large gradient", and "Near saddle") would greatly improve their impact.

Author response:
We thank the reviewer for their suggestion. In the revised manuscript, we refer to our figures after Remark 3.1 when discussing the theoretical results about the landscape of the objective $\ell$.

Reviewer comment (Writing):
I did not understand what was meant in the Introduction by the "spectral contrastive loss function $\mathcal{l}$ is non-convex in the 'ambient' space' " until the discussion about the need for the quotient geometry in Sec 3. I think making this point more clear earlier in the paper would help the reader understand why this is an interesting and tricky problem.

Author response:
We thank the reviewer for their suggestion. In the revised manuscript, we have highlighted this point in the paragraph mentioned by the reviewer.

Reviewer comment (Theory):
The theory established in this paper seems far from providing any practical insights on training spectral neural networks besides proving the feasibility of such an approach--at least the authors did not attempt to include argument like this in the paper. Not covering data-dependent kernels as those in HaoChen and most SSL work also reduce the significance of the result.

Author response:
We start by highlighting that SNNs are useful beyond SSL and the context of the work of HaoChen et al. 2021 Indeed, our results can justify the use of SNNs to solve eigenvalue problems of differential operators on manifolds. The use of SNN in this context is novel, and in particular the induced training procedure is quite different from popular approaches in the literature such as Physics Informed Neural Networks (PINNS). The use of SNN beyond standard tasks in machine learning is thus of interest to the broader scientific community. See, for example, our response to Reviewer hkrF and the Conclusions section in our paper.

On the other hand, we emphasize that studying the graph constructions in HaoChen et al. 2022 is an interesting research direction and we expect that a similar analysis to the one presented in our paper can be carried out in settings that are more useful for SSL. However, to achieve this, one would first need to generalize some of the theoretical results in Calder et al 2022. This mathematical task seems feasible, but only after a lengthy analysis. This can be explored in future work.

Reviewer comment (Theory):
Please see the above question about proof of Theorem 2.2.

Author response:
Recall that $\bar{U}$ is invertible. Hence, let $E = \bar{U}^{-1}Y_{\theta^*}$. We let $E^1$ be the first $r$ rows of $E$ with the rest of rows being equal to 0. Similarly, $E^2$ has the rest of the rows of $E$. 

> Remarks G.1 - G.3. 

More than being necessary for our proofs, these remarks help convey why our results are not direct consequences of existing results and that careful analysis is needed to deduce them. This is closely connected to the discussion in our general response to reviewers.

Reviewer comment (Novelty):
I believe that the manuscript is missing some works in the literature, and adding and discussing these will better guide the reader.

Author response:
We hope our edits improve our manuscript’s clarity and also better position our work in the landscape of existing results on similar problems. All references provided by the reviewer were incorporated in the paper. New text has been highlighted in blue.

Reviewer comment (Novelty):
Another line of research missing in the current paper is the recent work on generic NN-based eigensolvers [C], [D] that aim to recover the ordered top-$r$ eigenbasis (i.e., without modulo rotation) unlike the SNN and current work.

Author response:
> Other differential operators and Generic NN-based eigensolvers.

We have added references to the papers the reviewer mentioned. In Section A.2 we have added a brief discussion on the eigensolvers mentioned by the reviewer.

Reviewer comment (Theory):
Remark 2.2 is hard to appreciate. Can you explain the reasoning behind this in detail?

Author response:
>On Remark 2.2.  

Remark 2.2. aims at highlighting that the $\epsilon^2$ term in our error estimates in Theorem 2.1. is essential for our proof of Theorem 2.2. As discussed in our general response, this $\epsilon^2$ term could not have been obtained by directly fusing the results in Calder et al 2022 and Chen et al 2022. In order to get the term $\epsilon^2$, we needed to first obtain a stronger quantitative estimate for the regularity of the (discrete) graph Laplacian eigenvectors (stronger than the ones explicitly stated in Calder et al 2022) and then apply the results in Chen et al 2022 to find an NN approximator of a suitable extrapolation of the discrete eigenvector to the manifold (different from an associated eigenfunction!, for otherwise the error of approximation would not be $\epsilon^2$ but $\epsilon$). In order to obtain the better discrete regularity for graph Laplacian eigenvectors, we used the fact that graph Laplacian eigenvectors converge in a stronger semi-norm (almost $C^{0,1}$) toward Laplace-Beltrami eigenfunctions.

Reviewer comment (Theory):
The beginning of the introduction sounds like the goal of the paper is to develop and analyze a neural network-based eigensolver. My understanding of an eigensolver is that this is an algorithm that produces the full list of eigenvalues and eigenvectors. However, the method considered in the paper gives us much less: first, it is restricted to \( r \) largest eigenvalues and, second, the produced matrix \(\mathbf{Y}\) contains eigenvectors only up to an \( r \times r \) rotation, so there is still work to be done to extract the eigenvectors and eigenvalues. These points are not discussed; moreover, in the comparison of SNN with traditional eigensolvers only the advantages of the SNN are mentioned.

Author response:
> Only recovering eigenvectors up to rotation. 

We do not believe this is a limitation. First, notice that for many important downstream tasks, such as spectral clustering and spectral contrastive learning, it is sufficient to generate unions of eigenspaces, and thus, recovering individual eigenvectors may not be necessary for those tasks. On the other hand, if one indeed desires to generate the first K individual eigenvectors of $A_n$, it suffices to simply run the SNN algorithm for $r=1, \ldots , K$ and carry out a Gram-Schmidt like procedure with the outcomes. This pipeline would indeed avoid the use of eigensolvers, as claimed.

Reviewer comment (Theory):
*\(\eta\) is a decreasing, non-negative function* — where do you use that \(\eta\) is decreasing?

Author response:
> Assumption on $\eta$.

The non-increasing assumption on $\eta$ appears in multiple results in the manifold learning literature. We used some of these results in the proofs of our theorems. Notice that most $\eta$ used in practice (e.g., Gaussian, 0-1) satisfy this assumption.

Reviewer comment (Writing):
*\(D_G\) is the degree matrix associated to \(G\)* — what exactly is the definition of \(D_G\)?

Author response:
> The definition of $D_G$.

$D_G$ is the diagonal matrix whose $i$-th diagonal entry is the degree of the point $x_i$. $D_G$ was defined in the Appendix in the paragraph below (A.5). In the updated manuscript, we have added an explicit reference to its definition.

Reviewer comment (Writing):
In Theorem 3.2, the notation for matrices with the subscript \(S\) seems unexplained. Also, the matrix \(\Lambda\) is not defined (only \(\overline{\Lambda}\) is defined).

Author response:
> Notations in Theorem 3.2.

There is a typo in Theorem 3.2. We had not defined $\Lambda$. We thank the reviewer for pointing this out. We have now incorporated it in the updated manuscript.

Reviewer comment (Novelty):
Clarification is needed on the issue with the connection of the present paper to Luo & García Trillos, 2022.

Author response:
> About the related work and novelty.

First, we want to point out that the work Luo & Garcia Trillos 2022 that the reviewer mentions makes very different assumptions on the matrix $A_n$ than the ones we make in our paper. The implications of the different assumptions are indicated at multiple points in the paper: in Remark 3.2, the discussion below Theorem 3.2, and Remarks H.1 and H.2. 

Our paper studies SNNs comprehensively and combines multiple results from several subfields (such as manifold learning, neural network approximation theory, and Riemannian optimization) in a way that is original and far from trivial. Indeed, as we explain in our general response to reviewers (see also the response to Reviewer hkrF), we had to tackle multiple technical difficulties to bring together existing results in the literature. The paper Chen et al. 2022 that the reviewer mentions is indeed a useful auxiliary result for our work, but by no means are our Theorems 2.1 and 2.2 corollaries of it.