Paper: Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR-2024)

Reviewer comment (Experiments):
It's a shame that the authors did not include concrete experiments training their lifted models. I would be interested to see empirical tests validating the theory: i.e., actually do \( n \) random lifts, train via sgd, and see what percent of the time it converges.

Author response:
We agree that concrete experiments would be a valuable addition to this work, and initially did not include them due to space limitations. However, we have nonetheless added two small-scale experiments in the appendix. First, a theoretical validation, in the very simple setting of learning a one-dimensional function, of the fact that quantiles of the loss converge to arbitrary precision, and this convergence depends on the architecture. Second, a more realistic experiment (although of limited scale) of classification on MNIST, to investigate whether sparse networks behave in a significantly different way from their dense counterparts.
Note that these experiments remain small in scale, and thus admittedly not entirely convincing regarding the impact of sparsity on performance. However, these experiments are an indication that such sparse networks tend to have similar performance to their dense counterparts, and more extensive experiments on larger networks (for vision or NLP tasks) would be an interesting direction for future work.

Reviewer comment (Theory):
I have reservations about assumption A1. The set \( A_{s,g}(\kappa, \epsilon) \) echoes ideas from neural tangent kernel approximation -- as the authors note, it includes not only weights which are close to \( f^* \), but also weights which can be made close to \( f^* \) by moving a little bit \( u \) in the tangent approximation to the network. The NTK argument justifies this by showing that, under some random initialization, networks are close to their linear approximation as their width goes to infinity. Assumption 1 essentially starts by assuming that the tangent assumption is reasonable, and then states a lower bound on the size of the set of weights that can be made close to \( f^* \) by moving in the tangent plane. So in effect they seem to be _assuming_ what NTK analyses _concludes_ from infinite-width scaling. In that light, the fact that this paper claims convergence guarantees for finite-width networks does not seem that impressive.

Author response:
First, we would like to clarify the fact that Assumption A1 is not an assumption made throughout the paper. In particular, the proof of Theorem 4.3 relies on proving that random sparse lifts do verify this assumption, and are thus learnable via gradient flow according to Theorem 2.1.
Informally speaking, Theorem 2.1 has the form ["A1" implies "convergence"], while Theorem 4.3 proves ["Random sparse lift" implies "A1"], thus implying convergence.
We made this important point clearer in the text, and apologize for the potential confusion.
We have renamed "Assumption A1" to "Condition C1" in the updated version.

Second, regarding the link to NTK assumptions. Our analysis is indeed quite strongly linked to the proofs leveraging the NTK in the vastly overparameterized
regime (more parameters than there are samples), however assumption A1 is much weaker than positive-definiteness of the NTK. While positive definiteness *requires* vast overparameterization to ensure the NTK is not rank-deficient, assumption A1 can hold even with infinitely many samples, provided the architecture has universal approximation, which is indeed what we show later.
While our results are still ''overparameterized'' in the sense that they require
much more neurons to ensure learning than are needed to ensure universality,
we argue that this is a more satisfactory analysis because it lifts the assumption of vast overparameterization. As such, Assumption A1 is a reasonable candidate to replace the assumption of positive-definiteness and extend the analyses using NTKs to networks with finite width and infinite samples.

Reviewer comment (Writing):
What do the authors mean when they say that networks whose width is not increasing with depth "could be ill-behaved"?

Author response:
We show that for networks whose lift dimension satisfies certain condition, we can ensure convergence to arbitrarily good precision with arbitrarily high probability when scaled up.
In particular, the ``probability of failure'' (the loss stops decreasing at a non-vanishing threshold and thus does not go arbitrarily low) vanishes when scaling up: they are in this sense well-behaved, because this is a desirable property.
We cannot prove such claims for lift dimension which do not satisfy this condition,
in particular, it would be consistent with his theory that constant-width MLP
have an incompressible 1\% failure probability, which does not vanish when the
width tends to infinity: they would be ill-behaved.
It appears that this is not the case in experiments, but this theory is insufficient to explain this property.

Reviewer comment (Novelty):
The discussion in Section 2 seems to suggest that the convergence analysis is an extension of the existing NTK analysis, how does the results in this paper different?

Author response:
This theory attempts to describe a regime in which the NTK is not positive definite,
as such it can be understood as a generalization of NTK results to a setting in which they did not previously apply. Note, however that NTK results are typically restricted to dense architectures, while our convergence result is restricted to sparse architectures.
More precisely, the NTK is constructed as $dF_\theta \cdot dF_\theta^T \in \mathbb{R}^{n \times n}$, where $n$ is the number of samples, and $dF_\theta \in \mathbb{R}^{m \times n}$, where $m$ is the number of parameters.
In particular, by a rank argument, the NTK cannot have all eigenvalues positive if $m < n$, thus NTK convergence results only apply to the vastly overparameterized setting, of having more parameters than samples. In contrast, our theory extends
to infinitely many samples, and while our results hold only for large networks,
they do not require this vast overparameterization assumption.

Reviewer comment (Writing):
How should one understand the $\mathcal{C}$ in LiftPMod? The example given in the paper is not very helpful. It would be great if an example of random sparse lift is given for a specific NN architecture, say multi-layer feedforward networks.

Author response:
We have added in the appendix examples with classification on MNIST, where $\mathcal{C} = [28] \times [28] \subseteq \mathbb{N}^2$ are the coordinates of pixels.
The random sparse lifts then use $n_0 = 784$ as the lift dimension for the input.
However, one must specify how these are connected to the input grid (e.g. is the grid flattened by row or by column). The fact that $\\# \mathcal{C} = n_0$ is not sufficient to fully specify the network architecture.
Does this example help clarify your concerns?

Reviewer comment (Presentation):
It is not clear why a long list of citations from approximation theory (page 1) is given since the paper studies convergence.

Author response:
The reason we stressed prior work on approximation results is that our convergence guarantee is given with respect to the infimal loss of all lifts.
However, this infimal loss is guaranteed to be null only for architectures satisfying a universal
approximation property.
Thus, our result turns a  universal approximation guarantee into a universal learning guarantee, and would typically only be applied in settings where universal approximation has already been shown, which led to justify that many settings
already enjoy such guarantees, with links to the corresponding literature.

Reviewer comment (Theory):
In many places, claims are unjustified. For example, 'the class of neural networks that can be described .. more expressive than Tensor Programs". Why is that so? It would be OK not to explain this if it was a trivial statement.

Author response:
Regarding the link to tensor programs, there are two main differences.
First, tensor programs focus on only one parameter for scaling, whereas
we use lift dimensions that allow for more scaling dimension, for instance,
in the example of a multi-layer perceptron with non-constant widths.
Secondly, tensor programs use weight matrices (whose dimension scales quadratically with respect to the scaling parameter) in a linear fashion, such as to represent the operation $x \mapsto W \cdot x$ with weight matrix $W \in \mathbb{R}^{n \times n}$.
A random sparse lift similar to this would use a weight matrix $W \in \mathbb{R}^{n \times n}$ and a random connection matrix $B \in \{0,1\}^{n \times n}$ to compute
the operation $x \mapsto (W \odot B) \cdot x$.
The definition of random sparse lifts however also authorizes the operation
$ x\mapsto \left[\sum_j B_{i,j} e^{W_{i,j}} x_j / \sum_j B_{i,j} e^{W_{i,j}} \right]_{i \in [n]}$, realized with the maps $M: w, x \mapsto (e^w x, e^w)$ and $\sigma : u,v \mapsto u / v$.
While the operations described by tensor programs are by far the most common,
the operations described as random sparse lifts are, in this sense, more general.

Reviewer comment (Writing):
Informal text: "full-support type assumption" on page 2.

Author response:
Thank you for your time in reviewing this submission and your valuable comments. We have uploaded an updated version as supplementary material, with changes in red. In particular,
we apologize for the informal style in which a few passages were written, and have reformulated several parts of the introduction to use a more formal writing style. Please let us know if you think other passages should be reworked as well.

Reviewer comment (Theory):
I have not seen how the class of architectures introduced here includes/related to convolutional neural networks.

Author response:
We have added in the appendix an example of a convolutional network realized as a lift.
The lift dimension, in this case, corresponds to an increase in the number of channels.

Reviewer comment (Theory):
Do you have to maintain full support of the weight distribution throughout training (seeing that you have an assumption on initialization, similar to Nguyen and Pham, 2020)? If so, does sparsity of the network make maintaining full support harder than in the fully-connected case?

Author response:
Regarding your questions, the full-support of the weight distribution is an assumption of the initialization only, it is, for instance immediately
satisfied in the case of a multi-layer perceptron with normally-distributed weights,
because Gaussian distributions have full support.
We prove that this is sufficient to guarantee that the tangent approximation holds
for a long time after the initialization, due in part to the sparsification.

Reviewer comment (Theory):
Do you think this result can be extended to classification loss (e.g. logistic, exponential loss)? If so, how do you compare your result to existing convergence results for these loss (for instance, Ji and Telgarsky’s 2020 ‘Directional convergence and alignment in deep learning’)?

Author response:
We think this result cannot be extended as-is to classification loss,
but we conjecture that minimal additional assumptions, such as boundedness of 
the non-linearities associated to terminal vertices, will suffice
to guarantee convergence to low classification loss in a similar regime.
While we expect directional convergence, as studied by Ji and Telgarsky,
for the last layer weights, this might yield interesting extensions
to intermediate features which are not positively homogeneous (e.g. sigmoids instead of ReLUs, or more complicated activations).
Note that the convergence we expect may be only a convergence in loss,
and the convergence of all parameters, especially for intermediate layers,
may require stronger assumptions.

Reviewer comment (Theory):
It seems from Definition 4.2 that the sparsified graph would have bounded average degree. Is this level of sparsity tight?

Author response:
We have tightened our analysis to this specific kind of sparsity,
and we suspect that other sparse regimes may have similar guarantees with varying
details. As an example of varying detail, the set $S_0$ uses a growth
of the width like $n \log n$ between a node and its parent,
but other sparsifications may require
a growth like $n^2$.
Tightening the set of admissible lift dimensions may require tools that
are tailored to the particular sparsification distribution.

Reviewer comment (Writing):
Small typo: Page 2, last paragraph, “S_0 = {(s_0, s_1) \in \mathcal{N}^2 …” should be \in \mathbb{N}^2.

Author response:
Thank you for your time in reviewing this submission and your valuable comments. We have uploaded as supplementary material
an updated version, with all typos corrected and changes marked in red.

Reviewer comment (Writing):
Typo: In section 3, sometimes ‘Euclidean’ is written with a lower case ‘e’.

Author response:
Thank you for your time in reviewing this submission and your valuable comments. We have uploaded as supplementary material
an updated version, with all typos corrected and changes marked in red.