Paper: Data geometry and topology dependent bounds on network widths in deep ReLU networks (ICLR-2024)

Reviewer comment (Theory):
The relationship of novel terminologies used in this paper to other ideas in learning theory is unclear. For instance, what is the relationship between the margin of a "feasible architecture" on a manifold and the generalization error of that architecture on that manifold?

Author response:
>**W1, Q4. The relationship of novel terminologies used in this paper to other ideas learning theory is unclear. For instance, what is the relationship between the margin $\varepsilon$ of a "feasible architecture" on a manifold and the generalization error of that architecture on that manifold? Can you clarify the relationship of your novel terms to other terms in learning theory?**

In our work, the margin $\varepsilon$ in Definition 3.1 signifies that if $\varepsilon$ is large, the data manifold may have a 'simple' polytope-basis cover, involving fewer polytopes or a smaller total number of faces. Consequently, the network architecture proposed in Theorem 3.4 suggests that *'the large margin cover can be approximated with a reduced number of neurons.'* In other words, increasing the filtration value $\varepsilon$ generally leads to a decreased number of neurons, establishing a connection between generalization error and network architecture from the perspective of learning theory.

The insights from Theorem 3.5 can be grasped through a similar approach. In this theorem, we present a bound in terms of the dimension of the simplicial complex and the number of its faces. In TDA, the simplicial complex tends to simplify as the filtration value $\varepsilon$ increases, consequently, our bounds in Theorem 3.5 decrease. This highlights the correlation between the growth in the number of required neurons (the network width) and the increasing complexity of the dataset geometry.

Reviewer comment (Theory):
In Proposition C.1, where the author claims that "feasible architecture" is a sufficient condition for universal approximation, the proof is unconvincing. The existence of a function \( f_\delta \) which fits the indicator of \( \mathcal{X} \) is assumed, and then a neural network is defined to be equal to this \( f_\delta \). It is unclear how one knows there is such a neural network. Universal approximation would guarantee that, but that is what is being attempted to prove.

Author response:
>**W2. Proof of Proposition C.1 is not convincing, you assume existence of function then proe NN equals to this function. How do you know NN exists? UAP guarantees but it is what you try to prove indeed?**
 
Here we provide detailed explanations. What Proposition C.1 demonstrates is that if we can construct a constant function with its support sufficiently 'close' to $\mathcal{X}$, then we can also construct a function whose $L^p$ distance from $1_{\mathcal{X}}$ is sufficiently small. 
In simpler terms, the proposition establishes that *"constructing the indicator function"* is more difficult than *"approximating the indicator function in $L^p$ sense."* This is the reason why we solved the first problem.
In the revised paper, we have added some sentences to provide a more detailed understanding. Refer to the first paragraph on page 3 in the revised manuscript.

Reviewer comment (Theory):
The novel term "feasible architecture" is doing a lot of work in this theory and seems to obscure the question of how well an architecture actually fits a dataset. While it is clear that ReLU networks can fit polytopes exactly, it is not clear how well they can fit arbitrary manifolds.

Author response:
>**W3, W4, W6, Q2, Q3. It is clear that ReLU networks can fit polytopes exactly, but it is not clear how well they can fit arbitrary manifolds. What guarantees are there for finding a tight cover for a given manifold? How do you actually find a polytope-basis cover?**

Indeed, this is the common concern raised by all reviewers. Please refer to General Comments KR1 in the overall comment for the detailed response.

Reviewer comment (Theory):
For the term "polytope-basis cover," it is unclear what guarantees there are for finding tight covers for a given manifold.

Author response:
>**W3, W4, W6, Q2, Q3. It is clear that ReLU networks can fit polytopes exactly, but it is not clear how well they can fit arbitrary manifolds. What guarantees are there for finding a tight cover for a given manifold? How do you actually find a polytope-basis cover?**

Indeed, this is the common concern raised by all reviewers. Please refer to General Comments KR1 in the overall comment for the detailed response.

Reviewer comment (Novelty):
There is missing discussion of prior related work on relating topological characteristics to generalization theory, for example: "Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks" - Birdal et al. (NeurIPS 2021).

Author response:
>**W5, Q5. Missing some discussion of prior related work on relating topological characteristics to generalization theory.**
 
We have revised the paper to have relevant comparison with prior related work. See KR2 in General comments.

Reviewer comment (Reproducibility):
The practical applicability of the results is unclear, specifically how to actually find a polytope-basis cover.

Author response:
>**W3, W4, W6, Q2, Q3. It is clear that ReLU networks can fit polytopes exactly, but it is not clear how well they can fit arbitrary manifolds. What guarantees are there for finding a tight cover for a given manifold? How do you actually find a polytope-basis cover?**

Indeed, this is the common concern raised by all reviewers. Please refer to General Comments KR1 in the overall comment for the detailed response.

Reviewer comment (Evaluation):
Many strong claims are made in this paper, but it is not clear they are substantiated after reading.

Author response:
>**W7. Many strong claims are made in this paper, and after reading the paper it's not altogether clear to me they are substantiated.**
 
Throughout the revisions, we have improved the flow of our manuscript and emphasized our contributions. We highly recommend reviewing Section 1 in the revised manuscript for a comprehensive understanding. In summary, our main contribution lies in providing bounds on network widths that depend on the geometric characteristics of the dataset, described in terms of its polytope cover.

Reviewer comment (Theory):
Proposition 3.5 seems trivial. A very wiggly (but non-intersecting) decision boundary is homeomorphic to a linear decision boundary. The former cannot be solved by a linear classifier, but the latter can be.

Author response:
>**W8. Proposition 3.5 seems trivial.**

It might be understood trivially. In the revised paper, we have moved this result to the appendix, and removed from our main contributions.

Reviewer comment (Theory):
Is topological space really the right level of generality for this paper? Why not just consider manifolds? The former is substantially more general, and it appears only subsets of \( \mathbb{R}^n \) are considered.

Author response:
>**Q1. Is topological space really the right level of generality for this paper? Why not just consider manifolds? The former is substantially more general, and it appears you only consider subsets of $\mathbb{R}^{d}$.**
 
Thank you for highlighting a crucial aspect. As you suggested, the role of neural networks in real-world applications is not merely to distinguish a data point within $\mathbb{R}^d$, but rather to establish a decision boundary against other classes. In response to your feedback, we have slightly generalized the definition of a *'convex polytope'* in Definition E.1 in the revised paper. Then, our theoretical results still hold with this definition, where the setting becomes more practical.

Furthermore, we want to emphasize that our empirical findings in Appendix E demonstrate that each class in the real datasets (MNIST, Fashion-MNIST, CIFAR10) datasets can be effectively classified by a *'convex polytope'* characterized by a simple number of faces. Therefore, our result in Theorem 3.4 can provide the feasible architecture of neural networks on these datasets.

Reviewer comment (Experiments):
Lack of applications to practical real-world datasets. It is not clear how to count the numbers of polytopes, or j-dim facets in simplicial complex, necessary to approximate a given real-world dataset, so the practical application of the results is somewhat unclear.

Author response:
**W1. Lack of applications to practical real-world datasets.**

In the revised paper, we supplied empirical application of our results, involving seeking a polytope-basis cover of the real-world dataset.
See General Comment KR1 in the overall comments.

Reviewer comment (Presentation):
Presentation of the paper suffers from several drawbacks. The paper contributions are theoretical, but the presented in the article proofs are too sketchy. The article is intended for a wider ICLR community, but because it is too sketchy in the main part, a smooth reading even for experts is questionable.

Author response:
**W2. Paper does not provide the detail proofs in the main body, leading to bad presentation.**

Thanks for the constructive comment. We moved less important proofs to the appendix, and allocated more paragraphs to highlight our key ideas. See General Comment KR 2 in the overall comments.

Reviewer comment (Theory):
The numerous allusions to topological aspects are heavily overstated. The only place where some topological notion appears, namely Betti numbers, is in the fourth construction (Theorem 3.7) involving the polytopes with prism-shaped polytopes removed. However in this context the (d-i)-th Betti number is simply the number of the removed prism-shaped convex polytopes with maximum i unbounded axes, so the Betti numbers can be replaced by such simple counters in Theorem 3.7.

Author response:
**W4. The numerous allusions to topological aspects are heavily overstated. It only appears in Theorem 3.7 stating Betti numbers, with heavy assumptions on the shape.**

Recognizing that our results primarily pertain to geometric rather than topological aspects, we have revised our manuscript, including adjustments to the title, to emphasize the geometric perspective rather than topological analysis. See General Comment KR3 in the General comments.

Reviewer comment (Novelty):
Related work section does not mention vast literature on geometry and topology applications in analysis of data representations, e.g., Kim K et al. "Pllay: Efficient topological layer based on persistent landscapes." Advances in Neural Information Processing Systems 33 (2020), Barannikov, S et al. "Manifold Topology Divergence: a Framework for Comparing Data Manifolds." Advances in Neural Information Processing Systems 34 (2021), Barannikov, S et al. "Representation Topology Divergence: A Method for Comparing Neural Network Representations." ICML (2022).

Author response:
**W5. Related work section does not mention vast literature on geometry and topology applications in analysis of data representations.**

We have revised our paper with additional related work and detail comparison with our results. See General Comment KR2 in the General comments.

Reviewer comment (Presentation):
Page 1: The figure appearing on the article first page usually serves to highlight the principal contribution of the paper; does the standard Figure 1 with very well-known XOR dataset really serve such purpose?

Author response:
**● page 1: The figure appearing on the article first page usually serves to highlight the principal contribution of the paper, does the standard Figure 1 with very well-known XOR dataset really serve such purpose?**

We agree with the reviewers' comment, and have changed Figure 1 to highlight our contribution. We believe this will engage readers in understanding our motivation and contributions.

Reviewer comment (Writing):
Page 2: "considering the given dataset as a topological space" → considering the support of the data distribution as a topological space?

Author response:
**● page 2: "considering the given dataset as a topological space" -> considering the support of the data distribution as a topological space?**

We considered the data distribution $\mathcal{X}$ as a topological space. In practice, it refers the manifold of each class in the dataset.

Reviewer comment (Reproducibility):
Page 2: "We answer this question by constructing a collection of convex polytopes..." – where is it described in the paper how to construct such collection?

Author response:
**● page 2: "We answer this question by constructing a collection of convex polytopes..." - where is it described in the paper how to construct such collection?**

The sentence would be corrected to, "We answer the question when a polytope-basis cover of $\mathcal{X}$ is given." For the construction of these covers, please refer to Section 5 of the revised manuscript.

Reviewer comment (Theory):
Page 2: "forms m-simplicial complex..., we establish a novel topology-dependent bound" – the bound established in Theorem 3.6 concerning simplicial complexes is in terms of numbers of j-dimensional facets, which are not "topology-dependent" quantities.

Author response:
**● page 2: "forms m-simplicial complex..., we establish a novel topology-dependent bound" - the bound established in Theorem 3.6 concerning simplicial complexes is in terms of numbers of j-dimensional facets, which are not "topology-dependent" quantities.**

Indeed, rigorously speaking, the dimension of a simplicial complex is not a topology-dependent quantity, despite the use of simplicial complexes to explore topological properties. What we aimed to highlight was the demonstration that the bound depends on the dimension of the simplicial complex, which may be related to the topological complexity of $\mathcal{X}$. In response to your insightful suggestion (W4 above), we have removed topological aspects in the revised paper.

Reviewer comment (Theory):
Page 4: "from the volume identity of the polytope" – what is it? A reference is necessary here.

Author response:
**● page 4: "from the volume identity of the polytope"- what is it? a reference is necessary here.**

Thank you for a good comment. It means that the volume of a convex polytope is sum of the volume of small pyramids which is a partition of the polytope, which we only described in the proof in Appendix B.1.1. We allocated more space to explain this proof idea in the main body, in the revised version.

Reviewer comment (Theory):
Page 6: "the first result on the width of neural networks in terms of topological data structure" – the result in Theorem 3.6 is in terms of numbers of j-dimensional facets, which is not "in terms of topological data structure".

Author response:
**● page 6: "the first result on the width of neural networks in terms of topological data structure"- the result in Theorem 3.6 is in terms of numbers of j-dimensional facets, which is not "in terms of topological data structure"**

This is similar to one of the previous comments above. We initially considered it to be related to topological data structure, but we have removed some sentences that might be understood as an overstatement.

Reviewer comment (Theory):
Page 7: "This implies that the architecture outlined in Theorem 3.7 is dictated by the filtration parameter" – the topological space considered in Theorem 3.7 is a convex polytope with disjoint prism-shaped convex polytopes removed; it is not the Cech complex with respect to some parameter epsilon, so there is no architecture in Theorem 3.7 related to the filtration parameter.

Author response:
**● page 7: "This implies that the architecture outlined in Theorem 3.7 is dictated by the filtration parameter"- The topological space considered in Theorem 3.7 is a convex polytope with disjoint prism-shaped convex polytopes removed, it is not the Cech complex with respect to some parameter epsilon, so there is no architecture in Theorem 3.7 related to the filtration parameter.**

Thank you for the significant comment. We decided to remove this part from our draft.

Reviewer comment (Writing):
Page 8: "which completely classifying the given dataset" – perhaps, which classifies with zero error the given dataset?

Author response:
**● page 8: "which completely classifying the given dataset " - perhaps, which classifies with zero error the given dataset?**

The sentence means the classifier with zero error on the given dataset. We have modified the sentence in the revised version.

Reviewer comment (Theory):
It is often difficult for real data sets to satisfy Assumption 4.2 since the polytopes separating the clusters may not be convex. Are there any definitive modifications to the proposed convergence guarantees [Theorem 4.3] that the authors can suggest? How does the increase in the number of classes, and hence perhaps overlapping polytopes add to the complexity of the construction [Page 21]?

Author response:
>**Q1. How Theorem 4.3 can be improved, for example, when the dataset is non-convexly separated or the number of classes is greater than 2?**

Theorem 4.3 explains that if a two-layer ReLU network is initialized in proximity to the polytope cover, gradient descent may converge to the global minimum. This result readily extends to multiple polytopes or a polytope-basis cover.
Specifically, consider a scenario with $m$ polytopes covering a single class and a three-layer network $\mathcal{T}$ as defined in Proposition 3.7, equation (5) in the revised paper. Assuming the $m$ subnetworks, denoted as $\mathcal{N}_i$, are initialized near polytopes, satisfying the conditions in Theorem 4.3.
Then, we can apply Theorem 4.3 to each polytope individually. 
This is possible because $\mathcal{T}$ is the minimum of $m$ two-layer ReLU networks - it is important to note that the $\min$ operation in the network executes all gradients in backpropagation, except only one subnetwork for each input. This establishes the extension of Theorem 4.3 to a polytope-basis cover for three-layer ReLU networks.

Reviewer comment (Theory):
The optimality of $3$-layer ReLU networks for approximating indicators is clear from Proposition C.1 and C.2. Is it true for high-dimensional compactly supported functions ($f: R^d \to R^l$), $l>1$ in general, perhaps of some regularity in terms of smoothness? This seems crucial as there are numerous UA bounds for Lipschitz maps using ReLU feed-forward networks.

Author response:
>**Q2. Is it possible to obtain some UAP results of the 3-layer ReLU networks to approximate a compactly supported functions $f:\mathbb{R}^{d_x} \rightarrow \mathbb{R}^{d_y}$, in terms of smoothness?**

Thank you for a insightful question. We obtained that it is possible. 
Since we proposed a method to approximate the indicator function in $\mathbb{R}^d$ by a 3-layer ReLU network, a common idea in Lebesgue theory easily generalizes this result to approximate a compactly supported functions $f:\mathbb{R}^{d_x} \rightarrow \mathbb{R}^{d_y}$.
We propose the result in the appendix. See Theorem C.5 in the revised paper.
As the reviewer anticipated, the width is related with the error bound and the Lipshictz constant of $f$.

Remark. Notably, (Wang et al., 2022) recently proved that 2-layer ReLU networks cannot approximate a compactly supported function in $L^p(\mathbb{R}^d)$, while 3-layer ReLU networks can. However, their proof establishes only the existence of such networks without offering any width bounds. 
Consequently, our result in Theorem C.5 represents a refinement of their findings. 
This is one example that our results on polytope can be leveraged to derive width bounds for the UAP. 

**Reference.**

[Wang et al., 2022] Ming-Xi Wang and Yang Qu. Approximation capabilities of neural networks on unbounded domains. Neural Networks, 145:56–67, 2022.

Reviewer comment (Experiments):
Can the authors comment on how well the prescriptions regarding architectures hold up against simpler real datasets? As a practitioner, it is often frustrating to witness theoretical suggestions underperforming significantly. For example, to my knowledge, there exists no consistent method of estimating Betti numbers corresponding to even simpler real data distributions, if they at all have punctured supports.

Author response:
>**Q3. How can your results be applied to real datasets?**

In the revised paper, we supplied empirical application of our results, by constructing a polytope-basis cover of the given real  dataset.
See General Comment KR1 in the overall comments.

Reviewer comment (Evaluation):
The metrics in the paper used to quantify geometric structure in the input data (which the theoretical results reflect in terms of the rates for the network widths to achieve the feasible architecture property) seem that they may be hard to compute in moderate dimensions (presumably, one needs to fit a simplical complex to data, and calculate Betti numbers from it). This means it may be hard to verify the theory in cases beyond the low-dimensional examples highlighted in experiments. I hope the authors will correct me if I am mistaken here, and clarify this in the revision.

Author response:
>**W1. The geometric structure of the given dataset (the simplicial complex or Betti numbers) looks hard to be achieved in real dataset.**

To address your concern, in the revised paper, we present a way to find a polytope-basis cover for the given real dataset. See General Comment KR1 for more details of real experimental results.

Reviewer comment (Writing):
The non-technical writing in the paper (in contrast to the mathematical writing, highlighted above) suffers from a lack of precision in many areas. I would recommend rewriting the abstract to be more in line with the tone of the rest of the paper, tuning the first sentence of the introduction (I do not think this claim is universally accepted -- arguably, the ability to (efficiently) learn these networks is of far greater importance for understanding the successes of deep learning), and generally proofreading for typos.

Author response:
>**W2 & W3. The abstract and certain sentences in introduction is highly recommended to be revised. Further, there are some lack of introducing related works and detail comparison.**

We thank for suggesting relevant references and valuable feedback. We have revised Section 1 and 2 with additional references.
See General Comment KR2 in the overall comments.

Reviewer comment (Theory):
Can the authors clarify the reason for the focus on representing indicators for the relevant polytopes $\mathcal{X}$ (in the kind of $L^\infty$ sense mandated by Definition 3.1), and the limitations of this framework for general problems of interest? It seems to me that it might be too "hard" of a problem to characterize sufficient architectural configurations to fit data distributions if one's end goal is a machine learning task such as classification (for example, generally, one could classify $\mathcal{X}$ without exactly representing $\mathbb{1}(\mathcal{X})$). It also seems to me that representing $\mathbb{1}(\mathcal{X})$ may not be sufficient to solve general nonparametric regression tasks, i.e. to enjoy universal approximation of various nonparametric classes defined on $\mathcal{X}$ (please correct me if I am mistaken). A significant amount of work has been done in the latter setting, specifically on manifolds, which does not seem to have been discussed (e.g., [4] and many works by the same authors).

Author response:
>**Q1. Why did the authors choose to approximate the indicator function?**

Thank you for the valuable feedback. Indeed, approximating the indicator function is generally more difficult than just classifying finite classes in the dataset. However, if a neural network $\mathcal{N}$ can effectively approximate the indicator function on a given dataset class $\mathcal{X}$, it inherently possesses the capability to classify $\mathcal{X}$ from other classes. We intended to set the goal of our paper to cover not only classification tasks, but also regression tasks. We have indeed considered the MSE loss function in Section 4 (see Theorem 4.3), and we have included a simple approximation result in Theorem C.5, requested by Reviewer nRVF. See footnote 1 on page 4 of the revised manuscript for more detail.

Reviewer comment (Writing):
In Section 3.1, it is not exactly explicitly defined what an "architecture" is (relevant to understanding $\mathcal{A}$ in Definition 3.1), but it seems from context that it is a fixed choice of inter-layer maps and in particular of hidden layer dimensions. A limitation of this definition (c.f. footnote 1) seems to be that in general, universal approximation of various nonparmetric classes can only be enjoyed with neural networks when the hidden layer dimension is allowed to grow -- in other words, the "architecture" involves only (in a sense) the computational graph of the neural network, rather than particulars (such as input and output dimensions) about the maps corresponding to "edges" in the graph. Could the authors clarify this difference, and why they have opted to define an "architecture" in this way?

Author response:
>**Q2. What does the terminology `architecture' (denoted by $\mathcal{A}$) exactly mean?**

In our manuscript, the terminology `architecture' refers to the structure of neural networks, specifically the depth and width of each hidden layer. We introduced a notation, $d \rightarrow d_1 \rightarrow \cdots \rightarrow d_k \rightarrow 1$, to represent a $k$-layer neural network with widths $d_1, \cdots, d_k$ in the manuscript, and we did not fix the network architecture. 

As you commented, the UAP result can be achieved when the the hidden layer dimension is allowed, as presented in Theorem C.5.
More precisely, the hidden widths grows $O(\varepsilon^{-d_x})$ as the error bound  $\varepsilon$ reduces. Similarly, our other results (Theorem 3.2, 3.4, 3.5, 3.6) imply that the width increases as the number of polytopes and their faces increases. Precisely, Theorem 3.5 explicitly demonstrates how the width grows with the increasing complexity of the simplicial complex.
We have made this part clearer in the revised manuscript (see Section 3.1 and Definition 3.1 of the revised paper).