Paper: Topological Expressive Power of ReLU Neural Networks (ICLR-2024)

Reviewer comment (Theory):
Lower bound is obtained only for some specific weights of neural network. However, given a neural network architecture there are always weights that produce constant function and thus have strictly zero Betti numbers, so the meaning of paper's "lower bound" term is not quite clear.

Author response:
We are motivated to define the topological expressive power of an architecture as the representation of the "most complex" topology achievable by a neural network with that architecture. Consequently, our construction provides a lower bound for this concept.

Reviewer comment (Theory):
It is not clear whether the constructed network weights can be found via regular optimization algorithms.

Author response:
Proposition 10 suggests that in the parameter space, there are open neighborhoods around the weight vectors that arise from our construction such that every point in these open neighborhoods also represent functions with Betti numbers at least our bounds. Therefore, it is possible to find the weight representation of such a function in the parameter space by e.g. gradient descent methods.

Reviewer comment (Reproducibility):
The calculation of Betti numbers is difficult, which undermines the practical implications of the work.

Author response:
The main goal of this paper is not to suggest techniques relevant for practice, but rather to point out at theoretical barriers (or capabilities) of expressivity of a neural network. Furthermore, the topological theoretical insights might have connection to other aspects of deep learning, see e.g. the discussion with reviewer UFRF.  Moreover, Betti numbers are indeed used in practice, at least in dimensions up to 2.

Reviewer comment (Theory):
The claim of the exponential gap is somewhat unclear in the paper.

Author response:
The ''exponential gap'' refers to the fact that only exponentially sized shallow neural networks can encapsulate the Betti numbers that are already achievable by linear sized deep neural networks.  We can gladly elaborate further if the reviewer wishes to.

Reviewer comment (Theory):
The upper bound proof lacks some details; only about half a page is devoted to upper bounds, while the paper is mostly concerned with lower bounds.

Author response:
We have tried to keep the lower bound construction self-containing, which required multiple pages since definitions and figures had to be introduced. On the other hand, we have included a sketch for our upper bounds, missing steps being standard topological arguments. These can be found in the appendix.

Reviewer comment (Novelty):
The lower bounds found by the authors are similar to previous ones that have appeared in the literature, e.g., in Bianchini and Scarselli (2014). For example, the principal zeroth Betti number result is an extension of the loc.cit to ReLU activations.

Author response:
While the extension from sigmoid activation to ReLU activation may seem incremental, it is still a significant contribution since the ReLU activation function is widely used in modern neural networks. Neither in Bianchini-Scarselli nor in our case is it immediately clear that one must imply the other.  Moreover, we have extended the result to any Betti number, answering the question whether the expressivity results hold as well for ``holes'' affirmatively for any dimension.

Reviewer comment (Experiments):
The paper needs to show concrete real world examples with practical meaning of the paper's results.

Author response:
The main goal of this paper is not to suggest techniques relevant for practice, but rather to point out at theoretical barriers (or capabilities) of expressivity of a neural network. Furthermore, the topological theoretical insights might have connection to other aspects of deep learning, see e.g. the discussion with reviewer UFRF.  Moreover, Betti numbers are indeed used in practice, at least in dimensions up to 2.

Reviewer comment (Theory):
The reviewer encourages the authors to spell out the precise meaning of their notion "lower bound of" in concrete mathematical terms. It seems that this is the lower bound of the maximum of a certain quantity taken over some set, but what exactly is the set over which the maximum is taken? Does this maximum depend on the architecture only, or also on the dataset \(X\)? If it depends on the dataset \(X\), what is the meaning/effectiveness of this dependence on \(X\)?

Author response:
The intersection $X \cap F^{-1}((-\infty,0])$ does not coincide with $X^-$. The former represents the subset of $X$ mapped to the non-negative real line by $F$, while the latter is the set of negatively labeled data points. The construction of the lower bound is applicable in such a setting as well: If $k$ is the dimension of the manifold $X$, we have a $k$-dimensional cube in $X$, where our construction can be applied in the same manner. We acknowledge that the upper bound is indeed not directly applicable and depends more on the properties of the manifold $X$. Investigating upper bounds depending on $X$ would be an interesting follow-up research direction for our results. Furthermore, we want to emphasize that we do not propose a method for immediate practical applications. Instead, our focus is on exploring the fundamental nature of ReLU neural networks. We believe that our results, in their generality, contribute to theoretical applications, such as offering a stronger depth-distinguishing inapproximability result, as discussed with reviewer UFRF.

Reviewer comment (Novelty):
The reviewer encourages the authors to try to find a practical real-world case in which their results bring a novel perspective.

Author response:
Furthermore, we want to emphasize that we do not propose a method for immediate practical applications. Instead, our focus is on exploring the fundamental nature of ReLU neural networks. We believe that our results, in their generality, contribute to theoretical applications, such as offering a stronger depth-distinguishing inapproximability result, as discussed with reviewer UFRF.

Reviewer comment (Novelty):
The key takeaway of the paper is that depth is more important than width, and the paper proves this elegantly via the Betti numbers. However, similar depth-width tradeoffs were already known using different techniques and different connections, meaning that in some sense it was already known that depth is exponentially better than width. For example, Telgarsky's works used a basic triangle construction and measured complexity via the number of linear regions, and Montufar et al. also used the number of linear regions to show that depth is more important.

Author response:
Regarding the second question, for $d=1$, our bounds imply the depth separation given by Telgarsky and in fact even for shallower networks (i.e., the functions inapproximable by networks of depth $k$ and polynomial size(in $k$) can be represented by networks with a depth and size of $O(k^2)$). Since we showed that our construction is robust with respect to small perturbation, this separation holds for a full Lebesgue measure set of neural networks.
We believe that the reason why the bounds obtained in this way are slightly better is due to the simplicity of the analysis. Since all our connected components have a constant size and contain a point with a constant negative function value, the inapproximability by small shallow neural networks follows easily by the fact that they are not capable of expressing the same number of connected components.
For general $d$, our construction differs from the one of Telgarsky in the way that we apply the triangle map in every coordinate and make use of the arising subdivision of the unit cube.

Reviewer comment (Novelty):
There are generalizations of Telgarsky’s works that use connections to dynamical systems (Li-Yorke chaos, periodic orbits) and the notion of topological entropy, as in [1], [2], and [3]. Papers [1] and [2] provide lower bound constructions using more general functions than Telgarsky’s triangle, and [3] characterizes expressivity using topological entropy.

Author response:
We would like to thank the reviewer for the insightful review and especially for pointing at potential connections between our construction and concepts in dynamical systems used in the referenced works, as well as to the construction in Telgarksys work.

Regarding the first question, there are connections between our construction and the ones used in the mentioned references. For example, the coordinates of the cutting points (the points in our constructions that are the center of the resulting annuli) are non-periodic points of the simple triangle function(they are the points that get mapped to the fixed point $0$ after a certain number of application). Although this observation suggests a potential link between our construction and concepts in dynamical systems, we are not seeing a connection to the notion of topological entropy so far.

Reviewer comment (Writing):
I don't understand the notation $\beta_0(F) \in \Omega(M^d \cdot n_L)$. Is it the same as $\beta_0(F) = \Omega(M^d \cdot n_L)$ ? (that is, $C_1 M^d \cdot n_L \le \beta_0(F) \le C_2 M^d \cdot n_L$

Author response:
One standard definition of Landau symbols which is used often in complexity theory is that, given a function $f$, the terms $O(f)$ resp. $\Omega(f)$ denote sets for which $g=O(f)$ resp. $g=\Omega(f)$ (as the reviewer mentioned) would hold, see e.g. https://en.wikipedia.org/wiki/Big_O_notation#Matters_of_notation. We can, however, gladly switch to the equal sign, as it seems to be the better-known variant.

Reviewer comment (Experiments):
The most of the paper is dedicated to the proof of the **existence** of a network with a given topological expressivity. But in deep learning we are interested in a practical algorithm for finding such a network. The manuscript will benefit from computational experiments. You can use simple synthetic datasets with known Betti numbers (like in Naitzat et al. (2020)) and estimate depth/width of a network which is able to classify it with accuracy > 0.95, for example.

Author response:
Proposition 10 of the paper implies that there exist positive radius neighborhoods around each of our constructed functions that satisfy the same properties with respect to Betti numbers of the sublevel sets. Therefore, efficient algorithms based on e.g. gradient descent methods can be used for the detection of functions that prove our lower bounds (which also answers the reviewer's question). That being said, we agree that experiments would be an enrichment to our theoretical findings.

Reviewer comment (Presentation):
The manuscript is very long (30 pages), the Appendix is dedicated to proofs. Maybe some journal will be a better destination for such a manuscript.

Author response:
We believe that the short version of the paper is self-contained (at least with respect to the definitions and constructions) and includes all our results, and is therefore suitable for a conference. The interested reader shall be able to find the appendix online in the future.

Reviewer comment (Theory):
You explicitly construct a network with a given topological expressivity, but are this network's weights reachable by gradient optimization?

Author response:
Proposition 10 of the paper implies that there exist positive radius neighborhoods around each of our constructed functions that satisfy the same properties with respect to Betti numbers of the sublevel sets. Therefore, efficient algorithms based on e.g. gradient descent methods can be used for the detection of functions that prove our lower bounds (which also answers the reviewer's question). That being said, we agree that experiments would be an enrichment to our theoretical findings.