Paper: Characterizing ResNet's Universal Approximation Capability (ICLR-2024)

Reviewer comment (Writing):
Lack of Explanation in the "Perspectives on the Curse of Dimensionality" paragraph need clarification/explanation/motivation. The way this section is written leaves the reader, not familiar with constructive approximation, in the dark, thinking some "magic is happening" where "there be dragons". The curse of dimensionality is not only something that MLPs experience, but every (reasonable) class of function approximators which are uniformly approximating in the Lipschitz ball on some compact subset of a metric space. This is due to metric entropy limitations and the result is essentially due to Kolmogorov [1]. The authors should add one sentence at the end of page 2 to explain how this is a generic problem, and that the reader should expect this to always be the case. By the same token, the reason people consider specialized classes of functions as a viable way to circumvent the curse of dimensionality is also rooted in metric entropy number arguments. References [1] and [2] show that the metric entropy (a non-linear notion of comprehensibility of a function class), or linear notions such as Kolmogorov linear widths, are not cursed by dimensionality for smooth classes. For example, the metric entropy of the unit ball in \( C^k \) with respect to the uniform topology is \( \Theta(\varepsilon^{-r/d}) \) so one can expect that the optimal (non-linear) approximator can achieve such rates. These ideas should be communicated atop page 3 so that it does not seem that considering "special function classes" is a "magic trick". The relationship between linear and non-linear measures of compressibility is given in Carl's paper [3]. There is a similar reason justifying why MLPs with super expressive activation can circumvent the curse, namely due to the (near) discontinuity of these classes as a function of their parameters [4]. This explains why these approaches work but also why they should not be employed, since beating the curse this way is only possible with models whose input-output relation changes chaotically with tiny parameter perturbations, resulting in dramatically different functions. This is evident in Theorem 1 of [5]. The reviewer strongly feels that these points should be communicated to the reader, even concisely, to give a meaningful and critical discussion on the curse of dimensionality and its trade-offs: choosing between an unstable model class in its parameters or a restriction of the functions that can be effectively approximated.

Author response:
**Q1:Can you please incorporate the discussion I added above, in the weaknesses. I think the explanation of the curse of dimensionality is not satisfactory nor insightful, but it can be made 100\% by adding the above points.**

Response to Q1: We agree with your very helpful suggestions, and we have thoroughly implemented them in the revised manuscript  (see **Sec. 1.1, pages 2-3**). 

Specifically, we have made several modifications according to your suggestions to **the part 'Perspective on the curse of dimensionality'** in **Sec. 1.1, on pages 2-3**, to clarify and improve our discussion. 

**Response to comment 1.1**: *...I think the authors should add 1 sentence at the end of page 2 to explain how this is a generic problem, and that the reader should expect this to always be the case.*

We have added a sentence **at the end of the first paragraph** to tell readers that the curse of dimensionality is a pervasive challenge in function approximation, not exclusive to MLPs, resulting from metric entropy restrictions. 

**Response to comment 1.2**: *...These ideas should be communicated to the reader atop page 3 so that it does not seem that consider "special functions classes" is a "magic trick"*

We have expanded our discussion **at the beginning of the second paragraph** to explain why we consider specialized function classes as a strategy to mitigate the curse of dimensionality. This rationale is grounded on metric entropy number arguments, and we appreciate your suggestion to highlight this. 

**Response to comment 1.3**:*There similar reason justified why the MLPs with super expressive activation in the following paragraph. Namely, due to the (near) discontinuity of these classes, as a function of their parameters....*

Addressing your comment on the role of super-expressive activation functions, we have added a sentence **in the third paragraph.** We emphasize that these activation functions can potentially aid neural networks to overcome the curse of dimensionality due to their ability to become discontinuous function approximators. Finally, **at the end of the third paragraph**, we have pointed out that the choice between unstable model class approximators and constrained target function classes is a necessary balance to avoid the curse of dimensionality.

We believe these revisions substantially improve the paper quality. Thank you very much again!

Reviewer comment (Theory):
Suboptimal Rates Due to Polynomial Approximation and Not Quantization. The proof strategy is akin to [6] (or its quantitative version in Proposition 53 - [7]) where one first approximates polynomials using the network class (here ResNet, and in those papers deep but narrow MLPs), then relates the polynomials to the target function being approximated (e.g., via quantitative Stone-Weierstrass theorems using Bernstein polynomials). However, due to the "triangle inequality"-based argument, these rates are suboptimal compared to the more modern quantization-based arguments of [8], which are possible for structures with recurrence/depth like MLPs and ResNets. For this reason, the rates derived are suboptimal by an exponential factor of \( 1/2 \). For example, in Theorem 3.3 [9], the author exhibits an MLP with ReLU activation achieving the optimal depth of \( \mathcal{O}(\varepsilon^{-r/(2d)}) \) to approximate any function in the unit ball of \( H^r \), as opposed to the current manuscript’s ResNet guarantee of \( \tilde{\mathcal{O}}(\varepsilon^{-r/d}) \).

Author response:
**Comment 2**: *Suboptimal Rates Due to Polynomial Approximation and Not Quantization.*; and

**Q2: Why not deploy the quantization argument of [8] instead of approximating polynomials as in [6] (or its quantitative version in Proposition 53 of [7])?**

Response to Comment 2 and Q2: We appreciate your comments regarding the suboptimal rates in our work, particularly in comparison to modern quantization-based arguments. We acknowledge the limitations of the approach adopted in our paper and would like to clarify our initial selection of polynomial approximation was motivated by its capacity to construct a b-ResNet (ResNet with constant width) that achieves the approximation rate. In our view, this represents an intriguing result with potential independent interest. Furthermore, we can utilize this result to demonstrate that the upper bounds will experience a reduction by a factor of 'd' (as per Theorem 3).

However, we concur that pursuing the optimal approximation rate using ResNet represents a compelling research direction. Given the quantization-based arguments, we hypothesize that ResNets might also attain the optimal order, a topic we plan to explore in future work. We have addressed this point in our revised manuscript, specifically in the final paragraph of **Section 4.3, Page 8**, acknowledging the potential for improved rates with alternative methodologies.

Reviewer comment (Presentation):
Minor comment: Some figures, e.g., Figure 6 in the appendix, seem not to be exported as PDF or EPS files, making them pixelated. This should be fixed.

Author response:
On a separate note, in response to your minor comments on the issues related to the figures, we have address the concerns for some figures (see e.g., Fig. 6 in the updated supplementary materials). As it is a bit time consuming to, we decided to update those we have done first and continue to refine the remaining figures. We will be able to improve all the figures in the final version.

Reviewer comment (Theory):
For the lower bound (Theorem 2), it is not very useful given the product function is limited. The authors give a comprehensive discussion but the result extending this bound to the space of continuous functions is missing. It is good to discuss existing results, but a comparison is expected.

Author response:
**Comment 2**: *For the lower bound (Theorem 2), it is not very useful given the product function is limited. The authors give a comprehensive discussion but the result extending this bound to the space of continuous functions is missing. It is good to discuss existing results, but a comparison is expected.*

Response: Thanks for your attention to the details. We first note that the results in Theorem 2 establish a tight lower bound for NN to approximate product functions, and hence for polynomial functions. Such a tight lower bound is crucial for showing the upper bound of ResNet's approximation of polynomial functions in Theorem 4 is order optimal (in `ε`) and such a tight lower bound for NN approximation of polynomial functions is not available in the literature and directly using other lower bounds developed for general functions, e.g., those for continuous functions, does not serve our purpose. 

To this end, we would also like to clarify a confusion probably due to the way we organize Theorem 2 and the discussions below it. Our intended logic flow in Sec. 3 (Lower bounds on ResNet's approximation capability), after showing the ResNet can be implemented by an FNN (with extra tunable weights), is to (i) state that all lower bounds for FNN's approximation capability for different function classes can then be easily adapted to ResNets, and (ii) for a special class of functions, polynomials, we derive a tight lower bound. Such a lower bound matches the upper bound in Theorem 4, which will be mentioned following theorem 4. 

Last, we would like to note that there seems little room for extending our tight lower bound for polynomial functions to more general functions, e.g., continuous functions, as the existing lower bounds for continuous function is already proven to be order optimal (ref here). Nonetheless, this may be some room in improving the constants in the lower bounds, and we have included it as an interesting future direction in the conclusion section of the revised manuscript. 

We have revised the **Sec. 3** in the revised manuscript to make the above points clear and avoid potential future confusion; see **page 5-6**. Many thanks for your helpful comments!

Reviewer comment (Theory):
In Theorem 6, the upper bound for approximating continuous functions grows with the input dimension given that the width is bounded by $(d+1)$, provided the depth is sufficiently large. Such dependency on the input dimension is usually not friendly given that many neural networks in applications directly work on raw data. A dimension-independent bound (reference below) can be derived when the number of pieces in the continuous piecewise linear function is known.

Author response:
**Comment 3**: *In Theorem 6, the upper bound for approximating continuous functions grows with the input dimension given that the width is bounded by `d+1`, provided the depth is sufficiently large. Such dependency on the input dimension is usually not friendly given that many neural networks in applications directly work on raw data. A dimension-independent bound (reference below) can be derived when the number of pieces in the continuous piecewise linear function is known.*

*Chen, Kuan-Lin, Harinath Garudadri, and Bhaskar D. Rao. "Improved bounds on neural complexity for representing piecewise linear functions." Advances in Neural Information Processing Systems 35 (2022): 7167-7180.*

Response: Thank you for your thoughtful comment and the suggested reference. We've thoroughly reviewed the paper you mentioned and conjecture ResNet will have a similar result that a dimension-independent bound could be derived when the number of pieces in the continuous piecewise linear function is known. This indeed offers an interesting perspective on our results. In response to your feedback, we have included this limitation and related discussion following Theorem 6 (**Page 8**) in the updated version of our paper. We believe this will enrich our analysis and provide a more comprehensive view to our readers.

Reviewer comment (Theory):
On the other hand, the depth is not explicitly given in the theorem, which even reduces the significance of the statement. It would be clearer if the authors could explain this missing part in the discussion following Theorem 6.

Author response:
**Comment4**: *On the other hand, the depth is not explicitly given in the theorem, which even reduces the significance of the statement. It would be clearer if the authors could explain this missing part in the discussion following Theorem 6.*

Response: Thanks for your suggestion. The depth L=$O(Md)$ where $M$ is an $f$-dependent parameter. Please refer to Thm 6. and the clarification and discussion have been added in the revised manuscript following **Theorem 6, Page 8.**

Reviewer comment (Theory):
The function space considered in Theorem 8 is very limited. Such a space is even smaller than the space used by Theorem 4.1 in the following paper.

Author response:
**Comment 5 and 6**: *The function space considered in Theorem 8 is very limited. Such a space is even smaller than the space used by Theorem 4.1 in the following paper. In the above paper, the space of functions is defined based on the refinement levels of the inner functions which can be used to bound the Lipschitz constant of the outer functions. However, Theorem 8 directly limits the Lipschitz constant of the outer function. This assumption is unrealistic, and it avoids the main difficulty in approximating the outer function.*

A: We acknowledge this function space in Theorem 8 is limited. The motivation behind our approach in Theorem 8 was to explore conditions under which we could overcome the curse of dimensionality by using ResNet to approximate the inner and outer functions separately. In this context, limiting the Lipschitz constant of the ResNet function appeared to be a viable strategy, which led us to directly constrain the Lipschitz constant of the outer function.

We understand that this assumption may seem unrealistic and bypasses the central challenge in approximating the outer function. Your feedback gives us a valuable perspective on this, and we have provided further discussion in the revised version of our paper. Please refer to **Paragraph 2 following Theorem 7, Page 8-9.**

Reviewer comment (Theory):
In the above paper, the space of functions is defined based on the refinement levels of the inner functions which can be used to bound the Lipschitz constant of the outer functions. However, Theorem 8 directly limits the Lipschitz constant of the outer function. This assumption is unrealistic, and it avoids the main difficulty in approximating the outer function.

Author response:
**Comment 5 and 6**: *The function space considered in Theorem 8 is very limited. Such a space is even smaller than the space used by Theorem 4.1 in the following paper. In the above paper, the space of functions is defined based on the refinement levels of the inner functions which can be used to bound the Lipschitz constant of the outer functions. However, Theorem 8 directly limits the Lipschitz constant of the outer function. This assumption is unrealistic, and it avoids the main difficulty in approximating the outer function.*

A: We acknowledge this function space in Theorem 8 is limited. The motivation behind our approach in Theorem 8 was to explore conditions under which we could overcome the curse of dimensionality by using ResNet to approximate the inner and outer functions separately. In this context, limiting the Lipschitz constant of the ResNet function appeared to be a viable strategy, which led us to directly constrain the Lipschitz constant of the outer function.

We understand that this assumption may seem unrealistic and bypasses the central challenge in approximating the outer function. Your feedback gives us a valuable perspective on this, and we have provided further discussion in the revised version of our paper. Please refer to **Paragraph 2 following Theorem 7, Page 8-9.**

Reviewer comment (Evaluation):
Since the approximation quality is measured by the uniform norm, it seems not reasonable to measure the mean squared error. Measuring the maximum error could be more convincing.

Author response:
**Comment 7**: *Since the approximation quality is measured by the uniform norm, it seems not reasonable to measure the mean squared error. Measuring the maximum error could be more convincing.*

A: Your feedback on our choice of error measurement is valued. The reason we initially opted for the mean squared error is its widespread acceptance and preferred use in practice. However, we understand the limitations you have pointed out and agree that measuring the maximum error could provide a more convincing evaluation of our approach. We have supplemented our experimental results with measurements using the max error in the revised version of our manuscript (see **Sec. 6, Page 9** and Appendix H).

Reviewer comment (Novelty):
The most important weakness is with respect to novelty and technical innovation.

Author response:
**1) Concerns about the novelty and technical innovation.**

Response: Thanks for the note and we clarify in the following. Our paper builds upon existing techniques with new ideas to establish the universal approximation capability of ResNet FNN, addressing an important research gap in the literature and add to the theoretical justification of ResNet's excellent practical performance. While some techniques used in our paper are known in prior works, most proof ideas for existing approximation bounds are derived for networks without exploiting the skip connections unique to ResNet FNN. Our work extends these techniques to the case with skip connections, which involve some non-trivial development.

Particularly, in our construction, we leverage the identical mappings given by the skip connections in the construction of ResNets to approximate multivariate functions, which is the key to show that b-ResNet (ResNet with constant width) has powerful expressivity. This is one of our novel technical innovations in this paper.

Furthermore, We also prove that ResNet with single-neuron can approximate any continuous piece-wise linear function. The result is a non-trivial extension of the result in Lin & Jegelka (2018), which shows ResNet with single-neuron can approximate step functions. We also show how ResNet can be incorporated into a novel NN architecture suggested by the KST theorem to efficiently approximate a class of smooth functions.

All the above results are the first in the literature, addressing important research gaps and adding to the literature on ResNet's performance analysis.

Reviewer comment (Novelty):
The main concern is that many of the ideas/constructions upon which the paper relies to prove their results have been known in prior works. This is the case with the proposed lower bound, for example.

Author response:
**1) Concerns about the novelty and technical innovation.**

Response: Thanks for the note and we clarify in the following. Our paper builds upon existing techniques with new ideas to establish the universal approximation capability of ResNet FNN, addressing an important research gap in the literature and add to the theoretical justification of ResNet's excellent practical performance. While some techniques used in our paper are known in prior works, most proof ideas for existing approximation bounds are derived for networks without exploiting the skip connections unique to ResNet FNN. Our work extends these techniques to the case with skip connections, which involve some non-trivial development.

Particularly, in our construction, we leverage the identical mappings given by the skip connections in the construction of ResNets to approximate multivariate functions, which is the key to show that b-ResNet (ResNet with constant width) has powerful expressivity. This is one of our novel technical innovations in this paper.

Furthermore, We also prove that ResNet with single-neuron can approximate any continuous piece-wise linear function. The result is a non-trivial extension of the result in Lin & Jegelka (2018), which shows ResNet with single-neuron can approximate step functions. We also show how ResNet can be incorporated into a novel NN architecture suggested by the KST theorem to efficiently approximate a class of smooth functions.

All the above results are the first in the literature, addressing important research gaps and adding to the literature on ResNet's performance analysis.

Reviewer comment (Theory):
A related interesting question is about the depth-width tradeoffs for ResNets. Perhaps this follows from the work on just the tunable weights, but as far as can be told it is not obvious. Given that there are many results for FNNs and their depth/width tradeoffs, it is important to highlight this too.

Author response:
**2) Questions about the depth-width tradeoffs for ResNets.**

Response: Thanks for this insightful question! We are well aware of the depth-width trade-off in FNN's universal approximation capability as in the classical papers [1,2,3].

In this paper, we focus on a b-ResNet (ResNets with constant width) and show even this simple ResNet can achieve strong universal approximation capability. We briefly discuss the performance of deep ResNet and shallow ResNet following theorem 3, (at the 'Deep vs. shallow' bullet point, **Page 6-7**), in that deep ResNet can achieve better approximation error than the shallow one on the approximation of monomials/polynomials but maybe the function space is limited. We believe that a more thorough investigation on ResNet's depth-width tradeoffs is both practically important and theoretically interesting, and we have noted it as a future direction in the revised version of the paper.

[1] Shiyu Liang and Rayadurgam Srikant. Why deep neural networks for function approximation? arXiv
preprint arXiv:1610.04161, 2016.

[2] Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:
103–114, 2017

[3] Safran, Itay, and Ohad Shamir. Depth-width tradeoffs in approximating natural functions with neural networks. International conference on machine learning. PMLR, 2017.