From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper analyzes the capacity of narrow but deep CNNs to universally approximate shift-invariant functions, concluding that sufficiently deep CNNs with kernel and channel size greater than or equal to 2 can do so. The analysis is based on dynamical systems and flow maps and follows previous work for non-convolutional architectures.

The reviewers acknowledged that both the proofs and the results were non-trivial, and they found strength in the exposition and mathematical clarity of the presentation, though the intuition behind the approach may need further clarification. Some reviewers found the contributions to be marginal and somewhat incremental compared to prior work, and others questioned the practical relevance of infinite-depth limits.

During the discussion, it was suggested that the results are not surprising. This subjective judgement is debatable and does not fold into the decision; however, the underlying sentiment underscores the main weakness of the paper -- the results in this paper are unlikely to have significant impact on the field, even among theorists working in this direction. The conclusions do not break new ground, the techniques are not novel, and the practical implications are unclear. Having said that, it seems the results here are of high mathematical quality and might fit well at another venue.
justification_for_why_not_higher_score: The results and conclusions of this paper are fairly narrow and represent relatively incremental advances over prior work. The reviewers did not feel like this paper would have significant impact on the community.
justification_for_why_not_lower_score: N/A.

From: ICLR.cc/2023/Conference/Paper1291/Authors
title: A Point-by-point Reply to Reviewer iJgd (Part 2) 
comment: Q3(cont'd)

We now summarize the contribution of the paper listed in the comments and distinguish our contribution among them.
-  (Poggio et al. 2016) uses classical results (for smooth activation functions) to estimate the approximate rate of shallow neural networks. While in the deep part, they show that functions with special structures can be approximated well via deep neural networks. This work heavily relies on the hierarchical structure of functions, and the architecture used is fully connected. In our work, we consider convolutional approximations.
- (Okumoto et al. 2019) shows that with FCN and dilated CNN, the function with Besov-like regularity can be approximated even if the input data is of infinite dimension. The main problem addressed here is the infinite dimensionality, while the dilated convolution is used as a tool to extract data information, which mimics the harmonic analysis approach in the continuum. However, the use of FCN will break the shift invariant structure, which is considered in this work. 
-  (Schmidt-Hieber 2018) considers the neural networks with RELU activation functions in a statistical setting. They show that under some smoothness assumptions, the non-parametric regression of deep neural networks enjoys a nice bound on the prediction error. It again concerns fully connected neural networks, and the construction in the (approximation part of the) paper needs some layers must be wide enough. 
-  (Favero et al. 2021) analyzes the generalization error of the model via a kernel learning approach (teacher-student scenario). Some of its argument is empirical, and the kernel methods indicate an infinite-width relaxation. Here, we consider limited width/kernel sizes.
-  (Bietti 2022) also considers the generalization error of the model under a kernel method framework. It also focuses on shallow neural networks (one or two layers) and infinite width (such that the kernel method can be applied). 
-  (Xiao 2022) extends the result about the frequency of MLP to CNN setting, and also considers the generalization error of a convolutional neural network with infinite width. Again, the depth considered there is up to two layers.

The last three work considers a neural network model with one or two convolutional layers, and require that the width goes to infinity. In addition, all works listed here do not consider the approximation with respect to shift symmetry. In fact, this symmetry cannot be addressed there because they all use fully connected terminal layers. In contrast, our paper considers the approximation result that (1) uses depth (composition) for approximation (2) with limited width, and (3) is shift invariant.

References
Universality of deep convolutional neural networks. Zhou, ACHA 2020.
Deep Neural Network Approximation of Invariant Functions through Dynamical Systems. Qianxiao Li, Ting Lin, and Zuowei Shen. Arxiv: arXiv:2208.08707

From: ICLR.cc/2023/Conference/Paper1291/Authors
title: A Point-by-point Reply to Reviewer iJgd (Part 1)
comment: We thank the reviewer for their comments that helped improve our manuscript. Below is a point-by-point reply to the main issues and concerns raised by the reviewer. 

Q1. Can the result be used to improve some statistical rate?
- We agree that in addition to approximation error, it is very natural and useful to consider the statistical error (or generalization error) in the overall analysis of a machine learning model.
- Actually, as far as we know, few approximation or generalization results in the deep-but-narrow neural network (for layers greater than 3) have been established. This paper aims to solve the problem of the approximation property first and the statistical error is beyond the scope of this paper.

Q2. Benefits of this architecture, compared to full-width convolutions.
-  In this paper, we consider the case of constant, small kernel size. This is the most common setting in the practical application of deep convolutional networks. The work is devoted to explaining why the small kernel can still work, at least in the approximation sense. The approximation result with a small kernel is new to the community.
- To build such a result, the dynamical system approach is a way. To emphasize the expression/approximation power comes from the composition/depth, we limit the channel number and make the bounds sharp. For other kinds of architectures, since in general, they contain our model, the UAP can be directly established. In other words, our model can be regarded as a minimal model, in the sense that other families with practical settings can also possess UAP if they include our model as a subset.

Q3. Comparing the result to other result.

Thank you for these works which makes our literature review more informative. Some relative references have been added in the new version. The detailed contribution of each paper listed in the comments is summarized in Part 2 of the reply.

- The existing results around the approximation capabilities of convolutional neural networks can be categorized into several classes. One either (1) takes the kernel as full-width (same size as the input) (e.g. Li et al 2022) which is not often used in practice; (2) assumes a sufficiently large channel number in order to adopt some kernel learning methods (e.g. Bietti 2021); or (3) remove the nonlinear activation function and reduce to a linear approximation problem (e.g. Zhou 2020), then use complex fully connected terminal layer(s) to achieve approximation. In contrast, our result quantifies the minimal requirement of the width and the support number (and sacrifices the effective bound on the depth as compensation). Thus, our result cannot be derived from the techniques of these previous works. Moreover, since the existing theory relies on some relaxation (actually, the critical assumption of the relaxation is that each layer can be arbitrarily wide), they were not able to explore which condition is minimal in the setting of each layer.
- As a consequence, few, if any, results are obtained when the kernel size is small (and the channel number is fixed). Indeed, none of the results we are aware of have considered situations where both kernel size and width are limited. However, this is in fact the case when designing deep (residual) NNs, such as the ResNet family, where the primary change is increasing depth.
In contrast, our results analyze the optimality of both kernel size and the number of channels. The sharp result is non-trivial and new. The result indicates that even though each layer is relatively simple, much more complicated functions can be ultimately approximated via composition. In our opinion, this setting is important in order to uncover the theory behind the use of compositions to approximate functions -  a question that is yet to be understood and could be the crux of breaking the curse of dimensionality.
The analyses we present here are a step towards this goal.

From: ICLR.cc/2023/Conference/Paper1291/Authors
title: A Point-by-Point Reply to Reviewer RUpT (Part 2)
comment: Q3. Why wideness of the layers would have any effect on the symmetry.
- This is a general statement not limited to the shift invariance setting (explained below). We **remove** the sentence in the revised version since it may cause misunderstanding.
- The sentence is stated generally, indicating that for a general symmetry requirement (e.g. permutation invariance), simply enlarging the width is not helpful for approximation. (Bruna et al. 2022) has proved a simple case that there might exist exponential separation for different layer constructions. More precisely, the authors showed that for the same function, shallow networks with 2-Janossy pooling can use one channel to approximate it arbitrarily well, while using the 1-Janossy pooling will require an exponentially large channel size. In general, it is difficult to find a powerful layer architecture that is both symmetric and universal with one layer.
- If we relax the restriction on width, the universal approximation can be easily established for the kernel with full support: if we have a (shallow-but-wide) neural network, we can then copy all the layers and shift the parameters of all copies, then we can get a shift permutation invariant neural network, which is still shallow but wider. However, the aim of this paper is to investigate the minimal requirement of the width (here it means both kernel size and the number of channels) to achieve UAP through composition.
- Since our result concerns the minimal requirement and gives the optimal result, the other deep convolutional neural network family with more complex architectures also possesses UAP as a corollary of our results.  

Reference
- Exponential Separations in Symmetric Neural Networks. Aaron Zweig, Joan Bruna, Arxiv: 2206.01266

From: ICLR.cc/2023/Conference/Paper1291/Authors
title: A Point-by-point Reply to Reviewer RUpT (Part 1)
comment: We thank the reviewer for their comments that helped improve our manuscript. Below is a point-by-point reply to the main issues and concerns raised by the reviewer. 

Q1. The intuition behind the proofs are not provided. 
- We add a new paragraph in Section 3 as suggested, explaining the intuition of the sufficient condition.
- The mechanism behind the sharpness result on the kernel size (The second part of Theorem 1) is as follows: Suppose the kernel size is 1\*1, then the function is actually separated as a sum of some nonlinear univariate function, which is shown to be impossible to possess UAP at the beginning of Section 4.
- The intuition of the sharp result on the channel of number is that if the channel number is one, then in the expression $x \mapsto \sigma(w\ast x + b)$, we have no flexibility to adjust the coefficient $w$. The gradient with respect to $x$ of this layer will be either $0$ or the cyclic matrix $[w_{i-j}]_{ij}$. As a result, the whole function has only two values for the gradient, either $0$ or some non-zero $g$ (the mask not covered). A direct calculation shows that this kind of function cannot possess UAP, see Section 5.
- For the sufficient condition (in Appendix B), the proof is much more involved. The framework (developed by Li et al. 2022) is as follows. One first shows that the approximation result can be reduced into transporting arbitrarily many points in different orbits under the group action. Then, one constructs several operations to show that the point transporting can be achieved under the considered control family.

Q2. What's new in the proof. 
- For $\ge 2$ part (Section 3) the technical approach is similar, but for the sharpness part (Section 4 \& 5) they are new. 
- For the part around the sharpness (including the second and the third part of Theorem 1), all the techniques and proofs are new, and we believe that the examples constructed in these proofs are quite useful in analyzing the approximation property of other architectures. The construction and the analysis in this part are highly nontrivial. Thus, we expanded on the key ideas in the construction and proof ideas in the main body of the paper. 
- For the part around the sufficiency (the first part of Theorem 1), the idea behind the proof and the tools we used are quite similar to the previous work (Li et al. 2022).
However, the novelty here is that we address the minimal requirement in both kernel support and channel number, where the existing framework is *not applicable*. The key observation is that if we restrict the width and the kernel size, then the functions that can be expressed within single layers are limited. The framework reduces the $L^p$ approximation to transporting (finitely) many points, and if we can transport one point to its correct way within one (continuously idealized) layer then all things would get much simpler. This is the core idea of Li et al. to introduce the symmetric invariant well function. However, the current setting forbids us from constructing such invariant well function within a single layer. 
- To overcome this, the alternative way is to use more depth to transmit the discrepancy gradually, implying that we should use more layers to accomplish the point transportation. The main technical novelty here is to show that this is indeed possible, provided that we satisfy the minimal requirements laid out in terms of channel number and kernel size.

Reference 
- Deep Neural Network Approximation of Invariant Functions through Dynamical Systems. Qianxiao Li, Ting Lin, and Zuowei Shen. Arxiv:2208.08707

From: ICLR.cc/2023/Conference/Paper1291/Authors
title: A Point-by-point Reply to Reviewer gqo7 (Part 2) 
comment: Q3. The technical difference between this work and the previous work.
- For $\ge 2$ part (Section 3) the technical approach is similar, but for the sharpness part (Section 4 \& 5) they are new. 
- For the part around the sharpness (including {the second and the third part of Theorem 1}), all the techniques and proofs are new, and we believe that the examples constructed in these proofs are quite useful in analyzing the approximation property of other architectures. The construction and the analysis in this part are highly nontrivial. Thus, we expanded on the key ideas in the construction and proof ideas in the main body of the paper. 
- For the part around the sufficiency ({the first part of Theorem 1}), the idea behind the proof and the tools we used are quite similar to the previous work (Li et al. 2022).
However, the novelty here is that we address the minimal requirement in both kernel support and channel number, where the existing framework is *not applicable*. The key observation is that if we restrict the width and the kernel size, then the functions that can be expressed within single layers are limited. The framework reduces the $L^p$ approximation to transporting (finitely) many points, and if we can transport one point to its correct way within one (continuously idealized) layer then all things would get much simpler. This is the core idea of Li et al. to introduce the symmetric invariant well function. However, the current setting forbids us from constructing such invariant well function within a single layer. 
- To overcome this, the alternative way is to use more depth to transmit the discrepancy gradually, implying that we should use more layers to accomplish the point transportation. The main technical novelty here is to show that this is indeed possible, provided that we satisfy the minimal requirements laid out in terms of channel number and kernel size.

Q4.  It is possible to reformulate the current problem in the quotient space.

- Yes. For shallow networks, (Cohen et al. 2019) provided a general framework 
concerning the theory of quotient space. One can employ a corresponding approach here for dynamical systems. However,
we would like to avoid abstraction for the sake of abstraction. In this case, we found no new useful tools to deal with the approximation properties of flows in quotient spaces, since all operations will go back to the original $\mathbb R^n$ space.
-  We give a concrete example to show that we cannot directly deduce UAP from earlier results by a quotient argument. Observe that For the non-symmetric version, the result in (Li et al. 2019) requires that the control family $\mathcal F$ is (restricted) affine invariant, while if we directly adopt this in the quotient space, then it will be reduced to scaling invariant. However, the scaling invariant assumption cannot satisfy the UAP property, whose proof will be similar to those in Section 4.

Q5. Is the problem will be much simpler if the kernel can have full support?
- Yes, and it has been covered in the theory developed in (Li et al. 2022). The technical differences have been explained in the answer to Q3.
- Going back to the proof we provided in the Appendix, Step 2 of the proof of Theorem 4 can be simplified if the full-support kernel is adopted. However, the parameters in the full-support kernel increase as the input data increases, implying that the parameters in each layer are not bounded. As a result, such kernels are not recommended in practice. That's why we insist on the small kernel case and spent much more effort proving the result. 
- Also, the sharpness results are a consequence of other finite-size kernels, so it's important to consider them, as it reveals insights otherwise lost in the full-support analysis.

Q6. Can the problem be reduced to a one-dimensional problem?
- We do not think that it can, even if other approaches (including (Li et al. 2022) and the approach here) are considered.
- This is because since the shift-invariant functions/mapping can be uniquely determined in a slice $\Omega = \\\{ x_1 > x_i\\\}$ (here we take a one-dimensional convolutional example, where $i$'s are the spatial indices), which cannot be reduced to a domain with only one coordinate. All the coordinates are not decoupled, leading to technical challenges in such a theory. Therefore, building such a theory (especially in a narrow-and-deep setting) may not be possible.

References
- Deep Neural Network Approximation of Invariant Functions through Dynamical Systems. Qianxiao Li, Ting Lin, and Zuowei Shen. Arxiv:  arXiv:2208.08707
- A general theory of equivariant cnns on homo-geneous spaces. Taco S Cohen, Mario Geiger, and Maurice Weiler. NIPS 2019.
- Deep learning via dynamical systems: An approximation perspective. Qianxiao Li, Ting Lin, and Zuowei Shen, Journal of the European Mathematical Society, 2019.

From: ICLR.cc/2023/Conference/Paper1291/Authors
title: A Point-by-point Reply to Reviewer gqo7 (Part 1)
comment: We thank the reviewer for their comments that helped improve our manuscript. Below is a point-by-point reply to the main issues and concerns raised by the reviewer. 

Q1. The results are not surprising since there already exist many works about approximation in CNN. 
-  While there are many works concerning the approximation property of convolutional neural networks, to the best of our knowledge, ours is the first work to do so with a limited number of channels, fully highlighting the approximation capabilities of a *deep* neural network through function composition. This is arguably one of the most important features of deep learning that is worth understanding mathematically.
-  The existing results around the approximation capabilities of convolutional neural networks can be categorized into several classes. One either (1) takes the kernel as full-width (same size as the input) (e.g. Li et al 2022) which is not often used in practice; (2) assumes a sufficiently large channel number in order to adopt some kernel learning methods (e.g. Bietti 2021); or (3) remove the nonlinear activation function and reduce to a linear approximation problem (e.g. Zhou 2020), then use complex fully connected terminal layer(s) to achieve approximation. In contrast, our result quantifies the minimal requirement of the width and the support number (and sacrifices the effective bound on the depth as compensation). Thus, our result cannot be derived from the techniques of these previous works. Moreover, since the existing theory relies on some relaxation (actually, the critical assumption of the relaxation is that each layer can be arbitrarily wide), they were not able to explore which condition is minimal in the setting of each layer.
- As a consequence, few, if any, results are obtained when the kernel size is small (and the channel number is fixed). Indeed, none of the results we are aware of have considered situations where both kernel size and width are limited. However, this is in fact the case when designing deep (residual) NNs, such as the ResNet family, where the primary change is increasing depth.
In contrast, our results analyze the optimality of both kernel size and the number of channels. The sharp result is non-trivial and new. The result indicates that even though each layer is relatively simple, much more complicated functions can be ultimately approximated via composition. In our opinion, this setting is important in order to uncover the theory behind the use of compositions to approximate functions a question that is yet to be understood and could be the crux of breaking the curse of dimensionality.
The analyses we present here are a step towards this goal.

Q2. The contribution mainly lies in theory and not practical, the depth needs to be infinite.
- This paper's focus is theory, aiming to show the effectiveness of depth (composition) in the convolutional setting.
Nevertheless, understanding the approximation of shift-invariant function through compositional (deep) models is important in model selection in practice.
- For the concern of the **infinite** depth, we should point out that the majority of the approximation result requires either an arbitrarily large width or arbitrarily large depth. While CNN approximation when width increases are relatively well-understood, fewer results
are derived on how depth can ensure approximation, even at a constant width. Our results are initial steps in this direction.
- To emphasize the expression/approximation power comes from the composition/depth, we limit the channel number and make the bounds sharp. For other kinds of architectures, since in general, they contain our model, the UAP can be directly established. In other words, our model can be regarded as a simplified model and the approximation result of other more practical models can come from ours.

References
- Approximation and Learning with Deep Convolutional Models: a Kernel Perspective. Bietti, ICLR 2022.
- Universality of deep convolutional neural networks. Zhou, ACHA 2020.
- Deep Neural Network Approximation of Invariant Functions through Dynamical Systems. Qianxiao Li, Ting Lin, and Zuowei Shen. Arxiv:  arXiv:2208.08707

From: ICLR.cc/2023/Conference/Paper1291/Reviewer_gqo7
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper studies approximation power of (residual) vanilla convolutional networks. The authors show that any translational equivalent/invariant mapping/functions can approximated by such networks, in the $L^p$ sense, if 
- the number of channels is at least 2. 
- the kernel size is at least 2 (in each coordinates)
- the network is sufficiently deep (essentially, infinitely many of layers)   

The authors approach this problem from a dynamical viewpoint by viewing CNNs as a discretization of a certain equivalent flow. Although the results is not surprising, the proof is quite technical and non-trivial owing to the number of channels is small (>=2.) 
strength_and_weaknesses: # Strength 

- The overall mathematical clarity of the paper is far above typical ML conference papers. 
- Both the results and proofs are non-trivial. 

# Weaknesses: 
- the results are not surprising (as what was expected). The problem can be much simpler if allowing more channels (shallower networks) vs. the current setup ( == narrow + deeper). 
- The contribution mainly lies in theory and is not very relevant to practice, as it essentially requires the depth of the network to be infinite. 



clarity,_quality,_novelty_and_reproducibility: A few minor remarks.

1. I would like to know more about the technical difference between the current paper (with group equivalent) and Li et al., 2022a;b. Is it possible to `reformulate` the current problem in the quotient space $\mathbb R^k /\mathcal T$, where $\mathcal T$ is the group generated by translations and consider flows defined in this quotient space? 
2. Another related question. Is the problem much simpler if the kernel can have full support? Is it hard to show that any one-layer convolution kernel (with full support) can be approximated multi-layer with Conv + activation with kernel size 2?  
3. Can the problem be reduced to $\mathbb R$ from $\mathbb R^k$, namely, is the problem essentially a 1-D problem? 


summary_of_the_review: update

Thanks a lot for the detailed and insightful reply. I will keep my score. 

------

Although the results of the paper is not surprising, it resolves an important theoretical question regarding approximation power of shallow-and-deep CNNs. I think the result and its proof technique are valuable to the ICLR community.  
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1291/Reviewer_RUpT
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: Universal approximation results in deep learning stated that certain class of neural networks can approximate any function up to arbitrary low error. These results were originally proven for sufficiently wide networks but have been extended in 2 directions:
1. Proving them for small width networks (with arbitrary depth). For example, the work of Lu et al. 2017.
2. Proving them for invariant (equivariant) functions and for corresponding  invariant (equivariant) networks. For example, the work of Yarotsky 2018.

This paper combines these two extensions for shift invariant (equivariant) functions and FCNNs (or residual FCNNs). They prove that FCNNs of constant width can approximate any shift invariant (equivariant) function. They further show that the constant width they show this result for is tight.



strength_and_weaknesses: The authors do not provide intuition for the universal approximation result in the main body of the paper or in the Appendix. The authors should also describe what are the new ideas that go in the proof and what techniques are borrowed from earlier works, this will make the proof easier to verify. For example, in appendix B.5 (Proof of proposition 3) the authors use the approach of dividing the input space into small cubes which has been used in earlier works such as Lu et al. 2017. Same holds for the ‘coordinate zooming function’ which was used in Li et al. 2022.



Other comments:
1. ‘Deep, rather than wide, neural networks are thus a promising solution to this dilemma, using layer composition in the depth direction to build complexity, whereas each layer can be kept sufficiently simple to respect symmetry constraints.’ I do not see why wideness of the layers would have any effect on the symmetry constraints.
clarity,_quality,_novelty_and_reproducibility: As mentioned earlier the paper does not provide intuition for the proof and does not compare the proof techniques with related works. Other than that the paper is well-written.
summary_of_the_review: As the authors do not provide a comparison of techniques with prior works and a description of new ideas that go into their main result I cannot recommend acceptance.

Due to the lack of intuition provided in the paper and my unfamiliarity with the techniques used I was unable to verify the correctness of the proof (given in the appendix) of the main universal approximation result. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1291/Reviewer_iJgd
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper studies universal approximation of invariant functions, using fully convolutional networks with small filter size, small number of channels, but large depth. The theory is based on dynamical systems, using flow maps to construct deep residual networks approximating the target function, and follows previous work by Li et al, which studied such techniques on non-convolutional architectures.
strength_and_weaknesses: The paper makes interesting contributions to the line of work on approximation via dynamical systems, in particular, it extends previous work by Li et al to showing universal approximation results with convolutional networks with small filters, while Li et al. was restricted to full-width filters.

While this contributions can be of interest, it is nonetheless quite marginal and incremental over previous work, and may not be sufficient by itself for publication.

Here are some suggestions on what could strengthen the paper:
- approximation and statistical rates: the architecture used here is presumably more parameter-efficient than in other works -- can this be used to establish improved statistical rates?

- benefits of this architecture: are there specific function classes where such an architecture is particularly beneficial compared to full-width convolutions, or even fully-connected models?

- better comparison to related work on approximation with convolutional networks, e.g. [this](https://arxiv.org/abs/1611.00740), [this](https://openreview.net/pdf?id=dgxFTxuJ50e), [this](https://arxiv.org/abs/1708.06633), [this](https://arxiv.org/abs/2106.08619), [this](https://arxiv.org/abs/2102.10032), [this](https://arxiv.org/abs/2112.05611) (the last three are for convolutional kernels / infinite-width networks)

clarity,_quality,_novelty_and_reproducibility: ok
summary_of_the_review: Interesting result, but not useful enough by itself.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold