Paper: How many samples are needed to train a deep-ReLU neural network? (ICLR-2024)

Reviewer comment (Novelty):
There are many possible contributions mentioned, but none are convincing enough. For example, regarding the technical contributions, the authors claim that previous works cannot be applied to ReLU networks, but it is unclear whether this extension truly has many technical difficulties.

Author response:
We would like to note that the challenges arising from the ReLU activation function, particularly in the absence of the separation of the estimation method and the function space, and without considering any smoothness for the function space, become more pronounced in the calculation of step 1 in Lemma 6, specifically in Equation 7. However, for other activation functions, this equation can be computed much more easily with simple arithmetic.

Reviewer comment (Experiments):
Another possible contribution is that the theory matches the practice, but the empirical observations in this paper are not sufficient to support this claim, as only the MNIST and CHP datasets are used.

Author response:
Taking into account your valuable suggestions, we have introduced three additional network settings in the empirical part for the MNIST dataset. This addition aims to illustrate that network size is not a crucial factor in practice, aligning with our theoretical findings. Additionally, we conducted experiments with the CIFAR-10 dataset to augment our empirical results.

Reviewer comment (Theory):
In the theory part, the result requires d to be large enough, but it is not specified how large. It is unclear whether d would be exponential with the sample size n.

Author response:
We are appreciative of you for raising this meticulous point regarding how large d would be. Now, we first have added $d>=10$ (based on the proof of Lemma~6 in Appendix) have added the following explanations after Lemma~6:

By quantifying the lower bound of the packing number, it provides valuable insights into the capacity and potential complexity of these networks. For small values of $\\delta$, a sufficiently wide network becomes necessary.  This observation is particularly interesting as it provides valuable insights into selecting an appropriate width for the network based on the input dimension. The larger the input dimension $d$, the wider the network should be.

Reviewer comment (Experiments):
In the empirical part, more experiments should be added, for example on CIFAR-10 and ImageNet datasets.

Author response:
Taking into account your valuable suggestions, we have introduced three additional network settings in the empirical part for the MNIST dataset. This addition aims to illustrate that network size is not a crucial factor in practice, aligning with our theoretical findings. Additionally, we conducted experiments with the CIFAR-10 dataset to augment our empirical results.

Reviewer comment (Experiments):
In the empirical part, the theory shows that network size is not important, but the authors should provide more experiments to support this.

Author response:
Taking into account your valuable suggestions, we have introduced three additional network settings in the empirical part for the MNIST dataset. This addition aims to illustrate that network size is not a crucial factor in practice, aligning with our theoretical findings. Additionally, we conducted experiments with the CIFAR-10 dataset to augment our empirical results.

Reviewer comment (Writing):
This paper appears to overclaim its results. Titled "HOW MANY SAMPLES ARE NEEDED TO TRAIN A DEEPRELU NEURAL NETWORK?" it investigates a much narrower setting than suggested. Firstly, the paper overlooks the training process, assuming that the optimizer can derive the best learner from a given function class. Therefore, the term "train" in the title is misleading. Secondly, the scope is limited to feed-forward neural networks with square loss, neglecting widely used structures like CNNs and transformers. In light of this, I recommend that the authors either provide additional commentary within the paper or refine the title and contributions to reflect the actual scope of the study.

Author response:
1- Thank you for raising this point regarding the scope of our work. You are absolutely right. Now, we’ve changed the title of the paper to “How many samples are needed to train a ReLU feed-forward neural network? ” to make our setting more clear. For the future work, we are going to consider the setting for CNN and RNN as well.

Reviewer comment (Theory):
The paper adopts a teacher-student network setting with Gaussian input $x$ and noise $u$. The impact of the scale of $x$ and noise $u$ on the results in Theorem $1$ remains unclear. Additionally, the l1 constraint $v_1$ mentioned in Theorem 1 is not reflected in the final bound in (5), and the reason for this omission is not clear.

Author response:
2- Taking your thoughtful recommendation into account regarding directly reflecting the influence of $v_1$, we have revisited the final bound in Equation 5 to explicitly highlight the role of $v_1$. This revision not only enhances the clarity of our presentation but also provides a more transparent understanding of how the bound is influenced by the magnitude of $v_1$.

Reviewer comment (Theory):
What will happen if we consider a noiseless teacher-student model, i.e., $\sigma = 0$?

Author response:
4- One fundamental part of our theoretical framework involves calculating the KL divergence between two normal distributions and subsequently establishing an upper bound for mutual information. As expressed in the proof of Lemma 4, the symbol $\\sigma$ denotes the standard deviation of these normal distributions. The derived upper bound for the mutual information (Lemma 4) is crucial for applying Fano’s method. Although our method doesn't directly apply to a noiseless teacher-student model, exploring the noiseless teacher-student model in itself would be intriguing.

Reviewer comment (Theory):
For the proof of Lemma 4, the authors give an 8-step explanation which is simply reading through each step in the following inequalities. I'd prefer it if the most important step was emphasized and the other steps were left to the reader. For me the most important step there is the explicit integral formulation of \(D_\text{KL}\) but I have no idea where \(\frac{1}{\sigma^2}\) comes from.

Author response:
In response to your inquiry about the rationale behind $1/\sigma^{2}$: In Lemma~9, we compute the Kullback-Leibler divergence between two distinct multivariate normal distributions. The term $1/\sigma^{2}$ indeed arises from the definition of the KL divergence, considering that  $\mathbb{P}{f{\theta^{j}}}$ and $\mathbb{P}{f{\theta^{k}}}$ correspond to the distributions associated with ${f_{\theta^{j}}}(x)$ and ${f_{\theta^{k}}}(x)$, respectively. Reviewing Equation~6 could provide additional insight into this matter. The $\sigma$ is the variance of $n$ i$\cdot$i$\cdot$d$\cdot$\@ noises $u_{i}$. And based on the view of Lemma~9, we need to have a positive variance.

Reviewer comment (Experiments):
The empirical evaluation is limited as it is. There is only one architecture used for Figure 1 (a certain width and depth, which would be good to include in the caption). It is unclear whether the proposed scaling law is also valid for really narrow networks like width 20. The rate here does not depend on the architecture — is this also reflected in practice?

Author response:
We have introduced three different network settings in the empirical part, featuring three very narrow shallow-ReLU feed-forward neural networks. And demonstrate that the scaling is valid also for shallow- ReLU feed-forward neural networks.

Additionally, we conducted experiments with the CIFAR-10 dataset to provide more empirical results.

To enhance clarity and readability, we have included the structure of the networks (specifying the number of hidden layers and the number of hidden nodes) in the captions.

Reviewer comment (Writing):
In conclusion, please specify "networks" as deep ReLU feedforward networks.

Author response:
Given that our work focuses solely on ReLU feed-forward neural networks, we have explicitly considered this point throughout the paper to enhance clarity.

Reviewer comment (Theory):
I am confused about the input data distribution. It is specified as \(\mathcal{N}(0, I)\) at the beginning of Section 2; however, later the authors introduce \(L_2(\mathbb{P}_x)\) norm. Is \(\mathbb{P}_x\) here Gaussian? If so, \(h(x)\) comes from the metric \(\rho\), not from the density of the distribution \(\mathbb{P}_x\)?

Author response:
We appreciate your meticulous point of view regarding your question about the distribution of $x$.
As mentioned in Section~ Problem formulation and main result, $\mathbb{P}_{x}=\mathcal{N}(\boldsymbol{0},\Identity)$, and we need this assumption in the calculation of Lemma~12. Regarding the relationship between $h(x)$ and $\rho$:

$h(x)$ is not directly coming from $\rho$; instead $h(x)$ is used in the calculation of the $L_{2}(\mathbb{P}_{x})-$norm. 
The density function $h(x)$, plays a role in quantifying the "distance" of two functions $\widehat{f}, f^{*}$ with respect to the Gaussian distribution $\mathbb{P}_{x}$. We've also added the term ``fixed'' to emphasize that the marginal distribution $\mathbb{P}_{x}$ is fixed.

Reviewer comment (Experiments):
I would increase my score if the authors provided more experiments (see weaknesses) and/or provided an alternative in-depth discussion/intuition instead of the proof of Section 4.1.

Author response:
We have introduced three different network settings in the empirical part, featuring three very narrow shallow-ReLU feed-forward neural networks. And demonstrate that the scaling is valid also for shallow- ReLU feed-forward neural networks.

Additionally, we conducted experiments with the CIFAR-10 dataset to provide more empirical results.