Paper: Just How Flexible are Neural Networks in Practice? (ICLR-2024)

Reviewer comment (Experiments):
The paper takes the strategy of gradually increasing the number of training samples. However, the paper does not discuss how many different trials are run for each setting, and no error bars are plotted. In the very extreme case, there can be some unlucky bad initialization of the neural network that prevented a model from fitting training samples thus causing the EDC to stop growing for that setting. However, this rare case may not be representative of what happens in most cases.

Author response:
**a.  Multiple random seeds:**
We appreciate the reviewer's input and concur with their observation. To ensure the robustness of our results to random initialization, in the original paper, where our networks did not successfully reach 100% training accuracy, we reran the networks three more times with different random seeds. This was done to ensure that it not fit all the examples. We thank you for highlighting this aspect

Reviewer comment (Experiments):
I am not sure why the paper performs the sanity check that each training reached a minimum rather than a saddle point. Since an important consideration of the paper is to evaluate EDC based on the performance of different optimizers, whether the optimizer stopped at a local minimum, global minimum, or saddle points should be a property of the optimizer and it should not be influenced by the experiment design.

Author response:
**b. Consideration of optimization:**
We agree that we want our study to capture the properties of different optimizers, but in our experiments, none of the optimizers we tried got caught in saddle points.  We chose to check the Hessian spectrum because practitioners train with different optimizers, architectures, datasets, etc, for different numbers of epochs, and we need a way to ensure fair comparison.  Fixing the number of epochs, therefore, would not yield a fair comparison, and simply terminating when the gradient norm is sufficiently low may not yield a fair comparison either because we could end up at a saddle point in the middle of training or a flat region which we would have escaped during the length of a standard training routine, but a gradient-norm based stopping condition would cause us to terminate the training routine very early.

Reviewer comment (Evaluation):
The conclusion ”Flexibility across modalities” might be questionable. Is the result that tabular datasets have larger EDC a consequence of simpler tasks?

Author response:
**a. Larger EDC for tabular datasets is due to simpler tasks**: There are several ways to define simplicity level, with  pros and cons. However, controlling different factors that may impact the EDC is essential. Inspired by the reviewer's question, we conducted an experiment where we tried to control for properties such as class numbers. We transformed several datasets into binary classification problems, standardizing the number of classes. This modification allowed us to isolate the influence of the input distribution  on EDC more accurately. Our results revealed that the EDC across various image datasets is smaller than tabular datasets (see full details in the the paper). Therefore, we can assert that models can accommodate more examples for each model size for tabular datasets and, by some criteria, can be considered 'simpler'.

Reviewer comment (Experiments):
The conclusion ”CNN has larger EDC than MLP” might be questionable. It seems that the experiments are performed only on vision tasks, which makes the conclusion trivial. Does the same result hold on other tasks?

Author response:
**b. CNN has a larger EDC than MLP:** CNNs are recognized for their superior generalization on vision problems, attributed to their robust inductive bias for spatial relationships and locality (Taye MM, 2023). However, our work observed that CNNs exhibit greater parameter efficiency even on **randomly labeled data**, suggesting that their enhanced capacity extends beyond better generalization alone.
Regarding tabular data, in response to the reviewer's inquiry, we investigated the  EDC of MLP and CNNs for tabular datasets. Our findings, detailed in the updated version of our paper, reveal that the EDC is similar for the MLP and CNN. This outcome is unsurprising, as tabular datasets lack the spatial relationships and locality found in computer vision problems. Consequently, we did not anticipate a CNN advantage in fitting more examples than MLP in this context.

Reviewer comment (Experiments):
When investigating the influence of architecture and over-parameterization, it is unclear what optimizer settings the paper is using. Is the paper fixing an optimizer, or is the paper trying all optimizers and picking the best-performing one?

Author response:
**a. Clarify the optimizer settings**: For our experiments, unless otherwise specified, we utilized SGD for training. We highlighted this detail in the updated version.

Reviewer comment (Experiments):
Regarding the experiment of scaling up the image sizes, it is unclear whether the filter size is also scaled up accordingly.

Author response:
**b. Scaling filter sizes:** Yes, we increased the filter size proportionally with the input sizes.

Reviewer comment (Evaluation):
The fact that labels are fit to 100% accuracy may affect the conclusion of some experiments related to generalization since there can be outliers that negatively affect the generalization ability.

Author response:
In principle, it's true, and there might be outliers. During our experiments, we also checked our analysis without requiring 100% accuracy, allowing for a bit less (in order to accommodate outliers), which didn't significantly change our results.

Reviewer comment (Evaluation):
From Figure 1 (right) we see that a larger EDC improvement implies a better generalization ability. Figure 2(b) shows that scaling width under a fixed over-parameterization results in the largest EDC improvement. Do these two results imply that scaling the width leads to the best generalization?

Author response:
**Regarding the Figures** - Yes, the results imply that scaling the width leads to the best generalization.

Reviewer comment (Theory):
The paper could benefit from deeper theoretical exploration. While the metric's motivation isn't thoroughly convincing, there's an intriguing mention of PAC-Bayes and VC dimension. It would be valuable to elaborate on how their selected measure aligns with the current theory.

Author response:
Dear Reviewer 14iz,

We appreciate your comments and thoughtful considerations regarding the theoretical aspects of our paper. We acknowledge the importance of theoretical exploration in the realm of deep learning; however, we intentionally adopted a different approach in this work.

Our primary motivation was to explore the empirical side of network capacity rather than relying solely on theoretical frameworks such as PAC-Bayes and VC dimension. Historically, attempts to assess the capacity of deep networks have often been theoretical and, at times, not entirely aligned with empirical observations. We aimed to bridge this gap by providing an empirical perspective on network capacity.

Reviewer comment (Theory):
While I appreciate the importance of experiment-driven papers, it appears that the conclusions were drawn post-results, rather than being based on predefined hypotheses derived from established theory.

Author response:
While we recognize the value of predefined hypotheses derived from established theory, our focus on drawing conclusions post-results stems from our commitment to an experiment-driven methodology. This approach enhances transparency and clarity in the progress of scientific inquiry. The "double descent phenomenon" (Preetum, et al. 2021)  is a good example; its impact on the community was initially observed empirically, and the (partial) theoretical understanding followed in subsequent works.

Reviewer comment (Theory):
The relevance of model architectures to random labels remains unclear, and the frequent references to the linear model are perplexing, especially given that large parameterized neural networks don't typically align with traditional models—consider the double descent phenomenon and their tendency to yield smoother functions [1].

Author response:
Regarding **the relevance of model architectures to random labels and frequent references to linear models**, as mentioned in our paper, many theoretical works provide vague and impractical approximation theory bounds, and this is why it is so important to examine it in practice. We agree that modern neural networks do not align with traditional models, and our paper precisely demonstrates that the practical flexibility of modern networks is much higher than linear models. Concerning random labels, based on previous works (Zhang et al., 2016), we sought to distinguish between the input distribution and the label distribution, exploring how the network can fit the data regardless of the label’s distribution. Moreover, as demonstrated in our paper, the ability of a neural network to fit many more correctly labeled samples than incorrectly labeled samples predicts generalization.

Reviewer comment (Theory):
The description of the EDC and how it relates to quantifying capacity seems a little bit loose; it is unclear why it is a good metric for capacity. Furthermore, it would be good to split out step by step the computation.

Author response:
**EDC Metric Explanation:**

Defining the capacity of a model encompasses various perspectives. Intuitively, it refers to model complexity—a measure of how intricate a pattern or relationship a model can express. In simpler terms, it quantifies the number of examples a model can learn. While VC dimension is one method to define capacity, it may exhibit a considerable gap from the model's actual ability to fit the data and, therefore, not used in practice. In our work, we introduce EDC as an alternative measure for empirical capacity.

In essence, EDC is the maximum number of training data points a model can fit, representing its empirical capacity—the number of patterns it can learn. To elaborate, EDC is defined as the largest sample size where the model can perfectly fit randomly sampled training subsets of that size when optimized until convergence.

As explained in our paper, to compute the EDC, we adopt an iterative approach for each network size. Initially, we start with a small number of samples and train the model. Post-training, we verify if the model has reached 100% training accuracy. If met, we re-initialize the model with random initialization and train it again on a larger number of samples randomly drawn from the entire dataset. This process iteratively increases the number of samples in each iteration until the model fails to fit all the training samples perfectly. The largest sample size where the model achieves a perfect fit is taken as the EDC for that particular network size.  Please refer to section 3 in our paper for a detailed explanation.

Reviewer comment (Experiments):
When comparing different architectures such as CNNs and ViTs, it is unclear how the authors ensured that the architectures are compared fairly. For example, there is no mention of the number of parameters used in each architecture. Even if those architectures share the same parameter count, the shape of the architecture can have a big impact as the authors themselves study for CNNs. It is not clear if one can conclude that CNNs are more parameter efficient than ViTs, and there are not enough details in the paper for the reader.

Author response:
**Experimental Details and Fair Comparison of Architectures:** 
We apologize for any confusion caused by comparing the average EDC of different architectures in the paper. To provide clarification, for all figures where we plotted the average logarithmic EDC, we have included the complete, unaveraged results in the appendix. Specifically, Figure 5 in the appendix offers a detailed presentation of the EDC across the full spectrum of parameter counts for various models, including VITs, CNNs, and MLP. 
Our decision to feature averaged results in the main text was driven by the consistency of the observed phenomena across different model sizes, where no significant differences were noted between smaller and larger models. This choice aimed to balance clarity and conciseness in the paper's main body while providing extensive data in the appendix for readers seeking more detailed insights.
Regarding the reviewer’s concern about the impact of scaling laws on the comparison between different models, In our study, we scaled ViTs following the scaling laws proposed by Zhai et al., 2022. These laws advocate for scaling all aspects—depth, width, MLP width, and patch size—simultaneously and uniformly. We have expanded our revised results to include different scaling laws for VIT to address the reviewer's concerns and further elaborate on this aspect. Based on the reviewer's suggestion, we used both SoViT from  Ibrahim et al. (2023) and laws where the number of encoder blocks (depth) and the dimensionality of patch embeddings and self-attention (width) in the ViT are scaled separately.  Figure 22 in the appendix of our updated paper demonstrates that scaling each dimension independently leads to suboptimal results in line with our observations from the EfficientNet experiments and that SoViT has slightly better results to the ones from Zhai et al., 2022.
This expanded analysis strengthens our original findings and offers a more nuanced perspective on how different scaling strategies influence the EDC of various architectures. This elaboration adequately addresses the reviewer's concerns and provides a more comprehensive understanding of our comparative analysis.

Reviewer comment (Evaluation):
The authors repeatedly mention that ReLU improves EDC even though ReLU "were introduced to neural networks to prevent vanishing and exploding gradients." There does not appear to be a discrepancy here, as ReLU were introduced to prevent training instability issues, so it would be expected that ReLU improves training and EDC as a result. It is unclear why the authors highlight that the impact of ReLU on EDC is surprising.

Author response:
**Impact of ReLU on EDC:**
Thank you for pointing out this concern, and we would share your concern that exploding and vanishing gradient problems could in fact cause lower EDC if we were not successfully minimizing the loss function.  Whereas ReLUs were adopted to prevent exploding or vanishing gradient problems that can cause poor conditioning and hamper optimization, we actually ensure that we obtain a minimum of the loss function in each experiment.  That is, we find that ReLUs boost EDC even when we would be able to optimize successfully and converge to minima using competing activation functions.

Reviewer comment (Experiments):
When comparing different datasets, like CIFAR10/100, MNIST, and ImageNet, the authors use these results to argue for the impact of the input distribution. However, those datasets have different numbers of classes. One way to control for this is to fix the number of random labels in all datasets, but there is no discussion of this in the paper, and it seems that the authors kept the label space unchanged. This makes it unclear when the input distribution matters, especially when accounting for the differences in the number of classes.

Author response:
**Dataset Comparisons Considering Class Numbers:**
We appreciate the reviewer's observation. Our initial approach was to present the EDC for a variety of datasets and modalities, after which we delved into decomposing the effects of different dataset features on EDC. A key aspect of our exploration was assessing how the number of classes within a specific dataset influences EDC. We found that increasing the number of classes typically makes data fitting more challenging with semantic labels but easier with random labels. This counterintuitive finding underscores the complex interplay between class numbers and data-fitting capabilities. In response to the reviewer's point about the value of comparing datasets while controlling for properties like class numbers, we performed additional analyses for a more nuanced understanding. We converted several datasets into binary classification problems, standardizing the number of classes. This modification enabled us to assess the impact of class numbers on EDC more accurately and isolate the effects of input distribution.
Our results from this controlled comparison (Figure 22 in the appendix of the updated paper), showed that even though the EDC among various image datasets increased in the binary classification setting compared to the original setting, tabular datasets consistently demonstrated higher EDC. Furthermore, significant differences persisted among the different tabular datasets. These outcomes suggest that additional factors, perhaps intrinsic to the datasets themselves, significantly contribute to EDC beyond just the number of classes.
Incorporating the reviewer's feedback, we will clearly articulate these additional findings and analyses in the revised version of our paper. This enhancement will deepen the understanding of how diverse datasets, each with unique characteristics and complexities, influence the EDC of neural networks.

Reviewer comment (Writing):
The authors should provide precise definitions of some of the terms they use. For example, what is meant by "parameter-efficient"? One interpretation is that it is EDC divided by the parameter count—clarification is needed.

Author response:
**Precise Definitions:**  We acknowledge the need for precise definitions. Concerning "parameter-efficient," we clarify that a network is considered more parameter-efficient if it consistently demonstrates higher EDC compared to other models across a range of parameter counts.

Reviewer comment (Writing):
On page 6, there is a typo in "we three neural network."

Author response:
**Typos:** We thank you for catching the typo. The error has been corrected in the updated version of the paper.

Reviewer comment (Experiments):
It would be interesting to see if the scaling laws derived for ViT [1] would also increase EDC, similar to the analysis that was done on EfficientNet.

Author response:
**Scaling Laws Analysis:** 
Based on your comment, we conducted additional analyses following the scaling laws presented by Ibrahim et al. (2023). Similar to the EfficientNet analysis, the scaling laws improved ViT results. However, ViT still underperformed compared to CovNet. For further details, please see our response to Question 4 above.

Reviewer comment (Experiments):
Can you please clarify how the comparison across different architectures was done? Did you ensure that all architectures have the same size? How were the shapes selected? Are they based on some standard shapes, such as those used in the original ViT and EfficientNet papers? What about MLP?

Author response:
**Comparison across different architectures:** See our response to Question 2. We calculated their EDC across the entire range of model sizes for all architectures. See Figure 5 in the appendix for detailed results.

Reviewer comment (Experiments):
When using random labels, did you fix the label space in all datasets (e.g., binary classification) or was the same label space used in each dataset (e.g., 10 classes in CIFAR10 but 100 classes in CIFAR100)?

Author response:
**Fixing the label space when using random labels:** In our original results, we did not fix the number of classes. Following your suggestion, we conducted additional experiments by converting all datasets to binary classification. For further details, please see our response to Question 4 above.

Reviewer comment (Presentation):
Can you please clarify Figure 2(b)?

Author response:
**Figure 2(b) Clarification:** See our response in Answer 8 for clarification on Figure 2(b).