Paper: How does overparametrization affect features? (ICLR-2024)

Reviewer comment (Theory):
The research question has significant flaws. That is, independently trained underparameterized networks are likely to converge to similar feature representations, as each of them tries to minimize the target loss and, thus, only the features with the largest effect on the target loss are learned (given the more restrictive capacity constraint from the width scaling). On the other hand, overparameterized networks do not suffer from such an issue and can use their larger capacity during training to learn more (and other) features to further reduce the target loss. Consequently, this raises substantial concerns about the empirical findings.

Author response:
**"Independently trained underparameterized networks are likely to converge to similar feature representationss, as each of them tries to minimize the target loss "**\
Although the reviewer's claim may have a point, it does not necessarily hold. The learning process of each network can be different depending on their initialization. It is a noisy process and smaller networks may be learning some bad quality (noisy or redundant) features and these features may not help in the test set. 
\
Moreover, for instance, in our toy example (Section 4), all the networks have the capacity (enough neurons) to learn all the features but still they need to be large enough to learn all.

Reviewer comment (Novelty):
The introduced metric “feature performance” is only a linear probe and not an original contribution of the work.

Author response:
**"The introduced metric “feature performance” is only a linear probe and not an original contribution of the work."**\
We apologize for the confusion. We already state FSG is a linear probe in the paper and we will make it clearer. Our work differs from the previous work in how we utilize FSG.

Reviewer comment (Evaluation):
The feature residual analysis has contradictory results (Fig. 4a vs. 4b and Fig. 10). For the transformer setting, it is quite clear that the residual features help in predictive performance. However, for the ResNet setting this is not clear, as for $\alpha=1/8$ and $\alpha=1/16$ the difference is negligible. There is no discussion on this.

Author response:
**"The feature residual analysis has contradictory results (Fig. 4a vs. 4b and Fig. 10)."**\
We thank the reviewer for pointing this out. We apologize that Fig 4b is wrongly plotted. Plots in the figure are shown to be plotted for decreasing $\alpha$ but they are indeed plotted for increasing $\alpha$. That is why Figure 4b seems contradictory but the results in reality are not.

Reviewer comment (Writing):
Sec. 4 seems to reiterate the lottery ticket hypothesis (the initial weights are particularly effective or ineffective for training). It is unclear how this relates to the empirical findings of the present work.

Author response:
**"Sec. 4 seems to reiterate the lottery ticket hypothesis (the initial weights are particularly effective or ineffective for training). It is unclear how this relates to the empirical findings of the present work."**\
In section 4, our setup leads to many neurons to be dead at initialization. We wanted to point out that that is realistic as the lottery ticket hypothesis show.

Reviewer comment (Experiments):
The most basic requirement to verify this paper's point, "Do the features of overparameterized networks exhibit greater expressivity than the features of low-width networks, given that they tend to perform better?", is to have **a close training performance of overparameterized network and low-width networks** so that both networks are well-learned. Otherwise, the feature difference can come from well-learned / poorly-learned networks instead of overparameterized / underparameterized networks.

Author response:
**"The feature difference can come from well-learned / poorly-learned networks instead of overparameterized / underparameterized networks."** \
Our purpose is to create smaller (underparameterized) networks that has number of parameters comparable to the data size. Hence, we cannot use resnet18 as the lowest degree model in our experiments since it has much more paramaters than the data size. 
\
Moreover, we train all the networks for 300 epochs. Since smaller networks ($\alpha \le 1/8$) do not achieve a training accuracy close to 100 with 300 epochs, we also tried training these networks much longer. Although training much longer helped somewhat increasing the training accuracy, it either decreased or very marginally increased the validation accuracy. It is also natural that smaller (underparameterized) networks may not achieve a perfect training accuracy due to their capacity. 
\
Lastly, we concatenate many smaller networks to match the parameters of the large network. It could have been enough to match the number of features in the large network for a fair comparison. In that sense, we over-concatenate for a stronger empirical finding and still FSE scores of these concatenations are significantly high.

Reviewer comment (Evaluation):
Table 1 and Table 2 indicate that the feature difference actually comes from well-learned / poorly-learned networks. Table 1(b) shows the FSE feature difference starts to increase at $\alpha=1/8$, meanwhile, Table 2 shows the training accuracy starts to decrease at the same time ($\alpha=1/8$). When $\alpha < 1/8$, Table 2 shows a very similar training accuracy (99.81 - 99.99) but different validation accuracy (92.72 - 95.29), and Table 1 ($\alpha < 1/8$) does not reflect feature differences.

Author response:
We tune all the models using a variety of different hyperparameters and do not bias training process to favor larger size models.  We provide best possible training setting for each model to get their best predictive performance. We compare different size models with their best possible performances. So, they are comparable.\
As we stated before, it is expected that underparameterized smaller models may not reach 100 training accuracy since they have lower capacity. This doesn't mean that they are poorly trained.

Reviewer comment (Novelty):
The proposed FSE score (Definition 2.2.) is a common metric. [1 (iclr)] shows (almost) the same feature score. [3] computes a linear regression between two sets of features. Canonical Correlation Analysis [2] also shares a close idea.

Author response:
**"The proposed FSE score (Definition 2.2.) is a common metric."**\
We do not claim that FSE is a completely different metric from the previous ones. FSE is indeed used in [3] and we mentioned this (citing it) in our paper. The novelty of our paper lies in the use of FSE and FSG to compare the features between low- and high-width networks. Our papers also mentions some other similarity metrics too. Our work significantly differs from the literature in the way it utilizes FSE and FSG.

Reviewer comment (Experiments):
I suggest the author choose a regular network (e.g., ResNet18) as a low-width network and use a much wider (more channels) version as the base network (overparameterized), so that the well-learned / poorly-learned network problem can be avoided.

Author response:
**"The feature difference can come from well-learned / poorly-learned networks instead of overparameterized / underparameterized networks."** \
Our purpose is to create smaller (underparameterized) networks that has number of parameters comparable to the data size. Hence, we cannot use resnet18 as the lowest degree model in our experiments since it has much more paramaters than the data size. 
\
Moreover, we train all the networks for 300 epochs. Since smaller networks ($\alpha \le 1/8$) do not achieve a training accuracy close to 100 with 300 epochs, we also tried training these networks much longer. Although training much longer helped somewhat increasing the training accuracy, it either decreased or very marginally increased the validation accuracy. It is also natural that smaller (underparameterized) networks may not achieve a perfect training accuracy due to their capacity. 
\
Lastly, we concatenate many smaller networks to match the parameters of the large network. It could have been enough to match the number of features in the large network for a fair comparison. In that sense, we over-concatenate for a stronger empirical finding and still FSE scores of these concatenations are significantly high.

Reviewer comment (Theory):
The paper does not justify why the ridge regression is an appropriate method to measure the expressive power. As it is known to all, the network is a very complicated non-linear models. The true expressive power should be analyzed in terms of the function classes of these two kinds of networks.

Author response:
**"The true expressive power should be analyzed in terms of the function classes of these two kinds of networks."**\
In this paper (Section 2.2), the expressivity of a model follows from the set of functions that it can express through the linear combination of its features (that are fixed as they are learned features after training), which is different from any functions the model can express in theory without considering the optimization (learning) part. The true expressive power should be analyzed in terms of the function classes of these two kinds of networks.Optimization bias  may force models of different size to pick different features.

Reviewer comment (Experiments):
The paper does not justify why comparing overparameterized models with an ensemble of shallow models is important or meaningful. As an ensemble of small networks has fewer parameters than the large network, why is this a fair comparison?

Author response:
**"The paper does not justify why comparing overparameterized models with an ensemble of shallow models is important or meaningful. As an ensemble of small networks has fewer parameters than the large network, why is this a fair comparison?"**\
We include enough small networks in the ensemble to match the total number of parameters present in the large network for a fair comparison.

Reviewer comment (Theory):
Although the case analysis in section 4 is interesting, the result is only applicable to one very specific data distribution. Can the authors connect the data distribution to more general cases?

Author response:
**"Although the case analysis in section 4 is interesting, the result is only applicable to one very specific data distribution. Can the authors connect the data distribution to more general cases?"**\
Section 4 aims to give insights into why we observe the results in the previous sections. Some assumptions are needed to illustrate these observations in a simpler setting.

Reviewer comment (Experiments):
Why use a linear layer to define the FSE? Why not allow yourself to use a nonlinear layer? Other works from the model stitching literature have done this, e.g. [Bansal et al. (2021)](https://arxiv.org/abs/2106.07682). I would not be shocked if the main result from 3.2 didn’t hold much for a nonlinear version of FSE.

Author response:
As explained in Section 2.2, the model makes predictions based on the linear combination of its (last layer) features - each output neuron (before softmax) is linear combination of these features. Hence, the expressive power of a model is dictated by the linear span of the last hidden layer features (that is only the case for the last hidden layer), and this expressive power does not depend on non-linear relationships between the last hidden layer features. This expressive power's dependence on the linear combination of these features is the basis of the FSE measure, which quantifies one model's expressive capacity with respect to another.

Reviewer comment (Evaluation):
I see no error bars in some of the figures. Were these results based on one trial? Or are the error bars too small to see?

Author response:
Error bars are too small to see in most cases. Experiments are averaged over 5 trials.