Paper: On the Joint Interaction of Models, Data, and Features (ICLR-2024)

Reviewer comment (Theory):
Like any theoretical framework, many assumptions went into the combinatorial analysis. In particular, this framework assumes features and datapoints are each either dominant *or* rare, a dominant/rare datapoint always has the same number of dominant/rare features, etc.

Author response:
We fully agree that many assumptions were made. The way we see it is that deep learning's complexity necessitates some level of simplification for tractable analysis. In this case, since the resulting framework is still useful for understanding empirical phenomena, we feel the assumptions are justified.

Reviewer comment (Evaluation):
The framework is less useful for understanding the learning process of networks, such as why some runs might collapse while others successfully learn the desired features.

Author response:
This is correct. The framework as it stands does not describe the learning process, which is an extremely complicated subject itself. However, we postulate that techniques similar to [1] could be used to study the optimization of a feature model like ours in the future.

Reviewer comment (Writing):
Some minor typos: section 5, data generating process paragraph's second-to-last sentence samples $n_r$ rare, not dominant, features. Appendix C, equation line 34 the two $\not = \emptyset $ could maybe instead be $= \emptyset$.

Author response:
Thank you for the close reading and for catching these typos! We have fixed them in the revision.

Reviewer comment (Experiments):
While simplifying features and datapoints to be either dominant *or* rare is good enough for the section 6 experiments, have you considered modeling the rare-dominant variation as a spectrum instead of a binary? For instance, looking at figure 2b, I'm not sure where I would want to draw a line to separate the rare from the dominant features. Even putting the line somewhere around x=6.5, there is still a relatively large deal of variation of frequency of occurrence in both the rare and the dominant feature types.

Author response:
This is an excellent question! We have indeed considered the possibility of modeling it as a spectrum, which we discussed in Appendix D.1 and D.4. In our preliminary investigation, we found that modeling the feature spectrum may require introducing more free parameters (e.g., Zipf’s distribution), and can drastically increase the complexity of analysis. Given that this is our first foray into the topic, we opted for the simpler binary model. This is an important problem and we plan to explore it in more detail in future works.

Reviewer comment (Theory):
I definitely agree with "the empirical phenomena of deep learning can be understood at many different layers of abstraction". However, I think the model proposed in this paper is too simplistic. The implicit biases of deep learning are core to some of the merging phenomenon that we see these days and the models that the authors propose fails to capture that. I also think that a good toy model should leave the door open to generalizations and getting closer to real world practice (for example, for the random features model of deep learning, there is a very obvious way to move towards making it more realistic). But the models that the authors propose is too abstract and it is not clear what simplifications are made to the real problem to arrive at the proposed model.

Author response:
We definitely agree that our model is unconventional and more abstract than most existing works but similar assumptions have been made in prior work [6]. As is the case for all theoretical models, simplifications are required to keep analysis tractable. Despite the simplification, our model already exhibits some interesting behaviors and offers insight into empirical phenomena so we believe that it adds value to the current discussion on feature learning. 

In the case of random feature models, we feel that it is a little unfair to say in hindsight that there was a very obvious way to move towards making it more realistic. When NTK first came out in 2018, it was thought to be very promising. Indeed, despite its simplification, it offers clear value in the analysis of some phenomena in deep networks. However, it only later became clear that there was a big gap between NTK and real deep learning (despite many efforts to close this gap). More recent works (many of which you already brought up) rely on quite distinct analysis tools than those used in NTK and, still, it’s not clear if any of them is close to real-world practice.  As such, while we agree (and have acknowledged in Section 5, Appendix D.1 and D.4) that there is a gap between our proposed framework and practice, we feel it’s a tall order for the current state of deep learning theory to have no strong assumptions.  Just like NTK analysis, we hope that our approach will ultimately prove valuable (albeit along different dimensions), despite its simplicity.

Reviewer comment (Reproducibility):
It is not very clear how the authors set the hyperparameters in their model (e.g., the thresholds).

Author response:
As detailed in Appendix F.10 (referred to at the end of Section 4), our conclusions are robust across a range of parameter settings so we believe that the settings used in our paper should be appropriate for most situations.

Reviewer comment (Presentation):
The notations are a bit confusing (i, j, k, etc.). I suggest authors avoid using these generic letters. Also, the paper will benefit greatly from a figure that summarizes all the notations (p_d, p_r, etc.). It will also help explain the method.

Author response:
We apologize for the confusion. There is already a centralized list of notations in Appendix A which we could not include in the main text due to space constraints (this is referred to in the second paragraph of section 5). Beyond this, could you be more specific about where the $i,j,k$ are causing confusion so we could better address the confusion? $i,j,k$ are used to denote an arbitrary member of the ordered collection so we feel this is in accordance with the standard usage.

Reviewer comment (Theory):
The authors "prove" GDE in their model without the explicit assumption of calibration. But the model is very abstract/high-level and I'm not sure if the assumptions that they make are stronger or weaker than calibration.

Author response:
We want to clarify that we *do not* claim that the assumptions we make are weaker than calibration (since neither is a clear superset of the other), but we do believe that it sheds more light on the underlying reasons behind GDE. Only stating the model is calibrated does not tell us much about the model or the data, but our assumptions do. In fact, our framework tells us how one can break our assumption (and calibration) through data distribution interventions. Of course, it is an important open question why so many natural distributions seem to satisfy these assumptions.

Reviewer comment (Evaluation):
Looking at figure 3 (a) and 3 (c), it seems that the observations made from them are not that significant. Am I missing something? How does the choice of the thresholds affect your observations?

Author response:
Apologies for missing the first part of your question earlier. In 3 (a), the observation we are relying on is that data points with a small number of features are much more likely to have very high confidence under the ensemble and vice versa. For a concrete example, it is much more likely for a data point with 0.8 confidence to have 25 features than a data point with 0.4 confidence to have 25 features. As such, we believe that it’s fair to say that data with fewer features tend to have higher confidence. For 3(c), we claim that the more features the models share, the lower bound on their shared errors becomes higher, which can be clearly observed in the figure. This *does not* mean that the fewer features the models share, their shared errors would necessarily be smaller. Instead, both models would make random guesses (recall that we make similar assumptions in the conceptual model). This means that a pair of models can have a high or low amount of shared errors due to randomness, but the lower bound on the shared error will become higher if they share more features. We believe that the figure supports this claim.

Reviewer comment (Presentation):
Figure 1 is a bit vague. It is recommended to replace it with a clearer version.

Author response:
Thank you for the support of our paper! We are glad that you find our work valuable.

Regarding Figure 1, to better address your concerns, could you elaborate on which aspects of the figure you found unclear? This will help us make more targeted improvements. 

The goal of Figure 1 is to demonstrate that our definition of features can capture images with redundant information and images that contain uncommon instances of a given class. To make this clearer, we have updated the figure in the paper with texts to better reflect the core idea. Does this resolve your concerns? If not, we’d be happy to make more adjustments.

Reviewer comment (Theory):
Currently theoretical framework analyzes binary classification, does the analysis extend to multi-class classification?

Author response:
We discuss the necessary steps for the extension to multi-class classification in Appendix D.3.

Reviewer comment (Experiments):
Does the framework allow for introducing distribution shifts and etc?

Author response:
This is a great question. We believe our framework can accommodate distribution shifts. For instance, the simplest form of covariate shift one could introduce is to add a new set of features that are not present in the training data with some probability at test time. Since the model has not seen these new features during training, we may assume that individual members of the ensemble will make random predictions (but the prediction of different models will be correlated) if the data points only contain the new features (the same assumption for features that the models did not learn). One can also introduce a covariate shift on existing features but that would require imposing a non-uniform distribution over the existing features. We leave the exploration of these directions to future works.