Paper: Ask Your Distribution Shift if Pre-Training is Right for You (ICLR-2024)

Reviewer comment (Novelty):
**Unclear Contributions**. The contribution of the paper is unclear. Both theoretical and empirical contributions are mild. See next point for comment on theoretical analysis. The paper suggested to use initialization with a pretrained model and training on a balanced dataset, which is a standard practice anyway.

Author response:
The reviewer states that “the paper suggested to use initialization with a pretrained model and training on a balanced dataset, which is a standard practice anyway.” While training on balanced data has indeed been suggested by previous work, our work (in Section 6) suggests that, as a principle, this balanced dataset can be small and even non-diverse when fine-tuning a pre-trained model. We believe that this intuition is valuable for informing dataset curation processes.

Our work also offers an understanding of why pre-training in conjunction with a method for handling bias (such as balancing a dataset) is effective for robustness, which has not been previously studied to our knowledge.

Reviewer comment (Presentation):
**Presentation issues**. ER is a central measure used in the paper, but is not defined. How does the y axis label of Figure 5 (right) relate to MG defined in (4)?

Author response:
We are surprised by the reviewer’s comment that *effective robustness* (ER) is not defined, as it is defined in the  Section 2. To further clarify this metric, we added a longer definition in the Appendix in the revision.

Reviewer comment (Presentation):
**Q1** In Figure 2 (left), if the training data only has pictures of _indoor dogs_ and _outdoor cats_, then are the examples of _outdoor dogs_ and _indoor cats_ inside or outside the support of the training distribution?

Author response:
If the training data only has pictures of indoor dogs and outdoor cats, then examples of outdoor dogs and indoor cats are out-of-support. Here, we expect that a model could suffer both from biases and from poor extrapolation to this previously unseen group.

Reviewer comment (Experiments):
**Q2** In Figure 6(b), how does finetuning a pretrained model on female only dataset (i.e. female only examples from the original dataset) compare with the other results?

Author response:
We thank the reviewer for raising this point. In experiments that are not currently included in the paper, we found that fine-tuning a pre-trained model on a female only dataset is ineffective (the model still performs poorly on blond males). This might be because there are other attributes spuriously correlated with hair color even among just the female population, and this is why we opt for counterfactual editing. We will include this ablation in future revisions.

We would like to note that we are not claiming that counterfactual data is the only or most effective way to de-bias a dataset. The purpose of this experiment is to illustrate that de-biasing a small dataset and fine-tuning a pre-trained model on it can be effective, regardless of the de-biasing method.

Reviewer comment (Theory):
Another fundamental issue for Theorem 1 is that the model is considered deterministic. Even if you use random initialization and stochastic gradient solvers, you end up with the same solution. It is natural to use a stylized model as a starting point to build the analysis. Then I would expect to see how this analysis generalizes to the case of non-convex models where there is inherent learning stochasticity. Yet, without providing anything else, the paper jumps to experiments that are all based on neural network models. This is another major gap. Actually, there is not really a "pre-training" thing for convex models – regardless of the order you feed data to the model, it always converges to the same optimal solution. The resulting model solely depends on the training data and is irrelevant to initialization or the training process.

Author response:
While the model in Theorem 1 is deterministic, crucially, it depends on initialization (in our case, whether the model is pre-trained or randomly initialized). Even though the loss is convex, there are multiple equally good solutions and the initialization specifies which is selected. Pre-training thus can affect the learned model, even in this convex setting.

See the general response for concerns about the simplicity of the theoretical analysis.

Reviewer comment (Theory):
I don't understand why the proposed "in-support shifts" would change the classification boundary at all. Imagine a max-margin classifier (e.g., SVM) and a binary classification task for cat and dog images. (Note that Logistic Regression requires the same probability for both classes. You cannot directly apply it to unbalanced classification problems.) Regardless of the relative proportion for cat or dog images, the underlying distribution for cats and dogs is invariant. A proper classifier would find the decision boundary somewhere between the generating distributions for both classes. I don't see why this is considered "not robust". Or a more important question – what is the robustness considered in this work? The definition for the central notion of this work is not provided.

Author response:
We are confused by the reviewer’s comments about models being agnostic to in-support shifts. Consider [1], for example.

**Definition of robustness**

In our work, we give both an intuitive and formal definition of robustness. Intuitively, as described in our introduction, we consider the robustness of a method to be its ability to succeed on a shifted distribution that differs from the reference distribution the model is trained on. 

Formally, we define and evaluate the robustness of a model via the *effective robustness* metric [2] (we define this metric in Section 2). To further clarify this metric, we added a longer definition in the Appendix in the revision.

Reviewer comment (Reproducibility):
For the proposed approaches – use intervention techniques at pre-training to prevent exploiting biases; and fine-tune on small, non-diverse but debiased datasets – the paper merely cites existing works for these techniques without any description. This renders this work not self-contained and incomplete and this cannot be counted as technical contribution of this work.

Author response:
We would like to point the reviewer to Appendix B.4. Here, we indeed give a description of the *Deep Feature Reweighting* (which we assume the author is referring to) and also carefully describe our implementation of this method. Our contribution here is not a specific approach, but rather a guiding principle of combining pre-training with an intervention that can handle biases. Thus, we did not consider the details of the specific choice of this intervention to be worth including in the main paper.

Reviewer comment (Presentation):
Format: Appendix is not cut from the main paper. The PDF provided for the main paper is this 32-page document.

Author response:
We refer the reviewer to the ICLR 2024 call for papers and author guidelines:

- The ICLR 2024 author guide states: “We encourage authors to submit a single file (paper + supplementary text). Please mark the supplementary material clearly.”
- The ICLR 2024 call for papers states: “Authors may use as many pages of appendices (after the bibliography) as they wish.”

Reviewer comment (Writing):
Theorem 1, key variable $w_{ref}$ is undefined.

Author response:
The reviewer expressed confusion about $w^*_\text{ref}$ being undefined. We state that $w^*_\text{ref}$ does not depend on the initialization $w_\text{init}$. We also state that it is a property of the reference dataset. In the revision, we clarify this when we first refer to $w^*_\text{ref}$.

Reviewer comment (Presentation):
Appendix B 3.2, which is referred to in Page 6, does not exist.

Author response:
We are unsure why the reviewer is unable to view Appendix B.3.2. We have opened and download the pdf on multiple browsers, and in each case the Appendix is included in the file. We also note that, as we state in the main text in Section 4.2, we actually give the full details in the splitting method in Appendix B.3.1 and not B.3.2.

Reviewer comment (Presentation):
Appendix should not be submitted under the main paper.

Author response:
We refer the reviewer to the ICLR 2024 call for papers and author guidelines:

- The ICLR 2024 author guide states: “We encourage authors to submit a single file (paper + supplementary text). Please mark the supplementary material clearly.”
- The ICLR 2024 call for papers states: “Authors may use as many pages of appendices (after the bibliography) as they wish.”

Reviewer comment (Theory):
Although more details are presented in the Appendix, I don't think that the setup for Theorem 3.1 is presented well in the paper. It is not clear why the logistic regression assumption is needed and the proof is not referenced in the main paper. It is also unclear what $proj_{W_{ref}}$ refers to as it is never formally introduced. Is orthogonal to be interpreted mathematically or figuratively? Also "[...] while the initialization determines how the model extends outside of $W_{ref}$": since $W_{ref}$ is contained in $proj_{W_{ref}}$. It appears to me like both terms are influenced by $W_{ref}$, not just the first term.

Author response:
**Notation in Section 3**

To clarify the reviewer’s confusion about Theorem 1:

1. $proj_{W_\text{ref}}$ is the projection operator onto the subspace $W_\text{ref}$. We apologize for the confusion and clarify this in the revision.
2. Orthogonal is to be interpreted mathematically; $w_\text{init}-proj_{W_\text{ref}}w_\text{init}$ is orthogonal to $W_\text{ref}$ because we remove the component that lies within $W_\text{ref}$. This is also why this second term would not influence the model’s output on any point within $W_\text{ref}$ (because the product of this point and $w_\text{init}-proj_{W_\text{ref}}w_\text{init}$ would be zero).
3. The proof is in Appendix A and is referenced in the paper.

Reviewer comment (Presentation):
Figure 1's legend is unreadable due to overlapping text. I was still able to get the intuition but the authors should fix this.

Author response:
**Overlapping text in Figure 1**

We are confused by the author’s comments about the overlapping text in Figure 1. We were unable to reproduce this issue when

1. Downloading the PDF and viewing it locally.
2. Viewing the PDF in Safari, Chrome, Brave.

We suggest that the reviewer uses one of the methods above to view the PDF.

Reviewer comment (Experiments):
In **Section 4/Figure 3**, there are multiple variables which are being ablated at the same time. The measured effective robustness is with respect the performance of ResNet18 models, however the pretrained models, which are compared to, have various architectures. It is not clear whether the perceived robustness is due to purely transfer properties from pretraining strategies or from larger model size.

Author response:
**Measuring the robustness of pre-trained models with various architectures with respect to ResNet-18 models trained from scratch (1)**

The reviewer expressed concerns that we measure the robustness of pre-trained models with various architectures with respect to ResNet-18 models trained from scratch. We would like to point out that the pre-trained models include ResNet-18s (pre-trained on ImageNet) in addition to other, larger architectures, so there is already a direct comparison between models with the same architecture. That said, in future revisions we will augment this result by adding models trained from scratch with ResNet-50 and DenseNet-121 architectures (we also tried ViTs, but these tend to perform poorly when trained from scratch on small datasets [1]).

Reviewer comment (Experiments):
Biased datasets are also susceptible to an *in-support*/*out-of-support* analysis, however one is not given at the study. Specifically, one can imagine the extreme case where some minority group probabilities go to 0, yielding them completely *out-of-support*. For example, imagine the case in **Figure 2 (right)**, where cats are also observed during the night. How does this impact the pretraining algorithms? Current analysis seems to suggest that pretraining would be able to handle the more extreme cases better, which is counter-intuitive. I suggest studying systematic generalization tasks, where some combinations of generative attributes are completely not represented in the training set, but they exist in the test set in a balanced manner. See [1,2].

Author response:
**Biased datasets with zero minority sub-group probabilities (2)**

We thank the reviewer for pointing out that there might also be biased datasets in which the minority group probabilities are close to zero. This is an important point that we intend to clarify in future revisions. Indeed, in this extreme case a model would suffer from the bias. However, a model could also suffer from poor extrapolation, since the minority groups represent a previously unseen setting. We do not categorize such a shift as in-support for this reason.

The reviewer says that “current analysis seems to suggest that pre-training would be able to handle the more extreme cases better, which is counter-intuitive”. We agree that this result as stated by the reviewer would be counter-intuitive; however, this is not what our current analysis suggests. Our analysis suggests that pre-training can help with extrapolation, implying that a pre-trained model might be more robust *than a model trained from scratch under the same extreme case*. Notably, the analysis does not suggest that a pre-trained model would perform better in this extreme case than in the case where data from the minority group is available (since, indeed, a pre-trained model would likely suffer more from biases).

Reviewer comment (Evaluation):
In **Section 5/Figure 5**, the first correlation estimate is not strong enough to indicate negative correlation of (i) (and thus complementary effects between pretraining and debiasing methods). Please provide with a confidence interval of the correlation estimate, as it seems statistically possible that it is very close to 0. On the other hand, the correlation of (ii) is poorly motivated and it can be tautologically positive, in which case it is of no logical inference value.

Author response:
**Margin gain correlations (3)**

In the general response, we discuss why, in our view, even uncorrelated margin gains imply that interventions are complementary.

We are confused about why the reviewer believes that the correlation of (ii) is poorly motivated and might be tautologically positive. In our understanding, for something to be tautologically true means that they mean the same thing. In this case, applying two interventions separately and summing their independent gains and applying them together and measuring the gain are not the same. While some correlation is to be expected, the high positive correlation provides evidence that applying the two interventions together yields the combined benefits of applying each individually.

Reviewer comment (Experiments):
In **Section 6**, the proposed form of curated dataset is made so that the spurious feature which is balanced during test-time is completely omitted, namely the “gender” attribute of CelebA. If there is no spurious correlated feature during training, then the transfer problem corresponds to an extrapolation one, for which probably a classifier with simple augmentation/regularization might just work. The construction of counterfactual data, beyond being difficult to achieve in most cases, needs to be ablated to demonstrate that it is a necessary component of the curation process.

Author response:
**Counterfactual data in the curation process (4)**

The reviewer points out that “if there is no spurious correlated feature during training, then the transfer problem corresponds to an extrapolation one”. This is exactly the purpose of the dataset curation process; since our analysis suggests that pre-training can help with extrapolation, if we can set up a dataset where the problem is just extrapolation then a pre-trained and fine-tuned model might do well. While there may be other strategies besides pre-training that can help with extrapolation, we do not believe that “simple augmentations/regularization” would be an effective solution in general.

We are not claiming that counterfactual data is the only or most effective way to de-bias a dataset. The purpose of this experiment is to illustrate that de-biasing a small dataset and fine-tuning a pre-trained model on it can be effective, regardless of the de-biasing method. We intend to clarify this in future revisions.