Paper: Unraveling the Key Components of OOD Generalization via Diversification (ICLR-2024)

Reviewer comment (Novelty):
What are the unique advantages of “diversification” methods to solve OOD problems? Does it contribute much to OOD community to study the components of “diversification”? More discussion of motivation and related works are needed.

Author response:
> What are the unique advantages of “diversification” methods to solve OOD problems？Does it contribute much to the OOD community to study the components of “diversification”?

We kindly refer to the ‘The focused scope of our work’ section in the general response for a detailed answer. We will be sure to add more discussion in the next revision.

Reviewer comment (Experiments):
In addition to the methods used in the paper, does it need to consider more models to support the conclusions of the paper?

Author response:
> More complicated diversification loss, approaches, need to be included. In addition to the methods used in the paper, does it need to consider more models to support the conclusions of the paper?

At the time of writing, to the best of our knowledge, there were no other diversification losses or approaches that are fundamentally different from D-BAT and DivDis. The two methods differ not only on diversification losses but also on the optimization strategies, thus representing a spectrum of possible dynamics in this learning framework well.

Throughout the paper, we convey crucial messages that provide insights to guide the future design of diversification methods and the usage of current ones, and each message is supported by theoretical proofs (proposition 1, proposition 2), and experimental verifications as below:

- For proposition 1, we further justify the performance of diversification methods is highly dependent on spurious ratio by firstly verifying the solution of synthetic 2D illustrative example and secondly showing the conclusion holds in practice with datasets in different scales (M/C, M/F, and CeleA-CC).
- For proposition 2, we show with 10 models (spanning different training strategies) that diversification alone cannot lead to OOD generalization the inductive bias brought by the used learning algorithm is important.

For the reasons above, we believe our conclusions are well-supported. However, we would be very glad to provide further evidence, but was wondering which kind of models or approaches the reviewer would be interested to see?

Reviewer comment (Evaluation):
While the analysis is compelling, there is uncertainty about whether the identified limitations matter in practice. Specifically, what would happen if model selection is performed over multiple architectures and multiple diversity algorithms? Is there a risk that the results obtained on a cross-validation set may not generalize to the test set?

Author response:
> What if we do model selection over multiple architectures and multiple diversity algorithms? Is there risk that the results we get on a cross-validation set may not generalize to the test set?

As shown in Figure 3, diversification methods do not work well without pretraining, so the model selection boils down to choosing a diversification loss and a pre-trained model (which includes pretraining strategy and architecture).

As mentioned above (also shown in Figure 3), the pretraining performance on ImageNet is surprisingly not a good proxy for selecting the pretraining strategies. However, a validation set can be used as one of the ways of model selection, and the risk of results' generalization to the test set depends:

- When the validation set is i.i.d to the test set, we can safely do the model selection. Through this work, we have provided important axes (diversification loss, pretraining methods, and their combination) to identify the best model.
- In terms of cross-validation, i.e., assuming the validation set is i.i.d to the training set and not to the test set, it is not possible to do the model selection. To the best of our knowledge, existing methods in the field do not consider this case and there is generally no solution for this setting.
- Finally, when the validation set is i.i.d to the unlabeled data, which may be non-i.i.d to the test set, it depends on how many spurious features are present in the dataset.
    - When there is only 1 spurious feature, knowing the spurious ratio of the test set can help in simulating the corresponding validation set, and the model selection can be done across diversification loss, architectures, and pretraining strategies.
    - When there is more than 1 spurious feature, knowing such summary statistics is still good, but there is a risk that the results may not generalize to the test set. (see further discussion in the next answer)

Reviewer comment (Writing):
The main takeaway of the paper is unclear. Should diversification algorithms be avoided? What is the better alternative? A potentially better message from the paper could be that it identifies the axes on which model selection should be done, if some summary statistics of the test set can be provided in advance.

Author response:
> Should we not use diversification algorithms? What is the better alternative? I feel a better message from the paper can be that it identifies the axes on which model selection should be done, if some summary statistics of the test set can be provided in advance.

It is sensible to have multiple diverse hypotheses to tackle the spurious correlation problem, as the model does not know apriori which feature is semantic and which feature is spurious. Thus, we think that diversification methods can be useful in practice. However, further research would be needed to understand them better and develop better versions (e.g., ones that are more robust to the change in the spurious ratio)

The purpose of this work is exactly to provide a significant analysis of which axes we need to consider, to what extent (in terms of which statistic and models) we need to consider, and, as discussed above, the potential more efforts required for the second stage (disambiguation) to find the best model (i.e., the model that learns the true hypothesis). This also opens up new directions for developing better diversification methods that take all these axes into account.

We hope we have addressed your questions, and we remain at your disposal for any further questions you may have.

[1] Selvaraju et al., Grad-cam: Visual explanations from deep networks via gradient-based localization, CVPR 2017

Reviewer comment (Experiments):
Can the authors set up a model selection experiment where the spurious ratio of the unseen test set is known and the test set is simulated? Would this always lead to the correct model? What other information about the test set is needed? Can such information be summarized, or is that not possible—implying there is no way to know the best model a priori?

Author response:
> What are the summary statistics of the test set that the above procedure would need to know? For example, if the spurious ratio of the (unseen) test set is known, can that be used to simulate a pseudo-test set and then do model selection over it?
> 

As discussed above, the number of spurious hypotheses and the spurious ratio of each of those hypotheses are good summary statistics to know. In most cases, the number of spurious hypotheses is equal to 1 and there will therefore be only one spurious ratio.

However, as shown in Sec 5.3, when there are more than 1 (2 in this case) spurious hypotheses ($h_1$ & $h_2$) present in the training set, things can be a bit different. Having the spurious ratio for each spurious hypothesis may not help, as different types of the model may latch on a different spurious hypotheses because of their inductive bias, and it’s known to be hard to measure such inductive bias apriori. Thus, the given spurious ratio may not be valid to simulate a pseudo test-set which is valid for all models.

Although there is risk in some cases where no good model selection strategy is available, we would like to highlight and clarify that, the efforts done in this work are focused on the diversification stage (first stage of the two-stage framework shown in Sec 3.2). In cases where there is no reliable way of model selection, which can be known when given the above summary statistics, an ultimate way of model selection can be considering all the axes (diversification loss, pretraining, architecture) above and conducting the second stage (the disambiguation stage) with the hypotheses produced by spanning those axes. Therefore, in the second stage, the best diversification loss & pretraining strategy can still be easily selected with a small amount of additional human supervision, e.g., labeled data points or human observation on the Grad-CAM [1] feature map. This means the diversification method is still useful but may have to pass more burden to the second stage in order to reliably find the hypothesis that is close to the true one.

Reviewer comment (Writing):
In particular, there is some terminology and notation that can be improved for greater understanding. The "spurious ratio" index is a poorly named quantity because it's literally the accuracy of the selected hypothesis with respect to the true hypothesis h*. Namely, h* has the maximum spurious ratio value of 1.0, but it's definitely NOT spurious as it's the true hypothesis. Another name, like the "agreement ratio" would be much clearer.

Author response:
Thanks for pointing out the confusion. Indeed, this can be confusing and we will carefully consider a better name (e.g., agreement ratio as suggested) in the next revision. The new definition can be:

Definition 1 (Agreement Ratio) Given a hypothesis h, the agreement ratio $r_D^h$, with respect to a distribution $D$ and its true labeling function $h^∗$ is defined as the proportion of data points where $h^∗$ and $h$ agree, i.e., have the same prediction: $r_D^h = \mathbb{E}_{x \sim D}[h^*(x) = h(x)]$.

We also would like to comment that, though $h^*$ has $r_D^{h^*}=1$, this relation is not symmetric, i.e., a hypothesis $h$ with $r_D^{h}=1$ does not mean $h=h^*$. This is because the quantity is defined on a distribution $D$. For example, on $D_t$ where complete spurious correlation exists, there exists a $h$ such that $r_{D_t}^{h}=1$, but on unlabeled data $D_u$, the same $h$ may have $r_{D_u}^{h} \in [0,1]$. In the scope of our work, we mostly examine this quantity on $D_u$, and we assume the test set $D_{ood}$ is a distribution where no spurious correlation exists, i.e., $r_{D_ood}^{h} \approx 0.5$ (so that the test set can reflect how well the diversification methods find the true hypothesis $h^*$). We hope this could address your confusion.

Reviewer comment (Presentation):
Similarly, the plots on the left side of figure 2 do not seem to agree with the description of the synthetic problem in the first paragraph of section 4.1, which made it very hard to understand what was being communicated.

Author response:
As mentioned in the description of Sec 4.1, $x_1$ is the horizontal axis and in this case, the reviewer is correct that $h^*(x)=\mathcal{I}\{x_1 > 0\}$ would be 1 for the points on the right. This is indeed a typo, and we thank the reviewer for noticing it. We will correct it. However it should be noted that this doesn't make any difference in the conclusions made in Proposition 1, and in the behavior illustrated in Figure 2, given that we're in the symmetric binary classification case and the classes are balanced.

For defining the hyperplanes, we wanted to match what D-BAT does (in Figure 17) and therefore used the radian of the classification plane w.r.t horizontal axis $x_1$ as mentioned in Sec 4.1, instead of the normal. We hope this allows for a better readability.

Reviewer comment (Theory):
The no-free-lunch findings discussed in the paper are known in the supervised learning setting, so they definitely need to hold for this harder setting with labeled + unlabeled data. Could this research direction become even more constructive by making certain statistical assumptions? In the paper, D_u can be any "out of distribution" distribution; and the capacity of the learning algorithm is assumed to be infinite in proposition 2, even though in other parts of the paper, inductive biases of different model classes are highlighted (which suggests that there the model class has less than infinite capacity in practice).

Author response:
Dear reviewer shsk,

We thank you for acknowledging our clarifications and are glad they addressed most of the concerns. We also thank you for providing additional details on one of your questions about NFL, which we further address below.

If we consider a two-stage diversification method as one learning algorithm that outputs only a single hypothesis, then we agree that NFL is sufficient. However, this explanation does not consider the specific structure of diversification methods, which is 1) two-staged, where 2) the 1st diversification stage outputs multiple hypotheses K. **In contrast, our result takes this structure into account and does not follow directly from NFL. Below, we share our insights.**

1. As discussed in our rebuttal and the DivDis work, the 2nd stage does not need to be limited to statistical learning, and, e.g., human feedback (e.g., using Grad-CAM or other explainability methods) can be used to choose the resulting hypothesis out of K. Moreover, the 2nd stage can be applied to different downstream applications, e.g., in the Waterbirds case, one can choose between background or bird class predictors by only changing the 2nd stage. **Therefore, it is important to understand the performance of the 1st stage alone, which needs to output K “most useful” hypotheses (instead of a single one in the standard supervised learning setting). NFL does not cover this scenario** 
2. Our Proposition 2 focuses precisely on the 1st diversification stage and **provides an understanding of how its performance scales with K**, even when considering the best-case scenario for the 2nd stage, and **suggests that the 1st stage is the most crucial one** for the success of a diversification method as a whole, and it is more beneficial to focus on improving this stage instead of the 2nd one.

Overall, this theoretical result motivates our empirical study, where we find that in practice (Tab. 1), scaling K does not improve the performance of the whole diversification algorithm even when considering the 2nd stage to be an oracle, and the choice of the inductive bias, e.g., in the form of the architecture and pretraining method still plays a crucial role (Fig. 3 and Tab. 1).

We thank you for your comments, and we will incorporate these points into our updated manuscript to make it more clear.

Reviewer comment (Experiments):
The results for DivDis in table 1, section 5.2 for increasing K are not convincing because alpha needs to increase as the square of the number of hypotheses in the set (i.e., O(K^2)). Otherwise the regularization becomes too strong and dwarfs the empirical risk. But the text makes it seem like a fixed alpha was used, which would needlessly harm DivDis' performance as K gets larger.

Author response:
We assume the reviewer means alpha needs to *decrease* (rather than increase) when increasing K in order for the regularization not to become too strong. The reviewer's concerns are correct given Equation 2, as the diversification losses are summed, and the value is not normalized w.r.t the number of hypotheses K. In practice, however, we always *average* the diversification losses instead of summing them up, which results in having $\alpha_K = \frac{\alpha}{K(K-1)}$. The $\alpha$ in the numerator is what we keep fixed in practice, which corresponds to scaling the loss by a factor of $~1/K^2$ as the reviewer rightfully suggested. We thank the reviewer for noticing this confusion. We will update the loss function to reflect the normalization term.

Reviewer comment (Presentation):
Regarding figure 2, h* is defined in the text such that instances with x1 > 0 should be labeled as positive examples. I assume that x1 is the horizontal axis such that points to the right are positive. But the plot and its legend shows points to the left are positive (Class 1). Is my interpretation correct here?

Author response:
As mentioned in the description of Sec 4.1, $x_1$ is the horizontal axis and in this case, the reviewer is correct that $h^*(x)=\mathcal{I}\{x_1 > 0\}$ would be 1 for the points on the right. This is indeed a typo, and we thank the reviewer for noticing it. We will correct it. However it should be noted that this doesn't make any difference in the conclusions made in Proposition 1, and in the behavior illustrated in Figure 2, given that we're in the symmetric binary classification case and the classes are balanced.

For defining the hyperplanes, we wanted to match what D-BAT does (in Figure 17) and therefore used the radian of the classification plane w.r.t horizontal axis $x_1$ as mentioned in Sec 4.1, instead of the normal. We hope this allows for a better readability.

Reviewer comment (Theory):
Assuming it is correct, hyperplanes are typically defined by their normal, in which case it should be that h*(x) = h(x; 0), not h*(x) = h(x; pi/4), since theta=0 radians points in the positive horizontal direction to the right, whereas theta=pi/4 radians points up to the top of the page.

Author response:
As mentioned in the description of Sec 4.1, $x_1$ is the horizontal axis and in this case, the reviewer is correct that $h^*(x)=\mathcal{I}\{x_1 > 0\}$ would be 1 for the points on the right. This is indeed a typo, and we thank the reviewer for noticing it. We will correct it. However it should be noted that this doesn't make any difference in the conclusions made in Proposition 1, and in the behavior illustrated in Figure 2, given that we're in the symmetric binary classification case and the classes are balanced.

For defining the hyperplanes, we wanted to match what D-BAT does (in Figure 17) and therefore used the radian of the classification plane w.r.t horizontal axis $x_1$ as mentioned in Sec 4.1, instead of the normal. We hope this allows for a better readability.