Paper: Spurious Feature Diversification Improves Out-of-distribution Generalization (ICLR-2024)

Reviewer comment (Experiments):
The paper considers the setting of ensembling two individual models (FalseFalseTrue) and show that such ensemble indeed improves over single model's OOD performance on MultiColorMNIST. Is this finding generalizable to ensembles of more than two models? If we increase the number of models in Table 1, could the ensemble's performance potentially see further improvement?

Author response:
Thank you for your valuable question. We conducted experiments on MultiColorMNIST and observed that increasing the number of models in the ensemble leads to significantly improved OOD performances. The table below demonstrates the results for different ensemble sizes. When the ensemble number is 1, it signifies the use of a single model without performing model ensemble. On the other hand, an ensemble number of 16 indicates that we independently trained 16 models with different initializations and used their ensemble for predictions. It is evident that increasing the ensemble number greatly boosts OOD performance.

For instance, consider $p=0.8$ (as defined in Definition 1, where a larger $p$ indicates a larger distribution shift). The OOD performance of a single model (ensemble number equals 1) is 49.33\%. However, the ensemble of two models achieves 55.92\% OOD accuracy, while the ensemble of 16 models further increases the OOD accuracy to 64.85\%!

These findings also provide insights into the effectiveness of the model soup approach [1], which involves averaging multiple checkpoints trained with different hyper-parameters. We appreciate your suggestions, and we have included these results in Appendix C.3.1.

[1] Mitchell Wortsman, et.al., Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time


| $p$  | 0.70              | 0.75              | 0.80              | 0.85              | 0.90              |
|--------------|-----------------|-----------------|-----------------|-----------------|-----------------|
| Ensemble Number |                 |                 |                 |                 |                 |
| 1            | 71.66 ± 2.06 | 60.68 ± 2.23 | 49.33 ± 2.02 | 37.74 ± 1.58 | 26.74 ± 1.05 |
| 2            | 78.88 ± 1.24 | 68.34 ± 0.89 | 55.96 ± 0.77 | 42.91 ± 0.64 | 29.89 ± 0.63 |
| 4            | 84.39 ± 1.33 | 74.00 ± 1.26 | 62.04 ± 1.32 | 47.92 ± 1.17 | 32.74 ± 0.75 |
| 8            | 85.64 ± 1.22 | 75.73 ± 1.62 | 63.52 ± 1.61 | 49.15 ± 1.23 | 33.67 ± 0.93 |
| 16           | 86.76 ± 0.55 | 77.31 ± 0.87 | 64.85 ± 1.09 | 50.63 ± 0.69 | 34.47 ± 0.40 |

Table: Experiments on MultiColorMNIST

Reviewer comment (Evaluation):
I am not convinced that the performance gains of BANG are mainly due to better calibration as claimed in the paper and not because mixup/LS significantly improves OOD performance of finetuned models. While appendix E.6 shows that there are “oracle weights” to scale standard finetuned models when ensembling, it may not be that BANG models are bringing improvements for the same reason.

Author response:
We appreciate your insightful question and valuable suggestions regarding the "accuracy on the line" perspective. 
According to your suggestion, we have conducted additional experiments to address this aspect. To further investigate the performance of BANG, we examine the concept of ``Accuracy on the Line" [1, 2]. We try a lot of hyperparameters for the vanila fine-tuning. Specifically, we fine-tune the model using various hyperparameters, including learning rates ($1e^{-5}, 2e^{-5}, 3e^{-5}, 5e^{-5}$), training epochs ($4, 8, 10, 12, 16, 20$), and learning rate schedules (cosine, step decay). Notably, the default hyperparameters used in Section 4 and used in [3] are a learning rate of $3e^{-5}$, 10 training epochs, and a cosine scheduler. Weight averaging is applied to each fine-tuned checkpoint with the pretrained model. Figure 21 illustrates the OOD performance of each vanilla fine-tuned model, as well as the averaged model that combines the vanilla fine-tuned model with the pre-trained model. Interestingly, we observe that the OOD accuracy of the averaged model roughly forms a quadratic function with respect to the OOD accuracy of the vanilla fine-tuned model [2], rather than a linear relationship as described in [1].

Furthermore, we also show the performance of our BANG on the figure. BANG demonstrates significant robustness in OOD scenarios, surpassing the curve of expected performance. 

[1] John Miller et.al., Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization 

[2] Weixin Lian et.al., Accuracy on the Curve: On the Nonlinear Correlation of ML Performance Between Data Subpopulations

[3] Mitchell Wortsman et.al., Robust fine-tuning of zero-shot models

Reviewer comment (Experiments):
One way to demonstrate this would be analysis similar to “accuracy on the line” where there are models finetuned with mixup and LS at the same OOD accuracy as standard finetuned models. It would strengthen the claims of Section 4 if the BANG models in this setting outperform WiSE-FT even with finetuned models at the same OOD performance.

Author response:
We appreciate your insightful question and valuable suggestions regarding the "accuracy on the line" perspective. 
According to your suggestion, we have conducted additional experiments to address this aspect. To further investigate the performance of BANG, we examine the concept of ``Accuracy on the Line" [1, 2]. We try a lot of hyperparameters for the vanila fine-tuning. Specifically, we fine-tune the model using various hyperparameters, including learning rates ($1e^{-5}, 2e^{-5}, 3e^{-5}, 5e^{-5}$), training epochs ($4, 8, 10, 12, 16, 20$), and learning rate schedules (cosine, step decay). Notably, the default hyperparameters used in Section 4 and used in [3] are a learning rate of $3e^{-5}$, 10 training epochs, and a cosine scheduler. Weight averaging is applied to each fine-tuned checkpoint with the pretrained model. Figure 21 illustrates the OOD performance of each vanilla fine-tuned model, as well as the averaged model that combines the vanilla fine-tuned model with the pre-trained model. Interestingly, we observe that the OOD accuracy of the averaged model roughly forms a quadratic function with respect to the OOD accuracy of the vanilla fine-tuned model [2], rather than a linear relationship as described in [1].

Furthermore, we also show the performance of our BANG on the figure. BANG demonstrates significant robustness in OOD scenarios, surpassing the curve of expected performance. 

[1] John Miller et.al., Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization 

[2] Weixin Lian et.al., Accuracy on the Curve: On the Nonlinear Correlation of ML Performance Between Data Subpopulations

[3] Mitchell Wortsman et.al., Robust fine-tuning of zero-shot models

Reviewer comment (Evaluation):
In tables 2 and 7, we see that on the in-distribution ImageNet test set, finetuned(Mixup+LS) models outperform standard finetuned models (by ~1.5%), but BANG does not outperform WiSE-FT. Is there intuition for what BANG is doing in the in-distribution setting that explains this?

Author response:
Thank you for bringing up this intriguing phenomenon. We didn't notice this phenomenon before. We provide an possible explanation for this as follows: As we have demonstrated previously, the vanilla fine-tuned model tends to be over-confident. During the averaging process of WiSE-FT, there is a bias towards the fine-tuned model. Additionally, the fine-tuned model generally exhibits higher ID performance compared to the pre-trained model. Consequently, the ID performance of WiSE-FT can benefit from this bias towards the fine-tuned model. In contrast, BANG mitigates the issue of over-confidence in the fine-tuned model. By reducing the bias towards the fine-tuned model, larger weights are assigned to the pre-trained model. Again, it is worth noting that the pre-trained model has lower ID performance compared to the  fine-tuned model. As a result, BANG exhibits a relatively inferior ID performance compared to the model fine-tuned by MixUP and LS.

Reviewer comment (Experiments):
It would be good to see “FalseFalseTrue” analysis (like figure 2 left) of BANG models compared to WiSE-FT models. It appears from appendix E.6 that BANG models may be helpful in a “TrueFalseTrue” setting where the finetuned model was originally incorrect and overconfident. Does calibration help in the FalseFalseTrue setting?

Author response:
Thank you for your question. Let us use PM, FM and AM to denote the pre-trained, fine-tunede and averaged model, respectively.  We have found that the ratio of FalseFalseTrue (PM wrong, FM wrong and AM correct)increases from 2.5\% to 2.9\%. However, this increase is not as significant as the increase observed in the TrueFalseTrue (PM correct, FM wrong and AM correct) group, which goes from 10.6\% to 13.4\%. The visualization of the margin in Figure 20 of Appendix E.6 offers some insights. Specifically, in the TrueFalse group (PM correct, FM wrong), the Margin(PM) + Margin(FM) moves from the negative region to the positive region, crossing the boundary of y=-x. This indicates that BANG has a much higher probability of correcting the mistakes made by the fine-tuned model. Additionally, we also examine the FalseTrueTrue (PM wrong, FM correct and AM correct) group. This group demonstrates how much the mistakes made by the pre-trained model can be corrected by the fine-tuned model. The FalseTrueTrue ratio of BANG slightly decreases from 5.6\% to 4.4\% compared to vanilla fine-tuning. Importantly, this drop in the FalseTrueTrue ratio is much smaller than the increase observed in the TrueFalseTrue group (10.6\% to 13.4\%). These results suggest that alleviating the issue of over-confidence can benefit overall OOD performance. Furthermore, Figure 20 of Appendix E.6 provides further insights, showing that the Margin(PM) + Margin(FM) for BANG on the FalseTrue group (where the pre-trained model is false and the fine-tuned model is true) remains above the line y=-x. We hope this explanation clarifies the findings. If you have any more questions or need further clarification, we are more than willing to engage in further discussion.