Paper: Robustness May be More Brittle than We Think under Different Degrees of Distribution Shifts (ICLR-2024)

Reviewer comment (Experiments):
Most of the investigation is done on variations of MNIST that are likely to have only limited implications for real-world computer vision. For more reliable implications it would be recommended to use at least CIFAR-C or TinyImageNet-C that also include various levels of distribution shifts. Ideally a study would be conducted also on real-world data in addition to synthetic ones, even though those may be hard to find. To balance compute costs, a smaller number of methods could be evaluated.

Author response:
Thank you very much for your great advice. Due to the limited time frame, we were unable to experiment with CIFAR-10-C or TinyImageNet-C, but we did conduct further experiments on CIFAR-10 under a much more realistic shift. The distribution shift is a combination of two primitive types of distribution shifts which are shifts in brightness and shot-noise intensity. Since photos captured in darker environments tend to exhibit more intense shot noises, this dataset simulates realistic photographic effects in photos captured under low-light conditions and hence is much more realistic than the other datasets. Examples of this dataset can be found in Figure 8 (Appendix A.1). The results on this dataset (called LowLightCIFAR10) are shown in the updated Figure 3 and Figure 5. The new results are largely consistent with the previous ones. 

Regarding your concern that the results in this paper are likely to have only limited implications for real-world computer vision, we would like to argue that the implications may be more general than you might think. Our reasons are as follows: 

- The distribution shifts considered in this paper are very simple. We intend to show that the models are more brittle than we expect *even to these simple distribution shifts*. For more complicated shifts involving multiple types of inter-correlated shifts, the same general conclusions would likely follow. 

- Performance under synthetic shifts are correlated with performance under real-world (or natural) shifts. While [1] suggests that this correlation is weak, [2] delves deeper into the issue and provides evidence that synthetic corruptions sometimes do correlate well with corruptions that appear in the wild. 

- In this work, we focus on the *failures* of models under the distribution shifts as we vary their degrees. While robustness under synthetic shifts may not always imply robustness under natural shifts, failure under synthetic shifts almost certainly implies failure under natural shifts.

Reviewer comment (Experiments):
A small number of noise types is studied so it would be good to cover a larger variety.

Author response:
As you kindly suggested, we conducted additional experiments on a very different kind of noise, impulse noise. Examples of this dataset (called ImpulseNoiseMNIST) are shown in Figure 11 (Appendix A.1) of the updated paper. The experiment results are given in Figure 14 (Appendix B.2). In addition, the shot noise in LowLightCIFAR10 dataset is another type of noise. The results on these datasets are consistent with each other and have been partially discussed in our response to your previous comment. 

In particular, the new results on LowLightCIFAR10 (in Figure 5) and on ImpulseNoiseMNIST (in Figure 14, right) demonstrate an even more interesting pattern than we have previously shown. When the shift in the training data is not very strong (e.g., $D_0$ and $D_5$), the generalization pattern looks just like that of NoisyMNIST---robustness against milder shifts are not affected. On the other hand, when the shift in the training data is very strong ($D_0$ and $D_{10}$), the pattern looks like that of RotatedMNIST---robustness against milder shifts is significantly weakened. In addition, for LowLightCIFAR10, even the performance on the clean data is affected.

Reviewer comment (Experiments):
Is the behaviour similar also for more realistic datasets such as CIFAR-C?

Author response:
Thank you very much for your great advice. Due to the limited time frame, we were unable to experiment with CIFAR-10-C or TinyImageNet-C, but we did conduct further experiments on CIFAR-10 under a much more realistic shift. The distribution shift is a combination of two primitive types of distribution shifts which are shifts in brightness and shot-noise intensity. Since photos captured in darker environments tend to exhibit more intense shot noises, this dataset simulates realistic photographic effects in photos captured under low-light conditions and hence is much more realistic than the other datasets. Examples of this dataset can be found in Figure 8 (Appendix A.1). The results on this dataset (called LowLightCIFAR10) are shown in the updated Figure 3 and Figure 5. The new results are largely consistent with the previous ones. 

Regarding your concern that the results in this paper are likely to have only limited implications for real-world computer vision, we would like to argue that the implications may be more general than you might think. Our reasons are as follows: 

- The distribution shifts considered in this paper are very simple. We intend to show that the models are more brittle than we expect *even to these simple distribution shifts*. For more complicated shifts involving multiple types of inter-correlated shifts, the same general conclusions would likely follow. 

- Performance under synthetic shifts are correlated with performance under real-world (or natural) shifts. While [1] suggests that this correlation is weak, [2] delves deeper into the issue and provides evidence that synthetic corruptions sometimes do correlate well with corruptions that appear in the wild. 

- In this work, we focus on the *failures* of models under the distribution shifts as we vary their degrees. While robustness under synthetic shifts may not always imply robustness under natural shifts, failure under synthetic shifts almost certainly implies failure under natural shifts.

Reviewer comment (Experiments):
The performance of many domain generalization approaches improves with the number of training domains, but, in the current paper, the training domains of DG methods are limited to two (also valid for ERM, and the behavior can be observed in Fig. 6). Although a comprehensive list of domain generalization methods is considered (Appendix A.2), results are reported only on a subset of the methods, with part of the analysis being performed solely on ERM.

Author response:
**W1.1. “The performance of many domain generalization approaches improves with the number of training domains, but, in the current paper, the training domains of DG methods are limited to two.”**

Thank you for pointing this out. We have added the results of DG methods (and ERM) trained on more than two domains in Figure 12 and Figure 13 (Appendix B.1). The results show that more training domains do help. As more training domains are added, the gaps between the best-performing models gradually decrease. Nevertheless, the gaps seem to be closing at a slow rate. The discrepancy between the best-performing models is still largely present. In addition, the advantage of DG methods over ERM becomes less pronounced as the number of training domains increases. 

**W1.2. “Although a comprehensive list of domain generalization methods is considered (Appendix A.2), results are reported only on a subset of the methods, with part of the analysis being performed solely on ERM.”**

More results on this have been provided in the updated paper. For details about this part of the update, please refer to our responses below to your Q1 and Q2.

Reviewer comment (Experiments):
The analysis is performed over purely synthetic distribution shifts, considering a single type of shift.

Author response:
**W2.1. “natural shifts do not follow this 'single type' shift -- previous works, like [1], introduce benchmarks that address the natural distribution shifts while studying the continuous degradation under various degrees of shift”**

It is true that many natural shifts do not follow the 'single type' shift considered in our paper. Our main point, however, is to demonstrate even under these very simple shifts, the models are brittle to changes in the shift degrees. For more complicated shifts involving multiple types of shifts, it is almost certain that similar conclusions would follow. 

To verify this, we experimented with a new distribution shift on CIFAR10. The distribution shift is a combination of two primitive types of distribution shifts which are shifts in brightness and shot-noise intensity. Since photos captured in darker environments tend to exhibit more intense shot noises, this dataset simulates realistic photographic effects in photos captured under low-light conditions and hence is much more realistic than the other datasets. Examples of this dataset can be found in Figure 8 (Appendix A.1). The results on this dataset (called LowLightCIFAR10) are shown in the updated Figure 3 and Figure 5. The new results are largely consistent with the previous ones. Due to time constraint, the new results in Figure 3 are based on only the 11 algorithms listed in the figure. 

Thank you for letting us know the great work of Dragoi et al [1]. We will add a brief discussion about it in the related work.

Reviewer comment (Novelty):
Natural shifts do not follow this 'single type' shift — previous works, like [1], introduce benchmarks that address the natural distribution shifts while studying the continuous degradation under various degrees of shift.

Author response:
**W2.1. “natural shifts do not follow this 'single type' shift -- previous works, like [1], introduce benchmarks that address the natural distribution shifts while studying the continuous degradation under various degrees of shift”**

It is true that many natural shifts do not follow the 'single type' shift considered in our paper. Our main point, however, is to demonstrate even under these very simple shifts, the models are brittle to changes in the shift degrees. For more complicated shifts involving multiple types of shifts, it is almost certain that similar conclusions would follow. 

To verify this, we experimented with a new distribution shift on CIFAR10. The distribution shift is a combination of two primitive types of distribution shifts which are shifts in brightness and shot-noise intensity. Since photos captured in darker environments tend to exhibit more intense shot noises, this dataset simulates realistic photographic effects in photos captured under low-light conditions and hence is much more realistic than the other datasets. Examples of this dataset can be found in Figure 8 (Appendix A.1). The results on this dataset (called LowLightCIFAR10) are shown in the updated Figure 3 and Figure 5. The new results are largely consistent with the previous ones. Due to time constraint, the new results in Figure 3 are based on only the 11 algorithms listed in the figure. 

Thank you for letting us know the great work of Dragoi et al [1]. We will add a brief discussion about it in the related work.

Reviewer comment (Evaluation):
The paper does not benefit from a rigorous analysis of the results.

Author response:
**Q3. “Have you considered analyzing the performance decrease in correlation with the dataset distances (e.g. similar to the OTDD analysis from [3] or some perceptual similarity metrics [4])?”**

We deeply appreciate your great advice. Unfortunately, we are unable to conduct the analysis in time due to the limited time frame and the relatively low priority (we think) compared to your other great suggestions. We will include this analysis in the next version of this paper.

Reviewer comment (Experiments):
It seems that only ERM is used during linear probing or training of RI (randomly initialized) models (see question Q8).

Author response:
**Q8. “The analysis in Sec.5 is based solely on ERM? Have you considered using any DG method here?”**

We have considered using DG methods here in the case of NoisyMNIST. The results are given in Figure 13 (Appendix B.1). Some of the DG algorithms can further improve upon ERM, however, the improvements become smaller when more training domains are added. Please also note the main point of the analysis in Sec. 5 is to highlight the brittleness of large-scale pre-trained representations, so the results of DG algorithms are not quite relevant to Sec. 5.

Reviewer comment (Experiments):
RI performance is not reported for NoisyImageNet15 & LR-ImageNet15 (see question Q9).

Author response:
**Q9. “RI models are not reported for NoisyImageNet15 and LR-ImageNet15. Have you performed those experiments? Is there any limitation regarding those experiments?”**

Yes, we have performed those experiments. We omitted them because the results are not informative due to severe overfitting to the only 15 ImageNet classes in those datasets.

Reviewer comment (Experiments):
For RotatedMNIST — the authors consider a large range of rotation angles, some of which may induce significant confusion between class labels (e.g., numbers 6 and 9). The continuous covariate shift is less clear than in the case of the NoisyMNIST dataset (see Q7).

Author response:
**Q7. “Sec. 4.3, Fig. 5 … As ERM results are reported and we observe no drop on NoisyMNIST, it seems that RotatedMNIST is not actually capturing a continous covariate shift. Can you perform a comparative analysis of the two sets maybe based on dataset distances (similar to Q3)?”**

We indeed considered a large range of rotation angles; however, we do not think it would induce significant confusion between class labels (e.g., numbers 6 and 9). The rotation angle between $D_0$ and $D_8$ is 80 degrees. Even if we assume the original handwritten digits could tilt within a 30-degree range, which is a large range, it is very unlikely that there is room for confusion. In the worst case, for example, to confuse a 6 that is first tilted clockwise by 30 degrees and then further rotated clockwise by 80 degrees, with a 9 that is tilted counterclockwise by 30 degrees, the sum 30+80+30 needs to be at least 180 degrees. Clearly, this is far from the real case. 

Again, we would love to perform a comparative analysis of the two sets as you kindly suggested, but the time frame is too short for us to do so. We will include this analysis in the next version of this paper.

Reviewer comment (Experiments):
The studies conducted to support the argument that evaluations should be done over degrees of shifts are not very convincing. Would such controlled analysis be indicative of the performance of the model in the real world? Smoothly changing shifts implies that we should create synthetic shifts as it is expensive to create manually. Furthermore, only one type of shift is changing at a time, whereas in reality shifts are correlated, e.g., low light and ISO noise or motion blur tend to occur together.

Author response:
**1.1. “Would such controlled analysis be indicative of the performance of the model in the real world? Smoothly changing shifts implies that we should create synthetic shifts as it is expensive to create manually. Furthermore, only type one shift is changing at a time. In reality, shifts are correlated e.g., low light and iso noise or motion blur tends to occur together.”**

While we cannot be sure that the analysis in this paper would be indicative of the performance of the model in the real world, we are fairly confident that similar conclusions can still be drawn for the following reasons: 

- The distribution shifts considered in this paper are very simple. We intend to show that the models are more brittle than we expect *even to these simple distribution shifts*. For more complicated shifts involving multiple types of inter-correlated shifts, the same general conclusions would likely follow. 

- Performance under synthetic shifts are correlated with performance under real-world (or natural) shifts. While [1] suggests that this correlation is weak, [2] delves deeper into the issue and provides evidence that synthetic corruptions sometimes do correlate well with corruptions that appear in the wild. 

- In this work, we focus on the *failures* of models under the distribution shifts as we vary their degrees. While robustness under synthetic shifts may not always imply robustness under natural shifts, failure under synthetic shifts almost certainly implies failure under natural shifts. 

- In practice, properly constructed synthetic shifts are better than nothing, especially for high-stake applications such as self-driving. The key here is to make sure that the synthetic shifts are as faithful to the real shifts in consideration as possible. The more faithful they are, the more reliable and indicative the evaluation would be. In the meantime, we can also proactively collect more real data targeted at the specific distribution shifts in the studied problems (if it is possible). 

Finally, inspired by your constructive feedback, we experimented with a new distribution shift on CIFAR10. The distribution shift is a combination of two primitive types of distribution shifts which are shifts in brightness and shot-noise intensity. Since photos captured in darker environments tend to exhibit more intense shot noises, this dataset simulates realistic photographic effects in photos captured under low-light conditions and hence is much more realistic than the other datasets. Examples of this dataset can be found in Figure 8 (Appendix A.1). The results on this dataset (called LowLightCIFAR10) are shown in the updated Figure 3 and Figure 5. The new results are largely consistent with the previous ones. We will discuss more about the results on this new dataset in our response to some of your other comments below.

Reviewer comment (Experiments):
The paper seems to assume that distribution shifts are smoothly changing. What about semantic/categorical shifts, e.g., background changes like in NICO, changing shapes, etc.?

Author response:
**1.2. “The paper seems to assume that distribution shifts are smoothly changing. What about semantic/categorical shifts? e.g. background changes like in NICO, changing shapes etc.”**

From a theoretical perspective, we can always smoothly change a data distribution $p(x)$ into $p’(x)$ by properly parameterizing the distributions, even for categorical shifts. In reality, this is slightly more nuanced but still generally possible. For example, one can smoothly transition from “indoor” to “outdoor” (these are commonly regarded as categorical shifts) by walking out of the “door”. The real question, therefore, is which transitions are of real-world significance. For example, transitions from autumn to winter are meaningful, whereas direct transitions from autumn to spring (skipping winter) are not. 

It is possible that in some problems we do not care about the transitions due to their low real-world significance, so the corresponding evaluations are not needed. When we do care about the transitions, however, our paper suggests that those evaluations could be very informative and useful.

Reviewer comment (Evaluation):
There are different types of shifts, but are these shifts comparable? It seems like in Fig. 5, the trends from two different types of shifts are compared. Why do we expect the trends to be the same?

Author response:
**1.3. “There are different types of shifts, but are these shifts comparable? It seems like in Fig 5, its comparing the trends from two types different shifts, why do we expect the trends to be the same?”**

Comparing the different types of shifts is not our intention, nor do we expect the trends to be the same *or* different. For Fig. 5, we just would like to see if training under stronger shifts helps with performance under milder shifts (of the same type), and more importantly, how much it helps if it does. 

We experimented with two datasets to investigate this. As two opposite trends are shown in Fig. 5, the result is largely task-dependent: one demonstrates almost perfect generalization to milder shifts, while the other shows limited generalization. Admittedly, the limited generalization is not very surprising because, as you and another reviewer have pointed out, the model is trained on a multimodal distribution, so it is not expected to generalize in the middle. Nevertheless, this theory cannot explain why almost perfect generalization is observed in the former case. The fact that both phenomena, not just one of them, are observed is actually quite interesting. Furthermore, it supports our view that evaluating across different degrees of shifts should be encouraged as it may be difficult to predict how the model performs in the middle.

Reviewer comment (Theory):
Generalization to higher or lower shifts (sec 4.2, 4.3) is assumed without clear justification. Training on a certain level of shift makes that shift in-distribution; a different level of shift then becomes out-of-distribution to the model. Isn't this the same problem as training on clean data and testing on OOD data?

Author response:
**1.4. “Generalization to higher or lower shifts (sec 4.2, 4.3). Why do we assume that generalization will occur? Training on a certain level of shift makes that shift ID, a different level of shift then becomes OOD to the model. Isnt this the same problem as training on clean data and testing on OOD data?”**

In Sec. 4.2, it is *not* the same problem as training on clean data and testing on OOD data. The difference is that we allow model selection using another set of OOD data different from the test data. This kind of model selection is commonly practiced in large OOD benchmarks [3, 4]. More specifically, we trained the models on $D_0$ and $D_1$. Then, for example, we select the best-performing models on $D_4$. If the selected models have good performance on $D_4$ (which they do), then they have generalized from $D_0$ and $D_1$ to $D_4$. Intuitively, we would expect these models to also generalize better than other models in the test domains, at least in nearing ones such as $D_5$. However, as we shown in Sec. 4.2 with a very simple example (NoisyMNIST), this is not generally true. 

In Sec. 4.3, it is the same problem as training on clean data and testing on OOD data, but with a caveat. The caveat is that we look at generalization from stronger shifts to milder shifts. The models are trained with clean data ($D_0$) together with strongly shifted data (e.g., $D_8$, $D_{10}$). Intuitively, if a model has seen and overcome bad situations, then it should also be able to overcome better situations. This is indeed true in the case of NoisyMNIST (as we have shown in Sec. 4.3). In RotatedMNIST, however, we find that while training on strongly shifted data helps, the help is limited. 

The new results on LowLightCIFAR10 (in Figure 5) demonstrate an even more interesting pattern. When the shift in the training data is not very strong (e.g., $D_0$ and $D_5$), the generalization pattern looks just like that of NoisyMNIST---robustness against milder shifts is not affected. On the other hand, when the shift in the training data is very strong ($D_0$ and $D_{10}$), the pattern looks like that of RotatedMNIST---robustness against milder shifts is significantly weakened. In addition, even the performance on the clean data is affected.

Reviewer comment (Novelty):
It is not clear what the take-home message is. The results seem to indicate that the OOD performance of the model depends on its inductive biases and that it is hard to predict a model's performance OOD, which is quite generic.

Author response:
**2.1. “The results seems to say that the OOD performance of the model depends on its inductive biases and it is hard to predict a model's performance OOD, which is quite generic.”**

With all due respect, the message we tried to convey is more than that. It is not just hard, but even harder than it seems. To the best of our knowledge, this has not been clearly articulated or systematically examined before in the literature. That said, we do very much appreciate your comment. We will make this clearer in future revisions. Some of the more specific points have been discussed in our response to your previous questions.

Reviewer comment (Evaluation):
The overall goal seems to be more thorough evaluations (via fine-grained degrees of shifts) for models to be useful in the real world. While real-world datasets provide such evaluations, in some cases it is impossible to have smoothly changing shifts, e.g., smoothly changing animal occlusions in iwildcam. Furthermore, synthetic benchmarks like common corruptions already propose evaluating on different degrees of shifts for each corruption, although they do not provide analysis at each level of corruption.

Author response:
**2.2. “The overall goal seems to be more thorough evaluations ... Real world datasets provides such evaluations but some cases it is impossible to have smoothly changing shifts e.g. smoothly changing animal occlusions in iwildcam. Furthermore, synthetic benchmarks like common corruptions already proposes evaluating on different degrees of shifts for each corruption ...”**

It is true that we encourage more thorough evaluations. We also agree that it is impossible to observe smoothly changing shifts in some real-world scenarios. The real advice that we would like to convey is to conduct more thorough evaluations *whenever they are feasible*. This means that we should not be restricted to only using readily available data. In fact, we can proactively collect and/or synthesize more data for this purpose. For example, smoothly changing animal occlusions in iWildCam is actually possible if we maintain a series of shots each time an animal passes by. In addition, we can simulate occlusions by editing the images post-hoc. After all, while synthetic shifts are not as good as real shifts, they are at least better than nothing. 

Regarding your second argument, we would like to point out that previous synthetic benchmarks not only did not provide analysis at each level of corruption, but also did not articulate the importance of evaluating models on multiple degrees of shifts, let alone encouraging such practice to be followed in other tasks.

Reviewer comment (Evaluation):
It is unclear whether it is easy to anticipate the degree of shift at test-time such that such analysis would be useful. It seems more natural to anticipate the type of shift than its extent, e.g., adverse weather, rather than how adverse the weather will be.

Author response:
**2.3. “Furthermore is it easy to anticipate the degree of shift at test-time such that such analysis would be useful? It seems more natural to anticipate the type of shift than its extent e.g., adverse weather, than how adverse the weather will be.”**

Thank you for the great question. Indeed, anticipating how adverse the weather will exactly be is sometimes difficult. Fortunately, we do not need to be precise, we just need to ensure that almost every possible extent is approximately covered. In general, the more fine-grained it is, the better. Of course, the return of further increasing the granularity would diminish at some point. So, the real problem is to find the best trade-off between evaluation cost and informativeness. For high-stake applications, e.g., self-driving, the cost of conducting such evaluations is not really a concern. In conclusion, whether it is easy to anticipate the degree of shift at test-time usually does not have much to do with whether such analysis would be useful and whether it should be done.

Reviewer comment (Evaluation):
Regarding the statement "Similar problems can also arise when only the aggregate performance across multiple degrees is examined (Hendrycks & Dietterich, 2019)," the common corruptions benchmark measures performance over 5 degrees of shift and the paper measures it over 8/10. How do we know what is the ideal number of degrees to evaluate on?

Author response:
**3. “The common corruptions benchmark measures performance over 5 degrees of shift and the paper measures it over 8/10, how do we know what is the ideal number of degrees to evaluate on?”**

For our answer to this question, please first kindly refer to our response to your previous question (2.3). Operationally speaking, we can start with only 3 or 4 degrees, if the analysis done on some of the degrees is largely consistent with the evaluation results on the rest of the degrees, then we can stop. If not, we need to increase the number of degrees until the results are consistent or the trade-off between return and cost is unacceptable.

Reviewer comment (Writing):
The statement "robustness may be more brittle ..." sounds contradictory, as robustness is the opposite of brittle. It is our models that are brittle.

Author response:
**4. “The statement ‘robustness may be more brittle …' sounds contradictory as robustness is the opposite of brittle. It is our models that are brittle.”**

We agree that “models are brittle” would be technically more precise and perhaps a better choice. By “robustness may be brittle”, we aim to stress that the seemingly robustness of a model against a certain degree of shifts can easily break when the degree changes. We would love to consider revising the statement; however, we are not allowed to change the title of this submission. Finally, and perhaps most importantly, this statement does not allow much room for misinterpretation and should not cause much trouble for readers to understand the main ideas of this paper.