Paper: CIFAR-10-Warehouse: Broad and More Realistic Testbeds in Model Generalization Analysis (ICLR-2024)

Reviewer comment (Experiments):
The paper focuses on domain generalization within the context of CIFAR-10-Warehouse. However, CIFAR-10 is indeed a relatively small dataset with low-resolution images and a limited number of categories compared to larger-scale and more diverse datasets like ImageNet or the Wilds[1]. This work does not extensively discuss how the findings from this dataset can be applied to real-world scenarios or other domains.

Author response:
We agree with the reviewer that CIFAR-10 is a relatively small dataset compared with some existing datasets, such as ImageNet. However, it is one of the most commonly used data in the community and is an important benchmark for AccP and DG algorithm evaluation. Our choice of CIFAR-10 is because of its simplicity and manageability, which makes it a suitable starting point for foundational research of AccP and DG tasks. 

We start by using CIFAR-10 as a basic reference to build CIFAR-10 to solve the limitation that existing test data have a small number of domains or use synthetic data for evaluation. Despite its limitations, the insights gained from CIFAR-10 can provide valuable preliminary understanding, which can be a foundation for analyzing more complex datasets. The findings of AccP and DG observed in CIFAR-10 can offer indicative trends that might be observed in larger datasets. For example, if a method cannot work well on the CIFAR-10 setup, it is hard to imagine it works well on a dataset with a large number of categories. 


In future work, we will try to build testbeds with datasets of more classes. We will include the above discussion in our revised paper on the potential application of our findings from CIFAR-10 to other real-world scenarios and more diverse domains.

Reviewer comment (Experiments):
This work could be strengthened by discussing potential real-world applications beyond the scope of image classification. Evaluating and extending the methods on datasets with different characteristics and applications would provide a more practical significance for real-world applications.

Author response:
Thank you for your kind suggestion. Indeed, while our current research primarily centres on image classification, there is a large potential for applying our findings to a wider range of real-world applications, such as AccP and DG on object detection and image retrieval. Furthermore, we will consider conducting studies in our future work on these applications. In future revisions of the paper, we will add discussions about other potential applications.

Reviewer comment (Experiments):
While the paper introduces several domain generalization and accuracy prediction methods, it could benefit from including additional state-of-the-art baseline methods, e.g., GVRT[2], and VNE[3], for a more comprehensive comparison. This would help establish a clearer benchmark for the proposed methods.

Author response:
Thank you for this valuable suggestion. In response to your feedback, we have conducted additional experiments using the VNE method. The results are as follows, indicating VNE's performance across various subsets of the CIFAR-10-W dataset:

| dataset | DF (37)     | KW (95)     | KWC (48)   | All (180)   |
|---------|--------|--------|--------|--------|
| VNE     | 0.9396 | 0.9113| 0.7517 | 0.8768 |

Due to the time constraints of the rebuttal period, we were unable to complete experiments using GVRT. However, we have cited both GVRT and VNE (Min et al.,2022; Kim et al., 2023) as important baselines in our paper In the revised version, we will incorporate a comprehensive analysis that includes GVRT, further enriching our comparative study.

Reviewer comment (Experiments):
Can you elaborate on the potential biases introduced during data collection from search engines or how did you ensure the collected data is diverse and representative?

Author response:
Thank you for your insightful comments. This paper has provided some related discussions, such as distribution bias between CIFAR-10-Cs and CIFAR-10-W and the influence of noise data, in the submitted paper. Following the suggestion of the reviewer, we would like to provide additional explanations and discussions.

**Potential biases:** We agree with the reviewer that the dataset searched from search engines may be biased to the target sets and sets from different engines. However, our primary aim is to incorporate a wide range of distributions to evaluate the AccP and DG methods effectively. Hence, while these biases exist, they offer diverse perspectives for our evaluation, aligning with our objective to cover various distributions.

**Diverse and representative:** The diversity of our collected datasets has been demonstrated through the results in our paper. For example, Tables 2 and 3 show the diversity of datasets by presenting varying accuracy prediction and DG results on the CIFAR-10-W. Fig. 2 and Fig. 5 show statistical distributions of different outcomes across the datasets. Each data point represents results on one set, indicating the variety within our data.  Regarding representativeness, we try to follow the natural distribution of each search engine without imposing additional restrictions. We believe that distributions mirroring real-world scenarios are invaluable for comprehensive evaluations. 

**Potential limitations:** Noisy data should be one of the most important limitations of collecting data from search engines. Section 5 has provided some discussion about it. To remove noisy data, we have asked human annotators to review data for several around, and then use the CLIP model to select out data with multiple labels together with human double-check. By doing this, we think most of the noise data has been removed. On the other hand, dataset imbalance may be the other limitation of using search engines because the databases of different engines are different. The balanced distribution of data in the dataset is similar to the data distribution on the engines. At the same time, we discussed the results, such as that shown in Fig.4 (A) and Table.8 (Appendix), of the data being sampled to be balanced in Sections 3.3 and E. 

We will incorporate the aforementioned discussions into our paper to enhance the overall understanding of the proposed CIFAR-10-W dataset.

Reviewer comment (Experiments):
How might the findings from CIFAR-10-Warehouse generalize to other larger-scale datasets or tasks beyond image classification?

Author response:
We agree with the reviewer that CIFAR-10 is a relatively small dataset compared with some existing datasets, such as ImageNet. However, it is one of the most commonly used data in the community and is an important benchmark for AccP and DG algorithm evaluation. Our choice of CIFAR-10 is because of its simplicity and manageability, which makes it a suitable starting point for foundational research of AccP and DG tasks. 

We start by using CIFAR-10 as a basic reference to build CIFAR-10 to solve the limitation that existing test data have a small number of domains or use synthetic data for evaluation. Despite its limitations, the insights gained from CIFAR-10 can provide valuable preliminary understanding, which can be a foundation for analyzing more complex datasets. The findings of AccP and DG observed in CIFAR-10 can offer indicative trends that might be observed in larger datasets. For example, if a method cannot work well on the CIFAR-10 setup, it is hard to imagine it works well on a dataset with a large number of categories. 


In future work, we will try to build testbeds with datasets of more classes. We will include the above discussion in our revised paper on the potential application of our findings from CIFAR-10 to other real-world scenarios and more diverse domains.

Reviewer comment (Presentation):
The paper mentions addressing limitations and publishing future versions of CIFAR-10-Warehouse but does not provide a concrete roadmap for future research. A discussion of potential future directions, and how the dataset can evolve would be insightful.

Author response:
We appreciate your insightful suggestion. In our paper, particularly within Section 5, we have provided some discussions about the potential impact of CIFAR-10-W in various fields, including the challenges of learning from noisy data. Building on this foundation, and considering the reviewers' feedback, we recognize the critical importance of developing specialized datasets, particularly in fine-grained areas. For instance, the development of fine-grained datasets, as well as those covering a broad spectrum of categories, would be highly beneficial. These datasets are essential not only for addressing specific research queries but also for fostering innovation by introducing more complex and diverse scenarios for model evaluation and testing. We will add further discussion on these potential future directions and the importance of specialized datasets in advancing the field in Section 5 of our paper.

Reviewer comment (Experiments):
Can the authors give more analysis and justification on why they divide different domains based on color and cartoon/no cartoon? since there are a lot of other ways to categorize different domains, such as other styles besides cartoon.

Author response:
Great question. When dividing different domains, various factors can be considered, such as style, color, search engine features, and collection location. Our decision to focus on color and cartoony styles was driven by below reasons:

 1) Color: we chose to use color as a distinguishing factor because all the search engines we utilized come equipped with color filtering options. This functionality allows us to naturally leverage these pre-existing data divisions for our analysis.
 
 2) Cartoony Style: Cartoony images represent one of the most common styles beyond natural images. This makes them a valuable category for our study, providing a clear contrast with more realistic image styles. Meanwhile, there are also a large number of cartoon images on the Internet, making it easier for us to collect data.
 
Furthermore, to diversify our dataset, we also used diffusion models to generate additional datasets in different styles, such as oil painting and sculpture. This helps enhance the range and depth of our domain-based analysis.
 
We will integrate these discussions into our paper to provide a comprehensive understanding of our current findings. Additionally, will expand CIFAR-10-W in our future work to encompass a broader array of diverse distributions, enhancing its utility.

Reviewer comment (Evaluation):
Can the authors give more empirical analysis on the advantage and difference of the proposed dataset compared to existing datasets. For example, are the performance comparison on CIFAR-10-W and existing datasets aligned? Are there any contradicted conclusions or new observations based on the experiment results on CIFAR-10-W?

Author response:
Thanks for your kind suggestion. In our paper, we have provided experiments and empirical analysis for comparing CIFAR-10-W with CIFAR-10-Cs when they are used for evaluation. For example, Table 2 shows the results of using CIFAR-10 Cs and CIFAR-10-W respectively for AccP method evaluations, which are analysed in detail in Section 3.2. Fig. 2 (B) shows distributions of ground-truth accuracy and MAE values of method MS-AoL on 180 sets of CIFAR-10-W and 188 sets of CIFAR-10-Cs, respectively. we can observe that CIFAR-10-W has a wider range of dataset accuracy than CIFAR-20-Cs. 

During the rebuttal, we have provided new analyses in Section A of the Appendix. Fig. 7 shows the correlation between model accuracy and FD for both CIFAR-10-Cs and  CIFAR-10-W. Notably, CIFAR-10-W displays a broader range of FD values compared to CIFAR-10-Cs, indicating a larger variation in dataset distribution. For instance, CIFAR-10-W includes more sets with FD values exceeding 10. Moreover, the stronger correlation coefficients for CIFAR-10-Cs ($\rho = -0.93$ and $\tau = -0.77$) compared to CIFAR-10-W ($\rho = -0.81$ and $\tau = -0.62$) suggest that the distribution of CIFAR-10-Cs, which comprises simulated datasets, is less complex than the real-world data variant found in CIFAR-10-W. We also add t-SNE visualizations of variation within CIFAR-10-W data by color and by the search engine in Fig. 8, which shows that CIFAR-10-W provides a rich and complex dataset that extends beyond the variability of CIFAR-10.

In addition to the above new content, we will try to provide further analysis to enrich our paper with additional insights after the rebuttal process.

Thank you for this important question. Indeed, our analysis reveals some inconsistencies in the performance of methods when evaluated on CIFAR-10-W as compared to existing datasets like CIFAR-10-Cs. For instance, Tables 2 and 6 present differing rankings of AccP methods across these two datasets. Specifically, for AccP, we observed the following rank correlations across different models:

|             | ResNet44 | RepVGG-A0 | ShuffleNetv2 |
| ----------- | -------- | --------- | ------------  |
| correlation $\tau$ |  0.066    | 0.733   | 0.644       |

These correlations suggest a lack of strong rank consistency between performances on CIFAR-10-W and CIFAR-10-Cs. This is especially evident for ResNet44, where the correlation is notably low.

Moreover, compared with other domain generalization benchmark datasets (usually involving less than six domains), our proposed benchmark dataset empowers researchers to conduct more statistically meaningful analyses with 180 domains. Specifically, from Table 3, we can see that with the increased number of training domain, the test performances of model tend to be more stable. For example, the variance of test performance drops from approximately 3.9\% for single source domain to 0.67\% for four source domains. This outcome contrasts with results reported in other studies, indicating that CIFAR-10-W may present a more challenging and realistic testing environment by involving more test sets from different domains.

These observations suggest that conclusions drawn from CIFAR-10-W could differ from those based on existing datasets. In particular, CIFAR-10-W tends to magnify performance differences among methods, providing clearer insights into their robustness for AccP. For DG tasks, the lack of noticeable improvement over the ERM baseline in our dataset suggests that existing DG methods may need further refinement to handle the diverse and complex scenarios presented by CIFAR-10-W.

In summary, CIFAR-10-W serve as a new diverse benchmark dataset, offering a wide array of test environments that are helpful for future research in AccP and DG methodologies.  We will include the discussions highlighted above in the revised version of our paper, further indicating the relevance and utility of CIFAR-10-W for future studies.

Reviewer comment (Evaluation):
**[Critical] Why is MAE of prediction scores the main metric reported in the paper for comparing methods in Table 2?**  
Why I ask this (my understanding, could be caused by misinterpretation):  
- MAE of prediction scores, rather than prediction scores seems like a bad measure as indicated by Figure 2 and elaborated in Fig 9 – no correlation to accuracy, while Figure 7 shows prediction scores have high spearman rank order correlation (a strong measure of correlation!).  
- Hence, I can discern very little about the predictiveness of the accuracy from the MAE score as they’re uncorrelated!

Author response:
Thank you for your question. In the Accuracy Prediction (AccP) task, existing research often employs Mean Absolute Error (MAE) or Root Mean Square Error (RMSE) as the primary metrics. Consistent with this norm, we use MAE in our paper. The objective in AccP is to predict the classifier accuracy $ \hat{Acc}_i, i = 1, \ldots, n $, across $n $ given unlabeled sets. The MAE, defined as 

$$
\text{MAE} = \frac{1}{n}\sum_{i=1}^{n} |\text{Acc}_i - \hat{\text{Acc}}_i|,
$$

effectively indicates the gap between predicted accuracies and the ground-truth accuracies.

There is a misinterpretation regarding the results presented in Fig. 2 (also in Figs. 7 and 9). Fig. 2 (A) illustrates the correlation between prediction scores $ \hat{\text{Acc}_i} $ and actual accuracy $ \text{Acc}_i $ Fig. 2 (B) illustrates the relationship between the actual accuracy $ \text{Acc}_i $ and the prediction error $ |\text{Acc}_i - \hat{\text{Acc}}_i| $ of the AccP method on different datasets. It is expected that there may not be a clear correlation, as the value of prediction error is not necessarily directly influenced by the ground-truth accuracy of a dataset. 

We hope the above explanation can help the reviewer have a clear understanding of MAE.

Reviewer comment (Evaluation):
**[Important] Conclusions made from Task 1 need improvement — Similar but to a lesser degree for Task 2.**  

*(C1) This benchmark is a harder benchmark compared to CIFAR-10-Cs.*  
- Corruptions can be made harder by increasing the magnitude easily, increased hardness as the primary feature of the testbed seems weird. I would be interested in rather highlighting whether networks don't capture certain aspects introduced here.  

*(C2) Predictions are more consistent across classifiers here.*  
- One can easily argue that predictions are consistent here because the shift is not varied – it simply changes colours rather than the diverse, varied corruptions studied in CIFAR-10-C benchmarks.  
- Note that this does have a distribution shift across cartoons and SD classifier, but SD classifier results are varying across classifiers too!

Author response:
**1) This benchmark is a harder benchmark compared to CIFAR-10-Cs. The increasing hardness of the testbed seems weird. It is more interesting to highlight whether networks don't capture certain aspects introduced here.**

We agree with the reviewer that the goal of building a new testbed is not only to increase the difficulty of the dataset but to propose test data that are more indicative of algorithm performance. The primary aim of developing CIFAR-10-W is to offer a testbed that better reflects the complexities in real-world scenarios and provides a more comprehensive evaluation of algorithms.

Specifically, the proposed CIFAR-10-W can provide a more comprehensive evaluation of AccP methods by including more challenging sets. The challenge here not only means that the data set is more difficult but it is reflected in many aspects, such as more diverse data and more different predicted models. Meanwhile, the evaluation on CIFAR-10-W datasets indeed shows more obvious performance differences of existing methods, which is helpful for highlighting the performance of different AccP methods. 

For example, the ranges of y-axis of Fig. 2 (B) show that CIFAR-10-W have more diverse test sets for AccP algorithms evaluation, which helps indicate the robustness of AccP methods under variance environments. Meanwhile, the evaluation of diverse models provides a more accurate comparison of AccP methods when facing different target classifiers. For example, the standard deviations (SD) of different methods for ResNet44 on the CIFAR-10-Cs and CIFAR-10-W datasets, respectively, are **0.81** and **1.80**. Obviously, the results difference can be magnified on CIFAR-10-W.

In conclusion, CIFAR-10-W is not only a hard testbed, it is a more diverse and indicative testbed. We will include the above discussion in our revised paper.

**2) One can easily argue that predictions are consistent here because the shift is not varied – it simply changes colours rather than the diverse, varied corruptions studied in CIFAR-10-C benchmarks.** 

 Thank you for your comments. We respectfully disagree with the comment that the observed consistency is due to a lack of varied shifts in the data.

 In our paper, Fig. 2 and 5 illustrate the distribution of ground truth accuracy for ResNet44 and ERM accuracy on CIFAR-10-W. These figures clearly demonstrate varied performance across our datasets which can indicate the varied shifts among datasets. 

Additionally, during the rebuttal phase, the newly added Fig. 7 in the Appendix highlights the distribution differences between datasets from CIFAR-10-Cs and CIFAR-10-W by showing their FD with the CIFAR-10 dataset. It is evident from this analysis that CIFAR-10-W has more datasets with Fréchet Distance (FD) values greater than 10, suggesting a greater deviation from the CIFAR-10 dataset. In contrast, most datasets in CIFAR-10-Cs exhibit smaller FD values, indicating closer similarity to CIFAR-10. Meanwhile, the newly added Fig. 8 in the Appendix shows t-SNE visualizations of CIFAR-10-W and CIFAR-10, which indicates the diversity of images from the same color or different search engines.

Furthermore, we have calculated the rank correlations of different method performances on two subgroups of data from CIFAR-10-W:

|             | DF.h & DF | Goog. & Flic. | Bing C & Baid.C |
| ----------- | --------- | --------------- | ---------------------------- |
| correlation | 0.3050     | 0.4384          | 0.648                        |

There is no obvious correlation, which indicates the variance between these datasets. As for the consistency we mentioned in the paper is caused by the average statistics on \textbf{all} datasets. When methods are evaluated on \textbf{a large number of diverse datasets}, the rank of performance of methods will gradually be stable.  

Following the valuable suggestions of the reviewer, we will revise our conclusions and incorporate the above-discussed points into the revised version of our paper.

Reviewer comment (Experiments):
**[Important] Little analysis of the dataset itself, far more focus on task/models**  
- While this weakness is vaguely stated, I have fleshed out some components in Q1 to concretely ask what analysis would be helpful from my view.  
- However, I do not work in either of the fields so it is hard for me to accurately ask, but there seems to little analysis done which is concerning.

Author response:
Thank you for your constructive suggestion. In response to your feedback, we have enriched our paper with more analysis of the CIFAR-10-W dataset. The new contents are in Appendix Section A.

Specifically, we have included comparisons of distribution ranges between CIFAR-10 and CIFAR-10-W, highlighting the diversity present within CIFAR-10-W. Additionally, we present an analysis of the variance within datasets categorized by a single color option, illustrating the intricate differences that exist even within seemingly similar groupings.

We believe this added analysis will provide readers with a more comprehensive understanding of the CIFAR-10-W dataset, showing its complexity. This information should further clarify the relevance and applicability of CIFAR-10-W for AccP and DG evaluation.

Reviewer comment (Writing):
Only want to point out this one typo: you seem to have an unfinished sentence at the last line of page 6.

Author response:
Thanks. We have corrected the errors in the revised version and double-checked the paper to avoid typos.