Paper: Does Progress On Object Recognition Benchmarks Improve Generalization on Crowdsourced, Global Data? (ICLR-2024)

Reviewer comment (Novelty):
The experiment in Sec. 5.1. is consistent with prior/concurrent work: Rojas et al., DeVries et al., and Ramaswamy et al.

Author response:
Thank you for your response! We are glad you appreciate our efforts, and believe they have improved our work! We apologize for not making our work’s distinction clear in the draft, we hope to further clarify below and with added language in the paper (highlighted in related works, and in Sec 6.3). Please let us know if these distinctions make our contributions more clear: 

**Experiments in Ramaswamy et al:** 

Ramaswamy et al, perform last layer retraining on an ImageNet pretrained ResNet50 model with a mix of GeoDE and ImageNet, and show that doing so improves model per-region accuracy on GeoDE (Sec 6.1, Table 6). They show how this model improves class accuracy for the 10 out of the 96 classes in DollarStreet that overlap with GeoDE (Sec. 6.1, Table 7). 

**Experiment 6.3 in Our Work**: 

Our work studies *regional disparities*, and our contribution is to study these disparities in the context of the field’s progress, including as a function of scaling, benchmark progress, and recent interventions. Our experiment in Sec 6.3 shows the impact of last-layer retraining *on the geographical performance disparities* found in DollarStreet and GeoDE, which to our knowledge has not been reported directly in prior work. While Ramaswamy et al show how last layer retraining on GeoDe can improve a model’s per-class accuracy on 10 DollarStreet classes, **their experiment fails to answer our central question, which is asking if last layer retraining can improve the region disparity**. 

Our general goal is to highlight how **improving regional disparity is not simply a matter of improving total accuracy**. We see this in Fig. 2 of our paper, where despite progress on the DollarStreet dataset in total, the regional disparity has not only not been improved, but has been *exacerbated* over time. We hope that our last layer retraining experiment provides evidence that, in contrast to our results for scaling and ImageNet improvements, last layer retraining can practically improve regional disparities across datasets. We also hope it motivates future work in larger-scale data curation by providing evidence that training on a more geographically representative dataset can not only improve total performance, but improve the geographical fairness of the model. 

We believe that these regional disparities have important consequences for the field. Our work uncovers an alarming trend, finding geographic disparity as a pervasive problem unsolved by standard benchmark progress, scaling, or common interventions, and worsening over time. We hope that our work demonstrates the impact of this problem, and provides both motivation and insight for new directions of research in this area, including in careful data rebalancing and in design of specific geographic mitigations.

(Thank you also for your appendix formatting suggestion, we have changed the formatting in our revision to make navigation easier).

Reviewer comment (Novelty):
The experiment in Sec. 6.3. is also aligned with prior work. Rojas et al. find that training on DollarStreet "can improve the performance of classification tasks for items from lower-income homes", while Ramaswamy et al. "extract features using a ResNet50 model ... and retrain the final layer".

Author response:
Thank you for your response! We are glad you appreciate our efforts, and believe they have improved our work! We apologize for not making our work’s distinction clear in the draft, we hope to further clarify below and with added language in the paper (highlighted in related works, and in Sec 6.3). Please let us know if these distinctions make our contributions more clear: 

**Experiments in Ramaswamy et al:** 

Ramaswamy et al, perform last layer retraining on an ImageNet pretrained ResNet50 model with a mix of GeoDE and ImageNet, and show that doing so improves model per-region accuracy on GeoDE (Sec 6.1, Table 6). They show how this model improves class accuracy for the 10 out of the 96 classes in DollarStreet that overlap with GeoDE (Sec. 6.1, Table 7). 

**Experiment 6.3 in Our Work**: 

Our work studies *regional disparities*, and our contribution is to study these disparities in the context of the field’s progress, including as a function of scaling, benchmark progress, and recent interventions. Our experiment in Sec 6.3 shows the impact of last-layer retraining *on the geographical performance disparities* found in DollarStreet and GeoDE, which to our knowledge has not been reported directly in prior work. While Ramaswamy et al show how last layer retraining on GeoDe can improve a model’s per-class accuracy on 10 DollarStreet classes, **their experiment fails to answer our central question, which is asking if last layer retraining can improve the region disparity**. 

Our general goal is to highlight how **improving regional disparity is not simply a matter of improving total accuracy**. We see this in Fig. 2 of our paper, where despite progress on the DollarStreet dataset in total, the regional disparity has not only not been improved, but has been *exacerbated* over time. We hope that our last layer retraining experiment provides evidence that, in contrast to our results for scaling and ImageNet improvements, last layer retraining can practically improve regional disparities across datasets. We also hope it motivates future work in larger-scale data curation by providing evidence that training on a more geographically representative dataset can not only improve total performance, but improve the geographical fairness of the model. 

We believe that these regional disparities have important consequences for the field. Our work uncovers an alarming trend, finding geographic disparity as a pervasive problem unsolved by standard benchmark progress, scaling, or common interventions, and worsening over time. We hope that our work demonstrates the impact of this problem, and provides both motivation and insight for new directions of research in this area, including in careful data rebalancing and in design of specific geographic mitigations.

(Thank you also for your appendix formatting suggestion, we have changed the formatting in our revision to make navigation easier).

Reviewer comment (Evaluation):
There is no discussion on the differences between the DollarStreet and GeoDE datasets, despite some differing results and trends; are there any fundamental differences between the two datasets?

Author response:
Thank you for your appreciation of our work, and thoughtful questions. We are glad you found our work to address an important topic, containing interesting and important findings, providing a balanced view, and well-written. We hope to have addressed your questions, and welcome any additional questions or suggestions on how we can strengthen our work: 

**Difference Between DollarStreet and GeoDE and Figure 8**:

There are a few main drivers of the dataset differences from our perspective: 
1. DollarStreet is more diverse in terms of geography and household income. DollarStreet was curated to explicitly capture a broad set of households, sending photographers to over 63 countries to people’s homes, and selecting households with a variety of incomes. GeoDE was crowdsourced among internet users from 19 countries and was designed to show the importance of crowdsourced data compared to internet scraped data.
2. GeoDE has a 1-to-many label mapping, whereas DollarStreet has a 1:1 label mapping. As shown in Appendix A.1 , GeoDE classes are coarser as there are GeoDE classes that map to as many as 25 ImageNet classes, while DollarStreet has 1 ImageNet label for each DollarStreet label. This makes GeoDE an easier dataset than DollarStreet for ImageNet models (we also evaluate a large number of zero-shot models that use the ground truth labels, without requiring mapping).

Despite the large differences in dataset curation and difficulty, we find very similar results on DollarStreet and GeoDE, indicating that the problems we discover are not specific to a given dataset curation, or labeling. We find that both datasets have large geographic disparities, even with SOTA models. Most critically, we find for both benchmarks that disparities are not resolved by progress on standard generalization benchmarks, dataset scaling, standard robustness interventions, or architecture scaling.

Reviewer comment (Experiments):
GeoDE is omitted from Table 2, even though its rate of change seems closer to ImageNet-V2 and ImageNet-Sketch compared to DollarStreet.

Author response:
**GeoDE in Table 2**: 

We included only DollarStreet in Table 2 in order to avoid confusion in reader interpretation of our Progress gap measure, which is a ratio of the rate between the benchmarks with respect to one baseline (DollarStreet). We find similar trends hold, showing conclusions are consistent across both datasets - the progress rate on GeoDE is smaller compared to other benchmarks. We have these numbers in Figure 5 in the Appendix and would be happy to add them to the main table if helpful.

Reviewer comment (Evaluation):
Comparing Figs. 4 and 5 in Appendix B, the accuracies on GeoDE are significantly higher than those on DollarStreet and sometimes even higher than ImageNet-induced benchmarks; why is that? Is GeoDE "easier" because there are fewer labels or for another fundamental reason?

Author response:
Thank you for your appreciation of our work, and thoughtful questions. We are glad you found our work to address an important topic, containing interesting and important findings, providing a balanced view, and well-written. We hope to have addressed your questions, and welcome any additional questions or suggestions on how we can strengthen our work: 

**Difference Between DollarStreet and GeoDE and Figure 8**:

There are a few main drivers of the dataset differences from our perspective: 
1. DollarStreet is more diverse in terms of geography and household income. DollarStreet was curated to explicitly capture a broad set of households, sending photographers to over 63 countries to people’s homes, and selecting households with a variety of incomes. GeoDE was crowdsourced among internet users from 19 countries and was designed to show the importance of crowdsourced data compared to internet scraped data.
2. GeoDE has a 1-to-many label mapping, whereas DollarStreet has a 1:1 label mapping. As shown in Appendix A.1 , GeoDE classes are coarser as there are GeoDE classes that map to as many as 25 ImageNet classes, while DollarStreet has 1 ImageNet label for each DollarStreet label. This makes GeoDE an easier dataset than DollarStreet for ImageNet models (we also evaluate a large number of zero-shot models that use the ground truth labels, without requiring mapping).

Despite the large differences in dataset curation and difficulty, we find very similar results on DollarStreet and GeoDE, indicating that the problems we discover are not specific to a given dataset curation, or labeling. We find that both datasets have large geographic disparities, even with SOTA models. Most critically, we find for both benchmarks that disparities are not resolved by progress on standard generalization benchmarks, dataset scaling, standard robustness interventions, or architecture scaling.

Reviewer comment (Evaluation):
Geographic disparities are much more consistent (relatively constant with model improvements on ImageNet) for GeoDE compared to DollarStreet (Figure 8); why do the datasets differ in this way?

Author response:
Thank you for your appreciation of our work, and thoughtful questions. We are glad you found our work to address an important topic, containing interesting and important findings, providing a balanced view, and well-written. We hope to have addressed your questions, and welcome any additional questions or suggestions on how we can strengthen our work: 

**Difference Between DollarStreet and GeoDE and Figure 8**:

There are a few main drivers of the dataset differences from our perspective: 
1. DollarStreet is more diverse in terms of geography and household income. DollarStreet was curated to explicitly capture a broad set of households, sending photographers to over 63 countries to people’s homes, and selecting households with a variety of incomes. GeoDE was crowdsourced among internet users from 19 countries and was designed to show the importance of crowdsourced data compared to internet scraped data.
2. GeoDE has a 1-to-many label mapping, whereas DollarStreet has a 1:1 label mapping. As shown in Appendix A.1 , GeoDE classes are coarser as there are GeoDE classes that map to as many as 25 ImageNet classes, while DollarStreet has 1 ImageNet label for each DollarStreet label. This makes GeoDE an easier dataset than DollarStreet for ImageNet models (we also evaluate a large number of zero-shot models that use the ground truth labels, without requiring mapping).

Despite the large differences in dataset curation and difficulty, we find very similar results on DollarStreet and GeoDE, indicating that the problems we discover are not specific to a given dataset curation, or labeling. We find that both datasets have large geographic disparities, even with SOTA models. Most critically, we find for both benchmarks that disparities are not resolved by progress on standard generalization benchmarks, dataset scaling, standard robustness interventions, or architecture scaling.

Reviewer comment (Evaluation):
It is unclear whether the driving source of the disparities on DollarStreet is geographic region or family income; since these are correlated, could quantitative results be provided when also controlling for family income (e.g., comparing Europe and Africa at the same income level)?

Author response:
**Income Variation in DollarStreet**: 

We agree that controlling for income variation is very interesting, thank you for suggesting this analysis! We have run an analysis of OpenAI’s CLIP trained on LAION 2B, analyzing the per-region accuracies within each income quartile. For the middle two quartiles (the only quartiles with both Europe and Africa data), we found that there was still a substantial performance drop across regions. This indicates that even when controlling for income, there is a significant performance degradation across geographies. We have included the table in Appendix C, and below for reference - thank you for the excellent suggestion, and we welcome any further advice to clarify this point. 

| DS Income Quartile | Europe Avg Acc (\%) | Africa Average Accuracy (\%) | Region Disparity (\%) |
|--------------------|---------------------|------------------------------|:---------------------:|
| Q3                 | 51.9                | 45.2                         | 6.6                   |
| Q2                 | 61.1                | 52.5                         | 8.7                   |

Reviewer comment (Evaluation):
It is unclear whether "Progress Gap" is a fair or meaningful metric; could the authors discuss what might influence progress rates and contribute to differences across datasets, considering labeling biases and different levels of difficulty, and further motivate the usage of the metric?

Author response:
**Utility of Progress Gap Metric Across Datasets**: 

We completely agree that, along with any evaluation metric, our Progress Gap metric is best understood when considering the datasets used. However, we have defined our Progress Gap metric to be a generalizable and reliable measure. The progress gap will be a useful measure as long as 1) the accuracy trend is accurately described as linear, and 2) the labels are accurate. For current standard benchmarks, these conditions hold easily. We see very high R2 values for our linear fits (see Table 2), and the linear fit of this trend has been also shown in accuracy on the line works [1].  as can be shown in our high R2 values of our linear fit in, but we include some language in Section 3.2 clarifying these conditions when we introduce the metric. Thank you for the excellent suggestion to broaden the impact of our work, and we hope this addresses your question. 
For DollarStreet and GeoDE specifically, we emphasize that the benchmarks are reliable measures of geographical disparities, being 2-6x larger than existing robustness benchmarks and having labels that are grounded in real-world use and supported by a wealth of research use.  

1. John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. International Conference on Machine Learning, pages 7721–7735. PMLR, 2021.

Reviewer comment (Evaluation):
The paper really does not answer the questions as to why there is a disparity between progress in disparate geographical groups. Is it due to the dataset bias in training data? Is it because of the domain shifts/ label shift between different geographies? Although the current observation that there is an accuracy gap is important, it would be more useful if an insight into the possible causes is also presented.

Author response:
**Driving Mechanism of Geographical Disparity**:
 
Our work primarily focuses on highlighting geographical disparity as an understudied and open research challenge for the community by providing comprehensive empirical evidence that this problem is pervasive across architectures and training paradigms. While the exact underlying mechanisms for models’ disparities across geographies (as well as a solution to this problem) is an open question, below we outline our main hypotheses for the driving reasons behind these disparities: 

1. **Dataset imbalance:** Our work shows that the geographical disparities are persistent across 100+ models with a variety of training architectures and paradigms. This suggests that the disparities are driven by geographical data imbalance in pretraining datasets. Moreover, prior works showed ImageNet is heavily western biased [1], and in our paper Appendix F we present a kNN clustering experiment, where we analyze CLIP embeddings of LAION and DollarStreet to approximate the regional distribution of LAION. We find that our proxy measure indicates an extreme imbalance in LAION, which aligns with existing work on ImageNet. Additionally, in Section 6.3 we show that fine-tuning the last layer of a ViT model on DollarStreet improves disparities on **both** DollarStreet and GeoDE, showing the promise of interventions using geographically diverse data.

2. **Distribution shifts and factors of variations between geographies:** Gustafson et al [2] annotated factors of variation in DollarStreet and found that variations in texture, occlusion, and lighting were among the factors most associated with performance disparities. Models pre-trained on ImageNet may overrely on these factors and other spurious features such as backgrounds [3] which also contributes to geographical disparities.
3. **Household income:** In addition to geographical labels, DollarStreet also has income group labels, which is correlated with geography.  We hypothesize that the lower income group is underrepresented in pre-training data and also presents a distribution shift in terms of factors of variations discussed above.   

Importantly, scaling up model or dataset size as well as standard robustness interventions such as data augmentation, while being effective on standard ImageNet OOD benchmarks, don't lead to improvements in geographical generalization, highlighting the unique challenges of geographical distribution shifts. We add this discussion as a section in Appendix I.

[1] Shankar et al, No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World
[2] Gustafson et al, Pinpointing Why Object Recognition Performance Degrades Across Income Levels and Geographies
[3] Xiao et al, Noise or Signal: The Role of Image Backgrounds in Object Recognition

Reviewer comment (Experiments):
Adding to the above point, it would be useful to include another contemporary dataset GeoNet [1] into the evaluation since they seem to directly source their dataset based on labels from the ImageNet (thus avoiding relabeling).

Author response:
**Domain Adaptation References:** We greatly appreciate highlighting these works from domain adaptation, and we have **added them to our related works** (highlighted in blue in our revised submission). We agree that analyzing GeoNet would be an interesting addition to our analysis with DollarStreet and GeoDE. Given the large number of models in our evaluation suite, we don't believe we will have time to replicate our analysis on this additional dataset, but it is a useful additional analysis and are working on incorporating into future work. Thank you for the excellent suggestion!

Reviewer comment (Experiments):
The disparity difference is only computed against Europe and Africa. Do the observations also hold for, say, Europe and Asia?

Author response:
Including Other Region Comparisons: We agree that other subgroup comparisons would be interesting! We chose to focus on Europe-Africa disparities in our plots, as those correspond to best and worst performing regions across models, which also aligns  with worst-group accuracy in the robustness literature. However, we included the results with Europe vs each of the other regions, and find that our results hold. There are significant progress gaps across these regions and across both datasets, with Europe-Asia having a slope difference of 0.17 / 0.12 for DollarStreet and GeoDE respectively, and Europe-Americas having a slope difference of 0.09 / 0.09 for DollarStreet and GeoDE. As expected, these gaps are smaller than the best/worst subgroup comparison of Europe/Africa, but these progress gaps are just as pervasive across other region pairs and we agree it is interesting to analyze (See Appendix G for plot).

Reviewer comment (Evaluation):
Sec 5.3: Could the authors hypothesize why rates of improvement across regions in GeoDE is much more uniform compared to DollarStreet?

Author response:
**Comparing GeoDE and DollarStreet Results**: 

Great question - there are a few main drivers of this different from our perspective (below). We have added a discussion section of this in Appendix A. 
1. DollarStreet is more diverse in terms of geography and household income. DollarStreet was curated to explicitly capture a broad set of households, sending photographers to over 63 countries to people’s homes, and selecting households with a variety of incomes. GeoDE was crowdsourced among internet users from 19 countries and was designed to show the importance of crowdsourced data compared to internet scraped data. 

2. GeoDE has a 1-to-many label mapping, whereas DollarStreet has a 1:1 label mapping. As shown in Appendix A.1 , GeoDE classes are coarser as there are GeoDE classes that map to as many as 25 ImageNet classes, while DollarStreet has 1 ImageNet label for each DollarStreet label. This makes GeoDE an easier dataset than DollarStreet for ImageNet models (we also evaluate a large number of zero-shot models that use the ground truth labels, without requiring mapping).

Despite the large differences in dataset curation and difficulty, we find very similar results on DollarStreet and GeoDE, indicating that the problems we discover are not specific to a given dataset curation, or labeling. We find that both datasets have large geographic disparities, even with SOTA models. Most critically, we find for both benchmarks that disparities are not resolved by progress on standard generalization benchmarks, dataset scaling, standard robustness interventions, or architecture scaling.

Reviewer comment (Evaluation):
The paper does not provide a very clear and conclusive discussion to point out the potential direction or solutions for resolving the data challenges in evaluation. For instance, what factors should be considered in designing the proper benchmark datasets? What are the right evaluation metrics to consider for evaluating foundation models?

Author response:
**Discussion on Dataset factors and selection**: 

Thank you for this insightful suggestion! We absolutely want to emphasize the importance of data selection as a main takeaway of our work. We added some language to our discussion and an **additional section in Appendix H** describing these factors to our paper. To your specific questions: 
1. **Would dataset diversity be an important factor in evaluation?** Absolutely! Our empirical evidence indicates that geographical diversity is a missing axis from standard benchmarks, and we hope to encourage the field to consider geographical disparities (and all data subgroups) as important factors for evaluation. 
2. **What factors should be considered in designing proper evaluation benchmarks?** Our work provides evidence that data sources, label curation, geographic representation, and income information should all be closely considered. In Appendix H, we emphasize that the best evaluation metrics are crowdsourced and grounded in real-world use, which is a significant difference between standard benchmarks and our geographic benchmarks. 
3. **What are the right evaluation metrics to consider for evaluation foundation models?** We address this question by highlighting the need for using subgroup evaluation metrics in generalization literature, and the limitations of using average accuracy alone. We will cite relevant examples of subgroup analysis in fairness and spurious correlations. Similarly, our work highlights the impact of looking more closely at the kinds of accuracy gains made (improvement on which portions of data), which we demonstrate with our progress gap measure. Overall, we hope to suggest that there is no one particular benchmark or tool that can perfectly measure model reliability, but that having a comprehensive suite of measures, and by analyzing benchmarks more precisely, we can improve our understanding of model behavior and its reliability. 

Thank you for your insightful suggestions! We welcome any other questions or suggestions to improve the clarity and impact of our work.