From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: Summary:
This paper investigates whether models trained on ImageNet with increased accuracy also transfer their improved accuracy to multiple downstream datasets (what they call real-world datasets) via fine tuning. The authors performed extensive experiments testing many models on six publicly available datasets. They also tested some other common practices such as data augmentation in this context. The main finding is that models with higher ImageNet accuracy do not consistently yield performance improvements on the downstream datasets.

Strengths:
1. This paper addresses an interesting and important question: how well does the performance of the model on ImageNet correlate with the performance on the real tasks?
2. The formulation of the study is good, including many recent vision architectures and a variety of target datasets.
3. The paper is well written and easy to follow. 

Weaknesses:
1. The framing (or definition) of the problem as ‘web-scraped vs. real-world (non-scraped)’ is questionable. It is essentially a matter of distribution shift.
2. Related to this, the findings seem somewhat obvious. It is well-known (in particular to practitioners) that distribution shift can cause a significant performance drop and that the performance in the source dataset does not always transfer. 
3. Only the basic fine-tuning is tested and recent progress in transfer learning techniques such as domain adaptation are not considered.
4. The choice of the six datasets is not inclusive enough. The methodology of dataset selection can be made more formal. 
5. With regard to some of the above concerns, there are a little too assertive discussions which could misguide the community. 
justification_for_why_not_higher_score: This paper received split scores of (3,5,8,8), which means two are in favor of acceptance and two are opposed. The reviewers generally acknowledge the motivation and importance of the research, but some concerns are left such as the overall story and argument of the paper, the appropriateness of the experimental design, and the novelty of the results. In particular, one reviewer has raised strong concerns about the framing of the problem, the lack of methodology in terms of transfer learning, and the obviousness of the results, which make sense to the AC as well. It is a valid point that the problem is essentially a distribution shift problem, and it is well known that distribution shifts degrade performance, which is why measures such as domain adaptation are being actively studied. The six data sets selected are also of questionable comprehensiveness to be representative of the real-word tasks.
In the AC’s opinion, papers that, like this one, send a strong high-level message based on empirical results must be highly complete and convincing in terms of logic and supporting results in order not to lead the community in the wrong direction. If the paper cannot convince at least all the reviewers, it would be premature for acceptance. In the discussion between the reviewers and the AC, there was no strong support for the paper, so the AC would like to recommend rejection.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper5529/Authors
title: Rebuttal Discussion Check-in
comment: Dear Reviewers FJNw, yyk7, and ZbHU,

We recently posted a response to your comments on OpenReview and would like to kindly request a response from you. As we mentioned in our response, we have taken your feedback into consideration and have made additional changes to our paper.

We believe that our updated paper addresses the concerns raised in your reviews and would appreciate if you could take a moment to review our response and update your review accordingly. We are happy to answer any additional questions and concerns.

We understand that your time is valuable and appreciate your efforts in providing constructive feedback on our work. Your input is essential for improving the quality of our paper and we look forward to hearing from you.

Thank you for your time and feedback!

From: ICLR.cc/2023/Conference/Paper5529/Reviewer_QVyG
title: Thank you
comment: Thank you for addressing all my points and updating the paper.

From: ICLR.cc/2023/Conference/Paper5529/Authors
title: Response to Reviewer ZbHU (1/2) 
comment: We thank the reviewer for their time and helpful feedback. We are glad to hear that the reviewer believes our experiments are well conducted, and hope to clarify the points on which the reviewer has doubts.

*Why also not mentioning domain adaptation techniques that are beyond fine-tuning?*

We focus on fine-tuning because we believe it is the most commonly used strategy for most machine learning workflows. Evaluating further domain adaptation techniques would be an interesting direction for future work.

*"In particular, we study datasets collected with the goal of solving real-world tasks (e.g., classifying images from camera traps or satellites), as opposed to web-scraped benchmarks collected for comparing models." -> Dataset collected from the web also serve to structuring data in the web. This is a real-world task.*

While structuring web data is a real task, we do not consider it a “real-world task”. We define real-world tasks to be applications that can happen or affect things in the physical world, as opposed to those that happen virtually. An additional part of our framing is that the images used for these tasks must be created and collected for the purpose of solving this task. Of course, defining “real-world task” is open to interpretation, but we choose to define it this way in the context of our work.

*"In particular, we consider classification tasks derived from image data that were specifically collected with the goal of classification in mind. This is in contrast to many standard computer vision datasets – including ImageNet – where the constituent images were originally collected for a different purpose, posted to the web, and later re-purposed for benchmarking computer vision methods." -> There seems to be a confusion between the intent behind collecting the data and the intent behind the existence of the data. And what was the different purpose in the collection of data for ImageNet if not classification? The notion of intent is very subject to discussions…*

While ImageNet was built with the intent of classification, the images themselves were not uploaded to Flickr with the intention of using them for machine learning. The images existed before the concept of ImageNet was created. As you mention, there is a difference between the intent behind forming the dataset and the intent of collecting the original images and we are referring to the latter when it comes to defining real-world tasks.

*"Web-scraped target datasets are by definition within the distribution of data collected from the web, and a sufficiently large model can learn that distribution. This strategy may not be effective for non-web-scraped datasets, where there is no guarantee that it is possible to train on data that is close in distribution to the target data, even if we train on the entire web. Thus, it makes sense to distinguish these two types of datasets." -> While the argument has some logic, I do not find the distinction qualitatively very relevant. There might be a lot to argue. For instance, Camera Traps are a very common object now and many people are already posting it on the web, making the so called web distribution constantly changing.*

Recent works such as CLIP, ALIGN, BASIC, and CoCa have trained large models with large amounts of web-scraped datasets to achieve high performance results on a variety of web-scraped benchmarks. Follow-up work has shown that  simply scaling model and dataset size does not seem to have reached their limits yet. The suite of benchmarks in Kornblith et al. “Do Better ImageNet Models Transfer Better?” are all web-scraped and while better ImageNet models do better on those tasks, our results show that is not the case with all real-world datasets. These results suggest that this distinction is relevant. While an option is to ascribe these differences to different underlying distributions, we believe it is important to go explore further and try to characterize why these distributions are different.
As both you and reviewer yyk7 mentioned, the web distribution is constantly changing. An initial possible distinction is to categorize by intent behind the creation of the images, but this is overall a difficult task and one that we leave to future work.

From: ICLR.cc/2023/Conference/Paper5529/Authors
title: Response to Reviewer ZbHU (2/2)
comment: *"If improving ImageNet accuracy alone is insufficient to reach these well-tuned baselines, we can indeed conclude that architecture transfer to this target task is limited." -> Well, this is quite dubious. The improvment the authors perform may lack the power of the training techniques that have been used for these well-tuned baselines. These well-tuned baselines result from specific training approaches that may hypothetically depend on the task at hand, e.g., when employing curriculum learning (which is very much used for real-world applications), and hence not very much fall in the scope of the type of fine-tuning the authors are performing. I would hence be much more careful in their statement.*

We may be in agreement on this point, as our question is whether ImageNet with fine-tuning is enough to make progress on other tasks. Our models differ mostly by architecture. These architectures are primarily benchmarked against ImageNet, and we wanted to see if gains  obtained from architecture generalized to real-world applications. Because our results imply that these performance gains do not always carry over, the next step for practitioners may be to use additional training techniques, including the ones you suggested.

*It is not very novel and limited in scope (only picking 6 datasets) … Besides, the conclusion drawn by the paper feels already like common knowledge. For instance, in deep learning for satellite spectral imagery (like the Sentinel-2 example provided by the author), there are already many papers showing how to adapt the convolution specifically for that type of images that have a radically different distribution with respect to more classical images (scrapped or not scrapped...). The authors present some of their statements as if it is something that the community should change. But the truth is that it is already very clear for the practitioners of the various real-world tasks.*

We limited ourselves to six datasets in order to focus our resources towards a wider breadth of models. We agree that researchers and practitioners in the domains our datasets cover may well be aware of the limitations of ImageNet accuracy in predicting transfer accuracy. However, we do not believe that the poor predictive performance of ImageNet accuracy on these tasks is well-recognized within the broader community of researchers seeking to develop new architectures and improve training techniques because the broader community rarely benchmarks on datasets of the type we investigate. We believe that our results will be surprising to this community, particularly when compared against past works in transfer learning (Kornblith et al. “Do Better ImageNet Models Transfer Better?”) and robustness (Miller et al. “Accuracy on the Line…”), which have shown that better ImageNet models typically do better on other tasks.

To reconcile these differing perspectives on the external validity of ImageNet, we believe it is important to conduct a systematic empirical investigation such as ours. Many papers in our scientific community still focus their evaluation on web-scraped benchmarks like ImageNet and CIFAR, so while this may be a given for practitioners, these results may not be as obvious for researchers. We hope that our work will encourage better benchmarking practices that will lead to more scientific exploration and improvements like the examples you have mentioned.

From: ICLR.cc/2023/Conference/Paper5529/Authors
title: Response to Reviewer yyk7
comment: We thank the reviewer for their time and helpful feedback. We are especially encouraged by your agreement in the importance of studying benchmarking and real-world tasks.

*The choice of the six datasets while practical is not inclusive. For instance, there are a number of additional applications such as robotics (I am thinking indoor classification,), sports, or automotive applications that could also have been added. All of which can be classified as real-world applications that may more closely associate with ImageNet. I believe that the methodology of dataset selection can be made more formal while also including some dataset statistics.*

We agree more datasets would be nice, but given that our compute budget is limited, we had to balance the number of datasets with the number of models and the number of hyperparameter combinations we considered, and we decided to prioritize hyperparameter tuning and breadth of models before additional datasets. We believe that an in-depth investigation of six carefully chosen datasets is more informative than a more superficial analysis of more datasets.

On a side note, we originally intended to investigate an additional Kaggle competition on distracted driving, but were unable to do so because the terms of the competition did not permit using the data for research purposes. We would be happy to see future work investigate additional domains in a similar context.

*The work fails to also highlight another important of training using pretrained weights which is convergence time. Even though imagenet pretraining alone might not lead to better results, would a model trained from scratch also arrive to the same accuracy?*

Using pre-trained weights for faster convergence time is another important practical benefit of pre-training; however, training models from scratch is expensive and our work focuses on the transfer accuracy aspect. We will clarify in a future revision that transfer accuracy is not the only benefit of pre-training.

*In terms of clarity the definition of what is considered a real-world dataset is open to interpretation. Also, it is not unpoppable that images crawled through the web have been uploaded after been taken from “real-world” environments. This is not a weakness of the particular paper alone but rather a challenge that the community needs to face.*

While our definition of “real-world dataset” is imperfect, we hope that our work can lead to additional focus towards this area of machine learning. For now we focus on datasets that contain images that are both taken from real-world environments and collected for the sake of machine learning use. It is possible that such images are uploaded to the internet and used for purposes beyond their original task (perhaps pre-training), but this is for future work to investigate.

From: ICLR.cc/2023/Conference/Paper5529/Authors
title: Response to Reviewer QVyG
comment: We thank the reviewer for their time and helpful feedback. We are happy to hear that you liked our dataset selection and allocation of resources towards additional model configurations.

*I would have wanted to hear more about how your models compare to pre-existing models pretrained on ImageNet and finetuned with the same datasets that you tested in your experiments. Because it is almost like you are disregarding all the work that exists were other researchers have used ImageNet pretrained models(like how you did) on these same real world datasets. Such as: Naushad, R., Kaur, T. and Ghaderpour, E., 2021. Deep transfer learning for land use and land cover classification: A comparative study. Sensors, 21(23), p.8083. What are your comments on these previous works? Why did you just not take their published recorded accuracies instead of re-doing some of their experiments?*

Thank you for pointing us to this related work - we have updated our paper to reference this work. Direct comparisons are difficult with EuroSAT because there is no predefined train/test split. While the cited work has nice ablation experiments, their work focuses on two specific architectures, and they freeze a portion of the layers whereas we fine-tune end-to-end. More importantly, we run our own experiments in order to keep our experimental setup consistent for comparison across models, as well as across datasets. While we missed this work in particular, we would like to point out that we generally cite and compare our numbers with past work on the same benchmarks, such as competitors in the Kaggle competitions.

*In Section 4.1, you mentioned that you removed the background class but you do not mention why. Can you support this decision? If you are investigating transferability to the real world, surely the background class is very crucial to understanding how well it is performing.*

The empty class does not exist in the training data, and is also not used for evaluation in classification experiments in Beery et al. Recognition in Terra Incognita (Table 1).

*In section 5.1 , you say "We increase the number of epochs we fine-tune on from 30 to 50 to account for augmentation"- how do you support this statement?*

Augmentation leads to an increased number of possible examples, which is why we increased the number of epochs. Note that our augmentation experiments table includes reruns of the baselines with additional epochs, which are within error bars of the linear trends for models run on 30 epochs. This suggests that the additional epochs did not disadvantage the models run on 30 epochs. When we re-ran these experiments with 30 epochs (rather than 50 epochs), we found that while some of the numbers were slightly shifted, most of the patterns remained similar.

Table: Effect of pre-training augmentation and fine-tuning augmentation on downstream performance. First column includes the model and the pre-training augmentation done on ImageNet, while each column after indicates the dataset and the fine-tuning augmentation used.

| Model and Pre-training Aug | ImageNet Acc. | CCT20 Base Aug. | CCT20 AugMix | CCT20 RandAug | APTOS Base Aug. | APTOS AugMix | APTOS RandAug |
| ----------- | ----------- | ----------- |  ----------- | ----------- | ----------- | ----------- | ----------- |
| ResNet-50 (standard ImageNet aug.) | 76.1 | 73.60 | 73.16 | 74.10 | 0.9195 | 0.9204 | 0.9241 |
| ResNet-50 AugMix                              | 77.5 | 73.53 | 73.09 | 73.44 | 0.9195 | 0.9166 | 0.9216 |
| ResNet-50 RandAug                           | 78.8 | 73.96 | 73.42 | 74.47 | 0.9124 | 0.9191 | 0.9258 |
| Deit-tiny (RandAug + RandErasing)    | 72.2 | 66.94 | 66.83 | 67.23 | 0.9148 | 0.9117 | 0.9192 |
| Deit-small (RandAug + RandErasing) | 79.9 | 70.23 | 69.48 | 69.99 | 0.9227 | 0.9286 | 0.9248 |

*In section 6, you say "Asymptotically, it is probably possible to perform well on all web-scraped target datasets simply by collecting a very large amount of data from the Internet and training a very large model on it." Can you back this up by facts?*

Our statement is more conjecture than fact, but recent works such as CLIP, ALIGN, BASIC, and CoCa have trained large models with large amounts of web-scraped datasets to achieve high performance results on a variety of web-scraped benchmarks. Similarly, Google has scaled up large language models in PaLM and Minerva, showing that this is not only limited to image data, but more types of data found on the internet. In both cases, simply scaling model and dataset size does not seem to have reached their limits yet. We have made this statement more precise in the updated revision to make clear that this is conjecture rather than something that has been proven.

From: ICLR.cc/2023/Conference/Paper5529/Authors
title: Response to Reviewer FJNw
comment: We thank the reviewer for their time and helpful feedback. We are happy to hear that you found the paper clear and the study well formulated.

*Unclear if models are downloaded from torchvision with pre-trained weights or re-trained with a fixed set of parameters on Imagenet. This is especially troubling regarding the augmentation strategy that may vary in different pre-trained models (see discussion in ConvNext paper about the pre-training strategy)*

The models we used are downloaded from torchvision and timm, which typically use the augmentation strategy that is standard for pre-training each model. We agree that the augmentation strategy can affect the resulting model; however, part of the benefit of augmentation is reflected by the increase in ImageNet accuracy and thus does not change our analysis (i.e., whether ImageNet accuracy transfers to real-world tasks). The augmentation ablation in particular was to show that augmentations may affect ImageNet and downstream tasks in different ways, and that ImageNet accuracy alone is not a comprehensive benchmark.
We have included in Appendix O of our updated revision a table that describes the augmentation used for pre-training each model, as well as a version of our main figure that is colored by augmentation: cyan for simple augmentation (resize, crops, flips, etc.) and red for automatic augmentation (RandAugment, AutoAugment, TrivialAugmentWide). While the red points sometimes sag a bit below the general trend (most noticeably for CCT-20), our ablations suggest that adding additional augmentation at fine-tuning time would not noticeably affect this plot and the overall trends, at least for the DeiT models.

*The analysis could include some more low-level analysis of the architecture behavior. What are the layers that change the most during fine-tuning? How much feature representation shifts from the pre-trained network and the fine-tuned one?*

While it would be interesting to do additional low-level analysis of different architectures, it might be difficult to compare due to different layers and depths. Given your suggestion of measuring feature representation shifts between the pre-trained network and the fine-tuned network, we measured the linear centered kernel alignment (CKA) between the testset representations of the pre-trained and fine-tuned networks. We did this for four of the six datasets and half of our models. We do not find any obvious patterns within datasets, within architectures, and between datasets that transfer better and those that do not. 

The following table contains the CKA scores resulting from this experiment:

| Model | CCT20 | APTOS | HPA | Melanoma |
| ----------- | ----------- | ----------- |  ----------- | ----------- | 
| VGG13BN | 0.2180 | 0.4861 | 0.1751 | 0.0699 |
| DeiT-Tiny | 0.6268 | 0.4601 | 0.1711 | 0.2905 |
| DenseNet-121 | 0.2885 | 0.5065 | 0.1729 | 0.3309 |
| ResNet50 | 0.2879 | 0.5573 | 0.2214  | 0.2024 |
| EfficientNet-B0 | 0.2953 | 0.4670 | 0.1711 | 0.0380 |
| ResNet152 | 0.3073 | 0.6037 | 0.2148 | 0.2197 |
| DeiT-Small | 0.7081 | 0.5072 | 0.2687 | 0.3281 |
| ConvNext-tiny | 0.5004 | 0.5698 | 0.1363 | 0.2230 |
| PNASNet | 0.1807 | 0.3256 | 0.1002 | 0.1874 |
| EfficientNet-B4 | 0.2647 | 0.3797 | 0.1359 | 0.2358 |

From: ICLR.cc/2023/Conference/Paper5529/Reviewer_FJNw
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This work presents an extensive study  on the generalization of ImageNet pre-training. Six real-world datasets spanning from camera traps to satellite imagery are taken into account. The work is motivated by the fact that existing work on visual classifier generalization are often analyzing downstream tasks (e.g. detection) on dataset that are collected with the same strategy as imagenet (web scraping), moreover the datasets on which the transfer learning is performed are often object centric.

The analysis shows that high accuracy on the pre-training dataset not always correlate with high accuracy on real-world tasks for all architectures. Notably some models (E.g. ConvNext) show high correlation.


strength_and_weaknesses: Strenghts

- The study is well formulated and takes into consideration recent neural architectures and a good variety of target datasets
- The use of FID to show difference in dataset distribution is interesting, though the value of FID can be hard to interpret. It could be interesting to just train a linear classifier on each pre-trained representation (before fine-tuning) on the target dataset, this should be a good representative of distribution shift and also more understandable.

Weaknesses
- Unclear if models are downloaded from torchvision with pre-trained weights or re-trained with a fixed set of parameters on Imagenet. This is especially troubling regarding the augmentation strategy that may vary in different pre-trained models (see discussion in ConvNext paper about the pre-training strategy)
- The analysis  could include some more low-level analysis of the architecture behavior. What are the layers that change the most during fine-tuning? How much feature representation shifts from  the pre-trained network and the fine-tuned one?
clarity,_quality,_novelty_and_reproducibility: Paper is clear and well written. See above regarding the pre-training strategy (how are the network pre-trained?) 
The augmentation analysis is conducted on an ablation study, but still it is not clear if pre-trained network are all trained with the same data augmentation strategy.
summary_of_the_review: The paper is interesting and likely useful to the research community.  There are some issues in how experiments are formulated (augmentation) but this can be possibly be clarified in the rebuttal.

Some lower level insights on the learned representation would be interesting to better understand behavior of this models in fine-tuning.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5529/Reviewer_QVyG
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The work investigates whether models that perform well on the ImageNet dataset also perform as well on real world datasets. By real world datasets they mean datasets that were not scraped off the internet, their investigation includes datasets such as Caltech Camera Traps-20, the Human Protein Atlas Image Classification, EuroSat, APTOS 2019 Blindness, SIIM-ISIC Melanoma, Cassava leave datasets. They extensively fine tune models pre-trained on ImageNet using these datasets and report  that architectures that do well on ImageNet do not necessarily do well on other real world datasets and that the newer models that do well on ImageNet perform worse on these real world datasets than the older ones like VGG Networks.
strength_and_weaknesses: Strengths: 
--The work digs deeper into making the models work for these real world datasets. I really appreciate that the authors focused their efforts on a few datasets and focused their efforts on different model configurations before coming to their conclusions.
--The paper is well written and easy to follow.
--Their findings are some of the things the community has been talking about but had never really been scientifically proven extensively

Weaknesses:
-- I would have wanted to hear more about how your models compare to pre-existing models pretrained on ImageNet and finetuned with the same datasets that you tested in your experiments. Because it is almost like you are disregarding all the work that exists were other researchers have used ImageNet pretrained models(like how you did) on these same real world datasets. Such as: Naushad, R., Kaur, T. and Ghaderpour, E., 2021. Deep transfer learning for land use and land cover classification: A comparative study. Sensors, 21(23), p.8083.
What are your comments on these previous works? Why did you just not take their published recorded accuracies instead of re-doing some of their experiments?
clarity,_quality,_novelty_and_reproducibility: -- In Section 4.1, you mentioned that you removed the background class but you do not mention why. Can you support this decision?
If you are investigating transferability to the real world, surely the background class is very crucial to understanding how well it is performing.
 
-- In section 5.1 , you say "We increase the number of epochs we fine-tune on from 30 to 50 to account for augmentation"- how do you support this statement?
-- In section 6, you say "Asymptotically, it is probably possible to perform well on all web-scraped target datasets simply by collecting a very large amount of data from the Internet and training a very large model on it." Can you back this up by facts?
 
summary_of_the_review: I think the authors did a good job focusing on transferability of these real-world tasks and consolidating the experiments of diverse datasets into one paper. However, I am concerned about the fact that they do not mention other authors who ran some of their experiments on the same real world datasets that they used on some of the same architectures pretrained on ImageNet also. Overall I think this was a well written paper and all of their experiments are clear and sound easy to follow.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper5529/Reviewer_yyk7
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper investigates whether models trained on ImageNet with increased accuracy transfer their improved accuracy to multiple datasets. Many models are investigated on a benchmark of six publicly available datasets that try to capture the characteristics practical classification applications. The main finding is that models with higher ImageNet accuracy do not consistently yield performance improvements. 
strength_and_weaknesses: Strengths:
- This paper demonstrates that ImageNet pretrained models do not always lead to better accuracy for certain applications. Thus it highlights that architectural innovations might be overfitting on the ImageNet data highlighting the need for better benchmarks.
- A variety of models exhibiting large imagenet accuracy variance are studied. 
- The datasets under investigation can provide a useful initial benchmark for testing models on a variety of classification applications. 

Weaknesses:
- The choice of the six datasets while practical is not inclusive. For instance, there are a number of additional applications such as robotics (I am thinking indoor classification,), sports, or automotive applications that could also have been added. All of which can be classified as real-world applications that may more closely associate with ImageNet. I believe that the methodology of dataset selection can be made more formal while also including some dataset statistics. 
-  The work fails to also highlight another important of training using pretrained weights which is convergence time. Even though imagenet pretraining alone might not lead to better results, would a model trained from scratch also arrive to the same accuracy?

clarity,_quality,_novelty_and_reproducibility: - Overall, this study does not provide a rapidly new methodology of analysing but some aspects can be seen as novel such as the attempt to predict downstream task accuracy. The authors provide the hyperparameters and experiment details to reproduce the results. 

- In terms of clarity the definition of what is considered a real-world dataset is open to interpretation. Also, it is not unpoppable that images crawled through the web have been uploaded after been taken from “real-world” environments.  This is not a weakness of the particular paper alone but rather a challenge that the community needs to face. 

summary_of_the_review: Overall, the problem tackled in this paper is a challenging one as it attempts to find correlations of real-world tasks with ImageNet trained models. This is difficult to accomplish given that different datasets have different characteristics. But such studies are important, especially to understand the effect that improvements on ImageNet can lead to concrete improvements on real world applications. I fully agree with the final sentence of the paper that it is necessary to have a more diverse set of benchmarks that can improve learning algorithms more comprehensively. Such studies are thus important for the community.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper5529/Reviewer_ZbHU
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper provides experiments suggesting that there is only a very small correlation between performance of a neural classifier on ImageNet and its performance (after fine-tuning) on datasets that are not web-scraped.

strength_and_weaknesses: The numerical experiment seem to be well conducted. Yet, the results of the paper are not very surprising and some claims/conclusions of the authors seem strange or unnecessary. For instance, why this obsession with scrapped VS non-scrapped datasets? Isn't it enough to describe the reason why the 6 chosen datasets have implicitly different underlying distribution?

Why also not mentioning domain adaptation techniques that are beyond fine-tuning?

Some example of affirmation that are doubtful:

"In particular, we study datasets collected with the goal of solving real-world tasks (e.g., classifying images from
camera traps or satellites), as opposed to web-scraped benchmarks collected for
comparing models." -> Dataset collected from the web also serve to structuring data in the web. This is a real-world task.

"In particular, we consider classification tasks derived from image data that were specifically collected with the goal of classification in mind. This is in contrast to many standard computer vision datasets – including ImageNet – where the constituent
images were originally collected for a different purpose, posted to the web, and later re-purposed for benchmarking computer vision methods." -> There seems to be a confusion between the intent behind collecting the data and the intent behind the existence of the data. And what was the different purpose in the collection of data for ImageNet if not classification? The notion of intent is very subject to discussions...

"Web-scraped target datasets are by definition within the distribution of data collected from the web, and a sufficiently large model can
learn that distribution. This strategy may not be effective for non-web-scraped datasets, where there
is no guarantee that it is possible to train on data that is close in distribution to the target data, even
if we train on the entire web. Thus, it makes sense to distinguish these two types of datasets." -> While the argument has some logic, I do not find the distinction qualitatively very relevant. There might be a lot to argue. For instance, Camera Traps are a very common object now and many people are already posting it on the web, making the so called web distribution constantly changing.

"If improving ImageNet accuracy alone is insufficient to reach these well-tuned baselines, we can indeed conclude that architecture transfer to this target task is limited." -> Well, this is quite dubious. The improvment the authors perform may lack the power of the training techniques that have been used for these well-tuned baselines. These well-tuned baselines result from specific training approaches that may hypothetically depend on the task at hand, e.g., when employing curriculum learning (which is very much used for real-world applications), and hence not very much fall in the scope of the type of fine-tuning the authors are performing. I would hence be much more careful in their statement.
clarity,_quality,_novelty_and_reproducibility: The paper is quite clear, well written and surely reproductible. However, it is not very novel and limited in scope (only picking 6 datasets).

summary_of_the_review: Some affirmations of the paper are dubious, and the novelty is not very important; it is essentially an empirical work. Besides, the conclusion drawn by the paper feels already like common knowledge. For instance, in deep learning for satellite spectral imagery (like the Sentinel-2 example provided by the author), there are already many papers showing how to adapt the convolution specifically for that type of images that have a radically different distribution with respect to more classical images (scrapped or not scrapped...). The authors present some of their statements as if it is something that the community should change. But the truth is that it is already very clear for the practitioners of the various real-world tasks.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough