Paper: Does CLIP’s generalization performance mainly stem from high train-test similarity? (ICLR-2024)

Reviewer comment (Experiments):
Even though the results in Figure 4 show the near-pruned samples are crucial for OOD generalization of CLIP, they are not conclusive. In particular, the following experiment would be useful: Does a model *only* lose OOD generalization on one benchmark or are these near-pruned samples crucial for all sorts of image-classification performance on any benchmark? This requires plotting the accuracy on two datasets, e.g., ImageNet-Val and ImageNet-Sketch, and showing that after pruning samples similar to ImageNet-Sketch, we only lose performance on ImageNet-Sketch.

Author response:
We firstly thank the reviewer for suggesting to do near/far pruning experiments on ImageNet. **We ran these experiments and now include them in App. B.2. (especially Figs. 8 and 9)**. Notably, based on Fig. 9, nearest neighbors seem to matter more for CLIP than for ImageNet as near/far pruning has a stronger absolute effect on CLIP. We hope these experiments address weakness 1.ii).

Concerning weakness 1.i), we believe that Fig. 7 (in the appendix) already answers this: performance generally decreases across dataset. This is also true for the results of the main experiment summarized in Tab. 1.

Yet, we maintain that we see our main contribution not as showing that nearest neighbors are crucial for OOD generalization, but rather that **highly similar images** cannot sufficiently explain CLIP’s performance on common benchmarks. In this context, the pruning in Sec. 4 merely demonstrates one important observation, and it is not our main contribution. We want to clarify the goals of Sec. 4:

1. We show that **similar images**, specifically the nearest neighbors, generally matter for CLIP’s performance (Figs. 3b and 4), whether the test set is ID or OOD. We don’t take ImageNet-Train into account here, and we don’t consider whether the pruned subsets are ID or OOD wrt. the test sets. This is emphasized by the new results in Figs. 8 and 9 where we can see that near-pruning ImageNet-Train with ImageNet-Val (which is ID wrt. ImageNet-Train) also hurts ResNet’s performance.
2. We want to highlight that LAION contains many **similar images**, but especially also contains **highly similar images**, i.e., images that resemble the test datapoints closer than any of the images in ImageNet-Train do (Figs. 1 and 3a).

We make a distinction between **similar images** and **highly similar images**. In Sec. 4.2 and Fig. 4, we are pruning *any* images in the order of decreasing similarity, i.e., the pruning is unconstrained. In contrast, in the main experiment in Sec. 6 we only prune **highly similar images**, i.e. the pruning is constrained by the similarity thresholds from ImageNet. We formalize this distinction based on the generalization gap (since renamed to similarity gap) in Sec. 5.

We hope that this clarifies the goal of our work and especially Secs. 5 and 6. We acknowledge that our introduction and some paragraphs throughout the paper were worded in a way that could give a wrong impression of our aim, and apologize for the confusion. **We have edited the manuscript in Secs. 1, 4, 5, and 6 to better reflect our intention**.

Reviewer comment (Experiments):
Even though the results in Figure 4 show the near-pruned samples are crucial for OOD generalization of CLIP, they are not conclusive. In particular, the following experiment would be useful: Do CLIP models lose OOD generalization more quickly than classification models trained on ImageNet? For this, one would prune ImageNet with a similar procedure for X% of samples and compare whether the accuracy drop in ImageNet models is slower than CLIP models. If not, then we would know that any OOD generalization we have been seeing could be due to seeing similar samples in the training set.

Author response:
We firstly thank the reviewer for suggesting to do near/far pruning experiments on ImageNet. **We ran these experiments and now include them in App. B.2. (especially Figs. 8 and 9)**. Notably, based on Fig. 9, nearest neighbors seem to matter more for CLIP than for ImageNet as near/far pruning has a stronger absolute effect on CLIP. We hope these experiments address weakness 1.ii).

Concerning weakness 1.i), we believe that Fig. 7 (in the appendix) already answers this: performance generally decreases across dataset. This is also true for the results of the main experiment summarized in Tab. 1.

Yet, we maintain that we see our main contribution not as showing that nearest neighbors are crucial for OOD generalization, but rather that **highly similar images** cannot sufficiently explain CLIP’s performance on common benchmarks. In this context, the pruning in Sec. 4 merely demonstrates one important observation, and it is not our main contribution. We want to clarify the goals of Sec. 4:

1. We show that **similar images**, specifically the nearest neighbors, generally matter for CLIP’s performance (Figs. 3b and 4), whether the test set is ID or OOD. We don’t take ImageNet-Train into account here, and we don’t consider whether the pruned subsets are ID or OOD wrt. the test sets. This is emphasized by the new results in Figs. 8 and 9 where we can see that near-pruning ImageNet-Train with ImageNet-Val (which is ID wrt. ImageNet-Train) also hurts ResNet’s performance.
2. We want to highlight that LAION contains many **similar images**, but especially also contains **highly similar images**, i.e., images that resemble the test datapoints closer than any of the images in ImageNet-Train do (Figs. 1 and 3a).

We make a distinction between **similar images** and **highly similar images**. In Sec. 4.2 and Fig. 4, we are pruning *any* images in the order of decreasing similarity, i.e., the pruning is unconstrained. In contrast, in the main experiment in Sec. 6 we only prune **highly similar images**, i.e. the pruning is constrained by the similarity thresholds from ImageNet. We formalize this distinction based on the generalization gap (since renamed to similarity gap) in Sec. 5.

We hope that this clarifies the goal of our work and especially Secs. 5 and 6. We acknowledge that our introduction and some paragraphs throughout the paper were worded in a way that could give a wrong impression of our aim, and apologize for the confusion. **We have edited the manuscript in Secs. 1, 4, 5, and 6 to better reflect our intention**.

Reviewer comment (Experiments):
The objective of Sections 5 and 6 that “Correct for highly similar images” is not clear. Why would we want to prune our training datasets to get worse on some test benchmarks if those exact test samples do not appear in the training set? Why would we want models in Table 1 trained on pruned datasets that perform worse?

Author response:
Yet, we maintain that we see our main contribution not as showing that nearest neighbors are crucial for OOD generalization, but rather that **highly similar images** cannot sufficiently explain CLIP’s performance on common benchmarks. In this context, the pruning in Sec. 4 merely demonstrates one important observation, and it is not our main contribution. We want to clarify the goals of Sec. 4:

1. We show that **similar images**, specifically the nearest neighbors, generally matter for CLIP’s performance (Figs. 3b and 4), whether the test set is ID or OOD. We don’t take ImageNet-Train into account here, and we don’t consider whether the pruned subsets are ID or OOD wrt. the test sets. This is emphasized by the new results in Figs. 8 and 9 where we can see that near-pruning ImageNet-Train with ImageNet-Val (which is ID wrt. ImageNet-Train) also hurts ResNet’s performance.
2. We want to highlight that LAION contains many **similar images**, but especially also contains **highly similar images**, i.e., images that resemble the test datapoints closer than any of the images in ImageNet-Train do (Figs. 1 and 3a).

We make a distinction between **similar images** and **highly similar images**. In Sec. 4.2 and Fig. 4, we are pruning *any* images in the order of decreasing similarity, i.e., the pruning is unconstrained. In contrast, in the main experiment in Sec. 6 we only prune **highly similar images**, i.e. the pruning is constrained by the similarity thresholds from ImageNet. We formalize this distinction based on the generalization gap (since renamed to similarity gap) in Sec. 5.

We hope that this clarifies the goal of our work and especially Secs. 5 and 6. We acknowledge that our introduction and some paragraphs throughout the paper were worded in a way that could give a wrong impression of our aim, and apologize for the confusion. **We have edited the manuscript in Secs. 1, 4, 5, and 6 to better reflect our intention**.

Reviewer comment (Experiments):
Figure 4: What if we perform a similar process for training on the ImageNet dataset? Do we observe a similar accuracy drop?

Author response:
We firstly thank the reviewer for suggesting to do near/far pruning experiments on ImageNet. **We ran these experiments and now include them in App. B.2. (especially Figs. 8 and 9)**. Notably, based on Fig. 9, nearest neighbors seem to matter more for CLIP than for ImageNet as near/far pruning has a stronger absolute effect on CLIP. We hope these experiments address weakness 1.ii).

Reviewer comment (Experiments):
This paper only works on common classification ood benchmarks, but neglecting a whole bunch of other CLIP application domains. In the very least, retrieval is the most foundamental test ground to probe how good a CLIP model is. The authors could use MSCOCO as the in-domain dataset for sample filtering, and use Flickr-30k as the out-of-domain dataset. Also, experiments on more diverse benchmarks, like VTAB (check datacomp paper for current best practices), are encouraged.

Author response:
We agree with the reviewer that one of the interesting use cases of CLIP is image retrieval. However, we do not believe that this undermines the relevance zero-shot classification. We focus on this task as there has been a huge body of work on this topic, and CLIP demonstrated unprecedented performance on ImageNet-based distribution shifts. Additionally, in contrast to classification, retrieval tasks are more complex and highly sensitive to captions, demanding an analysis that factors in both images and texts. Therefore, we did not perform experiments on Flickr30 and MSCOCO.

We would also like to underline that not all ImageNet-based distribution shifts are characteristically similar. While ImageNet-Sketch and ImageNet-R have style changes, ObjectNet has shifts in rotation, background, viewpoints, and ImageNet-A has adversarial or hard images. Therefore our analysis and results are valid for a broad range of characteristically different distribution shifts in classification.

That said, we agree with the reviewer that a stronger case can be made with experiments on other datasets and for distribution shifts that are not based on ImageNet. At the moment, **as an additional experiment in Appx. C, we have repeated the similarity analysis from Fig. 3 for CelebA [1] and Waterbirds [2]**. For CelebA, we split along the factors `Eyeglasses`, `Hats` as distribution shifts and zero-shot predict the factor `Male`. For Waterbirds, we can either split based on the background (`land` vs. `water`), or based on the combination of bird and land (`landbird-on-land/waterbird-on-water` vs. `landbird-on-water/waterbird-on-land`). We observe the same trends (similarity distributions to the train set/LAION differ, similarity is correlated with accuracy) as we did in Sec. 4.

```python
| Dataset | Attributes | Correlation between similarity and zero-shot accuracy $\rho_S$ |
|-|-|-|
| CelebA | w/ eyeglasses | 1.0|
| CelebA | w/ hat | 0.78|
| | | |
| Waterbirds | Coregroup | 0.82 |
| Waterbirds| Land | 0.98 |
```

Given the rebuttal’s tight time frame and the computational cost of training CLIP, we cannot yet replicate the experiment from Sec. 6 on these datasets. But, if the reviewer feels strongly about additional experiments, we are happy to repeat the complete analysis from Sec. 6 on CelebA and Waterbirds (and possibly also iWILDCam) for the camera-ready version.

Reviewer comment (Reproducibility):
Some crucial experimental details seem to be missing. For instance, how the nearest neighbor sets in section 4.1 are constructed are not mentioned at all (or mentioned in the appendix). How many samples are chosen in this set, and what is the threshold of CLIP score used here?

Author response:
**We apologize for the confusion and have updated the manuscript to better explain what Sec. 4.1 is doing**. 

For the nearest neighbor sets in Fig. 1, we simply visualize the top six nearest neighbors to a given query sample (i.e., a sample from one of the test sets) in ImageNet-Train and LAION (after deduplication). **We have added more details in App. F regarding generating the nearest neighbors visualizations**. 

For the histograms in Fig 3., we simply take one of the six test sets and for each sample compute the similarity to its nearest neighbor in ImageNet or LAION (with duplicates removed). The histograms show the distribution over those similarities.

Neither of these cases in Sec. 4.1. imposes a threshold on the perceptual similarity.

Reviewer comment (Experiments):
The authors should compare the similarity of nearest neighbors to test sets between the whole LAION datset (after filtering in section 5/6) and ImageNet-Train. Otherwise, one could argue that, even though the closest sample in LAION and ImageNet training set is about the same far away to the test set, LAION has a lot more samples close to the closest sample than ImageNet (but not as close to the test set as the closest sample), and thus still has a unfair advantage on those ood benchmarks, compromising the validity of the conclusion.

Author response:
We thank the reviewer for suggesting to compare similarity distribution of the datasets after pruning, and we agree that this is an interesting analysis that strengthens our contribution. We have added the analysis in Appx. Section E. 

In this comparison, we show the nearest neighbor similarity of all samples in ImageNet/LAION as a ratio of the highest nearest neighbor similarity of the corresponding test sample. That is, points with a ratio  > 1 have been pruned, points with a ratio = 1 lie directly on the generalization gap (now renamed to similarity gap), and points with a ratio  < 1 lie farther away from the gap and remain after pruning. We observe:

1. Samples in ImageNet are proportionally much closer to the gap than samples in LAION.
2. Due to its immense size, LAION nevertheless has more points close to the gap in absolute numbers.

Thus, as the reviewer correctly notes, LAION’s large scale and data diversity give it **“an unfair advantage on the OOD benchmarks”**. However, we reiterate that this is not in conflict with our findings and instead corroborates our conclusion: **We aimed to investigate to what extent *highly similar training points (images that are both stylistically and semantically very close to the test points)* factor into CLIP’s performance**. We do not claim that our work examines all properties of the data distribution; in fact, we already hypothesize in the conclusion section that after correcting for this factor **data diversity** and **data scale** are the most likely remaining confounders for a fair comparison. Given the complexity and computation cost of our experiments, we leave further analysis of these confounders for future work.

Reviewer comment (Evaluation):
Further, there may be other objects, such as "space shuttle" which are not included in ImageNet, thus sketches of space shuttles can be present in the training data, in addition to natural images of the same. Thus the domain "sketch" is not unseen by CLIP, and hence the evaluation is not truly OOD.

Author response:
We thank reviewer **MLz9** for reviewing our paper, calling our experiments insightful, and providing helpful suggestions. In light of your concerns, we would first like to clarify the scope of our work.

Prior work has established that data distribution drives CLIP’s performance on ImageNet Out-of Distribution (OOD) benchmarks [1]. While precisely assessing how similar image distributions are is difficult, and (to the best of our knowledge), there is no consensus how the OOD-ness of a dataset could be quantified, it seems evident that these ImageNet OOD benchmarks cannot be considered to be OOD wrt. LAION. But **independent of whether these benchmarks can be considered OOD or not, it is interesting to examine what properties of its training distribution allow CLIP to reach such a high performance on them**.

The simplest hypothesis (a first-order explanation, if you will) is that CLIP just performs well because LAION contains many **highly similar images**, i.e images that are both semantically and stylistically similar to the test datasets, possibly even direct duplicates. Note that **to answer this hypothesis, we do not need to explicitly assume that the benchmarks are ID or OOD wrt. LAION**, neither before nor after our pruning intervention.

To summarize: **We agree with the reviewer that measuring CLIP’s “true OOD performance” is  interesting, but we do not claim to address this question in our work.** We realize that our introduction and some paragraphs throughout the paper were worded in a way that could give this impression, and apologize for the confusion. **We have edited the manuscript in Secs. 1, 4, 5, and 6 to better reflect our intention**. We would now like to also address the individual concerns here.


### Weaknesses 1, 2, 3 & Question 1: On a fair OOD generalization setting by removing all images of a certain domain

We completely agree with the reviewer that fair OOD testing would require removing all images pertaining to a certain domain. Indeed, we initially considered doing exactly this, but had to realize that this is far from trivial.

To remove all images of a certain domain, we need to be able to label each image as ‘ID’ or ‘OOD’. This essentially means that we need access to a domain classifier (which would also need to have near-perfect accuracy so that no images are overlooked). However, it is very unclear how such a classifier could be obtained. Even for the ‘sketch’ domain, where a classifier could conceivably be trained, it is unclear exactly how the classifier should demarcate this domain: Should the domain contain all sketches, even sketches with characteristics not present in ImageNet-Sketch? What about pencil sketches with a very limited color-palette (ImageNet-Sketch is mostly black-and-white)? What about tattoos or small sketches on objects in natural images (like printing on t-shirts? For other benchmarks, such as ImageNet-A, it is even less clear how the test images constitute a well-separable domain of images. It is precisely this vagueness in defining a domain based on a given test set that prevents us from building a fair OOD setting, which is why we do not claim to analyze this.

The reviewers’ example also touches on another interesting point: It could well be that after our pruning sketches of, say, space shuttles remain, while airplanes are only present in natural images. In this case, CLIP might have to compositionally generalize in order to reach high performance on the ‘airplane’ class in ImageNet-Sketch. It would indeed be very interesting to analyze how individual concepts in the data distribution can be combined by the model, but this is out of the scope of this work, and we are not even sure that such an analysis would be computationally feasible.

Reviewer comment (Writing):
It would be better to give a different name to "generalization gap" as this term is used in a different context.

Author response:
### Weakness 5:  It would be better to give a different name to "generalization gap" as this term is used in a different context

We have changed “generalization gap” to “similarity gap”.

Reviewer comment (Experiments):
Several additional experiments are required to be done, in order to emulate a true **OOD** setting, as discussed in the weaknesses section.

Author response:
We thank reviewer **MLz9** for reviewing our paper, calling our experiments insightful, and providing helpful suggestions. In light of your concerns, we would first like to clarify the scope of our work.

Prior work has established that data distribution drives CLIP’s performance on ImageNet Out-of Distribution (OOD) benchmarks [1]. While precisely assessing how similar image distributions are is difficult, and (to the best of our knowledge), there is no consensus how the OOD-ness of a dataset could be quantified, it seems evident that these ImageNet OOD benchmarks cannot be considered to be OOD wrt. LAION. But **independent of whether these benchmarks can be considered OOD or not, it is interesting to examine what properties of its training distribution allow CLIP to reach such a high performance on them**.

The simplest hypothesis (a first-order explanation, if you will) is that CLIP just performs well because LAION contains many **highly similar images**, i.e images that are both semantically and stylistically similar to the test datasets, possibly even direct duplicates. Note that **to answer this hypothesis, we do not need to explicitly assume that the benchmarks are ID or OOD wrt. LAION**, neither before nor after our pruning intervention.

To summarize: **We agree with the reviewer that measuring CLIP’s “true OOD performance” is  interesting, but we do not claim to address this question in our work.** We realize that our introduction and some paragraphs throughout the paper were worded in a way that could give this impression, and apologize for the confusion. **We have edited the manuscript in Secs. 1, 4, 5, and 6 to better reflect our intention**. We would now like to also address the individual concerns here.


### Weaknesses 1, 2, 3 & Question 1: On a fair OOD generalization setting by removing all images of a certain domain

We completely agree with the reviewer that fair OOD testing would require removing all images pertaining to a certain domain. Indeed, we initially considered doing exactly this, but had to realize that this is far from trivial.

To remove all images of a certain domain, we need to be able to label each image as ‘ID’ or ‘OOD’. This essentially means that we need access to a domain classifier (which would also need to have near-perfect accuracy so that no images are overlooked). However, it is very unclear how such a classifier could be obtained. Even for the ‘sketch’ domain, where a classifier could conceivably be trained, it is unclear exactly how the classifier should demarcate this domain: Should the domain contain all sketches, even sketches with characteristics not present in ImageNet-Sketch? What about pencil sketches with a very limited color-palette (ImageNet-Sketch is mostly black-and-white)? What about tattoos or small sketches on objects in natural images (like printing on t-shirts? For other benchmarks, such as ImageNet-A, it is even less clear how the test images constitute a well-separable domain of images. It is precisely this vagueness in defining a domain based on a given test set that prevents us from building a fair OOD setting, which is why we do not claim to analyze this.

The reviewers’ example also touches on another interesting point: It could well be that after our pruning sketches of, say, space shuttles remain, while airplanes are only present in natural images. In this case, CLIP might have to compositionally generalize in order to reach high performance on the ‘airplane’ class in ImageNet-Sketch. It would indeed be very interesting to analyze how individual concepts in the data distribution can be combined by the model, but this is out of the scope of this work, and we are not even sure that such an analysis would be computationally feasible.

Reviewer comment (Experiments):
For example, one should remove all "sketch" style images as well, from L-200M+IN-Train (sketch-pruned) in order to test on the ImageNet-sketch test set for a true OOD evaluation. Otherwise, other sketch images that are present in the dataset can help bridge the domain gap between sketches and natural images, leading to an unfair OOD evaluation.

Author response:
We thank reviewer **MLz9** for reviewing our paper, calling our experiments insightful, and providing helpful suggestions. In light of your concerns, we would first like to clarify the scope of our work.

Prior work has established that data distribution drives CLIP’s performance on ImageNet Out-of Distribution (OOD) benchmarks [1]. While precisely assessing how similar image distributions are is difficult, and (to the best of our knowledge), there is no consensus how the OOD-ness of a dataset could be quantified, it seems evident that these ImageNet OOD benchmarks cannot be considered to be OOD wrt. LAION. But **independent of whether these benchmarks can be considered OOD or not, it is interesting to examine what properties of its training distribution allow CLIP to reach such a high performance on them**.

The simplest hypothesis (a first-order explanation, if you will) is that CLIP just performs well because LAION contains many **highly similar images**, i.e images that are both semantically and stylistically similar to the test datasets, possibly even direct duplicates. Note that **to answer this hypothesis, we do not need to explicitly assume that the benchmarks are ID or OOD wrt. LAION**, neither before nor after our pruning intervention.

To summarize: **We agree with the reviewer that measuring CLIP’s “true OOD performance” is  interesting, but we do not claim to address this question in our work.** We realize that our introduction and some paragraphs throughout the paper were worded in a way that could give this impression, and apologize for the confusion. **We have edited the manuscript in Secs. 1, 4, 5, and 6 to better reflect our intention**. We would now like to also address the individual concerns here.


### Weaknesses 1, 2, 3 & Question 1: On a fair OOD generalization setting by removing all images of a certain domain

We completely agree with the reviewer that fair OOD testing would require removing all images pertaining to a certain domain. Indeed, we initially considered doing exactly this, but had to realize that this is far from trivial.

To remove all images of a certain domain, we need to be able to label each image as ‘ID’ or ‘OOD’. This essentially means that we need access to a domain classifier (which would also need to have near-perfect accuracy so that no images are overlooked). However, it is very unclear how such a classifier could be obtained. Even for the ‘sketch’ domain, where a classifier could conceivably be trained, it is unclear exactly how the classifier should demarcate this domain: Should the domain contain all sketches, even sketches with characteristics not present in ImageNet-Sketch? What about pencil sketches with a very limited color-palette (ImageNet-Sketch is mostly black-and-white)? What about tattoos or small sketches on objects in natural images (like printing on t-shirts? For other benchmarks, such as ImageNet-A, it is even less clear how the test images constitute a well-separable domain of images. It is precisely this vagueness in defining a domain based on a given test set that prevents us from building a fair OOD setting, which is why we do not claim to analyze this.

The reviewers’ example also touches on another interesting point: It could well be that after our pruning sketches of, say, space shuttles remain, while airplanes are only present in natural images. In this case, CLIP might have to compositionally generalize in order to reach high performance on the ‘airplane’ class in ImageNet-Sketch. It would indeed be very interesting to analyze how individual concepts in the data distribution can be combined by the model, but this is out of the scope of this work, and we are not even sure that such an analysis would be computationally feasible.

Reviewer comment (Experiments):
Even though authors perform many experiments, the experiments are performed on a single type of dataset, which are the variations of the ImageNet.

Author response:
We thank the reviewer for the suggestion to perform experiments on iWILDCam or FMoW. **We agree that experiments on distribution shifts not based on ImageNet are interesting to look at.** To provide additional context: We chose ImageNet-based distribution shifts because CLIP models perform extremely well on those, making them perfect candidates to test our primary hypothesis: whether **highly similar images drive performance**. We would also like to underline that not all ImageNet based distribution shifts are characteristically similar. While ImageNet-Sketch and ImageNet-R have style changes, ObjectNet has shifts in rotation/background/viewpoints, and ImageNet-A has adversarial or hard images. Therefore, our analysis and results are valid for a broad range of characteristically different distribution shifts.

That said, we agree with the reviewer that a stronger case can be made with experiments on other datasets. We find the proposed iWILDCam and FMoW datasets problematic since CLIP’s  (ViT-B/32) zero-shot performance on them is rather low (7.45% and 12.96%) [2]. As an alternative, we propose to look at two datasets with higher CLIP zero-shot accuracy. On CelebA [3], [4] report a zero-shot accuracy between 53% to 97%, and on Waterbirds [5], [6] report a zero-shot accuracy for ViT-B/16 and ViT-B/32 models of around 87.34%. We can split these datasets along some axes and analyze CLIP’s zero-shot prediction. 

At the moment, **as an additional experiment in Appendix Section C, we have repeated the similarity analysis from Fig. 3 for CelebA and Waterbirds.** For CelebA, we split along the factors Eyeglasses, Hats as distribution shifts and zero-shot predict the factor Male. For Waterbirds, we can either split based on the background (land vs. water), or based on the combination of bird and land (landbird-on-land/waterbird-on-water vs. landbird-on-water/waterbird-on-land). We observe the same trends (similarity distributions to the train set/LAION differ, similarity is correlated with accuracy) as we did in Sec. 4.

| Dataset | Attributes | Correlation between similarity and zero-shot accuracy $\rho_S$ |
|-|-|-|
| CelebA | w/ eyeglasses | 1.0|
| CelebA | w/ hat | 0.78|
| | | |
| Waterbirds | Coregroup | 0.82 |
| Waterbirds| Land | 0.98 |

Given the rebuttal’s tight time frame and the computational cost of training CLIP, we cannot yet replicate the experiment from Sec. 6 on these datasets. But, if the reviewer feels strongly about additional experiments, we are happy to repeat the complete analysis from Sec. 6 on CelebA and Waterbirds (and possibly also iWILDCam) for the camera-ready version.

Reviewer comment (Experiments):
It would have been to include experiments with other OOD datasets such as iWILDCam or FMoW. Without these experiments it is not sure if these analysis is limited to one type of dataset.

Author response:
We thank the reviewer for the suggestion to perform experiments on iWILDCam or FMoW. **We agree that experiments on distribution shifts not based on ImageNet are interesting to look at.** To provide additional context: We chose ImageNet-based distribution shifts because CLIP models perform extremely well on those, making them perfect candidates to test our primary hypothesis: whether **highly similar images drive performance**. We would also like to underline that not all ImageNet based distribution shifts are characteristically similar. While ImageNet-Sketch and ImageNet-R have style changes, ObjectNet has shifts in rotation/background/viewpoints, and ImageNet-A has adversarial or hard images. Therefore, our analysis and results are valid for a broad range of characteristically different distribution shifts.

That said, we agree with the reviewer that a stronger case can be made with experiments on other datasets. We find the proposed iWILDCam and FMoW datasets problematic since CLIP’s  (ViT-B/32) zero-shot performance on them is rather low (7.45% and 12.96%) [2]. As an alternative, we propose to look at two datasets with higher CLIP zero-shot accuracy. On CelebA [3], [4] report a zero-shot accuracy between 53% to 97%, and on Waterbirds [5], [6] report a zero-shot accuracy for ViT-B/16 and ViT-B/32 models of around 87.34%. We can split these datasets along some axes and analyze CLIP’s zero-shot prediction. 

At the moment, **as an additional experiment in Appendix Section C, we have repeated the similarity analysis from Fig. 3 for CelebA and Waterbirds.** For CelebA, we split along the factors Eyeglasses, Hats as distribution shifts and zero-shot predict the factor Male. For Waterbirds, we can either split based on the background (land vs. water), or based on the combination of bird and land (landbird-on-land/waterbird-on-water vs. landbird-on-water/waterbird-on-land). We observe the same trends (similarity distributions to the train set/LAION differ, similarity is correlated with accuracy) as we did in Sec. 4.

| Dataset | Attributes | Correlation between similarity and zero-shot accuracy $\rho_S$ |
|-|-|-|
| CelebA | w/ eyeglasses | 1.0|
| CelebA | w/ hat | 0.78|
| | | |
| Waterbirds | Coregroup | 0.82 |
| Waterbirds| Land | 0.98 |

Given the rebuttal’s tight time frame and the computational cost of training CLIP, we cannot yet replicate the experiment from Sec. 6 on these datasets. But, if the reviewer feels strongly about additional experiments, we are happy to repeat the complete analysis from Sec. 6 on CelebA and Waterbirds (and possibly also iWILDCam) for the camera-ready version.

Reviewer comment (Experiments):
Are there experiments on other datasets unrelated to ImageNet? Having these experiments would be better to make a strong case.

Author response:
We thank the reviewer for the suggestion to perform experiments on iWILDCam or FMoW. **We agree that experiments on distribution shifts not based on ImageNet are interesting to look at.** To provide additional context: We chose ImageNet-based distribution shifts because CLIP models perform extremely well on those, making them perfect candidates to test our primary hypothesis: whether **highly similar images drive performance**. We would also like to underline that not all ImageNet based distribution shifts are characteristically similar. While ImageNet-Sketch and ImageNet-R have style changes, ObjectNet has shifts in rotation/background/viewpoints, and ImageNet-A has adversarial or hard images. Therefore, our analysis and results are valid for a broad range of characteristically different distribution shifts.

That said, we agree with the reviewer that a stronger case can be made with experiments on other datasets. We find the proposed iWILDCam and FMoW datasets problematic since CLIP’s  (ViT-B/32) zero-shot performance on them is rather low (7.45% and 12.96%) [2]. As an alternative, we propose to look at two datasets with higher CLIP zero-shot accuracy. On CelebA [3], [4] report a zero-shot accuracy between 53% to 97%, and on Waterbirds [5], [6] report a zero-shot accuracy for ViT-B/16 and ViT-B/32 models of around 87.34%. We can split these datasets along some axes and analyze CLIP’s zero-shot prediction. 

At the moment, **as an additional experiment in Appendix Section C, we have repeated the similarity analysis from Fig. 3 for CelebA and Waterbirds.** For CelebA, we split along the factors Eyeglasses, Hats as distribution shifts and zero-shot predict the factor Male. For Waterbirds, we can either split based on the background (land vs. water), or based on the combination of bird and land (landbird-on-land/waterbird-on-water vs. landbird-on-water/waterbird-on-land). We observe the same trends (similarity distributions to the train set/LAION differ, similarity is correlated with accuracy) as we did in Sec. 4.

| Dataset | Attributes | Correlation between similarity and zero-shot accuracy $\rho_S$ |
|-|-|-|
| CelebA | w/ eyeglasses | 1.0|
| CelebA | w/ hat | 0.78|
| | | |
| Waterbirds | Coregroup | 0.82 |
| Waterbirds| Land | 0.98 |

Given the rebuttal’s tight time frame and the computational cost of training CLIP, we cannot yet replicate the experiment from Sec. 6 on these datasets. But, if the reviewer feels strongly about additional experiments, we are happy to repeat the complete analysis from Sec. 6 on CelebA and Waterbirds (and possibly also iWILDCam) for the camera-ready version.

Reviewer comment (Experiments):
Why do we use only CLIP ViT-B/16 embeddings to filter? Won't larger CLIP models have a better embedding to filter out the data points?

Author response:
**We agree with the reviewer that larger models would potentially capture image features and similarities better**. We originally went with the CLIP ViT-B/16-240+ model as
1. it was previously employed by Abbas et al [1] for semantic de-duplication,
2. it provides a good balance between performance and computational efficiency [2], 
3. our nearest neighbor visualizations using this model seemed to be human aligned,
Having said that, **based on the reviewer’s suggestion, we computed the nearest neighbor similarities of ImageNet-Train to each of the six different datasets using the image embedding space of ViT-L/14 as well — see the newly added Fig. 12 in Appendix Section D**. We then compute the correlation of the ViT-L/14 similarities to ViT-B/16-240+ similarities and observe very high correlation (in the range $\rho=0.8 \dots 0.93$) across datasets (see the newly added Tab. 5 in Appendix Section D). This indicates that the embedding space of ViT-B/16-240+ already sufficiently captures image features and similarities, and that using a bigger model wouldn’t change the trends or our conclusions.