Paper: SynBench: Evaluating Pretrained Representations for Image Classification using Synthetic Data (ICLR-2024)

Reviewer comment (Theory):
The paper assumes that the raw input data for each class has a Gaussian distribution. In computer vision, this means assuming that, in pixel space, two visual categories have Gaussian distributions. This is akin to assuming we start with quite a good representation. What's the point of representation learning if that's where we start? Why would you transform your data?

Author response:
## 1. Why Gaussuian distribution
We view SynBench as a "necessary" and "minimum" model test. We concur with the reviewer's argument that assuming a Gaussian distribution for raw input data in pixel space implies starting with a significantly structured and "good" representation. This assumption is intentional and serves the specific purpose of establishing a foundational test. If a pretrained representation network cannot preserve the quality of data drawn from such simple, well-defined distribution (indicated by a SynBench-score close to 1), it raises significant concerns about its potential performance in more complex, real-world scenarios. In our test, we do see that the seperability of the representations could degrade by significant margins compared with their reference behavior in the well-defined simple input space.

To draw an analogy for easier understanding, consider a hgh-performing language model. One might test its basic capabilities by checking whether it correctly writes "2" after "1+1=" or classifies "bad" as a negative word. Failure to perform these straightforward tasks with clear expected outcomes would suggest fundamental deficiencies in the model. 

Moreover, we also believe that our designed task with conditional Gaussian is feasible. For a feature extractor robust to noises/good at denoising, all class-1 synthetic samples shall be treated as point $\mu$ and all class-2 samples as point $-\mu$, which are then trivial to classify.

Reviewer comment (Evaluation):
One of the key claims of the paper is that the "Pearson correlation between SynBench-Scores and the average real-life task accuracy is larger than 0.9". Flipping to Table 7 in the appendix, we learn that these Pearson correlations are based on $n=5$ data points. It is statistically unacceptable to make this claim without reporting confidence intervals. Running through the [standard calculations](http://faculty.washington.edu/gloftus/P317-318/Useful_Information/r_to_z/PearsonrCIs.pdf), it seems that the confidence interval for the Pearson correlation of 0.92 would be $[-0.81, 1.0]$. That is, this correlation value is not very meaningful. This seems to be a highly misleading error, but the reviewer is eager to be corrected if off base on this point. In the absence of this claim, there is little evidence that "real world" task performance is related to the scores computed in this paper.

Author response:
## 2. Pearson and confidence interval
Let $r$ be the Pearson correlation coefficient, $p$ be the number of models. We ran the calculation based on the reference shared by the reviewer, and see that the upper and lower confidence interval limits in z-space are $0.5\ln(\frac{1+r}{1-r}) \pm 1.96\sqrt{\frac{1}{p-3}}=1.589\pm 1.386$. Translating to r-space by $r=\frac{e^{2z}-1}{e^{2z}+1}$ yields the upper limit of 0.995 and the lower limit of 0.203. That said, we believe the confidence interval for the Pearson correlation of 0.92 is instead $[0.203,0.995]$ if the desired confidence level is $95\%$. We are not sure how the reviewer obtained $[-0.81,1.0]$ and wish the reviewer can let us know to make sure we are on the same page. 

In the following Table 4.1, we added four efficient nets' SynBench-scores, together with the average of their reported performance on 27 downstream tasks in [Radford et al., 2021](https://arxiv.org/pdf/2103.00020.pdf), Table 10. We ran the same calculation for the Pearson correlation coefficient $r=0.87$ and $p=9$ to obtain the confidence interval of $[0.488,0.972]$ which suggest at least moderate correlation upto strong corelation.

| | ViT-B/16 | ViT-L/16 | ViT-B/32 | Resnet50-SimCLRv2 | Resnet101-SimCLRv2 | ENet_b0   | ENet_b1   | ENet_b2   | ENet_b3   | Pearson correlation |
|----------|----------|----------|----------|-------------------|--------------------|------|------|------|------|---------------------|
| Accuracy (%) | 74.3     | 75.5     | 72.6     | 75.4              | 75.4               | 72.5 | 72.6 | 73.1 | 73.9 | 1.0                 |
| SynBench (n=2048)  | 0.33     | 0.26     | 0.0      | 0.66              | 0.60               | 0.02 | 0.04 | 0    | 0    | 0.85                |
| SynBench (n=4096)  | 0.52     | 0.49     | 0.01     | 0.69              | 0.84               | 0.13 | 0.13 | 0.09 | 0.03 | 0.87                |

Table 4.1 The correlation between SynBench-score and the average accuracy on 27 real-life tasks.

Reviewer comment (Evaluation):
SimCLRv2 has the highest SynBench-Score in Table 1, but is generally understood to perform more poorly than the other techniques in the table in common evaluation protocols (see, e.g., the papers for DINO, BYOL, MAE, or other more recent methods). No mention is made of this in the discussion. This seems like a fairly large problem for the proposed method.

Author response:
## 4. SimCLRv2 has higher SynBench-Score than DINO
We need to note that SynBench is to evaluate specific models, not methodology. As there might have many subtle differences in training parameters/optimizers/datasets, we can only make conclusions for models used in our paper. In our paper, we use SimCLRv2 models from the [simclr official repository](https://github.com/google-research/simclr), specifically, we use SimCLRv2 models with resnet depth 50 and 100, both with width 2X and SK True. For example, if we try to compare with DINO models, we can refer to [DINO official repository](https://github.com/facebookresearch/dino) and see they are generally on par with the specific SimCLRv2 models we considered (see table 4.2). From this example, we don't see these two specific SimCLRv2 models to perform more poorly than DINO, and our paper makes no conclusions for BYOL and MAE. As such, we do not see there is a problem for our proposed method.

| | ViT-S/16-DINO | ViT-S/8-DINO | ViT-B/16-DINO |ViT-B/8-DINO | ResNet50-SimCLRv2 | ResNet101-SimCLRv2 |
|----------|----------|----------|----------|-------------------|--------------------|------|
ImageNet acc. | 77.0% | 79.7% | 78.2% | 80.1% | 77.7% | 79.0%|

Table 4.2 The linear probing accuracy on ImageNet1k reported in the original repository.

Reviewer comment (Experiments):
The paper considers CIFAR10 and TinyImageNet as "real-life downstream data" (see Sec. 4.3). Many in computer vision would likely disagree with this characterization.

Author response:
##  5. CIFAR10 and TinyImageNet as "real-life downstream data" in Section 4.3 
These are popular CV downstream datasets. We have updated our paper accordingly.

Reviewer comment (Writing):
I generally found the paper rather hard to follow. It is often unclear if the authors are targeting adversarial robustness, or how well a representation transfers, or both.

Author response:
## 1. SynBench's target
SynBench's target covers both adversarial robustness and representation seperability (accuracy). To be more precise, we believe a good pretrained representation network should produce representations that entail good seperability, either *with* or *without*, adversarial perturbations. Please see our paper section 1, the second paragraph, for motivations of our scope. SynBench-score as a metric is designed to serve this purpose and perform evaluations. By definition, SynBench-score is the relative AUC of expected bound-threshold accuracy curve (see equation 3). When SynBench is used to evaluate clean accuracy as in Section 4.2, we let $\epsilon=0$, whearas when it is used to suggest robust linear probing parameters $\epsilon$ as in Section 4.3, we let $\epsilon=\text{argmax}_\epsilon \text{SynBench-Score}$.

Reviewer comment (Theory):
I might well have missed a central point of the paper, but I fundamentally doubt that it is possible solely based on Gaussian mixtures with two components, without any knowledge about the target downstream tasks, to accurately predict classification performance of feature extractors across a broad range of downstream tasks.

Author response:
## 2. How can SynBench predict classification performance across a broad range of tasks
Think of how representation learning research typically evaluate a model for transfer learning - by running tests on broad range of downstream tasks. And the reason behind this is to see how the model behaves in different scenerios. To theorize things, we believe the general behavior of a pretrained representation is measured by how it perform on tasks of different difficulty levels. That is why we think a fundamental part of our design is to simulate tasks of different difficulty levels. One difference between SynBench and a traditional probing test is that, for example, we are using the classification problem of two highly overlapped Gassuain, instead of classifying ImageNet21k. We hope this clarification builds enough intuition to understand the following:
1. We vary $s$ in equation 2 from 0.1 to 5 in increments of 0.1, which correspond to optimal accuracy (groundtruth difficulty) ranging from 55% to 100% and 50 difficulty levels. If we refer to this [figure](https://imgur.com/a/rH819Jm), we see each of the red points correspond to one of our similated trials with difficulty levels (x-axis).
2. Baseline methods are task/data dependant, which means they are somewhat bound to tasks of that similar difficulty levels. If we refer to the same [figure](https://imgur.com/a/rH819Jm), it could be the single purple point with fixed level of difficulty.  
3. If we include certain knowledge of possible downstream data properties, say locality of pixel dependencies, then the prediction will indeed be more accurate (see our section 4.4).

Reviewer comment (Experiments):
The baselines such as (Whitney et al., 2020) all rely on measures that are derived from the feature extractor and the downstream task/data, whereas the current method performs the predictions solely based on the feature extractor and synthetic data, while claiming to outperform the baselines.

Author response:
We thank the reviewer for active engagement in the discussion.

> 2. How can SynBench predict classification performance across a broad range of tasks

SynBench is proposed for use in two main scenerio: (1) when we do not have any knowledge about the downsteam task; (2) when we possess some statistical information about the downstream task (e.g., covariance of data).
While we see the reviewer's concern pertains to the second scenario, we want to emphasize that our framework works for **arbitrary covariacne matrices that model structures in pixel spaces**. In this regard, we have provided an example in Section 4.4 showing the use of block heptadiagonal matrix to mimic locality. With the modeled covariance, we showed that the SynBench score can provide an even more accurate estimate of downstream performance, imroving the correlation from 0.79 (Gaussian-I) to 0.91 (Gaussian-H) when the synthetic sample size $n=2048$. This result exactly justifies the reviewer's intuition on the utility of SynBench in the second scenario.

> 3. Dependency on the properties of the pretrained feature extractor

We did not realize the reviewer was referring to robustness literature. Indeed, in quantifying the robustness of end-to-end neural networks, using the network's Lipschitzness is straightforward as it bounds the gradient and can connect variations in the output space back to input space safety regions. In our context, our goal is to assess the accuracy-robustness tradeoff induced by pretrained representations. Therefore, we measure the accuracy and robustness directly on the representations themselves, eliminating the need to propagate back to the input space and hence sparing the need for including the Lipschitz constant of the pretrained extractor.

> 4. Correlation breakdowns

We respectively disagree with the conclusion drawn. First and foremost, SynBench is designed to assess the **overall performance** of the pretrained representations on a set of basic tasks. If **there exist a metric that highly correlates with the linear probing performance on every single downstream task**, it would imply that **the linear probing performance on every single downstream task also correlates highly with each other**— which is not the case in reality. Therefore, we are seeking a metric that can inform on the potential overall performance. In the following table, we provide the average ranking of correlations with downstream tasks by SynBench and other baselines as a more robust and intuitive measure. It is clear that SynBench is able to give the overall best correlation with each individual downstream. 

| Correlation | Average ranking 
| -------- | -------- | 
| SynBench  | **2.11$\pm$ 0.976**
|Val loss |  3.68$\pm$ 1.166
|MDL| 3.57$\pm$ 1.613
|LogME| 3.00$\pm$ 1.488
|SFDA|2.64$\pm$ 1.076

Reviewer comment (Theory):
Another aspect that I found surprising is that the theory does not depend on the properties of the pretrained feature extractor, for example its Lipschitz constant, which is usually the case in similar contexts.

Author response:
## 3. Discussions: dependency on the properties of the pretrained feature extractor
We thank the reviewer for bringing this up. If possible, can the reviewer please share the literatures mentioned that use properties of the pretrained representations in similar contexts with us? In our theory, the reference optimal behavior is completely independant of the pretrained feature extractor (the denominator in Equation 3), and the evaluation depends on the feature extractor via the numerator. During this process, we do not explicitly use properties such as Lipschitzness. To our knowledge, the connection between Lipschitzness of the representation network and the downstream task accuracy via linear probing is not obvious.

Reviewer comment (Experiments):
The authors only consider the linear probing paradigm in evaluation of pretrained models. In practice, finetuning is also a common way to use these pretrained models. It is not clear how this metric would perform in the finetuning setup.

Author response:
## 1. How SynBench would perform in the finetuning setup
As discussed in specified in Section 3.3, SynBench is designed to evaluate the pretrained representations parameterized by $\theta$. In the scenerio when finetuning is performed, the parameters of the representation network will be updated, hence resulting in another pretrained representations (a separate model) from the perspective of SynBench. In our paper, we have shown how to use SynBench to compare representation networks before and after finetuning (Section 4.2, Comparing model attributes). In our example, ``ViT-B/16`` and ``ViT-B/16-in21k`` were both pretrained on ImageNet21k with supervision, whereas ``ViT-B/16`` is further finetuned on Imagenet 1k (name convention is adopted from [PyTorch Image Models](https://github.com/huggingface/pytorch-image-models)). The result shows the utility of SynBench in comparing original and finetuned models.

Reviewer comment (Writing):
In the Algorithm (A.6), what are $z_1$ and $z_2$ in Line 12?

Author response:
## 3. $z_1$ and $z_2$ in Line 12 of Algorithm A.6
The subscript denotes the class label of the representation. That is, $z_1^{train,i}$ is the $i$-th training sample from class 1, and $z_2^{train,j}$ is the $j$-th training sample from class 2. We have included this detail to the updated paper.

Reviewer comment (Evaluation):
It would be interesting to analyse the correlation on a subset of tasks instead of average of 27 tasks. For instance, it may be the case that metric performs better on OOD tasks compared to transfer learning on some datasets.

Author response:
## 4. How does SynBench inform model performance on OOD tasks vs. others
We thank the reviewer for raising this question. In the following Table 2.1, we computed the Frechet Inception Distance (FID) scores from ImageNet21k to the downstream tasks, and used them as the indicator of how OOD are the tasks. We then computed SynBench-score correlation with tasks that have FID scores larger than a threshold {50,100,150,200}. We do want to note that not all models in our analysis are pretrained with ImageNet21k; however, since ImageNet21k has become a go-to pretraining dataset, we assume samples therein are in-distribution.

From table 2.2, we see that if we don't apply filter on FID (or equivelantly let threshold be 0), the initial correlation was 0.92. As we gradually increase the threshold to 50, 100, 150, and even 200, the correlation stays above 0.8, indeed suggesting SynBench's robustness to OOD tasks.

| Datasets| Food101 | CIFAR10 | CIFAR100 | birdsnap | SUN397 | StanfordCars | Aircraft  |  VOC2007 |  DTD |  Pets | Caltech101 | Flowers | MNIST | FER2013 | 
| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |-------- | -------- | -------- | -------- | -------- | -------- | -------- |
| FID to ImageNet21k| 100.81 | 115.47 | 96.22 | 102.39 | 54.78 | 154.81 | 206.47 | 52.30 | 98.37 | 104.15 | 53.51 | 112.64 | 301.28 |175.75 |  
| Datasets| STL10  | EuroSAT | RESISC45 | GTSRB | KITTI | Country211 | PCAM | UCF101 | Kinetics700 | CLEVR | HatefulMemes | SST |  ImageNet | 
| FID to ImageNet21k| 71.19 | 142.62| 104.80 | 156.81 | 163.92 | 36.72 | 235.63 | 79.40 | time out | 194.64 | 86.64 | 368.13 | 17.78 | 

Table 2.1 The Frechet Inception Distance (FID) scores from ImageNet21k to 27 downstream tasks.

| FID| > 0 (all tasks)| > 50 | >100 | >150 | > 200 |
|-------- |-------- |-------- |-------- |-------- | -------- | 
| SynBench Correlation | 0.92 | 0.93 | 0.93 | 0.82 | 0.92|

Table 2.2 The correlation between SynBench-score and the average accuracy of FID-thresholded downstream tasks.

Reviewer comment (Experiments):
Robustness to Deviating Distributions  
It would be valuable to assess SynBench's robustness when facing uncommon real-world data distributions. For instance, applying SynBench to datasets like DomainNet, which contains diverse and domain-shifted data, can provide insights into its adaptability to varying data sources and distributions. Demonstrating SynBench's effectiveness under such conditions would strengthen its applicability and reliability.

Author response:
## 1. How does SynBench inform model performance on OOD tasks vs. others
We agree that showing SynBench's robustness to OOD tasks will strengthen the applicability and reliability. In the following Table 1.1, we computed the Frechet Inception Distance (FID) scores from ImageNet21k to the downstream tasks, and used them as the indicator of how OOD are the tasks. We then computed SynBench-score correlation with tasks that have FID scores larger than a threshold {50,100,150,200}. We do want to note that not all models in our analysis are pretrained with ImageNet21k; however, since ImageNet21k has become a go-to pretraining dataset, we assume samples therein are in-distribution.

From table 1.2, we see that if we don't apply filter on FID (or equivelantly let threshold be 0), the initial correlation was 0.92. As we gradually increase the threshold to 50, 100, 150, and even 200, the correlation stays above 0.8, indeed suggesting SynBench's robustness to OOD tasks.

| Datasets| Food101 | CIFAR10 | CIFAR100 | birdsnap | SUN397 | StanfordCars | Aircraft  |  VOC2007 |  DTD |  Pets | Caltech101 | Flowers | MNIST | FER2013 | 
| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |-------- | -------- | -------- | -------- | -------- | -------- | -------- |
| FID to ImageNet21k| 100.81 | 115.47 | 96.22 | 102.39 | 54.78 | 154.81 | 206.47 | 52.30 | 98.37 | 104.15 | 53.51 | 112.64 | 301.28 |175.75 |  
| Datasets| STL10  | EuroSAT | RESISC45 | GTSRB | KITTI | Country211 | PCAM | UCF101 | Kinetics700 | CLEVR | HatefulMemes | SST |  ImageNet | 
| FID to ImageNet21k| 71.19 | 142.62| 104.80 | 156.81 | 163.92 | 36.72 | 235.63 | 79.40 | time out | 194.64 | 86.64 | 368.13 | 17.78 | 

Table 1.1 The Frechet Inception Distance (FID) scores from ImageNet21k to 27 downstream tasks.

| FID| > 0 (all tasks)| > 50 | >100 | >150 | > 200 |
|-------- |-------- |-------- |-------- |-------- | -------- | 
| SynBench Correlation | 0.92 | 0.93 | 0.93 | 0.82 | 0.92|

Table 1.2 The correlation between SynBench-score and the average accuracy of FID-thresholded downstream tasks.