Paper: IS SYNTHETIC DATA USEFUL FOR TRANSFER LEARNING? AN INVESTIGATION INTO DATA GENERATION, VOLUME, AND UTILIZATION (ICLR-2024)

Reviewer comment (Experiments):
The main issue with the paper is that it is comparing:  
1) **Baseline**: a training procedure that only takes *Imagenet-1k* data (1.3M images) + downstream data  
2) **Method Proposed**: a training procedure that takes *Imagenet-1k* + *LAION-5B* (StableDiffusion).  
The proposed technique uses Imagenet-1k + LAION-5B (distilled into StableDiffusion), so it should only be compared against methods that use the same or similar amounts of *data* and *compute*. For example, how does a model pretrained on LAION-2/5B (like OpenCLIP, readily available online in [1]) perform when finetuned or even zero-shot? It is unsurprising that a machine learning method that has access to 5B images (with fine-grained labels) performs better than one that only has access to 1.3M images (with 1k discrete annotations).

Author response:
Thank you for raising this intriguing point. We concur with the reviewer that our "Baseline and Proposed Method involve different data sources." Nonetheless, we would like to clarify that our experiments reveal that "mixed transfer" (employing both ImageNet-1k and LAION-5B) often falls short compared to "vanilla transfer" (solely ImageNet-1k). This outcome suggests that the mere expansion of data does not guarantee enhanced performance. To explicitly reiterate the primary goals of this research: 1) We delve into how **off-the-shelf** stable diffusion can unlock new avenues for boosting transfer learning efficacy. 2) We examine the strategic leverage of stable diffusion through our novel "bridged transfer" and "dataset style inversion" techniques.

Furthermore, we believe it is essential to highlight that our study contributes to the ongoing efforts of integrating text-to-image models into established deep learning workflows. For instance, [1] harnesses synthetic images from Stable Diffusion in place of the CC3M dataset, thereby assessing the impact on model performance with synthetic against original CC3M images; [2] employs Stable Diffusion to enrich the ImageNet 1K dataset, leading to performance enhancements against original ImageNet 1K.

Reviewer comment (Experiments):
The paper does not provide statistics of the data trained on StableDiffusion with respect to the downstream tasks. For example, for the class Aircraft-Boeing 787 or for a specific car model, it may be the case that LAION-X has more than 3k examples. If one cannot generate more than 3k examples (as in Figure 5, because of computational constraints?), why not use the data directly available? Looking at this and pointing out clear examples where one can generate much more quantity/diversity data than that present in LAION-X would be a good insight. For example, is there a specific type of Aircraft model that is hardly present in LAION-X, but one can generate an infinite amount of realistic images for it?

Author response:
Thanks for your questions. To the best of our knowledge, the reviewer is asking the question of whether using Stable Diffusion data to perform is worth it or not when one can access some real data (for a specific class). To answer this question, we provide two perspectives. 

1. Would LAION-X images help for target dataset transfer learning? We do not verify this question since LAION-X does not give accurate labels for each image. Rather, each image is paired with a caption. For example, when we retrieve the images of class “747-200 Airplan” from here (https://rom1504.github.io/clip-retrieval/?back=https%3A%2F%2Fknn.laion.ai&index=laion5B-L-14&useMclip=false&query=747-200+Airplane), lots of different types of airplane have been retrieved in this case. Therefore, it is hard to determine how to adapt LAION-X images into the supervised training pipeline right now. And this is beyond the scope of our paper. 

2. To quantitatively convert the effect of synthetic data to the effect of real data, we can study this question in few-shot learning scenarios. We run the vanilla transfer from 1-shot to 50-shot one by one and match the accuracy of the bridged transfer. Therefore, we can measure additional samples by evaluating how much more data the vanilla transfer would need to realize the same accuracy as the bridged transfer. The results for the Aircraft dataset of ResNet-18 are shown below:

| Accuracy | Bridged Transfer (w. 3k images / cls) | Vanilla Transfer |
|----------|---------------------------------------|------------------|
| 25.1     | 1-shot                                | 6-shot           |
| 45.0     | 4-shot                                | 14-shot          |
| 58.8     | 8-shot                                | 24-shot          |
| 71.4     | 16-shot                               | 40-shot          |

It can be found that synthetic data can effectively improve the performance of vanilla transfer and equal to using more images. In reality, collecting the real data may have some other issues like costs and privacy, please see our reply to Reviewer HV3t.

Reviewer comment (Experiments):
There may be dataset contamination, which is not discussed, i.e., the test samples of the downstream tasks may be part of LAION-X, while this is not the case for the baseline. Does the model improve performance when prompted with "generate a * with the style of LSUN" without dataset adaptation?

Author response:
Thanks for your suggestion. We experimented with this prompt and rerun it on the Aircraft dataset, there was no explicit performance improvement with this prompt (within 0.1% accuracy difference).

Reviewer comment (Presentation):
Visual examples would make the paper stronger. For example, when sampling the different "aircraft", how different are the images from the downstream task? How do these images change when doing dataset Style Inversion?

Author response:
Thank you for your advice, we have added visualizations in Appendix D in our revised version, please check it.

Reviewer comment (Experiments):
Evaluating on tasks that are specialized (like medical, satellite imaging...) that may not be as prevalent on LAION-X as "natural" images (vehicles, animals...) would make for a stronger paper and ameliorate some of the concerns expressed in the points above.

Author response:
Thank you for your suggestions. We conducted EuroSAT image classification under the few-shot learning regime (since the full-shot test performance is nearly 100%), and we synthesized 1000 images per class for this dataset. Here are the results. 

| Method             | 1-shot | 2-shot | 4-shot | 8-shot | 16-shot |
|--------------------|--------|--------|--------|--------|---------|
| Vanilla Transfer   | 49.1   | 60.5   | 65.5   | 78.0   | 85.2    |
| Bridged Transfer++ | 64.2   | 66.7   | 72.2   | 79.9   | 86.8    |

Reviewer comment (Experiments):
What is the percentage of images from synthetic and real on the Mixed transfer experiment?

Author response:
In the mixed transfer experiments, we synthesized 1000 images per class. As for the real images per class, we refer you to Appendix A, Table 4 for a detailed description. Roughly speaking, the number of real images per class is within the range of 30~750.

Reviewer comment (Evaluation):
In LEEP analysis in Table 1, isn't it expected that a model that has lower train/test accuracy (as seen in Figure1 for imagenet pretrained vs bridged pertaining) performs worse on this metric too? Is the difference in transferability (when tested with this metric) also present when two models compared have the same training loss?

Author response:
The higher the LEEP score, the better the transferability. However, we want to emphasize that the LEEP score is evaluated without any finetuning, instead, it only involves forwarding the target dataset to the pre-trained model. Therefore, our Table 1 and Figure 1 results are the same.

Reviewer comment (Experiments):
The main body of experiments (for example when comparing transfer learning strategies), are performed with only text prompts as conditioning for generating synthetic data. As this is expected to generate large domain gaps with the downstream datasets, it is not surprising that the mixing of real and synthetic images performs worst. How would this compare if the style inversion was included, which reduces the domain gap between synthetic and downstream data? Or using some other alignment mechanisms such as textual inversion, dreambooth [1] or diffusion inversion [2].

Author response:
Q1: The main body of experiments (for example when comparing transfer learning strategies), are performed with only text prompts as conditioning for generating synthetic data. As this is expected to generate large domain gaps with the downstream datasets, it is not surprising that the mixing of real and synthetic images performs worst. How would this compare if the style inversion was included, which reduces the domain gap between synthetic and downstream data? Or using some other alignment mechanisms such as textual inversion, dreambooth [1] or diffusion inversion [2]. 

A1: Thank you for your question. We agree with the reviewer that the mixed transfer may benefit from DSI synthetic data. To validate this, we train the mixed transfer using either the DSI synthetic data. The results are shown below for the Aircraft and the Pets dataset.

| Transfer Method    | Syn Method      | Aircraft | Pets |
|--------------------|-----------------|----------|------|
| Vanilla Transfer   | N/A             | 79.7     | 91.3 |
| Mixed Transfer     | Single Template | 70.7     | 82.5 |
| Mixed Transfer     | DSI             | 81.8     | 91.2 |
| Bridged Transfer++ | DSI             | 85.8     | 91.9 |

We find that mixed transfer with DSI inversion indeed improves the performance of mixed transfer with Single Template synthesis. However, when compared to Bridged Transfer++, this method still has lower accuracy. In our view, the mixed transfer can be viewed as a type of data augmentation, which needs careful tuning to not differentiate between original images too much. Yet Bridged Transfer is a type of pre-training that has looser requirements in domain gap compared to data augmentations.

Reviewer comment (Experiments):
There should also be comparisons to vanilla transfer with FC reinit, which is not included in, e.g., Table 2 and Table 6. It seems that this makes a large difference, but it is only tested for the bridged transfer, not for the vanilla transfer.

Author response:
Q3: There should also be comparisons to vanilla transfer with FC reinit, which is not included in, e.g., Table 2 and Table 6. It seems that this makes a large difference, but it is only tested for the bridged transfer, not for the vanilla transfer.

A3: Since the ImageNet dataset has 1000 classes, which is different from the target datasets in transfer learning, we have to reinitialize the classifier in vanilla transfer. In other words, the vanilla transfer is inherently equipped with FC reinit.

Reviewer comment (Presentation):
The circle plots in Fig. 2 and Fig. 8 are not very intuitive to interpret. How are these produced? There is a normalization with vanilla transfer, but how are the axes shifted? For example, in Fig. 2 it seems that axes are also scaled to show mixed transfer on the inner ring, except for the Cars dataset. It would be easier to interpret the results on, e.g., bar plots or similar.

Author response:
Q5: The circle plots in Fig. 2 and Fig. 8 are not very intuitive to interpret. How are these produced? There is a normalization with vanilla transfer, but how are the axes shifted? For example, in Fig. 2 it seems that axes are also scaled to show mixed transfer on the inner ring, except for the Cars dataset. It would be easier to interpret the results on, e.g., bar plots or similar.

A5: Thanks for your comment. We apologize for not mentioning the axis normalization.  For Figure 2, we normalize each axis so that the vanilla transfer is fixed to the 3rd ring. Under this normalization, we scale the axis so that neither the mixed transfer goes inside the 1st ring nor the bridged transfer goes outside the 4th ring. We have changed Fig. 2 to prevent confusion on mixed transfer, where we normalize the mixed transfer to the 1st ring. 

For Figure 8, we normalize each axis so that the vanilla transfer is fixed to the 2nc ring. Then, we scale the axis so that the bridged transfer is fixed to the 4th ring unless the difference is less than 2%. If the difference is less than 2%, we will scale the interval to a 1% accuracy difference. We understand that It would be easier to interpret the results with bar plots, however, it may significantly increase the space of the plots. Therefore, we refrain from doing so.

Reviewer comment (Presentation):
It is a bit unclear what strategy has been used in figures. For example, in Fig. 2 the performance degrades with bridged transfer on DTD and Dogs, so I assume this is not with the bridged transfer++? However, in Fig. 3 and Fig. 9 bridged transfer performs equal or better on all datasets. So, here the method has been changed to the bridged transfer++? Please clarify around this.

Author response:
Q6: It is a bit unclear what strategy has been used in figures. For example, in Fig. 2 the performance degrades with bridged transfer on DTD and Dogs, so I assume this is not with the bridged transfer++? However, in Fig. 3 and Fig. 9 bridged transfer performs equal or better on all datasets. So, here the method has been changed to the bridged transfer++? Please clarify this.

A6: Thanks for your comment. We’d like to clarify that Fig 3 and Fig 9 actually report the training accuracy rather than the test accuracy. We did not employ Bridged Transfer++ in those two figures. We intended to show that Bridge Transfer has better convergence during training, but suffers from lower test accuracy due to overfitting.

Reviewer comment (Evaluation):
The high-quality synthetic images can always bring multi-fold benefits (cost reduction on real data collection, data shortage, privacy, IP, etc), it would be beneficial for understanding this work’s importance by clearly explaining or numerically analyze how this work can relate to these benefits.

Author response:
We quantitatively convert the effect of synthetic data to the effect of real data by studying this question in few-shot learning scenarios. We run the vanilla transfer from 1-shot to 40-shot one by one and match the accuracy of the bridged transfer. Therefore, we can evaluate how much data the vanilla transfer would need to realize the same accuracy as the bridged transfer. The difference between vanilla and bridged transfer is the gains of synthetic data. The results for the Aircraft dataset of ResNet-18 are shown below:

| Accuracy | Bridged Transfer (w. 3k images / cls) | Vanilla Transfer |
|----------|---------------------------------------|------------------|
| 25.1     | 1-shot                                | 6-shot           |
| 45.0     | 4-shot                                | 14-shot          |
| 58.8     | 8-shot                                | 24-shot          |
| 71.4     | 16-shot                               | 40-shot          |


Significantly, our findings reveal that augmenting a dataset with 3,000 synthetic images per class can be as effective as acquiring 24 new real images per class, an impactful insight considering our initial dataset comprises just 16 images per class. From a cost perspective, this is noteworthy: commercial image procurement may range from 1 to 3 dollars per image, whereas leveraging stable diffusion to generate 3,000 images entails a mere $1.50, as documented by AWS's cost analysis (https://aws.amazon.com/blogs/machine-learning/maximize-stable-diffusion-performance-and-lower-inference-costs-with-aws-inferentia2/). This cost efficiency could represent significant savings for researchers and commercial entities alike.

From the standpoint of privacy protection, synthetic image generation offers a substantial advantage. Traditional methods involving web-scraped or manually collected images mandate rigorous examination for personal identifiers and sensitive content, a process that can be both time-consuming and prone to error. In contrast, synthetic data not only circumvents many of these privacy concerns but also enables a mathematical assessment of potential risks. Studies such as [1, 2] have explored the concept of Differentially Private Stable Diffusion (DP-SD), suggesting that models trained on DP-SD-generated data inherit the framework's privacy-preserving attributes, thereby enhancing the overall security of the information.

[1] Lin, Z., Gopi, S., Kulkarni, J., Nori, H. and Yekhanin, S., 2023. Differentially Private Synthetic Data via Foundation Model APIs 1: Images. arXiv preprint arXiv:2305.15560.

[2] Sehwag, V., Panda, A., Pokle, A., Tang, X., Mahloujifar, S., Chiang, M., Kolter, J.Z. and Mittal, P., 2023. Differentially Private Generation of High Fidelity Samples From Diffusion Models.

Reviewer comment (Experiments):
More discussion is expected to better understand the scope and possibility of observed phenomenon, proposed framework, and strategy. For example, beyond the transfer learning paradigms, will this observation also hold in other learning paradigms? Beyond text-to-image generative models, will the idea of bridged transfer also work?

Author response:
Yes, we can apply synthetic images to a randomly initialized model (no real data pre-trained), and then transfer to target datasets. Here, we test the performance on Aircraft and Pets datasets. The results are shown below:

| Method                                | Aircraft | Pets |
|---------------------------------------|----------|------|
| Training from scratch                 | 66.3     | 68.5 |
| Vanilla transfer from ImageNet        | 79.7     | 91.3 |
| Vanilla transfer from Synthetic data  | 83.0     | 84.5 |
| Bridged transfer from Synthetic data  | 85.2     | 91.8 |

It can be found that our method can effectively improve the performance of a randomly initialized model. On the Aircraft dataset, the synthetic data even outperforms the ImageNet pretrained models.

Reviewer comment (Experiments):
Beyond the transfer learning paradigms, will this observation also hold in other learning paradigms?

Author response:
Yes, we can apply synthetic images to a randomly initialized model (no real data pre-trained), and then transfer to target datasets. Here, we test the performance on Aircraft and Pets datasets. The results are shown below:

| Method                                | Aircraft | Pets |
|---------------------------------------|----------|------|
| Training from scratch                 | 66.3     | 68.5 |
| Vanilla transfer from ImageNet        | 79.7     | 91.3 |
| Vanilla transfer from Synthetic data  | 83.0     | 84.5 |
| Bridged transfer from Synthetic data  | 85.2     | 91.8 |

It can be found that our method can effectively improve the performance of a randomly initialized model. On the Aircraft dataset, the synthetic data even outperforms the ImageNet pretrained models.

Reviewer comment (Experiments):
Beyond text-to-image generative models, will the idea of bridged transfer also work?

Author response:
Thanks for the question. Generally, we think bridged transfer will also work on other types of synthetic data, like GAN-based synthetic data. However, given the time limit in the rebuttal period and the synthesis model is a little out of our scope, we'd like to leave it as our future work.

Reviewer comment (Experiments):
It would be interesting to have the approach proposed in paper “IS SYNTHETIC DATA FROM GENERATIVE MODELS READY FOR IMAGE RECOGNITION?” as a baseline (let’s say this baseline as Transfer1). Specifically, it would be interesting to see the performance of Transfer1 on the same setting (i.e. having the same generative text2image model, same evaluation datasets and using the proposed approaches of dataset filtering RF, RG to close the domain gap).

Author response:
A1/2/3: We are grateful for the recommendation to juxtapose our method with Transfer 1. While acknowledging the importance of Transfer 1 in utilizing synthetic data for image recognition, we aim to delineate the distinctions and similarities between our approaches. 
We must emphasize that Transfer 1 utilizes classifier tuning tailored for the CLIP model; conversely, our work concentrates on whole network tuning applicable to a spectrum of models, such as ResNets. It is crucial to recognize the significant differences in pre-training, fine-tuning, and application methods between the models assessed by Transfer 1 and those in our study. For instance, it has been observed that fine-tuning an entire CLIP model may detrimentally impact its robustness and accuracy [1]. Consequently, not all insights from Transfer 1 are directly transferrable to our proposed bridge transfer frameworks, and vice versa—a point we will substantiate through our experiments. A direct comparison between the tuning of CLIP and general models falls beyond the scope of Transfer 1 and our work.

In this study, we delve into transfer learning for general models, as this scenario is not only prevalently employed in practical settings but also merits investigation to comprehend how synthetic data can enhance these applications.

[1] Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., ... & Schmidt, L. (2022). Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7959-7971).

Reviewer comment (Experiments):
It would be interesting to see the performance of Bridged Transfer on pre-trained CLIP models and then compare it with Transfer1 on the same proposed evaluation setting introduced by Transfer1.

Author response:
While it is challenging to draw an apple-to-apple comparison between Transfer 1 and our method, we have endeavored to adapt both for comparison: 1) We employed Stable Diffusion with RF and RG for Transfer 1, based on its official code, and 2) We performed bridged transfer (limited to the classifier) with CLIP. It's important to clarify that this experiment is not about directly contrasting the performance of Transfer 1 and our method in their unaltered states, but rather to understand how they operate within different configurations. Due to the time limit in the rebuttal period, we select Aircraft and Pets datasets to test the performance.

| Model     | Synthesis      | Transfer Method    | Aircraft 1-shot | Aircraft 8-shot | Pets 1-shot  | Pets 8-shot |
|-----------|----------------|--------------------|-----------------|-----------------|--------------|-------------|
| CLIP      | N/A            | Vanilla Transfer   | 20.2            | 32.7            | 86.2         | 87.9        |
| CLIP      | Basic          | Mixed (Transfer 1) | 21.9            | 31.8            | 87.1         | 87.8        |
| CLIP      | Basic          | Bridged Transfer   | 20.9            | 31.7            | 87.5         | 87.8        |
| CLIP      | Real Guidance  | Mixed (Transfer 1) | 22.6            | 33.4            | 87.5         | 89.5        |
| CLIP      | Real Guidance  | Bridged Transfer   | 21.8            | 32.8            | 87.3         | 89.3        |
| ResNet-18 | N/A            | Vanilla Transfer   | 6.5             | 32.2            | 38.5         | 80.4        |
| ResNet-18 | Basic          | Bridged Transfer++ | 25.0            | 58.8            | 54.8         | 87.5        |

Based on the results, we have several common observations with Transfer 1 and several new findings on top of it.

1. In few-shot recognition, Real Guidance synthesis is always better than Basic synthesis, which is also observed in Transfer 1. 

2. However, in 8-shot setting, we found that Basic synthesis + Mixed (Transfer 1) cannot improve the vanilla transfer, this is similar to our paper’s finding that regular synthetic data has a distribution shift and may lead to accuracy decreases in mixed transfer. 

3. The bridged transfer does not always perform better than the mixed transfer using the Transfer 1 framework (CLIP, classifier fine-tuning) which is also discovered by Transfer 1 in Table 7. 
We think the reasons are twofold: (1) Transfer 1 uses classifier tuning of the CLIP model, while our pipeline uses whole network tuning. In our paper, we found that synthetic data may lead to bias in the final classifier layer, therefore, we employ the FC reinitialization in bridged transfer++. In Transfer 1, since only the classifier is allowed to be tuned, it is not surprising that bridged transfer may have lower performance than mixed transfer when only optimizing the classifier. (2) Transfer 1 uses different batch sizes to balance the synthetic data and real data (512 for synthetic data, 32 for real data) so that the network weighs more in real data to prevent distribution shifts. This is why Transfer 1 needs to freeze BN, while we do not need to freeze BN layers in bridged transfer. 
Nonetheless, when we apply our bridged transfer approach to ResNet-18, it yields enhanced accuracy on the challenging fine-grained classification dataset 'Aircraft', despite ResNet-18 being significantly smaller than CLIP. Moreover, Transfer 1 exhibits superior performance on the 'Pets' dataset. These outcomes corroborate our earlier assertion: although Transfer 1 and our method focus on different models and scenarios, they both effectively demonstrate the potential of synthetic data to boost accuracy for their respective settings and tasks.

We hope our response can sufficiently address your concerns.

Reviewer comment (Writing):
It might be helpful for the reader if there was a subsection/paragraph that describes the difference between Bridged Transfer and Transfer1. For example, Bridged Transfer uses Dataset Style Inversion to reduce the domain gap whereas Transfer1 leans on filtering approaches to create a higher quality dataset.

Author response:
A4: Thanks for your suggestion, we have added a section in Appendix E showing the difference between our work and Transfer 1. In the previous experiments, we conducted experiments under the Transfer 1 framework and we believe these two papers share similar observations in leveraging the synthetic data. In addition to Transfer 1, our work studied how to more effectively utilize synthetic in the general transfer learning framework. We find that synthetic data promotes feature representation learning but may bias the FC layer. We will add these inspiring discussions to the main paper when we have more dataset results.

Reviewer comment (Presentation):
It might improve the readability of the paper if Bridged Transfer + FC Reinit & mixup is also included in Figure 2.

Author response:
A6: Thank you for your suggestion. We have added the Birdged Transfer++ (Bridged Transfer+FC Reinit & mixup) results into Fig 2.

Reviewer comment (Reproducibility):
How much training time/wall-clock time and memory resources does it take for Dataset Style Inversion (DSI) for each dataset to optimize the S* token? For example, it’s mentioned that it takes 20k training iterations for the Aircraft dataset.

Author response:
A7: For all datasets, we optimize the token for 20k training iterations (3 hours on one A100 GPU). Hence, the training wall-clock time is the same across different datasets. Note that when compared to class-wise textual inversion, the speed-up ratio will be correlated with a number of classes in that dataset.

Reviewer comment (Experiments):
Are there any qualitative reasons that for datasets like DTD and Pets, the improvement is kind of limited? Does the synthetic dataset have a higher domain gap or is this a fine-grained task for which text-to-image models can’t generate enough informative samples? Also, it would be interesting to see the impact of synthetic data volume increase for these classes similar to Figure 5(a).

Author response:
A8: During our experiments, we found that there could be two reasons that make DTD and Pets hard to improve with synthetic data. (1) The domain gap between synthetic and real data is too huge, or the label cannot describe the class very well. For example, in the DTD dataset, the label name is the adjective of each texture, e.g., “banded, braided”, which can correspond to lots of different banded textures and braided textures. In this case, we have to use DSI to mitigate the gap. (2) The second reason is the original datasets contain enough data samples for recognition, adding much more synthetic data might lead to severe overfitting, e.g. Pets. In this case, we can observe much more improvements in few-shot transfer, while moderate improvements in full-shot transfer.

Reviewer comment (Reproducibility):
In Table 3, what does Single Template correspond to (which Bridged Transfer version and what’s the network architecture)?

Author response:
A9/10: Thank you for your question. We apologize for the unclear description and wrong numbers in Table 3. In Table 3, the Single Template uses ResNet-18 as the architecture and both Single Template and DSI use Bridged Transfer ++, which is Bridged Transfer with FC Reinit and mixup. We revisited our experiments and found that we forgot to add a mixup loss function when running DTD datasets. We have changed the accuracy to new ones.