Paper: Real-Fake: Effective Training Data Synthesis Through Distribution Matching (ICLR-2024)

Reviewer comment (Novelty):
The overall idea of Maximum Mean Discrepancy (MMD) for learning data distribution is not novel, which has been explored before, like [1].

Author response:
**A1:** We kindly note the main idea of our work is not leveraging MMD loss as a standalone technique for learning data distribution but to formulate the training data synthesis as a distribution matching problem (Eq. 3 and Eq. 4). Our specific implementation of this framework on latent diffusion model involves using distribution matching loss (MMD) **along with latent prior initialization and conditional visual guidance**, integrated in Stable Diffusion for synthesizing informative training data. The significant performance improvements brought by our method empirically indicate the effectiveness of the proposed framework. We have also **validated the importance of all three components** of our method. We have revised our manuscript to emphasize and clarify our main idea.

Reviewer comment (Novelty):
The derived MMD loss based on diffusion model (Eq. (7)) is the same as the simplified diffusion model loss (Ho et al. 2020). This means the proposed loss does not bring anything new for the current diffusion model.

Author response:
**A2:** We would like to clarify that our derived MMD loss is **not** identical to the diffusion model loss presented by Ho et al. (2020). Their simplified diffusion model loss is a mean square error calculated on top of the predicted noise and added noise whereas the MMD loss, Eq. (7), is the L2-norm of the averaged difference between the predicted noise and added noise. 


$$
L_{DM} := ||\frac{1}{|N|} \sum_{i=1}^{|N|} (\epsilon - \epsilon_{\theta}(x_t, t))||^2 \leq 
\frac{1}{|N|} \sum_{i=1}^{|N|} ||(\epsilon - \epsilon_{\theta}(x_t, t))||^2 = L_{Diffusion}.
$$






**The diffusion model loss is an upper bound of the MMD loss** by Jensen’s inequality. The derived MMD loss is thus a tighter bound on the distribution discrepancy between real and synthetic data under the MMD measure. We have revised the relevant sections to better clarify this distinction.




[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems 33:6840–6851, 2020.

Reviewer comment (Evaluation):
The effectiveness of the proposed MMD loss (Eq. (7)) is doubtful. From Table 2, the performance gain brought by MMD is not significant compared to finetuning only.

Author response:
**A3:** As mentioned above, we show that the diffusion loss (Ho et al. 2020) is an upper bound of the MMD loss. Therefore, finetuning with the diffusion model loss still aligns with our distribution matching framework for training data synthesis. Our motivation (Section 3.1) to further include MMD loss rather than simply diffusion model loss is due to the known looseness problem of diffusion loss (Kingma et al. 2021). The results in Table 2 also show that the extra MMD loss can bring notable improvement in some settings, which are selected in the following table for illustration. MMD loss leads to consistent improvements of 0.6/1.1%, and 1.0/0.7%  on ImageNette/ImageNet100, on top of vanilla diffusion loss, accordingly. We revised our manuscript to clarify it and provided further analysis in Appendix G.5.




| Latent Prior | Visual Guidance | Distribution Matching | Finetune | ImageNette | ImageNet100 |
|--------------|-----------------|-----------------------|----------|------------|-------------|
|              | ✓               |                       | ✓        | 82.3       | 74.0        |
|              | ✓               | ✓                     | ✓        | 82.9       | 75.1        |
| -  |         -        |          -             |     -     | $\Delta$ + 0.6 | $\Delta$ + 1.1 |
| ✓            | ✓               |                       | ✓        | 89.5       | 79.3        |
| ✓            | ✓               | ✓                     | ✓        | 90.5       | 80.0        |
|     -             |        -         |             -          |     -     | $\Delta$ + 1.0 | $\Delta$+ 0.7 |




[2] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances
in neural information processing systems, 34:21696–21707, 2021.

Reviewer comment (Writing):
$\epsilon_0$ is not defined in Eq. (7).

Author response:
**A5:** Thank you for pointing out this problem. $\mathbf{\epsilon_{0}}$ represents the added noise $\mathbf{\epsilon}$ in the forward process of the diffusion model at each time step. We have corrected it in Eq.(7) in Section 3.1.

Reviewer comment (Experiments):
For the latent prior only in Table 2, does it mean simply adopting the proposed prior for the pretrained Stable Diffusion to synthesiz the target images?

Author response:
**A7:** Yes, the “Latent Prior” experiment setting only refers to adopting the prior for pre-trained Stable Diffusion without finetuning. We clarified it in the updated Table 2 and also Appendix G.5 for clarity.

Reviewer comment (Experiments):
The improvement over Imagen seems limited. The comparisons between the proposed method and Imagen are limited to ImageNet-1K, making the superiority unclear on other datasets.

Author response:
**A1:** Thank you for raising this point regarding the comparison to Imagen results (Azizi et al. 2023). We would like to kindly highlight several key aspects to consider Imagen (Saharia et al. 2022) in this context. 

Firstly, Imagen is a **closed-source** model trained on **private data**, which makes it impossible for us to obtain more experiment results of Imagen to compare except their reported ones on ImageNet-1K dataset in (Azizi et al. 2023). 

Secondly, the Imagen, trained on **private data**, involves a **more powerful generative model than Stable Diffusion** and is **fine-tuned with full parameters** on ImageNet-1K, which is computationally unaffordable by general researchers. In contrast, our approach leverages the open-source Stable Diffusion model coupled with a low-budget tuning (LoRA) and synthesis framework. 

Thirdly, despite the resource constraints, our method still achieves **notable performance improvements (1.7%)** on the very challenging large-scale dataset ImageNet-1K, underlining the effectiveness of our techniques. 

Last but not least, with a fair comparison with other baselines using the same generative model backbone, Stable Diffusion, our synthesis framework brings significant (10% ~ 35%) performance improvements on other datasets. 

[1] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479–36494, 2022

[2] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08466, 2023

Reviewer comment (Experiments):
This work utilizes the prompt in format of "photo of [classname], [Image Caption], [Intra-class Visual Guidance]". Are there any other comparative methods that also use the same "[Image Caption]"? Whether the improvement is mainly from "[Image Caption]" instead of "[Intra-class Visual Guidance]"?

Author response:
**A2:** Thanks for the question. Actually, the CiP (Lei et al. 2023) method generates images based on the "[Image Caption]". Specifically, they use the template “A photo of {class name}, {image caption}” to generate prompts. We provided a comparison to their results in Table 1 in the submission. Our method outperforms the baseline by 10%, 18%, and 17% on ImageNette, ImageNet-100 and ImageNet-1k datasets respectively. 
Furthermore, our ablation study in Table 2 has proven the importance of "[Intra-class Visual Guidance]" component. We further clarify it by showing the following selected results from Table 2. As shown, "[Intra-class Visual Guidance]" brings 1.5% and 0.7% performance improvements on ImageNette and ImageNet100 respectively, when cooperating with “Distribution Matching” components. To clarify it, we revised our manuscript and provided further analysis in Appendix G.5.

| Latent Prior | Visual Guidance | Distribution Matching | Finetune | ImageNette | ImageNet100 |
|--------------|-----------------|-----------------------|----------|------------|-------------|
|              |                 |                       | ✓        | 80.8       | 73.3        |
|              | ✓               |                       | ✓        | 82.3       | 74.0        |
|      -        |        -         |             -          |   -       |$\Delta$ + 1.5 |$\Delta$ +0.7 |

[3] Shiye Lei, Hao Chen, Sen Zhang, Bo Zhao, and Dacheng Tao. Image captions are natural prompts for text-to-image models. arXiv preprint arXiv:2307.08526, 2023.

Reviewer comment (Experiments):
It seems there is only 1 run on all the experiments. It is beneficial to include more than 1 repeats for the experiments to exclude the confounders;

Author response:
**Q1: More rounds of experiments**


**A2:** Thank you for reminding us of this problem. We have included additional multi-runs for ImageNet (IN-10, 1N-100, 1N-1K) experiments as shown in the table below, where we find the standard deviation of the three-seed experiment on all scales stays within a small range. This validates the stability of our method. We have added these results in our revised manuscript in Appendix G.6. 




Multi-Seed Experiment Results: Top-1 Classification Accuracy across various datasets (IN-10, IN-100, IN-1k) over multiple runs. *Average* shows the mean and standard deviation of multiple run results.




|       | IN-10      | IN-100     | IN-1k      |
|-------|------------|------------|------------|
| Run 1 | 90.5       | 80.0       | 70.9       |
| Run 2 | 91.2       | 79.9       | 70.6       |
| Run 3 | 90.7       | 80.2       | 70.9       |
| Average | 90.8 ± 0.29 | 80.0 ± 0.12 | 70.8 ± 0.14 |

Reviewer comment (Evaluation):
Details of privacy analysis are missing.

Author response:
**A1:**  Thank you for pointing out the insufficiency of details. We addressed this by updating a comprehensive privacy analysis details in Appendix G.4 in the revised manuscript.

Reviewer comment (Presentation):
Why are so many of the cells in table 1 empty?

Author response:
**A2:** We would like to kindly note a few reasons for the empty entries in Table 1. Firstly, direct evaluations against the Imagen model on diverse datasets are challenging due to its closed-source nature. Secondly, relatively older baselines in Table 1 have been surpassed by more advanced methods, making them less competitive for current benchmarks. Therefore, we focused our evaluations on the large-scale ImageNet-1K dataset, considering it the most representative and classical benchmark for evaluation in this domain. Also, regarding the benchmarking on other datasets, we compare with the current state-of-the-art *FakeIt*. It delivers a fair comparison given the same generative model backbone and demonstrates significant performance improvement.