Paper: CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling (ICLR-2024)

Reviewer comment (Experiments):
In Appendix G, this paper provides sampling and evaluation details, shown in Table 12. The sampling hyperparameters is set deliberately for these experiments. To my knowledge, the training setting is mostly fixed for the same dataset even though it is an ablation study. But, as shown in Table 12, this rule is broken. For example, the setting of ImageNet 256 in Table 1 is different from the setting of ImageNet 256 in Table 6a. How do you set the sampling hyperparameters?

Author response:
### **Ablations and hyperparameter selection**

The reviewer’s question brings up an interesting point about the hyperparameter settings in the present work.  The settings described in our paper were determined heuristically based on experimentation with the individual datasets.  It is important to note that, as our method is a *sampling* method, no hyperparameter choices need to be made at training time, and experimentation to help select hyperparameter settings is relatively inexpensive.  We also found that our method is quite robust to a range of settings.  When performing an ablation study on individual hyperparameters, the other hyperparameters must be kept fixed at predetermined values.  These values were selected to best illustrate the effects of exploring the range of the *ablated* hyperparameters, which in some cases differ from the settings used elsewhere in the paper.  However, we agree with the reviewer about the importance of maintaining a direct comparison between experiments, and we report below additional ablations with the fixed hyperparameters chosen to match those in Table 1.  These results comport with the conclusions of the original ablation study while, we hope, better connecting with the earlier experiments.

**New ablation on $\tau_1$**

| **$\tau_1$** | **FID** | **Recall** |
| :---: | :---: | :---: |
| 0.2 | 47.21 | 0.81 |
| 0.5 | 9.47 | 0.62 |
| 0.9 | 20.64 | 0.33 |

**New ablation on $\psi$**

| $\psi$ | FID | Recall |
| :---: | :---: | :---: |
| 0.0 | 111.39 | 0.81 |
| 0.5 | 11.55 | 0.60 |
| 1.0 | 9.47 | 0.62 |


**New ablation on $s$ with and without rescaling**

| $s$ | FID ($\psi=0$) | Recall ($\psi=0$) | FID ($\psi=1$) | Recall ($\psi=1$) |
| :---: | :---: | :---: | :---: | :---: |
| 0.025 | 16.88 | 0.43 | 12.94 | 0.50 |
| 0.05 | 14.96 | 0.48 | 12.69 | 0.52 |
| 0.10 | 14.13 | 0.63 | 10.18 | 0.60 |
| 0.15 | 111.39 | 0.81 | 9.47 | 0.62 |

Reviewer comment (Evaluation):
Empirically, with high guidance weights, the diffusion sampler can produce unnatural images and sometimes even diverges. There is a concern about the proposed method: is it easy to produce bad images?

Author response:
### **High guidance values and producing bad images**

The short answer is that, within a wide range of settings, it is quite difficult to produce bad images using our method.  None of the images in our paper were cherry-picked or otherwise curated.  However, it is important to note that CADS was developed to solve the issue of limited image diversity when using high guidance scales and was not designed to protect against the deleterious effects of ultra-high guidance scales.  As our method is built on top of CFG, we suspect that it is similarly susceptible to these effects.  However, our method provides practitioners with a simple tool for avoiding the diversity limitations of CFG, which we argue increases the range of scales that can be used in practice versus standard methods.

Reviewer comment (Writing):
“Adaptive” may not be an appropriate term. To my knowledge, ‘adaptive’ is used for a method of which parameters are dynamically changed depending on a given input value [1]. Here, $z_t$ is the input I was expecting rather than $t$ since $t$ is a value within a fixed time window.

Author response:
### **Adaptive CFG terminology**

We thank the reviewer for pointing this out.  We have changed the terminology to *dynamic* CFG to avoid confusion with adaptive normalization scenarios that are data dependent.

Reviewer comment (Experiments):
There are a lot of sampling methods including DDIM [2] while only the original DDPM sampler is used to compare. Considering the fact that DDIM is a more widely used sampling method than the original DDPM sampling method, additional comparison with DDIM is needed.

Author response:
### **More experiments with DDIM and other samplers**

We appreciate the reviewer’s point regarding the samplers one has to choose from.  Table 3 in our submission reports our experiments with other samplers based on the class-conditional ImageNet model. In our revised manuscript, we have added a section in the appendix that extends the experiments reported in Table 1 with additional samplers. Specifically, we compared our results on DDIM for both pose-to-image models and reported additional experiments using DPM++ [1] and PNDM [2] samplers with StableDiffusion. The table reporting these additional experiments is shown below.  In addition, our response to Reviewer fxon also includes an experiment of performance-versus-NFE based on DPM++ and DDIM samplers. We observed that across all tested scenarios and samplers, CADS consistently improves the diversity of output over that of the base sampler while maintaining image quality.

| **Dataset** | **Sampler** | **FID** ↓ | **Precision** ↑ | **Recall** ↑ | **MSS** ↓ | **Vendi Score** ↑ |
|:---------|:---------|:---------:|:---------:|:---------:|:---------:|:---------:|
| DeepFashion | DDIM | 14.60 | **0.93** | 0.02 | 0.80 | 1.03 |
| | CADS (Ours) | **7.90** | 0.76 | **0.49** | **0.35** | **2.30** |
| SHHQ | DDIM | 26.27 | **0.70** | 0.15 | 0.57 | 1.27 |
| | CADS (Ours) | **15.14** | 0.61 | **0.46** | **0.36** | **2.06** |
| StableDiffusion | DPM++ | 45.70 | **0.70** | 0.29 | 0.19 | 5.30 |
| | CADS (Ours) | **40.35** | 0.65 | **0.42** | **0.13** | **6.93** |
| | PNDM | 45.76 | **0.68** | 0.28 | 0.19 | 5.36 |
| | CADS (Ours) | **41.37** | 0.65 | **0.38** | **0.13** | **6.83** |

Reviewer comment (Experiments):
The definition of “diversity” is provided right under Section 3. What is the meaning of “random seed”? Does it mean the seed of the randomness (e.g., random.seed(1) in Python) or different $x_T$? To me, comparing samples from different seed is less clear than comparing different samples from the fixed seed. For example, we have two $x_T$ sampled from the standard normal, i.e., $x_T^{(0)}$, $x_T^{(1)}$. Let’s say the regular sampling method is $f$ (e.g., DDPM), the proposed sampling method is $g$, and an arbitrary metric of the semantic distance between two images is $h$. To me, the effect of the proposed method would be understandable if $h(f(x_T^{(0)}), f(x_T^{(1)})) < h(g(x_T^{(0)}), g(x_T^{(1)}))$ because we can consider that more modes in the real distribution are covered by the standard gaussian. However, if the noise samples $x_T^{(0)}$, $x_T^{(1)}$ are sampled from different random seed respectively, maybe it's a trivial issue, but it sounds somehow unclear to me. Further clarifications on this point are needed.

Author response:
### **Confusion about the random seed**

We thank the reviewer for catching this ambiguous terminology. We have changed the term “random seed” to “initial random sample” in our working draft, and we will pay additional attention to this section to ensure clarity when finalizing the manuscript.  The idea we wish to get across is that *low* diversity means that many different initial random samples are converted by the model to a small number of stereotyped outputs, while *high* diversity means that many different initial random samples are converted by the model to *just as many* distinct outputs.  However, when comparing CADS versus non-CADS sampling, we do fix the *local* random seed to guarantee that each method sees the same set of initial random samples, $x_T$.

Reviewer comment (Evaluation):
What is the justification for the higher IS of DDPM across most of $w_{\text{CFG}}$ in Fig. 5?

Author response:
### **Higher Inception Score (IS) values for DDPM**

As argued in [3], IS “does not reward covering the whole distribution or capturing diversity within a class.”  Indeed, one can construct a pathological scenario in which a very small set of exemplar images (say, one or a few for each class) can maximize IS while minimizing diversity by design.  Nevertheless, IS is still a popular measure of generative modeling output, which we included for completeness.  As IS is also a reasonable measure of image quality, it is worth noting that it consistently *increases* with $w_{\text{CFG}}$ in Figure 5.  That it is somewhat below DDPM may speak more to the nuances of IS than to the quality of output when using CADS.  Indeed, we report a similar phenomenon in Appendix F, where the IS value computed on *real* data from ImageNet was significantly lower than the values obtained for DDPM samples. As sampling with higher guidance tends to maximize the class probability $p(y|x)$, the IS of DDPM outputs is possibly artificially inflated.

Reviewer comment (Theory):
There is room for enhancement in articulating the motivation behind the methodology. While the issue of sample diversity has been clearly defined, the rationale for the solution could benefit from a stronger motivation. Ideally, the approach should be presented as a natural derivation from the first principle, rather than retroactively justified through theoretical exposition.

Author response:
### **Motivating the method**
We appreciate the reviewer’s suggestion to better motivate the method ahead of its introduction and to move some of the theoretical discussion from the appendix to the main paper.  To the best of our knowledge, accepted papers at ICLR do not get an extra page allowance, so we have to assume for the moment that we are working with limited space.  However, we agree with the reviewer that the placement of the method’s motivation is important, so we have added a paragraph above Section 3.1 in our working draft to provide readers a better intuition behind the rationale of the solution before introducing the solution itself. Please find the corresponding paragraph below:

> We conjecture that the low-diversity issue is due to a highly peaked conditional distribution, particularly at higher guidance scales, which leads the majority of samples toward certain modes during inference. One way to address this issue is by smoothing the conditional distribution with decreasing Gaussian noise, which breaks and then gradually restores the statistical dependency on the conditioning signal during inference. Next, we introduce a novel sampling technique that integrates this intuition into the inference process. The superiority of this method over the above approaches is depicted in Figure 2.c.

Reviewer comment (Experiments):
Notably, the experiments in this work utilize relatively large sampling steps (>= 100 NFEs). Will the proposed sampler deteriorate at a limited number of function evaluations? How does the conditional generation under CADS change along different sampler steps/NFEs?

Author response:
### **Behavior of CADS VS NFEs**
This is an excellent question. We have added a section to the appendix of the paper that explores the effectiveness of CADS with respect to different numbers of sampling steps (NFEs). The behavior of CADS vs NFEs is similar to the base sampler, and a consistent gap exists between sampling with and without condition annealing across different NFEs. Additionally, please note that as DDPM does not perform well when using low NFEs, we switched to DPM++ [2] and DDIM [3] to perform this comparison. Please find the results of this experiment in the tables below.

|  | DPM++ w/ CADS | DPM++ w/ CADS | DPM++ | DPM++ |
| --- | :---: | :---: | :---: | :---: |
| **NFEs** | **FID** | **Recall** | **FID** | **Recall** |
| 25 | 7.73 | 0.67 | 17.86 | 0.37 |
| 50 | 8.56 | 0.64 | 18.64 | 0.36 |
| 75 | 8.95 | 0.63 | 18.81 | 0.36 |
| 100 | 9.63 | 0.61 | 18.90 | 0.36 |

|  | DDIM w/ CADS | DDIM w/ CADS | DDIM | DDIM |
| --- | :---: | :---: | :---: | :---: |
| **NFEs** | **FID** | **Recall** | **FID** | **Recall** |
| 25 | 8.45 | 0.64 | 18.67 | 0.36 |
| 50 | 9.80 | 0.59 | 18.91 | 0.36 |
| 75 | 10.08 | 0.57 | 18.92 | 0.35 |
| 100 | 10.16 | 0.58 | 18.96 | 0.35 |

---
**References**

[1] Saharia, Chitwan, et al. "Image super-resolution via iterative refinement (2021)." *arXiv preprint arXiv:2104.07636* (2021).

[2] Lu, Cheng, et al. "Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models." *arXiv preprint arXiv:2211.01095* (2022).

[3] Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoising Diffusion Implicit Models." *International Conference on Learning Representations*. 2021.

We would like to thank the reviewer once again for the helpful and supportive comments. 

The NFE experiment we reported is based on class-conditional ImageNet generation using the DiT-XL/2 model. The reviewer’s intuition is indeed correct that reducing NFEs further would result in a U-shaped curve in the FID. We have performed follow-up tests at lower NFEs and have observed the U-shaped behavior that the reviewer suggested. We will report these results in the final version of the paper and would also be happy to extend this experiment beyond the ImageNet generation task.