Paper: Elucidating the design space of classifier-guided diffusion generation (ICLR-2024)

Reviewer comment (Experiments):
The paper focuses on the classic class-conditional diffusion generation, which is a simple case in the text-to-image generation. How about the performance of the proposed methods for the general case with text prompts as conditions? In particular, in section 5.4, the authors also provide a simple analysis of this case with the CLIP model. However, the results are not good.

Author response:
Thanks for the question. We agree that the more general text prompt condition is very important. The class-conditional setting, although simple, does provide a solid test bed for different kinds of guidance methods, with well-accepted quantitative measurements. 
If one method is not good at class-guided generation, we wouldn't expect it to excel at the more complicated text prompts. 
Therefore, we think our evaluations on the more typical class-guided generation case are valuable and the findings can potentially be generalized to more general settings. 


Nevertheless, we have explored the CLIP-guided generation task. 
In the added Figure 6 of our revised paper, we present a comparison of CLIP scores between the original guidance schedule and our proposed schedule. Notably, our updated schedule consistently achieves significantly higher CLIP scores throughout the sampling process. The objective of our CLIP-guided sampling approach is to showcase the capability of a traditional off-the-shelf CLIP model in guiding the ImageNet-trained diffusion model (ADM [Dhariwal et al. 2021]). By leveraging this guidance, our model generates images that surpass the limitations of the ImageNet dataset and align with the provided prompt.

We aim to investigate more complicated discriminative models such as BLIP-VQA, as well as discriminative models in other modalities as guidance in the future.

Reviewer comment (Evaluation):
Human-level metrics should be involved for clearer comparisons. It is well-known that some quantitative metrics like FID, may be problematic in some cases. In particular, the FID metric, used in ablation analysis in section 4 and experiments in section 5, cannot measure conditional adherence which is important for conditional generation. CLIP score should also be considered to evaluate the performance.

Author response:
Given the subjectivity of human-level metrics, we primarily focus on utilizing quantitative metrics to evaluate the performance of the classifier-guided diffusion (including our extensive experiments in evaluating the classifier-guided diffusion with quantitative metrics across diverse diffusion architecture: DDPM, EDM, DiT). For a more comprehensive evaluation, we add more quantitative metrics to the experiments. There are some values missing because the authors did not provide them. 


Table 1: ImageNet 64x64 evaluation: 250 steps of DDPM sampling results.  
|                    |   IS   |   FID  |   sFID  | Precision | Recall 
|---|---|---|---|---|---
| ADM                |   -    |  2.61  | 3.77  | 0.73  |  0.63 
| ADM-G              |   -    |  2.07  | 4.29  | 0.74  | 0.63  
| Resnet101-guided(Ours)   |  70.52 |  1.72  | 3.36  | 0.76  | 0.60 

Table 2: ImageNet 128x128 evaluation: 250 steps of DDPM sampling results.  
|                 |   IS   |   FID  |  sFID | Precision| Recall 
|---|---|---|---|---|---
| ADM             |    -   |  5.91  | 5.09  | 0.70  | 0.65 
| ADM-G           | 182.69 |  2.98  | 5.09  | 0.78  | 0.59  
| CFG             | 158.47 |  2.43  | -     |   -   |  -    
| Resnet101-guided(Ours)| 187.83 | 2.19   | 4.53  | 0.79 | 0.58 

Table 3: ImageNet 128x128 evaluation: 25 steps of DDIM sampling results.  
|                 |   IS   |   FID  |  sFID | Precision| Recall 
|---|---|---|---|---|---
| ADM-G           |   -    |  5.98  | 7.04  | 0.78  | 0.51 
| Resnet101-guided(Ours)|  139.88|  5.72  |  5.56 | 0.73  |  0.55 


In the added Figure 6 of our revised paper, we present a comparison of CLIP scores between the original guidance schedule and our proposed schedule. Notably, our updated schedule consistently yields significantly higher CLIP scores throughout the sampling process.

Reviewer comment (Evaluation):
How about the performance in terms of CLIP score and Human-level metrics?

Author response:
Given the subjectivity of human-level metrics, we primarily focus on utilizing quantitative metrics to evaluate the performance of the classifier-guided diffusion (including our extensive experiments in evaluating the classifier-guided diffusion with quantitative metrics across diverse diffusion architecture: DDPM, EDM, DiT). For a more comprehensive evaluation, we add more quantitative metrics to the experiments. There are some values missing because the authors did not provide them. 


Table 1: ImageNet 64x64 evaluation: 250 steps of DDPM sampling results.  
|                    |   IS   |   FID  |   sFID  | Precision | Recall 
|---|---|---|---|---|---
| ADM                |   -    |  2.61  | 3.77  | 0.73  |  0.63 
| ADM-G              |   -    |  2.07  | 4.29  | 0.74  | 0.63  
| Resnet101-guided(Ours)   |  70.52 |  1.72  | 3.36  | 0.76  | 0.60 

Table 2: ImageNet 128x128 evaluation: 250 steps of DDPM sampling results.  
|                 |   IS   |   FID  |  sFID | Precision| Recall 
|---|---|---|---|---|---
| ADM             |    -   |  5.91  | 5.09  | 0.70  | 0.65 
| ADM-G           | 182.69 |  2.98  | 5.09  | 0.78  | 0.59  
| CFG             | 158.47 |  2.43  | -     |   -   |  -    
| Resnet101-guided(Ours)| 187.83 | 2.19   | 4.53  | 0.79 | 0.58 

Table 3: ImageNet 128x128 evaluation: 25 steps of DDIM sampling results.  
|                 |   IS   |   FID  |  sFID | Precision| Recall 
|---|---|---|---|---|---
| ADM-G           |   -    |  5.98  | 7.04  | 0.78  | 0.51 
| Resnet101-guided(Ours)|  139.88|  5.72  |  5.56 | 0.73  |  0.55 


In the added Figure 6 of our revised paper, we present a comparison of CLIP scores between the original guidance schedule and our proposed schedule. Notably, our updated schedule consistently yields significantly higher CLIP scores throughout the sampling process.

Reviewer comment (Experiments):
How about the performance for the conditional generation with the general text prompts?

Author response:
Thanks for the question. We agree that the more general text prompt condition is very important. The class-conditional setting, although simple, does provide a solid test bed for different kinds of guidance methods, with well-accepted quantitative measurements. 
If one method is not good at class-guided generation, we wouldn't expect it to excel at the more complicated text prompts. 
Therefore, we think our evaluations on the more typical class-guided generation case are valuable and the findings can potentially be generalized to more general settings. 


Nevertheless, we have explored the CLIP-guided generation task. 
In the added Figure 6 of our revised paper, we present a comparison of CLIP scores between the original guidance schedule and our proposed schedule. Notably, our updated schedule consistently achieves significantly higher CLIP scores throughout the sampling process. The objective of our CLIP-guided sampling approach is to showcase the capability of a traditional off-the-shelf CLIP model in guiding the ImageNet-trained diffusion model (ADM [Dhariwal et al. 2021]). By leveraging this guidance, our model generates images that surpass the limitations of the ImageNet dataset and align with the provided prompt.

We aim to investigate more complicated discriminative models such as BLIP-VQA, as well as discriminative models in other modalities as guidance in the future.

Reviewer comment (Evaluation):
Please provide more discussions about Table 4 and Figure 4.

Author response:
In Figure 4 of our paper, the random seed was fixed to ensure a consistent and strict comparison, thereby minimizing dramatic changes. However, it is worth noting that some significant differences still exist between the two settings. For example, in the left-hand side figure representing the traditional guidance setting, the generated dog gradually loses the depiction of its feet from time 50 to 0. In contrast, in the right-hand side figure where the joint guidance was amplified, the dog's feet remain clear, and there is a noticeable enrichment of environmental details.

Reviewer comment (Evaluation):
The comparison is not so comprehensive. Since the main baseline should be the noise-aware classifier, the work provides little comparison in Table 5. More metrics such as IS, sFID, Precision and Recall should be included.

Author response:
In our research, we aim to demonstrate the potential of the more accessible traditional ResNet classifier and prove that even without specifically noise-finetuned, the off-the-shelf classifier can guide the diffusion to generate high-quality sampling: FID: 2.19 in ImageNet128x128) not only largely surpass the noise-aware fine-tuned classifier guided diffusion [Dhariwal et al. 2021] (FID: 2.97) but also the classifier-free trained diffusion (FID: 2.43) [Ho et al. 2022].

Due to the computation resource and time limits, we prioritize testing the ImageNet64x64 setting and we plan to include the ImageNet 256x256 results in our future research to provide a comprehensive analysis across different image resolutions. There are some values missing because the authors did not provide them.

Table 4: ImageNet 64x64 evaluation: 250 steps of DDPM sampling results.  
|                    |   IS   |   FID  |   sFID  | Precision | Recall |
|---|---|---|---|---|---
| ADM                |   -    |  2.61  | 3.77  | 0.73  |  0.63
| ADM-G              |   -    |  2.07  | 4.29  | 0.74  | 0.63  
| Resnet101-guided (Ours)   |  70.52 |  1.72  | 3.36  | 0.76  | 0.60 

Table 5: ImageNet 128x128 evaluation: 250 steps of DDPM sampling results.  
|                 |   IS   |   FID  |  sFID | Precision| Recall 
|---|---|---|---|---|---
| ADM             |    -   |  5.91  | 5.09  | 0.70  | 0.65 
| ADM-G           | 182.69 |  2.97  | 5.09  | 0.78  | 0.59 
| Resnet101-guided (Ours) | 187.83 | 2.19   | 4.53  | 0.79 | 0.58

Reviewer comment (Experiments):
Does off-the-shelf classifier guidance provide conditional information for the unconditional diffusion model? In my belief, the main objective of the guidance should be providing conditional information for the unconditional diffusion model rather than just improving the generated image quality. In Table 5, the Diffusion model on ImageNet128x128 is conditionally trained. Thus, the conditional information from an off-the-shelf classifier is not important. Results for combining off-the-shelf classifiers with unconditional models should be included.

Author response:
According to the findings presented in Table 4 of ADM [Dhariwal et al. 2021], classifier-guided unconditional diffusion falls short in terms of performance (FID: 12.0) compared to classifier-guided conditional diffusion (FID: 4.59). Consequently, the subsequent experiments in Table 5 of ADM [Dhariwal et al. 2021] specifically focus on classifier-guided conditional diffusion. 
To ensure a fair comparison with SOTA results, we have adopted the guided conditional diffusion setting in this study. 

Additionally, DiT is trained in a classifier-free manner, which encompasses both class-conditional and unconditional conditions. Table 9 of our revised paper demonstrates that off-the-shelf classifier guidance can also be employed in classifier-free diffusion.
However, we acknowledge that there are further settings and configurations to explore, which will be addressed in future research endeavors.

Reviewer comment (Theory):
There is a fatal gap in the modeling part in Algorithm 1, in the sampling equation $\hat{x}_{t-1} \sim \mathcal{N}(\mu + \gamma _t g, \sigma _t)$. In the original paper by Dhariwal and Nichol (2021), the sampling resulted from the $\log(p _{ \theta }(x_t|x _{t+1}) p _{\phi}(y|x_t))$. However, given the structure of Algorithm1, this equation is no longer valid since the gradient is taken regarding $x _0 (t)$ instead of $x_t$. In order to apply the same sampling process, even when $x_0(t)$ is forward through the classifier, the gradient should be taken regarding $x_t$.

Author response:
Here, $\hat{x}_0(t)$ refers to the denoised sample $\hat{x}_t$, denoted as $\hat{x}_0(t)$ = $(\hat{x}_t - (\sqrt{1-\alpha_t})$ $\epsilon \theta(\hat{x}_t,t))$ $ / \sqrt{\alpha_t}$. Ideally, the classifier gradient guides the diffusion denoising process towards the target image. The sampling trajectory from  $\hat{x}_0(T)$ to $\hat{x}_0(0)$ may not be fundamentally different from that from $\hat{x}_T$ to $\hat{x}_0$. 

Practically, we think that for off-the-shelf classifiers, choosing denoised $\hat{x}_0(t)$ as inputs may be more compatible since they are not specifically fine-tuned on noisy data. 
This choice is used in [Dhariwal et al. 2021] but the quantitative comparison between $\hat{x}_0(t)$ and $x_t$ is missing. 
In Table 2 of our revised paper, we select $\hat{x}_0(t)$ as the input to the classifier due to its superior guidance performance. 
In the ablation study table of classifier input types, we generate 10,000 samples from the 128x128 ImageNet dataset using ADM [Dhariwal et al. 2021], utilizing 250 DDPM steps.

Table: Classifier input types ablation study: 
| Inputs |   $\hat{x}_t$   | $\hat{x}_0(t)$ 
|---|---|---
| FID    |  8.61  | 7.17(16\%)     


To provide clarification of the off-the-shelf guidance in the DDIM sampling setting, we have included the 25 steps of DDIM guided-sampling experiments in the table below, it demonstrates that our off-the-shelf classifier guidance also outperforms the fine-tuned classifier-guided ADM-G [Dhariwal et al. 2021] in DDIM. 

Table 6: ImageNet 128x128 evaluation: 25 steps of DDIM sampling results.  
|                 |   IS   |   FID  |  sFID | Precision| Recall 
|---|---|---|---|---|---
| ADM-G           |   -    |  5.98  | 7.04  | 0.78  | 0.51 
| Resnet101-guided (Ours) |  139.88|  5.72  |  5.56 | 0.73  |  0.55

Reviewer comment (Theory):
I guess from the point (5), this is the main reason why the method can not be applied to DDIM?

Author response:
Here, $\hat{x}_0(t)$ refers to the denoised sample $\hat{x}_t$, denoted as $\hat{x}_0(t)$ = $(\hat{x}_t - (\sqrt{1-\alpha_t})$ $\epsilon \theta(\hat{x}_t,t))$ $ / \sqrt{\alpha_t}$. Ideally, the classifier gradient guides the diffusion denoising process towards the target image. The sampling trajectory from  $\hat{x}_0(T)$ to $\hat{x}_0(0)$ may not be fundamentally different from that from $\hat{x}_T$ to $\hat{x}_0$. 

Practically, we think that for off-the-shelf classifiers, choosing denoised $\hat{x}_0(t)$ as inputs may be more compatible since they are not specifically fine-tuned on noisy data. 
This choice is used in [Dhariwal et al. 2021] but the quantitative comparison between $\hat{x}_0(t)$ and $x_t$ is missing. 
In Table 2 of our revised paper, we select $\hat{x}_0(t)$ as the input to the classifier due to its superior guidance performance. 
In the ablation study table of classifier input types, we generate 10,000 samples from the 128x128 ImageNet dataset using ADM [Dhariwal et al. 2021], utilizing 250 DDPM steps.

Table: Classifier input types ablation study: 
| Inputs |   $\hat{x}_t$   | $\hat{x}_0(t)$ 
|---|---|---
| FID    |  8.61  | 7.17(16\%)     


To provide clarification of the off-the-shelf guidance in the DDIM sampling setting, we have included the 25 steps of DDIM guided-sampling experiments in the table below, it demonstrates that our off-the-shelf classifier guidance also outperforms the fine-tuned classifier-guided ADM-G [Dhariwal et al. 2021] in DDIM. 

Table 6: ImageNet 128x128 evaluation: 25 steps of DDIM sampling results.  
|                 |   IS   |   FID  |  sFID | Precision| Recall 
|---|---|---|---|---|---
| ADM-G           |   -    |  5.98  | 7.04  | 0.78  | 0.51 
| Resnet101-guided (Ours) |  139.88|  5.72  |  5.56 | 0.73  |  0.55

Reviewer comment (Experiments):
Ablation study is missing, it is quite vague to understand which proposed scheme is the main course for the improvement. From my understanding, there are three main differences from normal classifier guidance which are:  
* Gradients via $x_0(t)$ instead of $x_t$  
* Guidance schedule  
* $\tau _2$ temperature  

However, discussion on the contribution of each of them is missing.

Author response:
Thanks for the question. From Section 4.2 to Section 4.5, we have provided ablation studies for each of the design choices. We agree that it's important to look at all of them together in one task. Hence, we have provided additional ablation experiments.

The table below (Table 6 of our revised paper) showcases the results of our off-the-shelf classifier guided sampling in sequential ablation experiments, where we explore different design choices. The sequential parameter rule is employed to identify the selection that yields the best FID value. The percentage number within the brackets of each row indicates the improvement achieved by each design choice compared to the baseline setting, which is represented in the first column of each row. The evaluation process involves generating 10,000 samples from the 128x128 ImageNet dataset using ADM[Dhariwal et al. 2021], utilizing 250 DDPM steps.

Table: Sequential ablation: 
| Inputs |   $\hat{x}_t$   | **$\hat{x}_0(t)$** |  |  |   |
|---|---|---|---|---|---
| FID    |  8.61  | **7.17 (16\%)**    |   |   |   
| Softplus $\beta$ | inf (ReLU) |  5.0 | 4.0 | **3.0** |  2.0 |
| FID              | 7.17 |  6.89  |  6.73  |  **6.61 (8\%)** | 7.02  | 
| marginal temperature $\tau_2$| 1.0 | 0.8 | 0.7 | **0.5** | 0.3 | 
| FID                          | 6.61 |  5.62  | 5.45 | **5.27 (20\%)** | 5.30 |
| sin factor $\gamma$ | 0.0 | 0.1 | 0.2 | **0.3** | 0.4 | 
| FID                 | 5.27 | 5.30  | 5.27 | **5.24 (1\%)** | 5.38 |

Reviewer comment (Experiments):
Besides Resnet, how are other architectures such as DenseNet, Transformers? Do they also work with this scheme?

Author response:
In this research, our primary focus lies on the classical ResNet classifier, given its prominence as a fundamental architecture in classification models. We apply this classifier to various diffusion models, including DDPM, EDM, and DiT. However, we acknowledge the potential for further exploration of different classifier architectures, which we intend to pursue in future research endeavors.

Reviewer comment (Presentation):
The connection from 4.1, 4.2, 4.3 and 4.4, 4.5 as well as the design of the algorithm lacks some connections.

Author response:
The integral calibration error, denoted as $\overline{\text{ECE}}$, plays a crucial role in our research. It assesses the calibration of the off-the-shelf classifier throughout the diffusion process. Proposition 4.1 establishes a connection between classifier guidance and probability estimation, emphasizing the significance of achieving a lower calibration error. In Figure 1, we analyze $\overline{\text{ECE}}$ values and demonstrate that the off-the-shelf ResNet achieves lower $\text{ECE}_t$ values than the fine-tuned classifier when the noises are large. This finding provides a strong basis for asserting that the off-the-shelf classifier has the potential to match the performance of fine-tuned classifier guidance.

To leverage this potential, we utilize the predicted denoised image $\hat{x}_0(t)$ as the input to the classifier, as suggested in universal guidance [Bansal et al. 2023]. To achieve improved calibration, we propose three key design elements: smooth classifier guidance, consistent guidance direction, and an updated guidance schedule. These design choices aim to calibrate the off-the-shelf classifier with the diffusion models and enable the generation of high-quality guided samples. By incorporating these elements, we enhance the calibration of the off-the-shelf classifier and facilitate the production of superior samples.

We have refined the writing at the end of Section 4.1 to better transition to the sections afterward.

Reviewer comment (Theory):
Equation (2) should be $\log exp(\tau f _{y(x)})$? Check format of the equation (2) also.

Author response:
In equations (2) and (3) of the paper, you are absolutely correct that the numerator terms of the guidance are $\log{\exp(\tau f_y(x))}$ and $\log{\exp(\tau_1 f_y(x))}$. We simplify these terms by taking advantage of the fact that the exponential function is already contained within the logarithmic operation. Thank you for pointing it out.

Reviewer comment (Theory):
Equation (3) should be $\log exp(\tau _1 f _{y(x)})$?

Author response:
In equations (2) and (3) of the paper, you are absolutely correct that the numerator terms of the guidance are $\log{\exp(\tau f_y(x))}$ and $\log{\exp(\tau_1 f_y(x))}$. We simplify these terms by taking advantage of the fact that the exponential function is already contained within the logarithmic operation. Thank you for pointing it out.

Reviewer comment (Novelty):
Sec 4.3 "SMOOTH CLASSIFIER" seems trivial. I think there are already many works which use Softplus as activation function and explore its difference/advantage with ReLU. If I understand it correctly, the only contribution here is to replace ReLU with Softplus. The novelty point is not enough for an ICLR paper.

Author response:
The smoothness of the classifier is linked to its calibration. This may seem trivial but to the best of our knowledge, we are the first to point out calibration as one of the guidelines for classifier-guided diffusion generation. 
Further, we take the diffusion process into consideration and propose to look at the expected calibration error along the diffusion process. 

In Figure 1 of our paper, surprisingly, we found that off-the-shelf classifiers are more calibrated than the specifically fine-tuned ones when the noise levels are high. 
We followed up with this observation with guided generation experiments. 
In Appendix A.1, we found that the integral calibration error is strongly positively correlated with the performance 
This has been one of our biggest motivations for improving the classifier calibration. 
We have extended this part in Section 4.1 to make a more convincing case. 

On the practical side, we propose to augment the activation function for calibration, from ReLU to softplus, and choose the hyperparameter $\beta$ by examining the ECE error. 
This type of modification has been explored in [Zhu et al. 2021] for improving adversarial transferability but not for guiding diffusion generation. 
With this design choice, we are able to achieve better performance than well-established classifier guidance and classifier-free guidance on ImageNet benchmarks, which is not known to be possible before.

Reviewer comment (Experiments):
Sec 4.4 "JOINT VS CONDITIONAL DIRECTION", the author mention "we reduce the value of marginal temperature", which seems kind of manual tuning parameters. Is there some validation metric the authors use to determine the optimal temperature?

Author response:
Empirically, we determine the hyperparameter for the marginal temperature based on a lightweight guided sampling of 1000 generated samples. However, we emphasize that the true value lies in the underlying insight. This is supported by the ablation study presented in Table 4 of our paper (we put it below), where all reduced values of the marginal temperature exhibit significant improvements over the baseline($\tau_2=1$), differing only in magnitude.


Table: Ablation study of marginal logit temperature $\tau_2$ with respect to FID. $\tau_1$ is fixed as 1. The evaluation process involves generating 10,000 samples from the 128x128 ImageNet dataset using off-the-shelf classifier-guided ADM[4], utilizing 250 DDPM steps. 
|      Marginal temperature           |   1.0 (base)   |   0.8  |  0.7 |  0.5  | 0.3 
|---|---|---|---|---|---
| FID             |    6.20   | 5.62  | 5.45  | 5.27 | 5.30

Reviewer comment (Experiments):
Sec 5.3 "OFF-THE-SHELF GUIDANCE FOR DIT", the authors propose to incorporate classifier guidance g into classifier-free sampling. The idea is straightforward but the same question the authors introduce a parameter gamma_t here. How do the authors choose a proper gamma_t for a specific case?

Author response:
When it comes to selecting the appropriate choices for $\gamma_t$, we typically employ a lightweight guided sampling of 1000 generated samples for evaluation.

Reviewer comment (Theory):
It is unclear to me how Proposition 1 and ECE inform the design choices of the method. For Section 4.2, the same conclusion can be drawn with only FID. For Section 4.3, Proposition 1 only says “with smoothness $k>1$”, but it doesn’t claim better calibration/score prediction with higher smoothness. Section 4.4 and 4.5 seem to be completely irrelevant to ECE.

Author response:
The link between ECE and smoothness is twofold, as we have explained in the beginning of Section 4.3. Below, we provide further clarification.

First, notice that one of the assumptions for Proposition 4.1 to hold is that the classifier satisfies certain smoothness conditions (inside a Sobolev space). 
If the activation function is ReLU, the classifier will be piecewise linear with first-order derivatives being piece-wise constant. Not only will it **violate the smoothness assumption** of Proposition 4.1, but it is also bad for **gradient-based optimization**. 
Therefore, we considered changing the ReLU activation function to smoothed ones such as Softplus. 

Second, while trying different choices of the parameter $\beta$ within the Softplus function, we found that it also has a **significant effect on ECE**. Hence, we propose to directly choose $\beta$ by minimizing the ECE. We found the $\beta$ that minimizes the ECE also minimizes the generation FID (Please see the results in Figure 3). To be best of the authors' knowledge, tuning activation function for classifier calibration has not been explored before. 

To summarize, **Proposition 4.1 links classifier smoothness, calibration, and gradient estimation together.** 
By modifying the activation function of off-the-shelf classifiers, we are able to achieve both improved smoothness and calibration. 
 

### Marginal temperature
As for the marginal temperature, minimizing ECE is not its motivation. As stated in Section 4.4, this design choice is to **balance the joint vs. conditional direction of the classifier's gradient**. 

On a high level, the joint direction points toward the distribution mode while the conditional direction points away from the decision boundary. 
Inspired by Proposition 4.2 and the visualization in Figure 4, we **amplify the joint direction** relative to the marginal one to help guide the overall diffusion generation process toward the mode of
density. 

This is further supported by the ablation study presented in Table 4 of our paper, where reducing values of the marginal temperature ($\tau_2<1$) exhibit **significant improvements** over the baseline ($\tau_2=1$) in terms of FID. 
By choosing $\tau_2=0.5$ improves the performance (FID) by **20\%** over the baseline. 

We provide further visualization in Figure 4 of our paper. The left-hand side figure illustrates the traditional guidance setting, where the classifier gradient figure gradually fades from $t=50$ to 0, indicating a loss of dog depiction guidance during the sampling. In contrast, in the right-hand side figure where the joint guidance was amplified, the classifier gradient figure increasingly highlights the dog's outline, providing consistent and accurate guidance direction throughout the entire sampling process.

Reviewer comment (Evaluation):
The formula for $\text{ECE}_t$ is with respect to $\hat{x_t}$ but in Section 4.2 the authors talked about using $\hat{x_0}(t)$ will provide better calibration, so which sample did the authors use when providing the ECE results for the rest of the experiments?

Author response:
To estimate the integral calibration error ($\overline{\text{ECE}}$), we utilize the predicted probabilities from the off-the-shelf classifier obtained through reverse process samples at each timestep of the classifier-guided diffusion sampling. These predicted probability values are used to calculate $\overline{\text{ECE}}$, which serves as a metric to evaluate the calibration performance of both the classifier and diffusion models throughout the guided sampling process.

Reviewer comment (Reproducibility):
It is unclear to me how did the authors incorporate the Softplus activation function into the pre-trained classifiers, did they just replace all the activation functions in the pre-trained models? Or is there anything else that they did?

Author response:
We just replace the original ReLU activation functions with Softplus activation in the pre-trained ResNet classifier, eliminating the need for any additional training. This approach has been explored in [Zhu et al. 2021] for improving adversarial transferability.

Reviewer comment (Reproducibility):
How did the authors calculate the marginal likelihood for CLIP guidance generation?

Author response:
During the CLIP-guided diffusion process, we solely rely on the gradient of the joint likelihood as the guidance signal. This approach is adopted due to the absence of a marginal class in CLIP models, which influences our decision to utilize only the joint likelihood gradient for guidance.

It is correct that some of our design choices only apply to probabilistic models with assessable likelihood. 
Nevertheless, our proposed new guidance schedule (Section 4.5) is proven effective for CLIP-guided generation.  
Figure 6 of our revised paper demonstrates a comparison of the CLIP scores between the original guidance schedule and our proposed schedule during the sampling process. The results clearly indicate that our updated schedule consistently can achieve significantly higher CLIP scores throughout the entire sampling process.

Reviewer comment (Reproducibility):
What is the recurrent step in Table 1? Is it the same as the “backward guidance” in “Universal Guidance for Diffusion Model” paper? And why is the inference time not changed significantly with the calibrated method given there is extra marginalization required for all classes? What type of GPU did the authors use in their experiments?

Author response:
We apologize for the confusion. 
The recurrent step refers to the backward/forward guidance technique employed in the universal guidance paper [Bansal et al. 2023]. 
Simply put, if the classifier gradient is inferenced 3 times per step of the diffusion denoising process, the recurrent step is 3. 

Our calibrated designs can all be seen as a pre-conditioning step, which involves replacing the activation functions, changing the softmax temperature, etc. They do not introduce additional computational overhead, thus inference time remains unchanged. 
The purpose of increasing the number of recurrent steps is to demonstrate that augmenting the guidance steps alone does not substantially improve sampling quality. In contrast, our well-calibrated designs leverage the off-the-shelf classifier to enhance sampling performance without extending sampling time.

Regarding "marginalization," it typically refers to the exponential sum of logits of all classes, which is the denominator term used in the softmax function: $\frac{\exp(\tau f_y(x))}{\sum^N_{i=1} \exp(\tau f_i(x))}$. This traditional calculation is commonly employed in classifier-guided sampling [Dhariwal et al. 2021] and is not an additional computation we introduced.

For all of our experiments, we use NVIDIA V100 GPUs. We have added this information in the caption of Table 1 in our revision.

Reviewer comment (Experiments):
While I rate this study highly in general, it falls short in comparing with previous research utilizing off-the-shelf models.

Author response:
Thanks for pointing out the related works [1] and [2]. Both works investigated guided diffusion generation. However, their focus is different from ours, and hence their proposed methodologies are fundamentally different. Below we provide a detailed comparison to each of them.


The goal of [1] is to demonstrate the flexibility of conditional diffusion generation given specified constraints by introducing a regularization term $\log c(x, y)$ in the denoising process. Specifically, it explores constraints to control the digit shape in MNIST generation, facial specified attributes, infer semantic segmentations of city images, and solve a continuous relaxation of the traveling salesman problem. 

The take-home message of this work is that diffusion models, trained on domain-specific data and under carefully designed conditions, are **capable of generating samples that conform to specified constraints**. While this showcases the flexibility of diffusion models in producing constraint-compliant samples, it **doesn't test the limit of the performance in each of the categories**.
 
In comparison, we specifically investigate class-conditional image generation and demonstrated that with proper designs, off-the-shelf classifiers can beat existing state-of-the-art methods. 
Through comprehensive experiments on different architectures of diffusion models, including DDPM, EDM, and DiT, our off-the-shelf classifier-guided sampling consistently surpasses the baseline performance across various diffusion model architectures. 

Specifically, in the table below (Table 7 of our revised paper), our FID result of 2.19 on ImageNet128x128 not only surpasses the fine-tuned classifier-guided performance (FID: 2.97) [4], but also outperforms classifier-free trained diffusion models (FID: 2.43) [5]. This highlights that off-the-shelf guidance is not merely a substitute for fine-tuned guided diffusion to save training costs, but rather a better choice for generating high-quality samples.

Table: ImageNet 128x128 generation results.  
|                 |   IS   |   FID  |  sFID | Precision| Recall |
|---|---|---|---|---|---|
| ADM             |    -   |  5.91  | 5.09  | 0.70  | 0.65
| ADM-G           | 182.69 |  2.98  | 5.09  | 0.78  | 0.59 
| CFG             | 158.47 |  2.43  | -     |   -   |  -  
| Resnet101-guided (Ours)| 187.83 |  2.19  | 4.53  | 0.79  | 0.58 


[2] introduces a multi-expert strategy that utilizes multiple expert classifiers, each trained to handle a specific range of noise values. These "experts" guide the diffusion sampling process at their corresponding time steps. The authors propose a parameter-efficient knowledge transfer strategy to reduce the computational cost. Further, they proposed a teacher-student training approach to avoid the need for labeled data.

However, the experiments conducted in [2] do not provide conclusive evidence that multi-expert-guided sampling can compete with the fine-tuned classifier-guided diffusion ADM-G [4]. 
Moreover, though they employed parameter-efficient fine-tuning strategies like LORA, **their computation cost is still non-negligible and cannot be used in a plug-and-play fashion**. 

In contrast, our off-the-shelf guided sampling not only eliminates the need for further training but also delivers significantly higher sampling performance. As demonstrated in Table 7 of our revised paper, our FID result of 2.19 on ImageNet128x128 not only surpasses the performance of the fine-tuned classifier-guided approach (FID: 2.97) [4], but also outperforms diffusion models trained without the use of classifiers (FID: 2.43) [5].

Reviewer comment (Novelty):
Specifically, the omission of closely related prior works, [1, 2], is significant. PPAP [2] explores guidance using a wide range of modalities of off-the-shelf models in an efficient tuning and plug-and-play manner without significant computational costs, which is directly relevant to this paper's discussion. Observations such as the varying contributions of tuning guidance models across different time steps could also reinforce the evidence between the two studies.

Author response:
Thanks for pointing out the related works [1] and [2]. Both works investigated guided diffusion generation. However, their focus is different from ours, and hence their proposed methodologies are fundamentally different. Below we provide a detailed comparison to each of them.


The goal of [1] is to demonstrate the flexibility of conditional diffusion generation given specified constraints by introducing a regularization term $\log c(x, y)$ in the denoising process. Specifically, it explores constraints to control the digit shape in MNIST generation, facial specified attributes, infer semantic segmentations of city images, and solve a continuous relaxation of the traveling salesman problem. 

The take-home message of this work is that diffusion models, trained on domain-specific data and under carefully designed conditions, are **capable of generating samples that conform to specified constraints**. While this showcases the flexibility of diffusion models in producing constraint-compliant samples, it **doesn't test the limit of the performance in each of the categories**.
 
In comparison, we specifically investigate class-conditional image generation and demonstrated that with proper designs, off-the-shelf classifiers can beat existing state-of-the-art methods. 
Through comprehensive experiments on different architectures of diffusion models, including DDPM, EDM, and DiT, our off-the-shelf classifier-guided sampling consistently surpasses the baseline performance across various diffusion model architectures. 

Specifically, in the table below (Table 7 of our revised paper), our FID result of 2.19 on ImageNet128x128 not only surpasses the fine-tuned classifier-guided performance (FID: 2.97) [4], but also outperforms classifier-free trained diffusion models (FID: 2.43) [5]. This highlights that off-the-shelf guidance is not merely a substitute for fine-tuned guided diffusion to save training costs, but rather a better choice for generating high-quality samples.

Table: ImageNet 128x128 generation results.  
|                 |   IS   |   FID  |  sFID | Precision| Recall |
|---|---|---|---|---|---|
| ADM             |    -   |  5.91  | 5.09  | 0.70  | 0.65
| ADM-G           | 182.69 |  2.98  | 5.09  | 0.78  | 0.59 
| CFG             | 158.47 |  2.43  | -     |   -   |  -  
| Resnet101-guided (Ours)| 187.83 |  2.19  | 4.53  | 0.79  | 0.58 


[2] introduces a multi-expert strategy that utilizes multiple expert classifiers, each trained to handle a specific range of noise values. These "experts" guide the diffusion sampling process at their corresponding time steps. The authors propose a parameter-efficient knowledge transfer strategy to reduce the computational cost. Further, they proposed a teacher-student training approach to avoid the need for labeled data.

However, the experiments conducted in [2] do not provide conclusive evidence that multi-expert-guided sampling can compete with the fine-tuned classifier-guided diffusion ADM-G [4]. 
Moreover, though they employed parameter-efficient fine-tuning strategies like LORA, **their computation cost is still non-negligible and cannot be used in a plug-and-play fashion**. 

In contrast, our off-the-shelf guided sampling not only eliminates the need for further training but also delivers significantly higher sampling performance. As demonstrated in Table 7 of our revised paper, our FID result of 2.19 on ImageNet128x128 not only surpasses the performance of the fine-tuned classifier-guided approach (FID: 2.97) [4], but also outperforms diffusion models trained without the use of classifiers (FID: 2.43) [5].

Reviewer comment (Experiments):
Overall, I highly appreciate the paper's contribution to the completeness of the discussion on diffusion models' guidance. However, to properly highlight this paper's direct contributions, a comparison with prior studies attempting off-the-shelf guidance is crucial. Although the authors conducted experiments comparing various design choices of off-the-shelf guidance, it is not clearly presented how these relate to previous research, and there is no direct comparison of the final FID scores with prior methodologies. The lack of such comparisons has inclined my evaluation towards rejection.

Author response:
Thanks for pointing out the related works [1] and [2]. Both works investigated guided diffusion generation. However, their focus is different from ours, and hence their proposed methodologies are fundamentally different. Below we provide a detailed comparison to each of them.


The goal of [1] is to demonstrate the flexibility of conditional diffusion generation given specified constraints by introducing a regularization term $\log c(x, y)$ in the denoising process. Specifically, it explores constraints to control the digit shape in MNIST generation, facial specified attributes, infer semantic segmentations of city images, and solve a continuous relaxation of the traveling salesman problem. 

The take-home message of this work is that diffusion models, trained on domain-specific data and under carefully designed conditions, are **capable of generating samples that conform to specified constraints**. While this showcases the flexibility of diffusion models in producing constraint-compliant samples, it **doesn't test the limit of the performance in each of the categories**.
 
In comparison, we specifically investigate class-conditional image generation and demonstrated that with proper designs, off-the-shelf classifiers can beat existing state-of-the-art methods. 
Through comprehensive experiments on different architectures of diffusion models, including DDPM, EDM, and DiT, our off-the-shelf classifier-guided sampling consistently surpasses the baseline performance across various diffusion model architectures. 

Specifically, in the table below (Table 7 of our revised paper), our FID result of 2.19 on ImageNet128x128 not only surpasses the fine-tuned classifier-guided performance (FID: 2.97) [4], but also outperforms classifier-free trained diffusion models (FID: 2.43) [5]. This highlights that off-the-shelf guidance is not merely a substitute for fine-tuned guided diffusion to save training costs, but rather a better choice for generating high-quality samples.

Table: ImageNet 128x128 generation results.  
|                 |   IS   |   FID  |  sFID | Precision| Recall |
|---|---|---|---|---|---|
| ADM             |    -   |  5.91  | 5.09  | 0.70  | 0.65
| ADM-G           | 182.69 |  2.98  | 5.09  | 0.78  | 0.59 
| CFG             | 158.47 |  2.43  | -     |   -   |  -  
| Resnet101-guided (Ours)| 187.83 |  2.19  | 4.53  | 0.79  | 0.58 


[2] introduces a multi-expert strategy that utilizes multiple expert classifiers, each trained to handle a specific range of noise values. These "experts" guide the diffusion sampling process at their corresponding time steps. The authors propose a parameter-efficient knowledge transfer strategy to reduce the computational cost. Further, they proposed a teacher-student training approach to avoid the need for labeled data.

However, the experiments conducted in [2] do not provide conclusive evidence that multi-expert-guided sampling can compete with the fine-tuned classifier-guided diffusion ADM-G [4]. 
Moreover, though they employed parameter-efficient fine-tuning strategies like LORA, **their computation cost is still non-negligible and cannot be used in a plug-and-play fashion**. 

In contrast, our off-the-shelf guided sampling not only eliminates the need for further training but also delivers significantly higher sampling performance. As demonstrated in Table 7 of our revised paper, our FID result of 2.19 on ImageNet128x128 not only surpasses the performance of the fine-tuned classifier-guided approach (FID: 2.97) [4], but also outperforms diffusion models trained without the use of classifiers (FID: 2.43) [5].

Reviewer comment (Experiments):
In the CLIP guidance experiments, it seems that the authors did not apply the same level of guidance as they did with off-the-shelf classifiers. The CLIP guidance experiment does not appear to reflect the authors' contributions with a new methodology for off-the-shelf guidance.

Author response:
In the CLIP-guided diffusion section, we showcase the possibilities for future research in exploring other modalities. Figure 6 of our revised paper provides a comparison of the CLIP scores between the original guidance schedule and our proposed schedule during the sampling process. The results clearly indicate that our updated schedule consistently can achieve significantly higher CLIP scores throughout the entire sampling process.

Reviewer comment (Experiments):
Based on the various considerations and discoveries made by the authors, I am curious whether it is possible to extend off-the-shelf guidance beyond class conditions to various modalities of conditional generation. It seemed that the CLIP experiment was intended to demonstrate such a possibility; however, as mentioned in the weaknesses, it did not feel like an experiment based on a new methodology reflecting the authors' considerations and findings.

Author response:
This research primarily focuses on exploring classifier guidance, specifically in the context of text-to-image generation using an off-the-shelf CLIP model. However, in our future research endeavors, we intend to extend our investigation to encompass guidance from other modalities, such as leveraging the capabilities of large language models. By exploring these additional modalities, we aim to further enhance the performance and flexibility of our proposed approach.

We believe this is a very important future direction to explore. Currently, the major concern for state-of-the-art text-to-image models such as Stable Diffusion is not image quality, but controllability, i.e., the compatibility between the generated image and the input text. 
For instance, a simple prompt like "a green flower and a yellow leaf" is very challenging [7]. 

T2I-compbench is specifically designed to quantitatively measure text-image compatibility, which employs several image-to-text discriminative models, e.g., BLIP-VQA, to serve as the judge. 
This indicates that SOTA image-to-text discriminative models are better in terms of text-image compatibility, and have the potential to be used as guidance to improve the text-to-image generation. 
Our off-the-shelf guidance method can potentially be very helpful in improving the controllability of text-to-image generation. 

For all modalities, evaluating the generated samples is a challenging task. We have FIDs for images but such a well-accepted criterion is missing for 3D generation. 
Nevertheless, as long as there are powerful discriminative models that can tell whether the generated samples are good or not, they can be utilized in a training-free fashion by extending our method to achieve a significant boost in conditional generation across every modality.