Paper: Score-based Conditional Generation with Fewer Labeled Data by Self-calibrating Classifier Guidance (ICLR-2024)

Reviewer comment (Experiments):
Diffusion score-based models have exhibited state-of-the-art performance for years, showcasing exemplary results on numerous benchmarks with photo-realistic data. However, the experiments in this paper are confined to an initial setup, focusing on images with a 32x32 resolution and only encompassing 100 classes. The reviewer is uncertain whether the proposed approach can demonstrate superior performance among the baselines on widely recognized benchmarks such as ImageNet.

Author response:
We agree on the importance of evaluating our method on more datasets with higher resolution and more classes. Due to the limited computational resources in the academic environment, we were not able to do so previously. We are continuously using our available resources for additional experiments and will include them in future revisions upon completion. In the meantime, we hope to emphasize that the clear advantage of our proposed CG-SC in the fewer labeled data regime in CIFAR-10 and CIFAR-100 already provides strong support for the validity of the proposed method.

Reviewer comment (Theory):
The rationale behind the design of the proposed SC loss remains ambiguous. Could the authors elucidate with an analytical perspective why the proposed method should outperform the baselines (i.e., DLSM and JEM), which explicitly minimize the Fisher and KL divergences, respectively? Furthermore, could the authors theoretically justify the effectiveness of the proposed loss in a semi-supervised setup (e.g., defining the relationship of data-label pairs through optimal transport [2])?

Author response:
Advantage over JEM:
    	- Time efficiency: Since the training of JEM requires MCMC sampling, further interpreting classifiers as SGMs significantly reduces training time.
	- Advantage over DLSM:
		- Utilization of unlabeled data: The calculation of DLSM loss requires ground truth labels, while the proposed loss can be applied to both labeled and unlabeled data.
- To our understanding, the theoretical effectiveness of our method is not backed by optimal transport. We agree that given the promising results, further theoretical justifications would be beneficial. We hope our work could stimulate future theoretical analysis of the effect of the proposed regularization method.

Reviewer comment (Experiments):
How were the hyper-parameters for the baseline methods, specifically the balancing factors of CG-DLSM and CG-JR, and the smoothing factor in CG-LS, chosen to ensure a fair comparison in Section 4?

Author response:
Balancing factor of CG-DLSM is selected to be $1$, as suggested in the original paper.
	- Balancing factor of CG-JR is selected to be $0.01$, as suggested in the original paper.
	- The smoothing factor in CG-LS is tuned between $\{0.1, 0.05\}$ for the better intra-FID. Note that the performance of the two settings do not differ by much.

Reviewer comment (Novelty):
This work is the lack of technical novelty. The proposed self-calibration (SC) loss in this paper heavily depends on two prior works. One is EBMs, which connect probability distribution and energy function. The other one is JEM, which reinterprets classifiers $f(\boldsymbol{x}, y ; \boldsymbol{\phi})$ as EBMs and enforcing regularization with related objectives helps classifiers $f(\boldsymbol{x}, y ; \boldsymbol{\phi})$ to capture more accurate probability distributions.

Author response:
- [Weakness 1] Lack of novelty: depends on JEM and EBM
    - Although the proposed method utilizes the findings of JEM and EBM, we believe it is unfair to judge our novelty based on this. Our main contributions are that we proposed to further reinterpret classifiers as SGMs for regularization purposes, discovered the potential of CGSGM under semi-supervised settings, and designed experiments to verify the proposed method's effectiveness in improving CGSGM under both fully-supervised and semi-supervised settings.

Reviewer comment (Evaluation):
While the method proposed in this paper demonstrates stronger capabilities when labeled data is extremely scarce (only 5% of the data is labeled), in many instances where labeled data exceeds 50% (as shown in Sec 4.2 and Appendix A), the CFG method exhibits greater conditional generative power than the proposed CG-SC method and traditional CG methods. Currently, the CFG method is the more prevalent approach in conditional generation, mainly because it eliminates the need for training an additional classifier and can seamlessly integrate into the conventional DSM loss, which is much more easier to be applied in downstream tasks.

Author response:
- [Weakness 2] In instances where labeled data exceeds 50% (as shown in Sec 4.2 and Appendix A), the CFG method exhibits greater conditional generative power than the proposed CG-SC method and traditional CG methods.
    - We agree about the strength of CFG (and Cond) with more labeled data. Somehow we hope to re-emphasize that the paper does not focus on the regime of more labeled data. We believe it would be fairer if our proposed CG-SC could be discussed within the regime that we designed the method for. The results demonstrate CG-SC is more effective in the more realistic or common scenarios with less than 20% labeled data---the focus of our paper.

Reviewer comment (Experiments):
What is the hyperparameter $\lambda_{CG}$ for classifier guidance with and without self-calibration set in Table 1 for CG and CG-SC methods? As the traditional CG method regularizes the classifier externally by $\lambda_{CG}$ and the choice of the parameter $\lambda_{CG}$ chosen has a significant impact on the generation outcomes[1], there may be more experiment for CG by tuning the hyperparameter $\lambda_{CG}$ for a fair comparison. Moreover, there is also a hyperparameter classifier-free guidance scale $\lambda_{CFG}$ for CFG method [2], there should be more experiment on CFG methods by tuning the hyperparameter $\lambda_{CFG}$ for a fair comparison.

Author response:
- [Question 1] Tuning $\lambda_{CG}$ and $\lambda_{CFG}$
    - Both parameters are tuned to obtain the best intra-FID
    - $\lambda_{CG}$ is tuned within $\{0.5, 0.8, 1.0, 1.2, 1.5, 2.0, 2.5\}$
    - $\lambda_{CFG}$ is tuned within $\{0, 0.1, 0.2, 0.4\}$
    - We will include this in the Appendix of future revisions for further clarification of the experimental details.

Reviewer comment (Evaluation):
Why there is an inconsistency between conditional metrics and unconditional metrics? (Appendix H) The FID (Fréchet Inception Distance) serves as a metric to assess the diversity and authenticity of generated data distributions in comparison to true data distributions. In my view, if the conditional metrics for each category (intra FID) are better, then the overall unconditional metrics (FID) should also be better.

Author response:
- [Question 2] Why is there a mismatch between conditional and unconditional metrics?
    - Although the generated images exhibit high fidelity and cover a significant portion of the input data distribution (good FID), they might fail to accurately capture the class-conditional distributions (poor intra-FID). This limitation can result from either poor generation accuracy or low generation diversity in certain classes, leading to poor intra-FID scores for those specific classes, as shown in Appendix A. For instance, consider an extreme case where the generator mistakenly swapped images between the 'cat' and 'dog' classes, the FID may remain high, but the intra-FID would significantly decline. After we swap the incorrect images back, the intra-FID would become substantially better while the FID remains the same.
    - Tuning the scaling factor introduces a trade-off between accuracy and diversity. In Appendix H, as the scaling factor becomes larger, the generation accuracy increases while the diversity decreases. The mismatch between FID and intra-FID came from the fact that FID only considers the diversity of images, while intra-FID additionally takes into account how well the generated images represent each class (generation accuracy).

Reviewer comment (Experiments):
While the paper mentions that simply regularizing the classifier to prevent over-fitting suffers from distribution alignment, not much experiments are done to justify this. The only baseline which the paper considers is the adversarial training of the classifier. I would like to see more experiments done with strong regularizers such as using RandAugment or CutMix augmentations.

Author response:
- [Weakness 1] Experiments with RandAugment and CutMix
    - Indeed augmentation-based regularizations are important. We wanted to avoid augmentation-based regularization since it is hard to compare those methods with regularizations that use auxiliary losses as they are done in different stages (pre-processing v.s. training stage).

Reviewer comment (Evaluation):
For the CIFAR-100 results in table1 why doesn't it have the accuracy metric?

Author response:
- [Weakness 3] Why doesn't Cifar-100 have generation accuracy evaluation?
    - As mentioned in Section 4.1, nearly-accurate classification is desired for meaningful evaluation of the generation accuracy, but we were unable to locate such a classifier for the CIFAR-100 dataset. Therefore, generation accuracy is not included for the CIFAR-100 dataset.

Reviewer comment (Experiments):
For the baseline CG-DLSM, the classifier is regularized over only labeled data using external SGM. However, this classifier can be regularized over unlabeled data also? What happens if this baseline's classifier is also regularized over unlabeled data? This would be a fairer comparison with the paper's results.

Author response:
- [Weakness 4] Why not use DLSM on unlabeled data?
    - In the formulation of DLSM loss $\mathbb{E}\left[\frac{1}{2}\lVert \nabla_\tilde{x}\log p(\tilde{y}\vert \tilde{x}; \theta)+s(\tilde{x};\phi)-\nabla_\tilde{x}\log p_\sigma(\tilde{x}\vert x)\rVert^2\right]$, ground truth labels are required for calculation (the $\tilde{y}$ in $\nabla_\tilde{x}\log p(\tilde{y}\vert \tilde{x}; \theta)$). As a result, it is not applicable to unlabeled data.

Reviewer comment (Novelty):
The methodology appears incremental, building marginally upon JEM's foundation of interpreting classifiers as time-dependent EBMs. The newly introduced self-calibration loss primarily enhances this by applying a standard DSM technique to train the internal score function, thus lacking substantial novelty.

Author response:
- [Weakness 1] Our method is incremental (adding JEM and DSM to train the classifier)
    - Although the proposed method utilizes the findings of JEM and DSM, we believe it is unfair to judge our novelty based on this. Our main contributions are that we proposed to further reinterpret classifiers as SGMs for regularization purposes, discovered the potential of CGSGM under semi-supervised settings, and designed experiments to verify the proposed method's effectiveness in improving CGSGM under both fully-supervised and semi-supervised scenarios.

Reviewer comment (Presentation):
The allocation of Figure 1 is too naive. Overall, you could have edited the space of main paper more wisely.

Author response:
- Minor weaknesses
    - Thank you for the suggestions. We will improve the presentation of the mentioned point in future revisions.

Reviewer comment (Novelty):
I am afraid that the novelty may be limited since the techniques are common in deep-generation models.

Author response:
- [Weakness 1] Originality: The techniques are common in deep-generation models
    - Indeed, regularization methods are common in deep generative models, but whether a new regularization technique is effective needs verification through experiments. Our main contributions are that we proposed to further reinterpret classifiers as SGMs for regularization purposes, discovered the potential of CGSGM under semi-supervised settings, and designed experiments to verify the proposed method's effectiveness in improving CGSGM under both fully-supervised and semi-supervised scenarios. Furthermore, to our understanding, there is no other work that proposed to regularize classifiers as SGMs.

Reviewer comment (Novelty):
The proposed method may have little effect on other sub-fields of machine learning.

Author response:
- [Weakness 3] Little effect on other sub-fields of machine learning
    - We believe our work should already be considered significant as the sub-field we are studying ( conditional generation) is itself a significant sub-field of machine learning.