Paper: Label-Noise Robust Diffusion Models (ICLR-2024)

Reviewer comment (Novelty):
It is not clear what is the major difference from the methods that boosting the robustness of generative models on noisy labels.

Author response:
We appreciate the thorough reviews and valuable comments. We address the concerns below.


>**Q1. [Major difference from the other methods for generative model learning with noisy label]** *(Weaknesses 1) It is not clear what is the major difference from the methods that boosting the robustness of generative models on noisy labels.*

We compared our model with GAN-based generative models for the noisy label problem in the last paragraph of Section 2 and immediately below Proposition 2 in Section 3. For clarity, we present the detailed explanation of the GAN-based models and the comparison table in below and include it in the Appendix B.4 of the revised paper.

|                                 | [D1, D2]                                          | Ours                                   |
|---------------------------------|-----------------------------------------------|----------------------------------------|
| Base model                      | GAN                                           | Diffusion model                        |
| Objective                       | Noisy-label conditional **distribution** matching | Noisy-label conditional **score** matching |
| Label transition information    | $p(\tilde{y}\|y)$                             | $p_t(y\|\tilde{y},x_t)$                |
| &nbsp;&nbsp;&nbsp;&nbsp;Time-dependent label transition | X                                        | O                                 |
| &nbsp;&nbsp;&nbsp;&nbsp;Instance-wise label transition  | X                                        | O                                 |

There are a few studies for generative model learning from noisy labels. [D1, D2] proposed algorithms to overcome the difficulties of training GANs with noisy labels by matching the noisy-label conditional probability with the transition matrix $p(\tilde{y}\|y)$. On the other hand, our method focused on the robustness of diffusion models on noisy labels by matching the noisy-label conditional scores with the transition-aware weighted function $p_t(y\|\tilde{y},x_t)$.

Specifically, the structure of a generator in [D1, D2] is similar to the traditional GAN, but they modify the discriminator structure by incorporating the label transition matrix. When the discriminator is fed the samples generated by the generator and the corresponding labels, the labels are corrupted by the label transition matrix to become noisy labels. Since the given dataset is also a noisy labeled dataset, training the discriminator in this way makes it a match between the noisy labeled datasets. Then, the generator can have a clean label conditional distribution.

However, the diffusion model cannot completely mitigate the impact of noisy labels with the transition matrix alone because it targets the gradient of the log-likelihood, i.e., data score. We show theoretically the impact of noisy label on data score in Theorem 1. This impact could be represented by the transition-aware weighted function $w$, which is time- and instance-dependent unlike the transition matrix. We include further analysis of this function in the global response.
***
[D1] Thekumparampil, K. K., Khetan, A., Lin, Z., & Oh, S. (2018). Robustness of conditional gans to noisy labels. Advances in neural information processing systems, 31.   
[D2] Kaneko, T., Ushiku, Y., & Harada, T. (2019). Label-noise robust generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2467-2476).

Reviewer comment (Novelty):
The significance of this research topic is not clear, please explain it. Specifically, to the best of my knowledge, generative models are usually unsupervised, and thus there is only a few methods on boosting the model robustness against on noisy labels.

Author response:
>**Q2. [Significance of the research topic]** *(Weaknesses 2) The significance of this research topic is not clear, please explain it. Specifically, to the best of my knowledge, generative models are usually unsupervised, and thus there is only a few methods on boosting the model robustness against on noisy labels.*

While generative models are typically formulated on an unsupervised basis, their practical application often involves the generation of samples tailored to the user’s specific needs [D3, D4, D5, D6]. Therefore, conditional generative models play a crucial role in the practical application of generative models. In this case, the noisy label problem is a pervasive problem in the entire field of machine learning and is particularly critical in practical applications [D7, D8, D9]. Thus, as the reviewer points out, the lack of research on generative models dealing with noisy labels is a significant problem for the application of generative models.

Reviewer BRfo acknowledges that our model is the first work to consider the influence of noisy labels in diffusion models, and Reviewer Zgz7 values that our research topic targets an important problem because it is a common and practical challenge in real-world scenarios.
***
[D3] Perarnau, G., van de Weijer, J., Raducanu, B., & Álvarez, J. M. (2016). Invertible Conditional GANs for image editing. NIPS Workshop on Adversarial Training.   
[D4] Das, H. P., Tran, R., Singh, J., Yue, X., Tison, G., Sangiovanni-Vincentelli, A., & Spanos, C. J. (2022, June). Conditional synthetic data generation for robust machine learning applications with limited pandemic data. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 11, pp. 11792-11800).   
[D5] Pinaya, W. H., Tudosiu, P. D., Dafflon, J., Da Costa, P. F., Fernandez, V., Nachev, P., ... & Cardoso, M. J. (2022, September). Brain imaging generation with latent diffusion models. In MICCAI Workshop on Deep Generative Models (pp. 117-126).   
[D6] Iyer, A., Dey, B., Dasgupta, A., Chen, W., & Chakraborty, A. (2019). A conditional generative model for predicting material microstructures from processing methods. Second Workshop on Machine Learning and the Physical Sciences in NeurIPS 2019.   
[D7] Welinder, P., Branson, S., Perona, P., & Belongie, S. (2010). The multidimensional wisdom of crowds. Advances in neural information processing systems, 23.   
[D8] Xiao, T., Xia, T., Yang, Y., Huang, C., & Wang, X. (2015). Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2691-2699).   
[D9] Cheng, H., Zhu, Z., Li, X., Gong, Y., Sun, X., & Liu, Y. (2020, October). Learning with Instance-Dependent Label Noise: A Sample Sieve Approach. In International Conference on Learning Representations.

Reviewer comment (Experiments):
It is not clear the model performance on severe noisy labels, like 60%, 80%.

Author response:
>**Q3. [Performance on severe noisy labels]** *(Weaknesses 3) It is not clear the model performance on severe noisy labels, like 60%, 80%.*
 
To verify the robustness of the proposed model under extreme noise, we perform the experiments on the CIFAR-10 dataset with severe symmetric noise rates of 60% and 80%. The results are presented in the table below. These results show that TDSM consistently improves performance even at severe noise rates. We include these results in Appendix F.11 of the revised version.

|      | Metric                   | 60%      |            | 80%      |            |
|------|--------------------------|----------|------------|----------|------------|
|      |                          | DSM      | TDSM       | DSM      | TDSM       |
| un   | FID ($\downarrow$)       | **2.17** | 2.55       | **2.22** | 2.44       |
|      | IS ($\uparrow$)          | 9.75     | **10.09**  | 9.73     | **9.79**   |
|      | Density ($\uparrow$)     | 102.17   | **110.78** | 103.09   | **103.69** |
|      | Coverage ($\uparrow$)    | 81.00    | **81.65**  | 80.81    | **81.15**  |
| cond | CW-FID ($\downarrow$)    | 48.94    | **23.36**  | 72.51    | **53.57**  |
|      | CAS ($\uparrow$)         | 29.72    | **52.38**  | 9.93     | **24.68**  |
|      | CW-Density ($\uparrow$)  | 58.42    | **86.58**  | 43.97    | **54.30**  |
|      | CW-Coverage ($\uparrow$) | 61.88    | **75.08**  | 44.53    | **56.47**  |

Reviewer comment (Evaluation):
It is not clear whether the proposed method can boost the model classification performance?

Author response:
>**Q4. [Classification Performance]** *(Weaknesses 4) It is not clear whether the proposed method can boost the model classification performance.*

>*(Questions 2) Can the proposed method boost the model classification performance on noisy labels?*

We provide the response to this question in the Global Response. For your convenience, we also copy and paste the response provided in the Global Response.

***

**[Advantage of the proposed method]**

Label recovery is an intuitive way to overcome the problem of noisy labels in a dataset. Our method does not specifically focus on robustness to noisy labels through label recovery; instead, it focuses on robustness to noisy labels from a diffusion model training perspective. Consequently, our approach can be synergistically combined with existing label recovery methods to further improve performance. The analysis of this synergy is detailed in Section 4.4.

It is important to analyze how our method overcomes the noisy label robustness in diffusion models. Our analysis indicates that our proposed TDSM objective provides the diffusion model with information about the diffused label noise in the dataset. This information is represented by the transition-aware weight function $w$, depending upon the diffusion time. Therefore, we estimate the instance- and time-dependent label transition probability with the transition-aware weight function.

**[Transition-aware weight function plays a different role from a typical classifier]**

We want to emphasize that the transition-aware weight function $w$ plays a different role from a typical classifier. This probability represents the relationship between the noisy and clean label conditional scores (which are also time-dependent) and is an important element of the proposed training objective of a diffusion model. Therefore, we need to estimate this label transition probabilities over diffusion timesteps.

Specifically, our transition-aware weight function can be reformulated as follows:
$$
w(x_t,\tilde{y},y,t)=\frac{p_t(Y=y|x_t)p(\tilde{Y}=\tilde{y}|Y=y)}{p_t(\tilde{Y}=\tilde{y}|x_t)}.
$$
As seen in the expression, the weight function is composed of the clean label classifier $p_t(Y=y|x_t)$, label transition prior $p(\tilde{Y}=\tilde{y}|Y=y)$, and the noisy label classifier  $p_t(\tilde{Y}=\tilde{y}|x_t)$. Given the triplet of perturbed data instance and its corresponding noisy label and timestep, i.e., $(x_t, \tilde{y}, t)$, each component of $w$ has the following characteristics with respect to $y$: 1) $p(Y=y|x_t)$ is maximized when $y$ is the clean label of the clean data $x_0$ of $x_t$; 2) $p(\tilde{Y}=\tilde{y}|Y=y)$ is maximized when $y$ is the noisy label $\tilde{y}$ in general. The reason for 2) is that in general, a given noisy label is sufficiently likely to be a clean label. These two trade-offs imply that the $w$ function does not behave like a clean label classifier.

Furthermore, for large enough $t$, the distribution of $x_t$ converges to a label-independent prior distribution by the design of the diffusion process, so that $p_t(Y|x_t)$ converges to a uniform distribution. Therefore, for sufficiently large $t$, the $w$ function converges to a transition matrix $S$. This phenomenon can also be demonstrated in Figure 2 of the 2-D example.

In summary, the transition-aware weight function needed to overcome noisy labels in a diffusion model is not represented clean label recovery information only, and this function has time-varying information. 

**[Experimental results of clean label accuracy]**

It is possible to check the accuracy of the transition-aware weight function on the training dataset since the function provides the probability of a clean label given a noisy label and an instance. Figure 13 in the revised manuscript shows the clean label accuracy of the transition-aware weight function over diffusion timesteps. Since the noisy labels in the dataset are used as the input conditions of the baseline model for the entire diffusion timesteps, we plot the percentages of clean labels in the datasets as dots at $t=0$ and plot a dashed horizontal line for comparison.

Focusing on 'Ours (VolMinNet + TDSM)', which provides the best generation performance, we found that the behavior of the $w$ function varies with diffusion timesteps, as mentioned above. In particular, as $t$ increases, the clean label accuracy converges to the clean rate of the given dataset, which is 0.6. This is because the $w$ function converges to the label transition prior $p(\tilde{Y}|Y)$. 'Base (VolMinNet + DSM)', which does not take this into account, learns the diffusion model only with corrected labels, leading to the performance difference. Therefore, while existing noisy label methods that focus on clean label recovery certainly contribute significantly to generative performance, considering the TDSM objective in diffusion model training enables additional performance improvement independent of clean label recovery.

Reviewer comment (Experiments):
Can the proposed method boost the model classification performance on noisy labels?

Author response:
>**Q4. [Classification Performance]** *(Weaknesses 4) It is not clear whether the proposed method can boost the model classification performance.*

>*(Questions 2) Can the proposed method boost the model classification performance on noisy labels?*

We provide the response to this question in the Global Response. For your convenience, we also copy and paste the response provided in the Global Response.

***

**[Advantage of the proposed method]**

Label recovery is an intuitive way to overcome the problem of noisy labels in a dataset. Our method does not specifically focus on robustness to noisy labels through label recovery; instead, it focuses on robustness to noisy labels from a diffusion model training perspective. Consequently, our approach can be synergistically combined with existing label recovery methods to further improve performance. The analysis of this synergy is detailed in Section 4.4.

It is important to analyze how our method overcomes the noisy label robustness in diffusion models. Our analysis indicates that our proposed TDSM objective provides the diffusion model with information about the diffused label noise in the dataset. This information is represented by the transition-aware weight function $w$, depending upon the diffusion time. Therefore, we estimate the instance- and time-dependent label transition probability with the transition-aware weight function.

**[Transition-aware weight function plays a different role from a typical classifier]**

We want to emphasize that the transition-aware weight function $w$ plays a different role from a typical classifier. This probability represents the relationship between the noisy and clean label conditional scores (which are also time-dependent) and is an important element of the proposed training objective of a diffusion model. Therefore, we need to estimate this label transition probabilities over diffusion timesteps.

Specifically, our transition-aware weight function can be reformulated as follows:
$$
w(x_t,\tilde{y},y,t)=\frac{p_t(Y=y|x_t)p(\tilde{Y}=\tilde{y}|Y=y)}{p_t(\tilde{Y}=\tilde{y}|x_t)}.
$$
As seen in the expression, the weight function is composed of the clean label classifier $p_t(Y=y|x_t)$, label transition prior $p(\tilde{Y}=\tilde{y}|Y=y)$, and the noisy label classifier  $p_t(\tilde{Y}=\tilde{y}|x_t)$. Given the triplet of perturbed data instance and its corresponding noisy label and timestep, i.e., $(x_t, \tilde{y}, t)$, each component of $w$ has the following characteristics with respect to $y$: 1) $p(Y=y|x_t)$ is maximized when $y$ is the clean label of the clean data $x_0$ of $x_t$; 2) $p(\tilde{Y}=\tilde{y}|Y=y)$ is maximized when $y$ is the noisy label $\tilde{y}$ in general. The reason for 2) is that in general, a given noisy label is sufficiently likely to be a clean label. These two trade-offs imply that the $w$ function does not behave like a clean label classifier.

Furthermore, for large enough $t$, the distribution of $x_t$ converges to a label-independent prior distribution by the design of the diffusion process, so that $p_t(Y|x_t)$ converges to a uniform distribution. Therefore, for sufficiently large $t$, the $w$ function converges to a transition matrix $S$. This phenomenon can also be demonstrated in Figure 2 of the 2-D example.

In summary, the transition-aware weight function needed to overcome noisy labels in a diffusion model is not represented clean label recovery information only, and this function has time-varying information. 

**[Experimental results of clean label accuracy]**

It is possible to check the accuracy of the transition-aware weight function on the training dataset since the function provides the probability of a clean label given a noisy label and an instance. Figure 13 in the revised manuscript shows the clean label accuracy of the transition-aware weight function over diffusion timesteps. Since the noisy labels in the dataset are used as the input conditions of the baseline model for the entire diffusion timesteps, we plot the percentages of clean labels in the datasets as dots at $t=0$ and plot a dashed horizontal line for comparison.

Focusing on 'Ours (VolMinNet + TDSM)', which provides the best generation performance, we found that the behavior of the $w$ function varies with diffusion timesteps, as mentioned above. In particular, as $t$ increases, the clean label accuracy converges to the clean rate of the given dataset, which is 0.6. This is because the $w$ function converges to the label transition prior $p(\tilde{Y}|Y)$. 'Base (VolMinNet + DSM)', which does not take this into account, learns the diffusion model only with corrected labels, leading to the performance difference. Therefore, while existing noisy label methods that focus on clean label recovery certainly contribute significantly to generative performance, considering the TDSM objective in diffusion model training enables additional performance improvement independent of clean label recovery.

Reviewer comment (Evaluation):
In Table 2, the authors seem to only compare with DSM with non-class-aware evaluation metrics. What's the reason for this comparison with the specific metrics?

Author response:
>**Q2. [class-aware evaluation metrics for Table 2]** *(Weaknesses 2) In Table 2, the authors seem to only compare with DSM with non-class-aware evaluation metrics. What's the reason for this comparison with the specific metrics?*

The experiments for Table 2 in Section 4.2 applied our models to benchmark dataset with the conjecture that some annotated labels may be noisy. For the class-aware evaluation metrics, we need to have true labels, but due to the conjecture, we thought it was not appropriate to use these metrics with the annotated labels.

Having said that, it is possible to evaluate the class-aware evaluation metrics with the annotated labels. The results are shown below, and we find that our model further improves the intra-class sample quality. We add this reason and include the results in the Appendix F.10 and Table 12 in the revised manuscript.

|      | Metric                | MNIST     |           | CIFAR-10  |            | CIFAR-100 |           |
|------|-----------------------|-----------|-----------|-----------|------------|-----------|-----------|
|      |                       | Base      | Ours      | Base      | Ours       | Base      | Ours      |
| un   | FID ($\downarrow$)    | -         | -         | 1.92      | **1.91**   | **2.51**  | 2.67      |
|      | IS ($\uparrow$)       | -         | -         | 10.03     | **10.10**  | 12.80     | **12.85** |
|      | Density ($\uparrow$)   | 86.20     | **88.08** | 103.08    | **104.35** | 87.98     | **90.04** |
|      | Coverage ($\uparrow$)    | 82.90     | **83.69** | 81.90     | **82.07**  | 77.63     | **78.28** |
| cond | CW-FID ($\downarrow$) | -         | -         | 10.23     | **10.18**  | 66.97     | **66.68** |
|      | CAS ($\uparrow$) | **98.55** | 98.50     | **77.74** | 77.07      | **39.50** | 39.10     |
|      | CW-Density ($\uparrow$) | 85.79     | **87.96** | 102.63    | **103.69** | 82.58     | **84.96** |
|      | CW-Coverage ($\uparrow$)  | 82.09     | **82.98** | 81.57     | **81.88**  | 75.78     | **76.53** |

Reviewer comment (Novelty):
In the review of diffusion models, the authors seem to only review from the score matching networks, while omit the diffusion models derivated from optmizing the ELBO.

Author response:
>**Q5. [Review of diffusion models]** *(Weaknesses 4) In the review of diffusion models, the authors seem to only review from the score matching networks, while omit the diffusion models derivated from optmizing the ELBO.*

Thank you for pointing out the problem. Our method is developed through the relationship between conditional scores; therefore, we explained the diffusion model based on the score matching model. As suggested by the reviewer, we add the discussion of the diffusion models derived from optimizing the ELBO and the explanations of the relationship between the score matching model and the diffusion model in the new Appendix B.1 in the revised version.

Reviewer comment (Novelty):
Previous works are not correctly reviewed or cited. For example, the reference of denoising diffusion probabilistic model (Ho et al., 2020) is classified into video generation, in the introduction, while this is a fundamental work in diffusion models, and the authors may wanted to put video diffusion models (Ho et al., 2022) there. Moreover, some prior works that tackle the uncurated label distributions are not discussed and compared in the paper.

Author response:
>**Q6. [Previous works]** *(Weaknesses 5) Previous works are not correctly reviewed or cited. For example, the reference of denoising diffusion probabilistic model (Ho et al., 2020) is classified into video generation, in the introduction, while this is a fundamental work in diffusion models, and the authors may wanted to put video diffusion models (Ho et al., 2022) there. Moreover, some prior works that tackle the uncurated label distributions are not discussed and compared in the paper.*

Thank you for pointing out the incorrect citation. We modify this citation and confirm that there are no issues with other references. We also add discussion of studies on generative models that deal with uncurated label datasets in Appendix B.4 of the revised version.

Reviewer comment (Presentation):
It would be beneficial to include a color bar in Fig 8 to interprete the meaning of the colors presented.

Author response:
>**Q7. [Color bar]** *(Weaknesses 6) It would be beneficial to include a color bar in Fig 8 to interprete the meaning of the colors presented.*

Thank you for the suggestion. The color of the arrow means the norm of the score vector. We add a color bar and the meaning of the colors in Figure 8 of the revised version.

Reviewer comment (Novelty):
**Missing Citations:** The paper could be significantly enhanced by including additional relevant literature. Learning the transition relation from noisy labels to clean labels has been previously explored under the umbrella of mixture proportion estimation. Moreover, methods for estimating the transition matrix in class-dependent settings are also very related. Acknowledging popular works could provide a richer theoretical foundation.

Author response:
We briefly discussed the methods of existing noisy label learning studies in Appendix B.2 of the original manuscript. As the reviewer pointed out, the methods of mixture proportion estimation and transition matrix estimation are highly relevant to our method. We have added more detailed descriptions of these methods in Appendix B.3 of the revised manuscript, and we have mentioned this discussion in the main paper.

Reviewer comment (Experiments):
**Missing Baselines:** There are many methods for learning with noisy labels. It would be beneficial to combine existing state-of-the-art (SOTA) methods for learning with noisy labels to obtain estimated clean labels. Then, utilizing conditional diffusion models on these cleaned labels and comparing the performance with the author's method could offer a more comprehensive evaluation of the proposed method’s efficacy compared to current alternatives.

Author response:
In Section 4.4, we conducted experiments comparing our model with the baseline model proposed by the reviewer (*combining existing SOTA methods for learning with noisy labels to obtain estimated clean labels, and utilizing conditional diffusion models on these cleaned labels*). We applied the methods of the state-of-the-art models based on the transition matrix (VolMinNet, Li et al., ICML 2021 [B1]) and the latest model (DISC, Li et al., CVPR 2023 [B2]) to CIFAR datasets. The complete results are shown in Tables 10 and 11 of Appendix F.8 of the revised version. The experimental results indicate that applying our objective consistently improves the performance even with the cleaned labels. Therefore, our approach tackles the noisy label problem from a diffusion model learning perspective, providing an orthogonal direction compared to conventional noisy label methods.
***
[B1] Li, X., Liu, T., Han, B., Niu, G., & Sugiyama, M. (2021, July). Provably end-to-end label-noise learning without anchor points. In International conference on machine learning (pp. 6403-6413). PMLR.   
[B2] Li, Y., Han, H., Shan, S., & Chen, X. (2023). DISC: Learning from Noisy Labels via Dynamic Instance-Specific Selection and Correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 24070-24079).

Reviewer comment (Novelty):
**Unclear Advantage:** There is a need for a more detailed explanation regarding the advantage of the proposed method over existing methods, especially in recovering clean labels. Since the performance of diffusion models in noisy label scenarios hinges significantly on the accuracy of label recovery, explaining how the method enhances this aspect compared to others would greatly benefit readers in understanding the true potential and innovation of your approach.

Author response:
**[Advantage of the proposed method]**

Label recovery is an intuitive way to overcome the problem of noisy labels in a dataset. Our method does not specifically focus on robustness to noisy labels through label recovery; instead, it focuses on robustness to noisy labels from a diffusion model training perspective. Consequently, our approach can be synergistically combined with existing label recovery methods to further improve performance. The analysis of this synergy is detailed in Section 4.4.

It is important to analyze how our method overcomes the noisy label robustness in diffusion models. Our analysis indicates that our proposed TDSM objective provides the diffusion model with information about the diffused label noise in the dataset. This information is represented by the transition-aware weight function $w$, depending upon the diffusion time. Therefore, we estimate the instance- and time-dependent label transition probability with the transition-aware weight function.

**[Transition-aware weight function plays a different role from a typical classifier]**

We want to emphasize that the transition-aware weight function $w$ plays a different role from a typical classifier. This probability represents the relationship between the noisy and clean label conditional scores (which are also time-dependent) and is an important element of the proposed training objective of a diffusion model. Therefore, we need to estimate this label transition probabilities over diffusion timesteps.

Specifically, our transition-aware weight function can be reformulated as follows:
$$
w(x_t,\tilde{y},y,t)=\frac{p_t(Y=y|x_t)p(\tilde{Y}=\tilde{y}|Y=y)}{p_t(\tilde{Y}=\tilde{y}|x_t)}.
$$
As seen in the expression, the weight function is composed of the clean label classifier $p_t(Y=y|x_t)$, label transition prior $p(\tilde{Y}=\tilde{y}|Y=y)$, and the noisy label classifier  $p_t(\tilde{Y}=\tilde{y}|x_t)$. Given the triplet of perturbed data instance and its corresponding noisy label and timestep, i.e., $(x_t, \tilde{y}, t)$, each component of $w$ has the following characteristics with respect to $y$: 1) $p(Y=y|x_t)$ is maximized when $y$ is the clean label of the clean data $x_0$ of $x_t$; 2) $p(\tilde{Y}=\tilde{y}|Y=y)$ is maximized when $y$ is the noisy label $\tilde{y}$ in general. The reason for 2) is that in general, a given noisy label is sufficiently likely to be a clean label. These two trade-offs imply that the $w$ function does not behave like a clean label classifier.

Furthermore, for large enough $t$, the distribution of $x_t$ converges to a label-independent prior distribution by the design of the diffusion process, so that $p_t(Y|x_t)$ converges to a uniform distribution. Therefore, for sufficiently large $t$, the $w$ function converges to a transition matrix $S$. This phenomenon can also be demonstrated in Figure 2 of the 2-D example.

In summary, the transition-aware weight function needed to overcome noisy labels in a diffusion model is not represented clean label recovery information only, and this function has time-varying information.

Reviewer comment (Theory):
**Limited Application:** The focus on class-dependent noise settings might limit the broader applicability of the proposed method. The assumption of class-dependent noise could be strong and not verifiable in practical scenarios. It would be insightful if the paper could discuss the potential implications and limitations of this assumption, including how it might affect the generalizability of the proposed method to other noise settings or real-world applications where such assumptions may not be easily verified.

Author response:
In Section 4.3, we conducted experiments on the Clothing-1M dataset, which suffers from real-world label noise and does not adhere to the class-dependent noise assumption. Table 3 shows that our model even achieves superior performance on this dataset. While theoretical developments require assumptions about the noise setting, our model remains effective in real-world scenarios where these assumptions may not hold.

On the other hand, in the existing literature on noisy labels, there has been an extension from class-dependent label noise settings to instance-dependent label noise settings [B3, B4, B5, B6]. Especially in the context of learning diffusion models, there is a distinctive characteristic that, despite the class-dependent label noise assumption, the prediction of instance-dependent label transition probabilities is necessary. Therefore, investigating training methods under the current assumptions in the context of diffusion models lays the foundation for developing better methods. Similar to existing classification methods, extending our approach beyond the current assumption would be a valuable future work. 
***
[B3] Xia, X., Liu, T., Han, B., Wang, N., Gong, M., Liu, H., Niu, G., Tao, D., & Sugiyama, M. (2020). Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 33, 7597-7610.   
[B4] Berthon, A., Han, B., Niu, G., Liu, T., & Sugiyama, M. (2021, July). Confidence scores make instance-dependent label-noise learning possible. In International conference on machine learning (pp. 825-836). PMLR.   
[B5] Yao, Y., Liu, T., Gong, M., Han, B., Niu, G., & Zhang, K. (2021). Instance-dependent label-noise learning under a structural causal model. Advances in Neural Information Processing Systems, 34, 4409-4420.   
[B6] Cheng, D., Liu, T., Ning, Y., Wang, N., Han, B., Niu, G., Gao, X., & Sugiyama, M. (2022). Instance-dependent label-noise learning with manifold-regularized transition matrix estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16630-16639).

Reviewer comment (Evaluation):
Could the authors provide further details on how their method more effectively recovers clean labels compared to existing state-of-the-art methods VolMinNet (ICML 21), InstanceGM (WACV23), and DivideMix (ICLR20)?

Author response:
As we mentioned in the global response and Q3, our method does not focus on clean label recovery, which is the goal of existing methods for classification learning with noisy label such as VolMinNet, InstanceGM, and DividedMix, etc. Our method aims to overcome noisy labels that are characteristic of diffusion models. As a result, by combining our method with existing clean label recovery methods, we can further improve the robustness of the diffusion model to noisy labels. We demonstrated this in Section 4.4 of the manuscript, using the VolMinNet (Li et al., ICML 21) [B1] and DISC (Li et al., CVPR 23) [B2].

Reviewer comment (Experiments):
Would the authors consider adding additional experiments to demonstrate the performance when combining the existing state-of-the-art methods for learning with noisy labels with a conditional diffusion model? Additionally, could they compare the accuracy of clean label recovery with these methods?

Author response:
As we mentioned in the global response and Q3, our method does not focus on clean label recovery, which is the goal of existing methods for classification learning with noisy label such as VolMinNet, InstanceGM, and DividedMix, etc. Our method aims to overcome noisy labels that are characteristic of diffusion models. As a result, by combining our method with existing clean label recovery methods, we can further improve the robustness of the diffusion model to noisy labels. We demonstrated this in Section 4.4 of the manuscript, using the VolMinNet (Li et al., ICML 21) [B1] and DISC (Li et al., CVPR 23) [B2].

Reviewer comment (Evaluation):
Could the authors explain which specific mechanisms or features in their approach contribute to improved label recovery?

Author response:
**[Advantage of the proposed method]**

Label recovery is an intuitive way to overcome the problem of noisy labels in a dataset. Our method does not specifically focus on robustness to noisy labels through label recovery; instead, it focuses on robustness to noisy labels from a diffusion model training perspective. Consequently, our approach can be synergistically combined with existing label recovery methods to further improve performance. The analysis of this synergy is detailed in Section 4.4.

It is important to analyze how our method overcomes the noisy label robustness in diffusion models. Our analysis indicates that our proposed TDSM objective provides the diffusion model with information about the diffused label noise in the dataset. This information is represented by the transition-aware weight function $w$, depending upon the diffusion time. Therefore, we estimate the instance- and time-dependent label transition probability with the transition-aware weight function.

**[Transition-aware weight function plays a different role from a typical classifier]**

We want to emphasize that the transition-aware weight function $w$ plays a different role from a typical classifier. This probability represents the relationship between the noisy and clean label conditional scores (which are also time-dependent) and is an important element of the proposed training objective of a diffusion model. Therefore, we need to estimate this label transition probabilities over diffusion timesteps.

Specifically, our transition-aware weight function can be reformulated as follows:
$$
w(x_t,\tilde{y},y,t)=\frac{p_t(Y=y|x_t)p(\tilde{Y}=\tilde{y}|Y=y)}{p_t(\tilde{Y}=\tilde{y}|x_t)}.
$$
As seen in the expression, the weight function is composed of the clean label classifier $p_t(Y=y|x_t)$, label transition prior $p(\tilde{Y}=\tilde{y}|Y=y)$, and the noisy label classifier  $p_t(\tilde{Y}=\tilde{y}|x_t)$. Given the triplet of perturbed data instance and its corresponding noisy label and timestep, i.e., $(x_t, \tilde{y}, t)$, each component of $w$ has the following characteristics with respect to $y$: 1) $p(Y=y|x_t)$ is maximized when $y$ is the clean label of the clean data $x_0$ of $x_t$; 2) $p(\tilde{Y}=\tilde{y}|Y=y)$ is maximized when $y$ is the noisy label $\tilde{y}$ in general. The reason for 2) is that in general, a given noisy label is sufficiently likely to be a clean label. These two trade-offs imply that the $w$ function does not behave like a clean label classifier.

Furthermore, for large enough $t$, the distribution of $x_t$ converges to a label-independent prior distribution by the design of the diffusion process, so that $p_t(Y|x_t)$ converges to a uniform distribution. Therefore, for sufficiently large $t$, the $w$ function converges to a transition matrix $S$. This phenomenon can also be demonstrated in Figure 2 of the 2-D example.

In summary, the transition-aware weight function needed to overcome noisy labels in a diffusion model is not represented clean label recovery information only, and this function has time-varying information.

Reviewer comment (Experiments):
How do the authors verify the presence of class-dependent noise in practical applications?

Author response:
We believe that the class-dependent noise is still applicable in practical applications. One example is when classes are distinguished by fine-grained concepts such as breeds of dogs, species of birds, models of cars [B7, B8]. In this case, the distinction between classes is ambiguous, so that non-expert human labelers will have difficulty distinguishing between them, regardless of the data instances.
***
[B7] Krause, J., Sapp, B., Howard, A., Zhou, H., Toshev, A., Duerig, T., ... & Fei-Fei, L. (2016). The unreasonable effectiveness of noisy data for fine-grained recognition. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14 (pp. 301-320).   
[B8] Patrini, G., Rozza, A., Krishna Menon, A., Nock, R., & Qu, L. (2017). Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1944-1952).

Reviewer comment (Theory):
The proposed method is tailored for diffusion models which are conditioned on class, but most existing diffusion models are conditioned on text or other modalities, and the class label also can be expressed by language.

Author response:
We appreciate your feedback. We agree with the reviewer that recent diffusion model advancements have led to successful outcomes in text-conditional models, introducing notable models like Stable Diffusion and DALL-E3. However, it is important to note that class-conditional generative models have been widely implemented across various sectors of the industry: image editing [A1], medical data [A2, A3], and manufacturing [A4]. In fields where such models are essential, utilizing diffusion models, which have strong performances in generative tasks, could lead to improved results. Nevertheless, it is crucial to acknowledge that the problem of noisy labels in datasets continues to pose a significant challenge in real-world applications [A5, A6, A7].

One possible approach involves adapting publicly available text-conditional diffusion models for the purpose of class-conditional generation. However, in cases involving data privacy and label specificity, it may not be feasible to utilize the public text-conditional model. Therefore, it still becomes important to train a model from the incomplete dataset. Ongoing research on a class-conditional diffusion model that can effectively handle noisy labels from an application perspective is a significant endeavor. This research has the potential to solve a crucial problem in real-world scenarios and broaden the scope of diffusion models.

We would like to emphasize that there is no previous research that specifically addresses label noise in diffusion models. Much research on noisy label classification has extended from class-level studies to text-level or other modality-level studies [A8, A9]. Our work on the class-conditional is a necessary step to solidify the theoretical foundation on the noisy-image and noisy-text conditioned generation, which we believe that those issues will be published in the future.

Reviewer comment (Experiments):
Noisy label datasets are not common in diffusion model.

Author response:
First, we would like to mention the presence of noisy labels in the training datasets of diffusion models in practical situations. For example, the stable diffusion paper [1] mentions the use of the large-scale LAION dataset [2] in text-to-image generation experiments. Such large-scale datasets inevitably contain noisy labels introduced by human annotators or web crawlers during the data collection process. It has been reported that using pre-trained models on noisy labeled datasets can lead to performance degradation [3]. Therefore, we argue that label noise is also a significant problem for diffusion models, which are widely used in practical scenarios with large-scale datasets.

In this context, we claim that research considering noise in text labels builds on the foundation of noise in class labels. Text labels can be viewed as a continuous extension of the discrete class label $Y$, i.e., replacing the label space $\mathcal{Y}=${$1,…,c$} with $\mathbb{R}^m$. Consequently, the label transition probability $p(Y=y|\tilde{Y}=\tilde{y})$ can be extended from a probability mass function to a probability density function. Therefore, while our current research is based on class conditions, we are the first to consider the impact of label noise in diffusion models and to propose a method specific to diffusion models, and we argue that our study will serve as a foundation for extensions such as noisy-text conditional generation.