From: ICLR.cc/2023/Conference/Paper6473/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper6473/Reviewer_FCAb
title: Respond to the authors
comment: Thank you for carefully responding to my concerns and questions.
The paper has been improved significantly!

## Comments

* Dividing Table 3 into two will make the paper better, which clarifies the intent that WebVision and ILSVRC2012 are test sets while Clothing1M-I and Clothing1M-II are training sets.

From: ICLR.cc/2023/Conference/Paper6473/Authors
title: Response to Reviewer APd8
comment: Thank you very much for your careful review and constructive comments. We will try our best to answer all your concerns.

----

**Q1:** "The main weakness is the empirical evaluation. The method is only verified on three noisy datasets. It shows good improvements on two of them and none on the third."

**Response:** More experiments have been updated in the paper. Specifically, experiments are conducted on the (mini) WebVision dataset and evaluated on WebVision and ILSVRC12 validation set, as shown in Table 3.  Besides, Synthetic label noises (including uniform, symmetric, asymmetric, and class-dependent non-uniform noise etc.) on CIFAR-10 dataset have been evaluated Table 4 and 5. Empirical results indicate that our method performs well on different datasets and different noise types.



---

**Q2:** "Furthermore, it would have been interesting to further explore various distributional scenarios (eg varying skewness) of non-uniform label distributions both theoretically and empirically through simulations on synthetic data."

**Q3:** "The evaluation is limited to images, but it seems the method would generally apply (e.g., to tabular data)!?"

**Response:** As shown in Table 4 and 5, the proposed method and baseline methods have been evaluated on synthetic label noises with various distributional scenarios, including uniform, non-uniform, symmetric and asymmetric noise etc. Empirical results indicate that our method performs well with various distributional scenarios. Besides, we agree it is interesting to apply our method to more other forms of data and we would like to explore this in future work. Due to the limitation of time, tabular data (and language datasets) would be added in the final version if the paper is accepted.

---

**Q4:** "My final critique is more of a question to the authors: It seems the method could be more generally abstracted (could it?), but this route is not explored by the authors."

**Response:** Thanks for the suggestion. The abstraction of the proposed method would be an interesting question and need more exploration in our future work. The visualization in Figure 7 may give an intuitive illustration for the proposed cluster-dependent sample selection strategy, which actually adds another dimension in the clean sample selection process.

From: ICLR.cc/2023/Conference/Paper6473/Authors
title: Response to Reviewer FCAb
comment: Thank you very much for your careful review and constructive comments. We will try our best to answer all your concerns.

----

**Q1:** "The validity of the proposed noise model is not explained enough and confirmed."

**Response:** As shown in Figure 2 and Figure 3 (c), the label noise rate varies greatly among the data space. Therefore, the cluster-dependent noise is proposed to hold with the variability existing in the realistic noisy datasets. Compared with the previous noise model, the proposed cluster-dependent label noise (CluDN) makes a trade-off between the class- (CDN) and instance-dependent noise (IDN). The CluDN model can deal with the non-uniform noise better than the CDN model while keep much lower complexity than IDN model.

In practice, the proposed method treat samples in different clusters with different evaluation criteria, which based on the idea of  stratified sampling. As shown in Figure 7, compared with the naive small-loss trick, the proposed method sets different criteria for different clusters and thus can select clean samples more precisely.

----

**Q2:** "The experiments are limited; only CIFAR-N datasets and Clothing 1M with ResNet. Can ClusterMix alleviate other types of noise, such as uniform noise?"

**Response:** More experiments have been updated in the paper. Specifically, experiments are conducted on the (mini) WebVision dataset and evaluated on WebVision and ILSVRC12 validation set, as shown in Table 3. Besides, Synthetic label noises (including uniform, symmetric, asymmetric, and class-dependent non-uniform noise etc.) on CIFAR-10 dataset have been evaluated Table 4 and 5. Empirical results indicate that our method performs well on different datasets and different noise types.

---

**Q3:** "Figure 1 needs to include the contents in Figure 5 in the appendix; otherwise, it is unclear if the proposed method can handle this toy problem."

**Q4:** "I'm afraid if the definition of $\tilde{D}_{clean}$ is correct. $\hat{y}_n = y_n$ rather than $\hat{y}_n = \tilde{y}_n$ ?"

**Q5:** "I guess Lines 9 and 10 of Algorithm 2 need to be swapped."

**Response:** Really thanks for your careful checking for our paper. The mentioned details have been carefully revised in the new version.

From: ICLR.cc/2023/Conference/Paper6473/Authors
title: Response to Reviewer wj2B
comment: Thank you very much for your careful review and constructive comments. We will try our best to answer all your concerns.

---

**Q1:** "Based on the Figure 4, the ClusterMix share the same framework with DivideMix, and the only technical contribution is the cluster-based sample selection strategy. Such a contribution is incremental,"

**Response:** We agree that the key component of our solution is the cluster-based sample selection strategy. While the final solution is simple, what leads to it is the realization of the potential of the cluster-dependent noise model, which is conceptually different from existing class-dependent and instance-dependent models, and which we consider as one of our contributions. The class-dependent model has a coarse granularity and is not flexible enough to capture noise variation in different parts of the feature space. The instance-dependent model is very flexible but suffers the non-identifiability problem without extra information. Our cluster-dependent noise model works at an intermediate granularity, which provides a trade-off between flexibility and identifiability. Furthermore, our cluster is based on proximity in the latent space rather than class labels.

---

**Q2:** "The cluster-based sample selection strategy is the most important module in this paper, but I do not find the ablation study to evaluate effectiveness. In addition, this sample selection strategy should be able to use in any label noise learning methods with sample selection, and more experiments about this are needed. You should show how much improvements can ClusterMix bring over naive small-loss trick."

**Response:** Thanks for the suggestions. A group of contrast experiments have been updated in Table 8, Appendix H. In the experiments, we combine the naive/cluster-dependent sample selection with two different training mechanisms (i.e. supervised learning and semi-supervised learning). The experimental results validate the effectiveness and applicability of the proposed method.

- Effectiveness. Compared with the naive sample selection, the proposed cluster-based sample selection strategy can improve the performance both in the supervised and semi-supervised mechanisms. Besides, the proposed method plays a greater role when the training samples are inadequate.
- Applicability. The proposed cluster-based sample selection strategy can be used to two label noise learning methods (one is supervised and the other is semi-supervised). Both of methods can improve the performance over the naive sample selection.

---

**Q3:** "I find that ClusterMix cannot outperform DivideMix on the task 1 of Clothing1M. This phenomenon is abnormal and more analysis is needed to justify the effectiveness of cluster-based sample selection."

**Response:**  This phenomenon occurs reasonably due to the size of training set. As the empirical results on CIFAR-10 dataset (shown in Table 2), the proposed cluster-dependent sample selection strategy  brings a larger gain for a smaller training set. In contrast, the Clothing1M dataset has 1 million training samples which is a pretty huge number.  In the large dataset, the selected  clean samples can be redundant for the data space while the clustering algorithm would be limited by the number of data points. To verify the analysis, we conduct experiment task II on the Clothing1M dataset with limited training samples (5000 in practice). The experimental results show that our method performs the baseline methods again with the limited training samples.

---

**Q4:** "Some typos. For example, (1) in the line 8 of Abstract, "we propose...". (2) In the line 5 of Page 3, "inally,...". (3) Many reference formatting errors."

**Response:** Really thanks for your careful checking for our paper.  The mentioned details have been carefully revised in the new version.

From: ICLR.cc/2023/Conference/Paper6473/Authors
title: Response to Reviewer AjzT - Experiment Part.
comment: For the experiment part:

---

**Q3:** "Another concern is the experimental parts. The paper did not show the learning/test curve, or the standard deviation. Compared to methods such as SOP, the performance is actually quite close. "

**Response:** We would like to thank the reviewer for the constructive comments. We added the accuracy curves and standard deviation for CIFAR-10N in Figure 8, Appendix D. As shown in Table 2 and Figure 8, our method performs slightly better than the state-of-the-art baselines with all the 50000 training samples in CIFAR-10N worst label dataset. But when the training samples decrease to a relatively small amount, the gap between our method and baseline methods becomes substantially large. That is, our proposed method performs better when the training samples are inadequate. A reasonable explanation is that precise and adequate clean samples are more needed with limited training samples.

---

**Q4:** "Actually, one concern would be on the small-loss trick part. Since different clusters will have different noise ratios, as assumed in the paper, I think setting the percentage of data as clean or noisy in each cluster could be critical in determining the final performance. Are there any guides on how to set these hyperparameters？"

**Response:**

- For the small-loss trick,  an illustration is given in Figure 7, Appendix C. Compared with the naive small-loss trick, the proposed cluster-dependent sample selection strategy adds another dimension and allows the different clean threshold among different clusters.
- For the setting of the percentage of data as clean or noisy in each cluster, we use a 2-components Gaussian Mixture Model to fit the losses in each cluster and regard the component with lower mean value as the clean part. Then the clean probability (the probability belong to the clean component) of sample i is denoted as $\omega_i$. Finally, we just treat the samples with  $\omega_i>=0.5$ as the clean ones and otherwise noisy ones. There is no hyperparameter for the sample selection part other than this natural threshold of 0.5. The related contents have been discussed in Section 3.3.
- Besides the experiments in the original version, we show more experiment results on different datasets and noisy types in the revised draft. Specifically, Table 1, 2, and 3 for real noisy datasets, Table 4 and 5 for synthetic labels, Table 6 and 7 for sensitivity analysis, and Table 8 for contrast experiments of the proposed method.

From: ICLR.cc/2023/Conference/Paper6473/Authors
title: Response to Reviewer AJzT - Noise Model Part.
comment: Thank you very much for your careful review and constructive comments. We will try our best to answer all your concerns.

---

For the noise model part:

----

**Q1:** "The biggest concern may be on the cluster-dependent noise part. First, this is not a new assumption. It is a special case of class-dependent noise and has been considered in previous works' experiments, for example, Han et al. 2018, pairwise noise."

**Response:** We apologize for the confusion. The proposed cluster-dependent noise is conceptually different from class-dependent noise.  

On one hand, clusters are grouped by the feature similarity of data points, without any reference to the class information (neither noisy nor true labels). Thus, a class can contain multiple clusters and a cluster can cross class boundaries (i.e. contain points from different classes), as shown in Figure 3 of the revised draft.

On the other hand, the cluster-dependent noise model in our paper is at a different granularity than class-dependent noise model. The class-dependent noise model (e.g. in Han et al. 2018 and Berthon et al. 2021) assumes a constant noise rate within each class, which is at a much coarser granularity (since we have relatively few number of classes) than our cluster-dependent model and does not capture the variability within a class (see Figure 2, 3 and Appendix B of the revised draft). To the other extreme, the instance-dependent noise model has the finest granularity but suffers the non-identifiability problem without extra information (Liu et al. 2022). Our cluster-dependent noise model works at an intermediate granularity, which provides a trade-off between flexibility and identifiability. 

Actually the group-dependent noise model (Wang et al. 2021) is closer to our cluster-dependent model. In the group-dependent model, data points in the same class are grouped by the associated “sensitive information" (e.g. by age, gender, or race) and share different noise rate. However, it only uses a few hand-picked input features to group data and still works at a pretty coarse granularity, though finer than class-dependent model. In contrast, our model can have a much larger number of clusters at a finer granularity. Furthermore, the grouping is performed by clustering in the latent feature space without need for hand-picked group information.

----

**Q2:** "However, a more critical concern regarding this assumption is whether it holds with the "non-uniform" case argued earlier in this paper."

**Response:** Regarding to whether the assumption holds, we think there are several aspects here. First, as discussed in our response to Q1, cluster boundaries are in general different from class boundaries, and many cluster boundaries are actually in the interior of a class, so points on the boundary of a cluster are not necessarily more difficult than those in the interior of a cluster. Second, the assumption of uniform noise rate within a cluster is an approximation that strike a balance between flexibility and identifiability. Ideally we would like to use instance-dependent noise, but that's hard without extra information. Third, our cluster is based on similarity in latent space. It seems reasonable to assume that points with similar latent representations have similar difficulty and hopefully similar noise rate.




**References.**

*Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., ... & Sugiyama, M. (2018). Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31.*

*Berthon, A., Han, B., Niu, G., Liu, T., & Sugiyama, M. (2021, July). Confidence scores make instance-dependent label-noise learning possible. In International Conference on Machine Learning (pp. 825-836). PMLR.*

*Liu, Y. (2022). Identifiability of Label Noise Transition Matrix. arXiv preprint arXiv:2202.02016.*

*Wang, J., Liu, Y., & Levy, C. (2021, March). Fair classification with group-dependent label noise. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (pp. 526-536).*

From: ICLR.cc/2023/Conference/Paper6473/Reviewer_AJzT
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a method for non-uniform label noise. By stating that existing methods mostly are based on uniform label noise assumption, i.e, the noise rate of different data is the same, the paper makes a cluster-dependent label noise assumption: data are clustered, and different clusters of data have different noise rates. The paper then uses the small-loss trick to specify the clean and noisy samples within each cluster and adopts a DivideMix style algorithm to learn from the semi-supervised data. The paper finally demonstrates experimental results on three real datasets. 
strength_and_weaknesses: The paper made a new assumption on how the noise is generated. The assumption is different from the instance-dependent noisy assumption. However, it seems to be a class-dependent noise assumption considering the illustrated example in Figure 3. If the assumption holds well, the proposed method is technically correct and simple, although it is a direct combination of several existing methods of handling noisy labels: DivideMix and those based on the small-loss trick.

The biggest concern may be on the cluster-dependent noise part. First, this is not a new assumption. It is a special case of class-dependent noise and has been considered in previous works' experiments, for example, Han et al. 2018, pairwise noise. However, a more critical concern regarding this assumption is whether it holds with the "non-uniform" case argued earlier in this paper. Previously, the paper argues that the generation of non-uniform noise is because some of the instances are "difficult" and thus they have a high noise rate. However, I do not think data points in one cluster will have similar difficulty levels. Most of the cases, data points on the boundary of a cluster are considered as difficult compared to data points in the middle of a cluster. The assumption made in the paper seems to be incorrect, or at least inconsistent with the previous arguments. 

Another concern is the experimental parts. The paper did not show the learning/test curve, or the standard deviation. Compared to methods such as SOP, the performance is actually quite close. Actually, one concern would be on the small-loss trick part. Since different clusters will have different noise ratios, as assumed in the paper, I think setting the percentage of data as clean or noisy in each cluster could be critical in determining the final performance. Are there any guides on how to set these hyperparameters？
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow. The performance seems good but may need more reported data before determining its significance. The given assumption could be a concern for the paper. Given the assumption, the proposed method, although is not novel, but simple and correct. I have not seen any concern about reproducibility yet. 
summary_of_the_review: The correctness of the assumption made is a concern. The proposed method is straightforward but not novel enough. The paper may need more experiments to demonstrate its effectiveness. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6473/Reviewer_wj2B
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper aims to address non-uniform label noise learning, and reveal that the small-loss trick does not maintain the consistency of data distribution. Therefore, they propose the CluterMix, which combines the cluster-dependent sample selection method with a semi-supervised learning mechanism to distinguish and leverage the corrupt labels.
strength_and_weaknesses: Pros:
1. This paper proposes to treat the non-uniform label noise as group-dependent label noise based on the thought of divide-and-conquer. This assumption is interesting and significant.

Cons:
1. Based on the Figure 4, the ClusterMix share the same framework with DivideMix, and the only technical contribution is the cluster-based sample selection strategy. Such a contribution is incremental, 
2. The cluster-based sample selection strategy is the most important module in this paper, but I do not find the ablation study to evaluate  effectiveness. In addition, this sample selection strategy should be able to use in any label noise learning methods with sample selection, and more experiments about this are needed. You should show how much improvements can ClusterMix bring over naive small-loss trick. 
3. I find that ClusterMix cannot outperform DivideMix on the task 1 of Clothing1M. This phenomenon is abnormal and more analysis is needed to justify the effectiveness of cluster-based sample selection. 
4. Some typos. For example, (1) in the line 8 of Abstract, "we propose...". (2) In the line 5 of Page 3, "inally,...". (3) Many reference formatting errors.

Maybe I have some misunderstandings about the technical novelty. Please explain this problem in detail, and I am willing to raise my score.
clarity,_quality,_novelty_and_reproducibility: There exists many writing typos and the novelty seems a little bit limited.
summary_of_the_review: Please refer to Strength And Weaknesses.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper6473/Reviewer_FCAb
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper considers the cluster-dependent label noise model, where data can be divided into some clusters, and each cluster has its own label noise.
Based on this idea, they proposed ClusterMix, a method for learning from noisy labels.
This method extends CoTeaching by dividing training data into clusters and using GMM to split each cluster to possibly clean and corrupt sets.
Then, the label of the corrupt set is ignored, and the MixMatch-based semi-supervised learning algorithm trains learners.
They empirically confirmed the performance on CIFAR-10/100N and Clothing 1M and reported improvement especially when the number of training data is limited.

strength_and_weaknesses: Strength

* This paper introduces the cluster-dependent label noise model, which seems to be a realistic assumption of label corruption.
* The cluster-dependent sample selection is general and can be used with other methods.
* The experiments confirmed the method is effective especially when the number of training data is limited. Although I am not sure if this is practically appealing, as one can re-annotate by themselves for small data.

Weakness

* The validity of the proposed noise model is not explained enough and confirmed. 
* The experiments are limited; only CIFAR-N datasets and Clothing 1M with ResNet. Can ClusterMix alleviate other types of noise, such as uniform noise?
clarity,_quality,_novelty_and_reproducibility: Clarity

* Figure 1 needs to include the contents in Figure 5 in the appendix; otherwise, it is unclear if the proposed method can handle this toy problem.
* I'm afraid if the definition of $\tilde{D}_{\mathrm{clean}}$ is correct. $\hat{y}_n=y_n$ rather than $\hat{y}_n=\tilde{y_n}$?
* I guess Lines 9 and 10 of Algorithm 2 need to be swapped.

Quality

The paper is well written.

Novelty

A new noise model and a method to alleviate this issue is proposed.

Reproducibility

The experimental settings are described.
summary_of_the_review: The paper is generally well-written.
However, I want to know more about how the proposed method is general.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6473/Reviewer_APd8
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper contributes with a new method for classification from non-uniform label noise. The method is based on a cross-entropy neural network.
strength_and_weaknesses: Strengths:
- The problem is relevant, and the work contributes with an innovative method, which shows empirical promise. 
- The technical detail in the methodology is sufficiently high. 
- The quality of the datasets in the empirical evaluation is high. (They are taken from other work, including the noise generation scheme. This is contrast to many other papers, which generate themselves the noisy labels.)

Weaknesses:
- The main weakness is the empirical evaluation. The method is only verified on three noisy datasets. It shows good improvements on two of them and none on the third. 
- Furthermore, it would have been interesting to further explore various distributional scenarios (eg varying skewness) of non-uniform label distributions both theoretically and empirically through simulations on synthetic data. 
- The evaluation is limited to images, but it seems the method would generally apply (e.g., to tabular data)!? 

My final critique is more of a question to the authors: It seems the method could be more generally abstracted (could it?), but this route is not explored by the authors.
clarity,_quality,_novelty_and_reproducibility: The paper is - besides the occasional typo or grammatical mistake) - relatively well written and structured, and it is easy to read. The information on the experimental parameters is insufficient to reproduce the experiments. The code has not been uploaded as a supplement. The approach is novel and innovative to my knowledge.
summary_of_the_review: An innovative approach to learning with non-uniform label noise, with some weaknesses in abstraction of the method and the depth of the analysis, especially on the empirical side.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold