Paper: Early Stopping Against Label Noise Without Validation Data (ICLR-2024)

Reviewer comment (Experiments):
Since the metic is moveing average over k epochs, sensitivity analysis over k is needed.

Author response:
**Question 3 - PC Fluctuations & Sensitivity Analysis over k**
Here, we address both Weakness 3 and Question 3. Based on your constructive suggestions, we have conducted additional analysis.

1. Monotonicity of PC Values:
You inquired about the fluctuations in PC values and suggested setting some thresholds. We fully endorse your suggestion that applying a specific threshold to label wave in practice would further enhance the robustness of our method. In our existing experiments, we have found that Moving Averages and "Patience" are sufficient to accurately help us identity the appropriate early stopping point through PC. Therefore, to ensure the simplicity and straightforward of our proposed method, we did not add an additional threshold in this paper.

2. Sensitivity Analysis of k Values:
In response to your suggestion, we conducted a sensitivity analysis on the k value for Moving Averages. Specifically, we analyzed the correlation between the PC after applying different k values of moving averages and test accuracy:

| k   | Pearson Correlation Coefficient |
|-----------|-----------------------|
| 1        |      -0.8565      |
| 2        |      -0.9435      | 
| 3        |      -0.9637      |
| 5        |      -0.9367      |
| 10        |      -0.9406     |

(Experiment take the same settings as Sec. 4.1 and Appendix B)

We observed that as the k value changes, the Pearson correlation coefficient between the moving average of PC values and test accuracy exhibits some fluctuations but overall maintains a very strong negative correlation. This suggests that while considering the setting of k values is necessary, different k values do not significantly alter the relationship between the moving average of PC values and test accuracy. We will provide detailed sensitivity analysis charts in the further revision

Reviewer comment (Experiments):
It may not work well for extreme cases, e.g. class imbalanced dataset. Any solution for those settings? If not, some assumptions could be specified for the setting.

Author response:
**Question.1 - Class imbalanced dataset**
We have tested the performance of our method on class imbalanced datasets, by setting the Imbalance Factor to 0.1 and the Noise Ratio (Sym.) to 0.4, with other settings consistent with those described in Section 4.1. Our experiments were conducted using our method in Cross-Entropy (CE) and class imbalanced method LDAM [1].

| Methods   | Label Wave | Global Maximum |
|-----------|-----------------------|---------------------------|
| CE        |      55.31±0.86%      |       56.03±0.55%         |
| LDAM   |      61.07±0.88%      |      61.80±0.93%         |

The experimental results affirm that our method can effectively identify early stopping points when learning from class imbalanced noisy datasets. While we emphasize the effectiveness of the Label Wave method in such scenarios, we acknowledge, as your intuition, that it may not perform equally well in more extreme cases. For example, when applied to the CIFAR-100 training set with an Imbalance Factor of 0.01 and a Sym. Noise Ratio of 0.4, the model's test accuracy exhibited significant fluctuations, leading to the ineffectiveness of the Label Wave method. We wish to clarify that our method is intended to identify appropriate early stopping points; thus, scenarios where early stopping itself is not applicable fall outside the scope of our research.

[1] Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, NeurIPS 2019.

Reviewer comment (Novelty):
What is the difference between the previouse studies'[1,2] findings and what authors propose in section 3.2 (fitting mislabeled examples impairs the overall model’s fitting performance)?

Author response:
**Question.2 - Difference between the previous studies**
Our research, along with the findings of [1, 2], aims to deepen the understanding of how a model's performance is impacted by fitting incorrectly labeled examples. However, we believe that these studies contribute to this understanding in different ways.

Both [1] and [2] have introduced new methods to handle noisy labels based on their findings, and thus have made notable contributions to the field. Specifically, [1] discusses the effects of Label Smoothing (LS) and Negative Label Smoothing (NLS) in the context of label noise, mainly focusing on how to adjust label smoothing strategies. [2] tackles instance-dependent label noise, and concentrates on effectively sieving out wrongly labeled instances from the training data.

While [1] and [2] have significantly contributed to the field by exploring how noisy labels affect a model's generalization performance; our research distinctly points out how fitting noisy labels can impact the model's fitting performance on the training set. In Section 3.2, we delve deeper into the specific effects of fitting mislabeled examples on the overall fitting performance on the training set. Our study, by designing metrics called ''stability'' and ''variability'', discusses how the model's fitting performance on the training set is influenced by fitting incorrectly labeled examples as the training progresses. Moreover, we highlight a transitional phase, which we term ''learning confusion patterns''. We believe that this perspective has not been fully explored in previous studies.

[1] Wei, J., Liu, H., Liu, T., Niu, G., Sugiyama, M., & Liu, Y. (2022, June). To Smooth or Not? When Label Smoothing Meets Noisy Labels. In International Conference on Machine Learning.

[2] Cheng, H., Zhu, Z., Li, X., Gong, Y., Sun, X., & Liu, Y. (2020, October). Learning with Instance-Dependent Label Noise: A Sample Sieve Approach. In International Conference on Learning Representations.

Reviewer comment (Theory):
Will PC monotonically decrease before its local minima? In other words, are there no fluctuations? or should we need some threshold?

Author response:
**Question 3 - PC Fluctuations & Sensitivity Analysis over k**
Here, we address both Weakness 3 and Question 3. Based on your constructive suggestions, we have conducted additional analysis.

1. Monotonicity of PC Values:
You inquired about the fluctuations in PC values and suggested setting some thresholds. We fully endorse your suggestion that applying a specific threshold to label wave in practice would further enhance the robustness of our method. In our existing experiments, we have found that Moving Averages and "Patience" are sufficient to accurately help us identity the appropriate early stopping point through PC. Therefore, to ensure the simplicity and straightforward of our proposed method, we did not add an additional threshold in this paper.

2. Sensitivity Analysis of k Values:
In response to your suggestion, we conducted a sensitivity analysis on the k value for Moving Averages. Specifically, we analyzed the correlation between the PC after applying different k values of moving averages and test accuracy:

| k   | Pearson Correlation Coefficient |
|-----------|-----------------------|
| 1        |      -0.8565      |
| 2        |      -0.9435      | 
| 3        |      -0.9637      |
| 5        |      -0.9367      |
| 10        |      -0.9406     |

(Experiment take the same settings as Sec. 4.1 and Appendix B)

We observed that as the k value changes, the Pearson correlation coefficient between the moving average of PC values and test accuracy exhibits some fluctuations but overall maintains a very strong negative correlation. This suggests that while considering the setting of k values is necessary, different k values do not significantly alter the relationship between the moving average of PC values and test accuracy. We will provide detailed sensitivity analysis charts in the further revision

Reviewer comment (Experiments):
How will this pattern change when utilized with additional regularizations e.g. data augmentation? Will it be consistent or will it flutuate before it goes to local minima?

Author response:
**Question.4 - Additional regularizations**
Based on the reviewers' detailed comments, we tested the effectiveness of our Label Wave method with additional regularization methods. It is worth noting that our proposed Label Wave method itself includes BN and Data Augmentation, i.e., Label Wave + B, D as described below. Beyond the reviewer's comments, we have also tested the setting where these two regularization methods are not included. These regularization methods include:

*A. Mixup
B. BN
C. Dropout
D. Data Augmentation.*
(All experiments take the same settings as Sec. 4.1)

| Methods   | Label Wave | Global Maximum |
|-----------|-----------------------|---------------------------|
| Label Wave        |       66.79±0.39%      |    67.15±0.49%  |
| Label Wave + B, D |  81.61±0.44%    |    81.76±0.30%    |
| Label Wave + B, C, D     |   83.57±0.24%    |   83.77±0.32%    |
| Label Wave + A, B, D      |   82.38±0.56%      |     83.09±0.22%      |
| Label Wave + A, B, C, D  |83.67±0.45%|  84.05±0.35%     |

The experimental results show that adding or subtracting these regularization methods in the experimental setting does not affect the effectiveness of the label wave method for identifying early stopping points.

Reviewer comment (Experiments):
For Table 1 and Table 2, which algorithm is utilized (just Cross Entropy?)? If utilized with several algorithm managing noisy labels, how much different between best and label wave?

Author response:
**Question.5 - LNL algorithms**
Yes, Tables 1 and 2 evaluate the effectiveness of the label wave method when using Cross-Entropy Loss under various setting modifications. 
In Section 4.2, Tables 3 and 4 examine a range of algorithms for managing noisy labels. The differences between the maximum test accuracy and the label wave outcomes are shown below:

| Methods   | CIFAR10 - Label Wave | CIFAR10 - Global Maximum |
|-----------|-----------------------|---------------------------|
| CE        |        81.61±0.44%               |81.76±0.30% |
| Taylor-CE |        85.06±0.30%               |85.43±0.37%     |
| ELR       |        90.45±0.52%               |90.76±0.70% |
| CDR       |        87.69±0.10%               |87.80±0.24% |
| CORES     |        87.74±0.13%               |87.95±0.21% |
| NLS       |        83.45±0.19%               |83.62±0.37%     |
| SOP       |        88.42±0.38%               |88.82±0.46%     |


| Methods   | CIFAR100 - Label Wave | CIFAR100 - Global Maximum |
|-----------|-----------------------|---------------------------|
| CE        |        50.96±0.30%    |  51.05±0.33%   |
| Taylor-CE |        57.64±0.28%    |  57.99±0.30%   |
| ELR       |        65.36±0.39%    |  66.33±0.93%   |
| CDR       |        63.34±0.15%    |  63.54±0.28%   |
| CORES     |        45.03±0.38%    |  45.75±0.27%   |
| NLS       |        58.05±0.15%    |  58.32±0.35%   |
| SOP       |        68.53±0.30%    |  68.78±0.27%   |

Reviewer comment (Experiments):
Want to see result on more noise condition for table 3 and 4.

Author response:
**Question.6 - More noise condition for table 3 and 4**
Due to computational resource constraints, we will provide results with more noise condition for Tables 3 and 4 in subsequent Comments and further revision.

Reviewer comment (Experiments):
How about on real noise, e.g. Clothing1M?

Author response:
**Question.7 - Real noise**
We respectfully wish to highlight that our testing on the CIFAR-10N [1] dataset involved human-annotated real-world noise, showcasing the Label Wave method's effectiveness in real-world scenarios. Here, we present the outcomes of applying our label wave method on the Clothing1M dataset. 

| Methods   | Label Wave | Global Maximum |
|-----------|-----------------------|---------------------------|
| CE        |      70.12±0.34%      |       70.56±0.11%         |

[1] Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., & Liu, Y. (2021, October). Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations.

Reviewer comment (Experiments):
Will this criterion fit to another task, e.g. semantic segmentation?

Author response:
**Question.8 - Semantic segmentation**
Considering that our aim is to demonstrate that the concept of "learning confusion patterns" helps detect the transitional point in learning with noisy labels, we did not consider tasks like semantic segmentation and focused solely on classification tasks.


We extend our sincere thanks for your patience in reviewing our work and deeply appreciate your informative comments towards improving our manuscript.

Reviewer comment (Experiments):
Regarding applicability, my intuition is that the method only works well when there is “significant” amount of “random” label noise in the training set. “Random” is because the method relies on that the model fits simple patterns first and then learn random patterns from the noise. What if the label noise also only include simple patterns? E.g. black donkeys are mostly labeled as horses? The method also requires significant amount of label noise so that the model prediction fluctuate to a degree to be detected by the method. This is also reflected by the fact that experiments only considers >20% noise. Can the authors provide more insights into this through discussion or experiments?

Author response:
We appreciate the insightful concerns you have expressed about the applicability of the Label Wave method, particularly in scenarios with different types and amounts of label noise. We will detailedly address these concerns below in Part A - Types of Label Noise, and Part B - Amounts of Label Noise.

*Part A: Types of Label Noise*

1. Broad Applicability Across Noise Types: Our method is effective not only in the presence of random label noise but also in more complex scenarios. We have conducted extensive experiments to prove our method's applicability to both symmetric noise and instance-dependent label noise [1], as well as in real-world environments with label noise [2, 3] (refer to Section 4.1, Tables 1 and 2). These experiments demonstrate that our method is capable of effectively identifying early stopping points across various types of label noise studied in the community of learning with noisy labels.
2. Limitations in Specific Noise Types: While we highlight the effectiveness of the Label Wave method, it is consistent with your intuition that our method may not work equally well across all types of label noise. Recent research [4] highlights a type of noise known as subclass-dominant label noise, where early stopping has proven to be ineffective. We wish to clarify that our method is designed to identify appropriate early stopping points; hence, scenarios where early stopping itself is inapplicable fall outside the scope of our research.

*Part B: Amounts of Label Noise*

1. Effectiveness in Low Noise Scenarios: We would like to emphasize that our method remains effective in training sets with noise levels below 20%. Our experiments show that the method works well even with a noise rate as low as 10%. The choice to experiment with noise rates fixed between 20%-80% was based on the common range of dataset noise discussed in the research of learning with noisy labels.
2. Limitations at Very Low Noise Levels: However, we recognize that the concerns raised about the method's effectiveness at extremely low noise levels are valid. Our primary goal in this paper is to demonstrate how the concept of "learning confusion patterns" aids in detecting the transitional point in learning with noisy labels. To maintain simplicity, intuitiveness, and computational efficiency, we used the variability metric explored in Section 3.1, termed as prediction changes or PC (Section 3.3). Designing metrics that track the fluctuations of the model’s fitting performance on a broader range can be an effective way to improve our work. One initial intuition is to discuss the degree of fluctuations in prediction changes, which could help identify the early stopping point even when there is no obvious first local minimum in prediction changes.
3. 0% Label Noise: We have noted the suggestion to discuss scenarios where label noise is zero. However, as discussed in our paper, our method is not applicable to typical clean training sets. In cases of completely clean datasets, modern deep neural networks often exhibit benign overfitting [5, 6], making early stopping unnecessary. Scenarios where early stopping itself is inapplicable are beyond the scope of our method's intended use.

[1] Part-dependent label noise: Towards instance-dependent label noise, NeurIPS 2020.
[2] Learning with noisy labels revisited: A study using real-world human annotations, ICLR 2022.
[3] Learning from Massive Noisy Labeled Data for Image Classification, CVPR 2015.
[4] Subclass-Dominant Label Noise: A Counterexample for the Success of Early Stopping, NeurIPS 2023.
[5] Benign overfitting in linear regression, PNAS 2020.
[6] Benign overfitting in classification: Provably counter label noise with larger models, ICLR 2023.

Reviewer comment (Experiments):
Only datasets with synthetic noise are considered. How does the method work on real datasets with real-world noise?

Author response:
We respectfully wish to highlight that our testing on the CIFAR-10N dataset involved human-annotated real-world noise, showcasing the Label Wave method's effectiveness in real-world scenarios. Furthermore, our experiments conducted on the NEWS and Tiny-ImageNet datasets have demonstrated our method's broad applicability across various domains. Here, we present the outcomes of applying our Label Wave method on the Clothing1M dataset. We are committed to supplementing more results using real-world datasets in the further revision.

| Methods   | Label Wave | Global Maximum |
|-----------|-----------------------|---------------------------|
| CE        |      70.12±0.34%      |       70.56±0.11%         |

Reviewer comment (Experiments):
The baselines seem to use a noisy validation set, and evaluation is done on a clean test set. It makes more sense to split the clean test set to create a clean validation set or to create a clean validation set from the training set. This is because we want to ensure the validation set has a same distribution as the test set, i.e. to be clean.

Author response:
We really appreciate your wonderful suggestion about using a clean validation set to align with a clean test set distribution. However, in the context of our research on learning with noisy labels, we have intentionally chosen to maintain consistency between the distributions of the validation and training sets. This decision is grounded in the following considerations:

1. *Universality of Label Wave*: Our proposed method for identifying early stopping points using the Label Wave mthods aims to offer a universally applicable solution for learning with noisy labels. A prevalent assumption in this field, as reflected in numerous studies [7-10], is the presence of a training set with unreliable labels alongside a test set with reliable labels. In these cases, the training set is often divided into two parts: one for training and the other serving as a noisy validation set for early stopping. By adhering to this widely accepted setting, our method aligns with and can be directly compared with the method that utilizes a noisy validation set for identifying early stopping points.
2. *Label Wave in Real-World Applications*: In many real-world applications, it is common to encounter datasets where all available data is unreliable or noisy. This means that we cannot obtain an additional clean validation set for identifying early stopping points. Therefore, we only compare methods that utilize a noisy validation set for identifying early stopping points, as this reflects a more realistic and practical situation.

In conclusion, we use a noisy validation set to ensure a fair and relevant evaluation of our method within the specific context of learning with noisy labels.

[7] Early-learning regularization prevents memorization of noisy labels, NeurIPS 2020.
[8] Robust Early-learning: Hindering the Memorization of Noisy Labels, ICLR 2021.
[9] Mitigating Memorization of Noisy Labels by Clipping the Model Prediction, ICML 2023.
[10] Late stopping: Avoiding confidently learning from mislabeled examples, ICCV2023.

Reviewer comment (Experiments):
What happens if the amount of label noise is less than 20%? Including label noise from 0-20% can help better understand the method.

Author response:
Thank you for your suggestion regarding providing a more comprehensive view of the Label Wave method. Following your constructive feedback, to substantiate that the method works well even with a noise rate as low as 10%, we have now included experiments with <20% label noise, and the results are as follows:

| Settings   | Label Wave | Global Maximum |
|-----------|-----------------------|---------------------------|
| CIFAR-10N (Random 1)  |  83.03±0.35%    |    83.21±0.23%    |
| CIFAR-10N (Random 2)     |   82.53±0.23%    |   82.86±0.24%    |
| CIFAR-10N (Random 3)      |   82.92±0.50%      |     83.10±0.25%    |
| CIFAR-10 (Ins. 10%)  |85.56±0.72%|  85.91±0.23%     |
| CIFAR-10 (Sym. 10%)  |84.92±0.06%|  85.46±0.12%     |

It is noteworthy that the regularization approach of our baseline model is sufficiently robust to overfitting on CIFAR-10N-aggregate. We observed an average test accuracy of 84.03±0.92% in 30 to 50 epochs, and 83.97±0.50% in 130 to 150 epochs, making early stopping unnecessary. Scenarios where early stopping is inapplicable fall outside the intended use of our method.

Reviewer comment (Experiments):
In the motivation of not using a validation set is that using a validation set reduces training set size and thus decreases performance, by this motivation, the method is targeting domains with limited amount of training set. How small the dataset size should be when it’s preferred to consider this method? Could you provide some analysis on this?

Author response:
Thank you for clarifying the content of Question 5, which has prompted us to think more deeply about the applicability and limitations of our method. We concur with your intuition that when dealing with "sufficiently large" datasets, the benefits of Label Wave method in comparison to hold-out validation might not be as stark. However, the Label Wave method has no performance disadvantage in this case either.

Figuring out the 'sweet spot' in terms of dataset size for Label Wave is tricky. It’s not just about how big the dataset is, but also what kind of data we're dealing with and how much of it is held out. Natural datasets often exhibit a long-tail distribution, and in such cases, maintaining or not the identical distribution between the validation set and the training set can affect the effectiveness of hold-out validation. Furthermore, as shown in Figure 4a of our paper, the proportion of hold-out data is also a key factor in discussing the effectiveness of the Label Wave method compared to hold-out validation. It is also challenging to answer your question through empirical validation, as we currently lack datasets that meet the "super large" condition for a more comprehensive empirical study.

Although we are currently unable to precisely define the optimal size of datasets for which our method is best suited, we believe that this is an important research direction worth further exploration. It is noteworthy that our paper not only provides an effective early stopping method under conditions of limited training data but also introduces a new concept of "learning confusion patterns", which offers a new perspective for understanding and handling label noise.

Reviewer comment (Experiments):
In order to show the method’s effectiveness, more datasets from diverse domains should be considered.

Author response:
We respectfully wish to highlight that our testing on the CIFAR-10N dataset involved human-annotated real-world noise, showcasing the Label Wave method's effectiveness in real-world scenarios. Furthermore, our experiments conducted on the NEWS and Tiny-ImageNet datasets have demonstrated our method's broad applicability across various domains. Here, we present the outcomes of applying our Label Wave method on the Clothing1M dataset. We are committed to supplementing more results using real-world datasets in the further revision.

| Methods   | Label Wave | Global Maximum |
|-----------|-----------------------|---------------------------|
| CE        |      70.12±0.34%      |       70.56±0.11%         |

Reviewer comment (Novelty):
**Missing important reference:** I know references that are highly related to this work but unfortunately not mentioned and compared. These two papers also tackled exactly the same point and mentioned similar intuitions on what the best early stopping point is. These papers are worth mentioning and comparing.  
[1] How does early stopping help generalization against label noise, arXiv 2019  
[2] Robust learning by self-transition for handling noisy labels, KDD 2021

Author response:
**Weakness.1 - Related Work**

In revised manuscript, we have added the references [1] and [2] in Related Work.

[1] and [2] conceptualize model learning with noisy labels as two stages: the first stage learns from clean examples, and the second stage overfits to mislabeled examples. By identifying the largest safe set in the first stage and focusing on learning from that set in the latter stage, these methods effectively avoid the model's overfitting to mislabeled examples. Notably, [2] provides a method to identify the point where MP(t) equals MR(t) without additional supervision, thereby determining an early stopping point. [1] and [2] indeed provide good insights into the study of learning with noisy labels from two perspectives: the analysis of learning stages and the provision of practical learning with noisy label methods.

Here, we will emphasize the advantages of our proposed method from two aspects: the learning stages analysis and the model selection method.

*Learning Stages:*

We introduce a novel stage between the stage dominated by learning simple patterns and overfitting mislabeled examples, named "Learning Confused Patterns". In this stage, the model begins to learn from mislabeled examples, leading to simultaneous declines in generalization and fitting performance.

*Model Selection:*

The Label Wave method not only requires no additional supervision but also has the advantages of being simple and efficient:
1. Simple Concept: It is based on the principle that the first local minimum in prediction changes marks the best early stopping point. Hence, the Label Wave method does not need to estimate the noise rate and is effective across a wide range of noise rates.
2. Direct Application: It tracks prediction changes during the training process, allowing the model to halt before learning confusing patterns. This approach avoids the complexity and computational costs of statistical models like the GMM and EM algorithms used in [2] and [3].

[1] How does early stopping help generalization against label noise, arXiv 2019 
[2] Robust learning by self-transition for handling noisy labels, KDD 2021
[3] Selc: self-ensemble label correction improves learning with noisy labels, IJCAI 2022

Reviewer comment (Experiments):
**Unclear setup for practicality:** Detecting an early stop point is very important in the industry, especially when using strong regularization techniques together. For example, in the computer vision domain, using Mixup (or Cutmix, etc), Batch Norm, Dropout, and other architecture-specific regularization (e.g., stochastic depth for Vision Transformers) is a must-need. These kinds of strong regularization obviously change the learning behavior of DNNs like the training and testing curves. For the complete study toward practical methods, these recipes of training should be considered altogether, or theoretical support is needed.

Author response:
**Weakness.2 - Strong Regularization Techniques**

Based on the reviewers' detailed comments, we tested the effectiveness of our Label Wave method with additional regularization methods. It is worth noting that our proposed Label Wave method itself includes BN and Data Augmentation, i.e., Label Wave + B, D as described below. Beyond the reviewer's comments, we have also tested the setting where these two regularization methods are not included. These regularization methods include:

*A. Mixup
B. BN
C. Dropout
D. Data Augmentation.*
(All experiments take the same settings as Sec. 4.1)

| Methods   | Label Wave | Global Maximum |
|-----------|-----------------------|---------------------------|
| Label Wave        |       66.79±0.39%      |    67.15±0.49%  |
| Label Wave + B, D |  81.61±0.44%    |    81.76±0.30%    |
| Label Wave + B, C, D     |   83.57±0.24%    |   83.77±0.32%    |
| Label Wave + A, B, D      |   82.38±0.56%      |     83.09±0.22%    |
| Label Wave + A, B, C, D  |83.67±0.45%|  84.05±0.35%     |


The experimental results show that adding or subtracting these regularization methods in the experimental setting does not affect the effectiveness of the label wave method for identifying early stopping points.

In the end, we thank the reviewer for the two constructive comments mentioned above, and addressing them really improves our paper to a great extent.