Paper: Dirichlet-based Per-Sample Weighting by Transition Matrix for Noisy Label Learning (ICLR-2024)

Reviewer comment (Writing):
Since the final algorithm chooses resampling, my main concern is the advantage of resampling comparing to reweighting, which should be given more details about and hightlighted in some sections like Introduction.

Author response:
**W1: Advantage of resampling over reweighting**

-	**[The advantage of resampling comparing to reweighting]** Sorry for our possible misunderstanding to the reviewer’s question, and sorry for our not enough explanations that may lead the reviewer to be confused.

-	Actually, the final advantage of resampling over reweighting is simple: good performance when learning with noisy label. We also showed the performance empirically and reported that resampling may memorize noisy labelled samples less than reweighting by visualizing the confidence of wrong labelled samples in Section 4.5. We thought we explained why it works through section 3, but we explain why here and modified the manuscript.

1. In the previous study[1], the variance increase of the risk function has improved the model performance empirically. Since the variance of risk with resampling is larger than that with reweighting, we can explain the good performance of RENT following the previous study.

2. Empirically, per-sample weights should be different from the true weights. Nevertheless, per-sample weights from resampled dataset (in this case, it would mean the sampling ratio for each sample) is closer to the true weight than reweighting in mahalanobis distance manner. It means that the gap between the true risk and the empirical risk from per-sample weight estimation gap is smaller when resampling than reweighting.

3. Resampling can be interpreted as label perturbation. The superiority of resampling over reweighting is supported with several previous studies [2,3] showing that perturbations during training results in better performance for learning with noisy label

-	If it is not the answer that the reviewer wanted, can the reviewer explain in more details what the reviewer wants?

-	**[highlighting in Introduction]** Following the reviewer’s suggestion, we added the summary of what we explained in section 3 in the introduction part by changing 5-th paragraph and adding the contribution summary.

Reviewer comment (Presentation):
Figure 1 should be polished up to clearly illustrate how to implement the proposed framework. Otherwise, it is difficult to understand the 4-th and 5-th paragraphs of Section Introduction.

Author response:
**W2: Figure1**

-	Sorry for your confusion. we will represent the reviewer’s concern and change our figure in the final version. For helping the reviewer’s understanding possibly, we added a figure to clearly illustrate the implementation of RENT in Appendix A currently.

-	We wonder whether we represented the reviewer’s concern adequately. We hope the reviewer responds to us in more details so that we can relieve the reviewer’s concern.

Reviewer comment (Presentation):
It is suggested to summarize the contributions at the end of Section Introduction, such as extension of DWS, the analysis of reweighting and resampling, the new resampling method.

Author response:
**W3: Summarization of contribution at Introduction**

-	We represented the reviewer’s suggestion in the revised manuscript.

Reviewer comment (Reproducibility):
Some details about the implementation of the resampling should be given in the main body of the paper. For example, the reason why $\tilde{\mu}$ and the parameters of Categorical Distribution are calculated in the way illustrated in Algorithm 1.

Author response:
**W4&Q1: $\mu$ of algorithm 1**

-	Sorry for your confusion.
-	Due to the space issue, we updated why $\mu$ is calculated as algorithm 1 in appendix D.2. It may help you how the parameters of the categorical distribution are calculated.

Reviewer comment (Theory):
As it is mentioned above, why $\tilde{\mu}$ and the parameters of Categorical Distribution are calculated in the way illustrated in Algorithm 1?

Author response:
**W4&Q1: $\mu$ of algorithm 1**

-	Sorry for your confusion.
-	Due to the space issue, we updated why $\mu$ is calculated as algorithm 1 in appendix D.2. It may help you how the parameters of the categorical distribution are calculated.

Reviewer comment (Novelty):
I acknowledge the authors' theoretical contributions, but it must be said that the method proposed by the author lacks innovation. The approach in this paper bears similarity in concept to a significant category of noisy label learning methods based on sample selection, and their success strongly underscores the superiority of resampling methods.

Author response:
-	We admit RENT of our study shares some similarities with sample-selection based methods as the reviewer pointed out. Specifically, both RENT and sample selection methods will filter out noisy labelled samples. Therefore, if we interpret sample selection methods as giving $p(Y=\tilde{y}|x)$ to noisy labelled sample (or, giving 0 to the numerator of $\frac{p(Y=\tilde{y}|x)}{p(\tilde{Y}=\tilde{y}|x)}$), RENT may also be interpreted as sample selection methods. In this way, we propose that RENT suggests a new direction of sample selection utilizing the transition matrix ensuring statistical consistency of the risk function. Techniques to improve selection quality in sample selection based methods would also be applied to RENT, e.g. using two same structure networks with different initialization points [1], adding confidence regularizer[2], etc.

-	However, sample selection methods deterministically waste samples whose criteria value is below the threshold, while RENT would have possibility of sampling clean yet whose criteria is below the threshold. We admit this can also increase the possibility of sampling noisy data again, but we empirically showed good performance of RENT in the experiment that the reviewer HW3A suggested at Q2 as one case.

-	More analysis including how many clean samples are below the threshold of each sample selection method and how much RENT can resamples those samples may be an interesting study, and we will consider it as prominent future work.

-	Nevertheless, we want to underline our implemented method, RENT, is suggested based on the theoretical superiority of resampling over reweighting. If the reviewer knows any similar papers that theoretically proves the superiority of sample selection methods over other methods which shares the same form of objective function. please let us know so that we can refer to and upgrade our study.

Reviewer comment (Evaluation):
On CIFAR-10N dataset, RENT performs well on rand1-3 and worse scenarios but exhibits poor performance on aggre. In the context of crowdsourcing, aggre should be a more common noise setting. Therefore, it is essential to analyze the reasons for the subpar performance in this particular scenario.

Author response:
-	We basically think experimental results with Agree noise setting does not show statistically significant difference, and we conjecture that it is because the noise ratio of Aggregate is as small as 9.03% (from the original paper[1]).

-	We experimented to see the performance difference of each transition matrix utilization method changing the noise ratio. Figure 11 in Appendix F.2 of the updated manuscript shows this relation. As we can see in the figure, the performance gap between red lines and others are subtle when the noise ratio is small. Having said that, the performance gap between FL and RENT on Aggre setting is not statistically significant.

-	Still, RENT shows larger gap over two other methods under high noise ratio settings

Reviewer comment (Theory):
Why use the Dirichlet distribution to sample the weight?

Author response:
-	The motivation of introducing the concept of per-sample weight sampling is to integrate both reweighting and resampling. To explain both methods, we needed to express one-hot vector form (sampling) or generate the similar vectors to the mean vector  always (weighting). We thought Dirichlet distribution is a good choice since we can express both situations easily only by changing the shape parameter($\alpha$). With utilizing the Dirichlet distribution, we can construct a unique framework which connects the reweighting and resampling. Under this unified framework, we could directly compare reweighting and resampling strategies, and provide the rationale on why resampling generally performs better than reweighting for learning with noisy label.

Reviewer comment (Novelty):
The paper does not propose a "new" method. Reweighting and resampling are both existing methods, and may not be considered novel contributions.

Author response:
**W1: novelty**

-	If we misunderstood the reviewer’s intention, please let us know so that we can answer to the reviewer’s concerns and discuss those issues more deeply.

-	We want to know whether there is any resampling-based study to solve noisy label classification task. We think our study is the first solution, but if the reviewer knows another, please let us know so that we can check similarities and differences. Under this situation, our study should also be the first to combine reweighing and resampling in one framework. We think it is meaningful because based on this unified framework, we can directly compare reweighting and resampling, and explain the reason why resampling can be better than reweighting for learning with noisy label.

-	**[Comparison with importance sampling]** First, we want to underline considering importance sampling for noisy label task should be managed differently from the studies for different tasks. Considering what the traditional importance sampling method is, it assumes the source distribution (the distribution from which we draw samples, $p_{s}$) and the target distribution (the distribution for which you want to obtain information, $p_{t}$) is already given. For our problem setting, $p_{s}$ and $p_{t}$ would be the noisy label distribution and the clean label distribution, respectively. Since we know neither $p_{s}$ nor $p_{t}$ but have only $\tilde{D}$, the noisy labelled dataset, for learning with noisy label task, it makes the problem differently.

-	We admit that there is already a study [1] that solves noisy label classification task with importance reweighting. [1] tried to solve the problem that we pointed out above by using the transition matrix. Since $p(\tilde{Y}|X)=Tp(Y|X)$ according to the definition of the transition matrix, the problem will be solved if we can estimate the transition matrix well and know only one of $p(\tilde{Y}|X)$ or $p(Y|X)$. From several studies [1,2], $p(Y|X)$ has been approximated from the on-training classifier’s output, as we also reported in Section 2.3 of the manuscript. Therefore, it is natural to assume that there will be approximation error for true weights from $\tilde{\mu}=\frac{f(x)}{Tf(x)}$. In section 3, we analyzed resampling can be more robust than reweighting when these weights are different from the true weights, which explains why RENT outperforms RW in most cases experimentally. We think this point is the main difference of our study from [1].

-	**[Comparison with resampling/reweighting study]** Next, we admit that there are studies that compare and utilize resampling and reweighting, and [3] is one of those. [3] has studied resampling outperforms reweighting for correcting sampling bias with stochastic gradients and we are also motivated by them for hypothesizing that resampling may outperform reweighting even for learning with noisy label. However, their study and ours are different for the objective. The objective of resampling or reweighting in [3] is to find the optimal parameter which minimizes the risk over the whole population. If we translate this objective directly for learning with noisy label, the objective should be minimizing the risk over both clean label distribution and noisy label distribution. It is not the objective for noisy label learning, so [3] cannot be applied to learning with noisy label task. Under this situation, analyses in [3] should be modified to apply to noisy label learning.

-	**[Comparison with sample selection methods]** Finally, we admit our study have some similarities with sample selection-based methods. Sample selection methods for learning with noisy labels selects subset of samples with some criteria and thresholds. RENT and sample selection methods share similar concept in that both use subset of samples for training. However, RENT has the possibility of sampling difficult clean samples, while sample selection based methods cannot since they deterministically divide between the selected samples and the wasted samples. Also the fact that we can assure the resampled dataset with RENT can be considered as the sampled dataset from clean label distribution theoretically is also another strength of RENT.

Reviewer comment (Experiments):
Following the last point, how different is the Dirichlet-based resampling from just setting a threshold for $\mu$ and randomly sampling?

Author response:
**Q2:Dirichlet resampling v.s $w$ thresholding+random sampling**

-	Thanks review for their interesting question. We think the reviewer already knows the difference between Dirichlet-based resampling from setting \mu threshold and random sampling for equation, so we directly report the experiment results.
-	We think the motivation of this question is possibly the experiment result that we show at Section 4.5 ($w$ value), since $w$ values are quite divided according to the figure. Therefore, we set the threshold of $w$ as 1/(batch size) and randomly resampled dataset from whose $w$ value is larger than the threshold. We denote it as $w$+random in the table below.
 
**CIFAR10**
| **Base** | **Criterion** | **SN 20%** | **SN 50%** | **ASN 20%** | **ASN 40%** | 
| --- | --- | --- | --- | --- | --- |
| Forward | $w+$random | 74.0$\pm$0.8 | 45.7$\pm$1.6 | 80.0$\pm$0.8 | 71.3$\pm$2.0 | 
| Forward | **w/ RENT** | **78.7**$\pm$0.3 | **69.0**$\pm$0.1 | **82.0**$\pm$0.5 | **77.8**$\pm$0.5 |
| DualT | $w+$random | 74.8$\pm$0.6 | 48.2$\pm$1.2 | 80.1$\pm$0.6 | 72.4$\pm$0.9 | 
| DualT | **w/ RENT** | **82.0**$\pm$0.2 | **74.6**$\pm$0.4 | **83.3**$\pm$0.1 | **80.0**$\pm$0.9 | 
| TV | $w+$random | 70.7$\pm$1.8 | 45.5$\pm$1.6 | 78.7$\pm$0.9 | 71.9$\pm$1.7 | 
| TV | **w/ RENT** | **78.8**$\pm$0.8 | **62.5**$\pm$1.8 | **81.0**$\pm$0.4 | **74.0**$\pm$0.5 | 
| VolMinNet | $w+$random | 71.4$\pm$0.4 | 45.3$\pm$2.3 | 76.8$\pm$2.9 | 71.7$\pm$1.7 | 
| VolMinNet | **w/ RENT** | **79.4**$\pm$0.3 | **62.6**$\pm$1.3 | **80.8**$\pm$0.5 | **74.0**$\pm$0.4 | 
| Cycle | $w+$random | 70.8$\pm$12.2 | 44.7$\pm$0.8 | 78.1$\pm$0.7 | 69.2$\pm$1.2 | 
| Cycle | **w/ RENT** | **82.5**$\pm$0.2 | **70.4**$\pm$0.3 | **81.5**$\pm$0.1 | **70.2**$\pm$0.7 | 

**CIFAR100**
| **Base** | **Criterion** | **SN 20%** | **SN 50%** | **ASN 20%** | **ASN 40%** | 
| --- | --- | --- | --- | --- | --- |
| Forward | $w+$random | 30.8$\pm$1.8 | 16.1$\pm$1.1 | 30.6$\pm$2.2 | 24.6$\pm$0.5 |
| Forward | **w/ RENT** | **38.9**$\pm$1.2 | **28.9**$\pm$1.1 | **38.4**$\pm$0.7 | **30.4**$\pm$0.3 |
| DualT | $w+$random | 31.9$\pm$0.9 | 16.5$\pm$1.1 | 34.7$\pm$1.8 | 25.1$\pm$0.7 |
| DualT | **w/ RENT** | **39.8**$\pm$0.9 | **27.1**$\pm$1.9 | **39.8**$\pm$0.7 | **34.0**$\pm$0.4 |
| TV | $w+$random | 28.2$\pm$2.8 | 16.1$\pm$0.9 | 28.8$\pm$2.0 | 21.6$\pm$1.9 |
| TV | **w/ RENT** |  **34.0**$\pm$0.9 | **20.0**$\pm$0.6 | **34.0**$\pm$0.2 | **25.5**$\pm$0.4 |
| VolMinNet | $w+$random | 28.7$\pm$1.5 | 17.2$\pm$1.5 | 30.4$\pm$2.0 | 23.9$\pm$2.2 |
| VolMinNet | **w/ RENT** | **35.8**$\pm$0.9 | **29.3**$\pm$0.5 | **36.1**$\pm$0.7 | **31.0**$\pm$0.8 |
| Cycle | $w+$random | 31.5$\pm$0.8 | 16.3$\pm$1.4 | 33.2$\pm$1.6 | 24.5$\pm$2.3 |
| Cycle | **w/ RENT** | **40.7**$\pm$0.4 | **32.4**$\pm$0.4 | **40.7**$\pm$0.7 | **32.2**$\pm$0.6 |

-	RENT and $w$+random shows significant gap. We think this gap happens because during training, especially in the earlier learning iterations, w of clean samples and noisy samples may be more mixed (since the model parameter is yet more similar to the random assignment). Therefore, sorting with an arbitrary threshold may be riskier. However, Dirichlet-based resampling and RENT may be safer to this problem since the sampling probability of the samples below threshold is not 0.This would result in the overall performance differences.

-	Also, choosing a good threshold may be difficult, and we admit that $w$+random performance may be improved with threshold tuning. However, some training iteration  and noise condition adaptive strategy could be needed. Figure 17 in the appendix F.5 in the revised manuscript shows this need. It shows the number of samples whose $w_i$ is larger than 1/B, with clean samples and noisy samples, respectively, while training the classifier with RENT. Result shows that clean samples whose w is larger than 1/B is not as many as the final iteration (which is natural), which underlines the training process adaptive strategy for threshold value.

Reviewer comment (Experiments):
Will the method work better if combined with data cleaning or abstention kind of ideas? There is one reference that I found ([1] Koziarski, Michał, Michał Woźniak, and Bartosz Krawczyk. "Combined cleaning and resampling algorithm for multi-class imbalanced data with label noise." Knowledge-Based Systems 204 (2020): 106223.). Although I'm not familiar with this work, I think it might be interesting to the authors and other reviewers.

Author response:
**Q3: Learning with noisy label with class imbalace**

-	Thank the reviewer for suggesting the possible extension direction of our research.

-	First, we need to clarify the terminology difference between their study and our study. In their study, resampling is used to mitigate the sampling bias for imbalanced data. Otherwise, we use resampling to change the distribution from noisy label distribution to true label distribution. In other words, resampling in class imbalanced learning study considers $p(y)$, while we consider $p(y|x)$. We previously discussed this issue more in **W1**.

-	Considering the difference in terminology, what we are doing as resampling is more similar to data cleaning of their study.

-	However, we admit solving class imbalanced dataset with noisy label can be important [16, 17]. Therefore, we experimented with synthetic label noise with class imbalance. Figure 20 in Appendix F.8 the revised manuscript shows the performance comparison of FL, FW and RENT with several T estimation baselines. As we can see in the figure, Again, RENT shows consistently good result over FL and RW. However, we admit there is no treatment to solve class imbalance issue in RENT, and this direction could be an interesting future study. We will report more experimental details in the appendix.

Reviewer comment (Experiments):
Can the proposed method be compared with other noisy label learning methods (regularization, loss correction, or data cleaning)?

Author response:
**Q4:** We reported the model performance as above (**W3**)

-	Can the reviewer explain what exactly the loss correction is? According to [15], it is the other expression of the loss modification with the transition matrix, and we have already reported several loss correction baselines on the paper.

Reviewer comment (Evaluation):
Some experimental results are inconsistent with prior works, why are the performances of learning with true $T$ worse than Cycle consistency and Dual-$T$?

Author response:
-	Considering W1, We think the reviewer asks two things in this question, and we respond to those one by one respectively.

**W1: Experimental results inconsistency**
- We have reported the reason of experimental results inconsistency with regard to the prior works in Appendix F.2 (Performance reproduce & comparison with our experiment setting). In Table 8 in the revised manuscript, [1] means the reported performance from their original paper, [2] means reproduced performance following each paper’s experimental condition and [3] is the model performance we reported in the main paper following our experimental settings. Below we report table 8 for the reviewer’s convenience.

| T estimation | [1] ~ [3] | SN 20% | SN 50% | ASN 20% | ASN 40% |
|--------------|-----------|--------|--------|---------|---------|
| Forward| [1]  | 83.4$\pm$- | -      | 87.0$\pm$- | -      |
| Forward| [2]  | 82.6$\pm$0.3 | 67.4$\pm$1.0 | 87.9$\pm$0.2 | 82.9$\pm$0.4 |
| Forward |[3]  | 73.8$\pm$0.3 | 58.8$\pm$0.3 | 79.2$\pm$0.6 | 74.2$\pm$0.5 |
| DualT| [1]    | 78.4$\pm$0.3 | 70.0$\pm$0.7 | -      | -      |
| DualT |[2]    | 85.6$\pm$0.2 | 74.2$\pm$0.1 | 87.8$\pm$0.7 | 81.9$\pm$0.7 |
| DualT| [3]    | 79.9$\pm$0.5 | 71.8$\pm$0.3 | 82.9$\pm$0.2 | 77.7$\pm$0.6 |
| TV |[1]       | -            | 82.6$\pm$0.4 | -      | -      |
| TV |[2]       | 87.5$\pm$0.2 | 76.6$\pm$0.2 | 80.6$\pm$7.4 | 75.0$\pm$13.3 |
| TV |[3]       | 74.0$\pm$0.5 | 50.4$\pm$0.6 | 78.1$\pm$1.3 | 71.6$\pm$0.3 |
| VolMinNet| [1]| 89.6$\pm$0.3 | 83.4$\pm$0.3 | -      | -      |
| VolMinNet |[2]| 91.4$\pm$0.1 | 79.2$\pm$0.1 | 94.9$\pm$0.1 | 88.0$\pm$4.5 |
| VolMinNet |[3]| 74.1$\pm$0.2 | 46.1$\pm$2.7 | 78.8$\pm$0.5 | 69.5$\pm$0.3 |
| Cycle |[1]    | 90.4$\pm$0.2 | -            | 90.6$\pm$0.0 | 87.3$\pm$0.0 |
| Cycle| [2]    | 90.4$\pm$0.4 | -            | 86.5$\pm$0.1 | 66.4$\pm$0.4 |
| Cycle |[3]    | 81.6$\pm$0.5 | -            | 82.8$\pm$0.4 | 54.3$\pm$0.3 |

-	One of the main differences is evaluation time. We chose to report the test accuracy after convergence following the below reasons.

1. Utilizing validation dataset costs more, and when collecting the dataset is expensive, it will be an important issue.

2. The quality of the validation dataset may not be trustful enough, and considering noisy label learning setting, the problem may be more important than the clean dataset setting.

3. Evaluation period term would impact choosing the best model, e.g. if model evaluation happens every 1,000 epochs, it will be no useful; if the model is evaluated on every step, it will cost too much time.

4. Evaluation during training cost more time.

- Therefore, we suppose the model performance after finishing training is more reliable and realistic, so our reported performances are basically the performance after convergence.

**W1: Performance of True T vs. Cycle, DualT**

- Please check Appendix F.2 (True T is not the best? paragraph) on the revised manuscript for the related figures.

- First, we experimented to analyze the correlation between the performances of RENT and the estimation gap of the resulting transition matrix with regard to several estimation algorithms. Figure 12 shows the results.
 
- The model performance does not seem to have much correlation with T estimation error. We conjecture that this is due to two reasons: (1) since TV, VolMinNet and Cycle includes regularization terms for updating a classifier, those terms may have affected the classifier training process and (2) For TV, VolMinNet and Cycle, the transition matrix changes during training procedure, so the estimation gap would have reduced while training (according to their original paper, the estimation error seems to decrease with training process). Therefore, the transition matrix gap of those algorithms would have been larger.

- To solely compare the impact of T estimation error to the model performance under same risk function and learning procedures, we generated error-injected transition matrix arbitrarily and reported the model performances as Figure 13 of the revised manuscript. Specifically, we subtracted $\epsilon$ to diagonal terms and added $\epsilon/$(the number of classes-1) to others.

- Interestingly, we found out that the performance is not the best when the transition matrix estimation error is 0, showing the possible direction of improving RENT.
- Please check Appendix F.2. for more details of the experiment.

Reviewer comment (Writing):
The discussions in section B seems problematic, in $\sum_{j=1}^{C}\(max_{j} T_{\hat{y}_{i}j} \)$, are we trying to sum over all $j$, or finding the maximum $j$? It seems that you're trying to do both, can you instead give an intuitive explaination and refine your mathmatical statements?

Author response:
**W2: Notation error**
- Sorry for our mistakes. The j inside the parenthesis should not be j (the same notation as the outside). We modified the notation in the revised manuscript.

Reviewer comment (Theory):
The motivation of DWS is not strong enough, if we are going with the assumption that the classifier trained with noisy labels can not accurately estimate noise class posterior (poorly calibrated, high estimation error, etc.), then we can hardly assume that the transition matrix is accurate, as the transition matrix is usually estimated from the noise class posterior.

Author response:
**W3: Noisy class posterior estimation**

- First, we want to underline that the concept of “the estimation error of the noisy label posterior distribution from neural networks trained with noisy dataset could be high” is not new, and has already been studied in several previous studies [1,2,3]. Those studies have already suggested methods to estimate the transition matrix well under this situation, and we leveraged their methods for transition matrix estimation.

-	We also reported one of these studies([2]) as Remark B.1 in Appendix C. Again, the difference between their study and ours, is that they focused on how to estimate good transition matrix when the noisy posterior distribution is inaccurate (since the transition matrix is usually estimated from the noise class posterior as the reviewer pointed out), while we studied how to utilize the transition matrix well.

-	We admit we first assumed that the true T is accessible in section 2.3 to exclude the impact from the transition matrix estimation error and solely compare different T utilization. We demonstrated that current utilization can cause problem even with true transition matrix in Section 2.3, and this is the main motivation to find a new transition matrix utilization method. We are sorry if this causes any confusion for your idea.

-	However, we do not consider that the transition matrix is accurate after Section 2.3. Rather, we implicitly include the case when the transition matrix is inaccurate. In Section 3, we focus on the situation when per-sample weights vector, $\boldsymbol{\mu}$ is different from $\boldsymbol{\mu}^*$. We think both the estimation gap of noisy class posterior and the estimation gap of transition matrix are encompassed in $\boldsymbol{\mu}$. While we admit that we did not directly analyze the impact of the transition matrix estimation error and the calibration error of the classifier yet, Eq. (5) implicitly addresses the potential impact of these errors. Since mu is composed of the transition matrix and the classifier output, the possible errors in transition matrix estimation and classifier calibration are inherent in our analysis. If the reviewer has some ideas on how to directly analyze the impact of these errors to the classifier, we are happy to further discuss this issue so that we can relieve any of your concerns.

-	To represent the possible transition matrix estimation error from the previous transition matrix estimation methods, we reported experimental results with several transition matrix estimation baselines.

Reviewer comment (Experiments):
More recent and SOTA $T$ estimation method is not included [1].

Author response:
**W4: SOTA baseline**

-	Thank the reviewer for the suggestion. We report the experiment result as below, showing the experimental result over CIFAR10 and CIFAR100. We first checked whether the performance is reproducible, and we changed the Forward loss part as RENT and got the result. We used RCE as robust loss.

| **Dataset**  | | **CIFAR10**   |  | | | **CIFAR100**   |  | | |
|--------------  |--|------------|--|--|-|--------------|--|--|-| 
| **Noise**   |   | **UNI 20%**       | **UNI 50%**       | **FLIP 20%**      | **FLIP 45%**      |  **UNI 20%**       | **UNI 50%**       | **FLIP 20%**      | **FLIP 45%**      | 
| (ROBOT+RCE)    | w/FL (Reported)   | 92.1$\pm$0.0      | **88.5**$\pm$0.1  | 93.6$\pm$0.0      | 92.4$\pm$0.2      | 73.0$\pm$0.1      | 65.1$\pm$0.5      | 75.8$\pm$0.5      | 70.2$\pm$0.4      | 
|                | w/RW              | 88.9              | 84.5              | 72.7              | 86.9              | 6.3               | $-$               | 4.5               | 4.7               | 
|                | w/RENT            | **92.3**$\pm$0.3  | 87.5$\pm$0.4      | **93.9**$\pm$0.2  | **93.0**$\pm$0.2  | **74.1**$\pm$0.5  | **66.3**$\pm$0.4  | **76.9**$\pm$0.1  | **71.7**$\pm$1.2  |

-	RENT again improves the model performances with ROBOT.

Reviewer comment (Experiments):
It is well-known that instance-dependent transition matrix might exhibit high complexity when the class number increases, for instance, for cifar-100, we need to compute a tensor of size $50000 \times 100 \times 100$, which is prohibited in real-world applications, can DWS mitigate this issue? More discussions in this aspect might bring more contribution.

Author response:
**Q1: Intance dependent transition matrix estimation complexity**

-	Sorry for our possible misunderstanding to the reviewer’s question, but let us respond to the reviewer’s question based on what we understood

- Computation complexity issue: We want to clarify a tensor of size 50000 $\times$ 100 $\times$ 100 computation usually does not happen at one time for instance dependent transition matrix; we think the transition matrices would be calculated in batch-wise manner. For example, a baseline like [4] trains a new network that returns an instance dependent transition matrix as output. Therefore, it does not need to calculate the whole dataset at one time. If the reviewer knows some studies which compute the whole transition matrix in one time, please let us know so that we can understand how they could did that.

- Estimation gap issue: Then, the problem of instance-dependent transition matrix is the difficulty of estimating the transition matrix, since a tensor of size 50000 $\times$ 100 $\times$ 100 should be estimated by using 50000 samples, for example, for cifar-100, as the reviewer pointed out. Please not that we focused on how to utilize the transition matrix well, and it is orthogonal to various T estimation methods. Therefore, we think the difficulty of estimating the transition matrix is beyond of our research scope. Nevertheless, it can be an interesting direction to study further as a future work.

- Still, we want to underline again that RENT shows better performance than Forward loss (FL) even for instance dependent transition matrix based methods, as we reported in Table 2. (PDN[5] and BLTM[4]).

Reviewer comment (Writing):
While reviewing the paper, I encountered some challenges in comprehending the content, primarily due to the clarity and organization of the presentation and missing notations/definitions.

Author response:
**W1: Notations**

- Sorry for our mistakes or the reviewer's confusion. 

- In page 4, M is the sampling number of $\boldsymbol{w}$. It can be translated directly to a resampling budget of RENT as in page 6. We revised the manuscript.

- We modified our script with regard to the emprical risk of RW as the reviewer suggested (page 3)

- $\pi_N$ is a categorical distribution to resample data instances. It is not for resampling label for a specific sample, and also not for resampling a specific input $(x_1,…,x_M)$. We resample data instances, which is a joint of $(x_i,\tilde{y}_i)$. We also reported some explanations with regard to the parameters of the categorical distribution in Appendix D.2. It may help you understanding the sampling process of RENT.

Reviewer comment (Writing):
An explanation of $M$ seemed to be missing in page 4 but appears later on in page 6 (explained as a resampling budget).

Author response:
**W1: Notations**

- Sorry for our mistakes or the reviewer's confusion. 

- In page 4, M is the sampling number of $\boldsymbol{w}$. It can be translated directly to a resampling budget of RENT as in page 6. We revised the manuscript.

- We modified our script with regard to the emprical risk of RW as the reviewer suggested (page 3)

- $\pi_N$ is a categorical distribution to resample data instances. It is not for resampling label for a specific sample, and also not for resampling a specific input $(x_1,…,x_M)$. We resample data instances, which is a joint of $(x_i,\tilde{y}_i)$. We also reported some explanations with regard to the parameters of the categorical distribution in Appendix D.2. It may help you understanding the sampling process of RENT.

Reviewer comment (Reproducibility):
I wasn't sure how $x_1, \ldots, x_M$ are determined in Algorithm 1 page 6. Since $\pi_N$ is a categorical distribution (and not a joint distribution of instances and labels), it seems to me that we can only sample $\tilde{y}_1, \ldots, \tilde{y}_M$ (without $x_1, \ldots x_M$)?

Author response:
**W1: Notations**

- Sorry for our mistakes or the reviewer's confusion. 

- In page 4, M is the sampling number of $\boldsymbol{w}$. It can be translated directly to a resampling budget of RENT as in page 6. We revised the manuscript.

- We modified our script with regard to the emprical risk of RW as the reviewer suggested (page 3)

- $\pi_N$ is a categorical distribution to resample data instances. It is not for resampling label for a specific sample, and also not for resampling a specific input $(x_1,…,x_M)$. We resample data instances, which is a joint of $(x_i,\tilde{y}_i)$. We also reported some explanations with regard to the parameters of the categorical distribution in Appendix D.2. It may help you understanding the sampling process of RENT.

Reviewer comment (Presentation):
For Proposition 3.1, it might be more clear to present the assumptions directly within the proposition rather than explaining them in the appendix.

Author response:
**W2: Assumtion of proposition 3.1**

-	Sorry for not directly presenting the assumption in Proposition 3.1 We changed it as in the revised manuscript (page 6, blue color)

Reviewer comment (Theory):
Is Eq. 7 correct? Should we introduce a different notation for the quantity in Eq.7 instead of writing that this is equivalent with $R_{l, DWS}^{emp}$, since we are applying the CLT in the proof in Appendix C.1 and $N$ is finite?

Author response:
**W3: Correctness of Eq. 7**

-	First, Eq 7 is not ours; it is the empirical risk function of SNL. We think what the reviewer asks is related to Eq.6, so we answer to this question based on Eq. 6.
-	As the reviewer pointed out, we admit $R_{l,DWS}^{emp}$ can only be decomposed as the static risk and the stochastic normally distributed noise related term as Eq 6 when N goes to infinity. Therefore, we added the notation to show that N is infinite. We modified the manuscript in page 5, (Eq 6 part).

Reviewer comment (Presentation):
The 'et al.' in the references could be written out in full.

Author response:
**Q1: References**

-	Sorry. We copied and pasted the bibtex-form codes by finding each reference, so we did not recognize there were et al. sign in the references. The et al. mark was included in total of four references, so we checked whether the authors of those references were included in each reference and removed the et al. signal.

Reviewer comment (Experiments):
The comparative analysis of RENT is limited to FL and RW, utilizing transition matrices obtained through various methods. A more comprehensive evaluation, including a broader range of noise-label learning techniques, such as the approach proposed by Lin et al. in 2022, would provide a more comprehensive assessment of RENT's performance.

Author response:
**W1: Comparison with Lin et al. 2022**

-	Thank you for the reviewer’s suggestion. We reproduce and report the experiment result of Lin et al., which is VRNL (the following table). Since it can be applied to various transition matrix estimation methods orthogonally, we report the experimental results as we did in Table 1 on the main paper. We provide additional details of the experiment at Appendix F.2
| Base   | Risk        | SN 20%         | SN 50%         | ASN 20%        | ASN 40%        | SN 20%         | SN 50%         | ASN 20%        | ASN 40%        |
|--------|-------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|
| Forward| w/ FL       | 73.8$\pm$0.3   | 58.8$\pm$0.3   | 79.2$\pm$0.6   | 74.2$\pm$0.5   | 30.7$\pm$2.8   | 15.5$\pm$0.4   | 34.2$\pm$1.2   | 25.8$\pm$1.4   |
| Forward| w/ VRNL     | 76.9$\pm$0.4   | 64.8$\pm$2.3   | 54.2$\pm$9.2   | 59.3$\pm$12.7  | 34.1$\pm$3.4   | 18.4$\pm$2.8   | 35.5$\pm$2.4   | 27.2$\pm$1.6   |
| Forward| w/ RENT     | **78.7$\pm$0.3** | **69.0$\pm$0.1** | **82.0$\pm$0.5** | **77.8$\pm$0.5** | **38.9$\pm$1.2** | **28.9$\pm$1.1** | **38.4$\pm$0.7** | **30.4$\pm$0.3** |
| DualT  | w/ FL       | 79.9$\pm$0.5   | 71.8$\pm$0.3   | 82.9$\pm$0.2   | 77.7$\pm$0.6   | 35.2$\pm$0.4   | 23.4$\pm$1.0   | 38.3$\pm$0.4   | 28.4$\pm$2.6   |
| DualT  | w/ VRNL     | 81.2$\pm$0.3   | 73.7$\pm$0.8   | **83.5$\pm$0.1** | 78.1$\pm$2.2   | 38.2$\pm$1.4   | 25.2$\pm$3.7   | **40.0$\pm$1.2** | **34.4$\pm$1.3** |
| DualT  | w/ RENT     | **82.0$\pm$0.2** | **74.6$\pm$0.4** | 83.3$\pm$0.1   | **80.0$\pm$0.9** | **39.8$\pm$0.9** | **27.1$\pm$1.9** | 39.8$\pm$0.7   | 34.0$\pm$0.4   |
| TV         | w/ FL       | 74.0$\pm$0.5   | 50.4$\pm$0.6   | 78.1$\pm$1.3   | 71.6$\pm$0.3   | **34.5$\pm$1.4** | 21.0$\pm$1.4   | 33.9$\pm$3.6   | **28.7$\pm$0.8** |
| TV         | w/ VRNL     | 76.0$\pm$0.6   | 51.3$\pm$0.7   | 78.8$\pm$0.3   | 58.5$\pm$9.2   | 28.5$\pm$1.2   | **22.9$\pm$2.8** | 29.9$\pm$1.0   | 26.4$\pm$2.3   |
| TV         | w/ RENT     | **78.8$\pm$0.8** | **62.5$\pm$1.8** | **81.0$\pm$0.4** | **74.0$\pm$0.5** | 34.0$\pm$0.9 | 20.0$\pm$0.6   | **34.0$\pm$0.2** | 25.5$\pm$0.4   |
| VolMinNet  | w/ FL       | 74.1$\pm$0.2   | 46.1$\pm$2.7   | 78.8$\pm$0.5   | 69.5$\pm$0.3   | 29.1$\pm$1.5   | 25.4$\pm$0.8   | 22.6$\pm$1.3   | 14.0$\pm$0.9   |
| VolMinNet  | w/ VRNL     | 76.3$\pm$0.9   | 50.3$\pm$1.4   | 72.3$\pm$9.0   | 67.4$\pm$5.3   | 28.1$\pm$0.6   | 26.6$\pm$2.2   | 19.5$\pm$3.7   | 14.7$\pm$1.4   |
| VolMinNet  | w/ RENT     | **79.4$\pm$0.3** | **62.6$\pm$1.3** | **80.8$\pm$0.5** | **74.0$\pm$0.4** | **35.8$\pm$0.9** | **29.3$\pm$0.5** | **36.1$\pm$0.7** | **31.0$\pm$0.8** |
| Cycle      | w/ FL       | 81.6$\pm$0.5   | $-$            | 82.8$\pm$0.4   | 54.3$\pm$0.3   | 39.9$\pm$2.8   | $-$            | 39.4$\pm$0.2   | 31.3$\pm$1.2   |
| Cycle      | w/ VRNL     | 82.4$\pm$0.6   | $-$            | **83.0$\pm$0.4** | 54.3$\pm$0.4   | **41.9$\pm$2.1** | $-$            | **42.1$\pm$1.6** | **32.5$\pm$1.2** |
| Cycle      | w/ RENT     | **82.5$\pm$0.2** | **70.4$\pm$0.3** | 81.5$\pm$0.1   | **70.2$\pm$0.7** | 40.7$\pm$0.4 | **32.4$\pm$0.4** | 40.7$\pm$0.7 | 32.2$\pm$0.6 |
| True $T$   | w/ FL       | 76.7$\pm$0.2   | 57.4$\pm$1.3   | 75.0$\pm$11.9  | 70.7$\pm$8.6   | 34.3$\pm$0.5   | 22.0$\pm$1.5   | **35.8$\pm$0.5** | **31.9$\pm$1.0** |
| True $T$   | w/ VRNL     | 79.3$\pm$0.6   | 63.8$\pm$1.3   | 49.2$\pm$4.4   | 47.7$\pm$6.2   | **36.6$\pm$1.1** | **25.5$\pm$0.7** | 26.2$\pm$3.3   | 22.5$\pm$5.7   |
| True $T$   | w/ RENT     | **79.8$\pm$0.2** | **66.8$\pm$0.6** | **82.4$\pm$0.4** | **78.4$\pm$0.3** | 36.1$\pm$1.1 | 24.0$\pm$0.3   | 34.4$\pm$0.9 | 27.2$\pm$0.6   |

- Since there is no official code, we reproduced the code based on the paper explanation.

- VRNL improves the original $T$ estimation methods, and RENT tends to show better performances than VRNL on CIFAR10, and similar performances on CIFAR 100 (15 win, 9 lose cases over 24 settings). We suppose the estimation gap of per-sample weights may be large with regard to CIFAR100, so it caused lower performance in some cases.

- By the way, we think our method can also be integrated with VRNL. Specifically, this integration can be defined as utilizing the transition matrix as RENT (risk function part) and add variance of the risk as VRNL does (regularization). We may show the experiment results of such combination by the end of the rebuttal.

Reviewer comment (Evaluation):
In table 2, there are a couple of settings where RENT does not perform as well as the other two methods; can the authors discuss the underlying reasons?

Author response:
**Q1: Performance in Table 2**

-	First note that the noise ratio of Aggregate is as small as 9.03% (from the original paper[1]).

-	We reported the performance comparison of each transition matrix utilization method changing the noise ratio. Figure 11 in Appendix F.2 of the updated manuscript shows this relation. As we can see in the figure, the performance gap between red lines and others are subtle when the noise ratio is small. Having said that, the performance gap between FL and RENT on Aggre setting is not statistically significant.

-	Still, RENT shows larger gap over two other methods under high noise ratio settings.