Paper: Robust Classification via Regression for Learning with Noisy Labels (ICLR-2024)

Reviewer comment (Theory):
The primary limitation of the paper is the absence of a clear rationale behind the decision to transform the classification problem into a regression problem. The paper would significantly benefit from a more in-depth analysis of why this transformation is considered a "better" approach for handling noisy labels. Currently, the proposed method appears to be a compilation of established techniques, and addressing this aspect could enhance the quality and impact of the paper.

Author response:
**Rationale for transforming classification to regression**

We agree that the motivation for transforming classification to regression could be improved, which is why we have now provided a detailed discussion on this in Section A.7. We do not claim this view by default is a “better” approach for handling noisy labels in classification. However, we do claim the regression view that our adapted log-ratio transform approach provides, opens up a new research direction to leverage sound and time-tested methods from the (robust) regression literature. Please also see the “Clarification on our contributions and novelty” paragraph in the response to Reviewer U8oG.

Reviewer comment (Reproducibility):
The reviewer has a general impression that some key details are missing in the description of the method and the experiments.

Author response:
_Method description_

1. Above Equation 4, we now make it clearer that we use the standard label smoothing technique. 
2. In Section A.1, we now have a discussion on how to efficiently evaluate the loss, and that we use two distinct linear layers to predict the mean and variance.

Reviewer comment (Theory):
P4: The use of an equal weight vector $u$ in (4) for label smoothing should be clarified. What would be the effect of using a vector of all 1s scaled by other factors? Is this equivalent to adjust $\gamma$? The authors are encouraged to provide further clarification on this matter.

Author response:
**Clarification of label smoothing**

As the log-ratio transform is only defined for categorical distributions inside the probability simplex (no component can be zero), we need a way to transform class IDs from classification to the interior probability simplex. We achieve this by using the simple and common regularization technique called label smoothing [1], and we provide the original definition in Equation 4 (see Section 7 in the original paper). The idea is to smooth the original one-hot encoded version of the label by interpolating it with the uniform distribution over K classes (all components equal to 1/K).

The reviewer wants a clarification if it is possible to replace the uniform distribution with a vector of ones scaled by other factors than 1/K. An important property that is exploited in label smoothing is that interpolating two categorical distributions is another categorical distribution, for interpolation weights in $\[0,1\]$. So to clarify if we can use other scale factors, the answer is in general no, as a vector of ones can only be scaled by 1/K to be a valid categorical distribution (sum of components are one). However, it is possible to generalize label smoothing by replacing the uniform distribution with any other categorical distribution. For our purposes, the uniform distribution is a good choice though, as we require the interpolated distribution to have non-zero probability in all components and, a priori, we have no way of justifying putting more probability in any particular class.

Reviewer comment (Reproducibility):
P4: Regarding loss reweighting via a Gaussian noise model, there may be a computational challenge with evaluating the density function of a multivariate normal due to the inversion of covariance matrices, especially in high-dimensional cases. It appears that the matrix inversion lemma might have been employed as a workaround. Can the authors please confirm and elaborate on this technique? Additionally, it is not entirely clear how $\mu_{\theta}$ and $r_{\theta}$ correspond to the output of the neural network. Are they concatenated as a long vector, or are they separate outputs with distinct final linear layers?

Author response:
**Efficient evaluation of loss function**

The reviewer brings up an interesting question about how the negative log-likelihood of the Gaussian distribution can be efficiently evaluated. In short, to evaluate the loss function, inverses, and determinants of the covariance/scale matrices have to be calculated, which for general matrices are computationally heavy. Indeed, the parameterization we use makes inverse and determinant computations efficient through the Woodbury matrix identity, and the matrix determinant lemma, respectively. This is a common parameterization for the covariance/scale matrix of Gaussian distributions, that is implemented in probabilistic frameworks such as Tensorflow Probability. We agree that a motivation for the particular covariance/scale matrix parameterization was missing, which we have now added to Section A.1.

Regarding $\mu_\theta$ and $r_\theta$, we use a standard neural network (WideResNet 28-2, ResNet 50, or InceptionResNetV2) and replace the last linear layer with two linear layers (heads), one for predicting $\mu$ and one for predicting the rank-one factor $r$ of the scale matrix.

Reviewer comment (Experiments):
In Table 1, it is unclear why the symmetric and asymmetric noise rates are different. Moreover, it does not make sense to the reviewer that a noise rate is more than 50%, please clarify.

Author response:
**Clarifications on synthetic noises**

The reviewer wonders why the noise rates are different between the symmetric and asymmetric noise types, and how the noise rate can be higher than 50% in the symmetric case. 

To clarify this, we need to describe the different noise types. Symmetric noise of X% means that there is a X% risk for each example that their label is replaced by a uniformly random new label. Similarly, for asymmetric noise, there is a risk the label is replaced, but now to a perceptually similar class, e.g., images with cat labels are replaced with a dog label. 

In the symmetric case, even after applying noise with a noise rate above 50%, out of all examples with a particular label, the majority of them will still be correctly labelled. An example will make this clearer, consider the case with three classes, A, B and C, with 50% noise. Then, out of the examples that originally had a label of A, after the noise process, 50% are still As, 25% now are Bs, and 25% are Cs. With a similar argument for other classes, we find that out of all examples with a label of A after the noise process, 50% of the examples were originally As, 25% Bs, 25% Cs. In fact, as long as the noise rate is below (K-1)/K, the true class will still be in majority after the noise process, so there is still a signal to learn from.

The asymmetric noise case is different. As some classes are only changed to a particular other class (e.g., cat to dog), for the true class to still be in majority after the noise process, the noise rate must be below 50%. 

We hope the explanations of the different noise types above makes it clear that methods can be evaluated on higher symmetric noise rates (< (K-1)/K) than the asymmetric noise (< 50%).

Reviewer comment (Experiments):
A brief overview of the datasets would greatly assist readers who are new to the field of noisy label classification.

Author response:
**Clarification on the datasets used in experiments**

The reviewer suggests adding a brief overview of the datasets in the appendix to assist readers that are new to the noisy labels field. We wholeheartedly agree, and in fact, we provided such a section in Section A.4, which was referred to in the first paragraph of the experiments section (Section 4). As suggested by the reviewer, we will clarify that, e.g., “Random 1” and “Worst” correspond to different label sets of CIFAR-10N to reduce confusion for readers unfamiliar with the dataset.

Reviewer comment (Presentation):
The authors should point out aggregate, random, worst corresponding to three sets of noisy labels in Table 2. Otherwise, the table is difficult to understand for people who does not know the dataset well.

Author response:
_Experiments_

4. In Section 4.3’s result paragraph, we now mention that aggregate, random 1-3, and worst are label sets of CIFAR-10N. Due to layout constraints, we were not able to fit this detail into the table caption.

Reviewer comment (Writing):
P2: The last two sentences in the "compositional data" paragraph lack clarity regarding what "several problems" and "this problem" refer to.

Author response:
_Minor issues_

1. We have now reformulated the last sentences in Section 2’s compositional data paragraph.

Reviewer comment (Writing):
P2: The final sentence in the "transforms" paragraph could be more informative. Instead of stating that "several solutions exist," the authors could refer to the specific technique employed later in the paper for a more insightful description.

Author response:
_Minor issues_

2. The transforms paragraph in Section 2 now has a more informative last sentence, as it introduces our label smoothing solution.

Reviewer comment (Theory):
The expression in (2) on P3 is statistically incorrect. The conditional distribution of the transformed continuous label given features is a normal distribution rather than the marginal distribution of the label itself.

Author response:
_Minor issues_

3. We have now made the dependence on the input x explicit in Equation 2.

Reviewer comment (Writing):
P5: The first sentence "we describe our experimental setup of our experiments" in Section 4, appears redundant.

Author response:
_Minor issues_

4. The first sentence in Section 4 no longer includes “of our experiments”.

Reviewer comment (Presentation):
The LaTeX formatting for quotes, as exemplified by "worst" on P7, may require adjustment.

Author response:
_Minor issues_

5. We now adopt proper quotation marks throughout the manuscript.

Reviewer comment (Novelty):
The contribution is limited or needs refinement. To me, the log-ratio transformation, the Gaussian Noise Model for loss reweighting, and the estimation of the shift are all previous works. The authors should be more explicit on the novelty and contributions.

Author response:
**Clarification on our contributions and novelty**

Thanks for giving us the opportunity to more explicitly describe the novelty and contributions of our work. As the reviewer mentions, our use of a shifted Gaussian noise model for unified loss reweighting and label correction is interesting, meaningful and very different from previous methods. This is a novel contribution, despite building on existing notions such as the log-ratio transform and Gaussian noise models. All novel methods build upon existing mathematical notions.

We introduce and adapt the well-founded log-ratio transform to the modern problem of noisy labels in classification. Our approach makes it possible to view classification as regression, opening up a new research direction to leverage sound and time-tested methods from the (robust) regression literature that spans centuries. One of our contributions is to open up this research direction that we believe has immense potential, as we discuss in the added section in Section A.7. To the best of our knowledge, this is the first time the log-ratio transform has been used for this purpose.

Furthermore, we propose a specific method in this adapted log-ratio transform approach, which is a combination of the ilr transform with a shifted Gaussian noise model, that achieves loss reweighting and label correction in a single simple and unified approach. To the best of our knowledge, this is the first unified method for loss reweighting and label correction for label noise robustness in classification. In Section A.6 (referenced in the baselines paragraph in Section 4.2), we described, in detail, the connections to the most related works. As Reviewer QVE6 found this discussion interesting and valuable for understanding our method in the context of related work, we believe you will too. The closest work to the log-ratio transform approach we could find is the recent work on the loss reweighting using the Logistic-Normal (LN) distribution, which is a distribution over categorical distributions, similar to the Dirichlet distribution. We discuss LN in Section A.6, and even have it as a baseline. In short, LN can be seen as a special case of the more general log-ratio transform approach, corresponding to a particular transform (alr) and a Gaussian likelihood (zero-mean). In contrast, our adapted log-ratio transform approach, makes it possible to use any log-ratio transform (e.g., the ilr transform that does not suffer from the asymmetry issues of the alr transform), and also incorporate any regression technique, e.g., a shifted Gaussian noise model as in SGN, or Gaussian Processes, Student-T distributions, etc. Hence, in addition to the novelty of the adapted log-ratio transform, we propose the SGN method corresponding to the ilr transform and the shifted Gaussian noise model. Furthermore, we theoretically link the shift to changing the label, and propose to exploit this to achieve effects similar to the label correction methods from classification. To the best of our knowledge, this connection between the shift in the Gaussian noise and label correction is novel and one of our contributions. 

Another contribution is the practical algorithm for estimating the shift that differs from common label correction methods in classification that use buffers (e.g., ELR, SOP, NAL). In our work, we take a different approach, instead of using buffers, we use EMA networks to directly predict the labels/$\Delta$. Due to the memory requirements, we believe buffers do not scale to large datasets and or large number of classes, while our approach of using EMA networks do. Another limitation with the use of buffers is the storing of a single estimated label per example. This poses challenges, e.g., when using strong augmentation strategies. For example, with CutOut, there is a risk that a crucial part of the image is removed, leading to an unreliable prediction that still affects the estimate of the true label in the buffer. Similarly, the widely-used mixup technique encourages the prediction to have confidence in both of the mixed classes, potentially adding confidence in the wrong class, but the prediction still affects the buffer estimate for the example. This is not an issue of our approach, as the EMA network predicts labels for each augmented input separately. Furthermore, the buffers are useless at test time, whereas EMA networks can generalize significantly better than the original network.

Reviewer comment (Experiments):
The experiment results on Clothing1M and WebVision dataset with natural noise seem to fail compared with NAL, which though with a different evaluation setup, as Tabel 3 shows.

Author response:
**Empirical Performance on Clothing1M and WebVision**

We are glad the reviewer recognized the effectiveness of our method on the CIFAR datasets on synthetic and real noise. The reviewer claims SGN “fails” compared to NAL on Clothing1M and WebVision. This is incorrect, as on Clothing1M, SGN (73.9) outperforms/matches NAL (73.6). Furthermore, on WebVision, NAL has a different evaluation setup to all other methods, as mentioned in the text and the caption, and is why their results are marked with a star. All other methods evaluate the top 5 validation and top 1 and 5 test accuracies at the time of highest top 1 _validation_ accuracy. In contrast, NAL keeps track of the highest top 1 and 5 validation and test accuracies ever seen during training separately. That is, all results could come from different epochs (and even different training runs). Although it leads to higher reported results, we believe it is bad practice. With this in mind, the only valid comparison is on top 1 accuracy on the validation set, as there the evaluation setups are the same. There, NAL achieves 77.4, while SGN achieves 77.2. Is “fail” a fair description of SGN having 0.2 percentage points lower validation accuracy (i.e. having 5 fewer samples correctly predicted)?

Reviewer comment (Evaluation):
As Figure 2 shows, it takes around 400 epochs for the model to converge. However, figure 2 just depicts the train accuracy, and there can be a potential overfitting problem that the authors should consider.

Author response:
**Clarification on potential issue of overfitting**

The reviewer requests an investigation into any potential overfitting problem. To this end, we study the clean validation accuracy during training for various synthetic noise types and rates on CIFAR-10 and CIFAR-100, see Section A.8 in the appendix. Our results conclusively show that there are no signs of overfitting in any of these 16 cases.

Reviewer comment (Reproducibility):
It is not very clear to me that "The authors propose to have a two-layer neural network with parameters θ to output \mu and \sigma per example" in 3.3. How is the two-layer network trained and how does it work?

Author response:
**Clarification on network architecture**

We use a standard neural network (WideResNet 28-2, ResNet 50, and InceptionResNetV2) and replace the last linear layer with two separate linear layers (heads), one for predicting $\mu$ and one for predicting the rank-one factor $r$ of the scale matrix. The parameters of the networks are learnt exactly as in a standard classification with the cross-entropy loss, i.e., with backpropagation and SGD. We will make this clearer in the paper.

Reviewer comment (Experiments):
The experimental results is lower than SOTA performance, e.g., DivideMix, ELR, etc.

Author response:
**State-of-the-art Performance**

Our goal with this work is to introduce an adaptation of the well-founded log-ratio transform approach to the classification with noisy labels literature, and show the promise of the direction by devising a method unifying the ideas of loss reweighting and label correction from classification. Despite its simple design, our method empirically performs well against many strong and recent baselines, such as GCE, ELR, SOP, NAL, etc. The reviewer claims a weakness of our paper is that our method has lower performance compared to SOTA methods, and gives the example of DivideMix and ELR. In fact, we do compare with ELR, and our method achieves as good or better performance on CIFAR-10 and CIFAR-100 with synthetic noise, and on natural noise on the CIFAR-N datasets, Clothing1M, and WebVision, see Tables 1, 2, and 3.

We believe comparing SGN with DivideMix is unfair as it is a significantly more complex method as it combines ideas from sample selection GMMs to identify noisy labels (e.g., [1]), with a semi-supervised learning method [2], with co-training of two networks at the same time (e.g., [3]), and also use stronger data augmentation like mixup. Please see the “Rationale for chosen baselines” paragraph in the response to Reviewer QVE6 for a longer discussion.

We believe achieving strong performance is one important point of comparison, but does not take into account other possible empirical/practical advantages. Although DivideMix achieves excellent empirical performance, which is incredibly important for practitioners, we argue it is hard to fully understand due to all the different components. Our method is simpler in that it follows directly from a Gaussian noise model, leading to an interpretable robustness mechanism in terms of loss reweighting and label correction. Furthermore, DivideMIx was specifically created to be excellent at this particular task, but due to its complexity we believe it can be hard to extend/generalize to other problems. We believe our well-founded probabilistic method is easier to extend, e.g., it can naturally incorporate epistemic uncertainty (e.g., through Monte-Carlo Dropout [4] or Gaussian Processes [5]), making it interesting to both the noisy labels and the uncertainty communities.

Finally, we would like to point out that the reviewing guidelines (https://iclr.cc/Conferences/2024/ReviewerGuide) are clear that a lack of state-of-the-art results does not by itself constitute grounds for rejection, see the FAQ.

Reviewer comment (Novelty):
There exist some unified methods that combines loss reweighting and label correction to enhance robustness against label noise, e.g., CMW-Net [1].

Author response:
**Missing related work**

We performed a thorough literature study to select relevant baselines from the loss reweighting and or label correction categories of methods from top conferences and journals. Unfortunately, we missed this very recent work proposing the meta-learning based CMW-Net method.
 
CMW-Net is motivated in the context of the exciting research direction into learning loss weights using meta learning. Based on the title, abstract, introduction, and most of the method section, the main novelty seems to extend the meta-learned weight network of MW-Net [6] to also consider extra (task/class) information, leading to more flexibility in learning weighting schemes. However, in the last subsection of the method section (Section 3.6), CMW-Net is enhanced with label correction (denoted CMW-Net-SL). Instead of learning a loss weight, the CMW-Net now learns an interpolation parameter between the original one-hot label and a soft label estimate (Equation 14 can be rewritten to the form in Equation 13, in the same way as Equation 12 was), and the new interpolated label is then used in a standard CE loss (no loss reweighting). Therefore, although the proposed approaches are interesting and novel, we argue neither CMW-Net (loss reweighting) nor CMW-Net-SL (label correction) unifies loss reweighting and label correction. In future work, it would be interesting to explore ideas from CWM-Net-SL to learn $\Delta$ in SGN. Thank you for bringing our attention to this recent related work, which we will describe and reference as there are clear connections.

Reviewer comment (Novelty):
What's merit of the regression task. Some strong results and theoretical explanations are necessary.

Author response:
**Motivation for transforming classification to regression**

We agree that a motivation for transforming classification to regression deserves a thorough discussion, which we have now provided in Section A.7. In short, one of the main reasons is that this opens up a new research direction to leverage sound and time-tested methods from the (robust) regression literature that spans centuries. Thank you for pointing this out, as we believe this improves the motivation of our approach. 

Note that transforming the classification labels to regression labels can easily be done with a pre-processing step or on the fly during training, similar to label smoothing. At test time, instead of applying softmax to the logits as is done for standard softmax classification networks, we instead apply the ilr transform. This is trivial to implement as it requires minimal changes, and is therefore not cumbersome at all.

Our goal with this work is to introduce an adaptation of the well-founded log-ratio transform approach to the classification with noisy labels literature, and show the promise of the direction by devising a method unifying the ideas of loss reweighting and label correction from classification. Theoretically, it is not possible to achieve loss reweighting with the standard categorical likelihood, but our regression approach makes it possible to naturally achieve this with the Gaussian likelihood (higher variance corresponds to a lower weight). Furthermore, the Gaussian likelihood makes it possible to model shifted noise, which we theoretically show is similar to label correction methods in classification. Additionally, in Section A.7, we go over several other advantages of the  regression perspective, see e.g., “Uncertainty Estimation” and “Deep Learning Theory” paragraphs. Finally, we believe our experiments have clearly shown the effectiveness of our regression approach compared to several recent and strong baselines, e.g., ELR, SOP, and NAL.

Reviewer comment (Theory):
What are the advantages of using regression to solve classification tasks rather than directly solving classification tasks? Would it be too cumbersome to convert the classification problem into a regression problem, and then convert the regression problem into a classification problem?

Author response:
**Motivation for transforming classification to regression**

We agree that a motivation for transforming classification to regression deserves a thorough discussion, which we have now provided in Section A.7. In short, one of the main reasons is that this opens up a new research direction to leverage sound and time-tested methods from the (robust) regression literature that spans centuries. Thank you for pointing this out, as we believe this improves the motivation of our approach. 

Note that transforming the classification labels to regression labels can easily be done with a pre-processing step or on the fly during training, similar to label smoothing. At test time, instead of applying softmax to the logits as is done for standard softmax classification networks, we instead apply the ilr transform. This is trivial to implement as it requires minimal changes, and is therefore not cumbersome at all.

Our goal with this work is to introduce an adaptation of the well-founded log-ratio transform approach to the classification with noisy labels literature, and show the promise of the direction by devising a method unifying the ideas of loss reweighting and label correction from classification. Theoretically, it is not possible to achieve loss reweighting with the standard categorical likelihood, but our regression approach makes it possible to naturally achieve this with the Gaussian likelihood (higher variance corresponds to a lower weight). Furthermore, the Gaussian likelihood makes it possible to model shifted noise, which we theoretically show is similar to label correction methods in classification. Additionally, in Section A.7, we go over several other advantages of the  regression perspective, see e.g., “Uncertainty Estimation” and “Deep Learning Theory” paragraphs. Finally, we believe our experiments have clearly shown the effectiveness of our regression approach compared to several recent and strong baselines, e.g., ELR, SOP, and NAL.

Reviewer comment (Novelty):
The previous methods regard loss reweighting and label correction as two independent processes. So what is the advantage of treating them as a unified method in this paper?

Author response:
**Advantage of our unified method compared to combinations of separate methods**

In general, we believe unified methods are often simpler to implement, understand, analyse, and extend. Furthermore, in practice, combining different components that weren’t designed to work together can cause friction, e.g., due to power struggles between the two components that could require extra balancing hyperparameters.

In particular, NAL is a loss/gradient reweighting method that is also combined with a separate label correction method to improve performance. As these methods are separate, it is hard to understand the method from a probabilistic perspective. For example, it is unclear how soft labels in classification should be interpreted. How are these related to standard class ID labels? Should one interpret this as changing and having multiple new observed labels? In contrast, our simple method follows directly from a shifted Gaussian noise model, where both the label correction (shift $\Delta$) and the loss reweighting (covariance $\Sigma$) have clear roles. Hence, our approach has a different interpretation completely: instead of changing the observations, our view is strictly related to a label noise model with the original observations intact.

Reviewer comment (Experiments):
The main criticism is on the paper's experimental setup and results, specifically that the selected baselines to report appear to be a very biased set in an attempt to make the method look SOTA, unless the authors reimplemented the method and reran the numbers (which is unclear).

Author response:
**Training and Evaluation Setup**

Indeed, to have fair and conclusive experiments, we _reimplement all baseline methods_ into a shared code base where we have full control over the training and evaluation setup. This is mentioned in the first sentence in Section 4.1 and in the captions of Tables 1 & 2, but unfortunately the reviewer missed this. We have gone to great lengths to achieve the best possible results for the baselines. For example, in the literature, it is common to directly use the optimal hyperparameters reported in the baselines’ original papers. Instead, for our method and all baselines, we do a method-specific hyperparameter search (that includes the reported optimal HP) to find what works best in our training setup. Furthermore, to have conclusive results, we report the mean and standard deviation of five runs with different random seeds, and do statistical significance tests when comparing results. Due to the shared code base, all methods are equally affected by the random seed, leading to using the same weight initialization, data order in SGD, data augmentation, synthetic noisy labels, etc. 

Although we believe this setup leads to fair and conclusive results, training and evaluating all these methods require a lot of computational resources, which leads us to limit our studies to a smaller WideResNet 28-2 on the CIFAR datasets. As all methods use this network, our results are still fair and conclusive, but at the expense of a higher test error. Achieving excellent empirical performance is important for practitioners, but this was not the goal in this work. Our goal is to develop a simple well-founded statistical method that is still easy to implement, understand, extend, debug, and use in practice, and to compare this with existing related methods in an as fair and conclusive way as possible.

Reviewer comment (Experiments):
In most result tables, the proposed method appears as the best-performing method, but closer inspection suggests potential selection bias in the reported baselines. For example, at 60% symmetric noise in CIFAR10, Table 1 in the original ELR paper reports 86.12% mean accuracy, while Table 1 in the current paper reports 81.87%, which is significantly lower than the original authors' numbers.

Author response:
**Issues in directly comparing with results from other papers**

You bring up a good point about our results being lower than those reported in the ELR paper. As mentioned above, the main difference in our setups is that they use a larger network (ResNet 34) than us (WideResNet 28-2). To highlight the difference, their results with the notoriously overfitting CE loss on 10% asymmetric noise outperforms our CE results with no added label noise. This makes it clear that directly comparing results with other papers is not valid due to differences in training and evaluation setups. If one is only interested in comparing with results from other papers, our setup is very similar to NAN [4]. Our results with SGN are significantly better than all the reported results in that paper. For example, on 40% asymmetric noise on CIFAR-100, their best test accuracy is 63.90, whereas our SGN achieves 71.01.

Reviewer comment (Experiments):
The original ELR paper proposes a variant with add-ons, ELR+, which can outperform ELR, but the current authors did not include it. While such tricks might not be the main contribution of ELR, the reported SGN in the current paper also employed a few tricks (e.g., EMA) to improve upon its vanilla version. According to Table 5 of the original ELR paper, weight averaging alone can also boost ELR's vanilla performance by a few percentage points.

Author response:
**Response to: Training tricks/add-ons**

1. We interpret the reviewer’s argument to be that it is unclear how much of the improvement of SGN over ELR is due to using an EMA network instead of a buffer. If so, we believe this is a point that requires either changing our SGN method to use buffers, or ELR to use an EMA network instead of a buffer, and perform experiments to explore this. Unfortunately, there is not enough time left of the discussion period to properly perform experiments to look into this, and the reviewer mentioned they deemed such experiments unnecessary anyway. However, we believe the ablation study in Table 4 clearly shows that all components of SGN are important: loss reweighting, label correction and EMA predictions. Furthermore, our latest results suggest SGN has other benefits over just incorporating an EMA network, as SGN with mixup outperforms ELR+ (which includes mixup, EMA networks, and other add-ons) on the WebVision dataset. 
2. We thank the reviewer for clarifying their practical perspective on incorporating add-ons on top of vanilla methods. We agree that ablation studies are important in understanding the performance impact of different parts and add-ons of methods. Furthermore, we do see the relevance of incorporating add-ons for practitioners. Having said that, our point is that one has to strike a good balance between catering to practitioners and researchers. We are concerned that the research community may be heading towards overly catering to practitioners and making it virtually a requirement to incorporate add-ons and achieve state-of-the-art performance. We believe there should be room for novel directions that show conclusive promise but not yet necessarily achieve SOTA performance, as exploration is indeed a key aspect of science.

Reviewer comment (Experiments):
There are many other baselines in learning with label noise, such as DivideMix, that can achieve much higher accuracy, yet these methods are not included in the current paper. The reasons for their exclusion are not disclosed. Even if comparisons to such methods do not make sense or are unfair, the rationale behind why some methods are selected and others omitted should be provided.

Author response:
**Rationale for chosen baselines**

We agree that justifying the choice of baselines is important, we will add discussion similar to the one below to the final submission. Thanks for letting us clarify our rationale for choosing baselines.

_Methods that build on similar ideas._ We have tried our hardest to find relevant and recent baselines that are related to loss reweighting and label correction from top conferences and journals. For example, in Section A.6, we argue there are clear connections between our work and methods like ELR, SOP, LN, and NAL, that can either be seen as loss reweighting and or label correction methods. We argue this is the most important comparison, as we see how our method currently compares with the methods that have taken a similar approach.

_Other noisy labels methods._ Furthermore, we compare with many strong and commonly used baselines outside the loss reweighting and label correction categories, e.g., GCE from the robust loss functions category, and LS and NAN from the regularizations methods. This is informative to understand how our categories currently compare in terms of performance to other categories of algorithms to improve robustness against label noise. In our view, performing worse than baselines from other categories is not necessarily negative, as the research is in different stages of development, and categories of methods performing poorly today could achieve SOTA performance in the future. Therefore, these comparisons are intended for laying out context, and we believe it is important to consider and spread a diverse set of ideas and not limit the research field to what currently achieves the best performance. 

_SOTA methods._ Our goal in this work is to adapt the well-founded log-ratio transform approach to classification and apply it to the problem of label noise robustness by proposing a simple and unified method for loss reweighting and label correction. If our goal instead had been to achieve the current SOTA performance (a different but also important goal), we would have taken a different empirical direction. In general, the current SOTA methods are complex and compute-intensive that typically combine several methods. For example, in our training setup, our method has comparable training time to ELR, while ELR is twice and five times faster to train compared to ELR+ and DivideMix (Table I.1 in the appendix of the ELR paper), respectively.

Reviewer comment (Presentation):
Appendix A.6 contains a very interesting discussion that adds significant value to understanding the method in the context of related works. It is suggested that this discussion be moved to the main text, while some results/tables could be moved to the appendix instead.

Author response:
**Connections to related work**

We are excited that you also found our discussion on the connections to related work to be valuable for understanding. Based on your suggestion, we will incorporate this discussion in the main paper.