Paper: A Curriculum View of Robust Loss Functions (ICLR-2024)

Reviewer comment (Experiments):
The robust losses involves the curriculum view, i.e., a sample-weighting perspective, the paper should include more discussion and empirical comparisons with curriculum and reweighting noisy-label learning methods.

Author response:
We have discussed the relation to existing curriculum approaches in the related work section (now the second paragraph in the revised version). 

We believe that empirical comparisons with existing sample-weighting curricula is optional, or even orthogonal to our experiments aiming to **understand existing loss functions**:

- Results in section 4.1 analyze the effects of different implicit sample-weighting functions $w(\boldsymbol{s}, y)$ of loss functions **considered in this work**. We do no claim for a comprehensive analysis of all sample weighting curricula.
- Results in section 4.2 analyze the effects of the identified regularizers within loss functions considered in this work. They are orthogonal to existing sample-weighting curricula.
- Results in section 4.3 aim to identify the cause of underfitting. We first establish the relation between minimal sample weights to underfitting. We then theoretically connects minimal sample weights to the increased number of classes, which leads to minimal $\Delta(\boldsymbol{s}, y)$ and thus minimal weights at initialization with robust loss functions. We then test whether increasing the sample weights with small $\Delta(\boldsymbol{s}, y)$ helps mitigate underfitting:
  - Better results with our fixes compared to the underfitting loss functions support our explanation.
  - Results of previous state-of-the-art under the same experiment settings reflect the significance of the improvement  (we choose results from "Asymmetric loss functions for learning with noisy labels")
  - We never claim for a "novel approach for new state-of-the-art performance", which does require comprehensive comparison with major approaches of noisy-robust learning.

Given the page limitation, including additional empirical results of sample-weighting curriculums requires remove other important discussions of our unified standard form. We will reproduce some sample-weighting curricula in the appendix in later revisions.

Reviewer comment (Theory):
Eq.3 appears to rely on a number of assumptions, which should be clarified in the formulation of these assumptions.

Author response:
We have thoroughly revised our derivations in Section 3 to clarify the assumptions and intuitions. Eq.3 $L'(\boldsymbol{s}, y) = - w(\boldsymbol{s}, y) \cdot \Delta(\boldsymbol{s}, y)$ is based on a mild assumption that we are using first order optimizers based on gradients.

Reviewer comment (Theory):
As one of the main formulas of the paper, it is not quite clear how Eq.4 was obtained. More details should be provided.

Author response:
As mentioned above, the revised derivations in Section 3 makes Eq. 4 much clearer. We describe the example derivation of NCE towards Eq.4 before presenting the detailed standard form of Eq.4. Intuitively, we extract the regularizer $R(\boldsymbol{s})$ and rearrange other terms depending only on $\Delta(\boldsymbol{s}, y)$ into the primary loss function. If there is no explicit way to rearrange in the original loss terms, we turn to the gradients of $\boldsymbol{s}$, try rearranging the gradient terms, and then integrate over $\boldsymbol{s}$ after detaching weighting terms from derivative and integral. See the Appendix A.5 for detailed derivation for each loss functions.

Reviewer comment (Novelty):
Although a novel perspective that includes many loss functions is proposed, this perspective is unable to guide us to obtain any better robust loss functions, except for the simple fix to alleviate the underfitting issue.

Author response:
It would be unfair to state that 

> this perspective is unable to guide us to obtain any better robust loss functions, except for the simple fix to alleviate the underfitting issue

We should stress that:

- Unifying loss functions **with different theoretical motivations** into the **same** dimensions of design choices
- Connecting the design of loss functions to the design of learning curricula, which are commonly considered as distinct approaches.

are theoretically **nontrivial**. These approaches are previously understood and analyzed with **distinct** approaches and motivations. Our theoretical derivations provide a **common** ground to understand and analyze them.

As loss functions can be viewed as sample-weighting curricula, our findings suggest plenty of approaches to directly design them. As a quick list:

1. Blending the implcit curricula of robust loss functions to those explicitly designed, e.g., approach proposed in "DivideMix: Learning with Noisy Labels as Semi-supervised Learning"
2. We can use other metrics than $\Delta(\boldsymbol{s}, y)$ to identify the constantly unlearned samples, e.g., $\Delta(\boldsymbol{s}, y) - m$  where $m$ is a running average of $\Delta(\boldsymbol{s}, y)$, and assign small weights to small $\Delta(\boldsymbol{s}, y) - m$, which can avoid the underfitting issue due to minimal initial sample weights.
3. Use a proper regularizer for a good $\Delta(\boldsymbol{s}^*, y)$ with $\boldsymbol{s}^* = \arg\min_{\boldsymbol{s}} R(\boldsymbol{s})$ to facillitate learning with a large number of classes and avoid overfitting noisy samples. Existing confidence reducing regularizers all share the same minimum with $\Delta(\boldsymbol{s}^*, y) = - \log k$ where $k$ is the number of classes, which may risk underfitting with robust loss functions under large $k$.
4. Utilize the sample sifting effect of robust loss functions to **select** corrupted samples and either drop them or utilize semi-supervised learning to obtain surrogate labels.

Reviewer comment (Experiments):
To me, the fix derived from the curriculum view is the central part of this paper. However, this paper did not provide extensive experiments to empirically validate this method. Without clearly and extensively demonstrating the effectiveness, the key contribution of this paper seems limited.

Author response:
It would be unfair to view the fixes for underfitting the central part of the paper, which ignores the significance of our theoretical framework and the following empirical insights:

1. Robust loss functions act as a sift to neglect corrupted samples. This finding provides a distinct view compared to the theoretical bounds of these loss functions.
2. Regularizers like weight decay is important to the learning of robust loss functions. It suggests that one should carefully tune the regularization with robust loss functions.
3. Underfitting is due to minimal sample weights at initilizations, which is empirically supported by our fixes.

We have make these points more pronounced in the revised Introduction. 

We focus more on comphrehensively understand the loss functions considered in this work. The empirical performance of our fix aims to confirm whether avoiding minimal initial samples weights can improve the underfitting loss functions. The effectiveness on the underfitting MAE (and TCE in our revision) supports our explanation.



As for the questions,

> Is this method versatile enough? 

We don't understand what "versatile" means. Our fixes apply to underfitting with minimal intial sample weights. Nevertheless, our analysis of underfitting can help diagnose the cause in other settings. Notably, the novel curriculum view and the resulting analysis are more versatile than methods tailored for specific settings.

> Can any robust loss functions (excluding MAE) be equipped with this method? 

Yes. As shown in Table 4, TCE also suffers to moderate underfitting. We include the results of our fixes to TCE on CIFAR100 in the revised Tables 5,8,9:

|                         | Clean            | Symmmetric       | Symmetric        | Asymmetric       | Human            |
| ----------------------- | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- |
|                         | $\eta=0$         | $\eta =0.4$      | $\eta =0.8$      | $\eta =0.4$      | $\eta =0.4$      |
| CE                      | 77.44 $\pm$ 0.13 | 49.96 $\pm$ 0.02 | 11.23 $\pm$ 0.45 | 45.73 $\pm$ 0.49 | 54.40 $\pm$ 0.35 |
| GCE                     | 73.88 $\pm$ 0.25 | 64.67 $\pm$ 0.49 | 23.90 $\pm$ 2.69 | 45.14 $\pm$ 0.13 | 56.95 $\pm$ 0.63 |
| NCE+AGCE                | 76.37 $\pm$ 0.25 | 64.55 $\pm$ 0.46 | 26.19 $\pm$ 1.14 | 40.93 $\pm$ 1.22 | 53.67 $\pm$ 0.18 |
| TCE                     | 58.04 $\pm$ 1.15 | 45.91 $\pm$ 1.25 | 20.47 $\pm$ 1.45 | 28.35 $\pm$ 0.74 | 32.22 $\pm$ 1.22 |
| TCE shift, $\tau = 4.2$ | 77.14 $\pm$ 0.11 | 60.17 $\pm$ 0.47 | 18.16 $\pm$ 0.37 | 44.56 $\pm$ 0.71 | 55.07 $\pm$ 0.19 |
| TCE scale, $\tau = 4.2$ | 75.79 $\pm$ 0.17 | 62.88 $\pm$ 0.59 | 20.78 $\pm$ 1.53 | 43.57 $\pm$ 1.19 | 56.13 $\pm$ 0.22 |
| MAE                     | 7.46 $\pm$ 1.92  | 4.65 $\pm$ 1.55  | 3.21 $\pm$ 0.57  | 1.61 $\pm$ 0.53  | 1.54 $\pm$ 0.47  |
| MAE shift, $\tau = 3.4$ | 76.65 $\pm$ 0.30 | 61.29 $\pm$ 0.49 | 19.30 $\pm$ 1.00 | 44.06 $\pm$ 1.23 | 54.83 $\pm$ 0.49 |
| MAE scale, $\tau = 3.4$ | 73.54 $\pm$ 0.32 | 64.92 $\pm$ 0.20 | 23.00 $\pm$ 2.44 | 48.88 $\pm$ 0.79 | 57.56 $\pm$ 0.41 |

Results on WebVision will be included in later revision as they requires weeks to converge.

> Can this method work well on a variety of large-scale datasets?

Yes. To our knowledge, WebVision is the largest noisy dataset of image classification. We have extend the standard mini setting (50 classes) to include more classes (200 and 400), which are much more challenging than settings in previous research.

Reviewer comment (Experiments):
Is this method versatile enough?

Author response:
We don't understand what "versatile" means. Our fixes apply to underfitting with minimal intial sample weights. Nevertheless, our analysis of underfitting can help diagnose the cause in other settings. Notably, the novel curriculum view and the resulting analysis are more versatile than methods tailored for specific settings.

Reviewer comment (Experiments):
Can any robust loss functions (excluding MAE) be equipped with this method?

Author response:
Yes. As shown in Table 4, TCE also suffers to moderate underfitting. We include the results of our fixes to TCE on CIFAR100 in the revised Tables 5,8,9:

|                         | Clean            | Symmmetric       | Symmetric        | Asymmetric       | Human            |
| ----------------------- | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- |
|                         | $\eta=0$         | $\eta =0.4$      | $\eta =0.8$      | $\eta =0.4$      | $\eta =0.4$      |
| CE                      | 77.44 $\pm$ 0.13 | 49.96 $\pm$ 0.02 | 11.23 $\pm$ 0.45 | 45.73 $\pm$ 0.49 | 54.40 $\pm$ 0.35 |
| GCE                     | 73.88 $\pm$ 0.25 | 64.67 $\pm$ 0.49 | 23.90 $\pm$ 2.69 | 45.14 $\pm$ 0.13 | 56.95 $\pm$ 0.63 |
| NCE+AGCE                | 76.37 $\pm$ 0.25 | 64.55 $\pm$ 0.46 | 26.19 $\pm$ 1.14 | 40.93 $\pm$ 1.22 | 53.67 $\pm$ 0.18 |
| TCE                     | 58.04 $\pm$ 1.15 | 45.91 $\pm$ 1.25 | 20.47 $\pm$ 1.45 | 28.35 $\pm$ 0.74 | 32.22 $\pm$ 1.22 |
| TCE shift, $\tau = 4.2$ | 77.14 $\pm$ 0.11 | 60.17 $\pm$ 0.47 | 18.16 $\pm$ 0.37 | 44.56 $\pm$ 0.71 | 55.07 $\pm$ 0.19 |
| TCE scale, $\tau = 4.2$ | 75.79 $\pm$ 0.17 | 62.88 $\pm$ 0.59 | 20.78 $\pm$ 1.53 | 43.57 $\pm$ 1.19 | 56.13 $\pm$ 0.22 |
| MAE                     | 7.46 $\pm$ 1.92  | 4.65 $\pm$ 1.55  | 3.21 $\pm$ 0.57  | 1.61 $\pm$ 0.53  | 1.54 $\pm$ 0.47  |
| MAE shift, $\tau = 3.4$ | 76.65 $\pm$ 0.30 | 61.29 $\pm$ 0.49 | 19.30 $\pm$ 1.00 | 44.06 $\pm$ 1.23 | 54.83 $\pm$ 0.49 |
| MAE scale, $\tau = 3.4$ | 73.54 $\pm$ 0.32 | 64.92 $\pm$ 0.20 | 23.00 $\pm$ 2.44 | 48.88 $\pm$ 0.79 | 57.56 $\pm$ 0.41 |

Reviewer comment (Experiments):
Can this method work well on a variety of large-scale datasets?

Author response:
Yes. To our knowledge, WebVision is the largest noisy dataset of image classification. We have extend the standard mini setting (50 classes) to include more classes (200 and 400), which are much more challenging than settings in previous research.

Reviewer comment (Theory):
What is the motivation for rewriting the losses in this particular way? What novel perspective does this form give over something more intuitive like the $p_y$ and the gradient magnitude of the loss with respect to the logits? For example, in Figure 1, couldn’t one see the same thing with an x-axis with $p_y$ instead of $\Delta$, and the y-axis being the gradient magnitude with respect to the logits for the MAE green curve instead of $w$?

Author response:
The key motivation is to unify loss functions **with different theoretical motivations** into the **same** dimensions of design choices: the sample-weighting strategy and the output regularizer, where the shared implicit loss function $\Delta(\boldsymbol{s}, y)$ provides a common ground to analyze their properties

Regarding the mentioned metrics for analysis,

- Although $p_y$ and $\Delta$ can both track the learning of samples, $p_y$ applies a sigmoid transform on top of $\Delta$ that ignores the change of large $\|\Delta\|$, making the change around initialization and convergence too subtle to be noticed. For example, with CE + NLS (Figure 2 f), one cannot notice the improper scale and risk of numerical overflow when viewing the distributions in $p_y$.  We discuss the benefits of $\Delta$ in the revised paragraph after Eq. (3).

- Gradient magnitude can compare the loss functions **only when the shared gradient direction are extracted**, i.e., $\nabla_\boldsymbol{s} \Delta$. When regularizers are involved, e.g., MSE, our standard form helps separate the effects of regularizers, which is hard to achieve when only considering the gradient magnitude.

Reviewer comment (Theory):
How isn’t it trivial that different loss functions have different gradient weights, and therefore different sample weights? If all loss functions penalized the same, there would be no reason to have different ones?

Author response:
Different loss functions can have different gradient **directions** and **sample weights** simultaneously. The fact that loss functions with distinct theoretical motivations share the same $\nabla_\boldsymbol{s} \Delta$ gradient component is **nontrivial**.

Reviewer comment (Theory):
A single sentence motivating the inequality in Equation 5 would make it clearer.

Author response:
In our revision, we remove the inequality in Equation 5 and add "where $\mathbb{E}_k[\Delta(\boldsymbol{s}, y)]< 0$ with standard initialization $\sigma \approx 1$ and a large $k$." right after it.

Reviewer comment (Writing):
The clarity of several equations could be improved:  
i) The equation between Equations 2 and 3 is crucial and could be made much clearer: a single sentence discussion or motivation for using stop gradient, what $\Delta_y$ is, why two additional minus signs are added in the middle equation, and as $w$ is a key component of the paper, it deserves a proper definition.  
ii) The equation at the end of page 3: why are stop gradients used in that derivation and why is a factor of k and 1/k introduced?

Author response:
We have thoroughly revised Section 3 for clearer intuition of our derivations.

i): what we really mean is to detach $w(\boldsymbol{s}, y)$ from any computation of derivative & integral, thus preserving the same gradient in the standard form. We have rephrased "stop gradient operator" to "detach operator".

ii): $\Delta_y$ is simply a typo for $\Delta(\boldsymbol{s}, y)$

iii): it aims to make $w(\boldsymbol{s}, y)$ a positive function, and facilitates a direct substitution given that we do not explicitly define $w(\boldsymbol{s}, y)$

iv): we dub $w(\boldsymbol{s}, y)$ the **sample-weighting function** and include its definition in our revision right after Eq. (3)

i): the stop gradient issue is similar to Q4.1, i).

ii): it makes the later extraction of $R_{\mathrm{NCE}}(\boldsymbol{s}) = \sum_{i=1}^{k} \frac{1}{k}\log p_i$ more straightforward. It shares similar form as confidence regularizer $R_{\mathrm{CR}}(\boldsymbol{s}) =\sum_{i=1}^{k} P(y = i) \log p_{i}$, helping the readers to observe their connection -- we can obtain $R_{\mathrm{NCE}}(\boldsymbol{s})$ by setting $P(y = i) = \frac{1}{k}$.

Reviewer comment (Theory):
It was unclear why the more general form that accounts for regularization was introduced when its properties like $R(s)$ were never studied.

Author response:
We devote the entire section 4.2 to study the effect of different $R(\boldsymbol{s})$. At the beginning of section 4, we mention that $R(\boldsymbol{s})$ constrains the distributions of $\Delta(\boldsymbol{s}, y)$ towards a predefined optimum $\Delta(\boldsymbol{s}^*, y)$ where $\boldsymbol{s}^* = \arg\min_{\boldsymbol{s}} R(\boldsymbol{s})$. In our revision, we make it more pronounced right after the introduction of the standard form in Eq. (4).

Reviewer comment (Evaluation):
Reporting mean and standard deviation would improve the conclusiveness of the observations in the tables. Furthermore, the network predictions from several runs could be used to have more reliable histogram estimates (more data).

Author response:
Results in table 3-4 are reported with 3 different runs but there is no enough space to include the std. Results of WebVIsion in Table 6 are too expensive to conduct multiple random runs as stated in the caption.

We would update the histogram estimates in later revisions. However, the size of training set already bears a statistically reliable estimation of the histograms.

Reviewer comment (Novelty):
What novelty and significance does this work add over that of Wang et al. [1]?

Author response:
This question is similar to Weakness Q1 and Q2. Gradient magnitude can compare the loss functions **only when a shared gradient direction are extracted**, i.e., $\nabla_\boldsymbol{s} \Delta(\boldsymbol{s},y)$. We explicitly factorize the scale $w(\boldsymbol{s}, y)$ and direction $\nabla_\boldsymbol{s} \Delta(\boldsymbol{s}, y)$ of the gradient wrt logits $\boldsymbol{s}$ for a wide range of loss functions.

When no output regularizer $R(\boldsymbol{s})$ is involved, $w(\boldsymbol{s}, y)$ and gradient magnitude are equivalent. When $R(\boldsymbol{s})$ is implicitly involved, e.g., MSE and NCE, loss functions can have different gradient directions, making comparisons of gradient magnitudes improper. In contrast, $w(\boldsymbol{s}, y)$ and $R(\boldsymbol{s})$ leads to a proper comparison among loss functions.

Reviewer comment (Theory):
What are the benefits of viewing the losses in terms of the gradient of $\Delta$ rather than the more natural gradient magnitude with respect to the logits?

Author response:
This question is similar to Weakness Q1 and Q2. Gradient magnitude can compare the loss functions **only when a shared gradient direction are extracted**, i.e., $\nabla_\boldsymbol{s} \Delta(\boldsymbol{s},y)$. We explicitly factorize the scale $w(\boldsymbol{s}, y)$ and direction $\nabla_\boldsymbol{s} \Delta(\boldsymbol{s}, y)$ of the gradient wrt logits $\boldsymbol{s}$ for a wide range of loss functions.

When no output regularizer $R(\boldsymbol{s})$ is involved, $w(\boldsymbol{s}, y)$ and gradient magnitude are equivalent. When $R(\boldsymbol{s})$ is implicitly involved, e.g., MSE and NCE, loss functions can have different gradient directions, making comparisons of gradient magnitudes improper. In contrast, $w(\boldsymbol{s}, y)$ and $R(\boldsymbol{s})$ leads to a proper comparison among loss functions.

Reviewer comment (Novelty):
The finding that “MAE does not treat samples equally and instead focuses on examples with moderate loss/$\Delta$” is not novel in this work. This is one of the main findings of Wang et al., which is clearly shown in their Figure 1 (or 2 depending on the version of the paper), where examples with low and high $p_y$ have small gradients.

Author response:
We modify the statement to "In contrast, we show that MAE emphasizes samples with moderate $\Delta(\boldsymbol{s}, y)$ which is also observed by Wang et al. (2019a)."

Reviewer comment (Novelty):
The quote “.. attribute underfitting to their low variance, making clean and noise samples less distinguishable. But as shown in Table 4 MAE can underfit data with clean labels.” misrepresents the related work. Wang et al. did not mention that the variance was low between clean and noisy samples, but rather the more general “informative” and “uninformative” examples. Furthermore, the finding that MAE underfits clean examples is not novel, and Wang et al. clearly show this in their Table 1.

Author response:
We do misinterpret their statement.  In our revision in the first paragraph of section 4.3, we rephrased our statement to "Wang et al. (2019a) view $\|\nabla_{\boldsymbol{s}} L(\boldsymbol{s}, y)\|_1$ as weights for sample gradients and argue that their low variance makes informative and uninformative samples less distinguishable. However, it is unclear how low variance of gradient magnitudes leads to underfitting." Indeed, they do not explicitly explain why low variance of gradients can lead to underfitting.

We do not claim that MAE underfits clean data is novel -- it is well known in previous research.

Reviewer comment (Novelty):
Wang et al. already proposed a fix for MAE. Therefore, the novelty and significance of the proposed fix in this paper is unclear. A proper comparison (theoretically and experimentally) with other fixes for MAE is required.

Author response:
We should emphasize that we focus more on how our unified framework helps understanding the underfitting issue. Previous explanations have been thoroughly reviewed at the beginning of section 4.3, which are all flawed in various aspects. The proposed fixes merely aim to support the validity of our explanation:

- that minimal initial sample weight leads to underfitting as shown in our analysis
- that increasing initial sample weight indeed addresses underfitting

We stress this point in the revised abstract and introduction. A comparable performance of our fixes to selected state-of-the-art results already serves our purpose. Comparisons to other fixes would be nice to have but unnecessary. 

We include discussions and empirical comparisons of IMAE to Wang et al. in the revised section 4.3 using our experimental settings.  See the updated Figure 4(c) for a visualization of $w(\boldsymbol{s},y)$ and Table 5,8,9 for results on CIFAR100. We find that IMAE is very sensitive to hyperparameters. Results on WebVision requires a thorough tuning of hyperparameters for IMAE, which requires more time than the rebuttal period. We will include them in later revisions.

Reviewer comment (Novelty):
The significance of the proposed way of rewriting the loss functions is unclear. If one instead does similar studies in terms of loss values (e.g., Figure 3 in [1], and Figure 2 in [2]) or gradients (e.g., Theorem 1 in [3]), it seems many of the findings in the paper are already well-known — for example, that the gradients of clean examples dominate the early learning phase and then the gradients for noisy-labelled examples take over, resulting in overfitting [3]. Another example, that MAE focuses on moderate loss examples, is shown by Wang et al. (2019a). If there are any novel and significant findings, the authors should state, relate, and discuss them much more clearly compared to related work.

Author response:
We should again stress that:

- Unifying loss functions **with different theoretical motivations** into the **same** dimensions of design choices

- Connecting the design of loss functions to the design of learning curricula, which are commonly considered as distinct approaches. 

are theoretically **nontrivial**. These approaches are previously understood and analyzed with **distinct** approaches and motivations. Our theoretical derivations provide a **common** ground to understand and analyze them.

We have summarized our key empirical findings in the last paragraph of the revised introduction:

- Robust sample-weighting functions act as a sift to neglect corrupted samples
- Regularizers can help the learning of robust loss functions
- Underfitting is due to minimal sample weights at initializations

Regarding the listed minor empirical findings:

Yes, our observation is similar to [4]. They conduct theoretical analysis on linear binary classification and intuitively extend it to the multiclass setting. We remove the emphasis of this point in our revised introduction.

The discussion of the relation between $\lim_{\Delta(\boldsymbol{s}, y) \to \infty} w(\boldsymbol{s}, y) = 0$ and the memoization effect is necessary for a comprehensive analysis of sample weighting functions $w(\boldsymbol{s}, y)$. We relate our results to [4] by commenting "Such explanation complements the theoretical analysis for binary linear classification in [4], which reach similar conclusion that corrupted samples dominate the expected gradients in the late training stage." in the revised last paragraph of section 4.1. 

We don't understand the meaning of this question. As a speculative response, we only briefly mention that regularization preventing the memorization stage helps noise-robustness of CE in the last paragraph of section 4.1. It is included only for a more thorough discussions on the properties of $w(\boldsymbol{s}, y)$. We never claim that this observation is novel.

In our revision, this point is clearly stated in the first paragraph of section 4.3, "In contrast, we show that MAE emphasizes samples with moderate $\Delta(\boldsymbol{s}, y)$ which is also observed by Wang et al. (2019a)."

We should stress that our analysis shows that **ALL** robust loss functions has sample-weighting function $w(\boldsymbol{s}, y)$ focusing on samples with moderate $\Delta(\boldsymbol{s}, y)$ while Wang et al. (2019a) made similar observation **ONLY** for MAE. See the visualization of Figure 5 in appendix A. We mention this point in the revised caption of Table 1.

Reviewer comment (Novelty):
The list of robust loss functions is comprehensive but missing some relevant ones based on information theory: i) f-divergences [5], ii) Bregman divergences [6], and iii) Jensen-Shannon divergences [7].

Author response:
Thanks for providing the list of related work! We include Jensen-Shannon divergences in the revised Table 2, which involves a new implicit output regularizer $R(\boldsymbol{s}) = \sum_i p_i$. 

Our analysis does not apply to f-divergences [5] and Bregman divergences [6] without closed-form expressions. We clearly state this in the third paragraph of the revised section 3, "We examine a broad array of loss functions with closed-form expressions in this work", and the first paragraph of the related work, "We only consider loss functions with closed-form expressions and leave others (Amid et al., 2019; Wei & Liu, 2021) to future work."

Reviewer comment (Novelty):
Why would one use your rewriting of the loss function to study the training dynamics of robust loss functions over, say, the gradient perspective in Wang et al. or that in the GCE paper? What novel findings do you, and can you, make only because of this framework?

Author response:
By only considering the gradient magnitude, both Wang et al. and the GCE paper 

- cannot identify the effects of the implicit regularizers of loss functions, e.g., MSE, NCE
- cannot correctly compare different loss functions. Gradient directions of the implicit regularizers makes $\|\nabla_{\boldsymbol{s}} L({\boldsymbol{s}}, y)\|_1$ of Wang et al. not comparable. Even without implicit regularizers, $\mathrm{d}L({\boldsymbol{s}}, y)/\mathrm{d} p_y$ of the GCE paper has an additional $p_y(1-p_y)$ factor in the sample gradients.

Our standard form leads to proper comparison among loss functions with $w(\boldsymbol{s}, y)$ and $R(\boldsymbol{s})$.

Reviewer comment (Novelty):
Most experiments, and the only proposed fix based on the analysis, are related to MAE. As Wang et al. has studied and proposed a fix for MAE, could you clarify what novelty you bring to the understanding of MAE, and why it is significant?

Author response:
In the first paragraph of the revised section 4, we explicitly state that "We mainly use MAE and CE for illustration as they exhibit typical empirical observations. Similar analysis of other loss functions are left to Appendix B." 

Wang et al. does not provide an explicit explanation for the underfitting of MAE. Their empirical results only support the fact that their fix helps mitigate underfitting. However, their claim that "gradient magnitude’s variance matters" is not empirically supported. Based on their empirical results, one can also attribute other metrics varied due to their fix, e.g., "the expected gradient magnitude matters". Thus in the first paragraph of the revised section 4.3, we comment that "Wang et al. (2019a) view $\|\nabla_{\boldsymbol{s}} L({\boldsymbol{s}}, y)\|_1$ as weights for sample gradients and argue that their low variance makes informative and uninformative samples less distinguishable. However, it is unclear how low variance of gradient magnitudes leads to underfitting."

In contrast, we support our explanation explicitly with empirical and theoretical results. 

- We first establish the connection between underfitting (measured by train/test accuracy) to small sample weights (measured by effective learning rate $\alpha^*$ ).
- We then theoretically connects the increased number of classes to the decreased $\Delta(\boldsymbol{s},y)$ and sample weights at initialization, and empirically support it with results in Figure 3. 
- We finally establish the causal relation between minimal initial sample weights and underfitting. The fact that increased initial sample weights with our fixes eliminates underfitting support this claim well.

Reviewer comment (Novelty):
Why would one use your fix for MAE over the fix proposed by Wang et al.?

Author response:
Our fix are less sensitive to the hyperparameter $\tau$: the same $\tau = 3.4$ works well for a wide range of noise rates with MAE, while IMAE Wang et al. requires careful hyperparameter tuning for each noise setting (stated in the appendix D of their paper). In addition, under the same settings, our fix achieves better results than IMAE.

Nonetheless, we recommend designing new learning curricula based on our framework and empirical findings instead of sticking with the classic but flawed MAE. As loss functions can be viewed as sample-weighting curricula, our findings suggest plenty of approaches to design good curricula. As a quick list:

1. Blending the implicit curricula of robust loss functions to those explicitly designed, e.g., approach proposed in "DivideMix: Learning with Noisy Labels as Semi-supervised Learning"
2. We can use other metrics than $\Delta(\boldsymbol{s}, y)$ to identify the constantly unlearned samples, e.g., $\Delta(\boldsymbol{s}, y) - m$  where $m$ is a running average of $\Delta(\boldsymbol{s}, y)$, and assign small weights to small $\Delta(\boldsymbol{s}, y) - m$, which can avoid the underfitting issue due to minimal initial sample weights.
3. Use a proper regularizer for a good $\Delta(\boldsymbol{s}^*, y)$ with $\boldsymbol{s}^* = \arg\min_{\boldsymbol{s}} R(\boldsymbol{s})$ to facilitate learning with a large number of classes and avoid overfitting noisy samples. Existing confidence reducing regularizers all share the same minimum with $\Delta(\boldsymbol{s}^*, y) = - \log k$ where $k$ is the number of classes, which may risk underfitting with robust loss functions under large $k$.
4. Utilize the sample sifting effect of robust loss functions to **select** corrupted samples and either drop them or utilize semi-supervised learning to obtain surrogate labels.

Reviewer comment (Presentation):
The presentation of experiments could be further improved, i.e., what is $\tau$ in Figure 4.

Author response:
$\tau$ is the hyperparameter of the shifted/scaled weighting function $w^{+}(\boldsymbol{s}, y)$ and  $w^{*}(\boldsymbol{s}, y)$. It is denoted in the caption of Figure 4 in our revision

Reviewer comment (Presentation):
What is $\tau$ in Figure 4?

Author response:
See response for weakness Q1

Reviewer comment (Evaluation):
In Table 3, why MAE has such pretty bad performances under CIFAR-100?

Author response:
It is due to the minimal sample weights that $w_{\mathrm{MAE}}$ assign to samples at initialization, which has been thoroughly discussed in section 4.3:
- the first paragraph provide a quick review of existing observations and explanations
- the second confirms and intuitively explains it
- the third reveals the cause with increased number of classes $k$ and how it leads to minimal initial sample weights
- the fourth provide fixes to address it, further confirming that underfitting is indeed caused by minimal initial sample weights

Reviewer comment (Experiments):
Maybe I missed some important details, I was wondering how authors pick  $\tau$ for reporting experiment results in Table 5, 6.

Author response:
$\tau$ in Tables 5,8,9 is tuned on CIFAR100 with symmetric noise rate $\eta = 0.4$. Due to the excessive training time required to train on WebVision, $\tau$ in Table 6 is roughly tuned for each setting based on the observation that smaller $\tau$ helps avoid underfitting but undermines noise robustness.