Paper: Bounded Loss Robustness: Enhancing the MAE Loss for Large-Scale Noisy Data Learning (ICLR-2024)

Reviewer comment (Evaluation):
The proposed method dosen't achieve SOTA accuracy in Table 2, 3, 4. The method didn't show dominant advantages over other loss functions, further explorations are expected. (current improvements are not very significant)

Author response:
Given that we calculate the parameter $\epsilon$ and do not fine-tune it, we believe that even though our method is not superior in all cases, the fact that it is among the best on all data sets is a considerable achievement. This is true even more since it was believed that the MAE loss is incapable of training on data sets with many classes.

Reviewer comment (Experiments):
The estimation of "logit bias" is still empirical for tasks or number of classes, which is not easy to configure it for different tasks.

Author response:
The Reviewer's questions and weaknesses scrutinize our concept for choosing $\epsilon$. As a first step, our manuscript analyzes the overlap of the derivative of bounded loss functions with the logit values in the early learning phase. In this phase, the weights are still random such that it is possible to explicitly calculate how this overlap behaves as a function of the number of classes in the dataset. As more classes lead to a vanishing overlap of the two regions, we propose to revert this by shifting the logit encoding the correct class.
However, we do not use a heuristic value for the logit bias that must be tuned for each dataset,  but precisely calculate the $\epsilon$ value necessary to ensure optimal learning for differing class numbers using Eq. (4) of the manuscript. 

We argue that previous work used hyperparameters that must be tuned as soon as the number of classes in the dataset is changed. This greatly differs from our approach which provides a condition for choosing $\epsilon$. For example, if you were to train a network with 150 classes, we would suggest $\epsilon=3.25$ without training a neural network once. Fig. 5 in the appendix illustrates this behavior.
We believe that future work could benefit from our analysis by calculating how hyperparameters should be adjusted as a function of the number of classes in the dataset, therefore providing a more general framework that must not be fine-tuned for each individual dataset.

Reviewer comment (Evaluation):
"thus laying the foundation for a universal classification framework." Can this work support this strong conclusion?

Author response:
We believe that future work could benefit from our analysis by calculating how hyperparameters should be adjusted as a function of the number of classes in the dataset, therefore providing a more general framework that must not be fine-tuned for each individual dataset. In this spirit, we formulated our conclusion which the Reviewer finds too strong.
However, we accept the Reviewer's criticism and have somewhat softened our conclusion to "thus being a first step towards a class count independent framework." in the updated version of the manuscript.

Reviewer comment (Novelty):
one paper relates to this work, would you please give some comments (comparison): IMAGE for Noise-Robust Learning: Mean Absolute Error Does Not Treat Examples Equally and Gradient Magnitude's Variance Matters, Published at ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale ML Models.

Author response:
We thank the Reviewer for pointing out the interesting reference "IMAGE for Noise-Robust Learning: Mean Absolute Error Does Not Treat Examples Equally and Gradient Magnitude's Variance Matters", published in ICLR 2023.  The reference takes a similar first approach by analyzing the gradients with respect to the logits. By re-weighting the gradients using an exponential function, the MAE can train effectively - however at the price of changing the  MAE loss fundamentally. As a consequence, this approach has several drawbacks: the modified MAE loss loses the theoretically derived guarantees for noise robustness, as the modified loss is neither bounded nor symmetric. In addition, this approach introduces a hyperparameter $T$ which needs to be tuned. 

Besides these theoretical and practical considerations, we find that the final test accuracies in the presence of label noise are significantly better for MAE with logit bias as compared to the re-weighted gradient approach. We use the ResNet-32 architecture for Cifar-10 and ResNet-34 for Cifar-100 while the reference uses ResNet-20 and ResNet-56 on Cifar-10. When taking the better of their two final test accuracies, they achieve $84.0\\%$ on Cifar-10 at $40\\%$ label noise compared to our $87.3\\%$. For Cifar-100 they used ResNet-44 and found $63.4\\%$ compared to our $68.3\\%$ test accuracy for $20\\%$ noise. For $40\\%$ noise, we found $55.7\\%$ while their final accuracy is $54.7\\%$. Only in one case, $60\\%$ noise on Cifar-100, their results were better with an accuracy of $43.9\\%$ compared to our $35.5\\%$.

Considering that the above reference claims SOTA results while our method seems to perform better while also being more straightforward to implement, we believe that our paper is a worthy contribution to ICLR and kindly ask the Reviewer to reconsider the assessment of our work.

Reviewer comment (Novelty):
The proposed method "logit bias" seems too simple. It is definitely okay if it is effective enough, unfortunately, it is not.

Author response:
We politely disagree with the Reviewer's judgment that our method is "too simple". In our view, a simple method is preferable to a complex method as long as it performs as well as the complex method. With regards to the Reviewer's judgment that the "logit bias" is not effective enough, we showed that all of the proposed noise-robust loss functions struggled on one of the three data sets, while the MAE loss with a straightforward modification of the bias value remained functional. This was possible without tuning hyperparameter as the value of $\epsilon$ was determined by using Eq. (4). In addition,  we would like to argue that many of the more complex approaches to bounded loss functions, involving several hyperparameters, are finetuned to a specific problem while a "simple" method is able to perform well in many situations.

Reviewer comment (Experiments):
The current experimental results are not sufficient to demonstrate the effectiveness of the proposed method. First, most experiments are performed on symmetric label noise, and it is uncertain whether the proposed method can also handle asymmetric noise. Although WebVision contains asymmetric noise, its noise rate is relatively low. Moreover, even for symmetric label noise, the proposed method lags behind some previous methods such as genCE in many cases. Additionally, the authors claim that genCE has no hyper-parameter, but it appears that they set $q$ of GCE to 0.7 by default. In my experience, the performance of genCE can improve remarkably if $q$ is adjusted carefully. For example, if $q$ is set to 0.5 or a smaller value, genCE might outperform other robust losses on WebVision. Considering that MAE* has one hyper-parameter, it is unfair to freeze $q$ of genCE.

Author response:
As the Reviewer states, the dataset WebVision contains asymmetric noise and we do not find that this hurts the performance of our loss function. This is further supported by the theoretical results of Ref. [2], where it is established that the MAE is tolerant to asymmetric noise under general conditions.

Furthermore, we do not claim that genCE has no hyperparamers but explicitly write that we use genCE with a value of $q=0.7$ as suggested in the original paper [1]. We agree that tuning the parameter $q$ could improve the performance,  but as argued above,  there is a difference between setting the hyperparameter $q$ vis-a-vis using the $\epsilon$ value computed from Eq. (4). However, we are happy to include results for the suggested value $q=0.5$ in the final version of the manuscript. 

In light of the above arguments, we kindly ask the Reviewer to reconsider the assessment of our work.

Reference:
[2] Robust loss functions under label noise for deep neural networks, AAAI 2017

Reviewer comment (Writing):
The notations are not clearly defined; for example, the meaning of $\delta_{nj}$ is unclear.

Author response:
We thank the Referee for the comment that our notation which uses $\delta_n$ to denote the error in the backpropagation equations and $\delta_{ij}$ to denote the Kronecker delta can be confusing. In the updated Manuscript, we have modified the notation and now denote the Kronecker delta as $\delta[i,j]$ and explicitly state its definition.

The symbol $z_k$ denotes the logit at the neuron position $k$ while $a_j$ denotes the neuron activation at position $j$ after the activation function is applied, i.e, $a_k = \exp(z_k) / \sum_i   \exp(z_i)$ for the softmax activation function.
In the updated version we explicitly state the definitions for these quantities in the paragraph below Eq. (2), where they occur for the first time.

Reviewer comment (Experiments):
This method is only designed for MAE, could it be expanded to help other robust losses?

Author response:
As for improving the performance of other loss functions besides MAE, we believe that our analysis could be helpful both directly by applying an output bias to facilitate learning, but also indirectly: By providing a quantitative metric (the overlap between the logit distribution and the error) for how learning is impaired when the number of classes increases, the hyperparameters of other loss functions could potentially be adjusted based on the number of classes. This would potentially greatly improve their reliability and make them easier to use in practice.

Reviewer comment (Evaluation):
The empirical improvement is trivial. In the dataset CIFAR100 and with Resnet-34, it only achieves the state-of-the-art for clean data.

Author response:
As for the improvements on the Cifar-100 dataset, we agree that this is not a new state-of-the-art benchmark outperforming Cross-Entropy in the case of clean data. However, MAE with logit bias keeps a large degree of robustness against cross-entropy and clearly outperforms it when label noise is introduced. In addition, we would like to argue that achieving state-of-the-art results with the MAE loss on clean data is still impressive, considering that none of the training hyperparameters are optimized for the MAE loss.

Reviewer comment (Writing):
In figure 1, the label for x-axis is "pre activation $z_k$", but there is no information about the activation function, I wonder what is the specific meaning of $z_k$, $a_j$ and $\delta_{nj}$.

Author response:
The symbol $z_k$ denotes the logit at the neuron position $k$ while $a_j$ denotes the neuron activation at position $j$ after the activation function is applied, i.e, $a_k = \exp(z_k) / \sum_i   \exp(z_i)$ for the softmax activation function.
In the updated version we explicitly state the definitions for these quantities in the paragraph below Eq. (2), where they occur for the first time.

Following this notation, the "pre-activation $z_k$" in Fig. 1 is the logit at position $k$. We use Eq. (3) to calculate the error for each of the logit values, where we assume $a_k = \exp(z_k) / \sum_i  \exp(z_i)$, i.e., the softmax activation, which is more prominently mentioned in the updated version.

Reviewer comment (Theory):
How to get the output error of $\delta_n$ in Tabel 1? Is that related to activation function?

Author response:
Regarding the errors in Tab. 1, we use Eq. (2) to calculate these values. The assumed activation in the final layer is the softmax activation function. As mentioned, we tried to make this clearer in an updated version.

Reviewer comment (Experiments):
Should we not consider the two temperatures as parameters for the Bi-Tempered loss?

Author response:
We agree with the Referee that the two temperature values of the bi-tempered loss can be considered as two parameters. However, in our case, we did not actively fine-tune them on each dataset but chose two values that were proposed in the original reference, which is now explicitly mentioned in the updated version. We also summarize the parameters in Table 5 in the appendix.

Reviewer comment (Experiments):
How would the overlapping state change during training? Would a dynamically tuned $\epsilon$ be helpful? Would you consider other strategies for choosing $\epsilon$?

Author response:
This is a great question and we indeed believe that a dynamically tuned $\epsilon$ could further improve performance. We speculate that after the initial training phase, the network gains confidence in a specific label for most examples, i.e., it has a high logit value for one of the classes. 

If the largest logit points at the correct class (according to the label) the learning of the example might stop too early if $\epsilon$ is chosen too large, as the network trains to a lower level of confidence (observed for 1000 classes without decreasing $\epsilon$).
On the other hand, if the largest logit does not correspond to the correct label, which may be due to the label being wrong due to noise,
the logit bias leads to an increase in the otherwise small logit of the "correct" class. This increases the chances that the network relearns the examples and switches to the "correct" class. In the case of a mislabeled example, this might then reduce the robustness towards label noise, and again is an argument why a dynamical reduction of $\epsilon$ might be useful.  

We hence believe that the logit bias has its main use in the early stages of learning. In the later stages, learning is either possible without it as the correct label already gained in confidence, or should not be possible as the example has most likely an incorrect label. Therefore the referees suggestion of a decaying $\epsilon$ schedule could be a great addition to our method.
We added a brief discussion of this question to the appendix of our manuscript.