Paper: Probability-dependent gradient decay in large margin softmax (ICLR-2024)

Reviewer comment (Writing):
The paper suffers from an overall lack of clarity and is hard to read.

Author response:
Following the reviewer's guidance, we have recognized the poor presentation of our paper. Considering the readability of the paper, we have modified the presentation in section1 and section2 to improve presentation of the paper. In the original version, certain statements were confusing. We have made corresponding revisions to enhance clarity. Additionally, in response to the reviewer's advice, we have supplemented the standard deviation of the results in our experiments.

Reviewer comment (Writing):
The statement "MSE takes into account more complex optimization scenarios" is unclear; its meaning should be clarified.

Author response:
* Q1: "MSE takes into account more complex optimization scenarios": What do you mean by that?  
* A1: We have recognized that the meaning of this sentence was ambiguous. In the current version of the paper, this statement has been replaced with an introduction to model calibration, aiming to better elucidate the content presented in the paper.  

”However, the modern models often demonstrate inadequate confidence calibration in probability distribution through Softmax mapping. Specifically, these probability outputs display unwarranted over-confidence (Guo et al., 2017). Furthermore, researchers have identified that achieving high accuracy in classifiers and calibrating the model confidence are distinct objectives (Wenger et al., 2020). This scenario emphasizes the necessity to study the calibration of model output uncertainties in optimization.”

Reviewer comment (Writing):
The term "Hard mining strategy" is mentioned without a brief introduction or explanation.

Author response:
* Q2: "Hard mining strategy": you could briefly introduce what this is.  
* A2: Thank you for the suggestion. We have briefly introduced the hard mining strategy in the third paragraph of Section 1.

Reviewer comment (Writing):
In section 2, $J_j$ is referred to before being introduced or defined.

Author response:
* Q3: In section 2, you talk about J_i before introducing it  
* A3: We have noticed the poorly articulated expression in Section 2. In the current version, we have polished Eq. (1)-(6). Furthermore, we have added explanations regarding J in Eq. (1).

Reviewer comment (Presentation):
In figure 3, there are no legends for the top row, the caption does not clarify the different curves, and the main text does not explain them clearly (e.g., what are "post-training samples"—are these test samples?).

Author response:
* Q4: In figure 3: there are no legends for the top row, and the caption does not help to clarify the different curves being shown, the main text is also unclear about those i.e. what are "post-training samples", are those test samples?  
* A4: The term "post-training samples" refers to samples with low confidence in the early stages of training. We have revised the confusing expression to use "challenging samples." Additionally, we have updated the captions for each figure to better describe the content of the figures.

Reviewer comment (Presentation):
In figure 3, it should be mentioned in the legend or the caption that the different groups correspond to samples of varying difficulty.

Author response:
* Q4: In figure 3: there are no legends for the top row, and the caption does not help to clarify the different curves being shown, the main text is also unclear about those i.e. what are "post-training samples", are those test samples?  
* A4: The term "post-training samples" refers to samples with low confidence in the early stages of training. We have revised the confusing expression to use "challenging samples." Additionally, we have updated the captions for each figure to better describe the content of the figures.

Reviewer comment (Writing):
The statement "If we make excessive demands on the margin, some post-training samples cannot get any chance and will be 'sacrificed' according to the soft curriculum learning strategy [...]" is unclear and needs clarification.

Author response:
* Q5: "If we make excessive demands on the margin, some post-training samples cannot get any chance and will be "sacrificed" according to the soft curriculum learning strategy [...]", what do you mean?  
* A5: This sentence indicates that certain samples consistently exhibit low confidence when the margin is set too large. We have replaced this confusing statement with the following illustration.  

“If we make excessive demands on the margin, some challenging samples always remain low confidence $\hat p$ according to the stringent curriculum learning sequence.”

Reviewer comment (Writing):
The phrase "So it is convinced that the general softmax [...]" is unclear.

Author response:
* Q6: "So it is convinced that the general softmax [...]"  
* A6: The meaning of this sentence is that Softmax could be characterized by two hyperparameters, $\tau$ and $\beta$. In the current version, we have replaced some inappropriate terms, as illustrated by the following sentence:  

“So it is convinced that Softmax cross-entropy could be characterized with these two hyperparameters $\tau $ and $\beta$.”

Reviewer comment (Writing):
The statement "The beta smaller is, the gradient smoother is" is unclear.

Author response:
* Q7: "The beta smaller is, the gradient smoother is"  
* A7: Regarding the function $J\left( {{z_c}} \right)$ presented in equations (9)-(12), we determine that $\max {\left| {\nabla _{{z_c}}^2J\left( {{z_c}} \right)} \right|_2}$ equals a constant when ${p_c} = \frac{1}{{1 + \beta }}$. By considering the local range ${p_c} \in \left[ {0,0.5} \right]$, we can derive a local $L$-constraint for ${\left| {\nabla _{{z_c}}^2J\left( {{z_c}} \right)} \right|_2}$, as illustrated in equation (18). The original sentence was inaccurately expressed. We have replaced this confusing sentence with the following statement.  

“The $\beta$ smaller is, L-constrain in the early phase of optimization smaller is.”

Reviewer comment (Writing):
The statement "Warm-up strategy achieves even better results." is unclear in meaning and context.

Author response:
* Q8: "Warm-up strategy achieves even better results."  
* A8: As depicted in Figure 4, the warm-up strategy yields higher accuracy compared to a fixed gradient decay coefficient. We have replaced the original confusing sentence with the following statement:  

"As a result, the warm-up strategy achieves higher accuracy compared to a fixed gradient decay coefficient."

Reviewer comment (Writing):
The phrase "Curriculum design that divides samples is crucial to curriculum learning idea." is unclear.

Author response:
* Q9: "Curriculum design that divides samples is crucial to curriculum learning idea."  
* A9: Thank you for the reviewer's thorough examination. The original sentence aimed to convey that the design of the curriculum necessitates distinguishing between easy and challenging samples, which is crucial for the entire curriculum learning. We acknowledge the poor expression in the original sentence and have replaced it with the following clarification:  

“The design of the curriculum, specifically in evaluating the difficulty of individual samples, plays a pivotal role in curriculum learning idea.”

Reviewer comment (Writing):
The statement "Besides, based the previous analysis [...]" is unclear.

Author response:
* Q10: "Besides, based the previous analysis [...]"  
* A10: The original sentence aimed to convey that these challenging samples may exhibit low confidence when the gradient decay rate is excessively small. We recognize the inaccuracies in the original wording, and in the current version, this confusing sentence has been replaced with the following clarification.  

“Based the previous analysis in section 4.2, over-small gradient decay rate may impair the performance since some challenging samples containing important information remain low confidence $\hat p$.”

Reviewer comment (Presentation):
In figure 6, the y-axis mentions accuracy but also shows confidence, which is confusing; this should be clarified in the caption.

Author response:
* Q11: In figure 6: the y axis mentions accuracy and but also shows confidence, this is confusing and could be clarified in the caption.  
* A11:  Thank you for your suggestions. We have revised the caption of figure 6 to provide more detailed descriptions of their content. Additionally, to better introduce the concept of model calibration, we have incorporated relevant descriptions into section 1 and section 2. The modified caption is presented below:  

“Confidence and reliability diagrams with ResNet18 on CIFAR-100. ($bins=10$) In each subplot, the left plot illustrates the sample distribution in individual bins, while the right plot displays the average confidence and accuracy in each bin. Ideally, calibration aims for consistency between accuracy and average confidence in each bin. It indicates that a smaller gradient decay rate $\beta$ is associated with more pronounced miscalibration of the model, while a larger gradient decay rate mitigates this issue.”

Reviewer comment (Evaluation):
In table 2, the different values are quite close, making it difficult to evaluate the robustness of the improvement without standard deviation; it should be possible to run the same experiment with different seeds for some smaller datasets.

Author response:
* Q13: In table 2: The different values are quite close, and it is difficult to evaluate the robustness of the improvement without standard deviation. It should be possible to run the same experiment with different seeds for some of the smaller datasets.  
* A12 to Q12-15: We sincerely appreciate the constructive suggestions from the reviewer. The model employed in Fig. 4 is a three-layer fully connected network. Detailed parameter settings and all experimental details, including architecture, tuning, seeds, and optimizer, are provided in Appendix A.1. To offer more general conclusions, all hyperparameters are set without cumbersome tuning. In addition, all experimental results are averages of five different random seeds and the specific random seeds are given in Appendix A.1. The random seed configurations were consistent across different methods. Furthermore, following reviewer's recommendation, we have included standard deviation in the results of repeated experiments in Tables 2 and 3.

Reviewer comment (Evaluation):
In table 3, the same issue applies as in table 2: standard deviations should be provided to assess robustness.

Author response:
* Q14: In table 3: same as above, I would love to see standard deviations.  
* A12 to Q12-15: We sincerely appreciate the constructive suggestions from the reviewer. The model employed in Fig. 4 is a three-layer fully connected network. Detailed parameter settings and all experimental details, including architecture, tuning, seeds, and optimizer, are provided in Appendix A.1. To offer more general conclusions, all hyperparameters are set without cumbersome tuning. In addition, all experimental results are averages of five different random seeds and the specific random seeds are given in Appendix A.1. The random seed configurations were consistent across different methods. Furthermore, following reviewer's recommendation, we have included standard deviation in the results of repeated experiments in Tables 2 and 3.

Reviewer comment (Reproducibility):
For all experiments, the experimental protocol is not clearly described (architecture, tuning, seeds, optimizer, etc. are not detailed), and these details are also not found in the appendix.

Author response:
* Q15: For all the experiments, which experimental protocol was followed: which architecture, tuning, seeds, optimizer, ... I couldn't find those in the appendix either.  
* A12 to Q12-15: We sincerely appreciate the constructive suggestions from the reviewer. The model employed in Fig. 4 is a three-layer fully connected network. Detailed parameter settings and all experimental details, including architecture, tuning, seeds, and optimizer, are provided in Appendix A.1. To offer more general conclusions, all hyperparameters are set without cumbersome tuning. In addition, all experimental results are averages of five different random seeds and the specific random seeds are given in Appendix A.1. The random seed configurations were consistent across different methods. Furthermore, following reviewer's recommendation, we have included standard deviation in the results of repeated experiments in Tables 2 and 3.

Reviewer comment (Evaluation):
The warm-up scheme does not seem to provide a significant advantage over prior proposed modifications to softmax (e.g A-softmax) or does worse according to table 2 in the paper.

Author response:
We appreciate the reviewer for taking the time to review our paper.

* Firstly, in response to the reviewer's queries, we have added supplementary information regarding the warm-up training loss curve in Fig. 5.

* Secondly, considering review’s concerns, we would like to provide additional explanations about the significance of our work.

As mentioned by the reviewer in “Weaknesses”, the warm-up scheme shows an improvement over the previously proposed adjustment for Softmax, but the improvement is not significant. Addressing review’s concerns about the performance improvement, we repeated the results more times and presented the mean and standard deviation of the results in Tab.2 and Tab.3 to support our conclusions. Furthermore, considering that these two approaches address the problem from different perspectives, where A-Softmax statically considers the margin in the output space, while the warm-up scheme dynamically adjusts gradient decay coefficients during the optimization process. Therefore, we believe that the presentation of such results can provide some new insights.

On the other hand, our paper also focuses on the uncertainty estimation of the classification results, i.e., whether the model probability outputs can reflect the accuracy prediction in classification. Perfect calibration of neural network can be realized when the confidence score reflects the real probability that the classification is classified correctly. Formally, the perfectly calibrated network satisfied ${\rm P}\left( {\hat y = y|\hat p = p} \right) = p$ for all $p \in \left[ {0,1} \right]$. However, probability outputs of modern model display unwarranted over-confidence. Our work reveals that a probability-dependent gradient decay rate is a factor contributing to the over-confidence of modern models in classification. The optimization with a small gradient decay rate presents a more stringent curriculum learning for samples, meaning that challenging samples only gain confidence after easy samples have been sufficiently trained to a certain level of confidence. This leads to different samples having greater distinctiveness in the optimization. Correspondingly, our presented analysis, detailed in Section 3, along with more experimental results shown in Fig. 6, Table 3, and Appendix, demonstrates that a substantial increase in the model-calibrated criterion Expected Calibration Error (ECE) can be achieved when employing large probability-dependent gradient decay in modern model. To better present the work in model calibration, we have modified presentation in section 1 and section 2 to emphasize this point.

In addition to methodological innovations, our work can provide some insights and understanding, especially concerning concepts such as model calibration, large-margin Softmax, probability-dependent gradient decay, curriculum learning, and related topics in deep learning optimization.

Reviewer comment (Experiments):
What does the training loss look like across epochs for the warm-up schedule (more specifically could you plot another curve in figure 5 displaying the loss over epochs for the warm-up schedule)?

Author response:
We appreciate the reviewer for taking the time to review our paper.

* Firstly, in response to the reviewer's queries, we have added supplementary information regarding the warm-up training loss curve in Fig. 5.

* Secondly, considering review’s concerns, we would like to provide additional explanations about the significance of our work.

As mentioned by the reviewer in “Weaknesses”, the warm-up scheme shows an improvement over the previously proposed adjustment for Softmax, but the improvement is not significant. Addressing review’s concerns about the performance improvement, we repeated the results more times and presented the mean and standard deviation of the results in Tab.2 and Tab.3 to support our conclusions. Furthermore, considering that these two approaches address the problem from different perspectives, where A-Softmax statically considers the margin in the output space, while the warm-up scheme dynamically adjusts gradient decay coefficients during the optimization process. Therefore, we believe that the presentation of such results can provide some new insights.

On the other hand, our paper also focuses on the uncertainty estimation of the classification results, i.e., whether the model probability outputs can reflect the accuracy prediction in classification. Perfect calibration of neural network can be realized when the confidence score reflects the real probability that the classification is classified correctly. Formally, the perfectly calibrated network satisfied ${\rm P}\left( {\hat y = y|\hat p = p} \right) = p$ for all $p \in \left[ {0,1} \right]$. However, probability outputs of modern model display unwarranted over-confidence. Our work reveals that a probability-dependent gradient decay rate is a factor contributing to the over-confidence of modern models in classification. The optimization with a small gradient decay rate presents a more stringent curriculum learning for samples, meaning that challenging samples only gain confidence after easy samples have been sufficiently trained to a certain level of confidence. This leads to different samples having greater distinctiveness in the optimization. Correspondingly, our presented analysis, detailed in Section 3, along with more experimental results shown in Fig. 6, Table 3, and Appendix, demonstrates that a substantial increase in the model-calibrated criterion Expected Calibration Error (ECE) can be achieved when employing large probability-dependent gradient decay in modern model. To better present the work in model calibration, we have modified presentation in section 1 and section 2 to emphasize this point.

In addition to methodological innovations, our work can provide some insights and understanding, especially concerning concepts such as model calibration, large-margin Softmax, probability-dependent gradient decay, curriculum learning, and related topics in deep learning optimization.