From: ICLR.cc/2023/Conference/Paper197/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper197/Authors
title: Further comments about our response
comment: Dear Reviewer MXeN,

We would like to express our sincere gratitude for your time and efforts in reviewing our paper. Your comments and feedback are valuable in helping us improve the final version of our paper.

We have carefully thought about your feedback about our first response, and provided theoretical analysis about it. We are wondering if there are further comments about our work.

Thanks very much, and look forward to your reply!


Best,

The authors

From: ICLR.cc/2023/Conference/Paper197/Authors
title: Any concerns about our response?
comment: Dear Reviewer,

Thanks again for your valuable time in reviewing our paper. The response to your comments has been posted for nearly a month. If you have any concerns about our response, we would be happy to answer them.

Best regards,

The authors

From: ICLR.cc/2023/Conference/Paper197/Authors
title: Any concerns about our response
comment: Dear Reviewer,

Thanks again for your valuable time in reviewing our paper. The response to your comments has been posted for three weeks. If you have any concerns about our response, we would be happy to answer them.

Best regards,
The authors

From: ICLR.cc/2023/Conference/Paper197/Reviewer_X1fQ
title: Reviewer's response
comment: Thank you very much for the detailed response!
As someone who has been trying to study the learning dynamics of NNs under different loss functions, I can echo that a significant portion of the difficulty in analysis comes from the non-linear dynamics of the loss function itself. Hence, I believe, the introduced ASM loss would allow for further analytical studies of NNs. That being said, I believe that the ASM loss has two drawbacks: 1) It would be difficult to stabilize the training under such a loss function, and 2) The simplicity that it brings comes at the price of losing some interactions between the dynamics of the loss of different training examples. For example, with CE, some examples might have a lower contribution to the loss, while that would not be the case with the ASM loss as far as I can tell.

Nevertheless, I believe this work takes a step towards simplifying the learning dynamics of neural networks (at a price) and is valuable to the community. Hence, I increase my score by 1.

From: ICLR.cc/2023/Conference/Paper197/Authors
title: Response To Reviewer MXeN (Continued 2/2)
comment: 
**2) For the MSE loss** $L(\bf W^\top \bf h+\bf b,y)=\Vert \bf W^\top \bf h+\bf b-\bf y\Vert_2^2$ and $\mathcal L=\frac{1}{CN}\Vert\bf W^\top \bf H+\bf b \bf 1_{CN}^\top -\bf I\otimes \bf 1_N^\top \Vert_F^2$}, we can similarly derive that
$$
\bf H'(t)=\nabla_{\bf \Theta} \bf H^T\nabla_{\bf \Theta} \bf H \bf W(t)\bf C(t),\ \bf W'(t) = \bf H(t) \bf C(t)^T,\text{ and } \bf{b}'(t)=\bf C(t) \bf 1_{CN},
$$
where $\bf C(t)=\frac{2\eta }{CN}(\bf W(t)^T\bf H(t) + \bf b(t)\bf 1_{CN}^T- \bf I_{C}\otimes \bf 1_N^T)$. As can be seen, the above gradient flow is nonlinear and then intractable to analyze the exact dynamics though $\nabla_{\bf \Theta} \bf H^T\nabla_{\bf \Theta} \bf H$ stays constant.

Through the above analysis, it is clear to see that the proposed ASM loss offers more
mathematical opportunities than MSE to analyze the closed-form dynamics.

[1] Jacot, Arthur, Franck Gabriel, and Clément Hongler. "Neural tangent kernel: Convergence and generalization in neural networks." Advances in neural information processing systems 31 (2018).

From: ICLR.cc/2023/Conference/Paper197/Authors
title: Response To Reviewer MXeN (Continued 1/2)
comment: Many thanks to the reviewer for rapid feedback and constructive suggestions. We carefully thought about your comments in the past few days and would like to provide the following response.

We agree with you that "*assuming the representation $\bf H$ of the last hidden layer is a parameter that can be optimized directly is equivalent to assuming that $\bf H$ depends linearly on the parameters in the layers before*". Therefore, considering the linear approximation of last-layer features may be a promising direction to glimpse into deep learning. As suggested by the reviewer, we conducted some analysis through applying neural tangent kernel (NTK) up to the last hidden layer as linear approximation of last-layer features. The preliminary result shows that the ASM loss has more potential to derive exact dynamics than MSE.

Specifically, considering the last-layer feature $\bf h_{i,c}=\bf f_{\bf \Theta}(\bf x_{i,c})\in\mathbb{R}^p$ extracted from the example $\bf x_{i,c}$ by a number of parameterized layers $\bf f_{\bf \Theta}:\mathcal X\rightarrow \mathbb{R}^p$ and the empirical risk $\mathcal L=\frac{1}{CN}\sum_{i=1}^N\sum_{c=1}^CL(\bf W^T\bf h_{i,c} + \bf b,y_{i,c})$ w.r.t. the loss function $L(\bf W^T\bf h+\bf b,y)$, where $\bf H=[\bf h_{1,1},...,\bf h_{N,C}]$. When DNNs are trained using gradient descent on the composition $\mathcal{L}\circ \bf{W}\circ \bf f_{\bf \Theta}$, we have
$$
    \bf{\Theta}(t+1)-\bf{\Theta}(t)=-\eta \nabla_{\bf{\Theta}}\mathcal{L},\ 
    \bf{W}(t+1)-\bf{W}(t)=-\eta\nabla_{\bf{W}}\mathcal{L},
    \text{ and }
    \bf{b}(t+1)-\bf{b}(t)=-\eta\nabla_{\bf{b}}\mathcal{L}
$$
According to the first-order Taylor expansion, we can obtain
$$
\bf H(t+1)-\bf H(t)\approx \nabla_{\bf \Theta} \bf H^T[\bf{\Theta}(t+1)-\bf{\Theta}(t)] =-\eta \nabla_{\bf \Theta} \bf H^T\nabla_{\bf \Theta}\mathcal L.
$$

**1) For the ASM loss**, we can easily derive that
$$
\bf H'(t)=\nabla_{\bf \Theta} \bf H^T\nabla_{\bf \Theta} \bf H \bf W(t)\bf M,\text{ and }\bf W'(t) = \bf H(t) \bf M^T,
$$
where $\bf M=\tfrac{\eta}{CN}((1+\gamma)(\bf I_C\otimes \bf 1^T_{N})-\gamma\bf 1_C\bf 1_{CN}^T)$. 

Therefore, we have the following gradient flow:
$$
\begin{pmatrix}\bf{H}'(t) & 0\\\ 0 & \bf{W}'(t)\end{pmatrix} = 
\begin{pmatrix}0 & \nabla_{\bf \Theta}\bf H ^T\nabla_{\bf \Theta}\bf H\\\ \bf I & 0\end{pmatrix}
\begin{pmatrix}\bf{H}(t) & 0\\\ 0 & \bf{W}(t)\end{pmatrix}
\begin{pmatrix}0 & \bf M^T\\\ \bf M & 0\end{pmatrix}.
$$
In the infinite-width limit, we know that the neural tangent kernel $\nabla_{\bf \Theta}\bf H ^T\nabla_{\bf \Theta}\bf H$ stays asymptotically constant during training [1]. 

Let $\bf z(t)$ denote the row-first vectorization of $\begin{pmatrix}\bf{H}'(t) & 0\\\ 0 & \bf{W}'(t)\end{pmatrix}$, $\bf A=\begin{pmatrix}0 & \nabla_{\bf \Theta}\bf H ^T\nabla_{\bf \Theta}\bf H\\\ \bf I & 0\end{pmatrix}$, and $\bf B=\begin{pmatrix}0 & \bf M^T\\\ \bf M & 0\end{pmatrix}$, we have
$$
\bf z'(t)=(\bf A\otimes \bf{B}^T) \bf z(t)=(\bf A\otimes \bf{B}) \bf z(t),
$$
and then $\bf z(t)=e^{(\bf A\otimes \bf{B})t}\bf z(0)$. Considering the eigendecomposition $\bf A=\bf U_{\bf A}\bf \Lambda_{\bf A} \bf U_{\bf A}^{-1}$ and $\bf B=\bf U_{\bf B}\bf \Lambda_{\bf B} \bf U_{\bf B}^{-1}$, where $\bf \Lambda_{\bf A}=\mathop{\text{diag}}(\lambda_1^{\bf A},...,\lambda_n^{\bf A})$ and $\bf \Lambda_{\bf B}=\mathop{\text{diag}}(\lambda_1^{\bf B},...,\lambda_n^{\bf B})$, thus 
$$
\bf A\otimes \bf B= (\bf U_{\bf A}\bf \Lambda_{\bf A} \bf U_{\bf A}^{-1})\otimes \bf B=(\bf U_{\bf A}\otimes \bf I)(\bf \Lambda_{\bf A} \otimes \bf B)(\bf U_{\bf A}^{-1}\otimes \bf I),
$$
where $\otimes$ denotes Kronecker product. Moreover, we have
$$
\exp[(\bf A\otimes \bf{B})t]=(\bf U_{\bf A}\otimes \bf I)\exp[(\bf \Lambda_{\bf A}\otimes \bf{B})t](\bf U_{\bf A}^{-1}\otimes \bf I)=(\bf U_{\bf A}\bf U_{\bf B}\otimes \bf I)\exp[(\bf \Lambda_{\bf A}\otimes \bf \Lambda_{\bf B})t](\bf U_{\bf B}^{-1}\bf U_{\bf A}^{-1}\otimes \bf I).
$$
We know that $\exp[(\bf \Lambda_{\bf A}\otimes \bf \Lambda_{\bf B})t]$ is a diagonal matrix. Therefore, we have $(\bf U_{\bf B}^{-1}\bf U_{\bf A}^{-1}\otimes \bf I)\bf z(t)=\exp[(\bf \Lambda_{\bf A}\otimes \bf \Lambda_{\bf B})t](\bf U_{\bf B}^{-1}\bf U_{\bf A}^{-1}\otimes \bf I)\bf z(0)$, which leads to a concise closed-form dynamics on $(\bf U_{\bf B}^{-1}\bf U_{\bf A}^{-1}\otimes \bf I)\bf z(t)$.

From: ICLR.cc/2023/Conference/Paper197/Reviewer_MXeN
title: Reviewer's response
comment: My interpretation of 'strong reject' is not that it contributes nothing to the litterature, rather that this paper is not good enough for ICLR and that I do not see any small changes that would make it good enough before the end of the review period.

Thanks for the explanation about the ASM loss with constraints on the hidden representation, I can now see how the ASM can be meaningful in this setting.

Regarding the 'hidden layer as a free variables' assumption however, it seems to me that your only argument is 'some people have done it before' which does not convince me. Let me explain another issue I have with this assumption that I realized after writing my review, it requires a bit of context:
When you assume that the representation $H$ of the last hidden layer is a parameter than can be optimized directly, it is equivalent to assuming that $H$ depends linearly on the parameters in the layers before. The traditional tool to do this type of linear approximations is the Neural Tangent Kernel (NTK) (using the NTK up to the last hidden layer instead of up to the output layer), and we know that this is the best local linear approximation of a DNN. So what you are doing in this paper is assume a linear approximation and on top of this assume that the NTK equals the identity (or a scaling thereof since you allow different learning rates).

I think you should do this analysis with the true NTK instead of assuming that it is the identity. Everything will remain quadratic so the proofs will not change much. If you make an approximation, it should at the very least be the best amongst its class, and the best amongst linear approximations is the NTK. (In some sense what you are doing here instead is a first order Taylor approximation with the wrong derivative!)

However the results you obtain will be significantly different, at the very least for the non-regularized and regularized cases. Indeed the dynamics are heavily affected by the fact that in the resulting gradient flow system the eigenvalues have high multiplicity, this is why you can split the dynamics along 2 planes. However with the true NTK, you will in general loose this multiplicity (there will be many distinct eigenvalues) and this will change the results that you obtain (but it will be more accurate).

I think that this shows a big problem with your result: your result is not robust under small changes to the approximation you use (since it relies on some eigenvalues having large multiplicity), which is even more problematic considering that there is a better approximation (based on the NTK) that results in significantly different behavior (and this behavior would actually be robust under small changes to the eigenvalues).

From: ICLR.cc/2023/Conference/Paper197/Authors
title: Response to Reviewer MXeN (2/2)
comment: >
>[6] Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layer-peeled perspective on neural collapse. In *International Conference on Learning Representations*, 2022.
>
>[7] Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. *Applied and Computational Harmonic Analysis*, 59:224–241, 2022.
>
>[8] Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under MSE loss: Global optimality with unconstrained features. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 27179–27202. PMLR, 17–23 Jul 2022a.


Answers to specific comments are as follows:

**Q1**: "*A zero or negative ASM loss does not imply that the training set is labeled correctly, actually no value of the training ASM loss guarantees correct classification on the training set. I think this should be mentioned instead of saying that it is a variant of the cross-entropy loss. Considering these differences, the fact that the ASM loss is a reasonable approximation for the losses used in practice should be justified either theoretically or empirically.*"

**R**: We agree with the reviewer that the value of the ASM loss in Eq. 2.1 cannot be directly used as the criterion to judge whether a sample is classified correctly since the ASM loss can be arbitrarily negative in unconstrained cases. This is the main reason why the ASM loss cannot be used directly for practical training. To overcome this drawback, we turn to limit the rapid growth of norms. For instance, in the spherical constrained case, the features are constrained to the unit sphere, and class prototypes are fixed as a simplex equiangular tight frame (ETF). Moreover, under this case, the ASM loss would be lower bounded by $-(1+\gamma)$, and thus can be used in practical training (as shown in Fig. 2). We can also derive that $L_{ASM}< -(1+\gamma)\sqrt{\frac{C-2}{2(C-1)}}\Rightarrow \boldsymbol w_y^T\boldsymbol h>\max_{j\neq y} \boldsymbol w_j^T \boldsymbol h$ when these prototypes are fixed as a simple ETF, so the sample can be correctly classified if the value of the ASM loss is less than $-(1+\gamma)\sqrt{\frac{C-2}{2(C-1)}}$.


**Q2**: "*Actually after these simplification, the model studied is a special case of a shallow linear fully-connected network with a linear loss: $\mathcal{L}(W_1,W_2,b_2)=\text{Tr}(G^T(W_2W_1X+b_2))$ whose gradient dynamics can obviously be solved since the gradient is linear. One then recovers the result of the paper by taking $X=I_N$ and $G=Y$ (for the $Y$ matrix in the paper). In this light, the authors should at least cite and compare to the existing litterature on linear networks.*" 

**R**: Thanks for the kind suggestion. After carefully searching the literature, we found a very similar research area that investigates the convergence and implicit bias of overparameterized linear networks $f:x\rightarrow A_{\theta} x$ of depth $L\ge 1$, that is $A_{\theta}=W_L\dots W_1$ where $W_1,...,W_L$ are linear layers. For example, Min et al. [9] consider a single-hidden-layer linear network $f:x\rightarrow VU^T x$ that is trained with the square loss $\|Y-VU^TX\|_2^2$. There are also other similar works [10,11]. These works focus on regression and tensor factorization problems, while our work considers the classification problem where the label matrix $Y$ has a more special structure, leading to more specific conclusions, such as Neural Collapse [1].

> [9] Hancheng Min, Salma Tarmoun, Rene Vidal, Enrique Mallada. On the Explicit Role of Initialization on the Convergence and Implicit Bias of Overparametrized Linear Networks. *Proceedings of the 38th International Conference on Machine Learning*, PMLR 139:7760-7768, 2021.
>
> [10] Min H, Tarmoun S, Vidal R, et al. Convergence and Implicit Bias of Gradient Flow on Overparametrized Linear Networks[J]. arXiv preprint arXiv:2105.06351, 2022.
>
> [11] Razin N, Maman A, Cohen N. Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks[J]. arXiv preprint arXiv:2201.11729, 2022.

From: ICLR.cc/2023/Conference/Paper197/Authors
title: Response To Reviewer YmzU (2/2)
comment: **Q3**: "*In Fig. 1 (a), (b), and (d), it seems that $gamma$ has several values while only two curves are presented in these figures. Does the value of $gamma$ (from 0 to 0.05) have no relation with the corresponding parameters, e.g., loss, in these figures? If so, could you please give a qualitative or quantitative explanation on its effect?*"

**R**: We are very sorry that these figures are not clearly represented. If you enlarge the Fig. 1(d), you will find that some curves are overlapped and be covered by the purple line. We have repainted these figures in the revised manuscript.

 According to the proof of Corollary 3.2,  we can derive that $\|\boldsymbol Z(t)\|_2$ will increase with the rate $O(e^{\frac{(1+\gamma)\sqrt{\zeta_1(t)\zeta_2(t)}}{C\sqrt{N}}})$ when $0<\gamma<\frac{2}{C-2}$ and with the rate $O(e^{\frac{(\gamma C-\gamma-1)\sqrt{\zeta_1(t)\zeta_2(t)}}{C\sqrt{N}}})$ when $\gamma>\frac{2}{C-2}$. Therefore, for the setting in Fig. 1,  the exponential curve of $\|\boldsymbol Z(t)\|_2$ will be $O(e^{3.95 \times \sqrt{10}\times 10^{-4}t})$ when $\gamma=0.05$, which is faster than those curves of the rate roughly around $O(e^{\sqrt{10}\times 10^{-4}t})$ when $\gamma=0,0.001,0.005$ and $\frac{1}{C-1}=\frac{1}{99}$. This is the main reason why the the curves are overlapped when $\gamma=0,0.001,0.005$ and $\frac{1}{C-1}$.

**Q4**: "*I notice that through this paper no assumption is given. Since this is a theoretical paper, does that mean there is no limitation for applying the presented theorems?*"

**R**: In our work, we only take one assumption in studying the dynamics of neural network: modeling the last-layer features as free optimization variables. This assumption is called the layer-peeled model [1,2] or unconstrained features model [3], which has been widely adopted by a number of previous works, such as [4-6], to mention a few. 

>
>[1] Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. *Proceedings of the National Academy of Sciences*, 118(43): e2103091118, 2021.
>
>[2] XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. In *International Conference on Learning Representations*, 2022 (Outstanding Paper Award).
>
>
>[3] Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. *Sampling Theory, Signal Processing, and Data Analysis*, 20(2):1–13, 2022.
>
>[4] Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In *Proceedings of the 39th International Conference on Machine Learning*, volume 162 of *Proceedings of Machine Learning Research*, pages 21478–21505. PMLR, 2022.
>
>[5] Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layer-peeled perspective on neural collapse. In *International Conference on Learning Representations*, 2022.
>
>[6] Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. *Applied and Computational Harmonic Analysis*, 59:224–241, 2022.

**Q5**: "Reproducibility: The code is not provided thus I can’t verify this."

**R**: We have provided the code in the supplementary materials.

From: ICLR.cc/2023/Conference/Paper197/Authors
title: Response to Reviewer X1fQ (2/2)
comment: **Q4**: "*The improvement observed in Fig.2 and Table.1 are hard to measure since the standard deviation is not reported. Nevertheless, it appears that improvement is insignificant. However, I understand that the main point of the paper is to introduce a loss that is convenient for theoretical analysis and not practice.*"

**R**: Thanks for the kind comment. 

-- For the reason why standard deviation is not reported.
In our experiments, we set the random seed as 0 to ensure the results reproducible, which means that data generation, network initialization, and training settings are the same. We are working on setting up different random seeds so as to conduct experiments for many times. We will report standard deviation in the revised version.

-- About "it appears that improvement is insignificant"
We would like to bring your attention to Table 1. In long-tailed classification on CIFAR-100, explicit feature regularization achieves better results than baselines in all cases, and the accuracy is improved by 1.06-11.76\%. The improvements are significant actually.

From: ICLR.cc/2023/Conference/Paper197/Authors
title: Response To Reviewer YmzU (1/2)
comment: We thank the reviewer for the positive reviews and insightful questions. Responses to specific comments are below:

**Q1**: "*There is no theoretical comparison between the ASM loss and other loss functions, especially considering that the ASM loss resembles the hinge loss (by removing the max operator and the margin term of the hinge loss).*"

**R**: Thank very much for your kind comment! The ASM loss is a general form, which includes the unhinged loss as its special case. When $\gamma=\frac{1}{k-1}$, the ASM loss becomes the hinge loss by removing the max operator and the margin term. In this case, the loss is more suitable to be called as the unhinged loss.

**Q2**: "*Moreover, according to Eq. (2.1), $L_{ASM}$ for a single data point can be arbitrarily negative (e.g., let $w_y^Th\rightarrow \infty$ and $w_j^Th\rightarrow 0$ when$j\neq y$), this means that the overall empirical loss can be arbitrarily small even though only one example is classified correctly, which perhaps largely hinders the training. In a word, the ASM loss assigns correctly classified and wrongly classified data points with equal weights. This property is not suitable for the practical training of classifiers. The authors should provide more explanations on this point.*"

**R**: Thanks for your kind comment. 

Firstly, we would like to formally represent your concerns. Let us consider the gradient of the CE loss $L_{CE}=-\log p_y$, where $p_y=\frac{\exp(\boldsymbol w_y^T\boldsymbol h+b_y)}{\sum_{i=1}^C\exp(\boldsymbol w_i^T\boldsymbol h+b_i)}$, with respect to the feature $\boldsymbol h$:
$$
\frac{\partial L_{CE}}{\partial\boldsymbol h}=-\frac{\partial \log p_y}{\partial p_y}\frac{\partial p_y}{\partial y}=-\frac{1}{p_y}\cdot (p_y\boldsymbol w_y-\sum_{i=1}^C p_yp_i\boldsymbol w_i)=-(\boldsymbol w_y-\sum_{i=1}^Cp_i \boldsymbol w_i),
$$
For the ASM loss, we also derive its gradient as: 
$$
\frac{\partial L_{ASM}}{\partial \boldsymbol h}=-(\boldsymbol w_y-\gamma\sum_{i\neq y}\boldsymbol w_i)
$$

By comparing the gradients of the two losses, it can be seen that, the CE loss assigns different weight $p_i$ for each class prototype, while the ASM loss assigns a fixed weight $\gamma$ for all class prototypes. Therefore, the gradient of the  ASM loss cannot converge to $0$, leading to that the norm of features always grows, especially when the norm of $\boldsymbol w_i$ s also increases (please see verification in Fig. 19). This is one of the causes that the ASM loss cannot be directly used for practical training like CE. The other cause is the instability of these prototypes. As illustrated in Fig. 19 (c)),  the minimal angular between prototypes becomes very small quickly.

These two causes can be easily addressed. In our scheme, we turn to fix the class prototypes and then use the ASM loss for training of DNNs. In this way, we can kill two birds with one stone. As shown in Fig. 2, the ASM loss can achieve comparable results with CE, where the ASM loss still assigns correctly classified and wrongly classified data points with equal weights.  Moreover, we have further considered the spherical constrained case (see Sec. 3.3), in which the features are constrained to the unit sphere and class prototypes are fixed as a simplex equiangular tight frame (ETF). The ASM loss will be lower bounded by $-(1+\gamma)\boldsymbol w_y^T$, and can be used in practical training (as shown in Fig. 2). We can easily derive that, if the value of ASM loss is less than $-(1+\gamma)\sqrt{\frac{C-2}{2(C-1)}}$, we will have $\boldsymbol w_y^T\boldsymbol h>\boldsymbol w_i^T \boldsymbol h$ for all $i\neq y$, which indicates that the sample is correctly classified.

From: ICLR.cc/2023/Conference/Paper197/Authors
title: Response to Reviewer X1fQ (1/2)
comment: We thank the reviewer for the valuable comments as well as the suggestions for improvements. Our responses to the reviewer's comments are below:

**Q1**: "*The proposed ASM loss, as already pointed out by the authors, is unbounded and hence cannot be optimized without strong constrained on the weights. Could authors explain what characteristics does ASM have that a simple MSE doesn't? MSE, too, allows for analytical results and is not unbounded.*"

**R**: Thanks very much for your comment. Although the solution of MSE is analytical, it still cannot directly analyze the closed-form dynamics. Instead, MSE needs to make some simplifications or assumption about the learning process. For instance, under the MSE loss, Mixton et al. [1] formulate the gradient flow as a nonlinear ordinary differential equation and then linearize the equation by claiming that nonlinear terms are negligible; Han et al. [2] assume the gradient flow to be along the central path which requires the linear classifier to stay MSE-optimal for features throughout the dynamics. Thus, MSE is not so simple as expected to derive exact dynamics. This motivates us to develop the ASM loss, which does not require any simplifications or assumptions for the intermediate process in deriving the exact dynamics and works in a more analytical manner.

>[1] Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. *Sampling Theory, Signal Processing, and Data Analysis*, 20(2):1–13, 2022.
>
>[2] XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. In *International Conference on Learning Representations*, 2022 (Outstanding Paper Award).

**Q2**: "*Could authors elaborate on how different the ASM loss is from the Wasserstein loss? For example the loss introduced in Eq. 1 of (https://arxiv.org/pdf/1701.07875.pdf).*"

**R**: Thanks for the kind suggestion. The ASM loss is derived from the sample margin that measures the discriminativeness for a sample. Its final form $\frac{1}{N}\sum_{i=1}^N (\boldsymbol w_y^T\boldsymbol h+b_y - \gamma\sum_{j\neq y}(\boldsymbol w_y^T\boldsymbol h+b_y))$ resembles the Wasserstein loss $\mathbb E_{x\sim P_r}[f_w(x)]-\mathbb E_{x\sim P_\theta}[f_w(x)]$ if we regard positive logits $\boldsymbol w_y^T\boldsymbol h+b_y$ as scores from $P_r$  and negative logits $\boldsymbol w_i^T\boldsymbol h+b_i$  ($i\neq y$) as scores from $P_\theta$.  The main difference is that in the ASM loss the two logits $\boldsymbol w_y^T\boldsymbol h+b_y$ and  $\boldsymbol w_i^T\boldsymbol h+b_i$ ($i\neq y$)  are computed from $\boldsymbol h$ (i.e., the same input), rather than being sampled separably from two distributions as done in the Wasserstein loss. Moreover, the ASM loss in Eq.(2.2) does not constrain the values of these scores, in contrast the Wasserstein loss requires $f_w$ to be Lipschitz continuous.

**Q3**: "*According to Corollary 3.4, the l2-constraint setting appears very sensitive to the weight decay coefficient. Do authors agree that as a result, ASM + l2 is not a practical approach?*"

**R**: We agree with the reviewer that *ASM + L2* is not a practical approach, since the role of implicit penalization of features is difficult to measure when training neural networks. The sensitivity of weight decay on features will cause the norm of features to increase or decrease rapidly, especially when the norm of class prototypes is also changing. To promote the ASM loss to be practical, we turn to anchor the prototypes, which is demonstrated to be able to alleviate the instability and can help the ASM loss achieve comparable results with CE (please see Sec. 4.1).

From: ICLR.cc/2023/Conference/Paper197/Authors
title: Response to Reviewer MXeN (1/2)
comment: Thanks very much for your valuable time in reviewing our paper.
``Strong Reject" means that this work contributes nothing to the literature. This decision is too tough and unfair for us. XAI is an ongoing topic. There are many unknowns to explore. We expect to achieve one small step forward through our efforts.
In the following, we will try our best to address your concerns. We do not expect to totally reverse your recommendation, but just to demonstrate that this paper could contribute something useful to the literature.  

-- **About the motivation of this work:**
Before responding to the detailed comments, we would like to introduce the research timeline of neural collapse and the motivation of this work, which are helpful in understanding our contribution. 
As a pioneer, Papyan \& Han \& Donoho [1] empirically discovered the prevalence of Neural Collapse during the terminal phase of deep learning training.  Fang et al. [2] then introduced **the layer-peeled model** that regards last-layer features as free optimization variables and formulates the optimization as a nonconvex yet analytically tractable problem. These two works take the CE loss as the study object. Han et al. [3] revolved around the more tractable MSE loss for a better understanding of the behaviors of last-layer features and class prototypes under the layer-peeled model. However, this work still needs to make some simplifications and assumes that the gradient flow is along the central path that requires the linear classifier to stay MSE-optimal for features throughout the dynamics. Although CE and MSE are the most widely used losses, they are not simple enough to derive exact dynamics in certain frameworks, leading to difficulty in grasping the gap between modeling and practical optimization. In view of this, in this paper, we propose a new loss---the ASM loss---as an alternative study object, which is with a more concise form and is able to facilitate the subsequent analysis.
>[1] Papyan V, Han X Y, Donoho D L. Prevalence of neural collapse during the terminal phase of deep learning training[J]. *Proceedings of the National Academy of Sciences*, 2020, 117(40): 24652-24663.
>
>[2] Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. *Proceedings of the National Academy of Sciences*, 118(43): e2103091118, 2021.
>
>[3] XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. In *International Conference on Learning Representations*, 2022 (Outstanding Paper Award).

--**About the ASM loss:**
As stated above, the ASM loss serves as an alternative study object of Neural Collapse, which has the advantage of being linear in the outputs of the model. Thus, it can provide more opportunities to analyze more concrete behaviors and explore the shortcomings of the existing modeling. The main contribution of this work is to glimpse into deep learning through a concise form under the ASM loss, which enjoys the great advantage to simplify theoretical research very much. Moreover, it is not just a  tool for theoretical analysis. To make the ASM loss practical, we further provide several insights theoretically and empirically, including using prototype-anchored learning (PAL) with the ASM loss (Sec. 4.1), applying explicit regularization on features (Sec. 4.2), and rescaling the learning rates for the spherical constraint case (Sec. 4.3).

**About the assumption of the last-layer features:**

This assumption was first adopted by Fang et al. [2], which introduces the layer-peeled model that considers the last-layer features as free optimization variables. This assumption was also used in Han et al [3] (the outstanding paper of ICLR2022) and other papers published very recently [4-8].

In our work, we follow this assumption for our theoretical derivation. However, we did not fully adopt this assumption but turned to consider some practical issues: 1) We introduce different learning rates to distinguish last-layer features and class prototypes and draw more general conclusions. 2) To facilitate theoretical derivation, the previous works [2,8] claim that the norm of the features is implicitly penalized by penalizing the norm of network parameters.  We show empirically that implicit penalization is actually fragile (as depicted in Fig. 19). Instead, we add explicit regularization on features (see Sec. 4.2).

>[4] Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. *Sampling Theory, Signal Processing, and Data Analysis*, 20(2):1–13, 2022.
>
>[5] Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In *Proceedings of the 39th International Conference on Machine Learning*, volume 162 of *Proceedings of Machine Learning Research*, pages 21478–21505. PMLR, 2022.

From: ICLR.cc/2023/Conference/Paper197/Reviewer_YmzU
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper introduces a new loss ASM that utilizes the sample margin to simplify the derived gradient flow dynamics of last-layer features and prototypes in neural network training. With this new loss, the authors examine both unconstrained and regularized cases. In particular, they show that the solution depends on the initialization and can avoid inducing the neural collapse. The derived exponential convergence rate also partly explains the reason why neural network training does not need a lot of epochs. Furthermore, several numerical experiments are conducted to support the theoretical claims.
strength_and_weaknesses: Strength: 
1. This paper is well organized and clearly written. It is easy to read. The theoretical conclusions are also clearly summarized and further illustrated by corresponding remarks. 
2. The ASM loss is novel. The resulted training dynamics can lead to closed-form solutions which may be powerful for conducting more theoretical analysis on neural networks.
3. By showing that the ASM loss does not induce neural collapse, the authors clarify that the neural collapse is not common across all loss functions. 


Weakness:
1. There is no theoretical comparison between the ASM loss and other loss functions, especially considering that the ASM loss resembles the hinge loss (by removing the max operator and the margin term of the hinge loss). Moreover, according to Eq. (2.1), $L_{ASM}$ for a single data point can be arbitrarily negative (e.g., let $w^T_y h \to \infty$ and $w_j^T h \to 0$ when $j \neq y$), this means that the overall empirical loss can be arbitrarily small even though only one example is classified correctly, which perhaps largely hinders the training. In a word, the ASM loss assigns correctly classified and wrongly classified data points with equal weights. This property is not suitable for the practical training of classifiers. The authors should provide more explanations on this point.
2. In Fig. 1 (a), (b), and (d), it seems that $gamma$ has several values while only two curves are presented in these figures. Does the value of $gamma$ (from 0 to 0.05) have no relation with the corresponding parameters, e.g., loss, in these figures?  If so, could you please give a qualitative or quantitative explanation on its effect?
3. I notice that through this paper no assumption is given. Since this is a theoretical paper, does that mean there is no limitation for applying the presented theorems?

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clearly written.
Quality: The quality is high.
Novelty: The ASM loss and the corresponding results are novel to my best of knowledge.
Reproducibility: The code is not provided thus I can’t verify this.

summary_of_the_review: Overall, this paper provides solid and insightful analysis. However, more explanations regarding the ASM loss and its comparison with other loss functions should be added to further clarify its value to the research community.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper197/Reviewer_X1fQ
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This work introduces Averaged Sample Margin (ASM), a surrogate loss to the common cross-entropy loss (CE).
Unlike CE, the proposed ASM has a much simpler and linear derivative. From an analytical perspective, that allows for deriving the closed-form dynamics of gradient descent. Particularly, this paper studies the dynamics of the "layer-peeled model" in three setups:
1. ASM loss with unconstrained weights and prototypes.
2. ASM loss with l2-regularized weights and prototypes.
3. ASM loss with spherical constrained (normalized) weights and prototypes.

Following the analytical results, this work shows that gradient descent dynamics converge exponentially fast to the neural collapse solution, at a rate faster that previously analytically-derived rates and more consistent with practice.

Finally, given the insights from theory, the paper presents three practical recommendations:
1. Using prototype-anchored learning (PAL) with ASM loss.
2. Applying explicit regularization on features.
3. Rescaling the learning rate in the spherical constraint case.
strength_and_weaknesses: **Strengths**:
- The dynamics of CE is very complicated and prevents theorists from studying the learning of neural networks. Introducing ASM can definitely help investigating neural networks' learning dynamics.
- Deriving convergence rates in the layer-peeled setup is novel and interesting.
- The practical recommendations are valuable. Particularly, I found the "explicit regularization on features", something to be used in general.

**Weaknesses / Questions**:
- The proposed ASM loss, as already pointed out by the authors, is unbounded and hence cannot be optimized without strong constrained on the weights. Could authors explain what characteristics does ASM have that a simple MSE doesn't? MSE, too, allows for analytical results and is not unbounded.
- Could authors elaborate on how different the ASM loss is from the a Wasserstein loss? For example the loss introduced in Eq. 1 of (https://arxiv.org/pdf/1701.07875.pdf).
- According to Corollary 3.4, the l2-constraint setting appears very sensitive to the weight decay coefficient. Do authors agree that as a result, ASM + l2 is not a practical approach?
- The improvements observed in Fig. 2 and Table.1 are hard to measure since the standard deviation is not reported. Nevertheless, it appears that the improvement is insignificant. **However**, I understand that the main point of the paper is to introduce a loss that is convenient for theoretical analysis and not practice.

clarity,_quality,_novelty_and_reproducibility: Clarity and Quality:
The paper is well-motivated. The introduced loss and learning dynamics are presented in a nice order and the flow is good. However, I believe that the sentences are sometimes long and hard to follow. I suggest splitting longer sentences into shorter and more understandable sentences.

Novelty:
The ASM loss is novel. However, I do not immediately see a significant difference to the Wasserstein loss. I'm looking forward to reading the authors' response to the questions raised above.

Reproducibility:
The code is provided and is clear how to run. That being said, I haven't try running the code.
summary_of_the_review: In summary, this work introduces ASM, a surrogate loss to the common cross-entropy loss (CE). While interesting from an analytical perspective, ASM is unbounded and unstable in practice. From the analytical perspective, ASM allows for deriving the closed-form dynamics of gradient descent. Particularly, this paper studies the dynamics of gradient descent in unconstrained, l2-regularized, and spherical constrained settings. I have a hard time evaluating the novelty of ASM as I do not immediately see a significant difference to the Wasserstein loss. However, I will make sure to read authors response.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper197/Reviewer_MXeN
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The author study the training dynamics of a simplified model of DNNs:
- The first simplification is the use of the ASM loss, which is a classification loss which has the advantage of being linear in the outputs of the network.
- The second simplification is that the activations of the last hidden layer $H$ are taken to be parameters that evolve according to gradient descent directly (with a different learning rate).

Under these simplifications, the authors study the dynamics of gradient flow without $L_2$-regularization, with $L_2$-regularization (where the regularization is applied to $H$ directly) and with a constraint to the unit ball. They show some form of neural collapse for large training times.
strength_and_weaknesses: Thanks ot the above mentioned simplifications, the authors are able to obtain explicit formulas for the dynamics.

However these assumption are very strong, even though they are presented by the authors as if they were normal:
- The ASM loss is significantly different from the cross-entropy loss (which is non-linear in the labels, in contrast to the ASM), though the authors try to suggest that they are similar. For example a zero or negative ASM loss does not imply that the training set is labeled correctly, actually no value of the training ASM loss guarantees correct classification on the training set. I think this should be mentioned instead of saying that it is a variant of the cross-entropy loss. Considering these differences, the fact that the ASM loss is a reasonable approximation for the losses used in practice should be justified either theoretically or empirically.
- The assumption that the training dynamics of the activation of the last hidden layer $H$ are equivalent (up to changing the learning rate) to optimizing them directly is obviously not true in general. It might be approximately true in some case, but again it would need to be justified. 

Actually after these simplification, the model studied is a special case of a shallow linear fully-connected network with a linear loss:
$$
\mathcal{L}(W_1,W_2,b_2) = \mathrm{Tr}[G^T (W_2 W_1 X + b_2)]
$$
whose gradient dynamics can obviously be solved since the gradient is linear. One then recovers the result of the paper by taking $X=I_N$ and $G=Y$ (for the $Y$ matrix in the paper). In this light, the authors should at least cite and compare to the existing litterature on linear networks.

Finally, in spite of all these simplifications the paper does not offer much insight except a list of formulas.
clarity,_quality,_novelty_and_reproducibility: I find the presentation unclear and misleading. The results are presented in way that suggests that something stronger is proven, when in reality the authors make very strong assumptions. I was confused at first, but once I understood what model they were studying, all the results were obvious.

 
summary_of_the_review: After two very strong assumptions, the authors reduce the loss of DNNs to a quadratic loss and solve the (now linear) gradient flow. But all of this is hidden behind some cryptic explanations and big formulas.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 1: strong reject