Original: O N THE D YNAMICS UNDER THE A VERAGED S AMPLE M ARGIN L OSS AND B EYOND
Revised: O N THE D YNAMICS UNDER THE A VERAGED S AMPLE M ARGIN L OSS AND B EYOND

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent works have studied implicit biases in deep learning, especially the behavior of last-layer features and classifier weights.
Revised: Recent works have studied implicit biases in deep learning, especially the behavior of last-layer features and classifier weights.

Original: However, they usually need to simplify the dynamics under gradient descent due to the intractability of loss functions and neural architectures.
Revised: However, they usually need to simplify the dynamics under gradient descent due to the intractability of loss functions and neural architectures.

Original: In this paper, we introduce a concise loss function as a surrogate, namely the Averaged Sample Margin (ASM) loss, which offers more mathematical opportunities to analyze the closed-form dynamics while requiring few simplifications or assumptions, and allows for more practical considerations.
Revised: In this paper, we introduce a concise loss function as a surrogate, namely the Averaged Sample Margin (ASM) loss, which offers more mathematical opportunities to analyze the closed-form dynamics while requiring few simplifications or assumptions, and allows for more practical considerations.

Original: Based on the layer-peeled model that views last-layer features as free optimization variables, we build a complete analysis for the unconstrained, regularized, and spherical constrained cases.
Revised: Based on the layer-peeled model that views last-layer features as free optimization variables, we build a complete analysis for the unconstrained, regularized, and spherical constrained cases.

Original: We show that these dynamics mainly converge exponentially fast to a solution depending on the initialization of features and classifier weights, which can help explain why the training of deep neural networks usually takes only a few hundred epochs.
Revised: We show that these dynamics mainly converge exponentially fast to a solution depending on the initialization of features and classifier weights, which can help explain why the training of deep neural networks usually takes only a few hundred epochs.

Original: Our theoretical results can also aid in providing insights for improvements in practical training with the ASM loss or other losses, such as explicit feature regularization and rescaled learning rate for spherical cases.
Revised: Our theoretical results can also aid in providing insights for improvements in practical training with the ASM loss or other losses, such as explicit feature regularization and rescaled learning rate for spherical cases.

Original: Finally, we empirically demonstrate these theoretical results and insights with extensive experiments.
Revised: Finally, we empirically demonstrate these theoretical results and insights with extensive experiments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning with neural networks has achieved great success in a variety of tasks (Goodfellow et al., 2016), which, however, is not entirely understood in the interpolation and generalization of the learned models (Zhang et al., 2017; Neyshabur et al., 2017; Nakkiran et al., 2019;
Revised: Deep learning with neural networks has achieved great success in a variety of tasks (Goodfellow et al., 2016), which, however, is not entirely understood in the interpolation and generalization of the learned models (Zhang et al., 2017; Neyshabur et al., 2017; Nakkiran et al., 2019;

Original: Bubeck and
Revised: Bubeck and

Original: Sellke, 2021; Mei and Montanari, 2021).
Revised: Sellke, 2021; Mei and Montanari, 2021).

Original: Many modules, including loss functions (Lin et al., 2017;
Revised: Many modules, including loss functions (Lin et al., 2017;

Original: Hui and Belkin, 2021) and optimization algorithms (Auer et al., 2002; Duchi et al., 2011; Zeiler, 2012; Kingma and Ba, 2015), play a crucial role in the training of deep neural networks (DNNs), but lack convincing explanations.
Revised: Hui and Belkin, 2021) and optimization algorithms (Auer et al., 2002; Duchi et al., 2011; Zeiler, 2012; Kingma and Ba, 2015), play a crucial role in the training of deep neural networks (DNNs), but lack convincing explanations.

Original: Due to the complexity of multilayered architectures, recent works are devoted to simplifying modeling to better understand the behavior of neural networks, and then to gain insights for new algorithms, theoretical, and experimental investigations.
Revised: Due to the complexity of multilayered architectures, recent works are devoted to simplifying modeling to better understand the behavior of neural networks, and then to gain insights for new algorithms, theoretical, and experimental investigations.

Original: To understand the implicit regularization that improves generalization of the trained models in deep learning, many studies have investigated the implicit bias of gradient descent (Hardt et al., 2016;
Revised: To understand the implicit regularization that improves generalization of the trained models in deep learning, many studies have investigated the implicit bias of gradient descent (Hardt et al., 2016;

Original: Sekhari et al., 2021), with an emphasis on the behavior of linear predictors (or called classifiers) over linearly separable data (Soudry et al., 2018; Gunasekar et al., 2018; Nacson et al., 2019; Ji and Telgarsky, 2019; Ji et al., 2020; Shamir, 2021).
Revised: Sekhari et al., 2021), with an emphasis on the behavior of linear predictors (or called classifiers) over linearly separable data (Soudry et al., 2018; Gunasekar et al., 2018; Nacson et al., 2019; Ji and Telgarsky, 2019; Ji et al., 2020; Shamir, 2021).

Original: Soudry et al.
Revised: Soudry et al.

Original: (2018) proved that gradient descent iterates under exponentially-tailed loss minimization on separable data are biased toward ℓ 2 -maximum-margin solutions and further showed that continuing to optimize can still lead to performance improvements even if the validation loss increases.
Revised: (2018) proved that gradient descent iterates under exponentially-tailed loss minimization on separable data are biased toward ℓ 2 -maximum-margin solutions and further showed that continuing to optimize can still lead to performance improvements even if the validation loss increases.

Original: Shamir (2021) formally showed that standard gradient methods never overfit on separable data.
Revised: Shamir (2021) formally showed that standard gradient methods never overfit on separable data.

Original: These works impressively expose the implicit regularization induced by optimization algorithms and help to understand the generalization of the learned models, but they mainly focus on the behavior of linear classifiers that is only the last layer of neural networks, while the classifier actually interacts strongly with the features produced by many nonlinear layers and parameterized layers.
Revised: These works impressively expose the implicit regularization induced by optimization algorithms and help to understand the generalization of the learned models, but they mainly focus on the behavior of linear classifiers that is only the last layer of neural networks, while the classifier actually interacts strongly with the features produced by many nonlinear layers and parameterized layers.

Original: Thus, the relevant conclusions do not always apply to deep learning.
Revised: Thus, the relevant conclusions do not always apply to deep learning.

Original: For example, in (Soudry et al., 2018), the convergence rate of gradient descent is rather slow, wherein for almost all datasets, the distance to the maximum-margin solution decreases only as O (1 / log t ) .
Revised: For example, in (Soudry et al., 2018), the convergence rate of gradient descent is rather slow, wherein for almost all datasets, the distance to the maximum-margin solution decreases only as O (1 / log t ) .

Original: However, the training of DNNs typically takes only a few hundred epochs.
Revised: However, the training of DNNs typically takes only a few hundred epochs.

Original: In this paper, we show a exponential convergence that is more realistic.
Revised: In this paper, we show a exponential convergence that is more realistic.

Original: Another research line is after the empirical discovery of Neural Collapse by Papyan et al.
Revised: Another research line is after the empirical discovery of Neural Collapse by Papyan et al.

Original: (2020), which precisely characterizes a pervasive inductive bias of both features and linear classifiers at the terminal phase of training, and has opened a rich area of exploring this phenomenon in simplified mathematical frameworks (Mixon et al., 2022;
Revised: (2020), which precisely characterizes a pervasive inductive bias of both features and linear classifiers at the terminal phase of training, and has opened a rich area of exploring this phenomenon in simplified mathematical frameworks (Mixon et al., 2022;

Original: Fang et al., 2021;
Revised: Fang et al., 2021;

Original: Galanti et al., 2021; Zhu et al., 2021; Hui et al., 2022; Tirer and Bruna, 2022; Lu and Steinerberger, 2022; Kothapalli et al., 2022).
Revised: Galanti et al., 2021; Zhu et al., 2021; Hui et al., 2022; Tirer and Bruna, 2022; Lu and Steinerberger, 2022; Kothapalli et al., 2022).

Original: Neural Collapse provides a clear view of how the last-layer features and linear classifiers behave after interpolation and enables us to understand the benefit in generalization and robustness after achieving zero training error.
Revised: Neural Collapse provides a clear view of how the last-layer features and linear classifiers behave after interpolation and enables us to understand the benefit in generalization and robustness after achieving zero training error.

Original: Although we have a clear picture of the final phase, the intermediate dynamics is hard to analyze as a result of the intractability of CE.
Revised: Although we have a clear picture of the final phase, the intermediate dynamics is hard to analyze as a result of the intractability of CE.

Original: Therefore, some work (Mixon et al., 2022; Han et al., 2022; Zhou et al., 2022a; Tirer and Bruna, 2022; Kothapalli et al., 2022) revolves around the more tractable MSE loss that performs comparably to those trained with
Revised: Therefore, some work (Mixon et al., 2022; Han et al., 2022; Zhou et al., 2022a; Tirer and Bruna, 2022; Kothapalli et al., 2022) revolves around the more tractable MSE loss that performs comparably to those trained with

Original: CE (Demirkaya et al., 2020; Hui and Belkin, 2021), but they still need to make some simplifications or assumptions about the learning process.
Revised: CE (Demirkaya et al., 2020; Hui and Belkin, 2021), but they still need to make some simplifications or assumptions about the learning process.

Original: Mixon et al.
Revised: Mixon et al.

Original: (2022) formulate the gradient flow of the unconstrained feature model as a nonlinear ordinary differential equation and then linearize the equation by claiming that nonlinear terms are negligible for models initialized near the origin.
Revised: (2022) formulate the gradient flow of the unconstrained feature model as a nonlinear ordinary differential equation and then linearize the equation by claiming that nonlinear terms are negligible for models initialized near the origin.

Original: Furthermore, to derive exact dynamics, Han et al.
Revised: Furthermore, to derive exact dynamics, Han et al.

Original: (2022) assume that the gradient flow is along the central path which requires the linear classifier to stay MSE-optimal for features throughout the dynamics.
Revised: (2022) assume that the gradient flow is along the central path which requires the linear classifier to stay MSE-optimal for features throughout the dynamics.

Original: Therefore, MSE is still not simple enough to derive exact dynamics in certain mathematical frameworks, making it difficult to grasp the gap between the modeling and practical optimization.
Revised: Therefore, MSE is still not simple enough to derive exact dynamics in certain mathematical frameworks, making it difficult to grasp the gap between the modeling and practical optimization.

Original: In this paper, within the layer-peeled model (Fang et al., 2021) (or the unconstrained features model (Mixon et al., 2022)), we attempt to analyze the closed-form dynamics under gradient descent with as few simplifications as possible.
Revised: In this paper, within the layer-peeled model (Fang et al., 2021) (or the unconstrained features model (Mixon et al., 2022)), we attempt to analyze the closed-form dynamics under gradient descent with as few simplifications as possible.

Original: More specifically, we introduce a new loss function, namely the Averaged Sample Margin (ASM) loss, which has a concise form that intuitively expresses the objective of classification.
Revised: More specifically, we introduce a new loss function, namely the Averaged Sample Margin (ASM) loss, which has a concise form that intuitively expresses the objective of classification.

Original: Compared to CE and MSE, the ASM loss offers more mathematical opportunities to let us glimpse into deep learning with the closed-form dynamics while requiring few simplifications or assumptions, and will prepare us for more practical considerations and more reasonable designs in a later section.
Revised: Compared to CE and MSE, the ASM loss offers more mathematical opportunities to let us glimpse into deep learning with the closed-form dynamics while requiring few simplifications or assumptions, and will prepare us for more practical considerations and more reasonable designs in a later section.

Original: The main contributions of our work are highlighted as follows:
Revised: The main contributions of our work are highlighted as follows:

Original: • We derive exact dynamics of last-layer features and prototypes in unconstrained and regularized cases.
Revised: • We derive exact dynamics of last-layer features and prototypes in unconstrained and regularized cases.

Original: For spherical constrained cases that do not exhibit convexity, Lipschitzness, and β -smoothness, we also prove that gradient descent biases the normalized features towards a global minimizer.
Revised: For spherical constrained cases that do not exhibit convexity, Lipschitzness, and β -smoothness, we also prove that gradient descent biases the normalized features towards a global minimizer.

Original: • We provide the corresponding convergence analysis, which shows that the features and classifier weights converge to a solution depending on the initialization rather than induce the neural collapse solution that forms a simplex equiangular tight frame, suggesting that not all losses under gradient descent would lead to neural collapse (as verified in Section 3).
Revised: • We provide the corresponding convergence analysis, which shows that the features and classifier weights converge to a solution depending on the initialization rather than induce the neural collapse solution that forms a simplex equiangular tight frame, suggesting that not all losses under gradient descent would lead to neural collapse (as verified in Section 3).

Original: • We prove that the rate of convergence is exponential as a function of ζ ( t )
Revised: • We prove that the rate of convergence is exponential as a function of ζ ( t )

Original: =
Revised: =

Original: (cid:82) t 0 η ( τ )d τ , where η ( τ ) denotes the learning rate over time.
Revised: (cid:82) t 0 η ( τ )d τ , where η ( τ ) denotes the learning rate over time.

Original: This exponential convergence rate can help explain why the training of deep neural networks usually takes only a few hundred epochs.
Revised: This exponential convergence rate can help explain why the training of deep neural networks usually takes only a few hundred epochs.

Original: Moreover, we provide some insights for improvements in practical training with the ASM loss or other losses (cf.
Revised: Moreover, we provide some insights for improvements in practical training with the ASM loss or other losses (cf.

Original: Section 4).
Revised: Section 4).

Original:  T HE A VERAGED S AMPLE M ARGIN L OSS
Revised:  T HE A VERAGED S AMPLE M ARGIN L OSS

Original: In this paper, we mainly focus on the behavior of last-layer features and classifier weights in classification DNNs.
Revised: In this paper, we mainly focus on the behavior of last-layer features and classifier weights in classification DNNs.

Original: As described in prior works (Fang et al., 2021; Han et al., 2022), we also consider datasets containing inputs from C different classes with N examples in each class.
Revised: As described in prior works (Fang et al., 2021; Han et al., 2022), we also consider datasets containing inputs from C different classes with N examples in each class.

Original: The last-layer features h i,c = f Θ ( x i,c ) ∈
Revised: The last-layer features h i,c = f Θ ( x i,c ) ∈

Original: R p 1 extracted from the i -th example x i,c by a number of parameterized layers f Θ : X → R p are usually simplified as free optimization variables (Mixon et al., 2022; Fang et al., 2021; Han et al., 2022; Ji et al., 2022).
Revised: R p 1 extracted from the i -th example x i,c by a number of parameterized layers f Θ : X → R p are usually simplified as free optimization variables (Mixon et al., 2022; Fang et al., 2021; Han et al., 2022; Ji et al., 2022).

Original: The last layer of the network, i.e. , the linear classifier, possesses a class prototype w c ∈ R p and bias b c ∈ R for each class
Revised: The last layer of the network, i.e. , the linear classifier, possesses a class prototype w c ∈ R p and bias b c ∈ R for each class

Original: c ∈ [ C ] , which predicts a label using the rule arg max c ′ ( ⟨ w c ′ , h
Revised: c ∈ [ C ] , which predicts a label using the rule arg max c ′ ( ⟨ w c ′ , h

Original: i,c ⟩ + b c ′ ) .
Revised: i,c ⟩ + b c ′ ) .

Original: To better understand the dynamics of features and prototypes based on gradient descent, we consider a surrogate loss that offers more mathematical opportunities than the hard-to-analyze CE loss and the MSE loss.
Revised: To better understand the dynamics of features and prototypes based on gradient descent, we consider a surrogate loss that offers more mathematical opportunities than the hard-to-analyze CE loss and the MSE loss.

Original: Specifically, we introduce the Averaged Sample Margin (ASM) loss as follows:
Revised: Specifically, we introduce the Averaged Sample Margin (ASM) loss as follows:

Original: [Equation]
Revised: [Equation]

Original: where γ > 0 is the trade-off parameter and y denotes the class label of the feature h .
Revised: where γ > 0 is the trade-off parameter and y denotes the class label of the feature h .

Original: The sample margin m ( h , y ) =
Revised: The sample margin m ( h , y ) =

Original: w ⊤ y h
Revised: w ⊤ y h

Original: + b y − max j ̸ = y ( w ⊤ j h
Revised: + b y − max j ̸ = y ( w ⊤ j h

Original: + b j ) (Koltchinskii and Panchenko, 2002; Cao et al., 2019) is defined to measure the discriminativeness for a sample, which satisfies m ( h , y )
Revised: + b j ) (Koltchinskii and Panchenko, 2002; Cao et al., 2019) is defined to measure the discriminativeness for a sample, which satisfies m ( h , y )

Original: ≤k−
Revised: ≤k

Original: [Equation]
Revised: [Equation]

Original: ̸ = y [ w ⊤ y h 
Revised: , i.e. , L ASM [Equation] W h [Equation]

Original: + b y − ( w ⊤ j h + b j )] , i.e. , L ASM ( W h + b , y ) with γ = 1 k − 1 averaging the margins over all non-target classes is the lower bound of − m ( h , y ) . 
Revised: ) with γ = 1 k [Equation] 1 averaging the margins over all non-target classes is the lower bound of − m ( h , y ) . Moreover, 1 k − 1 (cid:80) j ̸ = y [ w ⊤ y h

Revised: + b y − ( w ⊤ j h + b j )] can also be regarded as the unhinged version that removes the max operator and margin term in the hinge loss 2 .

Original: Here, we replace 1 k − 1 with an additional parameter γ that balances positive and negative logits to draw general conclusions.
Revised: Here, we replace 1 k − 1 with an additional parameter γ that balances positive and negative logits to draw general conclusions.

Original: Furthermore, the ASM loss can also be regarded as a variant of CE and multi-binary CE loss:
Revised: Furthermore, the ASM loss can also be regarded as a variant of CE and multi-binary CE loss:

Original: [Equation]
Revised: [Equation]

Original: Intuitively, the ASM loss promotes the learned feature h to increase the logit with respect to the target class while decreasing the logits of the other classes.
Revised: Intuitively, the ASM loss promotes the learned feature h to increase the logit with respect to the target class while decreasing the logits of the other classes.

Original: If we follow up the layer-peeled model (Fang et al., 2021) to restrict the norms of both features and prototypes, the global minimizer of
Revised: If we follow up the layer-peeled model (Fang et al., 2021) to restrict the norms of both features and prototypes, the global minimizer of

Original: CN (cid:80) Ni =1 (cid:80)
Revised: CN (cid:80) Ni =1 (cid:80)

Original: Cc =1 L ASM ( W h
Revised: Cc =1 L ASM ( W h

Original: i,c , y i,c ) (the bias term b is omitted) will lead to Neural Collapse (Papyan et al., 2020; Han et al., 2022):
Revised: i,c , y i,c ) (the bias term b is omitted) will lead to Neural Collapse (Papyan et al., 2020; Han et al., 2022):

Original: Lemma 2.1 ( Neural Collapse under Averaged Sample Margin Loss ) .
Revised: Lemma 2.1 ( Neural Collapse under Averaged Sample Margin Loss ) .

Original: For norm-bounded prototypes and features, i.e., ∥ w c ∥ 2 ≤ E 1 and ∥ h i,c ∥ 2 ≤ E 2 , ∀ i ∈ [ N ] , ∀ c ∈ [ C ] , the global minimizer of1 CN (cid:80) Ni =1 (cid:80) Cc
Revised: For norm-bounded prototypes and features, i.e., ∥ w c ∥ 2 ≤ E 1 and ∥ h i,c ∥ 2 ≤ E 2 , ∀ i ∈ [ N ] , ∀ c ∈ [ C ] , the global minimizer of

Original: = L ASM ( W h
Revised: [Equation]

Original: i,c , y i,c ) implies neural collapse when p ≥ C − 1 .
Revised: ) implies neural collapse when p ≥ C − 1 .

Original: More specifically, the global minimizer is uniquely obtained at
Revised: More specifically, the global minimizer is uniquely obtained at

Original: [Equation]
Revised: [Equation]

Original: ∥ w c ∥ 2 = E 1 , and ∥ h i,c ∥ 2 = E 2 , ∀ i ∈ [ N ] , ∀ c ∈ [ C ] .
Revised: c [Equation] [ C ] .

Original: This lemma shows that the neural collapse solution is the only global optimal solution to minimize 1 CN (cid:80) i,
Revised: This lemma shows that the neural collapse solution is the only global optimal solution to minimize 1 CN (cid:80) i,

Original: c L ASM ( W h k,c , y i,c ) in the norm-bounded case.
Revised: c L ASM ( W h k,c , y i,c ) in the norm-bounded case.

Original: However, there exists an undesired direction to minimize the ASM loss in unconstrained cases, since the norm of features and prototypes tends to grow to infinity.
Revised: However, there exists an undesired direction to minimize the ASM loss in unconstrained cases, since the norm of features and prototypes tends to grow to infinity.

Original: For example, we can scale up W and b to obtain a smaller loss if
Revised: For example, we can scale up W and b to obtain a smaller loss if

Original: L ASM ( W h + b , y ) < 0 , which will happen analogously to CE (Liu et al., 2016; Wang et al., 2017;
Revised: L ASM ( W h + b , y ) < 0 , which will happen analogously to CE (Liu et al., 2016; Wang et al., 2017;

Original: Zhou et al., 2022c).
Revised: Zhou et al., 2022c).

Original: In this paper, we will analytically characterize the direction in which features
Revised: In this paper, we will analytically characterize the direction in which features

Original: H and prototypes W diverge.
Revised: H and prototypes W diverge.

Original: Specifically, we show that the continual gradient flow exhibits an implicit bias associated with the initialization of features and prototypes.
Revised: Specifically, we show that the continual gradient flow exhibits an implicit bias associated with the initialization of features and prototypes.

Original:  M AIN T HEORETICAL R ESULTS
Revised:  M AIN T HEORETICAL R ESULTS

Original: In this section, we build a complete analysis of the closed-form dynamics of the last-layer features and prototypes under the ASM loss in unconstrained, regularized, and spherical constrained cases.
Revised: In this section, we build a complete analysis of the closed-form dynamics of the last-layer features and prototypes under the ASM loss in unconstrained, regularized, and spherical constrained cases.

Original: We then derive the convergence analysis of these dynamics, which mainly shows exponential convergence.
Revised: We then derive the convergence analysis of these dynamics, which mainly shows exponential convergence.

Original: All proof can be found in the Appendix A.
Revised: All proof can be found in the Appendix A.

Original: 3. U NCONSTRAINED C ASE
Revised: 3. U NCONSTRAINED C ASE

Original: We first consider the unconstrained case (Mixon et al., 2022; Ji et al., 2022) in which there is no constraint or regularization on features and prototypes, i.e. , learning with the following objective 
Revised: We first consider the unconstrained case (Mixon et al., 2022; Ji et al., 2022) in which there is no constraint or regularization on features and prototypes, i.e. , learning with the following objective [Equation]

Original: [Equation]
Revised: [Equation]

Original: which can be reformulated as
Revised: which can be reformulated as

Original: [Equation]
Revised: [Equation]

Original: where H =
Revised: where H =

Original: [ h 1 , 1 , . . .
Revised: [ h 1 , 1 , . . .

Original: , h 1 ,C , h 2 , 1 , . . .
Revised: , h 1 ,C , h 2 , 1 , . . .

Original: , h N,C ] ∈ R p × CN is the matrix resulting from stacking together the feature vectors as columns, W = [ w 1 , w 2 , . . .
Revised: , h N,C ] ∈ R p × CN is the matrix resulting from stacking together the feature vectors as columns, W = [ w 1 , w 2 , . . .

Original: , w C ] ∈ R p × C is the matrix resulting from stacking stacking class prototypes as columns, ⊗ denotes the Kronecker product, I C is the identity matrix, 1 C , 1 N , and 1 CN are the length- C , - N , and - CN vectors of ones, respectively.
Revised: , w C ] ∈ R p × C is the matrix resulting from stacking stacking class prototypes as columns, ⊗ denotes the Kronecker product, I C is the identity matrix, 1 C , 1 N , and 1 CN are the length- C , - N , and - CN vectors of ones, respectively.

Original: Without loss of generality, we represent the label set { y i,c } 1 ≤ i ≤ N, 1 ≤ c ≤ C as the columns of the matrix I C ⊗ 1 ⊤ N .
Revised: Without loss of generality, we represent the label set { y i,c } 1 ≤ i ≤ N, 1 ≤ c ≤ C as the columns of the matrix I C ⊗ 1 ⊤ N .

Original: We follow the unconstrained features and prototypes modeling perspective (Mixon et al., 2022;
Revised: We follow the unconstrained features and prototypes modeling perspective (Mixon et al., 2022;

Original: Ji et al., 2022) that treats H as a free optimization variable.
Revised: Ji et al., 2022) that treats H as a free optimization variable.

Original: Within this model, we analyze the continuous dynamics of features H , prototypes W and biases b with gradient flow where time-oftraining is denoted by the variable t 2 .
Revised: Within this model, we analyze the continuous dynamics of features H , prototypes W and biases b with gradient flow where time-oftraining is denoted by the variable t 3 .

Original: Let Y = 1 CN ( γ 1 C 1 ⊤ CN − (1 + γ )( I C ⊗ 1 ⊤ N )) , then L ASM =
Revised: Let Y = 1 CN ( γ 1 C 1 ⊤ CN − (1 + γ )( I C ⊗ 1 ⊤ N )) , then L ASM =

Original: Tr( Y ⊤ W ⊤ H ) + γC − γ − 1 C 1 ⊤ C b .
Revised: Tr( Y ⊤ W ⊤ H ) + γC − γ − 1 C 1 ⊤ C b .

Original: Taking the partial derivative with respect to H , W , and b , respectively, we have:
Revised: Taking the partial derivative with respect to H , W , and b , respectively, we have:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and the corresponding learning dynamics following Gradient Descent (GD) is
Revised: and the corresponding learning dynamics following Gradient Descent (GD) is

Original: [Equation]
Revised: [Equation]

Original: where M = − Y = 1 CN ((1 + γ )( I C ⊗ 1 ⊤ N ) − γ 1 C 1 ⊤ CN ) , η 1 and η 2 are the learning rate of the features and prototypes, respectively.
Revised: where M = − Y = 1 CN ((1 + γ )( I C ⊗ 1 ⊤ N ) − γ 1 C 1 ⊤ CN ) , η 1 and η 2 are the learning rate of the features and prototypes, respectively.

Original: The reason to introduce different learning rates is that the representation H is actually the result of the interaction between a number of nonlinear layers and parameterized layers.
Revised: The reason to introduce different learning rates is that the representation H is actually the result of the interaction between a number of nonlinear layers and parameterized layers.

Original: This means that even if we use the same learning rate to optimize all parameters of the network, the feature H assumed to be a free optimization variable is almost impossible to be optimized at this learning rate 3 .
Revised: This means that even if we use the same learning rate to optimize all parameters of the network, the feature H assumed to be a free optimization variable is almost impossible to be optimized at this learning rate 4 .

Original: Moreover, we consider dynamic learning rates η 1 =
Revised: Moreover, we consider dynamic learning rates η 1 =

Original: η 1 ( t ) and η 2 = η 2 ( t ) that are usually adopted in practical implementations, such as the cosine annealing decay (Loshchilov and Hutter, 2017).
Revised: η 1 ( t ) and η 2 = η 2 ( t ) that are usually adopted in practical implementations, such as the cosine annealing decay (Loshchilov and Hutter, 2017).

Original: As can be seen, the dynamics of features H and prototypes W are independent of the bias term b , thus we can analyze the dynamics of H and W jointly, and analyze b independently:
Revised: As can be seen, the dynamics of features H and prototypes W are independent of the bias term b , thus we can analyze the dynamics of H and W jointly, and analyze b independently:

Original: Theorem 3.1 ( Dynamics of Features, Prototypes and Biases without Constraints ) .
Revised: Theorem 3.1 ( Dynamics of Features, Prototypes and Biases without Constraints ) .

Original: Consider the continual gradient flow (Equation 3.4) in which the dynamics follow the gradient descent direction of the Averaged Sample Margin loss in Eq.
Revised: Consider the continual gradient flow (Equation 3.4) in which the dynamics follow the gradient descent direction of the Averaged Sample Margin loss in Eq.

Original: Let Z ( t ) =
Revised: Let Z ( t ) =

Original: ( H ( t ) , W ( t )) , if η 1 ( t 1 ) η 2 ( t 2 ) =
Revised: ( H ( t ) , W ( t )) , if η 1 ( t 1 ) η 2 ( t 2 ) =

Original: η 1 ( t 2 ) η 2 ( t 1 ) for any t 1 , t 2 ≥ 0 , we have the following closed-form dynamics 
Revised: η 1 ( t 2 ) η 2 ( t 1 ) for any t 1 , t 2 ≥ 0 , we have the following closed-form dynamics [Equation] + Π − [Equation]

Revised: Z

Original: [Equation]
Revised: [Equation]

Revised: −

Revised: [Equation]

Original: 3.5) and
Revised: t [Equation] t [Equation]

Revised: −

Original: [Equation]
Revised: [Equation]

Original: [-where α ϵ 1 , α ϵ 2 , β ϵ 1 and β ϵ 2 for ϵ ∈ {±} are the scalars that only depend on C , N , γ , η 1 and η 2 (where the detailed forms can be seen in the appendix), Z 
 = ( H 0 , W 0 ) , C (-] t ) = (cid:16) ζ 1 ( t ) I CN
Revised: [-where α ϵ 1 , α ϵ 2 , β ϵ 1 and β ϵ 2 for ϵ ∈ {±} are the scalars that only depend on C , N , γ , η 1 and η 2 (where the detailed forms can be seen in the appendix), Z 
 = ( H 0 , W 0 ) , C (-] t [Equation]

Original: [Equation]
Revised: [Equation]

Revised: , (3.5) and

Original: [Equation]
Revised: [Equation]

Original: t )

Original: I

Original: [Equation]
Revised: [Equation]

Original: {+where α ϵ 1 , α ϵ 2 , β ϵ 1 and β ϵ 2 for ϵ ∈ {±} are the scalars that only depend on C , N , γ , η 1 and η 2 (where the detailed forms can be seen in the appendix), Z 
 = ( H 0 , W 0 ) , C ( t ) = (cid:16) ζ 1 ( t ) I CN 0ζ 2 ( t ) I C+} (cid:17) , ζ 1 ( t ) =
Revised: {+where α ϵ 1 , α ϵ 2 , β ϵ 1 and β ϵ 2 for ϵ ∈ {±} are the scalars that only depend on C , N , γ , η 1 and η 2 (where the detailed forms can be seen in the appendix), Z 
 = ( H 0 , W 0 ) , C ( t ) = (cid:16) ζ 1 ( t ) I CN 0ζ 2 ( t ) I C+} (cid:17) , ζ 1 ( t ) =

Original: (cid:82) t 0 η 1 ( τ )d τ , ζ 2 ( t ) =
Revised: (cid:82) t 0 η 1 ( τ )d τ , ζ 2 ( t ) =

Original: (cid:82) t 0 η 2 ( τ )d τ , Π ϵ 1 , Π ϵ 2 and Π 3 for ϵ ∈ {±} are orthogonal projection operators onto the following respective eigenspaces:
Revised: (cid:82) t 0 η 2 ( τ )d τ , Π ϵ 1 , Π ϵ 2 and Π 3 for ϵ ∈ {±} are orthogonal projection operators onto the following respective eigenspaces:

Original: [Equation]
Revised: [Equation]

Original: Remark.

Original: Note that E ϵ 1 , E ϵ 2 and E 3 are orthogonal to each other, E +1 (or E − 1 ) denotes the subspace where all features are in the same (or opposite) direction of their corresponding prototypes while the mean of prototypes is zero, E ϵ 2 denotes the subspace where all features and prototypes collapse respectively into two scaled versions of the same unit vector, and E 3 denotes the subspace where the

Original: [Equation]

Original: [Figure]
Revised: [Figure]

Original: a) The training loss. (b) The training accuracy with the prediction rule arg max c w ⊤ c h . As expected, the features align to their corresponding prototypes when γ < 2 C − 2 . (c) denote the distance between (cid:98) Z ( t )
Revised: [Equation]

Original: = Z ( t ) / ∥ Z ( t ) ∥ 2 and (cid:98) Z = Z / ∥ Z ∥ 2 . As expected in Eq.
Revised: [Equation]

Original: (3.9), the convergence rate is exponential when 0 < γ < 2 C − 2 , and will be fastest if γ = 1 C − 1 . (d) denotes the norm of Z ( t ) that increases exponentially. As can be noticed, (cid:98) Z ( t ) does not converge to (cid:98) Z but tend to be orthogonal to (cid:98) Z when γ = 0 . 05 >
Revised: Remark.

Original: [Equation] , that is, lim t [Equation] mean of all features from the same class is zero with all prototypes being zero.
Revised: Note that E ϵ 1 , E ϵ 2 and E 3 are orthogonal to each other, E +1 (or E − 1 ) denotes the subspace where all features are in the same (or opposite) direction of their corresponding prototypes while the mean of prototypes is zero, E ϵ 2 denotes the subspace where all features and prototypes collapse respectively into two scaled versions of the same unit vector, and E 3 denotes the subspace where the mean of all features from the same class is zero with all prototypes being zero.

Original: For classification tasks, we expect the features align to their corresponding prototypes with a cosine similarity of 1, i.e. , the solution in E =
Revised: For classification tasks, we expect the features align to their corresponding prototypes with a cosine similarity of 1, i.e. , the solution in E =

Original: ( H , W ) :
Revised: ( H , W ) :

Original: H = k W ⊗ 1 ⊤ N , W ∈ R p × C , k ∈ R + } that implies two manifestations of Neural Collapse: variability collapse and convergence to self-duality (Papyan et al., 2020).
Revised: H = k W ⊗ 1 ⊤ N , W ∈ R p × C , k ∈ R + } that implies two manifestations of Neural Collapse: variability collapse and convergence to self-duality (Papyan et al., 2020).

Original: In the following, we show that Z ( t ) ∥
Revised: In the following, we show that Z ( t ) ∥

Original: Z ( t ) ∥ will converge to a solution in E :
Revised: Z ( t ) ∥ will converge to a solution in E :

Original: Corollary 3.2 ( Convergence in the Unconstrained Case ) .
Revised: Corollary 3.2 ( Convergence in the Unconstrained Case ) .

Original: Under the conditions and notation of
Revised: Under the conditions and notation of

Original: Theorem 3.1, let s =
Revised: Theorem 3.1, let s =

Original: η 1 (0) η 2 (0) ,
Revised: η 1 (0) η 2 (0) ,

Original: if 0 < γ < 2 C − 2 (where C > 2 ) or C = 2 , and lim t →∞ ζ 1 ( t ) = ∞ , the gradient flow (as in Eq.
Revised: if 0 < γ < 2 C − 2 (where C > 2 ) or C = 2 , and lim t →∞ ζ 1 ( t ) = ∞ , the gradient flow (as in Eq.

Original: (3.4) ) will behave as:
Revised: (3.4) ) will behave as:

Original: [Equation]
Revised: [Equation]

Original: where Z=
Revised: where Z

Original: [Equation]
Revised: [Equation]

Original: = Π − 1 Z 0 , and the residual term ∆ ( t ) decreases as least as ∥ ∆ ( t ) ∥ =
Revised: Z 0 , and the residual term ∆ ( t ) decreases as least as ∥ ∆ ( t ) ∥ =

Original: O(cid:18) e √ [Equation] 1( t
Revised: O

Original: [Equation]
Revised: [Equation]

Original: · max {− γC, ( C − 2) γ − 2 } (cid:19) , and so the normalized Z ( t ) converges to Z ∥ Z ∥ in
Revised: , and so the normalized Z ( t ) converges to Z ∥ Z ∥ in

Original: [Equation]
Revised: [Equation]

Original: which further indicates lim t →∞ Z ( t )
Revised: which further indicates lim t →∞ Z ( t )

Original: ∥
Revised: ∥

Original: Z ( t ) ∥ ∈ E .
Revised: Z ( t ) ∥ ∈ E .

Original: Moreover, if γ ̸ = 1 C − 1 , then lim t →∞ max i b i ( t ) min i b i ( t ) = 1 .
Revised: Moreover, if γ ̸ = [Equation] 1 , then lim t →∞ max i b i ( t ) min i b i ( t ) = 1 .

Original: This corollary shows that even without any explicit regularization, when minimizing the ASM loss using gradient descent, we prove that Z ( t )
Revised: This corollary shows that even without any explicit regularization, when minimizing the ASM loss using gradient descent, we prove that Z ( t )

Original: ∥
Revised: ∥

Original: Z ( t ) ∥ converges to a special point Z ∥
Revised: Z ( t ) ∥ converges to a special point Z ∥

Original: Z ∥ ∈ E determined by the initialization of Z 0 and the ratio s , but does not necessarily perform as a neural collapse solution that forms a simplex equiangular tight frame as CE or MSE do (Papyan et al., 2020; Han et al., 2022).
Revised: Z ∥ ∈ E determined by the initialization of Z 0 and the ratio s , but does not necessarily perform as a neural collapse solution that forms a simplex equiangular tight frame as CE or MSE do (Papyan et al., 2020; Han et al., 2022).

Original: In addition, the rate of convergence is exponential as a function of the integral of the learning rate, i.e. , O(cid:18) [Equation] N
Revised: In addition, the rate of convergence is exponential as a function of the integral of the learning rate, i.e. , O

Original: [Equation]
Revised: [Equation]

Original: max [Equation] [Equation] 2 } (cid:19) , which indicates that the convergence of updating both features and prototypes by gradient descent is much faster than O (1 / log t ) that only updates prototypes (linear predictors) on linearly separable data (Soudry et al., 2018).
Revised: } (cid:19) , which indicates that the convergence of updating both features and prototypes by gradient descent is much faster than O (1 / log t ) that only updates prototypes (linear predictors) on linearly separable data (Soudry et al., 2018).

Original: In a sense, this convergence rate explains why training deep neural networks usually takes only several hundred or thousand epochs.
Revised: In a sense, this convergence rate explains why training deep neural networks usually takes only several hundred or thousand epochs.

Original: Moreover, if γ = 1 C − 1 , we can obtain the fastest convergence of
Revised: Moreover, if γ = 1 C − 1 , we can obtain the fastest convergence of

Original: Eq.
Revised: Eq.

Original: (3.9), that is, (cid:13)(cid:13)(cid:13) [Equation] t 
Revised: (3.9), that is, (cid:13)(cid:13)(cid:13) Z ( t )

Original: [Equation]
Revised: [Equation]

Original: For example, when η 1 ( t )
Revised: For example, when η 1 ( t )

Original: =
Revised: =

Original: η is a constant learning rate, then ζ 1 ( t ) = ηt → ∞ as t → ∞ , and the gradient flow in Eq.
Revised: η is a constant learning rate, then ζ 1 ( t ) = ηt → ∞ as t → ∞ , and the gradient flow in Eq.

Original: (3.4) shows an exponential convergence rate of Z ( t )
Revised: (3.4) shows an exponential convergence rate of Z ( t )

Original: ∥
Revised: ∥

Original: Z ( t ) ∥ to Z ∥
Revised: Z ( t ) ∥ to Z ∥

Original: Z ∥ , but it also leads to an exponential increase with the rate e (1+ γ ) √ ζ 1( t ) ζ 2( t ) C √ N in the norm of features and prototypes, which is almost unbearable for the practical training of the models.
Revised: Z ∥ , but it also leads to an exponential increase with the rate e (1+ γ ) √ ζ 1( t ) ζ 2( t ) C √ N in the norm of features and prototypes, which is almost unbearable for the practical training of the models.

Original: Therefore, we need to limit the excessive growth of these norms.
Revised: Therefore, we need to limit the excessive growth of these norms.

Original: 3. R EGULARIZED C ASE
Revised: 3. R EGULARIZED C ASE

Original: We then consider the following regularized optimization problem that explicitly introduces ℓ 2 regularization (often called “weight decay”) on features, prototypes, and biases to avoid large values:
Revised: We then consider the following regularized optimization problem that explicitly introduces ℓ 2 regularization (often called “weight decay”) on features, prototypes, and biases to avoid large values:

Original: [Equation]
Revised: [Equation]

Original: where ∥ · ∥ F denotes the Frobenius norm and λ > 0 is the regularization parameter.
Revised: where ∥ · ∥ F denotes the Frobenius norm and λ > 0 is the regularization parameter.

Original: Taking the partial derivative with respect to H , W , and b , we have
Revised: Taking the partial derivative with respect to H , W , and b , we have

Original: [Equation]
Revised: [Equation]

Original: and the corresponding learning dynamics following gradient descent with different learning rates and ℓ 2 -norm regularization can be formulated as
Revised: and the corresponding learning dynamics following gradient descent with different learning rates and ℓ 2 -norm regularization can be formulated as

Original: [Equation]
Revised: [Equation]

Original: The dynamics of this regularized gradient flow can be proved as follows
Revised: The dynamics of this regularized gradient flow can be proved as follows

Original: Theorem 3.3 ( Dynamics of Features, Prototypes, and Biases under Weight Decay ) .
Revised: Theorem 3.3 ( Dynamics of Features, Prototypes, and Biases under Weight Decay ) .

Original: Consider the continual gradient flow (Equation 3.11) in which the dynamics follows the gradient descent direction of the Averaged Sample Margin loss in Eq.
Revised: Consider the continual gradient flow (Equation 3.11) in which the dynamics follows the gradient descent direction of the Averaged Sample Margin loss in Eq.

Original: Let Z ( t ) =
Revised: Let Z ( t ) =

Original: ( H ( t ) , W ( t )) , if η 1 ( t 1 ) η 2 ( t 2 ) = 
Revised: ( H ( t ) , W ( t )) , if η 1 ( t 1 ) η 2 ( t 2 ) = η

Original: η 1 ( t 2 ) η 2 ( t 1 ) for any t 1 , t 2 ≥0 , we have the following closed-form dynamics.
Revised: [Equation] t 2 ) η 2 ( t 1 ) for any t 1 , t 2 ≥

Original: [Equation]
Revised: [Equation]

Revised: we have the following closed-form dynamics.

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where Π +1 Z 0 , Π − 1 Z 0 , Π +1 Z 0 , Π − 1 Z 0 , and Π 3 Z 0 follow the definition in Theorem 3.1, a ϵ 1 , a ϵ 2 , b ϵ 1 , b ϵ 2 , a 3 , and b 3 for ϵ ∈ {±} are the scalars that depend only on C , N , γ , λ , η 1 , and η 2 (where the detailed forms can be seen in Appendix A), ϕ ( t ) =
Revised: where Π +1 Z 0 , Π − 1 Z 0 , Π +1 Z 0 , Π − 1 Z 0 , and Π 3 Z 0 follow the definition in Theorem 3.1, a ϵ 1 , a ϵ 2 , b ϵ 1 , b ϵ 2 , a 3 , and b 3 for ϵ ∈ {±} are the scalars that depend only on C , N , γ , λ , η 1 , and η 2 (where the detailed forms can be seen in Appendix A), ϕ ( t ) =

Original: exp( − λ (cid:82) t 0 η 2 ( τ )d τ ) , and ψ ( t ) =(cid:82) t
Revised: exp( − λ (cid:82) t 0 η 2 ( τ )d τ ) , and ψ ( t ) =

Original: [Equation]

Original:  2 ( τ [Equation] exp( λ (cid:82) [Equation]
Revised: (cid:82) t 0 ζ 2 ( τ ) exp( λ (cid:82) τ 0 η 2 ( s )d s )

Original: d [Equation]
Revised: d τ .

Original: The convergence under the regularized case can also be derived as:
Revised: The convergence under the regularized case can also be derived as:

Original: Corollary 3.4 ( Convergence Under ℓ 2 Regularization ) .
Revised: Corollary 3.4 ( Convergence Under ℓ 2 Regularization ) .

Original: Under the conditions and notation of
Revised: Under the conditions and notation of

Original: Theorem 3.3, let s =
Revised: Theorem 3.3, let s =

Original: η 1 (0) η 2 (0) ,
Revised: η 1 (0) η 2 (0) ,

Original: if 0 < γ < 2 C − 2 (where C > 2 ) or C = 2 , and lim t →∞ ζ 1 ( t ) = ∞ , then there exist constants π + h , π − h , π + w , π − w , and ω only depending on λ , γ , s , C , and N , such that the gradient flow (as in Eq.
Revised: if 0 < γ < 2 C − 2 (where C > 2 ) or C = 2 , and lim t →∞ ζ 1 ( t ) = ∞ , then there exist constants π + h , π − h , π + w , π − w , and ω only depending on λ , γ , s , C , and N , such that the gradient flow (as in Eq.

Original: (3.11) ) behaves as:
Revised: (3.11) ) behaves as:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where ( H +1 , W +1 )
Revised: where ( H +1 , W +1 )

Original: =
Revised: =

Original: Π
Revised: Π

Original: +1 Z 0 , ( H − 1 , W − 1 ) =
Revised: +1 Z 0 , ( H − 1 , W − 1 ) =

Original: Π − 1 Z 0 .
Revised: Π − 1 Z 0 .

Original: Furthermore, we have the following convergence results for Z ( t ) :
Revised: Furthermore, we have the following convergence results for Z ( t ) :

Original: • If λ > 1+ γ C √ N , then lim t →∞ ∥ Z ( t ) ∥ = 0 ; • If λ = 1+ γ C √ N , then lim t →∞ Z ( t ) =
Revised: • If λ > 1+ γ C √ N , then lim t →∞ ∥ Z ( t ) ∥ = 0 ; • If λ = 1+ γ C √ N , then lim t →∞ Z ( t ) =

Original: (cid:16) H +1 + 1 − s 

Revised: (cid:16) H +1 + 1 − s

Original: + s H − 1 , W +1 − 1 − s 

Revised: + s H − 1 , W +1 − 1 − s

Original: + s W − 1 (cid:17) ;
Revised: + s W − 1 (cid:17) ;

Original: • If λ < 1+ γ C √ N , then lim t →∞ ∥ Z ( t ) ∥ = ∞ .
Revised: • If λ < 1+ γ C √ N , then lim t →∞ ∥ Z ( t ) ∥ = ∞ .

Original: Remark.
Revised: Remark.

Original: The results in Corollary 3.4 suggest that adding an appropriate weight decay on both features and prototypes can avoid impractical effects, since the norm of Z ( t ) shrinking to 0 or diverging toward infinity will significantly affect the training of DNNs.
Revised: The results in Corollary 3.4 suggest that adding an appropriate weight decay on both features and prototypes can avoid impractical effects, since the norm of Z ( t ) shrinking to 0 or diverging toward infinity will significantly affect the training of DNNs.

Original: Several recent works (Zhu et al., 2021;
Revised: Several recent works (Zhu et al., 2021;

Original: Zhou et al., 2022a) described that the features are implicitly penalized, but this implicit penalization is fragile, as depicted in Fig.
Revised: Zhou et al., 2022a) described that the features are implicitly penalized, but this implicit penalization is fragile, as depicted in Fig.

Original: As a consequence, we emphasize adding explicit regularization to features, not just implicit penalization attached by other components in Sec.
Revised: As a consequence, we emphasize adding explicit regularization to features, not just implicit penalization attached by other components in Sec.

Original: 3. S PHERICAL C ONSTRAINED C ASE
Revised: 3. S PHERICAL C ONSTRAINED C ASE

Original: We finally consider another constrained case in which features are restricted on the p -sphere S p − 1 = { x : ∥ x ∥ 2 = 1 , x ∈ R p } by explicitly performing ℓ 2 normalization to prevent arithmetic overflow or underflow happening in training DNNs, and we fix the prototypes 4 to satisfy W 1 C = 0 5 , then the optimization problem in Eq.
Revised: We finally consider another constrained case in which features are restricted on the p -sphere S p − 1 = { x : ∥ x ∥ 2 = 1 , x ∈ R p } by explicitly performing ℓ 2 normalization to prevent arithmetic overflow or underflow happening in training DNNs, and we fix the prototypes 5 to satisfy W 1 C = 0 6 , then the optimization problem in Eq.

Original: (3.1) can be reformulated as
Revised: (3.1) can be reformulated as

Original: [Equation]
Revised: [Equation]

Original: (3.15) where (cid:99) H =
Revised: (3.15) where (cid:99) H =

Original: ( (cid:98) h 1 , 1 , ..., (cid:98) h c,N ) , and (cid:98) h
Revised: ( (cid:98) h 1 , 1 , ..., (cid:98) h c,N ) , and (cid:98) h

Original: i,c = h i,c
Revised: i,c = h i,c

Original: ∥ h i,c ∥ 2 .
Revised: ∥ h i,c ∥ 2 .

Original: We then take the partial derivative with respect to H , and the discrete dynamical system based on gradient descent is formulated as
Revised: We then take the partial derivative with respect to H , and the discrete dynamical system based on gradient descent is formulated as

Original: [Equation]
Revised: [Equation]

Original: .16) where ( ∂ (cid:99) H ∂ H ) ⊤ is an element-wise operator, that is ( ∂ (cid:99) H ∂ H )
Revised: where ( ∂ (cid:99) H ∂ H ) ⊤ is an element-wise operator, that is ( ∂ (cid:99) H ∂ H )

Original: ⊤ H ′ 
Revised: ⊤ H ′ =

Original: = (cid:16)

Original: ∥ h

Original: i

Original: [Equation]

Original: c ∥

Original: [Equation]

Original: I p −

Original: [Equation]
Revised: [Equation]

Original: h i [Equation] c
Revised: for

Original: [Equation]

Original: (cid:17) for

Original: H ′
Revised: H ′

Original: ∈ R p × CN .
Revised: ∈ R p × CN .

Original: Despite the fact that the optimization objective in Eq.
Revised: Despite the fact that the optimization objective in Eq.

Original: (3.15) does not show convexity, Lipschitzness, and β -smoothness on H due to the ℓ 2 normalization operator, the normalized features that obey the gradient descent iterates in Eq.
Revised: (3.15) does not show convexity, Lipschitzness, and β -smoothness on H due to the ℓ 2 normalization operator, the normalized features that obey the gradient descent iterates in Eq.

Original: (3.16) can still converge to their corresponding normalized prototypes, i.e. , achieve the global minimum of Eq.
Revised: (3.16) can still converge to their corresponding normalized prototypes, i.e. , achieve the global minimum of Eq.

Original: Theorem 3.5.
Revised: Theorem 3.5.

Original: Considering the discrete dynamics in Eq.
Revised: Considering the discrete dynamics in Eq.

Original: (3.16) , if ∀ i ∈ [ N ] , c ∈
Revised: (3.16) , if ∀ i ∈ [ N ] , c ∈

Original: [ C ] , (cid:98) w ⊤ c (cid:98) h
Revised: [ C ] , (cid:98) w ⊤ c (cid:98) h

Original: i,c (0) > − 1 , the learning rate η ( t ) satisfies that
Revised: i,c (0) > − 1 , the learning rate η ( t ) satisfies that

Original: [Equation]
Revised: [Equation]

Original: -increasing
Revised: is non-increasing,

Original: [Equation]
Revised: [Equation]

Original: lim t →∞ η ( t +1) η ( t )
Revised: lim t [Equation] η ( t +1) η ( t )

Original: = 1 , and there exists a constant ε > 0 , s.t., η ( t ) >
Revised: = 1 , and there exists a constant ε > 0 , s.t., η ( t ) >

Original: ε , then we have
Revised: ε , then we have

Original: [Equation]
Revised: [Equation]

Original: and further if lim t →∞ ∥ H ( t ) ∥ < ∞ , then there exists a constant µ > 0 , such that the error above shows exponential decrease:
Revised: and further if lim t →∞ ∥ H ( t ) ∥ < ∞ , then there exists a constant µ > 0 , such that the error above shows exponential decrease:

Original: [Equation]
Revised: [Equation]

Original: Moreover, if (cid:98) w ⊤ c (cid:98) h i,c (0) = − 1 , then h i,c ( t )
Revised: Moreover, if (cid:98) w ⊤ c (cid:98) h i,c (0) = − 1 , then h i,c ( t )

Original: =
Revised: =

Original: h i,c (0) .
Revised: h i,c (0) .

Original:   E XPERIMENTS
Revised:  I NSIGHTS AND E XPERIMENTS

Original: In this section we provide some insights to better train DNNs according to the conclusions in Sec. 
Revised: In this section, we provide some insights to better train DNNs according to the conclusions in Sec. 3.

Original: We then corroborate our theoretical results and insights with extensive experiments.
Revised: We then corroborate our theoretical results and insights with extensive experiments.

Original: More details and results can be found in Appendix B.
Revised: More details and results can be found in Appendix B.

Original: [Figure]
Revised: [Figure]

Original: 4. T HE ASM L OSS WITH P ROTOTYPE - ANCHORED LEARNING
Revised: 4. T HE ASM L OSS WITH P ROTOTYPE - ANCHORED LEARNING

Original: Since directly using the ASM loss will lead to volatile effects, which is mainly reflected in the rapid increases of feature norms and the imbalance between class prototypes when training DNNs with the stochastic gradient method, as shown in Fig.
Revised: Since directly using the ASM loss will lead to volatile effects, which is mainly reflected in the rapid increases of feature norms and the imbalance between class prototypes when training DNNs with the stochastic gradient method, as shown in Fig.

Original: Inspired by recent works (Zhou et al., 2022b; Kasarla et al., 2022) that use the neural collapse solution as an inductive bias, we can anchor prototypes W during training, and then the dynamics of H with ℓ 2 -norm-based regularization in
Revised: Inspired by recent works (Zhou et al., 2022b; Kasarla et al., 2022) that use the neural collapse solution as an inductive bias, we can anchor prototypes W during training, and then the dynamics of H with ℓ 2 -norm-based regularization in

Original: Eq.
Revised: Eq.

Original: (3.10) can be formulated as the first-order non-homogeneous linear difference equation:
Revised: (3.10) can be formulated as the first-order non-homogeneous linear difference equation:

Original: [Equation]
Revised: [Equation]

Original: and the solution to the non-homogeneous linear DE can be easily derived:
Revised: and the solution to the non-homogeneous linear DE can be easily derived:

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Consider the continual gradient flow (Equation (4.1) ) in which the prototypes W are fixed, we have the closed-form dynamics:
Revised: Consider the continual gradient flow (Equation (4.1) ) in which the prototypes W are fixed, we have the closed-form dynamics:

Original: [Equation]
Revised: [Equation]

Original: which further indicates that (cid:13)(cid:13)
Revised: which further indicates that (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Prototype-anchored learning (PAL) is very effective in alleviating the instability by transforming the classification problem as a feature alighment problem (Zhou et al., 2022b).
Revised: Prototype-anchored learning (PAL) is very effective in alleviating the instability by transforming the classification problem as a feature alighment problem (Zhou et al., 2022b).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, the
Revised: 2, the

Original: ASM loss with PAL and FNPAL can achieves comparable or even better results compared to CE.
Revised: ASM loss with PAL and FNPAL can achieves comparable or even better results compared to CE.

Original: [Table]
Revised: [Table]

Original: λ
Revised: λ

Original: =67.
Revised: =67.

Original: 75.42 λ = 5 e −67.
Revised: 75.42 λ = 5 e −67.

Original: λ = 1 e −67.
Revised: λ = 1 e −67.

Original: λ = 5 e −69.
Revised: λ = 5 e −69.

Original: 4. E XPLICIT F EATURE R EGULARIZATION
Revised: 4. E XPLICIT F EATURE R EGULARIZATION

Original: In this paper, we directly consider explicit feature regularization to avoid excessive growth of feature norms, rather than fully adopt the implicit penalization described in prior works (Fang et al., 2021;
Revised: In this paper, we directly consider explicit feature regularization to avoid excessive growth of feature norms, rather than fully adopt the implicit penalization described in prior works (Fang et al., 2021;

Original: [Figure]
Revised: [Figure]

Original: , respectively.
Revised: , respectively.

Original: As can be
Revised: As can be

Original: , EFR can improve the performance of OOD detection by alleviating the over-confidence of OOD samples and making the energy scores  ID samples more concentrated.
Revised: , EFR can improve the performance of OOD detection by alleviating the over-confidence of OOD samples and making the energy scores of ID samples more concentrated.

Original: More intuitively, comparing (
Revised: More intuitively, comparing (

Original: ) to (e 
Revised: ) to (e to

Original: , EFR effectively limits the growth of feature norms and improves the between
Revised: , EFR effectively limits the growth of feature norms and improves the between

Original: Zhou et al., 2022a).
Revised: Zhou et al., 2022a).

Original: Explicit regularization on features can significantly remedy over-confidence and even improve generalization, which can be demonstrated in Tab. 1 and Fig.
Revised: Explicit regularization on features can significantly remedy over-confidence and even improve generalization, which can be demonstrated in Tab. 1 and Fig.

Original: 3, where experiments on long-tailed classification (Zhong et al., 2021) and out-of-distribution (OOD) detection (Liu et al., 2020) are conducted, respectively.
Revised: 3, where experiments on long-tailed classification (Zhong et al., 2021) and out-of-distribution (OOD) detection (Liu et al., 2020) are conducted, respectively.

Original: Moreover, adding explicit feature regularization can speed up the convergence of (cid:99) H ( t ) to (cid:92) W M according to Theorem 4.1, as verified in Fig. 15.
Revised: Moreover, adding explicit feature regularization can speed up the convergence of (cid:99) H ( t ) to (cid:92) W M according to Theorem 4.1, as verified in Fig. 15.

Original: 4. R ESCALED L EARNING R ATE FOR T HE S PHERICAL C ONSTRAINED C ASE
Revised: 4. R ESCALED L EARNING R ATE FOR T HE S PHERICAL C ONSTRAINED C ASE

Original: As shown in Sec. 3.3, the gradient of the objective in Eq.
Revised: As shown in Sec. 3.3, the gradient of the objective in Eq.

Original: (3.15) with respect to h i,c is −
Revised: (3.15) with respect to h i,c is −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: h i,c (cid:98) h
Revised: h i [Equation] c (cid:98) h

Original: ⊤ i,c ) w c , which shows that the gradient is significantly influenced by
Revised: ⊤ i [Equation] c ) w c , which shows that the gradient is significantly influenced by

Original: [Equation]
Revised: [Equation]

Original: For example, the gradient will disappear when ∥ h
Revised: For example, the gradient will disappear when ∥ h

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ∥ 2 is too large, and the gradient will be too large when ∥ h i,c ∥ is too small.
Revised: ∥ 2 is too large, and the gradient will be too large when ∥ h i,c ∥ is too small.

Original: A natural solution would be scaling up the learning rate η ( t ) with the feature norm ∥ h i,c ∥ 2 for each sample 6 , and we can still guarantee the convergence in Theorem 3.
Revised: A natural solution would be scaling up the learning rate η ( t ) with the feature norm ∥ h i,c ∥ 2 for each sample 7 , and we can still guarantee the convergence in Theorem 3.

Original: if η ( t ) is non-increasing and satisfies η (0)(1+ γ ) CN ≤ ∥ w c ∥ 2 .
Revised: if η ( t ) is non-increasing and satisfies η (0)(1+ γ ) CN ≤ ∥ w c ∥ 2 .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2c, the ASM loss with the rescaled learning rate (RLR) obviously converges faster than without it.
Revised: 2c, the ASM loss with the rescaled learning rate (RLR) obviously converges faster than without it.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduce the Averaged Sample Margin loss (ASM) as a surrogate to analyze the behavior of last-layer features and prototypes.
Revised: In this paper, we introduce the Averaged Sample Margin loss (ASM) as a surrogate to analyze the behavior of last-layer features and prototypes.

Original: Thanks to the conciseness of the ASM loss, we derive the exact dynamics under gradient descent in unconstrained, regularized, and spherical constrained cases.
Revised: Thanks to the conciseness of the ASM loss, we derive the exact dynamics under gradient descent in unconstrained, regularized, and spherical constrained cases.

Original: We then prove that these dynamics will converge exponentially to a particular solution relying on the initialization.
Revised: We then prove that these dynamics will converge exponentially to a particular solution relying on the initialization.

Original: Inspired by these results, we further provide some insights for improvements, such as the ASM loss with prototype-anchored learning, explicit feature regularization, and rescaled learning rate for spherical cases.
Revised: Inspired by these results, we further provide some insights for improvements, such as the ASM loss with prototype-anchored learning, explicit feature regularization, and rescaled learning rate for spherical cases.

Original: We finally verify these theoretical results and insights with extensive experiments, including numerical analysis, visual classification, imbalanced learning, and out-of-distribution detection.
Revised: We finally verify these theoretical results and insights with extensive experiments, including numerical analysis, visual classification, imbalanced learning, and out-of-distribution detection.

Original: We expect that the ASM loss can be an excellent tool to help the community further understand the behavior of deep neural networks and beyond the scope of the paper.
Revised: We expect that the ASM loss can be an excellent tool to help the community further understand the behavior of deep neural networks and beyond the scope of the paper.

Original: Appendix for “Dynamics of The ASM Loss and Beyond” A P ROOFS FOR L EMMAS , T HEOREMS , P ROPOSITIONS AND C OROLLARIES A. P ROOF OF L EMMA 2. 
Revised: Appendix for “Dynamics of The ASM Loss and Beyond” A P ROOFS FOR L EMMAS , T HEOREMS , P ROPOSITIONS AND C OROLLARIES A. P ROOF OF L EMMA 2.

Original: Lemma 2.1 (The Neural Collapse of The Averaged Sample Margin loss).
Revised: Lemma 2.1 (The Neural Collapse of The Averaged Sample Margin loss).

Original: For norm-bounded prototypes and features, i.e., ∥ w c ∥ 2 ≤ E 1 and ∥ h i,c ∥ 2 ≤ E 2 , ∀ i ∈ [ N ] , ∀ c ∈ [ C ] , the global minimizer of 1 CN (cid:80) Ni =1 (cid:80)
Revised: For norm-bounded prototypes and features, i.e., ∥ w c ∥ 2 ≤ E 1 and ∥ h i,c ∥ 2 ≤ E 2 , ∀ i ∈ [ N ] , ∀ c ∈ [ C ] , the global minimizer of 1 CN (cid:80) Ni =1 (cid:80)

Original: Cc
Revised: Cc

Original: =
Revised: =

Original: L ASM ( W h
Revised: L ASM ( W h

Original: i,c , y i,c ) implies neural collapse when p ≥ C − 1 .
Revised: i,c , y i,c ) implies neural collapse when p ≥ C − 1 .

Original: More specifically, the global minimizer is uniquely obtained at w ⊤ i w j ∥ w i ∥ 

Revised: More specifically, the global minimizer is uniquely obtained at w ⊤ i w j ∥ w i ∥

Original: ∥ w j ∥ 2 = − 1 C − 1 , ∀ i ̸
Revised: ∥ w j ∥ 2 = − 1 C − 1 , ∀ i ̸

Original: = j , w ⊤ yi,c h
Revised: = j , w ⊤ yi,c h

Original: i,c ∥ w yi,c ∥ 2 ∥ h i,c ∥ 2 = 1 , ∥ w c ∥ 2 = E 1 , and ∥ h i,c
Revised: i,c ∥ w yi,c ∥ 2 ∥ h i,c ∥ 2 = 1 , ∥ w c ∥ 2 = E 1 , and ∥ h i,c

Original: ∥ 2 = E 2 , ∀ i ∈
Revised: ∥ 2 = E 2 , ∀ i ∈

Original: [ N ] , ∀ c ∈
Revised: [ N ] , ∀ c ∈

Original: [ C ] .
Revised: [ C ] .

Original: Proof.
Revised: Proof.

Original: The proof is based on lower bounding the objective 1 CN (cid:80) Ni =1 (cid:80)
Revised: The proof is based on lower bounding the objective 1 CN (cid:80) Ni =1 (cid:80)

Original: Cc =1 L ASM ( W h
Revised: Cc =1 L ASM ( W h

Original: i,c , y i,c ) by a sequence of inequalities that holds if and only if the solution forms Neural Collapse (Papyan et al., 2020).
Revised: i,c , y i,c ) by a sequence of inequalities that holds if and only if the solution forms Neural Collapse (Papyan et al., 2020).

Original: Let (cid:98) w = 1 C (cid:80) C
Revised: Let (cid:98) w = 1 C (cid:80) C

Original: c
Revised: c

Original: =1 w c , according to the definition of L SM , we have
Revised: =1 w c , according to the definition of L SM , we have

Original: [Equation]
Revised: [Equation]

Original: where the first and second inequalities are based on the facts that w ⊤ y i,
Revised: where the first and second inequalities are based on the facts that w ⊤ y i,

Original: c h i,c ≤ E 1 E 2 and ( C (cid:98) w − w y
Revised: c h i,c ≤ E 1 E 2 and ( C (cid:98) w − w y

Original: i,c )
Revised: i,c )

Original: ⊤ h i,c ≥ − E 2 ∥ C (cid:98) w − w y i,c ∥ 2 , respectively.
Revised: ⊤ h i,c ≥ − E 2 ∥ C (cid:98) w − w y i,c ∥ 2 , respectively.

Original: In the third equality, we used the Cauchy-Schwarz inequality, and the last inequality we use the facts that ∥ w c ∥ 2 ≤ E 1 and ∥ (cid:98) w ∥ 2 ≥ 0 .
Revised: In the third equality, we used the Cauchy-Schwarz inequality, and the last inequality we use the facts that ∥ w c ∥ 2 ≤ E 1 and ∥ (cid:98) w ∥ 2 ≥ 0 .

Original: According the above derivation, the equality holds if and only if ∀ i ∈
Revised: According the above derivation, the equality holds if and only if ∀ i ∈

Original: [ N ] , ∀ c ∈
Revised: [ N ] , ∀ c ∈

Original: [ C ] , w ⊤ y i,c h i,c =
Revised: [ C ] , w ⊤ y i,c h i,c =

Original: E 1 E 2 , ( C (cid:98) w − w y i,c )
Revised: E 1 E 2 , ( C (cid:98) w − w y i,c )

Original: ⊤ h
Revised: ⊤ h

Original: i,c = − E 2 ∥ C (cid:98) w − w y
Revised: i,c = − E 2 ∥ C (cid:98) w − w y

Original: i,c ∥ 2 , ∥ C (cid:98) w − w c ∥ 2 = ∥ C (cid:98)
Revised: i,c ∥ 2 , ∥ C (cid:98) w − w c ∥ 2 = ∥ C (cid:98)

Original: w − w C ∥ 2 , ∥ w c ∥ 2 = E 1 , and ∥ (cid:98) w ∥ 2 = 0 .
Revised: w − w C ∥ 2 , ∥ w c ∥ 2 = E 1 , and ∥ (cid:98) w ∥ 2 = 0 .

Original: These equations can be simplified as
Revised: These equations can be simplified as

Original: [Equation]
Revised: [Equation]

Original: i w j ∥ w i | 2 ∥ w j ∥ 2 = − 1 C − 1 , ∀ i ̸
Revised: i w j ∥ w i | 2 ∥ w j ∥ 2 = − 1 C − 1 , ∀ i ̸

Original: = j , w ⊤ yi,c h
Revised: = j , w ⊤ yi,c h

Original: i,c ∥ w yi,c ∥ 2 ∥ h i,c ∥ 2 = 1 , ∥ w c ∥ 2 = E 1 , and ∥ h i,c
Revised: i,c ∥ w yi,c ∥ 2 ∥ h i,c ∥ 2 = 1 , ∥ w c ∥ 2 = E 1 , and ∥ h i,c

Original: ∥ 2 = E 2 , ∀ i ∈
Revised: ∥ 2 = E 2 , ∀ i ∈

Original: [ N ] , ∀ c ∈
Revised: [ N ] , ∀ c ∈

Original: [ C ] , which also implies neural collapse.
Revised: [ C ] , which also implies neural collapse.

Original: A. P ROOF OF T HEOREM 3. 
Revised: A. P ROOF OF T HEOREM 3.

Original: In this section, we will provide the proof of Theorem 3.1.
Revised: In this section, we will provide the proof of Theorem 3.1.

Original: Our analysis will actually rely on the eigenvalues and eigenspaces of five subspaces E +1 , E − 1 , E +2 , E − 2 and E 3 in Theorem 3.1.
Revised: Our analysis will actually rely on the eigenvalues and eigenspaces of five subspaces E +1 , E − 1 , E +2 , E − 2 and E 3 in Theorem 3.1.

Original: Their concrete projection operator can be found in Appendix A.8.
Revised: Their concrete projection operator can be found in Appendix A.8.

Original: In the following, we show that these five subspaces are orthogonal:
Revised: In the following, we show that these five subspaces are orthogonal:

Original: Lemma A.1.
Revised: Lemma A.1.

Original: The following five subspaces are orthogonal to each other and satisfy R p × C ( N +1) =
Revised: The following five subspaces are orthogonal to each other and satisfy R p × C ( N +1) =

Original: E +1 ⊕
Revised: E +1 ⊕

Original: [Equation]
Revised: [Equation]

Original: where ϵ ∈ {± 1 } , and k ̸ = 0 .
Revised: where ϵ ∈ {± 1 } , and k ̸ = 0 .

Original: Proof.
Revised: Proof.

Original: For ( H 1 , W 1 ) =
Revised: For ( H 1 , W 1 ) =

Original: ( 1 √ N ( W 1 ⊗ 1 ⊤ N ) , W 1 )
Revised: ( 1 √ N ( W 1 ⊗ 1 ⊤ N ) , W 1 )

Original: ∈
Revised: ∈

Original: E +

Revised: E +

Original: and ( H 2 , W 2 )
Revised: and ( H 2 , W 2 )

Original: = ( − 1 √ N ( W 2 ⊗ 1 ⊤ N ) , W 2 ) ∈
Revised: = ( − 1 √ N ( W 2 ⊗ 1 ⊤ N ) , W 2 ) ∈

Original: E − 1 , we have
Revised: E − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: W ⊤
Revised: W ⊤

Original: [Equation]
Revised: [Equation]

Original: = 0 .
Revised: = 0 .

Original: For ( H 1 , W 1 ) =
Revised: For ( H 1 , W 1 ) =

Original: ( 1 √ N h 1 1 ⊤ CN , h 1 1 ⊤ C ) ∈
Revised: ( 1 √ N h 1 1 ⊤ CN , h 1 1 ⊤ C ) ∈

Original: E +2 and ( H 2 , W 2 ) =
Revised: E +2 and ( H 2 , W 2 ) =

Original: ( − 1 √ N h 2 1 ⊤ CN ,
Revised: ( − 1 √ N h 2 1 ⊤ CN ,

Original: h 2 1 ⊤ C ) ∈
Revised: h 2 1 ⊤ C ) ∈

Original: E − 2 , we have
Revised: E − 2 , we have

Original: H 1 H ⊤ 2 + W 1 W ⊤ 2 = − 1 N h 1 1 ⊤ CN 1 CN
Revised: H 1 H ⊤ 2 + W 1 W ⊤ 2 = − 1 N h 1 1 ⊤ CN 1 CN

Original: h ⊤ 

Revised: h ⊤

Original: + h 1 1 ⊤ C 1 C
Revised: + h 1 1 ⊤ C 1 C

Original: h ⊤ 2 = 0 .
Revised: h ⊤ 2 = 0 .

Original: For ( H 1 , W 1 ) =
Revised: For ( H 1 , W 1 ) =

Original: ( ϵ 1 1 √ N ( W 1 ⊗ 1 ⊤ N ) , W 1 )
Revised: ( ϵ 1 1 √ N ( W 1 ⊗ 1 ⊤ N ) , W 1 )

Original: ∈
Revised: ∈

Original: E ϵ 

Revised: E ϵ

Original: and ( H 2 , W 2 )
Revised: and ( H 2 , W 2 )

Original: = ( ϵ 2 1 √ N h 2 1 ⊤ CN , h 2 1 ⊤ C )
Revised: = ( ϵ 2 1 √ N h 2 1 ⊤ CN , h 2 1 ⊤ C )

Original: ∈
Revised: ∈

Original: E ϵ 2 , since W 1 1 C = 

Revised: E ϵ 2 , since W 1 1 C =

Original: and ( W 1 ⊗ 1 ⊤ N ) 1 CN = N W 1 1 C = 0 , we have
Revised: and ( W 1 ⊗ 1 ⊤ N ) 1 CN = N W 1 1 C = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ⊤
Revised: ⊤

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: ⊤
Revised: ⊤

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 1 1 C h ⊤ 2 = 0 .
Revised: 1 1 C h ⊤ 2 = 0 .

Original: For ( H 1 , W 1 ) =
Revised: For ( H 1 , W 1 ) =

Original: ( ϵ 1 √ N ( W 1 ⊗ 1 ⊤ N ) , W 1 )
Revised: ( ϵ 1 √ N ( W 1 ⊗ 1 ⊤ N ) , W 1 )

Original: ∈
Revised: ∈

Original: E ϵ 

Revised: E ϵ

Original: and ( H 2 , W 2 )
Revised: and ( H 2 , W 2 )

Original: =
Revised: =

Original: ( H 2 , 0)
Revised: ( H 2 , 0)

Original: ∈ E 3 , we have
Revised: ∈ E 3 , we have

Original: [Equation]
Revised: [Equation]

Original: ⊤ = 0 .
Revised: ⊤ = 0 .

Original: For ( H 1 , W 1 ) =
Revised: For ( H 1 , W 1 ) =

Original: ( ϵ 1 √ N h 1 1 ⊤ CN , h 1 1 ⊤ C
Revised: ( ϵ 1 √ N h 1 1 ⊤ CN , h 1 1 ⊤ C

Original: ) ∈
Revised: ) ∈

Original: E ϵ 

Revised: E ϵ

Original: and ( H 2 , W 2 )
Revised: and ( H 2 , W 2 )

Original: =
Revised: =

Original: ( H 2 , 0)
Revised: ( H 2 , 0)

Original: ∈ E 3 , since H 2 ( I C ⊗ 1 N ) = 0 , we have
Revised: ∈ E 3 , since H 2 ( I C ⊗ 1 N ) = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: To sum up, we prove that the five subspaces E +1 , E − 1 , E +2 , E − 2 , E 3 are orthogonal to each other.
Revised: To sum up, we prove that the five subspaces E +1 , E − 1 , E +2 , E − 2 , E 3 are orthogonal to each other.

Original: Moreover, we have
Revised: Moreover, we have

Original: [Equation]
Revised: [Equation]

Original: Since these dimensions sum to pC ( N + 1)
Revised: Since these dimensions sum to pC ( N + 1)

Original: =
Revised: =

Original: dim
Revised: dim

Original: [Equation]
Revised: [Equation]

Original: R p × CN ⊕ R p × C ) , then R p × C ( N +1) =
Revised: R p × CN ⊕ R p × C ) , then R p × C ( N +1) =

Original: E +1 ⊕
Revised: E +1 ⊕

Original: E − 1 ⊕ E +2 ⊕ E − 2 ⊕ E 3 .
Revised: E − 1 ⊕ E +2 ⊕ E − 2 ⊕ E 3 .

Original: Theorem 3.1 (Dynamics of Features, Prototypes and Biases without Constraints).
Revised: Theorem 3.1 (Dynamics of Features, Prototypes and Biases without Constraints).

Original: Consider the continual gradient flow (Equation 3.4) in which the dynamics follow the gradient descent direction of Sample Margin loss in Eq.
Revised: Consider the continual gradient flow (Equation 3.4) in which the dynamics follow the gradient descent direction of Sample Margin loss in Eq.

Original: Let Z ( t ) =
Revised: Let Z ( t ) =

Original: ( H ( t ) , W ( t )) , if η 1 ( t 1 ) η 2 ( t 2 ) =
Revised: ( H ( t ) , W ( t )) , if η 1 ( t 1 ) η 2 ( t 2 ) =

Original: η 1 ( t 2 ) η 2 ( t 1 ) for any t 1 , t 2 ≥ 0 , we have the following closed-form dynamics
Revised: η 1 ( t 2 ) η 2 ( t 1 ) for any t 1 , t 2 ≥ 0 , we have the following closed-form dynamics

Original: Z ( t ) =
Revised: Z ( t ) =

Original: Π +1 Z
Revised: Π +1 Z

Original: [Equation]
Revised: [Equation]

Original: α +1 ( t )
Revised: α +1 ( t )

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t )
Revised: t )

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Π − 1 Z 0 (cid:0) α − 1 ( t ) C ( t ) + β − 1 ( t ) I C ( N +1) (cid:1) + Π 3 Z+ Π +2 Z 0 (cid:0) α +2 ( t ) C ( t ) + β +2 ( t )
Revised: Π − 1 Z 0 (cid:0) α − 1 ( t ) C ( t ) + β − 1 ( t ) I C ( N +1) (cid:1) + Π 3 Z+ Π +2 Z 0 (cid:0) α +2 ( t ) C ( t ) + β +2 ( t )

Original: I C ( N +1) (cid:1)
Revised: I C ( N +1) (cid:1)

Original: +
Revised: +

Original: Π − 2 Z 0 (cid:0) α − 2 ( t ) C ( t ) + β − 2 ( t ) I C
Revised: Π − 2 Z 0 (cid:0) α − 2 ( t ) C ( t ) + β − 2 ( t ) I C

Original: ( N +1) (cid:1) , (A.2) and
Revised: ( N +1) (cid:1) , (A.2) and

Original: [Equation]
Revised: [Equation]

Original: where α ϵ 1 , α ϵ 2 , β ϵ 1 and β ϵ 2 for ϵ ∈ {±} are the scalars that only depend on C , N , γ , ηand η 2 (where the detailed forms can be seen in the appendix), Z 

Revised: where α ϵ 1 , α ϵ 2 , β ϵ 1 and β ϵ 2 for ϵ ∈ {±} are the scalars that only depend on C , N , γ , ηand η 2 (where the detailed forms can be seen in the appendix), Z

Original: = ( H 0 , W 0 ) , C ( t ) =
Revised: = ( H 0 , W 0 ) , C ( t ) =

Original: [Equation]
Revised: [Equation]

Original: 2 and Π 3 for ϵ ∈ {±} are orthogonal projection operators onto the following respective eigenspaces:
Revised: 2 and Π 3 for ϵ ∈ {±} are orthogonal projection operators onto the following respective eigenspaces:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Writing Z ( t )
Revised: Writing Z ( t )

Original: =
Revised: =

Original: ( H ( t ) , W ( t )) , then the unsolved portion of the system is given by
Revised: ( H ( t ) , W ( t )) , then the unsolved portion of the system is given by

Original: [Equation]
Revised: [Equation]

Original: where M = 1 CN ((1 + γ )( I C ⊗ 1 ⊤ N ) − γ 1 C 1 ⊤ CN ) .
Revised: where M = 1 CN ((1 + γ )( I C ⊗ 1 ⊤ N ) − γ 1 C 1 ⊤ CN ) .

Original: Let A ( t ) =
Revised: Let A ( t ) =

Original: (cid:18)
Revised: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: , then the equation above can be reformulated as the initial-value problem associated with the linear ordinary differential equation:
Revised: , then the equation above can be reformulated as the initial-value problem associated with the linear ordinary differential equation:

Original: [Equation]
Revised: [Equation]

Original: For any t 1 , t 2 , we have the matrix commutator of A ( t 1 ) and A ( t 2 )
Revised: For any t 1 , t 2 , we have the matrix commutator of A ( t 1 ) and A ( t 2 )

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: where the last equality is based on the fact that η 2 ( t 1 ) η 1 ( t 2 ) =
Revised: where the last equality is based on the fact that η 2 ( t 1 ) η 1 ( t 2 ) =

Original: η 2 ( t 2 ) η 1 ( t 1 ) .
Revised: η 2 ( t 2 ) η 1 ( t 1 ) .

Original: Therefore, according to Magnus approach, we have
Revised: Therefore, according to Magnus approach, we have

Original: [Equation]
Revised: [Equation]

Original: Let ζ 1 ( t ) =
Revised: Let ζ 1 ( t ) =

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( t )
Revised: ( t )

Original: [Equation]
Revised: [Equation]

Original: ⊤ M
Revised: ⊤ M

Original: [Equation]
Revised: [Equation]

Original: = (cid:18) ζ 1 ( t ) I CN  ζ 2 ( t ) I C (cid:19) , and L ( t ) =
Revised: = (cid:18) ζ 1 ( t ) I CN  ζ 2 ( t ) I C (cid:19) , and L ( t ) =

Original: BC ( t ) , we have
Revised: BC ( t ) , we have

Original: [Equation]
Revised: [Equation]

Original: Moreover, we have
Revised: Moreover, we have

Original: [Equation]
Revised: [Equation]

Original: A.
Revised: A.

Original: [Equation]
Revised: [Equation]

Original: thus we obtain
Revised: thus we obtain

Original: [Equation]
Revised: [Equation]

Original: Looking at the above equation, we just need to analyze the eigenspaces and eigenvalues of B .
Revised: Looking at the above equation, we just need to analyze the eigenspaces and eigenvalues of B .

Original: Considering the following five subspaces:
Revised: Considering the following five subspaces:

Original: [Equation]
Revised: [Equation]

Original: where ϵ ∈ {±} .
Revised: where ϵ ∈ {±} .

Original: According to Lemma A.1, these five subspaces are orthogonal to each other and satisfy R p
Revised: According to Lemma A.1, these five subspaces are orthogonal to each other and satisfy R p

Original: [Equation]
Revised: [Equation]

Original: In the following, we will prove that E ϵ 1 , E ϵ 2 and E 3 are five eigenspaces of B .
Revised: In the following, we will prove that E ϵ 1 , E ϵ 2 and E 3 are five eigenspaces of B .

Original: More specifically, each nonzero member of each claimed eigenspace is an eigenvector, and the claimed eigenspaces have distinct eigenvalues.
Revised: More specifically, each nonzero member of each claimed eigenspace is an eigenvector, and the claimed eigenspaces have distinct eigenvalues.

Original: Note that for ( H , W ) ∈ R p × CN ⊕ R p × C , we have ( H , W ) B
Revised: Note that for ( H , W ) ∈ R p × CN ⊕ R p × C , we have ( H , W ) B

Original: = ( W M ⊤ , HW ) .
Revised: = ( W M ⊤ , HW ) .

Original: For ( H , W ) ∈
Revised: For ( H , W ) ∈

Original: E ϵ 1 , we have H =
Revised: E ϵ 1 , we have H =

Original: ϵ √ N W ⊗ 1 ⊤ N and W 1 C = 0 , thus
Revised: ϵ √ N W ⊗ 1 ⊤ N and W 1 C = 0 , thus

Original: [Equation]
Revised: [Equation]

Original: i.e. , ( H , W ) is an eigenvector of B with eigenvalue ϵ
Revised: i.e. , ( H , W ) is an eigenvector of B with eigenvalue ϵ

Original: (1+ γ ) C √ N .
Revised: (1+ γ ) C √ N .

Original: For ( H , W ) ∈
Revised: For ( H , W ) ∈

Original: E ϵ 2 , we have H = ϵ √ N h 1 ⊤ CN and W = h 1 ⊤ C , thus
Revised: E ϵ 2 , we have H = ϵ √ N h 1 ⊤ CN and W = h 1 ⊤ C , thus

Original: [Equation]
Revised: [Equation]

Original: W ,
Revised: W ,

Original: For ( H , W ) ∈
Revised: For ( H , W ) ∈

Original: E 3 , we have H ( I C ⊗ 1 N ) = 0 and W = 0 , thus
Revised: E 3 , we have H ( I C ⊗ 1 N ) = 0 and W = 0 , thus

Original: [Equation]
Revised: [Equation]

Original: i.e. , ( H , W ) is an eigenvector of B with eigenvalue 0 .
Revised: i.e. , ( H , W ) is an eigenvector of B with eigenvalue 0 .

Original: Overall, letting Π ϵi denote orthogonal projection onto E ϵi , we have the spectral decomposition
Revised: Overall, letting Π ϵi denote orthogonal projection onto E ϵi , we have the spectral decomposition

Original: [Equation]
Revised: [Equation]

Original: We then provide the concrete formulation of Z ( t ) =
Revised: We then provide the concrete formulation of Z ( t ) =

Original: Z 0 exp( L ( t )) by the orthogonal projection of
Revised: Z 0 exp( L ( t )) by the orthogonal projection of

Original: Z 0 onto each eigenspace of B ,
Revised: Z 0 onto each eigenspace of B ,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e. ,
Revised: e. ,

Original: [Equation]
Revised: [Equation]

Original: Decomposition along Π ϵ 1 Z 0 .
Revised: Decomposition along Π ϵ 1 Z 0 .

Original: First, Π
Revised: First, Π

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Π
Revised: Π

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: for k ≥ 0 , then
Revised: for k ≥ 0 , then

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: which is based on the facts that e x − e x=
Revised: which is based on the facts that e x − e x=

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +1)!
Revised: +1)!

Original: and e x + e
Revised: and e x + e

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: k (2 k )! .
Revised: k (2 k )! .

Original: Thus we have
Revised: Thus we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: 15) with
Revised: 15) with

Original: [Equation]
Revised: [Equation]

Original: Decomposition along Π ϵ 2 Z 0 .
Revised: Decomposition along Π ϵ 2 Z 0 .

Original: Similarly, for Π ϵ 2 Z 0 B = ϵ (1+
Revised: Similarly, for Π ϵ 2 Z 0 B = ϵ (1+

Original: [Equation]
Revised: [Equation]

Original: − γC
Revised: − γC

Original: [Equation]
Revised: [Equation]

Original: √ N Π ϵ 2 , we have Π
Revised: √ N Π ϵ 2 , we have Π

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Thus we have
Revised: Thus we have

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: Decomposition along Π 3 Z 0 .
Revised: Decomposition along Π 3 Z 0 .

Original: Since each vector in E 3 is a eigenvector of B with eigenvalue 0, then we have
Revised: Since each vector in E 3 is a eigenvector of B with eigenvalue 0, then we have

Original: [Equation]
Revised: [Equation]

Original: Note that E +1 , E − 1 , E +2 , E − 2 and E 3 are orthogonal subspace of R p × CN ⊕ R C × p , thus
Revised: Note that E +1 , E − 1 , E +2 , E − 2 and E 3 are orthogonal subspace of R p × CN ⊕ R C × p , thus

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: t ) I C ( N +1) (cid:1) (A.20)
Revised: t ) I C ( N +1) (cid:1) (A.20)

Original: Moreover, since b ′ ( t ) = − η 2 ( t ) γC − γ − 1 C 1 C , we obtain b ( t ) =
Revised: Moreover, since b ′ ( t ) = − η 2 ( t ) γC − γ − 1 C 1 C , we obtain b ( t ) =

Original: b (0) + (cid:90) t − η 2 ( τ ) γC − γ − 1 C d τ 1 C
Revised: b (0) + (cid:90) t − η 2 ( τ ) γC − γ − 1 C d τ 1 C

Original: =
Revised: =

Original: b
Revised: b

Original: + (1 + γ − γC ) ζ 2 ( t )
Revised: + (1 + γ − γC ) ζ 2 ( t )

Original: C 1 C , (A.21) with ζ 2 ( t ) = (cid:82) t 0 η 2 ( τ )d τ .
Revised: C 1 C , (A.21) with ζ 2 ( t ) = (cid:82) t 0 η 2 ( τ )d τ .

Original: A. P ROOF OF C OROLLARY 3. 
Revised: A. P ROOF OF C OROLLARY 3.

Original: Corollary 3.2 Under the conditions and notation of Theorem 3.1, let s =
Revised: Corollary 3.2 Under the conditions and notation of Theorem 3.1, let s =

Original: η 1 (0) η 2 (0) ,
Revised: η 1 (0) η 2 (0) ,

Original: if 0 < γ < 2 C − 2 (where C > 2 ) or C = 2 , and lim t →∞ ζ 1 ( t ) = ∞ , then the gradient flow (as in Eq.
Revised: if 0 < γ < 2 C − 2 (where C > 2 ) or C = 2 , and lim t →∞ ζ 1 ( t ) = ∞ , then the gradient flow (as in Eq.

Original: (3.4) ) will behave as:
Revised: (3.4) ) will behave as:

Original: [Equation]
Revised: [Equation]

Original: where ( H +1 , W +1 )
Revised: where ( H +1 , W +1 )

Original: =
Revised: =

Original: Π
Revised: Π

Original: +1 Z 0 , ( H − 1 , W − 1 ) =
Revised: +1 Z 0 , ( H − 1 , W − 1 ) =

Original: Π − 1 Z 0 , and the residual term ∆ ( t ) decreases at least as ∥ ∆ ( t ) ∥ =
Revised: Π − 1 Z 0 , and the residual term ∆ ( t ) decreases at least as ∥ ∆ ( t ) ∥ =

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: · max {− γC,
Revised: · max {− γC,

Original: [Equation]
Revised: [Equation]

Original: C − 2)
Revised: C − 2)

Original: [Equation]
Revised: [Equation]

Original: − 2 } (cid:19) , and so let Z = (cid:16) 1+ √ s 2 H + 1 + 1 − √ s 2 H − 1 , 1+ √ s 2 √ s W +1 − 1 − √ s 2 √ s W − 1 (cid:17) , the normalized Z ( t ) converges to the normalized Z in
Revised: − 2 } (cid:19) , and so let Z = (cid:16) 1+ √ s 2 H + 1 + 1 − √ s 2 H − 1 , 1+ √ s 2 √ s W +1 − 1 − √ s 2 √ s W − 1 (cid:17) , the normalized Z ( t ) converges to the normalized Z in

Original: [Equation]
Revised: [Equation]

Original: A.23) which further indicates lim t →∞ Z ( t )
Revised: A.23) which further indicates lim t →∞ Z ( t )

Original: ∥
Revised: ∥

Original: Z ( t ) ∥ ∈ E .
Revised: Z ( t ) ∥ ∈ E .

Original: Moreover, if γ ̸ = 1 C − 1 , then lim t →∞ max i b i ( t ) min i b i ( t ) = 1 .
Revised: Moreover, if γ ̸ = 1 C − 1 , then lim t →∞ max i b i ( t ) min i b i ( t ) = 1 .

Original: Proof.
Revised: Proof.

Original: Let ( H ϵ 1 , W ϵ 1 ) =
Revised: Let ( H ϵ 1 , W ϵ 1 ) =

Original: Π ϵ 1 Z 0 , ( H ϵ 2 , W ϵ 2 ) =
Revised: Π ϵ 1 Z 0 , ( H ϵ 2 , W ϵ 2 ) =

Original: Π ϵ 2 Z 

Revised: Π ϵ 2 Z

Original: and ( H 3 , W 3 )
Revised: and ( H 3 , W 3 )

Original: =
Revised: =

Original: Π 3 Z 0 for ϵ ∈ {± 1 } , according to Theorem 3.1, we have
Revised: Π 3 Z 0 for ϵ ∈ {± 1 } , according to Theorem 3.1, we have

Original: [Equation]
Revised: [Equation]

Original: Since η 1 ( t 1 ) η 2 ( t 2 ) =
Revised: Since η 1 ( t 1 ) η 2 ( t 2 ) =

Original: η 1 ( t 2 ) η 2 ( t 1 ) , then η 1 ( t ) =
Revised: η 1 ( t 2 ) η 2 ( t 1 ) , then η 1 ( t ) =

Original: η 1 (0) η 2 (0) η 2 ( t ) .
Revised: η 1 (0) η 2 (0) η 2 ( t ) .

Original: Let s =
Revised: Let s =

Original: η 1 (0) η 2 (0) , p ( t ) =
Revised: η 1 (0) η 2 (0) , p ( t ) =

Original: (1+ γ ) √ ζ 1 ( t ) ζ 2 ( t )
Revised: (1+ γ ) √ ζ 1 ( t ) ζ 2 ( t )

Original: C √ N , and q ( t ) =
Revised: C √ N , and q ( t ) =

Original: (1+ γ − γC ) √ ζ 1 ( t ) ζ 2 ( t )
Revised: (1+ γ − γC ) √ ζ 1 ( t ) ζ 2 ( t )

Original: C √ N , we have ζ 1 ( t ) =
Revised: C √ N , we have ζ 1 ( t ) =

Original: (cid:82) t 0 η 1 ( τ )d τ = s (cid:82) t 0 η 2 ( τ )d τ = sζ 2 ( t ) , and then α ϵ
Revised: (cid:82) t 0 η 1 ( τ )d τ = s (cid:82) t 0 η 2 ( τ )d τ = sζ 2 ( t ) , and then α ϵ

Original: [Equation]
Revised: [Equation]

Original: −√
Revised: −√

Original: [Equation]
Revised: [Equation]

Original: , α ϵ
Revised: , α ϵ

Original: [Equation]
Revised: [Equation]

Original: , α ϵ 2 (
Revised: , α ϵ 2 (

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: ϵ
Revised: ϵ

Original: [Equation]
Revised: [Equation]

Original: Since 0 < γ < 2 C − 2 ( where C > 2 ) or C = 2 , then we have p ( t ) − q ( t ) = γC
Revised: Since 0 < γ < 2 C − 2 ( where C > 2 ) or C = 2 , then we have p ( t ) − q ( t ) = γC

Original: [Equation]
Revised: [Equation]

Original: 1 ( t
Revised: 1 ( t

Original: [Equation]
Revised: [Equation]

Original: ζ
Revised: ζ

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: > 0 , p
Revised: > 0 , p

Original: [Equation]
Revised: [Equation]

Original: > 0 , and substitute these results into Equation (A.24) to obtain
Revised: > 0 , and substitute these results into Equation (A.24) to obtain

Original: [Equation]
Revised: [Equation]

Original: 25) where ∥ ∆ 1 ( t ) ∥
Revised: 25) where ∥ ∆ 1 ( t ) ∥

Original: =
Revised: =

Original: O ( e max { q ( t ) − p ( t ) , − q ( t ) − p ( t ) } and ∥ ∆ 2 ( t ) ∥
Revised: O ( e max { q ( t ) − p ( t ) , − q ( t ) − p ( t ) } and ∥ ∆ 2 ( t ) ∥

Original: =
Revised: =

Original: O ( e max { q ( t ) − p ( t ) , − q ( t ) − p ( t ) } .
Revised: O ( e max { q ( t ) − p ( t ) , − q ( t ) − p ( t ) } .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: where ∆ ( t ) =
Revised: where ∆ ( t ) =

Original: ( ∆ 1 ( t ) , ∆ 2 ( t )) and ∥ ∆ ( t ) ∥ ≤ ∥ ∆ 1 ( t ) ∥
Revised: ( ∆ 1 ( t ) , ∆ 2 ( t )) and ∥ ∆ ( t ) ∥ ≤ ∥ ∆ 1 ( t ) ∥

Original: + ∥ ∆ 2 ( t ) ∥
Revised: + ∥ ∆ 2 ( t ) ∥

Original: =
Revised: =

Original: O ( e max { q ( t ) − p ( t ) , − q ( t ) − p ( t ) } .
Revised: O ( e max { q ( t ) − p ( t ) , − q ( t ) − p ( t ) } .

Original: Let Z = (cid:16) 1+ √ s 2 H +

Revised: Let Z = (cid:16) 1+ √ s 2 H +

Original: + 1 −√ s 2 H − 1 , 1+ √ s 

Revised: + 1 −√ s 2 H − 1 , 1+ √ s

Original: √ s W +1 − 1 −√ s 2 √ s W − 1 (cid:17) , we have (cid:13) (cid:13)(cid:13) (cid:13)
Revised: √ s W +1 − 1 −√ s 2 √ s W − 1 (cid:17) , we have (cid:13) (cid:13)(cid:13) (cid:13)

Original: Z ( t ) ∥
Revised: Z ( t ) ∥

Original: Z ( t
Revised: Z ( t

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (cid:13) (cid:13)(cid:13) (cid:13) ≤
Revised: (cid:13) (cid:13)(cid:13) (cid:13) ≤

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ∥
Revised: ∥

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 ∥ ∆ ( t ) ∥
Revised: 2 ∥ ∆ ( t ) ∥

Original: ∥
Revised: ∥

Original: Z + ∆ ( t ) ∥ , (A.27) thus (cid:13)(cid:13)(cid:13) Z ( t ) ∥
Revised: Z + ∆ ( t ) ∥ , (A.27) thus (cid:13)(cid:13)(cid:13) Z ( t ) ∥

Original: Z ( t ) ∥ − Z ∥
Revised: Z ( t ) ∥ − Z ∥

Original: Z ∥ (cid:13)(cid:13)(cid:13) =
Revised: Z ∥ (cid:13)(cid:13)(cid:13) =

Original: O ( e max { q ( t ) − p ( t ) , − q ( t ) − p ( t ) } , and further lim t →∞ Z ( t ) ∥
Revised: O ( e max { q ( t ) − p ( t ) , − q ( t ) − p ( t ) } , and further lim t →∞ Z ( t ) ∥

Original: Z ( t ) ∥ =
Revised: Z ( t ) ∥ =

Original: Z ∥
Revised: Z ∥

Original: Z ∥ when lim t →∞ ζ 1 ( t ) = ∞ .
Revised: Z ∥ when lim t →∞ ζ 1 ( t ) = ∞ .

Original: According to the definition of E 1 and E 2 , we have
Revised: According to the definition of E 1 and E 2 , we have

Original: [Equation]
Revised: [Equation]

Original: thus we have Z ∈ E , then
Revised: thus we have Z ∈ E , then

Original: lim t →∞ Z ( t ) ∥
Revised: lim t →∞ Z ( t ) ∥

Original: Z ( t ) ∥
Revised: Z ( t ) ∥

Original: =
Revised: =

Original: Z ∥ Z ∥ ∈ E .
Revised: Z ∥ Z ∥ ∈ E .

Original: Moreover, we have b ( t ) =
Revised: Moreover, we have b ( t ) =

Original: b 

Revised: b

Original: + (1+ γ − γC ) ζ 2 ( t ) C 1 C , then ∀
Revised: + (1+ γ − γC ) ζ 2 ( t ) C 1 C , then ∀

Original: i, j , lim t →∞ b i ( t ) b j ( t ) =
Revised: i, j , lim t →∞ b i ( t ) b j ( t ) =

Original: lim t →∞ b i (0) + 1+ γ − γC C ζ 2 ( t ) b j (0) + 1+ γ − γC C ζ 2 ( t ) = 1 , thus lim t →∞ max i b i ( t ) max i b i ( t ) =1.
Revised: lim t →∞ b i (0) + 1+ γ − γC C ζ 2 ( t ) b j (0) + 1+ γ − γC C ζ 2 ( t ) = 1 , thus lim t →∞ max i b i ( t ) max i b i ( t ) =1.

Original: A. P ROOF OF T HEOREM 3. 
Revised: A. P ROOF OF T HEOREM 3.

Original: Theorem 3.3 (Dynamics of Features and Prototypes with ℓ 2 Regularization)
Revised: Theorem 3.3 (Dynamics of Features and Prototypes with ℓ 2 Regularization)

Original: Under the conditions and notations of Theorem 3.1, consider the continual gradient flow (Equation 3.11) in which the dynamics follow the gradient descent direction of Sample Margin loss in Eq.
Revised: Under the conditions and notations of Theorem 3.1, consider the continual gradient flow (Equation 3.11) in which the dynamics follow the gradient descent direction of Sample Margin loss in Eq.

Original: Let
Revised: Let

Original: Z ( t ) =
Revised: Z ( t ) =

Original: ( H ( t ) , W ( t )) , if η 1 ( t 1 ) η 2 ( t 2 ) =
Revised: ( H ( t ) , W ( t )) , if η 1 ( t 1 ) η 2 ( t 2 ) =

Original: η 1 ( t 2 ) η 2 ( t 1 ) for any t 1 , t 2 ≥ 0 , we have the following closed-form dynamics
Revised: η 1 ( t 2 ) η 2 ( t 1 ) for any t 1 , t 2 ≥ 0 , we have the following closed-form dynamics

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: and b ( t ) =
Revised: and b ( t ) =

Original: ϕ 3 ( t
Revised: ϕ 3 ( t

Original: [Equation]
Revised: [Equation]

Original: (cid:16) b
Revised: (cid:16) b

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where Z ( t ) is defined in Eq.
Revised: where Z ( t ) is defined in Eq.

Original: (3.5) , ϕ i ( t ) =
Revised: (3.5) , ϕ i ( t ) =

Original: e − (cid:82) t 0 λ i ( r )d r for i ∈ { 1 , 2 , 3 } , and ψ
Revised: e − (cid:82) t 0 λ i ( r )d r for i ∈ { 1 , 2 , 3 } , and ψ

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: According to the gradient flow in Eq.
Revised: According to the gradient flow in Eq.

Original: (3.11), and the notations in the proof of Theorem 3.1, we have:
Revised: (3.11), and the notations in the proof of Theorem 3.1, we have:

Original: [Equation]
Revised: [Equation]

Original: i.e. , Z ′ ( t ) =
Revised: i.e. , Z ′ ( t ) =

Original: Z ( t ) A ( t ) , where A ( t )
Revised: Z ( t ) A ( t ) , where A ( t )

Original: =
Revised: =

Original: A ( t ) − λ Λ ( t ) , and Λ ( t ) = (cid:16) η 1 ( t ) I CN 0η 2 ( t ) I C (cid:17)
Revised: A ( t ) − λ Λ ( t ) , and Λ ( t ) = (cid:16) η 1 ( t ) I CN 0η 2 ( t ) I C (cid:17)

Original: [Equation]
Revised: [Equation]

Original: For any t 1 , t 2 , we have the matrix commutator of A ( t 1 ) and A ( t 2 )
Revised: For any t 1 , t 2 , we have the matrix commutator of A ( t 1 ) and A ( t 2 )

Original: [ A ( t 1 ) , A ( t 2 )]
Revised: [ A ( t 1 ) , A ( t 2 )]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (A
Revised: (A

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the last equality is based on the fact that η 2 ( t 1 ) η 1 ( t 2 ) =
Revised: where the last equality is based on the fact that η 2 ( t 1 ) η 1 ( t 2 ) =

Original: η 2 ( t 2 ) η 1 ( t 1 ) .
Revised: η 2 ( t 2 ) η 1 ( t 1 ) .

Original: Therefore, according to Magnus approach, we have
Revised: Therefore, according to Magnus approach, we have

Original: Z ( t ) =
Revised: Z ( t ) =

Original: Z 0 exp (cid:18) (cid:90) t
Revised: Z 0 exp (cid:18) (cid:90) t

Original: [Equation]
Revised: [Equation]

Original: ζ
Revised: ζ

Original: [Equation]
Revised: [Equation]

Original: − λζ
Revised: − λζ

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: (cid:19) , (A.33) where ζ 1 ( t ) =
Revised: (cid:19) , (A.33) where ζ 1 ( t ) =

Original: (cid:82) t 0 η ( τ )d τ and ζ 2 ( t )
Revised: (cid:82) t 0 η ( τ )d τ and ζ 2 ( t )

Original: = (cid:82) t 0 η ( τ )d τ .
Revised: = (cid:82) t 0 η ( τ )d τ .

Original: Let B =
Revised: Let B =

Original: B − λ I C ( N +1) , we have
Revised: B − λ I C ( N +1) , we have

Original: [Equation]
Revised: [Equation]

Original: We again consider the orthogonal decomposition of Z 0 , i.e. , Z 0 =
Revised: We again consider the orthogonal decomposition of Z 0 , i.e. , Z 0 =

Original: (Π +1 +Π − 1 +Π +

Revised: (Π +1 +Π − 1 +Π +

Original: +Π − 2 +Π 3 )
Revised: +Π − 2 +Π 3 )

Original: Z 0 .
Revised: Z 0 .

Original: As mentioned in the proof of Theorem 3.1, we have
Revised: As mentioned in the proof of Theorem 3.1, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, for any D = ( H , W ) ∈
Revised: Therefore, for any D = ( H , W ) ∈

Original: { Π ϵ 1 Z 0 , Π ϵ 2 Z 0 , Π 3 Z 0 } (where H ∈ R p × CN and W ∈ R p × C ) and the corresponding eigenvalue σ ∈ { ϵ (1+ γ ) C √ N , ϵ (1+ γ − γC ) C √ N , 0 } , we have
Revised: { Π ϵ 1 Z 0 , Π ϵ 2 Z 0 , Π 3 Z 0 } (where H ∈ R p × CN and W ∈ R p × C ) and the corresponding eigenvalue σ ∈ { ϵ (1+ γ ) C √ N , ϵ (1+ γ − γC ) C √ N , 0 } , we have

Original: DB = σ D , HM ⊤ =
Revised: DB = σ D , HM ⊤ =

Original: σ W , and W M = σ H .
Revised: σ W , and W M = σ H .

Original: (A.36)
Revised: (A.36)

Original: In the following, we will prove that there exist two scalars a ( t ) and b ( t ) , such that D exp( BC ( t )) =
Revised: In the following, we will prove that there exist two scalars a ( t ) and b ( t ) , such that D exp( BC ( t )) =

Original: [Equation]
Revised: [Equation]

Original: b
Revised: b

Original: [Equation]
Revised: [Equation]

Original: ) I C (cid:19) .
Revised: ) I C (cid:19) .

Original: First, we prove that D ( BC ( t ))
Revised: First, we prove that D ( BC ( t ))

Original: k can be represented as D ( BC ( t ))
Revised: k can be represented as D ( BC ( t ))

Original: k
Revised: k

Original: = ( a k ( t ) H , b k ( t ) W ) by induction, where a k ( t ) , b k ( t ) ∈ R .
Revised: = ( a k ( t ) H , b k ( t ) W ) by induction, where a k ( t ) , b k ( t ) ∈ R .

Original: For k = 0 , we have D ( BC ( t )) 

Revised: For k = 0 , we have D ( BC ( t ))

Original: = ( H , W ) , i.e. , a 0 = b 0 = 1 .
Revised: = ( H , W ) , i.e. , a 0 = b 0 = 1 .

Original: Assume that D ( BC ( t )) n
Revised: Assume that D ( BC ( t )) n

Original: = ( a n ( t ) H , b n ( t ) W ) for k = n .
Revised: = ( a n ( t ) H , b n ( t ) W ) for k = n .

Original: Then for k = n + 1 , we have
Revised: Then for k = n + 1 , we have

Original: D ( BC ( t )) n + = D ( BC ( t )) n ( BC ( t )) =( a n ( t ) H , b n ( t ) W )( BC ( t ))
Revised: D ( BC ( t )) n + = D ( BC ( t )) n ( BC ( t )) =( a n ( t ) H , b n ( t ) W )( BC ( t ))

Original: [Equation]
Revised: [Equation]

Original: ) I C (cid:19) =( ζ 1 ( t )( σb n ( t ) − λa n ( t ))
Revised: ) I C (cid:19) =( ζ 1 ( t )( σb n ( t ) − λa n ( t ))

Original: H , ζ 2 ( t )( σa n ( t ) − λb n ( t )) H ) , (A.37) thus a n +1 ( t ) =
Revised: H , ζ 2 ( t )( σa n ( t ) − λb n ( t )) H ) , (A.37) thus a n +1 ( t ) =

Original: ζ 1 ( t )( σb n ( t ) − λa n ( t )) and b n +1 ( t ) =
Revised: ζ 1 ( t )( σb n ( t ) − λa n ( t )) and b n +1 ( t ) =

Original: ζ 2 ( t )( σa n ( t ) − λb n ( t )) .
Revised: ζ 2 ( t )( σa n ( t ) − λb n ( t )) .

Original: To sum up, we have shown by induction that D ( BC ( t ))
Revised: To sum up, we have shown by induction that D ( BC ( t ))

Original: k can be represented as D ( BC ( t ))
Revised: k can be represented as D ( BC ( t ))

Original: k
Revised: k

Original: = ( a k ( t ) H , b k ( t ) W ) =
Revised: = ( a k ( t ) H , b k ( t ) W ) =

Original: D (cid:18) a k ( t ) I CN  b
Revised: D (cid:18) a k ( t ) I CN  b

Original: k
Revised: k

Original: ( t ) I C
Revised: ( t ) I C

Original: (cid:19) , and (cid:18)
Revised: (cid:19) , and (cid:18)

Original: a k ( t ) b k ( t ) (cid:19) satisfies (cid:18) a k ( t ) b k ( t ) (cid:19) =
Revised: a k ( t ) b k ( t ) (cid:19) satisfies (cid:18) a k ( t ) b k ( t ) (cid:19) =

Original: (cid:18) − λζ 1 ( t ) σζ 1 ( t ) σζ 2 ( t ) − λζ 2 ( t ) (cid:19) (cid:18) a k − 1 ( t ) b
Revised: (cid:18) − λζ 1 ( t ) σζ 1 ( t ) σζ 2 ( t ) − λζ 2 ( t ) (cid:19) (cid:18) a k − 1 ( t ) b

Original: k − 1 ( t ) (cid:19) with (cid:18) ab(cid:19) =
Revised: k − 1 ( t ) (cid:19) with (cid:18) ab(cid:19) =

Original: (cid:18) 11 (cid:19) , (A.38) i.e. , (cid:18) a k ( t ) b k ( t ) (cid:19) =
Revised: (cid:18) 11 (cid:19) , (A.38) i.e. , (cid:18) a k ( t ) b k ( t ) (cid:19) =

Original: ( S ( σ, λ, ζ 1 ( t ) , ζ 2 ( t )))
Revised: ( S ( σ, λ, ζ 1 ( t ) , ζ 2 ( t )))

Original: k (cid:18) 11 (cid:19) , where S ( σ, λ, ζ 1 , ζ 2 ) =
Revised: k (cid:18) 11 (cid:19) , where S ( σ, λ, ζ 1 , ζ 2 ) =

Original: (cid:18) − λζσζ 1 σζ− λζ(cid:19) .
Revised: (cid:18) − λζσζ 1 σζ− λζ(cid:19) .

Original: Therefore, we have
Revised: Therefore, we have

Original: D exp( BC ( t ))
Revised: D exp( BC ( t ))

Original: =
Revised: =

Original: D ∞ (cid:88) k
Revised: D ∞ (cid:88) k

Original: = ( BC ( t ))
Revised: = ( BC ( t ))

Original: k k !
Revised: k k !

Original: [Equation]
Revised: [Equation]

Original: D
Revised: D

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: (cid:19) , (A
Revised: (cid:19) , (A

Original: [Equation]
Revised: [Equation]

Original: 39) with a ( t ) =
Revised: 39) with a ( t ) =

Original: (cid:80) ∞ k =

Revised: (cid:80) ∞ k =

Original: a k ( t ) k !
Revised: a k ( t ) k !

Original: and b ( t )
Revised: and b ( t )

Original: =
Revised: =

Original: (cid:80) ∞ k =

Revised: (cid:80) ∞ k =

Original: b k ( t ) k ! , i.e. , (cid:18) a ( t ) b ( t ) (cid:19) =
Revised: b k ( t ) k ! , i.e. , (cid:18) a ( t ) b ( t ) (cid:19) =

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Next, we are going to derive a concrete expression of exp( S ( σ, λ, ζ 1 ( t ) , ζ 2 ( t ))) .
Revised: Next, we are going to derive a concrete expression of exp( S ( σ, λ, ζ 1 ( t ) , ζ 2 ( t ))) .

Original: Let the determinant | S ( σ, λ, ζ 1 ( t ) , ζ 2 ( t ) − θ I | = 0 , we can derive that the eigenvalues of
Revised: Let the determinant | S ( σ, λ, ζ 1 ( t ) , ζ 2 ( t ) − θ I | = 0 , we can derive that the eigenvalues of

Original: S ( σ, λ, ζ 1 ( t ) , ζ 2 ( t ) are θ 1 = − λ
Revised: S ( σ, λ, ζ 1 ( t ) , ζ 2 ( t ) are θ 1 = − λ

Original: ( ζ 1 ( t )+ ζ 2 ( t ))
Revised: ( ζ 1 ( t )+ ζ 2 ( t ))

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: < 0 and θ 2 =
Revised: < 0 and θ 2 =

Original: [Equation]
Revised: [Equation]

Original: , and the corresponding eigenvectors are
Revised: , and the corresponding eigenvectors are

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: , and v 2 = (cid:18) 1 λζ
Revised: , and v 2 = (cid:18) 1 λζ

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: + θ 2 σζ 1 ( t
Revised: + θ 2 σζ 1 ( t

Original: [Equation]
Revised: [Equation]

Original: (cid:19) .
Revised: (cid:19) .

Original: Let P =
Revised: Let P =

Original: ( v 1 , v 2 )
Revised: ( v 1 , v 2 )

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: and exp
Revised: and exp

Original: [Equation]
Revised: [Equation]

Original: thus
Revised: thus

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: since E +1 , E − 1 , E +2 , E − 2 and E 3 are orthogonal subspace of R p × CN ⊕ R C × p , we obtain that
Revised: since E +1 , E − 1 , E +2 , E − 2 and E 3 are orthogonal subspace of R p × CN ⊕ R C × p , we obtain that

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where a ϵ 1 ( t ) , b ϵ 1 ( t ) , a ϵ 2 ( t ) ,
Revised: where a ϵ 1 ( t ) , b ϵ 1 ( t ) , a ϵ 2 ( t ) ,

Original: b ϵ 2 ( t ) , a 3 ( t ) and b 3 ( t ) satisfy
Revised: b ϵ 2 ( t ) , a 3 ( t ) and b 3 ( t ) satisfy

Original: [Equation]
Revised: [Equation]

Original: ϵ
Revised: ϵ

Original: [Equation]
Revised: [Equation]

Original: ) b
Revised: ) b

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: t ) b
Revised: t ) b

Original: ϵ
Revised: ϵ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) (cid:19)
Revised: ) (cid:19)

Original: =
Revised: =

Original: exp (cid:16)
Revised: exp (cid:16)

Original: [Equation]
Revised: [Equation]

Original: (cid:16)
Revised: (cid:16)

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: ( t )
Revised: ( t )

Original: [Equation]
Revised: [Equation]

Original: ( t ) (cid:17)(cid:17) (cid:18) 11 (cid:19)
Revised: ( t ) (cid:17)(cid:17) (cid:18) 11 (cid:19)

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: ( t ) b
Revised: ( t ) b

Original: [Equation]
Revised: [Equation]

Original: ( t ) (cid:19) =
Revised: ( t ) (cid:19) =

Original: exp ( S (
Revised: exp ( S (

Original: [Equation]
Revised: [Equation]

Original: ζ
Revised: ζ

Original: [Equation]
Revised: [Equation]

Original: ( t
Revised: ( t

Original: [Equation]
Revised: [Equation]

Original: Moreover, since b ′ ( t ) =
Revised: Moreover, since b ′ ( t ) =

Original: η 2 ( t ) 1+ γ − γC C 1 C − λη
Revised: η 2 ( t ) 1+ γ − γC C 1 C − λη

Original: [Equation]
Revised: [Equation]

Original: t ) b ( t ) is a first-order linear differential equation, then we have
Revised: t ) b ( t ) is a first-order linear differential equation, then we have

Original: [Equation]
Revised: [Equation]

Original: where ϕ ( t ) =
Revised: where ϕ ( t ) =

Original: exp( − λ (cid:82) t 0 η 2 ( τ )d τ ) and ψ ( t ) =
Revised: exp( − λ (cid:82) t 0 η 2 ( τ )d τ ) and ψ ( t ) =

Original: (cid:82) t 0 ζ 2 ( τ ) exp( λ (cid:82) τ 0 η 2 ( s )d s )
Revised: (cid:82) t 0 ζ 2 ( τ ) exp( λ (cid:82) τ 0 η 2 ( s )d s )

Original: d τ .
Revised: d τ .

Original: A. P ROOF OF C OROLLARY 3. 
Revised: A. P ROOF OF C OROLLARY 3.

Original: Corollary 3.4.
Revised: Corollary 3.4.

Original: Under the conditions and notation of Theorem 3.3, let s =
Revised: Under the conditions and notation of Theorem 3.3, let s =

Original: η 1 (0) η 2 (0) ,
Revised: η 1 (0) η 2 (0) ,

Original: if 0 < γ < 2 C −(where C > 2 ) or C = 2 , and lim t →∞ ζ 1 ( t ) = ∞ , then there exist constants π + h , π − h , π + w , π − w and ω only depending on λ , γ , s , C and N , such that the gradient flow behaves as:
Revised: if 0 < γ < 2 C −(where C > 2 ) or C = 2 , and lim t →∞ ζ 1 ( t ) = ∞ , then there exist constants π + h , π − h , π + w , π − w and ω only depending on λ , γ , s , C and N , such that the gradient flow behaves as:

Original: [Equation]
Revised: [Equation]

Original: (A.46) where ( H +1 , W +1 ) =
Revised: (A.46) where ( H +1 , W +1 ) =

Original: Π
Revised: Π

Original: +1 Z 0 , ( H − 1 , W − 1 ) =
Revised: +1 Z 0 , ( H − 1 , W − 1 ) =

Original: Π − 1 Z 0 .
Revised: Π − 1 Z 0 .

Original: Furthermore, we have the following results:
Revised: Furthermore, we have the following results:

Original: • If λ > 1+ γ C √ N , then lim t →∞ ∥ Z ( t ) ∥ = 0 ; • If λ = 1+ γ C √ N , then lim t →∞ H ( t ) =
Revised: • If λ > 1+ γ C √ N , then lim t →∞ ∥ Z ( t ) ∥ = 0 ; • If λ = 1+ γ C √ N , then lim t →∞ H ( t ) =

Original: H +1 + 1 − s 

Revised: H +1 + 1 − s

Original: + s H − 1 , lim t →∞ W ( t )
Revised: + s H − 1 , lim t →∞ W ( t )

Original: =
Revised: =

Original: W +1 − 1 − s 

Revised: W +1 − 1 − s

Original: + s W − 1 ; • If λ < 1+ γ C √ N , then lim t →∞ ∥ Z ( t ) ∥ = ∞ .
Revised: + s W − 1 ; • If λ < 1+ γ C √ N , then lim t →∞ ∥ Z ( t ) ∥ = ∞ .

Original: Proof.
Revised: Proof.

Original: Since ζ 1 ( t ) = sζ 2 ( t )
Revised: Since ζ 1 ( t ) = sζ 2 ( t )

Original: [Equation]
Revised: [Equation]

Original: then the eigenvalues of S ( σ, λ, ζ 1 ( t ) , ζ 2 ( t )) are θ
Revised: then the eigenvalues of S ( σ, λ, ζ 1 ( t ) , ζ 2 ( t )) are θ

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: Let ω 1 ( σ, λ, s )
Revised: Let ω 1 ( σ, λ, s )

Original: =
Revised: =

Original: − λ
Revised: − λ

Original: ( s +
Revised: ( s +

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: For brevity
Revised: For brevity

Original: [Equation]
Revised: [Equation]

Original: let ω 1 and ω 2 denote ω 1 ( σ, λ, s ) and ω 2 ( σ, λ, s ) , respectively
Revised: let ω 1 and ω 2 denote ω 1 ( σ, λ, s ) and ω 2 ( σ, λ, s ) , respectively

Original: [Equation]
Revised: [Equation]

Original: and then we can reformulate a ( t ) and b ( t ) as
Revised: and then we can reformulate a ( t ) and b ( t ) as

Original: [Equation]
Revised: [Equation]

Original: Moreover, according to Theorem 3.
Revised: Moreover, according to Theorem 3.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: (A
Revised: (A

Original: [Equation]
Revised: [Equation]

Original: Since 0 < γ < 2 C − 2 (where C > 2 ) or C = 2 , thus we have 1+ γ C √ N > | 1+ γ − γC | C √ N , and then ω 2 (cid:16) 1+ γ C √ N , λ, s (cid:17) > ω 2 (cid:16) | 1+ γ − γC | C √ N , λ, s (cid:17) .
Revised: Since 0 < γ < 2 C − 2 (where C > 2 ) or C = 2 , thus we have 1+ γ C √ N > | 1+ γ − γC | C √ N , and then ω 2 (cid:16) 1+ γ C √ N , λ, s (cid:17) > ω 2 (cid:16) | 1+ γ − γC | C √ N , λ, s (cid:17) .

Original: When t → ∞ , the dominant terms in H ( t ) and W ( t ) are the ones whose coefficient contains exp (cid:16) ω
Revised: When t → ∞ , the dominant terms in H ( t ) and W ( t ) are the ones whose coefficient contains exp (cid:16) ω

Original: 2 (cid:16) 1+ γ C √ N , λ, s (cid:17) ζ 2 ( t ) (cid:17) , i.e. , a +1 ( t ) , a − 1 ( t ) , b +1 ( t ) and b − 1 ( t ) .
Revised: 2 (cid:16) 1+ γ C √ N , λ, s (cid:17) ζ 2 ( t ) (cid:17) , i.e. , a +1 ( t ) , a − 1 ( t ) , b +1 ( t ) and b − 1 ( t ) .

Original: Let ( H ϵ 1 , W ϵ 1 ) =
Revised: Let ( H ϵ 1 , W ϵ 1 ) =

Original: Π
Revised: Π

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: H
Revised: H

Original: [Equation]
Revised: [Equation]

Original: , W
Revised: , W

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: Π
Revised: Π

Original: [Equation]
Revised: [Equation]

Original: Z 

Revised: Z

Original: and ( H 3 , W 3 )
Revised: and ( H 3 , W 3 )

Original: =
Revised: =

Original: Π 3 Z 0 for ϵ ∈ {± 1 } , thus we have
Revised: Π 3 Z 0 for ϵ ∈ {± 1 } , thus we have

Original: [Equation]
Revised: [Equation]

Original: where ∆ 1 and ∆ 2 decrease to zero as least as O
Revised: where ∆ 1 and ∆ 2 decrease to zero as least as O

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: and the rate of convergence is O
Revised: and the rate of convergence is O

Original: [Equation]
Revised: [Equation]

Original: Moreover
Revised: Moreover

Original: [Equation]
Revised: [Equation]

Original: we have the following conclusions:
Revised: we have the following conclusions:

Original: • If λ = 1+ γ C √ N , we have ω 2 = 0 , ω 1 = − λ
Revised: • If λ = 1+ γ C √ N , we have ω 2 = 0 , ω 1 = − λ

Original: ( s + 1) , π
Revised: ( s + 1) , π

Original: + h = 1 , π − h = 1 − s 

Revised: + h = 1 , π − h = 1 − s

Original: + s , π + w = 1 and π − w = − 1 − s 1+ s .
Revised: + s , π + w = 1 and π − w = − 1 − s 1+ s .

Original: Since lim t →∞ ζ 2 ( t ) = ∞ , we have
Revised: Since lim t →∞ ζ 2 ( t ) = ∞ , we have

Original: [Equation]
Revised: [Equation]

Original: • If λ > 1+ γ C √ N , we have ω 2 > 0 , and then lim t →∞ ∥ Z ( t ) ∥
Revised: • If λ > 1+ γ C √ N , we have ω 2 > 0 , and then lim t →∞ ∥ Z ( t ) ∥

Original: = 0 since lim t →∞ ζ 2 ( t ) = ∞ .
Revised: = 0 since lim t →∞ ζ 2 ( t ) = ∞ .

Original: • If λ < 1+ γ C √ N , we have ω 2 < 0 , and
Revised: • If λ < 1+ γ C √ N , we have ω 2 < 0 , and

Original: then lim t
Revised: then lim t

Original: →∞ ∥ Z ( t ) ∥ = ∞ since lim t →∞ ζ 2 ( t )
Revised: →∞ ∥ Z ( t ) ∥ = ∞ since lim t →∞ ζ 2 ( t )

Original: = ∞ .
Revised: = ∞ .

Original: So far the proof has been completed.
Revised: So far the proof has been completed.

Original: A. P ROOF OF T HEOREM 3. 
Revised: A. P ROOF OF T HEOREM 3.

Original: Lemma A.2.
Revised: Lemma A.2.

Original: For h ( t ) , w ∈ R p , η ( t ) > 0 , let (cid:98) v = (cid:98) v ∥ v ∥ 2 denote the ℓ 2 -normalized vector of v , considering the discrete dynamical system h
Revised: For h ( t ) , w ∈ R p , η ( t ) > 0 , let (cid:98) v = (cid:98) v ∥ v ∥ 2 denote the ℓ 2 -normalized vector of v , considering the discrete dynamical system h

Original: ( t + 1) =
Revised: ( t + 1) =

Original: h ( t ) + η ( t ) ∥
Revised: h ( t ) + η ( t ) ∥

Original: h ( t ) ∥(cid:16) I p − (cid:98) h ( t ) (cid:98) h ⊤ ( t ) (cid:17) w , if (cid:98) w ⊤ (cid:98) h (0) > − 1 , the learning rate η ( t ) satisfies that lim t →∞ η ( t +1)
Revised: h ( t ) ∥(cid:16) I p − (cid:98) h ( t ) (cid:98) h ⊤ ( t ) (cid:17) w , if (cid:98) w ⊤ (cid:98) h (0) > − 1 , the learning rate η ( t ) satisfies that lim t →∞ η ( t +1)

Original: η ( t )
Revised: η ( t )

Original: = 1 , η ( t )
Revised: = 1 , η ( t )

Original: ∥ h ( t ) ∥ 2 is non-increasing with η (0)
Revised: ∥ h ( t ) ∥ 2 is non-increasing with η (0)

Original: ∥ h (0) ∥ 2 < ∥ w ∥ 2 , and there exists a constant ε > 0 , s.t., η ( t ) > ε , s.t., η ( t ) >
Revised: ∥ h (0) ∥ 2 < ∥ w ∥ 2 , and there exists a constant ε > 0 , s.t., η ( t ) > ε , s.t., η ( t ) >

Original: ε , then we have
Revised: ε , then we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For brevity, let α t = η ( t )
Revised: For brevity, let α t = η ( t )

Original: ∥
Revised: ∥

Original: w ∥∥ h ( t ) ∥ 22 , ξ t = η
Revised: w ∥∥ h ( t ) ∥ 22 , ξ t = η

Original: ( t +1)
Revised: ( t +1)

Original: η ( t ) , and β t = (cid:98) w ⊤ (cid:98) h ( t ) denote the cosine similarity between (cid:98) w and (cid:98) h ( t ) , then we can easily derive that α t > 0 and β t > − 1 for all t ≥ 0 .
Revised: η ( t ) , and β t = (cid:98) w ⊤ (cid:98) h ( t ) denote the cosine similarity between (cid:98) w and (cid:98) h ( t ) , then we can easily derive that α t > 0 and β t > − 1 for all t ≥ 0 .

Original: We will show that α t is monotonically decreasing and β t is monotonically increasing .
Revised: We will show that α t is monotonically decreasing and β t is monotonically increasing .

Original: Note that h ( t ) is orthogonal with (cid:16)
Revised: Note that h ( t ) is orthogonal with (cid:16)

Original: I p − (cid:98) h ( t ) (cid:98) h ⊤ ( t ) (cid:17) w , thus we have
Revised: I p − (cid:98) h ( t ) (cid:98) h ⊤ ( t ) (cid:17) w , thus we have

Original: [Equation]
Revised: [Equation]

Original: , which indicates ∥ h ( t ) ∥ 2 is monotonically increasing as a function of t , and then α t is monotonically decreasing since η ( t ) ∥
Revised: , which indicates ∥ h ( t ) ∥ 2 is monotonically increasing as a function of t , and then α t is monotonically decreasing since η ( t ) ∥

Original: w ∥∥ h ( t ) ∥ 2 is non-increasing.
Revised: w ∥∥ h ( t ) ∥ 2 is non-increasing.

Original: Moreover, we can rearrange the discrete dynamics and formulate h ( t + 1) as a positive combination of h ( t ) and w :
Revised: Moreover, we can rearrange the discrete dynamics and formulate h ( t + 1) as a positive combination of h ( t ) and w :

Original: [Equation]
Revised: [Equation]

Original: so (cid:98) w ⊤ (cid:98) h ( t + 1) ≥ (cid:98) w ⊤ (cid:98) h ( t ) , i.e. , β t is monotonically increasing , which is based on the facts that 1 − η ( t ) w ⊤ (cid:98) h ( t )
Revised: so (cid:98) w ⊤ (cid:98) h ( t + 1) ≥ (cid:98) w ⊤ (cid:98) h ( t ) , i.e. , β t is monotonically increasing , which is based on the facts that 1 − η ( t ) w ⊤ (cid:98) h ( t )

Original: ∥ h ( t ) ∥≥ 1 − η ( t ) ∥
Revised: ∥ h ( t ) ∥≥ 1 − η ( t ) ∥

Original: w ∥∥ h ( t ) ∥ 2 ≥ 1 − η (0) ∥
Revised: w ∥∥ h ( t ) ∥ 2 ≥ 1 − η (0) ∥

Original: w ∥ 2 ∥ h (0) ∥ 2 > 0 , η ( t ) ∥
Revised: w ∥ 2 ∥ h (0) ∥ 2 > 0 , η ( t ) ∥

Original: h ( t ) ∥ 2 > 0 , and y ⊤ ( x + k y )
Revised: h ( t ) ∥ 2 > 0 , and y ⊤ ( x + k y )

Original: ∥ x + k y ∥ 2 ≥ y ⊤ x ∥ x ∥ 2 holds for all k > 0 and x , y ̸
Revised: ∥ x + k y ∥ 2 ≥ y ⊤ x ∥ x ∥ 2 holds for all k > 0 and x , y ̸

Original: = 0 .
Revised: = 0 .

Original: We can formulate the discrete iterations of α t and β t from the Eq.
Revised: We can formulate the discrete iterations of α t and β t from the Eq.

Original: (A.55) as follows:
Revised: (A.55) as follows:

Original: [Equation]
Revised: [Equation]

Original: with β 0 =
Revised: with β 0 =

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: ∥ w ∥ 2 ∥ h (0) ∥ 22 > 0 , ξ t ≤ 1 and lim t →∞ ξ t = 1 .
Revised: ∥ w ∥ 2 ∥ h (0) ∥ 22 > 0 , ξ t ≤ 1 and lim t →∞ ξ t = 1 .

Original: To prove lim t →∞ (cid:13)(cid:13)(cid:13) h ( t )
Revised: To prove lim t →∞ (cid:13)(cid:13)(cid:13) h ( t )

Original: ∥ h ( t ) ∥ 2 − w ∥ w ∥ 2 (cid:13)(cid:13)(cid:13) = 0 , we just need prove lim t →∞ β t = 1 .
Revised: ∥ h ( t ) ∥ 2 − w ∥ w ∥ 2 (cid:13)(cid:13)(cid:13) = 0 , we just need prove lim t →∞ β t = 1 .

Original: Note that α t is monotonic decreasing and lower bounded by 0 , then the sequence ( α t ) is convergent.
Revised: Note that α t is monotonic decreasing and lower bounded by 0 , then the sequence ( α t ) is convergent.

Original: Similarly, the sequence ( β t ) is convergent.
Revised: Similarly, the sequence ( β t ) is convergent.

Original: Let a = lim t → α t and b = lim t → β t , we obtain
Revised: Let a = lim t → α t and b = lim t → β t , we obtain

Original: [Equation]
Revised: [Equation]

Original: thus a = 0 or lim t →∞ α 2 t (1 − β 2 t )
Revised: thus a = 0 or lim t →∞ α 2 t (1 − β 2 t )

Original: = 0 , i.e. , a = 0 or b = 1 .
Revised: = 0 , i.e. , a = 0 or b = 1 .

Original: Therefore, the limits of α t and β t exist if and only if lim t →∞ α t = 0 or lim t →∞ β t = 1 .
Revised: Therefore, the limits of α t and β t exist if and only if lim t →∞ α t = 0 or lim t →∞ β t = 1 .

Original: In the following, we will prove that the limit of β t must be equal to 1.
Revised: In the following, we will prove that the limit of β t must be equal to 1.

Original: Firstly, we prove a simpler result when β 0 > 0 :
Revised: Firstly, we prove a simpler result when β 0 > 0 :

Original: Lemma A.3.
Revised: Lemma A.3.

Original: For the discrete dynamical system in Eq.
Revised: For the discrete dynamical system in Eq.

Original: (A.56) , if β 0 ≥ 0 , then lim t →∞ β t = 1 .
Revised: (A.56) , if β 0 ≥ 0 , then lim t →∞ β t = 1 .

Original: Proof.
Revised: Proof.

Original: As aforementioned, due to the existence of the limit of α t , we have lim t →∞ α t = 0 or lim t →∞ β t
Revised: As aforementioned, due to the existence of the limit of α t , we have lim t →∞ α t = 0 or lim t →∞ β t

Original: = 1 .
Revised: = 1 .

Original: Thus, we just need to prove that lim t →∞ β t = 1 as lim t →∞ α t = 0 .
Revised: Thus, we just need to prove that lim t →∞ β t = 1 as lim t →∞ α t = 0 .

Original: When lim t →∞ α t = 0 , then there exists τ , such that ∀ t > τ , α t ≤ 1 .
Revised: When lim t →∞ α t = 0 , then there exists τ , such that ∀ t > τ , α t ≤ 1 .

Original: According to the iterations in Eq.
Revised: According to the iterations in Eq.

Original: (A.56), we can derive that
Revised: (A.56), we can derive that

Original: [Equation]
Revised: [Equation]

Original: then ∀ t > τ ,
Revised: then ∀ t > τ ,

Original: [Equation]
Revised: [Equation]

Original: where the inequality is based on the fact that 1 − α i β i ∈ (0 , 1] when 0 ≤ β 0 ≤ β i ≤ 1 , η ( t +1) ≥ ε , and α i ≤ 1 for i > τ .
Revised: where the inequality is based on the fact that 1 − α i β i ∈ (0 , 1] when 0 ≤ β 0 ≤ β i ≤ 1 , η ( t +1) ≥ ε , and α i ≤ 1 for i > τ .

Original: Since η (0)(1 − β 20 ) εα· (cid:81)
Revised: Since η (0)(1 − β 20 ) εα· (cid:81)

Original: [Equation]
Revised: [Equation]

Original: 2 is a constant, we obtain
Revised: 2 is a constant, we obtain

Original: [Equation]
Revised: [Equation]

Original: as lim t →∞ α t +1 = 0 .
Revised: as lim t →∞ α t +1 = 0 .

Original: This reveals lim t →∞ β 2 t = 1 .
Revised: This reveals lim t →∞ β 2 t = 1 .

Original: Furthermore, since β t ≥ 0 , we then have lim t →∞ β t = 1 .
Revised: Furthermore, since β t ≥ 0 , we then have lim t →∞ β t = 1 .

Original: Next, we are going to prove lim t →∞ β t = 1 when − 1 < β 0 < 0 .
Revised: Next, we are going to prove lim t →∞ β t = 1 when − 1 < β 0 < 0 .

Original: According to Lemma A.3, we just need prove that ∃ τ > 0 , s.t. , β τ ≥ 0 .
Revised: According to Lemma A.3, we just need prove that ∃ τ > 0 , s.t. , β τ ≥ 0 .

Original: For the sake of contradiction, suppose that β t < 0 for all t > 0 , we then have lim t →∞ α t = 0 .
Revised: For the sake of contradiction, suppose that β t < 0 for all t > 0 , we then have lim t →∞ α t = 0 .

Original: As a consequence, we obtain
Revised: As a consequence, we obtain

Original: [Equation]
Revised: [Equation]

Original: and we know that ∃ t ′ > 0 , such that
Revised: and we know that ∃ t ′ > 0 , such that

Original: [Equation]
Revised: [Equation]

Original: According to the iterations in Eq.
Revised: According to the iterations in Eq.

Original: (A.56), we can derive that
Revised: (A.56), we can derive that

Original: [Equation]
Revised: [Equation]

Original: Since 1 − α t ( α t + β t − α t β 2 t ) 1+ α 2 t − α 2 t β 2 t ≥ 1 , then for t ≥ t ′ , α
Revised: Since 1 − α t ( α t + β t − α t β 2 t ) 1+ α 2 t − α 2 t β 2 t ≥ 1 , then for t ≥ t ′ , α

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ≥ ξ
Revised: ≥ ξ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: − β
Revised: − β

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: − β 20 )
Revised: − β 20 )

Original: [Equation]
Revised: [Equation]

Original: (A.61) which contradicts the fact in Eq.
Revised: (A.61) which contradicts the fact in Eq.

Original: (A.60).
Revised: (A.60).

Original: Thus, ∃ τ > 0 , s.t. β τ ≥ 0 .
Revised: Thus, ∃ τ > 0 , s.t. β τ ≥ 0 .

Original: Consider the dynamical system with an initial time τ , we have lim t →∞ β t = 1 according to Lemma A.3.
Revised: Consider the dynamical system with an initial time τ , we have lim t →∞ β t = 1 according to Lemma A.3.

Original: To sum up, we have proven that lim t →∞ β t = 1 when β 0 > − 1 and α 0 > 0 .
Revised: To sum up, we have proven that lim t →∞ β t = 1 when β 0 > − 1 and α 0 > 0 .

Original: Theorem 3.5 Considering the discrete dynamics in Eq.
Revised: Theorem 3.5 Considering the discrete dynamics in Eq.

Original: (3.16) , if ∀ i ∈ [ N ] , c ∈
Revised: (3.16) , if ∀ i ∈ [ N ] , c ∈

Original: [ C ] , (cid:98) w ⊤ c (cid:98) h
Revised: [ C ] , (cid:98) w ⊤ c (cid:98) h

Original: i,c (0) > − 1 , the learning rate η ( t ) satisfies that
Revised: i,c (0) > − 1 , the learning rate η ( t ) satisfies that

Original: [Equation]
Revised: [Equation]

Original: lim t →∞ η ( t +1) η
Revised: lim t →∞ η ( t +1) η

Original: ( t )
Revised: ( t )

Original: = 1 , and there exists a constant ε > 0 , s.t., η ( t ) >
Revised: = 1 , and there exists a constant ε > 0 , s.t., η ( t ) >

Original: ε , then we have
Revised: ε , then we have

Original: [Equation]
Revised: [Equation]

Original: and further if lim t →∞ ∥ H ( t ) ∥ < ∞ , then there exists a constant µ > 0 , such that the error above shows exponential convergence:
Revised: and further if lim t →∞ ∥ H ( t ) ∥ < ∞ , then there exists a constant µ > 0 , such that the error above shows exponential convergence:

Original: [Equation]
Revised: [Equation]

Original: Moreover, if (cid:98) w ⊤ c (cid:98) h i,c (0) = − 1 , then h i,c ( t )
Revised: Moreover, if (cid:98) w ⊤ c (cid:98) h i,c (0) = − 1 , then h i,c ( t )

Original: =
Revised: =

Original: h i,c (0) .
Revised: h i,c (0) .

Original: Proof.
Revised: Proof.

Original: Since H ( t + 1) =
Revised: Since H ( t + 1) =

Original: H ( t ) + (1+ γ ) η ( t ) CN (cid:16) ∂ (cid:99)
Revised: H ( t ) + (1+ γ ) η ( t ) CN (cid:16) ∂ (cid:99)

Original: H ∂ H (cid:12)(cid:12) H = H ( t ) (cid:17)
Revised: H ∂ H (cid:12)(cid:12) H = H ( t ) (cid:17)

Original: ⊤ W
Revised: ⊤ W

Original: ( I C ⊗ 1 ⊤ N ) , then for i ∈
Revised: ( I C ⊗ 1 ⊤ N ) , then for i ∈

Original: [ N ] , c ∈ [ C ] ,
Revised: [ N ] , c ∈ [ C ] ,

Original: [Equation]
Revised: [Equation]

Original: h ⊤
Revised: h ⊤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:17) w c .
Revised: (cid:17) w c .

Original: (A.64)
Revised: (A.64)

Original: According to Lemma A.2, when (cid:98) w ⊤ c (cid:98) h i,c (0) > − 1 and η (0)(1+ γ ) CN < ∥ h
Revised: According to Lemma A.2, when (cid:98) w ⊤ c (cid:98) h i,c (0) > − 1 and η (0)(1+ γ ) CN < ∥ h

Original: i,c (0) ∥∥
Revised: i,c (0) ∥∥

Original: w c ∥ 2 , we have lim t →∞ ∥ (cid:98) h
Revised: w c ∥ 2 , we have lim t →∞ ∥ (cid:98) h

Original: i,c − (cid:98) w c ∥ = 0 , then lim t →∞ (cid:13)(cid:13)(cid:13)(cid:99) H ( t ) − (cid:99) W
Revised: i,c − (cid:98) w c ∥ = 0 , then lim t →∞ (cid:13)(cid:13)(cid:13)(cid:99) H ( t ) − (cid:99) W

Original: ( I C ⊗ 1 ⊤ N ) (cid:13)(cid:13)(cid:13) = 0 .
Revised: ( I C ⊗ 1 ⊤ N ) (cid:13)(cid:13)(cid:13) = 0 .

Original: If further lim t →∞ ∥ H ( t ) ∥ 2 < ∞ , let L = sup
Revised: If further lim t →∞ ∥ H ( t ) ∥ 2 < ∞ , let L = sup

Original: i,c,t ∥ h
Revised: i,c,t ∥ h

Original: i,c ( t ) ∥ .
Revised: i,c ( t ) ∥ .

Original: According to the proof of Lemma
Revised: According to the proof of Lemma

Original: A.2, ∀ i, c , we have lim t →∞ (cid:98) w ⊤ c (cid:98) h
Revised: A.2, ∀ i, c , we have lim t →∞ (cid:98) w ⊤ c (cid:98) h

Original: i,c ( t ) = 1 , then for a given constant δ > 0 , ∃ τ > 0 , s.t. , ∀ t > τ , (cid:98) w ⊤ c (cid:98) h i,c ( τ ) ≥ δ .
Revised: i,c ( t ) = 1 , then for a given constant δ > 0 , ∃ τ > 0 , s.t. , ∀ t > τ , (cid:98) w ⊤ c (cid:98) h i,c ( τ ) ≥ δ .

Original: Consider t > τ , we have
Revised: Consider t > τ , we have

Original: [Equation]
Revised: [Equation]

Original: where the first, the second, and the third inequalities are based on the facts that ∥ h i,c ( t )
Revised: where the first, the second, and the third inequalities are based on the facts that ∥ h i,c ( t )

Original: ∥∥ h
Revised: ∥∥ h

Original: i,c ( t +1) ∥
Revised: i,c ( t +1) ∥

Original: 22 ≤ 1 , 1 − (1+ γ ) η ( j +1)
Revised: 22 ≤ 1 , 1 − (1+ γ ) η ( j +1)

Original: ∥ w c ∥ 2 CN ∥
Revised: ∥ w c ∥ 2 CN ∥

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: c (
Revised: c (

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: −
Revised: −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ∥
Revised: ∥

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: 2 CN ∥
Revised: 2 CN ∥

Original: h i,c ( j ) ∥ 22 for t > τ , and ∥ h
Revised: h i,c ( j ) ∥ 22 for t > τ , and ∥ h

Original: i,c ( j ) ∥ 2 ≤ L , respectively.
Revised: i,c ( j ) ∥ 2 ≤ L , respectively.

Original: Let c 1 = max i,
Revised: Let c 1 = max i,

Original: [Equation]
Revised: [Equation]

Original: ∥ w
Revised: ∥ w

Original: [Equation]
Revised: [Equation]

Original: ∥
Revised: ∥

Original: [Equation]
Revised: [Equation]

Original: CNL
Revised: CNL

Original: [Equation]
Revised: [Equation]

Original: (cid:17) −
Revised: (cid:17) −

Original: [Equation]
Revised: [Equation]

Original: and µ = min c − 2 log (cid:16) 1 − (1+ γ ) εδ ∥ w c ∥ 2 CNL(cid:17) , then 1 − (cid:98) w ⊤ c (cid:98) h
Revised: and µ = min c − 2 log (cid:16) 1 − (1+ γ ) εδ ∥ w c ∥ 2 CNL(cid:17) , then 1 − (cid:98) w ⊤ c (cid:98) h

Original: i,c ( t + 1) ≤ c 1 e − µt 1+ δ .
Revised: i,c ( t + 1) ≤ c 1 e − µt 1+ δ .

Original: Therefore, we have (cid:13)(cid:13)(cid:13)(cid:99) H ( t )
Revised: Therefore, we have (cid:13)(cid:13)(cid:13)(cid:99) H ( t )

Original: [Equation]
Revised: [Equation]

Original: (cid:99) W
Revised: (cid:99) W

Original: [Equation]
Revised: [Equation]

Original: I C
Revised: I C

Original: [Equation]
Revised: [Equation]

Original: 1 ⊤
Revised: 1 ⊤

Original: [Equation]
Revised: [Equation]

Original: i,c (cid:0)
Revised: i,c (cid:0)

Original: [Equation]
Revised: [Equation]

Original: (cid:98)
Revised: (cid:98)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: + 1) (cid:1)
Revised: + 1) (cid:1)

Original: [Equation]
Revised: [Equation]

Original: CNe
Revised: CNe

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: δ ) , (A.65) i.e. , (cid:13)(cid:13)(cid:13)(cid:99) H ( t ) − (cid:99) W ( I C ⊗ 1 ⊤ N ) (cid:13)(cid:13)(cid:13) =
Revised: δ ) , (A.65) i.e. , (cid:13)(cid:13)(cid:13)(cid:99) H ( t ) − (cid:99) W ( I C ⊗ 1 ⊤ N ) (cid:13)(cid:13)(cid:13) =

Original: O ( e − µt ) .
Revised: O ( e − µt ) .

Original: Moreover, if (cid:98) w ⊤ c (cid:98) h i,c (0) = − 1 , we have h i,c ( t + 1)
Revised: Moreover, if (cid:98) w ⊤ c (cid:98) h i,c (0) = − 1 , we have h i,c ( t + 1)

Original: =
Revised: =

Original: h i,c ( t ) + (1 + γ ) η ( t ) CN ∥
Revised: h i,c ( t ) + (1 + γ ) η ( t ) CN ∥

Original: h i,c ( t ) ∥(cid:16) I p − (cid:98) h i,c ( t ) (cid:98)
Revised: h i,c ( t ) ∥(cid:16) I p − (cid:98) h i,c ( t ) (cid:98)

Original: h ⊤ i,c ( t ) (cid:17) w c
Revised: h ⊤ i,c ( t ) (cid:17) w c

Original: = h
Revised: = h

Original: i,c ( t ) , (A.66) thus h i,c ( t ) =
Revised: i,c ( t ) , (A.66) thus h i,c ( t ) =

Original: h i,c (0) .
Revised: h i,c (0) .

Original: A. P ROOF OF T HEOREM 4. 
Revised: A. P ROOF OF T HEOREM 4.

Original: Theorem 4.1 Consider the continual gradient flow (Equation (4.1) ) in which the prototypes W is fixed, we have the closed-form dynamics:
Revised: Theorem 4.1 Consider the continual gradient flow (Equation (4.1) ) in which the prototypes W is fixed, we have the closed-form dynamics:

Original: [Equation]
Revised: [Equation]

Original: which further indicates that (cid:13)(cid:13) H ( t ) − 1 λ W M (cid:13)(cid:13)
Revised: which further indicates that (cid:13)(cid:13) H ( t ) − 1 λ W M (cid:13)(cid:13)

Original: =
Revised: =

Original: O (cid:16) e − λ (cid:82) t 0 η
Revised: O (cid:16) e − λ (cid:82) t 0 η

Original: ( τ )d τ (cid:17) .
Revised: ( τ )d τ (cid:17) .

Original: Proof.
Revised: Proof.

Original: For the first order non-homogeneous linear difference equation in Eq.
Revised: For the first order non-homogeneous linear difference equation in Eq.

Original: [Equation]
Revised: [Equation]

Original: the solution is
Revised: the solution is

Original: [Equation]
Revised: [Equation]

Original: then ∥ H ( t
Revised: then ∥ H ( t

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: 0 η ( τ )d τ ( H (0) − WM λ )
Revised: 0 η ( τ )d τ ( H (0) − WM λ )

Original: ∥
Revised: ∥

Original: =
Revised: =

Original: O ( e − λ (cid:82) t 0 η ( τ )d τ ) .
Revised: O ( e − λ (cid:82) t 0 η ( τ )d τ ) .

Original: A. T HE P ROJECTIONS ONTO E +1 , E − 1 , E +2 , E − 2 AND E
Revised: A. T HE P ROJECTIONS ONTO E +1 , E − 1 , E +2 , E − 2 AND E

Original: Lemma A.4.
Revised: Lemma A.4.

Original: Let S denote the subspace S
Revised: Let S denote the subspace S

Original: = { W : W 1 n = 0 , W ∈ R m × n } , then the projection of a point A ∈ R m × n onto S can be denoted as Π S A =
Revised: = { W : W 1 n = 0 , W ∈ R m × n } , then the projection of a point A ∈ R m × n onto S can be denoted as Π S A =

Original: A (cid:0) I n − 1 n 1 n 1 ⊤ n (cid:1) .
Revised: A (cid:0) I n − 1 n 1 n 1 ⊤ n (cid:1) .

Original: Proof.
Revised: Proof.

Original: Let W = ( w 1 , w 2 , ..., w n ) ∈ S and A
Revised: Let W = ( w 1 , w 2 , ..., w n ) ∈ S and A

Original: =
Revised: =

Original: ( a 1 , a 2 , ..., a n ) ∈ R m × n , we have
Revised: ( a 1 , a 2 , ..., a n ) ∈ R m × n , we have

Original: [Equation]
Revised: [Equation]

Original: where we used the Cauchy-Schwarz inequality, and the equality holds if and only if w i −
Revised: where we used the Cauchy-Schwarz inequality, and the equality holds if and only if w i −

Original: a i = w n − a n , ∀ i ∈ [ n ] , and (cid:80) ni
Revised: a i = w n − a n , ∀ i ∈ [ n ] , and (cid:80) ni

Original: =

Revised: =

Original: w i = 0 , i.e. , W = A (cid:0) I n − 1 n 1 n 1 ⊤ n (cid:1) .
Revised: w i = 0 , i.e. , W = A (cid:0) I n − 1 n 1 n 1 ⊤ n (cid:1) .

Original: Therefore, the projection of
Revised: Therefore, the projection of

Original: A onto S is Π S A = arg
Revised: A onto S is Π S A = arg

Original: min W ∈S
Revised: min W ∈S

Original: ∥ W − A ∥ 2 F = A (cid:0) I n − 1 n 1 n 1 ⊤ n (cid:1) .
Revised: ∥ W − A ∥ 2 F = A (cid:0) I n − 1 n 1 n 1 ⊤ n (cid:1) .

Original: Lemma A.5.
Revised: Lemma A.5.

Original: Let S denote the subspace { W : W
Revised: Let S denote the subspace { W : W

Original: ( I c ⊗ 1 n ) = 0 , W ∈ R m × cn } , then the projection of a point A ∈ R m × cn onto S can be denoted as Π S A
Revised: ( I c ⊗ 1 n ) = 0 , W ∈ R m × cn } , then the projection of a point A ∈ R m × cn onto S can be denoted as Π S A

Original: =
Revised: =

Original: A (cid:0)
Revised: A (cid:0)

Original: I cn − 1 n I c ⊗ 1 n 1 ⊤ n (cid:1) .
Revised: I cn − 1 n I c ⊗ 1 n 1 ⊤ n (cid:1) .

Original: Proof.
Revised: Proof.

Original: The proof is similar to Lemma A.4.
Revised: The proof is similar to Lemma A.4.

Original: We can simply let W =
Revised: We can simply let W =

Original: ( W 1 , ..., W n ) ∈ S and
Revised: ( W 1 , ..., W n ) ∈ S and

Original: A =
Revised: A =

Original: ( A 1 , ..., A n ) ∈ R m × cn , where W i , A i ∈
Revised: ( A 1 , ..., A n ) ∈ R m × cn , where W i , A i ∈

Original: R m × c , then we have
Revised: R m × c , then we have

Original: [Equation]
Revised: [Equation]

Original: where the equality holds if and only if W i −
Revised: where the equality holds if and only if W i −

Original: A i =
Revised: A i =

Original: W n − A n , ∀ i ∈ [ n ] , and (cid:80) ni
Revised: W n − A n , ∀ i ∈ [ n ] , and (cid:80) ni

Original: =1 W i = 0 , i.e. , W =
Revised: =1 W i = 0 , i.e. , W =

Original: A ( I cn − 1 n I c ⊗ 1 n 1 ⊤ n ) .
Revised: A ( I cn − 1 n I c ⊗ 1 n 1 ⊤ n ) .

Original: Therefore, the projection of A onto S is Π S A
Revised: Therefore, the projection of A onto S is Π S A

Original: = A (cid:0)
Revised: = A (cid:0)

Original: I cn − 1 n I c ⊗ 1 n 1 ⊤ n (cid:1) .
Revised: I cn − 1 n I c ⊗ 1 n 1 ⊤ n (cid:1) .

Original: Lemma A.6.
Revised: Lemma A.6.

Original: For H ∈ R p × CN , W ∈ R p × C , the projection of ( H , W ) onto E ϵ 1 is
Revised: For H ∈ R p × CN , W ∈ R p × C , the projection of ( H , W ) onto E ϵ 1 is

Original: [Equation]
Revised: [Equation]

Original: (A
Revised: (A

Original: [Equation]
Revised: [Equation]

Original: where P = 1 2 ( ϵ √ N H ( I C ⊗ 1 N ) + W )( I C − 1 C 1 C 1 ⊤ C ) .
Revised: where P = 1 2 ( ϵ √ N H ( I C ⊗ 1 N ) + W )( I C − 1 C 1 C 1 ⊤ C ) .

Original: Proof.
Revised: Proof.

Original: Let S
Revised: Let S

Original: =
Revised: =

Original: { Z : Z 1 C = 0 , Z ∈ R p × C } and H =
Revised: { Z : Z 1 C = 0 , Z ∈ R p × C } and H =

Original: { H 1 , ..., H N } (where H i ∈
Revised: { H 1 , ..., H N } (where H i ∈

Original: R p × C ), the minimizer of Z ∈ S is
Revised: R p × C ), the minimizer of Z ∈ S is

Original: [Equation]
Revised: [Equation]

Original: Thus, Π ϵ 1 ( H , W ) =
Revised: Thus, Π ϵ 1 ( H , W ) =

Original: ( ϵ √ N ( P ⊗ 1 ⊤ N ) , P ) with P = 12 ( ϵ √ N H ( I C ⊗ 1 N )+ W )( I C − 1 C 1 C 1 ⊤ C ) .
Revised: ( ϵ √ N ( P ⊗ 1 ⊤ N ) , P ) with P = 12 ( ϵ √ N H ( I C ⊗ 1 N )+ W )( I C − 1 C 1 C 1 ⊤ C ) .

Original: Lemma A.7.
Revised: Lemma A.7.

Original: For H ∈ R p × CN , W ∈ R p × C , the projection of ( H , W ) onto E ϵ 2 is
Revised: For H ∈ R p × CN , W ∈ R p × C , the projection of ( H , W ) onto E ϵ 2 is

Original: [Equation]
Revised: [Equation]

Original: 71) where h = 12 C ( ϵ √ N H 1 CN + W 1 C ) .
Revised: 71) where h = 12 C ( ϵ √ N H 1 CN + W 1 C ) .

Original: Proof.
Revised: Proof.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Lemma A.8.
Revised: Lemma A.8.

Original: For H ∈ R p × CN , W ∈ R p × C , the projection of ( H , W ) onto E 3 is
Revised: For H ∈ R p × CN , W ∈ R p × C , the projection of ( H , W ) onto E 3 is

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This can be easily derived by Lemma A.5.
Revised: This can be easily derived by Lemma A.5.

Original: B E XPERIMENTS
Revised: B E XPERIMENTS

Original: In this section, we provide experimental details, including datasets, network architectures, optimization methods, hyperparameter settings, and more results.
Revised: In this section, we provide experimental details, including datasets, network architectures, optimization methods, hyperparameter settings, and more results.

Original: B. N UMERICAL E XPERIMENTS
Revised: B. N UMERICAL E XPERIMENTS

Original: For numerical experiments in Figures 1, 4, 5, 6, 7, 8, and 9, we set p = 512 , C = 100 , N = 10 , and then randomly initialize H 0 and W 0 .
Revised: For numerical experiments in Figures 1, 4, 5, 6, 7, 8, and 9, we set p = 512 , C = 100 , N = 10 , and then randomly initialize H 0 and W 0 .

Original: We use the SGD optimizer to optimize these free variables.
Revised: We use the SGD optimizer to optimize these free variables.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: with γ ∈ { 0 , 0 . 1 , 0 . 01 , 0 .
Revised: with γ ∈ { 0 , 0 . 1 , 0 . 01 , 0 .

Original: 001 , 1 C − 1 } .
Revised: 001 , 1 C − 1 } .

Original: We set p = 512 , C = 100 , N = 10 , λ
Revised: We set p = 512 , C = 100 , N = 10 , λ

Original: = (1+ γ ) C √ N , η 1 ( t ) =
Revised: = (1+ γ ) C √ N , η 1 ( t ) =

Original: η 2 ( t ) = 0 .
Revised: η 2 ( t ) = 0 .

Original: 1 ( i.e. , s = 1 ), thus we have lim t →∞ Z ( t ) =
Revised: 1 ( i.e. , s = 1 ), thus we have lim t →∞ Z ( t ) =

Original: Π
Revised: Π

Original: +1 Z 0 , according to Corollary 3.4, and then randomly initialize H 0 and W 0 .
Revised: +1 Z 0 , according to Corollary 3.4, and then randomly initialize H 0 and W 0 .

Original: (a) The training accuracy with the prediction rule arg max c w ⊤ c h .
Revised: (a) The training accuracy with the prediction rule arg max c w ⊤ c h .

Original: As expected, the features align to their corresponding prototypes when γ < 2 C − 2 .
Revised: As expected, the features align to their corresponding prototypes when γ < 2 C − 2 .

Original: (b) The ℓ 2 distance between H ( t ) and H +1 .
Revised: (b) The ℓ 2 distance between H ( t ) and H +1 .

Original: As expected Theorem 3.4, the distance will decrease as exponential rate when 0 < γ < 2 C − 2 .
Revised: As expected Theorem 3.4, the distance will decrease as exponential rate when 0 < γ < 2 C − 2 .

Original: (c) The ℓ 2 distance between W ( t ) and W +1 .
Revised: (c) The ℓ 2 distance between W ( t ) and W +1 .

Original: (d) and (e) denote the norm of features and prototypes, respectively.
Revised: (d) and (e) denote the norm of features and prototypes, respectively.

Original: As can be seen, ∥ H ∥ 2 and ∥
Revised: As can be seen, ∥ H ∥ 2 and ∥

Original: W ∥ 2 do not grow exponentially as in the unconstrained case, which confirms that weight decay can avoid excessive growth of feature norm and prototype norm.
Revised: W ∥ 2 do not grow exponentially as in the unconstrained case, which confirms that weight decay can avoid excessive growth of feature norm and prototype norm.

Original: B. V ISUAL C LASSIFICATION
Revised: B. V ISUAL C LASSIFICATION

Original: For classification experiments in Fig.
Revised: For classification experiments in Fig.

Original: 2, Fig.
Revised: 2, Fig.

Original: 12, Fig. 13, Fig.
Revised: 12, Fig. 13, Fig.

Original: 14, we experiment with ResNet-18,
Revised: 14, we experiment with ResNet-18,

Original: ResNet-34, and ResNet-50 (He et al., 2016) trained on CIFAR-10, CIFAR-100 (Krizhevsky and
Revised: ResNet-34, and ResNet-50 (He et al., 2016) trained on CIFAR-10, CIFAR-100 (Krizhevsky and

Original: Hinton, 2009), and ImageNet-100 that takes the first 100 classes of ImageNet (Deng et al., 2009) , respectively.
Revised: Hinton, 2009), and ImageNet-100 that takes the first 100 classes of ImageNet (Deng et al., 2009) , respectively.

Original: The networks are trained for 200 epochs and 100 epochs for CIFAR-10/-100 and
Revised: The networks are trained for 200 epochs and 100 epochs for CIFAR-10/-100 and

Original: ImageNet-100, respectively.
Revised: ImageNet-100, respectively.

Original: For all training, we use SGD optimizer with momentum 0.9 and cosine
Revised: For all training, we use SGD optimizer with momentum 0.9 and cosine

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: η ( s = 1 ), γ = 1 C − 1 , and λ = 1+ γ C √ N .
Revised: η ( s = 1 ), γ = 1 C − 1 , and λ = 1+ γ C √ N .

Original: As can be seen, features and prototypes converge to ( H , W ) exponentially, and larger learning rates can accelerate convergence.
Revised: As can be seen, features and prototypes converge to ( H , W ) exponentially, and larger learning rates can accelerate convergence.

Original: [Figure]
Revised: [Figure]

Original: learning rate annealing Loshchilov and Hutter (2017) with T max being the corresponding epochs.
Revised: learning rate annealing Loshchilov and Hutter (2017) with T max being the corresponding epochs.

Original: The initial learning rate is set to 0 .
Revised: The initial learning rate is set to 0 .

Original: 1 , weight decay is set to 5 × 10 − 4 , and batch size is set to 256.
Revised: 1 , weight decay is set to 5 × 10 − 4 , and batch size is set to 256.

Original: Typical data augmentations including random width/height shift and horizontal flip are applied.
Revised: Typical data augmentations including random width/height shift and horizontal flip are applied.

Original: Moreover, to use the PAL and FNPAL (Zhou et al., 2022b) that anchors prototypes with a neural collapse solution, we remove the ReLU layer before the linear classifier in the last layer.
Revised: Moreover, to use the PAL and FNPAL (Zhou et al., 2022b) that anchors prototypes with a neural collapse solution, we remove the ReLU layer before the linear classifier in the last layer.

Original: B. I MBALANCED C LASSIFICATION
Revised: B. I MBALANCED C LASSIFICATION

Original: For the experiments of imbalanced learning in Tab. 1, Tab. 2, Tab. 3, Fig. 10, and Fig.
Revised: For the experiments of imbalanced learning in Tab. 1, Tab. 2, Tab. 3, Fig. 10, and Fig.

Original: 11, we utilize the same network architectures, and optimization settings as visual classification.
Revised: 11, we utilize the same network architectures, and optimization settings as visual classification.

Original: We only use the imbalanced versions of CIFAR-10 and CIFAR-100 by following the setting in (Zhou et al., 2022c).
Revised: We only use the imbalanced versions of CIFAR-10 and CIFAR-100 by following the setting in (Zhou et al., 2022c).

Original: The number of training examples is reduced for per class, and the test set keeps unchanged, where we use the imbalance ratio ρ = max i n i min i n i to denote the ratio between sample sizes of the most frequent and least frequent class.
Revised: The number of training examples is reduced for per class, and the test set keeps unchanged, where we use the imbalance ratio ρ = max i n i min i n i to denote the ratio between sample sizes of the most frequent and least frequent class.

Original: Moreover, long-tailed imbalance (Cui et al., 2019) that utilizes
Revised: Moreover, long-tailed imbalance (Cui et al., 2019) that utilizes

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: set p = 512 , C = 100 ,
Revised: set p = 512 , C = 100 ,

Original: N = 10 , η 2 = 0 .
Revised: N = 10 , η 2 = 0 .

Original: 5 , γ = 1 C − 1 , where H = π + h H
Revised: 5 , γ = 1 C − 1 , where H = π + h H

Original: +

Revised: +

Original: + π − h H − 1 and W = π + w W +

Revised: + π − h H − 1 and W = π + w W +

Original: + π − w W − 1 in
Revised: + π − w W − 1 in

Original: Corollary 3.4.
Revised: Corollary 3.4.

Original: As can be seen, larger scale parameter s can achieve faster convergence speed
Revised: As can be seen, larger scale parameter s can achieve faster convergence speed

Original: [Figure]
Revised: [Figure]

Original: an exponential decay in samples sizes and step imbalance (Buda et al., 2018)(that sets all minority classes to have the same number of samples, as do all majority classes) are considered.
Revised: an exponential decay in samples sizes and step imbalance (Buda et al., 2018)(that sets all minority classes to have the same number of samples, as do all majority classes) are considered.

Original: For imbalanced learning, we utilize expected calibration error (ECE) to measure calibration of the models (Zhong et al., 2021), where all predictions are grouped into several interval bins of equal size and then calculate the error between the accuracy and confidence for each interval bin, i.e ,
Revised: For imbalanced learning, we utilize expected calibration error (ECE) to measure calibration of the models (Zhong et al., 2021), where all predictions are grouped into several interval bins of equal size and then calculate the error between the accuracy and confidence for each interval bin, i.e ,

Original: ECE =
Revised: ECE =

Original: B (cid:88) b =
Revised: B (cid:88) b =

Original: [Equation]
Revised: [Equation]

Original: where N denotes the number of predictions, B is the number of interval bins, S b is the set of samples whose prediction scores fall into Bin- b , acc ( · ) and conf ( · ) denote the accuracy and predicted confidence of S b , respectively.
Revised: where N denotes the number of predictions, B is the number of interval bins, S b is the set of samples whose prediction scores fall into Bin- b , acc ( · ) and conf ( · ) denote the accuracy and predicted confidence of S b , respectively.

Original: As shown in Tab. 2 and Tab. 3, explicit feature regularization can improve imbalanced learning on
Revised: As shown in Tab. 2 and Tab. 3, explicit feature regularization can improve imbalanced learning on

Original: CIFAR-10/-100 in most cases.
Revised: CIFAR-10/-100 in most cases.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: B. O UT - OF -D ISTRIBUTION D ETECTION
Revised: B. O UT - OF -D ISTRIBUTION D ETECTION

Original: For the experiments of OOD detection in Fig.
Revised: For the experiments of OOD detection in Fig.

Original: 3, Tab. 4, Fig.
Revised: 3, Tab. 4, Fig.

Original: 16, Fig.
Revised: 16, Fig.

Original: [Figure]
Revised: [Figure]

Original: and Fig.
Revised: and Fig.

Original: 18, we use a
Revised: 18, we use a

Original: ResNet-18 on CIFAR-10 and a ResNet-34 on CIFAR-100 to train the classification models
Revised: ResNet-18 on CIFAR-10 and a ResNet-34 on CIFAR-100 to train the classification models

Original: [Figure]
Revised: [Figure]

Original: and use their test dataset as the in-distribution data D testin .
Revised: and use their test dataset as the in-distribution data D testin .

Original: For the OOD test dataset D testout , we simply use a common benchmark: SVHN (Netzer et al., 2011).
Revised: For the OOD test dataset D testout , we simply use a common benchmark: SVHN (Netzer et al., 2011).

Original: We measure the performance with the following metrics: (1) the false positive rate (FPR95) of OOD examples when true positive rate of in-distribution examples is at 95% ; (2) the area under the receiver operating characteristic curve (AUROC); and (3) the area under the precision-recall curve (AUPR).
Revised: We measure the performance with the following metrics: (1) the false positive rate (FPR95) of OOD examples when true positive rate of in-distribution examples is at 95% ; (2) the area under the receiver operating characteristic curve (AUROC); and (3) the area under the precision-recall curve (AUPR).

Original: We then consider the softmaxbased score (Hendrycks and Gimpel, 2016), energy-based score (Liu et al., 2020), and our proposed feature norm-based score to assessing the improvement of explicit feature regularization over the normal training.
Revised: We then consider the softmaxbased score (Hendrycks and Gimpel, 2016), energy-based score (Liu et al., 2020), and our proposed feature norm-based score to assessing the improvement of explicit feature regularization over the normal training.

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] 
Revised: [Table] [Figure] [Table]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: C O THER P OTENTIAL I NSIGHTS C. A G OOD I NITIALIZATION OF P ROTOTYPES
Revised: C O THER P OTENTIAL I NSIGHTS C. A G OOD I NITIALIZATION OF P ROTOTYPES

Original: As depicted in Sec. 3 and Appendix A.8, the dynamics under the ASM loss is dependent on the initialization of both features and prototypes, such as Π +1 Z 0 =
Revised: As depicted in Sec. 3 and Appendix A.8, the dynamics under the ASM loss is dependent on the initialization of both features and prototypes, such as Π +1 Z 0 =

Original: ( 1 √ N ( P ⊗ 1 ⊤ N ) , P ) , where P = 12 (cid:16) 1 √ N H 

Revised: ( 1 √ N ( P ⊗ 1 ⊤ N ) , P ) , where P = 12 (cid:16) 1 √ N H

Original: ( I C ⊗ 1 N ) + W 0 (cid:17) ( I C − 1 C 1 C 1 ⊤ C ) .
Revised: ( I C ⊗ 1 N ) + W 0 (cid:17) ( I C − 1 C 1 C 1 ⊤ C ) .

Original: However, these features H 0 extracted from a dataset by some nonlinear layers and parameterized layers are practically intractable, but we can elaborately initialize W 0 and highlight its role in the whole.
Revised: However, these features H 0 extracted from a dataset by some nonlinear layers and parameterized layers are practically intractable, but we can elaborately initialize W 0 and highlight its role in the whole.

Original: To do this, we consider two ways: (1) Initializing the structure of W 0 .
Revised: To do this, we consider two ways: (1) Initializing the structure of W 0 .

Original: Inspired by the neural collapse solution that maximizes class separation, we can initialize W 0 as this structure, i.e. , (cid:98) w ⊤ i (cid:98) w j = − 1 C − 1 , ∀ i ̸
Revised: Inspired by the neural collapse solution that maximizes class separation, we can initialize W 0 as this structure, i.e. , (cid:98) w ⊤ i (cid:98) w j = − 1 C − 1 , ∀ i ̸

Original: = j ; (2) Increasing the importance of W 0 .
Revised: = j ; (2) Increasing the importance of W 0 .

Original: A simple strategy is scaling up W 0 , thereby implicitly weakening the importance of H 0 .
Revised: A simple strategy is scaling up W 0 , thereby implicitly weakening the importance of H 0 .

Original: However, it is difficult to handle the initialization of features because they are obtained by a complex processing a large dataset, thus we seek to initialize the prototypes in the last layer of the network.
Revised: However, it is difficult to handle the initialization of features because they are obtained by a complex processing a large dataset, thus we seek to initialize the prototypes in the last layer of the network.

Original: [Figure]
Revised: [Figure]

Original: C. R EFINED D ECISION - MAKINGS
Revised: C. R EFINED D ECISION - MAKINGS

Original: Recalling the rule— arg max c ′ ⟨ w c ′
Revised: Recalling the rule— arg max c ′ ⟨ w c ′

Original: [Figure]
Revised: [Figure]

Original: h ⟩ + b c ′ that makes decision by selecting the class with the largest logit (where the inner product ⟨ w c ′ , h ⟩ is dominant), which may not be good to directly use the learned features and prototypes
Revised: h ⟩ + b c ′ that makes decision by selecting the class with the largest logit (where the inner product ⟨ w c ′ , h ⟩ is dominant), which may not be good to directly use the learned features and prototypes

Original: [Figure]
Revised: [Figure]

Original: since learning with the ASM within limited iterations (that means ζ 1 ( t ) < ∞ ) will introduce some residual ∆ ( t ) caused by gradient descent regardless of the unconstrained case or regularized case.
Revised: since learning with the ASM within limited iterations (that means ζ 1 ( t ) < ∞ ) will introduce some residual ∆ ( t ) caused by gradient descent regardless of the unconstrained case or regularized case.

Original: Example C.1.
Revised: Example C.1.

Original: If we add a perturbation ∆ for all features while adding s ∆ for all prototypes, then the perturbed decision-making will be arg max c ′ ⟨ w c ′ + s ∆ , h + ∆ ⟩ + b c ′ , which may not be equivalent to arg max c ′ ⟨ w c ′ , h ⟩ + b c ′ .
Revised: If we add a perturbation ∆ for all features while adding s ∆ for all prototypes, then the perturbed decision-making will be arg max c ′ ⟨ w c ′ + s ∆ , h + ∆ ⟩ + b c ′ , which may not be equivalent to arg max c ′ ⟨ w c ′ , h ⟩ + b c ′ .

Original: C. A DJUSTED S AMPLE M ARGIN L OSS
Revised: C. A DJUSTED S AMPLE M ARGIN L OSS

Original: As aforementioned in Sec.
Revised: As aforementioned in Sec.

Original: 3.3 and the proof of Theorem 3.5, we will encounter zero gradients when the cosine similarity (cid:98) w ⊤ y (cid:98) h is − 1 or 1 , so we can adjust the loss to avoid the issue by to w y and accelerate convergence:
Revised: 3.3 and the proof of Theorem 3.5, we will encounter zero gradients when the cosine similarity (cid:98) w ⊤ y (cid:98) h is − 1 or 1 , so we can adjust the loss to avoid the issue by to w y and accelerate convergence:

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: where ε ∈ (0 , 1) is a hyperparameter and δ = −
Revised: where ε ∈ (0 , 1) is a hyperparameter and δ = −

Original: [Equation]
Revised: [Equation]

Original: formed with a stop-gradient) satisfying ( w y + δ ) ⊤ (cid:98)
Revised: formed with a stop-gradient) satisfying ( w y + δ ) ⊤ (cid:98)

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (Liu et al., 2020) from models trained with different explicit feature regularization, where CE is the loss function
Revised: (Liu et al., 2020) from models trained with different explicit feature regularization, where CE is the loss function

Original: [Figure] [Figure]
Revised: [Figure] [Figure]