G ENERALIZING AND D ECOUPLING N EURAL C OLLAPSE VIA H YPERSPHERICAL U NIFORMITY G AP
Anonymous authors
Paper under double-blind review
A BSTRACT
The neural collapse (NC) phenomenon describes an underlying geometric symmetry for deep neural networks, where both deeply learned features and classifiers converge to a simplex equiangular tight frame.
It has been shown that both crossentropy loss and mean square error can provably lead to NC.
Inspired by how
NC characterizes the training target of neural networks, we decouple NC into two objectives: minimal intra-class variability and maximal inter-class separability.
We then introduce the concept of hyperspherical uniformity (which characterizes the degree of uniformity on the unit hypersphere) as a unified framework to quantify these two objectives.
Finally, we propose a generic objective – hyperspherical uniformity gap (HUG), which is defined by the difference between inter-class and intra-class hyperspherical uniformity.
HUG not only provably converges to NC, but also decouples NC into two separate objectives.
Unlike cross-entropy loss that couples intra-class compactness and inter-class separability, HUG enjoys more flexibility and serves as a good alternative loss function.
Empirical results show that HUG works well in terms of generalization, calibration and robustness.
I NTRODUCTION
Recent years have witnessed the great success of deep representation learning in a variety of applications ranging from computer vision [30], natural language processing [12] to game playing [46, 52].
Despite such a success, how deep representations can generalize to unseen scenarios and when they might fail remain a black box.
Deep representations are typically learned by a multi-layer network with cross-entropy (CE) loss optimized by stochastic gradient descent.
In this simple setup, [67] has shown that zero loss can be achieved even with arbitrary label assignment.
After continuing to train the neural network past zero loss with CE, [49] discovers an intriguing phenomenon called neural collapse (NC).
NC can be summarized as the following characteristics:
• Intra-class variability collapse : Intra-class variability of last-layer features collapses to zero, indicating that all the features of the same class concentrate to their intra-class feature mean.
• Convergence to simplex ETF : After being centered at their global mean, the class-means are both linearly separable and maximally distant on a hypersphere.
Formally, the class-means form a simplex equiangular tight frame (ETF) which is a symmetric structure defined by a set of maximally distant and pair-wise equiangular points on a hypersphere.
• Convergence to self-duality : The linear classifiers, which live in the dual vector space to that of the class-means, converge to their corresponding class-mean and also form a simplex ETF.
• Nearest decision rule :
The linear classifiers behave like nearest class-mean classifiers.
The NC phenomenon suggests two general principles for deeply learned features and classifiers: minimal intra-class compactness of features ( i.e. , features of the same class collapse to a single point), and maximal inter-class separability of classifiers / feature mean ( i.e. , classifiers of different classes have maximal angular margins).
While these two principles are largely independent, popular loss functions such as CE and square error (MSE) completely couple these two principles together.
Since there is no trivial way for CE and MSE to decouple these two principles, we identify a novel quantity – hyperspherical uniformity gap (HUG), which not only characterizes intra-class feature compactness and inter-class classifier separability as a whole, but also fully decouples these two principles.
The decoupling enables HUG to separately model intra-class compactness and inter-class separability, making it highly flexible.
More importantly, HUG can be directly optimized and used to train neural networks, serving as an alternative loss function in place of CE and MSE for classification.
HUG is formulated as the difference between inter-class and intra-class hyperspherical uniformity.
Hyperspherical uniformity [40] quantifies the uniformity of a set of vectors on a hypersphere and is used to capture how diverse these vectors are on a hypersphere.
Thanks to the flexibility of
HUG, we are able to use many different formulations to characterize hyperspherical uniformity, including (but not limited to) minimum hyperspherical energy (MHE)
[37], maximum hyperspherical separation (MHS) [40] and maximum gram determinant (MGD) [40].
Different formulations yield different interpretation and optimization difficulty ( e.g. , HUG with MHE is easy to optimize, HUG with MGD has interesting connection to geometric volume), thus leading to different performance.
Similar to CE loss, HUG also provably leads to NC under the setting of unconstrained features [44].
Going beyond NC, we hypothesize a generalized NC (GNC) that extends the original NC to the scenario where there is no constraint for the number of classes and the feature dimension.
NC requires the feature dimension no smaller than the number of classes while GNC no longer requires this.
We further prove that HUG also leads to GNC at its objective minimum.
Another motivation behind HUG comes from the classic Fisher discriminant analysis (FDA)
[14] where the basic idea is to find a projection matrix T that maximizes between-class variance and minimizes within-class variance.
What if we directly optimize the input data (without any projection) rather than optimizing the linear projection in FDA?
We make a simple derivation below:
Projection FDA:
[Equation]
where the between-class scatter matrix is S w = (cid:80) Ci =1 (cid:80)
j ∈
A c ( x j − µ i )( x j − µ i ) ⊤ , the within-class scatter matrix is S b =
(cid:80) Ci =1 n i ( µ i − ¯ µ )( µ i − ¯ µ ) ⊤ , n i is the number of samples in the i -th class, n is the total number of samples, µ i = n − 1 i (cid:80)
j ∈
A c x j is the i -th class-mean, and ¯ µ = n − 1 (cid:80) nj
=1 x j is the global mean.
By considering class-balanced data on the unit hypersphere, optimizing data
FDA is equivalent to simultaneously maximizing tr ( S b ) and minimizing tr ( S w ) .
Maximizing tr ( S b ) encourages inter-class separability and is a necessary condition for hyperspherical uniformity.
Minimizing tr ( S w ) encourages intra-class feature collapse, reducing intra-class variability.
Therefore,
HUG can be viewed a generalized FDA criterion for learning maximally discriminative features.
However, one may ask the following questions: Why is HUG useful if we already have the FDA criterion?
Could we simply optimize data FDA?
In fact, the FDA criterion has many degenerate solutions.
For example, we consider a scenario of 10-class balanced data where all features from the first 5 classes collapse to the north pole on the unit hypersphere and features from the rest 5 classes collapse to the south pole on the unit hypersphere.
In this case, tr ( S w ) is already minimized since it achieves the minimum zero.
tr ( S b ) also achieves its maximum n at the same time.
In contrast, HUG naturally generalizes FDA without having these degenerate solutions and serves as a more reliable criterion for training neural networks.
We summarize our contributions below:
• We decouple the NC phenomenon into two separate learning objectives: maximal inter-class separability ( i
e. , maximally distant class feature mean and classifiers on the hypersphere) and minimal intra-class variability ( i.e. , intra-class features collapse to a single point on the hypersphere).
• Based on the two principled objectives induced by NC, we hypothesize the generalized NC which generalizes NC by dropping the constraint on the feature dimension and the number of classes.
• We identify a general quantity called hyperspherical uniformity gap, which well characterizes both inter-class separability and intra-class variability.
Different from the widely used CE loss, HUG naturally decouples both principles and thus enjoys better modeling flexibility.
• Under the HUG framework, we consider three different choices for characterizing hyperspherical uniformity: minimum hyperspherical energy, maximum hyperspherical separation and maximum Gram determinant.
There also exist many other choices that may work equally well.
HUG provides a unified framework for different characterizations of hyperspherical uniformity to supervise neural networks.
We give both theoretical and empirical insights to validate HUG’s effectiveness.
O N G ENERALIZING AND D ECOUPLING N EURAL C OLLAPSE
NC describes an intriguing phenomenon for the distribution of last-layer features and classifiers in overly-trained neural networks, where both features and classifiers converge to ETF.
However, ETF can only exist when the feature dimension d and the number of classes C satisfy
d ≥ C − 1 .
This is not always true for deep neural networks.
For example, neural networks for face recognition are usually trained by classifying large number of classes ( e.g. , more than 85K classes in [17]), and the feature dimension ( e.g. , 512 in SphereFace [36]) is usually much less than the number of classes.
In general, when the number of classes is already large, it is infeasible to use a larger feature dimension.
Thus a natural question arises: what will happen in this case if a neural network is fully trained?
[Figure]
[Figure]
Motivated by this question, we conduct a simple experiment to simulate the case of d ≥ C − 1 and the case of d<C − 1 .
Specifically, we train a convolutional neural network (CNN) on MNIST with feature dimension 2.
For the case of d ≥ C − 1 , we use only 3 classes (digit 0,1,2) as the training set.
For the case of d<C − 1 , we use all 10 classes as the training set.
We visualize the learned features of both cases in Figure 1.
The results verify the case of d ≥ C − 1 indeed approaches to NC, and
ETF does not exist in the case of d<C − 1 .
Interestingly, one can observe that learned features in both cases approach to the configuration of equally spaced frames on the hypersphere.
To accommodate the case of d<C − 1 , we extend NC to the generalized NC by hypothesizing that last-layer inter-class features and classifiers tend to converge to equally spaced points on the hypersphere, which can be characterized by hyperspherical uniformity.
Generalized Neural Collapse (GNC)
We define the feature global mean as µ G =
Ave i,c x i,c where x i,c ∈ R d is the last-layer feature of the i -th sample in the c -th class, the feature class-mean as µ c =
Ave i x i,c for different classes c ∈{ 1 , · · · , C } , the feature within-class covariance as Σ W =
Ave i,c ( x i,c − µ c )( x i,c − µ c ) ⊤ and the feature between-class covariance as Σ B = Ave c ( µ c − µ G )( µ c − µ G ) ⊤ .
GNC states that
• (1) Intra-class variability collapse : Intra-class variability of last-layer features collapse to zero, indicating that all the features of the same class converge to their intra-class feature mean.
Formally, GNC has that Σ † B Σ W → 0 where † denotes the Moore-Penrose pseudoinverse.
• (2) Convergence to hyperspherical uniformity : After being centered at their global mean, the class-means are both linearly separable and maximally distant on a hypersphere.
Formally, the class-means converge to equally spaced points on a hypersphere,
i
e. ,
(cid:88)
[Equation]
where ˆ µ i = ∥ µ i − µ G ∥ − 1 ( µ i − µ G ) .
E =
(cid:80)
c ̸ = c ′ ∥
ˆ µ c − ˆ µ c ′ ∥
− 2 is a variational characterization of hyperspherical uniformity using Coulomb potentials (defined as hyperspherical energy [37]).
• (3) Convergence to self-duality : The linear classifiers, which live in the dual vector space to that of the class-means, converge to their corresponding class-means, leading to hyperspherical uniformity.
Formally, GNC has that ∥ w c ∥ − 1 w c − ˆ µ c → 0 where w c ∈ R
d
is the c -th classifier.
• (4) Nearest decision rule : The learned linear classifiers behave like the nearest class-mean classifiers.
Formally, GNC has that arg max c ⟨ w c , x ⟩ + b c →
arg min c ∥ x − µ c ∥ .
In contrast to NC, GNC further considers the case of d<C − 1 and hypothesizes that both feature class-means and classifiers converge to hyperspherically uniform point configuration that minimizes the Coulomb potential.
We discuss the theoretical properties of GNC by connecting to classical potential theory [3].
We show in Theorem 1 that GNC reduces to NC in the case of d ≥ C − 1 :
Theorem 1 (Regular Simplex Optimum for GNC) Let f : (0 , 4]
→ R be a convex and decreasing function defined at v =0 by lim v → 0 + f ( v ) .
If 2 ≤ C ≤ d +1 , then we have that the vertices of regular ( C − 1) -simplices inscribed in S d − 1 with centers at the origin (equivalent to simplex ETF) minimize the hyperspherical energy (cid:80) c
̸ = c ′ K ( ˆ µ c , ˆ µ c ′ ) on the unit
hypersphere S
d − 1 ( d ≥ 3 ) with the kernel as K (ˆ µ c , ˆ µ c ′ )
=
f ( ∥ ˆ µ c − ˆ µ c ′ ∥ 2 ) .
If f is strictly convex and strictly decreasing, then these are the only energy minimizing C -point configurations.
Thus GNC reduces to NC when d ≥ C − 1 .
vvv-v-v-vvv 2 vv(a)
Regular simplex (b)
[Figure]
polytope
We note that Theorem 2 not only guarantees the equiangular
ETF as the minimizer of the hyperspherical energy in Eq.
1, but also shows that a general family of energies also share the same minimizer (as long as the potential function f is convex and decreasing).
The case of d<C − 1 is where GNC really gets interesting but complicated.
Other than the regular simplex case, we also highlight a special uniformity case of 2 d = C .
In this case, we can prove in Theorem 2 that GNC(2) converges to the vertices of a cross-polytope as hyperspherical energy gets minimized.
As the number of classes gets infinitely large, we show in Theorem 3 that GNC(2) leads to a point configuration that is uniformly distributed on S d − 1 .
Additionally, we show a simple yet interesting result in Proposition 1 that the last-layer classifiers are initialized to be uniformly distributed on the hypersphere.
Theorem 2 (Cross-polytope Optimum for GNC)
If C =2 d , then the vertices of the cross-polytope are the minimizer of the hyperspherical energy in GNC(2).
The cross-polytope optimum for GNC(2) is in fact quite intuitive, because it corresponds to the
Cartesian coordinate system (up to a rotation).
For example, the vertices of the unit cross-polytope in Rare ( ± 1 , 0 , 0) , (0 , ± 1 , 0) , (0 , 0 , ± 1) .
These 6 vectors minimize the hyperspherical energy on S 2 .
We illustrate both the regular simplex and cross-polytope cases in Figure 2.
For the other cases of d<C − 1 , there exists generally no simple point structure that minimizes the hyperspherical energy, as heavily studied in [10, 21, 31, 51].
For the point configurations that asymptotically minimize the hyperspherical energy as C grows larger, Theorem 3 can guarantee that these configurations asymptotically converge to a uniform distribution on the hypersphere.
Theorem 3 (Asymptotic Convergence to Hyperspherical Uniformity) Consider a sequence of point configurations { ˆ µ C 1 , · · · , ˆ µ CC } ∞ C =2 that asymptotically minimizes the hyperspherical energy on S d −as C → ∞ , then { ˆ µ C 1 , · · · , ˆ µ CC } ∞ C
=2 is uniformly distributed on the hypersphere S d − 1 .
Proposition 1 (Minimum Energy Initialization) With zero-mean Gaussian initialization (e.g., [16, 22]), the C last-layer classifiers of neural networks are initialized as a uniform distribution on the hypersphere.
The expected initial energy is C ( C − 1) (cid:82) S
[Equation] [Equation]
[Equation] [Equation]
µ
[Equation]
dσ
[Equation]
1 (ˆ µ
[Equation]
With Proposition 1, one can expect that the hyperspherical energy of the last-layer classifiers will first increase and then decrease to a lower value than the initial energy.
To validate the effectiveness of our GNC hypothesis, we conduct a few experiments to show how both class feature means and classifiers converge to hyperspherical uniformity ( i.e. , minimizing the hyperspherical energy), and how intra-class feature variability collapses to almost zero.
We start with an intuitive understanding about GNC from Figure 1.
The results are directly produced by the learned features without any visualization tool (such as t-SNE [60]), so the feature distribution can reflect the underlying one learned by neural networks.
We observe that GNC is attained in both d<C − 1 and d ≥ C − 1 , while
NC is violated in d<C − 1 since the learned feature class-means can no longer form a simplex ETF.
To see whether the same conclusion holds for higher feature dimensions
[Figure]
we also train two CNNs on
CIFAR-100 with feature dimension as 64 and 128, respectively.
The results are given in Figure
[Figure]
A v e r a g e H yp e r s ph e
r
i ca l E n e r gy
[Figure]
Figure 3 shows that GNC captures well the underlying convergence of the neural network training.
Figure 3(a,c) shows that the hyperspherical energy of feature class-means and classifiers converge to a small value, verifying the correctness of GNC(2) which indicates feature class-means converge to hyperspherical uniformity, and GNC(3) which indicates classifiers also converge to hyperspherical uniformity.
More interestingly, in the MNIST experiment, we can compute the exact minimal energy on S 1 : 2 in the case of d =2 , C =
( 1 / 3 for average energy) and ≈ 82 .
5 in the case of d =2 , C
= ( ≈ 0 .
917 for average energy).
The final average energy in Figure 3(a) matches our theoretical minimum well.
From Figure 3(c), we observe that the classifier energy stays close to its minimum at the very beginning, which matches our Proposition 1 that vectors initialized with zero-mean
Gaussian are uniformly distributed over the hypersphere (this phenomenon becomes more obvious in higher dimensions).
To evaluate the intra-class feature variability, we consider a hyperspherical reverse-energy E r = (cid:80) i ̸
=
j ∈
A c ∥
ˆ x i − ˆ x j ∥ 2 where ˆ x i
= x i ∥ x i ∥ and A c denotes the sample index set of the c -th class.
The smaller this reverse-energy gets, the less intra-class variability it implies.
We plot the reverse-energy in Figure 3(b,d) and show that the intra-class feature variability approaches to zero, as GNC(1) suggests.
We perform a comprehensive empirical study on GNC in Appendix B
[Figure]
After establishing the GNC hypothesis, we discuss how to decouple the GNC phenomenon and how such a decoupling can enable us to propose a new objective to train neural networks.
GNC(1) and GNC(2) suggest to minimize intra-class feature variability and maximize interclass feature separability, respectively. GNC(3) and GNC(4) are in fact natural consequences if GNC(1) and GNC(2)
hold.
It has long been discovered [35, 55, 65] that last-layer classifiers serve as class proxies to represent the corresponding class of features, and they are also an approximation to the feature class-means.
Therefore, GNC(3) is a direct consequence of GNC(1) and GNC(2).
GNC(3) indicates the classifiers converge to hyperspherical uniformity, which, together with GNC(1), implies GNC(4).
[68] has discussed NC under the case of d<C − 1 , and shown that the global solution in this case yields the best rand- d approximation of the simplex ETF.
Along with [68], GNC gives a more comprehensive characterization of the convergence of class-means.
We show a special case of d =2 , C =4 .
It is easy to see that hyperspherical uniformity in this case forms four vectors with adjacency ones being perpendicular.
This is also the case captured by the best rank- 2 approximation ( i.e. , a 2-dimensional hyperplane with simplex ETF projected onto it).
Figure 4 gives a geometric interpretation for the connection between [68] and GNC.
Until now, it has been clear that GNC really boils down to two decoupled objectives: maximize inter-class separability and minimize intra-class variability , which again echos the goal of FDA.
The problem reduces to how to effectively characterize these two objectives while being decoupled for flexibility (unlike CE or MSE).
In the next section, we propose to address this problem by characterizing both objectives with a unified quantity - hyperspherical uniformity.
H YPERSPHERICAL U NIFORMITY G AP 3. G ENERAL F RAMEWORK
As GNC(2) suggests, the inter-class separability is well captured by hyperspherical uniformity of feature class-means, so it is natural to directly use it as a learning target.
On the other hand,
GNC(1) does not suggest any easy-to-use quantity to characterize intra-class variability.
We note that minimizing intra-class variability is actully equivalent to encouraging features of the same class to concentrate on a single point, which is the opposite of hyperspherical uniformity.
Therefore, we can unify both intra-class variability and inter-class separability with a single characterization of hyperspherical uniformity.
We propose to maximize the hyperspherical uniformity gap:
[Equation]
where α, β are hyperparameters, ˆ µ c = µ c ∥ µ c ∥ is the feature class-mean projected on the unit hypersphere, µ c = (cid:80) c
∈
A c x c is the feature class-mean, x i is the last-layer feature of the i -th sample and
A c denotes the sample index set of the c -th class.
HU ( { v i } mi =1 ) denotes some measure of hyperspherical uniformity for vectors { v 1 , · · · , v m } .
Eq.
2 is the general objective for HUG.
Without loss of generality, we assume that the larger it gets, the stronger hyperspherical uniformity we have.
We mostly focus on supervised learning with parameteric class proxies 2 where the CE loss is widely used as a de facto choice, although HUG can be used in much broader settings as discussed later.
In the HUG framework, there is no longer a clear notion of classifiers (unlike the CE loss), but we still can utilize class proxies ( i.e. , a generalized concept of classifiers) to help with the optimization.
We can observe that Eq.
2 directly optimizes the feature class-means for inter-class separability, but they are usually inefficient to compute during training (we need to compute them in every iteration).
Therefore it is nontrivial to optimize the original HUG for training neural networks.
A naive solution is to approximate feature class-mean with a few mini-batches such that the gradients of inter-class hyperspherical uniformity ( T b ) can be still back-propagated to the last-layer features.
However, it may takes many mini-batches in order to obtain a sufficiently accurate class-mean, and the approximation gets much more difficult with large number of classes.
To address this, we employ parametric class proxies to act as representatives of intra-class features and optimize them instead of feature class-means.
To make these parametric proxies become representatives of each class, we need to connect the class proxies to the features such that the gradient of T b can have an influence on the features.
To achieve this, we modify the HUG objective to cope with parametric class proxies:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where ˆ w c ∈ S
d − 1 is the learnable parametric proxy for the c -th class.
The intra-class hyperspherical uniformity term connects the class proxies with features by minimizing the joint hyperspherical uniformity of intra-class features and their class proxy, guiding the intra-class features to move towards their corresponding class proxy.
When training a neural network, Eq. 3 will instead optimize network weights and proxies (the gradients w.r.t.
features will be back-propagated to previous layers).
Learnable proxies .
We can view the class proxy ˆ µ i as learnable parameters and update them with stochastic gradients, similarly to the parameters of neural networks.
In fact, learnable proxies play a role similar to the last-layer classifiers in the CE loss, improving the optimization by aggregating intra-class features.
The major difference between learnable proxies and moving-averaged proxies is the way we update them.
As GNC(3) implies, class proxies in HUG can also be used as classifiers.
Static proxies .
Eq.
3 is decoupled into maximal inter-class separability and minimal intra-class variability.
These two objects are independent and do not affect each other.
We can thus optimize them independently.
This suggests a even simpler way to assign class proxies – initializing class proxies with prespecified points that have attained hyperspherical uniformity, and fixing them in the training.
There are two simple ways to obtain these class proxies: (1) minimizing their hyperspherical energy beforehand; (2) using zero-mean Gaussian to initialize the class proxies (Proposition 1 guarantees the probabilistic hyperspherical uniformity).
After initialized with minimum energy, these class proxies will stay fixed and only the features are optimized towards their class proxies.
Partially learnable proxies .
After the class proxies are initialized using the static way above, we can increase its flexibility by learning an orthogonal matrix for the class proxies to find a suitable orientation for them.
Specifically, we can learn this orthogonal matrix using methods in [39].
3. V ARIATIONAL C HARACTERIZATIONS OF H YPERSPHERICAL U NIFORMITY
While there exist many choices to measure hyperspherical uniformity, we seek variational characterizations due to simplicity.
As examples, we consider minimum hyperspherical energy [37] which is inspired by Thomson problem [54, 57] and measures hyperspherical uniformity by Coulomb potential energy, maximum hyperspherical separation [40] which is inspired by Tammes problem [56] and captures hyperspherical uniformity by the smallest pairwise distance, and maximum gram determinant [40] which measures hyperspherical uniformity with the volume of the formed parallelotope.
Minimum hyperspherical energy .
MHE seeks to find an equilibrium state with minimum potential energy that distributes n electrons on a unit hypersphere as evenly as possible.
Hyperspherical uniformity is characterized by minimizing the hyperspherical energy for n vectors V n = { v 1 , · · · , v n ∈ R d } :
[Equation]
where ˆ v i := v i ∥ v i ∥ is the i -th vector projected onto the unit hypersphere.
With HU ( ˆ V )=
− E s ( ˆ V ) , we apply MHE to HUG and formulate the new objective as follows ( s b =2 , s w = − 1 ):
[Equation]
which can already be used as to train neural networks.
The intra-class variability term in Eq.
5 can be relaxed to a lower bound such that we can instead minimize a simple upper bound of L MHE-HUG :
[Equation]
which is much more efficient to compute in practice and thus can serve as a relaxed HUG objective.
Moreover, L MHE-HUG and L
′ MHE-HUG share the same minimizer.
Detailed derivation is in Appendix H.
Maximum hyperspherical separation .
MHS uses a maximum geodesic separation principle by maximizing the separation distance ϑ ( ˆ V n ) ( i.e. , the smallest pairwise distance in V n = { v 1 , · · · , v n ∈
R d } ): max ˆ V { ϑ ( ˆ V n ) :
= min i ̸ =
j ∥
ˆ v i − ˆ v j ∥} .
Because ϑ ( ˆ V n ) is another variational definition, we cannot naively set HU ( · ) = ϑ ( · ) .
We define ϑ − 1 ( ˆ V n ):=max i ̸
= j ∥ ˆ v i − ˆ v j ∥ and HUG becomes
[Equation]
which, by replacing intra-class variability with its surrogate, results in a more efficient form:
[Equation]
which is a max-min optimization with a simple nearest neighbor problem inside.
We note that
L MHS-HUG and L ′ MHS-HUG share the same maximizer.
Detailed derivation is in Appendix H.
Maximum gram determinant .
MGD characterizes the uniformity by computing a proxy to the volume of the parallelotope spanned by the vectors.
MGD is defined with kernel gram determinant:
[Equation]
where we use a simple Gaussian kernel with parameter ϵ and define G ( { ˆ v 1 , · · · , ˆ v n } ) as the kernel gram matrix for ˆ V n = { ˆ v 1 , · · · , ˆ v n } .
With HU ( ˆ V n )=det G ( ˆ V n ) , we have the HUG objective as
[Equation]
(10) where we drop log( · ) for the intra-class variability term to avoid numerical issues.
With MGD, HUG has interesting geometric interpretation – it encourages the volume spanned by class proxies to be as large as possible and the volume spanned by intra-class features to be as small as possible.
3. T HEORETICAL I NSIGHTS AND D ISCUSSIONS
There are many interesting theoretical questions concerning HUG, and this framework is highly related to a few topics in mathematics, such as tight frame theory [61], potential theory [32], sphere packing and covering [3, 13, 19].
The depth and breath of these topics are beyond imagination.
In this section, we focus on discussing some highly related yet intuitive theoretical properties of HUG.
Theorem 4 (Order of Minimum Hyperspherical Energy)
If d − 1 >s> 0 or − 2 <s< 0 and d ∈
N , we have that lim n →∞ { n − 2 · min ˆ V n E s ( ˆ V n ) } =
c ( s, d ) where c ( s, d ) is a constant involving s, d .
The result above shows that the leading term of the minimum energy grows of order O ( n 2 ) as n →∞ .
Theorem 4 generally holds with a wide range of s for the Riesz kernel in hyperspherical energy.
Moreover, the following result shows that MHS is in fact a limiting case of MHE as s →∞ .
Proposition 2 (MHS is a Limiting Case of MHE) Let n ∈ N , n ≥ 2 be fixed and ( S d − 1 , L 2 ) be a compact metric space.
We have that lim s →∞ (min ˆ V n ⊂ S
d − 1 E s ( ˆ V n ))
1 s = (max ˆ V n ⊂ S d − 1 ϑ ( ˆ V n ))
− 1 .
Proposition
The HUG objectives in both Eq.
5 and Eq.
6 converge to simplex ETF when 2 ≤ C ≤ d + 1 , converge to cross-polytope when C = 2 d and asymptotically converge to GNC as C → ∞ .
Proposition 3 shows that HUG not only decouples GNC but also provably converges to GNC.
Since
GNC indicates that the CE loss eventually approaches to the maximizer of HUG, we now look into how the CE loss implicitly maximizes the HUG objective in a coupled way.
Proposition 4 The CE loss is L CE =
(cid:80) ni
=1 log(1+ (cid:80) Cj ̸
= c i exp( ⟨ w j , x i ⟩−⟨ w c i , x i ⟩ )) where n is the number of samples, x i is the i -th sample with
label c
i and w c is the last-layer linear classifier for the j -th class.
Bias is omitted for simplicity.
L CE is bounded by ( ρ = C − 1 ) n (cid:88)
[Equation]
From Proposition 4, we can see that the CE loss inherently couples intra-class variability (IV) and inter-class separability (IS).
With normalized classifiers and features, we can see that Q 1 and Q 3 have the same minimum
where x i = w c i and w i , ∀ i attain hyperspherical uniformity (Appendix K).
[41] has proved that the minimizer of a normalized form of the CE loss converges to hyperspherical uniformity.
Built upon this nice result, we can easily extend it to the following theorem:
Theorem 5 (CE Asymptotically Converges to HUG’s Maximizer)
Considering unconstrained features of C classes (each class has the same number of samples), with features and classifiers normalized on some hypersphere, we have that, for the minimizer of the CE loss, classifiers converge weakly to the uniform measure on S d − 1 as C →∞ and features collapse to their corresponding classifiers.
The minimizer of CE also asymptotically converges to the maximizer of HUG.
Theorem 5 shows that the minimizer of the CE loss with unconstrained features [44] asymptotically converges to the maximizer of HUG ( i.e. , GNC).
Till now, we show that HUG shares the same optimum with CE (with hyperspherical normalization), while being more flexible for decoupling inter-class feature separability and intra-class feature variability.
Therefore, we argue that HUG can be an excellent alternative for the widely used CE loss in classification problems.
The role of feature and class proxy norm .
Both NC and GNC do not take the norm of feature and class proxy into consideration.
HUG also assume both feature and class proxy norm are projected onto some hypersphere.
Although dropping these norms usually improves generalizability [7, 8, 11, 36, 62], training neural networks with standard CE loss still yields different class proxy norms and feature norms.
We hypothesize that this is due to the underlying difference among training data distribution of different classes.
One empirical evidence to support this is that average feature norm of different classes is consistent across training under different random seeds ( e.g. , average feature norm for digit 1 on MNIST stays the smallest in different run).
[29, 38, 43] empirically show that feature norm corresponds to the quality of the sample, which can also viewed as a proxy to sample uncertainty.
[47] theoretically shows that the norm of neuron weights ( e.g. , classifier) matters for its Rademacher complexity.
As a trivial solution to minimize the CE loss, increasing the classifier norm (if the feature is correctly classified) can easily decrease the CE loss to zero for this sample, which is mostly caused by the softmax function.
Taking both feature and class proxy norm into account greatly complicates the analysis ( e.g. , it results in weighted hyperspherical energy where the potentials between vectors are weighted) and seem to yield little benefit.
We defer this to future investigation.
Connection to contrastive learning .
The goal of contrastive learning [7, 18, 24, 59, 64] is to learn discriminative features through instance-wise discrimination and contrast.
Despite the lack of class labels, [64] discovers that contrastive learning performs sample-wise alignment and sample-wise uniformity, sharing a high-level spirit to intra-class variability and inter-class separability.
HUG as a general framework for designing loss functions .
HUG can be viewed as an inherently decoupled way of designing new loss functions.
As long as we design a measure of hyperspherical uniformity, then HUG enables us to effortlessly turn it into a loss function for neural networks.
We show that, both theoretically and empirically, HUG delivers better loss functions than the CE loss.
E XPERIMENTS AND R ESULTS
Our experiments aims to demonstrate the empirical effectiveness of HUG, so we focus on the fair comparison to the popular CE loss under the same setting.
Experimental details are in Appendix A.
4. E XPLORATORY E XPERIMENTS AND A BLATION S TUDY
[Table] [Table]
Different HUG variants .
We compare different HUG variants and the
CE loss on CIFAR-10 and CIFAR-100 with ResNet-18 [23].
Specifically, we use Eq.
6, Eq. 6 and Eq.
10 for MHE-HUG, MHS-HUG and MGD-HUG, respectively.
The results are given in Table 1.
We can observe that all HUG variants outperform the CE loss.
Among all, MHE-HUG achieves the best testing accuracy with considerable improvement over the CE loss.
We note that all HUG variants are used without the CE loss.
The performance gain of HUG are actually quite significant, since the CE loss is currently a default choice for classification problems and serves as a very strong baseline.
[Table]
Different methods to update proxies .
We also evaluate how different proxy update methods will affect the classification performance.
We use the same setting as Table 1.
For all the proxy update methods, we apply them to MHE-HUG (Eq. 6) under the same setting.
The results are given Table 2.
We can observe that all the propose proxy update methods work reasonably well.
More interestingly, static proxies work surprisingly well and outperform the CE loss even when all the class proxies are randomly initialized and then fixed throughout the training.
The reason the static proxies work for MHE-HUG is due to Proposition 1.
This result is significant since we no longer have to train class proxies in
HUG (unlike CE).
When trained with large number of classes, it is GPU-memory costly for learning class proxies, which is also known as one of the bottlenecks for face recognition [1].
HUG could be a promising solution to this problem.
[Figure] [Figure]
[Figure]
Loss landscape and convergence .
We perturb neuron weights (refer to [33]) to visualize the loss landscape of HUG and CE in Figure 5.
We use MHE in HUG here.
The results show that HUG yields much flatter local minima than the CE loss in general, implying that HUG has potentially stronger generalization [28, 48].
We show more visualizations and convergence dynamics in Appendix M.
[Table]
Learning with different architectures .
We evaluate HUG with different network architectures such as VGG-16 [53],
ResNet-18 [23] and DenseNet-121 [25].
Results in Table(Left number: CIFAR-10, right number: CIFAR-100) show that HUG is agnostic to different network architectures and outperforms the CE loss in every case.
Although HUG works well on its own, any other methods that improve CE can also work with HUG.
4. G ENERALIZATION AND A DVERSARIAL R OBUSTNESS
[Table]
Long-tailed recognition .
We consider the task of long-tailed recognition, where the data from different classes are imbalanced.
The settings generally follow [5], and the dataset gets more imbalanced if the imbalance ratio (IR) gets smaller.
The potential of HUG in imbalanced classification is evident, as the inter-class separability in the HUG is explicitly modeled and can be easily controlled.
Experimental results in Table 4 show that HUG can consistently outperform the CE loss in the challenging long-tailed setting under different imbalanced ratio.
[Table]
Continual learning .
We also demonstrate the potential of HUG in the class-continual learning setting, where the training data is not sampled i.i.d.
but comes in class by class.
Since training data is highly biased, hyperspherical uniformity among class proxies is crucial.
Due to the decoupled nature of HUG, we can explicitly increase the importance of inter-class separability, unlike the CE loss.
We use a simple continual learning method – ER [50] where the CE loss with memory buffer is used.
We simply replace it with HUG.
Table 5 shows HUG is able to consistently improve the performance of ER method under different size of memory buffer.
[Table]
Adversarial robustness .
We further test HUG’s adversarial robustness.
In our experiments, we consider the classical white-box PGD attack [42] on ResNet18.
The PGD attack iteration is set as 100 and the attack strength level is set as 2/255, 4/255, 8/255 in l ∞ norm.
All networks are naturally training with either HUG or the CE loss.
Results in Tabledemonstrates that HUG yields consistently stronger adversarial robustness than the CE loss.
R ELATED W ORK AND C ONCLUDING R EMARKS
We start by generalizing and decoupling the NC phenomenon, obtaining two basic principles for loss functions.
Based on these principles, we identify a quantity hyperspherical uniformity gap, which not only decouples NC but also provides a general framework for designing loss functions.
We demonstrate a few simple HUG variants that outperform the CE loss in terms of generalization and adversarial robustness.
There is a large body of excellent work in NC that is related to HUG, such as [20, 27, 58, 70].
[69] extends the study of NC to more practical loss functions ( e.g. , focal loss and losses with label smoothing).
Different from existing work in hyperspherical uniformity [34, 37, 40] and generic diversity (decorrelation) [2, 6, 9, 45, 63, 66], HUG works as a new learning target (used without CE) rather than acting as a regularizer for the CE loss (used together with CE).
Following the spirit of [26], we demonstrate the effectiveness of HUG as a valid substitute for CE.