On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Data
Abstract
It has been empirically observed that training large models with weighted cross-entropy (CE) beyond the zero-training-error regime is not a satisfactory remedy for label-imbalanced data.
Instead, researchers have proposed the vector-scaling (VS) loss, as a parameterization of the
CE loss that is tailored to this modern training regime.
The driving force to understand the impact of such parameterizations on the gradient-descent path has been the theory of implicit bias .
Specifically for linear(ized) models, this theory allows to explain why weighted CE fails and how the VS-loss biases the optimization path towards solutions that favor minorities.
However, beyond linear models the description of implicit bias is more obscure.
In order to gain insights on the impact of different CE-parameterizations in non-linear models, we investigate their implicit geometry of learned classifiers and embeddings.
Our main result characterizes the global minimizers of a non-convex cost-sensitive SVM classifier for the so-called unconstrained features model, which serves as an abstraction of deep models.
We also study empirically the convergence of SGD to this global minimizer observing slow-downs with increasing imbalance ratios and scalings of the loss hyperparameters.
In deep-nets, we show preliminary results on the empirical convergence to the predicted geometry.
1. Introduction
It has been shown that in the modern overparameterized regime, where training continues beyond zero-training error, traditional techniques, such as oversampling minorities or minimizing a weighted cross-entropy (CE) loss can be ineffective in mitigating label-imbalances [1, 22].
In a growing literature, several alternatives have been proposed to guarantee equitable performance across majorities and minorities, e.g. [2, 11–15, 18, 27].
Among these, the vector-scaling (VS)-loss [14, 27] introduces multiplicative hyper-parameters on the logits of the CE loss.
The idea behind this parameterization is rooted in the theory of implicit bias , which seeks characterizing the bias introduced by gradient-based algorithms during training [9, 17, 23].
Specifically for binary linear models, [14] uncovers a favorable bias of the
VS-loss towards classifiers with larger margin for the minority.
However, this leaves open the question how the VS-loss changes the learnt model in non-linear settings where embeddings and classifiers are jointly learnt.
Unfortunately, implicit bias characterizations for non-linear models are more obscure compared to the linear case [10, 17].
In particular, it is unclear how to gain from them concrete insights on the way the learnt models affect minorities.
This paper undertakes the investigation of the implicit geometry of classifiers and embeddings learnt by CE-parameterizations when training on imbalanced data.
The notion
[Figure]
of implicit geometry, 1 pioneered by [20] and further investigated by [3–8, 16, 19, 24–26, 28, 29], is intimately related to that of implicit bias.
On the one hand, it is more restrictive as it focuses only on the classifiers and on the embeddings, rather than the weights of the entire model.
Also, it is insensitive to the specific architecture or dataset.
On the other hand, it offers a more explicit characterization that describes the involved geometry of the weights and promises to be “cross-situationally invariant” across architectures and datasets [20].
Contributions.
We study two parameterizations of the CE loss.
First, the class-dependent temperature (CDT)-loss [27], which is a special case of the VS-loss [14].
Second, the label-dependent temperature (LDT)-loss, which we introduce here as an alternative to the
CDT-loss.
For both losses, we investigate the implicit geometry of learnt features and classifiers when training on label-imbalanced data without explicit regularization beyond zero training error.
To do this, we rely on the unconstrained features model (UFM) [3, 19], a proxy for large overparameterized models, that has been used recently to study the implicit geometry of the CE loss [8, 24, 28, 29].
Relying on implicit-bias results, we relax the question of implicit geometry of the solutions found by SGD, to a question about the geometry of the global minimizers of a non-convex cost-sensitive support-vector machines (SVM) problem.
Our main result characterizes the global minimizers of this problem, which takes different form for the CDT and LDT losses.
The characterization is explicit in terms of the number of classes, the minority and imbalance ratio, and the hyperparameters of the losses.
It also leads to closed-form expressions for the norms and pairwise angles of the learned geometries.
This allows identifying how to tune the hyperparameters to induce symmetry in the geometry (see Fig. 1).
We also show numerically that SGD training on the UFM converges to the uncovered geometries.
However, we observe that convergence slows down for increasing values of the imbalance ratio and of the hyperparameters.
We believe this motivates further theoretical and algorithmic investigations towards faster training.
Finally, as evidence of the utility of our geometry characterizations for the UFM, we present preliminary results on deep-learning architectures and complex imbalanced datasets.
2. Background
The VS-loss is the following parameterization of the CE-loss [14]:
[Equation]
Here x i , i ∈ [ n ] are n examples, h θ ( . ) is the feature map parameterized by trainable parameters θ (weights of the hidden layers of a neural network),
y i ∈
[ k ] are the labels, and w c , c ∈ [ k ] are the classifier vectors (head of the network) in a k -class classification setting.
The parameters δ c , and ι c , c ∈
[ k ] are multiplicative and additive hyperparameters, respectively.
Setting δ c = 1 , ι c = 0 , ∀ c ∈ [ k ] recovers the CE loss.
Prior art: Binary linear classification.
In a binary setting with fixed feature map (i.e. non-trainable θ ) [14] studies the implicit bias of gradient descent (GD) on binary VS loss.
Proposition 1 ([14]) For fixed feature map h θ , v i ∈ {± 1 } , and h i ∶=
h
θ ( i ) , i ∈ [ n ] , GD with sufficiently small learning rate on the binary VS-loss L VS,binary ( w ) ∶=
∑ i ∈[ n ] log ( 1 + e − δ vi v i w T h
i + ι vi ) , converges (asymptotically in the number of training steps) in direction to the
Cost-Sensitive SVM (CS-SVM) classifiers: arg min w ∥ w ∥ 22 subj.
to v i δ v i
w T h i ≥ 1 , i ∈ [ n ] .
Prop. 1 explicitly describes how the hyperparameters affect training asymptotically: the
GD path is implicitly biased towards a classifier that assigns margins to the two classes with relative ratio δ − 1 / δ + 1 .
Thus, tuning δ − 1 > δ + 1 if class v = + 1 is minority, favors the minority by assigning larger margin to it [14].
Note, the additive hyperparameters ι c do not have any effect on the implicit bias asymptotically.
Our focus here is on the asymptotic training regime, hence onwards we restrict attention to the multiplicative hyperparameters δ y .
Open problem: Beyond linear models.
Prop. 1 is limited to a setting with fixed features.
While an extension of the loss itself to the learned-feature setting is easy to heuristically derive (see Eqn.
, it is an open question to explicitly characterize the effect of the hyperparameters on the learnt solution.
For instance, how do they affect the relative margin between majorities and minorities or between minorities vs minorities?
3. An implicit-geometry view
To better understand the impact of different CE modifications, we propose studying their implicit geometry, i.e. the geometry of classifiers and embeddings learned (asymptotically in the number of training steps) by GD.
For this, we adopt the unconstrained features model (UFM) [3, 19].
To describe the model, let W d × k
=
[ w 1 , w 2 , ⋯ , w k ] and H d × n =
[ h 1 , h 2 , ⋯ , h n ] be the matrix of k classifier weights and n feature embeddings corresponding to each example in the training set.
Here, d ≥ k − 1 is the feature dimension.
We assume each class c ∈ [ k ] has n c ≥ 1 examples so that ∑ c ∈[ k ] n c = n. Without loss of generality, we assume examples are ordered, i.e. samples i = 1 , . . .
, n 1 have label y i
= 1 and so on.
In the UFM the features h
i , i ∈ [ n ] are trained jointly with the weights w c , c ∈ [ k ] and are trained unconstrained , i.e. without abiding by an explicit parameterization by some weight vector θ (as in (1)).
Previous works use the UFM to study the implicit geometry of CE loss on balanced (e.g.
[3, 19, 28, 29]), and more recently, on imbalanced data [24].
Instead, we study different parameterizations of the CE loss.
CDT/LDT losses on the UFM.
Consider training the UFM by minimizing the following two parameterization of the CE loss:
[Equation]
Both losses are parameterized by a vector δ =
[ δ 1 , δ 2 , . . .
, δ k ] T ∈ R k of positive multiplicative hyperparameters, i.e. δ > 0 .
The CDT loss in (2a) was previously introduced by [14, 27] (note this is a special case of (1) when ignoring the additive ι c ).
Here, we also introduce the
LDT loss in (2b) as an alternative parameterization.
Observe the following subtle distinction:
CDT associates the hyperparameters δ with the class label of the classifier vectors w c , while
LDT assigns the same δ to the label of the feature vector h i .
Unconstrained-features cost-sensitive SVM.
We minimize the losses in (2) without explicit regularization.
Note that in the UFM, minimization over the embedding map is not parameterized in terms of θ , as say in (1).
Thus, the minimization is (joint) over classifiers W and embeddings H .
Specifically, consider performing this minimization using gradient flow (i.e. GD with infinitesimal step-size.)
Then, by interpreting the UFM as a two-layer linear model (e.g. see [24]) it can be shown following [17] that gradient flow will converge (asymptotically in time) in direction to a KKT point of the following two non-convex minimizations for CDT and LDT losses respectively:
[Equation]
Note the resemblence of the above to the CS-SVM minimization of Prop. 1.
But unlike in
Prop. 1, the problems here are non-convex since minimization is also over H .
We refer to (3) as CS-SVM.
Our main result characterizes their global solutions.
4. Global structure of the CS-SVM
In this section, we use M d × k = [ µ 1 , ⋯ , µ k ] to denote the mean-embeddings of H , i.e. µ c = ( 1 / n c )
∑ i ∶ y i =
c h
i , ∀ c ∈ [ k ] .
For simplicity, we focus on a ( R, ρ )-STEP-imbalanced setting, where we have ρk minority classes with n min samples each, and ρk =
1 − ρ ) k majority classes with Rn min samples.
In this case, it is reasonable to assume that δ also shares this STEP structure, i.e. for majorities δ c
= δ maj > 0 and for minorities δ c = δ min > 0.
We describe the geometry of the unconstrained CS-SVM solutions ( W ∗ , H ∗ ) of (3a) and (3b) in terms of an encoding matrix and a geometry induced by the SVD factors of this matrix.
Definition 1 For hyperparameters δ ∈ R k + , minority fraction ρ ( ¯ ρ ∶= 1 − ρ ), k classes, and a rational imbalance ratio R 2 , the ( δ , R ) -Simplex-Encoding Label (SEL) matrix and the corresponding Simplex-Encoded-Labels Interpolation (SELI) geometry are defined as follows:
(a) ( δ , R ) -SEL matrix.
Define Ξ k × k such that ∀ c, j ∈ [ k ] ,
[Equation]
Choose α ∈ N such that αR ∈ N , and let n ∶= αk ( Rρ + ρ ) .
Then, the ( δ , R ) -SEL matrix, along with its SVD, is defined as follows,
[Equation]
where Λ ∈ R ( k − 1 )×( k − 1 ) + is a diagonal matrix and U , V ∈ R k ×( k − 1 ) have orthonormal columns.
(b) ( δ , R ) -SELI geometry.
The classifier and mean-embeddings matrices W , M ∈ R d × k follow the ( δ , R )
-SELI geometry if the following conditions are satisfied.
[Equation]
The ( δ , R ) -SELI geometry is specified by the SVD factors of the corresponding SEL matrix.
The first two properties describe the relative norms and pair-wise angles of classifiers/mean embeddings, and the third one determines the relative margins between classes.
The pattern of the ( δ , R ) -SEL matrix is determined by R and δ , as well as ρ and k .
However, we choose to drop the latter dependence in the name, since they are understood from the context and our results focus on the role of R and δ .
When δ = 1 k , ˆ Z takes a special form: it reduces to a matrix with entries 1 − 1 / k and − 1 / k , which [24] calls the SEL matrix and shows that it characterize the implicit geometry of the CE loss for imbalanced data, i.e. SELI geometry [24].
Further, considering the balanced case R = 1, the ETF geometry [20] is recovered.
Our definition is strictly more general allowing us to describe the implicit geometry learned by
CDT/LDT losses.
We note that ˆ Z T diag ( δ ) − 1 1 k = 0.
Thus, rank ( ˆ Z ) =
k − 1.
Theorem 2 Suppose d ≥ k − 1 in an ( R, ρ ) -STEP imbalance setting.
Let ( W ∗ , H ∗ ) be any minimizers of either (3a) or (3b) , and M ∗ be the optimal class-wise mean-embeddings.
Then, all embeddings collapse to their class means, i.e., ∀ i ∈ [ n ] , h ∗ i = µ ∗ y
i .
Also,
(i) [CDT (3a) ] ( W ∗ , M ∗ ) follow the ( δ , R )
-SELI geometry.
(ii) [LDT (3b) ] ( W ∗ , M ∗ diag ( δ )) follow ( 1 k , ˜ R ) -SELI geometry, where ˜ R ∶= R ( δ min / δ maj ) 2 , provided ˜ R is rational.
Thm. 2 describes the geometry of both classifiers and embeddings that correspond to the solutions of the non-convex CS-SVM for either CDT or LDT.
From statements (i) and (ii) we can find the optimal classifiers and mean-embeddings, their norms (up to a constant) and pair-wise angles in terms of the geometry in Defn.
It is also easy to see that the geometry only depends on the ratio ∆ ∶=
δ maj / δ min and not on the absolute magnitude δ .
The special case of δ = 1 k and R = 1 recovers the SELI [24] and ETF [20] geometry.
For general R and tuning of δ , the LDT/CDT geometries are different than both the SELI and
ETF geometries.
We visualize changes in the geometry in Fig.
Closed-form geometry formulas.
Explicit computation of the SVD factors of the ( δ, R ) -SEL matrix yields closed-form expressions for the norms and pair-wise angles of the mean-embeddings and classifiers.
These formulas can then be evaluated in the limit R → ∞ to give insights for mitigating collapse of minority classifiers into each other, called the minority collapse [3].
Special tunings.
Two notable special cases arise when setting δ c = √ n c , as shown below:
Corollary 3 ( ∆ = √ R : Alignment with CDT, ETF with LDT)
(i)
In (3a) , set ∆ ∶= δ maj / δ min = √ R .
Then, cos ( w ∗ y
i , h ∗ i ) = 1 , ∀ i ∈ [ n ] , i.e. each feature embedding h ∗ i perfectly aligns with its corresponding classifier w ∗
y i .
(ii)
In (3b) , set ∆ = √ R .
Then, ( W ∗ , M ∗ diag ( δ )) follows the ETF geometry.
In this case, the geometry of LDT is exactly the same (for all values of R ) as the perfectly symmetric ETF geometry modulo a scaling on the norms of the embeddings (minority norms are larger than majorities by a factor of √ R ).
See Fig.
1 for an illustration.
Numerical experiments.
We investigate convergence of SGD trajectory for unregularized
CDT/LDT-loss in (2a)/(2b) to the implicit geometries in Thm.
2 for both UFM and deepnets with standard datasets.
In both cases, we parameterize ∆ =
R γ with γ ∈ [− 1 . 5 , 1 . 5 ] .
For each choice of γ and loss function, we compute the average norm ratios and angles between each pair of majority and minority classifiers/embeddings and compare it to CDT/LDT geometries.
Fig.
2 shows the results for ( R = 10 , ρ = 1 / 2 ) -STEP imbalanced UFM, while
Fig.
3 in the SM shows the results for ResNet-18 model on imbalanced CIFAR-10 dataset.
The details of the experiments are given in the SM.
Additional theoretical and experimental results are available in the long version of the paper.
CS-SVM SGD
ETF
Norm Ratios
Majority Angles
Minority Angles Majority/Minority Angles
C D
T ∥
w m a j ∥ 2 / ∥
w m i n o r
∥∥
h m
a j ∥ 2 / ∥
h m i n o r ∥ c o s ( w m a j , w m a j ) c
o s ( h m a j , h m a j ) c
o s ( w m i n o r , w m i n o r ) c
o s ( h m i n o r , h m i n o r ) c
o s ( w m a j , w m i n o r ) c
o s ( h m a j , h m i n o r ) γ γ γ γ     1.0 0.
CS-SVM SGD
ETF
Norm Ratios
Majority Angles
Minority Angles Majority/Minority Angles
L D
T ∥
w m a j ∥ 2 / ∥
w m i n o r
∥∥
h m
a j ∥ 2 / ∥
h m i n o r ∥ c o s ( w m a j , w m a j ) c
o s ( h m a j , h m a j ) c
o s ( w m i n o r , w m i n o r ) c
o s ( h m i n o r , h m i n o r ) c
o s ( w m a j , w m i n o r ) c
o s ( h m a j , h m i n o r ) γ γ γ γ
Concluding remarks.
In this work, we extend the scope of the geometry characterizations of the embeddings and classifiers learned by deep-nets initiated by [20].
We study the geometry of deep-nets trained on imbalanced data with CE parameterizations.
We use the UFM for uncovering geometries in deep-nets trained with CDT and LDT losses.
We encountered the following challenges when showing convergence empirically: the level of convergence that can be reached in realistic training settings generally varies between architectures, data models and the loss that is optimized.
For example, we find that CDT classifier geometry converges very well to its prescribed limit, but the same is not true for the embeddings geometry for the same loss or for the classifiers geometry for the LDT.
Consistently, the experiments in [20] show different levels of convergence between different metrics (e.g. classifiers vs embeddings, norms vs angles) and different architectures/datasets.
Similar to the observation by [24] of slow convergence under high imbalance ratios, our results indicate worse convergence under “strong” parameterizations of CE as well.
Despite these limitations, our results indicate that our characterization of the geometries with these losses is remarkably well predictive of the behavior in deep-nets.
Paraphrasing [20]: one might anticipate that the classifier and embeddings being by-product of training in such complex environments display no underlying structure.
In view of this, we find the level of agreement of the empirically measured angles/norms to the respective (closed-form) ( δ , R ) -SELI geometry values rather striking.
Appendix A. Experiment details
In this section, we provide additional details and discussions on our experiments.
A.1. UFM experimental setting
We investigate convergence of SGD trajectory for unregularized CDT/LDT loss in (2a)/(2b) to the implicit geometries in Thm. 2.
We train the UFM as a two-layer network (no biases) with n = 275 inputs, d = 20 hidden units and k = 10 classes, trained on the basis vectors in
R n .
We measure the properties (i.e norms and angles) of the trained geometry and compare them with the statement in Thm. 2.
The labels for each vector are chosen such that the dataset is ( R = 10 , ρ = 1 / 2 )
-STEP imbalanced, with n min = 5, and choose ∆ =
δ maj δ min =
R γ with γ ∈ [− 1 . 5 , 1 . 5 ] .
Models are trained with constant learning rate for 6000 epochs.
We normalize δ so that 1 Tk δ =
k , since we empirically observe for a fixed ratio ∆ that the convergence speed depends on the magnitude of δ .
For the trained classifiers and embeddings, we compute: (1) squared ratios of majorityminority norms, (2) cosine of angles between pairs of majority-majority, minority-minority, majority-minority classifiers and mean-embeddings.
For each choice of γ and loss function, we compute each metric on all the respective pairs, and compare their average to the values predicted by Thm. 2.
As ∆ gets far from 1, some models fail to achieve zero-training-error in this setting: We differentiate these cases with the purple (rather than green) markers.
A.2. Deep-net experimental setting
We train a ResNet-18 on imbalanced CIFAR-10 with R = 10.
We train the model forepochs with an initial learning rate of 0 .
1 reduced at epochs 116 and 232 by a factor of 10, with a batch size of 128.
Following the same setting as in [20, 24], momentum and weight decay are set to 0 .
9 and 10 − 5 respectively.
Similar to UFM, we normalize δ to sum to k .
At the end on the training, we compute the same metrics described in Sec.
A.1. A.3. Discussion on experimental results
Figs.
2 and 3 illustrate the empirical geometry vs the prediction of Thm. 2.
The convergence to the theoretical values is more challenging for the deep-net models, particularly for large ∣ γ ∣ .
Further, the theory gives a more accurate prediction of the mean-embeddings’ geometry in case of the LDT, and of the classifiers’ in case of the CDT loss. This is consistent for both
UFM and deep-net models. For LDT, the prediction is well followed by UFM and ResNet empirics around the special case of γ = 0 . 5, with an exception of the majority classifier angles in the ResNet experiments.
Also, as predicted by the theorem, for γ = 0 . (∆ = √ R ), the LDT geometry is the Simplex ETF, up to a scaling on the features: In Figs. 2 and 3 the LDT cosine plots intersect with the ETF angles, i.e. − 1 /( k − 1 ) , thus achieving equiangularity and maximal angular separation.
The classifier norm ratios also attain the value 1, which along with the equiangularity describe an ETF structure for classifiers.
{+Remark 
 In our experiments with CDT and LDT, we center the embeddings before computing the geometrical quantities like norms and angles. This is consistent with centering performed for experiments with balanced data by [21, 24, 29], although the exact centering+} CS-SVM SGD
ETF
Norm Ratios
Majority Angles
Minority Angles Majority/Minority Angles
C
D
T ∥
w m a j ∥ 2 / ∥
w m i n o r
∥∥
h m
a j ∥ 2 / ∥
h m i n o r ∥ c o s ( w m a j , w m a j ) c
o s ( h m a j , h m a j ) c
o s ( w m i n o r , w m i n o r ) c
o s ( h m i n o r , h m i n o r ) c
o s ( w m a j , w m i n o r ) c
o s ( h m a j , h m i n o r ) γ γ γ γ     1.0 0.
CS-SVM SGD
ETF
Norm Ratios
Majority Angles
Minority Angles Majority/Minority Angles
L D
T ∥
w m a j ∥ 2 / ∥
w m i n o r
∥∥
h m
a j ∥ 2 / ∥
h m i n o r ∥ c o s ( w m a j , w m a j ) c
o s ( h m a j , h m a j ) c
o s ( w m i n o r , w m i n o r ) c
o s ( h m i n o r , h m i n o r ) c
o s ( w m a j , [-thus achieving equiangularity and maximal angular separation. The classifier norm ratios also attain the value 1, which along with the equiangularity describe an ETF structure for classifiers. Remark 
 In our experiments with CDT and LDT, we center the embeddings before computing the geometrical quantities like norms and angles.-]w m i n o r ) c
o s ( h m a j , h m i n o r ) γ γ γ γ vector is now as discussed in the section below.
Further, even in the UFM experiments with
LDT, we observe that the centered mean embeddings follow the predicted implicit geometry more closely.
Thus, we employ the centering as shown in (7) .
CDT centering : Classifiers that follow the geometry in Thm. 2, w ∗ c , c ∈ [ k ] are centered around zero after some re-weighting, i.e. ∑ c ∈[ k ] w ∗ c / δ c = 0.
The embeddings h i , i ∈ [ n ] are also not centered around zero in general.
Instead, it holds that ∑
[Equation]
Note that this reduces to ∑ i ∈[ n ] h ∗ i for balanced data, and remains unchanged for any choice of the hyper-parameters δ .
Eqn.
(6) is also equivalent to ∑ c ∈[ k ] µ ∗ c = 0, with µ ∗ c , c ∈ [ k ] the mean embeddings of each class.
LDT centering : The optimal classifiers and features ( W ∗ , M ∗ D ) follow the ( 1 k , ˜ R )
-SELI structure.
Thus (see [24, Sec.
B.1.4]), the classifiers w ∗ c , c ∈
[ k ] are centered around zero.
However the embeddings are centered around zero after a reweighting that depends both on δ c and n c , c ∈
[ k ] .
Specifically, ∑ c ∈[ k ] δ c µ ∗ c = 0, or equivalently,
[Equation]
R = 20, gamma = 0.5 R = 20, gamma = 1.0 R = 2, gamma = 0.
R = 2, gamma = 1.
[Figure]
A.4. Optimization Concerns
In our experiments [Equation] we observe that large R and ∆ affect the convergence negatively.
This observation is also made in [24].
We compare the convergence speed for different values of R and γ in Fig. 4.
To measure the distance of the SGD solution to the predicted implicit geometry at each epoch [Equation]
we compute ∥ W T W ∥ W T W ∥ − W ∗ T W ∗ ∥
W ∗ T W ∗ ∥
∥ F for the classifier weights, ∥ M T M [Equation] for centered mean embeddings and
[Equation] for logits, where ( W ∗ , M ∗ ) are as described by Thm. 2. In addition to the worse convergence, as suggested by Fig.
[Equation] larger ∣ γ ∣ values make it more challenging to reach models with zero training error by SGD. In particular, for a ResNet-18 model trained with CDT on imbalanced CIFAR-
[Equation] Fig.
5 illustrates how larger γ values significantly reduce the training accuracy of minority classes, further emphasizing the difficulties with optimization in this setting
[Equation]
[Figure]