Generalization for Multiclass Classification withOverparameterized Linear Models
Anonymous for review
August 2,
Abstract
Via an overparameterized linear model with Gaussian features, we provide conditions for good generalization for multiclass classification of minimum-norminterpolating solutions in an asymptotic setting where both the number of underlying features and the number of classes scale with the number of training points.
The survival/contamination analysis framework for understanding the behaviorof overparameterized learning problems is adapted to this setting, revealing thatmulticlass classification qualitatively behaves like binary classification in that, aslong as there are not too many classes (made precise in the paper), it is possibleto generalize well even in some settings where the corresponding regression taskswould not generalize.
Besides various technical challenges, it turns out that the key difference from the binary classification setting is that there are relatively fewer positive training examples of each class in the multiclass setting as the number ofclasses increases, making the multiclass problem “harder” than the binary one
Introduction
Multiclass classification on standardized datasets is where the current deep-learning revolution reallymade the community take notice with previously unattainable levels of performance.
Contemporarysystems have demonstrated tremendous success at these tasks, typically using gigantic models withparameters that vastly exceed the (also large) number of data points used to train these models.
Indefiance of traditional statistical wisdom regarding overfitting, these big models can be trained toachieve zero training error even with noisy labels, but still generalize well in practice [84, 28].
To better understand this empirical phenomenon, one line of work uses appropriate high-dimensionallinear models for regression problems to show how benign fitting of noise in training data is possible[31, 52, 4, 9, 55].
Essentially, the model must have enough "non-preferred" degrees of freedom to beable to absorb the training noise without contaminating predictions by too much.
Simultaneously,there has to be enough of a preference for degrees of freedom that can capture the true pattern toenable it to survive the learning procedure and be well represented in the final learned model.
A subsequent line of work studies binary classification [56, 13, 76] and shows that binary classificationcan generalize well beyond what can be proved by classical margin-based bounds [3] and there existregimes where binary classification can even succeed in generalizing where regression fails — lesspreference is required for the degrees of freedom that capture the true pattern [56].
Very recently, thegeneralization of multiclass classification in similar models was studied in Wang et al.
[77] but theanalysis was limited to a fixed finite number of classes.
In practice, we see that larger datasets oftencome with more classes and are tackled with even bigger models and so it is important to see whathappens to generalization when everything scales together.
To have a crisply understandable approachthat allows everything to scale, this paper also adopts the bi-level covariance model with Gaussianfeatures that is used in Muthukumar et al.
[55, 56], Wang et al.
[75], Wang et al.
To understand classification, we must understand the role of training loss functions in determiningwhat is learned.
Empirical evidence shows that least-squares can yield classification performancecompetitive to cross-entropy minimization [64, 35, 11].
Muthukumar et al.
[56], Hsu et al.
[33]show that indeed with sufficient overparameterization, the support vector machine (SVM) solution,which also arises from minimizing the logistic loss using gradient descent [68, 36], is identical tothat obtained by the minimum-norm interpolation (MNI) of binary labels — what would be obtained by gradient descent while minimizing the squared loss.
A similar equivalence 1 holds for differentvariations of multiclass SVMs and the MNI of one-hot-encoded labels [77].
Consequently, this paperfocuses on the MNI approach to overparameterized learning for multiclass classification.
Our contributions
Our study provides an asymptotic analysis of the error of the minimum-norm interpolating classifier for the multiclass classification problem with weighted Gaussian features.
We consider anoverparameterized setting using a bi-level feature weighting model where the number of features,classes, favored features, and the feature weights themselves all scale with the number of trainingpoints.
Under this model, Theorem 5.1 provides sufficient conditions for good generalization in theform of a region in which as the number of training points increase, the number of classes growsslowly enough, the total number of features (i.e. level of overparameterization) grows fast enough,the number of favored features grows slowly enough, and the amount of favoring of those favoredfeatures is sufficient to allow for asymptotic generalization.
We assume that our labels are generatednoiselessly based on which of the first k features is the largest.
To prove our main result, Theorem 5.1, we present a novel typicality-style argument featuringthe feature margin (gap between the largest and second-largest feature) for computing sufficientconditions for correct classification utilizing the signal-processing inspired concepts of survivaland contamination from Muthukumar et al.
[55, 56] and leveraging the random-matrix analysistools sharpened in Bartlett et al.
The survival concept relates to the shrinkage induced by theregularizing effect of having lots of features in the context of min-norm interpolation — survivalcaptures what is left of the true pattern after shrinkage.
Contamination reflects the consequenceof overparameterization when training via optimization: in addition to the true pattern, there is aninfinite family of other 3 false patterns (aliases) that also happen to explain the limited training data,and the optimizer ends up hedging its bet across the true pattern and these other competing falseexplanations.
The learned false patterns contaminate the predictions on test points, and this can bequantified by the relevant standard deviation.
The key is analyzing what happens with multiclass training data where there are relatively fewerpositive examples of each class, and where the training data for a particular class is not independentof the features corresponding to other classes.
The analysis shows that as a result of having fewerpositive exemplars for a class relative to the total size of the training data, the survival drops by afactor of k (the number of classes), while the contamination only drops by a factor of √ k .
As inbinary classification, the ratio of the relevant survival to contamination terms plays the role of theeffective signal-to-noise ratio and shows up as a key quantity in our error analysis (Equation (22) from Section 5.1).
When this ratio grows asymptotically to ∞ , multiclass classification generalizes well.
Tothe best of our knowledge, this is the first work that quantifies this effect of fewer informative samplesper class and in what sense that makes multiclass classification harder than binary classification.
Theclosest related work ([77]) only considers multiclass classification in the fixed finite class setting andconsequently, doesn’t compute exact dependencies on the number of classes k .
We provide a moredetailed comparison of our work with Wang et al.
[77] and Muthukumar et al.
[56] in Appendix H ofthe Supplemental material.
Related Work
The present work is situated within a larger stream of theoretical research trying to understand whyoverparameterized learning works and its limits.
The limited page budget here forces brevity, but werecommend the recent surveys Bartlett et al.
[5], Belkin [6], Dar et al.
[20] for further context.
Classically, by either operating in the underparameterized regime or by performing explicit regularization, we can force the training procedure to average out the harmful effects of training noise andthereby hope to obtain good generalization.
The present cycle of seeking a deeper understanding began after it was observed that modern deep networks were overparameterized, capable of memorizing noise, and yet still generalized well, even when they were trained without explicit regularization[59, 84].
Experiments in Geiger et al.
[28], Belkin et al.
[8] observed a double-descent behavior ofthe generalization error where in addition to the traditional U-shaped curve in the underparameterizedregime, the error decreases in the overparameterized regime as we increase the number of model parameters.
This double descent phenomenon is not unique to deep learning models and was replicatedfor kernel learning [7].
Further, the good generalization performance in the overparameterized regimecannot be explained by traditional worst-case generalization bounds based on Rademacher complexityor VC-dimension since the models have the capacity to fit purely random labels.
Overparameterizedmodels must therefore have some fortuitous combination of the model architecture with the trainingalgorithm that leads us to a particular solution that generalizes well.
To understand the phenomenon better, several works study the simpler setting of overparameterizedlinear regression.
The minimum- ℓ 2 norm 4 interpolator is of particular interest since gradient descenton the squared loss has an implicit 5 bias towards this solution in the overparameterized regime[24] and has been studied extensively.
(An incomplete list is Hastie et al.
[31], Mei and Montanari[52], Bartlett et al.
[4], Belkin et al.
[9], Muthukumar et al.
[55], Bibas et al.
[10], Kobak et al.[41], Wu and Xu [81], Richards et al.
To generalize well, the underlying feature family mustsatisfy a balance between having a few important directions that sufficiently favor the true pattern,and a large number of unimportant directions that can absorb the noise in a harmless manner.
3.1 High dimensional binary classification
Both concurrently with and subsequent to the wave of analyses on overparameterized regression,researchers turned their attention to binary classification.
A line of work poses the overparameterizedbinary classification problem as an optimization problem and analyzes it directly to obtain preciseasymptotic behaviours of the generalization error [22, 66, 37, 69, 54, 38, 70].
The key technicaltool employed in these works is the Convex Gaussian Min-max Theorem and the resultant errorformulas involve solutions to a system of non-linear equations that typically do not admit closed-formexpressions.
The generalization error of the max-margin SVM has also been analyzed directly bystudying the iterates of gradient descent in [13] and leveraging the implicit regularization perspectiveof optimization algorithms.
However, although the above works did significantly enhance our understanding of binary classification in the overparameterized regime, a fundamental question was not answered: “Is classificationeasier than regression?"
While the classification task is easier than the regression task at test time(regression requires us to correctly predict a real value while binary classification requires us to onlypredict its sign correctly), the training data for classification is less informative than that for regressionsince the labels are also binary.
As described earlier, this question was answered in Muthukumaret al.
[56], by exhibiting an asymptotic regime where binary classification error goes to zero, but theregression error does not.
This was shown using Gaussian features with a bi-level covariance model.
It turns out that the level of anisotropy (favoring of true features) required to perform regressioncorrectly is significantly higher than that required for binary classification.
The key to the result in Muthukumar et al.
[56] was the signal-processing inspired survival/contamination framework introduced in Muthukumar et al.
[55] as a reconceptualization of the “effectiveranks” perspective of Bartlett et al.
For binary classification to succeed, what matters is that thesurvival exceed the contamination so that the sign of the prediction remains correct.
Meanwhile,regression is harder since for regression to succeed, the survival must also tend to 1 .
3.2 Multiclass classification and the role of training loss function
There is a large classical body of work on multiclass classification algorithms [79, 12, 23, 18, 46],with further works giving computationally efficient algorithms for extreme multiclass problemswith a huge number of classes [15, 83, 62].
Numerous theoretical works investigate the consistency of classifiers [85, 60, 61, 71, 14].
Finite-sample analysis of the generalization error in multiclassclassification problems in the underparameterized regime has been studied in Koltchinskii and
Panchenko [42], Guermeur [29], Allwein et al.
[1], Li et al.
[49], Cortes et al.
[16], Lei et al.[47], Maurer [51], Lei et al.
[48], Kuznetsov et al.
[44, 45] and includes both data dependentbounds using Rademacher complexity, Gaussian complexity and covering numbers as well as dataindependent bounds using the VC dimension.
Recent work [72] leverages the Convex Gaussian
Min-max Theorem to precisely characterize the asymptotic behaviour of the least-squares classifierin underparameterized multiclass classification.
So, how different is multiclass classification from binary classification?
The test time task ismore difficult and for the same total number of training points, we have fewer positive trainingexamples from each class.
Several empirical studies comparing the performances of multiclassclassification via learning multiple binary classifiers have been undertaken [64, 25, 1].
The effectsof the loss function while using deep nets to perform classification has also been investigated[32, 26, 43, 11, 21, 40, 35, 39].
Empirical evidence of least-squares minimization yielding competitivetest classification performance to cross-entropy minimization has been presented in Rifkin and Klautau[64], Hui and Belkin [35], Bosman et al.
More recently, Wang et al.
[77] makes progress towards bridging the gap between empirical observations and theoretical understanding by proving that in certain overparameterized regimes thesolution to a multiclass SVM problem is identical to the one obtained by minimum-norm interpolationof one-hot encoded labels (equivalently, that gradient descent on squared loss leads to the samesolution as gradient descent on cross-entropy loss as a result of implicit bias of these algorithms[24, 36, 68]).
In addition, Wang et al.
[77] extends the analysis presented in Muthukumar et al.
[56]for the binary classification problem to the multiclass problem with finitely many classes via aninteresting reduction to analyzing a finite set of pairwise competitions, all of which must be won formulticlass classification to succeed.
(We give further comments on the relationship of the presentpaper with Wang et al.
[77] in Appendix H of the Supplemental material.)
Problem setup
We consider the multiclass classification problem with k classes.
The training data consists of n pairs{ x i , ℓ i } ni =1 where x i ∈ R d are i.i.d Gaussian vectors drawn from distribution,
[Equation]
N [Equation] [Equation] We make the following assumption on how the labels ℓ i ∈ [ k ] are generated.
Assumption 4.1.
Orthogonal classes noiseless model
The class labels ℓ i are generated based onwhich of the first k dimensions of a point x i has the largest value,
[Equation]
We use the notation x i [ m ] to refer to the m th element of vector x i .
For clarity of exposition, wemake explicit a feature weighting that transforms the training points as follows:
[Equation]
Here λ ∈ R d contains the squared feature weights.
The feature weighting serves the role of favoringthe true pattern, something that is essential for good generalization.
The weighted feature matrix X w ∈ R n × d is given by,
[Equation]
where z j ∈ R n contains the j th features from the n training points.
Note that z j ∼ N (0 , I n ) are i.i.d
Gaussians.
We use a one-hot encoding for representing the labels as the matrix Y oh ∈ R n × k ,
[Equation] [Equation]
where,
[Equation]
A zero-mean variant of the encoding where we subtract the mean 1 k from each entry is denoted:
[Equation] [Equation]
Our classifier consists of k coefficient vectors ˆ f m for m ∈ [ k ] that are learned by minimum-norminterpolation of the zero-mean one-hot variants using the weighted features.
[Equation]
We can express these coefficients in closed form as,
[Equation] [Equation]
On a test point x test ∼ N (0 , I d ) we predict a label as follows:
First, we transform the test pointinto the weighted feature space to obtain x wtest where x wtest [ j ] = (cid:112) λ j x test [ j ] for j ∈ [ d ] .
Then wecompute k scalar “scores” and assign the class based on the largest score as follows:
[Equation]
The true label of the test point is ℓ test = argmax 1 ≤ m ≤ k x test [ m ] .
A misclassification event E erroccurs iff
[Equation]
In our work we determine sufficient conditions under which the probability of misclassification(computed over the randomness in both the training data and test point) goes to zero in an asymptoticregime where the number of training points, number of features, number of classes and featureweights scale according to the bi-level ensemble model.
Definition 4.2.
( Bi-level ensemble ): The bi-level ensemble is parameterized by p, q, r and t wherep > 1 , 0 ≤ r < 1 , 0 < q < ( p − r ) and 0 ≤ t < r .
Here, parameter p controls the extent ofoverparameterization, r determines the number of favored features, q controls the weights on favoredfeatures and t controls the number of classes.
The number of features ( d ), number of favored features( s ), number of classes ( k ) and feature weights ( (cid:112) λ j ) all scale with the number of training points ( n )as follows:
[Equation]
[Equation]
where c k is a positive integer.
The feature weights are given by,
[Equation]
We provide a visualization of the bi-level model in Figure 1.
Main result
Theorem 5.1.
( Asymptotic classification region in the bi-level model ): Under the bi-level ensemblemodel 4.2, when the true data generating process is 1-sparse (Assumption 4.1), the probability ofmisclassification P ( E err )[-! 
-]
→ 0 as n → ∞ if the following conditions hold:
[Equation]
[Equation]
Note that from Muthukumar et al.
[56], the condition q + r > 1 corresponds to the regime wherethe corresponding regression does not generalize well and thus our result shows that multiclassclassification can generalize in regimes where the regression problem does not.
Figure 2 visualizesthe regimes by considering slices of the four dimensional scaling parameter space of p, q, r and t .
(1a)and (2a) fix the value of q to 0 .
75 and 0 .
95 respectively and contrast the multiclass problem with afixed finite number of classes ( t = 0 ) to the binary classification and regression problems.
From theseplots we observe that if we fix p, q, t and increase r , i.e. increasing how many features are favored(and thereby each of them less), we transition from the regime where both regression andbinary classification work, into the regime where binary classification works but regression does not,then the regime where this paper can prove multiclass classification works and finally to the regimewhere neither regression nor binary classification works.
In Figure 2, subplots (1b),(1c),(2b) and (2c) each visualize a slice along the r and t (class scaling)dimensions with fixed p and q .
The x axis itself in these plots corresponds to a fixed finite classessetting.
From (1b) we observe that the right-hand boundary of the region where multiclass classification generalizes well contains two slopes.
These slopes arise from the two conditions t < 1 − r andt <
p + 1 − 2( q + r ) in Theorem 5.1 and are a result of either contamination from favored (but nottrue) features dominating or contamination from the unfavored features dominating.
In (1c) we arein the regime where binary classification works for all values of r < 1 .
However, as we increase t ,eventually multiclass classification stops working.
When we go from the binary problem to a multiclass problem with k classes, the survival dropsby a factor of k as a consequence of having only 1 k fraction of positive training examples per class.
This is because the one-hot labels we interpolate while training have fewer large values close to 1that are able to positively correlate with the true feature vector.
Having fewer positive exemplarsalso reduces the total energy in the training vector by a factor of k , and because of the square-rootrelationship of the standard deviation to the energy, the contamination only shrinks by a factor of 231 √ k .
The overall survival/contamination ratio decreases by a factor of √ k making the multiclassclassification task more difficult.
An interesting observation here is the amount of favoring requiredfor good generalization is linked to the number of positive training examples per class.
Indeed, ifwe consider a setting where the binary classification problem generalizes well, and we switch tothe k class multiclass problem, then by increasing the number of training samples k fold (and thusmatching the number of positive training examples per class in the multiclass case to the binarycase) and keeping the number of features and feature weights constant we can generalize well formulticlass classification.
(Appendix G of the Supplemental material elaborates on this phenomenon,as well as why it is somewhat surprising.)
Next, we present a brief overview of our proof that utilizes the survival/contamination analysisframework from Muthukumar et al.
[56] along with a typicality-inspired argument where the featuremargin (difference between largest and second largest feature) on the test point plays a key role.
Thecomplete proof is provided in Appendices B, C, D, and E of the Supplemental material.
5.1 Proof sketch
Assume without loss of generality that for the test point x test ∼ N (0 , I d ) , the true class is α for someα ∈ [ k ] .
Let x wtest be the weighted version of this test point.
A necessary and sufficient condition forclassification error is that for some β ̸ = α, β ∈ [ k ] ,
[Equation]
By converting into the unweighted feature space we obtain
[Equation]
where
[Equation]
Performing some algebraic manipulations and because λ α = λ β = λ since both α and β are favoredfeatures [Equation] we can rewrite this as
[Equation]
where
[Equation]
We divide by CN α,β to normalize the RHS above to have a standard normal distribution.
Next, byremoving the dependency on β
[Equation]
we obtain a sufficient condition for correct classification:
[Equation]
Here the min and max are over all competing features: 1 ≤ β ≤ k, β ̸ = α and the sum is overall d feature indices except α and β , but we simplify the notation for convenience.
We show viaintermediate lemmas introduced in Appendix B of the Supplemental material that under the conditionsspecified in Theorem 5.1, with sufficiently high probability 11 , the relevant survival to contamination
SU/CN ratio grows at a polynomial rate n v for some v > 0 , the closest feature margin shrinks at aless-than-polynomial rate 1 / √ ln nk , the survival variation decays at a polynomial rate n u for some u > 0 .
Further, the magnitudes of the largest competing feature and the normalized contaminationare no more than (cid:112) ln( nk ) .
This implies that the left-hand side of Equation (22) grows at a polynomial rate n v (ignoringlogarithmic terms) and dominates the right-hand side which grows at the much slower rate √ ln nk .
Asurvival/contamination ratio also plays a key role in the analysis of the binary classification problem in
Muthukumar et al.
[56] but in the multiclass setting, we additionally have the survival variation termand feature margin playing important roles since we are comparing different scores while predictingthe class label.
For correct classification, the survival/contamination ratio must be sufficiently large,the survival variation must be small enough and the feature margin must be sufficiently large.
Conclusion
In this work we compute sufficient conditions for good generalization of multiclass classificationin a bi-level overparameterized linear model with Gaussian features.
We observed that multiclassclassification can generalize even when the regression problem does not generalize (for q + r > 1 ).
Further, the multiclass problem is “harder” than the binary problem because we have fewer positivetraining examples per class.
The nature of the training data complicates our analysis in the multiclasssetting since the true class labels are generated by comparing k features and thus we no longer haveindependence of the encoded class label y with any of these features.
This becomes relevant when wecompute bounds on the survival and contamination quantities since the Hanson-Wright inequality [65]is no longer applicable directly on the quantities of interest as was the case for the binary classificationproblem in prior work [56].
As a consequence of working around this non-independence we believethat our sufficient conditions for good generalization in the regime q + r > 1 are loose.
Even though in our work we focus on the regime where regression does not work, q + r > 1 , we canextend the analysis to the regime where q + r < 1 by grinding through the expressions for survivaland contamination in this regime.
Even in this regime, for multiclass training data, survival is of theorder 1 k while contamination scales similarly to the regime q + r > 1 .
Thus, while it is true that forbinary classification or a fixed number of classes, the regime where regression works is a regimewhere classification also works, this need not be true if there are too many classes.
We conjecture that the following is a set of necessary and sufficient conditions for asymptoticallygood generalization (We elaborate on this in Appendix F in the Supplemental material):
Conjecture 6.1.
( Conjectured bi-level regions ): Under the bi-level ensemble model 4.2, whenthe true data generating process is 1-sparse (Assumption 4.1), as n → ∞ , the probability ofmisclassification event P ( E err ) behaves as follows:
[Equation]
The conjectured regions are visualized in (1d),(1e),(1f),(2d),(2e) and (2f) in Figure 2.
Subfigures (1d)and (2d) illustrate that we believe multiclass classification with finitely many classes works if binaryclassification works.
Further, comparing (1e) to (2e) when we increase q , the conjectured parameterregion where multiclass classification works shrinks since we decrease the amount of favoring of truefeatures.
Interestingly, the nature of the looseness in our approach is such that our proof technique isable to recover a larger fraction of the conjectured region for larger q which intuitively is a result ofless favoring leading to stronger concentration of certain random quantities.
Tightening the potentiallooseness in our analysis and proving the converse result by computing sufficient conditions for poorgeneralization of multiclass classification are interesting avenues of future work.
Further, although the present analysis focuses on solutions that exactly interpolate the training data,we can extend our results to account for additional ridge regularization by viewing ridge regularizationas minimum-norm interpolation using augmented contamination-free features as in the Appendixof Muthukumar et al.
[55] and computing bounds leveraging tools from Tsigler and Bartlett [74].
Our assumption of the strict bi-level weighting model is largely to simplify the calculations andby substituting terms appropriately in our lemmas from Appendix B in the Supplemental material,it should be possible to compute results for other weighting models.
Finally, exploring the newphenomena that can be encountered as we go beyond the 1-sparse noiseless model is an excitingdirection for future work.