Generalization for Multiclass Classiﬁcation withOverparameterized Linear Models
Anonymous for review
May 19,
Abstract
Via an overparameterized linear model with Gaussian features, we provide con- ditions for good generalization for multiclass classiﬁcation of minimum-norm interpolating solutions in an asymptotic setting where both the number of under- lying features and the number of classes scale with the number of training points.
The survival/contamination analysis framework for understanding the behavior of overparameterized learning problems is adapted to this setting, revealing that multiclass classiﬁcation qualitatively behaves like binary classiﬁcation in that, as long as there are not too many classes (made precise in the paper), it is possible to generalize well even in some settings where the corresponding regression tasks would not generalize.
Besides various technical challenges, it turns out that the key difference from the binary classiﬁcation setting is that there are relatively fewer positive training examples of each class in the multiclass setting as the number of classes increases, making the multiclass problem “harder” than the binary one.
Introduction
Multiclass classiﬁcation on standardized datasets is where the current deep-learning revolution reallymade the community take notice with previously unattainable levels of performance.
Contemporarysystems have demonstrated tremendous success at these tasks, typically using gigantic models withparameters that vastly exceed the (also large) number of data points used to train these models.
Indeﬁance of traditional statistical wisdom regarding overﬁtting, these big models can be trained toachieve zero training error even with noisy labels, but still generalize well in practice [81, 27].
To better understand this empirical phenomenon, one line of work uses appropriate high-dimensionallinear models for regression problems to show how benign ﬁtting of noise in training data is possible[30, 51, 4, 9, 54].
Essentially, the model must have enough "non-preferred" degrees of freedom to beable to absorb the training noise without contaminating predictions by too much.
Simultaneously,there has to be enough of a preference for degrees of freedom that can capture the true pattern toenable it to survive the learning procedure and be well represented in the ﬁnal learned model.
A subsequent line of work studies binary classiﬁcation [55, 13, 74] and shows that binary classiﬁcationcan generalize well beyond what can be proved by classical margin-based bounds [3] and there existregimes where binary classiﬁcation can even succeed in generalizing where regression fails — lesspreference is required for the degrees of freedom that capture the true pattern [55].
Very recently, thegeneralization of multiclass classiﬁcation in similar models was studied in Wang et al.
[75] but theanalysis was limited to a ﬁxed ﬁnite number of classes.
In practice, we see that larger datasets oftencome with more classes and are tackled with even bigger models and so it is important to see whathappens to generalization when everything scales together.
To have a crisply understandable approachthat allows everything to scale, this paper also adopts the bi-level covariance model with Gaussianfeatures that is used in Muthukumar et al.
[54, 55], Wang et al.
[73], Wang et al.
To understand classiﬁcation, we must understand the role of training loss functions in determiningwhat is learned.
Empirical evidence shows that least-squares can yield classiﬁcation performancecompetitive to cross-entropy minimization [62, 34, 11].
Muthukumar et al.
[55], Hsu et al.
[32]show that indeed with sufﬁcient overparameterization, the support vector machine (SVM) solution,which also arises from minimizing the logistic loss using gradient descent [66, 35], is identical tothat obtained by the minimum-norm interpolation (MNI) of binary labels — what would be obtained by gradient descent while minimizing the squared loss.
A similar equivalence 1 holds for differentvariations of multiclass SVMs and the MNI of one-hot-encoded labels [75].
Consequently, this paperfocuses on the MNI approach to overparameterized learning for multiclass classiﬁcation.
Our contributions
Our study provides an asymptotic analysis of the error of the minimum-norm interpolating classiﬁer for the multiclass classiﬁcation problem with weighted Gaussian features.
We consider anoverparameterized setting using a bi-level feature weighting model where the number of features,classes, favored features, and the feature weights themselves all scale with the number of trainingpoints.
Under this model, Theorem 5.1 provides sufﬁcient conditions for good generalization in theform of a region in which as the number of training points increase, the number of classes growsslowly enough, the total number of features (i.e. level of overparameterization) grows fast enough,the number of favored features grows slowly enough, and the amount of favoring of those favoredfeatures is sufﬁcient to allow for asymptotic generalization.
We assume that our labels are generatednoiselessly based on which of the ﬁrst k features is the largest.
To prove our main result, Theorem 5.1, we present a novel typicality-style argument featuringthe feature margin (gap between the largest and second-largest feature) for computing sufﬁcientconditions for correct classiﬁcation utilizing the signal-processing inspired concepts of survival andcontamination from Muthukumar et al.
[54, 55] and leveraging the random-matrix analysis toolssharpened in Bartlett et al.
The key is analyzing what happens with multiclass training data wherethere are relatively fewer positive examples of each class, and where the training data for a particularclass is not independent of the features corresponding to other classes.
The analysis shows that asa result of having fewer positive exemplars for a class relative to the total size of the training data,the survival drops by a factor of k (the number of classes), while the contamination only drops by afactor of p k .
As in binary classiﬁcation, the ratio of the relevant survival to contamination termsplays the role of the effective signal-to-noise ratio and shows up as a key quantity in our error analysis(Equation (22) from Section 5.1).
When this ratio grows asymptotically to 1 , multiclass classiﬁcationgeneralizes well.
To the best of our knowledge, this is the ﬁrst work that quantiﬁes this effect offewer informative samples per class and in what sense that makes multiclass classiﬁcation harderthan binary classiﬁcation.
The closest related work ([75]) only considers multiclass classiﬁcation inthe ﬁxed ﬁnite class setting and consequently, doesn’t compute exact dependencies on the number ofclasses k .
We provide a more detailed comparison of our work with Wang et al.
[75] and Muthukumaret al.
[55] in Appendix H in the Supplemental materials.
Related Work
The present work is situated within a larger stream of theoretical research trying to understand whyoverparameterized learning works and its limits.
The limited page budget here forces brevity, but werecommend the recent surveys Bartlett et al.
[5], Belkin [6], Dar et al.
[20] for further context.
Classically, by either operating in the underparameterized regime or by performing explicit regularization, we can force the training procedure to average out the harmful effects of training noise andthereby hope to obtain good generalization.
The present cycle of seeking a deeper understandingbegan after it was observed that modern deep networks were overparameterized, capable of memorizing noise, and yet still generalized well, even when they were trained without explicit regularization[57, 81].
Experiments in Geiger et al.
[27], Belkin et al.
[8] observed a double-descent behavior ofthe generalization error where in addition to the traditional U-shaped curve in the underparameterizedregime, the error decreases in the overparameterized regime as we increase the number of model parameters.
This double descent phenomenon is not unique to deep learning models and was replicatedfor kernel learning [7].
Further, the good generalization performance in the overparameterized regimecannot be explained by traditional worst-case generalization bounds based on Rademacher complexityor VC-dimension since the models have the capacity to ﬁt purely random labels.
Overparameterizedmodels must therefore have some fortuitous combination of the model architecture with the trainingalgorithm that leads us to a particular solution that generalizes well.
To understand the phenomenon better, several works study the simpler setting of overparameterizedlinear regression.
The minimum- ` 2 norm 3 interpolator is of particular interest since gradient descenton the squared loss has an implicit 4 bias towards this solution in the overparameterized regime[24] and has been studied extensively.
(An incomplete list is Hastie et al.
[30], Mei and Montanari[51], Bartlett et al.
[4], Belkin et al.
[9], Muthukumar et al.
[54], Bibas et al.
[10], Kobak et al.[40], Wu and Xu [78], Richards et al.
To generalize well, the underlying feature family mustsatisfy a balance between having a few important directions that sufﬁciently favor the true pattern,and a large number of unimportant directions that can absorb the noise in a harmless manner.
3.1 High dimensional binary classiﬁcation
Both concurrently with and subsequent to the wave of analyses on overparameterized regression,researchers turned their attention to binary classiﬁcation.
A line of work poses the overparameterizedbinary classiﬁcation problem as an optimization problem and analyzes it directly to obtain preciseasymptotic behaviours of the generalization error [22, 64, 36, 67, 53, 37, 68].
The key technicaltool employed in these works is the Convex Gaussian Min-max Theorem and the resultant errorformulas involve solutions to a system of non-linear equations that typically do not admit closed-formexpressions.
The generalization error of the max-margin SVM has also been analyzed directly bystudying the iterates of gradient descent in [13] and leveraging the implicit regularization perspectiveof optimization algorithms.
However, although the above works did signiﬁcantly enhance our understanding of binary classiﬁcation in the overparameterized regime, a fundamental question was not answered: “Is classiﬁcationeasier than regression?"
While the classiﬁcation task is easier than the regression task at test time(regression requires us to correctly predict a real value while binary classiﬁcation requires us to onlypredict its sign correctly), the training data for classiﬁcation is less informative than that for regressionsince the labels are also binary.
As described earlier, this question was answered in Muthukumaret al.
[55], by exhibiting an asymptotic regime where binary classiﬁcation error goes to zero, but theregression error does not.
This was shown using Gaussian features with a bi-level covariance model.
It turns out that the level of anisotropy (favoring of true features) required to perform regressioncorrectly is signiﬁcantly higher than that required for binary classiﬁcation.
The key to the result in Muthukumar et al.
[55] was the signal-processing inspired survival/contamination framework introduced in Muthukumar et al.
[54] as a reconceptualization of the “effectiveranks” perspective of Bartlett et al.
The survival concept relates to the shrinkage induced by theregularizing effect of having lots of features in the context of min-norm interpolation — survivalcaptures what is left of the true pattern after shrinkage.
Contamination reﬂects the consequenceof overparameterization when training via optimization: in addition to the true pattern, there is aninﬁnite family of other 5 false patterns (aliases) that also happen to explain the limited training data,and the optimizer ends up hedging its bet across the true pattern and these other competing falseexplanations.
The learned false patterns contaminate the predictions on test points, and this can bequantiﬁed by the relevant standard deviation.
For binary classiﬁcation to succeed, what matters is thatthe survival exceed the contamination so that the sign of the prediction remains correct.
Meanwhile,regression is harder since for regression to succeed, the survival must also tend to 1 .
3.2 Multiclass classiﬁcation and the role of training loss function
There is a large classical body of work on multiclass classiﬁcation algorithms [76, 12, 23, 18, 45],with further works giving computationally efﬁcient algorithms for extreme multiclass problemswith a huge number of classes [15, 80, 60].
Numerous theoretical works investigate the consistencyof classiﬁers [82, 58, 59, 69, 14].
Finite-sample analysis of the generalization error in multiclassclassiﬁcation problems in the underparameterized regime has been studied in Koltchinskii and
Panchenko [41], Guermeur [28], Allwein et al.
[1], Li et al.
[48], Cortes et al.
[16], Lei et al.
been In fact, there is an important complementary literature that brings out the implicit regularization performed by training methods, especially variants of gradient descent and stochastic gradient descent, and how the underlying architecture of the model shapes this implicit regularization [29, 66, 35, 77, 56, 2, 79]. This is related to what is called the challenge of “underspeciﬁcation” in ML [19], and this in turn is one aspect of the challenge of covariate shifts [71]. [46], Maurer [50], Lei et al.
[47], Kuznetsov et al.
[43, 44] and includes both data dependentbounds using Rademacher complexity, Gaussian complexity and covering numbers as well as dataindependent bounds using the VC dimension.
Recent work [70] leverages the Convex Gaussian
Min-max Theorem to precisely characterize the asymptotic behaviour of the least-squares classiﬁerin underparameterized multiclass classiﬁcation.
So, how different is multiclass classiﬁcation from binary classiﬁcation?
The test time task ismore difﬁcult and for the same total number of training points, we have fewer positive trainingexamples from each class.
Several empirical studies comparing the performances of multiclassclassiﬁcation via learning multiple binary classiﬁers have been undertaken [62, 25, 1].
The effectsof the loss function while using deep nets to perform classiﬁcation has also been investigated[31, 26, 42, 11, 21, 39, 34, 38].
Empirical evidence of least-squares minimization yielding competitivetest classiﬁcation performance to cross-entropy minimization has been presented in Rifkin and Klautau[62], Hui and Belkin [34], Bosman et al.
More recently, Wang et al.
[75] makes progress towards bridging the gap between empirical observations and theoretical understanding by proving that in certain overparameterized regimes thesolution to a multiclass SVM problem is identical to the one obtained by minimum-norm interpolationof one-hot encoded labels (equivalently, that gradient descent on squared loss leads to the samesolution as gradient descent on cross-entropy loss as a result of implicit bias of these algorithms[24, 35, 66]).
In addition, Wang et al.
[75] extends the analysis presented in Muthukumar et al.
[55]for the binary classiﬁcation problem to the multiclass problem with ﬁnitely many classes via aninteresting reduction to analyzing a ﬁnite set of pairwise competitions, all of which must be won formulticlass classiﬁcation to succeed.
(We give further comments on the relationship of the presentpaper with Wang et al.
[75] in the Supplemental materials in Appendix H.)
Problem setup
We consider the multiclass classiﬁcation problem with k classes.
The training data consists of n pairs{ x i , ` i } ni =1 where x i 2 R d are i.i.d Gaussian vectors drawn from distribution,
[Equation]
We make the following assumption on how the labels ` i 2 [ k ] are generated.
Assumption 4.1.
1-sparse noiseless model
The class labels ` i are generated based on which of the ﬁrst k dimensions of a point x i has thelargest value,
[Equation]
We use the notation x i [ m ] to refer to the m th element of vector x i .
For clarity of exposition, wemake explicit a feature weighting that transforms the training points as follows:
[Equation]
Here  2 R d contains the squared feature weights.
The feature weighting serves the role of favoringthe true pattern, something that is essential for good generalization.
The weighted feature matrix X w 2 R n ⇥ d is given by,
[Equation]
where z j 2 R n contains the j th features from the n training points.
Note that z j ⇠ N (0 , I n ) are i.i.d
Gaussians.
We use a one-hot encoding for representing the labels as the matrix Y oh 2 R n ⇥ k ,
[Equation] [Equation]
m x i
but we consider the simpliﬁed case where µ m are unit vectors, and are orthogonal to each other for different classes m .
For the bi-level ensemble model (Deﬁnition 4.2) that we consider in this paper, this is exactly equivalent to the 1-sparse model deﬁned here.
Our feature model is equivalent to the one used in other works (e.g. [55]) that assume that the covariates come from an anisotropic Gaussian with a covariance matrix that favors the truly important directions.
where,
[Equation]
[Equation]
A zero-mean variant of the encoding where we subtract the mean 1 k from each entry is denoted:
[Equation] [Equation]
Our classiﬁer consists of k coefﬁcient vectors ˆ f m for m 2 [ k ] that are learned by minimum-norminterpolation of the zero-mean one-hot variants using the weighted features.
[Equation]
We can express these coefﬁcients in closed form as,
[Equation] [Equation]
On a test point x test ⇠ N (0 , I d ) we predict a label as follows:
First, we transform the test pointinto the weighted feature space to obtain x wtest where x wtest [ j ] = p  j x test [ j ] for j 2 [ d ] .
Then wecompute k scalar “scores” and assign the class based on the largest score as follows:
[Equation]
The true label of the test point is ` test = argmax 1  m  k x test [ m ] .
A misclassiﬁcation event E err 189 occurs iff
[Equation]
[Equation] In our work we determine sufﬁcient conditions under which the probability of misclassiﬁcation(computed over the randomness in both the training data and test point) goes to zero in an asymptoticregime where the number of training points, number of features, number of classes and featureweights scale according to the bi-level ensemble model.
Deﬁnition 4.2.
( Bi-level ensemble ): The bi-level ensemble is parameterized by p, q, r and t wherep > 1 , 0  r < 1 , 0 < q < ( p  r ) and 0  t < r .
Here, parameter p controls the extent ofoverparameterization, r determines the number of favored features, q controls the weights on favoredfeatures and t controls the number of classes.
The number of features ( d ), number of favored features( s ), number of classes ( k ) and feature weights ( p  j ) all scale with the number of training points ( n )as follows:
[Equation]
[Equation]
where c k is a positive integer.
The feature weights are given by,
[Equation]
We provide a visualization of the bi-level model in Figure 1.
Main results
Theorem 5.1.
( Asymptotic classiﬁcation region in the bi-level model ): Under the bi-level ensemblemodel 4.2, when the true data generating process is 1-sparse (Assumption 4.1), the probability ofmisclassiﬁcation P ( E err ) {+! 
+}
as n ! 1 if the following conditions hold:
[Equation]
[Equation]
[Equation]
Note that from Muthukumar et al.
[55], the condition q + r > 1 corresponds to the regime where theregression problem does not generalize well and thus our result shows that multiclass classiﬁcationcan generalize in regimes where the regression problem does not.
Figure 2 visualizes the regimes byconsidering slices of the four dimensional scaling parameter space of p, q, r and t .
(1a) and (2a) ﬁx the value of q to 0 .
75 and 0 .
95 respectively and contrast the multiclass problem with a ﬁxed ﬁnitenumber of classes ( t = 0 ) to the binary classiﬁcation and regression problems.
From these plotswe observe that if we ﬁx p, q, t and increase r , i.e. increasing how many features are favored (andthereby favoring each of them less), we transition from the regime where both regression and binaryclassiﬁcation work, into the regime where binary classiﬁcation works but regression does not, thenthe regime where this paper can prove multiclass classiﬁcation works and ﬁnally to the regime whereneither regression nor binary classiﬁcation works.
In Figure 2, subplots (1b),(1c),(2b) and (2c) each visualize a slice along the r and t (class scaling)dimensions with ﬁxed p and q .
The x axis itself in these plots corresponds to a ﬁxed ﬁnite classessetting.
From (1b) we observe that the right-hand boundary of the region where multiclass classiﬁcation generalizes well contains two slopes.
These slopes arise from the two conditions t < 1  r andt <
p + 1  2( q + r ) in Theorem 5.1 and are a result of either contamination from favored (but nottrue) features dominating or contamination from the unfavored features dominating.
In (1c) we arein the regime where binary classiﬁcation works for all values of r < 1 .
However, as we increase t ,eventually multiclass classiﬁcation stops working.
When we go from the binary problem to a multiclass problem with k classes, the survival dropsby a factor of k as a consequence of having only 1 k fraction of positive training examples per class.
This is because the one-hot labels we interpolate while training have fewer large values close to 1that are able to positively correlate with the true feature vector.
Having fewer positive exemplarsalso reduces the total energy in the training vector by a factor of k , and because of the square-rootrelationship of the standard deviation to the energy, the contamination only shrinks by a factor ofp k .
The overall survival/contamination ratio decreases by a factor of p k making the multiclassclassiﬁcation task more difﬁcult.
An interesting observation here is the amount of favoring requiredfor good generalization is linked to the number of positive training examples per class.
Indeed, ifwe consider a setting where the binary classiﬁcation problem generalizes well, and we switch tothe k class multiclass problem, then by increasing the number of training samples k fold (and thusmatching the number of positive training examples per class in the multiclass case to the binarycase) and keeping the number of features and feature weights constant we can generalize well formulticlass classiﬁcation.
(Appendix G in the Supplemental material elaborates on this phenomenon,as well as why it is somewhat surprising.)
Next, we present a brief overview of our proof that utilizes the survival/contamination analysisframework from Muthukumar et al.
[55] along with a typicality-inspired argument where the featuremargin (difference between largest and second largest feature) on the test point plays a key role.
Thecomplete proof is provided in Appendices B, C, D, E in the Supplemental material.
5.1 Proof sketch
Assume without loss of generality that for the test point x test ⇠ N (0 , I d ) , the true class is ↵ for some↵ 2 [ k ] .
Let x wtest be the weighted version of this test point.
A necessary and sufﬁcient condition forclassiﬁcation error is that for some  6 = ↵ ,  2 [ k ] ,
[Equation]
By converting into the unweighted feature space we obtain
[Equation]
where
[Equation]
Performing some algebraic manipulations and because  ↵ =   =  since both ↵ and  are favoredfeatures, we can rewrite this as
[Equation]
where
[Equation]
[Equation]
We divide by CN ↵ ,  to normalize the RHS above to have a standard normal distribution.
Next, byremoving the dependency on  , we obtain a sufﬁcient condition for correct classiﬁcation: [Equation] survival
[Equation]
normalized contamination
[Equation]
Here the min and max are over all competing features: 1    k,  6 = ↵ and the sum is overall d feature indices except ↵ and  , but we simplify the notation for convenience.
We show viaintermediate lemmas introduced in Appendix B of the Supplemental material that under the conditionsspeciﬁed in Theorem 5.1, with sufﬁciently high probability 11 , the relevant survival to contamination
SU/CN ratio grows at a polynomial rate n v for some v > 0 , the closest feature margin shrinks at aless-than-polynomial rate 1 / p ln nk , the survival variation decays at a polynomial rate n  u for some u > 0 .
Further, the magnitudes of the largest competing feature and the normalized contaminationare no more than p ln( nk ) .
This implies that the left-hand side of Equation (22) grows at a polynomial rate n v (ignoringlogarithmic terms) and dominates the right-hand side which grows at the much slower rate p ln nk .
Asurvival/contamination ratio also plays a key role in the analysis of the binary classiﬁcation problem in
Muthukumar et al.
[55] but in the multiclass setting, we additionally have the survival variation termand feature margin playing important roles since we are comparing different scores while predictingthe class label.
For correct classiﬁcation, the survival/contamination ratio must be sufﬁciently large,the survival variation must be small enough and the feature margin must be sufﬁciently large.
Conclusion
In this work we compute sufﬁcient conditions for good generalization of multiclass classiﬁcationin a bi-level overparameterized linear model with Gaussian features.
We observed that multiclassclassiﬁcation can generalize even when the regression problem does not generalize (for q + r > 1 ).
Further, the multiclass problem is “harder” than the binary problem because we have fewer positivetraining examples per class.
The nature of the training data complicates our analysis in the multiclasssetting since the true class labels are generated by comparing k features and thus we no longer haveindependence of the encoded class label y with any of these features.
This becomes relevant when wecompute bounds on the survival and contamination quantities since the Hanson-Wright inequality [63]is no longer applicable directly on the quantities of interest as was the case for the binary classiﬁcationproblem in prior work [55].
As a consequence of working around this non-independence we believethat our sufﬁcient conditions for good generalization in the regime q + r > 1 are loose.
Even though in our work we focus on the regime where regression does not work, q + r > 1 , we canextend the analysis to the regime where q + r < 1 by grinding through the expressions for survivaland contamination in this regime.
Even in this regime, for multiclass training data, survival is of theorder 1 k while contamination scales similarly to the regime q + r > 1 .
Thus, while it is true that forbinary classiﬁcation or a ﬁxed number of classes, the regime where regression works is a regimewhere classiﬁcation also works, this need not be true if there are too many classes.
We conjecture that the following is a set of necessary and sufﬁcient conditions for asymptoticallygood generalization (We elaborate on this in Appendix F in the Supplemental material):
Conjecture 6.1.
( Conjectured bi-level regions ): Under the bi-level ensemble model 4.2, whenthe true data generating process is 1-sparse (Assumption 4.1), as n ! 1 , the probability ofmisclassiﬁcation event P ( E err ) behaves as follows: [Equation]
[Equation]
The conjectured regions are visualized in (1d),(1e),(1f),(2d),(2e) and (2f) in Figure 2.
Subﬁgures (1d)and (2d) illustrate that we believe multiclass classiﬁcation with ﬁnitely many classes works if binaryclassiﬁcation works.
Further, comparing (1e) to (2e) when we increase q , the conjectured parameterregion where multiclass classiﬁcation works shrinks since we decrease the amount of favoring of truefeatures.
Interestingly, the nature of the looseness in our approach is such that our proof technique isable to recover a larger fraction of the conjectured region for larger q which intuitively is a result ofless favoring leading to stronger concentration of certain random quantities.
Tightening the potentiallooseness in our analysis and proving the converse result by computing sufﬁcient conditions for poorgeneralization of multiclass classiﬁcation are interesting avenues of future work.
Further, although the present analysis focuses on solutions that exactly interpolate the training data,we can extend our results to account for additional ridge regularization by viewing ridge regularizationas minimum-norm interpolation using augmented contamination-free features as in the Appendixof Muthukumar et al.
[54] and computing bounds leveraging tools from Tsigler and Bartlett [72].
Our assumption of the strict bi-level weighting model is largely to simplify the calculations andby substituting terms appropriately in our lemmas from Appendix B in the Supplemental material,it should be possible to compute results for other weighting models.
Finally, exploring the newphenomena that can be encountered as we go beyond the 1-sparse noiseless model is an excitingdirection for future work.