Mirror Descent Maximizes Generalized Margin and Can Be Implemented Efficiently
Anonymous Author(s)
AffiliationAddress
Abstract
Driven by the empirical success and wide use of deep neural networks, understanding the generalization performance of overparameterized models has become anincreasingly popular question.
To this end, there has been substantial effort tocharacterize the implicit bias of the optimization algorithms used, such as gradientdescent (GD), and the structural properties of their preferred solutions.
This paperanswers an open question in this literature: For the classification setting, whatsolution does mirror descent (MD) converge to?
Specifically, motivated by itsefficient implementation, we consider the family of mirror descent algorithms withpotential function chosen as the p -th power of the ℓ p -norm, which is an importantgeneralization of GD.
We call this algorithm p - GD .
For this family, we characterizethe solutions it obtains and show that it converges in direction to a generalizedmaximum-margin solution with respect to the ℓ p -norm for linearly separable classification.
While the MD update rule is in general expensive to compute and perhapsnot suitable for deep learning, p - GD is fully parallelizable in the same manner as
SGD and can be used to train deep neural networks with virtually no additionalcomputational overhead .
Using comprehensive experiments with both linear anddeep neural network models, we demonstrate that p - GD can noticeably affect thestructure and the generalization performance of the learned models.
Introduction
Overparameterized deep neural networks have enjoyed a tremendous amount of success in a widerange of machine learning applications [Brown et al., 2020, Dosovitskiy et al., 2020, Ramesh et al.
,2021, Schrittwieser et al., 2020].
However, as these highly expressive models have the capacity tohave multiple solutions that interpolate training data, and not all these solutions perform well on testdata, it is important to characterize which of these interpolating solutions the optimization algorithmsconverge to.
Such characterization is important as it helps understand the generalization performanceof these models, which is one of the most fundamental questions in machine learning.
Notably, it has been observed that even in the absence of any explicit regularization, the interpolatingsolutions obtained by the standard gradient-based optimization algorithms, such as (stochastic)gradient descent, tend to generalize well.
Recent research has highlighted that such algorithms favorparticular types of solutions, i.e., they implicitly regularize the learned models.
Importantly, suchimplicit biases are shown to play a crucial role in determining generalization performance, e.g.,[Donhauser et al., 2022, Neyshabur et al., 2014, Wilson et al., 2017, Zhang et al
[Table]
In the literature
[Table]
the implicit bias of first-order methods is first studied in linear settings since theanalysis is more tractable
[Table]
and also, there have been several theoretical and empirical evidence thatcertain insights from linear models translate to deep learning, e
[Table]
g. [Allen-Zhu et al., 2019, Bartlettet al., 2017, Belkin et al., 2019, Jacot et al., 2018, Lyu and Li, 2019, Nakkiran et al., 2021].
In thelinear setting, it is easier to establish implicit bias for regression tasks
[Table]
where square loss is typicallyused and it attains its minimum at a finite value.
For example, the implicit bias of gradient descent(GD) for square loss goes back to Engl et al.
Beyond GD, analysis of other popular algorithmssuch as the family of mirror descent (MD), which is an important generalization of GD, is moreinvolved and was established only recently by [Azizan and Hassibi, 2019a, Gunasekar et al., 2018].
Specifically, those works showed that mirror descent converges to the interpolating solution thatis closest to the initialization in terms of a Bregman divergence.
Thus, the implicit bias in linearregression is relatively well-understood by now.
On the other hand, in the classification setting, the implicit bias analysis becomes significantlymore challenging, and several questions remain open despite significant progress in the past fewyears.
A key differentiating factor in the classification setting is that the loss function does not attainits minimum at a finite value, and the weights have to grow to infinity.
It has been shown that forthe logistics loss, the gradient descent iterates converge to the ℓ 2 -maximum margin SVM solution indirection
[Ji and Telgarsky, 2019, Soudry et al., 2018].
However, such characterizations for mirrordescent are missing in the literature.
Because it is possible for optimization algorithms to exhibitimplicit bias in regression but not in classification (and vice versa)
[Gunasekar et al., 2018], resolvingthis gap of knowledge warrants careful analysis.
See Table 1 for a summary.
In this paper, we advance the understanding of the implicit regularization of mirror descent in theclassification setting.
In particular, inspired by their practicality, we focus on mirror descents withpotential function ψ ( · ) =
1 p ∥·∥ pp for p > 1 .
More specifically, such choice of potential results in anupdate rule that can be applied coordinate-wise , in the sense that updating the value at one coordinatedoes not depend on the values at other coordinates.
Thanks to this property, this subclass of mirrordescent can be implemented with no additional computational overhead , making it much morepractical than other algorithms in the literature; see Remark 10 for more details.
Our contributions.
In this paper, we make the following contributions:
■ We study mirror descent with potential 1 p ∥·∥ pp for p > 1 , which will call p -norm GD , andabbreviated as p - GD , as a practical and efficient generalization of the popular gradient descent.
■ We show that for separable linear classification with logistics loss, p - GD
exhibits implicit regularization by converging in direction to a ªgeneralizedº maximum-margin solution with respect to the ℓ p norm.
More generally, we show that, for monotonically decreasing loss functions, p - GDfollows the so-called regularization path, which is defined in Section 2.
■ We investigate the implications of our theoretical findings with two sets of experiments: Ourexperiments involving linear models corroborate our theoretical results, and real-world experiments with deep neural networks and popular datasets suggest that our findings carry over to suchnonlinear settings.
Our deep learning experiments further show that p - GD with different p lead tosignificantly different generalization performance.
Additional related work.
We remark that recent works also attempt to accelerate the convergenceof gradient descent to its implicit regularization, either by using an aggressive step size schedule[Ji and Telgarsky, 2021, Nacson et al., 2019] or with momentum [Ji et al., 2021].
Further, therehave been several results for other optimization methods, including steepest descent, AdaBoost, andvarious adaptive methods such as RMSProp and Adam [Gunasekar et al., 2018, Rosset et al., 2004,
Telgarsky, 2013, Wang et al., 2021].
However, to the best of our knowledge, there is no result formirror descent in the classification setting.
Background and Problem Setting
We are interested in the well-known classification setting.
Consider a collection of input-labelpairs { ( x i , y i ) } ni =1 ⊂ R d × {± 1 } and a classifier f w ( x ) , where w ∈ W .
For some loss functionℓ : R × {± 1 } → R , our goal is to minimize the empirical loss:
[Equation]
Throughout the paper, we assume that the classification loss function ℓ is decreasing, convex anddoes not attain its minimum, as in most common loss functions in practice (e.g., logistics loss andexponential loss).
Without loss of generality, we assume that inf ℓ ( · ) = 0 .
For our theoretical analysis, we consider a linear model, where the models can be expressed by [Equation]
w ⊤ x and w ∈ R d .
We also make the following assumptions about the data.
First, sincewe are mainly interested in the over-parameterized setting where d > n , we assume that the datais linearly separable, i.e., there exists w ∗ ∈ R d s.t. sign( ⟨ w ∗ , x i ⟩ ) =
y i for all i ∈ [ n ] .
We alsoassume that the inputs x i ’s are bounded.
More specifically, for our later purpose, we assume that forp > 0 , there exists some constant C so that max i ∥ x i ∥ q < C , where 1 /q + 1 /p = 1 .
Preliminaries on mirror descent.
The key component of mirror descent is a potential function .
Inthis work, we will focus on differentiable and strictly convex potentials defined on the entire domain 94 R n .
1 We call ∇ ψ the corresponding mirror map .
Given a potential, the natural notion of ªdistanceºassociated with the potential ψ is given by the Bregman divergence.
Definition 1 (Bregman divergence [Bregman, 1967]) .
For a mirror map ψ , the Bregman divergence
D ψ ( · , · ) associated to ψ is defined as
[Equation]
An important case is the potential ψ = ∥·∥ 2 2 , where ∥·∥ denotes the Euclidean norm.
Then, the Bregman divergence becomes D ψ ( x, y )
=
12 ∥ x − y ∥ 2 .
For more background on Bregman divergenceand its properties, see, e.g., [Bauschke et al., 2017, Section 2.2] and [Azizan and Hassibi, 2019b].
Mirror descent (MD) with respect to the mirror map ψ is a generalization of gradient descent wherewe use Bregman divergence as a measure of distance:
[Equation]
( MD) Equivalently, MD can be written as ∇ ψ ( w t +1 )
=
∇ ψ ( w t ) − η ∇ L ( w t ) .
We refer readers to [Bubeck,2015, Figure 4.1] for a nice illustration of mirror descent.
Also, see [Juditsky et al., 2011, Section5.7] for various examples of potentials depending on applications.
One property we will repeatedly use is the following [Azizan and Hassibi, 2019a]:
Lemma 2 ( MD identity) .
For any w ∈ R n , the following identities hold for MD :
[Equation]
Using Lemma 2, we make several new observations and prove the following useful statements.
Lemma 3.
For sufficiently small step size η such that ψ − ηL is convex, the loss is monotonicallydecreasing after each iteration of MD , i.e., L ( w t +1 ) ≤ L ( w t ) .
Lemma 4.
In a separable linear classification problem, if η is chosen sufficiently small s.t.
ψ − ηLis convex, then we have L ( w t ) → 0 as t → ∞ .
Hence, lim t →∞ ∥ w t ∥ = ∞ for any norm ∥·∥ .
The formal proofs of these lemmas can be found in Appendix A.
Remark 5.
We can relax the condition in Lemma 3 and 4 such that for a sufficiently small step sizeη , ψ − ηL is only locally convex at the iterates w t .
The relaxed condition allows us to analyze lossessuch as the exponential loss (see, e.g. footnote 2 of Soudry et al.
This condition can beconsidered as the mirror descent counterpart to the standard smoothness assumption in the analysisof gradient descent (see Lu et al.
Preliminaries on the convergence of linear classifier.
As we discussed above, the weights vectorw t diverges for mirror descent.
Here the main theoretical question is:
What direction does MD diverge to?
In other words, can we characterize w t / ∥ w t ∥ as t → ∞ ?
To answer this question, we define two special directions whose importance will be illustrated later.
Definition 6.
The regularization path with respect to the ℓ p -norm is defined as
[Equation]
And if the limit lim B →∞ ¯ w p ( B ) /B exists, we call it the regularized direction and denote it by u r p .
Definition 7.
The margin γ of the a linear classifier w is defined as γ ( w ) = min i =1 ,...,n y i ⟨ x i , w ⟩ .
The max-margin direction with respect to the ℓ p -norm is defined as:
[Equation]
[Equation]
And let ˆ γ p be the optimal value to the equation above.
Note that u m p is parallel to the hard-margin SVM solution w.r.t.
ℓ p -norm: argmin w {∥ w ∥ p : γ ( w ) ≥1 } .
Also note that the superscripts in u r p and u m p are not variables and we only use this notation todifferentiate the two definitions.
Prior results had shown that, in linear classification, gradient descentconverges in direction.
Theorem 8 (Soudry et al.
For separable linear classification with logistics loss, the gradientdescent iterates with sufficiently small step size converge in direction to u m 2 , i.e., lim t →∞ w t
∥
w t ∥ 2 = u m 2 .
Theorem 9 (Ji et al.
If the regularized direction u r p with respect to the ℓ 2 -norm exists, thenthe gradient descent iterates with sufficiently small step size converge to the regularized direction u r 2 ,
Mirror Descent with the p -th Power of ℓ p -norm
In this section, we investigate theoretical properties of the main algorithm of interest, namely mirrordescent with ψ ( · ) = 1 p ∥·∥ pp and for p > 1 .
We shall call this algorithm p -norm GD because itnaturally generalizes gradient descent to ℓ p geometry, and for conciseness, we will refer to thisalgorithm by the shorthand p - GD .
As noticed by Azizan et al.
[2021], this choice of mirror potential isparticularly of practical interest because the mirror map ∇ ψ updates becomes separable in coordinatesand thus can be implemented coordinate-wise independent of other coordinates:
[Equation]
Furthermore, we can extend upon the observation of Azizan et al.
[2021] and derive these identitiesthat allow us to better manipulate p - GD :
[Equation]
Remark 10.
Note that the coordinate-wise separability property is not shared among other relatedalgorithms in the literature.
For instance, the choice ψ = 12 ∥·∥ 2 q for 1 /p + 1 /q = 1 , which is referredto as the p -norm algorithm
[Gentile, 2003, Grove et al., 2001] is not fully coordinate-wise separablesince it requires computing ∥ w t ∥ p at each step (see, e.g., [Gentile, 2003, eq.
Another relatedalgorithm is steepest descent, where the Bregman divergence in MD is replaced with ∥·∥ 2 for generalnorm ∥·∥ .
However, for similar reasons, the update rule is not fully separable.
3.1 Main theoretical results
We extend Theorems 8 and 9 to the setting of p - GD .
We will resolve two major obstacles in theanalysis of implicit regularization in linear classification:
■ Our analysis approaches the classification setting as a limit of the regression implicit bias.
This argument gives stronger theoretical justification for utilizing the regularized direction (asemployed by Ji et al.
[2020]) and partially addresses the concern from Gunasekar et al.
[2018]that the implicit bias of regression and classification problems are ªfundamentally different.º
■ On a more technical note, analyzing the implicit bias requires handling the cross terms of the form⟨∇ ψ ( w ) , w ′ ⟩ , which lack direct geometric interpretations.
We demonstrate that for our potentialfunctions of interest, these terms can be nicely written and can be handled in the analysis.
We begin with the motivation behind the regularized direction, and consider the regression setting inwhich there exists some weight vector w such that L ( w ) = 0 .
Then, we can apply Lemma 2 to get
[Equation]
Since we assumed L ( w ) = 0 , the equation above implies that D ψ ( w, w t ) ≥ D ψ ( w, w t +1 ) forsufficiently small step-size η .
This can be interpreted as MD having a decreasing ªpotentialº of thefrom D ψ ( w, · ) during each step.
Using this property, Azizan and Hassibi [2019a] establishes theimplicit bias results of mirror descent in the regression setting.
However, such weight vector w does not exist in the classification setting.
One natural workaroundwould then be to choose a vector w so that L ( w ) ≤ L ( w t ) for all t ≤ T .
The following result, whichis a generalization of [Ji et al., 2020, Lemma 9], shows that one can in fact choose the referencevector w as a scalar multiple of the regularized direction.
Lemma 11.
If the regularized direction u r p exists, then ∀ α > 0 , there exists r α such that for any wwith ∥ w ∥ p > r α , we have L ((1 + α )
∥
w ∥ p u r p ) ≤ L ( w ) .
However, this does not resolve the issue altogether.
Recall from Lemma 4 that the loss approaches 0,and therefore one cannot choose a fixed reference vector w in the limit as T → ∞ .
But due to thehomogeneity of Bregman divergence (4b), we can scale u r p by a constant factor during each iteration,and, by doing so, we choose the reference vector w to be a ªmoving target.º
In other words, the ideabehind our analysis is that the classification problem is chasing after a regression one and wouldbehave similar to it in the limit.
Let us formalize this idea.
We begin with the following inequality:
[Equation]
where c t is taken to be ≈ ∥ w t ∥ p .
Now we modify (5) so that it can telescope over different iterations.
One way is to add
D
[Equation]
on both sides of (5) and move D ψ (cid:0) c t u
[Equation]
p
[Equation]
w
[Equation]
+
[Equation]
to the right-hand side as follows:
[Equation]
Summing over t = 0 , . .
, T − 1 gives us
[Equation]
The rest of the argument deals with simplifying quantities that do not cancel under telescoping sum.
For instance, in order to deal with
[Equation]
u [Equation] , we invoke the MD update rule as follows
[Equation]
where the last inequality follows from the intuition that u r p is the direction along which the lossachieves the smallest value and hence ∇ L ( w t ) must point away from u r p , i.e., it must be that [Equation] ∇ L ( w t ) , u r p (cid:11) ≲ ⟨∇ L ( w t ) , u ⟩ for any direction u .
The following result formalizes this intuition.
Corollary 12.
For w so that ∥ w ∥ p > r α , we have ⟨∇ L ( w ) , w ⟩ ≥ (1 + α ) ∥ w ∥ p (cid:10) ∇ L ( w ) , u r p (cid:11) .
Proof.
This follows from the convexity of L and Lemma 11: (cid:10)
[Equation]
−
[Equation]
(cid:11) ≥
L ( w ) − L ((1 + α ) ∥ w ∥ u r p ) ≥ 0 .
Now we are left with the terms ⟨− η ∇ L ( w t ) , w t ⟩ .
For general potential ψ , the quantity⟨− η ∇ L ( w t ) , w t ⟩ = ⟨∇ ψ ( w t +1 ) − ∇ ψ ( w t ) , w t ⟩ cannot be simplified.
On the other hand, dueto our choice of potential, one can invoke Lemma 2 to lower bound these quantities in terms of∥ w t +1 ∥ p and ∥ w t ∥ p , and this step is detailed in Lemma 18 in Appendix B.2.
Once we have established a lower bound on (cid:10) ∇ [Equation] , u
[Equation]
, we can turn (6) entirely into a telescoping sum andunwind the above process to show that D ψ
[Equation]
must converge to zero in the limit ast → ∞ .
Putting this all together, we obtain the following result.
Theorem 13.
For a separable linear classification problem, if the regularized direction u r p exists,then with sufficiently small step size, the iterates of p - GD converge to u r p in direction:
[Equation]
A formal proof of this theorem can be found in Appendix B.3.
We note that our proof furthersimplifies derivations using the separability of the mirror map.
The final missing piece would be theexistence of the regularized direction.
In general, finding the limit direction u r p would be difficult.
Fortunately, we can sometimes appeal to the max-margin direction that is much easier to compute.
The following result is a generalization of [Ji et al., 2020, Proposition 10] and shows that for commonlosses in classification, the regularized direction and the max-margin direction are the same, henceproving the existence of the former.
Proposition 14.
If we have a loss with exponential tail, e.g. lim z →∞ ℓ ( z ) e az =
b , then the regularized direction exists and it is equal to the max-margin direction u m p .
The proof of this result can be found in Appendix B.5.
Note that many commonly used losses inclassification, e.g., logistic loss, have exponential tail.
3.2 Asymptotic convergence rate
In this section, we characterize the rate of convergence in Theorem 13.
Following the proof of
Theorem 13, one can show the following result in the case of linearly separable data.
[-0 − 1 .− 0 .0 .0 .
-]
Corollary 15.
The following rate of convergence holds: D ψ
[Equation]
In order to fully understand the convergence rate, we need to characterize the asymptotic behavior of∥ w t ∥ p .
The next result precisely does that.
Recall that we assumed the dataset is bounded so thatmax i ∥ x i ∥ q ≤ C for 1 /p + 1 /q
=
1 , and the max-margin direction u m p satisfies (cid:10) x
i , u m p (cid:11) ≥ ˆ γ p ∀ i ∈[ n ] .
Then, we have the following bound on ∥ w t ∥ p .
Lemma 16.
For exponential loss ℓ ( z ) =
exp( − z ) , the asymptotic growth of ∥ w t ∥ p is contained inΘ(log t ) .
In particular, we have
[Equation]
The proof of this lemma can be found in Appendix C. Similar conclusions can be reached for otherlosses with exponential tail.
Therefore, in such cases, p - GD has poly-logarithmic rate of convergence.
Corollary 17.
For exponential loss, we have convergence rate D
ψ
[Equation]
Experiments
In this section, we investigate the behavior and performance of p - GD for various values of p .
Wenaturally pick p = 2 that corresponds to gradient descent, Because p - GD does not directly supportp = 1 and ∞ , we choose p = 1 . 1 as a surrogate for ℓ 1 , and p = 10 as a surrogate for ℓ ∞ .
We alsoconsider p = 1 . 5 , 3 , 6 to interpolate these points.
This section will present a summary of our results;the complete experimental setup and full results can be found in Appendices E and F.
4.1 Linear classification
Visualization of the convergence of p - GD .
To visualize the results of Theorem 13 and Corollary 17,we randomly generated a linearly separable set of 15 points in R 2 .
We then employed p - GD on thisdataset with exponential loss ℓ ( z ) =
exp( − z ) and fixed step size η = 10 − 4 .
We ran this experimentfor p = 1 . 5 , 2 , 3 and for 10 6 iterations.
In the illustrations of Figure 1, the mirror descent iterates w t have unbounded norm and convergein direction to u m p .
These results are consistent with Lemma 4 and with Theorem 13.
Moreover, aspredicted by Corollary 17, the exact rate of convergence for D ψ
[Equation]
is poly-logarithmicwith respect to the number of iterations.
Corollary 17 also indicates that the convergence rate wouldbe faster for larger p due to the larger exponent, and this is consistent with our observation in thesecond plot of Figure 1.
Finally, in the third plot of Figure 1, the norm of the iterates w t grows at alogarithmic rate, which is the same as the prediction by Lemma 16. {+− 1 .− 0 .0 .0 .1 . 0 − 1 .− 0 .0 .0 .
+}
.101010 5 Iteration 0 .0 .0 .0 .0 .
C o n v e r g e n ce ga p p = 1 .
5 p =p =101010 5 Iteration   || w t || p p = 1 .p = p =
u r p , w t / ∥ w t ∥ t (cid:1)
Implicit bias of p - GD in linear classification.
We now verify the conclusions of Theorem 13.
Tothis end, we recall that u m p is parallel to the SVM solution argmin w {∥ w ∥ p : γ ( w ) ≥ 1 } .
Hence, wecan exploit the linearity and rescale any classifier so that its margin is equal to 1 .
If the prediction of
Theorem 13 holds, then for each fixed value of p , the classifier generated by p - GD should have thesmallest ℓ p -norm after rescaling.
To ensure that u m p are sufficiently different for different values of p , we simulate an over-parameterizedsetting by randomly select 15 points in R 100 .
We used fixed step size of 10 − 4 and ran 250 thousanditerations for different p ’s. shows the results for p = 1 . 1 , 2 , 3 and 10; under each norm, we highlight the smallestclassifier in boldface .
Among the four classifiers we presented, p - GD with p = 1 . 1 has the smallestℓ 1 .
1 -norm.
And similar conclusions hold for p = 2 , 3 , 10 .
Although p - GD converges to u m p at avery slow rate, we are able to observe a very strong implicit bias of p - GD classifiers toward theirrespective ℓ p geometry in a highly over-parameterized setting.
This suggests we should be able totake advantage of the implicit regularization in practice and at a moderate computational cost.
Dueto space constraints, we defer a more complete result with additional values of p to Appendix F.1.
[Table]
4.2 Deep neural networks
[Table] Going beyond linear models, we now investigatep - GD in deep-learning settings in its impact onthe structure of the learned model and potentialimplications on the generalization performance.
As we had discussed in Section 3, the implementation of p - GD is straightforward ; to illustrate simplicity of implementation, we provide codesnippets in Appendix D.
Thus, we are able to effectively experiment with the behaviors p - GD in neuralnetwork training.
Specifically, we perform a set of experiments on the CIFAR-10 dataset [Krizhevskyet al., 2009].
We use the stochastic version of p - GD with different values of p .
We choose a variety ofnetworks: VGG [Simonyan and Zisserman, 2014], R ES N ET [He et al., 2016], M OBILE N ET [Sandleret al., 2018] and R EG N ET [Radosavovic et al., 2020].
Implicit bias of p - GD in deep neural networks.
Since the notion of margin is not well-defined inthis highly nonlinear setting, we instead visualize the impacts of p - GD ’s implicit regularization onthe histogram of weights (in absolute value) in the trained model. [Figure] [Table] = 1 .
[Table] p = 2 (SGD) [Table] p =
[Table]
[Table]
In Figure 2, we report the weight histograms of R ES N ET -18 models trained under p - GD withp = 1 . 1 , 2 , 3 and 10 .
Depending on p , we observe interesting differences between the histograms.
Note that the deep network is most sparse when p = 1 . 1 as most weights clustered around 0 .
Moreover, comparing the maximum weights, one can see that the case of p = 10 achieves thesmallest value.
Another observation is that the network becomes denser as p increases; for instance,there are more weights away from zero for the cases p = 3 , 10 .
These overall tendencies are alsoobserved for other deep neural networks; see Appendix F.2.
Generalization performance.
We next investigate the generalization performance of networkstrained with different p ’s. To this end, we adopt a fixed selection of hyper-parameters and then trainfour deep neural network models to 100% training accuracy with p - GD with different p ’s. As Table 3shows, interestingly the networks trained by p - GD with p = 3 consistently outperform other choicesof p ’s; notably, for M OBILE N ET and R EG N ET , the case of p = 3 outperforms the others by morethan 1%.
Somewhat counter-intuitively, the sparser network trained by p - GD with p = 1 . 1 does notexhibit better generalization performance, but rather shows worse generalization than other valuesof p .
Although these observations are not directly predicted by our theoretical results, we believethat understanding them would establish an important step toward understanding generalization ofoverparameterized models.
Due to space limit, we defer other experimental results to Appendix F.3.
I MAGE N ET experiments.
We also perform a similar set of experiments on the I MAGE N ETdataset [Russakovsky et al., 2015], and these results can be found in Appendix F.4.
Conclusion and Future Work
In this paper, we establish an important step towards better understanding implicit bias in theclassification setting, by showing that p - GD converges in direction to the generalized regularized/maxmargin directions.
We also run several experiments to corroborate our main findings along with thepracticality of p - GD .
The experiments are conducted in various settings: (i) linear models in both lowand high dimensions, (ii) real-world data with highly over-parameterized nonlinear models.
We conclude this paper with several important future directions:
■ Our analysis holds for ψ ( · ) =
∥·∥ pp , where we argued that this choice is key practical interest dueto its efficient algorithmic implementations.
It is mathematically interesting to generalize ouranalysis to other potential functions regardless of practical interest.
■ As we discussed in Section 4.2, different choices of p ’s for our p - GD algorithm result in differentgeneralization performance.
It would be interesting to investigate this phenomenon and to developtheory that explains why certain values of p lead to better generalization performance.
■ Another interesting question is to further investigate how practical techniques used in trainingneural networks (such as weight decay and adaptive learning rate) can affect the implicit bias andgeneralization properties of p - GD .
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] All of our claims accurately reflect the results inSections 3 and 4.
(b) Did you describe the limitations of your work?
[Yes] In Section 5, we described severaldirections where we can improve this work.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our paper investigates the foundational properties of mirror descent algorithms in learning; we do not believe there are any direct societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
for the assumptions we used and the motivation behind them.
(b) Did you include complete proofs of all theoretical results?
[Yes] All proofs are included
in the Appendix, and we have referenced them as we present our claims.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] They are
included in the supplemental material.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We gave an overview of our training setup in Section 4 and the
full details are given in Appendix E.
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[Yes] We reported the standard deviation whenever multiple
trials were performed.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] They are reported in Appendix E.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A] We only used public datasets.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]We did not curate any new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
The datasets we used are well-known; so, we
did not feel it was necessary to repeat that they do not contain sensitive information.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
There were no human subjects in our work.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]