Mirror Descent Maximizes Generalized Margin and Can Be Implemented Efﬁciently
Abstract
Driven by the empirical success and wide use of deep neural networks, understanding the generalization performance of overparameterized models has become an increasingly popular question.
To this end, there has been substantial effort to characterize the implicit bias of the optimization algorithms used, such as gradient descent (GD), and the structural properties of their preferred solutions.
This paper answers an open question in this literature: For the classiﬁcation setting, what solution does mirror descent (MD) converge to?
Speciﬁcally, motivated by its efﬁcient implementation, we consider the family of mirror descent algorithms with potential function chosen as the p -th power of the ℓ p -norm, which is an important generalization of GD.
We call this algorithm p - GD .
For this family, we characterize the solutions it obtains and show that it converges in direction to a generalized maximum-margin solution with respect to the ℓ p -norm for linearly separable classiﬁcation.
While the MD update rule is in general expensive to compute and perhaps not suitable for deep learning, p - GD is fully parallelizable in the same manner as
SGD and can be used to train deep neural networks with virtually no additional computational overhead .
Using comprehensive experiments with both linear and deep neural network models, we demonstrate that p - GD can noticeably affect the structure and the generalization performance of the learned models.
Introduction
Overparameterized deep neural networks have enjoyed a tremendous amount of success in a wide range of machine learning applications [Brown et al., 2020, Dosovitskiy et al., 2020, Ramesh et al., 2021, Schrittwieser et al., 2020].
However, as these highly expressive models have the capacity to have multiple solutions that interpolate training data and not all these solutions perform well on test data, it is important to characterize which of these interpolating solutions the optimization algorithms converge to.
Such characterization is important as it helps understand the generalization performance of these models, which is one of the most fundamental questions in machine learning.
Notably, it has been observed that even in the absence of any explicit regularization, the interpolating solutions obtained by the standard gradient-based optimization algorithms, such as (stochastic) gradient descent, tend to generalize well.
Recent research has highlighted that such algorithms favor particular types of solutions, i.e., they implicitly regularize the learned models.
Importantly, such implicit biases are shown to play a crucial role in determining generalization performance, e.g., [Donhauser et al., 2022, Neyshabur et al., 2014, Wilson et al., 2017, Zhang et al., 2021].
In the literature, the implicit bias of ﬁrst-order methods is ﬁrst studied in linear settings since the analysis is more tractable, and also, there have been several theoretical and empirical evidence that certain insights from linear models translate to deep learning, e.g. [Allen-Zhu et al., 2019, Bartlett et al., 2017, Belkin et al., 2019, Jacot et al., 2018, Lyu and Li, 2019, Nakkiran et al., 2021]. In the linear setting, it is easier to establish implicit bias for regression tasks, where square loss is typically [Table]
This work
[Table] and
[Table]
used and it attains its minimum at a ﬁnite value.
For example, the implicit bias of gradient descent (GD) for square loss goes back to Engl et al.
Beyond GD, analysis of other popular algorithms such as the family of mirror descent (MD), which is an important generalization of GD [Table] is more involved and was established only recently by [Azizan and Hassibi, 2019a, Gunasekar et al., 2018].
Speciﬁcally, those works showed that mirror descent converges to the interpolating solution that is closest to the initialization in terms of a Bregman divergence.
Thus, the implicit bias in linear regression is relatively well-understood by now.
On the other hand, in the classiﬁcation setting, the implicit bias analysis becomes signiﬁcantly more challenging [Table] and several questions remain open despite signiﬁcant progress in the past few years.
A key differentiating factor in the classiﬁcation setting is that the loss function does not attain its minimum at a ﬁnite value, and the weights have to grow to inﬁnity.
It has been shown that for the logistics loss, the gradient descent iterates converge to the ℓ 2 -maximum margin SVM solution in direction
[Ji and Telgarsky, 2019, Soudry et al., 2018].
However, such characterizations for mirror descent are missing in the literature.
Because it is possible for optimization algorithms to exhibit implicit bias in regression but not in classiﬁcation (and vice versa)
[Gunasekar et al., 2018], resolving this gap of knowledge warrants careful analysis.
See Table 1 for a summary.
In this paper, we advance the understanding of the implicit regularization of mirror descent in the classiﬁcation setting.
In particular, inspired by their practicality, we focus on mirror descents with potential function ψ ( · ) = 1 p k·k pp for p > 1 .
More speciﬁcally, such choice of potential results in an update rule that can be applied coordinate-wise , in the sense that updating the value at one coordinate does not depend on the values at other coordinates.
Thanks to this property, this subclass of mirror descent can be implemented with no additional computational overhead , making it much more practical than other algorithms in the literature; see Remark 10 for more details.
Our contributions.
In this paper, we make the following contributions:
(cid:4) We study mirror descent with potential 1 p k·k pp for p > 1 , which will call p -norm GD , and abbreviated as p - GD , as a practical and efﬁcient generalization of the popular gradient descent.
(cid:4) We show that for separable linear classiﬁcation with logistics loss, p - GD exhibits implicit regularization by converging in direction to a “generalized” maximum-margin solution with respect to the ℓ p norm.
More generally, we show that, for monotonically decreasing loss functions, p - GD follows the so-called regularization path, which is deﬁned in Section 2.
(cid:4)
We investigate the implications of our theoretical ﬁndings with two sets of experiments: Our experiments involving linear models corroborate our theoretical results, and real-world experiments with deep neural networks and popular datasets suggest that our ﬁndings carry over to such nonlinear settings.
Our deep learning experiments further show that p - GD with different p lead to signiﬁcantly different generalization performance.
Additional related work.
We remark that recent works also attempt to accelerate the convergence of gradient descent to its implicit regularization, either by using an aggressive step size schedule
[Ji and Telgarsky, 2021, Nacson et al., 2019] or with momentum [Ji et al., 2021].
Further, there have been several results for other optimization methods, including steepest descent, AdaBoost, and various adaptive methods such as RMSProp and Adam [Gunasekar et al., 2018, Min et al., 2022,
Rosset et al., 2004, Telgarsky, 2013, Wang et al., 2021].
A mirror-descent-based algorithm for explicit regularization was recently proposed by Azizan et al.
[2021a].
Comparatively, there has been very little progress on mirror descent in the classiﬁcation setting.
Li et al.
[2021] consider a mirror descent, but their assumptions are not applicable beyond the ℓ 2 geometry.
To the best of our knowledge, there is no result for more general mirror descent algorithms in the classiﬁcation setting.
Background and Problem Setting
We are interested in the well-known classiﬁcation setting.
Consider a collection of input-label pairs { ( x i , y i ) } ni =1 ⊂ R d × {± 1 } and a classiﬁer f w ( x ) , where w ∈ W .
For some loss function ℓ : R × {± 1 } → R , our goal is to minimize the empirical loss:
[Equation]
Throughout the paper, we assume that the classiﬁcation loss function ℓ is decreasing, convex and does not attain its minimum, as in most common loss functions in practice (e.g., logistics loss and exponential loss).
Without loss of generality, we assume that inf ℓ ( · ) = 0 .
For our theoretical analysis, we consider a linear model, where the models can be expressed by f w ( x ) =
w ⊤ x and w ∈ R d .
We also make the following assumptions about the data.
First, since we are mainly interested in the over-parameterized setting where d > n , we assume that the data is linearly separable, i.e., there exists w ∗ ∈ R d s.t. sign( h w ∗ , x i i ) =
y i for all i ∈ [ n ] .
We also assume that the inputs x i ’s are bounded.
More speciﬁcally, for our later purpose, we assume that for p > 0 , there exists some constant C so that max i k x i k q < C , where 1 /q + 1 /p = 1 .
Preliminaries on mirror descent.
The key component of mirror descent is a potential function .
In this work, we will focus on differentiable and strictly convex potentials deﬁned on the entire domain R n .
We call ∇ ψ the corresponding mirror map .
Given a potential, the natural notion of “distance” associated with the potential ψ is given by the Bregman divergence.
Deﬁnition 1 (Bregman divergence [Bregman, 1967]) .
For a mirror map ψ , the Bregman divergence
D ψ ( · , · ) associated to ψ is deﬁned as
[Equation]
An important case is the potential ψ = k·k 2 2 , where k·k denotes the Euclidean norm.
Then, the Bregman divergence becomes D ψ ( x, y )
=
12 k x − y k 2 .
For more background on Bregman divergence and its properties, see, e.g., [Bauschke et al., 2017, Section 2.2] and [Azizan and Hassibi, 2019b].
Mirror descent (MD) with respect to the mirror map ψ is a generalization of gradient descent where we use Bregman divergence as a measure of distance:
[Equation]
Equivalently, MD can be written as ∇ ψ ( [Equation]
=
[Equation] ∇ L ( w t ) .
We refer readers to [Bubeck, 2015, Figure 4.1] for a nice illustration of mirror descent.
Also, see [Juditsky et al., 2011, Section 5.7] for various examples of potentials depending on applications.
One property we will repeatedly use is the following [Azizan and Hassibi, 2019a]:
Lemma 2 ( MD identity) .
For any w ∈ R n , the following identities hold for MD :
[Equation]
Using Lemma 2, we make several new observations and prove the following useful statements.
Lemma 3.
For sufﬁciently small step size η such that ψ − ηL is convex, the loss is monotonically decreasing after each iteration of MD , i.e., L ( w t +1 ) ≤ L ( w t ) .
Lemma 4.
In a separable linear classiﬁcation problem, if η is chosen sufﬁciently small s.t. ψ − ηL is convex, then we have L ( w t ) → 0 as t → ∞ .
Hence, lim t →∞ k w t k = ∞ for any norm k·k .
The formal proofs of these lemmas can be found in Appendix A.
Remark 5.
We can relax the condition in Lemma 3 and 4 such that for a sufﬁciently small step size η , ψ − ηL is only locally convex at the iterates w t .
The relaxed condition allows us to analyze losses such as the exponential loss (see, e.g. Footnote 2 of Soudry et al.
This condition can be considered as the mirror descent counterpart to the standard smoothness assumption in the analysis of gradient descent (see Lu et al.
Preliminaries on the convergence of linear classiﬁer.
As we discussed above, the weights vector w t diverges for mirror descent.
Here the main theoretical question is:
What direction does MD diverge to?
In other words, can we characterize w t / k w t k as t → ∞ ?
To answer this question, we deﬁne two special directions whose importance will be illustrated later.
Deﬁnition 6.
The regularization path with respect to the ℓ p -norm is deﬁned as
[Equation]
And if the limit lim B →∞ ¯ w p ( B ) /B exists, we call it the regularized direction and denote it by u r p .
Deﬁnition 7.
The margin γ of the a linear classiﬁer w is deﬁned as γ ( w ) =
min i =1 ,...,n y i
h x i , w i .
The max-margin direction with respect to the ℓ p -norm is deﬁned as:
[Equation]
And let ˆ γ p be the optimal value to the equation above.
Note that u m p is parallel to the hard-margin SVM solution w.r.t.
ℓ p -norm: argmin w {k w k p : γ ( w ) ≥ 1 } .
Also note that the superscripts in u r p and u m p are not variables and we only use this notation to differentiate the two deﬁnitions.
Prior results had shown that, in linear classiﬁcation, gradient descent converges in direction.
Theorem 8 (Soudry et al.
For separable linear classiﬁcation with logistics loss, the gradient descent iterates with sufﬁciently small step size converge in direction to u m 2 , i.e., lim t →∞ w t k w t k 2 = u m 2 .
Theorem 9 (Ji et al.
If the regularized direction u r p with respect to the ℓ 2 -norm exists, then the gradient descent iterates with sufﬁciently small step size converge to the regularized direction u r 2 ,
i.e., lim t →∞ w t k w t k 2 = u r 2 .
Mirror Descent with the p -th Power of ℓ p -norm
In this section, we investigate theoretical properties of the main algorithm of interest, namely mirror descent with ψ ( · ) = 1 p k·k p p and for p > 1 .
We shall call this algorithm p -norm GD because it naturally generalizes gradient descent to ℓ p geometry, and for conciseness, we will refer to this algorithm by the shorthand p - GD .
As noticed by Azizan et al.
[2021b], this choice of mirror potential is particularly of practical interest because the mirror map ∇ ψ updates becomes separable in coordinates and thus can be implemented coordinate-wise independent of other coordinates:
[Equation]
Furthermore, we can extend upon the observation of Azizan et al.
[2021b] and derive these identities that allow us to better manipulate p - GD :
[Equation]
Remark 10.
Note that the coordinate-wise separability property is not shared among other related algorithms in the literature.
For instance, the choice ψ = 12 k·k 2 q for 1 /p
+ 1 /q
= 1 , which is referred to as the p -norm algorithm
[Gentile, 2003, Grove et al., 2001] is not fully coordinate-wise separable since it requires computing k w t k p at each step (see, e.g., [Gentile, 2003, eq.
Another related algorithm is steepest descent, where the Bregman divergence in MD is replaced with k·k 2 for general norm k·k .
However, for similar reasons, the update rule is not fully separable.
3.1 Main theoretical results
We extend Theorems 8 and 9 to the setting of p - GD .
We will resolve two major obstacles in the analysis of implicit regularization in linear classiﬁcation: (cid:4) Our analysis approaches the classiﬁcation setting as a limit of the regression implicit bias.
This argument gives stronger theoretical justiﬁcation for utilizing the regularized direction (as employed by Ji et al.
[2020]) and partially addresses the concern from Gunasekar et al.
[2018] that the implicit bias of regression and classiﬁcation problems are “fundamentally different.”
(cid:4) On a more technical note, analyzing the implicit bias requires handling the cross terms of the form h∇ ψ ( w ) , w ′ i , which lack direct geometric interpretations.
We demonstrate that for our potential functions of interest, these terms can be nicely written and can be handled in the analysis.
We begin with the motivation behind the regularized direction, and consider the regression setting in which there exists some weight vector w such that L ( w ) = 0 .
Then, we can apply Lemma 2 to get
[Equation]
Since we assumed L ( w ) = 0 , the equation above implies that D ψ ( w, w t ) ≥ D ψ ( w, w t +1 ) for sufﬁciently small step-size η .
This can be interpreted as MD having a decreasing “potential” of the from D ψ ( w, · ) during each step.
Using this property, Azizan and Hassibi [2019a] establishes the implicit bias results of mirror descent in the regression setting.
However, such weight vector w does not exist in the classiﬁcation setting.
One natural workaround would then be to choose a vector w so that L ( w ) ≤ L ( w t ) for all t ≤ T .
The following result, which is a generalization of [Ji et al., 2020, Lemma 9], shows that one can in fact choose the reference vector w as a scalar multiple of the regularized direction.
Lemma 11.
If the regularized direction u r p exists, then ∀ α > 0 , there exists r α such that for any w with k w k p > r α , we have L ((1 + α ) k w k p u r p ) ≤ L ( w ) .
However, this does not resolve the issue altogether.
Recall from Lemma 4 that the loss approaches 0, and therefore one cannot choose a ﬁxed reference vector w in the limit as T → ∞ .
But due to the homogeneity of Bregman divergence (4b), we can scale u r p by a constant factor during each iteration, and, by doing so, we choose the reference vector w to be a “moving target.”
In other words, the idea behind our analysis is that the classiﬁcation problem is chasing after a regression one and would behave similar to it in the limit.
Let us formalize this idea.
We begin with the following inequality:
[Equation]
where c t is taken to be ≈ k w t k p .
Now we modify (5) so that it can telescope over different iterations.
One way is to add
D
[Equation]
c t +1 u r p , w t + (cid:1) on both sides of (5) and move D ψ (cid:0) c t u r p , w t + (cid:1) to the right-hand side as follows:
D ψ (cid:0)
[Equation]
t
[Equation]
(cid:10)
[Equation] [Equation]
( c
[Equation]
+1 − c
[Equation]
u
[Equation]
(cid:11) Summing over t = 0 , . .
, T − 1 gives us
D ψ
[Equation]
(cid:0)
[Equation]
The rest of the argument deals with simplifying quantities that do not cancel under telescoping sum.
For instance, in order to deal with
[Equation]
, we invoke the MD update rule as follows
[Equation]
where the last inequality follows from the intuition that u r p is the direction along which the loss achieves the smallest value and hence ∇ L ( w t ) must point away from u r p , i.e., it must be that (cid:10) ∇ L ( w t ) , u r p
[Equation]
h∇ L ( w t ) , u i for any direction u .
The following result formalizes this intuition.
Corollary 12.
For w so that k w k p > r α , we have h∇ L ( w ) , w i ≥ (1 + α ) k w k p
[Equation]
Proof.
This follows from the convexity of L and Lemma 11:
[Equation]
u
[Equation]
Now we are left with the terms h− η ∇ L ( w t ) , w t i .
For general potential ψ , the quantity h− η ∇ L ( w t ) , w t i =
h∇ ψ ( w t +1 )
− ∇ ψ ( w t ) , w t i cannot be simpliﬁed.
On the other hand, due to our choice of potential, one can invoke Lemma 2 to lower bound these quantities in terms of k w t +1 k p and k w t k p , and this step is detailed in Lemma 18 in Appendix B.2.
Once we have established a lower bound on
[Equation]
p (cid:11) , we can turn (6) entirely into a telescoping sum and unwind the above process to show that D ψ
[Equation]
must converge to zero in the limit as t → ∞ .
Putting this all together, we obtain the following result.
Theorem 13.
For a separable linear classiﬁcation problem, if the regularized direction u r p exists, then with sufﬁciently small step size, the iterates of p - GD converge to u r p in direction:
[Equation]
A formal proof of this theorem can be found in Appendix B.3.
We note that our proof further simpliﬁes derivations using the separability of the mirror map.
The ﬁnal missing piece would be the existence of the regularized direction.
In general, ﬁnding the limit direction u r p would be difﬁcult.
Fortunately, we can sometimes appeal to the max-margin direction that is much easier to compute.
The following result is a generalization of [Ji et al., 2020, Proposition 10] and shows that for common losses in classiﬁcation, the regularized direction and the max-margin direction are the same, hence proving the existence of the former.
Proposition 14.
If we have a loss with exponential tail, e.g. lim z →∞ ℓ ( z ) e az =
b , then the regularized direction exists and it is equal to the max-margin direction u m p .
The proof of this result can be found in Appendix B.5.
Note that many commonly used losses in classiﬁcation, e.g., logistic loss, have exponential tail.
− 1 .− 0 .0 .0 .1 .
0 − 1 .− 0 .0 .0 .
.101010 5 Iteration 0 .0 .0 .0 .0 .
C o n v e r g e n ce ga p p
= 1 .p =p =101010 5 Iteration   || w t
|| p p
= 1 .p =p =
(cid:0) u r p , w t / k w t k t (cid:1)
3.2 Asymptotic convergence rate
In this section, we characterize the rate of convergence in Theorem 13.
Following the proof of
Theorem 13, one can show the following result in the case of linearly separable data.
Corollary 15.
The following rate of convergence holds:
[Equation]
In order to fully understand the convergence rate, we need to characterize the asymptotic behavior of k w t k p .
The next result precisely does that.
Recall that we assumed the dataset is bounded so that max i k x i k q ≤ C for 1 /p
+ 1 /q
= 1 , and the max-margin direction
u m p satisﬁes
[Equation]
[ n ] .
Then, we have the following bound on k w t k p .
Lemma 16.
For exponential loss ℓ ( z ) =
exp( − z ) , the asymptotic growth of k w t k p is contained in Θ(log t ) .
In particular, we have
[Equation]
The proof of this lemma can be found in Appendix C. Similar conclusions can be reached for other losses with exponential tail.
Therefore, in such cases, p - GD has poly-logarithmic rate of convergence.
Corollary 17.
For exponential loss, we have convergence rate
[Equation]
Experiments
In this section, we investigate the behavior and performance of p - GD for various values of p .
We naturally pick p = 2 that corresponds to gradient descent, Because p - GD does not directly support p = 1 and ∞ , we choose p = 1 . 1 as a surrogate for ℓ 1 , and p = 10 as a surrogate for ℓ ∞ .
We also consider p = 1 . 5 , 3 , 6 to interpolate these points.
This section will present a summary of our results; the complete experimental setup and full results can be found in Appendices E and F.
4.1 Linear classiﬁcation
Visualization of the convergence of p - GD .
To visualize the results of Theorem 13 and Corollary 17, we randomly generated a linearly separable set of 15 points in R 2 .
We then employed p - GD on this dataset with exponential loss ℓ ( z ) =
exp( − z ) and ﬁxed step size η = 10 − 4 .
We ran this experiment for p = 1 . 5 , 2 , 3 and for 10 6 iterations.
In the illustrations of Figure 1, the mirror descent iterates w t have unbounded norm and converge in direction to u m p .
These results are consistent with Lemma 4 and with Theorem 13.
Moreover, as predicted by Corollary 17, the exact rate of convergence for D ψ (cid:0) u m p , w t / k w t k t (cid:1) is poly-logarithmic with respect to the number of iterations.
Corollary 17 also indicates that the convergence rate would be faster for larger p due to the larger exponent, and this is consistent with our observation in the second plot of Figure 1.
Finally, in the third plot of Figure 1, the norm of the iterates w t grows at a logarithmic rate, which is the same as the prediction by Lemma 16.
Implicit bias of p - GD in linear classiﬁcation.
We now verify the conclusions of Theorem 13.
To this end, we recall that u m p is parallel to the SVM solution argmin w {k w k p : γ ( w ) ≥ 1 } .
Hence, we can exploit the linearity and rescale any classiﬁer so that its margin is equal to 1 .
If the prediction of
Theorem 13 holds, then for each ﬁxed value of p , the classiﬁer generated by p - GD should have the smallest ℓ p -norm after rescaling.
To ensure that u m p are sufﬁciently different for different values of p , we simulate an over-parameterized setting by randomly select 15 points in R 100 .
We used ﬁxed step size of 10 − 4 and ran 250 thousand iterations for different p ’s.
Table 2 shows the results for p = 1 . 1 , 2 , 3 and 10; under each norm, we highlight the smallest classiﬁer in boldface .
Among the four classiﬁers we presented, p - GD with p = 1 . 1 has the smallest ℓ 1 .
1 -norm.
And similar conclusions hold for p = 2 , 3 , 10 .
Although p - GD converges to u m p at a very slow rate, we are able to observe a very strong implicit bias of p - GD classiﬁers toward their respective ℓ p geometry in a highly over-parameterized setting.
This suggests we should be able to take advantage of the implicit regularization in practice and at a moderate computational cost.
Due to space constraints, we defer a more complete result with additional values of p to Appendix F.1.
[Table]
4.2 Deep neural networks
Going beyond linear models, we now investigate p - GD in deep-learning settings in its impact on the structure of the learned model and potential implications on the generalization performance.
As we had discussed in Section 3, the implementation of p - GD is straightforward ; to illustrate simplicity of implementation, we provide code snippets in Appendix D.
Thus, we are able to effectively experiment with the behaviors p - GD in neural network training.
Speciﬁcally, we perform a set of experiments on the CIFAR-10 dataset [Krizhevsky et al., 2009].
We use the stochastic version of p - GD with different values of p .
We choose a variety of networks: VGG [Simonyan and Zisserman, 2014], R ES N ET [He et al., 2016], M OBILE N ET [Sandler et al., 2018] and R EG N ET [Radosavovic et al., 2020].
Implicit bias of p - GD in deep neural networks.
Since the notion of margin is not well-deﬁned in this highly nonlinear setting, we instead visualize the impacts of p - GD ’s implicit regularization on the histogram of weights (in absolute value) in the trained model.
In Figure 2, we report the weight histograms of R ES N ET -18 models trained under p - GD with p = 1 . 1 , 2 , 3 and 10 .
Depending on p , we observe interesting differences between the histograms.
Note that the deep network is most sparse when p = 1 . 1 as most weights clustered around 0 .
Moreover, comparing the maximum weights, one can see that the case of p = 10 achieves the smallest value.
Another observation is that the network becomes denser as p increases; for instance, there are more weights away from zero for the cases p = 3 , 10 .
These overall tendencies are also observed for other deep neural networks; see Appendix F.2.
Generalization performance.
We next investigate the generalization performance of networks trained with different p ’s. To this end, we adopt a ﬁxed selection of hyper-parameters and then train four deep neural network models to 100% training accuracy with p - GD with different p ’s. As
[Figure]
p
[Figure]
VGG
[Table] [Table]
p = 1 .88.19 ± .
92.63 ± .
91.16 ± .
91.21 ± .
p = 2 (SGD) 90.15 ± .
93.90 ± .
91.97 ± .
92.75 ± .
p =90.85 ± .
94.01 ± .
[Table]
=
[Table]
Table 3 shows, interestingly the networks trained by p - GD with p = 3 consistently outperform other choices of p ’s; notably, for M OBILE N ET and R EG N ET , the case of p = 3 outperforms the others by more than 1%.
Somewhat counter-intuitively, the sparser network trained by p - GD with p = 1 .does not exhibit better generalization performance, but rather shows worse generalization than other values of p .
Although these observations are not directly predicted by our theoretical results, we believe that they nevertheless establish an important step toward understanding generalization of overparameterized models.
Due to space limit, we defer other experimental results to Appendix F.3.
I MAGE N ET experiments.
We also perform a similar set of experiments on the I MAGE N ET dataset [Russakovsky et al., 2015], and these results can be found in Appendix F.4.
Conclusion and Future Work
In this paper, we establish an important step towards better understanding implicit bias in the classiﬁcation setting, by showing that p - GD converges in direction to the generalized regularized/maxmargin directions.
We also run several experiments to corroborate our main ﬁndings along with the practicality of p - GD .
The experiments are conducted in various settings: (i) linear models in both low and high dimensions, (ii) real-world data with highly over-parameterized nonlinear models.
We conclude this paper with several important future directions:
(cid:4) Our analysis holds for ψ ( · )
= k·k pp , where we argued that this choice is key practical interest due to its efﬁcient algorithmic implementations.
It is mathematically interesting to generalize our analysis to other potential functions regardless of practical interest.
(cid:4) As we discussed in Section 4.2, different choices of p ’s for our p - GD algorithm result in different generalization performance.
It would be interesting to investigate this phenomenon and to develop theory that explains why certain values of p lead to better generalization performance.
(cid:4) Another interesting question is to further investigate how practical techniques used in training neural networks (such as weight decay and adaptive learning rate) can affect the implicit bias and generalization properties of p - GD .
Acknowledgement
The authors thank MIT UROP students Tiffany Huang and Haimoshri Das for contributing to the experiments in Section 4.2.
The authors acknowledge the MIT SuperCloud and Lincoln Laboratory
Supercomputing Center for providing computing resources that have contributed to the results reported within this paper.
This work was supported in part by MathWorks and the MIT-IBM Watson
AI Lab. K.A. acknowledges support through graduate assistantship in part from the NSF grant 1846088, the ONR grant N00014-20-1- and the VBFF.
N.A. also acknowledges support from the Edgerton Career Development Professorship.