Mirror Descent Maximizes Generalized Margin and Can Be Implemented Efficiently
Anonymous Author(s)
AffiliationAddress
Abstract
Driven by the empirical success and wide use of deep neural networks, understanding the generalization performance of overparameterized models has become anincreasingly popular question.
To this end, there has been substantial effort tocharacterize the implicit bias of the optimization algorithms used, such as gradientdescent (GD), and the structural properties of their preferred solutions.
This paperanswers an open question in this literature: For the classification setting, whatsolution does mirror descent (MD) converge to?
Specifically, motivated by itsefficient implementation, we consider the family of mirror descent algorithms withpotential function chosen as the p -th power of the ℓ p -norm, which is an importantgeneralization of GD.
We call this algorithm p - GD .
For this family, we characterizethe solutions it obtains and show that it converges in direction to a generalizedmaximum-margin solution with respect to the ℓ p -norm for linearly separable classification.
While the MD update rule is in general expensive to compute and notsuitable for deep learning, p - GD is fully parallelizable in the same manner as
SGD and can be used to train deep neural networks with virtually no additionalcomputational overhead .
Using comprehensive experiments with both linear anddeep neural network models, we demonstrate that p - GD can noticeably affect thestructure and the generalization performance of the learned models.
Introduction
Overparameterized deep neural networks have enjoyed a tremendous amount of success in a widerange of machine learning applications [Brown et al., 2020, Dosovitskiy et al., 2020, Ramesh et al.
,2021, Schrittwieser et al., 2020].
However, as these highly expressive models have the capacity tohave multiple solutions that interpolate training data and not all these solutions perform well on testdata, it is important to characterize which of these interpolating solutions the optimization algorithmsconverge to.
Such characterization is important as it helps understand the generalization performanceof these models, which is one of the most fundamental questions in machine learning.
Notably, it has been observed that even in the absence of any explicit regularization, the interpolating solutions obtained by the standard gradient-based optimization algorithms, such as (stochastic) gradient descent, tend to generalize well.
Recent research has highlighted that such optimization algorithms favor particular types of solutions, i e., they implicitly regularize the learned models
Importantly, such implicit biases are shown to play a crucial role in determining generalization
[Table]
In the literature
[Table]
the implicit bias of first-order methods is first studied in linear settings since theanalysis is more tractable
[Table]
and also, there have been several theoretical and empirical evidence thatcertain insights from linear models translate to deep learning, e
[Table]
g. [Allen-Zhu et al., 2019, Bartlettet al., 2017, Belkin et al., 2019, Jacot et al., 2018, Lyu and Li, 2019, Nakkiran et al., 2021].
In thelinear setting [Table] it is easier to establish implicit bias for regression tasks
[Table]
where square loss is typicallyused and it attains its minimum at a finite value.
For example, the implicit bias of gradient descent(GD) for square loss goes back to Engl et al.
Beyond GD, analysis of other popular algorithmssuch as the family of mirror descent (MD), which is an important generalization of GD, is moreinvolved and was established only recently by [Azizan and Hassibi, 2019, Gunasekar et al., 2018].
Specifically, those works showed that mirror descent converges to the interpolating solution thatis closest to the initialization in terms of a Bregman divergence.
Thus, the implicit bias in linearregression is relatively well-understood by now.
On the other hand, in the classification setting, the implicit bias analysis becomes significantlymore challenging, and several questions remain open despite significant progress in the past fewyears.
A key differentiating factor in the classification setting is that the loss function does not attainits minimum at a finite value, and the weights have to grow to infinity.
It has been shown that forthe logistics loss, the gradient descent iterates converge to the ℓ 2 -maximum margin SVM solution indirection
[Ji and Telgarsky, 2019, Soudry et al., 2018].
However, such characterizations for mirrordescent are missing in the literature.
Because it is possible for optimization algorithms to exhibitimplicit bias in regression but not in classification (and vice versa)
[Gunasekar et al., 2018], resolvingthis gap of knowledge warrants careful analysis.
See Table 1 for a summary.
In this paper, we advance the understanding of the implicit regularization of mirror descent in theclassification setting.
In particular, inspired by their practicality, we focus on mirror descents withpotential function ψ ( · ) =
1 p ∥·∥ pp for p > 1 .
More specifically, such choice of potential results in anupdate rule that can be applied coordinate-wise , in the sense that updating the value at one coordinatedoes not depend on the values at other coordinates.
Thanks to this property, this subclass of mirrordescent can be implemented with no additional computational overhead , making it much morepractical than other algorithms in the literature; see Remark 2 for more details.
Our contributions.
In this paper, we make the following contributions:
■ We study mirror descent with potential 1 p ∥·∥ pp for p > 1 , which will call p -norm GD , andabbreviated as p - GD , as a practical and efficient generalization of the popular gradient descent.
■ We show that for separable linear classification with logistics loss, p - GD exhibits implicit
regularization by converging in direction to a ªgeneralizedº maximum-margin solution with respect to the ℓ p norm.
More generally, we show that, for monotonically decreasing loss functions, p - GDfollows the so-called regularization path, which is defined in Section 2.
■ We further investigate the implications of our theoretical findings with a set of experiments. Our experiments involving linear models corroborate our theoretical results, and real-worldexperiments with deep neural network models and popular datasets suggest that our findings carryover to such nonlinear settings.
Additional related work.
We remark that recent works also attempt to accelerate the convergenceof gradient descent to its implicit regularization, either by using an aggressive step size schedule[Ji and Telgarsky, 2021, Nacson et al., 2019] or with momentum [Ji et al., 2021].
Further, therehave been several results for other optimization methods, including steepest descent, AdaBoost, andvarious adaptive methods such as RMSProp and Adam [Gunasekar et al., 2018, Rosset et al., 2004,
Telgarsky, 2013, Wang et al., 2021].
However, to the best of our knowledge, there is no result formirror descent in the classification setting.
Background and Problem Setting
We are interested in the well-known classification setting.
Consider a collection of input-labelpairs { ( x i , y i ) } ni =1 ⊂ R d × {± 1 } and a classifier f w ( x ) , where w ∈ W .
For some loss functionℓ : R × {± 1 } → R , our goal is to minimize the empirical loss:
[Equation]
Throughout the paper, we assume that the classification loss function ℓ is decreasing, convex anddoes not attain its minimum, as in most common loss functions in practice (e.g., logistics loss andexponential loss).
Without loss of generality, we assume that inf ℓ ( · ) = 0 .
For our theoretical analysis, we consider a linear model, where the models can be expressed byf w ( x ) =
w ⊤ x and w ∈ R d .
We also make the following assumptions about the data.
First, weassume that the data is linearly separable, i.e., there exists w ∗ ∈ R d s.t. sign( ⟨ w ∗ , x i ⟩ ) =
y i for alli ∈ [ n ] .
We also assume that the inputs x i ’s are bounded.
More specifically, for our later purpose weassume that for p > 0 , there exists some constant C so that max i ∥ x i ∥ q < C , where 1 /q + 1 /p = 1 .
Preliminaries on mirror descent.
The key component of mirror descent is a potential function .
Inthis work, we will focus on differentiable and strictly convex potentials defined on the entire domain 92 R n .
1 We call ∇ ψ the corresponding mirror map .
Given a potential, the natural notion of ªdistanceºassociated with the potential ψ is given by the Bregman divergence.
Definition 1 (Bregman divergence [Bregman, 1967]) .
For a mirror map ψ , the Bregman divergence
D ψ ( · , · ) associated to ψ is defined as
[Equation]
An important case to keep in mind is the potential ψ = ∥·∥ 2 2 , where ∥·∥ denotes the Euclidean norm, inwhich case the Bregman divergence simply becomes D ψ ( x, y )
=
12 ∥ x − y ∥ 2 .
For more backgroundon Bregman divergence, see, e.g., [Bauschke et al., 2017, Section 2.2].
Mirror descent (MD) with respect to the mirror map ψ is a generalization of gradient descent wherewe use Bregman divergence as a measure of distance:
[Equation]
Equivalently, MD can be written as ∇ ψ ( w t +1 )
=
∇ ψ ( w t ) − η ∇ L ( w t ) .
We refer readers to [Bubeck,2015, Figure 4.1] for a nice illustration of mirror descent.
Also, see [Juditsky et al., 2011, Section5.7] for various examples of potentials depending on applications.
One property we will repeatedly use is the following [Azizan and Hassibi, 2019]:
Lemma 1 ( MD identity) .
For any w ∈ R n , the following identities hold for MD :
[Equation]
Using Lemma 1, we make several new observations and prove the following useful statements.
Lemma 2.
For sufficiently small step size η such that ψ − ηL is convex, the loss is monotonicallydecreasing after each iteration of MD , i.e., L ( w t +1 ) ≤ L ( w t ) .
Lemma 3.
In a separable linear classification problem, if η is chosen sufficiently small s.t.
ψ − ηLis convex, then we have L ( w t ) → 0 as t → ∞ .
Hence, lim t →∞ ∥ w t ∥ = ∞ for any norm ∥·∥ .
The formal proofs of these lemmas can be found in Appendix A.
Remark 1.
Throughout the paper, we consider sufficiently small step size η such that ψ − ηL islocally convex at the iterates w t .
This can be considered as the mirror descent counterpart to thestandard smoothness assumption in the analysis of gradient descent (see Lu et al.
Preliminaries on the convergence of linear classifier.
As we discussed above, the weights vectorw t diverges for mirror descent.
Here the main theoretical question is:
What direction does MD diverges to?
In other words, can we characterize w t / ∥ w t ∥ as t → ∞ ?
To answer this question, we define two special directions whose importance will be illustrated later.
Definition 2.
The regularization path with respect to the ℓ p -norm is defined as¯
[Equation]
And if the limit lim B →∞ ¯ w p ( B ) /B exists, we call it the regularized direction and denote it by u r p .
Definition 3.
The margin γ of the a linear classifier w is defined as γ ( w ) = min i =1 ,...,n ⟨ x i , w ⟩ .
The max-margin direction with respect to the ℓ p -norm is defined as:
[Equation]
And let ˆ γ p be the optimal value to the equation above.
Note that u m p is parallel to the hard-margin SVM solution w.r.t.
ℓ p -norm: argmin w {∥ w ∥ p : γ ( w ) ≥1 } .
Prior results had shown that, in linear classification, gradient descent converges in direction.
Theorem 4 (Soudry et al.
For separable linear classification with logistics loss, the gradientdescent iterates with sufficiently small step size converge in direction to u m 2 , i.e., lim t →∞ w t
∥
w t ∥ 2 = u m 2 .
Theorem 5 (Ji et al.
If the regularized direction u r p with respect to the ℓ 2 -norm exists, thenthe gradient descent iterates with sufficiently small step size converge to the regularized direction u r 2 , i.e., lim t →∞ w t ∥ w t ∥ 2 = u r 2 .
Mirror Descent with the p -th Power of ℓ p -norm
In this section, we investigate theoretical properties of the main algorithm of interest, namely mirrordescent with ψ ( · ) = 1 p ∥·∥ pp and for p > 1 .
We shall call this algorithm p -norm GD because it naturallygeneralizes gradient descent to ℓ p geometry, and for conciseness, we will refer to this algorithm bythe shorthand p - GD .
As noticed by Azizan et al.
[2021], this choice of mirror potential is particularlyof practical interest because the mirror map ∇ ψ updates becomes separable in coordinates and thuscan be implemented coordinate-wise independent of other coordinates:
[Equation]
Furthermore, we can extend upon the observation of Azizan et al.
[2021] and derive these identitiesthat allow us to better manipulate p - GD :
[Equation]
Remark 2.
Note that the coordinate-wise separability property is not shared among other relatedalgorithms in the literature.
For instance, the choice ψ = 12 ∥·∥ 2 q for 1 /p + 1 /q = 1 , which is referredto as the p -norm algorithm
[Gentile, 2003, Grove et al., 2001] is not fully coordinate-wise separablesince it requires computing ∥ w t ∥ p at each step (see, e.g., [Gentile, 2003, eq.
Another relatedalgorithm is steepest descent, where the Bregman divergence in MD is replaced with ∥·∥ 2 for generalnorm ∥·∥ .
However, for similar reasons, the update rule is not fully separable.
3.1 Main theoretical results
We extend Theorems 4 and 5 to the setting of p - GD .
We will resolve two major obstacles in theanalysis of implicit regularization in linear classification:
■ Our analysis approaches the classification settings as a limit of regression implicit bias.
Thisargument gives stronger theoretical justification for utilizing regularized direction (as employedby Ji et al.
[2020]) and partially addresses the concern from Gunasekar et al.
[2018] that theimplicit bias of regression and classification problems are ªfundamentally different.º
■ On a more technical note, analyzing the implicit bias requires handling the cross terms of the form⟨∇ ψ ( w ) , w ′ ⟩ , which lacks direct geometric interpretations.
We demonstrate that for our potentialfunctions of interest, these terms can be nicely written and can be handled in the analysis.
We begin with the motivation behind the regularized direction, and consider the regression setting inwhich there exists some weight vector w such that L ( w ) = 0 .
Then, we can apply Lemma 1 to get
[Equation]
Since we assumed L ( w ) = 0 , the equation above implies that D ψ ( w, w t ) ≥ D ψ ( w, w t +1 ) forsufficiently small step-size η .
This can be interpreted as MD having a decreasing ªpotentialº of thefrom D ψ ( w, · ) during each step.
Using this property, Azizan and Hassibi [2019] establishes theimplicit bias results of mirror descent in the regression setting.
However, such weight vector w does not exist in the classification setting.
One natural workaroundwould then be to choose a vector w so that L ( w ) ≤ L ( w t ) for all t ≤ T .
The following result, whichis a generalization of [Ji et al., 2020, Lemma 9], shows that one can in fact choose the referencevector w as a scalar multiple of the regularized direction.
Lemma 6.
If the regularized direction u r p exists, then ∀ α > 0 , there exists r α such that for any wwith ∥ w ∥ p > r α , we have L ((1 + α )
∥
w ∥ p u r p ) ≤ L ( w ) .
However, this does not resolve the issue altogether.
Recall from Lemma 3 that the loss approaches 0,and therefore one cannot choose a fixed reference vector w in the limit as T → ∞ .
But due to thehomogeneity of Bregman divergence (4b), we can scale u r p by a constant factor during each iteration,and, by doing so, we choose the reference vector w to be a ªmoving target.º
In other words, the ideabehind our analysis is that the classification problem is chasing after a regression one and wouldbehave similar to it in the limit.
Let us formalize this idea.
We begin with the following inequality:
[Equation]
where c t is taken to be ≈ ∥ w t ∥ p .
Now we modify (5) so that it can telescope over different iterations.
One way is to add
D ψ
[Equation]
c
[Equation]
[Equation]
(cid:1) on both sides of (5) and move D ψ
[Equation]
[Equation]
(cid:1) to the right hand side as follows:
[Equation]
Summing over t = 0 , . .
, T − 1 gives us
[Equation]
The rest of the argument deals with simplifying quantities that do not cancel under telescoping sum.
For instance, in order to deal with
[Equation]
) , u r p (cid:11) , we invoke the MD update rule as follows
[Equation]
where the last inequality follows from the intuition that u r p is the direction along which the lossachieves the smallest value and hence ∇ L ( w t ) must point away from u r p , i.e., it must be that(cid:10) ∇ L ( w t ) , u r p (cid:11) ≲ ⟨∇ L ( w t ) , u ⟩ for any direction u .
The following result formalizes this intuition.
Corollary 7.
For w so that ∥ w ∥ p > r α , we have ⟨∇ L ( w ) , w ⟩ ≥ (1 + α [Equation] ∥ w ∥ p [Equation]
L
[Equation] u r p [Equation] Proof.
This follows from the convexity of L and Lemma 6:
[Equation]
[Equation]
Now we are left with the terms ⟨− η ∇ L ( w t ) , w t ⟩ .
For general potential ψ , the quantity⟨− η ∇ L ( w t ) , w t ⟩ = ⟨∇ ψ ( w t +1 ) − ∇ ψ ( w t ) , w t ⟩ cannot be simplified.
On the other hand, dueto our choice of potential, one can invoke Lemma 1 to lower bound these quantities in terms of∥ w t +1 ∥ p and ∥ w t ∥ p , and this step is detailed in Lemma 13 in Appendix B.2.
Once we have established a lower bound on
[Equation]
, we can turn (6) entirely into a telescoping sum andunwind the above process to show that D ψ (cid:16) u r p , w T / ∥ w t ∥ p (cid:17) must converge to zero in the limit ast → ∞ .
Putting this all together, we obtain the following result.
Theorem 8.
For a separable linear classification problem, if the regularized direction u r p exists, thenwith sufficiently small step size, the iterates of p - GD converge to u r p in direction:
[Equation]
A formal proof of this theorem can be found in Appendix B.3.
We note that our proof furthersimplifies derivations using the separability of mirror map.
The final missing piece would be theexistence of the regularized direction.
In general, finding the limit direction u r p would be difficult.
Fortunately, we can sometimes appeal to the max-margin direction that is much easier to compute.
The following result is a generalization of [Ji et al., 2020, Proposition 10] and shows that for commonlosses in classification, the regularized direction and the max-margin direction are the same, henceproving the existence.
Proposition 9.
If we have a loss with exponential tail, e.g. lim z →∞ ℓ ( z ) e az =
b , then the regularizeddirection exists and it is equal to the max-margin direction u m p .
The proof of this result can be found in Appendix B.5.
Note that many commonly used losses inclassification, e.g., logistic loss, has exponential tail.
3.2 Asymptotic convergence rate
In this section, we characterize the rate of convergence in Theorem 8.
Following the proof of
Theorem 8, one can show the following result in the case of linearly separable data.
− 1 .− 0 .0 .0 .1 .
{+0 − 1 .− 0 .0 .0 .
+}
.101010 5 Iteration 0 .0 .0 .0 .0 .
C o n v e r g e n ce ga p p = 1 .
5 p =p =101010 5 Iteration   || w t
|| p p = 1 .p =p = [Equation] Corollary 10.
The following rate of convergence holds: D ψ
[Equation]
In order to fully understand the convergence rate, we need to characterize the asymptotic behavior of∥ w t ∥ p .
The next result precisely does that.
Recall that we assumed the dataset is bounded so thatmax i ∥ x i ∥ q ≤ C for 1 /p + 1 /q
=
1 , and max-margin direction u m p satisfies
[Equation] n ] .
Then, we have the following bound on ∥ w t ∥ p .
Lemma 11.
For exponential loss ℓ ( z ) =
exp( − z ) , the asymptotic growth of ∥ w t ∥ p is contained inΘ(log t ) .
In particular, we have
[Equation]
The proof of this lemma can be found in Appendix C. Similar conclusions can be reached for otherlosses with exponential tail.
Therefore, in such cases, p - GD has poly-logarithmic rate of convergence.
Corollary 12.
For exponential loss, we have convergence rate D ψ (cid:16)
[Equation]
Experiments
In this section, we investigate the behavior and performance of p - GD for various values of p .
Wenaturally pick p = 2 that corresponds to gradient descent, Because p - GD does not directly supportp = 1 and ∞ , we choose p = 1 . 1 as a surrogate for ℓ 1 , and p = 10 as a surrogate for ℓ ∞ .
We alsoconsider p = 1 . 5 , 3 , 6 to interpolate these points.
This section will present a summary of our results;the complete experimental setup and full results can be found in Appendices E and F.
4.1 Linear classification
Visualization of the convergence of p - GD .
To visualize the results of Theorem 8 and Corollary 12,we randomly generated a linearly separable set of 15 points in R 2 .
We then employed p - GD on thisdataset with exponential loss ℓ ( z ) =
exp( − z ) and fixed step size η = 10 − 4 .
We ran this experimentfor p = 1 . 5 , 2 , 3 and for 10 6 iterations.
In the illustrations of Figure 1, the mirror descent iterates w t have unbounded norm and convergein direction to u m p .
These results are consistent with Lemma 3 and with Theorem 8.
Moreover, aspredicted by Corollary 12, the exact rate of convergence for D ψ (cid:0) u m p , w t / ∥ w t
∥ t (cid:1) is poly-logarithmicwith respect to the number of iterations.
Corollary 12 also indicates that the convergence rate wouldbe faster for larger p due to the larger exponent, and this is consistent with our observation in thesecond plot of Figure 1.
Finally, in the third plot of Figure 1, the norm of the iterates w t grows at alogarithmic rate, which is the same as the prediction by Lemma [-16. − 1 .− 0 .0 .0 .1 . 0 − 1 .− 0 .0 .0 .
-]11.
Implicit bias of p - GD in linear classification.
We now verify the conclusions of Theorem 8.
Tothis end, we recall that u m p is parallel to the SVM solution argmin w {∥ w ∥ p : γ ( w ) ≥ 1 } .
Hence, wecan exploit the linearity and rescale any classifier so that its margin is equal to 1 .
If the prediction of
Theorem 8 holds, then for each fixed value of p , the classifier generated by p - GD should have thesmallest ℓ p -norm after rescaling.
To ensure that u m p are sufficiently different for different values of p , we simulate an over-parameterizedsetting by randomly select 15 points in R 100 .
We used fixed step size of 10 − 4 and ran 250 thousanditerations for different p ’s. Table 2 shows the results for p = 1 . 1 , 2 , 3 and 10; under each norm, we highlight the smallestclassifier in boldface .
Among the four classifiers we presented, p - GD with p = 1 . 1 has the smallestℓ 1 .
1 -norm.
And similar conclusions hold for p = 2 , 3 , 10 .
Although p - GD converges to u m p at avery slow rate, we are able to observe a very strong implicit bias of p - GD classifiers toward theirrespective ℓ p geometry in a highly over-parameterized setting.
This suggests we should be able totake advantage of the implicit regularization in practice and at a moderate computational cost.
Dueto space constraint, we defer a more complete result with additional values of p to Appendix F.1. [Table] [Table] [Table]
[Table]
4.2 Deep neural networks
Going beyond linear models, we now investigatep - GD in deep-learning settings in its impact onthe structure of the learned model and potentialimplications on the generalization performance.
As we had discussed in Section 3, the implementation of p - GD is straightforward ; to illustrate simplicity of implementation, we provide codesnippets in Appendix D.
Thus, we are able to effectively experiment with the behaviors p - GD in neuralnetwork training.
Specifically, we perform a set of experiments on the CIFAR-10 dataset [Krizhevskyet al., 2009].
We use the stochastic version of p - GD with different values of p .
We choose a variety ofnetworks: VGG [Simonyan and Zisserman, 2014], R ES N ET [He et al., 2016], M OBILE N ET [Sandleret al., 2018] and R EG N ET [Radosavovic et al., 2020].
Implicit bias of p - GD in deep neural networks.
Since the notion of margin is not well-defined inthis highly nonlinear setting, we instead visualize the impacts of p - GD ’s implicit regularization onthe histogram of weights (in absolute value) in the trained model.
In Figure 2, we report the weight histograms of R ES N ET -18 models trained under p - GD withp = 1 . 1 , 2 , 3 and 10 .
Depending on p , we observe interesting differences between the histograms.
Note that the deep network is most sparse when p = 1 . 1 as most weights clustered around 0 .
Moreover, comparing the maximum weights, one can see that the case of p = 10 achieves thesmallest value.
Another observation is that the network becomes denser as p increases; for instance,there are more weights away from zero for the cases p = 3 , 10 .
These overall tendencies are alsoobserved for other deep neural networks; see Appendix F.2.
Generalization performance.
We next investigate the generalization performance of networkstrained with different p ’s. To this end, we adopt a fixed selection of hyper-parameters and then trainfour deep neural network models to 100% training accuracy with p - GD with different p ’s. As Table 3shows, interestingly the networks trained by p - GD with p = 3 consistently outperform other choicesof p ’s; notably, for M OBILE N ET and R EG N ET , the case of p = 3 outperforms the others by morethan 1%.
Somewhat counter-intuitively, the most sparse network trained by p - GD with p = 1 . 1does not exhibit better generalization performance, but rather shows worse generalization than othervalues of p .
Although these observations are out of the scope of our theoretical results, we believethat understanding them would establish an important step toward understanding generalization ofoverparameterized models.
Due to space limit, we defer other experimental results to Appendix F.3. [Figure] [Table] p = 1 .
[Table]
p = 2 (SGD) [Table] [Table] p =90.
[Table]
[Table]
[Table]
[Table]
94.07 ± .
p =88.78 ± .
93.55 ± .
92.60 ± . 92.97 ± . I MAGE N ET experiments. We also perform a similar set of experiments on the I MAGE N ETdataset [Russakovsky et al., 2015], and these results can be found in Appendix F.4.
Conclusion and Future Work
In this paper, we establish an important step toward understanding the implicit bias in the classificationsetting, by showing that p - GD converges in direction to the generalized regularized/max-margindirections.
We also run several experiments to corroborate our main findings along with the practicalityof p - GD .
The experiments are conducted in various settings: (i) linear models in both low and highdimensions, (ii) real-world data with highly over-parameterized nonlinear models.
We conclude this paper with several important future directions:
■ Our analysis is for ψ ( · ) =
∥·∥ pp .
It would be interesting to generalize our results to the case of
other potential functions of practical interest
■ As we discussed in Section 4.2, different choices of p ’s for our p - GD algorithm result in differentgeneralization performance.
It would be interesting to theoretically investigate these results.
■ Another interesting question is to investigate how practical techniques used in training neuralnetworks (such as weight decay and adaptive learning rate) can affect the implicit bias andgeneralization properties of p - GD .