Combining Implicit and Explicit Regularization for Efﬁcient Learning in Deep Networks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Recent work on implicit regularization has focused on gradient trajectories duringthe optimization process in attempting to explain why deep networks favor certainkinds of solutions over others.
For deep linear neural networks, it has been shownthat gradient descent/ﬂow implicitly regularizes toward low-rank solutions inmatrix completion/factorization tasks, akin to an accelerative pre-conditioning,whose effects become more pronounced with increased depth.
In light of this,we propose an explicit penalty that mirrors the rank minimization behavior andgeneralization performance independently of depth, but interestingly only takeseffect with Adam and some of its close variants—it outperforms many approachesin matrix completion and is robust to a wide range of parameter and data regimes.
Our ﬁndings suggest that explicit regularization can play a key role together withthe choice of optimization algorithm in designing different, desirable forms ofregularization, and that a more nuanced understanding of this interplay may benecessary.
Introduction
Much work has poured into understanding and explaining how and why highly over-parameterizedneural networks generalize so effectively with more parameters than training examples despite longheld notions to the contrary.
This generalization puzzle has deepened as over-parameterized deeplearning models often generalize well even without any explicit penalization, simply by optimizingits training error on an under-determined problem.
To explain this and related phenomena [8, 9], previous works have focused on how gradient-basedoptimization induces an implicit bias on optimization trajectories, tending towards solutions withcertain properties [6, 19] like those that generalize well or are simpler in some respect.
In contrast,although explicit regularization has seen wide-spread usage for improving performance in a varietyof model frameworks such as ` p norm weight constraints or dropout [43], its effectiveness and rolein generalization in deep networks is not certain given its inability to prevent over-ﬁtting to randomlabels [50] or its absence in deep models which generalize well on their own.
Some more recent work has focused on a simple test-bed setting that admits a crisp formulationfor understanding the mechanisms through which implicit regularization operates, namely, matrixcompletion : given some subset of an unknown, low-rank matrix W ? , the task is to recover the unseenentries.
A key observation has focused on how gradient descent on a shallow linear neural network,with sufﬁciently small learning rate and near-zero initialization, pushes towards low-rank solutionswithout explicit regularization or constraints [19].
This phenomenon led [19] to conjecture thatgradient descent is inducing an implicit regularization that minimizes the nuclear norm.
This conjecture has been put into doubt by work showing that implicit regularization inﬂuences thetrajectory of gradient descent in such a way so as to (a) promote solutions with few large singularvalues and many small ones [6], with the gap becoming more pronounced as depth increases (i.e.deep matrix factorization by [6]), and (b) resemble a combination of momentum and adaptiveregularization [5], itself akin to explicit accelerated gradient methods [28] popular in deep learning.
Contributions Despite these ﬁndings, questions still remain:
• Are there explicit norm-based penalties that can mirror the effect of implicit regularization soas to better harness this phenomenon for better or more efﬁcient learning?
Previous work [6] has conjectured that implicit regularization cannot be characterized by explicit norm-basedpenalties, but whether norm-based penalties can produce similar effects is unclear.
• What role does acceleration play, if at all?
Similarly, can this accelerative effect be replicatedor, at least, imitated?
Previous work [5] conjectured that the implicit preconditioning oraccelerative effect from depth is somewhat orthogonal to explicit acceleration methods.
To try and shed more light on these questions, we propose and consider the following penalty thattakes the ratio between the nuclear norm of a matrix and its Frobenius norm:
k W k ? / k W k F .
This 49 penalty can be viewed as an adaptation of the Hoyer measure [23] but for matrices in the spectraldomain, measuring the relative spectral sparsity of a non-zero matrix W .
We show the following:
A depth 1 linear neural network (i
e. a degenerate network without any depth) equipped withour penalty can produce the same rank reduction with comparable, or better, generalizationperformance as deep matrix factorization.
In short, depth is no longer necessary for learning.2.
The above result only occurs under Adam and its close variants.
This suggests that dif-
ferent optimizers may have different inductive biases that interact differently with explicitregularizers in preferring certain solutions over others.
With the penalty, we achieve better generalization and rank-reduction performance beyondmany techniques (Fig. 4), even in low data regimes (i.e. fewer observed entries during
training) where the nuclear norm minimizer has no recovery guarantees.
In settings with non-trivial depth, this penalty induces the same overall effect on rank and generalization largely independent of depth, showing a degree of depth invariance
Previous conjectures in related work [5, 6, 50] have largely dismissed the importance of explicitregularization in understanding generalization in deep learning, leaving its overall role unclear.
Similarly, despite its relative popularity, Adam and other adaptive optimizers have received their shareof doubt [44, 45] regarding their effectiveness in producing desirable solutions.
Our results suggest asubtle but perhaps important interplay between the choice of the optimizer, its inductive bias, and theexplicit penalty in producing different trajectories, and, hence, different kinds of solutions.
If so, thenin these interactions, explicit regularization may yet have a role to play.
Paper Overview In Section 2, we review previous work.
Section 3 details our experiments andﬁndings.
In Section 4, we extend our experiments to an empirical benchmark and we conclude in
Section 5 with a discussion on future work.
Related Work
Implicit regularization and its role in explaining optimization and generalization have been studiedextensively, with more recent work focusing on optimization trajectories in deep, over-parameterizedsettings [5–7].
One avenue has focused on linear neural networks (LNNs) [18, 21, 26, 41] given theirrelative tractability and the similarity in learning dynamics between linear and non-linear networks.
For instance, in settings with multiple optima, [20] has shown how LNNs trained on separable datacan converge with an implicit bias towards the maximum margin solution.
Others [6] have demonstrated how gradient ﬂow converges towards low-rank solutions where increased depth acts as an accelerative pre-conditioning [5].
[2] has shown more directly, in the case ofnatural and vanilla gradient descent, how different choices of pre-conditioning impact bias-variancetrade-offs and population risk in over-parameterized linear regression, but leaves the question openfor pre-conditioning in adaptive gradient optimizers.
Acceleration in optimization dates back to Nesterov [36] and even Newton’s method, using secondorder information or a limited memory of past quantities (e.g. gradients) to inform optimization as aform of pre-conditioning.
Adam’s [28] effectiveness—and its closely inspired variants [16, 31, 32]—in optimizing deep networks faster makes the importance of adaptive pre-conditioning more explicit.
Though some [44, 45] have doubted the effectiveness of adaptive gradient methods and illustratedpotential issues in certain cases, such as harming generalization, others [30, 48, 51] have shownadvantages from preconditioning and adaptive optimizers; despite speeding up optimization, however,the effect of pre-conditioning on generalization has been less clear.
Some [1, 15, 47] have argued thatthe “sharpness” of minima can vary depending on the choice of optimizer.
Matrix completion and factorization have themselves long been an important focus for areas likesignal recovery [11] and recommendation systems [10] while theoretical bounds for exact recoveryand convergence [12, 13, 33, 38] have led to practical algorithms [24, 34].
We refer to [14] for a morecomprehensive survey.
These and related tasks have also served as test-beds to better understandimplicit regularization.
[6, 19] use matrix factorization and sensing to study gradient ﬂow’s implicitbias towards low-rank solutions and conjecture that solutions produced by algorithmic regularizationmay not correspond to any minimization of norm-based penalties at all.
More recently, [46] studiedthe implicit bias of mirror descent on matrix sensing, showing its correspondence to a solution thateither interpolates between the nuclear and Frobenius norms depending on the underlying matrix.
Some have started to explicitly compare implicit and explicit regularization; in certain least squaresproblems, [52] showed how SGD’s implicit biases can generalize no worse or even outperform ridgeregression, requiring quadratically more samples to compete with SGD.
To the best of our knowledge,our paper is the ﬁrst to propose the ratio of the nuclear norm to the Frobenius norm as a penalty inmatrix completion to study the interplay between explicit and implicit regularization.
Experiments and Findings
3.1 Setup
Formally, we have a ground-truth matrix W ? 2 R m ⇥ n whose observed entries are indexed by the set⌦ .
We deﬁne the projection P ⌦ ( W ? ) to be a m ⇥ n matrix such that the entries with indices in ⌦remain while the rest are masked with zeros.
We are interested in the following optimization:
[Equation]
where k · k F is the Frobenius norm, R ( · ) is an explicit penalty, and   0 is the tuning parame- 113 ter.
Following [3], we deﬁne a deep linear neural network (DLNN) through the following overparameterization, or deep factorization, of W :
[Equation]
under the loss function in (1) where W i 2 R d i ⇥ d i  1 , i 2 { 1 , . . .
, N } is the i -th layer/weight matrix.
Here, N also denotes the depth of the network/factorization where N = 2 corresponds to matrixfactorization , N  3 corresponds to deep matrix factorization , and N = 1 is the degenerate case(i.e.
no depth).
We refer to the matrix W , the product of the N weight matrices in Eq.
(2), as theend-product matrix as per [6], which is the network’s produced solution in estimating W ? .
In thefollowing analyses, we focus on rank-5 matrices with d 0 = . . .
=
d N = n = 100 (i.e. layers/weightmatrices W i 2 R 100 ⇥ 100 , 8 i ) for illustrative purposes, but our results extend similarly to other ranks(e.g.
see Appendix A.2).
We leave a detailed discussion of experiment settings to Appendix A.1.
, without penalty
We ﬁrst establish a baseline by reviewing the dynamics under both un-regularized gradient descentand un-regularized Adam to better understand their dynamics in the presence of our penalty later on.
Gradient Descent Previous work [6] has shown that, via gradient ﬂow, depth enhances the implicitregularization towards low rank, characterized by the following singular value trajectory:
[Equation]
where ˙  i is the time derivative of  i ( t ) , the i -th singular value of W ( t ) , { u i , v i } are the left and rightsingular vectors of W ( t ) corresponding to  ( t ) i , N is the network’s depth, and r W L ( W ( t )) is the
[Figure] loss gradient with respect to the end-product matrix W at time t .
Moving forward, we suppress thedependency on t for simplicity.
Note that full dynamics require non-degeneracy (i [Figure]
e. depth N > 1 );otherwise, they reduce down to the product of the loss gradient and singular vectors.
We see thesedynamics at play (Fig. 1, top row) where higher depths produce solutions with lower test error andrank (i.e. better singular value separation).
For non-degenerate cases ( N  2 ), as depth increases, larger singular values and their changesbecome more pronounced and enhanced than their smaller counterparts, driving singular valueseparation and rank decrease—this is seen empirically (Fig. 1 top row), resulting in a decrease inthe solution’s effective rank.
The singular values also evolve at uneven paces across depth levels;increasing depth increases the gap in the time it takes between the 1 st and 5 th largest singular valuesto develop while also taking longer to stabilize—these effects are even more pronounced whencomparing the ﬁve largest singular values to the remaining ones.
This differential pace by which thesingular values evolve is largely shaped, and constrained, by depth.
Only with sufﬁcient depth ( N  3 ) do solutions produced by un-penalized gradient descent minimizerank so as to recover the original rank of the matrix in question and generalize better than the nuclearnorm solution.
Additionally, [5, 6] have also described the trajectory dynamics of the end-productmatrix W itself (i.e. the produced solution to Eq. under the same assumptions:
[Equation] where vec ( · ) denotes (column-ﬁrst order) vectorization, ˙ W = dW/dt , P W = P
Nj =1 ( W > W ) N  j N ⌦ ( WW > ) j  1 N , and ⌦ denotes the Kronecker product. P W can seen as a pre-conditoning onto thegradient that, with sufﬁcient depth ( N  2) , accelerates movements already taken in the optimization[5].
As depth (i.e. over-parameterization) increases, this acceleration intensiﬁes.
Adam Analyzing Adam can be difﬁcult given its exponentially moving average of gradients; tosimplify our analysis, we borrow the assumptions from [4] to, though imperfect, characterize Adam’sdynamics via gradient ﬂow (see Appendix A.4 for more details).
We set the discount factorsequal  1 =  2 =  and assume that the estimate of discounted squared gradients ˆ v can be wellapproximated by its expectation.
Lemma 1.
Under the assumptions above and of [6], the trajectory of the end-product matrix W andits singular values  i , respectively, can be approximately characterized as:
[Equation]
(5) where P W,G = P Nj =1 (( W > W ) N  j N ⌦ ( WW > )
j  1 N ) G j is p.s.d.
and G j is a diagonal positive matrix.
Speciﬁcally, G j = diag(vec( S j )) , [ S j ] m,n = [( r W j L ( W ) [-
-]2 + s 2 j )  1 / 2 ] m,n , r W j L ( W ) =@
L ( W ) / @
W j , and s 2 j = var( r W j L ( W )) .
Proof.
See Appendix A.4.
Via our approximation, the pre-conditioning induced by Adam can be characterized as a modiﬁcationof gradient descent’s P W , normalizing each layer-component by the square-root of its sum of itssquared layer-wise loss gradient ( @ L ( W ) / @
W j ) 2 and the gradient variance, s 2 j , before summingacross all depth levels.
Unlike before in the regularized case, the variance of the loss gradient comesinto play—whereas before the pre-conditioning serves as a purely accelerative effect that intensiﬁeswith depth, normalization by a function of the variance can either dampen or further acceleratethe trajectory.
Empirically, in the deep regime ( N > 2 ), we see that Adam minimizes rank toexact/near-exact rank recovery more efﬁciently and uniformly than gradient descent via faster ( 10 4vs. 10 5 ⇠ 10 6 ), more uniform convergence (Fig. 1, bottom row).
Singular value dynamics exhibitsigniﬁcantly more uniform evolution regardless of depth in contrast to gradient descent (Fig. 1, right).
The trajectory and convergence of singular values, the effective rank of W , and test error appearaccelerated .
Given these differences, Adam appears to arrive at different kinds of solutions thangradient descent.3.3 Depth, with penalty
[Figure]
Gradient Descent We ﬁrst characterize dynamics under gradient ﬂow with our proposed penalty.
Lemma 2.
Under the assumptions of [5], with the proposed penalty, the singular values of theend-product matrix W under gradient descent with penalty (i.e. ˙  reg r ) evolve in the following fashion:
[Equation]
(6) where  > 0 is the regularization parameter and ˙
GF r denotes the singular value trajectory underun-regularized gradient ﬂow (Eq.
Proof.
Follows from the gradient of our penalty, applied to Eq.
The penalty can be seen as an additive component to the original trajectories in Eq.
This newcomponent also intensiﬁes with increased depth, making movements in larger singular values morepronounced than those of smaller ones, which together can drive faster, more pronounced singularvalue separation than before ( 2 N  1 N vs. 3 N  2 2 ), depending on  .
Empirically, comparing the un-regularized case (Fig. 1 top row) to the now regularized case (Fig.
2top row), we indeed see that the regularized case features faster convergence in rank and fasterdevelopment of the top singular values as a result of this accentuation.
Here, increased depthcontinues to push down rank but, unlike before (Eq.
(3), Eq.
(4)), the penalty now allows singular value trajectories to depend on their own magnitudes even without depth ( N = 1 ), providing anadditional degree of freedom.
The penalty also allows each singular value to depend on its relativeweight within the distribution of singular values through (1  || W
||
⇤ / || W || F ) rather than just its 194 own absolute magnitude. We can similarly characterize the trajectory of the end-product matrix W . Lemma 3.
Under the same assumptions, with the proposed penalty, the end-product matrix Wevolves in the following fashion: [Equation] where U , V contain the left and right singular vectors of W , P W is the pre-conditioning in Eq.
(4) ,and ˜ ⌃ = || W ||
⇤ || W || F ⌃ where ⌃ contains the singular values of W along its diagonal.
Proof.
Follows from the gradient of our penalty, applied to Eq. We see that the depth-dependent accelerative pre-conditioning P W now acts on a new term, similar to
Eq.
(6), which serves as an additional, spectral-based component that can accelerate the trajectory ofthe produced solution W , especially at higher depths. Looking at the diagonals, the new term can beseen as a spectrally re-scaled version of W that inﬂuences its trajectory in a way that accounts foreach singular value’s weight relative to its entire spectrum: u > i v i
[Equation]
⌘ . Empirically, wesee (Fig. 2 top row, left plot) that the end-product matrix W produced achieves better test error andconverges faster at higher depths ( N  4 ) compared to analogous depths in the un-regularized case(Fig.
1 top row, left plot).
Adam With Adam, the penalty’s effect differs considerably in terms of the solutions found.
Lemma 4.
Under the same assumptions, with the proposed penalty, the end-product matrix W andits singular values evolves in the following fashion:
[Equation]
Proof.
Follows from Eq.
(5) with our penalty.
See Appendix A.4 for details.
Empirically, we note a new development: namely, the network displays a large degree of depthinvariance in that it can now push rank down almost independently of depth (Fig. 2, bottom row),even at depth 2.
All depths recover the rank of the underlying matrix (i.e. 5) unlike gradient descent.
Starting with a shallow network ( N = 2 ), increasing the depth does not induce any material changesin the produced solutions as the penalized DLNN under Adam produces low-rank solutions thatachieve exact rank recovery and low generalization error on par with, if not better than before (Fig. 2).
It also retains some of the accelerative effect of Adam with respect to the speed of convergenceand behavior of effective rank and how the singular values develop.
Compared to before, we seea beneﬁcial accelerative effect that endows more favorable generalization properties and smootherdevelopment of singular values whose convergence speed is at least an order of magnitude faster thanunder gradient descent—which do not change much with depth, if at all.
As singular values evolvenear identically, we see that their paths are relatively robust to depth—all producing the near-identicalbehavior without much dependence on depth (Fig. 2, right).
3.4 Degenerate case, with penalty
Given the non-trivial interaction effects seen thus far, we now consider the degenerate depth 1 network.
It is known [50] that gradient descent in this setting converges to the minimum Frobenius (i.e. ` 2 )norm solution, which does not necessarily induce spectral sparsity or low ranked-ness.
Applying ourpenalty to this network under gradient descent does little to affect the solution meaningfully (Fig. 3).
This similarly holds for un-regularized Adam—without depth, learning is heavily constrained.
Yet, surprisingly, despite being ineffective with gradient descent, our penalty is effective under
Adam (Fig. 3, bottom) even without depth or over-parameterization, achieving generalization onpar with, if not better than, deep matrix factorization ( N > 2 ) under gradient descent.
We note that [Figure] replacing our penalty with other proxies of rank/spectral sparsity such as the nuclear norm doesnot work (Table 1).
Under un-regularized gradient ﬂow, at depth 1, singular value and end-productdynamics collapse: singular value trajectories reduce to u > i r W L ( W ( t )) v i , as noted earlier [Figure] and theaccelerative pre-conditioning reduces to just the identity P W =
I mn .
We see this empirically (e.g.
Fig.
3 and Table 1) where solutions produced un-regularized gradient descent at depth 1 generalizepoorly and fail to reduce rank.
By removing depth, we see that a key component in helping adepth 1 network learn is its variance-weighted gradient term r W L ( W ) · G , as deﬁned in Eq.
(5),along with the term P W,G
[Equation]
⌘ , which reduces to G ⇣
[Equation]
|
[Equation]
⌘ withoutdepth.
Interestingly, the variance of the loss gradient and the ratio ⌘ 2 = var( r W L ( W )) / r W L ( W ) 2formed from r W L ( W ) · G , akin to a signal-to-noise ratio, have come up in other works as importantquantities that is strongly predictive of generalization capabilities [25] or is essential to ﬁndingoptimal variance adaption factors of stochastic gradients [4].
3.5 Comparative Performance
[Table] [Table] [Table]
[Table] Adagrad [17] [Table] [Table]
[Table]
[Table]
Adamax [28]
[Table]
[Table]
Other Combinations Our results do not preclude the possibility that other optimizers and penaltiescan produce similar or better effects.
For completeness, inspired by the properties of our proposedpenalty (e.g. ratio-like, spectral, non-convex), we experiment with various optimizers and penalties in
Table 1 to compare optimizer-penalty interactions across shallow ( N = 1 ) and deep ( N = 3 ) settings.
We note that our ratio penalty under Adam and its close variant Adamax, both initially proposed in [Figure] Matrix Completion: [Figure]
SI
[28], are the only combinations that largely show depth invariance in test error and rank recoverywhereas others require depth to either push down rank or generalize better.
Although the nuclearnorm shows a degree of depth invariance in rank reduction, it is unable to ﬁnd well-generalizingsolutions and it fails in rank reduction under gradient descent where depth is still necessary.
Given thespace considerations of this paper, we leave a fuller treatment and characterization of each optimizer’sinductive biases and their interactions with different regularizers for future work.
Comparative Performance Finally, we note our penalty’s comparative performance (Fig. 4) againstother methodologies for synthetic matrix completion across various sample sizes (i.e. the numberof entries out of 100,000 made available for training).
Our depth 1 network with Adam and thepenalty outperforms other methods including the un-regularized DLNN in test error, degree of rankcompression, and convergence speed across several data regimes.
At lower sample sizes, even atdepth 1, the penalized network generalizes better than methods such as SoftImpute [34], OptSpace[27], the minimum nuclear norm solution [13], and a DLNN with gradient descent ( N  3 ) by atleast an order of magnitude.
Its performance over other methods not only persists across various dataregimes but it continues to improve with larger sample sizes in addition to scarcer data regimes.
Empirical Results
Lastly, a natural question might be: how well do the effects of our penalty extend to real-world data?
To answer this, we consider MovieLens100k, a popular benchmark for recommendation systems,consisting of 944 users and ratings on 1,683 movies that form an interaction matrix M 2 R 944 ⇥ 1683with the goal of predicting the rest of M after having observed some subset of M .
Unlike our earlier experiments, however, the values are discrete in the range { 1 , 2 , 3 , 4 , 5 } and M is full-rank.
Given these differences and conventional approaches in tackling more empiricalrecommendation system problems, we apply our penalty in two ways.
The ﬁrst way is as before,training a simple depth 1 network with Adam and the penalty (Depth 1 LNN, Table 2).
The secondway is to impose our penalty on a classic user-item embedding model (User-Item Embedding, Table 2) that combines user-speciﬁc and item-speciﬁc biases with a dot product between a latent userembedding and a latent item embedding, similar to a shallow explicit factorization model; we applyour penalty separately on either the item embedding layer or the user embedding layer.
Althoughthis relatively simple approach utilizing explicit ratings has fallen out of favor relative to more recentones (e.g. graph-based approaches, implicit feedback, deep networks, etc.)
utilizing additionalinformation and more complex designs, we nonetheless evaluate the effects of our penalty withthis simple framework.
With these two approaches, we compare their results with those from botholder and more recent state-of-the-art techniques for ratings prediction, reﬂecting a wide variety ofapproaches, architectures, and methods that use additional side information/features beyond M toimprove performance (Table 2).
Surprisingly, we see that the test performance of a simple depth 1 network with Adam and thepenalty ( w. Adam+penalty ) can come close to other more specialized approaches—but, in contrast
[Table]
to the other methodologies, it does so without any specialized architecture or parameterization,additional side information beyond M , creation of new features, or even any factorization at all.
Moreover, we see that the performance of the simple user-embedding model also comes close to theother approaches, even one (AutoSVD++) that leverages additional information/features, despite notneeding to do so itself.
We leave details of our experiment settings to Appendix A.1.
Discussion
The dynamics of optimization trajectories—inﬂuenced by the choice of optimizer, parameterization,explicit penalties, and even architecture—can play an important role in generalization.
With neuralnetworks, depth and gradient descent-like algorithms have been the key ingredients to some of itssuccesses.
In our work focusing on matrix completion, our penalty is able to achieve well-generalizingsolutions and achieve perfect rank recovery even with a depth 1 (i.e. no depth) network.
Does thatmean our penalty, together with Adam’s own inductive bias, is producing an effect similar to implicitregularization under gradient descent with depth, but better?
Is it a coincidence that our penalty onlyobtains such performance with Adam?
We suspect not.
While we concur with the overall spirit of the conjecture in [6]—namely, that areductionist view of implicit regularization which suggests that it can be entirely encapsulated by anexplicit norm-based penalty is likely an incorrect over-simpliﬁcation—we believe that there is meritin studying and relating implicit regularization to its similar explicit counterparts, as our expositionhighlights, and examining their interplay.
Our work suggests that we may be able to partially replicatethe empirical successes of deep learning by selectively combining optimization methods with explicitpenalties via better model design or encoding of inductive biases for speciﬁc tasks.
Naturally, many questions remain open from our limited analyses which, due to space considerations,we leave as possible directions for future work.
For instance, how well do lessons from DLNNstranslate to their non-linear counterparts on other tasks (e.g. classiﬁcation)?
How does this relate tolearning regimes with larger learning rates (i.e. beyond gradient ﬂow)?
A more rigorous analysis ofthe properties (e.g. convergence) of Adam, adaptive gradient methods, and other optimizers in thepresence of explicit regularizers and under different settings may better our understanding.
It remainsunclear whether implicit regularization is a bug or a feature in deep, over-parameterized settings.
Nonetheless, our ﬁndings suggest that it can be harnessed and transformed to desirable effect.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 3, Section 5, andAppendix.(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
Our workdoes not present any foreseeable societal impacts or consequences.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3and Appendix.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 3 andAppendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See supplemen-
tary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running exper- iments multiple times)?
[N/A] Results under our proposed penalty are valid even across varying degrees of regularization strength (  ) and several initial learning rates ↵ and across several initializations.
Other results that were run with random seeds had
negligible standard errors (see implementation details under the Appendix).
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
[Table]
c) Did you include the estimated hourly wage paid to participants and the total amount
[-where ⊙ denotes the Hadamard product and S j is a matrix that contains the corresponding entry-wiseelements of 1 / (cid:113) ∇ W j L ( W ) 
-]
[Table]
spent on participant compensation?
[N/A]
[-( S j )) , and S j is defined as amatrix whose elements are: [ S j ] m,n = [( ∇ W L ( W ) 
-]