Combining Implicit and Explicit Regularization for Efficient Learning in Deep Networks
Anonymous Author(s)
AffiliationAddress
Abstract
Recent work on implicit regularization has focused on gradient trajectories duringthe optimization process in attempting to explain why deep networks favor certainkinds of solutions over others.
For deep linear neural networks, it has been shownthat gradient descent/flow implicitly regularizes toward low-rank solutions in matrixcompletion/factorization tasks, akin to an accelerative pre-conditioning, whoseeffects become more pronounced with increased depth.
In light of this, we proposean explicit penalty that mirrors the rank minimization behavior and generalizationperformance independently of depth, but interestingly only takes effect with Adamand its close variants—it outperforms many approaches in matrix completion andis robust to a wide range of parameter and data regimes.
Our findings suggest thatexplicit regularization can play a key role together with the choice of optimizationalgorithm in designing different, desirable forms of regularization and that a morenuanced understanding of this interplay may be necessary
Introduction
Much work has poured into understanding and explaining how and why highly over-parameterizedneural networks generalize so effectively with more parameters than training examples despite longheld notions to the contrary.
This generalization puzzle has deepened as over-parameterized deeplearning models often generalize well even without any explicit penalization, simply by optimizingits training error on an under-determined problem.
To explain this and related phenomena [8, 9], previous works have focused on how gradient-basedoptimization induces an implicit bias on optimization trajectories, tending towards solutions withcertain properties [6, 19] like those that generalize well or are simpler in some respect.
In contrast,although explicit regularization has seen wide-spread usage for improving performance in a varietyof model frameworks such as ℓ p norm weight constraints or dropout [45], its effectiveness and rolein generalization in deep networks is not certain given its inability to prevent over-fitting to randomlabels [52] or its absence in deep models which generalize well on their own.
Some more recent work has focused on a simple test-bed setting that admits a crisp formulationfor understanding the mechanisms through which implicit regularization operates, namely, matrixcompletion : given some subset of an unknown, low-rank matrix W ⋆ , the task is to recover the unseenentries.
A key observation has focused on how gradient descent on a shallow linear neural network,with sufficiently small learning rate and near-zero initialization, pushes towards low-rank solutionswithout explicit regularization or constraints [19].
This phenomenon led [19] to conjecture thatgradient descent is inducing an implicit regularization that minimizes the nuclear norm.
This conjecture has been put into doubt by work showing that implicit regularization influences thetrajectory of gradient descent in such a way so as to (a) promote solutions with few large singular values and many small ones [6], with the gap becoming more pronounced as depth increases (i.e.deep matrix factorization by [6]), and (b) resemble a combination of momentum and adaptiveregularization [5], itself akin to explicit accelerated gradient methods [28] popular in deep learning.
Contributions Despite these findings, questions still remain:
• Are there explicit norm-based penalties that can mirror the effect of implicit regularization so as to better harness this phenomenon for better or more efficient learning?
Previous work [6]has conjectured that implicit regularization cannot be characterized by explicit norm-based penalties, but whether norm-based penalties can produce similar effects is unclear.
• What role does acceleration play, if at all?
Can this accelerative effect be replicated or, at least, imitated?
Previous work [5] conjectured that the implicit preconditioning oraccelerative effect from depth is somewhat orthogonal to explicit acceleration methods.
To try and shed more light on these questions, we propose and consider the following penalty thattakes the ratio between the nuclear norm of a matrix and its Frobenius norm:
∥ W ∥ ⋆ / ∥ W ∥ F .
Thispenalty can be viewed as an adaptation of the Hoyer measure [23] but for matrices in the spectraldomain, measuring the relative spectral sparsity of a non-zero matrix W .
We show the following:
A depth 1 linear neural network (i.e. a degenerate network without any depth) equipped with
our penalty can produce the same rank reduction with comparable, or better, generalization performance as deep matrix factorization.
In short, depth is no longer necessary for learning. Given that shallow networks are faster and more lightweight
, these attributes also make them more efficient than their deeper counterparts. The above result only occurs under Adam and, to some extent, its close variants.
This
suggests that different optimizers may have different inductive biases that interact differently with explicit regularizers in preferring certain solutions over others.
With the penalty, we achieve better generalization and rank-reduction performance beyond many techniques (Fig. 4), even in low data regimes (i e. fewer observed entries during
training) where the nuclear norm minimizer has no recovery guarantees.4.
In settings with non-trivial depth, this penalty induces the same overall effect on rank andgeneralization largely independent of depth, showing a degree of depth invariance.
Previous conjectures and works [5, 6, 52] have largely dismissed the importance of explicit regularization in understanding generalization in deep learning, leaving its overall role unclear.
Similarly,despite its relative popularity, Adam and other adaptive optimizers have received their share of doubt[46, 47] regarding their effectiveness in producing desirable solutions.
Our results suggest a subtle butperhaps important interplay between the choice of the optimizer, its inductive bias, and the explicitpenalty in producing different trajectories, and, hence, different kinds of solutions.
If so, then in theseinteractions, explicit regularization may yet have a role to play.
Paper Overview In Section 2, we review previous work.
Section 3 details our experiments andfindings.
In Section 4, we extend our experiments to an empirical benchmark and we conclude in
Section 5 with a discussion on future work.
Related Work
Implicit regularization and its role in explaining optimization and generalization have been studiedextensively, with more recent work focusing on optimization trajectories in deep, over-parameterizedsettings [5–7].
One avenue has focused on linear neural networks (LNNs) [18, 21, 26, 43] given theirrelative tractability and the similarity in learning dynamics between linear and non-linear networks.
For instance, in settings with multiple optima, [20] has shown how LNNs trained on separable datacan converge with an implicit bias towards the maximum margin solution.
Others [6] have demonstrated how gradient flow converges towards low-rank solutions where increased depth acts as an accelerative pre-conditioning [5].
[2] has shown more directly, in the case ofnatural and vanilla gradient descent, how different choices of pre-conditioning impact bias-variancetrade-offs and population risk in over-parameterized linear regression, but leaves the question openfor pre-conditioning in adaptive gradient optimizers.
Acceleration in optimization dates back to Nesterov [37] and even Newton’s method, using secondorder information or a limited memory of past quantities (e.g. gradients) to inform optimization as aform of pre-conditioning.
Adam’s [28] effectiveness—and its closely inspired variants [16, 32, 33]—in optimizing deep networks faster makes the importance of adaptive pre-conditioning more explicit.
Though some [46, 47] have doubted the effectiveness of adaptive gradient methods and illustratedpotential issues in certain cases, such as harming generalization, others [31, 50, 53] have shownadvantages from preconditioning and adaptive optimizers; despite speeding up optimization, however,the effect of pre-conditioning on generalization has been less clear.
Some [1, 15, 49] have argued thatthe “sharpness” of minima can vary depending on the choice of optimizer.
Matrix completion and factorization have themselves long been an important focus for areas likesignal recovery [11] and recommendation systems [10] while theoretical bounds for exact recoveryand convergence [12, 13, 34, 40] have led to practical algorithms [24, 35].
We refer to [14] for a morecomprehensive survey.
These and related tasks have also served as test-beds to better understandimplicit regularization.
[6, 19] use matrix factorization and sensing to study gradient flow’s implicitbias towards low-rank solutions and conjecture that solutions produced by algorithmic regularizationmay not correspond to any minimization of norm-based penalties at all.
More recently, [48] studiedthe implicit bias of mirror descent on matrix sensing, showing its correspondence to a solution thateither interpolates between the nuclear and Frobenius norms depending on the underlying matrix.
Some have started to explicitly compare implicit and explicit regularization; in certain least squaresproblems, [54] showed how stochastic gradient descent’s (SGD) implicit biases can generalize noworse or even outperform ridge regression, requiring quadratically more samples to compete with SGD.
To the best of our knowledge, our paper is the first to propose the ratio of the nuclear norm tothe Frobenius norm as a penalty in matrix completion to study the interplay between explicit andimplicit regularization.
Experiments and Findings
3.1 Setup
Formally, we have a ground-truth matrix W ⋆ ∈ R m × n whose observed entries are indexed by the setΩ .
We define the projection P Ω ( W ⋆ ) to be a m × n matrix such that the entries with indices in Ωremain while the rest are masked with zeros.
We are interested in the following optimization:
[Equation]
where ∥ · ∥ F is the Frobenius norm, R ( · ) is an explicit penalty, and λ ≥ 0 is the tuning parameter. While we consider various penalties, our main focus is demonstrating the effects of our proposedpenalty R ( W ) =
|| W || ∗ / || W || F . Following earlier works such as [3], we define a deep linear neuralnetwork (DLNN) through the following over-parameterization, or deep factorization, of W :
[Equation]
under the loss function in (1) where W i ∈ R d i × d i − 1 , i ∈ { 1 , . . .
, N } is the i -th layer/weight matrix.
Here, N also denotes the depth of the network/factorization where N = 2 corresponds to matrixfactorization , N ≥ 3 corresponds to deep matrix factorization , and N = 1 is the degenerate case(i.e.
no depth).
We refer to the matrix W , the product of the N weight matrices in Eq.
(2), as theend-product matrix as per [6], which is the network’s produced solution in estimating W ⋆ .
In thefollowing analyses, we focus on rank 5 matrices as the ground truth W ∗ and parameterize our DLNN W with d 0 = . . .
=
d N = n = 100 (i.e. layers/weight matrices W i ∈ R 100 × 100 , ∀ i ) for illustrativepurposes, but our results extend to other ranks (e.g. see Appendix A.2). We follow previous workssuch as [6] and employ the effective rank of a matrix [42] to track and quantify the rank of W inour experiments, which is defined as: e-rank( W ) =
exp { H ( p 1 , . . . , p n ) } where H is the Shannonentropy, p i = σ i / ∥ σ ∥ 1 , σ = { σ i } are the unsigned singular values of W , and ∥·∥ 1 is the ℓ 1 norm. The numerical instability of the numeric rank measure is a known issue [42], resulting in unreliable,high-variance rank estimates.
We leave a detailed discussion of experiment settings to Appendix A.1.
Depth, without penalty
We first establish a baseline by reviewing the dynamics under both un-regularized gradient descentand un-regularized Adam to better understand their dynamics in the presence of our penalty later on.
Gradient Descent Previous work [6] has shown that depth enhances the implicit regularizationtowards low rank, characterized by the following trajectories on the end-product matrix W and itssingular values { σ i } (for details see [5, 6]):
[Equation]
where ˙ σ i is the time derivative of σ i ( t ) , the i -th singular value of W ( t ) , { u i , v i } are the left andright singular vectors of W ( t ) corresponding to σ ( t ) i , N is the network’s depth, ∇ W L ( W ( t )) isthe loss gradient with respect to the end-product matrix W at time t , vec ( · ) denotes (column-firstorder) vectorization, ˙ W = dW/dt , P W = (cid:80) Nj =1 ( W ⊤ W ) N − j N ⊗ ( WW ⊤ ) j − 1 N , and ⊗ denotes the
Kronecker product.
Moving forward, we suppress the explicit dependency on t for simplicity.
We note that full dynamics in Eq.
(4) require non-degeneracy (i.e. depth N > 1 ); otherwise, theyreduce down to the product of the loss gradient and singular vectors.
We see these dynamics at play(Fig. 1, top row) where higher depths produce solutions with lower test error and rank (i.e. bettersingular value separation).
For non-degenerate cases ( N ≥ 2 ), as depth increases, larger singular values and their changesbecome more pronounced and enhanced than their smaller counterparts, driving singular valueseparation and rank decrease—this is seen empirically (Fig. 1 top row), resulting in a decrease inthe solution’s effective rank.
The singular values also evolve at uneven paces across depth levels;increasing depth increases the gap in the time it takes between the 1 st and 5 th largest singular valuesto develop while also taking longer to stabilize—these effects are even more pronounced whencomparing the five largest singular values to the remaining ones.
This differential pace by whichthe singular values evolve is largely shaped, and constrained, by depth.
Only with sufficient depth( N ≥ 3 ) do solutions produced by un-penalized gradient descent minimize rank so as to recover theoriginal rank of the matrix in question and generalize better than the nuclear norm solution.
For Eq.
(3), the trajectory dynamics of the end-product matrix W itself (i.e. the produced solutionto Eq.
(1)), P W can seen as a pre-conditoning onto the gradient that, with sufficient depth ( N ≥ 2) ,accelerates movements already taken in the optimization [5].
As depth (i.e. over-parameterization)increases, this acceleration intensifies.
Adam Analyzing Adam can be difficult given its exponentially moving average of gradients; tosimplify our analysis, we borrow some assumptions from [4] to characterize Adam’s dynamics via gradient flow, though imperfect, by specifically assuming that the discounted gradients can bewell-approximated by their expectation. (see Appendix A.4 for more details).
Theorem 1.
Under the assumptions above and of [6], the trajectory of the singular values σ i of theend-product matrix W can be approximately characterized as: [Equation] Similarly, the trajectory of the end-product matrix W itself can be approximately characterized as:
[Equation]
) where P W,G = (cid:80) Nj =1 (( W ⊤ W ) N − j N ⊗ ( WW ⊤ )
j − 1 N ) G j is p.s.d.
and G j is a diagonal positivematrix.
Specifically, G j = diag(vec( S j )) , [ S j ] m,n = [( ∇ W j L ( W ) {+
+} + s 2 j ) − 1 / 2 ] m,n , ∇ W j L ( W ) =
∂ L ( W ) /∂W
j , and s 2 j = var( ∇ W j L ( W )) .
Proof.
See Appendix A.4.
Via this approximation, the pre-conditioning induced by Adam can be characterized as a modificationof gradient descent’s P W , normalizing each layer-component by the square-root of its sum of itssquared layer-wise loss gradient ( ∂ L ( W ) /∂W
j ) 2 and the gradient variance, s 2 j , before summingacross all depth levels.
Unlike before, the variance of the loss gradient comes into play—whereasbefore the pre-conditioning serves as a purely accelerative effect that intensifies with depth, normalization by a function of the variance can either dampen or further accelerate the trajectory.
Empirically,in the deep regime ( N > 2 ), we see that Adam minimizes rank to exact/near-exact rank recoverymore efficiently and uniformly than gradient descent via faster ( 10 4 vs. 10 5 ∼ 10 6 ), more uniformconvergence (Fig. 1, bottom row).
Singular value dynamics exhibit significantly more uniformevolution regardless of depth in contrast to gradient descent (Fig. 1, right).
Given these differences, Adam appears to arrive at different kinds of solutions than gradient descent. [Figure] 3.3 Depth, with penalty
[Figure]
Gradient Descent We first characterize dynamics under gradient flow with our proposed penalty.
Theorem 2.
Under the assumptions of [5], with the proposed penalty, the singular values of theend-product matrix W under gradient descent with penalty (i.e. ˙ σ reg r ) evolve in the following fashion:
[Figure] [Equation]
where λ > 0 is the regularization parameter and ˙ σ GF r denotes the singular value trajectory undergradient flow (Eq. Similarly, the end-product matrix W evolves in the following fashion: [Equation]
where U , V contain the left and right singular vectors of W , P W is the pre-conditioning in Eq. (3) ,and ˜Σ = || W || ∗ || W || F Σ where Σ contains the singular values of W along its diagonal.
Proof.
See Appendix A.4 for details.
The penalty can be seen as an additive component to the original trajectories in Eq.
This newcomponent also intensifies with increased depth, making movements in larger singular values morepronounced than those of smaller ones, which together can drive faster, more pronounced singularvalue separation than before ( 2 N − 1 N vs. 3 N − 2 2 ), depending on λ .
Empirically, comparing the un-regularized case (Fig. 1 top row) to the now regularized case (Fig.
2top row), we indeed see that the regularized case features faster convergence in rank and fasterdevelopment of the top singular values as a result of this accentuation.
Here, increased depthcontinues to push down rank but, unlike before (Eq.
(4), Eq.
, the penalty now allows singularvalue trajectories to depend on their own magnitudes even without depth ( N = 1 ), providing anadditional degree of freedom.
The penalty also allows each singular value to depend on its relativeweight within the distribution of singular values through (1 −
|| W
|| ∗ / || W || F ) rather than just itsown absolute magnitude.
We see that the depth-dependent accelerative pre-conditioning P W nowacts on a new term, similar to Eq.
(7), which serves as an additional, spectral-based component thatcan accelerate the trajectory of the produced solution W , especially at higher depths.
Looking atthe diagonals, the new term can be seen as a spectrally re-scaled version of W that influences itstrajectory in a way that accounts for each singular value’s weight relative to its entire spectrum:
[Equation]
|
[Equation]
i
[Equation]
The end-product matrix W produced now (Fig. 2 top row, left plot) achievesbetter test error and converges faster at higher depths ( N ≥ 4 ) compared to analogous depths in theun-regularized case (Fig.
1 top row, left plot).
Adam With Adam, the penalty’s effect differs considerably in terms of the solutions produced.
Theorem 3.
Under the same assumptions, with the proposed penalty, the end-product matrix and itssingular values evolve in the following fashion:
[Equation]
Proof.
Follows from Eq.
(5) and Eq. (6) with our penalty.
See Appendix A.4 for details.
Empirically, we note a new development: there is a larger degree of depth invariance in that rank ispushed down almost independently of depth (Fig. 2, bottom row), even at depth 2.
All depths recoverthe rank of the underlying matrix (i.e. 5) unlike gradient descent.
Starting with a shallow network( N = 2 ), increasing the depth does not induce any material changes in the produced solutions as thepenalized DLNN under Adam produces low-rank solutions that achieve exact rank recovery and lowgeneralization error on par with, if not better than, before (Fig. 2).
It retains some of the accelerativeeffect of Adam with respect to the speed of convergence and behavior of effective rank and how thesingular values develop.
Compared to before, we see a beneficial accelerative effect that endows morefavorable generalization properties and smoother development of singular values whose convergencespeed is at least an order of magnitude faster than under gradient descent—which do not changemuch with depth.
As singular values evolve, we see that their paths are relatively robust to depth, allproducing the near-identical behavior with much less dependence on depth (Fig. 2, right).
3.4 Degenerate case, with penalty
Given the non-trivial interaction effects seen thus far, we now consider the degenerate depth 1 network.
It is known [52] that gradient descent in this setting converges to the minimum Frobenius (i.e. ℓ 2 )norm solution, which does not necessarily induce spectral sparsity or low ranked-ness.
Applying our penalty to this network under gradient descent does little to affect the solution meaningfully (Fig. 3).
In this case, without depth, learning is heavily constrained.
Yet, surprisingly, despite being ineffective with gradient descent, our penalty is effective under
Adam (Fig. 3, bottom) even without depth or over-parameterization, achieving generalization onpar with, if not better than, deep matrix factorization ( N > 2 ) under gradient descent.
We note thatreplacing our penalty with other proxies of rank/spectral sparsity such as the nuclear norm doesnot work (Table 1).
Under un-regularized gradient flow, at depth 1, singular value and end-productdynamics collapse: singular value trajectories reduce to u ⊤ i ∇ W L ( W ( t )) v i , as noted earlier, and theaccelerative pre-conditioning reduces to just the identity P W =
I mn .
We see this empirically (e.g.
Fig.
3 and Table 1) where solutions produced un-regularized gradient descent at depth 1 generalizepoorly and fail to reduce rank.
By removing depth, we see that a key component in helping adepth 1 network learn is its variance-weighted gradient term ∇ W L ( W ) · G , as defined in Eq.
(6),along with the term P W,G (cid:16)
[Equation]
(cid:17) , which reduces to G (cid:16) λ vec( UV
[Equation]
− U ˜Σ
[Equation]
) |
[Equation]
W |
[Equation]
(cid:17) withoutdepth.
Interestingly, the variance of the loss gradient and the ratio η 2 = var( ∇ W L ( W )) / ∇ W L ( W ) 2formed from ∇ W L ( W ) · G , akin to a signal-to-noise ratio, have come up in other works as importantquantities that is strongly predictive of generalization capabilities [25] or is essential to findingoptimal variance adaption factors of stochastic gradients [4].
Other Combinations Our results do not preclude the possibility that other optimizers and penaltiescan produce similar or better effects.
For completeness, inspired by the properties of our proposedpenalty (e.g. ratio-like, spectral, non-convex), we experiment with various optimizers and penalties in
Table 1 to compare optimizer-penalty interactions across shallow ( N = 1 ) and deep ( N = 3 ) settings.
We note that our ratio penalty under Adam and its close variant Adamax, both initially proposed in[28], are the only combinations that largely show depth invariance in test error and rank recoverywhereas others require depth to either push down rank or generalize better.
Although the nuclearnorm shows a degree of depth invariance in rank reduction, it is unable to find well-generalizingsolutions and it fails in rank reduction under gradient descent where depth is still necessary.
Given thespace considerations of this paper, we leave a fuller treatment and characterization of each optimizer’sinductive biases and their interactions with different regularizers for future work.
Comparative Performance Finally, we note our penalty’s comparative performance (Fig. 4) againstother methodologies for synthetic matrix completion across various sample sizes (i.e. the numberof entries out of 100,000 made available for training).
Our depth 1 network with Adam and thepenalty outperforms other methods including the un-regularized DLNN in test error, degree of rankcompression, and convergence speed across several data regimes. At lower sample sizes, even at [Table]
1.8e +  0. 8e − 2e − 1. GD 0. 1e − depth 1, the penalized network generalizes better than methods such as SoftImpute [35], OptSpace[27], the minimum nuclear norm solution [13], and a DLNN with gradient descent ( N ≥ 3 ) by atleast an order of magnitude.
Its performance over other methods not only persists across various dataregimes including under scarce data but also improves with larger sample sizes.
Results on real world data
Lastly, a natural question might be: how well do the effects of our penalty extend to real-world data?
To answer this, we consider MovieLens100k, a popular benchmark for recommendation systems,consisting of 944 users and ratings on 1,683 movies that form an interaction matrix M ∈ R 944 × 1683with the goal of predicting the rest of M after having observed some subset of M .
Unlike our earlier experiments, the values here are discrete in the range { 1 , 2 , 3 , 4 , 5 } and M isfull-rank.
Given these differences and conventional approaches taken in empirical recommendationsystems, we apply our penalty in two ways.
The first way is as before, training a depth 1 network with Adam and the penalty (Depth 1 LNN, Table 2).
The second way is to impose our penalty on a classicuser-item embedding model (User-Item Embedding, Table 2 [29]) that combines user-specific anditem-specific biases with a dot product between a latent user embedding and a latent item embedding,similar to a shallow explicit factorization model; we apply our penalty separately on either the itemembedding layer or the user embedding layer.
Though this approach utilizing explicit ratings hasfallen out of favor in lieu of more recent ones (e.g. graph-based, implicit feedback, deep networks) [Table]
utilizing additional information and complex designs, we nonetheless evaluate the effects of ourpenalty with this simple framework.
With these two approaches, we compare their results with thosefrom both older and more recent state-of-the-art techniques, reflecting a wide variety of approachesand architectures that use additional side information beyond M to improve performance (Table 2).
Surprisingly, we see that the test performance of a simple depth 1 network with Adam and the penalty( w. Adam+penalty ) can come close to other more specialized approaches—but, in contrast to theothers, it does so without any specialized architecture or parameterization, additional side informationbeyond M , creation of new features, or even any factorization at all.
Moreover, we see that theperformance of the simple user-embedding model also comes close to the other approaches, evenone (e.g. AutoSVD++) that leverages additional information/features, despite not doing so itself.
Weleave details of our experimental settings to Appendix A.1.
Discussion
The dynamics of optimization trajectories—influenced by the choice of optimizer, parameterization,explicit penalties, and even architecture—can play an important role in generalization.
With neuralnetworks, depth and gradient descent-like algorithms have been key ingredients to some of itssuccesses.
In our work focusing on matrix completion, our penalty achieves well-generalizingsolutions and perfect rank recovery even with a depth 1 (i.e. no depth) network.
Does that meanour penalty, together with Adam’s own inductive bias, is producing an effect similar to implicitregularization under gradient descent with depth, but better?
Is it a coincidence that our penalty onlyobtains such performance with Adam?
We suspect not.
While we concur with the overall spirit of the conjecture in [6]—namely, a reductionist view of implicit regularization which suggests that it can be entirely encapsulated by anexplicit norm-based penalty is likely an incorrect over-simplification—we believe that there is meritin studying and relating implicit regularization to its similar explicit counterparts, as our expositionhighlights, to examine their interplay.
Our work suggests that we may be able to partially replicatethe empirical successes of deep learning by selectively combining optimization methods with explicitpenalties via better model design or encoding of inductive biases for specific tasks.
Many questions remain open from our limited analyses which, due to space considerations, we leaveas directions for future work.
For instance, how well do lessons from DLNNs translate to theirnon-linear counterparts on other tasks (e.g. classification)?
How does this relate to learning regimeswith larger learning rates (i.e. beyond gradient flow)?
A more rigorous analysis of the properties (e.g.convergence) of Adam, adaptive gradient methods, and other optimizers in the presence of explicitregularizers may better our understanding.
It remains unclear whether implicit regularization is a bugor a feature in deep, over-parameterized settings.
Nonetheless, our findings suggest that it can beharnessed and transformed to desirable effect.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](
b) Did you describe the limitations of your work
[Yes] See Section 3, Section 5, and Appendix.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work does not present any foreseeable societal impacts or consequences.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section and Appendix.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 3 andAppendix. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See supplemen- tary material.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they were chosen)?
[Yes] See Appendix.
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)? [N/A] Results under our proposed penalty are valid evenacross varying degrees of regularization strength ( λ ) and several initial learning rates αand across several initializations. Other results that were run with random seeds had
negligible standard errors (see implementation details under the Appendix). (d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Implementation Details, Further Experiments, & DerivationsA.1 Implementation Details
In this section, we provide implementation details of our experiments. We use PyTorch for implementing the linear neural network, CVXPY for finding the minimum nuclear norm solution, the Rpackage ROptSpace for running the OptSpace algorithm, and the Python package fancyImputefor running the SoftImpute algorithm. Additionally, for experiments on MovieLens100k, we modifyand build upon the Spotlight repo 1 which uses PyTorch to provide common building blocks fordifferent kinds of recommendation models. Experiments were run on a NVIDIA Tesla V100 GPU.
Synthetic Data For synthetic data experiments, our implementation details mirror the experimentaldesign of [6], which we briefly detail. When referring to a random rank r matrix with size m × n ,we mean a product UV ⊤ , where the entries of U ∈ R m × r and V ∈ R n × r are drawn independentlyfrom a standard normal distribution. The observed entries are uniformly sampled at random withoutrepetition. The goal is to recover the underlying matrix by observing a portion of it and trying torecover the remaining entries as well as producing a solution matrix whose overall rank is low or isclose to the rank of the underlying ground truth matrix.
During training, deep linear neural networkswere trained by gradient descent, or whichever other optimizer under consideration, via full-batchtraining applied to the Frobenius loss (i.e. the ℓ 2 loss) over the observed entries. Weights are initialized via independent samples from a Gaussian distribution with zero mean andstandard deviation 10 − 3 . Learning rates are fixed through training; specifically, aligning withassumptions of gradient flow, we set our initial learning rates (for gradient descent, Adam, and ourother optimizers) α = 10 − 3 for the results shown in this paper, but we also conducted the sameexperiments with α ∈ { 5 · 10 − 4 , 10 − 4 } and saw no qualitative differences. During training, stoppingcriteria consist of either reaching a total of 5 · 10 6 iterations or training loss reaching values below10 − 7 , whichever applies sooner.
Moreover, results relating to the ratio penalty are robust acrossvarying strengths of regularization λ ∈ [10 − 6 , 10 − 1 ] and training is full-batch, leaving no room fornoise like that in mini-batch training. For Adam, aside from the initial learning rate, the defaulthyper-parameters under PyTorch’s implementation were used, which include ( β 1 , β 2 ) = (0 . 9 , 0 . 999) ,no amsgrad , etc. unless otherwise stated.
Similarly for experiments involving other optimizers,aside from the initial learning rate, we use the default hyper-parameter settings unless otherwisespecified. Test error is measured via mean-squared error, unless stated otherwise. To quantify therank of solutions produced, we use the effective rank measure [42] due to its favorable propertiesover numerical rank in simulation settings and its usage in [6]. The test error with respect to a groundtruth matrix W ⋆ is given by the mean-squared error i.e. for a solution W it is 1 N ∥ W − W ⋆ ∥ 2 F butresults continue to hold across other measures of test error.
For rank 5 experiments where the samplesize was fixed, sample size is set at 2000 (again, following [6]) but results are similar for sample sizesat 2500, 3000, etc. The sample size for rank 10 experiments were set at 3000 and 3500.
For our experiments on MovieLens100k, we use the base explicit factorizationbase available in Spotlight , as mentioned earlier, in conjunction with Adam and our ratio penalty. The explicit factorization model (i.e. baseline) consists of several components akin to a shallow,depth 2 LNN model (with the exception of bias terms and separate inputs into separate embeddinglayers): a user-embedding layer, an item-embedding layer, a user-specific bias, and an item-specificbias. With our penalty applied to either the user-embedding layer or the item-embedding layer, weevaluate values of λ ∈ [10 − 6 , 10 − 1 ] , embedding dimensions in { 2 , 4 , 8 , 16 , 32 , 64 , 128 } , and batchsizes of { 128 , 256 , 512 , 1024 } for initial learning rates α ∈ [5 · 10 − 3 , 10 − 5 ] .
For hyper-parameterscombination, the model is trained for 100 epochs. The best model is check-pointed and training isterminated if training error begins increasing for three epochs. The best performing model with thereported test error was with learning rate α = 10 − 3 , batch size 256, effective dimension 64, andλ = 0 . For our depth 1 LNN, we take an approach near-identical to our synthetic experiments (e.g. sameinitialization and training/stopping criteria, full-batch training, etc.).
For our hyper-parameter space,we explore values for regularization strengths λ ∈ [2 · 10 1 , 10 − 6 ] and initial learning rate α ∈[5 · 10 − 3 , 10 − 5 ] . The best performing model with the reported test error was with learning rateα = 5 · 10 − 4 and λ = 1 . 5 though many similar configurations came close to the same performance.
The following details the same set of experiments as those in Section 3 but for rank 10 matrices as anexample to illustrate that our results generalize beyond rank 5. Overall, comparative performance results very much resemble those in Fig.
4 for rank 5 completion. As seen in Fig. 5, our penalty’s comparative performance relative to other techniques remainsunchanged from rank 5. Like before, Adam with penalty, at depth 1, outperforms all other approachesacross all data regimes by at least two orders of magnitude, with reconstruction error decreasing evenfurther as sample size is increased.
A similar result is seen in terms of its performance on rank: itreduces rank to the point of exact rank recovery independently of sample size. Extending our experiments to rank 10 matrix completion, we see extreme similarity with resultsunder rank 5 (Fig. 6). For singular values, we focus on the 1st and 10th largest values in accordancewith the changed rank setting (rank 5 to better illustrate the differential behavior of themost prominent values For regularized gradient descent and Adam (Fig. 7), results again resemble the rank 5 case.
Undergradient descent, the proposed penalty induces only slight effects on quickening and tightening theconvergence behavior of singular values and test error. In contrast, we see nearly identical effects ofour penalty under Adam when compared to the rank 5 case. We also see that, like in rank 5, depth 2becomes effective in generalizing and reducing rank when compared to depth 2 under Adam alone; indepth 2, our proposed penalty is able to push its singular value trajectories of the 1 st and 10 th largestsingular values towards the other respective singular values in higher depth levels as if getting depth2 to behave similarly as higher depth levels in contrast to un-penalized Adam.
In this section, we expand upon our experiments evaluating different combinations of optimizers andexplicit penalties beyond those in Table 1. In Table 3 and Table 4, we include other optimizers andevaluate them against several explicit penalties. For all optimizer choices, hyper-parameters, aside
[Table] [Table]
[Table] [Table]
from the initial learning rate which is set to 10 − 3 , are set to their default values in PyTorch unlessotherwise specified (e.g. default momentum values for GD). For all experiments evaluating optimizerpenalty combinations, we try λ ∈ [10 − 6 , 10 − 1 ] which all delivered similar results, especially in termsof observed depth invariance (or lack thereof) and generalization/rank reduction performance. Theresults shown in Table 1 in Appendix A.3 as well as in Table 3 and Table 4 here have λ = 0 . 05 . Results shown are trained on a sample size of 2000 entries (out of a 100 × 100 matrix); results showno meaningful difference on other sample sizes (2500, 3000).
We note that Adam-like variants such as AdamW, NAdam, and RAdam also, unsurprisingly, exhibitvarying degrees of depth invariance with respect to both generalization error and rank reduction withour ratio penalty though their solutions fail to generalize as well as Adam and Adamax under thepenalty—this makes sense as Adam and Adamax are much more closely related, and both proposedin the same paper with only slight differences in their update rules, than subsequent variants that havesignificantly different iterative updates (see [16, 28, 32, 33] for further details).
A.4 Supplemental Derivations
Here, we outline and detail the derivations behind some of our work describing the trajectory of theend-product matrix W and its singular values σ i under gradient flow, largely building off of [5, 6].
Wehighlight the main derivations as they relate to the theorems in this paper and defer to the appendicesof [5, 6] for additional details and a fuller treatment of gradient flow derivations.
We also provideadditional commentary and discussion here due to space constraints in the main body of the paper.
A.4.1 Theorem 2 (Gradient flow, with penalty)
Setup & preliminaries.
We omit the details of deriving the ˙ σ and ˙ W under un-regularized gradientflow; instead, we defer to the appendix of [5] for a detailed treatment and build upon their results forour derivations.
Evolution of W and σ under gradient flow without penalty.
To begin, we recall the dynamics ofthe end-product matrix W and its singular values σ under un-regularized gradient flow (for details ona full derivation, see the appendices of [5, 6] for a detailed treatment):
[Equation]
Since the explicit regularization simply adds a regularization term to the original loss function,we can re-calculate the gradient of the new, regularized loss and substitute the gradients back intothe dynamics above in order to re-characterize the above dynamics under the penalty. In moreformal terms, if we let the new modified loss function be represented by ˜ L ( W ) : = L ( W ) + R ( W ) [Table]
NAdam [16]
[Table]
[Table]
[Table]
[Table]
RAdam [32]
[Table]
[Table]
[Table]
[Table]
where R ( W ) =
||
W || ∗ /
||
W || F denotes the explicit penalty, then we can calculate ∇ W ˜ L ( W )
=∇ W L ( W )
+ λ ∇ W R ( W ) to plug back into the un-regularized dynamics in order to characterize thetrajectories under the effect of the penalty.
Evolution of W and σ under gradient flow with penalty.
To characterize the dynamics of gradientflow in presence of the penalty, we first derive the gradient of the penalty (i.e. ∇ W R ( W ) )
ignoringthe regularization strength (hyper)parameter λ .
Via the chain rule and the sub-gradient of the nuclearnorm, we can express the gradient of the penalty as:
[Equation]
where U , V , Σ are the singular matrices of the singular value decomposition of W (i.e. W = U Σ V ⊤ ).
To first characterize the dynamics of the end-product matrix W , we can use the gradient of the penaltyand substitute it back into Eq.
(11) along with the loss gradient.
Taking Eq.
(12), plugging it backinto Eq.
(11) along with the original loss gradient ∇ W L ( W ) , and re-arranging terms, we have
[Equation]
where ˜Σ = || W
||
∗
||
W || F Σ , producing the expression in Eq.
Turning now to the singular values, for analytical ease in characterizing the dynamics of the singularvalues σ i under gradient flow with the penalty (i.e., ˙ σ reg r ), we can equivalently express Eq.
(12) in terms of its diagonal elements; for instance, if we focus on the r -th diagonal element, this produces
[Equation]
where { u r , v r } are the i -th left and right singular vectors of W and σ r is the r -th diagonal element of Σ .
Substituting this term along with the original loss gradient back into Eq.
(10) andre-writing/grouping terms, we obtain our desired result in the form of Eq.
(7) for the r -th singularvalue, i.e.,
[Equation]
where ˙ σ GF r is defined in Eq.
(10) (i.e. the trajectory of singular values under un-regularized gradientflow).
A.4.2 Theorem 1 (Adam, no penalty)
Setup & preliminaries.
We omit the details of deriving P W from the beginning and defer to theappendix of [5] for a more comprehensive review.
Operations like divisions, squaring, and squareroots on vectors and matrices are to assumed to be performed element-wise unless otherwise stated.
In this case, since parameters are matrices, loss gradients will generally be matrices; in this case,when referring to the variance of said objects, the variance is taken to mean the variance of thevectorized matrix (i.e. condensed into a vector) drawn from some distribution.
Per-layer weight updates under Adam in discrete time take the form, assuming no weight-decay:
[Equation]
for each layer j ∈ { 1 , . . .
, N } in a depth
N network where α is the initial learning rate, ˆ m t is thebias-corrected first moment, ˆ v t is the bias-corrected second moment, and ε is a division-by-zeroadjustment which we assume to be zero to simplify our later analysis.
In turn, m t is the movingaverage of stochastic gradients g t and v t is the moving average of their element-wise square g 2 t ,where g t = ∇ W j L ( W ( t ) ) , the gradient of the loss with respect to the j -th layer weight, g 2 t is theelement-wise squaring of g t , and β 1 , β 2 ∈ (0 , 1) are discount factors for m t and v t respectively.
Moving forward, for notational simplicity, we suppress the dependence of the W and W j on t unlessotherwise stated and set β 1 = β 2 = β
= 1 − ϵ for sufficiently small epsilon (e.g. ϵ = 0 .
Given the loss in Eq.
(1), the gradient of the loss with respect to the j -th layer weight matrix is(suppressing notation on time t ):
[Equation]
(13) where ∇ W L ( W ) is the gradient of the loss with respect to the end-product matrix W .
Whilegradient flow uses this loss directly in its iterative updates for each j -th weight matrix, Adam makesadjustments to this loss gradient in its calculation of ˆ m and ˆ v .
Assumptions.
As noted earlier in Section 3, to approximately characterize the trajectory under
Adam in closed form, albeit imperfectly, we make a few additional assumptions.
More formally, inthe same spirit as [4]
, we assume that the underlying distribution of stochastic gradients is stationaryand approximately constant over the time horizon of the moving average of gradients so that ˆ m t andˆ v t approximate estimate their respective moments of g t .
Namely:
[Equation]
where s 2 t,j is the (element-wise) variance of ∇ W j L ( W ( t ) ) with the assumption that each W j is drawni.i.d.
from a common stationary distribution.
Naturally, this assumption can only hold approximately, at best, and is clearly inaccurate for gradients far in the past—to that end, we assume some sufficientdegree of “burn-in” or progression so that past some time t , gradients near initialization or extremelydistant past gradients do not contribute meaningfully to the moving average.
As noted in [4], thisassumed approximation is more realistic and more likely to hold in certain cases (e.g. high noise orsmall step size, the latter of which aligns with the spirit of gradient flow).
Evolution of W under Adam without penalty.
Under the assumptions above and those in [6], toapproximately characterize the trajectory of the end-product matrix, we can rewrite Adam’s updateof each weight layer W j using our assumptions above in continuous time (i.e. α → 0 ), in a fashionsimilar to gradient flow, as (suppressing notational dependence on time t ):
[Equation]
where, again, division, etc.
here denote element-wise operations.
As an aside, we note that theexpression in Eq.
(14) can also be written as ˙ W j = − (
[Equation]
− 1 / 2 where η 2 j : =
[Equation]
,resembling a variance adaptation factor for each layer j that shortens the update in directions of high(relative) variance as an adaptive means to account for varying reliability of the gradient throughoutthe optimization process or an approximate signal-to-noise ratio of the gradient [4, 28].
Using thedefinition of ∇ W j L ( W ) from Eq.
(13), we can rewrite the right-hand side of Eq.
(14) as:
[Equation]
{+where ⊙ denotes the Hadamard product and S j is a matrix that contains the corresponding entry-wiseelements of 1 / (cid:113) ∇ W j L ( W ) 
+}
+ s 2 j to make the element-wise operations more explicit.
To derivethe evolution of the end-product matrix ˙
W from ˙ W j , we follow the approach in [6] but with severalmodifications due to presence of S j .
For the full details, we defer to the appendix of [6].
We left andright multiply by a product of other weight layers and sum across the depth of the network:
[Equation]
To simplify notation for the proceeding steps, we define the following:
[Equation]
Re-writing the above expression with these new definitions, we have:
[Equation]
Ignoring the negative sign to focus on the summation and taking the (column-major order) vectorization of the above expression, we then have:
[Equation]
where the first two equalities is due to the additivity of the vectorization operator and the associationbetween vectorized products and the Kronecker product ( vec( ABC )
=
( A ⊤ ⊗ C )vec( B ) ), the fourthequality is due to the preservation of the Hadamard product over vectorization, the fifth equalityis the result of applying the same logic as the first equality to vec( ALB ) , and the sixth equality isthe result of the mixed-product property of Kronecker products.
Previous work [5] has shown that,in un-regularized gradient flow, P W := (cid:80) Nj =1 (( WW ⊤ )
N − j N ⊗ ( W ⊤ W ) j − 1 N ) is a p.s.d.
matrix thatserves as an accelerative pre-conditioning that acts on the loss gradient.
However, now there is anadditional component S j so we can further simplify the previous expression:
[Equation]
where G j = diag ( vec ( S j ) denotes taking the elements of vec( S j )) and placing them along thediagonal of a zero matrix.
Finally, substituting the term back in for L , we have a approximatecharacterization of the trajectory of the end-product matrix W under Adam:
[Equation]
where P W,G := (cid:80) Nj =1 (( WW ⊤ ) N − j N ⊗ ( W ⊤ W ) j − 1 N ) G j .
Positive semi-definiteness of P W,G .
Under gradient flow, [5] has shown that P W = (cid:80) Nj =1 P jwhere P
j
=
( W ⊤ W ) N − j N ⊗ ( WW ⊤ )
j − 1 N and P j is symmetric; to show that P j is p.s.d.
, note that,suppressing notation for time dependence t , we can equivalently write for some j ,
[Equation]
(18) where A = V ⊗ U , the Kronecker product of the matrices containing the left and right singularvectors of W respectively, and D is the diagonal matrix containing the singular values of W .
Sincethe term in the middle is the Kronecker product of two diagonal matrices (with non-negative singularvalues along the diagonal), it’s also a diagonal matrix.
Therefore, for non-zero x and some fixed j ,
[Equation]
where c = σ 2 j −
[Table]
[Equation]
and σ 1 is the smallest singular value of W or, equivalently, the smallestvalue of the diagonal matrix D which is by definition non-negative.
G j is also symmetric, since it is diagonal, and p.s.d.
since its diagonal entries (and therefore eigenvalues) are by definitionnon-negative.
Since P j G j is the product of a real symmetric p.s.d.
matrix P j and a real positive diagonal matrix
G j , the product is also p.s.d.
Therefore, P W,G = (cid:80) Nj =1 P j G j is also p.s.d.
as a sum of p.s.d.matrices.
Similar to P W under un-regularized gradient flow, P W,G acts as a pre-conditioning on theloss gradient which now also includes the variance of the loss gradient with respect to each layer j .
Implications of P W,G .
As noted in [5], P W in the original case of un-regularized gradient flowserves as an accelerative pre-conditioning that intensifies with depth and can only exist with sufficientdepth ( N ≥ 2 ).
More specifically, they show that the eigenvalues of P W are (cid:80) Nj =1 σ 2 N − j N r σ 2 j − 1 N r ′ ,corresponding to its eigenvectors vec( u r v ⊤ r ′ ) where u r and v ⊤ r ′ are the left and right singular vectorsof the end-product matrix W ∈ R m × n .
With non-trivial depth ( N ≥ 2 ), increasing σ r or σ r ′ resultsin an increase in the eigenvalue corresponding to the eigen-direction (i.e. rank one matrix)
u r v ⊤ r ′ ,which can be interpreted as the pre-conditioning favoring or accelerating in directions that correspondto singular vectors whose presence in the end-product matrix W is stronger.
In the degenerate case N = 1 , however, the pre-conditioning P W collapses into the identity, causingall of its non-zero eigenvalues to reduce down to just unity—resulting in no accelerative or favoringof any eigen-direction(s) during the optimization process.
In contrast, the eigenvalues of the preconditioning P W,G induced by Adam results in a re-scaling of the original eigenvalues of P W .
Atnon-trivial depth N ≥ 2 , the eigenvalues of P W,G corresponding to its eigenvectors vec( u r v ⊤ r ′ )
takethe form: (cid:80) Nj =1 σ 2 N − j N r
σ 2 j − 1 N r ′
[(1 + η 2 j ) − 1 / 2 ] r,r ′ for r ∈ { 1 , . . .
, m } and r ′ ∈ { 1 , . . . , n } , whereη 2 j = s 2 j / ∇ W j L ( W ) 2 (again, element-wise division) and [ · ] r,r ′ denotes the ( r, r ′ ) -th diagonal entryof the matrix.
As such, the eigen-values are now scaled or normalized by the square root of 1 + η 2 j , aquantity akin to a measure of relative variance, noise-to-signal ratio, or a variance adaptation factorthat can help further promote acceleration in directions of low relative variance, or more reliableupdates, and vice versa in directions of high relative variance.
At degenerate depth, P W,G ’s non-zeroeigenvalues, unlike P W , are no longer just unity and static, producing no accelerative favoring inany direction; instead, its eigenvalues become [(1 + η 2 ) − 1 / 2 ] r,r ′ where now η 2 = s 2 / ∇ W L ( W ) 2 ,allowing for continued variation and accelerative effects even at depth 1.
Evolution of σ under Adam without penalty.
For the trajectories of the singular values { σ i } ,we largely build atop the approach in [6], whose key steps we highlight here.
We start by definingthe analytic singular value decomposition of the end-product matrix as W ( t ) =
U ( t ) D ( t ) V ( t ) ⊤ .
Differentiating W ( t ) with respect to time t :
[Equation]
If we then left-multiply the above expression by U ( t ) ⊤ and right-multiply by V ( t ) , we see that:
[Equation]
where the orthonormal columns of U ( t ) and V ( t ) have helped simplify the earlier expression.
Focusing on the diagonal elements of the above expression and suppressing the notation for timedependence, we see that they are:
[Equation]
where u i and v i are the i -th left and right singular vectors associated with the i -th diagonal elementor, equivalently, the i -th singular value σ i .
Since the columns of U are orthonormal by definition, andbecause u i and v i has constant unit length by definition (i.e. u ⊤ i u
i = 12 ddt || u
i ( t ) || 22 = 0 ), the aboveequation can be distilled down into:
[Equation]
Taking the (column-order first) vectorization of the expression above, then:
[Equation]
We can then substitute our expression for vec( ˙ W ) from Eq.
(17) to characterize the singular valuesunder Adam:
[Equation]
A.4.3 Lemma 4 (Adam, with penalty)
We omit the details of deriving ˙
σ under gradient flow from the beginning and defer to the appendix of[5] for a more comprehensive review.
We highlight the key parts.
Evolution of W under Adam with penalty.
To characterize the evolution of the end-product matrix
W under Adam in presence of the penalty, we can simply leverage our earlier results from Eq.
(17)and combine them with the gradient of the penalty since the penalty is simply an additive componentto the same loss function.
Denoting our penalty by R ( W ) , we can characterize ˙ W as:
[Equation]
Substituting in the gradient of the penalty and vectorizing accordingly, we have:
[Equation]
where U and V ⊤ are the matrices containing the left and right singular vectors of the end-productmatrix W at time t (time notation suppressed in the expression above), λ is the regularization strength,and ˜Σ is a re-weighted version of a rectangular diagonal matrix containing the singular values of W(i.e.
˜Σ =
|| W || ∗ || W || F Σ ).
Here, we can see that P W,G in Eq.
(21) is accelerating not just the loss gradientin helping with optimization with respect to performance via the loss but also the penalty gradient inits tendency towards low-rankedness.
Evolution of σ under Adam with penalty.
Similar to the approach taken to characterize ˙ W underthe penalty, we can re-use the expression in Eq.
(19) and substitute in the appropriate expression forvec( ˙ W ) in the case of Adam with the penalty:
[Equation]
A.4.4 Depth 1 dynamics
[Figure]
Lastly, we clarify the dynamics of the end-product matrix W and its singular values σ in the case of adepth 1 network.
Un-regularized gradient flow and Adam.
As mentioned in Section 3, for un-regularized gradientflow, at N = 1 , the trajectories of the end-product matrix and its singular values reduce down to:
[Equation]
24) where P W =
I mn has now reduced down to the identity; as such, the accelerative pre-conditioningthat typically strengthens with depth no longer exists, as expected.
For un-regularized Adam, at N = 1 , we have:
[Equation]
[Equation]
26) where P W,G
= I mn , G j = G since now
N = j = 1 , G = diag(vec
{+( S j )) , and S j is defined as amatrix whose elements are: [ S j ] m,n = [( ∇ W L ( W ) 
+}
+ s 2 ) − 1 / 2 ] m,n as defined in Eq.
Whilethe depth-dependent accelerative pre-conditioning still no longer exists, we have a new sort of “preconditoning” in the form of a p.s.d.
matrix G that is a function of the squared loss gradient and itsvariance.
Regularized gradient flow and Adam.
Finally, we describe the dynamics for the end-productmatrix and its singular values under gradient flow and Adam at depth 1, but now with the penalty.
For gradient flow with the penalty, we see that:
[Equation]
As noted earlier, in the absence of any pre-conditioning, we do have an additional degree of freedomprovided by the penalty both in terms of the evolution of ˙ W (i.e. via the penalty gradient and not justthe loss gradient) and ˙ σ (i.e. to depend on its own relative magnitude unlike un-regularized gradientflow).
For Adam, we have:
[Equation]
As noted earlier in Section 3, the key is the combination of G , a pre-conditioning that is a function ofthe variance of the loss gradient and appears under Adam and Adam-like variants, and the gradientpenalty from our normalized nuclear norm ratio, that allows a depth 1 network (i.e. no depth) togeneralize as well as a deep network, or deep factorization, and perform the same extent of rankreduction, as gradient flow/descent’s implicit regularization in the presence of depth, to the point ofperfect rank recovery as measured by effective rank like in [6].