A S IMPLE D EEP E QUILIBRIUM M ODEL C ONVERGES TO G LOBAL O PTIMA WITH W EIGHT T YING
A BSTRACT
A deep equilibrium linear model is implicitly deﬁned through an equilibrium point of an inﬁnite sequence of computation.
It avoids any explicit computation of the inﬁnite sequence by ﬁnding an equilibrium point directly via root-ﬁnding and by computing gradients via implicit differentiation.
It is a simple deep equilibrium model with nonlinear activations on weight matrices.
In this paper, we analyze the gradient dynamics of this simple deep equilibrium model with non-convex objective functions for a general class of losses used in regression and classiﬁcation.
Despite non-convexity, convergence to global optimum at a linear rate is guaranteed without any assumption on the width of the models, allowing the width to be smaller than the output dimension and the number of data points.
Moreover, we prove a relation between the gradient dynamics of the simple deep equilibrium model and the dynamics of trust region Newton method of a shallow model.
This mathematically proven relation along with our numerical observation suggests the importance of understanding implicit bias and a possible open problem on the topic.
Our proofs deal with nonlinearity and weight tying, and differ from those in the related literature.
I NTRODUCTION
A feedforward deep neural network consists of a stack of H layers, where H is the depth of the network.
The value for the depth H is typically a hyperparameter and is chosen by network designers (e.g., ResNet-101 in He et al. 2016).
Each layer computes some transformation of the output of the previous layer.
Surprisingly, several recent studies achieved results competitive with the state-ofthe-art performances by using the same transformation for each layer with weight tying (Dabre &
Fujita, 2019; Bai et al., 2019b; Dehghani et al., 2019).
In general terms, the output of the l -th layer with weight tying can be written by
[Equation]
where x is the input to the neural network, z ( l ) is the output of the l -th layer (with z (0) =
x ), θ represents the trainable parameters that are shared among different layers (i.e., weight tying), and z ( l − 1) (cid:55)→ h
( z ( l − 1) ; x, θ ) is some continuous function that transforms z ( l − 1) given x and θ .
With weight tying, the memory requirement does not increase as the depth H increases in the forward pass.
However, the efﬁcient backward pass to compute gradients for training the network usually requires to store the values of the intermediate layers.
Accordingly, the overall computational requirement typically increases as the ﬁnite depth H increases even with weight tying.
Instead of using a ﬁnite depth H , Bai et al. (2019a) recently introduced the deep equilibrium model that is equivalent to running an inﬁnitely deep feedforward network with weight tying.
Instead of running the layer-by-layer computation in equation (1), the deep equilibrium model uses rootﬁnding to directly compute a ﬁxed point z ∗ =
lim l →∞ z ( l ) , where the limit can be ensured to exist by a choice of h .
We can train the deep equilibrium model with gradient-based optimization by analytically backpropagating through the ﬁxed point using implicit differentiation (e.g., Griewank & Walther, 2008; Bell & Burke, 2008; Christianson, 1994).
With numerical experiments, Bai et al.
(2019a) showed that the deep equilibrium model can improve performance over previous state-ofthe-art models while signiﬁcantly reducing memory consumption.
Despite the remarkable performances of deep equilibrium models, our theoretical understanding of its properties is yet limited.
Indeed, immense efforts are still underway to mathematically understand deep linear networks, which have ﬁnite values for the depth H without weight tying (Saxe et al., 2014; Kawaguchi, 2016; Hardt & Ma, 2017; Laurent & Brecht, 2018; Arora et al., 2018; Bartlett et al., 2019; Du & Hu, 2019; Arora et al., 2019a;
Zou et al., 2020b).
In deep linear networks, the function h at each layer is linear in θ and linear in x ; i.e., the map ( x, θ ) (cid:55)→ h ( z ( l − 1) ; x, θ ) is bilinear.
Despite this linearity, several key properties of deep learning are still present in deep linear networks.
For example, the gradient dynamics is nonlinear and the objective function is nonconvex.
Accordingly, understanding gradient dynamics of deep linear networks is considered to be a valuable step towards the mathematical understanding of deep neural networks (Saxe et al., 2014;
Arora et al., 2018; 2019a).
In this paper, inspired by the previous studies of deep linear networks, we initiate a theoretical study of gradient dynamics of deep equilibrium linear models as a step towards theoretically understanding general deep equilibrium models.
As we shall see in Section 2, the function h at each layer is nonlinear in θ for deep equilibrium linear models, whereas it is linear for deep linear networks.
This additional nonlinearity is essential to enforce the existence of the ﬁxed point z ∗ .
The additional nonlinearity, the inﬁnite depth, and weight tying are the three key proprieties of deep equilibrium linear models that are absent in deep linear networks.
Because of these three differences, we cannot rely on the previous proofs and results in the literature of deep linear networks.
Furthermore, we analyze gradient dynamics, whereas Kawaguchi (2016); Hardt & Ma (2017); Laurent & Brecht (2018) studied the loss landscape of deep linear networks.
We also consider a general class of loss functions for both regression and classiﬁcation, whereas Saxe et al.
; Arora et al.
; Bartlett et al.
(2019); Arora et al. (2019a); Zou et al.
(2020b) analyzed gradient dynamics of deep linear networks in the setting of the square loss.
Accordingly, we employ different approaches in our analysis and derive qualitatively and quantitatively different results when compared with previous studies.
In Section 2, we provide theoretical and numerical observations that further motivate us to study deep equilibrium linear models.
In Section 3, we mathematically prove convergence of gradient dynamics to global minima and the exact relationship between the gradient dynamics of deep equilibrium linear models and that of the adaptive trust region method.
Section 5 gives a review of related literature, which strengthens the main motivation of this paper along with the above discussion (in Section 1).
Finally, Section 6 presents concluding remarks on our results, the limitation of this study, and future research directions.
P RELIMINARIES
We begin by deﬁning the notation.
We are given a training dataset (( x i , y i ))
ni
=1 of n samples where x i ∈ X ⊆
R m x and y i ∈ Y ⊆
R m y are the i -th input and the i -th target output, respectively.
We would like to learn a hypothesis (or predictor) from a parametric family H =
{ f θ : R m x → R m y | θ ∈ Θ } by minimizing the objective function L (called the empirical loss) over θ ∈ Θ :
L ( θ ) = (cid:80) ni =1 (cid:96) ( f θ ( x i ) , y i ) , where θ is the parameter vector and (cid:96) : R m y × Y
→ R ≥ 0 is the loss function that measures the difference between the prediction f θ ( x i ) and the target
y i for each sample.
For example, when the parametric family of interest is the class of linear models as H =
{ x (cid:55)→ Wφ ( x ) | W ∈ R m y × m } , the objective function L can be rewritten as:
[Equation]
where the feature map φ is an arbitrary ﬁxed function that is allowed to be nonlinear and is chosen by model designers to transforms an input x ∈ R m x into the desired features φ ( x ) ∈ R m .
We use vec( W ) ∈
R m y m to represent the standard vectorization of a matrix W ∈ R m y × m .
Instead of linear models, our interest in this paper lies on deep equilibrium models .
The output z ∗ of the last hidden layer of a deep equilibrium model is deﬁned by
[Equation]
where the last equality follows from the continuity of z (cid:55)→ h ( z ; x, θ ) (i.e., the limit commutes with the continuous function).
Thus, z ∗ can be computed by solving the equation z ∗ =
h ( z ∗ ; x, θ ) without running the inﬁnitely deep layer-by-layer computation.
The gradients with respect to parameters are computed analytically via backpropagation through z ∗ using implicit differentiation.
2. D EEP E QUILIBRIUM L INEAR M ODELS
A deep equilibrium linear model is an instance of the family of deep equilibrium models and is deﬁned by setting the function h at each layer as follows:
[Equation]
where θ =
( A, B ) with two trainable parameter matrices A ∈ R m × m and B ∈ R m y × m .
Along with a positive real number γ ∈ (0 , 1) , the nonlinear function σ is used to ensure the existence of the ﬁxed point and is deﬁned by σ ( A ) ij = exp
[Equation]
A
[Equation]
mk =1 exp( A kj ) .
The class of deep equilibrium linear models is given by H = { x (cid:55)→ B (cid:0) lim l →∞ z ( l ) ( x, A ) (cid:1) | A ∈ R m × m , B ∈ R
m y × m } , where z ( l ) ( x, A )
=
γσ ( A ) z ( l − 1) + φ ( x ) .
Therefore, the objective function for deep equilibrium linear models can be written as
[Equation]
The outputs of deep equilibrium linear models f θ ( x ) = B (cid:0)
lim l →∞ z ( l ) ( x, A ) (cid:1) are nonlinear and non-multilinear in the optimization variable A .
This is in contrast to linear models and deep linear networks.
From the optimization viewpoint, linear models Wφ ( x ) are called linear because they are linear in the optimization
variables W .
Deep linear networks W ( H ) W ( H − 1) · · · W (1) x are multilinear in the optimization variables ( W (1) , W (2) , . . .
, W ( H ) )
(this holds also when we replace x by φ ( x ) ).
This difference creates a challenge in the analysis of deep equilibrium linear models.
Following previous works on gradient dynamics of different machine learning models (Saxe et al., 2014; Ji & Telgarsky, 2020), we consider the process of learning deep equilibrium linear models via gradient ﬂow:
[Equation]
where ( A t , B t ) represents the model parameters at time t with an arbitrary initialization ( A 0 , B 0 ) .
Throughout this paper, a feature map φ and a real number γ ∈ (0 , 1) are given and arbitrary (except in experimental observations) and we omit their universal quantiﬁers for the purpose of brevity.
2. P RELIMINARY O BSERVATION FOR A DDITIONAL M OTIVATION
Our analysis is chieﬂy motivated as a step towards mathematically understanding general deep equilibrium models (as discussed in Sections 1 and 5).
In addition to the main motivation, this section provides supplementary motivations through theoretical and numerical preliminary observations.
In general deep equilibrium models, the limit, lim l →∞ z ( l ) , is not ensured to exist (see Appendix C).
In this view, the class of deep equilibrium linear models is one instance where the limit is guaranteed to exist for any values of model parameters as stated in Proposition 1:
Proposition 1.
Given any ( x, A ) , the sequence ( z ( l ) ( x, A )) l in Euclidean space R m converges.
Proof.
We use the nonlinearity σ to ensure the convergence in our proof in Appendix A.5.
Proposition 1 shows that we can indeed deﬁne the deep equilibrium linear model with lim l →∞ z ( l ) =
z ∗ ( x, A ) .
Therefore, understanding this model is a sensible starting point for theory of general deep equilibrium models.
As our analysis has been mainly motivated for theory, it would be of additional value to discuss whether the model would also make sense in practice, at least potentially in the future.
Consider an (unknown) underling data distribution P ( x, y )
=
P ( y | x ) P ( x ) .
Intuitively, if the mean of the P ( y | x ) is approximately given by a (true unknown) deep equilibrium linear model, then it would make sense to use the parametric family of deep equilibrium linear models to have the inductive bias in practice.
To conﬁrm this intuition, we conducted numerical simulations.
To generate datasets, we ﬁrst drew uniformly at random 200 input images for input data points x
i from a standard image dataset —
CIFAR-10, CIFAR-100 or Kuzushiji-MNIST (Krizhevsky & Hinton, 2009; Clanuwat et al., 2019).
We then generated targets as y i
=
B ∗ (lim l →∞ z ( l ) ( x i , A ∗ ))
+ δ i where δ i i.i.d.
∼ N (0 , 1) .
Each entry of the true (unknown) matrices A ∗ and B ∗ was independently drawn from the standard normal distribution.
For each dataset generated in this way, we used stochastic gradient descent (SGD) to train linear models, fully-connected feedforward deep neural networks with ReLU nonlinearity [Figure] [Figure] [Figure]
[Figure]
[Figure] [Figure] [Figure] (DNNs), and deep equilibrium linear models.
For all models, we ﬁxed φ ( x ) = x .
See Appendix for of the experimental settings.
The results of this numerical test are presented in Figure 1.
In the ﬁgure, the plotted lines indicate the mean values over ﬁve random trials whereas the shaded regions represent error bars with one standard deviation.
The plots for linear models and deep equilibrium linear models are shown with the best and worst learning rates (separately for each model in terms of the ﬁnal test errors at epoch = 5000 ) from the set of learning rates S LR = { 0 . 01 , 0 .
The plots for DNNs are shown with the best learning rates (separately for each depth H ) from the set S LR .
As can be seen, all models preformed approximately the same at initial points, but deep equilibrium linear models outperformed both linear models and DNNs in test errors after training, conﬁrming our intuition above.
Moreover, we conﬁrmed qualitatively same behaviors with four more datasets as well as for DNNs with and without bias terms in Appendix D. These observations additionally motivated us to study deep equilibrium linear models to obtain our main results in the next section.
The purpose of these experiments is to provide a secondary motivation for our theoretical analyses.
M AIN R ESULTS
In this section, we establish mathematical properties of gradient dynamics for deep equilibrium linear models by directly analyzing its trajectories.
We prove linear convergence to global minimum in Section 3.1 and further analyze the dynamics from the viewpoint of trust region in Section 3.2.
3. C ONVERGENCE A NALYSIS
We begin in Section 3.1.1 with a presentation of the concept of the Polyak-Łojasiewicz (PL) inequality and additional notation.
The PL inequality is used to regularize the choice of the loss functions (cid:96) in our main convergence theorem for a general class of losses in Section 3.1.2.
We conclude in
Section 3.1.3 by providing concrete examples of the convergence theorem with the square loss and the logistic loss, where the PL inequality is no longer required as the PL inequality is proven to be satisﬁed by these loss functions.
3.1. T HE P OLYAK -Ł OJASIEWICZ I NEQUALITY AND A DDITIONAL N OTATION
In our context, the notion of the PL inequality is formally deﬁned as follows:
Deﬁnition 1.
The function L 0 is said to satisfy the Polyak-Łojasiewicz (PL) inequality with radius
R ∈ (0 , ∞ ] and parameter κ > [-
-]0 if 12 (cid:107)∇ L vec0 (vec( W ))
(cid:107) 22 ≥ κ ( L vec0 (vec( W ))
− L ∗ 0 ,R ) for all (cid:107) W (cid:107) 1 < R , where L vec0 (vec( · )) :
=
L 0 ( · ) and L ∗ 0 ,R :=
inf W : (cid:107) W (cid:107) 1 <R L 0 ( W ) .
With any radius R > 0 sufﬁciently large (such that it covers the domain of L 0 ), Deﬁnition 1 becomes equivalent to the deﬁnition of the PL inequality in the optimization literature (e.g., Polyak, 1963;
Karimi et al., 2016).
See Appendix C for additional explanations on the equivalence.
In general, the non-convex objective function L of deep equilibrium linear models does not satisfy the PL inequality.
Therefore, we cannot assume the inequality on L .
However, in order to obtain linear convergence for a general class of the loss functions (cid:96) , we need some assumption on (cid:96) : otherwise, we can choose a loss (cid:96) to violate the convergence.
Accordingly, we will regularize the choice of the loss (cid:96) through the PL inequality on the function L 0 : W (cid:55)→ (cid:80) ni
=1 (cid:96) ( Wφ ( x i ) , y i ) .
The PL inequality with a radius R ∈ (0 , ∞ ] (Deﬁnition 1) leads to the notion of the global minimum value in the domain corresponding to the radius in our analysis: L ∗ R = inf A ∈ R m × m ,B ∈B R L ( A, B ) , where B R = { B ∈ R m y × m | (cid:107) B (cid:107) 1 < (1 − γ ) R } .
With R = ∞ , this recovers the global minimum value L ∗ in the unconstrained domain as L ∗ R =
L ∗ :=
inf A ∈ R m × m ,B ∈ R my × m L ( A, B ) .
Furthermore, if a global minimum ( A ∗ , B ∗ ) ∈ R m × m × R m y × m exists, there exists ¯ R < ∞ such that for any R ∈ [ ¯ R, ∞ ) , we have B ∗ ∈ B R and thus L ∗ R =
L ∗ .
In other words, if a global minimum exists, using a (sufﬁciently large) ﬁnite radius R < ∞ sufﬁces to obtain L ∗ R =
L ∗ .
We close this subsection by introducing additional notation.
For a real symmetric matrix M , we use λ min ( M ) to represent its smallest eigenvalue.
For an arbitrary matrix M ∈ R d × d (cid:48) , we let rank( M ) be its rank, (cid:107) M (cid:107) p be its matrix norm induced by the vector p -norm, σ min ( M ) be its smallest singular value (i.e., the min( d, d (cid:48) ) -th largest singular value), M ∗ j be its j -th column vector in R d , and M i ∗ be its i -th row vector in R d (cid:48) .
For d ∈ N > 0 , we denote by I
d the identify matrix in R d × d .
We deﬁne the Jacobian matrix J k,t ∈ R m × m of the vector-valued function A ∗ k (cid:55)→ σ ( A ) ∗ k by ( J k,t ) ij = ∂σ ( A ) ik ∂A jk |
A =
A t for all t ≥ 0 and k = 1 , . . .
, m .
Finally, we deﬁne the feature matrix Φ ∈ R m × n by Φ ki = φ ( x i ) k for k = 1 , . . .
, m and i = 1 , . . . , n .
3.1. M AIN C ONVERGENCE T HEOREM
Using the PL inequality only on the loss function (cid:96) through L 0 (Deﬁnition 1), we present our main theorem — a guarantee on linear convergence to global minimum for the gradient dynamics of the non-convex objective L for deep equilibrium linear models:
Theorem 1.
Let (cid:96) : R m y × Y
→ R ≥ 0 be arbitrary such that the function q (cid:55)→ (cid:96) ( q, y i ) is differentiable for any i ∈ { 1 , . . . , n } (with an arbitrary m y ∈ N > 0 and an arbitrary Y ).
Then, for any T > 0 , R ∈ (0 , ∞ ] and κ > 0 such that (cid:107) B t (cid:107) 1 < (1 − γ ) R for all t ∈ [0 , T ] and L 0 satisﬁes the PL inequality with the radius R and the parameter κ , the following holds:
[Equation]
where λ T :=
inf t ∈ [0 ,T ] λ min ( D t ) >
and D t is a positive deﬁnite matrix deﬁned by
[Equation]
with U t := I m − γσ ( A t ) .
Furthermore, λ T ≥ 1 m (1+ γ
[Equation]
2 for any {+T ≥ 
+}
(lim T →∞ λ T ≥ 1 m (1+ γ ) 2 ) .
Proof.
The additional nonlinearity σ creates a complex interaction among m hidden neurons.
This interaction is difﬁcult to be factorized out for the gradients of L with respect to A .
This is different from but analogous to the challenge to deal with nonlinear activations in the loss landscape of (nonoverparameterized) deep nonlinear networks, for which previous works have made assumptions of sparse connections to factorize the interaction (Kawaguchi et al., 2019).
In contrast, we do not rely on sparse connections.
Instead, we observe that although it is difﬁcult to factorize this complex interaction (due to the nonlinearity σ ) in the space of loss landscape, we can factorize it in the space of gradient dynamics.
See Appendix A.1 for the proof overview and the complete proof.
Theorem 1 shows that in the worst case for λ T , the optimality gap decreases exponentially towards zero as L ( A T , B T ) − L ∗ R ≤ C
e − 2 κ m (1+ γ ) 2 T , where C
=
L ( A 0 , B 0 ) − L ∗ 0 ,R .
Therefore, for any desired accuracy (cid:15) > 0 , setting C 0 e −
[Equation]
≤ (cid:15) and solving for T yield that
[Equation]
for any T ≥ m
[Equation]
[Equation] Theorem 1 also states that the rate of convergence improves further depending on the quality of the matrix D t (deﬁned in equation (8)) in terms of its smallest eigenvalue over the particular trajectory ( A t , B t ) up to the speciﬁc time t ≤ T ; i.e., λ T = inf t ∈ [0 ,T ] λ min ( D t ) .
This opens up the direction of future work for further improvement of the convergence rate through the design of initialization ( A 0 , B 0 ) to maximize λ T for trajectories generated from a speciﬁc initialization scheme.
3.1. E XAMPLES : S QUARE L OSS AND L OGISTIC L OSS
The main convergence theorem in the previous subsection is stated for any radius R ∈ (0 , ∞ ] and parameter κ > 0 that satisfy the conditions on (cid:107) B t (cid:107) 1 and the PL inequality (see Theorem 1).
The values of these variables are not completely speciﬁed there as they depend on the choice of the loss functions (cid:96) .
In this subsection, we show that these values can be speciﬁed further and the condition on PL inequality can be discarded by considering a speciﬁc choice of loss functions (cid:96) .
In particular, by using the square loss for (cid:96) , we prove that we can set R = ∞ and κ = 2 σ min (Φ) 2 :
Corollary 1.
Let (cid:96) ( q, y i )
= (cid:107) q − y
i
(cid:107) 22 where y i ∈
R m y for i = 1 , 2 , . .
. , n (with an arbitrary m y ∈ N > 0 ).
Assume that rank(Φ)
=
min( n, m ) .
Then for any T > 0 ,
[Equation]
where σ min (Φ) > 0 , L ∗ 0 :=
inf W ∈ R my ×
m L 0 ( W ) , and λ T :=
inf t ∈ [0 ,T ] λ min ( D t ) ≥ 1 m (1+ γ
[Equation]
Proof.
This statement follows from Theorem 1.
The conditions on (cid:107) B t (cid:107) 1 and the PL inequality (in
Theorem 1) are now discarded by using the property of the square loss (cid:96) .
See Appendix A.3 for the complete proof.
In Corollary 1, the global linear convergence is established for the square loss without the notion of the radius R as we set R = ∞ .
Even with the square loss, the objective function L is non-convex.
Despite the non-convexity, Corollary 1 shows that for any desired accuracy (cid:15) > 0 ,
[Equation]
for any T ≥ m
[Equation]
1 + γ )
[Equation]
Corollary 1 allows both cases of m ≤ n and m > n .
In the case of over-parameterization m > n , the covariance matrix ΦΦ (cid:62) ∈ R m × m (or XX (cid:62) with φ ( x ) = x ) is always rank deﬁcient because rank(ΦΦ (cid:62) ) =
rank(Φ) ≤ n < m .
This implies that the Hessian of L 0 is always rank deﬁcient, because the Hessian of L 0 is ∇ 2 L vec0 (vec( W ))
= 2[ΦΦ (cid:62) ⊗ I m y
] ∈ R m y m × m y m (see Appendix
A.3 for its derivation) and because rank([ΦΦ (cid:62) ⊗ I m y ]) = rank(ΦΦ (cid:62) ) rank( I m y ) ≤ m y n < m y m .
Since the strong convexity on a twice differentiable function requires its Hessian to be of full rank, this means that the objective L 0 for linear models is not strongly convex in the case of overparameterization m > n .
Nevertheless, we establish the linear convergence to global minimum for deep equilibrium linear models in Corollary 1 for both cases of m > n and m ≤ n by using
Theorem 1.
For the logistic loss for (cid:96) , the following corollary proves the global convergence at a linear rate:
Corollary 2.
Let (cid:96) ( q, y i
[Equation]
=
[Equation]
+ τ
[Equation]
q
[Equation]
22 with an arbitrary τ ≥ 0 where y i ∈ { 0 , 1 } for i = 1 , 2 , . . . , n .
Assume that rank(Φ)
= m .
Then for any T > 0 and
R ∈ (0 , ∞ ] such that (cid:107) B t (cid:107) 1 < (1 − γ ) R for all t ∈ [0 , T ] , the following holds:
[Equation]
where σ min (Φ) > 0 , λ T := inf t ∈ [0 ,T ] λ min ( D t ) ≥ 1 m (1+ γ ) 2 , and ρ (
[Equation]
[Equation]
Proof.
This statement follows from Theorem 1 by proving that the condition on PL inequality is satisﬁed with the parameter κ
[Equation]
In Corollary 2, we can also set R = ∞ to remove the notion of the radius R from the statement of the global convergence for the logistic loss.
By setting R = ∞ , Corollary 2 states that for any
T > 0 ,
[Equation]
for the logistic loss.
For any τ > 0 , this implies that for any desired accuracy (cid:15) > 0 ,
[Equation]
In practice, we may want to set τ > 0 to regularize the parameters (for generalization) and to ensure the existence of global minima (for optimization and identiﬁability).
That is, if we set τ = 0 instead, the global minima may not exist in any bounded space, due to the property of the logistic loss.
This is consistent with Corollary 2 in that if τ = 0 , equation (11) does not hold and we must consider the convergence to the global minimum value L ∗ R deﬁned in a bounded domain with a radius R < ∞ .
In the case of τ = 0 and R < ∞ , Corollary 2 implies that for desired accuracy (cid:15) > 0 ,
[Equation]
where we have ρ ( R ) > 0 because R < ∞ .
Therefore, Corollary 2 establish the linear convergence to global minimum with both cases of τ > 0 and τ = 0 for the logistic loss.
3. U NDERSTANDING D YNAMICS T HROUGH T RUST R EGION N EWTON M ETHOD In
this subsection, we analyze the dynamics of deep equilibrium linear models in the space of the hypothesis, f θ t : x (cid:55)→ B t (cid:0) lim l →∞ z ( l ) ( x, A t ) (cid:1) .
For any functions g and ¯ g with a domain X ⊆
R m x , we write g = ¯ g if g ( x ) = ¯ g ( x ) for all x ∈ X .
The following theorem shows that the dynamics of deep equilibrium linear models f θ t can be written as ddt f θ t = 1 δ t V t φ where 1 δ t is scalar and V t follows the dynamics of a trust region Newton method of shallow models with the (non-standard) adaptive trust region V t .
This suggests potential beneﬁts of deep equilibrium linear models in two aspects: when compared to shallow models, it can sometimes accelerate optimization via the effect of the implicit trust region method (but not necessarily as the trust region method does not necessarily accelerate optimization) and induces novel implicit bias for generalization via the non-standard implicit trust region V t .
Theorem 2.
Let (cid:96) : R m y ×
Y
→ R ≥ 0 be arbitrary such that the function q (cid:55)→ (cid:96) ( q, y i ) is differentiable for any i ∈ { 1 , . . . , n } with m y = 1 and (an arbitrary Y ).
Then for any time t ≥ 0 , there exist a real number ¯ δ t > 0 such that for any δ t ∈ (0 , ¯ δ t ] ,
[Equation]
(13) where V t := { v ∈ R m : (cid:107) v (cid:107) G t ≤ δ t (cid:107) ddt vec( B t U − 1 t ) (cid:107) G t } , G t :=
U t (cid:0) S − 1 t − δ t F t (cid:1) U (cid:62) t (cid:31) 0 , and
L t 0 ( v ) : =
L vec0 (vec( B t U − 1 t ))
+ ∇ L vec0 (vec( B t U − 1 t )) (cid:62) v + 1 2 v (cid:62) ∇ 2 L vec0 (vec( B t U − 1 t )) v.
Here, F t := (cid:80) ni
=1 ∇ 2 (cid:96) i ( f θ t ( x i ))
(lim l →∞ z ( l ) ( x i , A t ))
(lim l →∞ z ( l ) ( x i , A t )) (cid:62) with (cid:96) i ( q ) :
= (cid:96) ( q, y i ) and S t := I m + γ 2 diag( v St ) with v St ∈ R m and ( v St ) k := (cid:107)
J (cid:62) k,t ( B t U − 1 t ) (cid:62) (cid:107) 22 ∀ k .
Proof.
This is proven with the Karush–Kuhn–Tucker (KKT) conditions for the constrained optimization problem: minimize v ∈V t L t 0 ( v ) .
See Appendix A.2.
When many global minima exist, a difference in the gradient dynamics can lead to a signiﬁcant discrepancy in the learned models: i.e., two different gradient dynamics can ﬁnd signiﬁcantly different global minima with different behaviors for generalization and test accuracies (Kawaguchi et al., 2017).
In machine learning, this is an important phenomenon called implicit bias — inductive bias induced implicitly through gradient dynamics — and is the subject of an emerging active research area (Gunasekar et al., 2017; Soudry et al., 2018; Gunasekar et al., 2018;
Woodworth et al., 2020;
Moroshko et al., 2020).
As can be seen in Theorem 2, the gradient dynamics of deep equilibrium linear models f θ t differs from that of linear models W t φ with any adaptive learning rates, ﬁxed preconditioners, and existing variants of Newton methods.
This is consistent with our experiments in Section 2.2 and Appendix D where the dynamics of deep equilibrium linear models resulted in the learned predictors with higher test accuracies, when compared to linear models with any learning rates.
In this regard, Theorem 2 provides a partial explanation (and a starting point of the theory) for the observed generalization behaviors, whereas Theorem 1 (with Corollaries 1 and 2) provides the theory for the global convergence observed in the experiments.
Theorem 2, along with our experimental results, suggests the importance of theoretically understanding implicit bias of the dynamics with the time-dependent trust region.
In Appendix B, we show that Theorem 2 suggests a new type of implicit bias towards a simple function as a result of inﬁnite depth, whereas understanding this implicit bias in more details is left as an open problem for future work.
E XPERIMENTS
In this section, we conduct experiments to further verify and demonstrate our theory.
To compare with the previous ﬁndings, we use the same synthetic data as that in the previous work (Zou et al., 2020b): i.e., we randomly generate x i ∈ R 10 from the standard normal distribution and set y i =
− x i + 0 .
1 ς i for all i ∈ { 1 , 2 , . . . , n } with n = 1000 , where ς i is independently generated by the standard normal distribution.
We set φ ( x ) = x and use the square loss (cid:96) ( q, y i ) =
(cid:107) q − y
i
(cid:107) 22 .
As in the previous work, we consider random initialization and identity initialization (Zou et al., 2020b) and report the results in Figure 2 (a).
As can be seen in the ﬁgure, deep equilibrium linear models converges to the global minimum value with all initialization and random trials, whereas linear ResNet converges to a suboptimal value with identity initialization.
This is consistent with our theory for deep equilibrium linear models and the previous work for ResNet (Zou et al., 2020b).
We repeated the same experiment by generating ( x i ) k independently from the uniform distribution of the interval [ − 1 , 1] instead for all i ∈ { 1 , . . . , n } and k ∈ { 1 , . . .
, m } with n = 1000 and m = 10 .
Figure 2 (b) shows the results of this experiment with the uniform distribution and conﬁrm the global convergence of deep equilibrium linear models again with all initialization and random trials.
In this case, linear ResNet with identity initialization also converged to the global minimum value.
These observations are consistent with Corollary 1 where deep equilibrium linear models are guaranteed to converge to the global minimum value without any condition on the initialization.
We now consider the rate of the global convergence.
In Corollary 1, we can set λ T = 1 m (1+ γ ) 2 to get a guarantee for the global linear convergence rate for all initializations in theory.
However, in practice, this is a pessimistic convergence rate and we may want to choose λ T depending on a initialization.
To demonstrate this, using the same data as that in Figure 2 (a), Figure 2 (c) reports the numerical training trajectory along with theoretical upper bounds with initialization-independent λ T = 1 m (1+ γ ) 2 and initialization-dependent λ T = inf t ∈ [0 ,T ] λ min ( D t ) .
As can be seen in Figure(c), the theoretical bound with initialization-dependent λ T demonstrates a faster and more accurate convergence rate.
A qualitatively same observation is reported for the logistic loss in Appendix D.2.
[Figure]
R ELATED W ORK
The theoretical study of gradient dynamics of deep networks with some linearized component is a highly active area of research.
Recently, Bartlett et al.
; Du & Hu (2019); Arora et al.
(2019a)
; Zou et al.
(2020b) analyzed gradient dynamics of deep linear networks and proved global convergence rates for the square loss under certain assumptions on the dataset, initialization, and network structures.
For example, the dataset is assumed to be whitened (i.e., ΦΦ (cid:62)
= I m or XX (cid:62) =
I m x ) and the initial loss is assumed to be smaller than the loss of any rank-deﬁcient solution by Arora et al.
(2019a): the input and output layers are assumed to represent special transformations and are ﬁxed during training by Zou et al.
(2020b).
Deep networks are also linearized implicitly in the neural tangent kernel (NTK) regime with significant over-parameterization m (cid:29) n (Yehudai & Shamir, 2019; Lee et al., 2019).
By signiﬁcantly increasing model parameters (or more concretely the width m ), we can ensure deep features or corresponding NTK to stay nearly the same during training.
In other words, deep networks in this regime are approximately linear models with random features corresponding to the NTK at random initialization.
Because of this implicit linearization, deep networks in the NTK regime are shown to achieve globally minimum training errors by interpolating all training data points (Zou et al., 2020a;
Li & Liang, 2018; Jacot et al., 2018; Du et al., 2019; 2018; Chizat et al., 2019; Arora et al., 2019b;
Allen-Zhu et al., 2019; Lee et al., 2019; Fang et al., 2020; Montanari & Zhong, 2020).
These previous studies have signiﬁcantly advanced our theoretical understanding of deep learning through the study of deep linear networks and implicitly linearized deep networks in the NTK regime.
In this context, this paper is expected to contribute to the theoretical advancement through the study of a new and signiﬁcantly different type of deep models — deep equilibrium linear models.
In deep equilibrium linear models, the function at each layer A (cid:55)→ h ( z ( l − 1) ; x, θ ) is nonlinear due to the additional nonlinearity σ : A (cid:55)→ h ( z ( l − 1) ; x, θ ) :
= γσ ( A )
z ( l − 1)
+ φ ( x ) .
In contrast, for deep linear networks, the function at each layer W ( l ) (cid:55)→ h ( l ) ( z ( l − 1) ; x, W ( l ) ) :
= W ( l ) z ( l − 1) is linear (it is linear also with skip connection).
Furthermore, the nonlinearity σ is not an element-wise function, which poses an additional challenge in the mathematical analysis of deep equilibrium linear models.
The nonlinearity σ , the inﬁnite depth, and weight tying in deep equilibrium linear models necessitated us to develop new approaches in our proofs.
The differences in the models and proofs naturally led to qualitatively and quantitatively different results.
For example, we do not require any of over-parameterization m (cid:29) n , interpolation of all training data points, and any assumptions mentioned above for deep linear networks.
Unlike previous papers, we also related the dynamics of deep equilibrium linear models to that of a trust region Newton method of shallow models with G t -quadratic norm.
This suggested potential beneﬁts of deep equilibrium linear models.
Our theory is consistent with our numerical observations.
C ONCLUSION
For deep equilibrium linear models, despite the non-convexity, we have rigorously proven convergence of gradient dynamics to global minima, at a linear rate, for a general class of loss functions, including the square loss and logistic loss.
Moreover, we have proven the relationship between the gradient dynamics of deep equilibrium linear models and that of the adaptive trust region method.
These results apply to models with any conﬁguration on the width of hidden layers, the number of data points, and input/output dimensions, allowing rank-deﬁcient covariance matrices as well as both under-parameterization and over-parameterization.
The crucial assumption for our analysis is the differentiability of the function q (cid:55)→ (cid:96) ( q, y i ) , which is satisﬁed by standard loss functions, such as the square loss, the logistic loss, and the smoothed hinge loss (cid:96) ( q, y i ) =
(max { 0 , 1 − y i q } ) k with k ≥ 2 .
However, it is not satisﬁed by the (non-smoothed) hinge loss (cid:96) ( q, y i )
=
max { 0 , 1 − y i q } , the treatment of which is left to future work.
Future work also includes corresponding theoretical analyses with stochastic gradient descent.
Our theoretical results (in Section 3) and numerical observations (in Section 2.2 and Appendix D) uncover the special properties of deep equilibrium linear models, providing a basis of future work for theoretical studies of implicit bias and for further empirical investigations of deep equilibrium models.
In our proofs, the treatments of the additional nonlinearity σ , the inﬁnite depth, and weight tying are especially unique, and we expect our new proof techniques to be proven useful in further studies of gradient dynamics for deep models.
A P ROOFS
In this appendix, we complete the proofs of our theoretical results.
We present the proofs of Theorem 1 in Appendix A.1, Theorem 2 in Appendix A.2, Corollary 1 in Appendix A.3, Corollary 2 in
Appendix A.4, and Proposition 1 in Appendix A.5.
We also provide a proof overview of Theoremin the beginning of Appendix A.1.
Before starting our proofs, we ﬁrst introduce additional notation used in the proofs and then discuss alternative proofs using the implicit function theorem to avoid relying on the convergence of
Neumann series.
Additional notation.
Given a scalar-valued function a ∈ R and a matrix M ∈ R d × d (cid:48) , we write
[Equation]
where M ij represents the ( i, j ) -th entry of the matrix M .
Given a vector-valued function a ∈ R d and a column vector b ∈ R d (cid:48) , we write
[Equation]
where b i represents the i -th entry of the column vector b .
Similarly, given a vector-valued function a ∈ R d and a row vector b ∈ R 1 × d (cid:48) , we write
[Equation]
where b 1 i represents the i -th entry of the row vector b .
We use ∇ A L to represent the map ( A, B ) (cid:55)→ ∂L∂A
( A, B ) (without the usual transpose used in vector calculus).
Given a matrix M and a function ϕ , we deﬁne ∇ M ϕ similarly as the map M (cid:55)→ ∂ϕ∂M ( M ) .
Our proofs also use the indicator function:
[Equation]
Finally, we recall the deﬁnition of the Kronecker product of two matrices: for matrices M ∈
R d M × d (cid:48) M and ¯ M ∈ R d ¯ M × d (cid:48) ¯ M ,
[Equation]
On alternative proofs using the implicit function theorem.
In our default proofs, we utilize the Neumann series (cid:80) ∞ k =0 γ k σ ( A ) k when deriving the formula of the gradients with respect to A .
Instead of using the Neumann series, we can alternatively use the implicit function theorem to derive the formula of the gradients with respect to A .
Speciﬁcally, in this alternative proof, we apply the implicit function theorem to the function ψ deﬁned by
[Equation]
where vec[ A ] and z ∈ R m are independent variables of the function ψ : i.e., ψ (vec[ A ] , z ) is allowed to be nonzero.
On the other hand, the vector z satisfying ψ (vec[ A ] , z ) = 0 is the ﬁxed point z ∗ =
lim l →∞ z ( l ) based on equation (3).
Therefore, by applying the implicit function theorem to the function ψ , it holds that if the the Jacobian matrix ∂ψ
[Equation]
vec
[Equation] [Equation]
is invertible, then
[Equation]
Since ∂ψ (vec[ A ] ,z ) ∂z
[Equation]
) is invertible, it holds that
[Equation]
Moreover, since σ ( A ) z ∈ R m is a column vector,
[Equation]
Combining equations (15) and (16) [Equation] we have
[Equation]
In our proofs [Equation] whenever we require the gradients with respect to A
[Equation]
we can directly use equation (17) [Equation] instead of relying on the convergence of the Neumann series.
For example, equation (21) in the proof of Theorem 1 is identical to equation (17) with additional multiplication of B q ∗ :
i
[Equation]
e., for the left hand side,
[Equation]
and for the right hand side,
[Equation]
A. P ROOF OF T HEOREM
We begin with a proof overview of Theorem 1.
We ﬁrst compute the derivatives of the output of deep equilibrium linear models with respect to the parameters A in Appendix A.1.1.
Then using the derivatives, we rearrange the formula of ∇ A L such that it is related to the formula of ∇ L 0 in
Appendices A.1.1–A.1.3.
Intuitively, we then want to understand ∇ A L through the property of ∇ L 0 , similarly to the landscape analyses of deep linear networks by Kawaguchi (2016).
However, we note there that the additional nonlinearity σ creates a complex interaction over the dimension m to prevent us from using such a proof approach.
Instead, using the proven relation of ∇ A L and ∇ Lfrom Appendices A.1.1–A.1.3, we directly analyze the trajectories of the dynamics over time t in
Appendices A.1.4–A.1.5, which results in a partial factorization of the iteration over the dimension m .
Using such a partial factorization, we derive the linear convergence rate in Appendices A.1.6–
A.1.7 by using the PL inequality and the properties of induced norms.
Before getting into the details of the proof, we now brieﬂy discuss the property of our proof in terms of the tightness of a bound.
In the condition of (cid:107) B t (cid:107) 1 < (1 − γ ) R in the statement of Theorem 1, the quantity (1 − γ ) comes from the proof in Appendix A.1.7: i.e., it is the reciprocal of the quantity 11 − γ in the upper bound of (cid:107) ( I m − γσ ( A ))
− 1 (cid:107) 1 ≤ 11 − γ .
Therefore, a natural question is whether or not we can improve this bound further.
This bound turns out to be tight based on the following lower bound.
The matrix I m − γσ ( A ) is a Z -matrix since off-diagonal entries are less than or equal to zero.
Furthermore, I m − γσ ( A ) is M -matrix since eigenvalues of I m − γσ ( A ) are the eigenvalues of I m − γσ ( A ) (cid:62) and the eigenvalues of I − γσ ( A ) (cid:62) are lower bounded by 1 − γ > 0 .
This is because σ ( A ) (cid:62) is a stochastic matrix with the largest eigenvalue being one.
Moreover, in the proof in Appendix A.1.7, we showed that | I − γσ ( A ) | jj − (cid:80) i (cid:54) =
j | I − γσ ( A ) | ij = 1 − γ for all j .
Therefore, using the lower bound by Moraˇca (2008), we have
[Equation]
which matches with the upper bound of (cid:107) ( I m − γσ ( A )) − 1 (cid:107) 1 ≤ 11 − γ .
Therefore, we cannot further improve the our bound on (cid:107) B t (cid:107) 1 in general without making some additional assumption.
A.1. R EARRANGING THE FORMULA OF ∇ A L
We will use the following facts for matrix calculus (that can be derived by using deﬁnition of derivatives: e.g., see Barnes, 2006): [Equation] ( a
[Equation]
Recall that U
=
I − γσ ( A ) .
From the above facts, given a function g , we have
[Equation]
Using the quotient rule,
[Equation]
[Equation] [Equation]
[Equation]
Thus, ∂g ( U ) ∂A
kl = − γ m (cid:88) i = ∂g ( U ) ∂U il ∂σ ( A ) il ∂A kl = − γ (cid:18) ∂g ( U ) ∂U ∗ l (cid:19) (cid:62) ∂σ ( A ) ∗ l ∂A kl ∈ R , where ∂g ( U ) ∂U ∗ l ∈ R m × 1 and ∂σ ( A ) ∗ l ∂A kl ∈ R m × 1 .
This yields
[Equation]
where ∂σ ( A ) ∗ l ∂A ∗ l ∈ R m × m .
Now we want to set g ( U ) to be the output of deep equilibrium linear models as g ( U ) =
B q ∗ (cid:0)
lim l →∞ z ( l ) ( x, A ) (cid:1) for each q ∈ { 1 , . . .
, m y } .
To do this, we ﬁrst simplify the formula of the output B q ∗ (cid:0)
lim l →∞ z ( l ) ( x, A ) (cid:1) using the following:
[Equation]
Therefore,
[Equation]
where the ﬁrst line, the second line and the last line used the fact that γσ ( A ) ij ≥ 0 ,
[Equation]
1 , and hence (cid:107) γσ ( A ) (cid:107) 1 < 1 for γ ∈ (0 , 1) .
This shows that B (cid:0)
lim
l →∞ z ( l ) ( x, A ) (cid:1) =
BU − 1 φ ( x ) , where the inverse U − 1 exists as the corresponding Neumann series converges {+(cid:80) ∞ k =
+}
γ k σ ( A ) k since (cid:107) γσ ( A ) (cid:107) 1 < 1 .
Therefore, we can now set g ( U ) =
B q ∗ (cid:0)
lim l →∞ z ( l ) ( x, A ) (cid:1) =
B q ∗ U − 1 φ ( x ) .
Then, using ∂a (cid:62) M − 1 b ∂M
[Equation]
which implies that
[Equation]
Combining (18) and (20),
[Equation] [Equation]
where we used ( U −(cid:62) )
∗ l
= (( U − 1 ) (cid:62) )
[Equation] = (( U − 1 ) l ∗ ) (cid:62) and (( U −(cid:62) ) ∗ l ) (cid:62) = ((( U − 1 ) l ∗ ) (cid:62) ) (cid:62) =
( U − 1 ) l ∗ .
By taking transpose,
[Equation]
By rearranging this to the matrix form,
[Equation]
[Equation] where ∂B [Equation]
∈ R m × m .
Each entry of this matrix represents the derivatives of the model output with respect to the parameters A .
We now use this to rearrange ∇ A L
( A, B ) :
= ∂L ( A,B ) ∂A .
We set ˆ y iq =
[Equation]
∗ U − 1 φ ( x ) and ˆ y i = BU − 1 φ ( x ) and deﬁne J k : [Equation] ∂σ ( A )
[Equation]
∂A
[Equation] ∈ R [Equation] m , and
Q := n
[Equation]
=
[Equation]
Then, using the chain rule and the above formula of ∂B q ∗ U − 1 φ ( x ) ∂A , ∂L ( A, B ) [Equation] = [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
Summarizing the above
[Equation]
we have that
[Equation]
[Equation]
where ∇ A L ( A, B ) ∈
R m × m .
A.1. R EARRANGING THE FORMULA OF ∇ L
In order to relate L 0 to the gradient dynamics of L
[Equation]
we now rearrange the formula of ∇ L 0 .
We set ˆ y iq =
W q ∗ φ ( x i ) ∈
R and ˆ y i
= Wφ ( x i ) ∈
R m y for linear models.
Then, by the chain rule,
[Equation]
Since
[Equation]
we have
[Equation]
By rearranging this into the matrix form
[Equation]
(ˆ y i ,y i ) ∂
ˆ y
i
[Equation]
∂(cid:96) (ˆ y i
[Equation]
y i
[Equation]
∂
[Equation]
(cid:62)
[Equation]
R m
[Equation]
m where ∂(cid:96) (ˆ y i ,y i ) ∂
ˆ y i ∈ R 1 × m y .
Thus,
[Equation]
A.1. C OMBINING THE FORMULA OF ∇ A L AND ∇ L
Combining (22) and (23) by resolving the different deﬁnitions of ˆ y i yields that
[Equation]
[Equation] Here, if there is no additional nonlinearity σ , the matrices J k = ∂σ ( A ) ∗ k
∂A ∗ k become identity for all k .
In that case, ∇ A L ( A, B ) can be further simpliﬁed and factorize over m , which is desired for the analysis of gradient dynamics.
However, due to the additional nonlinearity, we cannot factorize ∇ A L ( A, B ) over m .
One of the key techniques in our analysis is to keep this un-factorized ∇ A L ( A, B ) and ﬁnd a way to factorize it during the update of parameters ( A t , B t ) in the gradient dynamics, as shown later in this proof.
To do so, we now start considering the dynamics over time t .
A.1. A NALYSING (cid:0) lim l →∞ z ( l ) ( x, A t ) (cid:1)
Now let us temporarily consider a gradient dynamics discretized by the Euler method as
[Equation]
with some step size α > 0 .
Then,
[Equation]
where we used (cid:0)
lim l →∞ z
[Equation]
l ) ( x, A t ) (cid:1) = U − 1 t φ ( x ) from Section A.1.1.
By setting ϕ ij ( α ) = σ ( A −
[Equation]
By using the chain rule and setting M =
A − α ∇ A L ( [Equation] [Equation] × n
[Equation]
Therefore,
[Equation]
Recalling the deﬁnition of J k :=
∂σ ( A ) ∗ k ∂A ∗ k ∈ R m × m ,
[Equation]
∗ j
[Equation]
Rearranging it into the matrix form,
[Equation]
Putting the above equations together,
[Equation]
Thus, [ I
[Equation]
By setting M =
[ J 1 ∇ A L ( A, B ) ∗· · ·
J m ∇ A L ( A, B ) ∗ m ] and ϕ ( α ) =
[ U + αγM
+ o ( α 2 )] −and by using ∂ ¯ M − 1 ∂a = − ¯ M − 1 ∂
¯ M∂a ¯ M − 1 , [ I − γσ ( A − α ∇ A L ( A, B ))] − 1 =
[ U + αγM
+ O ( α 2 )] −
[Equation]
Summarizing above,
[Equation]
P UTTING RESULTS TOGETHER FOR INDUCED DYNAMICS
We now consider the dynamics of Z t :=
B t U − 1 t in R m y × m that is induced by the gradient dynamics of ( A t , B t ) :
[Equation]
Continuing the previous subsection
[Equation]
we ﬁrst consider the dynamics discretized by the Euler method:
[Equation]
where α > 0 .
Then, substituting (25) into the right-hand side of this equation,
Z t +
[Equation]
] U − 1 t
+ O ( α 2 ) .
Using B t +1 =
[ B t − α ∇ B L ( A t , B t )] , we have αγB t +1 U − 1 t
[ J 1 ,t ∇ A L ( A t , B t ) ∗· · · J m,t ∇ A L ( A t , B t ) ∗ m ] U − 1 t = αγB t U − 1 t [ J 1 ,t ∇ A L ( A t , B t ) ∗· · · [Equation] m,t
[Equation]
A
[Equation]
( A t ,
[Equation]
t )
[Equation] m ] U [Equation] 1 t + O (
[Equation]
) = αγZ [Equation] [Equation] [Equation]
[Equation] [Equation]
) ∗· [Equation] J m, [Equation]
[Equation] [Equation] m ] U [Equation]
t + O ( α 2 ) . Since ( U −(cid:62) ) ∗ k = (( U − 1 ) (cid:62) ) ∗ k = (( U − 1 ) k ∗ ) (cid:62) , U − 1 =   ( U − 1 ) 1 ∗ ...
( U − 1 ) m ∗ 
 and ∇ A L ( A, B ) ∗ k = (cid:16) ∂L ( A,B ) ∂A (cid:17) ∗ k = γJ
(cid:62) k ( BU − 1 ) (cid:62) ∇ L 0 ( BU − 1 )( U −(cid:62) ) ∗ k from (24), we have that
Z t [ J 1 ,t ∇ A L ( A t , B t ) ∗· · ·
J m,t ∇ A L ( A t , B t ) ∗ m ] U − 1 t = γZ t (cid:2) J 1 ,t J (cid:62) 1 ,t Z (cid:62) t ∇ L 0 ( Z t )( U −(cid:62) t ) ∗· · ·
J m,t J (cid:62) m,t Z (cid:62) t ∇ L 0 ( Z t )( U −(cid:62) t ) ∗
[Equation]
 
[Equation]
[Equation]
U
[Equation]

 =
[Equation]
On the other hand, using B t + =
[ B t − α ∇ B L ( A t , B t )] and ∇ B L ( A, B ) :
= ∂L ( A,B ) ∂B = (cid:18)(cid:80) ni =1 (cid:16) ∂(cid:96) (ˆ y i ,y i ) ∂
ˆ y i (cid:17) (cid:62) φ ( x i ) (cid:62) (cid:19)
U −(cid:62)
= ∇ L 0 ( BU − 1 ) U −(cid:62) , we have
[Equation]
U
[Equation]
Summarizing these equations by noticing
[Equation]
yields that
[Equation]
By vectorizing both sides,
[Equation]
By deﬁning
[Equation]
we have
[Equation]
This implies that
[Equation]
where α > 0 .
By recalling the deﬁnition of the Euler method and deﬁning Z ( t ) =
Z t , we can rewrite this as
[Equation]
By taking the limit for α → 0 and going back to continuous-time dynamics
[Equation]
this implies that
[Equation]
Here, we note that the complex interaction over m due to the nonlinearity is factorized out into the matrix D t .
Furthermore, the interaction within the matrix D t has more structures when compared with that in the gradients themselves from (24).
For example, unlike the gradients, the interaction over m even within D t can be factorized out in the case of m y = 1 as:
[Equation]
Although we do not assume m y = [Equation] this illustrates the additional structure well.
A.1. A NALYSIS OF THE MATRIX D t
From the deﬁnition of D t , we have that
[Equation]
[Equation]
Since U −(cid:62) U − 1 is positive deﬁnite
[Equation]
I m y is positive deﬁnite
[Equation]
and a Kronecker product of two positive deﬁnite matrices is positive deﬁnite (since the eigenvalues of Kronecker product are the products of eigenvalues of the two matrices), we have
[Equation]
Since ( U −(cid:62) ) ∗ k
( U − 1 ) k ∗ is positive semideﬁnite, γ 2 Z t J k,t J (cid:62) k,t Z (cid:62) t is positive semideﬁnite, and
a
Kronecker product of two positive semideﬁnite matrices is positive semideﬁnite (since the eigenvalues of Kronecker product are the products of eigenvalues of the two matrices), we have
[Equation]
Since a sum of positive semideﬁnite matrices is positive semideﬁnite (from the deﬁnition of positive semi-deﬁniteness:
[Equation]
Since a sum of a positive deﬁnite matrix and positive semideﬁnite matrix is positive deﬁnite (from the deﬁnition of positive deﬁniteness and positive deﬁniteness: ( x (cid:62) M 1 x > 0 ∧ x (cid:62) M 2 x ) ⇒ x (cid:62) ( M 1 + M 2 ) [Equation] M 2 x > 0 ),
[Equation]
Therefore, D t is a positive deﬁnite matrix for any t and hence
[Equation]
A.1. C ONVERGENCE RATE VIA P OLYAK -Ł OJASIEWICZ INEQUALITY AND NORM BOUNDS
Let R ∈
(0 , ∞ ] and T > 0 be arbitrary.
By taking derivative of L 0 ( Z t ) − L ∗ 0 ,R with respect to time t with Z t :=
B t U − 1 t ,
[Equation]
where we used the chain rule and the fact that ddt L ∗ 0 ,R = 0 .
By using the vectorization notation with [Equation] L
[Equation]
By using (26) for the equation of vec (cid:2) ddt ( Z t ) (cid:3) ,
[Equation]
Using the condition that ∇ L 0 satisﬁes the the Polyak-Łojasiewicz inequality with radius R , if (cid:107) Z t (cid:107) 1 < R , then we have that for all t ∈ [0 , T ] ,
[Equation]
By solving the differential equation, this implies that if (cid:107) Z t (cid:107) 1 < R ,
[Equation]
Since L ( A t , B t ) =
L 0 ( Z t ) , if (cid:107) Z t (cid:107) 1 < R , L ( A
[Equation]
We now complete the proof of the ﬁrst part of the desired statement by showing that (cid:107) B (cid:107) 1 < (1 − γ ) R implies (cid:107) Z t (cid:107) 1 < R .
With Z = BU − 1 , since any induced operator norm is a submultiplicative matrix norm,
[Equation]
We can then rewrite
[Equation]
Here, the matrix I m − γσ ( A ) (cid:62) is strictly diagonally dominant: i.e., | I m − γσ ( A ) (cid:62) | ii > (cid:80) j (cid:54)
=
i | I m − γσ ( A ) | ij for any i .
This can be shown as follows: for any j ,
[Equation]
This calculation also shows that | I m − γσ ( A ) | jj − (cid:80) i (cid:54) =
j | I m − γσ ( A ) | ij = 1 − γ for all j .
Thus, using the Ahlberg–Nilson–Varah bound for the strictly diagonally dominant matrix (Ahlberg & Nilson, 1963; Varah, 1975; Moraˇca, 2008), we have
[Equation]
By taking transpose,
[Equation]
Summarizing above,
[Equation]
Therefore, if (cid:107) B (cid:107) 1 < R (1 − γ ) , then (cid:107) Z (cid:107) 1 = (cid:107) B ( I m − γσ ( A )) − 1 (cid:107) 1 ≤ (cid:107) B (cid:107) 1 11 − γ < R, as desired.
Combining this with (31) implies that if (cid:107) B (cid:107) 1 < R (1 − γ ) ,
[Equation]
Recall that L ∗ 0 ,R = inf W : (cid:107) W (cid:107) 1 <R L 0 ( W ) and L ∗ R = inf A ∈ R m × m ,B ∈B R L ( A, B ) where B R = { B ∈
R m y × m | (cid:107) B (cid:107) 1 < (1 − γ ) R } .
Here, B ∈ B R implies that (cid:107) Z {+(cid:107) 
+} = (cid:107) BU − 1 (cid:107) 1 ≤ (cid:107) B (cid:107) 1 (cid:107) U − 1 {+(cid:107) 
 <+}
(1 − γ ) R (cid:107) U − 1 (cid:107) 1 ≤ R , using the above upper bond (cid:107) U − 1 {+(cid:107) 
+}
= (cid:107) ( I m − γσ ( A ))
− 1 (cid:107) 1 ≤ 11 − γ .
Since
L ( A, B ) =
L 0 ( Z ) with Z = BU − 1 , this implies that L ∗ 0 ,R ≤ L ∗ R and thus
[Equation]
This completes the ﬁrst part of the desired statement of Theorem
[Equation] The remaining task is to lower bound λ T , which is completed as follows: for any ( A, B ) ,
[Equation]
where the third line follows from (27)–(29), the ﬁfth line follows from the property of Kronecker product (the eigenvalues of Kronecker product of two matrices are the products of eigenvalues of the two matrices) [Equation] and the last inequality follows from the relation between the spectral norm and the norm (cid:107) · (cid:107) 1 .
We now compute (cid:107) U (cid:107) 1 {+as: for any ( A, B ) , (cid:107) U (cid:107) 
 = (cid:107) I m − γσ ( A ) (cid:107)= max j (cid:88) i | ( I m − γσ ( A )) ij | = max j (cid:88) i | ( I m ) ij − γσ ( A ) ij | = max j | ( I m )+}
jj − γσ ( A ) jj | + (cid:88) i (cid:54) =
j | ( I m ) ij − γσ ( A ) ij | = max j | 1 − γσ ( A ) jj | + (cid:88) i (cid:54) = j | − γσ ( A ) ij | = max j 1 − γσ ( A )
jj + (cid:88) i (cid:54) =
j γσ ( A ) ij = max j 1 + γ    (cid:88)
i (cid:54) =
j σ ( A ) ij 
 − σ ( A ) jj 
 ≤ 1 + γ.
By substituting this into (32), we have that for any ( A, B ) (and hence for any t ),
[Equation]
This completes the proof for both the ﬁrst and second parts of the statement of Theorem 1.
A. P ROOF OF T HEOREM
We ﬁrst show that with δ t > 0 sufﬁciently small, we have G t (cid:31) 0 .
Recall that
[Equation]
Thus, with δ t > 0 sufﬁciently small, for any v (cid:54) = 0 ,
[Equation]
which is dominated by the ﬁrst term v (cid:62) U t S − 1 t U (cid:62) t v if the matrix U t S − 1 t U (cid:62) t is positive deﬁnite.
Since S t := I m + γ 2 diag( v St ) with v St ∈ R m and ( v St ) k := (cid:107) J (cid:62) k,t ( B t U − 1 t ) (cid:62) (cid:107) 2 2 for k = 1 , 2 , . .
, m , the matrix U t S − 1 t U (cid:62) t is positive deﬁnite.
Thus, with δ t > 0 sufﬁciently small, v (cid:62) Gv is dominated by the ﬁrst term, which is positive (since U t S − 1 t U (cid:62) t is positive deﬁnite), and thus we have G t (cid:31) 0 .
Then we observe that the output of argmin v : (cid:107) v (cid:107) Gt [Equation] vec [Equation]
L [Equation] 0 ( v ) is the set of solutions of the following constrained optimization problem:
[Equation]
Since this optimization problem is convex, one of the sufﬁcient conditions for global optimality is the KKT condition with a multiplier µ ∈ R :
[Equation]
Therefore, the desired statement is obtained if the above KKT condition is satisﬁed by v = δ t (cid:0) ddt vec( B t U − 1 t ) (cid:1) with some multiplier µ . The rest of this proof shows that the KKT condition is satisﬁed by setting v = δ t (cid:0) ddt vec( B t U − 1 t ) (cid:1) and µ = 12 δ t . With this choice, the last two conditions of the KKT condition hold, since
[Equation]
, and
[Equation]
t (cid:13)(cid:13)(cid:13)(cid:13) d dt
[Equation]
B
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
t
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
d dt
[Equation]
( B t U − 1 t ) (cid:13)(cid:13)(cid:13)(cid:13)
G t = 0 .
The remaining task is to show that ∇ L t 0 ( v ) + 2 µG t v = 0 with v = δ t (cid:0) ddt vec( B t U − 1 t ) (cid:1) and µ = 12 δ t .
From the deﬁnition of L t 0 ,
[Equation]
[Equation]
We now compute and ∇ L vec0 and ∇ 2 L vec0 .
Since ∇ L 0 ( W ) :
= ∂L 0 ( W ) ∂W
= (cid:80) ni
=1 (cid:16) ∂(cid:96) (ˆ y i ,y i ) ∂
ˆ y i (cid:17) (cid:62) x (cid:62) i , vec( ∇ L 0 ( W ))
=
n (cid:88) i = vec (cid:32) I m y (cid:18) ∂(cid:96)
(ˆ y i , y i ) ∂ ˆ y i (cid:19) (cid:62) φ ( x i ) (cid:62) (cid:33) =
n (cid:88)
i = [ φ ( x i ) ⊗ I
m y ] (cid:18) ∂(cid:96)
(ˆ y i , y i )
∂ ˆ y i (cid:19) (cid:62) , where
[Equation]
Therefore,
[Equation]
For the Hessian,
[Equation]
i
[Equation]
[Equation] [Equation] [Equation]
x
[Equation]
m y ]
By deﬁning (cid:96) i ( z ) =
(cid:96) ( z, y i ) and ∇ 2 (cid:96) i ( z )
[Equation]
∂∂z (cid:16) ∂(cid:96) i (
[Equation]
) ∂z (cid:17) (cid:62) ,
[Equation] [Equation]
Since we have that
[Equation]
A )
[Equation]
+
[Equation]
, we can write:
[Equation]
where we used the fact that γσ ( A ) ij ≥ 0 , (cid:107) σ ( A ) (cid:107) 1 = max j (cid:80) i | σ ( A ) ij | = 1 , and thus (cid:107) γσ ( A ) (cid:107) 1 < 1 for any γ ∈ (0 , 1) .
This shows that (cid:0) lim
l →∞ z ( l ) ( x, A ) (cid:1) = z ∗ ( x, A ) =
U − 1 φ ( x ) , from which we have φ ( x i ) =
Uz ∗ ( x i , A ) .
Substituting φ ( x i )
=
Uz ∗ ( x i , A ) into (35),
[Equation]
n (cid:88) i =
[ U ⊗ I m y ]
[ z ∗ ( x i , A ) ⊗ I m y ] ∇ 2 (cid:96) i
( Wφ ( x i ))
[ z ∗ ( x i , A ) (cid:62) ⊗ I
m y ]
[ U (cid:62) ⊗ I m y ] .
= [ U ⊗ I m y ] (cid:32) n (cid:88) i =
[ z ∗ ( x i , A ) ⊗ I m y ] ∇ 2 (cid:96) i
( Wφ ( x i ))
[ z ∗ ( x i , A ) (cid:62) ⊗ I m y ] (cid:33)
[ U (cid:62) ⊗ I m y ] .
In the case of m y = 1 , since I m y = 1 , ∇ 2 L vec0 (vec( W )) is further simpliﬁed to:
[Equation]
Therefore,
[Equation]
where
[Equation]
By plugging µ = 12 δ t and ∇ 2 L vec0 (vec( B t U − 1 t ))
=
U t F t U (cid:62) t into (34),
[Equation]
By using v = δ t (cid:0) ddt vec( B t U − 1 t ) (cid:1) ,
[Equation]
By plugging (26) into ddt vec( B t U − 1 t ) with Z t =
B t U [Equation]
[Equation]
Recall that
[Equation]
In the case of m y =
[Equation]
the matrix D t can be simpliﬁed as:
[Equation]
Plugging this into the above equation for D t ,
[Equation]
Therefore
[Equation]
the constrained optimization problem at time t is solved by
[Equation]
which implies that
[Equation]
By multiplying φ ( x ) (cid:62) ⊗ I m y to each side of the equation, we have
[Equation]
yielding that
[Equation]
This proves the desired statement of Theorem 2.
A. P ROOF OF C OROLLARY
The assumption rank(Φ) =
min( n, m ) implies that σ min (Φ) > 0 .
Moreover, the square loss (cid:96) satisﬁes the assumption of the differentiability.
Thus, Theorem 1 implies the statement of this corollary if L 0 with the square loss satisﬁes the Polyak-Łojasiewicz inequality for any W ∈ R m y × m with parameter κ = 2 σ min (Φ) 2 .
This is to be shown in the rest of this proof.
By setting ϕ =
Lin Deﬁnition 1, we have (cid:107)∇ ϕ vec (vec( q ))
(cid:107)
=
(cid:107)∇ L 0 ( W ) (cid:107) 2 F .
With the square loss, we can write
L 0 ( W ) =
(cid:80) ni =1 (cid:107) Wφ ( x i ) − y
{+i (cid:107) 
+}
= (cid:107) W Φ − Y (cid:107) 2 F where Φ ∈ R m × n and Y ∈ R m y × n with Φ ji = φ ( x i ) j and Y ji
= ( y i ) j .
Thus,
[Equation]
We ﬁrst consider the case of m ≤ n .
In this case, we consider the vectorization L vec0 (vec( W )) =
L 0 ( W ) and derive the gradient with respect to vec( W ) :
[Equation]
Then, the Hessian can be easily computed as
[Equation]
where I m y is the identity matrix of size m y by m y .
Since the singular values of Kronecker product of the two matrices is the product of singular values of each matrix, we have
[Equation]
where we used the fact that m ≤ n in this case.
Since W is arbitrary, this implies that L vec0 is strongly convex with parameter 2 σ min (Φ) 2 > 0 in R m y × m .
Since a strongly convex function with some parameter satisﬁes the Polyak-Łojasiewicz inequality with the same parameter (Karimi et al., 2016), this implies that L vec0 (and hence L 0 ) satisﬁes the Polyak-Łojasiewicz inequality with parameter 2 σ min (Φ) 2 > 0 in R m y × m in the case of m ≤ n .
We now consider the remaining case of m > n .
In this case, using the singular value decomposition of Φ = U Σ V (cid:62) ,
[Equation]
for any L ∗∗ 0 ≥ 0 , where the ﬁrst line uses (cid:107) q (cid:107) 2 F = (cid:107) q (cid:62) (cid:107) 2 F , the second line uses the singular value decomposition, and the third and fourth line uses the fact that U and V are orthonormal matrices.
The forth line uses the fact that m > n in this case.
Therefore, since W is arbitrary, we have shown that L 0 satisﬁes the Polyak-Łojasiewicz inequality for any W ∈ R m y × m with parameter κ = 2 σ min (Φ) 2 in both cases of m > n and m ≤ n .
A. P ROOF OF C OROLLARY
The assumption rank(Φ)
= m implies that σ min (Φ) > 0 .
Moreover, the logistic loss (cid:96) satisﬁes the assumption of the differentiability.
Thus, Theorem 1 implies the statement of this corollary if Lwith the logistic loss satisﬁes the Polyak-Łojasiewicz inequality with the given radius R ∈ (0 , ∞ ] and the parameter κ = (2 τ + ρ ( R ))
σ min (Φ) 2 where ρ ( R ) depends on R .
Let R ∈
(0 , ∞ ] be given.
Note that we have ρ ( R ) >
if R < ∞ , and ρ ( R ) =
if R = ∞ .
If ( R, τ ) = ( ∞ , 0) , then 2 τ + ρ ( R ) = 0 for which the statement of this corollary trivially holds (since the bound does not decrease).
Therefore, we focus on the remaining case of ( R, τ ) (cid:54) = ( ∞ , 0) .
Since ( R, τ ) (cid:54) = ( ∞ , 0) , we have 2 τ + ρ ( R ) > 0 .
We ﬁrst compute the Hessian with respect to W as:
[Equation]
where ˆ p i = 1 1+ e − Wφ
( xi ) .
Therefore,
[Equation]
i = v (cid:62) φ ( x i ) φ ( x i ) (cid:62) v + 2 τv (cid:62) (cid:32) n (cid:88)
i = φ ( x i ) φ ( x i ) (cid:62) (cid:33) v = (2 τ + ρ ( R )) v (cid:62) ΦΦ (cid:62) v ≥ (2 τ + ρ ( R ))
σ min (Φ) 2 > 0 .
Therefore, L 0 is strongly convex with parameter (2 τ + ρ ( R ))
σ min (Φ) 2 > 0 .
Since a strongly convex function with a parameter satisﬁes the Polyak-Łojasiewicz inequality with the same parameter (Karimi et al., 2016), this implies that L 0 satisﬁes the Polyak-Łojasiewicz inequality with the given radius R ∈ (0 , ∞ ] and parameter (2 τ + ρ ( R ))
σ min (Φ) 2 > 0 .
Since R ∈ (0 , ∞ ] is arbitrary, this implies the statement of this corollary by Theorem 1.
A. P ROOF OF P ROPOSITION
Let ( x, A ) be given.
By repeatedly applying the deﬁnition of z ( l ) ( x, A ) , we obtain
[Equation]
(36) where σ ( A ) k represents the matrix multiplications of k copies of the matrix σ ( A ) with σ ( A )
= I m .
In general, if σ is identity, this sequence does not converge.
However, with our deﬁnition of σ , we have
[Equation]
Therefore, (cid:107) γσ ( A ) (cid:107) 1 = γ (cid:107) σ ( A ) (cid:107) 1 = γ < 1 for any γ ∈ (0 , 1) .
Since an induced matrix norm is sub-multiplicative, this implies that
[Equation]
In other words, each term (cid:107) γ k σ ( A ) k (cid:107) 1 in the series (cid:80) ∞ k =0 (cid:107) γ k σ ( A ) k (cid:107) 1 is bounded by γ k .
Since the series (cid:80) ∞ k =0 γ k converges in R with γ ∈ (0 , 1) , this implies, by the comparison test, that the series (cid:80) ∞ k =0 (cid:107) γ k σ ( A ) k (cid:107) 1 converges in R .
Thus, the sequence ( (cid:80) lk =0 (cid:107) γ k σ ( A ) k (cid:107) 1 ) l is a Cauchy sequence.
By deﬁning s l = (cid:80) lk =
γ k σ ( A ) k , we have (cid:107) s l − s l [-0 k 
-](cid:48)
(cid:107) 1 = (cid:107) (cid:80) lk
= l (cid:48) +1 γ k σ ( A ) k (cid:107) 1 ≤
(cid:80) lk
= l (cid:48) +1 (cid:107) γ k σ ( A ) k (cid:107) 1 for any l (cid:48) > l by the triangle inequality of the (matrix) norm.
Since ( (cid:80) lk [-=
 k-]=0 (cid:107) γ k σ ( A ) k (cid:107) 1 ) l is a Cauchy sequence, this inequality implies that ( (cid:80) lk =
γ k σ ( A ) k ) l is a
Cauchy sequence (in a Banach space ( R m × m , (cid:107)·(cid:107) 1 ) , which is isometric to R mm under (cid:107)·(cid:107) 1 ).
Thus, the sequence ( (cid:80) lk [-=
-]=0 γ k σ ( A ) k ) l converges.
From (36), this implies that the sequence ( z ( l ) ( x, A )) l converges.
B O N THE I MPLICIT B IAS
In this section, we show that Theorem 2 suggests an implicit bias towards a simple function as a result of inﬁnite depth, whereas understanding this bias in more details is left as an open problem.
This section focuses on the case of the square loss (cid:96) ( q, y i ) = (cid:107) q − y
i
(cid:107) 22 with m y = 1 .
By solving vec( V t ) ∈ argmin v ∈V L t 0 ( v ) in Theorem 2 for the direction of the Newton method, Theoremimplies that
[Equation]
(37) where r t ∈ R m is an error vector with each entry being a function of the residuals f θ t ( x i ) − y i .
Since σ ( A t ) is a positive matrix and is a left stochastic matrix due to the nonlinearity σ , the Perron–
Frobenius theorem (Perron, 1907; Frobenius, 1912) ensures that the largest eigenvalue of σ ( A t ) is one, any other eigenvalue in absolute value is strictly smaller than one, and any left eigenvector corresponding the largest eigenvalue is the vector η
= η
, 1] (cid:62) ∈ R m where ζ ∈ R is some scalar.
Thus, the largest eigenvalue of the matrix ( I m − γσ ( A t )) − 1 is 11 − γ , any other eigenvalue is in the form of 1 1 − λ k γ with | λ k | < 1 , and any left eigenvector corresponding the largest eigenvalue is η 1 ∈ R m .
By decomposing the error vector as r t = P 1 r t
+ (1 − P 1 ) r t , this implies that vec( V t ) =
V (cid:62) t = 11 − γ P 1 r t + g γ
(1 − P 1 ) r t ) ∈ R m , where P 1 = 1 m 11 (cid:62) is a projection onto the column space of 1 = [1 , 1 , . . . , 1] (cid:62) ∈ R m , and g γ is a function such that for any q in its domain, (cid:107) g γ ( q ) (cid:107) < c for all γ ∈ (0 , 1) with some constant c in γ .
In other words, vec( V t ) in Theorem 2 can be decomposed into the two terms: 11 − γ P 1 r t (the projection of the error vector onto the column space of 1 ) and g γ ((1 − P 1 ) r t ) (a function of the projection of the error vector onto the null space space of 1 ).
Here, as γ → 1 , (cid:107) 11 − γ P 1 r t (cid:107) → ∞ and (cid:107) g γ ((1 − P 1 ) r t ) (cid:107) < c .
This implies that with γ < 1 sufﬁciently large, the ﬁrst term 11 − γ P 1 r t dominates the second term g γ ((1 − P 1 ) r t ) .
Since ( P 1 r t ) (cid:62) φ ( x ) = ¯ µ t m 1 (cid:62) φ ( x ) with ¯ µ t = (cid:80) mk =1 ( r t ) k ∈ R , this implies that with γ < 1 sufﬁciently large, the dynamics of deep equilibrium linear models ddt f θ t = 1 δ t V t φ learns a simple shallow function ˆ µ T m 1 (cid:62) φ ( x ) ﬁrst before learning more complicated components of the functions through g γ ((1 − P 1 ) r t ) , where ˆ µ T = (cid:82) T 0 ¯ µ t dt ∈ R .
[-T 
 ˉ μ t dt ∈ R .-] Here, ˆ µ T m 1 (cid:62) φ ( x ) is a simple average model that averages over the features 1 m 1 (cid:62) φ ( x ) and multiplies it by a scaler ˆ µ t .
Moreover, large γ <means that we have large effective depth or large weighting for deeper layers since we have a shallow model with γ = 0 and γ is a discount factor of the inﬁnite depth.
C A DDITIONAL D ISCUSSION
On existence of the limit.
When Bai et al. (2019a) introduced general deep equilibrium models, they hypothesized that the limit lim l →∞ z ( l ) exists for several choices of h , and provided numerical results to support this hypothesis.
In general deep equilibrium models, depending on the values of model parameters, the limit is not ensured to exists.
For example, when h increases the norm of the output at every layer, then it is easy to see that the sequence diverge or explode.
This is also true when we set h to be that of deep equilibrium models without the nonlinearity σ (or equivalently redeﬁning σ to be the identity function): if the operator norms on A are not bounded by one, then the sequence can diverge in general.
In other words, in general, some trajectory of gradient dynamics may potentially violate the assumption of the existence of the limit when learning models.
In this view, the class of deep equilibrium linear models is one instance of general deep equilibrium models where the limit is guaranteed to exist for any values of model parameters as stated in Proposition 1.
On the PL inequality.
With R sufﬁciently large, the deﬁnition of the PL inequality in this paper is simply a rearrangement in the form where L 0 is allowed to take matrices as its inputs through
L vec0 (vec( · ))
=
L 0 ( · ) , where vec( M ) represents the standard vectorization of a matrix M .
See (Polyak, 1963; Karimi et al., 2016) for more detailed explanations of the PL inequality.
On the reditus R for the logistic loss.
As shown in Section 3.1.3, we can use R = ∞ for the square loss and the logistic loss, in order to get a prior guarantee for the global linear convergence in theory.
In practice, for the logistic loss, we may want to choose R depending on the different scenarios, because of the following observation.
For the logistic loss, we would like to set the radius
R to be large so that the trajectory on B is bounded as (cid:107) B t (cid:107) 1 < (1 − γ ) R for all t ∈ [0 , T ] and the global minimum value on the constrained domain to decrease: i.e., L ∗ R → L ∗ as R → ∞ .
However, unlike in the case of the squared loss, the convergence rate decreases as we increase R in the case of the logistic loss, because ρ ( R ) decreases as R increases.
This does not pose an issue because we can always pick R < ∞ so that for any t > 0 and T > 0 , we have ρ ( R ) >
c ρ for some constant c ρ > 0 .
Moreover, this tradeoff does not appear for the square loss: i.e., we can set R = ∞ for the square loss without decreasing the convergence rate.
We can also avoid this tradeoff for the logistic loss by simply setting R = ∞ and τ > 0 .
On previous work without implicit linearization.
In Section 5, we discussed the previous work on deep neural networks with implicit linearization via signiﬁcant over-parameterization.
Kawaguchi & Huang (2019) observed that we can also use the implicit linearization with mild over-parameterization by controlling learning rates to guarantee global convergence and generalization performances at the same time.
On the other hand, there is another line of previous work where deep nonlinear neural networks are studied without any (implicit or explicit) linearization and without any strong assumptions; e.g., see the previous work by Shamir (2018)
; Liang et al.
Nguyen (2019); Kawaguchi & Bengio (2019); Kawaguchi & Kaelbling (2020); Nguyen (2021).
Whereas the conclusions of these previous studies without strong assumptions can be directly applicable to practical settings, their conclusions are not as strong as those of previous studies with strong assumptions (e.g., implicit linearization via signiﬁcant over-parameterization) as expected.
The direct practical applicability, however, comes with the beneﬁt of being able to assist the progress of practical methods (Verma et al., 2019; Jagtap et al., 2020b;a).
D E XPERIMENTS
The purpose of our experiments is to provide a secondary motivation for our theoretical analyses, instead of claiming the immediate beneﬁts of using deep equilibrium linear models.
D. E XPERIMENTAL S ETUP
For data generation and all models, we set φ ( x ) = x .
Therefore, we have m = m x .
Data.
To generate datasets, we ﬁrst drew uniformly at random 200 input images from a standard image dataset — CIFAR-10 (Krizhevsky & Hinton, 2009), CIFAR-100 (Krizhevsky & Hinton, 2009) or Kuzushiji-MNIST (Clanuwat et al., 2019) — as pre input data points x pre i ∈ R m (cid:48) x .
Out of 200 images, 100 images to be used for training were drawn from a train dataset and theother images to be used for testing were drawn from the corresponding test dataset.
Then, the input data pints x i ∈ R m x with m x = 150 were generated as x i
=
R x pre i where each entry of a matrix
R ∈ R m x × m (cid:48) x was set to δ/ √ m x with δ i.i.d.
∼ N (0 , 1) and was ﬁxed over the indices i .
We then generated the targets as y i =
B ∗ (lim l →∞ z ( l ) ( x i , A ∗ ))
+ δ i
∈
R with γ = 0 .
8 where δ i i.i.d.
∼ N (0 , 1) .
Each entry of the true (unknown) matrices A ∗ ∈ R 1 × m and B ∗ ∈ R m × m was independently drawn from the standard normal distribution.
Model.
For DNNs, we used ReLU activation and W ( l ) ∈
R m × m for l = 1 , 2 . . .
, H −( W ( H ) ∈ R 1 × m ).
Each entry of the weight matrices W ( l ) for DNNs was initialized to δ/ √ m where δ i.i.d.
∼ N (0 , 1) for all l = 1 , 2 , . . .
, H .
Similarly, for deep equilibrium linear models, each entry of
A and B was initialized to δ/ √ m where δ i.i.d.
∼ N (0 , 1) .
Linear models were initialized to represent the exact same functions as those of initial deep equilibrium linear models: i.e., W 0 =
B 0 U − 1 0 .
We used γ = 0 .
8 for deep equilibrium linear models.
Training.
For each dataset, we used stochastic gradient descent (SGD) to train linear models, deep equilibrium linear models, and fully-connected feedforward deep neural networks (DNNs).
We used the square loss (cid:96) ( q, y i )
= (cid:107) q − y i (cid:107) 2 2 .
We ﬁxed the mini-batch size to be 64 and the momentum coefﬁcient to be 0.8.
Under this setting, linear models are known to ﬁnd a minimum norm solution (with extra elements from initialization) (Gunasekar et al., 2017; Poggio et al., 2017).
Similarly,
DNNs have been empirically observed to have implicit regularization effects (although the most well studied setting is with the loss functions with exponential tails)
(e.g., see discussions in Poggio et al., 2017; Moroshko et al., 2020;
Woodworth et al., 2020).
In order to minimize the effect of learning rates on our conclusion, we conducted experiments with all the values of learning rates from the choices of 0 . 01 , 0 .
001 , 0 . 0005 , 0 . 0001 and 0 .
00005 , and reported the results with both the worst cases and the best cases separately for each model (and each depth H for DNNs).
All experiments were implemented in PyTorch (Paszke et al., 2019).
D. A DDITIONAL E XPERIMENTS
In this subsection, we report additional experimental results.
Additional datasets.
We repeated the same experiments as those for Figure 1 with four additional datasets – modiﬁed MNIST (LeCun et al., 1998), SVHN (Netzer et al., 2011), SEMEION (Srl &
Brescia, 1994), and Fashion-MNIST (Xiao et al., 2017).
We report the result of this experiment in
Figure 4.
As can be seen from Figures 4, we conﬁrmed qualitatively the same observations as in
Figure 1: i.e., all models preformed approximately the same at initial points, but deep equilibrium linear models outperformed both linear models and nonlinear DNNs in test errors after training.
DNNs without bias terms.
In Figures 1–4, the results of DNNs are reported with bias terms.
To consider the effect of discarding bias term, we also repeated the same experiments with DNNs without bias term and reported the results in Figure 5.
As can be seen from Figures 5, we conﬁrmed qualitatively the same observations: i.e., deep equilibrium linear models outperformed nonlinear
DNNs in test errors.
DNNs with deeper networks.
To consider the effect of deeper networks, we also repeated the same experiments with deeper DNNs with depth H = 10 , 100 and 200 , and we reported the results in Figures 6–7.
As can be seen from Figures 6–7, we again conﬁrmed qualitatively the same observations: i.e., deep equilibrium linear models outperformed nonlinear DNNs in test errors, although DNNs can reduce training errors faster than deep equilibrium linear models.
We experienced gradient explosion and gradient vanishing for DNNs with depth H = 100 and H = 200 .
Larger datasets.
In Figures 1, we used only 200 data points so that we can observe the effect of inductive bias and overﬁtting phenomena under a small number of data points.
If we use a large number of data points, it is expected that the beneﬁt of the inductive bias with deep equilibrium linear models tends to become less noticeable because using a large number of data points can reduce the degree of overﬁtting for all models, including linear models and DNNs.
However, we repeated the same experiments with all data points of each datasets: for example, we usetraining data points and 10000 test data points for MNIST.
Figure 8 reports the results where the values are shown with the best learning rates for each model from the set of learning rates S LR = { 0 . 01 , 0 .
001 , 0 . 0005 , 0 . 0001 , 0 . 00005 } (in terms of the ﬁnal test errors at epoch = 100).
As can be seen in the ﬁgure, deep equilibrium linear models outperformed both linear models and nonlinear DNNs in test errors.
Logistic loss and theoretical bounds.
In Corollary 2, we can set λ T = 1 m (1+ γ ) 2 to get a guarantee for the global linear convergence rate for any initialization in theory.
However, in practice, this is a pessimistic convergence rate and we may want to choose λ T depending on initializations.
To demonstrate this, Figure 3 reports the numerical training trajectory along with theoretical upper bounds with initialization-independent λ T = 1 m (1+ γ ) 2 and initialization-dependent λ T = inf t ∈ [0 ,T ] λ min ( D t ) .
As can be seen in Figure 3, the theoretical upper bound with initialization-dependent λ T demonstrates a faster convergence rate.
[Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]