T OWARDS U NDERSTANDING G ENERALIZATION VIA D ECOMPOSING E XCESS R ISK D YNAMICS
Anonymous authors
Paper under double-blind review
A BSTRACT
Generalization is one of the fundamental issues in machine learning.
However, traditional techniques like uniform convergence may be unable to explain generalization under overparameterization (Nagarajan & Kolter, 2019).
As alternative approaches, techniques based on stability analyze the training dynamics and drive algorithm-dependent generalization bounds.
Unfortunately, the stability-based bounds are still far from explaining the surprising generalization in deep learning since neural networks usually suffer from unsatisfactory stability.
This paper proposes a novel decomposition framework to improve the stability-based bounds via a more ﬁne-grained analysis of the signal and noise, inspired by the observation that neural networks converge relatively slowly when ﬁtting noise (which indicates better stability).
Concretely, we decompose the excess risk dynamics and apply stability-based bound only on the noise component.
The decomposition framework performs well in both linear regimes (overparameterized linear regression) and non-linear regimes (diagonal matrix recovery).
Experiments on neural networks verify the utility of the decomposition framework.
I NTRODUCTION
Generalization is one of the essential mysteries uncovered in modern machine learning (Neyshabur et al., 2014; Zhang et al., 2016; Kawaguchi et al., 2017), measuring how the trained model performs on unseen data.
One of the most popular approaches to generalization is uniform convergence (Mohri et al., 2018), which takes supremum over parameter space to decouple the dependency between the training set and the trained model.
However, Nagarajan & Kolter (2019) pointed out that uniform convergence itself might not be powerful enough to explain generalization, because the uniform bound can still be vacuous under overparameterized linear regimes even if we reduce the parameter space to the minimum.
One alternative solution beyond uniform convergence is to analyze the generalization dynamics , which measures the generalization gap during the training dynamics.
Stability -based bound is among the most popular techniques in generalization dynamics analysis (Lei & Ying, 2020), which is derived from algorithmic stability (Bousquet & Elisseeff, 2002).
Fortunately, one can derive nonvacuous bounds under general convex regimes using stability frameworks (Hardt et al., 2016).
However, stability is still far from explaining the remarkable generalization abilities of neural networks, mainly due to two obstructions.
Firstly, stability-based bound depends heavily on the gradient norm in non-convex regimes (Li et al., 2019), which is typically large at the beginning phase in training neural networks.
Secondly, stability-based bound usually does not work well under general nonconvex regimes (Hardt et al., 2016; Charles & Papailiopoulos, 2018; Zhou et al., 2018b)
but neural networks are usually highly non-convex.
The aforementioned two obstructions mainly stem from the coarse-grained analysis of the signal and noise.
As Zhang et al.
(2016) argued, neural networks converge fast when ﬁtting signal but converge relatively slowly when ﬁtting noise 1 , indicating that the training dynamics over signal and noise are signiﬁcantly different.
Consequently, on the one hand, the fast convergence of signal-related training contributes to a large gradient norm at the beginning phase (see Figure 1a), resulting in
̂
[Figure]
̂ θ
[Figure]
poor stability.
On the other hand, the training on signal forces the trained parameter away from the initialization, making the whole training path highly non-convex (see Figure 1b).
The above two phenomena inspire us to decompose the training dynamics into noise and signal component and only apply the stability-based analysis over the noise component.
To demonstrate that such decomposition generally holds in practice, we conduct several experiments of neural networks on both synthetic dataset and real-world dataset (see Figure 2 for more details).
Based on the above discussion, we improve the stability-based analysis by proposing a decomposition framework on excess risk dynamics 2 , where we handle the noise and signal components separately via bias-variance decomposition.
In detail, we decompose the excess risk into variance excess risk (VER) and bias excess risk (BER), where VER measures how the model ﬁts noise and BER measures how the model ﬁts signal.
Under the decomposition, we apply the stability-based techniques to VER and apply uniform convergence to BER inspired by Negrea et al.
The decomposition framework accords with the theoretical and experimental evidence surprisingly well, providing that it outperforms stability-based bounds in both linear (overparameterized linear regression) and non-linear (diagonal matrix recovery) regimes.
We summarize our contributions as follows:
• We propose a new framework aiming at improving the traditional stability-based bounds, which is a novel approach to generalization dynamics analysis focusing on decomposing the excess risk dynamics into variance component and bias component.
Starting from the overparameterized linear regression regimes, we show how to deploy the decomposition framework in practice, and the proposed framework outperforms the stability-based bounds.
• We theoretically analyze the excess risk decomposition beyond linear regimes.
As a case study, we derive a generalization bound under diagonal matrix recovery regimes.
To our best knowledge, this is the ﬁrst work to analyze the generalization performance of diagonal matrix recovery.
• We conduct several experiments on both synthetic datasets and real-world datasets (MINIST, CIFAR-10) to validate the utility of the decomposition framework, indicating that the framework provides interesting insights into the generalization community. [Figure]
1. R ELATED W ORK
Stability-based Generalization.
The stability-based researches can be roughly split into two branches.
One branch is about how algorithmic stability leads to generalization (Feldman & Vondrak, 2018; 2019; Bousquet et al., 2020).
Another branch focus on how to calculate the stability parameter for speciﬁc problems, e.g., Hardt et al.
(2016) prove a generalization bound scales linearly with time in convex regimes.
Furthermore, researchers try to apply stability techniques into more general settings, e.g., non-smooth loss (Lei & Ying, 2020; Bassily et al., 2020), noisy gradient descent in non-convex regimes (Mou et al., 2018;
Li et al., 2019) and stochastic gradient descent in non-convex regimes (Zhou et al., 2018b; Charles & Papailiopoulos, 2018; Zhang et al., 2021).
In this paper, we mainly focus on applying the decomposition framework to improve the stability-based bound.
Uniform Convergence is widely used in generalization analysis.
For bounded losses, the generalization gap is tightly bounded by its Rademacher Complexity (Koltchinskii & Panchenko, 2000;
Koltchinskii, 2001; Koltchinskii et al., 2006).
Furthermore, we reach faster rates under realizable assumption (Srebro et al., 2010).
A line of work focuses on uniform convergence under neural network regimes, which is usually related to the parameter norm (Bartlett et al., 2017; Wei & Ma, 2019).
However, as Nagarajan & Kolter (2019) pointed out, uniform convergence may be unable to explain generalization.
Therefore, more techniques are explored to go beyond uniform convergence.
Other Approaches to Generalization.
There are some other approaches to generalization, including PAC-Bayes (Neyshabur et al., 2017a; Dziugaite & Roy, 2017; Neyshabur et al., 2017b; Dziugaite & Roy, 2018;
Zhou et al., 2018a; Yang et al., 2019), information-based bound (Russo & Zou, 2016;
Xu & Raginsky, 2017; Banerjee & Montúfar, 2021; Haghifam et al., 2020; Steinke & Zakynthinou, 2020), and compression-based bound (Arora et al., 2018; Allen-Zhu et al., 2018; Arora et al., 2019).
Bias-Variance Decomposition .
Bias-variance decomposition plays an important role in statistical analysis (Lehmann & Casella, 2006; Casella & Berger, 2021; Geman et al., 1992).
Generally, high bias indicates that the model has poor predicting ability on average and high variance indicates that the model performs unstably.
Bias-variance decomposition is widely used in machine learning analysis, e.g., adversarial training (Yu et al., 2021), double descent (Adlam & Pennington, 2020), uncertainty (Hu et al., 2020).
Oymak et al.
(2019) applied bias-variance decomposition on Jacobian of neural networks to explain their different performances on clean and noisy data.
This paper considers a slightly different bias-variance decomposition following the analysis of SGD (Dieuleveut et al., 2016; Jain et al., 2018;
Zou et al., 2021), focusing on the decomposition of the noisy output.
Matrix Recovery.
Earlier methods for solving matrix recovery problems rely on the convex relaxation techniques for minimum norm solutions (Recht et al., 2010; Chandrasekaran et al., 2011).
Recently, a branch of works focuses on the matrix factorization techniques with simple local search methods.
It has been shown that there is no spurious local minima in the exact regime (Ge et al., 2016; 2017; Zhang et al., 2019).
In the overparameterized regimes, it was ﬁrst conjectured by Gunasekar et al.
and then answered by Li et al.
(2018) that gradient descent methods converge to the low-rank solution efﬁciently.
Later, Zhuo et al.
(2021) extends the conclusion to the noisy settings.
P RELIMINARY
In this section, we introduce necessary deﬁnitions, assumptions, and formally introduce previous techniques in dealing with generalization.
Data distribution.
Let x ∈ X
⊂ R p be the input and y ∈ Y ⊂ R be the output, where x , y are generated from a joint distribution ( x , y ) ∼ P .
Deﬁne P x , P y , and P y | x as the corresponding marginal and conditional distributions, respectively.
Given n training samples D ≜ { x i , y i } i ∈ [ n ] generated from distribution P , we denote its empirical distribution by P n .
To simplify the notations, we deﬁne X ∈ R n × p as the design matrix and Y ∈ R n as the response vector.
Excess Risk.
Given the loss function ℓ ( θ ; x , y ) with parameter θ and sample ( x , y ) , we deﬁne the population loss as L ( θ ; P ) ≜
E ( x , y ) ∼P [ ℓ ( θ ; x , y )] and its corresponding training loss as L ( θ ; P n ) ≜ 1 n P i ℓ ( θ ; x i , y i ) .
Let A t denote the optimization algorithm which takes dataset D as input and return the trained parameter ˆ θ ( t ) at time t , namely, A t ( D )
=
ˆ θ ( t ) .
During the analysis, we focus on the excess risk dynamics E L (ˆ θ ( t ) ; P ) , which measures how the trained parameter ˆ θ ( t ) performs on the population loss:
[Equation]
Although the minimizer can be not unique, we deﬁne L ( θ ∗ ; P n ) ≜ min θ L ( θ ; P ) where θ ∗ denotes arbitrarily one of the minimizers.
We additionally remark that bounding the generalization gap
L (ˆ θ ( t ) ; P ) −L (ˆ θ ( t ) ; P n ) sufﬁces to bound the excess risk under Empirical Risk Minimization (ERM) framework by the following split (see Equation 1).
Therefore, we mainly discuss the excess risk following Bartlett et al.
[Equation]
VER and BER.
As discussed in Section 1, we aim to decompose the excess risk into the variance excess risk (VER) and the bias excess risk (BER) deﬁned in Deﬁnition 1.
The decomposition focuses on the noisy output regimes and we split the output y into the signal component E
[ y | x ] and the noise component y − E
[ y | x ] .
Informally, VER measures how the model performs on pure noise, and BER measures how the model performs on clean data.
Deﬁnition 1 (VER and BER)
Given ( x , y ) ∼ P , let ( x , E
[ y | x ]) ∼ P b denote the signal distribution and ( x , y − E
[ y | x ]) ∼ P v denote the noise distribution.
The variance excess risk (VER) E v
L ( θ ; P ) and bias excess risk (BER)
E b L ( θ ; P ) are deﬁned as:
[Equation]
To better illustrate VER and BER, we consider three surrogate training dynamics:
Standard Training,
Variance Training, and Bias Training, corresponding to ER, VER, and BER, respectively.
Standard Training: Training process over the noisy data ( X, Y ) from the initialization θ (0) .
We denote the trained parameter at time t by ˆ θ ( t ) .
Variance Training: Training process over the pure noise ( X, Y −
E [ Y | X ] from the initialization θ (0) v .
We denote the trained parameter at time t by ˆ θ ( t ) v .
Bias Training: Training process over the clean data ( X, E
[ Y | X ]) from the initialization θ (0) b .
We denote the trained parameter at time t by ˆ θ ( t ) b .
When the context is clear, although the trained parameters ˆ θ ( t ) , ˆ θ ( t ) v , ˆ θ ( t ) b are related to the corresponding initialization and the algorithms, we omit the dependency.
Besides, we denote θ ∗ , θ ∗ v and θ ∗ b the optimal parameters which minimize the corresponding population loss L ( θ ; P ) , L ( θ ; P v ) , and L ( θ ; P b ) , respectively.
Techniques in generalization.
We next introduce two techniques in generalization analysis including stability -based techniques in Proposition 1 and uniform convergence in Proposition 2.
These techniques will be revisited in Section 3.
Proposition 1 (Stability Bound from Feldman & Vondrak (2019)) Assume that algorithm A t is ϵ -uniformly-stable at time t , meaning that for any two dataset D and D ′ with only one different data point, we have
[Equation]
Then the following inequality holds 3 with probability at least 1 − δ :
[Equation]
Proposition 2 (Uniform Convergence from Wainwright (2019))
Uniform convergence decouples the dependency between the trained parameter and the training set by taking supremum over a parameter space that is independent of training data, namely
[Equation]
where B is independent of dataset D and A t ( D ) ∈ B for any time t .
W ARM - UP : D ECOMPOSITION UNDER L INEAR R EGIMES
In this section, we consider the overparameterized linear regression with linear assumptions, where the sample size is less than the data dimension, namely, n < p .
We will show how the decomposition framework works and the improvements compared to traditional stability-based bounds.
Before diving into the details, we emphasize the importance of studying overparameterized linear regression.
As the arguably simplest models, we expect a framework can at least work under such regimes.
Besides, recent works (e.g., Jacot et al.
(2018)) show that neural networks converge to overparametrized linear models (with respect to the parameters) as the width goes to inﬁnity under some regularity conditions.
Therefore, studying overparameterized linear regression provides enormous intuition to neural networks.
Settings.
When the context is clear, we reuse the notations in Section 2.
Without loss of generality, we assume that E [ x ] = 0 .
Besides, we assume a linear ground truth y
=
x ⊤ θ ∗ + ϵ where E [ ϵ | x ] = 0 .
Let Σ x ≜ E [ xx ⊤ ] denote the covariance matrix.
Given ℓ 2 loss ℓ ( θ ; x , y ) =
( y − x ⊤ θ ) 2 , we apply gradient descent (GD) with constant step size λ and initialization θ (0) = 0 , namely, ˆ θ ( t +1)
=
ˆ θ ( t ) + λn P
i x i ( y i − x ⊤ i ˆ θ ( t ) ) .
We provide the main results of overparameterized linear regression in Theorem 1 and then introduce how to apply the decomposition framework.
Theorem 1 (Linear Regression Regimes)
Under the overparameterized linear regression settings, assume that w = θ ∗ , ⊤ x √ θ ∗ , ⊤ Σ x θ ∗ is σ 2 w -subgaussian.
If ∥ x ∥ ≤ 1 , | ϵ | ≤ V , sup t ∈ [ T ] ∥ ˆ
θ ( t ) v ∥ ≤ B are all bounded, the following inequality holds with probability at least 1 − δ :
[Equation]
∥λT + Tλ [ V + B ]n ! , where the probability is taken over the randomness of training data.
By setting T = Θ( √ n ) in Theorem 1, the upper bound is approximately ˜ O
( 1 √ n ) , indicating that the estimator at time T is consistent 4 .
Below we show the comparison with some existing bounds, including benign overﬁtting bound and traditional stability-based bound 5 .
Comparison with benign overﬁtting.
The bound in Bartlett et al.
(2020) mainly focus on the minnorm solution (as T → ∞ , the parameter trained in GD converges to the min-norm solution).
Instead, the bound in Theorem 1 does not require the min-norm solution assumption.
Comparison with stability-based bound.
The stability-based bound (without applying the decomposition framework) is ˜
O [Equation] , where B ′ =
sup t ∥ ˆ θ ( t ) ∥ denotes the bound of the trained parameter in standard training.
As a comparison, the notation B in Theorem 1 denote the bound of the trained parameter in variance training .
One major difference between B and B ′ is that: B is independent of ∥ θ ∗ ∥ while B ′ is closely related to ∥ θ ∗ ∥ .
Therefore, with a large time T , the bound in Theorem 1 outperforms the stability-based bound when ∥ θ ∗ ∥ is large compared to V (namely, large signal-to-noise ratio) 6 .
Proof Sketch.
We defer the proof details to Appendix C due to space limitations.
Firstly, we provide
Lemma 1 to show that the excess risk can be decomposed into VER and BER.
Lemma 1 Under the overparameterized linear regression settings with initialization θ (0) =
θ (0) b
= θ (0) v = 0 , for any time T , the following decomposition inequality holds:
[Equation]
We next bound VER and BER separately in Lemma 2 and Lemma 3.
When applying the stabilitybased bound on VER, we ﬁrst convert VER into a generalization gap (see Equation 1) and then bound three terms individually.
Lemma 2 Under the overparameterized linear regression settings, assume that ∥ x ∥ ≤ 1 , | ϵ | ≤ V , sup t ∥ ˆ θ ( t ) v ∥ ≤ B are all bounded.
Consider the gradient descent algorithm with constant stepsize λ , with probability at least 1 − δ :
[Equation]
In Lemma 3, we apply uniform convergence to bound BER.
Although we can bound it by directly calculating the excess risk, we still apply uniform convergence, because exact calculating is limited to a speciﬁc problem while uniform convergence is much more general.
To link the uniform convergence and excess risk, we split the excess risk into two pieces inspired by Negrea et al.
For the ﬁrst piece, which stands for the generalization gap, we bound it by uniform convergence.
For the second piece, which stands for the (surrogate) training loss, we derive a bound decreasing with time T .
Lemma
Under the overparameterized linear regression settings, assume that w =
[Equation]
σ 2 w -subgaussian, then with probability at least 1 − δ :
[Equation]
We remark that the assumptions in Theorem 1 can be relaxed.
Firstly, assuming x is subGaussian sufﬁces
Although we do not present it explicitly, the results in Lemma 3 are derived under uniform convergence frameworks.
Combining Lemma 1, Lemma 2 and Lemma 3 leads to Theorem 1.
□
D ECOMPOSITION UNDER G ENERAL R EGIMES
This section mainly discusses how to decompose the excess risk into variance component (VER) and bias component (BER) beyond linear regimes.
Due to a ﬁne-grained analysis on the signal and noise, the decomposition framework improves the stability-based bounds under the general regimes.
We emphasize that the methods derived in this section are nevertheless the only way to reaching the decomposition goal and we leave more explorations for future work.
To simplify the discussion, we assume the minimizer θ ∗ of excess risk exists and is unique.
Before diving into the main theorem, it is necessary to introduce the deﬁnition of local sharpness for the excess risk function, measuring the smoothness of the excess risk function around the minimizer θ ∗ .
Deﬁnition 2 (Local Sharpness)
For a given excess risk function E L ( θ ; P ) with E L ( θ ∗ ; P ) = 0 , we deﬁne s > 0 as the local sharpness factor when the following equation holds 7 :
[Equation]
Assumption 1 (Uniform Bound) Assume that for excess risk function E L ( θ ; P ) with local sharpness s , given P x , for any P y | x and θ , the uniform bounds 0 < m u ( P x ) ≤ M u
( P x ) exist:
[Equation]
When the context is clear, we omit the dependency of P x and denote the bounds by m u and M u .
Roughly speaking, the uniform bounds act as the maximal and minimal eigenvalues of the excess risk function E L ( θ ; P ) .
For example, under the overparameterized linear regression regimes (Section 3), the uniform bounds are the maximum and minimum eigenvalues of the covariance matrix Σ x .
Assumption 2 (Dynamics Decomposition Condition, DDC)
We assume the trained parameters ˆ θ ( T ) are ( a, C, C ′ )
-bounded at time T , namely:
[Equation]
where n denotes the size of training samples, T denotes the training time, and θ ∗ , θ ∗ v and θ ∗ b denote the minimizers of the corresponding population loss L ( θ ; P ) , L ( θ ; P v ) , and L ( θ ; P b ) , respectively.
We emphasize that although we use C √ T + C ′ √ n in Assumption 2, it can be replaced with any o (1) term which converges to zero when T, n go to inﬁnity.
Empirically, several experiments demonstrate the generality of DDC and its variety (see Figure 3).
From the theoretical perspective, we derive that overparameterized linear regression is (1 , 0 , 0) -bounded in DDC, and diagonal matrix recovery regimes satisfy DDC (see Section 4.1).
We next show how to transfer DDC to the Decomposition
Theorem in Theorem 2.
Theorem 2 (Decomposition Theorem) Assume that the excess risk has a unique minimizer with local sharpness factor s .
Under Assumption 1 and Assumption 2, the following decomposition inequality holds:
[Equation]
We defer the proofs to Appendix D due to limited space.
In Theorem 2, Assumption 1 and Assumption 2 focus on the population loss and the training dynamics individually.
We remark that our proposed generalization bound is algorithm-dependent due to parameters ( a, C, C ′ ) in DDC. [Figure]
By Theorem 2, we successfully decompose the excess risk dynamics into VER and BER.
The remaining question is how to bound VER and BER individually.
As we will derive in Section 4.1 via a case study on diagonal matrix recovery, we bound VER by stability techniques and BER by uniform convergence.
4. D IAGONAL M ATRIX R ECOVERY
From the previous discussion, we now conclude the basic procedure when applying the decomposition framework under Theorem 2.
Generally, there are three questions to be answered in a speciﬁc problem: (a) Does the problem satisfy DDC?
(b) How to bound VER using stability?
(c) How to bound BER using uniform convergence?
To better illustrate how to solve the three questions individually, we provide a case study on diagonal matrix recovery, a special and simpliﬁed case of a general low-rank matrix recovery problem but still retains the key properties.
The diagonal matrix recovery regime and its variants are studied as a prototypical problem in Gissin et al.
HaoChen et al.
Settings.
Let X ⋆ =
Diag { σ ⋆ 1 , · · · , σ ⋆r , 0 , · · · , 0 } ∈ R d × d denote a rank- r (low rank) diagonal matrix, where σ ⋆ 1 ≥ · · · ≥ σ ⋆r > 0 .
We deﬁne A ∈ R d × d as the measurement matrix which is diagonal and y ∈ R d as the measurement vector.
Assume the ground truth y i
= A i σ ⋆i + ε i where y i , A i denotes their corresponding i -th element, ϵ i denotes the noise, and σ ⋆i = [-
-]0 if r < i ≤ d .
Our goal is to recover X ⋆ from dataset D
= { A ( i ) , y ( i ) } i ∈ [ n ] with n training samples.
In the training process, we use the diagonal matrix U =
Diag { u 1 , · · · , u d } to predict X ⋆ via ˆ X = UU ⊤ .
Given the loss function 8 L n ( U ) =
1 n P n j =1 (cid:13)(cid:13)
y ( j ) − A ( j ) ⊙ UU ⊤ (cid:13)(cid:13) 2 , we train the model with Gradient Flow 9 ˙
U t = −∇L n ( U t ) .
During the analysis, we assume that A i is O (1) - uniformly bounded with unit variance, and ε i is ν 2 -subGaussian with uniform bounds | ε ( j ) i | ≤ V .
We emphasize that the training procedure of diagonal matrix recovery is fundamentally different from linear regimes since L n ( U ) is non-convex with respect to U .
Theorem 3 (Decomposition Theorem in Diagonal Matrix Recovery)
In the diagonal matrix recovery settings, if we set the initialization U 0 = αI where α > 0 is the initialization scale, then for any 0 < t < ∞ , with probability at least 1 − δ , we have:
[Equation]
By setting α = Θ (cid:16) ( d 2 n [Equation] −(cid:17) and t =
Θ (log ( dnσ r ) /σ r ) in Theorem 3, the upper bound is approximately ˜ O [Equation] n
[Equation]
, indicating that the estimator at time t is consistent.
Comparison with stability-based bound.
The stability-based bound (without applying the decomposition framework) is ˜
O
[Equation]
As a comparison, the bound in Theorem 3 outperforms the stability-based bound in the sense that we deduce the coefﬁcient of the blowing-up term t +1 n from dV 2 + ∥ X ⋆ ∥ ⋆ to dV 2 , which is a signiﬁcant improvement with a large signal-to-noise ratio under large time t .
Besides, the bound in Theorem 3 captures the bias-variance tradeoff, meaning that the excess risk curve falls ﬁrst and then rises.
Proof Sketch.
We defer the proof details to Appendix E due to space limitations.
Similar to Section 3, the proof of Theorem 3 consists of the following three lemmas, corresponding to the DDC,
BER, and VER terms separately.
We ﬁrst prove that diagonal matrix recovery regimes satisfy DDCin Lemma 4.
Lemma 4 Under the assumptions in Theorem 3, for any 0 ≤ t < ∞ , with probability at least 1 − δ , we have
[Equation]
Different from the overparameterized linear regression regimes, Lemma 4 suffers from a O (cid:16) [Equation] factor due to the non-linearity.
This phenomenon can be veriﬁed empirically in Figure 3b and
Figure 3e.
We next apply stability techniques and uniform convergence to tackle VER and BER in
Lemma 5 and Lemma 6, respectively.
Lemma 5 For VER, under the assumptions in Theorem 3, with probability at least 1 − δ , we have
[Equation]
Lemma 6 For BER, under the assumptions in Theorem 3, with probability at least 1 − δ , we have
[Equation]
Combining the above lemmas leads to Theorem 3.
□
Remark:
Directly applying Theorem 2 under overparameterized linear regression regimes costs a loss on condition number.
The cost comes from the DDC condition where we transform the function space (excess risk function) to the parameter space (the distance between the trained parameter and the minimizer).
We emphasize that the ﬁeld of decomposition framework is indeed much larger than
Theorem 2, and there are other different approaches beyond Theorem 2.
For example, informally, by introducing a pre-conditioner matrix A , changing DDC from the form ∥ ˆ θ − θ ⋆ ∥ to ∥
A ˆ θ − Aθ ⋆ ∥ and setting A = Σ 1 / 2 x , one can avoid the cost of the condition number.
We leave the technical discussion for future work.
C ONCLUSION
In this paper, inspired by the distinct performances of neural networks on signal and noise, we improve the traditional stability-based bounds by proposing a decomposition framework.
Under the framework, we decompose the excess risk into variance excess risk (VER) and bias excess risk (BER), representing the model ability to ﬁt noise and signal, respectively.
We then bound VER using stability-based techniques and bound BER using uniform convergence.
Our bound captures the tendency of the excess risk and blows up much slower than the traditional stability-based bounds.
We apply the decomposition framework into a linear case (overparameterized linear regression with gradient descent) and a non-linear case (matrix recovery with gradient ﬂow), showing that the framework accords better with the theoretical and empirical evidence.
We ﬁnally conduct several experiments on both synthetic datasets and real-world datasets to validate the utility of the decomposition framework on neural networks.
Supplementary Materials
In this section, we ﬁrst provide additional numerical experiment in Section A and ﬁgure illustration in Section B. We then give all the omitted proofs of lemmas and theorems in Section C, Section D, and Section E. We ﬁnally provide some further discussions, including why we apply stability on VER, the motivation behind DDC and a relaxed version of DDC.
A N UMERICAL E XPERIMENT
In this section we provide additional numerical experiments on neural network to verify that our theory is satisﬁed for general neural network architectures and optimization algorithms.
[Figure]
A. S YNTHETIC E XPERIMENT
In this section, our goal is to recover a sparse linear function f ⋆ ( x ) = ⟨ θ ⋆ , x ⟩ ( x ∈ R 30 ) with fully connected ReLU network.
For each setting, we run 5 independent trials and report their average performances (the solid line) and the corresponding standard deviations (the error bar).
Effects of depth: We ﬁrst explore the effect of depth and width for the excess risk decomposition property.
In this experiment, we ﬁx the width of each layer to be 64 , and use SGD as the optimizer with Gaussian initialization ( N (0 , σ 2 ) where σ = 1 × 10 − 3 ) and stepsize η = 1 × 10 − 2 .
For the dash line, we plot a ( BER + VER ) to verify the performance of our risk decomposition framework.
Here a can be regarded as a measure of the success of our theory.
Ideally a should be a small constant so that our bound is non-vacuous.
Speciﬁc to this experiment, for Figure 4a, 4b, 4c, we set a = 1 . 3 , 2 . 0 , 2 . 3 , respectively.
Overall, these results are well-predicted by our theory.
Moreover, it is remarkable to notice that this constant becomes large once increasing the depth.
This mainly contributes to the fact that deeper neural network has higher non-linearity.
Effects of width: In this experiment, we explore the effect of width for 2 -layer ReLU network with SGD optimizer (the initialization and the stepsize are the same as above).
For Figure 4d, 4e, 4e, we set a = 1 . 3 , 1 . 0 , 1 . 0 , respectively.
As can be seen, the wider the neural network, the better our risk decomposition works.
We comment that this behavior also comes to the level of non-linearity.
It is now well-known that when the width tends to inﬁnity, the neural network converges to a linear function (e.g. (Jacot et al., 2018)).
Effects of optimizer:
In this experiment, we try three different optimizers: SGD (the same hyperparameters as before), Adam (stepsize η = 0 .
002 , ( β 1 , β 2 ) =
ϵ = 1 e − 08 , no weight decay), Rprop (learning rate η = 5 × 10 − 4 , ( η 1 , η 2 ) =
(0 . 5 , 1 . 2) , stepsizes (1 × 10 − 6 , 50) ).
For
Figure 4g, 4h, 4i, we set a = 1 . 3 , 1 . 2 , 1 . 1 .
It can be seen that our risk decomposition framework is robust to different optimization algorithms, demonstrating its generality and potential to extend to general optimization problems.
A. CIFAR-
In this section we deliver additional experiment in CIFAR-10 dataset.
In this experiment, we use
ResNet-18 and choose Adam as the optimizer.
We regard that optimizing over the original dataset as the bias training (in the sense that all the label is noiseless, referring to the bias part).
As for the variance training, we set the ground truth to be the uniform logit v = [0 . 1 , · · · , 0 . 1] ⊤ (here we use one-hot coding and choose the categorical cross entropy loss).
As for the labels in the training dataset, we generate it uniformly from { e 1 , · · · , e 10 } , where e i is the standard basis in R 10 .
Under such data generating model, its Bayes risk is provided by − 10 × 0 .
1 × log(0 .
≈ 2 .
3026 for categorical cross entropy loss.
For the standard training, we generate the training label as follows:
y = (1 − p )
y ⋆ + p e j .
Here p ∈ [0 , 1] is the corruption probability, y ⋆ is the true label and j is uniformly sampled from { 1 , · · · , 10 } .
In this experiment, we set the corruption probability to be 0 . 4 .
The numerical result is provided in Figure 5 .
Different from the performance on MINIST, here the variance training actually converges to the ground truth.
We conjecture that this is due to the structure of ResNet, resulting to some kind of implicit regularization towards the training process.
B F IGURE I LLUSTRATION
[Equation]
Figure 1a conveys an important message that variance training has a much smaller gradient norm at the beginning compared to standard training.
This mainly comes from the difference in their landscapes.
Since there is no signal term in the variance learning, we can regard the landscape corresponding to the variance training as just a translation transformation of the original landscape (Figure 1a, the blue line is just a translation transformation of the green line).
Based on this difference, they pursue different optima.
We denote ˆ θ v , ˆ θ the empirical loss minimizer, respectively.
Due to concentration, we have ˆ θ v ≈ 0 , and ˆ θ ≈ θ ⋆ .
Note that we use a near-zero initialization θ (0)
= θ (0) v ≈ 0 , we immediately have that θ (0) v
≈ ˆ θ v , indicating that the gradient norm is small.
On the other hand, since θ (0) is far away from the optimum, its initial gradient norm can be very large.
To better illustrate this principle, we use linear regression to provide a simple example.
For the standard training, the empirical loss is L n ( θ ; P ) =
1 n P ni =1 ( ⟨ θ, X i ⟩ − ⟨ θ ⋆ , X i ⟩ − ε i ) 2 .
Hence, we have
[Equation]
if we assume X i ∼ N (0 , I d × d ) .
On the other hand, for the variance training, the empirical loss is
L n ( θ ; P v ) = 1 n P ni =1 ( ⟨ θ, X i ⟩ − ε i ) 2 .
Therefore, we have
[Equation]
In conclusion, we have
[Equation]
Obviously, there is a huge initial gradient norm gap between standard and variance training, showing that our bound is signiﬁcantly better than the direct stability-based bound under such regimes.
B. F IGURE
Figure 1b provides an example showing that the trajectory experienced by the variance training is likely to stay a convex region, though the global landscape can be highly non-smooth and nonconvex.
The reason is similar to that of Figure 1a.
Since θ (0) v ≈ ˆ θ v , the gradient norm is small and the training dynamics may stay close to the optimum, which lies on a smooth and convex region.
On the contrary, the standard training needs to cross potentially many bad local minimum or saddle points, suffering from large cumulative gradient norms.
B. F IGURE
The setting of Figure 2 is similar to that of Section A. For MINIST dataset, we use three layer convoluted neural network with Adam optimizer.
Figure 3 provides the numerical veriﬁcation of Dynamics Decomposition Condition, and our upper bounds for both linear regression and matrix recovery problems 11 .
Linear Regression We set the dimension d = 500 , the sample sizes are n = 300 , 800 , respectively.
Samples X 1 · · · , X n are i.i.d.
from standard Gaussian distribution N (0 , I d × d ) , and the output y i = ⟨ X i , θ ⋆ ⟩ + ε i where ε i ∼ N (0 , σ 2 ) and σ = 2 .
We set the initialization θ (0)
=
θ (0) b
= θ (0) v =and the learning rate λ = 0 .
Matrix Recovery
We set the dimension d = 20 , rank r = 3 , the ground truth X ⋆ = V Σ V ⊤ , where Σ =
Diag { 5 , 3 , 1 , 0 , · · · , 0 } and V is an orthonormal matrix.
The numbers of the measurement matrices are 200 , 600 , respectively.
The measurement matrices A 1 , · · · , A n are i.i.d.
standard
Gaussian matrices.
The corresponding measurement y i =
⟨ A i , X ⋆ ⟩ + ε i , where ε i ∼ N (0 , 1) .
In this numerical experiment, we use the initialization U (0) =
U b (0)
=
U v (0)
=
αI d × d where α = 0 .
We solve this problem via gradient descent with constant stepsize η = 0 . 1 .
For the upper bound, it is calculated via upper bound =
(cid:13)(cid:13)
X bt − X ⋆ (cid:13)(cid:13) F + ∥ X vt ∥ F + 8 . 5 √ n , where the last term is corresponding to the additional term in Lemma 4.
Neural Network We the dimension d = 30 , the ground truth function is f ⋆ ( x ) =
⟨ θ ⋆ , x ⟩ where θ ⋆ is sparse and ∥ θ ⋆ ∥ = Θ(1) .
The covariate x is generated from Gaussian distribution.
The label for bias training is exactly y i =
f ⋆ ( x i ) .
For the variance trainng, its label is generated from an additional Gaussian distribution ε i ∼ N (0 , σ 2 ) where σ = 1 . 5 .
For the standard training, its label is y i =
f ⋆ ( x i )+ ε i .
For the upper bound, it is calculated via upper bound =
(cid:13)(cid:13)(cid:13)
f θ bt − f ⋆ (cid:13)(cid:13)(cid:13) P + (cid:13)(cid:13)
f θ vt (cid:13)(cid:13) P .
Here ∥·∥ P is deﬁned to be the L 2 ( P ) norm, i.e., ∥ g ∥ 2 P = E x P (cid:2) f ( x ) 2 (cid:3) .
For these settings, we choose SGD (with stepsize η = 1 × 10 − 3 ) as the optimizer where the initialization is randomly generated from a Gaussian distribution.
C P ROOF FOR O VERPARAMETERIZED L INEAR R EGRESSION C. P ROOF OF T HEOREM
In this section, we prove Theorem 1 stated as follows:
Theorem 1 (Linear Regression Regimes)
Under the overparameterized linear regression settings, assume that w = θ ∗ , ⊤ x √ θ [Equation] Σ [Equation]
is σ [Equation] w -subgaussian.
If ∥ x ∥ ≤ 1 , | ϵ | ≤ V , sup t ∈ [ T ] ∥
ˆ θ ( t ) v ∥ ≤ B are all bounded, the following inequality holds with probability at least 1 − δ :
[Equation]
where the probability is taken over the randomness of training data.
Proof:
We ﬁrst calculate the population loss under linear regression regime, namely, for any time t :
[Equation]
where (a) we use the condition that y = x ⊤ θ ∗ + ϵ and (b) we use the assumption that E [ ϵ | x ] = 0 .
Therefore, the excess risk can be written as
[Equation]
Now plugging the Lemma 1, Lemma 2, Lemma 3, into Theorem 2, we derive that with probability at least 1 − δ :
[Equation]
which completes the proof.
□
C. P ROOF OF L EMMA
We ﬁrst recall Lemma 1 as follows:
Lemma 1 Under the overparameterized linear regression settings with initialization θ (0) =
θ (0) b
= θ (0) v = 0 , for any time T , the following decomposition inequality holds:
[Equation]
Proof: Firstly, when we use gradient descent, we have that:
[Equation]
We next prove the conclusion by induction.
Since we choose θ (0)
=
θ (0) v = θ (0) b = 0 , the conclusion holds at the initialization θ (
[Equation]
[Equation]
b .
Now assume that ˆ θ ( t )
=
ˆ θ ( t ) v + ˆ θ ( t ) b holds at time t , we have
[Equation]
In summary, we have: ˆ θ ( t ) =
ˆ θ ( t ) v + ˆ θ ( t ) b holds for any time t .
Besides, we easily calculate that θ ∗ =
θ ∗ v + θ ∗ b since θ ∗ =
θ ∗ b = θ ∗ and θ ∗ v = 0 .
Therefore, we have ˆ θ ( t ) − θ ⋆
= ˆ θ ( t ) v − θ ⋆v
+ ˆ θ ( t ) b − θ ⋆b holds for any t .
By CauchySchwarz inequality, we derive that for a speciﬁc time T :
[Equation]
where we denote the norm ∥ u ∥ 2 A = u ⊤ Au .
□
C. P ROOF OF L EMMA
Lemma 2 Under the overparameterized linear regression settings, assume that ∥ x ∥ ≤ 1 , | ϵ | ≤ V , sup t ∥ ˆ θ ( t ) v ∥ ≤ B are all bounded.
Consider the gradient descent algorithm with constant stepsize λ , with probability at least 1 − δ :
[Equation]
Proof: When the context is clear, we omit v in this section and denote the trained parameter by ˆ θ ( t ) .
For any time t , we ﬁrst split the excess risk into three components following Equation 1.
E
[Equation]
Part (I).
We bound the ﬁrst component in Equation 13 using stability.
Fact A: the loss ℓ ( θ ; x , y ) is
[Equation]
Fact B: the stability of the parameter is 2 λtn [ V + B ] .
Denote the parameter trained on dataset S and the parameter trained on dataset S ′
as ˆ θ ( t ) S and ˆ θ ( t ) S ′ , respectively.
We will show that, when S and S ′
[Table]
[Equation]
Summation over time t provides the following result:
[Equation]
Therefore, the total stability with respect to the ℓ 2 -loss is
[Equation]
By Proposition 1, we have that
[Equation]
Part II.
We next show that the second component in Equation 13 is less than zero based on Lemma 7.
[Equation]
The i -th eigenvalue of the matrix [ − 2 X
[ I − [ I − λn X ⊤ X ] t ] X † + X † , ⊤ [ I − [ I − λn X ⊤ X ] t ] X ⊤ X
[ I − [ I − λ n X ⊤ X t ] X † ] can be calculated as follows
[Equation]
where σ i refers to the i -th eigenvalue of the matrix 1 n X ⊤ X .
Therefore, we have
[Equation]
Part III.
We bound the third component in Equation 13 using concentration.
Note that | ℓ 2 ( θ ; x , y ) | = | [Equation] y − x ⊤ θ ) 2 | ≤ [ V + B ] 2 , so we can use Hoeffding’s inequality,
[Equation]
By setting ϵ =
[ V + B ]
[Equation]
log
[Equation]
2 /δ ) 2 n , then with probability at least 1 − δ , we have
[Equation]
Combining Part (I), (II), and (III) together, at time T , we have:
[Equation]
C. P ROOF OF L EMMA
Lemma
Under the overparameterized linear regression settings, assume that w = θ ∗ , ⊤ x √ θ ∗ , ⊤ Σ x θ ∗ is σ 2 w -subgaussian, then with probability at least 1 − δ :
[Equation]
Proof: Since we use the noiseless data ( x , E ( y | x )) and E ( y | x ) = x ⊤ θ ∗ , by Lemma 7 we have
[Equation]
where we use the fact that
[Equation]
Therefore, BER can be expressed as:
[Equation]
We split the excess risk into two parts.
Intuitively, it is just like the decomposition in Equation 1.
The ﬁrst part is similar to the generalization gap to use uniform convergence to bound it.
Note that here we require that λ < sup X1 ∥ 1 n X 0 ⊤ X 0 ∥ .
Hence, we have
[Table]
[Equation]
[Table]
[Equation]
where we apply Lemma 8 repeatedly since rank
[Equation]
1 where A stands for arbitrary matrix.
We highlight the difference between X 0 and X .
The notation X represents the contribution of the training set to the training error.
The notation X 0 represents the contribution of the training set to the training estimator ˆ θ .
The uniform convergence is taken over the estimator θ .
Therefore, we take sup operator on X 0 .
We next bound θ ∗ , ⊤ (cid:2) Σ x − 1 n X ⊤ X (cid:3) θ ∗ .
Assume that w = θ ∗ , ⊤ x / p θ ∗ , ⊤ Σ x θ ∗ is SubGaussian with subGaussian norm σ w .
Obviously, E [ w ] = 0 , E (cid:2) w 2 (cid:3) = 1 .
Therefore, we have that: w 2 − 1 is sub-Exponential distributions with sub-Exponential norm (cid:13)(cid:13) w
[Equation]
− 1 (cid:13)(cid:13) ψ 1 ≤ σ 2 w .
While ϵ/ (cid:2) θ ⊤ Σ x θ (cid:3) < σ 2 w , we have the following tail bound
[Equation]
where c is a universal constant.
Therefore, by setting ϵ = (cid:2) θ ⊤ Σ x θ (cid:3) σ 2 w log[2 /δ ] √ c √ n with probability at least 1 − δ , we have
[Equation]
⊤
[Equation]
With abuse to use c as a constant, we have with probability at least ≥ 1 − δ
[Equation]
For the second part, which is closely related to the empirical loss of θ ( t ) b , We will show that: for a general case, the empirical loss converges with rate O (1 /t ) .
Denote the eigenvalues of 1 n X ⊤ X as σ 1 , · · · , σ n , 0 , · · · , 0 .
Then the corresponding eigenvalues of matrix (cid:2)
I
[Equation]
⊤
[Equation]
X ⊤ X (cid:3) t are ( [Equation] i ) 2 t σ i .
Therefore, we have:
[Equation]
The maximum is attained while σ i = 1 λ (2 t +1) .
Combining these terms leads to the conclusion.
□
C
A UXILIARY L EMMAS
Lemma 7 The dynamics of ˆ θ ( t ) using GD is:
[Equation]
where X † represents the Pseudo inverse of matrix X .
Proof: The proof is based on induction.
We notice that when t = 0 , the equation directly follows.
Assume when t = t , the equation holds.
Then at time t + 1 , we have
[Equation]
where we use the fact that X ⊤ XX † = X ⊤ .
□
Lemma 8 For rank-1 matrix A, we have that ∥ A ∥ = | tr [ A ] | .
D P ROOF OF T HEOREM
In this section, we provide the proof of Theorem 2.
We ﬁrst recall the formal statement of Theorem 2.
Theorem 2 (Decomposition Theorem) Assume that the excess risk has a unique minimizer with local sharpness factor s .
Under Assumption 1 and Assumption 2, the following decomposition inequality holds:
[Equation]
Proof: Under Assumption 1 (upper bound), we have:
[Equation]
Combining Assumption 2 (DDC) and the lower bound in Assumption 1, we have
[Equation]
Therefore
[Equation]
we have that
[Equation]
n (cid:21) s . where we use the fact that ( a + b + c + d ) p ≤ max { (4 a ) p , (4 b ) p , (4 c ) p , (4 d ) p } ≤ (4 a ) p + (4 b ) p + (4 c ) p + (4 d ) p when a, b, c, d, p > 0 .
□
E P ROOF FOR D IAGONAL M ATRIX R ECOVERY
In this section, we prove Theorem 3 via proving the proceeding three lemmas separately.
E. P ROOF OF T HEOREM
Theorem 3 (Decomposition Theorem in Diagonal Matrix Recovery)
In the diagonal matrix recovery settings, if we set the initialization U 0 = αI where α > 0 is the initialization scale, then for any 0 < t < ∞ , with probability at least 1 − δ , we have:
[Equation]
Note that although we do optimization based on U , we consider its square X = UU ⊤ as the parameter since we require the solution is unique during the analysis.
Note that we do not change the update rule (where GD is still taken on U ).
One can directly check that under Diagonal Matrix Recovery regimes, s = 2 and M u
= m u = 1 .
Therefore, combining Lemma 4, Lemma 5 and Lemma 6 leads to Theorem 3.
E. P ROOF OF L EMMA
Lemma 4 Under the assumptions in Theorem 3, for any 0 ≤ t < ∞ , with probability at least 1 − δ , we have
[Equation]
Proof: Since all the measurement matrices A (1) , · · · , A ( n ) are diagonal, we can analyze the dynamic of each singular value separately.
For a given singular value σ , assume a i ∼ N (0 , 1) , ε i ∼ subG( ν 2 ) .
Then we deﬁne σ 1 =n P ni
=1 a 2 i σ , and σ 2 = 1 n P ni {+=
+}
a i ε i , denote σ = σ 1 + σ 2 , and ξ = 1 n P ni
=1 a 2 i .
Denote u 2 ( t ) , u 2 b ( t ) , u 2 v ( t ) the dynamics of standard training, bias training, and variance training, respectively.
Then our goal is to show that | σ − u 2 ( t ) | ≤ | σ − u 2 b ( t ) | + u 2 v ( t ) + small term holds with high probability.
Before diving into the details, we ﬁrst give the dynamics of u 2 ( t ) , u 2 b ( t ) , u 2 v ( t ) .
If σ > 0 , we have
[Equation]
Furthermore, if σ 2 > 0 , we have
[Equation]
Otherwise, if σ 2 < 0 we have
[Equation]
If σ ⋆ = 0 , we have
[Equation]
Now we turn to the proof of Lemma 4.
If σ = 0 , then σ = σ 2 , which implies u 2 ( t ) =
u 2 v ( t ) , indicating that the conclusion holds.
Hence, we only consider the case that σ ≥ σ r > 0 .
By triangle inequality, it sufﬁces to show
[Equation]
Without loss of generality [Equation] we assume σ > σ 1 > 0 .
Note that
[Equation]
Via integration-by-parts formula
[Equation]
we have
[Equation]
Therefore
[Equation]
it sufﬁces to show that
[Equation]
We prove that there exists a universal constant C , such that
[Equation]
which is equivalent to show
[Equation]
Note that ϕ (0) =
C
[Equation]
(cid:0) σ 1 + ξα 2 (cid:1) ≥ 0 , and its gradient is
[Equation]
By concentration of Chi-square distribution, we have ξ ≥ 1 − q [Equation] 1 /δ
[Equation]
n = Ω(1) with probability at least 1 − δ , providing that n ≳ log (cid:0) 1 δ (cid:1) .
It sufﬁces to show that ϕ ′ ( t ) ≥ 0 , which is equivalent to ψ ( t ) = Cα 2 e [Equation]
[Equation] + [Equation] 1 − σσ 1 t ≥ 0 .
It attains its minimum σ 2 + Cσ 1 − σ 2 log (cid:0)2 Cα(cid:1) at the point t = log (cid:0)2 Cα(cid:1) / 2 σ 1 .
Therefore, it sufﬁces to set C = log (cid:0) [Equation]
Overall, we have
[Equation]
Hence, with proper choice of probability, we have with probability at least 1 − δ , the following holds
[Equation]
n
[Equation]
To summarize, the case is[-(cid:16) a = 1 , C = 0 , C (cid:48) = O (cid:16) log (cid:0) 
 α 2 (cid:1)-]
[Equation]
[Equation]
(See
DDC).
□
E. P ROOF OF L EMMA
Lemma 5 For VER, under the assumptions in Theorem 3, with probability at least 1 − δ , we have
[Equation]
Proof:
Similar to the proof in overparameterized linear regression regimes, we ﬁrst split the excess risk in three componenets:
[Equation]
Part (I).
We bound the ﬁrst component in Equation 13 using stability.
We deﬁne σ 2 = 1 n P ni {+=
+}
a i ε i , σ ′ 2 = 1 n P n − 1 i =
a i ε i + a ′ n ε ′ n .
Similarly, we deﬁne ξ = 1 n P ni
=1 a 2 i , and ξ ′ = 1 n P n − 1 i =1 a [-
-]2 i + a ′ 2 n .
In the ﬁrst case , we assume both σ 2 , σ ′ 2 > 0 , so that their difference can be bounded | u 2 v ( t ) − u ′ 2 v ( t ) | ≲
[Equation]
WLOG, we assume σ 2 > σ ′ 2 .
Then, similar to the proof in Lemma
[Equation]
we have
[Equation]
The second term is dominated by the ﬁrst term.
Hence, we only need to handle the third term.
Note that
[Equation]
we have
[Equation]
Combined the above three parts, we ﬁnally have
[Equation]
In the second case , we assume both σ 2 , σ ′ 2 < 0 .
Since both of them would decrease to 0 , we have | u 2 b ( t ) − u ′ 2 b ( t ) | ≤ α 2 where α
= O ( 1 n 1 / 4 ) .
In the last case , without loss of generality, we assume σ 2 < 0 < σ ′ 2 .
Since u 2 v ( t ) decreases to 0 , and u ′ 2 v ( t ) increases to σ ′ 2 , we have | u 2 b ( t ) − u ′ 2 b ( t ) | ≤ σ ′ 2 ≤ | σ ′ 2 − σ 2 | .
Overall, we have
[Equation]
Besides, since the loss is O ( V ) -Lipschitz:
[Equation]
where u 2 is bounded upper bounded by O ( V ) during the training analysis due to a small initialization.
Then the total stability is
[Equation]
Summation over the dimension d , the whole loss ℓ d ( u 2 v ( t )) has stability
[Equation]
By Proposition 1, we have that
[Equation]
Part (II).
We next calculate the second term.
Take the explicit formula into the equation, we have
[Equation]
If σ 2 = 1 n P ni {+=
+} a i ε i >
[Equation]
we have [Equation] If σ 2 < 0 , we have
[Equation]
In conclusion [Equation] we have L n [Equation]
Part (III).
For the third term [Equation] we just need one single concentration inequality.
Note that for the variance training, the loss function is of the order O (cid:0) V 2 (cid:1) .
Hence, via the Hoeffding inequality, with probability at least 1 − δ , we have
[Equation]
Combining the three terms leads to Lemma 5.
□
E. P ROOF OF L EMMA
Lemma 6 For BER, under the assumptions in Theorem [Equation] with probability at least 1 − δ , we have
[Equation]
Proof:
Similar to linear regression [Equation] we decompose the excess risk into two parts:
[Equation]
We use the uniform convergence to bound the ﬁrst term.
Assume that σ > 0 , we have
[Equation]
Now assume σ = [Equation] we have
[Equation]
The second term refers to the training loss, if σ > 0 , we have
[Equation]
If σ =
[Equation]
we have
[Equation]
Finally
[Equation]
by assigning probability properly
[Equation]
we have
[Equation]
F A DDITIONAL D ISCUSSION
In this section
[Equation]
we discuss more the details in the main text.
We ﬁrst validate the rationality of applying stability-based bound by showing that the generalization gap (under variance regime) increases with time.
We next provide some motivation behind Dynamics Decomposition Condition by showing that it indeed holds as the beginning of the training phase.
We ﬁnally introduce a related version of Dynamics Decomposition Condition which does not require the information about θ ∗ , θ ∗ v [Equation] and θ ∗ .
F. T HE GENERALIZATION GAP UNDER VARIANCE INCREASES WITH TIME
In this part, we will prove that the (variance regime) generalization gap indeed increases with time, which validates that applying stability-based bound (which also increases with time) is rational.
We provide the following Lemma 9 which validates the above statement.
Lemma 9 Let ∆ v ( t ) =
L (ˆ θ ( t ) v ; P ) − L (ˆ θ ( t ) v ; P n ) be the generalization gap under variance regime.
Assume that Σ x = I .
Then under linear regression settings as in Section 3, we have
[Equation]
Proof: Note that the population loss is calculated as follows by Equation C.
[Equation]
And the training loss is calculated as follows:
[Equation]
Note that by Lemma 7, we have
[Equation]
Therefore, we have:
[Equation]
By omitting the terms in ∆ v ( t ) that is uncorrelated to t
[Equation]
we have (denote as ¯∆ v ( t ) ): [Equation] where we use the fact that [Equation]
We next focus on the matrix
[Equation]
where we plug in Σ x
= I by assumption.
Denote the SVD of 1 n X ⊤ X as U ⊤ Σ U where the i -th eigenvalue is σ i .
Then when σ i = 0 , the i − th eigenvalue of M a is also equal to zero (due to the persudo inverse X † ).
When σ i > 0 , the i − th eigenvalue of M a can be derived as
[Equation]
Its derivation is then
[Equation]
since log(1 − λσ i ) ≤ 0 and (1 − σ i )(1 − λσ i ) t − 1 ≤ 0 .
Therefore, the derivation of each eigenvalue is larger than zero.
We rewrite ¯∆ v ( t ) =
z ⊤ Σ Ma z ⊤ where z ⊤ =
UE is a vector independent of time t and Σ Ma is the diagonal matrix with diagonal σ i ( M a ) .
As a result, we have
[Equation]
since ∂∂t Σ Ma [-(cid:23) 
-]⪰ 0 according to Equation 15.
Note that ¯∆ v ( t ) and ∆ v ( t ) only differ with a term independent of time, we have
[Equation]
□
F. M OTIVATION BEHIND D YNAMICS D ECOMPOSITION C ONDITION
For a dataset S = {
( X i , y i ) } ni =1 , where y i =
f θ ⋆ ( X i )
+ ε i , and θ ⋆ ∈ Θ .
We choose the ℓ 2loss to solve this problem L ( θ ) = {+12 n P ni =
+}
( f θ ( X i ) − y i ) 2 .
Then we can show that the Dynamics Decomposition Condition holds at the beginning phase if we initialize θ (0) v = 0 , and assume f 0 ( X ) = 0 , ∀ X ∈ supp {P} .
For sufﬁcient small t > 0 , the dynamics are well approximated by
[Equation]
Suppose θ (0)
=
θ (0) b [Equation] then based on Taylor expansion [Equation]
we have the approximation
[Equation]
Therefore, the Dynamics Decomposition Condition holds at least at the beginning phase.
Moreover, we discuss in the main text that for a branch of learning tasks, such as linear regression and matrix recovery, these Dynamics Decomposition Condition uniformly holds for arbitrary 0 ≤ t < ∞ .
F. R ELAXED V ERSION OF DDC
One may notice that there exist θ ∗ , θ ∗ v , θ ∗ b in DDC (See Assumption 2), which is the optimal solution to the corresponding population loss.
They are sometimes unavailable in practice and even have no explicit solution.
For more convenient analysis, we propose a relaxed version of DDC in Lemma 10.
Lemma 10 (Relaxed DDC)
Let the loss be ℓ 2 loss where ℓ ( y , ˆ y )
=
( y − ˆ y ) 2 .
We assume the predicting model ˆ y =
f ˆ θ ( x ) is unique with respect to ˆ θ (when θ i ̸
= θ j , f θ i ̸
= f θ j ).
Besides, assume f θ =0 ( x ) = 0 for all x .
For the ground truth, we assume an additive noise, namely, y =
f θ ∗ ( x ) + ϵ where ϵ is independent of x .
Then the following Equation 18 sufﬁces to prove (max { a, 1 } , C, C ′ ) - boundedness for DDC (See Assumption 2).
[Equation]
Based on Lemma 10, one can verify DDC without knowing the optimal solution parameter θ ∗ , θ ∗ v , θ ∗ b .
Besides, since we usually have the iteration equation (the relationship between ˆ θ ( t +1) and ˆ θ ( t ) ) given an algorithm, it is possible to verify DDC based on the iteration forms.
Note that we may not directly verify the condition in practice based on the relaxed DDC if we do not have the explicit split on signal E [ y | x ] and noise y − E
[ y | x ] .