C OMPRESSION BASED BOUND FOR NON - COMPRESSED NETWORK : UNIFIED GENERALIZATION ERROR ANALYSIS OF LARGE COMPRESSIBLE DEEP NEURAL NETWORK
A
BSTRACT
One of the biggest issues in deep learning theory is the generalization ability of networks with huge model size.
The classical learning theory suggests that overparameterized models cause overﬁtting.
However, practically used large deep models avoid overﬁtting, which is not well explained by the classical approaches.
To resolve this issue, several attempts have been made.
Among them, the compression based bound is one of the promising approaches.
However, the compression based bound can be applied only to a compressed network, and it is not applicable to the non-compressed original network.
In this paper, we give a uniﬁed framework that can convert compression based bounds to those for non-compressed original networks.
The bound gives even better rate than the one for the compressed network by improving the bias term.
By establishing the uniﬁed framework, we can obtain a data dependent generalization error bound which gives a tighter evaluation than the data independent ones.
I NTRODUCTION
Deep learning has shown quite successful results in wide range of machine learning applications.
such as image recognition (Krizhevsky et al., 2012), natural language processing (Devlin et al., 2018) and image synthesis tasks (Radford et al., 2015).
The success of deep learning methods is mainly due to its ﬂexibility, expression power and computational efﬁciency for large dataset training.
Due to its signiﬁcant importance in wide range of application areas, its theoretical analysis is also getting much important.
For example, it has been known that the deep neural network has universal approximation capability (Cybenko, 1989; Hornik, 1991; Sonoda & Murata, 2015) and its expressive power grows up in an exponential order against the number of layers (Montufar et al., 2014; Bianchini & Scarselli, 2014;
Cohen et al., 2016; Cohen & Shashua, 2016; Poole et al., 2016; Suzuki, 2019).
However, theoretical understandings are still lacking in several important issues.
Among several topics of deep learning theories, a generalization error analysis is one of the biggest issues in the machine learning literature.
An important property of deep learning is that it generalizes well even though its parameter size is quite large compared with the sample size (Neyshabur et al., 2019).
This can not be well explained by a classical VC-dimension type theory (Harvey et al., 2017) which suggests that overparameterized models cause overﬁtting and thus result in poor generalization ability.
For this purpose, norm based bounds have been extensively studied so far (Neyshabur et al., 2015;
Bartlett et al., 2017b; Neyshabur et al., 2017; Golowich et al., 2018).
These bounds are beneﬁcial because the bounds are not explicitly dependent on the number of parameters and thus are useful to explain the generalization error of overparameterized network (Bartlett, 1998; Neyshabur et al., 2015; 2019).
However, these bounds are typically exponentially dependent on the number of layers and thus tends to be loose for deep network situations (Dziugaite & Roy, 2017; Arora et al., 2018;
Nagarajan & Kolter, 2019).
As a result, Arora et al.
(2018) reported that a simple VC-dimension bound (Li et al., 2018; Harvey et al., 2017) can still give sharper evaluations than these norm based bounds in some practically used deep networks.
Wei & Ma (2019) improved this issue by involving a data dependent Lipschitz constant as performed in Arora et al.
; Nagarajan & Kolter (2019).
On the other hand, compression based bound is another promising approach for tight generalization error evaluation which can avoid the exponential dependence on the depth.
The complexity of deep neural network model is regulated from several aspects.
For example, we usually impose explicit regularization such as weight decay (Krogh & Hertz, 1992), dropout (Srivastava et al., 2014; Wager et al., 2013), batch-normalization (Ioffe & Szegedy, 2015), and mix-up (Zhang et al., 2018; Verma et al., 2018).
Zhang et al.
(2016) reported that such explicit regularization does not have much effect but implicit regularization induced by SGD (Hardt et al., 2016;
Gunasekar et al., 2018; Ji &
Telgarsky, 2019) is important.
Through these explicit and implicit regularizations, deep learning tends to produce a simpler model than its full expression ability (Valle-Perez et al., 2019; Verma et al., 2018).
To measure how “simple” the trained model is, one of the most promising approaches currently investigated is the compression bounds (Arora et al., 2018; Baykal et al., 2019; Suzuki et al., 2018).
These bounds measure how much the network can be compressed and characterize the size of the compressed network as the implicit effective dimensionality.
Arora et al.
characterized the implicit dimensionality based on so called layer-cushion quantity and suggested to perform random projection to obtain a compressed network.
Along with a similar direction,
Baykal et al.
(2019) proposed a pruning scheme called Corenet and derived a bound of the size of the compressed network.
Suzuki et al.
(2018) has developed a spectrum based bound for their compression scheme.
Unfortunately, all of these bounds guarantee the generalization error of only the compressed network, not the original network.
Hence, it does not give precise explanations about why large network can avoid overﬁtting.
In this paper, we derive a uniﬁed framework to obtain a compression based bound for a noncompressed network .
Unlike the existing researches, our bound is valid to evaluate the original network before compression, and thus gives a direct explanation about why deep learning generalizes despite its large network size.
The difﬁculty to apply the compression bound to the original network lies in evaluation of the population L 2 -bound between the compression network and the original network.
A naive evaluation results in the VC-bound which is not preferable.
This difﬁculty is overcome by developing novel data dependent capacity control technique using local Rademacher complexity bounds (Mendelson, 2002; Bartlett et al., 2005; Koltchinskii, 2006; Giné & Koltchinskii, 2006).
Then, the bound is applied to some typical situations where the network is well compressed.
Our analysis stands on the implicit bias hypothesis (Gunasekar et al., 2018; Ji & Telgarsky, 2019) that claims deep learning tends to produce rather simple models.
Actually, Gunasekar et al.
Ji & Telgarsky (2019) showed gradient descent results in (near) low rank parameter matrices in each layer in linear network settings.
Martin & Mahoney (2018) evaluated the eigenvalue decays of the weight matrix through random matrix theories and several numerical experiments.
These observations are also supported by the ﬂat minimum analysis (Hochreiter & Schmidhuber, 1997; Wu et al., 2017; Langford & Caruana, 2002), that is, the product of the eigenvalues of the Hessian around the
SGD solution tends to be small, which means SGD converges to a ﬂat minimum and possess stability against small perturbations leading to good generalization.
Based on these observations, we make use of the eigenvalue decay of the weight matrix and the covariance matrix among the nodes in each layer (this assumption is actually veriﬁed by numerical experiments in Appendix D).
The eigenvalue decay speed characterizes the redundancy in each layer and thus is directly relevant to compression ability.
Our contributions in this paper are summarized as follows:
• We give a uniﬁed framework to obtain a compression based bound for non-compressed network which properly explains that a compressible network can generalizes well.
The bound can convert several existing compression based bounds to that for non-compressed one in a unifying manner.
The bound is applied to near low rank models as concrete examples.
• We develop a data dependent capacity control technique to bound the discrepancy between the original network and compressed network.
As a result, we obtain a sharp generalization
[Table] Thm [Table] error bound which is even better than that of the compressed network.
All derived bounds are characterized by data dependent quantities.
Other related work Recently, the role of over-parameterization for two layer networks has been extensively studied (Neyshabur et al., 2019; Arora et al., 2019).
These are for the shallow network and the generalization error is essentially given by the norm based bounds.
It is not obvious that these bounds also give sharp bounds for deep models.
PAC-Bayes bound is also applied to obtain a non-vacuous compression based bound (Zhou et al., 2019).
However, the bound is still for the compressed (quantized) models and it is not obvious that that bound can be converted to that for the original network.
Relation between compression and learnability was traditionally studied in a different framework as in Littlestone & Warmuth (1986) and minimum description code length (Hinton & Van Camp, 1993).
Our bound would share the same spirits with these studies but give a new analysis by incorporating recent observations in deep learning researches.
P RELIMINARIES : P ROBLEM FORMULATION AND NOTATIONS
In this section, we give the problem setting and notations that will be used in the theoretical analysis.
We consider the standard supervised leaning formulation where data consists of input x ∈ R d and output (or label)
y ∈ R .
We consider a single output setting, i.e., the output y is a 1-dimensional real value, but it is straight forward to generalize the result to a multiple output case.
Suppose that we are given n i.i.d.
observations D n = ( x i , y i ) ni
=1 distributed from a probability distribution P .
To measure the performance of a trained function f , we use a loss function ψ : R × R → R and deﬁne a training error and its expected one as [Equation] E
[Equation]
ψ ( Y, f ( X ))] , where the expectation is taken with respect to ( X, Y ) ∼ P .
Basically, we are interested in the generalization error Ψ( (cid:98) f )
− (cid:98) Ψ
( (cid:98) f ) for an estimator (cid:98) f .
We denote the empirical L 2 -norm by (cid:107) f (cid:107) n := (cid:112)(cid:80) ni
=1 f ( z i ) 2 /n for an empirical observation z i =
( x i , y i )
( i = 1 , . . . , n ) .
The population L 2norm is denoted by (cid:107) f (cid:107) L 2 :=
(cid:112) E Z ∼ P [ f ( Z ) 2 ] .
This paper deals with deep neural networks as a model.
The activation function is denoted by η which will be assumed to be 1-Lipschitz as satisﬁed by ReLU (Assumption 1).
Let the depth of the network be L and the width of the (cid:96) -th internal layer be m (cid:96) ( (cid:96) = 1 , . . .
, L + 1) where we set m
1 =
d (dimension of input) and m L +
= 1 (dimension of output) for convention.
Then, the set of networks having depth L and width m = ( m 1 , . . .
, m L ) with norm constraint as
[Equation]
where (cid:107) W (cid:107) 2 :=
sup u : (cid:107) Wu (cid:107)(cid:54) =
(cid:107) Wu (cid:107) / (cid:107) u (cid:107) 1 is the operator norm (the maximum singular value), (cid:107) W (cid:107) F :=
(cid:113)(cid:80) i,j W 2 i,j is the Frobenius norm, and G is the “clipping” operator that is deﬁned by
G ( x ) = max {− M, min { x, M }} for a constant M .
The reason why we put the clipping operator G on top of the last layer is because the clipping operator restricts the L ∞ -norm by a constant M and then we can avoid unrealistically loose generalization error.
Note that the clipping operator does not change the classiﬁcation error for binary classiﬁcation.
We express F to represent the “full model”:
F = NN( m , R 2 , R F ) for a given R 2 , R F > 0 .
Here, we implicitly suppose that R 2 is close toso that the norm of the output from internal layers is not too much ampliﬁed, while R F could be moderately large.
The Rademacher complexity is the typical tool to evaluate the generalization error on a function class
F (cid:48) , which is denoted by ˆ R n ( F (cid:48) ) :
= E (cid:15) (cid:2) sup f ∈F (cid:48) 1 n (cid:80) ni =1 (cid:15) i f
( z i ) | D n (cid:3) where D n =
( z i ) ni
=
= ( x i , y i ) ni
=1 , and (cid:15)
i
( i = 1 , . . .
.n )
is an i.i.d.
Rademacher sequence ( P ( (cid:15) i = 1) = P ( (cid:15) i = − 1) = 1 / 2) .
This is also called conditional Rademacher complexity because the expectation is taken conditioned on ﬁxed D n .
Its expectation with respect to D n is denoted by ¯ R n ( F (cid:48) ) :
= E D n [ ˆ R n ( F (cid:48) )] .
Roughly speaking the Rademacher complexity measures the size of the model
and it gives an upper bound of the generalization error (Vapnik, 1998; Mohri et al., 2012).
The main difﬁculty in generalization error analysis of deep learning is that the Rademacher complexity of the full model F is quite large.
One of the successful approaches to avoid this difﬁculty is the compression based bound (Arora et al., 2018; Baykal et al., 2019; Suzuki et al., 2018) which measures how much the trained network (cid:98) f can be compressed.
If the network can be compressed to much smaller one, then its intrinsic dimensionality can be regarded as small.
To describe it more precisely, suppose that the trained network (cid:98) f is included in a subset of the neural network model: (cid:98) f ∈ (cid:98) F ⊂ F .
For example, (cid:98) F can be a set of networks with weight matrices that have bounded norms and are near low rank (Sec. 3.1 or Sec. 3.2).
We do not assume a speciﬁc type of training procedure, but we give a uniform bound valid for any estimator (cid:98) f that falls into (cid:98) F and satisﬁes the following compressibility condition.
We suppose that the network (cid:98) f is easy to compress, that is, (cid:98) f can be compressed to a smaller network (cid:98) g which is included in a submodel: (cid:98) g ∈ (cid:98) G .
For example, (cid:98) G can be a set of networks with a smaller size than (cid:98) f .
How small the trained network (cid:98) f can be compressed has been characterized by several notions such as “layer-cushion” (Arora et al., 2018).
Typical compression based bounds give generalization errors of the compressed model (cid:98) g , not the original network (cid:98) f .
Our approach converts an error bound of (cid:98) g to that of (cid:98) f and eventually obtains a tighter evaluation.
The biggest difﬁculty for transforming the compression bound to that of (cid:98) f lies in evaluation of the population L 2 -norm between (cid:98) f and (cid:98) g .
Basically, the compression based bounds are given as
[Equation]
for a constant C > 0 under some assumptions (Table 1).
The term (cid:107) (cid:98) f − (cid:98) g
(cid:107) n appears to adapt the empirical error of (cid:98) f to that of (cid:98) g , that is called “compression error” which can be seen as a bias term.
We see that, in the right hand side, there appears the complexity of (cid:98) G which is assumed to be much smaller than that of the full model F .
However, the left hand side is not the expected error of (cid:98) f but that of (cid:98) g .
One way to transfer this bound to that of (cid:98) f is that we have | Ψ( (cid:98) g ) [Equation] Ψ [Equation]
[Equation] ≤ [Equation] g −
[Equation]
f (cid:107) Lby assuming Lipschitz continuity of the loss function and then convert the bound (1) to
[Equation]
However, to bound the term (cid:107) (cid:98) f
− (cid:98)
[Equation] (cid:107) [Equation]
− (cid:98)
[Equation] L 2 , there typically appears the complexity of the model (cid:98) F which is larger than the compressed model (cid:98) G like (cid:107) [-
-][Equation]
, which results in slow convergence rate.
To overcome this difﬁculty, we need to carefully control the difference between the training and test error of (cid:98) f and (cid:98) g by utilizing the local Rademacher complexity technique (Mendelson, 2002; Bartlett et al., 2005; Koltchinskii, 2006; Giné & Koltchinskii, 2006).
The local Rademacher complexity of a model F (cid:48) with radius r > 0 is deﬁned as
[Equation]
The main difference from the standard Rademacher complexity is that the model is localized to a set of functions satisfying (cid:107) f (cid:107) L 2 ≤ r .
As a result, we obtain a tighter error bound.
Throughout this paper, we always assume the following assumptions.
Let P X and P Y denote the marginal distribution of x and that of y respectively.
Assumption 1 (Lipschitz continuity of loss and activation functions) .
The loss function ψ is 1Lipschitz continuous with respect to the function output:
[Equation]
The activation function η is also 1-Lipschitz continuous:
[Equation] (cid:48) )
where d (cid:48) is any positive integer.
Assumption 2.
The norm of input is bounded by B x > 0 : (cid:107) x (cid:107) ≤ B x ( ∀ x ∈ supp( P X )) .
Assumption 3.
The L ∞ -norms of all elements in (cid:98) F and (cid:98) G are bounded by M ≥ 1 : (cid:107) f (cid:107) ∞ , (cid:107) g (cid:107) ∞ ≤
M for all f ∈ (cid:98) F and g ∈ (cid:98) G .
This assumption can be ensured by applying the clipping operator G to the output of the functions.
In this paper, all the variables L, m (cid:96) , R 2 , R F , M, B x are supposed to be o ( n ) .
What we will derive in the following is a bound which has mild dependency on the depth L and depends on the width ( m (cid:96) )
L(cid:96)
=1 in a sub-linear order by using the compression based approach.
Existing bounds for no-compressed network Here we give a brief review of the generalization error bound for non-compressed models.
(i) VC-bound:
The Rademacher complexity of the full model F can be bounded by a naive VC-dimension bound (Harvey et al., 2017) which is ¯ R n ( F ) =
O (cid:18)(cid:113) L (cid:80) n(cid:96)
=1 m (cid:96) m (cid:96) +1 n log( n ) (cid:19) .
In this bound, there appears the number of parameters (cid:80) n(cid:96)
=1 m (cid:96) m (cid:96) +1 in the numerator.
However, the number of parameters is often larger than the sample size n in practical use.
Hence, this bound is not appropriate to evaluate generalization ability of overparameterized networks.
(ii) Norm-based bound: Golowich et al.
(2018) showed the norm based bound which is given as ¯ R n ( F ) =
O (cid:18)(cid:113) LR L F n (cid:19) .
However, this is exponentially dependent on the depth as R L F resulting in quite loose bound.
Neyshabur et al.
(2017) showed a norm based bound of ¯ R n ( F ) =
O (cid:18)(cid:113)
[Equation] (cid:19) which avoids the exponential dependency.
However, there is still dependency on the width, (cid:80) (cid:96) m (cid:96) R 2 F , which is larger than the linear order of the width since R F could be moderately large.
Bartlett et al.
(2017b) showed ¯ R n ( F )
=
O (cid:32) R [Equation] [Equation] n (cid:18)
[Equation]
R 2 / 3(cid:19) 3 / 2 (cid:33) .
The norm constraint on R 2 → 1 implicitly assumes sparsity on the weight matrix and R 2 → 1 typically depends on the width linearly.
Wei & Ma (2019) improved the exponential dependency R L 2 appearing in this bound (Bartlett et al., 2017b) to obtained a bound
O (cid:18) 1 √ n (cid:16)
+ Lκ 43 R 2 / 3 2 → 1 + Lκ 23 R 2 / 3 1 → 1 (cid:17) 3 / 2 (cid:19) where κ is the Lipschitz continuity between layers.
We can see that R 22 → 1 and R 21 → 1 can depend on the width linearly and quadratically respectively even though R 2 is bounded.
C OMPRESSION BOUND FOR NONCOMPRESSED NETWORK
Here, we give a general theoretical tool that converts a compression based bound to that for the original network (cid:98) f .
We suppose the model classes (cid:98) F and (cid:98) G are ﬁxed independently on each data observation 2 .
For sets of functions, F (cid:48) and G (cid:48) , we denote the Minkowski difference of them by
F (cid:48) − G (cid:48) :
= { f − g | f ∈ F (cid:48) , g ∈ G (cid:48) } .
We assume that the local Rademacher complexity of (cid:98) F − (cid:98) G has a concave shape with respect to r > 0 : Suppose that there exists a function φ : [0 , ∞ ) → [0 , ∞ ) such that
[Equation]
This condition is not restrictive, and usual bounds for the local Rademacher complexity satisfy this condition (Mendelson, 2002; Bartlett et al., 2005).
Using this notation, we deﬁne r ∗ =
r ∗ ( t ) as
[Equation]
This is roughly given by the ﬁxed point of a function r 2 (cid:55)→ φ ( r ) , and it is useful to bound the ratio of the empirical L 2 -norm and the population L 2 -norm of an element h in (cid:98) F − (cid:98) G : (cid:107) h (cid:107)
L 2 / ( (cid:107) h (cid:107) 2 n + r 2 ∗ ) ≤ 1 / 2 with high probability.
Finally, we denote ψ ( F (cid:48) ) :
= { ψ ( · , f ( · )) | f ∈ F (cid:48) } for a set F (cid:48) of functions.
Then, we obtain the following theorem that gives the compression based bound for noncompressed networks.
Theorem 1.
Suppose that the empirical L 2 -distance between (cid:98) f and (cid:98) g is bounded by (cid:107) (cid:98) f − (cid:98) g (cid:107) n ≤ ˆ rfor a ﬁxed ˆ r > 0 almost surely.
Let ˙ r :=
(cid:112) 2(ˆ r 2 + r 2 ∗ ) , then, under Assumptions 1, 2, 3, there exists a universal constant C > 0 such that
[Equation]
with probability at least 1 − 3 e − t for all t ≥ 1 .
The proof is given in Appendix A.
The bound consists of two terms: “main term” and “bias term.”
The main term represents the complexity of the compressed model (cid:98) G which could be much smaller than (cid:98) F .
The bias term represents a sample complexity to bridge the original model and the compressed model.
Typically we have r 2 ∗
=
o (1 / √ n ) , and if we set ˆ r = o p (1) , then the bias term can be faster than the main term which is O (1 / √ n ) .
The term ˙ R ˙ r ( ψ ( (cid:98) F ) − ψ ( (cid:98) G )) can be reﬁned a little bit and the reﬁned term can be evaluated by using the covering number of the model.
The reﬁned version is given in Appendix A (Theorem 5).
This bound is general, and can be combined with the compression bounds derived so far such as Arora et al.
; Baykal et al.
; Suzuki et al.
(2018) where the complexity of (cid:98) G and the bias ˆ r are analyzed for their generalization error bounds.
The main difference from the compression bound (1) for (cid:98) g is that the bias term ˆ r = (cid:107) (cid:98) f − (cid:98) g (cid:107) n is replaced by 1 √ n (cid:107) (cid:98)
f − (cid:98) g
(cid:107) n which is √ n times smaller.
Since r 2 ∗ is typically o (1 / √ n ) and ˙ R ˙ r ( ψ ( (cid:98) F ) − ψ ( (cid:98) G )) can be made in the same order as the main term or even faster by setting ˆ r appropriately, we may neglect these terms.
Then, the bound is informally written as
[Equation]
This allows us to obtain tighter bound than the compression bound for (cid:98) g because the bias term ˆ r/ √ n is much smaller than ˆ r
and eventually we can let the variance term ¯ R n ( (cid:98) G ) much smaller by taking small compressed model (cid:98) G when we balance the bias and variance trade-off.
This is an advantageous point of directly bounding the generalization error of (cid:98) f instead of (cid:98) g .
Finally, we note that some existing bounds such as Arora et al.
; Bartlett et al.
(2017b)
; Wei & Ma (2019) assumes a constant margin so that the bias term can be a sufﬁciently small constant (which does not need to converge to 0).
On the other hand, our bound does not assume it and the bias term should converge to 0 so that the bias is balanced with the variance term, which is a more difﬁcult problem setting.
Example 1.
In practice, a trained network can be usually compressed to one with sparse weight matrix via pruning techniques (Denil et al., 2013; Denton et al., 2014).
Based on this observation,
Baykal et al.
(2019) derived a compression based bound based on a pruning procedure.
In this situation, we may suppose that (cid:98) G is the set of networks with S non-zero parameters where S is much smaller than the total number of parameters: (cid:98) G =
{ f ∈ NN
( m , R 2 , R F ) | (cid:80) L(cid:96)
=1 (cid:107) W ( (cid:96) )
(cid:107) 0 ≤ S } where (cid:107) W ( (cid:96) )
(cid:107)
is the number of nonzero parameters of the weight matrix W ( (cid:96) ) .
In this situation, its Rademacher complexity is bounded by ¯ R ( (cid:98) G ) ≤ CM (cid:113)
L Sn log( n ) (see Appendix B. for the proof).
This is much smaller than the VC-dimension bound (cid:113) L (cid:80) n(cid:96)
=1 m (cid:96) m (cid:96) +1 n log( n ) if
S (cid:28) (cid:80) n(cid:96)
=1 m (cid:96) m (cid:96) +1 .
Although our bound can be adopted to several compression based bounds, we are going to demonstrate how small the obtained bound can be through some typical situations in the following.
3. C OMPRESSION BOUND WITH NEAR LOW RANK WEIGHT MATRIX
Here, we analyze the situation where the trained network has near low rank weight matrices ( W ( (cid:96) ) )
L(cid:96)
=1 .
It has been reported that the trained network tends to have near low rank weight matrices experimentally (Gunasekar et al., 2018; Ji & Telgarsky, 2019) (see Appendix D for the empirical veriﬁcation).
This situation has been analyzed in Arora et al.
(2018) where the low rank property is characterized by their original quantities such as layer cushion.
However, we employ a much simpler and intuitive condition to highlight how the low rank property affects the generalization.
Assumption 4.
Assume that each of weight matrices W ( (cid:96) )
( (cid:96) = 1 , . . .
, L ) of any f ∈ (cid:98) F is near low rank, that is, there exists α > 1 / 2 and V 0 > 0 such that
[Equation]
where σ j ( W ) is the j -th largest singular value of a matrix W ( σ 1 ( W ) ≥ [Equation] ≥ · · · ≥ 0 ).
In this situation, we can see that for any 1 ≤ s ≤ min { m (cid:96) , m (cid:96) +1 } , we can approximate W ( (cid:96) ) by a rank s matrix W (cid:48) as (cid:107) W ( (cid:96) )
− W (cid:48) (cid:107) 2 ≤ V 0 s − α .
Let the set of networks with exactly low rank weight matrices be NN( m , s , R 2 , R F ) :
= { f ∈ NN( m , R 2 , R F ) | the weight matrix W ( (cid:96) ) of f has rank s (cid:96) } for s =
( s 1 , . . .
, s L ) .
If we set (cid:98) G = NN( m , s , R 2 , R F ) , then we have the following theorem.
Theorem 2.
The compressed model (cid:98) G = NN( m , s , R 2 , R F ) has the following complexity:
[Equation]
If (cid:98) F satisﬁes Assumption 4, we can set ˆ r = V 0 R L −
B x (cid:80) L(cid:96)
=1 s − α (cid:96) : for any (cid:98) f ∈ (cid:98) F , there exists (cid:98) g ∈ (cid:98) G such that (cid:107) (cid:98) f
− (cid:98) g
(cid:107) n ≤ ˆ r .
Then, letting A 1 = L (cid:80) L(cid:96)
=1 s (cid:96) ( m (cid:96) + m (cid:96) +1 ) n log( n ) and
[-(2 LV 0 R L − 
-]
[Equation]
, the overall generalization error is bounded by
[Equation]
with probability 1 − 3 e − t for any t > 1 where C > 0 is a constant depending on α .
See Appendix B.3 for the proof.
This indicates that, if α > 1 / 2 is large (in other words, each weight matrix is close to rank 1 ), then we have a better generalization error bound.
Note that the rank s (cid:96) can be arbitrary chosen and ˆ r and A 1 are in a trade-off relation.
Hence, by selecting the rank appropriately so that this trade-off is balanced, then we obtain the optimal upper bound as in the following corollary.
Corollary 1.
Under Assumption 4, using the same notation as Theorem 2, it holds that
[Equation]
with probability 1 − 3 e − t for any t > 1 where C is a constant depending on α .
An important point here is that the bound is O ( (cid:113) L (cid:80) L(cid:96)
=1 m (cid:96) n ) which has linear dependency on the width m (cid:96) in the square root, but the naive VC-dimension bound has quadratic dependency
O ( (cid:113) L (cid:80) L(cid:96)
=1 m (cid:96) m (cid:96) n ) .
In other words, the term in the square root has linear dependency to the number of nodes instead of the number of parameters .
This is huge gap because the width can be quite large in practice.
This result implies that a compressible model achieves much better generalization than the naive VC-bound.
In the generalization error bound, there appears R L 2 .
Even though R 2 can be much smaller than
R F , the exponential dependency R L 2 can give loose bound as pointed out in Arora et al.
This is due to a rough evaluation of the Lipschitz continuity between layers, but the practically observed Lipschitz constant is usually much smaller.
To ﬁx this issue, we give a reﬁned version of
Corollary 1 in Appendix B.4 by using data dependent Lipschitz constants such as interlayer cushion and interlayer smoothness introduced by Arora et al.
The reﬁned bound does not involve the exponential term R L 2 , but instead κ 2 ( κ : Lipschitz continuity) appears.
3. C OMPRESSION BOUND WITH NEAR LOW RANK COVARIANCE MATRIX
Strictly speaking, the near low rank condition on the weight matrix in the previous section can be dealt with a standard Rademacher complexity argument.
Here, we consider more data dependent bound: We assume the near low rank property of the covariance matrix among the nodes in an internal layer (see Appendix D for the empirical veriﬁcation).
A compression based bound for (cid:98) g using the low rank property of the covariance has been studied by Suzuki et al.
, but their analysis requires a bit strong condition on the weight matrix.
In this paper, we employ a weaker assumption.
Let (cid:98) Σ ( (cid:96) )
= 1 n (cid:80) ni =1 φ (cid:96) ( x i ) φ (cid:96) ( x i ) (cid:62) be the covariance matrix of the nodes in the (cid:96) -th layer where φ (cid:96) [Equation] = η ◦
[Equation]
Assumption 5.
Suppose that the trained network (cid:98) f satisﬁes the following conditions:
[Equation]
for a ﬁxed β > 1 and U 0 > 0 .
If (cid:98) f satisﬁes this assumption, then we can show that (cid:98) f can be compressed to a smaller one f (cid:93) that has width ( m (cid:93)(cid:96) )
L(cid:96)
=2 with compression error roughly evaluated as (cid:107) (cid:98) f − f (cid:93) (cid:107) n (cid:46) (cid:80) (cid:96) ( m (cid:93)(cid:96) )
− β/ 2 .
More precisely, for given ˜ r (cid:96) > 0 ( (cid:96) = 1 , . . .
, L ) which corresponds to the compression error in the (cid:96) - th layer, let ˙ m (cid:96) :
= max { 1 ≤ j ≤ m (cid:96) | ˙ µ ( (cid:96) )
j ≥ ˜ r 2 (cid:96) / 4 } .
Then, we deﬁne N (cid:96) ( ˜r ) =
β +1 β − 1 ˙ m (cid:96) + [Equation]
R F ˜ [Equation] 2 ˜
[Equation] [Equation] , for ˜r =
(˜ r 1 , . .
˜ r L ) .
Correspondingly we set
[Equation]
Then, we obtain the following theorem.
Theorem 3.
Let ˆ r := (cid:80)
Lk =1 R
( L − k )
R F ˜ r k .
Then, under Assumption 5, there exists (cid:98) g with width m (cid:93)
=
( m 1 , m (cid:93) 2 , . . .
, m (cid:93)L ) that satisﬁes (cid:98) g ∈ NN( m (cid:93) , (cid:113) 203 max (cid:96) m (cid:96) R 2 (cid:113) 203 max (cid:96) m (cid:96) R F ) and (cid:107) (cid:98) f − (cid:98) g
(cid:107) n ≤ ˆ r.
In particular, we may set (cid:98) G =
NN
( m (cid:93) , (cid:113) 203 max (cid:96) m (cid:96) R 2 , (cid:113) 203 max (cid:96) m (cid:96) R F ) , and then it holds that ¯ R n ( (cid:98) [Equation]
(cid:93)(cid:96) [Equation] (cid:93)(cid:96) n log( n ) for a constant C > 0 .
See Appendix B.5 for the proof.
Here, we again observe that there appears a trade-off between ˆ r and m (cid:93)(cid:96) because as ˜ r (cid:96) becomes small, then ˙ m (cid:96) becomes large and thus m (cid:93)(cid:96) becomes large.
The evaluation given in Theorem 3 can be substituted to the general bound (Theorem 1).
If (cid:98) F is the full model F , then there appears the number (cid:80) L(cid:96)
=1 m (cid:96) m (cid:96) +1 of parameters which could be larger than n , which is unavoidable.
This dependency on the number of parameters becomes much milder if both of Assumptions 4 and 5 are satisﬁed.
Theorem 4.
Under Assumptions 4 and 5, it holds that Ψ
[Equation]
f
[Equation]
≤
[Equation]
for P L =
(2 LV 0 R L −
B x ) 1 /α and Q L =
[Equation]
with probability 1 − 3 e − t ( t > 1) , where C is a constant depending on α, β .
If we omit L and log( n ) terms for simplicity of presentation, then the bound can be written as
[Equation]
where the ˜ O ( · ) symbol hides the poly-log order.
This is tighter than that of Corollary 1.
We can see that as β and α get large, the bound becomes tighter.
Actually, by taking the limit of α, β → ∞ , then the bound goes to L 2 (cid:113) log( n ) n
[Equation]
Moreover, the term dependent on the width is O (1 /n ) with respect to the sample size n which is faster than the rate O ( (cid:113) (cid:80) L(cid:96)
[Equation]
n ) which was presented in
Corollary 1.
Hence, the low rank property of both the covariance matrix and the weight matrix helps to obtain better generalization.
Although the bound contains the exponential term R L 2 , we can give a reﬁned version that does not contain the exponential term by assuming interlayer cushion (Arora et al., 2018).
See Appendix B.6 for the reﬁned version.
There appears exp( 14 (2 √ L − 1)) which is exponentially dependent on L .
However, this term is moderately small for realistic settings of the depth L .
Actually, it is 7.27 for L = 20 and 26.7 for
L = 50 (we can replace this term in exchange for larger polynomial dependency on L ).
The bound is not optimized with respect to the dependency on the depth L .
In particular, the term L 2 (cid:112) log( n ) /n could be an artifact of the proof technique and the L 2 term would be improved.
Finally, we compare our bound with the following norm based bounds; O
[Equation]
by Bartlett et al.
(2017b)
and O (cid:18)
[Equation]
by Wei & Ma (2019).
Since our bound and their bounds are derived from different conditions, we cannot tell which is better.
Here, we consider a special case where m (cid:96) = m ( ∀ (cid:96) ) and W ( (cid:96) )
= 1 m 11 (cid:62) ∈
R m × m which is an extreme case of low rank settings (note that W ( (cid:96) ) has rank 1).
Then, R 2 = 1 , R F = 1 ,
R 2 → 1 = √ m and R 1 → 1 = m , and thus their bounds are O ( (cid:112) m/n ) and O ( (cid:112) ( m + m 2 ) /n ) respectively.
However, β and α in our bound (cid:113) m 4 /β 4 /β
+2(1 − 1 / 2 α ) /n = (cid:113) m 1 1+ β (1 − 1 / 2 α ) /
2 /n can be arbitrary large in this situation, so that our bound has much milder dependency on the width m .
On the other hand, if the weight matrix has small norm and has no spectral decay (corresponding to small α and β ), then our bound can be looser than theirs.
Combining compression based bounds and norm based bounds would be interesting future work.
C ONCLUSION
In this paper, we derived a compression based error bound for non-compressed network.
The bound is general and it can be adopted to several compression based bound derived so far.
The main difﬁculty lies in evaluating the population L 2 -norm between the original network and the compressed network, but it can be overcome by utilizing the data dependent bound by the local Rademacher complexity technique.
We have applied the derived bound to a situation where low rank properties of the weight matrices and the covariance matrices are assumed.
The obtained bound gives much better dependency on the parameter size than ever obtained compression based ones.
Acknowledgment We thank the anonymous reviewers for their valuable comments.
TS was partially supported by MEXT Kakenhi (15H05707, 18K19793 and 18H03201), Japan Digital Design, and JST-CREST, Japan .
P ROOF OUTLINE OF T HEOREM
Remember that for a trained network (cid:98) f , (cid:98) g is its compressed version that is included in a submodel (cid:98) G . First, we decompose the generalization gap as [Equation]
The second block in the right hand side is easy to bound, i.e., by applying the standard Rademacher complexity bound (Theorem 3.1 of Mohri et al.
with the contraction inequality (Theorem 11.6 of Boucheron et al.
(2013) or Theorem 4.12 of Ledoux & Talagrand (1991))
, it holds that
[Equation]
with probability 1 − e − t .
Since (cid:98) G is a small model, this bound could be much smaller than a naive
VC-dimension bound.
The ﬁrst block “bridges” the generalization gap of (cid:98) g to that of (cid:98) f , but bounding the ﬁrst block is more involved.
We make use of the local Rademacher complexity to bound the term.
Suppose that the event in which (cid:107) (cid:98) f − (cid:98) g (cid:107) L 2 ≤ ˙ r holds has high probability (which should be proven later), then it is also expected that ψ ( y, f ) − ψ ( y, g ) has small L 2 -norm.
This is true because of the
Lipschitz continuity assumption (Assumption 1).
Actually, the Talagrand’s concentration inequality yields that
[Equation]
for Φ
[Equation]
r ) :=
[Equation]
) with probability 1 − e − t .
Since Φ( ˙ r ) requires the restriction (cid:107)
f − g (cid:107) L 2 ≤ ˙ r , this quantity is much smaller than the standard
Rademacher complexity ¯ R n ( ψ ( (cid:98) F ) − ψ ( (cid:98) G )) , which yields fast convergence rate.
Finally, we should bound the probability of (cid:107) (cid:98) f − (cid:98) g (cid:107) L 2 ≤ ˙ r .
This can be bounded by utilizing the ratio type empirical process.
Actually, we can show that
P
[Equation]
sup
[Equation]
for r ∗ deﬁned in Eq.
(2), where P n f := 1 n (cid:80) ni =1 f
( z i ) and Pf = E
[ f ( Z )] .
This yields that (cid:107) (cid:98) f − (cid:98) g (cid:107) 2 L 2 −(cid:107) (cid:98) f − (cid:98) g (cid:107) 2 n ≤ 12 ( (cid:107) (cid:98) f − (cid:98) g (cid:107) 2 L 2 + r 2 ∗ ) with probability 1 − e − t and equivalently (cid:107) (cid:98) f − (cid:98) g (cid:107) 2 L 2 ≤ 2( (cid:107) (cid:98) f − (cid:98) g (cid:107) 2 n + r 2 ∗ ) .
Since (cid:107) (cid:98) f − (cid:98) g (cid:107) 2 n ≤ ˆ r 2 a.s., we have (cid:107) (cid:98) f − (cid:98) g
{+(cid:107) 
+}
L 2 ≤ 2(ˆ r 2 + r 2 ∗ ) = ˙
r 2 .
We can show that Φ( ˙ r ) ≤ ˙ R ˙ r ( ψ ( (cid:98) F ) − ψ ( (cid:98) G )) by using the Lipschitz continuity assumption (Assumption 1).
Then, we obtain the assertion.
N OTATION LISTS
Since we use plenty of notations, we give the notation list in Table 2.
A PPENDIX
In the appendix, we give the proofs of the main text.
We use the following notation throughout the appendix:
[Equation]
To evaluate it, the covering number is useful (van der Vaart & Wellner, 1996).
Deﬁnition 1 (Covering number) . For a metric space ˜ F equipped with a metric ˜ d , the (cid:15) -covering number N ( ˜ F , ˜ d, (cid:15) ) is deﬁned as the minimum number of balls with radius (cid:15) (measured by the metric ˜ d ) to cover the metric space ˜ F .
Hereafter, C denotes a constant which will be dependent on the context.
We let a ∧ b := min { a, b } and a ∨ b := max { a, b } for a, b ∈ R .
A P ROOF OF T HEOREM
Denote the local Rademacher complexity of { ψ ( f ) − ψ ( g ) | f ∈ (cid:98) F , (cid:98) g ∈ (cid:98) G , (cid:107) f − g (cid:107) L 2 ≤ r } by Φ( r ) :
= ¯ R n ( { ψ ( f ) − ψ ( g ) | f ∈ (cid:98) F , (cid:98) g ∈ (cid:98) G , (cid:107)
f − g (cid:107) L 2 ≤ r } ) .
Here, we restate Theorem 1 in the following in more complete form. Remember that (cid:98) f ∈ (cid:98) F and (cid:98) g ∈ (cid:98) G are the trained original network and the compressed network respectively.
[Table]
[Equation]
[Equation] [Table]
[Table] [Equation] the whole [Table]
networks
[Table]
width
[Table]
[Equation]
set
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
an upper bound of the j -th largest eigenvalue of the covariance matrix in the (cid:96) -th layer of (cid:98) f
[Table]
decreasing rate of the eigenvalues of W ( (cid:96) )
[Table] decreasing rate of the eigenvalues of (cid:98) Σ ( (cid:96) ) Theorem 5. Suppose that the empirical L 2 -distance between (cid:98) f and (cid:98) g is bounded by (cid:107) (cid:98) f − (cid:98) g (cid:107) n ≤ ˆ rfor a ﬁxed ˆ r > 0 almost surely.
Let ˙ r :=
(cid:112) 2(ˆ r 2 + r 2 ∗ ) , then, under Assumptions 1, 2, 3, there exists a universal constant C > 0 such that
[Equation]
with probability at least 1 − 3 e − t for all t ≥ 1 .
Note that ˙ R ( ψ ( (cid:98) F ) − ψ ( (cid:98) G )) in the statement of Theorem 1 of the main body is replaced by reﬁned quantity Φ( ˙ r ) (we can show Φ( ˙ r ) ≤ ˙ R ( ψ ( (cid:98) F ) − ψ ( (cid:98) G ))
from the Lipschitz continuity of ψ ). Proof of Theorems 1 and 5.
First, by the standard Rademacher complexity analysis, we have that
[Equation]
[Equation] with probability 1 − e − t , where we used the Rademacher concentration inequality (Theorem 3. of Mohri et al.
in the third line and the contraction inequality (Theorem 11.6 of Boucheron et al.
(2013) or Theorem 4.12 of Ledoux & Talagrand (1991) and its proof) in the last line.
We let this event be E 0 ( t ) .
Next, we observe that
[Equation]
where we used Eq.
(5) in the last line.
Here, it should be noticed that it is not a good strategy to bound the ﬁrst term Ψ
( (cid:98) f ) − Ψ( (cid:98) g ) − (ˆΨ( (cid:98) f ) − ˆΨ( (cid:98) g )) by bounding Ψ( (cid:98) f ) − ˆΨ( (cid:98) f ) and Ψ( (cid:98) g ) − ˆΨ( (cid:98) g ) independently.
Instead, we should bound them simultaneously to obtain tighter bound.
This can be accomplished by using the local Rademacher complexity technique.
Note that (cid:98) f and (cid:98) g are date dependent and we can only bound the empirical L 2 -distance between them.
On the other hand, the local Rademacher complexity is characterized by the population L 2norm.
To bridge this gap, we need to bound the population L 2 -distance (cid:107) (cid:98) f − (cid:98) g (cid:107) L 2 between (cid:98) f and (cid:98) g in terms of the empirical L 2 -norm bound (cid:107) (cid:98) f − (cid:98) g (cid:107) n ≤ ˆ r .
To do so, we also bound the local Rademacher complexity of (cid:98) F − (cid:98) G : ˙
R r
( (cid:98) F − (cid:98) G ) .
Suppose that there exists a function φ : [0 , ∞ ) → [0 , ∞ ) such that the the following conditions are satisﬁed:
[Equation]
and
[Equation]
Note that Eq.
(11) gives one example of φ ( r ) .
Then, by the so-called peeling device , we can show that for any r > 0 ,
[Equation]
for all t > 0 (Theorem 7.7 and Eq.
(7.17) of Steinwart & Christmann (2008)).
Hence, if we choose r ∗ = r ∗ ( t ) so that
[Equation]
then it holds that
[Equation]
uniformly over all h ∈ (cid:98) F − (cid:98) G with probability greater than 1 − e − t .
We let this event as E 1 ( t ) .
In this event, if (cid:107) (cid:98) f − (cid:98) g
[Equation]
2 n ≤ ˆ r 2 , then it holds that
[Equation]
Next, we bound Ψ( (cid:98) f ) − Ψ
( (cid:98) g ) − (ˆΨ( (cid:98) f ) − ˆΨ( (cid:98) g )) .
To bound this term, we apply the Talagrand’s concentration inequality (Proposition 2 and Talagrand (1996);
Bousquet (2002)).
To apply it, we should bound the variance and L ∞ -norm of ψ ( y, f ( x ))
− ψ ( y, g ( x )) − (E[ ψ ( Y, f ( X ))] − E[ ψ ( Y, g ( X ))]) for any f ∈ (cid:98) F , g ∈ (cid:98) G with (cid:107)
f − g (cid:107)
L 2 ≤ r (where r will be set 2(ˆ
r 2 + r 2 ∗ ) ).
Due to the Lipschitz continuity of ψ , we have that
[Equation]
Similarly, it holds that
[Equation]
Hence
[Equation]
by the Talagrand’s concentration inequalit (Proposition 2 and Talagrand (1996) ; Bousquet (2002)) [Equation] it holds that
[Equation]
with probability at least 1 − e − t for any t > 0 .
The ﬁrst term in the right hand side can be bounded as
[Equation]
where we used the standard symmetrization argument (see Lemma 11.4 of Boucheron et al.
for example).
Combining these inequalities, it holds that
[Equation]
for a universal constant C > 0 with probability at least 1 − e − t for all t > 0 .
We denote by this event as E 2 ( t, r ) .
We deﬁne an event E 3 ( t ) =
E 1 ( t ) ∩ E 2 ( t, ˙ r ) .
Then P ( E 3 ( t )) ≥ 1 − 2 e − t for all t > 0 .
In this event, (cid:107) [Equation] f −
[Equation]
g (cid:107) 2 L 2 ≤ ˙ r 2 and thus it holds that
[Equation]
for a universal constant C > 0 .
Combining this and Eq.
(6), we obtain the assertion on the event
E 0 ( t ) ∩ E 3 ( t ) .
This gives the proof of Theorem 5.
To show Theorem 1, note that (cid:107) ψ ( f ) − ψ ( g ) (cid:107) L 2 ≤ (cid:107)
f − g
(cid:107) L 2 by the Lipschitz continuity of ψ and this yields { ψ ( f ) − ψ ( g ) | f ∈ (cid:98) F , g ∈ (cid:98) G , (cid:107) f − g (cid:107) L 2 ≤ r } ⊂ { ψ ( f ) − ψ ( g ) | f ∈ (cid:98) F , g ∈ (cid:98) G , (cid:107) ψ ( f ) − ψ ( g ) (cid:107) L 2 ≤ r } .
Therefore, we have
[Equation]
Hereafter, we derive some upper bounds of the (local) Rademacher complexities under some covering number conditions.
Lemma 1.
For a given r > 0 , let ˆ γ n
[Equation]
Then, it holds that
[Equation]
and
[Equation]
where C, C (cid:48) are universal constants
[Equation]
Proof.
The conditional Rademacher complexity of the set { ψ ( y, f ( x )) − ψ ( y, g ( x )) | f ∈ (cid:98) F , g ∈ (cid:98) G , (cid:107) f − g (cid:107) L 2 ≤ r } can be bounded by a constant times the following Dudley integral (see Theorem 5.22 of Wainwright (2019) or Lemma A.5 of Bartlett et al. (2017a) for example): [Equation] n
[Equation]
where we used (cid:107) ψ ( f ) − ψ ( g ) − ( ψ ( f
[Equation]
) − ψ
[Equation]
g
[Equation]
[Equation]
n ≤
[Equation]
ψ
[Equation]
f
[Equation]
− ψ
[Equation]
f
[Equation]
[Equation]
n
[Equation]
ψ ( g ) − ψ
[Equation]
g
[Equation]
) (cid:107) n ≤ (cid:15) for f, f (cid:48) ∈ (cid:98) F and g, g (cid:48) ∈ (cid:98) G with (cid:107)
f − f (cid:48) (cid:107) n ≤ (cid:15)/ 2 and (cid:107)
g − g (cid:48) (cid:107) n ≤ (cid:15)/ 2 in the third line, and 1-Lipschitz continuity of the loss function ψ in the fourth line (i.e., | ψ ( y, f ( x ))
− ψ ( y, g ( x )) | ≤ | f ( x ) − g ( x ) | which yields (cid:107)
[Equation]
In the same way, we can see that ˙ R r ( (cid:98) F − (cid:98) G ) is bounded by the Dudley integral as
[Equation]
where we used the same argument as Eq.
(10) and C > 0 is a universal constant.
Then, we conclude
Eq.
Next, we show Eq.
The term Eq.
(9) can be evaluated by using the local Rademacher complexity of ψ ( (cid:98) F ) − ψ ( (cid:98) G ) .
Note that (cid:107) ψ ( f ) − ψ ( g ) (cid:107) L 2 ≤ (cid:107)
f − g
(cid:107) L 2 by the Lipschitz continuity of ψ and this yields { ψ ( f ) −
ψ ( g ) | f ∈ (cid:98) F , g ∈ (cid:98) G , (cid:107)
f − g (cid:107) L 2 ≤ r } ⊂ { ψ ( f ) − ψ ( g ) | f ∈ (cid:98) F , g ∈ (cid:98) G , (cid:107) ψ ( f ) − ψ ( g ) (cid:107)
L 2 ≤ r } .
Then, the Sudakov’s minoration (Corollary 4.14 of Ledoux & Talagrand (1991)) gives an upper bound of the right hand side of Eq.
[Equation]
ˆ γ n 1 /n
[Equation]
n
[Equation]
[Equation]
n d
[Equation]
[Equation]
≤
[Equation]
where ˆ R n,r ( ψ ( (cid:98) F )
[Equation]
ψ ( (cid:98)
[Equation]
=
[Equation]
Since ˆ γ n ≤ 2 M , the expectation of the right hand side with respect to D n is ˙ R r
( ψ ( (cid:98) F ) − ψ ( (cid:98) G ))
(cid:112) log( n ) log(2 nM ) .
This gives an upper bound of the right hand side of
Eq.
(9) and yields Eq.
Lemma 2.
[Equation]
Proof.
By the contraction inequality of the Rademacher complexity (Theorem 4.12 of Ledoux &
Talagrand (1991) and its proof), we have
[Equation]
r }
[Equation]
(symmetrization; Lemma 11.4 of Boucheron et al.
[Equation]
inequality
[Equation]
Lemma 3.
Suppose that
[Equation]
for q < 1 .
Then, for a universal constant C > 0 and a constant C q > 0 which depends on q <
[Equation]
it holds that
[Equation]
In particular,
[Equation]
Proof.
Remember that ˆ γ n := sup {(cid:107) f − g (cid:107) n : (cid:107) f − g (cid:107) L 2 ≤ r, f ∈ (cid:98) F , g ∈ (cid:98) G} for a given r > 0 . Under the assumption
[Equation]
we may take φ ( r ) (deﬁned below) as an upper bound of Φ( r ) by Lemma 1:
[Equation]
where C > 0 is a universal constant. The right hand side can be evaluated as E
[Equation]
[Equation]
where we used Lemma 2. Hence
[Equation]
if the ﬁrst term is larger than the second term
[Equation]
we have that
[Equation] Therefore
[Equation]
we obtain that
[Equation]
On the other hand, if the second term in Eq.
(12) is larger than the ﬁrst one, then Young’s inequality gives that
[Equation]
where c (cid:48) > 0 is any positive real.
Thus taking c (cid:48) sufﬁciently large (which depends on M, q ), we conclude that
[Equation]
where C q is a constant depending only on q < 1 .
These two inequalities (Eq.
(13) and Eq.
(14)) give the ﬁrst assertion.
By noticing the assumption M ≥ 1 , r ∗ can be derived from a simple calculation.
B D ERIVATION OF COMPRESSION BASED BOUND FOR NON - COMPRESSED NETWORKS B. F ULL MODEL BOUND
Here, we assume that the model (cid:98) F of the trained network is the full model (cid:98) F = F =
NN( m , R 2 , R F ) and (cid:98) G is included in NN( m , ˆ R 2 , ˆ R F ) .
Then, their covering entropy is bounded by
[Equation]
Hence, the condition in Lemma 3 holds for S 1 = (cid:80) L(cid:96)
[Equation]
m
[Equation]
m (cid:96) +1 , S 2 = LS 1 log( L ( R 2 ∨ ˆ R 2 ∨ 1)(max (cid:96) m (cid:96) + 1)) and S 3 = 0 .
In this case, we can set
[Equation]
for a constant C > 0 .
B. C OMPLEXITY OF A SPARSE MODEL (P ROOF OF E XAMPLE 1)
Suppose that (cid:98) G is the model with sparse weight matrices given in Example 1.
Let m = max (cid:96) m (cid:96) and B = R 2 , then we can see that [Equation] where the deﬁnition of Φ( L, m, S, B ) is given in Appendix C.1.
Therefore, its covering number is bounded by
[Equation]
by Lemma 4.
Hence, the Rademacher complexity is bounded as
[Equation]
by the Dudley integral, ¯ R ( (cid:98) G
[Equation]
d (cid:15) , where C is a universal constant.
B. N EAR LOW RANK CONDITION ON THE WEIGHT MATRIX (P ROOF OF T HEOREM 2 AND C OROLLARY 1 )
Here, we give proofs of Theorem 2 and Corollary 1 which give a generalization error bound when the trained network has near low rank weight matrices ( W ( (cid:96) ) )
L(cid:96)
=
1 (Assumption 4).
Under Assumption 4, we can see that for any 1 ≤ s ≤ min { m (cid:96) , m (cid:96) +1 } , we can approximate W ( (cid:96) ) by a rank s matrix W (cid:48) as
[Equation]
[Equation]
This can be checked by discarding the singular vectors corresponding to the singular values smaller than the s -th largest one.
This ensures that, for any f ∈ (cid:98) F , there exists f (cid:48)
∈ F such that it has width s =
( s 1 , . . .
, s L ) , weight matrix W (cid:93) (
[Equation]
) with (cid:107) W (cid:93) ( (cid:96) ) (cid:107) 2 ≤ R 2 , (cid:107) W (cid:93) ( (cid:96) ) (cid:107) F ≤ R F and (cid:107)
[Equation]
This can be proved as follows.
Let f (cid:48)
( x ) =
G ◦ ( W (cid:93) ( L ) η ( · )) ◦ · · · ◦ ( W (cid:93) (1) x ) where W (cid:93) ( (cid:96) ) is a rank s (cid:96) matrix that satisﬁes Eqs.
(15) and (16) for W (cid:48)
=
W (cid:93) ( (cid:96) ) .
Let f (cid:96) ( x ) =
G ◦ ( W (cid:93) ( L ) η ( · )) ◦· · ·◦ ( W (cid:93) ( (cid:96) +1) η
( · )) ◦ ( W ( (cid:96) )
η ( · )) ◦· · ·◦ ( W (1) x ) and f 0 ( x ) = f (cid:48) .
Then, (cid:107) f − f (cid:48) (cid:107) ∞ ≤ (cid:80) L(cid:96)
=1 (cid:107) f
(cid:96) − f (cid:96) − 1 (cid:107) ∞ . We can see that (cid:107) ( W ( (cid:96) ) η ( · )) ◦· · ·◦ ( W (1) x ) (cid:107) ≤ (cid:81) (cid:96)k =1 (cid:107) W ( k ) (cid:107) 2 (cid:107) x (cid:107) ≤ R (cid:96) 2 B x , (cid:107) ( W (cid:93) ( (cid:96) ) η ( · )) ◦ ( W ( (cid:96) − 1) η ( · )) ◦ · · · ◦ ( W (1) x ) − ( W ( (cid:96) )
η ( · )) ◦ ( W ( (cid:96) − 1) η ( · )) ◦ · · · ◦ ( W (1) x ) (cid:107) ≤ (cid:107) W ( (cid:96) ) − W (cid:93) ( (cid:96) ) (cid:107) 2 (cid:107) ( W ( (cid:96) − 1) η ( · )) ◦ · · · ◦ ( W (1) x ) (cid:107) ≤ V 0 s − α (cid:96) R
{+(cid:96) − 
+}
B x .
This gives (cid:107) f (cid:96) − f (cid:96) (cid:107) ∞ ≤ (cid:81)
{+Lk = (cid:96) +1 (cid:107) W ( k ) (cid:107) 2 V 0 s − α (cid:96) R (cid:96) − 
+}
B {+x ≤ R L − 
+}
V 0 s − α (cid:96) B x .
Finally, by summing up this from (cid:96) = 1 to (cid:96) =
L , we obtain Eq.
In particular, for any (cid:15) > 0 , by setting s (cid:48) (cid:96) =
s (cid:48) (cid:96) ( (cid:15) )
=
{+min {(cid:100) ( (cid:15)/ ( LV 0 R L − 
 B x )) − 1 /α (cid:101) , m (cid:96)+}
∧ m (cid:96) +1 } for all (cid:96) , then (cid:107) f − f (cid:48) (cid:107) ∞ ≤ (cid:15) .
This indicates that, by Lemma 5, the covering entropy of (cid:98) F is bounded by
[Equation]
[Equation]
≤ 2(
L (cid:88)
[Equation]
As the compressed network (cid:98) G , we may choose (cid:98) G = NN( m , s , R 2 , R F ) for s = ( s 1 , . .
, s L ) so that, for all f ∈ (cid:98) F , there exists g ∈ (cid:98) G satisfying
[Equation]
Hence, we may set ˆ r 2 =
[( V 0 R L −
B x
[Equation]
(cid:80) L(cid:96)
=1 s − α (cid:96) ] 2 .
In this case, the covering number of (cid:98) G is bounded as (18) by replacing s (cid:48) (cid:96) with s (cid:96) .
Therefore, Lemma 3 gives that
[Equation]
where
S
[Equation]
where q = 1 / 2 α was used.
This indicates that, if α > 1 / 2 is large (in other words, each weight matrix is close to rank 1 ), then the local Rademacher complexity can be small.
Actually, the bound is smaller than (cid:80) L(cid:96)
[Equation] 1 m (cid:96)
[Equation]
(cid:96)
[Equation]
n because each rank s (cid:96) must satisfy s (cid:96) ≤ min { m (cid:96) , m (cid:96) +1 } . Finally, we observe that
[Equation]
by Lemma 5 for (cid:98) G = NN ( m , s , R 2 , R F ) and the Dudley integral (van der Vaart & Wellner, 1996):
[Equation]
d (cid:15) .
This gives Theorem 2.
Corollary 1 can be obtained by substituting s
[Equation]
B. I MPROVED BOUND WITH L IPSCHITZ CONTINUITY CONSTRAINT
In the generalization error bound of Theorem 2 and Corollary 1, there appears R L 2 .
Even though
R 2 can be much smaller than R F , the exponential dependency R L 2 could give lose bound as pointed out in Arora et al.
We improve this exponential dependency by assuming the following condition.
Assumption 6 (Lipschitz continuity between layers: Interlayer cushion, interlayer smoothness (Arora et al., 2018)) .
For the trained network (cid:98) f
= G ◦ ( W ( L ) η ( · )) ◦ · · · ◦ ( W (1) x ) , let φ (cid:96) ( x ) =
η ◦ ( W ( (cid:96) − 1) η ( · )) ◦· · ·◦ ( W (1) x ) be the input to the (cid:96) -th layer and M (cid:96),(cid:96) (cid:48)
( x ) = ( W ( (cid:96) (cid:48) )
η ( · )) ◦· · ·◦ ( W ( (cid:96) )
x ) be the transformation from the (cid:96) -th layer to (cid:96) (cid:48) -th layer.
Then, we assume that there exists κ, τ >such that τ ≤ 1 / (2 κ 2 L ) and for any (cid:96), (cid:96) (cid:48) ∈ [ L ] , n (cid:88) i = [ M (cid:96)
[Equation]
(cid:96)
[Equation]
( φ (cid:96) (
[Equation]
i
[Equation]
+
[Equation]
[Equation]
i [Equation] M [Equation]
(cid:96) (cid:48)
[Equation]
φ (cid:96)
[Equation]
x i ) + ξ (1) i [Equation] +
[Equation]
(cid:17) 2 , for all ξ (1) =
( ξ (1) 1 , . .
, ξ (1) n ) (cid:62) ∈ R n and ξ (2) =
( ξ (2) 1 , . .
. , ξ (2) n ) (cid:62) ∈ R n .
This assumption is a simpliﬁed version of the interlayer cushion and the interlayer smoothness introduced in Arora et al.
Although a trivial bound of κ is κ ≤ R L 2 , the practically observed
Lipschitz constant is usually much smaller.
Assumption 6 captures this point and gives better dependency on the depth L .
Actually, we can remove the exponential dependency on R 2 as in the following corollary.
Corollary 2.
Under Assumptions 4 and 6, it holds that
[Equation]
for A
[Equation]
=
[Equation]
with probability 1 − 3 e − t for any t > 1 where C is a constant depending on α .
This is almost same as Corollary 1, but the exponential dependency on R L 2 is replaced by the Lipschitz continuity κ 2 .
Proof of Corollary 2.
Suppose that
[Equation]
then we show that Eq.
(17) can be replaced by
[Equation]
where if s (cid:96) = min { m (cid:96) , m (cid:96) +1 } , then s − α (cid:96) term can be replaced by 0 (which means no-compression in the layer (cid:96) ).
Once we obtain this evaluations, then the following argument is same as the proof of
Theorem 2 and Corollary 1 (Sec. B.3).
Let φ (cid:96) ( x ) =
η ◦ ( W ( (cid:96) )
η ( · )) ◦ · · · ◦ ( W (1) x ) and φ (cid:93)(cid:96) ( x ) =
η ◦ ( W (cid:93) ( (cid:96) )
η ( · )) ◦ · · · ◦ ( W (cid:93) (1) x ) for (cid:96) = 1 , . . .
, L − 1 , and let φ L ( x ) =
G ◦ ( W ( L ) η ( · )) ◦ · · · ◦ ( W (1) x ) and φ (cid:93)L ( x ) =
G ◦ ( W (cid:93) ( L ) η ( · )) ◦ · · · ◦ ( W (cid:93) (1) x ) .
Let C B :=
2 κB x .
We will show that
[Equation]
for all k = 1 , . . .
, L .
We show this by inductive reasoning.
To do so, we assume that, for k = 1 , . . .
, (cid:96) − 1 , this is satisﬁed, and then we show this for k = (cid:96) .
Note that, for all k with k < (cid:96) , it holds that, for any (cid:96) (cid:48) > k ,
[Equation]
the assumption of τ ) [Equation]
Note that the term s − α k can be replaced by 0 if s k = min { m k , m k +1 } which corresponds to the full rank setting ( W (cid:93) ( k )
=
W ( k ) ).
Therefore, we have that
[Equation]
Under the setting (19), this gives that
[Equation]
Finally, noting that (cid:107) φ (cid:96) (cid:107) n ≤ C B / 2 , we have (cid:107)
[Equation]
n ≤ C B .
This concludes the inductive reasoning.
Finally, noting that f = φ L and f (cid:48) =
φ (cid:93)L , we have Eq.
B. N EAR LOW RANK CONDITION ON THE COVARIANCE MATRIX (P ROOF OF T HEOREMAND T HEOREM 4)
Under Assumption 5, (cid:98) f can be compressed as follows.
Suppose that the network is compressed to smaller one upto the (cid:96) − 1 -th layer and the weight matrix of the compressed one is denoted by ( W (cid:93) ( k ) )
(cid:96) − 1 k =1 where each W (cid:93) ( k ) has size m (cid:93)k +1 × m (cid:93)k (here, m (cid:93)k ≤ m k is assumed), and, in the (cid:96) − 1 -th layer, W (cid:93) ( (cid:96) − 1) has size m (cid:96) × m (cid:93)(cid:96) − 1 .
The input to the (cid:96) -th layer of the compressed networkis denoted by φ (cid:93)(cid:96) ( x ) =
η ( W (cid:93) ( (cid:96) − 1) η
( · · · W (cid:93) (1) x ) · · · ) .
Let r 2 (cid:96) =
(cid:107)(cid:107) φ (cid:96) − φ (cid:93)(cid:96) (cid:107)(cid:107) 2 n and Σ (cid:93) ( (cid:96) ) :
= 1 n (cid:80) ni =1 φ (cid:93)(cid:96) ( x i )( φ (cid:93)(cid:96) ( x i ))
(cid:62) .
For a given matrix Σ and a precision r 2 > 0 , the degrees of freedom 3 are deﬁned as
[Equation]
[Equation] Since the degrees of freedom are monotonically increasing with respect to each σ j (Σ) , we can see that N (cid:96) ( r 2 , Σ) ≥ N (cid:96) ( r 2 , Σ (cid:48) )
if Σ (cid:23) Σ (cid:48) .
Let
[Equation]
then Proposition 1 tells that there exits a matrix ˆ A (cid:96) ∈ m (cid:96) × m (cid:93)(cid:96) and J (cid:96) ⊂ { 1 , . . .
, m (cid:96) } m (cid:93)(cid:96) such that
[Equation]
for any w ∈ R m (cid:96) 5 , and the norm of ˆ A (cid:96) is bounded as
[Equation]
Next, we evaluate the degrees of freedom of Σ (cid:93) ( (cid:96) ) .
We bound this by using the degrees of freedom of (cid:98) Σ ( (cid:96) ) .
First note that r 2 (cid:96) =
(cid:107)(cid:107) φ (cid:96) − φ (cid:93)(cid:96) (cid:107)(cid:107) 2 n .
Let s ≤ m .
For any matrix U ∈ R m (cid:96) × s such that
U (cid:62) U
= I s ,
Tr[ U (cid:62) Σ (cid:93) ( (cid:96) ) U ]
= P n [ φ (cid:93)(cid:96) (cid:62) UU (cid:62) φ (cid:93)(cid:96) ] ≤ 2 { P n [ φ (cid:62) (cid:96) UU (cid:62) φ (cid:96) ]
+ P n [( φ (cid:96) − φ (cid:93)(cid:96) )
(cid:62) UU (cid:62) ( φ (cid:96) − φ (cid:93)(cid:96) )] } by the Cauchy-Schwartz inequality.
Here, let U be the matrix that gives P n [ φ (cid:62) (cid:96) UU (cid:62) φ (cid:96) ]
= (cid:80) m
(cid:96) j
= m (cid:96) − s +1 σ j ( (cid:98) Σ ( (cid:96) ) )
=
inf U : U (cid:62) U =
I s P n [ φ (cid:62) (cid:96) UU (cid:62) φ (cid:96) ] , then by noticing P n [( φ (cid:96) − φ (cid:93) (cid:96) ) (cid:62) UU (cid:62) ( φ (cid:96) − φ (cid:93)(cid:96) )] ≤ P n (cid:107) φ (cid:96) − φ (cid:93)(cid:96) (cid:107) 2 ≤ r 2 (cid:96) , we obtain that P n [ φ (cid:93)(cid:96) (cid:62)
[Equation]
r [Equation]
Finally, by minimizing the left hand side with respect to U , we obtain that
[Equation]
By setting s = m (cid:96) − m + 1 for 1 ≤ m ≤ m (cid:96) , this indicates that
[Equation]
Now, let ˙ m (cid:96) :
= min { j ∈ { 1 , . . .
, m (cid:96) } | ˙ µ ( (cid:96) )
j ≤ r 2 } (if ˙ µ ( (cid:96) [Table] m (cid:96) > r 2 , then we set ˙ m (cid:96) = m (cid:96) ).
Then,
[Table]
[Equation]
Now, let
[Equation]
then
[Equation]
We deﬁne the right hand side as m (cid:93)(cid:96) :
[Equation]
We have, by Eq.
(22), m
[Equation]
=
[Equation]
By the induction assumption [Equation] we also have that
[Equation]
Combining these inequalities, if we deﬁne φ
[Equation]
(cid:93)(cid:96)
[Equation]
x )) , and set W (cid:93) ( (cid:96) )
=
W ( (cid:96) )
ˆ A (cid:96) and reset W (cid:93) ( (cid:96) − 1) ← W (cid:93) ( (cid:96) − 1) J (cid:96) , : , then it holds that
[Equation]
where we let
[Equation]
Letting r 0 = 0 , by an induction argument, we obtain
[Equation]
R
[Equation]
˜ r
[Equation]
Finally, we obtain
[Equation]
, for a compressed network f (cid:93) that has width m (cid:93) = ( m (cid:93) 1 , . . .
, m (cid:93)L ) with parameters W (cid:93) ( (cid:96) ) =
W ( (cid:96) )
J (cid:96) +1 , : ˆ A (cid:96) .
Note that
[Equation]
[Equation]
Therefore, if we set (cid:98) G = NN [Equation] m
[Equation] ) , then there exists [Equation] G such that
[Equation]
where
[Equation]
Moreover, applying Lemma 5 to (cid:98) G and the Dudley integral yields
[Equation]
This gives the assertion of Theorem 3.
Here, we consider a situation where R 2F ˜ r
[Equation]
=
[Equation]
[Equation]
for some constant c 0 > 0 .
Then it holds that
[Equation]
Therefore, by setting C L := (1 ∨ R 2 )
L exp (cid:16) c 0 (2 √ L − 1) (cid:17) , it holds that r (cid:96) ≤ C L r 1 for (cid:96) = 1 , . . .
, L , in particular, we have
[Equation]
In this situation [Equation] the degrees of freedom are bounded by
[Equation]
Next [Equation] we bound ˙ m (cid:96) .
To do so, we should bound ˜ r (cid:96) from below.
Note that
[Equation]
Hence,
[Equation]
By Lemma
[Equation]
we can evaluate r 2 ∗ for (cid:98) F satisfying Assumption 4 as
[Equation]
where
[Equation]
[Equation]
Then, the overall generalization error is upper bounded by
[Equation]
  [Equation] with probability 1 − 3 e − t for all t ≥ 1 .
By letting Q (cid:48)
L,α,n := L (2 LV 0 R L
[Equation]
B
[Equation]
log( n
[Equation]
n and assuming ˆ r ≤ 1 , the second and third terms in C [ · ] is bounded by
[Equation]
Hence, by setting C L r
[Equation]
= (cid:16) [Equation] L (cid:17) − [Equation]
[Equation]
2 α ) which balances the ﬁrst and the second terms, then ˆ r ≤ C L r 1 ≤ 1 and the right hand side is bounded by
[Equation]
CM [Equation] n
[Equation]
Finally, by setting c 0 = 1 / 4 , we obtain the assertion for
Q L =
[Equation]
This gives Theorem 4.
B. I MPROVED BOUND OF T HEOREM 4 WITH L IPSCHITZ CONTINUITY CONSTRAINT
Here, we again note that there appears R L 2 in P L and Q L in the bound of Theorem 4.
This is due to a rough evaluation of the interlayer Lipschitz continuity.
We can reduce this exponential dependency under Assumption 6.
Corollary 3.
Assume Assumption 6 in addition to Assumptions 4 and 5, then the bound in Theorem 4 holds for the following redeﬁned P L and Q L :
[Equation]
except that the term M R 2F L 2 R 22 (cid:113) log( n ) 3 n is replaced by Mκ 2 R 2F L 2 (cid:113) log( n ) 3 n : Ψ( (cid:98) f ) ≤ (cid:98) Ψ( (cid:98) f ) + C (cid:34) M (cid:118)(cid:117)(cid:117)(cid:117)(cid:116) [ P L ∨ Q L ] L 1+ β [Equation]
n (cid:32)
[Equation]
1 − 1 / 2 α ) log( n )
[Equation]
Proof of Corollary 3.
To show Corollary 3, we set
[Equation]
where c 0 is a constant, and by the same argument as in the proof of Corollary 2, we can show that
[Equation]
Then, through a cumbersome calculation, we have that
[Equation]
for a universal constant C .
Moreover, we can show that r (cid:96) can be bounded as
[Equation]
This also gives
[Equation]
for a universal constant C (cid:48) .
Then, redeﬁning C L = κ exp( c
[Equation]
2 √ L − 1)) , we can apply the same argument as in the proof of Corollary 2.
Indeed, we can show
[Equation]
From the above argument, if we set (cid:98) G = NN( m (cid:93) , (cid:113) 203 max (cid:96) m (cid:96) R 2 , (cid:113) 203 max (cid:96) m (cid:96) R F ) , then there exists (cid:98) g ∈ (cid:98) G such that
[Equation]
where ˆ r
[Equation]
= r
[Equation]
Moreover, we can show
[Equation]
where S 1 =
L (cid:88) (cid:96) =
[Equation]
n [Equation] S 3 = (
L (cid:88) (cid:96)
[Equation]
[Equation]
(cid:96)
[Equation]
1 /α
[Equation]
( n
[Equation]
L
[Equation]
[Equation]
[Equation]
[Equation]
+ 1) 2 )] = O (cid:32) L ( L (cid:88) (cid:96) =
[Equation]
(cid:96)
[Equation]
0 κ 2 B x ) 1 /α log( n ) (cid:33) .
Here, to evaluate S 3 , we used the argument in Sec.
B.4 (proof of Corollary 2).
The remaining argument is the same as the proof of Theorem 3 and Theorem 4 (Sec. B.5).
C A UXILIARY LEMMAS
In this section, we give several auxiliary lemmas that are used in the proof of the theorems.
These results are not new at all, but we explicitly present them for completeness.
C. C OVERING NUMBER OF DEEP NETWORK MODELS
Deﬁne the neural network with height L , width m , sparsity constraint S and norm constraint B as Φ( L, m, S, B ) :
= { G ◦ ( W
[Equation]
L
[Equation]
η
[Equation]
[Equation]
L
[Equation]
where (cid:107) · (cid:107) 0 is the (cid:96) 0 -norm of the matrix (the number of non-zero elements of the matrix) and (cid:107) · (cid:107) ∞ is the (cid:96) ∞ -norm of the matrix (maximum of the absolute values of the elements).
The following evaluation of the covering number of the model Φ( L, m, S, B ) is shown by SchmidtHieber (2019); Suzuki (2019).
Lemma 4 (Covering number evaluation (Schmidt-Hieber, 2019; Suzuki, 2019)) .
The covering number of Φ( L, m, S, B ) can be bounded by
[Equation]
∞ , δ ) ≤ S log( δ − 1 L ( B ∨ 1) L − 1 ( m + 1) 2 L ) ≤ 2 SL log(( B ∨ 1)( m + 1))
+ S log( δ − 1 L ) .
Proof of Lemma 4.
Given a network f ∈ Φ( L, m, S, B ) expressed as
[Equation]
let
A k
[Equation]
◦ [Equation]
· ◦
[Equation]
W [Equation] [Equation] b
[Equation] [Equation] and B k ( f )( x ) =
G ◦ ( W ( L ) η ( · ) + b ( L ) ) ◦ · · · ◦ ( W ( k ) η
( x ) + b ( k ) ) , for k = 2 , . . . , L .
Corresponding to the last and ﬁrst layer, we deﬁne B L +1 ( f )( x ) = x and
A 1 ( f )( x ) = x .
Then, it is easy to see that f ( x ) =
B k +1 ( f ) ◦ ( W ( k ) · + b ( k ) ) ◦
A k ( f )( x ) .
Now, suppose that a pair of different two networks f, g ∈ Φ( L, m, S, B ) given by f ( x ) =
G ◦ ( W ( L ) η ( · )+ b ( L ) ) ◦· · ·◦ ( W (1) x + b (1) ) , g ( x ) =
G ◦ ( W ( L ) (cid:48)
η ( · )+ b ( L ) (cid:48) ) ◦· · ·◦ ( W (1) (cid:48) x
+ b (1) (cid:48) ) , has a parameters with distance δ : (cid:107) W ( (cid:96) )
− W ( (cid:96) ) (cid:48) (cid:107) ∞
≤
δ and (cid:107) b ( (cid:96) ) − b ( (cid:96) ) (cid:48) (cid:107) ∞
≤ δ .
Now, not that (cid:107)A k ( f ) (cid:107) ∞ ≤ max j (cid:107) W ( k − 1) j, : (cid:107) 1 (cid:107)A k − 1 ( f ) (cid:107) ∞ + (cid:107) b ( k − 1) (cid:107) ∞ ≤ mB
cid:107)A k − 1 ( f ) (cid:107) ∞ + B ≤ ( B ∨ 1)( m + 1) (cid:107)A k − 1 ( f ) (cid:107) ∞ ≤ ( B ∨ 1) k − 1 ( m + 1) k − 1 , and similarly the Lipshitz continuity of B k ( f ) with respect to (cid:107) · (cid:107) ∞ -norm is bounded as ( Bm ) L − k +1 .
Then, it holds that
[Equation]
L
[Equation]
k =
[Equation]
≤
L [Equation] k = ( Bm
[Equation]
W [Equation] [Equation] k
[Equation]
k )
[Equation]
k
[Equation]
∞ ≤
L [Equation]
k = ( Bm ) L − k
[Equation]
[ m ( B [Equation] 1) k − 1 m k − 1 + 1] ≤
L (cid:88) k
= ( Bm ) L − k δ
( B ∨ 1) k − 1 m k ≤ δL ( B ∨ 1) L − 1 ( m + 1) L .
Thus, for a ﬁxed sparsity pattern (the locations of non-zero parameters), the covering number is bounded by (cid:0) δ/
[ L ( B ∨ 1) L − 1 ( m + 1) L ] (cid:1) − S .
There are the number of conﬁgurations of the sparsity pattern is bounded by (cid:0)
( m +1)
L S (cid:1) ≤ ( m + 1) LS .
Thus, the covering number of the whole space Φ is bounded as
[Equation]
which gives the assertion.
Lemma 5 (Covering number evaluation) .
Let NN( m , R 2 , R F ) be the set of neural networks with depth m ( m 1 , . . .
, m L ) , (cid:107) W ( (cid:96) )
(cid:107) ∞
≤ R 2 and (cid:107) W ( (cid:96) )
(cid:107) F ≤ R F .
The covering number of
NN( m , R 2 , R F ) can be bounded by
[Equation]
Moreover, the set of networks with low rank weight matrices, NN( m , s , R 2 , R F ) , has the following covering number bound:
[Equation]
Proof of Lemma 5.
Let B =
R 2 , m = max (cid:96) m (cid:96) , and S =
(cid:80) L(cid:96)
=1 m (cid:96) m (cid:96) +1 , then we can see that NN( m , R 2 , R F ) is a subset of Φ( L, m, S, B ) because (cid:107) W (cid:107) ∞ ≤ (cid:107) W (cid:107) 2 .
Hence Lemma 4 gives the ﬁrst assertion.
As for the second one, we can easily check that the covering number of NN( m , s , R 2 , R F ) can be bounded by the one given in Lemma 5 for Φ(2 L, m, S, B ) with
S = (cid:80) L(cid:96) =
[Equation]
s [Equation] ( m (cid:96) + m (cid:96) +1 ) .
Then, we obtain the second assertion.
C. C OMPRESSION ERROR BOUND FOR ONE LAYER
The following proposition was shown by Bach (2017); Suzuki et al.
Let (cid:98) Σ
I,I (cid:48)
∈ R K
× H for integers K, H ∈ N and a matrix (cid:98) Σ
I,I (cid:48)
∈ R K
× H be a matrix ( (cid:98) Σ i,j ) i ∈ I,j ∈ I (cid:48) for the index sets
I ∈ [ m (cid:96) ]
K and I (cid:48) ∈ [ m (cid:96) ] H .
Let F = { 1 , . . .
, m (cid:96) } be the full index set.
Let the degrees of freedom corresponding to (cid:98) Σ be ˆ N ( λ ) :
= N (cid:96) ( λ, (cid:98) Σ) (see Eq.
(21)) for λ > 0 .
Proposition 1.
Let U =
( U j,l ) j,l is the orthogonal matrix that diagonalizes (cid:98) Σ , that is, (cid:98) Σ =
U diag (ˆ µ 1 , . . .
, ˆ µ m (cid:96) )
U (cid:62) for ˆ µ j ≥
( j = 1 , . . .
, m (cid:96) ) .
Deﬁne
[Equation]
For λ > 0 , if m ≥ 5 ˆ N ( λ ) log(80 ˆ N ( λ )) , then there exist v 1 , . .
. , v m ∈ { 1 , . . .
, m (cid:96) } such that, for every α ∈ R m (cid:96) , inf
[Equation]

[Equation]
and (cid:80) mj
=1 τ (cid:48)
j − 1 ≤ 53 m × m (cid:96) , where (cid:107) β (cid:107) 2 τ (cid:48) :
= (cid:80) mj
=1 β 2 j τ (cid:48)
j .
Let τ := mλτ (cid:48)
and I τ = diag ( τ ) .
Then, ˆ A := (cid:98) Σ F,J ( (cid:98) Σ J,J + I τ ) − 1 for J = { v 1 , . .
. , v m } satisﬁes (cid:107) ˆ A (cid:107) 2 ≤ (cid:114)3 m (cid:96) , and the optimal β that achieves the inﬁmum is given by ˆ β =
ˆ A (cid:62) α for any α ∈ R m (cid:96) .
C. C ONCENTRATION INEQUALITY
Proposition 2 (Talagrand’s Concentration Inequality (Talagrand, 1996; Bousquet, 2002)) .
Let G be a function class on X that is separable with respect to ∞ -norm, and { x i } ni =1 be i.i.d.
random variables with values in X .
Furthermore, let B ≥
and U ≥ 0 be B := sup g ∈G E
[( g − E[ g ]) 2 ] and
U := sup g ∈G (cid:107) g
(cid:107) ∞ , then for Z :
= sup g ∈G (cid:12)(cid:12) 1 n (cid:80) ni
=1 g ( x i ) − E
[ g ] (cid:12)(cid:12) , we have
[Equation]
for all t > 0 .
D N UMERICAL EXPERIMENTS
In this section, we experimentally validate the assumptions we made in the theoretical analysis and investigate how large the intrinsic dimensionality becomes.
We use VGG-19 network trained on CIFAR-10.
The VGG-19 network have 16 convolution layers (named c0 , . .
. , c15) and 3 fully connected layers (named l16 , . .
. , l18).
The size of each ﬁlter in each convolution layer is 3 × 3 .
Our theory does not support a convolution layer in a strict sense, but we adopt it as follows.
If the convolution layer in the (cid:96) -th layer has the input channel size m (cid:96) and the output channel size m (cid:96) + with the ﬁlter size k × k (in our case k = 3 ), then the weight matrix is given as a 4-way tensor with the size m (cid:96) +
× m (cid:96) × k × k : W [Equation]
) ∈ R m
[Equation]
[-+
-]+1 × m [Equation] × k × k .
Although a singular value of a 4-way tensor is not well-deﬁned, we can perform a low rank approximation of the weight matrix by folding out the tensor to a large matrix.
Actually, considering “similarity” between the ﬁlters as
[Equation]
then we can easily see that, for s (cid:96) ∈ [ m (cid:96) +1 ] , it holds that
[Equation]
( K ( (cid:96) ) ) , where P ∈ R s (cid:96) × m (cid:96) +1 is a projection matrix to the eigen-space corresponding to the s (cid:96) largest singular values of K ( (cid:96) ) , W (cid:48) :
=
PW ( (cid:96) )
=
( (cid:80) m (cid:96) +1 i (cid:48)
=1 P i,i (cid:48) W ( (cid:96) )
i (cid:48) ,j,k,k (cid:48) )
i,j,k,k (cid:48) ∈ [ s (cid:96) ] ×
[ m (cid:96) ] × [ k ] ×
[ k ] and the
Frobenius norm (cid:107) · (cid:107) F of a tensor is the Euclidean norm as a vector.
Therefore, we can use the eigenvalues of K ( (cid:96) ) to evaluate the redundancy of parameters among ﬁlters.
As for the covariance matrix in a convolution layer, we also apply the same argument.
That is, the input to the (cid:96) -th layer (which is a convolution layer) is given by φ (cid:96) ( x
) ∈ R m (cid:96) × I × J where I and J are the width and height of the input, and we deﬁne the following “covariance” matrix as a similarity measure between the channels:
[Equation]
This
Near low rank properties of the covariance matrix and weight matrix Here, we see plausibility of the near low rank assumptions we made in the analysis.
Figure 1 presents the eigenvalues of covariance matrix (cid:98) Σ ( (cid:96) ) in each of layer c2, c7, c12 and l16.
The eigenvalues are sorted in decreasing order.
We can see that the eigenvalue distributions are highly concentrated around 0 and the eigenvalues decrease quickly, which indicates the near low rank property of (cid:98) Σ ( (cid:96) )
[Figure]
Next, we plot the eigenvalues of K ( (cid:96) ) in Figure 2 for layer c2, c7, c12 and l16.
We again observe a rapid decrease of the eigenvalues.
These results justify our theoretical assumptions.
[Figure]
Intrinsic dimensionality Here, we calculate the intrinsic dimensionalities of the VGG-19 network.
For that purpose
[Figure]
we set a threshold parameter ν ∈ { 10 − 1 , 10 − 2 , 10 − 3 } and compute
[Table]
) ) ≥ ν × max j (cid:48)
σ j (cid:48)
( K ( (cid:96) ) ) } and ˙ m (cid:96) :=
# { j ∈ [ m (cid:96) ]
| σ j ( (cid:98) Σ ( (cid:96) ) ) ≥ ν × max j (cid:48)
σ j (cid:48) ( (cid:98) Σ ( (cid:96) ) ) }
(which corresponds to setting ˜ r 2 (cid:96)
= 4 ν × max j (cid:48)
σ j (cid:48) ( (cid:98) Σ ( (cid:96) ) ) ).
Table 4 summarizes the effective ranks ˙ m (cid:96) , s (cid:96) of all layers.
We can see that the effective ranks can be much smaller than the channel sizes in several layers (especially layers from c8 to l18,), which indicates the network has high redundancy and its intrinsic dimensionality could be much smaller than the actual number of parameters.
Next, we compute the intrinsic dimensionality of each layer based on the effective ranks calculated above.
Basically, the main term of our bound (Theorem 4) is given by (cid:113)(cid:80) L(cid:96)
=
1 ˙ m (cid:96) +1 ˙ m (cid:96) n (note that m (cid:93)(cid:96) is essentially controlled by ˙
m (cid:96) ).
Hence, we employ ˙ m (cid:96) +1 × ˙ m (cid:96) as the intrinsic dimensionality of each fully connected layer.
As for a convolution layer, we employ ˙ m (cid:96) +1 ˙ m (cid:96) k 2 as the intrinsic dimensionality which is the number of parameters of compressed network.
We also calculate the intrinsic dimensionality obtained by compressing only the weight matrix.
That is given by s (cid:96) m (cid:96) k 2 + m (cid:96) +1 s (cid:96) .
Both of them are summarized in Table 4.
We can see that the intrinsic dimensionality is smaller than the actual number of parameters.
In particular, it is much smaller for higher layers such as c8 to l18.
This indicates that the information required for classiﬁcation is almost distilled in the ﬁrst few layers and the contribution of the subsequent layers would be much smaller than the earlier layers.
Moreover, we see that compressing the network using the covariance matrix gives smaller intrinsic dimensionalities than the weight matrix.
This is because
the improvement induced by decreasing ˙ m (cid:96) is a quadratic order but that by s (cid:96) is just a linear order.
Another reason is that the effective rank of the covariance matrix is more data dependent in a sense that it is strongly dependent on the distribution of the data, and thus it can capture data dependent redundancy more efﬁciently (see also Figures 1 and 2).
Since the intrinsic dimensionality is much smaller than the actual number of parameters, the VC-dimension bound is too pessimistic and a compression based bound like ours gives a better generalization error bound.
Finally, we give a comparison of intrinsic dimensionalities calculated by Arora et al.
(2018) and ours.
We borrowed the values presented in the paper (Arora et al., 2018).
We would like to note that the comparison is not completely fair because the intrinsic dimensionality of both our analysis and that of Arora et al.
(2018) neglect constant functors (such as depth), and thus the ﬁnal generalization error is not merely determined by the raw values.
However, the comparison offers better understanding of our analysis by observing difference and similarity between them.
We can see
[Table]
[Table] [Table]
[Table]
[Table]
512/ 2,359, 22,   3, c 512/ 2,359, 26,   4, that our intrinsic dimensionality gives a smaller number than theirs.
This is because compression through the covariance matrix gives quadratic factor improvement while compression through low rank property of the weight matrices gives linear order improvement.
This indicates considering near low rank properties of both of weight matrices and covariance matrices yields sharper bounds.
It can be realized by our uniﬁed theoretical frame-work.