R ETHINKING C OMPRESSED C ONVOLUTIONAL N EURAL N ETWORKS FROM A S TATISTICAL P ERSPECTIVE
Anonymous authors
Paper under double-blind review
A BSTRACT
Many designs have recently been proposed to improve the model efﬁciency of convolutional neural networks (CNNs) at a ﬁxed resource budget, while there is a lack of theoretical analysis to justify them.
This paper ﬁrst formulates CNNs with high-order inputs into statistical models, which have a special ”Tucker-like” formulation.
This makes it possible to further conduct the sample complexity analysis to CNNs as well as compressed CNNs via tensor decomposition.
Tucker and CP decompositions are commonly adopted to compress CNNs in the literature.
The low rank assumption is usually imposed on the output channels, which according to our study, may not be beneﬁcial to obtain a computationally efﬁcient model while a similar accuracy can be maintained.
Our ﬁnding is further supported by ablation studies on CIFAR10, SVNH and UCF101 datasets.
I NTRODUCTION
The introduction of AlexNet (Krizhevsky et al., 2012) spurred a line of research in 2 D CNNs, which progressively achieve high levels of accuracy in the domain of image recognition (Simonyan &
Zisserman, 2015; Szegedy et al., 2015; He et al., 2016; Huang et al., 2017).
The current stateof-the-art CNNs leave little room to achieve signiﬁcant improvement on accuracy in learning stillimages, and attention has hence been diverted towards two directions.
The ﬁrst is to deploy deep
CNNs on mobile devices by removing redundancy from the over-parametrized network, and some representative models include MobileNetV1 & V2 (Howard et al., 2017; Sandler et al., 2018).
The second direction is to utilize CNNs to learn from higher-order inputs, for instance, video clips (Tran et al., 2018;
Hara et al., 2017) or electronic health records (Cheng et al., 2016; Suo et al., 2017).
This area has not yet seen a widely-accepted state-of-the-art network.
High-order kernel tensors are usually required to account for the multiway dependence of the input.
This notoriously leads to heavy computational burden, as the number of parameters to be trained grows exponentially with the dimension of inputs.
Subsequently, model compression becomes the critical juncture to guarantee the successful training and deployment of tensor CNNs.
Tensor methods for compressing CNNs.
Denil et al.
(2013) showed that there is huge redundancy in network weights such that the entire network can be approximately recovered with a small fraction of parameters.
Tensor decomposition recently has been widely used to compress the weights in a CNN network (Lebedev et al., 2015;
Kim et al., 2016; Kossaiﬁ et al., 2020b; Hayashi et al., 2019).
Speciﬁcally, the weights at each layer are ﬁrst summarized into a tensor, and then tensor decomposition, CP or Tucker decomposition, can be applied to reduce the number of parameters.
Different tensor decomposition to convolution layers will lead to a variety of compressed CNN block designs.
For instance, the bottleneck block in ResNet (He et al., 2016) corresponds to the convolution kernel with a special Tucker low-rank structure, and the depthwise separable block in MobileNetV1 (Howard et al., 2017) and the inverted residual block in MobileNetV2 (Sandler et al., 2018) correspond to the convolution kernel with special CP forms.
All the above are for 2D
CNNs, and Kossaiﬁ et al. (2020b) and Su et al.
(2018) considered tensor decomposition to factorize convolution kernels for higher-order tensor inputs.
Tensor decomposition can also be applied to fully-connected layers since they may introduce a large number of parameters (Kossaiﬁ et al., 2017; 2020a); see also the discussions in Section 5.
Moreover,
Kossaiﬁ et al.
(2019) summarized all weights of a network into one single high-order tensor, and then directly imposed a low-rank structure to achieve full network compression.
While the idea is highly motivating, the proposed structure of the high-order tensor is heuristic and can be further improved; see the discussions in Section 2.4.
Parameter efﬁciency of the above proposed architectures was heuristically justiﬁed by methods, such as FLOPs counting, naive parameter counting and/or empirical running time.
However, there is still lack of a theoretical study to understand the mechanism of how tensor decomposition can compress
CNNs.
This paper attempts to ﬁll this gap from statistical perspectives.
Sample Complexity Analysis.
Du et al.
(2018a)
ﬁrst characterized the statistical sample complexity of a CNN; see also Wang et al.
for compact autoregressive nets.
Speciﬁcally, consider a CNN model, y = F CNN ( x , W ) + ξ , where y and x are output and input, respectively, W contains all weights and ξ is an additive error.
Given the trained and true underlying networks F CNN ( x , (cid:99) W ) and
F CNN ( x , W ∗ ) , the root-mean-square prediction error is deﬁned as
[Equation]
where (cid:99) W and W ∗ are trained and true underlying weights, respectively, and E x is the expectation on x .
The sample complexity analysis is to investigate how many samples are needed to guarantee a given tolerance on the prediction error.
It can also be used to detect the model redundancy.
Consider two nested CNNs, where F 1 is more compressed than F 2 .
Given the same true underlying networks, when the prediction errors from trained F 1 and F 2 are comparable, we then can argue that F 2 has redundant weights comparing with F 1 .
As a result, conducting sample complexity analysis to CNNs with higher order inputs will shed light on the compressing mechanism of popular compressed CNNs via tensor decomposition.
The study in Du et al.
(2018a) is limited to 1-dimensional convolution with a single kernel, followed by weighted summation, and its theoretical analysis cannot be generalized to CNNs with compressed layers.
In comparison, our paper presents a more realistic modeling of CNN by introducing a general N -dimensional convolution with multiple kernels, followed by an average pooling layer and a fully-connected layer.
The convolution kernel and fully-connected weights are in tensor forms, and this allows us to explicitly model compressed CNNs via imposing low-rank assumption on weight tensors.
Moreover, we used an alternative technical tool, and a sharper upper bound on the sample complexity can be obtained.
Our paper makes three main contributions:
We formulate CNNs with high-order inputs into statistical models, and show that they have an explicit “Tucker-like” form.
The sample complexity analysis can then be conducted to CNNs as well as compressed CNNs via tensor decomposition, with weak conditions allowing for time-dependent inputs like video data.
From theoretical analysis, we draw an interesting ﬁnding that forcing low dimensionality on output channels may introduce unnecessary parameter redundancy to a compressed network.
1. C OMPARISON WITH OTHER EXISTING WORKS
Deep neural networks are usually over-parametrized, yet empirically, they can generalize well.
It is an important topic in the literature to theoretically study the generalization ability of deep neural networks, including deep CNNs (Li et al., 2020; Arora et al., 2018).
The generalization error, deﬁned as the difference between test and training errors, is commonly used to evaluate such ability, and many techniques have been developed to control its bound; see, for example, the VC dimension (Vapnik, 2013), the Rademacher complexity & covering number (Bartlett & Mendelson, 2002), the norm-based capacity control (Neyshabur et al., 2017; Golowich et al., 2018; Bartlett et al., 2017;
Neyshabur et al., 2015) and low-rank compression based methods (Li et al., 2020; Zhou & Feng, 2018; Arora et al., 2018).
These works use a model-agnostic framework, and hence relies heavily on explicit regularization such as weight decay, dropout or data augumentation, as well as algorithmbased implicit regularization to remove the redundancy in the network.
We, however, attempt to theoretically explain how much compressibility is achieved in a compressed network architecture.
Speciﬁcally, we make a comparison between a CNN and its compressed version, and makes theoretically-supported modiﬁcation to the latter to further increase efﬁciency.
Subsequently, our analysis requires an explicit formulation for the network architecture, which is provided in Section 2, and the prediction error at (1) is adopted as our evaluation criteria.
We notice that Li et al.
(2020) also proposes to use the CP Layers to compress the weights in each convolution layer.
But their study is still model-agnostic, since the ranks of the underlying CP Layers depend on the trained weights.
In details, their proposed approach uses regularization assumptions on the weights and hence, their derived theoretical bound is inﬂuenced by training and not suitable to analyze the network design exclusively.
Other existing works, that aim to provide theoretical understanding for the neural networks, include the understanding of parameter recovery with gradient-based algorithms for deep neural networks (Zhong et al., 2017b;
Fu et al., 2020;
Goel et al., 2018; Zhong et al., 2017a); the development of other provably efﬁcient algorithms (Cao & Gu, 2019; Du & Goel, 2018); and the investigation of convergence in an over-parameterized regime (Allen-Zhu et al., 2019; Li & Liang, 2018; Du et al., 2018b).
Our work differs greatly from these works in both target and methodology.
We do not consider computational complexity or algorithm convergence.
Instead, we focus on the statistical sample complexity to depict the mechanism of compressed block designs for CNNs.
F ORMULATING CNN S WITH HIGHER - ORDER INPUTS 2. N OTATION
Tensor notations.
We follow the notations in Kolda & Bader (2009) to denote vectors by lowercase boldface letters, e.g. a ; matrices by capital boldface letters, e.g. A ; tensors of orderor higher by Euler script boldface letters, e.g. A .
For an N th -order tensor A ∈ R l 1 ×···× l N , denote its elements by A ( i 1 , i 2 , . .
, i N ) and the n -mode unfolding by A ( n ) , where the columns of A ( n ) are the n -mode vectors of A , for 1 ≤ n ≤ N .
And A (: , · · · , i n , · · · , :) denotes the subtensor of A , holding only the n th index ﬁxed.
The vectorization operation is denoted by vec ( · ) .
The inner product of two tensors A , B ∈ R l 1 ×···× l N is deﬁned as (cid:104) A , B (cid:105) =
(cid:80) i 1 · · · (cid:80) i N A ( i 1 , . .
, i N ) B ( i 1 , . . .
, i N ) , and the Frobenius norm is (cid:107) A (cid:107) F = (cid:112)
(cid:104) A , A (cid:105) .
The mode- n multiplication × n of a tensor A ∈ R l 1 ×···× l N and a matrix B ∈ R p n × l n is deﬁned as ( A × n B )( i 1 , . .
. , j n , . .
, i N )
= (cid:80) l n i n =1 A ( i 1 , . .
, i n , . .
, i N ) B ( j n , i n ) , for 1 ≤ n ≤ N , respectively.
The mode- n multiplication ¯ × N of a tensor A ∈ R l 1 ×···× l N and a vector b ∈ R l n is deﬁned as ( A ¯ × n b )( i 1 , . .
, i n − 1 , i n +1 , . .
, i N )
= (cid:80) l n i n =1 A ( i 1 , . .
, i n , . .
, i N ) b ( i n ) , for 1 ≤ n ≤ N .
The symbol “ ⊗ ” is the Kronecker product and “ ◦ ” is the outer product.
We extend the deﬁnition of Khatri-Rao product to tensors: given tensors A ∈ R l 1 × l 2 ×··· l N × K and B ∈ R p 1 × p 2 ×··· p N × K , their Khatri-Rao product is a tensor of size R l 1 p 1 × l 2 p 2 ×··· l N p N × K , denoted by C
= A (cid:12) B , where
C (: , · · · , k ) =
A (: , · · · , k ) ⊗ B (: , · · · , k ) , 1 ≤ k ≤ K .
CP decomposition.
The Canonical Polyadic (CP) decomposition (Kolda & Bader, 2009) factorizes the tensor A ∈ R l 1 ×···× l N into a sum of rank-1 tensors, i.e. A = (cid:80) R r =1 α r h (1) r ◦ h (2) r ◦ · · · ◦ h ( N ) r , where h ( j ) r is a unit norm vector of size R l j for all 1 ≤ j ≤ N .
The CP rank is the number of ranktensors R .
Tucker decomposition.
The Tucker ranks of an N th -order tensor A ∈ R l 1 ×···× l N are deﬁned as the matrix ranks of the unfoldings of A along all modes.
If the Tucker ranks of A are ( R 1 , . . .
, R N ) , then there exist a core tensor G ∈ R R 1 ×···× R N and matrices H ( i
) ∈
R l i × R i , such that A =
G × 1 H (1) × 2 H (2) · · · × N H ( N ) , known as Tucker decomposition (Tucker, 1966),
2. B ASIC THREE - LAYER CNN S
Consider a three-layer CNN with one convolution, one average pooling and one fully-connected layers, and we assume linear activations for simplicity.
Speciﬁcally, for a general tensor-structured input X ∈ R d 1 × d 2 ×···× d N , we ﬁrst perform its convolution with an N th -order kernel tensor A ∈ R l 1 × l 2 ×···× l N to get an intermediate output X c ∈ R m 1 × m 2 ×···× m N , and then use average pooling with pooling sizes ( q 1 , · · · , q N ) to derive another intermediate output X cp ∈ R p 1 × p 2 ×···× p N .
Finally, X cp goes through a fully-connected layer, with weight tensor B ∈ R p 1 × p 2 ×···× p N to produce a scalar output; see Figure 1(1).
We consider the convolution layer with the stride size of s c along each dimension.
Assume that m j = ( d j − l j ) /s
c
+ 1 are integers for 1 ≤ j ≤ N , otherwise zero-padding will be needed.
Let
[Equation]
which act as positioning factors to stretch the kernel tensor A into a tensor of the same size as X , while the rest of entries are ﬁlled with zeros; see Figure 1(2).
As a result, X c has entries
[Equation]
For the pooling layer with the stride size of s p along each dimension, we assume the pooling sizes { q j } Nj =1 satisfy the relationship of m j = q j + ( p j − 1) s p , so the sliding windows can be overlapped.
But for ease of notation, we can simply take q j = m j /p j .
The average pooling operation is equivalent to forming p 1 p 2 p 3 · · · p N consecutive blocks within X c , each of size R q 1 × q 2 ×···× q N , and then take the average per block.
The resulting tensor X cp has entries
X cp ( i 1 , i 2 , . .
, i N ) =
( (cid:81) j q j ) − 1 (cid:80) j
(cid:80) i
j q j k j =(
i j − 1) q j +1 X c ( k 1 , k 2 , . . . , k N ) , for 1 ≤ i j ≤ p j and 1 ≤ j ≤ N .
If we denote U ( j ) F ,i j = q − 1 j (cid:80) i j q j k =( i
j − 1) q j +1 U ( j ) k , then equivalently,
[Equation]
The fully-connected layer performs a weighted summation over X cp , with weights given by entries of the tensor B ∈ R p 1 × p 2 ×···× p N .
Denote U ( j ) F =
( U
[Equation]
j ) F , 1 , · · · , U ( j ) F ,p j ) for 1 ≤ j ≤ N , and the predicted output has the form of
[Equation]
Similarly, for a CNN with K kernels, denote { A k , B k } Kk =1 to be the set of kernels and fullyconnected weights, where B k ∈ R p 1 × p 2 ×···× p N and A k ∈ R l 1 × l 2 ×···× l N .
The model can then be explicitly represented by
[Equation]
where ξ i is the additive error,
X i ∈ R d 1 × d 2 ×···× d N , and the composite weight tensor
[Equation]
The innate weight-sharing compactness of CNN can be equivalently represented as a “Tucker-like” form at (4).
The factor matrices { U ( j ) F ∈ R d j × l j p j } Nj =1 are ﬁxed and solely determined by CNN operations on the inputs.
They have full column ranks given that p j l j ≤ d j always holds.
The core tensor is a special Kronecker product that depicts the layer-wise interaction between weights.
2. S AMPLE COMPLEXITY ANALYSIS
We can now derive the non-asymptotic error bound for the CNN model.
Let Z i
=
X i × 1 U (1) (cid:48)
F ×
U (2) (cid:48)
F × 3 · · · × N U ( N ) (cid:48) F ∈ R l 1 p 1 × l 2 p 2 ×···× l N p N .
Model (3) is equivalent to
[Equation]
where W = (cid:80) Kk =1 B k ⊗ A k .
The trained weights have the form of (cid:99) W = (cid:80)
Kk =1 (cid:98) B k ⊗ (cid:98) A k , where {
[Equation]
k ,
[Equation]
Denote x i = vec( X i ) and z i = vec( Z i ) , and it can be veriﬁed that z
i
=
U (cid:48)
G x i , where U G =
U (1) F ⊗ { U ( N ) F ⊗ [ U ( N − 1) F ⊗ · · · ⊗ ( U (3) F ⊗ U (2) F )] } represents the CNN operations on the inputs.
Let ¯ x = ( x 1 (cid:48) , x 2 (cid:48) , . . .
, x n (cid:48) )
(cid:48) , and we make the following technical assumptions.
Assumption 1.
(Time-dependent inputs) ¯ x is normally distributed with mean zero and variance Σ
=
E (¯ x ¯ x (cid:48) ) ,
where c x I ≤ Σ ≤ C x I for some 0 < c x < C x .
Denote by λ max ( A ) and λ min ( A ) the maximum and minimum eigenvalues of a symmetric matrix
A , respectively.
When { x i } is a stationary time series with spectral density function f X ( θ ) , we can take c x = inf − π ≤ θ ≤ π λ min ( f X ( θ )) and C x = sup − π ≤ θ ≤ π λ max ( f X ( θ )) ; see Basu & Michailidis (2015).
For independent inputs, Σ = diag { Σ 11 , . . .
, Σ nn } is block diagonal, and it holds that c x = min 1 ≤ j ≤ n λ min ( Σ jj ) and C x = max 1 ≤ j ≤ n λ max ( Σ jj ) , where Σ jj = E ( x j x j (cid:48) ) .
Assumption 2.
(Sub-Gaussian errors) { ξ i } are independent σ 2 -sub-Gaussian random variables with mean zero, and is independent of { X j , 1 ≤ j ≤ i } for all 1 ≤ i ≤ n .
Assumption 3.
(Restricted isometry property)
c u I ≤ U (cid:48)
G U G ≤ C u
I for some 0 < c u < C u .
Denote κ U
= C x C u
and κ L = c x c u .
Let W ∗ =
(cid:80)
Kk =1 B ∗ k ⊗ A ∗
k be the true weight.
The model complexity of CNN at (5) is d M = K ( P + L + 1) ,
where P = p 1 p 2 · · · p N and L = l 1 l 2 · · · l N .
Theorem 1 (CNN) .
Suppose that Assumptions 1-3 hold and n (cid:38) ( κ U /κ L ) 2 d M .
Then,
[Equation]
(cid:114)
[Equation]
n (cid:33) and, E ( (cid:99) W ) ≤ [Equation] F with probability 1 − 4 exp {− [ c H (0 .
25 κ L /κ U )
2 n − 9 d M ] } − exp {− d M − 8 δ } , where δ = O p (1) ,
c H is a positive value, α RSC
= κ L / 2 , and α RSM = 3 κ U / 2 .
It can be seen that the prediction error E ( (cid:99) W ) , is O p ( (cid:112)
d M /n ) , and hence the sample complexity is of order O
( d M /ε 2 ) , to achieve a prediction error ε .
Technical proofs of all theorems and corollary in the paper are deferred to Appendix A.3.
Note that, for simplicity, we assume a simple regression model at (5).
The theoretical analysis can indeed be established for classiﬁcation problems as well.
In details, we provide corresponding theorem and corollaries for both binary and multiclass classiﬁcation problems in Appendix A.4.
2. CNN S WITH MORE LAYERS
Consider a 5-layer CNN with “convolution → pooling → convolution → pooling → fully connected” layers, where the settings for the ﬁrst “convolution → pooling” layers are the same as those in Section 2.2.
Denote the kernel for the second convolution by (cid:101)
A ∈ R ˜ l 1 × ˜ l 2 ×···× ˜ l N , and the fully-connected weight tensor B is of size R ˜ p 1 × ˜ p 2 ×···× ˜ p N .
We similarly deﬁne matrices (cid:101) U ( j ) i j and (cid:101) U ( j ) F ,i j , which are both of size
R p j × ˜ l j with 1 ≤ i j ≤ ˜ p
j and 1 ≤ j ≤ N , and can be used to represent the second convolution and pooling operations, respectively.
Note that the output from the ﬁrst pooling layer, X cp , is the input to the second convolution layer, and hence the output from the second pooling layer has entries
[Equation] N
[Equation]
Stack the matrices { U ( j ) F ,i j ∈ R d j × l j } 1 ≤ i j ≤ p j and { (cid:101) U ( j ) F ,i j ∈ R p j × ˜ l j } 1 ≤ i j ≤ ˜ p
j into 3D tensors U ( j ) ∈
R d j × l j × p j and (cid:101) U
( j ) ∈
R p j × ˜ l
j × ˜ p j , respectively.
Let U ( j ) DF =
( U ( j ) × 3 (cid:101) U ( j ) )
(1) for 1 ≤ j ≤ N and, by some algebra in Appendix A.2, we can show that the predicted output has the form of
[Equation]
[Equation]
where U ( j ) × 3 (cid:101) U
( j ) yields a tensor of size R d j × l j × (cid:101)
l j × (cid:101) p j , with (cid:80) p
j i =
1 U ( j ) ( k 1 , k 2 , i ) (cid:101) U ( j ) ( i, k 3 , k 4 ) as its ( k 1 , k 2 , k 3 , k 4 ) th entry.
The case with multiple kernels will have a form similar to (4).
Following the same logic, it can indeed be generalized to an even deeper CNN by adding more “convolutionpooling” layers.
Since the composite weight tensor always has a “Tucker-like” form, techniques in proving Theorem 1 can be adopted to conduct the sample complexity analysis for deep CNNs.
Note that, at model (7), the kernels at different convolution layers appear in the form of (cid:101)
A ⊗ A , which actually provides a theoretical justiﬁcation for exploring the layer-wise low-dimensional structure in the literature.
On the other hand, we summarize all weights of a network into one single tensor, akin to Kossaiﬁ et al.
But our summarized tensor has an explicit “nested doll” structure, where the weight structure of the previous layer is nested within that of the current layer.
In practice, zero-padding can be used before each convolution layer to preserve the input size.
This is compatible with our framework, since we can propagate the padding of zeros to previous layers and all the way back to the input tensor X .
C OMPRESSED CNN S
For a high-order input, a deep CNN with a larger number of kernels may involve heavy computation, which renders its training difﬁcult for portable devices with limited resources.
In real applications, many compressed CNN block designs have been proposed to improve the efﬁciency, and most of them are based on either matrix factorization or tensor decomposition; see Lebedev et al.
Kim et al.
; Astrid & Lee (2017); Kossaiﬁ et al. (2020b).
Tucker decomposition can be used to compress CNNs, which is same as introducing a multilayer
CNN block; see Figure 3 in Kim et al.
Speciﬁcally, we stack the kernels
A k ∈ R l 1 × l 2 ×···× l N with 1 ≤ k ≤ K into a higher order tensor A stack ∈ R l 1 × l 2 ×···× l N × K , and assume that A stack has
Tucker ranks of ( R 1 , R 2 , · · · , R N , R N +1 ) .
As a result,
[Equation]
where G ∈ R R 1 ×···× R N × R N +1 is the core tensor, and
H ( j ) ∈ R
l j × R j are factor matrices for 1 ≤ j ≤ N + 1 .
From Section 2, to train such a compressed CNN block with linear activations is equivalent to search for the least-square estimators at (6) with A stack being constrained to have the form of (8).
Denote the estimators by { (cid:98) B TU k , (cid:98)
A TU k } 1 ≤ k ≤ K .
The trained weights then have a form of (cid:99) W TU =
(cid:80)
Kk =1 (cid:98) B TU k ⊗ (cid:98)
A TU k , and the model complexity is d TU M
= (cid:81) N +1 j =1 R j + (cid:80) Ni =1 l i R i + R N +1 P .
CP decomposition is more popular in compressing CNNs; see Kossaiﬁ et al.
(2020b)
; Lebedev et al.
; Astrid & Lee (2017).
Let { (cid:98) B CP k , (cid:98)
A CP k } 1 ≤ k ≤ K be the least-square estimators at (6) with A stack having a CP decomposition with a rank of R .
Then, the trained weights are (cid:99) W CP = (cid:80) K k
=1 (cid:98) B CP k ⊗ (cid:98)
A CP k , and the model complexity is d CP M = R N +
+ R ( (cid:80) N i =
1 l i + P ) .
Theorem 2 (Compressed CNN) .
Let ( (cid:99) W , d M ) be ( (cid:99) W TU , d TU M ) for Tucker decomposition, or ( (cid:99) W CP , d CP M ) for CP decomposition.
Suppose that Assumptions 1-3 hold and n (cid:38) ( κ U /κ L ) 2 c N d M .
Then, [Equation]
with probability 1 − 4 exp {− [ c H (0 .
25 κ L /κ U )
2 n − 3 c N d M ] }− exp {− d M − 8 δ } , where δ = O p (1) , c N = 2 N +1 log( N + 2) ,
c H is a positive value and α RSC , α RSM are deﬁned in Theorem 1.
Theorem 2 shows that, as expected, the sample complexity is proportional to the model complexity.
Compared to Theorem 1, we see that the sample complexity of the compressed CNN depends on (cid:80)
Ni =1 l i instead of (cid:81) Ni =
1 l i .
When the input dimension N is large, compressed CNN can indeed reduce a large number of parameters.
However, we notice that the sample complexity of a compressed CNN depends on the rank R N + or R , rather than the number of kernels K .
This differs from the naive parameter counting, and we hence provide the rationale behind this counter-intuitive observation in the corollary below.
Corollary 1.
(a)
If A has a Tucker decomposition with the ranks of ( R 1 , R 2 , · · · , R N +1 ) , the CNN is equivalent to one with R N +1 kernels, and each kernel A r , where 1 ≤ r ≤ R N +1 , has a Tucker decomposition with ranks of ( R 1 , R 2 , · · · , R N ) .
Moreover, (b) if the stacked kernel tensor A has a CP decomposition with rank R , the corresponding CNN can be reparameterized into one with R kernels, and each kernel A r , where 1 ≤ r ≤ R , has a CP decomposition form with rank R .
Deﬁne the K/R ratio to be the value of K/R N +1 for Tucker decomposition or K/R for CP decomposition, and in practice, it always holds that K ≥ R .
Corollary 1 essentially states that when
K/R > 1 , there exists model redundancy in the CNN model with linear activations.
In other words, we can obtain a more compressed network by setting K
= R such that it has the same sample complexity as the previous one.
The K/R ratio appears in various block designs; see Figure 2.
And we can use this ﬁnding to evaluate their model efﬁciency.
(T1)
Standard bottleneck block.
The basic building block in ResNet (He et al., 2016) can be exactly replicated by a Tucker decomposition on A stack ∈ R l 1 × l 2 × C × K , with ranks of ( l 1 , l 2 , R, R ) , where C is the number of input channels (Kossaiﬁ et al., 2020b).
As shown by the ablation studies in Session 4, when K (cid:29) R , this design may not be the most parameter efﬁcient.
(T2) Funnel block.
As a straightforward revision on a standard bottleneck block, the funnel block maintains an output channel size of K = R .
This is hence an efﬁcient block design.
Also, when
R = C , we obtain a normalized funnel block, with the same number of input and output channels.
(C1) Depthwise separable block.
This block is the basic module in MobileNetV1 (Howard et al., 2017) with a pair of depthwise and pointwise separable convolution layers.
It is equivalent to assuming a CP decomposition on A stack ∈ R l 1 × l 2 × C × K with the rank of C (Kossaiﬁ et al., 2020b).
The 1 × 1 pointwise convolution expands the numbers of channels from C to K .
When K (cid:29) C , this design is parameter inefﬁcient from a computation viewpoint.
(C2) Inverted residual block.
Sandler et al.
(2018) later proposed this design in MobileNetV2.
It includes expansive layers between the input and output layers, with the channel size of x · C ( x ≥ 1) , where x represents the expansion factor.
As discussed by Kossaiﬁ et al.
(2020b), it can heuristically correspond to a CP decomposition on A stack , with CP rank equals to x · C .
Since, the rank of output channel dimension can be at most x · C , as long as K ≤ x · C holds, it is theoretically efﬁcient and provides leeway for exploring thicker layers within blocks.
With nonlinear activations, the model redundancy may bring some beneﬁt.
Such beneﬁt is often not guaranteed.
And when K (cid:29) R , it will, without a doubt, greatly hinder computational efﬁciency.
We will show, in our ablation studies in the next section, that under realistic settings with nonlinear activations, our ﬁnding on the K/R ratio still applies.
Speciﬁcally, networks with K/R =maintains comparable performance as networks with K/R > 1 , while using much less parameters.
E XPERIMENTS
This section ﬁrst veriﬁes the theoretical results with synthetic datasets, and then conducts ablation studies on the bottleneck blocks in ResNet (He et al., 2016) with different K/R ratios.
4. N UMERICAL ANALYSIS FOR THEORETICAL RESULTS
We choose four settings to verify the sample complexity in Theorem 1; see Table 1.
The parameter tensors { A k , B k } Kk =1 are generated to have standard normal entries.
We consider two types of
[Table]
inputs: the independent inputs are generated with entries being standard normal random variables, and the time-dependent inputs { x i } are generated from a stationary VAR(1) process.
The random additive noises are generated from the standard normal distribution.
The number of training samples n varies such that (cid:112) d M /n is equally spaced in the interval [0 . 15 , 0 .
60] , with all other parameters ﬁxed.
For each n , we generate 200 training sets to calculate the averaged estimation error in Frobenius norm.
It can be seen that the estimation error increases linearly with the square root of d M /n , which is consistent with our ﬁnding in Theorem 1.
For Theorem 2, we also adopt four different settings; see Table 1.
Here, we consider 4D input tensors.
The stacked kernel A stack is generated by (8), where the core tensor has standard normal entries and the factor matrices are generated to have orthonormal columns.
The number of training samples n is similarly chosen such that (cid:113)
d TU M /n is equally spaced.
A linear trend between the estimation error and the square root of d TU M /n can be observed, which veriﬁes Theorem 2.
For details of implementation, we employed the gradient descent method for the optimization with a learning rate of 0.01 and a momentum of 0.9.
The procedure is deemed to have reached convergence if the target function drops by less than 10 − 8 .
4. A BLATION STUDIES ON COMPUTATIONALLY - EFFICIENT BLOCK DESIGNS
Following the notation in Section 3, we denote by K , the number of output channels of a bottleneck block, and by R , the Tucker rank of A stack along the output channels dimension.
Since K ≥ R , we let t =
K/R be the expansive ratio of the block.
As t increases, the number of parameters within the block increases.
Here, ablation studies are conducted to show that the increase in t does not necessarily lead to the increase in test accuracy.
We analyze two image recognition datasets, CIFAR10 (Krizhevsky et al., 2009) and Street View House Numbers (SVHN) (BNetzer et al., 2011), and an action recognition dataset, UCF101 (Soomro et al., 2012).
Standard data pre-processing and augmentation techniques are adopted to all three datasets (He et al., 2016;
Hara et al., 2017).
Network architecture.
Two network architectures are considered in our study; as shown in Figure 4.
For image recognition datasets, we adopt a 41-layer Residual Network: it consists of a 3 ×convolution layer, a max pooling layer, followed by 3 groups of A + x × B residual blocks with different R , and ends with an average pooling and fully-connected layer.
Block B represents the bottleneck structure that we are interested in.
The standard bottleneck block corresponds to t = 4 .
When t = 1 , it corresponds to the normalized funnel block in Section 3 (T2).
For sake of
[Table]
comparison, we also take t = 8 or 16.
When t > 1 , Block A is the standard downsampling block that contains convolution with the stride size of 2.
It is inefﬁcient according to our study.
So when t = 1 , we propose minor revisions to simultaneously increase the channels while performing the stride-2 3 × 3 convolution.
For UCF101 dataset, we use 3D ResNet-18 as the basic framework and insert 4 stacks of Block B with R = 256 before the average pooling and fully-connected layer.
Results.
The results in Table 2 provide empirical support for our theoretical ﬁnding.
Though the number of parameters in the network increases with larger t , the overall test accuracy remains roughly comparable.
For CIFAR-10, it may appear that the test accuracy for t = 16 is slightly better than t =
but it uses 5 times as many parameters.
Implementation details.
All experiments are conducted in PyTorch and on Tesla V100-DGXS.
Following the practice in He et al.
(2016), we adopt batch normalization(BN)
(Ioffe & Szegedy, 2015) right after each convolution and before the ReLU activation.
For network (1), we initialize the weights as in He et al.
We use stochastic gradient descent with weight decay 10 − 4 , momentum 0.9 and mini-batch size 128.
The learning rate starts from 0.1 and is divided by 10 for every 100 epochs.
We stop training after 300 epochs, since the training accuracy hardly changes.
The training accuracy is approximately 93% for CIFAR-10 and 96% for SVHN.
We set seeds 1-5 and report the worst case scenario of the test accuracy in Table 2.
For network (2), we follow Hara et al.
and use the Kinetics pretrained 3D ResNet-18.
The weights are ﬁxed for the ﬁrst few layers, and backward propagation is only applied to the added Block B layers and the fully-connected layer.
We use stochastic gradient descent with weight decay 10 − 5 , momentum 0.9 and mini-batch sizewith 16 frames per clip.
The learning rate starts from 0.001 and divided by 10 for every 30 epochs.
Training is stopped after 80 epochs and top-1 clip accuracy is around 79% for UCF
C ONCLUSION AND DISCUSSION
Our paper proposes a uniﬁed theoretical framework that can adopt sample complexity analysis as a tool to study the effective number of parameter in a tensor decomposed CNN block.
The main practical takeaways are: (i) For a large kernel tensor A (Peng et al., 2017), it is always effective to impose low-rank structure on its input dimensions, such as height, width or time-length, etc; and (ii) It is essential to maintain K/R = 1 for a CNN block in order to increase accuracy in a more parameter efﬁcient way.
In this regard, one can either choose a small R with deeper networks or a larger R with shallower networks.
In fact, when K/R = 1 , the block corresponds exactly to the “straightened” bottleneck in Zagoruyko & Komodakis (2016), who showed in their empirical study that by increasing R in shallow networks, the test accuracy can indeed improve quite signiﬁcantly.
Since the low-dimensional structure at convolution layers is our focus, tensor decomposition has been applied to the kernel weights A stack only in Section 3.
Kossaiﬁ et al.
(2017) applied tensor decomposition to fully-connected layers since they may contain a large number of parameters as well.
Along the line, we can stack the fully-connected weights B k ∈ R p 1 × p 2 ×···× p N with 1 ≤ k ≤
K into a higher order tensor B stack
∈
R p 1 × p 2 ×···× p N × K and further assume it to have a Tucker or
CP form.
This corresponds exactly to the Tensor Regression Layer (TRL) introduced in (Kossaiﬁ et al., 2020a), and similar theoretical analysis can be obtained.
We leave it for future research.
A A PPENDIX
This appendix contains ﬁve sections.
In the ﬁrst section, we provide details for our mathematical formulation for a high-order 3-layer CNN.
In the next section, we discuss the formulation of a 5layer CNN as an illustrating example of how to add more layers to our framework.
A complete version of Theorems 1&2 together with the technical proofs for Theorems 1&
and Corollaryare provided in the third section.
In the fourth section, we extend our settings to classiﬁcation and subsequently establish theorem and corollaries for binary and multiclass classiﬁcation problem.
More implementation details and results for additional experiments are presented in the last section.
A. CNN FORMULATION
For a tensor input X ∈ R d 1 ×···× d N , it ﬁrst convolutes with an N -dimensional kernel tensor A ∈ R l 1 ×···× l N with stride sizes equal to s c , and then performs an average pooling with pooling sizes equal to ( q 1 , · · · , q N ) .
It ends with a fully-connected layer with the weight tensor B and produces a scalar output.
We assume that m j = ( d j − l j ) /s
c
+ 1 are integers for 1 ≤ j ≤ N , otherwise zero-padding will be needed.
For ease of notation, we take the pooling sizes { q j } Nj =1 to satisfy the relationship m j =
p
j q j .
To duplicate the operation of the convolution layer using a simple mathematical expression, we ﬁrst need to deﬁne a set of matrices { U ( j ) i j ∈ R d j × l j } , where
[Equation]
U ( j ) i j acts as a positioning factor to transform the kernel tensor A into a tensor of same size as the input X , with the rest of the entries equal to zero.
We are ready to construct our main formulation for a 3-layer tensor CNN.
To begin with, we ﬁrst illustrate the process using a vector input x ∈ R d , with a kernel vector a ∈ R l ; see Fig 5.
Using the i th positioning matrix U (1) i , we can propagate the small kernel vector a , into a vector U (1) i a of size
R d , by ﬁlling the rest of the entries with zeros.
The intermediate output vector has entries given by x c ( i ) =
(cid:104)
x , U (1)
i a (cid:105) , for 1 ≤ i ≤ m .
The average pooling operation is equivalent to forming p consecutive vectors within x c , each of size R q , and then take the average.
This results in a vector x cp of size R p , with i th entry equal to x cp ( i ) =
q
[Equation]
[Equation]
The fully-connected layer performs a weighted summation over the p vectors, with weights given by entries of the vector b ∈ R p .
This gives us the predicted output (cid:98)
y = (cid:104) b , x cp (cid:105) =
(cid:104) x , w X (cid:105) , where
[Equation]
and U
[Equation] [Equation]
[Equation]
1 ” represents the mode-1 product.
For matrix input X with matrix kernel A , however, we need 2 sets of positioning matrices, { U (1) i 1 } m 1 i 1 =1 and { U (2) i 2 } m 2 i 2 =1 , one for the height dimension and the other for width.
Then, the intermediate output from convolution has entries given by X c ( i 1 , i 2 ) =
(cid:104) X , U (1)
i 1 AU (2) (cid:48)
i 2 (cid:105) , for 1 ≤ i 1 ≤ p 1 , 1 ≤ i 2 ≤ p 2 .
For the average pooling, we form p 1 p 2 consecutive matrices from X c , each of size R q 1 × q 2 and take the average.
This results in X cp ∈ R p 1 × p 2 , with X cp
( i 1 , i 2 )
= (cid:104) X , ( q − 1 1 (cid:80) i 1 q
[Equation]
=(
i
[Equation]
+
[Equation]
U (
[Equation] [Equation]
A ( q − 1 2 (cid:80) i 2 q 2 k 2 =( i 2 − 1) q 2 +1 U (2) k 2 ) (cid:48) (cid:105) .
The output X cp goes through fully-connected layer with weight matrix B and gives the predicted output (cid:98)
y = (cid:104) X , W X (cid:105) , where
W X
= p
[Equation]
i
[Equation]
, with U ( j ) F = q − 1 j ( (cid:80) q j k =1 U ( j ) k , · · · , (cid:80) m j k = m
From here, together with the case of high-order tensor input discussed in Session 2.2, we can then derive the form of predicted outcome as (cid:98)
y = (cid:104) X , W single X (cid:105) , where
[Equation]
With K kernels, we denote the set of kernels and the corresponding fully-connected weight tensors as { A k , B k } Kk =1 .
Since the convolution and pooling operations are identical across kernels, we can use a summation over kernels to derive the weight tensor for multiple kernels, which is
[Equation]
and we arrive at the formulation for 3-layer tensor CNN.
A. F IVE - LAYER CNN FORMULATION
Consider a 5-layer CNN with ”convolution → pooling → convolution → pooling → fully connected” layers and a 3D tensor input X ∈ R d 1 × d 2 × d 3 .
Here, we denote the intermediate output from the ﬁrst convolution by X c ∈ R m 1 × m 2 × m 3 , from the second convolution by (cid:101)
X c ∈ R ˜ m 1 × ˜ m 2 × ˜ m 3 , and the output from the ﬁrst pooling by X cp ∈ R p 1 × p 2 × p 3 and from the second pooling by (cid:101)
X cp ∈ R ˜ p 1 × ˜
p 2 × ˜ p 3 .
We can ﬁrst see that the predicted output from the 5-layer CNN is the same as directly feeding X cp to the second convolution layer, followed by an average pooling layer and a fully-connected layer.
Denote the ﬁrst convolution kernel tensor by A ∈ R l 1 × l 2 × l 3 , the second convolution kernel tensor by A ∈ R ˜ l 1 × ˜ l 2 × ˜ l 3 , and the fully-connected weight tensor by B ∈ R ˜ p 1 × ˜ p 2 × ˜ p 3 .
Deﬁne the set of matrices { U ( j ) i j ∈ R d j × l j } , for 1 ≤ i j ≤ m j , 1 ≤ j ≤ N as in (2).
And similarly, deﬁne { (cid:101) U ( j ) i j ∈ R p
j × ˜ l j } , with
[Equation]
where ˜ s c is the stride size for the second convolution.
Let (cid:101) U
[Equation]
˜ q j +1 (cid:101) U
( j ) k ) .
We further stack the matrices { U ( j ) F ,i j ∈
R d j × l j } 1 ≤ i j ≤ p j and { (cid:101) U ( j ) F ,i
j ∈ R p
j × ˜ l j } 1 ≤ i j ≤ ˜ p
j into 3D tensors U ( j ) ∈ R
d j × l j × p j and (cid:101) U ( j ) ∈
R p j × ˜ l j × ˜ p
j .
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where (cid:101) u i j = vec( (cid:101) U (1) ( i j , : , :)) ∈ R ˜
l j ˜ p
j and U ( j ) DF
=
( U ( j ) × 3 (cid:101) U ( j ) )
(1) , for 1 ≤ i j ≤ p j and 1 ≤ j ≤ 3 .
A. T HEORETICAL RESULTS AND TECHNICAL PROOFS
Let Z i
=
X i × 1 U (1) (cid:48)
F × 2 U (2) (cid:48)
F × 3 · · · × N U ( N ) (cid:48) F ∈ R l 1 p 1 × l 2 p 2 ×···× l N p N .
The trained weights have the form of (cid:99) W = (cid:80)
Kk =1 (cid:98) B k ⊗ (cid:98) A k , where { (cid:98) B k , (cid:98)
A
[Equation] [Equation]
= arg min B k , A
[Equation] [Equation]
K n n
[Equation]
i =
[Equation]
(cid:32)
y
[Equation]
−
K (cid:88) k
=
[Equation]
(cid:104)
[Equation]
B
[Equation]
(cid:33)
[Equation]
Given a test sample ( X , y ) , where y = (cid:104) X , W X (cid:105) + ξ , and ( X , ξ ) satisﬁes Assumptions 1 and 2.
The mean-square training and test error are deﬁned as err
[Equation]
Denote κ U
= C x C u
and κ L = c x c u .
Let W ∗ =
(cid:80)
Kk =1 B ∗ k ⊗ A ∗
k be the true weights, and (cid:107) W
[Equation]
2 n := n − 1 (cid:80) ni
=1 ( (cid:104)
Z i , W (cid:105) ) 2 be the empirical norm with respect to W .
The model complexity of CNN at equation (1) or (2) is d M = K ( P + L + 1) .
Theorem 1 (Complete Version) .
Suppose that Assumptions 1-3 hold and n (cid:38) ( κ U /κ L ) 2 d M .
Then, for some δ > 0 ,
[Equation]
a )
[Equation]
with probability 1 − 4 exp {− [ c H (0 .
25 κ L /κ U )
2 n − 9 d M ] }− exp {− d M − 8 δ } , where c H is a positive value, α RSC = κ L / 2 , and α RSM = 3 κ U / 2 .
It can be seen that the estimation error, (cid:107) (cid:99) W − W ∗ (cid:107) F , is O p ( (cid:112)
d M /n ) , and hence the sample complexity is of order O
( d M /ε 2 ) , to achieve a training error ε .
The empirical norm is equivalent to the training error minus the sample standard deviation of noise.
And we can see that the training error err train ( (cid:99) W )
=
O p ( (cid:112) d M /n )
+ (cid:98) σ and test error err ( (cid:99) W ) =
O p ( (cid:112) d M /n )
+ σ , where (cid:98) σ and σ represent the sample and population standard deviation of noise, respectively.
Now we proceed to provide a complete version of the sample complexity of compressed CNNs.
Let d TU M = (cid:81)
N +1 j =1 R j + (cid:80) Ni =1 l i R i + R N +1 P and d CP M = R N +
+ R ( (cid:80)
Ni =
1 l i + P ) .
Theorem 2 (Complete Version) .
Let ( (cid:99) W , d M ) be ( (cid:99) W TU , d TU M ) for Tucker decomposition, or ( (cid:99) W CP , d CP M ) for CP decomposition.
Suppose that Assumptions 1-3 hold and n (cid:38) ( κ U /κ L ) 2 c N d M .
Then, for some δ > 0 ,
[Equation]
with probability 1 − 4 exp {− [ c H (0 .
25 κ L /κ U )
2 n − 3 c N d M ] } − exp {− d M − 8 δ } , where c N = 2 N +1 log( N + 2) , c H is a positive value and α RSC , α RSM are deﬁned in Theorem 1.
Because CP decomposition can be considered as a special case of Tucker decomposition where
R j = R , for 1 ≤ j ≤ N + 1 .
We only provide the proof for Tucker CNN.
A.3. P ROOF OF T HEOREM
Denote the sets (cid:98) S K = { (cid:80)
Kk =1 B k ⊗ A k : A k ∈ R l 1 × l 2 ×···× l N and B k ∈ R p 1 × p 2 ×···× p N } and
S K = { W ∈ (cid:98)
S K : (cid:107) W (cid:107) F = 1 } .
Let (cid:98) ∆ =
(cid:99) W − W ∗ , and then
[Equation]
which implies that
[Equation]
(11) where (cid:107) ∆ (cid:107) 2 n := n − 1 (cid:80) ni
=1 ( (cid:104) Z i , ∆ (cid:105) ) 2 is the empirical norm with respect to ∆ , and (cid:98) ∆ ∈ (cid:98) S 2 K .
Consider a ε -net ¯ S 2 K , with the cardinality of N (2 K, ε ) , for the set S 2 K .
For any ∆ ∈ S 2 K , there exists a ¯ ∆ j ∈ ¯ S 2 K such that (cid:107) ∆ − ¯ ∆ j (cid:107) F ≤ ε .
Note that ∆ − ¯ ∆ j ∈ (cid:98) S 4 K and, from Lemma 1(a), we can further ﬁnd ∆ 1 , ∆ 2 ∈ (cid:98) S 2 K such that (cid:104) ∆ 1 , ∆ 2 (cid:105) = 0 and ∆ − ¯ ∆ j = ∆ 1 + ∆ 2 .
It then holds that (cid:107) ∆ 1 (cid:107) F + (cid:107) ∆ 2 (cid:107) F ≤ √ 2 (cid:107) ∆ − ¯ ∆ j (cid:107) F ≤ √ 2 ε since (cid:107) ∆ − ¯ ∆ j (cid:107) 2F = (cid:107) ∆ 1 (cid:107) 2F + (cid:107) ∆ 2 (cid:107) 2F .
As a result,
[Equation]
which leads to
[Equation]
Note that, from Lemma 1(b), log N (2 K, ε ) ≤ 2 d M log(9 /ε ) , where d M = K ( P + L + 1) .
Let ε
=
(2 √ 2) − 1 , and then 8 − 2 log(9 /ε ) > 1 .
As a result, by (12) and Lemma 3,
[Equation]
Note that
[Equation] [Equation]
≤ P (cid:34)
[Equation]
+
[Equation]
(cid:107) ∆ (cid:107)
[Equation]
n
[Equation]
(cid:19) .
From (13) and Lemma 2, we then have that, with probability 1 − 4 exp {− [ c H (0 .
25 κ L /κ U )
2 n − 9 d M ] } − exp {− d M − 8 δ } , sup ∆ ∈S 2 K n
[Equation]
i =
[Equation]
(cid:104)
[Equation]
(cid:105)
[Equation]
√
[Equation]
RSM (cid:32)(cid:114)
d M n + (cid:114) δ n (cid:33) and (cid:107) ∆ (cid:107) 2 n ≥ α RSC (cid:107) ∆ (cid:107) 2F , which, together with (11), leads to (cid:107) (cid:98) ∆ (cid:107) F ≤ 16 σ √ α RSM α RSC (cid:32)(cid:114)
d M n + (cid:114) δ n (cid:33) and (cid:107) (cid:98) ∆ (cid:107) n ≤ 16 σ α RSM α RSC (cid:32)(cid:114)
[Equation]
Finally we prove (c).
Given a test sample ( X , y ) , and let Z = X
× 1 U (1) (cid:48)
F × 2 U (2) (cid:48)
F × 3 · · ·× N U ( N ) (cid:48) F .
It holds that
[Equation]
and
E
[Equation]
As a result, from (a) of this theorem, we have
[Equation]
This accomplishes the proof.
A.3. P ROOF OF T HEOREM
Denote the sets (cid:98) S TU ( R 1 , · · · , R N +1 ) =
{ (cid:80) Kk =1 B k ⊗ A k : the stacked kernel A stack ∈
R l 1 × l 2 ×···× l N × K has the ranks of ( R 1 , ..., R N , R N +1 ) and B k ∈ R p 1 × p 2 ×···× p N }
, (cid:98) S TU2 K =
{ W 1 + W 2 : W 1 , W 2 ∈ (cid:98) S TU ( R 1 , · · · , R N +1 ) } , and S TU2 K = { W ∈ (cid:98) S TU2 K : (cid:107) W (cid:107) F = 1 } .
Note that (cid:99) W TU , W ∗ ∈ (cid:98) S TU ( R 1 , · · · , R N +1 ) , and (cid:98) ∆ =
(cid:99) W TU − W ∗ ∈ (cid:98) S TU 2 K .
We ﬁrst consider a ε -net for S TU2 K .
For each 1 ≤ k ≤ K , Let b k
=
b k 1 , . . .
, b kP ) (cid:48)
= vec( B k ) , and we can rearrange B k ⊗ A k into the form of A k ◦ b k , which is a tensor of size R l 1 × l 2 ×···× l N × P , where P = p 1 p 2 · · · p N .
Denote B = ( b kj ) ∈
R P × K , and it holds that
[Equation]
B , which is a tensor with the size of R l 1 × l
[Equation]
l N × P and the ranks of ( R 1 , ..., R N , R N +1 ) where (cid:101) B
= BH ( N +1) ∈ R P × R N
+1 .
Essentially, in this step, we rewrite the model into one with R k + kernels instead.
Speciﬁcally, we now have W = (cid:80) R
N +1 r
=
(cid:101)
A r ⊗ (cid:101) B r , where (cid:101)
A r = G r × 1 H (1) ×
H (2) · · · × N H ( N ) with G r = G (: , : , · · · , r ) and the r -th column of (cid:101) B is the vectorization of (cid:101) B r for 1 ≤ r ≤ R N +1 .
As a result, (cid:98) S TU2 K consists of tensors with the ranks of (2 R 1 , ..., 2 R N , 2 R N +1 ) at most.
Denote S Tucker ( r 1 , · · · , r N , r N +1 )
= { T ∈ R
[Equation] [Equation]
N +1 : (cid:107) T (cid:107) F = 1 , T has the Tucker ranks of ( r 1 , · · · , r N , r N +1 ) } , where l N +1 = P = p 1 p 2 · · · p N .
Then the ε -covering number for S TU2 K satisﬁes
[Equation]
For each T ∈ S Tucker ( r 1 , · · · , r N , r N +1 ) , we have
[Equation]
U
[Equation]
) , where G ∈ R r 1 ×···× r N × r N +1 with (cid:107) G (cid:107) F = 1 , and U ( i ) ∈
R l i × r i with 1 ≤ i ≤ N + 1 are orthonormal matrices.
We now construct an ε -net for S Tucker ( r 1 , · · · , r N , r N +1 ) by covering the sets of G and all U ( i ) s, and the proof hinges on the covering number of low-multilinear-rank tensors in Wang et al.
Treating G as (cid:81)
N +1 j =1 r j -dimensional vector with (cid:107) G (cid:107) F = 1 , we can ﬁnd an ε/ ( N + 2) -net
for it, denoted by ¯ G , with the cardinality of | ¯ G | ≤ (3( N + 2) /ε )
(cid:81)
N +1 j =1 r j .
Next, let O n,r = { U ∈ R n × r : U (cid:62) U
=
I r } .
To cover O n,r , it is beneﬁcial to use the (cid:107) · (cid:107) 1 , 2 norm, deﬁned as
[Equation]
where X i denotes the i th column of X .
Let Q n,r = { X ∈ R n × r : (cid:107) X (cid:107) 1 , 2 ≤ 1 } .
One can easily check that O n,r ⊂ Q n,r , and then an ε/ ( N + 2) -net
¯ O n,r for O n,r has the cardinality of | ¯ O n,r | ≤ (3( N + 2) /ε )
nr .
Denote ¯ S Tucker ( r 1 , · · · , r N , r N +1 )
=
{ G × 1 U (1) × 2 · · · × N +
U ( N +1) :
G ∈ ¯ G, U ( i ) ∈
¯ O l i ,r i , 1 ≤ i ≤ N + 1 } .
By a similar argument presented in
Lemma A.1 of Wang et al.
(2019), we can show that ¯ S Tucker ( r 1 , · · · , r N , r N +1 ) is a ε -net for the set S Tucker ( r 1 , · · · , r N , r N +1 ) with the cardinality of
[Equation]
where l N +1 = P .
Thus, the ε -covering number of S TU2 K is
N TU ( ε ) =
|S Tucker (2 R
[Equation] [Equation]
) |
[Equation]
Let ε = 1 / 2 .
It then holds that log(3( N + 2) /ε ) <
3 log( N + 2) and log N TU ( ε ) ≤ 2 N +1 d TU M log[(3 N + 6) /ε ] , where d TU M = (cid:81) N +1 j =1 R j + (cid:80) Ni =1 l i R i + R N +1 P .
By a method similar to the proof of Lemma 2, we can show that
[Equation]
and
[Equation]
where c N = 2 N +1 log( N + 2) .
Moreover, similar to equation 13, by applying Lemma 3, we can show that
[Equation] [Equation]
where ε = (2 √ 2) − 1 and 2 N +4 log( N + 2) − 2 N +1 log(3( N + 1) /ε ) > 1 .
By a method similar to the proof of Theorem 1, we can show that, with probability 1 − 4 exp {− [ c H (0 .
25 κ L /κ U )
2 n − 3 c N log( N + 2)
d TU M ] } − exp {− d TU M − 8 δ } , sup
[Equation] [Equation]
and
[Equation]
2F , which, together with equation 11, leads to
[Equation]
We accomplished the proof.
A.3. P ROOFS OF C OROLLARY
When the kernel tensor A has a Tucker decomposition form of A = G × 1 H (1) × 2 H (2) × 3 · · ·× N +
H ( N +1) , and the multilinear rank is ( R 1 , R 2 , · · · , R N +1 ) .
Let H =
G × 1 H (1) × 2 H (2) × 3 · · ·× N
H ( N ) , which is a tensor of size R l
[Table]
l N × R N +1 .
The mode- ( N + 1) unfolding of H is a matrix with R N +1 row vectors, each of size R L , and we denote them by g 1 , · · · , g R N +1 .
Fold g r s back into tensors, i.e. g r = vec( (cid:101) H r ) , where (cid:101) H r ∈ R l 1 ×···× l N and 1 ≤ r ≤ R N +1 .
Moreover, let
H ( N +1)
=
( h ( N +1), · · · , h ( N +1) K ) (cid:48) , where h ( N +1)
k is a vector of size R R N +
and we denote its r th entry as h ( N +1) k,r , for 1 ≤ r ≤ R N +1 .
It can be veriﬁed that, for each 1 ≤ k ≤ K ,
A k = G × 1 H (1) × 2 H (2) · · · × N H ( N )
¯ × N +1 h ( N +1) k
= H ¯ × N +1 h ( N +1) k , and hence,
K (cid:88) k
=
B k ⊗ A k =
R N +1 (cid:88) r =
K (cid:88) k
=
B k ⊗ (cid:16) (cid:101) H r · h ( N +1) k,r (cid:17) =
R N +1 (cid:88)
r = (cid:32) K (cid:88) k
= h ( N +1) k,r
B k (cid:33) ⊗ (cid:101) H r .
By letting (cid:101) B r = (cid:80) K k =1 h ( N +1) k,r
B k and (cid:101)
A r = (cid:101) H r for 1 ≤ r ≤ R N +1 , we can reformulate the model into y i
= (cid:104)
R N +1 (cid:88)
r = (cid:101)
[Equation]
and the proof of (a) is then accomplished.
Suppose that the kernel tensor A has a CP decomposition form of A = (cid:80)
Rr =1 α r h
(1) r ◦ h (2) r ◦ · · · ◦ h ( N +1) r , where h ( j ) r ∈ R l j and (cid:107) h
( j ) r (cid:107) =
1 for all 1 ≤ j ≤ N .
Moreover, h ( N +1) r =
( h ( N +1) r,, ..., h ( N +1) r,K ) (cid:48)
∈ R K and (cid:107) h
( N +1) r (cid:107) = 1 .
Note that
[Equation]
for all 1 ≤ k ≤ K , and hence
[Equation]
By letting (cid:101) B r = h ( N +1) r,k
B k and (cid:101)
A r = α r h (1) r ◦ h (2) r ◦ · · · ◦ h ( N ) r for all 1 ≤ r ≤ R , we can reparameterize the model into
[Equation]
+ ξ i , and the proof of (b) is then accomplished.
A.3. L EMMAS U SED IN THE P ROOFS OF T HEOREMS
Lemma 1.
(Partition and covering number of restricted spaces) Consider (cid:98) S K = { (cid:80)
Kk =1 B k ⊗ A k :
A k ∈ R l 1 × l 2 ×···× l N and B k ∈ R p 1 × p 2 ×···× p N } and S K
= { W ∈ (cid:98) S K : (cid:107) W (cid:107) F = 1 } .
(a) For any ∆ ∈ (cid:98) S 2 K , there exist W 1 , W 2 ∈ (cid:98) S K such that ∆ =
W 1 + W 2 and (cid:104) W 1 , W 2 (cid:105)
= 0 .
(b) The ε -covering number of the set S K is
[Equation] [Equation]
[Equation]
, where L = l 1 l 2 · · · l N and P = p 1 p 2 · · · p N .
Proof.
For each K , we ﬁrst deﬁne a map T K : (cid:98) S K
→ R P × L .
For any W = (cid:80) Kk =1 B k ⊗ A k ∈ (cid:98) S K , we deﬁne that T K ( W ) =
(cid:80) Kk =1 vec ( B k ) vec( A k ) (cid:48)
∈ R P × L , which has the rank of at most K .
For any ∆ ∈ (cid:98) S 2 K , the rank of matrix T K ( ∆ ) is at most 2 K .
As shown in Figure 6, we can split the singular value decomposition (SVD) of T K ( ∆ ) into two parts, and it can be veriﬁed that T K ( ∆ ) =
U 1 Λ 1 V 1 (cid:48)
+ U 2 Λ 2 V 2 (cid:48) and (cid:104)
U 1 Λ 1 V 1 (cid:48) , U 2 Λ 2 V 2 (cid:48) (cid:105) = 0 .
Denote U
i
=
( u ( i ) 1 , u ( i ) 2 , . . .
, u ( i ) K ) and
V i Λ (cid:48)
i =
( v ( i ) 1 , v ( i ) 2 , . . .
, v ( i ) K ) for i = 1 and 2.
We then fold u ( i ) j s and v ( i )
j s into tensors, i.e. u ( i ) k
= vec( B ( i ) k )
and v ( i ) k
= vec( A ( i ) k ) with 1 ≤ k ≤ K .
Let W 1 = (cid:80)
Kk =1 B (1) k ⊗ A (1) k and
W 2 = (cid:80)
Kk =1 B (2) k ⊗ A (2) k .
It then can be veriﬁed that W 1 , W 2 ∈ (cid:98) S K , ∆ =
W 1 + W 2 and (cid:104) W 1 , W 2 (cid:105)
= 0 .
Thus, we accomplish the proof of (a).
Denote by S matrix ⊂ R P
×
L the set of matrices with unit Frobenius norm and rank at most K .
Note that T ( S K ) ⊂
S matrix , while the ε -covering number for S matrix is |S matrix | ≤ (9 /ε )
K ( L + P +1)
; see Candes & Plan (2011).
This accomplishes the proof of (b).
Lemma 2.
(Restricted strong convexity and smoothness) Suppose that n (cid:38) ( κ U /κ L ) 2 d M .
It holds that, with probability at least 1 − 4 exp {− [ c H (0 .
25 κ L /κ U )
2 n − 9 d M ] } ,
[Equation]
for all ∆ ∈ (cid:98) S 2 K , where (cid:98) S 2 K is deﬁned in Lemma 1, c H is a positive constant from the HansonWright inequality, α RSC
= κ L / 2 and α RSM = 3 κ U / 2 .
Proof.
It is sufﬁcient to show that sup ∆ ∈S 2 K (cid:107) ∆ (cid:107) 2 n ≤ α RSM and inf ∆ ∈S 2 K (cid:107) ∆ (cid:107) 2 n ≥ α
RSC hold with a high probability, where (cid:107) ∆ (cid:107) 2 n := n − 1 (cid:80) ni =1 ( (cid:104)
Z i , ∆ (cid:105) ) 2 is the empirical norm, and S 2 K is deﬁned in Lemma 1.
Without confusion, in this proof, we will also use the notation of ∆ for its vectorized version, vec( ∆ ) .
Note that
[Equation]
where Q = n − 1 Σ 12 [ I n ⊗ ( U G ∆∆ (cid:48) U (cid:48) G )] Σ 12 , ¯ w ∈ R nD is a random vector with i.i.d.
standard normal entries, z i
=
U (cid:48)
G x i , x i = vec( X i ) , ¯ x = ( x 1 (cid:48) , x 2 (cid:48) , . . .
, x n (cid:48) )
(cid:48) , and Σ = E (¯ x ¯ x (cid:48) )
∈ R nD × nD .
Denote w G = U G ∆ ∈ R D and S n − 1 = { v ∈ R n : (cid:107) v (cid:107) 2 = 1 } .
Let λ max ( Σ ii ) be the maximum eigenvalue of Σ ii
= E (vec ( X i ) vec ( X i ) (cid:48) ) for 1 ≤ i ≤ n , and it holds that λ max ( Σ ii )
≤ C x for all 1 ≤ i ≤ n .
For matrix Q , we have
[Equation]
[Equation]
and similarly, we can show that tr( Q ) ≥ κ L , where κ L = c
x c u , κ U
= C x C u and κ L ≤ κ U .
Thus,
[Equation]
Moreover
[Equation]
denote Q 1 = n − 12 Σ 12 [ I n ⊗ w G ] and note that Q = Q 1 Q (cid:48) 1 .
To bound the operator norm of Q
[Equation]
we have
[Equation] [Equation]
Finally we can use (18) and (20) to bound the Frobenius norm of Q .
By some algebra, for any square matrices A , B ∈ R n × n , (cid:107) AB (cid:107)
[Equation]
2op
[Equation]
B
[Equation]
2F holds.
Hence,
[Equation]
This, together with (17), (20) and the Hanson-Wright inequality (Vershynin, 2018, Chapter 6), leads to
[Equation]
where c H is a positive constant.
On the other hand,
[Equation]
where (cid:98) Γ = n − 1 (cid:80) ni
=
[ z i z i (cid:48)
− E ( z i
z i (cid:48) )] is a symmetric matrix.
Consider a ε -net ¯ S 2 K , with the cardinality of N (2 K, ε ) , for the set S 2 K .
For any ∆ ∈ S 2 K , there exists a ¯ ∆ j ∈ ¯ S 2 K such that (cid:107) ∆ − ¯ ∆ j (cid:107) F ≤ ε .
Note that ∆ − ¯ ∆ j ∈ (cid:98) S 4 K and, from Lemma 1, we can further ﬁnd ∆ 1 , ∆ 2 ∈ (cid:98) S 2 K such that (cid:104) ∆ 1 , ∆ 2 (cid:105) = 0 and ∆ − ¯ ∆ j = ∆ 1 + ∆ 2 , and it then holds that (cid:107) ∆ 1 (cid:107) F + (cid:107) ∆ 2 (cid:107) F ≤ √ 2 (cid:107) ∆ − ¯ ∆ j (cid:107) F ≤ √ 2 ε .
Moreover, for a general real symmetric matrix A ∈ R d × d , u ∈ R d and v ∈ R d , sup (cid:107) u (cid:107) 2 = (cid:107) v (cid:107) 2 =1 u (cid:48)
Av = sup (cid:107) u (cid:107) 2 =1 u (cid:48)
Au .
As a result,
[Equation]
, where ( (cid:107) ∆ 1 (cid:107) F + (cid:107) ∆ 2 (cid:107) F ) 2 + 2( (cid:107) ∆ 1 (cid:107) F +
[Equation]
∆
[Equation]
F
[Equation]
∆
j
[Equation]
F ≤
[Equation]
ε + √
[Equation]
ε ≤ 5 ε as ε ≤ 1 , and this leads to sup
[Equation]
Note that, from Lemma 1(b), log N (2 K, ε ) ≤ 2 d M log(9 /ε ) , where d M = K ( L + P + 1) .
Let ε = 1 / 10 , and then 2 log(9 /ε ) < 9 .
Combining (21) and (22) and letting t = κ L / 2 , we have
[Equation]
which, together with (19) and the fact that κ L ≤ κ U , implies that
[Equation]
23) where α RSM = 1 .
5 κ U ≥ κ U
+ κ L / 2 .
By a method similar to (22), we can also show that sup ∆ ∈S
[Equation]
which, together with (21)
[Equation]
leads to
[Equation]
where t = κ L / 2 and ε = 1 / 10 .
As a result,
[Equation]
where α RSC = κ L − κ L / 2 .
This, together with (23), accomplishes the proof.
Lemma 3.
(Concentration bound for martingale) Let { ξ i , 1 ≤ i ≤ n } be independent σ 2 -subGaussian random variables with mean zero, and { z i , 1 ≤ i ≤ n } is another sequence of random variables.
Suppose that ξ i is independent of { z i ,
z i − 1 , . . .
, z 1 } for all 1 ≤ i ≤ n .
It then holds that, for any real numbers α, β > 0 ,
[Equation]
Proof.
We can prove the lemma by a method similar to Lemma 4.2 in Simchowitz et al.
A. C LASSIFICATION P ROBLEMS
Starting from Section 2.2, we know that, for each input tensor X , the intermediate scalar output after convolution and pooling has the following form
[Equation]
where W = (cid:80) Kk =1 B k ⊗ A k and Z =
X × 1 U (
[Equation]
) (cid:48)
F
[Equation]
N is the k th kernel and B k ∈ R p 1 × p 2 ×···× p N is the corresponding k th fully-connected weight tensor.
Consider a binary classiﬁcation problem.
We have the binary label output y ∈ { 0 , 1 } with p ( y | Z )
=
(cid:18)
[Equation]
Suppose we have samples { Z i , y i } ni =1 , we use the negative log-likelihood function as our loss function.
It is given, up to a scaling of n − 1 by
[Equation]
where φ ( z ) =
log(1 + e z ) and its gradient and Hessian matrix is ∇L
[Equation]
[Equation] [Equation]
with φ (cid:48) ( z ) = 1 / (1 + e − z ) ∈
(0 , 1) and φ (cid:48)(cid:48)
[Equation]
[because e − z + e z ≥ 2 ].
Since H n ( W ) is a positive semi-deﬁnite matrix, our loss function in (24) is convex.
Suppose (cid:99) W is the minimizer to the loss function
[Equation]
where (cid:98) S K
=
(cid:80) Kk =1 B k ⊗ A k : A k ∈ R l 1 × l 2 ×···× l N and B k ∈ R p 1 × p 2 ×···× p N } , and B ( R ) is a
Frobenius ball of some ﬁxed radius R centered at the underlying true parameter.
Similar to Fan et al.
(2019), we need to make two additional assumptions to guarantee the locally strong convexity condition.
Assumption 4 (Classiﬁcation) .
Apart from Assumptions 2-3, we additionally assume that (C1) { vec( X i ) } ni
=1 are i.i.d gaussian vectors with mean zero and variance Σ , where Σ ≤ C x I .
(C2) the Hessian matrix at the underlying true parameter W ∗ is positive deﬁnite and there exists some κ 0 > κ U > 0 , such that H ( W ∗ )
=
E ( H n ( W ∗ )) ≥ κ 0 · I , where κ U =
C x C u ; (C3) (cid:107) W ∗ (cid:107) F ≥ α √ d M for some constant α , where d M = K ( P + L + 1) .
Notice that we can relax (C1) into Assumption 1, but it will require more techincal details.
Also, { vec( X i ) } ni =1 can be sub-gaussian random vectors instead of gaussian random vectors.
Denote d M = K ( P + L + 1) .
Theorem 3 (Classiﬁcation: CNN) .
Suppose that Assumptions 2&3 and Assumption 4 hold and n (cid:38) d M .
Then, for some δ > 0 ,
[Equation]
with probability 1 − 4 exp {− 0 .
25 cn [-( κ L /κ U ) 
-]+ 9 d M } − 2 exp {− c γ d M − cδ } , where δ = O p (1) , c and c γ are some positive constants. Denote d TU M
= (cid:81) N +1 j =1 R j + (cid:80) Ni =1 l i R i + R N +1 P and d CP M = {+R N +
 + R ( (cid:80) Ni = 1 l i + P ) .+}
Corollary 2 (Classiﬁcation: Compressed CNN) .
Let ( (cid:99) W , d M ) be ( (cid:99) W TU , d TU M ) for Tucker decomposition, or ( (cid:99) W CP , d CP M ) for CP decomposition.
Suppose that Assumptions in Theorem 3 hold and n (cid:38) c N d M .
Then, for some δ > 0 ,
[Equation]
with probability 1 − 4 exp {− 0 .
25 cn + 3 c N d M } − 2 exp {− c γ d M − cδ } , where δ = O p (1) , c and c γ are some positive constants, and c N is deﬁned as in Theorem 2.
We consider a binary classiﬁcation problem as a simple illustration.
In fact, the analysis framework can be easily extended to a multiclass classiﬁcation problem.
Here, we consider a M -class classiﬁcation problem.
Because we need our intermediate output after convolution and pooling to be a vector of length M , instead of a scalar, we need to introduce one additional dimension to the fully-connected weight tensor.
Hence, we introduce another subscript m to represent the class label.
And the set of fullyconnected weights is represented as { B k,m } 1 ≤ k ≤ K, 1 ≤ m ≤ M where each B k,m is of size p 1 × p 2 × · · · × p N .
Then, for each input tensor X , our intermediate output is a vector of size M , where the m th entry is represented by
[Equation]
where W m =
(cid:80)
Kk =1 ( B k,m ⊗ A k ) is an N -th order tensor of size l 1 p 1 × l 2 p 2 × · · · × l N p N .
For a M -class classiﬁcation problem, we have the vector label output y ∈ (0 , 1) M .
Essentially, each entry of y comes from a different binary classiﬁcation problem, with M problems in total.
We can model it as
[Equation]
If we stack { W m } Mm =1 into a tensor W stack , which is an N + 1 -order tensor of size l 1 p 1 × l 2 p 2 × · · · × l N p N × M .
And we further introduce some natural basis vectors { e m ∈ R M } Mm =1 .
It can be shown that
[Equation]
where ◦ is the outer product.
We can then recast this model into one with nM samples { Z im , y im : 1 ≤ k ≤ K, 1 ≤ m ≤ M } .
The corresponding loss function is
[Equation]
We can now use the techniques in Theorem 3 to show the following corollaries for multiclass classiﬁcation problem.
Denote d MC M = K ( MP + L + 1) .
Corollary 3 (Multiclass Classiﬁcation: CNN) .
Under similar assumptions as in Theorem 3, suppose that n (cid:38) d MC M .
Then, for some δ > 0 ,
[Equation]
with probability 1 − 4 exp (cid:8) − 0 .
25 cn[-( κ L /κ U ) 
-] + 9 d MC M (cid:9) − 2 exp (cid:8) − c γ d MC M − cδ (cid:9) , where δ = O p (1) , c and c γ are some positive constants.
Denote d MC − TU M = (cid:81)
N +1 j =1 R j + (cid:80) Ni =1 l i R i + R N +1 PM and d {+MC − CP M = R N +
 + R ( (cid:80)+}
Ni = 1 l i +
PM ) .
Corollary 4 (Multiclass Classiﬁcation:
Compressed CNN) .
Let ( (cid:99) W stack , d MC M ) be ( (cid:99) W stack , TU , d MC − TU M ) for Tucker decomposition, or ( (cid:99) W stack , CP , d MC − CP M ) for CP decomposition.
Suppose that Assumptions in Theorem 3 hold and n (cid:38) c N d multi M .
Then, for some δ > 0 , (cid:107)
[Equation]
√
[Equation]
(cid:32)(cid:114)
[Equation]
with probability 1 − 4 exp (cid:8) − 0 .
25 cn + 3 c N d MC M (cid:9) − 2 exp (cid:8) − c γ d MC M − cδ (cid:9) , where δ = O p (1) , c and c γ are some positive constants, and c N is deﬁned as in Theorem 2.
Proof of Theorem 3.
Denote the sets (cid:98) S K = { (cid:80)
Kk =1 B k ⊗ A k : A k ∈ R l 1 × l 2 ×···× l N and B k ∈
R p 1 × p 2 ×···× p N } and S K =
{ W ∈ (cid:98) S K : (cid:107) W (cid:107) F = 1 } .
We further denote ∆ = vec( W − W ∗ ) , where W ∗ is the underlying true parameter and W , W ∗ ∈ (cid:98) S K , and deﬁne the ﬁrst-order Taylor error
[Equation]
Suppose (cid:99) W is the minimizer for the loss function, i.e.,
[Equation]
Denote (cid:98) ∆ = vec( (cid:99) W − W ∗ ) .
We then have
L n ( (cid:99) W ) − L n ( W ∗ ) ≤ 0 , which can be rearranged into E n ( (cid:98) ∆ ) ≤ − (cid:68) ∇L n ( W ∗ ) , (cid:98) ∆ (cid:69) .
Then, for some (cid:102) W between (cid:99) W and W ∗ ,
[Equation]
which leads to (cid:107)
[Equation]
∆
[Equation]
sup [Equation] K ∆
[Equation]
| . From Lemma 4 and Lemma 5, when n (cid:38) d M , we obtain that, for some δ > 0 , (cid:107) (cid:99)
[Equation]
∗
[Equation]
√
[Equation]
κ
[Equation]
(cid:32) (cid:114)
[Equation]
M n + (cid:114) δ n (cid:33) , with probability 1 − 4 exp {− 0 .
25 cn + 9 d M } − 2 exp {− c γ d M − cδ } .
Now we proof several lemmas to be used in Theorem 3.
For simplicity of notation, denote x i = vec( X i ) and z i = vec( Z i ) .
It holds that z i
=
U
G x i , for 1 ≤ i ≤ n .
Lemma 4 (Deviation bound) .
Under Assumption 4(C3), suppose that n (cid:38) d M , then, for some δ > 0 ,
[Equation]
where δ = O p (1) , d M = K
( P + L + 1) , κ U
= C x C u and c γ is some positive constant.
Proof.
Let η i = (cid:104)
Z i , W ∗ (cid:105) , and from (25),
[Equation]
and we can observe that
[Equation]
It implies (i) E (cid:104)∇L n ( W ∗ ) , ∆ (cid:105)
= 0 , (ii) the independence between φ (cid:48)
( η i ) − y i and (cid:104)
z i , ∆ (cid:105) .
And from Lemma 6, the independence leads to
[Equation]
Denote κ U
= C x C u .
For any ﬁxed ∆ such that (cid:107) ∆ (cid:107) 2 = 1 ,
[Equation]
This, together with (cid:107) φ [Equation] i ) − y i (cid:107) ψ 2 ≤ 0 .
25 in Lemma [Equation] gives us
[Equation]
Then [Equation] we can use the Beinstein-type inequality, namely Corollary 5.17 in (Vershynin, 2010) to derive that, for any ﬁxed ∆ with unit l 2 -norm,
[Equation]
Consider a ε -net ¯ S 2 K , with the cardinality of N (2 K, ε ) , for the set S 2 K .
For any ∆ ∈ S 2 K , there exists a ¯ ∆ j ∈ ¯ S 2 K such that (cid:107) ∆ − ¯ ∆ j (cid:107) F ≤ ε .
Note that ∆ − ¯ ∆ j ∈ (cid:98) S 4 K and, from Lemma 1(a), we can further ﬁnd ∆ 1 , ∆ 2 ∈ (cid:98) S 2 K such that (cid:104) ∆ 1 , ∆ 2 (cid:105) = 0 and ∆ − ¯ ∆ j = ∆ 1 + ∆ 2 .
It then holds that (cid:107) ∆ 1 (cid:107) F + (cid:107) ∆ 2 (cid:107) F ≤ √ 2 (cid:107) ∆ − ¯ ∆ j (cid:107) F ≤ √ 2 ε since (cid:107) ∆ − ¯ ∆ j (cid:107) 2F = (cid:107) ∆ 1 (cid:107) 2F + (cid:107) ∆ 2 (cid:107) 2F .
As a result,
[Equation]
which leads to
[Equation]
Note that, from Lemma 1(b), log N (2 K, ε ) ≤ 2 d M log(9 /ε ) , where d M = K ( P + L + 1) .
Let ε
=
(2 √ 2) − 1 and then 2 log(9 /ε ) < 7 .
With (28), we can show that
[Equation]
Take t = 0 .
25 √ κ U ( (cid:112)
d M /n
+ (cid:112) δ/n )
where δ =
O p (1) , and there exists some γ such that (cid:112) d M /n ≤ γ holds.
We can ﬁnally show that
[Equation]
where c γ is some positive constant related to γ .
Lemma 5 (LRSC) .
Suppose that n (cid:38) d M , under Assumptions 4, there exists some constant R > 0 , such that for any W ∈ (cid:98) S 2 K satisfying
[Equation]
holds with probability 1 − 4 exp {− 0 .
25 cn + 9 d M } , where (cid:101) κ 1 = κ 1 − κ U .
And κ 1 is deﬁned in Lemma 8.
Proof.
We divide this proof into two parts.
1. RSC of L n ( W ) at W = W ∗ .
We ﬁrst show that, for all ∆ ∈ S 2 K , the following holds with probability at least 1 −
[Equation]
where (cid:101) κ = κ 0 − κ L > 0 .
Let η i = (cid:104)
Z i , W ∗ (cid:105) and denote (cid:101) z i = (cid:112) φ (cid:48)(cid:48) ( η i )
z i and we can see that
[Equation] [Equation]
z i (cid:101) z
i (cid:48) and, H ( W ∗ ) =
E H n ( W ∗ ) .
Denote x i = vec( X i ) .
Here, for simplicity, we assume { x i } ni =1 to be independent gaussian vectors with mean zero and covariance matrix Σ , where c x I ≤ Σ ≤ C x
I for some 0 < c x < C x .
We will also use the notation of ∆ for its vectorized version, vec( ∆ ) , and we consider ∆ with unit l 2 -norm.
Since (cid:107)(cid:104) ∆ , (cid:101)
z
[Equation]
25 √ κ U , where w i is a standard gaussian vector and κ U
=
C x C u , we can show that
[Equation]
where the ﬁrst inequality comes from Remark 5.18 in (Vershynin, 2010) and second inequality comes from Lemma 5.14 in (Vershynin, 2010).
And hence, by the Beinstein-type inequality in Corollary 5.17 in (Vershynin, 2010), for any ﬁxed ∆ such that (cid:107) ∆ (cid:107) 2 = 1 , we have
[Equation]
With similar covering number argument as presented in Lemma 2 in our paper, we can show that,
P
[Equation]
where d M = K ( P + L + 1) .
Llet t = 0 .
5 κ U .
By Assumption 4(C1), we can obtain that, when n (cid:38) d M ,
[Equation]
where (cid:101) κ = κ 0 − κ U > 0 .
2. LRSC of L n ( W ) around W ∗ .
Deﬁne the event A = {|(cid:104) W ∗ , Z i (cid:105)| > τ sup ∆ ∈S 2 K |(cid:104) ∆ , z i (cid:105)|} and construct the functions h n ( W ) =
1 n n (cid:88) i = φ (cid:48)(cid:48)
( (cid:104)
Z i , W (cid:105) )
I
A z i z i (cid:48) and, h ( W ) =
E h n ( W ) , where τ is some positive constant to be selected according to Lemma 8.
Since the difference between h n ( · ) and H n ( · ) is the indicator function, it holds that H n ( · ) ≥ h n ( · ) .
We will ﬁnish the proof of LRSC in two steps.
Firstly, we show that, with high probability, h n ( W ∗ ) is positive deﬁnite on the restricted set S 2 K .
Secondly, we bound the difference between ∆ (cid:48)
h n ( W ) ∆ and ∆ (cid:48)
h n ( W ∗ ) ∆ , and hence show that h n ( W ) is locally positive deﬁnite around W ∗ .
This naturally leads to the LRSC of L n ( W ) around W ∗ .
From Lemma 8, we can select τ , such that h ( W ∗ ) ≥ κ 1 I .
Following similar arguments as in the ﬁrst part, we can show that for all ∆ ∈ S 2 K , the following holds with probability at least
[Equation]
where (cid:101) κ 1 = κ 1 − κ U > 0 .
In the meanwhile, for any W ∈ (cid:98) S K such that (cid:107) W − W ∗ (cid:107) F ≤ R , where R can be speciﬁed later to satisfy some conditions,
[Equation]
where ¯ W lies between W and W ∗
[Equation]
and φ (cid:48)(cid:48)(cid:48) ( z ) =
e z (1 − e z ) /
(1 + e z ) 3 .
Given the event A holds, choose R < τ
[Equation]
we can lower bound the term,
[Equation]
Notice that, for all z ∈ R , the third order derivative of the function φ ( z ) is upper bounded as | φ
[Equation]
( z ) | ≤ 1 / | z | .
This relationship helps us further bound the term,
[Equation]
φ
[Equation]
Hence
[Equation]
we can show that
[Equation]
By setting t = α RSM R/ ( τ − R )
[Equation]
where α RSM = 3 κ U / 2 , we can use the equation (16) in Lemmato obtain, as long as n (cid:38) d M ,
[Equation]
By rearranging terms
[Equation]
this is equivalent to
[Equation]
If we deﬁne the event B 2 = { inf ∆ ∈S 2 K ∆ (cid:48)
h n ( W ∗ ) ∆ ≤ (cid:101) κ 1 } and denote its complementary event by B c
[Equation]
and from (29)
[Equation]
we know that P ( B 2 ) ≤ 2 exp {− 0 .
25 cn + 9 d M } .
It can be seen that
[Equation]
So, if we choose R to be sufﬁciently small, such that α RSM R/ ( τ − R ) ≤ (cid:101) κ 1 / 2 , it holds that,
P
[Equation]
inf
[Equation]
This, together with H n ( · ) ≥ h n ( · ) , leads us to conclude that, when n (cid:38) d M , there exists some
R > 0 , such that for any W ∈ (cid:98) S 2 K satisfying
[Equation]
holds with probability
[Equation]
We accomplished our proof of the LRSC of L n ( W ) around W ∗ .
Lemma 6.
For two sub-gaussian random variables, X and Y , when E ( XY ) =
E X E Y , i.e. X is independent to Y , it holds that
[Equation]
Lemma 7.
[Equation]
Proof.
Firstly, we observe that
[Equation]
It then holds that E (cid:0) exp { λ (cid:2) φ (cid:48) ( η i ) −
y i (cid:3) } (cid:1) =
E { E (cid:0) exp { λ (cid:2) φ (cid:48) ( η i ) − y i (cid:3) }|
z i (cid:1) } ≤ exp { 0 .
125 λ 2 } , and this implies that (cid:107) φ (cid:48) ( η i ) − y
i (cid:107) ψ 2 ≤ 0 .
Lemma 8.
Under Assumption 4, there exists a universal constant τ > 0 such that h ( W ∗ ) ≥ κ 1 I , where κ 1 is a positive constant.
Proof.
We ﬁrst show that for any p 0 ∈ (0 , 1) , there exists a constant τ such that
[Equation]
We would separately show that
[Equation]
by
[Equation]
for some positive constants c 1 and c 2 .
Using the relationship P ( D 1 ∩ D 2 ) =
P ( D 1 ) + P ( D 2 ) −
P ( D c 1 ∪ D c 2 ) ≥ P ( D 1 ) + P ( D 2 ) − 1 , it follows naturally that
[Equation]
sup
[Equation]
where τ = c 1 /c 2 .
Since x i is a gaussian vector with mean zero and covariance Σ , z i
= U (cid:48)
G x i is a zero-mean gaussian vector with covariance given by U (cid:48) G Σ U G , and (cid:104) W ∗ , Z i (cid:105) =
(cid:104) vec( W ∗ ) , z i (cid:105) also follows a normal distribution with mean zero and variance (also its sub-Gaussian norm) upper bounded by κ U (cid:107) W ∗ (cid:107) 2F , where κ U =
C x C u .
Since from Assumption 4(C2), (cid:107) W ∗ (cid:107) 2F ≥ α √ d M , we can take c 1 to be sufﬁciently small such that
[Equation]
where x is a gaussian variable with variance upper bounded by 1.
Then, we can also observe that, for any ﬁxed ∆ ∈ S 2 K , (cid:104) ∆ , z i (cid:105) is a gaussian variable with zero mean and variance upper bounded by κ U .
We can use the concentration inequality for gaussian random variable to establish that
[Equation]
for all t ∈ R .
We can further use the union bound to show that
[Equation]
Let t = c 2 √ d M for some positive constant c 2 > √ 7 κ U .
We can choose c 2 large enough such that
[Equation]
The probability at (30) is hence shown.
Now, we will take a look at the matrix h
( W ∗ ) and show that it is positive deﬁnite.
Same as in
Lemma 5, we denote the event
A
= {|
[Equation]
(cid:105)|}
[Equation]
and its complement by
A c , then
[Equation] [Equation]
where (i) follows from Assumption 4(C1),
And since (cid:104) ∆ , z i (cid:105) is a gaussian variable with mean zero and variance bounded by κ U , its fourth moment is bounded by 3 κ U .
Also, by φ (cid:48)(cid:48) ( z ) ∈ (0 , 0 . 25) for all z ∈ R , (ii) can be shown.
Here, we can take p 0 to be small enough such that 0 .
25 √
3 κ
U (1 − p 0 ) ≤ 0 .
5 κ 0 holds.
It follows that
[Equation]
with κ 1 = 0 .
5 κ 0 .
We hence accomplished the proof of the lemma.
A. A DDITIONAL E XPERIMENTS
We ﬁrst provide some additional implementation details for Section 4.1 in the paper.
We consider two types of inputs for veriﬁcation of the sample complexity in Theorem 1.
For the time-dependent inputs, we generate a sequence of vectors { x i } with size R d from a stationary VAR(1) process, where d = 245 or 192.
The VAR(1) process has a transition matrix with spectral norm less than 1 to ensure stationarity, and a white noise sequence which follows multivariate standard normal distribution.
The ﬁrst 50 simulated data points in the sequence are discarded to alleviate the inﬂuence of initialization, and the length of the resulting sequence is equal to the training sample size plus the testing sample size, i.e. n + 200 .
We then reshape { x i } into a sequence of tensors of shape (7, 5, 7) or (8, 8, 3), which are the time-dependent inputs for experiments under S1/S2 or S3/S4.
Next, we conduct extra experiments for efﬁcient number of kernels for a CP block design.
This study uses 32 × 32 inputs with 16 channels, and we set stride to 1 and pooling sizes to (5 , 5) .
We generate the orthonormal factor matrices { H ( j ) , 1 ≤ j ≤ 4 } , where H (1) is of size R 8 × R , H (2) is of size
R 8 × R , H (3) is of size R 16 × R and H (4) is of size
R K × R where R = 8 and K ∈ { 8 , 16 , 24 , 32 } .
If we denote the orthonormal column vectors of H ( j ) by h ( j ) r where 1 ≤ r ≤ R and 1 ≤ j ≤ 4 , the stacked kernel tensor A can then be generated as A = (cid:80)
Rr =1 h (1) r ◦ h (2) r ◦ h (3) r ◦ h (4) r .
And it can be seen that A has a CP rank of R = 8 .
We split the stacked kernel tensor A along the kernel dimension to obtain { A k , 1 ≤ k ≤ K } and generate the corresponding fully-connected weight tensors { B k , 1 ≤ k ≤ K } with standard normal entries.
The parameter tensor W is hence obtained, and we further normalize it to have unit Frobenius norm to ensure the comparability of estimation errors between different K s. The block structure in Figure 7(1) is employed to train the network, and it is equivalent to the bottleneck structure with a CP decomposition on A ; see Kossaiﬁ et al.
(2020b).
We can see that as K increases, there is more redundancy in the network parameters.
Fifty training sets are generated for each training size, and we stop training when the target function drops by less than 10 − 5 .
From Figure 7(2), the estimation error increases as K is larger, and the difference is more pronounced when training size is small.