M ULTI -M ODE D EEP M ATRIX AND T ENSOR F ACTORIZATION
A BSTRACT
Recently, deep linear and nonlinear matrix factorizations gain increasing attention in the area of machine learning.
Existing deep nonlinear matrix factorization methods can only exploit partial nonlinearity of the data and are not effective in handling matrices of which the number of rows is comparable to the number of columns.
On the other hand, there is still a gap between deep learning and tensor decomposition.
This paper presents a framework of multi-mode deep matrix and tensor factorizations to explore and exploit the full nonlinearity of the data in matrices and tensors.
We use the factorization methods to solve matrix and tensor completion problems and prove that our methods have tighter generalization error bounds than conventional matrix and tensor factorization methods.
The experiments on synthetic data and real datasets showed that the proposed methods have much higher recovery accuracy than many baselines
I NTRODUCTION
Low-rank matrices and tensors are pervasive in sciences and engineering.
Low-rank matrix completion (LRMC) (Srebro & Shraibman, 2005; Cand ` es & Recht, 2009; Recht, 2011; Hu et al., 2013;
Hardt, 2014; Chen et al., 2014; Shamir & Shalev-Shwartz, 2014; Sun & Luo, 2016; Fan & Chow, 2017; Fan et al., 2020; K ¨ ummerle & Sigl, 2018; Fan et al., 2019) and low-rank tensor completion (LRTC) (Gandy et al., 2011; Acar et al., 2011; Liu et al., 2012; Kressner et al., 2014; Yuan & Zhang, 2016; Foster & Risteski, 2019), as shown in Figure 1, aim to recover the missing entries of a lowrank matrix or tensor.
LRMC and LRTC are very usefully in data preprocessing, image and video inpainting, and collaborative ﬁltering (Fan & Cheng, 2018;
Liu et al., 2012).
For instance, in collaborative ﬁltering (e.g. the recommendation problem of Netﬂix), the rating matrix given by users on items is often highly incomplete because each user can only rate a few items and each item is often rated by a few users.
If the unknown entries of the rating matrix are predicted, recommendation can then be made, accordingly, for users or items.
Since rating matrices often have potentially low-rank structures, we can use LRMC or even LRTC to recover the missing entries for recommendation.
[Figure]
Notation Throughout the paper, ‘ x ’, ‘ x ’, ‘ X ’, and ‘ X ’ denote scalar, columns vector, matrix, and tensor respectively.
‘ X ( j ) ’ (or ‘ X j ’) denotes a matrix with index j . (cid:107) · (cid:107) F denotes the Frobenius norm of matrix or tensor.
(cid:107) · (cid:107) ∗ denotes the nuclear norm of matrix, i.e. sum of the singular values.
(cid:107) · (cid:107) ∞ denotes the maximum absolute element in a matrix or tensor.
‘ C × j A ’ denotes the jmode product of a tensor C with a matrix A .
P Ω ( X ) denotes the projection onto the set Ω , i.e., [ P Ω ( X )] ij =
[ X ] ij if ( i, j ) ∈ Ω and [ P Ω ( X )] ij = 0 otherwise.
| Ω | denotes the cardinality of Ω . g ( · ) denotes an activation function and h l denotes the number of units in layer l of a neural network.
LRMC
In the past decade, a lot of algorithms with theoretical guarantee or/and empirical success have been proposed for LRMC.
For example, Cand`es & Recht (2009) proved that a low-rank matrix can be recovered exactly with high probability from a few entries sampled uniformly at random, via nuclear norm minimization.
Nie et al.
(2012) proposed to minimize the Schatten- p quasi-norm for LRMC.
Nuclear norm and Schatten- p quasi-norm minimizations are based on singular value decomposition and hence have high computational cost when the size of the matrix is very large.
In addition, they do not incorporate the rank prior that are possibly available in practice, though could be inaccurate.
In contrast, low-rank factorization based methods (Srebro & Shraibman, 2005; Srebro & Salakhutdinov, 2010; Wen et al., 2012; Hardt, 2014; Sun & Luo, 2016; Jin et al., 2016; Shang et al., 2016) are scalable to big matrices and can provide high recovery accuracy if the factorization size is properly determined.
In fact, regularized matrix factorizations are closely related to nuclear norm and Schatten- p quasi-norm minimizations.
For example, it is known (Srebro et al., 2005;
Rennie & Srebro, 2005) that (cid:107) X (cid:107) ∗ = min AB [Equation] X (cid:107) A (cid:107)
[Equation]
(cid:107) B (cid:107) [Equation] (cid:0) (cid:107) A (cid:107) [Equation]
(cid:107)
[Equation]
(cid:1) .
For LRMC, we can solve minimize A , B
[Equation]
where A ∈ R m × d , B ∈ R d × n , and 0 < d ≤ min { m, n } .
Interestingly, Gunasekar et al.
(2017) found and proved that gradient descent for (1) with λ = 0 and d = m = n converges to the minimum nuclear norm solution, provided that the learning rate is small enough and the initializations are close enough to the origin.
Fan et al.
(2019) proposed a class of rank regularizers called factor group sparse regularizer (FGSR) as the variational forms for Schatten- p quasi-norms, e.g., [Equation] X [Equation]
AB
[Equation]
where α > 0 and q ∈ { 1 , 12 , 14 , . . . }
When q = 1 , one gets Schatten- 1 / 2 quasi-norm.
Later,
Giampouras et al.
(2020) extended the variational form to arbitrary p ∈ (0 , 1] .
LRTC A few LRMC methods have been directly extended to LRTC (Acar et al., 2011; Liu et al., 2012; Kressner et al., 2014).
One may categorize LRTC algorithms according to the different tensor factorization models such as CP (CANDECOMP/PARAFAC) decomposition (Kolda & Bader, 2009;
Jain & Oh, 2014), Tucker decomposition (Xu et al., 2013; Xie et al., 2018), tensor singular value decomposition (Kilmer et al., 2013; Zhang & Aeron, 2016), and tensor ring decomposition (Zhao et al., 2016; Huang et al., 2020).
Take the CP decomposition based LRTC as an example, one can just learn a CP decomposition from the partial observations (Acar et al., 2011).
Yuan & Zhang (2016) provided the sample complexity for the exact recovery of LRTC via nuclear norm minimization, though the optimization is intractable because computing the nuclear norm of a tensor is NP-hard (Hillar & Lim, 2013).
Barak & Moitra (2016) and Foster & Risteski (2019) exploit the sum-ofsquares hierarchy for LRTC and provided theoretical guarantees for the recovery.
Razin et al.
(2021) studied the implicit regularization in tensor factorization and proved that gradient descent with small learning rate and near-zero initialization gives rise to a bias towards low tensor rank solutions.
Liu et al.
(2019) introduced convolutional neural network to tensor completion.
The method can model the possibly complex interactions inside tensors while preserving the desired low-rank structure.
To complete a third-order (w.l.g) tensor based on the Tucker decomposition, we may solve minimize
[Equation]
subject to P Ω ( X ) = P [Equation] ( Y ) , (3) where we can also consider regularization or constraint on the factors A 1 , A 2 , A 3 , and C .
For instance, Xu et al.
(2013) reformulated (3) to all-mode matricization and solved the problem by alternating minimization.
Xie et al.
(2018) proposed to minimize the number of nonzero elements of C and the ranks of the matricizations of X , in addition to (3).
Deep Matrix Factorization Recently, deep matrix factorization (DMF) (Fan & Cheng, 2018;
Arora et al., 2019) gains increasing attention in machine learning.
There are two types of DMF methods: linear DMF (Trigeorgis et al., 2016;
Zhao et al., 2017; Arora et al., 2019) and nonlinear
DMF (Xue et al., 2017; Wang et al., 2017; Fan & Cheng, 2018).
A general formulation is
[Equation]
[Equation]
where { g i } L − 1 l =1 are activation functions.
When all g i are linear, (4) is linear DMF.
Otherwise, (4) is nonlinear DMF.
Arora et al.
(2019) found that adding depth to a linear matrix factorization can enhance an implicit tendency towards low-rank and can provide higher recovery accuracy than depthfactorization, though the implicit regularization in linear DMF may not be captured using simple mathematical norms.
In (Fan & Cheng, 2018), the numerical results showed that nonlinear DMF is able to recover (with high accuracy) the missing entries of a full-rank matrix of which the columns are generated by a nonlinear low-dimensional latent variable model.
Contributions of this paper This paper ﬁrst provides theoretical analysis for why and when nonlinear DMF outperforms linear DMF in matrix completion.
Second, this paper proposes a new matrix factorization method called two-mode nonlinear DMF.
The method can exploit the full nonlinearity of the data while classical nonlinear DMF uses only partial nonlinearity of the data.
Third, the paper presents a new tensor decomposition method called multi-mode nonlinear deep tensor factorization, which can be regarded as a high-order generalization of the two-mode nonlinear DMF.
Finally, the paper shows that the multi-mode deep matrix and tensor factorization methods have tighter generalization error bounds than conventional factorization methods in matrix and tensor completion.
Extensive numerical results veriﬁed the effectiveness of the proposed methods.
W HY AND WHEN DOES NONLINEAR DMF OUTPERFORM MF?
Nonlinear DMF methods can outperform linear MF methods empirically in many real tasks such as collaborative ﬁltering (Xue et al., 2017; Fan & Cheng, 2018), image inpainting (Fan & Cheng, 2018), and disease modeling (Wang et al., 2017).
The fundamental reason is that the data matrices in these tasks indeed have some nonlinear structures that cannot be exploited by linear MF methods.
Nevertheless, in nonlinear DMF, the theoretical guarantee is very limited.
In contrast, linear MF or even linear DMF have been well studied and guaranteed with sample complexity (Hardt, 2014; Sun & Luo, 2016; Jin et al., 2016), generalization bound (Srebro & Shraibman, 2005; Fan et al., 2019), or convergence (Gunasekar et al., 2017; Arora et al., 2019).
In this section, we will provide some theoretical guarantee for nonlinear DMF in matrix completion.
Assumption 1.
Suppose Y = X + E , where Y ∈ R n 1 × n 2 .
The columns of X are generated by x = f ( z ) , where f : R d → R n 1 denotes a nonlinear mapping, d < min( n 1 , n 2 ) , and z ∈ R d denotes a latent variable.
The entries of the noise matrix E are drawn from N (0 , σ 2 ) .
Suppose we observed a few entries of Y randomly (sampling without replacement) and want to recover X from the incomplete Y .
Consider the following method (Fan & Cheng, 2018): minimize
[Equation]
where Z ∈ R d × n 2 , W l ∈ R
h l × h l − 1 , l = 1 , . . .
, L , h L = n 1 , h
= d , and h − 1 = n 2 .
λ w and λ z are regularization parameters.
Here we have let all activation functions be the same without loss of generality and omitted the bias terms of the neural network for simplicity.
According to the universal approximation theorems (Cybenko, 1989; Pinkus, 1999; Lu et al., 2017; Hanin & Sellke, 2017), f can be well approximated by the neural network provided that L is sufﬁciently large or at least one of h 1 , . . .
, h L − 1 is sufﬁciently large.
The following theorem provides the excess risk bound for (5).
Theorem 1.
Suppose Z and { W l } Ll =1 are given by (5) .
Let ˆ X
=
W L g ( W L − 1 · · · g ( W 1 Z ) · · · ) .
Suppose (cid:107) Z (cid:107) F (cid:81) Ll =1 (cid:107) W l (cid:107) F ≤ κ , max( (cid:107) Y (cid:107) ∞ , (cid:107) ˆ X (cid:107) ∞ ) ≤ ξ , the Lipschitz constant of g is η .
Let
N = n 1 n 2 .
Then with probability at least 1 − 2 N − 1 , there exists a numerical constant c such that 1 √
[Equation]
In Theorem 1, the last term of the RHS of the inequality is equivalent to
[Equation]
Most major activation functions (e.g. ReLU and sigmoid) are, at worst, 1-Lipschitz with respect to (cid:96) 2 norm, which indicates that η L − 1 is not large.
When n 2 is large enough, ψ 1 is linear with the latent dimension d of the columns of X when using (5).
In practice, it is difﬁcult to know the true d in advance.
But the Frobenius norm regularizations on Z and W 1 are able to reduce the nuclear norm or even the rank of W 1 Z theoretically, which means the d in formulation (5) and Theoremcan be larger than the d (ground truth) in Assumption 1.
Note that classical MF (depth-1) is a special case of (5)
when
L = 1 and g is linear.
However, because of the nonlinearity in the data generating model of Assumption 1, the rank of X can be much higher than the intrinsic dimension of the data, i.e. r := rank ( X ) (cid:29) d .
For example, if f is a q -order polynomial function, r can be as large as (cid:0) d
+ qq (cid:1) .
For classical MF, the last term of the RHS of the inequality in Theorem 1 reduces to
[Equation]
Here κ (cid:48) may be less than the κ in (6) but they are in the log operator.
Suppose all the Frobenius norms are the same (the worst case, may not happen because the norms of the factors in a deep factorization are usually smaller than the norms of the factors in a shallow factorization), denoted by β .
Then κ = β L +1 and κ (cid:48)
=
β 2 , but r (cid:29) d .
In the experiments, we observed that κ ≈ ( κ (cid:48) ) 1 . 3 when L =and κ ≈ ( κ (cid:48) ) 1 .
when L = 3 , which are much better than the worst cases.
Now comparing (7) with (6), we conclude that the nonlinear DMF provides a tighter generalization bound than the classical
MF when n 2 is much larger than n 1 and (cid:80) L −
1 l =1 h l h l − 1 is not too large.
The second condition is realistic because we usually let h 1 , . . .
, h L − 1 < n 1 or we can just assume h 1 , . . .
, h L − 1 < r .
Consider the case that n 2 is not large compared to n 1 , e.g. n 2 = n 1 .
In (6), suppose d = r/ 2 , L = 3 , h 1 = 2 d , h 2 = 3 d , and η = 1 , we have ψ 1 = cξ (cid:0) | Ω | − 1 (2 n 2 r + 2 r 2 ) log( κ ) (cid:1) 1 / 4 and ψ [-
-]2 = cξ (cid:0) | Ω | − 1 (2 n 2 r ) log( κ (cid:48) )
(cid:1) 1 / 4 ,
where ψ 1 > ψ 2 .
It means the upper bound given by the classical MF is tighter than that given by the nonlinear DMF.
In general, the superiority of the nonlinear DMF over the classical MF increases when n 2 /n 1 increases, which is consistent with the numerical results shown in Figure 5 of the appendix.
T WO - MODE NONLINEAR DEEP MATRIX FACTORIZATION
Data matrices in many problems are square or nearly square, which restricts the capability of nonlinear DMF.
For instance, the difference between the width and height of an image is often small.
The covariance matrices and graph adjacency matrices are square matrices.
In recommendation system, the number of users and the number of items may be of the same order of magnitude.
A covariance matrix or graph adjacency matrix may be constructed naturally by data with nonlinear latent structures.
Similarly, one may imagine that the rating given by a user on an item is an interaction of the user’s features and item’s features and the features may have some nonlinear low-dimensional latent structures.
The nonlinear DMF in (5) exploits only partial nonlinearity of the data.
Formally, we make the following assumption.
Assumption 2.
Suppose Y = X + E , where Y ∈ R n 1 × n 2 and X =
U (cid:62) QV .
The columns of
U ∈ R m 1 × n 1 and V ∈ R m 2 × n 2 are generated respectively by [Equation]
) and v = f 2 ( s ) , where f 1 : R d 1 → R m 1 and f 2 : R d 2 → R m 2 are nonlinear mappings, d 1 < min( m 1 , n 1 ) , d 2 < min( m 2 , n 2 ) , and z ∈ R d 1 and s ∈ R d 2 denote latent variables.
The entries of Q ∈ R m 1 × mand E ∈ R n 1 × n 2 are drawn from N (0 , σ 2 Q ) and N (0 , σ 2 E ) respectively.
Suppose we have a matrix Y given by Assumption 2, we can factorize it as
[Equation]
In (8), f θ 1 : R d 1 → R m 1 and f θ 2 : R d 2 → R m 2 are multi-layer neural networks with parameter sets θ
1 = { W (1)1 , . . .
, W (1) L 1 } and θ 2 = { W (2)1 , . . .
, W (2) L 2 } , where W ( j ) i denotes the weight matrix in the i -th layer of neural network j , and L j denotes the number of layers of neural network j , j = 1 , 2 . Y Q g( [Equation]
g(
[Equation]
g( ) g( )
Z T
S
We call (8) two-mode or two-way nonlinear deep matrix factorization.
Figure 2 shows an intuitive example of (8) with depth-three factorizations in both directions.
Suppose we observed a few entries of Y randomly (sampling without replacement).
Based on
Assumption 2, we propose the following matrix completion method
[Equation]
where
[Equation]
and Z ∈ R d 1 × n 1 , S ∈ R d 2 × n 2 , W ( j ) l
[Equation] [Equation]
h ( j ) − 1 = n j , j = 1 , 2 , and Q ∈ R m 1 × m 2 .
We have let all activation functions be the same without loss of generality.
In (9), R ( · ) is a regularization operator to reduce overﬁtting.
For instance,
[Equation]
2 F (cid:1) .
Compared to (5), (9) exploits the full nonlinearity of the data.
The following theorem provides the excess risk for (9).
Theorem 2.
Suppose Z , S , { W (1) l } L 1 l =1 , { W (2) l } L 2 l =1 , and Q are given by (9) .
Let ˆ X
[Equation]
Suppose
[Equation]
max
[Equation]
Y [Equation] ∞ , (cid:107) ˆ X (cid:107) ∞ ) ≤ ξ , and the Lipschitz constant of g is η .
Let N = n 1 n 2 .
Then with probability at least 1 − 2 N − 1 , there exists a numerical constant c such that
[Equation]
In Theorem 2, the last term of the RHS of the inequality can be written as
[Equation]
Since the h L − 1 in (6) is much larger than the d 2 in (10), we have ψ 3 < ψ 1 .
It means the two-mode nonlinear DMF (9) provides a tighter generalization bound than classical nonlinear DMF (5).
It is worth mentioning that one may formulate DMF in a two-sided form, e.g., X ≈ g (cid:0) W (2) L g ( W (1) L ZW (1) R ) W (2) R (cid:1) or X ≈ g (cid:0) W (1)
L g ( Z L Z R ) W (1) R (cid:1) equivalently.
However, this model has two shortcomings compared to the one-sided DMF (4) and two-mode DMF (8).
First, (4) and (8) are more compact (in terms of the number of parameters) than the two-sided DMF.
For example, assume the sizes of X , Z L , and Z R are 500 × 500, 100 × 10, and 10 × 100 respectively, then the number of parameters in the two-sided DMF is larger than 100,000.
In the one-sided DMF (4), let X
=
g
( W 2 g ( W 1 Z )) and let the sizes of W 1 , W 2 , and Z be 500 × 100, 100 × 10, and 10 × 500 respectively, then the number of parameters is 56,000, which is much smaller than that of the two-sided DMF.
The other shortcoming is that the two-sided DMF is difﬁcult to interpret in real applications.
In contrast, (4) and (8) are much easier to interpret and more realistic.
M ULTI - MODE NONLINEAR DEEP TENSOR FACTORIZATION
In this section, we extend the two-mode deep matrix factorization to multi-mode nonlinear deep tensor factorization.
First, we make the following assumption.
Assumption 3.
Suppose Y = X + E , where Y ∈ R n 1 × n 2 ···× n k and X =
C × 1 U 1 × 2 U 2 · · ·× k U k .
For j = 1 , . . .
, k , the rows of U j ∈ R n j × m j are generated by u ( j ) =
f j ( z ( j ) ) , where f j : R d j →
R m j is a nonlinear mapping, d j < m j , and z ( j ) ∈ R
d j are latent variables.
The entries of core tensor C ∈ R m 1 × m 2 ···× m k and noise tensor E ∈ R n 1 × n 2 ···× n k are drawn from N (0 , σ 2 C ) and
N (0 , σ 2 E ) respectively.
Although { f j } kj =1 , C , and { Z ( j ) } kj =1 are unknown, we can take advantage of neural networks and tensor decomposition to model the data, i.e.,
[Equation]
where f θ j : R d j → R d j × m j is a neural network with parameter set θ j .
Figure 3 shows an intuitive example of the proposed factorization method for a third-order tensor.
Different from Tucker decomposition, we perform nonlinear DMF on the factor matrices and do not constrain the norms of the columns of the factor matrices.
This factorization method can take advantage of the nonlinear structures of the tensor fully.
[Equation]
To recover X from a few entries of Y , we propose to solve the following problem minimize
[Equation]
[Equation]
where f θ j ( Z ( j ) )
=
g ( W ( j ) L j g ( W ( j ) L j − 1 · · · g ( W ( j ) 1 Z ( j ) ) · · · )) and θ j = { W ( j ) 1 , . .
, W ( j ) L j } , j = 1 , . .
, k .
h denotes the regularization on the parameters, e.g.,
[Equation]
The following theorem shows the excess risk bound for (12).
Theorem 3.
Suppose C and { θ j , Z ( j ) } kj =1 are given by (12) .
Let ˆ X =
C ×
1 f θ 1 ( Z (1) )
×f θ 2 ( Z (2) ) · · · × k f θ k
( Z ( k ) ) .
Suppose (cid:107) C (cid:107) F ≤ β C , (cid:107) W ( j ) l (cid:107) F ≤ β ( j ) l , l = 1 , . . .
, L j , (cid:107) Z ( j ) (cid:107) F ≤ β ( j ) 0 , j = 1 , . .
, k , max( (cid:107) Y (cid:107) ∞ , (cid:107) ˆ X (cid:107) ∞ ) ≤ ξ , and the Lipschitz constant of g is η .
Let
N = (cid:81) kj =1 n j .
Then with probability at least 1 − 2 N − 1 , there exists a numerical constant c such that
[Equation]
Remark 1.
One can obtain slightly tighter bounds in Theorems 1, 2, and 3 when using spectral norms instead of Frobenius norms.
But in (5) , (9) , and (12) , the regularizations only control the
Frobenius norms explicitly.
In addition, (12) and Theorem 3 are also applicable to CP-like decomposition based tensor completion if C is super-diagonal.
We see that Theorem 2 is a special case of Theorem
when k = 2 .
Denote ψ 4 the last term of the
RHS of the inequality in Theorem 3.
Since h ( j )
=
d j and h ( j ) − 1 = n j , we have ψ
[Equation]
[Equation]
If g is linear and L = 0 , (12) degenerates into the Tucker decomposition based LRTC, e.g. (3).
Accordingly, ψ 4 becomes
[Equation]
Since m j (cid:29) d j , we have ψ 5 > ψ 4 .
It means the nonlinear method (12) has a tighter generalization error bound than linear tensor completion methods.
Therefore, we expect that (12) outperforms linear LRTC methods when the data have nonlinear structures.
For convenience, in this paper, we call (9) and (12) multi-mode deep matrix and tensor factorizations (M 2 DMTF).
Optimization The optimization for (9) and (12) are non-trivial because there are high-order tensors and neural networks with unknown inputs and incomplete outputs.
We tried alternating minimization, nonlinear conjugate descent, L-BFGS (Liu & Nocedal, 1989), iRprop+ (Igel & H ¨ usken, 2000), and Adam (Kingma & Ba, 2014), where all variables are randomly initialized from Gaussian distribution.
It is found that iRProp+ outperforms other algorithms especially on real datasets.
The details of optimization and the analysis of time and space complexity are presented in Appendix A.
N UMERICAL RESULTS 5. S YNTHETIC DATA
We use the following nonlinear model to generate matrices:
[Equation]
where Z ∈ R 2 × 100 , W (1)
[Equation]
R 10 × 2 ,
[Equation]
∈ R 20 × 10 , and W (2) ∈ R 30 × 20 are drawn from U ( − 1 , 1) .
σ denotes the sigmoid fucntion.
The exact rank of X is
but the smallest 10 nonzero singular values are close to zero.
Such a synthetic matrix is realistic because the nonlinearity in real data may not be very strong and the matrices are often low-rank.
Similarly, the following nonlinear model is used to generate third-order tensors:
[Equation]
where Z (1) ∈ R 2 × 50 , W (1) ∈
[Equation]
[Equation]
10 , Z (2) ∈ R 2 × 50 , W (2) ∈ R 10 × 2 , W (2) ∈
R 20 × 10 , Z (3) ∈ R 2 × 50 , W (
3) ∈ R 10 × 2 , and W (3) ∈ R 20 × 10 are drawn from U ( − 1 , 1) and
C ∈ R 20 × 20 × 20 are drawn from N (0 , 1) .
The exact n -rank of X is (20 , 20 , 20) but it can be well approximated by a tensor with n -rank (10 , 10 , 10) because here the nonlinearity is not very strong.
First, we compare the proposed method with linear matrix factorization (MF (1)), nuclear norm minimization (NNM) (Cand`es & Recht, 2009), TNNR (Hu et al., 2013), HM-IRLS (K¨ummerle &
Sigl, 2018), FGSR (Fan et al., 2019), (stacked) Autoencoders (SAE) (Sedhain et al., 2015), and
DMF (Fan & Cheng, 2018).
We use the tanh activation function in SAE, DMF, and M 2 DMTF.
We determine the hyper parameters of all methods via cross-validation to ensure they perform as well as possible.
Details about the optimizations and parameter settings are in Appendix B.
We use the relative recovery error (Fan et al., 2019)
(cid:107)P ¯Ω ( ˆ X − X ) (cid:107) F / (cid:107)P ¯Ω ( X ) (cid:107) F to evaluate the performance of matrix completion, where ¯Ω consists of the locations of the missing entries.
Figure 4(a) shows the average performance of 20 repeated trials when the missing rate increases from 0.8 to 0.99.
MF and NNM have much higher recovery error than FGSR.
This result is consistent with the results of (Fan et al., 2019; Arora et al., 2019).
Although SAE is a nonlinear method, it still has high recovery error when the missing rate is less than 0.9.
The reason is that ﬁlling the missing entries of the input of the encoder with zero introduces relatively large bias, which may not be eliminated effectively by the hidden layers.
DMF and FGSR have quite similar performance, which is consistent with the analysis at the end of Section 2.
Our M 2 DMTF outperformed other methods signiﬁcantly especially when the missing rate is high.
The reason is that M 2 DMTF is able to exploit the full nonlinearity of the data and has tighter generalization bound than other methods.
Figure 4(b) shows the performance (average of 20 trials) of FaLRTC (Liu et al., 2012), TenALS (Jain & Oh, 2014), TMac (Xu et al., 2013),
KBR-TC (Xie et al., 2018), TRLRF (Yuan et al., 2019),
CoSTCo (Liu et al., 2019), OITNN-O
(Wang et al., 2020), and M 2 DMTF.
Since the rank of the tensor is quite low, TMac, KBR-TC, and M 2 DMTF have high recovery accuracy when the missing rate is less than 0.96.
Note that TMac and KBR-TC slightly outperformed M 2 DMTF when the missing rate is less than 0.96.
The reason is that the optimization of M 2 DMTF is more difﬁcult.
In Figure 4(b), when the missing rate is higher than 0.97, the proposed M 2 DMTF outperformed all methods largely.
The optimization curves of M 2 DMTF are shown in Figure 7 of Appendix B.
[Figure]
5. R EAL MATRICES
We consider two benchmark datasets: MovieLens-100k and MovieLens-1M.
They consist of the ratings ( 1 to 5 ) for movies given by users.
The sizes (movies × users) of the rating matrices in the datasets are 1682 × 943 and 3706 × 6040 respectively.
For each matrix (highly incomplete), we randomly split the known entries to a training set and a test set.
The relative recovery errors (average of 10 repeated trials) on the two datasets are reported in Tables 1 and 2 respectively.
FGSR,
SAE, and M 2 DMTF outperformed MF and DMF in all cases.
When the training-test ratio is 9/ or 7/3, SAE has less recovery error than M 2 DMTF.
This is quite different from the results on the synthetic data in Section 5.1.
Currently, it is unknown why SAE has such a good performance on the MovieLens data.
One possible reason is that the MovieLens data are quite noisy and SAE can successfully reduce the inﬂuence of the noise when the missing rate is relatively low.
When the training-test ratio is smaller, i.e., 5/5, 3/7, or 1/9, M 2 DMTF outperformed SAE.
[Table] [Table] [Table]
We compare the proposed method M 2 DMTF with the baselines on the following datasets: Amino acid ﬂuorescence (Bro, 1997) ( 5 × 201 × 61 ), Flow injection (Nørgaard & Ridder, 1994) ( 12 × 100 × 89 ), and SW-NIR kinetic data (Bijlsma & Smilde, 2000) ( 301 × 241 × 8 ).
More details about the experimental settings are in Appendix B.2.
The relative recovery error (average of 10 trials) are reported in Table 3.
The results of FaLRTC and CoSTCo are not reported because they have much higher recovery error than other methods in most cases and we want to ﬁt the table to the width of the page.
On Amino, our M 2 DMTF outperformed other methods signiﬁcantly in all cases.
On Flow, when the missing rate is 0.95 or 0.97, M 2 DMTF has much lower recovery error than other methods.
On SW-NIR, M 2 DMTF has low recovery error consistently while most baselines failed totally.
[Table]
> [Table] [Table] [Table]
> 0 . [Table] > [Table]
> 0 . >
[Table] [Table] [Table]
Comparative studies were also conducted on three larger datasets.
The results are reported in Appendix B.2.4.
The experiments of tensor completion based recommendation system are in Section
B.2.5.
The visualization of the learned tensor and factors of M 2 DMTF are in Appendix B.2.6.
C ONCLUSION
This paper has proposed a framework of multi-mode deep matrix and tensor factorizations.
The proposed factorization methods can explore and exploit the full nonlinearity of the data effectively.
We applied the factorization methods to matrix and tensor completion and provided the generalization error bounds, which veriﬁed the superiority of the proposed methods over conventional linear and nonlinear matrix factorization methods and linear tensor completion methods.
The proposed framework reduced the gap between tensor decomposition and deep learning and showed that their combinations can provide better performance in tensor recovery.
On some datasets, when the missing rates were low, our methods did not achieve the best performance.
The reason is that the nonlinearity in the data is not very strong and the problem is not difﬁcult for the baselines methods of which the optimizations are easier than our methods.
Future work may focus on improving the optimization efﬁciency of M 2 DMTF.
A CKNOWLEDGEMENTS
The work was supported by the research funding T00120210002 of Shenzhen Research Institute of
Big Data and the Youth program 62106211 of National Natural Science Foundation of China.
A O PTIMIZATION DETAILS AND COMPUTATIONAL COMPLEXITY
In our M 2 DMTF, the matrix model is a special case of the tensor model.
Therefore, we only show the optimization detail and the computational complexity of the tensor case.
The gradient-based optimization is shown in Algorithm 1.
We suggest using iRprop+ as the optimizer in the algorithm because we found empirically that iRprop+ is more stable and efﬁcient than other optimizers such as Adam in solving our M 2 DMTF.
Input: Y ∈ R n 1 ×···× n k , Ω , λ 1 , λ 2 , g , network structures (cid:8) m j , L j , { h ( j ) [Equation]
[Equation] , T .
{ W
[-j =1 , (cid:8) U ( j )= f θ ( j ) 
-][Equation] [Equation]
[Equation]
U
[Equation]
L
2 F +
R ( { θ ( j ) t , Z ( j ) } k
j =1 , C t ) and V t
= U ( j − 1) t ⊗ · · · U (1)⊗ U ( k ) t · · · ⊗ U ( j +1) t .
4: end for 5: ∇ C ←− ∂ L k /∂ C 6: (cid:8) { θ ( j ) t , Z ( j ) t } kj =1 , C t (cid:9) ←− gradient-based optimizer (cid:0) {∇ θ ( j ) , ∇ Z ( j ) } kj =1 , ∇ C (cid:1) 7: end for
Output: ˆ Y =
C T × 1 U (1) T × 2 U (2) T · · · × k U ( k ) T .
Without loss of generality, we here assume that in (12), n 1 = · · · n k = n , m 1 = · · · m k = m ,
L 1 = · · · = L k = L , and h (1) l = · · · = h ( k ) l = h l , l = 0 , 1 , . . .
, L .
The time cost in each iteration of the gradient-based optimization is
[Equation]
in which the ﬁrst term is from the Tucker-like decomposition while the second term is from the forward and backward propagations of the neural networks.
Since the second term is much smaller than the ﬁrst one and m < n , the time complexity per iteration is about O (cid:0) kn k m + kn k − 1 m k (cid:1) .
As we need to store the observed elements of Y , core tensor C , latent factors Z ( j ) , network parameter, their gradients, and the values of hidden units, the space complexity is O (cid:0) | Ω | + m k + k (cid:80) Ll
=
h l − 1 h l + kn (cid:80)
Ll =1 h l (cid:1) , which is further simpliﬁed to O (cid:0) | Ω | + m k + kn (cid:80) Ll
=1 h l (cid:1) because n is larger than all h l .
Now we see that the time (per iteration in the optimization) and space complexity of our M 2 DMTF are slightly higher than the Tucker decomposition based tensor completion methods.
However, M 2 DMTF requires much more iterations than the baseline methods because it is composed of tensor decomposition and neural networks.
B M ORE ABOUT THE EXPERIMENTS B. M ATRIX COMPLETION
In the proposed method (9), the regularization is
[Equation]
Z (cid:107) 2 F + (cid:107) S (cid:107) 2 F )
+ λ 2 2 (cid:16) (cid:107) Q (cid:107) 2 F + 2 (cid:88)
j = L j (cid:88) l = (cid:107) W ( j ) l (cid:107) 2 F (cid:17) . We let λ 2 = λ (cid:48) 2 √ n 1 n 2 and determine λ 1 and λ (cid:48) 2 instead.
In the optimization, S , Z , and { W ( j ) l } are initialized randomly, Q is initialized to an identity matrix.
B.1. P ARAMETER SETTING ON THE SYNTHETIC DATA
• In MF (problem (1) in the main paper), the factorization dimension d is 5 because it outperforms other choices.
The λ is chosen from { 0 . 01 , 0 . 1 , 1 } and the optimizer is iRprop+.
The maximum iteration is 2000.
• In FGSR (Fan et al., 2019), we use the variational form of Schatten- 1 / 2 norm, set d = 20 , and choose λ from { 0 . 01 , 0 . 1 } .
The maximum iteration is 2000.
• In SAE (Sedhain et al., 2015), the network structure is [100 , 20 , 10 , 5 , 10 , 20 , 100] .
The weight decay parameter is chosen from { 0 . 1 , 0 . 5 } .
• In DMF (Fan & Cheng, 2018), the network structure is [3 , 10 , 20 , 100] .
The weight decay parameters are chosen from { 0 . 01 , 0 . 1 } .
• In M 2 DMTF, L = 2 , d 1 =
d 2 = 3 , h (1)1 = h (2)1 = 10 , m 1 = m 2 = 20 , and λ 1 = λ (cid:48) 2 = 1 .
In SAE, DMF, and M 2 DMTF, the activation function is the hyperbolic tangent function.
The opti mizer is iRprop+ and the maximum iteration is 3000.
B.1. M ORE RESULTS ON THE SYNTHETIC DATA
Figure 5 compares DMF with linear MF (FGSR) in the cases of different n 2 /n 1 .
We see that when n 2 /n 1 is too small, FGSR outperforms DMF.
When n 2 /n 1 increases, the superiority of DMF over
FGSR becomes more signiﬁcant.
These empirical results are consistent with our theoretical analysis in Section 2 of the main paper.
Figure 6 shows the performance of M 2 DMTF with different number of hidden layers and different activation function.
it can be found that tanh and sigmoid functions lead to lower recovery error than
ReLU in all cases.
The main reason is that the data were generated by smooth functions that can be [Figure] v [Figure]
r [Figure] [Figure]
well approximated by and increasing the the is large error increased slightly, which might be caused by overﬁtting.
Figure 7 shows the values of the objective functions and relative recovery errors against the iterations in matrix completion and tensor completion when the missing rate is 0 .
The recovery error decreases slightly when the number of iterations is very large.
In general, the optimization efﬁciency is not very high because M 2 DMTF is composed of multiple neural networks with unknown inputs and incomplete outputs.
We replace the sin , cos , exp , and σ in (16) and (17) with tanh such that the nonlinearity in the data generating models are consistent with the activation functions used in our matrix and tensor completion method M 2 DMTF.
The average results of 10 trials are reported in Figure 8.
We see that
M 2 DMTF outperformed other methods signiﬁcantly when the missing rate is high enough.
B.1. P ARAMETER SETTING ON THE M OVIE L ENS DATASETS
• In MF (problem (1) in the main paper), we set d = 10 and λ = 10 .
In stead of iRprop+, we use alternating minimization because it outperformed iRprop+ in this case.
The maximum iteration is 2000.
• In FGSR Fan et al.
(2019), we use the variational form of Schatten- 1 / 2 norm, set d = 100 , and set λ = 0 .
015 or 0 .
007 (for MovieLens-100k or MovieLens-1M).
The maximum iteration is 2000.
• In SAE Sedhain et al.
, we use a single hidden layer of size 500.
The weight decay parameter is 0 . 5 .
[Figure]
[Figure]
[Figure] [Figure] [Figure] • In DMF, Fan & Cheng (2018), the network structure is [10 or 20 , 50 , 100 , 1682] .
The weight decay parameters are chosen from { 0 . 05 , 0 . 1 } .
• In M 2 DMTF, , we set L = 2 , d
= 10 or 20 , h (1) =
h (2) = 50 , m 1 = m 2 = 100 , and λ
1 = λ (cid:48) 2 = 1 .
B.1. M ORE RESULTS ON THE M OVIE L ENS DATASETS
The RMSE (average of 10 trials) on the MovieLens-100k are reported in Table 4.
The proposed
M 2 DMTF outperformed MF, FGSR and DMF in all cases and outperformed SAE when the missing rate was high.
Note that RMSE is a monotonic transformation of the relative recovery error used in the main paper.
B. T ENSOR COMPLETION
In the proposed method (12), the regularization is
[Equation]
[Equation]
[Table]
We let λ 2 = λ (cid:48) 2 (cid:16) (cid:81) kj =1 n j (cid:17) 1 /k and determine λ 1 and λ (cid:48) 2 instead.
In the optimization, U j is initialized by the singular value decomposition of the j -mode matricization of Y , C is initialized to zero, and { Z ( j ) } , { W ( j ) l } are initialized randomly.
B.2. P ARAMETER SETTING ON THE SYNTHETIC DATA
• In FaLRTC (Liu et al., 2012), we set α 1 = α
= α 3 = 13 choose µ from { 0 . 5 , 1 , 2 , 5 , 10 } .
• In TenALS (Jain & Oh, 2014), the initial rank is 5 or 10 and the max iteration is 200.
• In TMac (Xu et al., 2013)
, the rank is initialized to [10 , 10 , 10] or [5 , 5 , 5] and adjusted adaptively.
The maximum iteration is 500.
• In KBR-TC
(Xie et al., 2018), ρ = 1 . 01 and λ = 0 .
The maximum iteration is
300. • TRLRF (Yuan et al., 2019), the rank is set to [5 , 5 , 5] or [10 , 10 , 10] .
We choose λ from { 0 . 01 , 0 . 1 , 1 , 10 , 50 } .
• In CoSTCo (Liu et al., 2019), the learning rate is chosen from { 0 . 01 , 0 . 001 , 0 . 001 } , the rank is chosen from { 2 , 3 , 5 , 10 , 20 } , the epoch number is 50 or 200, and the batch size is 256 × 2 or 256 × 5 .
In OITNN-O
(Wang et al., 2020), we choose α from { 0 . 0001 , 0 . 001 , 0 . 01 , 0 . 1 } .
The max iteration is 500.
• In M 2 DMTF, L = 2 , d 1 =
d 2 = d 3 = 3 , h (1)1 = h (2)1 = h (3)1 = 10 , m
= m
= m 3 = 20 , and λ
1 = λ (cid:48) 2 = 1 .
The optimizer is iRprop+ and the maximum iteration is 3000.
B.2. T IME COST COMPARISON ON THE SYNTHETIC DATA
We report the average time costs (second) and the recovery errors of all methods in Table 5, which also includes the performance of M 2 DMTF with different iteration numbers.
We see that our method with 3000 iterations is slower than other methods.
However, when we use less iterations (e.g.or 200), the time cost of our method is comparable with or even lower than the baselines and its recovery error is still much lower than the baselines.
FaLRTC
[Table]
1.1s 2.0s
[Table]
.1s
[Table]
9s 9.9s
[Table]
6s 32.3s
B.2. E XPERIMENTAL SETTINGS ON REAL TENSORS
We normalize each tensor by X ← X / (cid:107) X (cid:107) ∞ .
In M 2 DMTF, we set L =
and λ 1 = λ (cid:48)
2 = 0 .
For i = 1 , 2 , 3 , we set m i ≈ n i 5 , d i ≈ n i 10 if n i > 10 and set m i ≈ 0 .
8 n i , d i ≈ 0 .
7 n i if n i ≤ 10 .
The parameters of other methods are carefully determined to provide their best performance.
B.2. M ORE RESULTS ON TENSOR DATASETS
In addition to the three datasets considered in the main paper [Table] we test the proposed method on three more datasets: Porphyrin (Wold et al., 2006)
× 274 × 35 , Sugar (Bro, 1999) 256 × 571 × 7 , and Bonnie (Acar et al., 2008) 89 × 97 × 549 .
To reduce the computational costs of all methods on
Bonnie [Table] we only consider a subset of size 89 × 97 × 100 .
The results are reported in Table 6.
We see that M 2 DMTF outperforms other methods when the missing rate is high enough [Table] [Table] B.2. T
ENSOR COMPLETION BASED AS A DOWNSTREAM TASK
We split the MovieLens-100k dataset by time to form a tensor of size 943 × 1682 × 31 , where 31 denotes the number of weeks as a dimension of time.
Since the baselines KBR-TC, TRLRF,
OITNN-O, and the proposed M 2 DMTF have high time cost on large tensors, we reduce the size of the tensor by removing the users giving 49 or less ratings and the movies rated by 49 or less users.
Then we get a tensor of size 563 × 593 × 31 .
The relative recovery error (average of 5 trials) are reported in Table 7.
Our M 2 DMTF outperformed other methods.
In addition, comparing Tablewith Table [Table] we found that tensor completion methods are more effective than matrix completion methods in recommendation.
[Table]
V ISUALIZATION OF FACTORS LEARNED BY M 2 DMTF
Here we take the ‘Flow injection’ tensor (Nørgaard & Ridder, 1994) as an example.
The data is from measuring samples with three different analytes on a ﬂow injection analysis (FIA) system where a pH-gradient is imposed.
The three analytes in twelve measured samples are 2-,
3-, and 4-hydroxy-benzaldehyde.
They have different absorption spectra depending on whether they are in their acidic or basic form.
The data set is a 12 (samples) × 100 (wavelengths) × 89 (times) array.
Figure 9 presents the original data, incomplete data, and recovered data for the ﬁfth horizontal slice (sample-5) of the ‘Flow’ tensor, where the missing rate is 0.97.
We see that FaLRTC, TRLRF, and
CoSTCo failed totally.
TenALS, TMac, and KBR-TC have high recovery error visually.
In contrast, the data recovered by our M 2 DMTF is very close to the original data.
More importantly, the result of M 2 DMTF indicates that there are three analytes, while other methods failed to reveal the fact.
We analyze the characteristics of the low-dimensional factors learned by our M 2 DMTF.
Figurepresents the values of U 2 ∈ R 100 × 20 and Z (
) ∈
R 100 × 10 as well as the correlation coefﬁcients [Figure] [Figure] and mutual information.
In the ﬁrst column, we see that the values of U 2 are more chaotic than the values Z (2) .
The reason is that the 20 variables in U 2 are correlated mutually.
In the second column of Figure 10, we see that the autocorrelation of the samples (rows) of U 2 is similar to that of Z (2) , because U 2 relies on Z (2) .
The third column of the ﬁgure indicates that the correlation between the variables in U 2 are higher than that in Z (2) .
The last column compares the mutual information that can quantify nonlinear relationship.
Now comparing column 4 with column 3, we conclude that there exist signiﬁcant nonlinear relationship between the variables in U 2 and it is necessary to represent U 2 by a lower-dimensional latent factor Z (2) via deep factorization.
These results veriﬁed that the nonlinear latent factor assumption of our M 2 DMTF is practical and useful.
C P ROOF OF THEOREMS
First of all, we provide the following lemma.
Lemma 1.
Let S be a set deﬁned over tensors of size n 1 × n 2 × · · · × n k .
Let |S| be the (cid:15) - covering number of S with respect to the Frobenius norm.
Let N = (cid:81)
ki = 1 n i .
Suppose X ∈ S and max {(cid:107) Y (cid:107) ∞ , (cid:107) X (cid:107) ∞ } ≤ ξ .
Then with probability at least 1 − 2 N − 1 , there exists a constant c such that
[Equation]
The proof of the lemma is in Appendix D.1.
C. P ROOF FOR T HEOREM
The following lemma (proved in Appendix 2) shows an upper bound for the covering number of the deep factorization matrix set.
Lemma 2.
Let S = { X ∈ R m × n : X = W L g ( W L − 1 · · · g ( W 1 Z )) , (cid:107) Z (cid:107) F ≤ β 0 , (cid:107) W l (cid:107) F ≤ β l , 1 ≤ l ≤ L } , where Z ∈ R h 0 × n , W l ∈ R h l × h l − 1 , l = 1 , . . .
, L , h L = m ,
h
= d , and h − 1 = n .
Suppose the Lipschitz constant of g is η .
Then the covering numbers of S with respect to the Frobenius norm satisfy
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] Now using Lemma 1 and Lemma
[Equation]
we have [Equation] In the fourth inequality, we have set (cid:15) = 3 ξ ( L + 1) .
The last inequality holds because N is much smaller than |S| and c can be a constant large enough.
This ﬁnished the proof.
C. P ROOF FOR T HEOREM
The following lemma (proved in Appendix D.3) provides an upper bound for the covering number of the two-mode deep factorization matrix set.
Lemma 3.
Let S = { X ∈ R n 1 × n 2 : X = f θ 1 ( Z ) (cid:62) Q f θ 2 ( S ) , (cid:107) Q (cid:107) F ≤ β Q , (cid:107) Z (cid:107) F ≤ β (1)0 , (cid:107) S (cid:107) F ≤ β (2)0 , (cid:107) W ( j ) l (cid:107) F ≤ β ( j ) l ,≤ l ≤
L j , j = 1 , 2; f θ 1 ( Z ) =
g ( W (1) L 1 g ( W (1) L 1 − 1 · · · g ( W (1)1 Z ))) , f θ 2 ( S ) = g ( W (2)
L 2 g ( W (2) L 2 − 1 · · · g ( W (2)1 S ))) , } , where Z ∈ R d 1 × n 1 , S ∈ R d 2 × n 2 , W ( j ) l ∈ R h
[Equation] [Equation]
=
[Equation]
, L
[Equation]
, h
[Equation]
= m
[Equation]
j = 1 , 2 , and Q ∈ R m 1 × m 2 .
Suppose the Lipschitz constant of g is η .
Then the covering numbers of S with respect to the Frobenius norm satisfy
[Equation]
where p =[-m 1 m 2 + (cid:80) 
-]
[-j =1 (cid:80) L j l =
 h ( j ) l h ( j ) l − 1 and s = β Q η L 1 + L 2 (cid:0) (cid:81) L 1 l =0 β (1) l (cid:1)(cid:0) (cid:81) L 2 l =0 β (2) l (cid:1) .-][Equation]
Similar to the proof of Theorem [Equation] using Lemma
[Equation]
we have
[Equation]
where N = n 1 n 2 . Let (cid:15) = 3 ξ ( k + 1)(max( L 1 , L 2 ) + 1) .
According to Lemma 3 and we obtain
[Equation]
Now we have 1 √
[Equation]
This ﬁnished the proof.
C. P ROOF FOR T HEOREM
The following lemma (proved in Appendix D.4) provides an upper bound for the covering number of the multi-mode deep factorization tensor set.
Lemma 4.
Let S = { X ∈ R n 1 × n 2 ···× n k : X = C ×
1 f θ 1 ( Z (1) ) × {+2 f θ 
+}
( Z (2) ) · · · × k f θ k
( Z ( k ) ); f θ j ( Z ) = g (
[Equation]
( j ) L
[Equation]
g (
[Equation]
( j ) L j − 1 · · · g ( W ( j ) 1 Z ( j ) )
[Equation]
j = 1 , . .
, k ; (cid:107) C (cid:107) F ≤ β C , (cid:107) Z ( j ) (cid:107) F ≤ β ( j ) 0 , (cid:107) W ( j ) l (cid:107) F ≤ β ( j ) l , 1 ≤ l ≤ L j , j = 1 , . . .
, k } , where Z ( j ) ∈ R d j × n j ,
W ( j ) l ∈
[Equation] [Equation]
=
[Equation]
[Equation] [Equation]
=
[Equation]
= 1 , . .
k , and
C ∈ R m 1 × m 2 ···× m k .
Suppose the Lipschitz constant of g is η .
Then the covering numbers of S with respect to the Frobenius norm satisfy
[Equation]
where p =
[Equation]
The proof is similar to that for Theorem 2.
Using Lemma 1 and Lemma 4 , we arrive at
[Equation]
where c is a constant.
D P ROOF FOR LEMMAS D. P ROOF FOR L EMMA
We give the following lemma.
Lemma 5 (Hoeffding inequality for sampling without replacement (Serﬂing, 1974)) .
Let
X 1 , X 2 , . .
, X s be a set of samples taken without replacement from a distribution { x 1 , x 2 , . . .
, x N } of mean u and variance σ 2 .
Denote a = min
i x i and b = max i x i .
Then
[Equation]
We deﬁne
[Equation]
where N = (cid:81) ki =1 n i .
Suppose max {(cid:107) Y (cid:107) ∞ , (cid:107) X (cid:107) ∞ } ≤ ξ .
According to Lemma [Equation] we have
[Equation]
where δ = 4 ξ 2 .
Using union bound for all ¯ X ∈ S yields
[Equation]
(cid:19) .
Or equivalently, with probability at least 1 − 2 N − 1 , sup ¯
{+X ∈S | ˆ L ( ¯ X ) − L ( ¯ X ) | ≤ (cid:115) δ 2 log ( |S| N ) (cid:18)| Ω | − N + N | Ω | (cid:19) ≤ (cid:115) δ 2 log ( |S| N ) 
 | Ω | (cid:44) ∆ .+}
Since |√ u − √ v | ≤ (cid:112) | u − v | holds for any non-negative u and v , we have sup ¯ X ∈S (cid:12)(cid:12)(cid:12)(cid:12)(cid:113)
ˆ L ( ¯ X ) − (cid:113) L ( ¯ X ) (cid:12)(cid:12)(cid:12)(cid:12) ≤ √ ∆ .
Recall that (cid:15) ≥ (cid:107)
X
− ¯ X (cid:107) F ≥ (cid:107)P ( X − ¯ X ) (cid:107) F , we have (
cid:12)(cid:12)(cid:12)(cid:12)(cid:112) L ( X ) − (cid:113) L ( ¯ X ) (cid:12)(cid:12)(cid:12)(cid:12)
= 1 √
N (cid:12)(cid:12)(cid:12) (cid:107) Y − X (cid:107) F − (cid:107)
Y − ¯ X (cid:107) F (cid:12)(cid:12)(cid:12) ≤ (cid:15) √
N and (cid:12)(cid:12)(cid:12)(cid:12)(cid:113) ˆ L ( X ) − (cid:113)
ˆ L ( ¯ X ) (cid:12)(cid:12)(cid:12)(cid:12) =
(cid:112) | Ω | (cid:12)(cid:12)(cid:12) (cid:107)P
[Table]
It follows that
[Equation]
This ﬁnished the proof
[Equation]
D. P ROOF FOR L EMMA
Let X
=
W L g ( W L − 1 · · · g ( W 1 Z )) [Equation] where Z ∈ R d × n , W l ∈ R h l × h l − 1 , l = 1 , . .
, L [Equation] and h
= d .
Let S ab :
= { W ∈ R a × b : (cid:107) W (cid:107) F ≤ β } .
Then there exists an (cid:15) -net
¯ S ab obeying
N ( S ab , (cid:107) · (cid:107) F , (cid:15) ) ≤ (cid:18) 3 β (cid:15) (cid:19) ab such that (cid:107) W − ¯ W (cid:107) F ≤ (cid:15) .
Now replace (cid:15) with (cid:15)/γ and let (cid:107) W l − ¯ W l (cid:107) F ≤ (cid:15) γ l , l = 1 , . . .
, L , (cid:107) Z − ¯ Z (cid:107) F ≤ (cid:15) γ 0 .
Let γ l =
( L + 1)
η L − 1 (cid:81) i (cid:54) =
l β i , l = 0 , . . .
, L .
(cid:107)
[Equation]
F
[Equation]
F ≤(cid:107)
[Equation]
W L g ( ¯ W L − 1 · · · ¯ W 2 g ( W 1 Z ))
− ¯ W L g ( ¯ W L − 1 · · · ¯ W 2 g ( ¯ W 1 Z ))
(cid:107) F + (cid:107) ¯ W L g ( ¯ W L − 1 · · · ¯ W 2 g ( ¯ W 1 Z ))
− ¯ W L g ( ¯ W L − 1 · · · ¯ W 2 g ( ¯ W 1 ¯ Z ))
(cid:107) F ≤ η L − 1 (cid:107) W L − ¯ W L (cid:107) F (cid:107) Z (cid:107) F
L − 1 (cid:89) l
= (cid:107) W l (cid:107) F + η L − 1 (cid:107) W L − 1 − ¯ W L − 1 (cid:107) F (cid:107) Z (cid:107) F (cid:107) ¯ W L (cid:107) F
L − 2 (cid:89) l = (cid:107) W l (cid:107) F + · · · + η L − 1 (cid:107) W 1 − ¯ W 1 (cid:107) F (cid:107) Z (cid:107) F
L − 1 (cid:89) l
= (cid:107) ¯ W l (cid:107) F + η L − 1 (cid:107)
Z − ¯ Z (cid:107) F
L (cid:89) l
= (cid:107) ¯ W l (cid:107) F ≤ η L − 1 (cid:15) γ
L
L − 1 (cid:89) l = β l + η L −(cid:15)
γ L −(cid:89) l (cid:54) =
L −β l + · · · + η L − 1 (cid:15) γ 1 (cid:89) l (cid:54) =
β l + η L − 1 (cid:15) γ (cid:89) l (cid:54) =
β l = (cid:15).
The second inequality utilized the Lipschitz continuity of g and the submultiplicativity of the Frobenius norm.
Therefore, ¯ S is an (cid:15) -cover of S .
We have
[Equation]
D. P ROOF FOR L EMMA
Lemma 3 is a special case of Lemma 4.
D. P ROOF FOR L EMMA
Let ρ
= (cid:107) C (cid:107) F .
For j = 1 , . . .
, k , we have
[Equation]
Denote τ = (cid:81) kj =
ρ j .
Similar values can be deﬁned for ¯ C and ¯ U j .
Let S jl :=
{ W ( j ) l ∈ R a × b : (cid:107) W ( j ) l (cid:107) F ≤ β ( j ) l } .
Then there exists an (cid:15) -net
¯ S jl obeying
N ( S jl , (cid:107) · (cid:107) F , (cid:15) ) ≤ (cid:32) 3 β ( j ) l (cid:15) (
cid:33) h ( j ) l h ( j ) l −such that (cid:107) W ( j ) l − ¯ W ( j ) l (cid:107) F ≤ (cid:15) .
Let Γ ( j ) l =
( L j + 1) η L j (cid:81) i (cid:54) =
l β i , l = 0 , . . . , L j , and Υ j =
( k + 1) (cid:81) t (cid:54) =
j ρ t , j = 0 , 1 , . . .
, k .
Replace (cid:15) with (cid:15) Γ ( j ) l Υ j , we have (cid:107) W ( j ) l
[Equation]
≤
[Equation]
Υ j and have
[Equation]
It follows that
[Equation]
W ( j ) 2 g ( W ( j ) 1 Z ( j ) )))
− g ( ¯ W ( j ) L j g ( ¯ W ( j ) L j − 1 · · · W ( j ) 2 g ( W ( j ) 1 Z j )))
(cid:107) F + · · · + (cid:107) g
( ¯ W ( j ) L j g ( ¯ W ( j ) L j − 1 · · · ¯ W ( j ) 2 g ( W ( j ) 1 Z ( j ) )))
− g ( ¯ W ( j ) L j g ( ¯ W ( j ) L j − 1 · · · ¯ W ( j ) 2 g ( ¯ W ( j ) 1 Z j )))
(cid:107) F + (cid:107) g
( ¯ W ( j ) L j g ( ¯ W ( j ) L j − 1 · · · ¯ W ( j ) 2 g ( ¯ W ( j ) 1 Z ( j ) )))
− g ( ¯ W ( j ) L j g ( ¯ W ( j ) L j − 1 · · · ¯ W ( j ) 2 g ( ¯ W ( j )¯ Z ( j ) )))
(cid:107) F ≤ η L j (cid:107) W ( j ) L j − ¯ W ( j ) L j (cid:107) F (cid:107) Z ( j ) (cid:107) F
L j − 1 (cid:89) l
= (cid:107) W ( j ) l (cid:107) F + η L j (cid:107) W ( j ) L j − 1 −
¯ W ( j ) L j − 1 (cid:107) F (cid:107) Z ( j ) (cid:107) F (cid:107) ¯ W ( j ) L j (cid:107) F
L j − 2 (cid:89) l = (cid:107) W ( j ) l (cid:107) F + · · · + η L j (cid:107) W ( j )− ¯ W ( j ) 1 (cid:107) F (cid:107) Z ( j ) (cid:107) F
[-β ( j ) l + η L j (cid:15) Γ ( j ) 
-]
L − 1 (cid:89) l = (cid:107)
[Equation]
Υ j
[Equation]
[Equation]
Now we have
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Therefore ¯ S is an (cid:15) -cover of S .
Then the covering number of S dnk can be bounded as
[Equation]
j =
L j (cid:89) l = (cid:32) 3Γ ( j ) l Υ j β ( j ) l (cid:15) (cid:33) h ( j ) l h ( j ) l −.
= (cid:18) 3Υ 0 ρ(cid:15) (cid:19) (cid:81) kj =1 m j k (cid:89)
j =
L j (cid:89) l = (cid:18) 3( L j + 1) ρ j Υ j (cid:15) (cid:19) h ( j ) l h ( j ) l −= (cid:18)
( k + 1) τ (cid:15) (cid:19) (cid:81) kj =
m j k (cid:89)
j =
L j (cid:89) l = (cid:18) 3( k + 1)( L j + 1) τ (cid:15) (cid:19) h ( j ) l h ( j ) l −.
≤ (cid:18)
( k + 1) τ (cid:15) (cid:19) (cid:81) kj =
m j k (cid:89)
j = (cid:18) 3( k + 1)( L max + 1) τ (cid:15) (cid:19) (cid:80) Ljl =
0 h ( j ) l h ( j ) l −= (cid:18) 3( k + 1) τ (cid:15) (cid:19) (cid:81) k
j =1 m j (cid:18) 3( k + 1)( L max + 1) τ (cid:15) (cid:19) (cid:80) k
j =1 (cid:80)
Lj l =
h
( j ) l h ( j ) l −≤ (cid:18) 3( k + 1)( L max + 1) τ (cid:15) (cid:19) (cid:81) kj = 1 m j
[Equation]