Subquadratic Kronecker Regression with Applications to Tensor Decomposition
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Kronecker regression is a highly-structured least squares problem min x k Kx  b k 2 2 ,where the design matrix K
= A (1) ⌦ · · · ⌦ A ( N ) is a Kronecker product of factormatrices.
This regression problem arises in each step of the widely-used alternatingleast squares (ALS) algorithm for computing the Tucker decomposition of a tensor.
We present the ﬁrst subquadratic-time algorithm for solving Kronecker regressionto a (1+ " ) -approximation that avoids the exponential term
O ( "  N ) in the runningtime.
Our techniques combine leverage score sampling and iterative methods.
Byextending our approach to block-design matrices where one block is a Kroneckerproduct, we also achieve subquadratic-time algorithms for (1) Kronecker ridgeregression and (2) updating the factor matrix of a Tucker decomposition in ALS,which is not a pure Kronecker regression problem, thereby improving the runningtime of all steps of Tucker ALS.
We demonstrate the speed and accuracy of this
Kronecker regression algorithm on synthetic data and real-world image tensors.
Introduction
Tensor decomposition has a rich multidisciplinary history with countless applications in data mining,machine learning, and signal processing [31, 51, 53, 28].
The most widely-used tensor decompositions are the CP decomposition and the Tucker decomposition.
Similar to the singular valuedecomposition of a matrix, both decompositions have natural analogs of low-rank structure.
Unlikematrix factorization, however, computing the rank of a tensor is NP-hard [24].
Therefore, most lowrank tensor decomposition algorithms decide on the rank structure in advance, and then optimize thevariables of the decomposition to ﬁt the data.
While conceptually simple, this approach is extremelyeffective in many applications.
The alternating least squares (ALS) algorithm is the main workhorse for low-rank tensor decomposition (e.g., it is the ﬁrst algorithm mentioned in the MATLAB Tensor Toolbox [5]).
For both CPand Tucker decompositions, ALS cyclically optimizes disjoint blocks of variables while keeping allothers ﬁxed.
As the name suggests, each step solves a linear regression problem.
The core tensorupdate step in ALS for Tucker decompositions is notoriously expensive but highly structured.
In fact,the design matrix of this regression problem is the Kronecker product of the factor matrices of the
Tucker decomposition K =
A (1) ⌦ · · · ⌦ A ( N ) .
Our work builds on a line of Kronecker regressionalgorithms [15, 16, 43] to give the ﬁrst subquadratic-time algorithm for solving Kronecker regressionto a (1 + " ) -approximation while avoiding an exponential term of O ( "  N ) in the running time.
We combine leverage score sampling with iterative methods to fully exploit the Kronecker structureof the design matrix.
We also extend our approach to block-design matrices where one block is a
Kronecker product, achieving subquadratic-time algorithms for (1) Kronecker ridge regression and(2) updating the factor matrix of a Tucker decomposition in ALS, which is not a pure Kroneckerregression problem.
Putting everything together, this work improves the running time of all steps of
ALS for Tucker decompositions and runs in time that is sublinear in the size of the input tensor, linearin the error parameter "  1 , and subquadratic in the number of columns of the design matrix in eachstep.
Our algorithms support L2 regularization in the Tucker loss function, so the decompositionscan readily be used in downstream learning tasks (e.g., using the factor matrix rows as embeddingsfor clustering [62]).
Regularization also plays a critical role in the more general tensor completionproblem to prevent overﬁtting when data is missing and has applications in differential privacy [8, 6].
The current fastest Kronecker regression algorithm of Diao et al.
[16] solves this problem by leveragescore sampling and achieves the following running times for A ( n ) 2 R I n ⇥ R n with I n  R n , for alln 2 [ N ] , and R =
Q
Nn =1 R n :
˜ O
( P Nn =1 ( nnz ( A ( n ) )
+ R ! n )+ R ! "
1 ) by sampling ˜ O
( R "  1 ) rows from K by their leveragescores.2.
˜ O
( P Nn =1 ( nnz ( A ( n ) )
+ R !
n "  1 ) + R "  N ) by sampling ˜ O
( R n "  1 ) rows from each factormatrix
A ( n ) and taking the Kronecker product of the sampled factor matrices.
Note that the second approach is linear in R , but we have to pay an exponential cost in the numberof factor matrices for the error parameter.
In this paper, we show that the running time of the ﬁrstapproach can be improved to subquadratic in R without increasing the running time dependence on " ,simultaneously improving on both approaches.
Theorem 1.1.
For n 2 [ N ] , let A ( n ) 2 R I n ⇥ R n , I n  R n , and b 2 R I 1 ··· I n .
There is a (1 + " ) -approximation algorithm for solving arg min x k ( A (1) ⌦ · · · ⌦ A ( N ) )
x  b k 22 that runs in time
[Equation]
where MM ( a, b, c ) is the running time of multiplying an a ⇥ b matrix with a b ⇥ c matrix.
Without using the fast matrix multiplication of Gall and Urrutia [21] and Alman and Williams [4], thelast term in (1) is ˜
O
( R 2 "  1 ) , which is already an improvement over the normal ˜ O
( R 3 "  1 ) runningtime.
With fast matrix multiplication, MM (
[Equation]
) is subquadratic in R ,for any nontrivial subset S 62 { ; , [ N ] } , which is an improvement over ˜ O
( R ! "
[Equation]
⇡ ˜ O
( R 2 .
373 "  1 ) .
In cases when there exists a “balanced” subset S such that
[Equation]
, our running time goesas low as ˜ O
( R 1 .
626 "  1 ) using [21].
For ease of notation, we denote the subquadratic improvementby the constant ✓ ⇤ > 0 , where R 2  ✓ ⇤ =
min S ✓ [ N ]
[Equation]
Although updating the core tensor in the ALS algorithm for Tucker decomposition is a pure Kroneckerproduct regression as described in Theorem 1.1, updating the factor matrices is a regression problemof the form arg min x k KMx  b k 22 , where K is a Kronecker product and M is a matrix withoutany particular structure.
We show that such problems can be converted to block regression problemswhere one of the blocks is the Kronecker product K .
We then design sublinear-time leverage scoresampling techniques for these block matrices, which leads to the following theorem about acceleratingall ALS steps.
Theorem 1.2.
There is an ALS algorithm for L2-regularized Tucker decompositions that takes a tensor 72 X 2 R I 1 ⇥ ··· ⇥ I N and returns N factor matrices A ( n ) 2 R I n ⇥ R n and a core tensor G 2 R R 1 ⇥ ··· ⇥ R nsuch that each factor matrix and core update is a (1 + " ) -approximation to the optimum with highprobability.
The running times of each step are:
• Core tensor:
[Equation]
• Factor matrix:
[Equation]
where R =
Q
N n =1 R n , R 6 = k = R/R k , and ✓ ⇤ > 0 is a constant derived from fast rectangular matrixmultiplication.
Note that for tensors of relatively large order, the superlinear term in R is the bottleneck in manyapplications since R is exponential in the order of the tensor.
Thus, our improvements are signiﬁcantin both theory and practice as illustrated in our experiments in Section 6.
1.1 Our Contributions and Techniques
We present several new results about approximate Kronecker regression and the ALS algorithm for
Tucker decompositions.
Below is a summary of our contributions:
Our main technical contribution is the algorithm FastKroneckerRegression in Section
This Kronecker regression algorithm builds on the block-sketching tools introduced in Section 3, and combines iterative methods with fast rectangular matrix multiplication to achievea running time that is subquadratic in the number of columns in the Kronecker matrix.
Ourkey insight is to use the original (non-sketched)
Kronecker product as the preconditionerin the Richardson iterations when solving the sketched problem.
This allows us to fullyexploit the Kronecker structure through fast Kronecker matrix-vector multiplications (e.g.,the KronMatMul algorithm in Lemma 4.4), instead of computing the pseduoinverse of thesketched normal matrix (( SK )
| SK +  I ) + as in Diao et al.
We generalize our Kronecker regression techniques to work for Kronecker ridge regressionand the factor matrix updates in ALS for Tucker decomposition.
We show that a factormatrix update is equivalent to solving an equality-constrained Kronecker regression problemwith a low-rank update to the preconditioner in the Richardson iterations.
We can implementthese new matrix-vector products nearly as fast by using the Woodbury matrix identity.
Thus,we provably speed up each step of Tucker ALS (i.e., the core tensor and factor matrices).3.
We give a block-sketching toolkit in Section 3 that states we can sketch blocks of a matrixby their leverage scores (i.e., their leverage scores in isolation, not with respect to the entireblock matrix).
This is one of the ways we exploit the Kronecker product structure of thedesign matrix.
This approach can be useful for constructing spectral approximations andfor approximately solving block regression problems.
One corollary is that we can use the“sketch-and-solve” method for any ridge regression problem (Corollary 3.5).4.
We compare FastKroneckerRegression with Diao et al.
[16, Algorithm 1] on a synthetic
Kronecker regression task studied in [15, 16] and as a subroutine in ALS for computing
the
Tucker decomposition of the image tensors [40, 46, 47].
Our results highlight the importanceof reducing the runtime dependency on the number of columns in the Kronecker product.
1.2 Related Work
Kronecker Regression.
Diao et al.
[15] recently gave the ﬁrst Kronecker regression algorithmbased on TensorSketch [49] that is faster than forming the Kronecker product.
Diao et al.
[16]improved this by removing the dependence on O ( nnz ( b )) from the running time, where b 2 R I 1 ··· I Nis the response vector.
Marco et al.
[43] have studied the generalized Kronecker regression problem.
Ridge Leverage Scores.
Alaoui and Mahoney [3] extended the notion of statistical leverage scoresto account for L2 regularization.
Sampling from approximate ridge leverage score distributions hassince played a key role in sparse low-rank matrix approximation [14], the Nyström method [45],bounding statistical risk in ridge regression [44], and ridge regression [12, 44, 37, 29].
Fast recursivealgorithms for computing approximate leverage scores [13] and for solving overconstrained leastsquares [36] are also closely related.
Tensor Decomposition.
Cheng et al.
[11] and Larsen and Kolda [34] used leverage score samplingto speed up ALS for CP decomposition.
1 Song et al.
[54] gave a polynomial-time, relative-errorapproximation algorithm for several low-rank tensor decompositions, which include CP and Tucker.
Frandsen and Ge [20] showed that if the tensor has an exact Tucker decomposition, then all localminima are globally optimal.
Randomized low-rank Tucker decompositions based on sketching havebecome increasingly popular, especially in streaming applications: [41, 56, 9, 55, 28, 42, 40, 2].
Themore general problem of low-rank tensor completion is also a fundamental approach for estimatingthe values of missing data [1, 39, 26, 25, 19].
Fundamental algorithms for tensor completion are basedon ALS [63, 22, 38], Riemannian optimization [33, 30, 48], or projected gradient methods [60].
Input: X 2 R I 1 ⇥ ··· ⇥ I N , ( R 1 , R 2 , . . .
, R N ) ,  1: Initialize core tensor G 2 R R 1 ⇥ R 2 ⇥ ··· ⇥ R n 2: Initialize factors A ( n ) 2 R I n ⇥ R n for n 2 [ N ] 3: repeat 4: for n = 1 to N do 5:
K
A (1) ⌦ · · · ⌦ A ( n  1) ⌦ A ( n +1) ⌦ · · · ⌦ A ( N ) 6:
B
X ( n ) 7: for i = 1 to I n do 8:
y ⇤ arg min y k KG | ( n ) y  b
| i : k 22 +  k y k9:
Update factor row a ( n ) i : y ⇤ | 10:
K
A (1) ⌦ A (2) ⌦ · · · ⌦ A ( N ) 11: g ⇤ arg min g k Kg  vec ( X ) k 2
+  k g k 212:
Update core tensor G vec  1 ( g ⇤ ) 13: until convergence 14: return G , A (1) , A (2) , . .
A ( N )
Notation.
The order of a tensor is the number of its dimensions.
We denote scalars bynormal lowercase letters x 2 R , vectors byboldface lowercase letters x 2 R n , matricesby boldface uppercase letters X 2 R m ⇥ n , andhigher-order tensors by boldface script letters
X 2 R
I 1 ⇥ I 2 ⇥ ··· ⇥ I N .
We use normal uppercaseletters to denote the size of an index set (e.g.,[ N ] = { 1 , 2 , . . .
, N } ).
The i -th entry of a vector x is denoted by x i , the ( i, j ) -th entry of amatrix X by x ij , and the ( i, j, k ) -th entry of athird-order tensor X by x ijk .
Linear Algebra.
Let I n denote the n ⇥ n identity matrix and 0 m ⇥ n denote the m ⇥ n zero 145 matrix.
The transpose of a matrix A 2 R m ⇥ nis A | and the Moore–Penrose inverse is A + .
The singular value decomposition (SVD) of Ais a factorization of the form U ⌃ V | , where U 2 R m ⇥ m and V 2 R n ⇥ n are orthogonal matrices, and ⌃ 2 R m ⇥ n is a diagonal matrix withnon-negative real numbers on its diagonal.
The entries  i ( A ) of ⌃ are the singular values of A ,and the number of non-zero singular values is equal to r = rank ( A ) .
The compact SVD is a relateddecomposition where ⌃ 2 R r ⇥ r is a diagonal matrix containing the non-zero singular values.
The Kronecker product of two matrices A 2 R m ⇥ n and B 2 R p ⇥ q is denoted by A ⌦ B 2 R ( mp ) ⇥ ( nq ) .
Tensor Products.
Fibers of a tensor are the vectors we get by ﬁxing all but one index.
If X is athird-order tensor, we denote the column, row, and tube ﬁbers by x : jk , x i : k , and x ij : , respectively.
The mode- n unfolding of a tensor X 2 R I 1 ⇥ I 2 ⇥ ··· ⇥ I N is the matrix X ( n ) 2 R I n ⇥ ( I 1 ··· I n  1 I n +1 ··· I N )that arranges the mode- n ﬁbers of X as columns of X ( n ) ordered lexicographically by index.
Thevectorization of X 2 R I 1 ⇥
I 2 ⇥ ··· ⇥ I N is the vector vec ( X ) 2 R
I 1 I 2 ··· I N formed by vertically stackingthe entries of X ordered lexicographically by index.
For example, this transforms X 2 R m ⇥ n into atall vector vec ( X ) by stacking its columns.
We use vec  1 ( x ) to undo this operation when it is clearfrom context what the shape of the output tensor should be.
The n -mode product of tensor X 2 R I 1 ⇥
I 2 ⇥ ··· ⇥ I N and matrix A 2 R J ⇥
I n is denoted by Y
= X ⇥ n Awhere Y 2 R
I 1 ⇥ ··· ⇥ I n  1 ⇥ J ⇥ I n +1 ⇥ ··· ⇥ I N .
This operation multiplies each mode- n ﬁber of X by thematrix A .
This operation is expressed elementwise as
[Equation]
The Frobenius norm k X k F of a tensor X is the square root of the sum of the squares of its entries.
Tucker Decomposition.
The Tucker decomposition decomposes tensor X 2 R
I 1 ⇥ I 2 ⇥ ··· ⇥ I N intoa core tensor G 2 R R 1 ⇥ R 2 ⇥ ··· ⇥ R N and N factor matrices A ( n )
2 R
I n ⇥ R n .
We compute a Tuckerdecomposition by minimizing the nonconvex loss function
[Equation]
A
[Equation]
[Equation]
[Equation]
[Equation]
where  is the L2 regularization. Elements in the reconstructed tensor [Equation] A ( N ) 171 are
[Equation]
Equation (2) demonstrates that b
X is the sum of R 1 · · · R N rank-1 tensors.
The tuple ( R 1 , R 2 , . . . , R N )is the multilinear rank of the decomposition.
The multilinear rank is typically chosen in advance andmuch smaller than the dimensions of X .
Alternating Least Squares.
We present TuckerALS in Algorithm 1 and highlight its connectionsto Kronecker regression.
The core tensor update (Lines 10–12) is a ridge regression problem wherethe design matrix K core 2 R I 1 ··· I N ⇥ R 1 ··· R N is a Kronecker product of the factor matrices.
Each factormatrix update (Lines 5-9) also has Kronecker product structure, but there are additional subspaceconstraints we must account for.
We describe these constraints in more detail in Section 5.
Row Sampling and Approximate Regression
Here we establish our sketching toolkit.
The  -ridge leverage score of the i -th row of A 2 R n ⇥ d is
[Equation]
The matrix of cross  -ridge leverage scores is A ( A | A +  I )
+ A | .
We denote its diagonal by `  ( A )because it contains the  -ridge leverage scores of A .
Ridge leverage scores generalize statisticalleverage scores in that setting  = 0 gives the leverage scores of A .
We denote the vector of statisticalleverage scores by ` ( A ) .
If A = U ⌃ V | is the compact SVD of A , then for all i 2 [ n ] , we have
[Equation]
(4) where r = rank ( A ) .
It follows that every `  i ( A )  1 since U is an orthogonal matrix.
We direct thereader to Alaoui and Mahoney [3] or Cohen et al.
[13] for further details.
The main results in this paper build on approximate leverage score sampling for block matrices.
The -ridge leverage scores of A 2 R n ⇥ d can be computed by appending p  I d to the bottom of A toget A 2 R ( n + d ) ⇥ d and considering the leverage scores of A , so we state the following results interms of statistical leverage scores without loss of generality.
Deﬁnition 3.1.
For any A 2 R n ⇥ d , the vector ˆ ` ( A ) 2 R n is a  -overestimate for the leverage scoredistribution of A if, for all
i 2 [ n ] , it satisﬁes
[Equation]
Next we describe the approximate leverage score sampling algorithm in Woodruff [59, Section 2.4].
The core idea here is that if we sample ˜ O ( d/  ) rows and reweight them appropriately, this smallersketched matrix can be used instead of A to give provable guarantees for many problems.
Deﬁnition 3.2 (Leverage score sampling) .
Let A 2 R n ⇥ d and p 2 [0 , 1] n be a  -overestimatefor the leverage score distribution of A .
SampleRows ( A , s, p ) denotes the following procedure.
Initialize sketch matrix S = [-
-]0 s ⇥ n .
For each row i of S , independently and with replacement, selectan index j 2 [ n ] with probability p j and set s ij = 1 / p p j s .
Return sketch S .
The main result in this section is that we can choose to sketch a single block of a matrix by theleverage scores of that block in isolation.
This sketched submatrix can then be used with the other(non-sketched) block to give a spectral approximation to the original matrix or for approximate linearregression.
The notation A 4 B is the Loewner order and means B  A is positive semideﬁnite.
Lemma 3.3.
Let A =
[ A 1 ; A 2 ] be vertically stacked with A 1 2 R n 1 ⇥ d and A 2 2 R n 2 ⇥ d .
Let p 2[0 , 1] n 1 be a  -overestimate for the leverage score distribution of A 1 .
If s > 144 d ln(2 d/  ) /
( " 2 ) ,the sketch S returned by SampleRows ( A 1 , s, p ) guarantees, with probability at least 1   , that
[Equation]
Lemma 3.4 (Approximate block regression) .
Consider the problem arg min x 2 R d k
Ax  b k 22 where
A =
[ A 1 ; A 2 ] and b = [ b 1 ; b 2 ] are vertically stacked and A 1 2 R n 1 ⇥ d , A 2 2 R n 2 ⇥ d , b 1
R n 1 , b 2 2 R n 2 .
Let p 2 [0 , 1] n 1 be a  -overestimate for the leverage score distribution of A 1 .
Lets  1680 d ln(40 d ) /
( " ) and let S be the output of SampleRows ( A 1 , s, p ) .
If˜x ⇤ = arg [Equation] x
[Equation]
, then, with probability at least 9 / 10 , we have k
A˜x
⇤
b k 22  (1 + " ) min x 2 R d k Ax  b k 22 .
We defer the proofs of these results to Appendix A.
The key idea behind Lemma 3.4 is that leveragescores do not increase if rows are appended to the matrix.
This then allows us to prove a sketchedsubmatrix version of Drineas et al.
[17, Lemma 8] for approximate matrix multiplication and satisfythe structural conditions for approximate least squares in Drineas et al.
One consequence is thatwe can “sketch and solve” ridge regression problems, which was recently shown in [58, Theorem 1].
Corollary 3.5.
For any A 2 R n ⇥ d , b 2 R d ,   0 , consider arg min x 2 R d ( k Ax  b k [-
-]22 +  x 22 ) .
219 Let p 2 [0 , 1] n 1 be a  -overestimate for the leverage scores of A and s  1680 d ln(40 d ) /
( " ) .
If
S is the output of SampleRows ( A , s, p ) , then, with probability at least 9 / 10 ,
the sketched solution˜x ⇤ = arg min x 2 R d ( k S
( Ax  b ) k
+  x 22 ) gives a (1 + " ) -approximation to the original problem.
Remark 3.6.
The success probability of the sketch can be boosted from 9 / 10 to 1   by sampling afactor of O (log(1 /  )) more rows.
See the discussion in Chen and Price [10, Section 2] about matrixconcentration bounds for more details.
Kronecker Regression
Now we describe the key ingredients that allow us to design an approximate Kronecker regressionalgorithm whose running time is subquadratic in the number of columns in the design matrix.1.
The leverage score distribution of a Kronecker product matrix K
= A (1) ⌦ · · · ⌦ A ( N ) is aproduct distribution of the leverage score distributions of its factor matrices.
Therefore, wecan sample rows of K from ` ( K ) with replacement in ˜ O
(1) time after a preprocessing step.2.
The normal matrix K | K
+  I in the ridge regression problem
min x k Kx  b k 22 +  k x k 22 isa O (1) -spectral approximation to the sketched version ( SK )
| SK +  I by Lemma 3.3.
Thismeans we can use Richardson iteration with ( K | K +  I ) + as the preconditioner to solve thesketched instance , which guarantees us a (1+ " ) -approximation.
Using ( K | K +  I ) + as thepreconditioner allows us to heavily exploit the Kronecker structure with fast matrix-vectormultiplications.3.
At this point, Kronecker matrix-vector multiplications are still the bottleneck, so we partitionthe factor matrices into two groups by their number of columns and use fast rectangularmatrix multiplication to get a subquadratic running time.
This ﬁrst result shows how  -ridge leverage scores of a Kronecker product matrix decompose according to the SVDs of its factor matrices.
All missing proofs in this section are deferred to Appendix B.
Lemma 4.1.
Let K
= A (1) ⌦ A (2) ⌦ · · · ⌦ A ( N ) , where each factor matrix A ( n ) 2 R I n ⇥ R n .
Let( i 1 , i 2 , . .
, i N ) be the natural row indexing of K by its factors.
Let the factor SVDs be A ( n ) =
U ( n ) ⌃ ( n ) V ( n ) | .
For any   0 , the  -ridge leverage scores of K are
[Equation]
where the sum is over T
=
[ R 1 ] ⇥
[ R 2 ] ⇥ · · · ⇥ [ R N ] .
For statistical leverage scores, this simpliﬁesto ` ( i 1 ,...,i N ) ( K ) =
Q
Nn =1 ` i n ( A ( n ) ) .
This proof repeatedly uses the mixed-product property for Kronecker products and the deﬁnition of -ridge leverage scores in Equation (3).
4.1 Iterative Methods
Now we state a result for the convergence rate of preconditioned Richardson iteration [52], whichuses the notation k x k 2 M = x | Mx .
Lemma 4.2 (Preconditioned Richardson iteration) .
Let M be a matrix where A | A 4 M 4  · A | Afor some   1 .
Let x ( k +1)
= x ( k )  M +
( A | Ax ( k )  A | b ) .
Then,
[Equation] where x ⇤ =
arg min x 2 R d
k
Ax  b k 22 .
Remark 4.3.
The ridge regression algorithm in Chowdhury et al.
[12] is also based on sketching andpreconditioned Richardson iteration.
They consider short and wide matrices where d  n and usethe sketched normal matrix as the preconditioner to solve the original problem.
One of our maintechnical contributions is to use the original normal matrix as the preconditioner to solve the sketchedproblem.
[-( R n N 2 "  2 ) rows by Lemma 3.
-]
Reversing this is advantageous because computing the pseduoinverse and matrix-vectorproducts with the original Kronecker matrix is substantially less inexpensive due to its Kroneckerstructure.
However, this still motivates the need for faster Kronecker matrix-vector multiplications.
4.2 Fast Kronecker-Matrix Multiplication
The next result is a simple but useful observation about extracting the rightmost factor matrix from the
Kronecker product and recursively computing a new less expensive Kronecker-matrix multiplication.
Lemma 4.4.
Let A ( n ) 2 R I n ⇥ J n , for n 2 [ N ] , and B 2 R J 1 ··· J N ⇥ K .
There is an algorithm
KronMatMul ([ A (1) , . . .
, A ( N ) ] , B ) that computes
[Equation]
The following theorem is more sophisticated.
We write the statement in terms of rectangular matrixmultiplication time MM ( a, b, c ) , which is the time to multiply an a ⇥ b matrix by a b ⇥ c matrix.
Theorem 4.5.
Let A ( n ) 2 R I n ⇥ R n , for n 2 [ N ] , I = I 1 · · · I N , R = R 1 · · · R N , b 2 R
I , c 2 R R ,and S 2 R
I ⇥ I be a diagonal matrix with ˜ O
( R "  1 ) nonzeros.
The vectors ( A 1 ⌦ · · · ⌦ A N ) | Sb and
S ( A 1 ⌦ · · · ⌦ A N ) c can be computed in time
˜ O
(min T ✓ [ N ] MM (
[Equation]
The core idea behind Theorem 4.5 is that the factor matrices can be partitioned into two groups toachieve a good “column-product” balance, i.e., min T ✓ [ N ] max {
[Equation]
Q n 62 T R n } is close top R .
Then we use the fact that nnz ( S ) =
˜ O ( R "  1 ) with a sparsity-aware KronMatMul to solve eachpart of this partition separately, and combine them with fast rectangular matrix multiplication.
Ifwe achieve perfect balance, the running time is ˜ O
( R 1 .
626 "  1 ) using results of Gall and Urrutia [21],which are explained in detail in van den Brand and Nanongkai [57, Appendix C].
If one of these twofactor matrix groups has at most 0 .
9 of the “column-product mass,” the running time is ˜ O
( R 1 . 9 "  1 ) .
4.3 Main Algorithm
We are now ready to present our main algorithm for solving approximate Kronecker regression.
Theorem 4.6.
For any Kronecker product matrix K
= A (1) ⌦ · · · ⌦ A ( N ) 2 R I 1 ··· I N ⇥ R 1 ··· R N ,b 2 R I 1 ··· I N ,   0 , " 2 (0 , 1 / 4] , and  > 0 , FastKroneckerRegression returns x ⇤ 2 R R 1 ··· R Nin ˜ O (( R 1 · · · R N ) 2 "  1 log(1 /  )
+
P
[Equation]
I n R 2 n ) time such that, with probability at least 1   ,
[Equation]
We defer the proof to Appendix B.2. The core idea is that the matrix-vector products with M + ineach Richardson iteration (Line 13 of Algorithm 2) uses two fast Kronecker matrix-vector products.
The approximation guarantee follows from the fact that the original normal matrix K | K +  I is aspectral approximation (hence a valid preconditioner) for the sketched Kronecker regression problem. Next, by using the theoretically faster MM-based Kronecker matrix-vector products in Theorem 4.5instead of KronMatMul (Lemma 4.4), we decrease the time complexity of Algorithm 2 from quadraticto subquadratic.
Corollary 4.7. Let R = R 1 · · · R N . The algorithm FastKroneckerRegression can run in time˜ O ( R 2  ✓ ⇤ "  1 log(1 /  ) +
{+P Nn =
 I n R 2 n ) , where ✓ ⇤ > 0 is a constant derived from+} the optimallybalanced MM expression in Theorem 4.5, i.e., R 2  ✓ ⇤ = min T ✓ [ N ] MM (
Q n 2 T R n , R, Q n/ 2 T R n ) . Input: Factor matrices A ( {+n ) 
 R I n ⇥+} R n , response vector b 2 R I 1 ··· I N , L2 regularization strength  , error " , failure probability  1: Set R
R 1 R 2 · · · R N 2: for n = 1 to N do 3:
Compute the SVD of ( A ( n ) ) | A ( n ) =
V ( n ) (( ⌃ ( n ) ) | ⌃ ( n ) )
( V ( n ) ) | 4: Compute the leverage scores ` ( A ( n ) ) using Equation (3) 5: Initialize product distribution data structure P to sample indices from ( ` ( A (1) ) , · · · , ` ( A ( N ) )) 6: Set D ( ⌃ (1) | ⌃ (1) ⌦ · · · ⌦ ⌃ ( N ) | ⌃ ( N ) +  I R ) + 7: Let M + =
( V (1) ⌦ · · · ⌦ V ( N ) )
D ( V (1) ⌦ · · · ⌦ V ( N ) ) | 8: Set s d 1680 R ln(40 R ) ln(1 /  ) / " e 9: Set S SampleRows ( K , s, P ) 10: Let ˜K = SK and ˜b = Sb 11: Initialize x 0 R 12: repeat 13: x x  (1  p " ) M + ( ˜K | ˜Kx +  x  ˜K | ˜b ) using fast Kronecker-matrix multiplication 14: until convergence 15: return x
Applications to Low-Rank Tucker Decomposition
Now we apply our fast Kronecker regression algorithm to TuckerALS and prove Theorem 1.2.
Welist the running times of different factor matrix and core update algorithms in Table 2 (Appendix C)and analyze these subroutines in Appendix C.3.
Core Tensor Update.
The core update running time in Theorem 1.2 is a direct consequence ofour algorithm for fast Kronecker regression in Theorem 4.6.
The only difference is that we avoidrecomputing the SVD and Gram matrix of each factor since these are computed at the end of eachfactor matrix update and stored for future use.
Factor Matrix Update.
The factor matrix updates require more work because of the G | ( n ) y termin Line 8 of TuckerALS .
To overcome this, we substitute variables and recast each factor update asan equality-constrained Kronecker regression problem with an appended low-rank block to accountfor the L2 regularization of the original variables.
To support this new low-rank block, we use the
Woodbury matrix identity to extend the technique of using Richardson iterations with fast Kroneckermatrix-vector multiplication for solving sketched regression instances.
The next result formalizes this substitution and reduces the problem to block Kronecker regressionwith a new subspace constraint.
This relies on the fact that the least squares solution to k
Mx  z k 22with minimum norm is M + z
All proofs in this section are deferred to Appendix C.
Lemma 5.1.
Let A 2 R n ⇥ m , M 2 R m ⇥ d , b 2 R n , and   0 .
For any ridge regression problemof the form arg min x 2 R d ( k AMx  b k
+
k x k 22 )
, we can solve z opt = arg min
Nz =
k Az  b k 22 + k M + z k 22 , where
N =
I m  MM + , and return vector M + z opt instead.
Letting z
=
G | ( n ) y in Line 8 of TuckerALS and modifying FastKroneckerRegression to supportadditional low-rank updates to the preconditioner, we get the FastFactorMatrixUpdate algorithm,presented as Algorithm 3 in Appendix C.2.
The analysis is similar to the proofs of Theorem 4.6and Corollary 4.7.
The factor matrix updates beneﬁt in the same way as before from fast Kroneckermatrix-vector products, and new low-rank block updates are supported via the Woodbury identity.
Theorem 5.2.
For any   0 , " 2 (0 , 1 / 4] , and  > 0 , the FastFactorMatrixUpdate algorithmupdates A ( n ) 2 R I n ⇥ R n in TuckerALS with a (1 + " ) -approximation, with probability at least 1   ,in time
˜ O
( I n R 2 6 = n "  1 log(1 /  )
+ I n R
P
Nk =
1 R k + R ! n ) .
Corollary 5.3.
FastFactorMatrixUpdate updates A ( n ) 2 R I n ⇥ R n in ˜ O
( I n R 2  ✓ ⇤ 6 = n "  1 log(1 /  )
+
I n R
P
Nk =
1 R k + R ! n ) time, where ✓ ⇤ > 0 is the optimally balanced MM exponent in Theorem 4.5.
Experiments
All experiments use NumPy [23] with an Intel Xeon W-2135 processor (8.25MB cache, 3.70 GHz)and 128GB of RAM.
The FastKroneckerRegression -based ALS experiments for low-rank Tuckerdecomposition on image tensors are deferred to Appendix D.2.
Kronecker regression.
We build on the numerical experiments in [15, 16] for Kronecker regressionthat use two random factor matrices.
We generate matrices A (1) , A (2) 2 R n ⇥ d where each entry isdrawn i.i.d.
from the normal distribution N (1 , 0 . 001)
and compare several algorithms for solvingmin x k ( A (1)
⌦ A (2) )
x  1 n 2 k
+  k x k 22 as we increase n, d .
The running times are plotted in Figure 1.
The algorithms we compare are: (1) a baseline that solves the normal equation ( K | K +  I )
+ K | b andfully exploits the Kronecker structure of K | K before calling np.linalg.pinv() ; (2) an enhancedbaseline that combines the SVDs of A ( n ) with Lemma 4.4, e.g., KronMatMul ([( U (1) )
| , ( U (2) ) | ] , b ) ,using only Kronecker-vector products; (3) the sketching algorithm of Diao et al.
[16, Algorithm 1];and (4) our FastKroneckerRegression algorithm in Algorithm 2.
For both sketching algorithms,we use " = 0 .
1 and  = 0 .
We reduce the number of row samples in both algorithms by ↵ = 10  5so that the algorithms are more practical and comparable to the earlier experiments in [15, 16].
Lastly,we set  = 10  3 .
We discuss additional parameter choice details and the full results in Appendix D.1.
The running times in Figure 1 demonstrate several different behaviors.
The naive baseline quicklybecomes impractical for moderately large values of n or d .
KronMatMul is competitive for n  10 4 ,especially since it is an exact method.
The runtimes of the sketching algorithms are nearly-independentof n .
Diao et al.
[16] works well for small d , but deteriorates tremendously as d grows because itcomputes (( SK ) | SK +  I ) + 2 R d 2 ⇥ d 2 and cannot exploit the Kronecker structure of K , whichtakes O ( d 6 ) time.
FastKroneckerRegression , on the other hand, runs in O ( d 4 ) time because ituses quadratic-time Kronecker-vector products in each Richardson iteration step (Line 13).
[Table]
These experiments also show that combining sketching with iterative methods can give better sketchefﬁciency .
Table 1 compares the loss of [16, Algorithm 1] and FastKroneckerRegression to anexact baseline OPT for d = 64 .
Both algorithms use the exact same sketch SK for each value of n .
Our algorithm uses the original ( K | K +  I )
+ as a preconditioner to solve the sketched problem,whereas Diao et al.
[16, Algorithm 1] computes (( SK ) | SK +  I ) + ( SK )
|
Sb exactly and becomesnumerically unstable for n  2048 when d 2 { 16 , 32 , 64 } .
This raises the question about how to 354 combine sketched information with the original data to achieve more efﬁcient algorithms, even whensolving sketched instances.
We leave this question of sketch efﬁciency as an interesting future work.