Dynamic Tensor Product Regression
Abstract
In this work, we initiate the study of Dynamic Tensor Product Regression .
One has matrices A 1 ∈ R n 1 × d 1 , . . .
, A q ∈ R n q × d q and a label vector b ∈ R n 1 ...n q , and the goal is to solve the regression problem with the design matrix
A being the tensor product of the matrices A 1 , A 2 , . . . ,
A q i.e. min x ∈ R d 1 ...dq ∥ ( A 1 ⊗ . . .
⊗ A q ) x − b ∥ 2 .
At each time step, one matrix A i receives a sparse change, and the goal is to maintain a sketch of the tensor product A 1 ⊗ . . .
⊗ A q so that the regression solution can be updated quickly.
Recomputing the solution from scratch for each round is very slow and so it is important to develop algorithms which can quickly update the solution with the new design matrix.
Our main result is a dynamic tree data structure where any update to a single matrix can be propagated quickly throughout the tree.
We show that our data structure can be used to solve dynamic versions of not only Tensor Product Regression, but also Tensor Product Spline regression (which is a generalization of ridge regression) and for maintaining Low
Rank Approximations for the tensor product.
Introduction
The task of fitting data points to a line is well-known as the least-squares regression problem [Sti81] which has a wide range of applications in signal processing [RGlY78], convex optimization [Bub15], network routing [Mad13, Mad16] and training neural networks [BPSW21, SZZ21].
In this work, we study a generalized version of least-squares regression where the design matrix
A is a tensor product of q smaller matrices A 1 ⊗ A 2 ⊗ . . .
⊗ A q .
Tensor products have been extensively studied in mathematics and the physical sciences since their introduction more than a century ago by Whitehead and Russell in their Principia Mathematica [WR12].
They have been shown to have a humongous number of applications in several areas of mathematics like applied linear algebra and statistics [VL00].
In particular, machine learning applications include image processing [NK06], multivariate data fitting [GVL13], and natural language processing [PSA21].
Furthermore, regression problems involving tensor products arise in surface fitting and multidimensional density smoothing [EM06], and structured blind deconvolution problems [OY05], among several other applications, as discussed in [DSSW18, DJS + 19].
Solving tensor product regression in ℓ 2 norm [DJS + 19, FFG22] to a (1 ± ε ) precision takes time (cid:101) O
( (cid:80) qi
=1 nnz( A i ) + poly( dq/ ( δε ))) , where each matrix A i ∈ R n i × d i and d = d 1 d 2 . . .
d q , and nnz( A ) denotes the number of nonzero entries of A .
However, these algorithms are inherently static , meaning that even if there is a sparse (or low-rank) update to a single design matrix A i , it needs to completely recompute the new solution from scratch.
In modern machine learning applications, such static algorithms are not practical due to the fact that data points are always evolving and recomputing the solution from scratch every time is computationally too expensive.
Hence, it is important to develop efficient dynamic algorithms that can adaptively update the solution.
For example, graphs for real-world network data are modeled as the tensor product of a large number of smaller graphs [LCK + 10].
Many important problems on these graphs can be solved by regression of the adjacency and Laplacian matrices of these graphs [ST04].
Real-world network data is always time-evolving and so it is crucial to develop dynamic algorithms for solving regression problems where the design matrix is a tensor product of a large number of smaller matrices.
Hence, we ask the following question:
Can we design a dynamic data structure for tensor product regression that can handle updates to the design matrix and quickly updates the solution?
We provide a positive answer to the above question.
At the heart of our data structure is a binary tree that can maintain a succinct sketch of A 1 ⊗ . . .
⊗ A q and supports an update to one of the matrices quickly.
Such tree structures have been useful in designing efficient sketching algorithms for tensor products of vectors [AKK + 20, SWYZ21].
However, the goal of these works has been in reducing the sketching dimension from an exponential dependence on q to a polynomial dependence on q .
Their results can be directly generalized to solving tensor product regression statically but not dynamically.
In this work, we build upon the tree structure to solve the Dynamic Tensor Product Regression problem and other related dynamic problems like Dynamic Tensor Spline Regression and Dynamic Tensor Low Rank Approximation.
Our key observation is that updating the entire tree is efficient when a single leaf of the tree gets an update: specifically, we only need to update the nodes which fall on the path from the leaf to the root.
Technical Contributions
• We design a dynamic tree data structure D YNAMIC T ENSOR T REE that maintains a succinct representation of the tensor product
A 1 ⊗ . . .
⊗ A q and supports efficient updates to any of the A i ’s. •
Consequently, we develop a dynamic algorithm for solving Tensor Product Regression, when one of the matrices A i ’s is updated
and we need to output an estimate of the new solution to the regression problem quickly.
• We also show that we can use our D YNAMIC T ENSOR T REE data-structure in a fast dynamic algorithm to solve the Tensor Product Spline Regression problem, which is a generalization of the classic Ridge Regression problem.
• We also initiate the study of Dynamic Tensor Low Rank Approximation , where the goal is to maintain a low rank approximation (LRA) of the tensor product with dynamic updates, and show how we can use our D YNAMIC T ENSOR T REE data structure to solve this problem.
Roadmap.
In section 2, we first discuss some related work.
Then, we provide notation, some background for our work, and the main problem formulation in section 3.
In section 4, we provide a technical overview of our paper.
Following that, we provide our dynamic tree data structure in section 5.
We then show how it can be used for Dynamic Tensor Product Regression, Dynamic
Tensor Spline Regression, and Dynamic Tensor Low Rank Approximation in section 6.
We end with our conclusion and discuss some future directions in section 7.
Related Work
Sketching.
Sketching techniques have many applications in numerical linear algebra, such as linear regression, low-rank approximation [CW13, NN13, MM13, BW14, RSW16, SWZ17, HLW17,
ALS + 18, BBB + 19, IVWW19, SWZ19a, SWZ19b, MW21, WY22, CSTZ22], distributed problems [WZ16, BWZ16], reinforcement learning [WZD + 20, SSX21], projected gradient descent [XSS21], training over-parameterized neural networks [SYZ21, SZZ21], tensor decomposition [SWZ19c], clustering [EMZ21], cutting plane method [JLSW20], generative adversarial networks [XZZ18], recommender systems [RRS + 22], and linear programming [LSZ19, JSWZ21, SY21].
Dynamic Least-squares Regression.
A dynamic version of the ordinary least-squares regression problem (without a tensor product design matrix) was recently studied in [JPW22].
In their model, at each iteration, a new row is prepended to the design matrix A and a new coordinate is prepended to the vector b .
To efficiently maintain the necessary parts of the design matrix and the solution, they make use of a fast leverage score maintenance data structure by [CMP20] and achieve a running time of (cid:101)
O (nnz( A ( T ) )
+ poly( d, ε − 1 )) where T is the number of time steps.
There are several major differences between their work and ours.
Tensor/Kronecker Product Problems.
Many machine learning problems involve tensor/Kronecker product computations such as, regression [HLW17, DSSW18, DJS + 19], lowrank approximation [SWZ19c], fast kernel computation [SWYZ21], semi-definite programming [JKL + 20, HJS + 21], and training over-parameterized neural networks [BPSW21, SZZ21].
Apart from solving problems which directly involve tensor/Kronecker products, another line of research focuses on constructing polynomial kernels efficiently so that the computation of various kernel problems can be improved [AKK + 20, SWYZ21].
Preliminaries 3.1 Notation
For any natural number n , we use [ n ] to denote the set { 1 , 2 , . . . , n } .
We use E [ · ] to denote expectation of a random variable if it exists.
We use 1 [ · ] and Pr[ · ] to denote the indicator function and probability of an event respectively.
For any natural numbers a, b, c , we use T mat ( a, b, c ) to denote the time it takes to multiply two matrices of sizes a × b and b × c .
For a vector x , we use ∥ x ∥to denote its ℓ 2 norm.
For a matrix A , we use ∥ A ∥ F to denote its Frobenius norm.
For a matrix
A , we use A ⊤ to denote the transpose of A .
Given two matrices A ∈ R n 1 × d 1 and B ∈ R n 2 × d 2 , we use A ⊗ B to denote their tensor product (which is also referred to as Kronecker product), i.e.,
[Equation] 1 =
A i 1 ,j 1 · B i 2 ,j 2 .
For example, for 2 × 2 matrices A and B , (cid:20) aa 12 aa(cid:21) ⊗ B
= (cid:20) a 11 B a 12 B a 21 B a 22 B (cid:21) =
  a 11 ba 11 ba 12 ba 12 b 12 a 11 ba 11 ba 12 ba 12 b 22 a 21 ba 21 ba 22 ba
b 12 a 21 ba 21 ba 22 ba 22 b 
We use (cid:78) to denote tensor product of more than two matrices, i.e., (cid:78) qi
=1 A i
=
A 1 ⊗ A 2 ⊗· · ·⊗ A q .
For non-negative real numbers a and b , we use a = (1 ± ε ) b to denote (1 − ε ) · b ≤ a ≤ (1 + ε ) · b .
For a real matrix A , we use σ i ( A ) to denote its i -th largest singular value.
For any function f , we use (cid:101) O ( f ) to denote O ( f poly(log f )) .
3.2 Background
In this section, we provide definitions for some terms used throughout our paper.
Definition 3.1 (Statistical Dimension) .
For any λ ≥ 0 and any positive semidefinite matrix M ∈ R n × n , the λ -statistical dimension of M is
[Equation]
Definition 3.2 (Oblivious Subspace Embedding [Sar06, AKK + 20]) .
Let ε, δ ∈ (0 , 1) , µ > 0 , and d, n ≥ 1 be integers.
An ( ε, δ, µ, d, n ) -Oblivious Subspace Embedding (OSE) is a distribution over m × d matrices with the guarantee that for any fixed matrix A ∈ R d × n with sd λ ( A ⊤ A ) ≤ µ , we have for any vector x ∈ R n , with probability at least 1 − δ ,
[Equation]
Definition 3.3 (Approximate Matrix Product) .
Given ε, δ > 0 , we say that a distribution D over m × d matrices Π has the ( ε, δ ) -approximate matrix product property if for every C, D ∈ R d × n , with probability at least 1 − δ ,
[Equation]
Definition 3.4 (OSNAP matrix [NN13]) .
For every sparsity parameter s , target dimension m , and positive integer d , the OSNAP transform with sparsity s is defined as
[Equation]
for all r ∈ [ m ] , j ∈ [ d ] , where σ r,j ∈ {± 1 } are independent and uniform Rademacher random variables and δ r,j are Bernoulli random variables satisfying
• For every i ∈ [ d ] , (cid:80) r ∈ [ m ] δ r,i = s .
In other words, every column has exactly s nonzero entries.
• For all r ∈ [ m ] and all i ∈ [ d ] , E [ δ r,i ]
= s/m .
• The δ r,i ’s are negatively correlated: ∀ T ⊂ [ m ] × [ d ] , E
[ (cid:81)
( r,i ) ∈ T δ r,i ] ≤ (cid:81) ( r,i
) ∈ T E [ δ r,i ] =
( sm ) | T | .
Definition 3.5 (T ENSOR SRHT [AKK + 20]) .
We define the T ENSOR SRHT S :
R d × R d → R m as
S = 1 √ m P · ( HD 1 × HD 2 ) , where each row of P ∈ { 0 , 1 } m × d contains only one 1 at a random coordinate, one can view P as a sampling matrix.
H is a d × d Hadamard matrix, and D 1 , Dare two d × d independent diagonal matrices with diagonals that are each independently set to be a
Rademacher random variable (uniform in {− 1 , 1 } ).
For completeness, we define other sketching matrices such as C OUNT S KETCH [CCFC02],
SRHT [LDFU13] and T ENSOR S KETCH [Pag13] in Appendix A.
3.3 Problem Formulation
In this section, we define our task.
We start with the static version of tensor product regression.
Definition 3.6 (Static version) .
In the ℓ 2 Tensor Product Regression problem, we are given matrices
A 1 , A 2 , . . .
, A q where A i ∈ R n i ×
d i and b ∈ R n 1 n 2 ...n q .
Let us define n := n 1 n 2 . . .
n q and d :=
d 1 d 2 . . .
d q .
The goal is to output x ∈ R d such that we minimize the objective:
[Equation]
Our dynamic version of ℓ 2 Tensor Product Regression involves updating one of the matrices
A i with an update matrix B .
We formally define the problem as follows:
Definition 3.7 (Dynamic version) .
In the Dynamic ℓ 2 Tensor Product Regression problem, we are given matrices A 1 , A 2 , . . .
, A q where A i ∈ R n i ×
d i , a sequence { ( i 1 , B 1 ) , ( i 2 , B 2 ) , . . .
, ( i T , B T ) } with i t ∈
[ q ] , B t ∈ R n it ×
d it for all t ∈
[ T ] , and the goal is to design a data structure with the following procedures:
• I NITIALIZE :
The data structure initializes A 1 , . .
A q and maintains an approximation to (cid:78) qi
=1 A i .
• U PDATE : At time step t , given an index i t and an update matrix B t , the data-structure needs to update A i t ← A i t + B t , and maintain an approximation to A 1 ⊗ . . .
⊗ ( A i t + B t ) ⊗ . . .
⊗ A q . • Q UERY :
The data structure outputs a vector (cid:98) x ∈ R d such that
[Equation]
In addition to the standard regression problem, we also consider the spline regression (which is a generalization of ridge regression) and low rank approximation problems.
We will provide the definitions for these problems in their respective sections.
Technical Overview
The two main design considerations for our data structure are as follows: 1).
We want a data structure that stores a sketch of the tensor product, this has the advantage of having a smaller storage space and a faster time to form the tensor product. 2).
The update time to one of the matrices
A
i should be fast, ideally we want to remove the linear dependence on q .
With these two goals in mind, we introduce a dynamic tree data structure that satisfies both of them simultaneously, building on the sketch of [AKK + 20].
Specifically, we use each leaf of the tree to represent S
i A
i , a sketch of the base matrix that only has m rows instead of n rows, where m is proportional to the small dimension d .
Each internal node represents a sketch of the Tensor product of its two children.
To speed up computation, we adapt tensor-typed sketching matrices to avoid the need to explicitly form the Tensor product of the two children.
The root of the tree will store a sketch for the tensor product (cid:78) qi
=1 A i .
We point out that in order for this tree to work, we need to apply an independent sketching matrix to each tree node.
Since there are 2 q − 1 nodes in total, we will still have a dependence on q in the initialization phase (which will be dominated by the d term).
However, during the update phase, our tree has a clear advantage: if one of the leaves has been updated, we only need to propagate the change along the path to root.
Since the height of the tree is at most O (log q ) , we remove the linear dependence on q in the update phase.
This is the key machinery of obtaining a faster update time in our dynamic data structure.
Moreover, the correctness follows naturally from the guarantee of the tree: as shown in [AKK + 20], the tree itself is an OSE for matrices of proper size, hence, it also preserves the subspace after updating one of the leaves.
Figure 1 is an example tree for tensor product of 4 matrices.
S S
T T
S
T T
With this data structure, we provide efficient algorithms for dynamic Tensor product regression, dynamic Tensor Spline regression, and dynamic Tensor low rank approximation.
For all these problems, we do not explicitly maintain the solutions, rather, we use our tree to quickly update a sketch of the design matrix.
In the query phase (when we need to actually solve the regression problems), we use this sketch of updated design matrix to quickly.
Dynamic Tree Data Structure
In this section, we introduce our data structure, which contains I NITIALIZE and U PDATE procedures.
We’ll define task-specific Q UERY procedures later.
We defer proofs in this section to Appendix B.
We start with an outline of Algorithm 1.
Outline of Tree: For simplicity, let us assume that the number of matrices q is a power of 2 .
Also, let us assume that all the matrices are of the same dimension n 1 × d 1 .
The output is a matrix of dimension m × d where m = poly( q, d, ε − 2 ) .
First, we apply q independent base sketch matrices (such as C OUNT S KETCH (Def. A.1), SRHT (Def. A.3))
{ T i ∈ R m × n 1 , i ∈ [ q ] } to each of the matrices
A 1 , A 2 , . . .
, A q .
After this step, all the leaf nodes are of size m × d 1 .
Then, we have a tree of height log q .
Let us use ℓ = 0 → log q to represent the level of the tree nodes where ℓ = 0 represents the leaves and ℓ = log q represents the root node.
Let us use J k,ℓ ∈ R n × d 2 ℓ to denote the matrix stored at the node k, ℓ .
For leaf nodes, we choose base sketches
T k ∈ R m × nand compute J k, 0 =
T k A k .
At each internal node, we use a sketch for tensor-typed inputs (such as T ENSOR S KETCH (Def. A.2) or T ENSOR SRHT (Def. 3.5)), apply it to its two children. The sketching matrix S i ∈ R m × m 2 , and it can be applied to its two inputs fast, obtaining a running time of (cid:101) O ( m ) instead of O ( m 2 ) . We inductively compute the tree all the way up to the root for initialization.
Note that at the root, we end up with a matrix of size m × d , which is significantly smaller than the tensor product n × d .
The running time guarantees of Algorithm 1, which we will prove in Appendix B, are as follows:
1: data structure D YNAMIC T ENSOR T REE ▷ Theorem 5.
2: 3: members 4:
J k,ℓ ∈ R m × d 2 ℓ for 0 ≤ ℓ ≤ log q and 0 ≤ k ≤ q/ 2 ℓ −▷ ℓ is the level of the tree.
5: end members 6: 7: procedure I NITIALIZE
( A 1 ∈ R n 1 × d 1 , . . .
, A q ∈ R n q × d q , m ∈ N + , C base , T base ) 8: for ℓ =
→ log q do 9: for k =
→ q/ 2 ℓ − 1 do 10: if ℓ = 0 then 11:
Choose a C k ∈ R m × n k from C base .
▷ C base can be C OUNT S KETCH , SRHT or
OSNAP.
J k,ℓ ← C k
A k 13: else 14:
Choose a T k,ℓ ∈ R m × m 2 from T base .
▷ T base can be T ENSOR S KETCH or
T ENSOR SRHT.
J k,ℓ ← T k,ℓ ( J 2 k,ℓ − 1 ⊗ J 2 k
+1 ,ℓ − 1 )
▷ Sketch of tensor product of its children 16: end if 17: end for 18: end for 19: end procedure 20: 21: procedure U PDATE ( i ∈ [ q ] , B ∈ R n i × d i ) 22: for ℓ =
→ log( q ) do 23: if ℓ = 0 then 24:
J new i, 0 ← C i B 25:
J i, 0 ← J i, 0 + J new i, 0 26: else 27:
J new ⌈ i/ 2 ℓ ⌉ ,ℓ ← T k,ℓ ( J new ⌈ i/ 2 ℓ − 1 ⌉ ,ℓ − 1 ⊗ J ⌈ i/ 2 ℓ − 1 ⌉ +1 ,ℓ − 1 )
▷ Sibling is to the left also sometimes 28:
J ⌈ i/ 2 ℓ ⌉ ,ℓ ← J ⌈ i/ 2 ℓ ⌉ ,ℓ + J new ⌈ i/ 2 ℓ ⌉ ,ℓ
▷ Sketch of tensor product of its children 29: end if 30: end for 31: end procedure
Theorem 5.1 (Running time of our main result.
Informal version of Theorem B.1) .
There is an algorithm (Algorithm 1) that has the following procedures:
• I NITIALIZE
( A 1 ∈ R n 1 × d 1 , A 2 ∈ R n 2 × d 2 , . . .
, A q ∈ R n q × d q , m ∈ N + , C base , T base ):
Given matrices A 1 ∈ R n 1 × d 1 , A 2 ∈ R n 2 × d 2 , . . .
, A q ∈ R n q × d q , , a sketching dimension m , families of base sketches C base and T base , the data-structure pre-processes in time (cid:101) O
( (cid:80) qi
=1 nnz( A i ) + qmd ) .
• U PDATE ( i ∈ [ q ] , B ∈ R n i × d i ) :
Given a matrix B and an index i ∈
[ q ] , the data structure updates the approximation to A 1 ⊗ . . .
⊗ ( A i + B ) ⊗ . . .
⊗ A q in time (cid:101)
O (nnz( B ) + md ) .
Remark 5.2.
In our data structure, we only pay the linear q term in I NITIALIZATION (which will be dominated by the term d ), since we are maintaining a tree with 2 q − 1 nodes, each node corresponds to a matrix of m rows.
In the U PDATE phase, we shave off the linear dependence on q .
We also want to point out that the sketching dimension m is typically at least linear on q , however, it only depends on q , the small dimension d and the precision parameter ε , hence it is much efficient to use this sketch representation compared to the original tensor product.
Also, note that though we only design a data structure for q being a power of 2, it is not hard to generalize to q being an arbitrary positive integer via the technique introduced in [SWYZ21].
We only incur a log q factor by doing so.
The approximation guarantee for our dynamic tree, which follows from [AKK + 20] and for which we provide a brief sketch in Appendix B are as follows:
Theorem 5.3 (Approximation guarantee of our dynamic tree.
Informal version of Theorem B.1) .
Let Π q denote the sketching matrix generated by the dynamic tree, we then have that:
[Equation]
Choices of parameter m : An important parameter to be chosen is the sketching dimension m .
Intuitively, it should depend polynomially on ε − 1 , d, q .
We list them in the following table.
[Equation]
Init time
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Against an Adaptive Adversary:
A common issue in dynamic algorithm is the presence of adaptive adversary .
Consider a sequence of update pairs { ( i, B i ) }
Ti =1 in which the adversary can choose the pair ( i, B i ) based on the output of our data structure on ( i − 1 , B i − 1 ) .
Such adversary model naturally captures many applications, in which the data structure is used in an iterative process , and the input to the data structure depends on the output from last iteration.
Our data structure can be extended to handle adaptive adversary as follows.
We store all initial factor matrices A 1 , . . .
, A q .
During an update, suppose we are given a pair ( i, B i ) .
Instead of using the sketching matrices correspond to the i -th leaf during initialization, we instead choose fresh sketching matrices along the path from the leaf to the root.
Since the only randomness being used is the sketching matrices along the path, by using fresh randomness, our data structure works against adaptive adversary.
Such modification does not affect the update time too much: during each update, we need to apply the newly-generated base sketch C new i to both A i and B , incurring a time of (cid:101)
O
(nnz( A i )+nnz( B )) .
Along the path, the data structure repeatedly apply T new k,ℓ to its two children, which takes (cid:101) O
( md ) time.
The overall time is thus (cid:101)
[Equation]
Thus, in later applications of our data structure, we present the runtime without this modification.
Faster Dynamic Tensor Product Algorithms with Dynamic Tree
In this section, we instantiate the dynamic tree data structure introduced in section 5 for three prominent applications: dynamic tensor product regression, dynamic tensor spline regression and dynamic tensor low rank approximation.
6.1 Dynamic Tensor Product Regression
We present an algorithm (Algorithm. 2) for solving the dynamic Tensor product regression problem.
Our algorithm is to use the dynamic tree data structure to maintain the design matrix and update it efficiently.
Additionally, we maintain the label vector via Π q b , which is a sketch of b .
During the query phase, we output the solution using the sketched tensor product we maintained and the sketched label vector.
We have the following guarantees, the proof of which we defer to Appendix C:
Theorem 6.1 (Informal version of Theorem C.1) .
There is an algorithm (Algorithm 2) for dynamic
Tensor product regression problem with the following procedures:
• I NITIALIZE
( A 1 ∈ R n 1 × d 1 , A 2 ∈ R n 2 × d 2 , . . .
, A q ∈ R n q × d q , m ∈ N + , C base , T base , b ∈ R n 1 ...n q ): Given matrices A 1 ∈ R n 1 × d 1 , A 2 ∈ R n 2 × d 2 , . . .
, A q ∈ R n q × d q , , a sketching dimension m , families of base sketches C base , T base and a label vector b ∈ R n 1 ...n q , the data-structure pre-processes in time (cid:101) O
( (cid:80) qi
=1 nnz( A i ) + qmd + m · nnz( b )) .
• U PDATE ( i ∈ [ q ] , B ∈ R n i × d i ) :
Given a matrix B and an index i ∈
[ q ] , the data structure updates the approximation to A 1 ⊗ . . .
⊗ ( A i + B ) ⊗ . . .
⊗ A q in time (cid:101)
O (nnz( B ) + md ) .
• Q UERY :
Query outputs an approximate solution (cid:98) x to the Tensor product regression problem where ∥ (cid:78) qi
=1 A i (cid:98) x − b ∥ 2 = (1 ± ε ) ∥ (cid:78) qi
=
A i x ∗ − b ∥ 2 with probability at least 1 − δ in time O
( md ω − 1 + d ω ) where x ∗ is an optimal solution to the Tensor product regression problem i. [Equation]
[Equation]
To realize the input sparsity time initialize and update time, we can use the combination of OSNAP and T ENSOR SRHT, which gives a sketching dimension
[Equation]
Hence, the update time is (cid:101)
[Equation]
log(1 /δ )) .
Compared to the static algorithm [DJS + 19], we note that their algorithm is suitable to be modified into dynamic setting, with a running time of (cid:101)
O (nnz( A i ) + nnz( B ) + ε − 2 ( q + d ) d/δ )) , their algorithm cannot accommodate general dynamic setting due to the 1 /δ dependence in running time.
Suppose the length dynamic sequence is at least d , then one requires the failure probability to be at most 1 d for union bound.
In such scenario, the algorithm of [DJS + 19] is strictly worse.
[FFG22] improves the dependence on d , yielding a subquadratic update time in the form of (cid:101)
O
(nnz( A i )+nnz( B )+ ε − 2 q 2 d ωi + ε − 1 d 2 − θ /δ ) .
The 1 /δ dependence follows from the same reason as [DJS + 19]: the algorithm relies on leverage score sampling to solve regression, therefore requires approximate matrix product property.
However, leverage score only provides guarantees on first and second moment, so to achieve ( √ ε, δ ) - approximate matrix product, they need ε − 1 /δ rows.
This also makes their algorithm unsuitable for dynamic setting. 6.2 Dynamic Tensor Spline Regression
Spline regression models are a well studied class of regression models [MC01].
In particular,
Spline regression problems involving tensor products arise in the context of B-Splines and PSplines [EM96].
For a very brief but relevant introduction to Spline regression, we refer the reader to [DSSW18].
In this section, we first define the Spline regression problem as it pertains to our sketching application.
Algorithm 3, which shows how we can use our D YNAMIC T ENSOR T REE data structure to solve a dynamic version of the Spline regression problem is provided in Appendix D.
Definition 6.2 (Spline Regression) .
In the Spline regression problem, given a design matrix A ∈ R m × n , a target vector x ∈ R n , a regularization matrix L ∈ R p × d , and a non-negative penalty coefficient λ ∈ R , the goal is to optimize
[Equation]
Notice that the classic ridge regression problem is a special case of spline regression (when L = I ).
In the T ENSOR S PLINE R EGRESSION problem, we want to solve the Spline regression problem where the design matrix A can be decomposed as the tensor product of multiple smaller matrices i.e. A =
A 1 ⊗· · ·⊗ A q .
Our D YNAMIC T ENSOR T REE data structure can be used to solve a dynamic version of T ENSOR S PLINE R EGRESSION .
In our dynamic model, at each time step, the update is of the form i ∈ [ q ] , B ∈ R n i ×
d i , and the data structure has to update A i ←
A i + B .
Before introducing our main result, we define an important metric to measure the rank of a Spline regression:
Definition 6.3 (Statistical dimension of Splines) .
For matrix A ∈ R n × d and L ∈ R p × d with rank( L ) = p and rank (cid:18)(cid:20)
AL (cid:21)(cid:19) =
d .
Given the generalized singular value decomposition [GVL13] of ( A, L ) such that A
= U (cid:20)
[Equation]
(cid:21)
RQ ⊤ , L = V (cid:2) Ω [Equation] × ( n − p ) (cid:3) RQ ⊤ , where Σ = diag( σ 1 , . . .
, σ p ) , Ω = diag( µ 1 , . .
. , µ p ) ∈
R p × p are diagonal matrices.
Finally, let γ i
=
σ i /µ i for i ∈
[ p ] .
The statistical dimension of the Spline is defined as sd λ ( A, L ) =
(cid:80) pi
=1 1 / (1 + λ/γ 2 i )
+ d − p .
The statistical dimension of the Spline can be much smaller than d , and we will show that the sketching dimension depends on the statistical dimension instead of d .
Let x ∗ denote an optimal solution of the Spline regression problem, i.e.
[Equation]
and let OPT :
=
∥
Ax ∗ − b ∥
+ λ · ∥ Lx ∗ ∥ 22 .
We can compute x ∗ given A, b, L, λ by x ∗ =
( A ⊤ A + λL ⊤ L ) − 1 A ⊤ b .
Theorem 6.4 (Guarantees for DTSR EGRESSION .
Informal version of Theorem D.5) .
There exists an algorithm (Algorithm 3) with the following procedures:
• I NITIALIZE
( A 1 ∈ R n 1 × d 1 , . . .
, A q ∈ R n q × d q ) : Given matrices A 1 ∈ R n 1 × d 1 , . . .
, A q ∈ R n q × d q , the data structure pre-processes in time
[Equation]
• U PDATE ( i ∈ [ q ] , B ∈ R n i × d i ) :
Given an index i ∈ [ q ] and an update matrix B ∈ R n × d , the data structure updates in time (cid:101)
O (nnz( B ) + md ) .
• Q UERY : Let A denote the tensor product (cid:78) qi
=1 A i .
Query outputs a solution (cid:101) x ∈ R n with constant probability such the (cid:101) x is an ε approximate to the Tensor Spline Regression problem i.e.
[Equation]
where OPT = min x ∈ R n ∥ Ax − b ∥ 22 + λ ·∥ Lx ∥ 22 .
Query takes time md ( ω − 1)
+ pd ( ω − 1) + d ω .
We defer the proof of the above theorem to Appendix D.
The sketching dimension m here depends on the statistical dimension of the Spline, which is at most d .
To realize the above runtime, one can pick OSNAP and T ENSOR SHRT, which gives m = ε − 1 q sd 2 λ ( A, L ) log(1 /δ ) and an update time (cid:101) O
(nnz( B ) + ε − 1 q sd 2 λ ( A, L ) d log(1 /δ )) .
The improved dependence on ε − 1 comes from using approximate matrix product directly, instead of
OSE.
6.3 Dynamic Tensor Low Rank Approximation
In this section, we consider a problem studied in [DJS + 19]: given matrices A 1 , . . .
, A q , the goal is to compute a low rank approximation for the tensor product matrix A 1 ⊗ . . .
⊗ A q ∈ R n × d , specifically, a rank- k approximation B ∈ R n × d such that
[Equation]
where OPT k = min rank - k A ′ ∥
A ′ − A ∥ F and A = (cid:78) qi
=1 A i .
The [DJS + 19] algorithm works as follows: they pick q independent C OUNT S KETCH matrices [CCFC02] with O ( ε − 2 qk 2 ) rows, then apply each sketch matrix S i to A i .
After that, they form the tensor product M = (cid:78) qi
=1 S i A i and run SVD on M = U Σ V ⊤ .
Finally, they output B =
AU ⊤ k U k in factored form (as matrices
A 1 , . . .
, A q , U k ), where U k ∈ R k × d denotes the top k right singular vectors.
By doing so, they achieve an overall running time of
[Equation]
Two central issues around their algorithm are 1).
exponential dependence on parameters ε − 1 , q, k and 2).
the algorithm is inherently static, so a natural way to make it dynamic is to apply S i to the update B , reforming the tensor product of sketches and compute the SVD.
We address these two issues simultaneously by using our tree data structure.
For the sketching dimension, we observe it is enough to design a sketch that is ( ε, δ, k, d, n ) -OSE and ( ε/k, δ ) - approximate matrix product, therefore, we can set sketching dimension m = ε − 2 qk 2 log(1 /δ ) .
As we will show later, this gives a much improved running time.
By using the dynamic tree, we also provide a dynamic algorithm.
Theorem 6.5 (Guarantees for L OW R ANK M AINTENANCE .
Informal version of Theorem E.3) .
There exists an algorithm (Algorithm 4) that has the following procedures
• I NITIALIZE
( A 1 ∈ R n 1 × d 1 , . . .
, A q ∈ R n q × d q ) : Given matrices A 1 ∈ R n 1 × d , . . .
, A q ∈ R n q × d q , the data structure processes in time (cid:101) O ( q (cid:88) i = nnz
( A i ) + qmd ) .
• U PDATE ( i ∈ [ q ] , B ∈ R n i × d i ) :
Given an index i ∈ [ q ] and an update matrix B ∈ R n i × d i , the data structure updates the approximation for A 1 ⊗ . . .
⊗ ( A i + B ) ⊗ . . .
⊗ A q in time (cid:101)
O (nnz( B ) + md ) .
• Q UERY : Let A denote the tensor product (cid:78) qi
=1 A i .
The data structure outputs a rank- k approximation C such that
[Equation]
The time to output C is (cid:101) O
( md ω − 1 ) .
By choosing m = ε − 2 qk 2 log(1 /δ ) , we can provide good guarantees for low rank approximation.
This gives
• Initialization time in (cid:101) O
( (cid:80) qi
=1 nnz( A i ) + ε − 2 qdk 2 log(1 /δ )) ; • Update time in (cid:101) O
(nnz( B ) + ε − 2 qdk 2 log(1 /δ )) ; • Query time in (cid:101) O ( ε − 2 qd ω − 1 k 2 log(1 /δ )) .
Without using fast matrix multiplication, our algorithm improves upon the prior state-of-theart [DJS + 19] on all parameters even in the static setting.
To the best of our knowledge, our tensor product low rank approximation algorithm is the fastest, both in static and dynamic settings.
Conclusion & Future Directions
In this work, we initiate the study of the Dynamic Tensor Regression, Dynamic Tensor Spline Regression, and Dynamic Tensor Low-Rank Approximation problems, and design a tree data structure
D YNAMIC T ENSOR T REE which can be used to provide fast dynamic algorithms for these problems.
The problems we study are very interesting and there are a lot of exciting future directions, some of which we discuss below.
First, it is unclear whether the algorithms we proposed are optimal and so having lower bounds for the update times would be an important direction.
Second, we have focused exclusively on regression problems with the ℓ 2 norm whereas solving Dynamic Tensor Regression and Dynamic Tensor Spline Regression with respect to the more robust ℓ 1 -norm as well as for more general p -norms are also very appealing.
Acknowledgments
We would like to thank all the NeurIPS 2022 reviewers of our paper and also the ICMLreviewers who reviewed an earlier version of this work.
Most of this work was done while LZ was at CMU.
AR was supported in-part by NSF grants CCF-1652491, CCF-1934931, and CCFduring the preparation of this manuscript.
Appendix
Roadmap.
In this appendix, we provide omitted details and proofs for our main paper.
In particular, section A of our appendix corresponds to section 3 of our main paper, section B to section 5, section C to section 6.1, section D to section 6.2, and section E to section 6.3.
A Preliminaries: Other Sketching Matrices
Here, we will define some sketches which we use throughout our paper:
Subsampled Randomized
Hadamard Transform (SRHT) [LDFU13], Count Sketch (C OUNT S KETCH )
[CCFC02] and Tensor
Count Sketch (T ENSOR S KETCH )
[Pag13, PP13].
Next, we introduce C OUNT S KETCH and T ENSOR S KETCH , which are important sketching matrices we will be using across our paper.
Definition A.1 (C OUNT S KETCH matrix [CCFC02]) .
Let h : [ n ] → [ b ] be a random 2 -wise independent hash function and σ : [ n ] → {− 1 , +1 } be a random 4 -wise independent hash function.
Then R ∈ R b × n is a C OUNT S KETCH matrix if we set R h ( i ) ,i =
σ ( i ) for all i ∈ [ n ] and all the other entries of R to zero.
Definition A.2 (T ENSOR S KETCH matrix [Pag13, PP13]) .
Let h 1 , h 2 : [ m ] → [ s ] be 3-wise independent hash functions, also let σ : [ m ] → {± 1 } be a 4-wise independent random sign function.
The degree two T ENSOR S KETCH transform, S : R m × R m → R s is defined as follows: for any i, j ∈ [ m ] and r ∈ [ s ] ,
[Equation]
+
[Equation]
= r mod s ] .
Definition A.3 (Subsampled Randomized Hadamard Transform (SRHT) matrix [LDFU13]) .
We say
R ∈ R b
× n is a subsampled randomized Hadamard transform matrix 4 if it is of the form
R = (cid:112) n/b · SHD , where S ∈ R b × n is a random matrix whose rows are b uniform samples (without replacement) from the standard basis of R n , H ∈ R n × n is a normalized Walsh-Hadamard matrix, and D ∈ R n × n is a diagonal matrix whose diagonal elements are i.i.d.
Rademacher random variables.
B Proofs for Dynamic Tree Data Structure (section 5)
In this section, we provide proofs for the running time and approximation guarantees of our dynamic tree data structure (Algorithm 1).
B.1 Running Time of Algorithm
Theorem B.1 (Formal version of Theorem 5.1) .
There is an algorithm (Algorithm 1) that has the following procedures:
• I NITIALIZE
( A 1 ∈ R n 1 × d 1 , A 2 ∈ R n 2 × d 2 , . . .
, A q ∈ R n q × d q , m ∈ N + , C base , T base ):
Given matrices A 1 ∈ R n 1 × d 1 , A 2 ∈ R n 2 × d 2 , . . .
, A q ∈ R n q × d q , a sketching dimension m , families of base sketches C base and T base , the data-structure pre-processes in time (cid:101) O
( (cid:80) qi
=1 nnz( A i ) + qmd ) where d = d 1 d 2 . . .
d q . • U PDATE ( i ∈ [ q ] , B ∈ R n i × d i ) :
Given a matrix B and an index i ∈
[ q ] , the data structure updates the approximation to A 1 ⊗ . . .
⊗ ( A i + B ) ⊗ . . .
⊗ A q in time (cid:101)
O (nnz( B ) + md ) .
Note: We will assume that n 1 = n 2 = · · · = n q and d 1 = d 2 = · · · = d q for the following computations.
To modify our D YNAMIC T ENSOR T REE to work for the non-uniform case, we only need to choose the base sketches at the leaves of the appropriate dimension and also have different dimensions for the internal nodes.
Proof.
Proof of I NITIALIZATION .
The initialization part has 2 key steps - applying the T base sketches to the leaves and applying S base sketches to each of the internal nodes.
• Applying a C OUNT S KETCH matrix C
i : R m × n i to any matrix
A i : R n i ×
d i takes input sparsity time
O (nnz( A i )) .
Therefore, the total time for this step is O ( (cid:80) qi
=1 nnz( A i )) .
If OSNAP is used, this is true up to polylogarithmic factors.
• For all internal nodes, let us consider the time of computing one node at level ℓ ∈ [log 2 q ] using T ENSOR SRHT. Consider the computation of J k,ℓ ∈ R m × d 2 ℓ 1 , we pick a T ENSOR SRHT T
k,ℓ ∈ R m × m 2 and apply it to J 2 k,ℓ − 1 ⊗ J 2 k +1 ,ℓ − 1 .
This can be viewed as computing the [-time for this step is O (2 md 3 · 2 ℓ − 
 + 2 md 3 ·-]tensor product of column vectors of J 2 k,ℓ − 1 and J 2 k +1 ,ℓ − 1 : each column of J k,ℓ is a tensor product of two columns of its two children after applying T k,ℓ .
Each column takes time O ( m log m ) and the total number of columns of J k,ℓ is d 2 ℓ 1 .
Hence, the time of forming J k,ℓ is O ( md 2 ℓ 1 log m ) .
• For each level, there are q/ 2 ℓ nodes, so O ( q/ 2 ℓ md 2 ℓ 1 ) time in total for level ℓ . Sum over all levels for ℓ ∈ [log q ] , we have
[Equation]
• Thus, the total time for initialization is
[Equation]
Proof of U PDATE .
We note that U PDATE can be decomposed into the following parts:
• Apply C OUNT S KETCH C
i to the update matrix B , which takes O (nnz( B )) time.
If OSNAP or SRHT are used, then this is true up to polylogarithmic factors.
• Updating the leaf node takes O ( md i ) time because C i B ∈ R m × d i .
• For each level ℓ ∈ [log 2 q ] , we first compute the update tree by applying T ENSOR S KETCH T
k,ℓ ∈ R m × m 2 to J 2 k,ℓ − 1 ⊗ J 2 k +1 ,ℓ − 1 which takes time O ( md 2 ℓ 1 log m ) .
• We also need to update one internal node for each level, which takes (cid:101) O
( md 2 ℓ 1 ) time.
This is at most O ( md log m ) for the root.
Since there are log q levels, the total time for updating all internal nodes is also (cid:101) O
( md ) .
Hence, the total runtime of U PDATE is
[Equation]
B.2 Approximation Guarantee of Algorithm
In this section, we provide the approximation guarantee of Algorithm 1.
Theorem B.2.
Let A 1 ∈ R n 1 × d 1 , . . .
, A q ∈ R n q × d q be the input matrices, m ∈ N + be the sketching dimension, let C base , T base be appropriate family of sketching matrices.
Let Π q be the sketching matrix generated by Algorithm 1.
Then for any x ∈ R d , we have that with probability at least 1 − δ ,
∥ Π q q (cid:79) i = A i x ∥ 2 = (1 ± ε ) ∥ q (cid:79) i =
A i x ∥ 2 , if
• C base is C OUNT S KETCH , T base is T ENSOR S KETCH and m
= Ω( ε − 2 qd 2 1 /δ ) .
• C base is OSNAP , T base is T ENSOR SRHT and m = (cid:101) Ω( ε − 2 q 4 d log(1 /δ )) .
• C base is OSNAP , T base is T ENSOR SRHT and m = (cid:101) Ω( ε − 2 qd 2 log(1 /δ )) .
First we will discuss the theorems of [AKK + 20] which lead to our approximation guarantees and then discuss the specifics for each choice of base sketches seperately.
Lemma B.3 (Theorem 1 of [AKK + 20]) .
For every positive integers n, q, d , every ε, sd λ > 0 , there exists a distribution on linear sketches Π q ∈ R m ×
d q such that: 1).
If m = Ω( ε − 2 q sd 2 λ ) , then Π q is an ( ε, 1 / 10 , sd λ , d q , n ) -OSE (Def. 3.2).
If m = Ω( ε − 2 q ) , then Π q has the ( ε, 1 / 10) -approximate matrix product (Def. 3.3).
Lemma B.4 (Theorem 2 of [AKK + 20]) .
For every positive integers n, q, d , every ε, sd λ > 0 , there exists a distribution on linear sketches Π q ∈ R m ×
d q such that: 1).
If m = (cid:101) Ω( ε − 2 q sd 2 λ ) , then Π q is an ( ε, 1 / poly( n ) , sd λ , d q , n ) -OSE (Def. 3.2).
If m = (cid:101) Ω( ε − 2 q ) , then Π q has the ( ε, 1 / poly( n )) - approximate matrix product (Def. 3.3).
Lemma B.5 (Theorem 3 of [AKK + 20]) .
For every positive integers n, q, d , every ε, sd λ > 0 , there exists a distribution on linear sketches Π q ∈ R m × d q which is an ( ε, 1 / poly( n ) , sd λ , d q , n ) -OSE (Def. 3.2), provided that the integer m satisfies m = (cid:101) Ω( q 4 sd λ /ε 2 )
We note that the 3 results due to [AKK + 20] are actually stronger than what we need here, since the statements say that they are subspace embedding for all matrices of proper size.
Also, when the statistical dimension sd λ is smaller than the rank of the design matrix, the sketching dimension is even smaller, which is important in our Spline regression application.
We then proceed to prove the individual guarantees for different choice of base sketches item by item.
• By Lemma B.3, we know that in order for Π q to be an OSE, we will require m = Ω( ε − 2 qd 2 1 /δ ) .
Note that the statistical dimension sd λ is d in this case.
We point out that the dependence on success probability is 1 /δ , this means that we can not obtain a high probability version.
This is the main reason that Lemma B.3 only provides an OSE with constant probability.
• By Lemma B.5, we need m = (cid:101) Ω( ε − 2 q 4 d log(1 /δ )) .
We get a nearly linear dependence on d in the expense of worse dependence on q .
• By Lemma B.4, we will require m = (cid:101) Ω( ε − 2 qd 2 log(1 /δ )) .
In this scenario, the dependence on δ is log(1 /δ ) , hence we can aim for a high probability guarantee for our OSE.
C Proofs for Dynamic Tensor Product Regression (section 6.1)
Theorem C.1 (Formal version of Theorem 6.1) .
There is a dynamic data structure for dynamic
Tensor product regression problem with the following procedures:
• I NITIALIZE
( A 1 ∈ R n 1 × d 1 , A 2 ∈ R n 2 × d 2 , . . .
, A q ∈ R n q × d q , m ∈ N + , C base , T base , b ∈ R n 1 ...n q ): Given matrices A 1 ∈ R n 1 × d 1 , A 2 ∈ R n 2 × d 2 , . . .
, A q ∈ R n q × d q , , a sketching dimension m , families of base sketches C base , T base and a label vector b ∈ R n 1 ...n q , the data-structure pre-processes in time (cid:101) O
( (cid:80) qi
=1 nnz( A i ) + qmd + m · nnz( b )) .
• U PDATE ( i ∈ [ q ] , B ∈ R n i × d i ) :
Given a matrix B and an index i ∈
[ q ] , the data structure updates the approximation to A 1 ⊗ . . .
⊗ ( A i + B ) ⊗ . . .
⊗ A q in time (cid:101)
O (nnz( B ) + md ) .
• Q UERY :
Query outputs an approximate solution (cid:98) x to the Tensor product regression problem where ∥ (cid:78) qi
=1 A i (cid:98) x − b ∥ 2 = (1 ± ε ) ∥ (cid:78) qi
=
A i x ∗ − b ∥ 2 with probability at least 1 − δ in time O
( md ω − 1 + d ω ) where x ∗ is an optimal solution to the Tensor product regression problem i.e. x ∗ = arg min x ∈ R d ∥ (cid:78) qi
=
A i x − b ∥ 2 .
Proof.
Proof of I NITIALIZATION .
The initialization part has 3 steps:
1: data structure DTR EGRESSION 2: 3: members 4:
D YNAMIC T ENSOR T REE DTT 5:
A 1 ∈ R n 1 × d 1 , . . .
, A q ∈ R n q × d q 6: (cid:101) b ∈ R m 7: end members 8: 9: procedure I NITIALIZE
( A 1 ∈ R n 1 × d 1 , . . .
, A q ∈ R n q × d q , m, C base , T base , b ∈ R n ) 10:
DTT.I NITIALIZE ( A 1 , . . .
, A q , m, C base , T base ) 11:
A 1 ← A 1 , . . .
, A q ← A q 12:
Let Π q ∈ R m × n be the sketching matrix corresponds to DTT 13: (cid:101) b ← Π q b 14: end procedure 15: 16: procedure Q UERY () 17:
M ← DTT.
J 0 ,18: (cid:101) x ← arg min x ∈ R n ∥ Mx − (cid:101) b ∥19: return x 20: end procedure
• Initializing DTT takes time (cid:101) O
( (cid:80) qi
=1 nnz( A i ) + qmd ) from Theorem B.1.
• Initializing all the matrices
A i , i ∈ [ q ] takes time (cid:101) O
( (cid:80) qi
=1 nnz( A i )) .
• Computing (cid:98) b takes time (cid:101) O ( m · nnz( b )) .
Therefore, the total time for initialization is (cid:101) O
( q (cid:88)
i = nnz
( A i ) + qmd + q (cid:88)
i = nnz
( A i ) + m · nnz( b ))
= (cid:101) O
( q (cid:88)
i = nnz
( A i ) + qmd + m · nnz( b ))
Proof of U PDATE .
We just update our DTT data structure.
This takes time (cid:101)
O (nnz( B )
+ md 3 / 2 ) from Theorem B.1.
Proof of Q UERY time .
We can view the Query part as having the following 5 steps:
• Initializing M takes time (cid:101) O
( md ) since M ∈ R m × d . • M ⊤ ( (cid:101) b ) , this takes O ( md ) time because we are multiplying a d × m matrix with a m ×vector.
• M ⊤ · M , this step takes T mat ( d, m, d ) time as we are multiplying an d × m matrix with an m × d matrix.
• ( M ⊤ M ) − 1 this step takes T mat ( d, d, d ) time as it is inverting M ⊤ M which is of size d × d . • ( M ⊤ M ) − 1 ( M ⊤ (cid:101) b ) .
In this step, we are multiplying ( M ⊤ M ) − 1 which is a d × d matrix with a vector M ⊤ (cid:101) b of size d .
This takes time O
( d 2 ) .
Overall, the total running time of Q UERY is
[Equation]
[Equation]
Proof of Correctness for Q UERY .
The proof of correctness for Query follows from Theorem B. identically to the proof of Theorem 3.1 in [DSSW18].
D Omitted details and proofs for Dynamic Tensor Spline Regression (section
To specify the sketching dimension m needed , we use the following definition of Statistical Dimension for Splines [DSSW18]:
Definition D.1 (Statistical Dimension for Splines) .
For the Spline regression problem specified by
Eq. 1, the statistical dimension is defined as sd λ ( A, L ) =
(cid:80) i 1 / (1+ λ/γ 2 i )
+ d − p .
Here { γ i , i ∈
[ p ] } are the generalized singular values of ( A, L ) which are defined as follows.
Definition D.2 (Generalized singular values) .
For matrices A ∈ R n × d and L ∈ R p ×
d such that rank( L ) = p and rank (cid:18)(cid:20)
A L (cid:21)(cid:19) =
d , the generalized singular value decomposition (GSVD) [GVL13] of ( A, L ) is given by the pair of factorizations A =
U (cid:20) Σ 0 p × ( d − p )( d
[Equation]
p
[Equation]
p I d
[Equation] p (cid:21)
RQ ⊤ and L = V (cid:2) Ω 0 p × ( d − p ) (cid:3) RQ ⊤ , where U ∈ R n × d has orthonormal columns, V ∈ R p × p ,
Q ∈ R d × d are orthogonal,
R ∈ R d × d is upper triangular and nonsingular, and Σ and Ω are p ×
p diagonal matrices: Σ = diag( σ 1 , σ 2 , . . .
, σ p ) and Ω = diag( µ 1 , µ 2 , . . . , µ p ) with 0 ≤ σ 1 ≤ σ 2 ≤ . . .
≤ σ p < 1 and 1 ≥ µ 1 ≥ µ 2 ≥ . .
≥ µ p > 0 , satisfying Σ ⊤ Σ+Ω ⊤ Ω
= I p .
The generalized singular values γ i of ( A, L ) are defined by the ratios γ i =
σ i /µ i for i ∈
[ p ] .
We prove that for the spline regression, our sketching matrix can have even smaller dimension.
The following is due to [DSSW18].
Lemma D.3.
Let x ∗ =
arg min x ∈ R d ∥
Ax − b ∥ 22 + λ ∥ Lx ∥ 22 , A ∈ R n × d and b ∈ R n .
Let U 1 ∈ R n × d be the first n rows of an orthogonal basis for (cid:20) A √ λL (cid:21) ∈ R ( n + p ) ×
d .
Let sketching matrix S ∈ R m × n have a distribution such that with probability 1 − δ ,
• ∥
U ⊤ 1 S ⊤ SU 1 − U ⊤
1 U 1 ∥ 2 ≤ 1 / 4 , • ∥
U ⊤ 1 ( S ⊤ S − I )( b − Ax ∗ ) ∥ 2 ≤ (cid:112) ε OPT / 2 .
Let (cid:101) x denote arg min x ∈ R d ∥ S
( Ax − b ) ∥ 22 + λ ∥ Lx ∥ 22 .
Then with probability at least 1 − δ ,
[Equation]
Lemma D.4.
For U 1 as in Lemma D.3, we have
[Equation]
We are now ready to prove the main theorem.
Theorem D.5 (Formal version of Theorem 6.4) .
There exists an algorithm (Algorithm 3) with the following procedures:
• I NITIALIZE
( A 1 ∈ R n 1 × d 1 , . . .
, A q ∈ R n q × d q ) : Given matrices A 1 ∈ R n 1 × d 1 , . . .
, A q ∈ R n q × d q , the data structure pre-processes in time (cid:101) O
( (cid:80) qi
=1 nnz( A i ) + qmd + m · nnz( b )) .
• U PDATE ( i ∈ [ q ] , B ∈ R n i × d i ) :
Given an index i ∈ [ q ] and an update matrix B ∈ R n × d , the data structure updates in time (cid:101)
O (nnz( B ) + md ) .
• Q UERY : Let A denote the Tensor product (cid:78) qi
=1 A i . Query outputs a solution (cid:101) x ∈ R n with constant probability such the (cid:101) x is an ε approximate to the Tensor Spline Regression problem i.e. ∥ A (cid:101) x − b ∥ 22 + λ ·∥ L (cid:101) x ∥ 22 ≤ (1+ ε ) · OPT where OPT = min x ∈ R n ∥ Ax − b ∥ 22 + λ ·∥ Lx ∥ 22 .
Query takes time (cid:101) O ( md ( ω − 1) + pd ( ω − 1) + d ω ) .
1: data structure DTSR EGRESSION 2: 3: members 4:
D YNAMIC T ENSOR T REE DTT 5:
A 1 ∈ R n 1 × d 1 , . . .
, A q ∈ R n q × d q 6: (cid:101) b ∈ R m 7: end members 8: 9: procedure I NITIALIZE
( A 1 ∈ R n 1 × d 1 , . . .
, A q ∈ R n q × d q , m, C base , T base , b ∈ R n ) 10:
DTT.I NITIALIZE ( A 1 , . . .
, A q , m, C base , T base ) 11:
A 1 ← A 1 , . . .
, A q ← A q 12:
Let Π q ∈ R m × n be the sketching matrix corresponds to DTT 13: (cid:101) b ← Π q b 14: end procedure 15: 16: procedure Q UERY () 17:
M ← DTT.
J 0 ,18: (cid:101) x ← arg min x ∈ R n ∥ Mx − (cid:101) b ∥ 22 + ∥ Lx ∥19: return x 20: end procedure
Moreover, we have that if
• C base is C OUNT S KETCH , T base is T ENSOR S KETCH , then m = Ω( ε − 1 q sd 2 λ ( A, L )1 /δ ) .
• C base is OSNAP , T base is T ENSOR SRHT , then m = (cid:101) Ω( ε − 1 q 4 sd λ
( A, L ) log(1 /δ )) .
Proof.
Proof of Correctness for Q UERY .
To prove that it is enough to use sketching whose dimension depends on sd λ ( A, L ) , we just need to show that the sketching matrix related to the dynamic tree satisfies the two guarantees of Lemma D.3.
We will use approximate matrix product (Def. 3.3) to prove this guarantee.
By Lemma D.4, we have that ∥ U 1 ∥ 2 F = sd λ
( A, L ) , and consider an ( √ ε/ sd λ ( A, L ) , δ ) -approximate matrix product sketching, we have that
[Equation]
setting ε ≤ 1 / 2 , we obtain desired result.
For the second part, we set C
=
U 1 and D =
b − Ax ∗ , then
[Equation]
This means that to obtain good approximation for Spline regression, it is enough to employ (
√ ε/ sd λ
( A, L ) , δ ) -approximate matrix product.
The dimension follows from Lemma B.3 and B.4.
We note that even though the definition of statistical dimension in [AKK + 20] and ours is different, our definition is more general and the [AKK + 20] definition can be viewed as picking L ⊤ L to be the identity matrix.
Hence, one can easily generalize the conclusion of Lemma B.3, B.4 and B.5 to our definition of statistical dimension via the argument used in [ACW17, DSSW18, AKK + 20].
Proof of I NITIALIZATION .
The initialization part of Algorithm 3 is identical to the initialization part of Algorithm 2.
Therefore, the time for initialization is (cid:101) O
( (cid:80) qi
=1 nnz( A i ) + qmd + m · nnz( b )) from Theorem C.1.
Proof of U PDATE .
We just update our DTT data structure.
This takes time (cid:101)
O (nnz( B ) + md ) from Theorem B.1.
Proof of Q UERY time .
As mentioned in Section 6.2, we can compute (cid:101) x ← arg min x ∈ R n ∥ Mx − (cid:101)
b
∥ 22 + ∥ Lx ∥ 22 by (cid:101) x ← ( M ⊤ M + λL ⊤ L ) − 1 M ⊤ (cid:101) b .
We can view Query as consisting of the following 5 steps:
• Initializing M takes time (cid:101) O
( md ) since M ∈ R m × d . • M ⊤ ( (cid:101) b ) , this takes O ( md ) time because we are multiplying a d × m matrix with a m ×vector.
• M ⊤ · M , this step takes T mat ( d, m, d ) time as we are multiplying a d × m matrix with an m × d matrix.
• L ⊤ L , this step takes T mat ( d, p, d ) time as we are multiplying a d × p matrix with a p × d matrix.
• M ⊤ M + λL ⊤ L , this takes O ( d 2 ) time as we are adding two d × d size matrices.
• ( M ⊤ M + λL ⊤ L ) − 1 , this step takes T mat ( d, d, d ) time as it is inverting a matrix of size d × d . • ( M ⊤ M + λL ⊤ L ) − 1 ( M ⊤ (cid:101) b ) .
In this step, we are multiplying ( M ⊤ M + λL ⊤ L ) − 1 which is a d × d matrix with a vector M ⊤ (cid:101) b of size d .
This takes time O
( d 2 ) .
Overall, the total running time of Q UERY is
[Equation]
E Omitted details and proofs for Dynamic Tensor Low Rank Approximation (section 6.3)
For low rank approximation, we show it is enough to design sketching matrix whose dimension depends on k instead of d .
Definition E.1 (Projection Cost Preserving Sketch) .
Let A ∈ R n × d , we say a sketching matrix
S ∈ R m × n is a k Projection Cost Preserving Sketch (PCPSketch if for any orthogonal projection
P ∈ R d × d of rank k , we have
[Equation]
where c ≥ 0 is some fixed constant independent of P , but may depend on A and SA .
For low rank approximation, we note that the optimal low rank approximation for A is projecting onto the top k left singular vectors.
Hence, if we can prove that the sketching corresponds to the dynamic tree is a k -PCPSketch, then we are done.
We will require a few technical tools.
Lemma E.2 (Theorem 12 of [CEM + 15]) .
Let A ∈ R n × d .
Suppose S ∈ R m × n is an ( ε, δ, k, d, n ) -
OSE and ( ε/ √ k, δ ) -approximate matrix product, then S is a k -PCPSketch for A . Theorem E.3 (Formal version of Theorem 6.5) . There exists an algorithm (Algorithm 4) that has the following procedures
1: data structure L OW R ANK M AINTENANCE 2: 3: members 4:
D YNAMIC T ENSOR T REE DTT 5:
A 1 ∈ R n 1 × d 1 , . . .
, A q ∈ R n q × d q 6: end members 7: 8: procedure I NITIALIZE
( A 1 ∈ R n 1 × d 1 , . . .
, A q ∈ R n q × d q ) 9:
DTT.I NITIALIZE ( A 1 , . . .
, A q ) 10:
A 1 ← A 1 , . . .
, A q ← A q 11: end procedure 12: 13: procedure U PDATE ( i ∈ [ q ] , B ∈ R n i × d i ) 14:
DTT.U PDATE ( i, B ) 15: end procedure 16: 17: procedure Q UERY 18:
M ← DTT.
J 0 ,19:
Compute SVD of M such that M = U Σ V ⊤ 20:
U k ← top k right singular vectors of M 21: return A 1 ⊗ . . . ⊗ A q U
⊤ k U k in factored form 22: end procedure
• I NITIALIZE
( A 1 ∈ R n 1 × d 1 , . . .
, A q ∈ R n q × d q ) : Given matrices A 1 ∈ R n × d , . . .
, A q ∈ R n × d , the data structure processes in time (cid:101) O ( (cid:80) qi
=1 nnz( A i ) + qmd + m · nnz( b )) .
• U PDATE ( i ∈ [ q ] , B ∈ R n i × d i ) :
Given an index i ∈ [ q ] and an update matrix B ∈ R n × d , the data structure updates the approximation for A 1 ⊗ . . .
⊗ ( A i + B ) ⊗ . . .
⊗ A q in time (cid:101)
O (nnz( B ) + md )) .
• Q UERY : Let A denote the tensor product (cid:78) qi
=1 A i .
The data structure outputs a rank- k approximation C such that
[Equation]
The time to output C is (cid:101) O
( md ω −
Moreover, we have that if
• C base is C OUNT S KETCH , T base is T ENSOR S KETCH and m
= Ω( ε − 2 qk 2 1 /δ ) .
• C base is OSNAP , T base is T ENSOR SRHT and m = (cid:101) Ω( ε − 2 qk 2 log(1 /δ )) .
Proof.
Proof of Correctness for Q UERY .
We first show that if we have a k -PCPSketch for A , then we can compute a matrix C that is a good approximation to rank- k low rank approximation.
Recall that for any projection matrix, we have
[Equation]
we need to design the projection P .
Let SA =
U Σ V ⊤ be the singular value decomposition of SA , and let
U k ∈ R k × d be the top k left singular vectors of SA .
Set P =
U ⊤ k U k , we note that SAU ⊤ k
U k is the optimal rank- k approximation for SA .
On the other hand, let Q ∈ R
d
×
d be the projection such that AQ is the optimal rank- k approximation for A .
We have
[Equation]
By Lemma E.2, to obtain a k -PCPSketch, we need ( ε, δ, k, d, n ) -OSE and ( ε/ √ k, δ ) -approximate matrix product, choosing dimension according to Lemma B.3 and B.4 gives the desired result.
Proof of I NITIALIZATION .
The initialization part of Algorithm 3 is identical to the initialization part of Algorithm 2.
Therefore, the time for initialization is (cid:101) O
( (cid:80) qi
=1 nnz( A i ) + qmd + m · nnz( b )) from Theorem C.1.
Proof of U PDATE .
We just update our DTT data structure.
This takes time (cid:101)
O (nnz( B ) + md ) from Theorem B.1.
Proof of Q UERY time .
We can view Query as consisting of the following 5 steps:
• Initializing M takes time (cid:101) O
( md ) since M ∈ R m × d . •
Computing the SVD of M takes time
O
( md ω − 1 ) since M is m × d matrix and m ≥ d . • U k can be obtained directly from the SVD computation so this step doesn’t take any additional time.
• Once the algorithm has U k , it can terminate as it outputs ( A 1 , ..., A q , U k ) in factored form.
Overall, the total running time of Q UERY is
[Equation]