Error Analysis of Tensor-Train Cross Approximation
Anonymous Author(s)
Abstract
Tensor train decomposition is widely used in machine learning and quantum physicsdue to its concise representation of high-dimensional tensors, overcoming the curseof dimensionality.
Cross approximation—originally developed for representinga matrix from a set of selected rows and columns—is an efficient method forconstructing a tensor train decomposition of a tensor from few of its entries.
While tensor train cross approximation has achieved remarkable performance inpractical applications, its theoretical analysis, in particular regarding the errorof the approximation, is so far lacking.
To our knowledge, existing results onlyprovide element-wise approximation accuracy guarantees, which lead to a veryloose bound when extended to the entire tensor.
In this paper, we bridge this gap byproviding accuracy guarantees in terms of the entire tensor for both exact and noisymeasurements.
Our results illustrate how the choice of selected subtensors affectsthe quality of the cross approximation and that the approximation error caused bymodel error and/or measurement error may not grow exponentially with the orderof the tensor.
These results are verified by numerical experiments, and may haveimportant implications for the usefulness of cross approximations for high-ordertensors, such as those encountered in the description of quantum many-body states.
Introduction
As modern big datasets are typically represented in the form of huge tensors, tensor decompositionhas become ubiquitous across science and engineering applications, including signal processing andmachine learning [1, 2], communication [3], chemometrics [4, 5], genetic engineering [6] and so on.
The most widely used tensor decompositions include the canonical polyadic (CP) [7], Tucker [8]and tensor train (TT) [9] decompositions.
In general, the CP decomposition provides a compactrepresentation but comes with computational difficulties in finding the optimal decomposition forhigh-order tensors [10, 11].
The Tucker decomposition can be approximately computed via the higherorder singular value decomposition (SVD) [12] but it is inapplicable for high-order tensors sinceits number of parameters scales exponentially with the tensor order.
The TT decomposition sits inthe middle between CP and Tucker decompositions and combines the best of the two worlds: thenumber of parameters does not grow exponentially with the tensor order, and it can be approximatelycomputed by the SVD-based algorithms with a guaranteed accuracy [9].
(See the monograph [13] fora detailed description.)
As such, TT decomposition has attracted tremendous interest over the pastdecade [14–26].
Notably, the TT decomposition is equivalent to the matrix product state (MPS) ormatrix product operator (MPO) introduced in the quantum physics community to efficiently representone-dimensional quantum states, where the number of particles in a quantum system determines theorder of the tensor [27–29].
The use of TT decomposition may thus allow one to classically simulatea one-dimension quantum many-body system with resources polynomial in the number of particles[29], or to perform scalable quantum state tomography [30].
While an SVD-based algorithm [9] can be used to compute a TT decomposition, this requires information about the entire tensor.
However, in many practical applications such as low-rank embeddingsof visual data [31], density estimation [32], surrogate visualization modeling [33], quantum statetomography [30], multidimensional harmonic retrieval [34], parametric partial differential equations[35], interpolation [36], and CMOS ring oscillators [37], one can or it is desirable to only evaluate asmall number of elements of a tensor in practice, since a high-order tensor has exponentially manyelements.
To address this issue, the work [38] develops a cross approximation technique for efficientlycomputing a TT decomposition from selected subtensors.
Originally developed for matrices, crossapproximation represents a matrix from a selected set of its rows and columns, preserving structuresuch as sparsity and non-negativity within the data.
Thus, it has been widely used for data analysis[39, 40], subspace clustering [41], active learning [42], approximate SVDs [43, 44], data compressionand sampling [45–47], and so on.
See Section 2 for a detailed description of cross approximation formatrices and Section 3 for its extension to TT decomposition.
Although the theoretical analysis and applications of matrix cross approximation [48–59] have beendeveloped for last two decades, to the best of our knowledge, the theoretical analysis of TT crossapproximation, especially regarding its error bounds, has not been well studied [60, 61].
Althoughcross approximation gives an accurate TT decomposition when the tensor is exactly in the TT format[38], the question remains largely unsolved as to how good the approximation is when the tensoris only approximately in the TT format, which is often the case for practical data.
In particular, animportant question is whether the approximation error grows exponentially with the order due tothe cascaded multiplications between the factors; this exponential error growth is proved to occurwith the Tucker decomposition [62, 63]. Fortunately, for TT cross approximation, the works [60, 61]establish element-wise error bounds that do not grow exponentially with the order.
Unfortunately,when one extends these element-wise results to the entire tensor, the Frobenius norm error boundgrows exponentially with the order of the tensor, indicating that the cross approximation couldessentially fail for high-order tensors.
However, in practice cross approximation is often found towork well even for high-order tensors, leading us to seek for tighter error bounds on the entire tensor.
Specifically, we ask the following main question:
Question : Does cross approximation provide a stable tensor train (TT) decomposition of a high-order tensor with accuracy guaranteed in terms of the Frobenius norm?
In this paper, we answer this question affirmatively by developing accuracy guarantees for tensorsthat are not exactly in the TT format.
Our results illustrate how the choice of subtensors affects thequality of the approximation and that the approximation error need not grow exponentially with theorder.
We also extend the approximation guarantee to noisy measurements.
The rest of this paper is organized as follows.
In Section 2, we review and summarize existing errorbounds for matrix cross approximation.
Section 3 introduces our new error bounds for TT crossapproximation.
Section 4 includes numerical simulation results that supports our new error boundsand Section 5 concludes the paper.
Notation : We use calligraphic letters (e.g., A ) to denote tensors, bold capital letters (e.g., A ) todenote matrices, bold lowercase letters (e.g., a ) to denote vectors, and italic letters (e.g., a ) to denotescalar quantities.
A − 1 and A † represent the inverse and the Moore–Penrose pseudoinverse matricesof A , respectively.
Elements of matrices and tensors are denoted in parentheses, as in Matlab notation.
For example, A ( i 1 , i 2 , i 3 ) denotes the element in position ( i 1 , i 2 , i 3 ) of the order-3 tensor A ,
A ( i 1 , :)denotes the i 1 -th row of A , and A (: , J ) denotes the submatrix of A obtained by taking the columnsindexed by J . ∥ A ∥ and ∥
A ∥ F respectively represent the spectral norm and Frobenius norm of A .
Cross Approximation for Matrices
As a tensor can be viewed as a generalization of a matrix, it is instructive to introduce the principlesbehind cross approximation for matrices.
Cross approximation, also known as skeleton decompositionor CUR decomposition , approximates a low-rank matrix using a small number of its rows and columns.
Formally, following the conventional notation for the CUR decomposition, we construct a crossapproximation for any matrix A ∈ R m × n as
[Equation]
where † denotes the pseudoinverse (not to be confused with the Hermitian conjugate widely usedin quantum physics literature), C
=
A (: , J ) , U =
A ( I, J ) , and R =
A ( I, :) with I ∈ [ m ] and
J ∈ [ n ] denoting the indices of the selected rows and columns, respectively.
See Figure 1 for anillustration of matrix cross approximation.
While the number of selected rows and columns (i.e., | I |and | J | ) can be different, they are often the same since the rank of CU † R is limited by the smallernumber if they are different.
When U is square and non-singular, the cross approximation (1) takesthe standard form A ≈ CU − 1 R .
Without loss of generality, we assume an equal number of selectedrows and columns in the sequel.
We note that one may replace U † in (1) by C † AR † to improvethe approximation, resulting in the CUR approximation CC † AR † R , where CC † and R † R areorthogonal projections onto the subspaces spanned by the given columns and rows, respectively[64, 56].
However, such a CUR approximation depends on knowledge of the entire matrix A .
As wemainly focus on scenarios where such knowledge is prohibitive, in the sequel we only consider thecross approximation (1) with U
=
A ( I, J ) .
While in general the singular valuedecomposition (SVD) gives the bestlow-rank approximation to a matrix interms of Frobenius norm or spectralnorm, the cross approximation shownin Figure 1 has several notable advantages: ( i ) it is more interpretablesince it is constructed with the rowsand columns of the matrix, and hencecould preserve structure such as sparsity, non-negativity, etc., and ( ii ) it only requires selected rowsand columns while other factorizations often require the entire matrix.
As such, cross approximation has been widely used for data analysis [39, 40], subspace clustering [41], active learning [42],approximate SVDs [43, 44], data compression and sampling [45–47], and so on; see [56, 57] for adetailed history of the use of cross approximation.
The following result shows that the cross approximation is exact (i.e., (1) becomes equality) as longas the submatrix U has the same rank as A .
Theorem 1 For any A ∈ R m × n , suppose we select rows and columns C
=
A (: , J ) , U =
A ( I, J ) ,
R =
A ( I, :) from A such that rank( U ) =
rank( A ) .
Then
[Equation]
119 Proof As R is a submatrix of A and U is a submatrix of R , rank( U ) ≤ rank( R ) ≤ rank( A )always holds.
Now if rank( U ) = rank( A ) , we have rank( U ) =
rank( R ) and thus any columnsin R can be linearly represented by the columns in U .
Using this observation and noting that U = UU † U , we can obtain that R = UU † R .
Likewise, since rank( R ) =
rank( A ) , any rows in A can be linearly represented by the rows in R .
Thus, we can conclude that A = CU † R .
In words, Theorem 1 implies that for an exactly low-rank matrix A with rank r , we can sampler (or more) rows and columns to ensure exact cross approximation.
In practice, however, A isgenerally not exactly low-rank but only approximately low-rank.
In this case, the performance ofcross approximation depends crucially on the selected rows and columns.
Column subset selection isa classical problem in numerical linear algebra, and many methods have been proposed based on themaximal volume principle [65, 66] or related ideas [67–69]; see [69] for more references representingthis research direction.
Efforts have also been devoted to understand the robustness of cross approximation since it onlyyields a low-rank approximation to A when A is not low-rank.
The work [48] provides an elementwise accuracy guarantee for the rows and columns selected via the maximal volume principle.
Therecent work [53] shows there exist a stable cross approximation with accuracy guaranteed in the
Frobenius norm, while the work [54] establishes a similar guarantee for cross approximations
ofrandom matrices using the maximum projective volume principle.
A derandomized algorithm isproposed in [55] that finds a cross approximation achieving similar performance guarantee.
The work[57] establishes an approximation guarantee for cross approximation using any sets of selected rowsand columns.
See Appendix A for a detailed description of these results.
Cross Approximation for Tensor Train Decomposition
In this section, we consider cross approximation for tensor train (TT) decomposition.
We beginby introducing tensor notations.
For an N -th order tensor T ∈ R d 1 ×···× d N , the k -th separation or unfolding of T , denoted by T ⟨ k ⟩ , is a matrix of size ( d 1 · · · d k ) ×
( d k +1 · · · d N ) with its( i 1 . . .
i k , i k +1 · · · i N ) -th element defined by
[Equation]
Here, i 1 · · · i k is a single multi-index that combines indices i 1 , . . .
, i k .
As for the matrix case, wewill select rows and columns from T ⟨ k ⟩ to construct a cross approximation for all k = 1 , . . .
, N − 1 .
To simplify the notation, we use I ≤ k and I >k to denote the positions of the selected r ′ k rows andcolumns in the k -th unfolding.
With abuse of notation, I ≤ k and I >k are also used as the positionsin the original tensor.
Thus, both T ⟨ k ⟩ ( I ≤ k , I >k ) and T ( I ≤ k , I >k ) represent the sampled r ′ k × r ′ kintersection matrix.
Likewise, T ( I ≤ k − 1 , i k , I >k ) represents an r ′ k − 1 × r ′ k matrix whose ( s, t ) -thelement is given by T ( I ≤ k − 1 s , i k , I >kt ) for s = 1 , . .
, r ′ k − 1 and t = 1 , . . .
, r ′ k .
3.1 Tensor Train Decomposition
We say that T ∈ R d 1 ×···× d N is in the TT format if we can express its ( i 1 , . .
, i N ) -th element as thefollowing matrix product form [9]
[Equation]
where X j ∈ R r j − 1 × d j × r j , j = 1 , . . .
, N with r 0 = r N = 1 and X j (: , i j , :)
∈ R r j − 1 × r j , j =1 , . . .
, N denotes one “slice” of X j with the second index being fixed at i j .
To simplify notation, weoften write the above TT format in short as T =
[ X 1 , . . .
, X N ] .
In quantum many-body physics, (3)is equivalent to a matrix product state or matrix product operator, where T ( i 1 , . . .
, i N ) denotes anelement of the many-body wavefunction or density matrix, N is the number of spins/qudits, and r k( k = 1 , 2 , · · · , N − 1) are known as the bond dimensions.
Note that in general the decomposition(3) is not invariant to dimensional permutation due to strictly sequential multilinear products overthe latent cores.
The tensor ring decomposition generalizes (3) by taking the trace of the RHS; thisallows r 0 , r N ≥ 1 and is invariant to circular dimensional permutation [70].
While there may exist infinitely many ways to decompose a tensor T as in (3), we say the decomposition is minimal if the rank of the left unfolding of each X k (i.e.,  
X k (: , 1 , :) ...
X k (: , d k , :) 
 ) is r k and the rankof the right unfolding of each X k (i.e., [ X k (: , 1 , :) · · ·
X k (: , d k , :)] ) is r k − 1 .
The dimensionsr =
( r 1 , . . .
, r N − 1 ) of such a minimal decomposition are called the TT ranks of T .
According to[71], there is exactly one set of ranks r that T admits a minimal TT decomposition.
Moreover, r kequals the rank of the k -th unfolding matrix T ⟨ k ⟩ , which can serve as an alternative way to define the
TT rank.
Efficiency of Tensor Train Decomposition
The number of elements in the N -th order tensor
T grows exponentially in N , a phenomenon known as the curse of dimensionality .
In contrast,a TT decomposition of the form (3) can represent a tensor T with d 1 · · · d N elements using only
O ( dNr 2 ) elements, where d = max { d 1 , . . .
d N } and r = max { r 1 , . .
, r N − 1 } .
This makes the
TT decomposition remarkably efficient in addressing the curse of dimensionality as its number ofparameters scales only linearly in N .
For this reason, the TT decomposition has been widely used forimage compression [14, 15], analyzing theoretical properties of deep networks [16], network compression (or tensor networks) [17–22], recommendation systems [23], probabilistic model estimation[24], learning of Hidden Markov Models [25], and so on.
The work [26] presents a library for TTdecomposition based on TensorFlow.
Notably, TT decomposition is equivalent to the matrix productstate (MPS) introduced in the quantum physics community to efficiently and concisely representquantum states; here N represents the number of qubits of the many–body system [27–29].
Theconcise representation by MPS is remarkably useful in quantum state tomography since it allowsus to observe a quantum state with both experimental and computational resources that are onlypolynomial rather than exponential in the number of qubits N [30].
Comparison with Other Tensor Decompositions
Other commonly used tensor decompositionsinclude the Tucker decomposition, CP decomposition, etc.
The Tucker decomposition is suitableonly for small-order tensors (such as N = 3 ) since its parameterization scales exponentially with N[8].
Like the TT decomposition, the canonical polyadic (CP) decomposition [7] also represents atensor with a linear (in N ) number of parameters.
However, TT has been shown to be exponentiallymore expressive than CP for almost any tensor [16, 72].
Moreover, computing the CP decompositionis computationally challenging; even computing the canonical rank can be NP-hard and ill-posed[10, 11].
In contrast, one can use a sequential SVD-based algorithm [9] to compute a quasioptimal
TT decomposition (the accuracy differs from that of the best possible approximation by at most afactor of √ N ).
We refer to [13] for a detailed comparison.
3.2 Cross Approximation For Tensor Train Decomposition
While the SVD-based algorithm [9] can be used to find a TT decomposition, that algorithm requiresthe entire tensor and is inefficient for large tensor order N .
More importantly, in applications likelow-rank embeddings of visual data [31], density estimation [32], surrogate visualization modeling[33], and quantum state tomography [30], it is desirable to measure a small number of tensor elementssince the experimental cost is proportional to the number of elements measured.
In such scenarios,we can use TT cross approximation to reconstruct the full tensor using only a small number of knownelements, similar to the spirit of matrix cross approximation.
To illustrate the main ideas behind TT cross approximation, we first consider the case where a tensor
T is in the exact TT format (3), i.e., the unfolding matrix T ⟨ k ⟩ is exactly low-rank with rank r k .
Inthis case, by Theorem 1, we can represent T ⟨ 1 ⟩ with a cross approximation by choosing r ′ 1 rows(with r ′ 1 ≥ r 1 ) indexed by I ≤ 1 and r ′ 1 columns indexed by I > 1 such that
[Equation]
… … … … … reshape
Here T ⟨ 1 ⟩ (: , I > 1 ) has d 1 r ′ 1 entries, while T ⟨ 1 ⟩ ( I ≤ 1 , :) is anextremely wide matrix.
However, one only needs to samplethe elements in T ⟨ 1 ⟩ (: , I > 1 ) andcan avoid sampling the entireelements of T ⟨ 1 ⟩ ( I ≤ 1 , :) by further using cross approximation .
In particular, one can reshape T ⟨ 1 ⟩ ( I ≤ 1 , :) into another matrixof size r ′ 1 d 2 × d 3 · · · d N and thenrepresent it with a cross approximation by choosing r ′ 2 rows andcolumns.
This procedure can beapplied recursively N − 1 times,and only at the last stage doesone sample both r ′ N − 1 rows andcolumns of an r ′ N − 2 d N − 1 × d N 226 matrix.
This is the procedure developed in [38] for TT cross approximation.
The process of TT crossapproximation is shown in Fig.
The procedure illustrated in Figure 2 produces nested interpolation sets as one recursively appliescross approximation on the reshaped versions of previously selected rows.
But the interpolation sets{ I ≤ k , I >k } are not required to be nested.
In general, one can slightly generalize this procedure andobtain a TT cross approximation (cid:98) T with elements given by [61](cid:98)
[Equation]
[Equation] (4) where I ≤ k denotes the selected row indices from the multi-index { d 1 · · · d k } , I >k is the selectionof the column indices from the multi-index { d k +1 · · · d N } , and I ≤ 0 =
I >N = ∅ .
Here A † τ denotes the τ -pseudoinversion of A that sets the singular values less than τ to zero before calculating thepseudoinversion.
Similar to Theorem 1, the TT cross approximation (cid:98) T equals T when T is in the TT format
(3)and the intersection matrix T ( I ≤ k , I >k ) has rank r k for k = 1 , . .
. , N − 1 [38].
Nevertheless, inpractical applications, tensors may be only approximately but not exactly in the TT format (3), i.e.,the unfolding matrices T ⟨ k ⟩ may be only approximately low-rank.
Moreover, in applications suchas quantum tomography, one can only observe noisy samples of the tensor entries.
We provideguarantees in terms of ∥T − (cid:98) T ∥
F for these cases in the following subsections.
3.3 Error Analysis of TT Cross Approximation with Exact Measurements
Assume each unfolding matrix T ⟨ k ⟩ is approximately low-rank with residual captured by ∥ T ⟨ k ⟩ −
T ⟨ k ⟩ r k ∥ F , where T ⟨ k ⟩ r k denotes the best rank- r k approximation of T ⟨ k ⟩ .
For convenience, we denote by
[Equation]
where ϵ captures the largest low-rank approximation error in all the unfolding matrices.
The work [60, 61] develops an element-wise error bound for TT cross approximation based on themaximum-volume choice of the interpolation sets.
However, if we are interested in approximationguarantee for the entire tensor (say in the Frobenius norm) instead of each entry, then directly applyingthe above result leads to a loose bound with scaling at least proportional to √ d 1 · · · d N .
Our goal isto provide guarantees directly for ∥T − (cid:98) T ∥ F . Note that (5) implies that a fundamental lower bound∥T − (cid:98) T ∥ F ≥ ϵ since ∥T − (cid:98) T ∥ F = ∥ T ⟨ k ⟩ − (cid:98) T ⟨ k ⟩ ∥ F [Equation]
holds for any TT format tensor (cid:98) T with ranks r (not only those constructed by cross approximation).
Fortunately, tensor models in practical applications such as image and video [14, 15] and quantumstates [27–29] can be well represented in TT format with very small ϵ .
Our first main result proves the existence of stable TT cross approximation with exact measurementsof the tensor elements:
Theorem 2 Suppose T can be approximated by a TT format tensor with ranks r and approximationerror ϵ defined in (5) .
For sufficiently small ϵ , there exists a cross approximation (cid:98) T of the form (4)with τ k = 0 that satisfies
[Equation]
where κ := max k (cid:8)(cid:13)(cid:13) T ⟨ k ⟩ (: , I >k ) · T ⟨ k ⟩ ( I ≤ k , I >k ) − 1 (cid:13)(cid:13) , (cid:13)(cid:13) T ⟨ k ⟩ ( I ≤ k , I >k ) − 1 · T ⟨ k ⟩ ( I ≤ k , :) (cid:13)(cid:13)(cid:9) .
We use sufficiently small ϵ to simplify the presentation.
This refers to the value of ϵ that(cid:13)(cid:13)(cid:13)(cid:16) (cid:98) T ⟨ k ⟩ (: , I >k ) − T ⟨ k ⟩ (: , I >k ) (cid:17) · T ⟨ k ⟩ ( I ≤ k , I >k ) − 1 (cid:13)(cid:13)(cid:13) ≤ κ , which can always be guaranteed sinceby (7) the distance between (cid:98) T and T (and hence the distance between (cid:98) T ⟨ k ⟩ (: , I >k ) and T ⟨ k ⟩ (: , I >k ) )approaches to zero when ϵ goes to zero.
The full proof is provided in Appendix B.
On one hand, the right hand side of (7) grows only linearly with the approximation error ϵ , with a scalarthat increases only logarithmically in terms of the order N .
This shows the TT cross approximationcan be stable for well selected interpolation sets.
Recall that by (6), for any cross approximation (cid:98) T ofthe form (4) with r k selected rows and columns in the sets { I ≤ k , I >k } , ∥T − (cid:98) T ∥
F ≥ ϵ necessarilyholds since (cid:98) T ⟨ k ⟩ has rank at most r k .
While it may be unfair to compare the element-wise errorbounds [60, 61], we present [60, Theorem 1] below which also involves ϵ and a similar quantity to κ [Equation] where ∥T ∥ max = max i 1 ,...,i N |T ( i 1 , . . , i N ) | and (cid:101) κ = max k r k ∥ T ⟨ k ⟩ ∥ max · ∥ T ⟨ k ⟩ ( I ≤ k , I >k ) − 1 ∥ max .
Loosely speaking, κ and (cid:101) κ can be approximately viewed as the condition number of the submatrix T ⟨ k ⟩ ( I ≤ k , I >k ) with respect to the spectral norm and the Chebyshev norm, respectively.
Again, we note that our bound (7) is for the entire tensor,while ∥T − (cid:98) T ∥ max only concerns the largest element-wise error.
On the other hand, Theorem 2 only shows the existence one such stable cross approximation, butit does not specify which one.
As a consequence, the result may not hold for the nested samplingstrategy illustrated in Figure 2 and the parameter κ maybe out of one’s control.
The following resultaddresses these issues by providing guarantees for any cross approximation.
Theorem 3 Suppose T can be approximated by a TT format tensor with rank r and approximationerror ϵ defined in (5) .
Let T ⟨ k ⟩ r k be the best rank r k approximation of the k -th unfolding matrix T ⟨ k ⟩and T ⟨ k ⟩ r k = W ( k ) Σ ( k ) V ( k ) T be its compact SVD.
For any interpolation sets { I ≤ k , I >k } such thatrank( T ⟨ k ⟩ r k ( I ≤ k , I >k )) =
r k , k = 1 , . . .
, N − 1 , denote bya =
[Equation]
Then the cross approximation (cid:98) T in (4) with appropriate thresholding parameters τ k for the truncatedpseudo-inverse satisfies
[Equation]
(8) where ≲ means less than or equal to up to some universal constant.
The proof of Theorem 3 is given in Appendix C. As in (7), the right hand side of (8) scales onlypolynomially in ϵ and logarithmically in terms of the order N .
Compared with (7), the guaranteein (8) holds for any cross approximation, but the quality of the cross approximation depends onthe tensor T as well as the selected rows and columns as reflected by the parameters a and c .
Onone hand, (cid:13)(cid:13)
[ W ( k ) ( I ≤ k , :)] † (cid:13)(cid:13) and (cid:13)(cid:13)
[ V ( k ) ( I >k , :)] † (cid:13)(cid:13) can achieve their smallest possible value 1 whenthe singular vectors W ( k ) and V ( k ) are the canonical basis.
On the other hand, one may constructexamples with large (cid:13)(cid:13)
[ W ( k ) ( I ≤ k , :)] † (cid:13)(cid:13) and (cid:13)(cid:13) [ V ( k ) ( I >k , :)] † (cid:13)(cid:13) .
Nevertheless, these terms can beupper bounded by selecting appropriate rows and columns.
For example, for any orthonormal matrix
W ∈ R m × r , if we select I such that W ( I, :) has maximal volume among all | I | × r submatrices of W , then (cid:13)(cid:13) W ( I, :) † (cid:13)(cid:13) ≤ (cid:113) 1 + r ( m −| I | ) | I |− r {++
 [57,+} Proposition 5.1].
In practice, without knowing W ( k )and V ( k ) , we may instead find the interpolation sets by maximizing the volume of T ⟨ k ⟩ ( I ≤ k , I >k )using algorithms such as greedy restricted cross interpolation (GRCI) [60].
We may also exploitderandomized algorithms [55] to find interpolation sets that could potentially yield small quantitiesa, b, c .
We finally note that Theorem 3 uses the truncated pseudo-inverse in (4) with appropriate τ k .
We also observe from the experiments in Section 4 that the truncated pseudo-inverse could lead tobetter performance than the inverse (or pseudo-inverse) for TT cross approximation (4).
3.4 Error Analysis for TT Cross Approximation with Noisy Measurements
We now consider the case where the measurements of tensor elements are not exact, but noisy.
Thishappens, for example, in the measurements of quantum state where each entry of the many-bodywavefunction or density matrix can only be measured up to some statistical error that depends onthe number of repeated measurements.
In this case, the measured entries T ( I ≤ k − 1 , i k , I >k ) and[ T ( I ≤ k , I >k )]
τ k in the cross approximation (4) are noisy.
Our goal is to understand how such noisymeasurements will affect the quality of cross approximation.
To simplify the notation, we let E denote the measurement error though it has non-zero values only atthe selected interpolation sets { I ≤ k , I >k } .
Also let (cid:101) T =
T + E and (cid:101) T ⟨ k ⟩ = T ⟨ k ⟩ + E ⟨ k ⟩ , where E ⟨ k ⟩denotes the k -th unfolding matrix of the noise.
Then, the cross approximation of T ∈ R d 1 ×···× d Nwith noisy measurements, denoted by (cid:98) T , has entries given by
[Equation]
One can view (cid:98) T as a cross approximation of (cid:101) T with exact measurements of (cid:101) T .
However, we can notdirectly apply Theorem 3 since it would give a bound for ∥ (cid:101) T − (cid:98) T ∥
F and that would also depend onthe singular vectors of (cid:101) T ⟨ k ⟩ .
The following result addresses this {+issue. 8 r ′ - -  τ =
 τ =10 -7 τ+}
{+=10 - (a)    22 N - -5050 100 τ =
+}
( r '=2) τ
{+=
+}
( r '=4) τ
=10 -7 ( r '=4) (b) 1010 - 1010 -4 η - -80 - - r'=3r'=4r'= (c) 1010 - 1010 -4 µ - -80 - -40r'=3r'=4r'= (d) 2 3 4 5 6 7 8 r - - - -65 r'=8r'=9r'= (e)
Theorem 4 Suppose T can be approximated by a TT format tensor with rank r and approximationerror ϵ defined in (5) .
Let T ⟨ k ⟩ r k be the best rank r k approximation of the k -th unfolding matrix
T ⟨ k ⟩ and T ⟨ k ⟩ r k = W ( k ) Σ ( k ) V ( k ) T be its compact SVD.
Let (cid:101) T =
T + E denote the noisy tensor where E represents the measurement error.
For any interpolation sets { I ≤ k , I >k } such thatrank(
[Equation]
( I ≤ k , I >k )) =
r k , k = 1 , . . .
, N − 1 , denote bya =
[Equation]
Denote by ϵ = ϵ + ∥E∥ F .
Then the noisy cross approximation (cid:98) T in (9) appropriate thresholdingparameters τ k for the truncated pseudo-inverse satisfies
[Equation]
The proof of Theorem 4 is given in Appendix D. In words, Theorem 4 provides a similar guarantee tothat in Theorem 3 for TT cross approximation with noisy measurements and shows that it is also stablewith respect to measurement error.
The parameter c in Theorem 4 depends on the spectral norm of[ (cid:101) T ⟨ k ⟩ ( I ≤ k , I >k )] † and is defined in this form for simplicity.
It can be bounded by [ T ⟨ k ⟩ ( I ≤ k , I >k )] †and the noise and is expected to be close to [ T ⟨ k ⟩ ( I ≤ k , I >k )] † for random noise.
We finally note thatthe measurement error E only has O ( Nr 2 ) non-zero elements, and thus ∥E∥ F is not exponentiallylarge in terms of N .
Numerical Experiments
In this section, we conduct numerical experiments to evaluate the performance of the cross approximation (9) with noisy measurements.
We generate an N -th order tensor T ∈ R d 1 ×···× d N approximatelyin the TT format as T =
T r + η F , where T r is in the TT format with mode ranks [ r 1 , . .
, r N − 1 ] ,which is generated from truncating a random Gaussian tensor using a sequential SVD [9], and F is a random tensor with independent entries generated from the normal distribution.
We thennormalize T r and F to unit Frobenius norm.
Thus η controls the low-rank approximation error.
Tosimplify the selection of parameters, we let d = d 1 = · · · = d N = 2 , r = r 1 = · · · = r N − 1 , andτ = τ 1 = · · · = τ N − 1 for the cross approximation (9).
We apply the greedy restricted cross interpolation (GRCI) algorithm in [60] for choosing the interpolation sets I ≤ k and I >k , k = 1 , . . .
, N .
For convenience, we set r ′ =
r ′ 1 = · · · = r ′ N − 1 forthe number of selected rows and columns indexed by I ≤ k
and I >k .
During the procedure of crossapproximation, for each of the ( N − 2) dr ′ 2 + 2 dr ′ − r ′ 2 observed entries, we add measurement errorrandomly generated from the Gaussian distribution of mean 0 and variance 1 ( N − 2) dr ′ 2 +2 dr ′ − r ′ 2 .
Asin Section 3.4, for convenience, we model the measurement error as a tensor E such that E ∥E∥ 2 F = 1 .
To control the signal-to-noise level, we scale the noise by a factor µ .
Thus, we have(cid:101) T = [Equation] , (11) where ∥T r ∥ 2 F = ∥F∥ 2 F = E ∥E∥ 2 F = 1 . We measure the normalized mean square error (MSE) between T and the cross approximation (cid:98) T by
[Equation]
In the first experiment, we set N = 10 , d = 2 , r = 2 and µ = η = 10 − 5 and compare theperformance of tensor cross approximations with different τ and r ′ .
From Fig.
3a, we observe that when r ′ is specified as the exact rank (i.e., r ′ = r ), all of the TT cross approximations arevery accurate except when τ is relatively large and thus useful information is truncated.
On theother hand, when r ′ > r , the performance of cross approximations with the exact pseudo-inverse( τ = 0 ) degrades.
This is because the intersection matrices T ( I ≤ k , I >k ) in (4) or (cid:101) T ( I ≤ k , I >k ) in(9) become ill-conditioned.
Fortunately, since the last r ′ − r singular values of T ( I ≤ k , I >k ) and(cid:101) T ( I ≤ k , I >k ) only capture information about the approximation error and measurement error, thisissue can be addressed by using the truncated pseudo-inverse with suitable τ > 0 which can make TTcross approximation stable. We notice that larger r ′ gives better cross approximation due to moremeasurements.
In the second experiment, we test the performance of tensor cross approximations with differenttensor orders N .
For N ranging from 6 to 22 , we set all mode sizes d = 2 and µ = η = 10 − 5 .
Here d = 2 corresponds to a quantum state, and while d = 2 is small, the tensor is still huge forlarge
N (e.g., N = 22 ).
We also set all ranks r = 2 .
From Fig.
3b, we can observe that the TTcross approximation with the pseudo-inverse is stable as N increases when r ′ =
r (black curve) butbecomes unstable when r ′ >
r (blue curve).
Again, for the latter case when r ′ > r , the TT crossapproximation achieves stable performance by using the truncated pseudo-inverse.
In the third experiment, we demonstrate the performance for different η , which controls the levelof low-rank approximation error.
Here, we set µ = 10 − 7 , r = 2 , τ = 10 − 2 η , and consider theover-specified case where r ′ ranges from 3 to 5.
Fig. 3c shows that the performance of TT crossapproximation with the truncated pseudo-inverse is stable with the increase of η and different r ′ .
In the forth experiment, we demonstrate the performance for different measurement error levels µ .
Similar to the third experiment, we set η = 10 − 7 , r = 2 , τ = 10 − 2 µ , and consider the over-specifiedcase where r ′ ranges from 3 to 5.
We observe from Fig.
3d that tensor train cross approximation stillprovides stable performance with increasing µ and different r ′ .
In the fifth experiment, we test the performance for different r .
We set µ = η = 10 − 5 , d = 2 ,τ = 10 − 2 η and consider the case where r ′ ranges from 8 to 10 .
Fig.
3e shows that the peformance of
TT cross approximation is stable with the increase of r , with recovery error in the curves roughlyincreasing as O ( r 3 ) which is consistent with Theorem 4.
Note that in practice r is often unknown apriori.
For this case, the results in the above figures show that we can safely use a relatively largeestimate r ′ for tensor train cross approximation with the truncated pseudo-inverse. [Figure]
In the final experiment, we test the performance with further larger tensororders N .
To achieve this goal, we work on tensors in the exact TT format(i.e.
, η = 0 ) since in this case we only need to store the tensor
factorswith O ( Nr 2 ) elements rather than the entire tensor. We set r ′ = r = 2 ,µ = 10 − 5 and consider one case where N varies from 5 to 50 .
Weobserve from Fig.
4 that the approximation error curve increases onlylogarithmically in terms of the order N , which is consistent with (10).
Conclusion and Outlook
While TT cross approximation has achieved excellent performance inpractice, existing theoretical results only provide element-wise approximation accuracy guarantees.
Inthis paper, we provide the first error analysis in terms of the entire tensor for TT cross approximation.
For a tensor in an approximate TT format, we prove that the cross approximation is stable in the sensethat the approximation error for the entire tensor does not grow exponentially with the tensor order,and we extend this guarantee to the scenario of noisy measurements.
Experiments verified our results.
An important future direction we will pursue is to study the implications of our results for quantumtomography.
In particular, we aim to find out whether our result imply that the statistical errorsencountered in quantum measurements will not proliferate if we perform cross approximation of thequantum many-body state.
Another future direction of our work is to design optimal strategies forthe selections of the tensor elements to be measured.
One possible approach is to extend methodsdeveloped for matrix cross approximation [55], and another approach is to incorporate leverage scoresampling [73–75].
It is also interesting to use local refinement to improve the performance of TTcross approximation.
We leave these explorations to future work.