Error Analysis of Tensor-Train Cross Approximation
Anonymous Author(s)
Abstract
Tensor train decomposition is widely used in machine learning and quantum physicsdue to its concise representation of high-dimensional tensors, overcoming the curseof dimensionality.
Cross approximation—originally developed for representinga matrix from a set of selected rows and columns—is an efficient method forconstructing a tensor train decomposition of a tensor from few of its entries.
While tensor train cross approximation has achieved remarkable performance inpractical applications, its theoretical analysis, in particular regarding the errorof the approximation, is so far lacking.
To our knowledge, existing results onlyprovide element-wise approximation accuracy guarantees, which lead to a veryloose bound when extended to the entire tensor.
In this paper, we bridge this gap byproviding accuracy guarantees in terms of the entire tensor for both exact and noisymeasurements.
Our results illustrate how the choice of selected subtensors affectsthe quality of the cross approximation and that the approximation error caused bymodel error and/or measurement error may not grow exponentially with the orderof the tensor.
These results are verified by numerical experiments, and may haveimportant implications for the usefulness of cross approximations for high-ordertensors, such as those encountered in the description of quantum many-body states.
Introduction
As modern big datasets are typically represented in the form of huge tensors, tensor decompositionhas become ubiquitous across science and engineering applications, including signal processing andmachine learning [1, 2], communication [3], chemometrics [4, 5], genetic engineering [6] and so on.
The most widely used tensor decompositions include the canonical polyadic (CP) [7], Tucker [8]and tensor train (TT) [9] decompositions.
In general, the CP decomposition provides a compactrepresentation but comes with computational difficulties in finding the optimal decomposition forhigh-order tensors [10, 11].
The Tucker decomposition can be approximately computed via the higherorder singular value decomposition (SVD) [12] but it is inapplicable for high-order tensors sinceits number of parameters scales exponentially with the tensor order.
The TT decomposition sits inthe middle between CP and Tucker decompositions and combines the best of the two worlds: thenumber of parameters does not grow exponentially with the tensor order, and it can be approximatelycomputed by the SVD-based algorithms with a guaranteed accuracy [9].
(See the monograph [13] fora detailed description.)
As such, TT decomposition has attracted tremendous interest over the pastdecade [14–26].
Notably, the TT decomposition is equivalent to the matrix product state (MPS) ormatrix product operator (MPO) introduced in the quantum physics community to efficiently representone-dimensional quantum states, where the number of particles in a quantum system determines theorder of the tensor [27–29].
The use of TT decomposition may thus allow one to classically simulatea one-dimension quantum many-body system with resources polynomial in the number of particles[29], or to perform scalable quantum state tomography [30].
While an SVD-based algorithm [9] can be used to compute a TT decomposition, this requiresinformation about the entire tensor.
However, in many practical applications such as quantum statetomography [30], multidimensional harmonic retrieval [31], low-rank embeddings of visual data[32], surrogate visualization model [33], parametric partial differential equations [34], interpolation[35], and CMOS ring oscillators [36], one can only evaluate a small number of elements of a tensorin practice, since a high-order tensor has exponentially many elements.
To address this issue, thework [37] develops a cross approximation technique for efficiently computing a TT decompositionfrom selected subtensors.
Originally developed for matrices, cross approximation represents a matrixfrom a selected set of its rows and columns, preserving structure such as sparsity and non-negativitywithin the data.
Thus, it has been widely used for data analysis [38, 39], subspace clustering [40],active learning [41], approximate SVDs [42, 43], data compression and sampling [44–46], and soon.
See Section 2 for a detailed description of cross approximation for matrices and Section 3 for itsextension to TT decomposition.
Although the theoretical analysis and applications of matrix cross approximation [47–58] have beendeveloped for last two decades, to the best of our knowledge, the theoretical analysis of TT crossapproximation, especially regarding its error bounds, has not been well studied [59, 60].
Althoughcross approximation gives an accurate TT decomposition when the tensor is exactly in the TT format[37], the question remains largely unsolved as to how good the approximation is when the tensor isonly approximately in the TT format, which is often the case for practical data.
One existing work [59]develops an element-wise error bound for TT cross approximation based on the maximum-volumechoice of the interpolation sets; another existing work [60] provides a similar element-wise errorbound for truncated TT cross approximation, in which a threshold is set to truncate small singularvalues of intersection matrices.
However, when one extends these element-wise results to the entiretensor, the Frobenius norm error bound grows exponentially with the order of the tensor, indicatingthat the cross approximation will essentially fail for high-order tensors.
However, in practice crossapproximation is often found to work well even for high-order tensors, leading us to seek for tightererror bounds on the entire tensor.
Specifically, we ask the following main question:
Question : Does cross approximation provide a stable tensor train (TT) decomposition of a high-order tensor with accuracy guaranteed in terms of the Frobenius norm?
In this paper, we answer this question affirmatively by developing accuracy guarantees for tensorsthat are not exactly in the TT format.
Our results illustrate how the choice of subtensors affects thequality of the approximation and that the approximation error need not grow exponentially with theorder.
We also extend the approximation guarantee to noisy measurements.
The rest of this paper is organized as follows.
In Section 2, we review and summarize existing errorbounds for matrix cross approximation.
Section 3 introduces our new error bounds for TT crossapproximation.
Section 4 includes numerical simulation results that supports our new error boundsand Section 5 concludes the paper.
Notation : We use calligraphic letters (e.g., A ) to denote tensors, bold capital letters (e.g., A ) todenote matrices, bold lowercase letters (e.g., a ) to denote vectors, and italic letters (e.g., a ) to denotescalar quantities.
A − 1 and A † represent the inverse and the Moore–Penrose pseudoinverse matricesof A , respectively.
Elements of matrices and tensors are denoted in parentheses, as in Matlab notation.
For example, A ( i 1 , i 2 , i 3 ) denotes the element in position ( i 1 , i 2 , i 3 ) of the order-3 tensor A ,
A ( i 1 , :)denotes the i 1 -th row of A , and A (: , J ) denotes the submatrix of A obtained by taking the columnsindexed by J .
Cross Approximation for Matrices
As a tensor can be viewed as a generalization of a matrix, it is instructive to introduce the principlesbehind cross approximation for matrices.
Cross approximation, also known as skeleton decompositionor CUR decomposition , approximates a low-rank matrix using a small number of its rows and columns.
Formally, following the conventional notation for the CUR decomposition, we construct a crossapproximation for any matrix A ∈ R m × n as
[Equation]
where † denotes the pseudoinverse (not to be confused with the Hermitian conjugate widely usedin quantum physics literature), C
=
A (: , J ) , U =
A ( I, J ) , and R =
A ( I, :) with I ∈ [ m ] and
J ∈ [ n ] denoting the indices of the selected rows and columns, respectively.
See Figure 1 for anillustration of matrix cross approximation.
While the number of selected rows and columns (i.e., | I |and | J | ) can be different, they are often the same since the rank of CU † R is limited by the smallernumber if they are different.
When U is square and non-singular, the cross approximation (1) takesthe standard form A ≈ CU − 1 R .
Without loss of generality, we assume an equal number of selectedrows and columns in the sequel.
We note that one may replace U † in (1) by C † AR † to improvethe approximation, resulting in the CUR approximation CC † AR † R , where CC † and R † R areorthogonal projections onto the subspaces spanned by the given columns and rows, respectively[61, 55].
However, such a CUR approximation depends on knowledge of the entire matrix A .
As wemainly focus on scenarios where such knowledge is prohibitive, in the sequel we only consider thecross approximation (1) with U
=
A ( I, J ) .
While in general the singular value decomposition (SVD) gives the best low-rank approximation toa matrix in terms of Frobenius norm or spectral norm, the cross approximation shown in Figure 1has several notable advantages: ( i ) it is more interpretable since it is constructed with the rows andcolumns of the matrix, and hence could preserve structure such as sparsity, non-negativity, etc., and( ii ) it only requires selected rows and columns while other factorizations often require the entirematrix.
As such, cross approximation has been widely used for data analysis [38, 39], subspaceclustering [40], active learning [41], approximate SVDs [42, 43], data compression and sampling[44–46], and so on; see [55, 56] for a detailed history of the use of cross approximation.
The following result shows that the cross approximation is exact (i.e., (1) becomes equality) as longas the submatrix U has the same rank as A .
Theorem 1 For any A ∈ R m × n , suppose we select rows and columns C
=
A (: , J ) , U =
A ( I, J ) ,
R =
A ( I, :) from A such that rank( U ) =
rank( A ) .
Then
[Equation]
Proof Since rank( U ) =
rank( A ) , we have rank( U ) =
rank( R ) and thus any columns in R can belinearly represented by the columns in U .
Using this observation and noting that U = UU † U , wecan obtain that R = UU † R .
Likewise, since rank( R ) =
rank( A ) , any rows in A can be linearlyrepresented by the rows in R .
Thus, we can conclude that A = CU † R .
In words, Theorem 1 implies that for an exactly low-rank matrix A with rank r , we can sampler (or more) rows and columns to ensure exact cross approximation.
In practice, however, A isgenerally not exactly low-rank but only approximately low-rank.
In this case, the performance ofcross approximation depends crucially on the selected rows and columns.
Column subset selection isa classical problem in numerical linear algebra, and many methods have been proposed based on themaximal volume principle [62, 63] or related ideas [64–66]; see [66] for more references representingthis research direction.
Efforts have also been devoted to understand the robustness of cross approximation since it onlyyields a low-rank approximation to A when A is not low-rank.
The work [47] provides an element- wise accuracy guarantee for the rows and columns selected via the maximal volume principle.
Therecent work [52] shows there exist a stable cross approximation with accuracy guaranteed in the
Frobenius norm, while the work [53] establishes a similar guarantee for cross approximations
ofrandom matrices using the maximum projective volume principle.
A derandomized algorithm isproposed in [54] that finds a cross approximation achieving similar performance guarantee.
The work[56] establishes an approximation guarantee for cross approximation using any sets of selected rowsand columns.
See Appendix A for a detailed description of these results.
Cross Approximation for Tensor Train Decomposition
In this section, we consider cross approximation for tensor train (TT) decomposition.
We beginby introducing tensor notations.
For an N -th order tensor T ∈ R d 1 ×···× d N , the k -th separation or unfolding of T , denoted by T ⟨ k ⟩ , is a matrix of size ( d 1 · · · d k ) ×
( d k +1 · · · d N ) with its( i 1 . . .
i k , i k +1 · · · i N ) -th element defined by
[Equation]
Here, i 1 · · · i k is a single multi-index that combines indices i 1 , . . .
, i k .
As for the matrix case, wewill select rows and columns from T ⟨ k ⟩ to construct a cross approximation for all k = 1 , . . .
, N − 1 .
To simplify the notation, we use I ≤ k and I >k to denote the positions of the selected r ′ k rows andcolumns in the k -th unfolding.
With abuse of notation, I ≤ k and I >k are also used as the positionsin the original tensor.
Thus, both T ⟨ k ⟩ ( I ≤ k , I >k ) and T ( I ≤ k , I >k ) represent the sampled r ′ k × r ′ kintersection matrix.
Likewise, T ( I ≤ k − 1 , i k , I >k ) represents an r ′ k − 1 × r ′ k matrix whose ( s, t ) -thelement is given by T ( I ≤ k − 1 s , i k , I >kt ) for s = 1 , . .
, r ′ k − 1 and t = 1 , . . .
, r ′ k .
3.1 Tensor Train Decomposition
We say that T ∈ R d 1 ×···× d N is in the TT format if we can express its ( i 1 , . .
, i N ) -th element as thefollowing matrix product form [9]
[Equation]
where X j ∈ R r j − 1 × d j × r j , j = 1 , . . .
, N with r 0 = r N = 1 and X j (: , i j , :)
∈ R r j − 1 × r j , j =1 , . . .
, N denotes one “slice” of X j with the second index being fixed at i j .
To simplify notation, weoften write the above TT format in short as T =
[ X 1 , . . .
, X N ] .
In quantum many-body physics, (3)is equivalent to a matrix product state or matrix product operator, where T ( i 1 , . . .
, i N ) denotes anelement of the many-body wavefunction or density matrix, N is the number of spins/qudits, and r k( k = 1 , 2 , · · · , N − 1) are known as the bond dimensions.
While there may exist infinitely many ways to decompose a tensor T as in (3), we say the decomposition is minimal if the rank of the left unfolding of each X k (i.e.,  
X k (: , 1 , :) ...
X k (: , d k , :) 
 ) is r k and the rankof the right unfolding of each X k (i.e., [ X k (: , 1 , :) · · ·
X k (: , d k , :)] ) is r k − 1 .
The dimensionsr =
( r 1 , . . .
, r N − 1 ) of such a minimal decomposition are called the TT ranks of T .
According to[67], there is exactly one set of ranks r that T admits a minimal TT decomposition.
Moreover, r kequals the rank of the k -th unfolding matrix T ⟨ k ⟩ , which can serve as an alternative way to define the
TT rank.
Efficiency of Tensor Train Decomposition
The number of elements in the N -th order tensor
T grows exponentially in N , a phenomenon known as the curse of dimensionality .
In contrast,a TT decomposition of the form (3) can represent a tensor T with d 1 · · · d N elements using only
O ( dNr 2 ) elements, where d = max { d 1 , . . .
d N } and r = max { r 1 , . .
, r N − 1 } .
This makes the
TT decomposition remarkably efficient in addressing the curse of dimensionality as its number ofparameters scales only linearly in N .
For this reason, the TT decomposition has been widely used forimage compression [14, 15], analyzing theoretical properties of deep networks [16], network compression (or tensor networks) [17–22], recommendation systems [23], probabilistic model estimation[24], learning of Hidden Markov Models [25], and so on.
The work [26] presents a library for TTdecomposition based on TensorFlow.
Notably, TT decomposition is equivalent to the matrix product state (MPS) introduced in the quantum physics community to efficiently and concisely representquantum states; here N represents the number of qubits of the many–body system [27–29].
Theconcise representation by MPS is remarkably useful in quantum state tomography since it allowsus to observe a quantum state with both experimental and computational resources that are onlypolynomial rather than exponential in the number of qubits N [30].
Comparison with Other Tensor Decompositions
Other commonly used tensor decompositionsinclude the Tucker decomposition, CP decomposition, etc.
The Tucker decomposition is suitableonly for small-order tensors (such as N = 3 ) since its parameterization scales exponentially with N[8].
Like the TT decomposition, the canonical polyadic (CP) decomposition [7] also represents atensor with a linear (in N ) number of parameters.
However, TT has been shown to be exponentiallymore expressive than CP for almost any tensor [16, 68].
Moreover, computing the CP decompositionis computationally challenging; even computing the canonical rank can be NP-hard and ill-posed[10, 11].
In contrast, one can use a sequential SVD-based algorithm [9] to compute a quasioptimal
TT decomposition (the accuracy differs from that of the best possible approximation by at most afactor of √ N ).
We refer to [13] for a detailed comparison.
3.2 Cross Approximation For Tensor Train Decomposition
While the SVD-based algorithm [9] can be used to find a TT decomposition, that algorithm requiresthe entire tensor and is inefficient for large tensor order N .
More importantly, in applications likequantum state tomography [30], one can only measure a small number of tensor elements since theexperimental time cost is proportional to the number of elements measured.
In such scenario, wecan use TT cross approximation to reconstruct the full tensor using only a small number of knownelements, similar to the spirit of matrix cross approximation.
To illustrate the main ideas behind TT cross approximation, we first consider the case where a tensor
T is in the exact TT format (3), i.e., the unfolding matrix T ⟨ k ⟩ is exactly low-rank with rank r k .
Inthis case, by Theorem 1, we can represent T ⟨ 1 ⟩ with a cross approximation by choosing r ′ 1 rows(with r ′ 1 ≥ r 1 ) indexed by I ≤ 1 and r ′ 1 columns indexed by I > 1 such that
[Equation]
Here T ⟨ 1 ⟩ (: , I > 1 ) has d 1 r ′ 1 entries, while T ⟨ 1 ⟩ ( I ≤ 1 , :) is an extremely wide matrix.
However, oneonly needs to sample the elements in T ⟨ 1 ⟩ (: , I > 1 ) and can avoid sampling the entire elements of T ⟨ 1 ⟩ ( I ≤ 1 , :) by further using cross approximation .
In particular, one can reshape T ⟨ 1 ⟩ ( I ≤ 1 , :) intoanother matrix of size r ′ 1 d 2 × d 3 · · · d N and then represent it with a cross approximation by choosingr ′ 2 rows and columns.
This procedure can be applied recursively N − 1 times, and only at the laststage does one sample both r ′ N − 1 rows and columns of an r ′ N − 2 d N − 1 × d N matrix.
This is theprocedure developed in [37] for TT cross approximation.
The process of TT cross approximation isshown in Fig.
The procedure illustrated in Figure 2 produces nested interpolation sets as one recursively appliescross approximation on the reshaped versions of previously selected rows.
But the interpolation sets{ I ≤ k , I >k } are not required to be nested.
In general, one can slightly generalize this procedure andobtain a TT cross approximation (cid:98) T with elements given by [60]
[Equation]
where I ≤ k denotes the selected row indices from the multi-index { d 1 · · · d k } , I >k is the selectionof the column indices from the multi-index { d k +1 · · · d N } , and I ≤ 0 =
I >N = ∅ .
Here A † τ denotesthe τ -pseudoinversion of A that sets the singular values less than τ to zero before calculating thepseudoinversion.
Similar to Theorem 1, the TT cross approximation (cid:98) T equals T when T is in the TT format
(3)and the intersection matrix T ( I ≤ k , I >k ) has rank r k for k = 1 , . .
. , N − 1 [37].
Nevertheless, inpractical applications, tensors may be only approximately but not exactly in the TT format (3), i.e.,the unfolding matrices T ⟨ k ⟩ may be only approximately low-rank.
Moreover, in applications suchas quantum tomography, one can only observe noisy samples of the tensor entries.
We provideguarantees in terms of ∥T − (cid:98) T ∥
F for these cases in the following subsections.
… … … … … reshape
3.3 Error Analysis of TT Cross Approximation with Exact Measurements
Assume each unfolding matrix T ⟨ k ⟩ is approximately low-rank with residual captured by ∥ T ⟨ k ⟩ −
T ⟨ k ⟩ r k ∥ F , where T ⟨ k ⟩ r k denotes the best rank- r k approximation of T ⟨ k ⟩ .
For convenience, we denote by
[Equation]
where ϵ captures the largest low-rank approximation error in all the unfolding matrices.
The work [59, 60] develops an element-wise error bound for TT cross approximation based on themaximum-volume choice of the interpolation sets.
However, if we are interested in approximationguarantee for the entire matrix (say in the Frobenius norm) instead of each entry, then directlyapplying the above result leads to a loose bound with scaling at least proportional to √ d 1 · · · d N .
Ourgoal is to provide guarantees directly for (cid:13)(cid:13)(cid:13) T − (cid:98) T (cid:13)(cid:13)(cid:13) F .
Our first main result proves the existence of stable TT cross approximation with exact measurementsof the tensor elements:
Theorem 2 Suppose T can be approximated by a TT format tensor with rank r and approximationerror ϵ defined in (5) .
For sufficiently small ϵ , there exists a cross approximation (cid:98) T of the form (4)with τ k = 0 that satisfies
[Equation]
where κ := max k (cid:8)(cid:13)(cid:13) T ⟨ k ⟩ (: , I >k ) · T ⟨ k ⟩ ( I ≤ k , I >k ) − 1 (cid:13)(cid:13) , (cid:13)(cid:13) T ⟨ k ⟩ ( I ≤ k , I >k ) − 1 · T ⟨ k ⟩ ( I ≤ k , :) (cid:13)(cid:13)(cid:9) .
We use sufficiently small ϵ to simplify the presentation.
This refers to the value of ϵ that(cid:13)(cid:13)(cid:13)(cid:16) (cid:98) T ⟨ k ⟩ (: , I >k ) − T ⟨ k ⟩ (: , I >k ) (cid:17) · T ⟨ k ⟩ ( I ≤ k , I >k ) − 1 (cid:13)(cid:13)(cid:13) ≤ κ , which can always be guaranteed sinceby (6) the distance between (cid:98) T and T (and hence the distance between (cid:98) T ⟨ k ⟩ (: , I >k ) and T ⟨ k ⟩ (: , I >k ) )approaches to zero when ϵ goes to zero.
The full proof is provided in Appendix C.
On one hand, the right hand side of (6) grows only linearly with the approximation error ϵ , witha scalar that increases only logarithmically in terms of the order N .
This shows the TT crossapproximation can be stable for well selected interpolation sets.
On the other hand, Theorem 2 onlyshows the existence one such stable cross approximation, but it does not specify which one.
As aconsequence, the result may not hold for the nested sampling strategy illustrated in Figure 2 and theparameter κ maybe out of one’s control.
The following result addresses these issues by providingguarantees for any cross approximation.
Theorem 3 Suppose T can be approximated by a TT format tensor with rank r and approximationerror ϵ defined in (5) .
Let T ⟨ k ⟩ r k be the best rank r k approximation of the k -th unfolding matrix T ⟨ k ⟩and T ⟨ k ⟩ r k = W ⟨ k ⟩ r k Σ ⟨ k ⟩ r k V ⟨ k ⟩ r k T be its compact SVD.
For any interpolation sets { I ≤ k , I >k } such thatrank( T ⟨ k ⟩ r k ( I ≤ k , I >k )) =
r k , k = 1 , . . .
, N − 1 , denote bya =
[Equation]
,...,N
[Equation]
Then the cross approximation (cid:98) T in (4) with appropriate thresholding parameters τ k for the truncatedpseudo-inverse satisfies
[Equation]
where ≲ means less than or equal to up to some universal constant.
The proof of Theorem 3 is given in Appendix D. As in (6), the right hand side of (7) scales onlypolynomially in ϵ and logarithmically in terms of the order N .
Compared with (6), the guaranteein (7) holds for any cross approximation, but the quality of the cross approximation depends on thetensor T as well as the selected rows and columns as reflected by the parameters a and c .
On onehand, (cid:13) (cid:13)(cid:13)
[ W ⟨ k ⟩ r k ( I ≤ k , :)] † (cid:13) (cid:13)(cid:13) and (cid:13) (cid:13)(cid:13)
[ V ⟨ k ⟩ r k ( I >k , :)] † (cid:13) (cid:13)(cid:13) can achieve their smallest possible value 1 whenthe singular vectors W ⟨ k ⟩ r k and V ⟨ k ⟩ r k are the canonical basis.
On the other hand, one may constructexamples with large (cid:13)(cid:13)(cid:13)
[ W ⟨ k ⟩ r k ( I ≤ k , :)] † (cid:13)(cid:13)(cid:13) and (cid:13)(cid:13)(cid:13) [ V ⟨ k ⟩ r k ( I >k , :)] † (cid:13)(cid:13)(cid:13) .
Nevertheless, these terms can beupper bounded by selecting appropriate rows and columns.
For example, for any orthonormal matrix
W ∈ R m × r , if we select I such that W ( I, :) has maximal volume among all | I | × r submatricesof W , then (cid:13)(cid:13) W ( I, :) † (cid:13)(cid:13) ≤ (cid:113) 1 + r ( m −| I | ) | I |− r [-+
 [57,-]+1 [56, Proposition 5.1].
We finally note that Theorem 3uses the truncated pseudo-inverse in (4) with appropriate τ k .
We also observe from the experimentsin Section 4 that the truncated pseudo-inverse could lead to better performance than the inverse (orpseudo-inverse) for TT cross approximation (4).
3.4 Error Analysis for TT Cross Approximation with Noisy Measurements
We now consider the case where the measurements of tensor elements are not exact, but noisy.
Thishappens, for example, in the measurements of quantum state where each entry of the many-bodywavefunction or density matrix can only be measured up to some statistical error that depends onthe number of repeated measurements.
In this case, the measured entries T ( I ≤ k − 1 , i k , I >k ) and[ T ( I ≤ k , I >k )]
τ k in the cross approximation (4) are noisy.
Our goal is to understand how such noisymeasurements will affect the quality of cross approximation.
To simplify the notation, we let E denote the measurement error though it has non-zero values only atthe selected interpolation sets { I ≤ k , I >k } .
Also let (cid:101) T =
T + E and (cid:101) T ⟨ k ⟩ = T ⟨ k ⟩ + E ⟨ k ⟩ , where E ⟨ k ⟩denotes the k -th unfolding matrix of the noise.
Then, the cross approximation of T ∈ R d 1 ×···× d Nwith noisy measurements, denoted by (cid:98) T , has entries given by
[Equation]
One can view (cid:98) T as a cross approximation of (cid:101) T with exact measurements of (cid:101) T .
However, we can notdirectly apply Theorem 3 since it would give a bound for ∥ (cid:101) T − (cid:98) T ∥
F and that would also depend onthe singular vectors of (cid:101) T ⟨ k ⟩ .
The following result addresses this [-issue. 8 r ′ - -  τ =
 τ =10 -7 τ-]issue and establishes a bound for∥T − (cid:98) T ∥ F .
[-=10 - (a)    22 N - -5050 100 τ =
-]
[-=
-]
Theorem 4 Suppose T can be approximated by a TT format tensor with rank r and approximationerror ϵ defined in (5) .
Let T ⟨ k ⟩ r k be the best rank r k approximation of the k -th unfolding matrix
T ⟨ k ⟩ and T ⟨ k ⟩ r k = W ⟨ k ⟩ r k Σ ⟨ k ⟩ r k V ⟨ k ⟩ r k T be its compact SVD.
Let (cid:101) T =
T + E denote the noisy tensor where E represents the measurement error.
For any interpolation sets { I ≤ k , I >k } such that rank( T [Equation] k
[Equation]
k ( I ≤ k , I >k )) =
r k , k = 1 , . . .
, N − 1 , denote by
[Equation]
Then the noisy cross approximation (cid:98) T in (8) appropriate thresholding parameters τ k for the truncatedpseudo-inverse satisfies
[Equation]
where ϵ = ϵ + ∥E∥ F . The proof of Theorem 4 is given in Appendix E. In words, Theorem 4 provides a similar guaranteeto that in Theorem 3 for TT cross approximation with noisy measurements and shows that it is alsostable with respect to measurement error.
The parameter c in Theorem 4 depends on the spectral normof [ (cid:101) T ⟨ k ⟩ ( I ≤ k , I >k )] † and is defined in this form for simplicity.
It can bounded by [ T ⟨ k ⟩ ( I ≤ k , I >k )] †and the noise and is expected to be close to [ T ⟨ k ⟩ ( I ≤ k , I >k )] † for random noise.
Numerical Experiments
In this section, we conduct numerical experiments to evaluate the performance of the cross approximation (8) with noisy measurements.
We generate an N -th order tensor T ∈ R d 1 ×···× d N approximatelyin the TT format as T =
T r + η F , where T r is in the TT format with mode ranks [ r 1 , . .
, r N − 1 ] ,which is generated from truncating a random Gaussian tensor using a sequential SVD [9], and F is arandom tensor with independent entries generated from the normal distribution.
We then normalize T r and F to unit Frobenius norm.
Thus η controls the low-rank approximation error.
During theprocedure of cross approximation, we add measurement error randomly generated from the normaldistribution to the observed entries.
As in Section 3.4, for convenience, we model the measurementerror as a tensor E .
To control the signal-to-noise level, we first normalize E to have unit Frobeniusnorm (recall that it is nonzero only on the observed entries) and then scale it by a factor µ .
Thus, wehave
[Equation] where ∥T r ∥ F = ∥F∥ F = ∥E∥ F = 1 . To simplify the selection of parameters, we let d = d 1 = · · · =d N , r = r 1 = · · · = r N − 1 , and τ = τ 1 = · · · = τ N − 1 for the cross approximation (8).
We apply the greedy restricted cross interpolation (GRCI) algorithm in [59] for choosing the interpolation sets I ≤ k and I >k , k = 1 , . . .
, N .
For convenience, we set r ′ =
r ′ 1 = · · · = r ′ N − 1 for thenumber of selected rows and columns indexed by I ≤ k
and I >k .
We measure the normalized meansquare error (MSE) between T and the cross approximation (cid:98) T by
[Equation]
In the first experiment, we set N = 9 , d = 7 , r = 2 and µ = η = 10 − 4 and compare the performanceof tensor cross approximations with different τ and r ′ .
From Fig.
3a, we observe that when r ′ isspecified as the exact rank (i.e., r ′ = r ), all of the TT cross approximations are very accurate exceptwhen τ is relatively large and thus useful information is truncated.
On the other hand, when r ′ > r ,the performance of cross approximations with the exact pseudo-inverse ( τ = 0 ) degrades.
This isbecause the intersection matrices T ( I ≤ k , I >k ) in (4) or (cid:101) T ( I ≤ k , I >k ) in (8) become ill-conditioned.
Fortunately, this issue can be addressed by using the truncated pseudo-inverse with suitable τ > 0which can make TT cross approximation stable.
In the second experiment, we test the performance of tensor cross approximations with differenttensor orders N .
For N ranging from 6 to 22 , we set all mode sizes d = 2 and µ = η = 10 − 4 .
Here d = 2 corresponds to a quantum state, and while d = 2 is small, the tensor is still huge forlarge
N (e.g., N = 22 ).
We also set all ranks r = 2 .
From Fig.
3b, we can observe that the TTcross approximation with the pseudo-inverse is stable as N increases when r ′ =
r (black curve) butbecomes unstable when r ′ >
r (blue curve).
Again, for the latter case when r ′ > r , the TT crossapproximation achieves stable performance by using the truncated pseudo-inverse.
[Figure]
In the third experiment, we demonstrate the performance for different η , which controls the levelof low-rank approximation error.
Here, we set µ = 10 − 6 , r = 2 , τ = 10 − 2 η , and consider theover-specified case where r ′ ranges from 3 to 5.
Figure 3c shows that the performance of TT crossapproximation with the truncated pseudo-inverse is stable with the increase of η and different r ′ .
In the final experiment, we demonstrate the performance of TT cross approximation for differentmeasurement error levels µ .
Similar to the third experiment, we set η = 10 − 6 , r = 2 , τ = 10 − 2 µ ,and consider the over-specified case where r ′ ranges from 3 to 5.
We observe from Fig.
3d that tensortrain cross approximation still provides stable performance with increasing µ and different r ′ .
Wefinally note that in practice r is often unknown a priori.
For this case, the results in Fig.
3 show thatwe can safely use a relatively large estimate r ′ for tensor train cross approximation with the truncatedpseudo-inverse.
Conclusion and Outlook
While TT cross approximation has achieved excellent performance in practice, existing theoreticalresults only provide element-wise approximation accuracy guarantees.
In this paper, we providethe first error analysis in terms of the entire tensor for TT cross approximation.
For a tensor inan approximate TT format, we prove that the cross approximation is stable in the sense that theapproximation error for the entire tensor does not grow exponentially with the tensor order, and weextend this guarantee to the scenario of noisy measurements.
Experiments verified our results.
An important future direction we will pursue is to study the implications of our results for quantumtomography.
In particular, we aim to find out whether our result imply that the statistical errorsencountered in quantum measurements will not proliferate if we perform cross approximation of thequantum many-body state.
Another future direction of our work is to design optimal strategies forthe selections of the tensor elements to be measured.
One possible approach is to extend methodsdeveloped for matrix cross approximation [54], and another approach is to incorporate leverage scoresampling [69–71].
It is also interesting to use local refinement to improve the performance of TTcross approximation.
We leave these explorations to future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
In the abstract, introduction, main
results, and conclusion, we explicitly state that our results are about the analysis of
tensor train cross approximation, not about the optimal sampling strategy.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This paper
focuses on the analysis of tensor train cross approximation and bridges the gap between
theory and implementation.
So no potential negative societal impact is expected of this
work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We explicitly
mention the assumptions in Theorem 2, Theorem 3 and Theorem 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] We include all the
proofs in the Appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We detailedly
describe the experimental setting in Section 4.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] We detailedly describe the experimental setting in Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] All the results are reported in terms of the accuracyof the tensor cross approximation, so error bars are not reported.
But we ran these
experiments many times and observed similar results.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite them in
Section 4.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]