Error Analysis of Tensor-Train Cross Approximation
Abstract
Tensor train decomposition is widely used in machine learning and quantum physics due to its concise representation of high-dimensional tensors, overcoming the curse of dimensionality.
Cross approximation—originally developed for representing a matrix from a set of selected rows and columns—is an efficient method for constructing a tensor train decomposition of a tensor from few of its entries.
While tensor train cross approximation has achieved remarkable performance in practical applications, its theoretical analysis, in particular regarding the error of the approximation, is so far lacking.
To our knowledge, existing results only provide element-wise approximation accuracy guarantees, which lead to a very loose bound when extended to the entire tensor.
In this paper, we bridge this gap by providing accuracy guarantees in terms of the entire tensor for both exact and noisy measurements.
Our results illustrate how the choice of selected subtensors affects the quality of the cross approximation and that the approximation error caused by model error and/or measurement error may not grow exponentially with the order of the tensor.
These results are verified by numerical experiments and may have important implications for the usefulness of cross approximations for high-order tensors, such as those encountered in the description of quantum many-body states.
Introduction
As modern big datasets are typically represented in the form of huge tensors, tensor decomposition has become ubiquitous across science and engineering applications, including signal processing and machine learning [1, 2], communication [3], chemometrics [4, 5], genetic engineering [6], and so on.
The most widely used tensor decompositions include the canonical polyadic (CP) [7], Tucker [8] and tensor train (TT) [9] decompositions.
In general, the CP decomposition provides a compact representation but comes with computational difficulties in finding the optimal decomposition for high-order tensors [10, 11].
The Tucker decomposition can be approximately computed via the higher order singular value decomposition (SVD) [12], but it is inapplicable for high-order tensors since its number of parameters scales exponentially with the tensor order.
The TT decomposition sits in the middle between CP and Tucker decompositions and combines the best of the two worlds: the number of parameters does not grow exponentially with the tensor order, and it can be approximately computed by the SVD-based algorithms with a guaranteed accuracy [9].
(See the monograph [13] for a detailed description.)
As such, TT decomposition has attracted tremendous interest over the past decade [14–26].
Notably, the TT decomposition is equivalent to the matrix product state (MPS) or matrix product operator (MPO) introduced in the quantum physics community to efficiently represent one-dimensional quantum states, where the number of particles in a quantum system determines the order of the tensor [27–29].
The use of TT decomposition may thus allow one to classically simulate a one-dimension quantum many-body system with resources polynomial in the number of particles [29], or to perform scalable quantum state tomography [30].
While an SVD-based algorithm [9] can be used to compute a TT decomposition, this requires information about the entire tensor.
However, in many practical applications such as low-rank embeddings of visual data [31], density estimation [32], surrogate visualization modeling [33], quantum state tomography [30], multidimensional harmonic retrieval [34], parametric partial differential equations [35], interpolation [36], and CMOS ring oscillators [37], one can or it is desirable to only evaluate a small number of elements of a tensor in practice, since a high-order tensor has exponentially many elements.
To address this issue, the work [38] develops a cross approximation technique for efficiently computing a TT decomposition from selected subtensors.
Originally developed for matrices, cross approximation represents a matrix from a selected set of its rows and columns, preserving structure such as sparsity and non-negativity within the data.
Thus, it has been widely used for data analysis [39, 40], subspace clustering [41], active learning [42], approximate SVDs [43, 44], data compression and sampling [45–47], and it has been extended for tensor decompositions [38, 48–52].
See Sectionfor a detailed description of cross approximation for matrices and Section 3 for its extension to TT decomposition.
Although the theoretical analysis and applications of matrix cross approximation [53–64] have been developed for last two decades, to the best of our knowledge, the theoretical analysis of TT cross approximation, especially regarding its error bounds, has not been well studied [65, 66].
Although cross approximation gives an accurate TT decomposition when the tensor is exactly in the TT format [38], the question remains largely unsolved as to how good the approximation is when the tensor is only approximately in the TT format, which is often the case for practical data.
In particular, an important question is whether the approximation error grows exponentially with the order due to the cascaded multiplications between the factors; this exponential error growth is proved to occur with the Tucker decomposition [48, 49].
Fortunately, for TT cross approximation, the works [65, 66] establish element-wise error bounds that do not grow exponentially with the order.
Unfortunately, when one extends these element-wise results to the entire tensor, the Frobenius norm error bound grows exponentially with the order of the tensor, indicating that the cross approximation could essentially fail for high-order tensors.
However, in practice cross approximation is often found to work well even for high-order tensors, leading us to seek for tighter error bounds on the entire tensor.
Specifically, we ask the following main question:
Question : Does cross approximation provide a stable tensor train (TT) decomposition of a high-order tensor with accuracy guaranteed in terms of the Frobenius norm?
In this paper, we answer this question affirmatively by developing accuracy guarantees for tensors that are not exactly in the TT format.
Our results illustrate how the choice of subtensors affects the quality of the approximation and that the approximation error need not grow exponentially with the order.
We also extend the approximation guarantee to noisy measurements.
The rest of this paper is organized as follows.
In Section 2, we review and summarize existing error bounds for matrix cross approximation.
Section 3 introduces our new error bounds for TT cross approximation.
Section 4 includes numerical simulation results that supports our new error bounds and Section 5 concludes the paper.
Notation : We use calligraphic letters (e.g., A ) to denote tensors, bold capital letters (e.g., A ) to denote matrices, bold lowercase letters (e.g., a ) to denote vectors, and italic letters (e.g., a ) to denote scalar quantities.
A − 1 and A † represent the inverse and the Moore–Penrose pseudoinverse matrices of A , respectively.
Elements of matrices and tensors are denoted in parentheses, as in Matlab notation.
For example, A ( i 1 , i 2 , i 3 ) denotes the element in position ( i 1 , i 2 , i 3 ) of the order-3 tensor A ,
A ( i 1 , :) denotes the i 1 -th row of A , and A (
Cross Approximation for Matrices
As a tensor can be viewed as a generalization of a matrix, it is instructive to introduce the principles behind cross approximation for matrices.
Cross approximation, also known as skeleton decomposition or CUR decomposition , approximates a low-rank matrix using a small number of its rows and columns.
Formally, following the conventional notation for the CUR decomposition, we construct a cross approximation for any matrix A ∈ R m × n as
[Equation]
where † denotes the pseudoinverse (not to be confused with the Hermitian conjugate widely used in quantum physics literature),
C =
A (: , J ) , U =
A ( I, J ) , and R =
A ( I, :) with I ∈ [ m ] and
J ∈ [ n ] denoting the indices of the selected rows and columns, respectively.
See Figure 1 for an illustration of matrix cross approximation.
While the number of selected rows and columns (i.e., | I | and | J | ) can be different, they are often the same since the rank of CU † R is limited by the smaller number if they are different.
When U is square and non-singular, the cross approximation (1) takes the standard form A ≈ CU − 1 R .
Without loss of generality, we assume an equal number of selected rows and columns in the sequel.
We note that one may replace U † in (1) by C † AR † to improve the approximation, resulting in the CUR approximation CC † AR † R , where CC † and R † R are orthogonal projections onto the subspaces spanned by the given columns and rows, respectively [67, 61].
However, such a CUR approximation depends on knowledge of the entire matrix A .
As we mainly focus on scenarios where such knowledge is prohibitive, in the sequel we only consider the cross approximation (1) with U
=
A ( I, J ) .
While in general the singular value decomposition (SVD) gives the best low-rank approximation to a matrix in terms of Frobenius norm or spectral norm, the cross approximation shown in Figure 1 has several notable advantages: ( i ) it is more interpretable since it is constructed with the rows and columns of the matrix, and hence could preserve structure such as sparsity, nonnegativity, etc., and ( ii ) it only requires selected rows and columns while other factorizations often require the entire matrix.
As such, cross approximation has been widely used for data analysis [39, 40], subspace clustering [41], active learning [42], approximate SVDs [43, 44], data compression and sampling [45–47], and so on; see [61, 62] for a detailed history of the use of cross approximation.
The following result shows that the cross approximation is exact (i.e., (1) becomes equality) as long as the submatrix U has the same rank as A .
Theorem 1 For any A ∈ R m × n , suppose we select rows and columns C
=
A (: , J ) , U =
A ( I, J ) ,
R =
A ( I, :) from A such that rank( U ) =
rank( A ) .
Then
[Equation]
Proof As R is a submatrix of A and U is a submatrix of R , rank( U ) ≤ rank( R ) ≤ rank( A ) always holds.
Now if rank( U ) = rank( A ) , we have rank( U ) =
rank( R ) and thus any columns in R can be linearly represented by the columns in U .
Using this observation and noting that
U = UU † U , we can obtain that R = UU † R .
Likewise, since rank( R ) =
rank( A ) , any rows in
A can be linearly represented by the rows in R .
Thus, we can conclude that A = CU † R .
In words, Theorem 1 implies that for an exactly low-rank matrix A with rank r , we can sample r (or more) rows and columns to ensure exact cross approximation.
In practice, however, A is generally not exactly low-rank but only approximately low-rank.
In this case, the performance of cross approximation depends crucially on the selected rows and columns.
Column subset selection is a classical problem in numerical linear algebra, and many methods have been proposed based on the maximal volume principle [68, 69] or related ideas [70–72]; see [72] for more references representing this research direction.
Efforts have also been devoted to understand the robustness of cross approximation since it only yields a low-rank approximation to A when A is not low-rank.
The work [53] provides an elementwise accuracy guarantee for the rows and columns selected via the maximal volume principle.
The recent work [58] shows there exist a stable cross approximation with accuracy guaranteed in the
Frobenius norm, while the work [59] establishes a similar guarantee for cross approximations of random matrices using the maximum projective volume principle.
A derandomized algorithm is proposed in [60] that finds a cross approximation with a similar performance guarantee.
The work [62] establishes an approximation guarantee for cross approximation using any sets of selected rows and columns.
See Appendix A for a detailed description of these results.
Cross Approximation for Tensor Train Decomposition
In this section, we consider cross approximation for tensor train (TT) decomposition.
We begin by introducing tensor notations.
For an N -th order tensor T ∈ R d 1 ×···× d N , the k -th separation or unfolding of T , denoted by T ⟨ k ⟩ , is a matrix of size ( d 1 · · · d k ) ×
( d k +1 · · · d N ) with its ( i 1 . . .
i k , i k +1 · · · i N ) -th element defined by
[Equation]
Here, i 1 · · · i k is a single multi-index that combines indices i 1 , . . .
, i k .
As for the matrix case, we will select rows and columns from T ⟨ k ⟩ to construct a cross approximation for all k = 1 , . . .
, N − 1 .
To simplify the notation, we use I ≤ k and I >k to denote the positions of the selected r ′ k rows and columns in the k -th unfolding.
With abuse of notation, I ≤ k and I >k are also used as the positions in the original tensor.
Thus, both T ⟨ k ⟩ ( I ≤ k , I >k ) and T ( I ≤ k , I >k ) represent the sampled r ′ k × r ′ k intersection matrix.
Likewise, T ( I ≤ k − 1 , i k , I >k ) represents an r ′ k − 1 × r ′ k matrix whose ( s, t ) -th element is given by T ( I ≤ k − 1 s , i k , I >kt ) for s = 1 , . . .
, r ′ k − 1 and t = 1 , . . .
, r ′ k .
3.1 Tensor Train Decomposition
We say that T ∈ R d 1 ×···× d N is in the TT format if we can express its ( i 1 , . .
, i N ) -th element as the following matrix product form [9]
[Equation]
where X j ∈ R r j − 1 × d j × r j , j = 1 , . . .
, N with r 0 = r N = 1 and X j (: , i j , :)
∈ R r j − 1 × r j , j = 1 , . . .
, N denotes one “slice” of X j with the second index being fixed at i j .
To simplify notation, we often write the above TT format in short as T =
[ X 1 , . . .
, X N ] .
In quantum many-body physics, (3) is equivalent to a matrix product state or matrix product operator, where T ( i 1 , . .
, i N ) denotes an element of the many-body wavefunction or density matrix, N is the number of spins/qudits, and r k ( k = 1 , 2 , · · · , N − 1) are known as the bond dimensions.
Note that in general the decomposition (3) is not invariant to dimensional permutation due to strictly sequential multilinear products over the latent cores.
The tensor ring decomposition generalizes (3) by taking the trace of the RHS; this allows r 0 , r N ≥ 1 and is invariant to circular dimensional permutation [73].
While there may exist infinitely many ways to decompose a tensor T as in (3), we say the decomposition is minimal if the rank of the left unfolding of each X k (i.e., L ( X k ) =  
X k (: , 1 , :) ...
X k (: , d k , :) 
 ) is r k and the rank of the right unfolding of each X k (i.e., R ( X k ) = [ X k (: , 1 , :) · · ·
X k (: , d k , :)] ) is r k − 1 .
The dimensions r = ( r 1 , . . .
, r N − 1 ) of such a minimal decomposition are called the TT ranks of T .
According to [74], there is exactly one set of ranks r that T admits a minimal TT decomposition.
Moreover, r k equals the rank of the k -th unfolding matrix T ⟨ k ⟩ , which can serve as an alternative way to define the TT rank.
Efficiency of Tensor Train Decomposition
The number of elements in the N -th order tensor
T grows exponentially in N , a phenomenon known as the curse of dimensionality .
In contrast, a TT decomposition of the form (3) can represent a tensor T with d 1 · · · d N elements using only
O ( dNr 2 ) elements, where d = max { d 1 , . . .
d N } and r = max { r 1 , . .
, r N − 1 } .
This makes the
TT decomposition remarkably efficient in addressing the curse of dimensionality as its number of parameters scales only linearly in N .
For this reason, the TT decomposition has been widely used for image compression [14, 15], analyzing theoretical properties of deep networks [16], network compression (or tensor networks) [17–22], recommendation systems [23], probabilistic model estimation [24], learning of Hidden Markov Models [25], and so on.
The work [26] presents a library for TT decomposition based on TensorFlow.
Notably, TT decomposition is equivalent to the matrix product state (MPS) introduced in the quantum physics community to efficiently and concisely represent quantum states; here N represents the number of qubits of the many–body system [27–29].
The concise representation by MPS is remarkably useful in quantum state tomography since it allows us to observe a quantum state with both experimental and computational resources that are only polynomial rather than exponential in the number of qubits N [30].
Comparison with Other Tensor Decompositions
Other commonly used tensor decompositions include the Tucker decomposition, CP decomposition, etc.
The Tucker decomposition is suitable only for small-order tensors (such as N = 3 ) since its parameterization scales exponentially with N [8].
Like the TT decomposition, the canonical polyadic (CP) decomposition [7] also represents a tensor with a linear (in N ) number of parameters.
However, TT has been shown to be exponentially more expressive than CP for almost any tensor [16, 75].
Moreover, computing the CP decomposition is computationally challenging; even computing the canonical rank can be NP-hard and ill-posed [10, 11].
In contrast, one can use a sequential SVD-based algorithm [9] to compute a quasioptimal
TT decomposition (the accuracy differs from that of the best possible approximation by at most a factor of √ N ).
We refer to [13] for a detailed comparison.
3.2 Cross Approximation For Tensor Train Decomposition
While the SVD-based algorithm [9] can be used to find a TT decomposition, that algorithm requires the entire tensor and is inefficient for large tensor order N .
More importantly, in applications like lowrank embeddings of visual data [31], density estimation [32], surrogate visualization modeling [33], and quantum state tomography [30], it is desirable to measure a small number of tensor elements since the experimental cost is proportional to the number of elements measured.
In such scenarios, we can use TT cross approximation to reconstruct the full tensor using only a small number of known elements, similar to the spirit of matrix cross approximation.
To illustrate the main ideas behind TT cross approximation, we first consider the case where a tensor
T is in the exact TT format (3), i.e., the unfolding matrix T ⟨ k ⟩ is exactly low-rank with rank r k .
In this case, by Theorem 1, we can represent T ⟨ 1 ⟩ with a cross approximation by choosing r ′ 1 rows (with r ′ 1 ≥ r 1 ) indexed by I ≤ 1 and r ′ 1 columns indexed by I > 1 such that
[Equation]
… … … … … reshape
Here T ⟨ 1 ⟩ (: , I > 1 ) has d 1 r ′entries, while T ⟨ 1 ⟩ ( I ≤ 1 , :) is an extremely wide matrix.
However, one only needs to sample the elements in T ⟨ 1 ⟩ (: , I > 1 ) and can avoid sampling the entire elements of T ⟨ 1 ⟩ ( I ≤ 1 , :) by further using cross approximation .
In particular, one can reshape T ⟨ 1 ⟩ ( I ≤ 1 , :) into another matrix of size r ′ 1 d 2 × d 3 · · · d N and then represent it with a cross approximation by choosing r ′ 2 rows and columns.
This procedure can be applied recursively N −times, and only at the last stage does one sample both r ′ N − 1 rows and columns of an r ′ N − 2 d N − 1 × d N matrix.
This is the procedure developed in [38] for TT cross approximation.
The process of TT cross approximation is shown in Fig.
The procedure illustrated in Figure 2 produces nested interpolation sets as one recursively applies cross approximation on the reshaped versions of previously selected rows.
However, the interpolation sets { I ≤ k , I >k } are not required to be nested.
In general, one can slightly generalize this procedure and obtain a TT cross approximation (cid:98) T with elements given by [66]
[Equation]
where I ≤ k denotes the selected row indices from the multi-index { d 1 · · · d k } , I >k is the selection of the column indices from the multi-index { d k +1 · · · d N } , and I ≤ 0 =
I >N = ∅ .
Here A † τ denotes the τ -pseudoinversion of A that sets the singular values less than τ to zero before calculating the pseudoinversion.
Similar to Theorem 1, the TT cross approximation (cid:98) T equals T when T is in the TT format (3) and the intersection matrix T ( I ≤ k , I >k ) has rank r k for k = 1 , . .
. , N − 1 [38].
Nevertheless, in practical applications, tensors may be only approximately but not exactly in the TT format (3), i.e., the unfolding matrices T ⟨ k ⟩ may be only approximately low-rank.
Moreover, in applications such as quantum tomography, one can only observe noisy samples of the tensor entries.
We provide guarantees in terms of ∥T − (cid:98) T ∥
F for these cases in the following subsections.
3.3 Error Analysis of TT Cross Approximation with Exact Measurements
Assume each unfolding matrix T ⟨ k ⟩ is approximately low-rank with residual captured by ∥ T ⟨ k ⟩ −
T ⟨ k ⟩ r k ∥ F , where T ⟨ k ⟩ r k denotes the best rank- r k approximation of T ⟨ k ⟩ .
For convenience, we define
[Equation]
where ϵ captures the largest low-rank approximation error in all the unfolding matrices.
The work [65, 66] develops an element-wise error bound for TT cross approximation based on the maximum-volume choice of the interpolation sets.
However, if we are interested in an approximation guarantee for the entire tensor (say in the Frobenius norm) instead of each entry, then directly applying the above result leads to a loose bound with scaling at least proportional to √ d 1 · · · d N .
Our goal is to provide guarantees directly for ∥T − (cid:98) T ∥ F .
Note that (5) implies that a fundamental lower bound ∥T [Equation]
ϵ since ∥T
[Equation]
holds for any TT format tensor (cid:98) T with ranks r (not only those constructed by cross approximation).
Fortunately, tensor models in practical applications such as image and video [14, 15] and quantum states [27–29] can be well represented in the TT format with very small ϵ .
Our first main result proves the existence of stable TT cross approximation with exact measurements of the tensor elements:
Theorem 2 Suppose T can be approximated by a TT format tensor with ranks r and approximation error ϵ defined in (5) .
For sufficiently small ϵ , there exists a cross approximation (cid:98) T of the form (4) with τ k = 0 that satisfies
[Equation]
where κ := max k [Equation] T ⟨ k ⟩ ( I ≤ k , I >k ) − 1 · T ⟨ k ⟩ ( I ≤ k , :) (cid:13)(cid:13)(cid:9) .
We use sufficiently small ϵ to simplify the presentation.
This refers to the value of ϵ that (cid:13) (cid:13) (cid:13)(cid:16) (cid:98) T ⟨ k ⟩ (: , I >k ) − T ⟨ k ⟩ (: , I >k ) (cid:17) · T ⟨ k ⟩ ( I ≤ k , I >k ) − 1 (cid:13) (cid:13) (cid:13) ≤ κ , which can always be guaranteed since by (7) the distance between (cid:98) T and T (and hence the distance between (cid:98) T ⟨ k ⟩ (: , I >k ) and T ⟨ k ⟩ (: , I >k ) ) approaches zero when ϵ goes to zero.
The full proof is provided in Appendix B.
On one hand, the right hand side of (7) grows only linearly with the approximation error ϵ , with a scalar that increases only logarithmically in terms of the order N .
This shows the TT cross approximation can be stable for well selected interpolation sets.
Recall that by (6), for any cross approximation (cid:98) T of the form (4) with r k selected rows and columns in the sets { I ≤ k , I >k } , ∥T − (cid:98) T ∥
F ≥ ϵ necessarily holds since (cid:98) T ⟨ k ⟩ has rank at most r k .
While it may be unfair to compare the element-wise error bounds [65, 66], we present [65, Theorem 1] below which also involves ϵ and a similar quantity to κ : ∥T − (cid:98) T ∥ max ≤ (2 r + r (cid:101) κ + 1) ⌈ log 2 N ⌉ ( r + 1) ϵ, where ∥T ∥ max = max i 1 ,...,i N |T ( i 1 , . . .
, i N ) | and (cid:101) κ = max k r k ∥ T ⟨ k ⟩ ∥ max · ∥ T ⟨ k ⟩ ( I ≤ k , I >k ) − 1 ∥ max .
Loosely speaking, κ and (cid:101) κ can be approximately viewed as the condition number of the submatrix T ⟨ k ⟩ ( I ≤ k , I >k ) with respect to the spectral norm and the Chebyshev norm, respectively.
Again, we note that our bound (7) is for the entire tensor, while ∥T − (cid:98) T ∥ max only concerns the largest element-wise error.
On the other hand, Theorem 2 only shows the existence of one such stable cross approximation, but it does not specify which one.
As a consequence, the result may not hold for the nested sampling strategy illustrated in Figure 2 and the parameter κ maybe out of one’s control.
The following result addresses these issues by providing guarantees for any cross approximation.
Theorem 3 Suppose T can be approximated by a TT format tensor with rank r and approximation error ϵ defined in (5) .
Let T ⟨ k ⟩ r k be the best rank r k approximation of the k -th unfolding matrix T ⟨ k ⟩ and T ⟨ k ⟩ r k = W ( k ) Σ ( k ) V ( k ) T be its compact SVD.
For any interpolation sets { I ≤ k , I >k } such that rank( T ⟨ k ⟩ r k ( I ≤ k , I >k )) =
r k , k = 1 , . . .
, N − 1 , denote by a =
[Equation]
Then the cross approximation (cid:98) T in (4) with appropriate thresholding parameters τ k for the truncated pseudo-inverse satisfies
[Equation]
where ≲ means less than or equal to up to some universal constant.
The proof of Theorem 3 is given in Appendix C. As in (7), the right hand side of (8) scales only polynomially in ϵ and logarithmically in terms of the order N .
Compared with (7), the guarantee in (8) holds for any cross approximation, but the quality of the cross approximation depends on the tensor T as well as the selected rows and columns as reflected by the parameters a and c .
On one hand, (cid:13)(cid:13)
[ W ( k ) ( I ≤ k , :)] † (cid:13)(cid:13) and (cid:13)(cid:13)
[ V ( k ) ( I >k , :)] † (cid:13)(cid:13) can achieve their smallest possible value 1 when the singular vectors W ( k ) and V ( k ) are the canonical basis.
On the other hand, one may construct examples with large (cid:13)(cid:13)
[ W ( k ) ( I ≤ k , :)] † (cid:13)(cid:13) and (cid:13)(cid:13)
[ V ( k ) ( I >k , :)] † (cid:13)(cid:13) .
Nevertheless, these terms can be upper bounded by selecting appropriate rows and columns.
For example, for any orthonormal matrix
W ∈ R m × r , if we select I such that W ( I, :) has maximal volume among all | I | × r submatrices of
W , then (cid:13)(cid:13) W ( I, :) † (cid:13)(cid:13) ≤ (cid:113) 1 + r ( m −| I | ) | I |− r
+
[62, Proposition 5.1].
In practice, without knowing W ( k ) and V ( k ) , we may instead find the interpolation sets by maximizing the volume of T ⟨ k ⟩ ( I ≤ k , I >k ) using algorithms such as greedy restricted cross interpolation (GRCI) [65].
We may also exploit derandomized algorithms [60] to find interpolation sets that could potentially yield small quantities a, b, c .
We finally note that Theorem 3 uses the truncated pseudo-inverse in (4) with appropriate τ k .
We also observe from the experiments in Section 4 that the truncated pseudo-inverse could lead to better performance than the inverse (or pseudo-inverse) for TT cross approximation (4).
3.4 Error Analysis for TT Cross Approximation with Noisy Measurements
We now consider the case where the measurements of tensor elements are not exact, but noisy.
This happens, for example, in the measurements of a quantum state where each entry of the many-body wavefunction or density matrix can only be measured up to some statistical error that depends on the number of repeated measurements.
In this case, the measured entries T ( I ≤ k − 1 , i k , I >k ) and [ T ( I ≤ k , I >k )]
τ k in the cross approximation (4) are noisy.
Our goal is to understand how such noisy measurements will affect the quality of cross approximation.
To simplify the notation, we let E denote the measurement error though it has non-zero values only at the selected interpolation sets { I ≤ k , I >k } .
Also let (cid:101) T =
T + E and (cid:101) T ⟨ k ⟩ = T ⟨ k ⟩ + E ⟨ k ⟩ , where E ⟨ k ⟩ denotes the k -th unfolding matrix of the noise.
Then, the cross approximation of T ∈ R d 1 ×···× d N with noisy measurements, denoted by (cid:98) T , has entries given by
[Equation]
One can view (cid:98) T as a cross approximation of (cid:101) T with exact measurements of (cid:101) T .
However, we cannot directly apply Theorem 3 since it would give a bound for ∥ (cid:101) T − (cid:98) T ∥
F and that would also depend on the singular vectors of (cid:101) T ⟨ k ⟩ .
The following result addresses this issue.
[-8 r ′ - -  τ =
-]
[-=10 - (a)    22 N - -5050 100 τ =
-]
[-=
-]
Theorem 4 Suppose T can be approximated by a TT format tensor with rank r and approximation error ϵ defined in (5) .
Let T ⟨ k ⟩ r k be the best rank r k approximation of the k -th unfolding matrix
T ⟨ k ⟩ and T ⟨ k ⟩ r k = W ( k ) Σ ( k ) V ( k ) T be its compact SVD.
Let (cid:101) T =
T + E denote the noisy tensor where E represents the measurement error.
For any interpolation sets { I ≤ k , I >k } such that rank( T ⟨ k ⟩ r k ( I ≤ k , I >k )) =
r k , k = 1 , . . .
, N − 1 , denote by a =
[Equation]
Denote by ϵ = ϵ + ∥E∥ F .
Then the noisy cross approximation (cid:98) T in (9) with appropriate thresholding parameters τ k for the truncated pseudo-inverse satisfies
[Equation]
The proof of Theorem 4 is given in Appendix D. In words, Theorem 4 provides a similar guarantee to that in Theorem 3 for TT cross approximation with noisy measurements and shows that it is also stable with respect to measurement error.
The parameter c in Theorem 4 depends on the spectral norm of [ (cid:101) T ⟨ k ⟩ ( I ≤ k , I >k )] † and is defined in this form for simplicity.
It can be bounded by [ T ⟨ k ⟩ ( I ≤ k , I >k )] † and the noise and is expected to be close to [ T ⟨ k ⟩ ( I ≤ k , I >k )] † for random noise.
We finally note that the measurement error E only has O ( Nr 2 ) non-zero elements, and thus ∥E∥ F is not exponentially large in terms of N .
Numerical Experiments
In this section, we conduct numerical experiments to evaluate the performance of the cross approximation (9) with noisy measurements.
We generate an N -th order tensor T ∈ R d 1 ×···× d N approximately in the TT format as T =
T r + η F , where T r is in the TT format with mode ranks [ r 1 , . .
, r N − 1 ] , which is generated from truncating a random Gaussian tensor using a sequential SVD [9], and
F is a random tensor with independent entries generated from the normal distribution.
We then normalize T r and F to unit Frobenius norm.
Thus η controls the low-rank approximation error.
To simplify the selection of parameters, we let d = d 1 = · · · = d N = 2 , r = r 1 = · · · = r N − 1 , and τ = τ 1 = · · · = τ N − 1 for the cross approximation (9).
We apply the greedy restricted cross interpolation (GRCI) algorithm in [65] for choosing the interpolation sets I ≤ k and I >k , k = 1 , . . .
, N .
For convenience, we set r ′ =
r ′ 1 = · · · = r ′ N − 1 for the number of selected rows and columns indexed by I ≤ k
and I >k .
During the procedure of cross approximation, for each of the ( N − 2) dr ′ 2 + 2 dr ′ − r ′ 2 observed entries, we add measurement error randomly generated from the Gaussian distribution of mean 0 and variance [Equation]
As in Section 3.4, for convenience, we model the measurement error as a tensor E such that E ∥E∥ 2 F = 1 .
To control the signal-to-noise level, we scale the noise by a factor µ .
Thus, we have
[Equation]
where ∥T r ∥ 2 F = ∥F∥ 2 F = E ∥E∥ 2 F = 1 .
We measure the normalized mean square error (MSE) between T and the cross approximation (cid:98) T by
[Equation]
{+8 r ′ - -  τ = τ =10τ =10 - (a)    22 N - -    τ =
+}
( r '=2) τ
{+=
+}
( r '=4) τ
{+=10 -7 ( r '=4) (b) 1010 - 1010 - η - - - - r'=3 r'= r'= (c) 1010 - 1010 - µ - - - - - r'=3 r'= r'= (d)      
+}
r - - - - r'=8 r'= r'= (e) 5 10 15 20 25 30 35 40 45 50 N - -  τ =0, r'= 30 log 2 N-1 × 10 - (f)
In the first experiment, we set N = 10 , d = 2 , r = 2 and µ = η = 10 − 5 and compare the performance of tensor cross approximations with different τ and r ′ .
From Fig.
3a, we observe that when r ′ is specified as the exact rank (i.e., r ′ = r ), all of the TT cross approximations are very accurate except when τ is relatively large and thus useful information is truncated.
On the other hand, when r ′ > r , the performance of cross approximations with the exact pseudo-inverse ( τ = 0 ) degrades.
This is because the intersection matrices T ( I ≤ k , I >k ) in (4) or (cid:101) T ( I ≤ k , I >k ) in (9) become ill-conditioned.
Fortunately, since the last r ′ − r singular values of T ( I ≤ k , I >k ) and (cid:101) T ( I ≤ k , I >k ) only capture information about the approximation error and measurement error, this issue can be addressed by using the truncated pseudo-inverse with suitable τ > 0 which can make TT cross approximation stable.
We notice that larger r ′ gives better cross approximation due to more measurements.
In the second experiment, we test the performance of tensor cross approximations with different tensor orders N .
For N ranging from 6 to 22 , we set all mode sizes d = 2 and µ = η = 10 − 5 .
Here d = 2 corresponds to a quantum state, and while d = 2 is small, the tensor is still huge for large N (e.g., N = 22 ).
We also set all ranks r = 2 .
From Fig.
3b, we can observe that the TT cross approximation with the pseudo-inverse is stable as N increases when r ′ =
r (black curve) but becomes unstable when r ′ >
r (blue curve).
Again, for the latter case when r ′ > r , the TT cross approximation achieves stable performance by using the truncated pseudo-inverse.
In the third experiment, we demonstrate the performance for different η , which controls the level of low-rank approximation error.
Here, we set µ = 10 − 7 , r = 2 , τ = 10 − 2 η , and consider the over-specified case where r ′ ranges from 3 to 5.
Fig.
3c shows that the performance of TT cross approximation with the truncated pseudo-inverse is stable with the increase of η and different r ′ .
In the fourth experiment, we demonstrate the performance for different measurement error levels µ .
Similar to the third experiment, we set η = 10 − 7 , r = 2 , τ = 10 − 2 µ , and consider the over-specified case where r ′ ranges from 3 to 5.
We observe from Fig.
3d that tensor train cross approximation still provides stable performance with increasing µ and different r ′ .
In the fifth experiment, we test the performance for different r .
We set µ = η = 10 − 5 , d = 2 , τ = 10 − 2 η and consider the case where r ′ ranges from 8 to 10 .
Fig.
3e shows that the peformance of
TT cross approximation is stable with the increase of r , with recovery error in the curves roughly increasing as O ( r 3 ) which is consistent with Theorem 4.
Note that in practice r is often unknown a priori.
For this case, the results in the above figures show that we can safely use a relatively large estimate r ′ for tensor train cross approximation with the truncated pseudo-inverse.
In the final experiment, we test the performance with further larger tensor orders N .
To achieve this goal, we work on tensors in the exact TT format (i.e., η = 0 ) since in this case we only need to store the tensor factors with O ( Nr 2 ) elements rather than the entire tensor.
We set r ′ =
r = 2 , µ = 10 −and consider one case where N varies from 5 to 50 .
We observe from Fig.
3f that the approximation error curve increases only logarithmically in terms of the order N , which is consistent with (10).
Conclusion and Outlook
While TT cross approximation has achieved excellent performance in practice, existing theoretical results only provide element-wise approximation accuracy guarantees.
In this paper, we provide the first error analysis in terms of the entire tensor for TT cross approximation.
For a tensor in an approximate TT format, we prove that the cross approximation is stable in the sense that the approximation error for the entire tensor does not grow exponentially with the tensor order, and we extend this guarantee to the scenario of noisy measurements.
Experiments verify our results.
An important future direction we will pursue is to study the implications of our results for quantum tomography.
In particular, we aim to find out whether our results imply that the statistical errors encountered in quantum measurements will not proliferate if we perform cross approximation of the quantum many-body state.
Another future direction of our work is to design optimal strategies for the selections of the tensor elements to be measured.
One possible approach is to extend methods developed for matrix cross approximation [60], and another approach is to incorporate leverage score sampling [76–78].
It is also interesting to use local refinement to improve the performance of TT cross approximation.
We leave these explorations to future work.
Acknowledgment
We acknowledge funding support from NSF Grants
No.
CCF-1839232, PHY-2112893, CCFand CCF-2106881, as well as the W. M. Keck Foundation.
We also thank the four anonymous reviewers for their constructive comments.