Conventional algorithms for finding low-rank canonical polyadic (CP) tensor decompositions are unwieldy for large sparse tensors. The CP decomposition can be computed by solving a sequence of overdetermined least problems with special Khatri-Rao structure. In this work, we present an application of randomized numerical linear algebra to fitting the CP decomposition of sparse tensors, solving a significantly smaller sampled least squares problem at each iteration with probabilistic guarantees on the approximation errors. Prior work has shown that sketching is effective in the dense case, but the prior approach cannot be applied to the sparse case because a fast Johnson-Lindenstrauss transform (e.g., using a fast Fourier transform) must be applied in each mode, causing the sparse tensor to become dense. Instead, we perform sketching through leverage score sampling, crucially relying on the fact that the structure of the Khatri-Rao product allows sampling from overestimates of the leverage scores without forming the full product or the corresponding probabilities. Naive application of leverage score sampling is ineffective because we often have cases where a few scores are quite large, leading to repeatedly sampling the few entries with large scores. We improve the speed by combining repeated rows. Additionally, we propose a novel hybrid of deterministic and random leverage-score sampling which consistently yields improved fits . Numerical results on real-world large-scale tensors show the method is significantly faster than competing methods without sacrificing accuracy.
The low-rank canonical polyadic tensor decomposition is useful in data analysis and can be computed by solving a sequence of overdetermined least squares subproblems. Motivated by consideration of sparse tensors, we propose sketching each subproblem using leverage scores to select a subset of the rows, with probabilistic guarantees of the solution accuracy. We randomly sample rows proportional to leverage score upper bounds that can be efficiently computed using the special Khatri-Rao subproblem structure inherent in tensor decomposition. Crucially, the number of rows in the sketched system is independent of the number of nonzeros in the full tensor and the number of rows in the full system. Along the way, we provide a practical solution to the generic matrix sketching problem of sampling overabundance for high-leverage-score rows, proposing to include such rows deterministically and combine repeated samples in the sketched system; we conjecture that this can lead to improved theoretical bounds . Numerical results on real-world large-scale tensors show the method is significantly faster than deterministic methods without sacrificing accuracy.
R
Conventional algorithms for finding
The
R
(CP) tensor decompositions are unwieldy for large sparse tensors. The CP decomposition
tensor decomposition is useful in data analysis and
R
problems with special Khatri-Rao structure. In this work, we present an application of randomized numerical linear algebra to fitting the CP decomposition of
squares subproblems. Motivated by consideration of
R
solving a significantly smaller sampled least squares problem at each iteration
we propose sketching each subproblem using leverage scores to select a subset of the rows,
R
on the approximation errors. Prior work has shown that sketching is effective in the dense case, but the prior approach cannot be applied to the sparse case because a fast Johnson-Lindenstrauss transform (e.g., using a fast Fourier transform) must be applied in each mode, causing the sparse tensor to become dense. Instead, we perform sketching through leverage score sampling, crucially relying on the fact that the structure of the
of the solution accuracy. We randomly sample rows proportional to leverage score upper bounds that can be efficiently computed using the special
R
product allows sampling from overestimates of
subproblem structure inherent in tensor decomposition. Crucially,
R
leverage scores without forming the full product or the corresponding probabilities. Naive application of leverage score sampling is ineffective because we often have cases where a few scores are quite large, leading to repeatedly sampling the few entries with large scores. We improve the speed by combining repeated rows. Additionally, we propose a novel hybrid of deterministic and random leverage-score sampling which consistently yields improved fits
number of rows in the sketched system is independent of the number of nonzeros in the full tensor and the number of rows in the full system. Along the way, we provide a practical solution to the generic matrix sketching problem of sampling overabundance for high-leverage-score rows, proposing to include such rows deterministically and combine repeated samples in the sketched system; we conjecture that this can lead to improved theoretical bounds
R
competing
deterministic