We give a new approach to the dictionary learning (also known as "sparse coding") problem of recovering an unknown n × m matrix A (for m n) from examples of the form y = Ax + e, where x is a random vector in R m with at most τ m nonzero coordinates, and e is a random noise vector in R n with bounded magnitude. For the case m = O(n), our algorithm recovers every column of A within arbitrarily good constant accuracy in time m O(log m/ log(τ −1 )) , in particular achieving polynomial time if τ = m −δ for any δ > 0, and time m O(log m) if τ is (a sufficiently small) constant. Prior algorithms with comparable assumptions on the distribution required the vector x to be much sparser-at most √ n nonzero coordinates-and there were intrinsic barriers preventing these algorithms from applying for denser x. We achieve this by designing an algorithm for noisy tensor decomposition that can recover, under quite general conditions, an approximate rank-one decomposition of a tensor T , given access to a tensor T that is τ -close to T in the spectral norm (when considered as a matrix). To our knowledge, this is the first algorithm for tensor decomposition that works in the constant spectral-norm noise regime, where there is no guarantee that the local optima of T and T have similar structures. Our algorithm is based on a novel approach to using and analyzing the Sum of Squares semidefinite programming hierarchy (Parrilo 2000, Lasserre 2001), and it can be viewed * A full version of this paper is available at http://arxiv.org/abs/1312.6652 † Supported by NSF grant 1111109. ‡ Supported by a Microsoft Research Fellowship, a Alfred P. Sloan Fellowship, an NSF CAREER award, and the Simons Collaboration for Algorithms and Geometry. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. as an indication of the utility of this very general and powerful tool for unsupervised learning problems.
A sum of squares approach is proposed for dictionary learning in REF .
Dictionary Learning and Tensor Decomposition via the Sum-of-Squares Method
STOC '15
Mathematics Computer Science