We study " learning-based" sketching approaches for diverse tasks in numerical linear algebra: least-squares regression, \ell_p regression, Huber regression, low-rank approximation (LRA), and k-means clustering. Sketching methods are used to quickly and approximately compute properties of large matrices. Linear maps called " sketches " are applied to compress data, and these concise representations are used to compute the desired properties. Specifically, we consider sparse sketches (such as CountSketch). Recent works have dealt with optimizing sketches for data distributions to perform better than their random counterparts. We extend this theme to several important and ubiquitous tasks, each of which requires a new analysis and novel practical methods. Specifically, our contributions are : 1) For all tasks, we introduce fast algorithms using learned sketches with worst-case guarantees. We give a simple task-agnostic method for retaining the worst-case guarantees of randomized sketching, which yields time-optimal algorithms for LRA and least-squares regression . Also, for k-means clustering , we give a faster alternative for retaining worst-case guarantees. 2) We show empirically that learned sketches are reliable in improving approximation accuracy, with comparison against "non-learned" sketching baselines. 3) We introduce a greedy algorithm for optimizing the location of the nonzero entries of a sparse sketch and prove guarantees for certain distributions on the LRA task. Previous work only looked at optimizing the values rather than the locations. Also, we show empirically that it further improves learned sketch performance . \\%DIF > and 20\\%, respectively. (Interestingly, we can also prove that we get a strict improvement for LRD under certain conditions.) Finally, we design two sketching algorithm modifications that leverage the strong expected performance of learned sketches, provide worst-case performance guarantees, and have the same time complexity as classical sketching. We prove the worst-case property for each of the problems and their modified algorithms.
A sketching algorithm is a way to solve an optimization problem approximately and in a fraction of the usual time. We consider classical sketching algorithms which first compress data by multiplication with a random " sketch matrix". Our work improves and extends the " learned sketch " paradigm, in which sketch matrices are optimized to yield better expected performance. This technique has only been used for a suboptimal variant of sketched low-rank decomposition (LRD). Our work extends the problem coverage to optimal sketched LRD, least-squares regression (LS), and k-means clustering . We improve sketch learning for all three problems and very significantly for LS and LRD: experimental performance increases by 12\\%DIF > and 20\\%, respectively. (Interestingly, we can also prove that we get a strict improvement for LRD under certain conditions.) Finally, we design two sketching algorithm modifications that leverage the strong expected performance of learned sketches, provide worst-case performance guarantees, and have the same time complexity as classical sketching. We prove the worst-case property for each of the problems and their modified algorithms.
R
We study
A sketching algorithm is a way to solve an optimization problem approximately and in a fraction of the usual time. We consider classical sketching algorithms which first compress data by multiplication with a random
R
learning-based" sketching approaches for diverse tasks in numerical linear algebra: least-squares regression, \ell_p regression, Huber regression, low-rank approximation (LRA), and k-means clustering. Sketching methods are used to quickly and approximately compute properties of large matrices. Linear maps called
sketch matrix". Our work improves and extends the
R
sketches
learned sketch
R
are applied to compress data, and these concise representations are used to compute the desired properties. Specifically, we consider sparse sketches (such as CountSketch). Recent works have dealt with optimizing sketches for data distributions to perform better than their random counterparts. We extend this theme to several important and ubiquitous tasks, each of which requires a new analysis and novel practical methods. Specifically, our contributions are : 1) For all tasks, we introduce fast algorithms using learned sketches with worst-case guarantees. We give a simple task-agnostic method for retaining the worst-case guarantees of randomized sketching, which yields time-optimal algorithms for LRA and
paradigm, in which sketch matrices are optimized to yield better expected performance. This technique has only been used for a suboptimal variant of sketched low-rank decomposition (LRD). Our work extends the problem coverage to optimal sketched LRD,
R
. Also, for
(LS), and
D
, we give a faster alternative for retaining worst-case guarantees. 2) We show empirically that learned sketches are reliable in improving approximation accuracy, with comparison against "non-learned" sketching baselines. 3) We introduce a greedy algorithm for optimizing the location of the nonzero entries of a sparse sketch and prove guarantees for certain distributions on the LRA task. Previous work only looked at optimizing the values rather than the locations. Also, we show empirically that it further improves learned sketch performance
A
We improve sketch learning for all three problems and very significantly for LS and LRD: experimental performance increases by 12