Paper:
Scale Up Nonlinear Component Analysis with Doubly Stochastic Gradients
Nonlinear component analysis such as kernel Principle Component Analysis (KPCA) and kernel Canonical Correlation Analysis (KCCA) are widely used in machine learning, statistics and data analysis, but they cannot scale up to big datasets. Recent attempts have employed random feature approximations to convert the problem to the primal form for linear computational complexity. However, to obtain high quality solutions, the number of random features should be the same order of magnitude as the number of data points, making such approach not directly applicable to the regime with millions of data points. We propose a simple, computationally efficient, and memory friendly algorithm based on the "doubly stochastic gradients" to scale up a range of kernel nonlinear component analysis, such as kernel PCA, CCA and SVD. Despite the non-convex nature of these problems, our method enjoys theoretical guarantees that it converges at the rateÕ(1/t) to the global optimum, even for the top k eigen subspace. Unlike many alternatives, our algorithm does not require explicit orthogonalization, which is infeasible on big datasets. We demonstrate the effectiveness and scalability of our algorithm on large scale synthetic and real world datasets.

Review:
This paper proposes a computationally efficient way of scaling kernel non-linear componentanalysis. Previous attempts of scaling kernel PCA-like eigenproblems in machine learningresorted to Nystrom-variant subsampling or random feature linearization of the problem. Each ofthese methods works effectively in certain settings but the number of samples/features may beprohibitively large in some settings.This paper proposes a doubley-stochastic gradient algorithm for solving kernel eigenvalue problems encompassing kernel PCA, CCA, and SVD. It is called doubly-stochastic because sampling is performed in the data samples and the random features (for this paper, limited tostationary kernels whose Fourier transform is well-defined). The paper claims convergence rate of $\tilde{O}(1/t)$ to the global optimum for the recovered eigensubspace (in terms of principal angle), but with a caveat being that the step sizes are chosen properly and the mini-batch size is sufficiently large.Quality: The paper is reasonably well-written with no glaring grammar errors/typos.Clarity: Clear in most parts, some suggestions for improvement: - In Line 300: "which outside the RKHS" -> "which is outside the RKHS" - Please provide the citation for the "median trick" - just to make the paper self-contained. - Please provide the dimensionality of the datasets used in the experiment. For example, the synthetic dataset is mentioned without such information. - Figure 1 and Figure 2 and Figure 3 are hard to look at, when printed on the paper. - Equation (6), Line 305: the Part II should read $ 2 | \tilde{g}_t(x) - h_t(x) |^2 $Originality: The idea of doubly-stochastic gradient descent has appeared in the convex setting where the objective function can be expressed as a convex function (kernel SVM, etc.). This paper attempts to prove the global convergence rate for the principal angle bound on the recovered kernel eigensubspace.Significance: Kernel eigenvalue problems are notoriously hard to scale and achieving high accuracy is challenging. Addressing this problem is a worthy effort.