Paper:
Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passes
We consider stochastic gradient descent (SGD) for least-squares regression with potentially several passes over the data. While several passes have been widely reported to perform practically better in terms of predictive performance on unseen data, the existing theoretical analysis of SGD suggests that a single pass is statistically optimal. While this is true for low-dimensional easy problems, we show that for hard problems, multiple passes lead to statistically optimal predictions while single pass does not; we also show that in these hard models, the optimal number of passes over the data increases with sample size. In order to define the notion of hardness and show that our predictive performances are optimal, we consider potentially infinite-dimensional models and notions typically associated to kernel methods, namely, the decay of eigenvalues of the covariance matrix of the features and the complexity of the optimal predictor as measured through the covariance matrix. We illustrate our results on synthetic experiments with non-linear kernel methods and on a classical benchmark with a linear model.

Review:
This paper identifies and separates (kernel) linear least-squares regression problems wherein carrying out multiple passes of stochastic gradient descent (SGD) over a training set can yield better statistical error than only a single pass. This is relevant to the core of machine learning theory, and relates to a line of work published at NIPS, ICML, COLT, and similar conferences in the past several years about the statistical error of one-pass, many-pass, and ERM-based learning.  The authors focus on regression problems captured, by assumption, by two parameters: alpha, which governs the exponent of a power-law eigenvalue decay, and r, which governs a transformation under which the Hilbert norm of the optimal predictor is bounded. They refer to problems where r <= (alpha - 1) / (2 * alpha) as "hard". The main result of the paper is to show that for these "hard" problems, multiple SGD passes either achieve (minimax) optimal rates of statistical estimation, or at least improve the rate relative to a single pass.  The results are interesting and might address an unanswered core question in machine learning, and the mathematical presentation is clear, with assumptions upfront. What's most needed is clarification on whether this applies to core machine learning problems, and how this relates to the previous results on optimality of single-pass SGD. What problem instances are included in the assumptions and then in the different "easy" and "hard" subclasses? In what way were the "hard" problems not also solved by single-pass SGD?  Two examples are given in the paper. Example 1 is a learning problem over the one-dimensional unit interval. Example 2 states a family of learning problems for which the authors claim alpha = 2m/d and r = s/(2m), provided d < 2m (indeed if d >= 2m then alpha is nonpositive and so there is no valid r to make for a "hard problem"). These examples left me confused about the introductory claim (e.g. in the abstract) that single pass SGD is better "for low-dimensional easy problems." Both examples are learning problems over input domains of inherently low dimension.  Experimentally, least-squares regression is an interesting core problem for the NIPS community. What is a reasonable choice of the parameters alpha and r for different common datasets, or even simply MNIST? What if a feature map or kernel known to be useful for MNIST is used? The paper currently has an experiment performed on MNIST (with no feature map or kernel), showing out-of-sample error in response to number of passes. However, the experiment does not validate any of the assumptions or explain whether the observed outcome was predicted by the paper's theory.  Finally, Section 4.2 describes in its final paragraph a setting where multi-pass SGD performs as well as the empirical risk minimizer, and that both are minimax optimal. Do we know whether a single-pass cannot achieve this result as well? If so, then this paragraph seems to imply, independently of the multi-pass SGD result, a separation between single-pass SGD and empirical risk minimization, which would be a result worth highlighting or explaining on its own. If not, then seeing as how both single-pass and empirical risk minimization work, it seems less surprising that multi-pass (which is in a sense between the two) works as well. Either way, it seems important to clarify which of the two scenarios holds.  ----  Edited after author response: It appears that, overall, the author response answers/addresses most of my questions/concerns above.