Paper:
Simultaneous Model Selection and Optimization through Parameter-free Stochastic Learning
Stochastic gradient descent algorithms for training linear and kernel predictors are gaining more and more importance, thanks to their scalability. While various methods have been proposed to speed up their convergence, the model selection phase is often ignored. In fact, in theoretical works most of the time assumptions are made, for example, on the prior knowledge of the norm of the optimal solution, while in the practical world validation methods remain the only viable approach. In this paper, we propose a new kernel-based stochastic gradient descent algorithm that performs model selection while training, with no parameters to tune, nor any form of cross-validation. The algorithm builds on recent advancement in online learning theory for unconstrained settings, to estimate over time the right regularization in a data-dependent way. Optimal rates of convergence are proved under standard smoothness assumptions on the target function as well as preliminary empirical results.

Review:
Added post rebuttal: After considering the rebuttal and taking another look at the paper, I may have been a bit harsh in my criticism. I still think the paper relies a lot on techniques from previous papers, but the connections between them are indeed novel. I thank the author\s for the clarifications.---------------------------------------------This paper proposes a parameter-free algorithm for stochastic gradient descent of linear and kernel predictors. In particular, the step size is automatically determined and does not rely on the (initially unknown) norm of the competitor, nor on a doubling trick. The algorithm builds on recently developed online convex optimization algorithms with similar behavior, with the contributions being:- Convergence bounds depending on the magnitude of the subgradients rather than the time horizon. In particular, for smooth losses, this leads to bounds which depend on the risk of the competitor, and interpolate between O(T^{-1/2}) and O(T^{-2/3}) rates.- Convergence to the risk of the "Bayes-optimal" function (w.r.t. the loss function used), assuming this function lies in a function space with an appropriate spectral decay.- Convergence bounds under a Tsybakov noise condition.- A few experiments showing the competitiveness of the proposed algorithm and cross-validated SVM.The question of how to avoid cross-validation and parameter tuning is certainly an interesting one and has practical implications. Moreover, the algorithm is simple and easy to implement. On the flip side, there has already been a string of recent works on online learning methods with such behavior, which can be readily applied to stochastic and batch problems (e.g. [14,16,25] which the paper cites). The proposed algorithm also builds on those methods, so the conceptual novelty is perhaps not as strong as the abstract and introduction seem to suggest. As far as I understand, the main improvement over them consists of the refined bound depending on the subgradient magnitudes, which for smooth losses lead to bounds depending on the loss of the competitor. This is an interesting contribution, although the bounds are probably suboptimal (e.g. when the loss is zero, we only get O(T^{-2/3}) rates, but with standard algorithms can get O(T^{-1}). A more minor issue is that it isn't clear if the refined bound and algorithmic change leads to anything practically noticeable (some experiments would be useful in that regard). The results regarding convergence to the Bayes-optimal and Tsybakov noise condition are nice and technically non-trivial, but unless I missed something, appear to consist mostly of plugging the new bound into existing techniqus. Overall, I would say that the contributions are solid but a bit incremental in light of previous work.  In terms of presentation, the paper is mostly well-written, but I felt it was unnecessarily complicated at times. For example:- In section 2, the definitions of the function spaces are very abstract, and would probably be hard to grasp for non-experts. Some specific examples would be very useful. - Theorem 1 could be written in a much simpler way by choosing a specific value for "a", e.g. a=2.25L, and replacing \phi(x) by the resulting constant. I failed to see what additional insight is gained by keeping "a" as free parameter. Technical and Other Comments----------------------------- The setting focuses on classification losses, which are a function of y*prediction. Can similar results be shown for regression or more general losses?- Line 46: "While in finite hypothesis spaces simple optimal strategies are known, in infinite dimensional spaces..." - should "finite" be "finite dimensional"?- Line 82: Since you focus on scalar-input loss functions, perhaps it's best to present things w.r.t. scalars rather than vectors, and use "sub-derivative" instead of "sub-gradient". For example, I was confused at first by algorithm 3, which talks about s_t as subgradients but are in fact used as scalars.- Line 125: "By the Mercer's theorem"- LIne 203: "We will present such algorithm"- Line 315: "that is low optimal \ell-risk" -> "that has low optimal \ell-risk" (?)- Line 408: "run few" -> "run a few"