Paper:
SGD Algorithms based on Incomplete U -statistics: Large-Scale Minimization of Empirical Risk
In many learning problems, ranging from clustering to ranking through metric learning, empirical estimates of the risk functional consist of an average over tuples (e.g., pairs or triplets) of observations, rather than over individual observations. In this paper, we focus on how to best implement a stochastic approximation approach to solve such risk minimization problems. We argue that in the largescale setting, gradient estimates should be obtained by sampling tuples of data points with replacement (incomplete U -statistics) instead of sampling data points without replacement (complete U -statistics based on subsamples). We develop a theoretical framework accounting for the substantial impact of this strategy on the generalization ability of the prediction model returned by the Stochastic Gradient Descent (SGD) algorithm. It reveals that the method we promote achieves a much better trade-off between statistical accuracy and computational cost. Beyond the rate bound analysis, experiments on AUC maximization and metric learning provide strong empirical evidence of the superiority of the proposed approach.

Review:
-- Review after rebuttal -- I thank the authors for their responses. I notched up my recommendation a bit, but still I cannot offer a clear accept for this interesting paper. My quibble with Prop. 1 was not fully resolved. The authors said "the variances \sigma are w.r.t. the gradient of the *true* loss L(\theta)". This is a bit unclear but I take it to mean that\theta should be \theta^* in Prop. 1?That would be an important typo that was not acknowledged. Furthermore, in later search through the citation the authors provide and other works, I found an inconsistency in the results, which basically boils down to the \sum n'_k being a productin Prop. 1. Unfortunately, I cannot disclose other works to maintain author anonymity. Together with the way the authors originally presented this result, this makes me skepticalabout Prop. 1. Regarding my comments on excess mathematical clutter, these were generally acknowledged by the authors but it is unclear how much can be done in such limited time.Still, I decided to increase my rating for this paper, so if other reviewers are happy with therebuttal the paper could get through.--  In this paper the authors investigate ways to perform stochastic approximation that use U-statistics in a way that is computationally and statistically efficient. The quite surprising result is that using "incomplete U-statistics" is significantlymore efficient than "complete U-statistics", in both efficiency aspects. This is surprising because complete U-statistics use averages over all"data tuples", albeit reducing the sample size.I am skeptical about this result, which begins in Proposition 1. Specifically the variance for the complete U-statistics is proportional to 1/\sum n_k' whereas the variance for the incomplete U-statistic is 1/\prod {\choose..}. (see Eq. in Prop. 1) This difference is huge and the mathematical formulation is counter-intuitive. If we set n_k'=n_kshouldn't the complete U-statistic be efficient like theory suggests? It seems that efficiency should depend onterms like n_k' / n_k. This would not follow from Prop. 1. Unfortunately there is no proof for Prop. 1, but a citation that is not very helpful.I strongly suggest that the authors provide a simple proof for that fact because it is alsoimportant for Theorem 2, and for the paper in general.The other major problem is mathematical clutter. For example, Def. 1 is a quarter page long and overwhelming. A more intuitive explanation of U-statistics is possible since this is a well-developed subject in statistics. I would recommend to build up from primitives, e.g., introduce U-statistics when K=1, and thengeneralize. I also think that Proposition 2 and Theorem 1 are standard theoryin stochastic approximation, and should be in the Appendix. This sectionconfused me the most. How is normality used?Is the purpose of this section to show that the conditional variance is importantfor convergence of SGD? If it is, then the section is unnecessary because it replicates standard theoryresults. If not, the authors should try to be more explicit and precise of what this section isabout and what are the new results.The experimental results are limited and they don't seem to supportthe claim "the incomplete U-estimator significantly outperforms its complete variant". For example, the results in Figure 1 show that the difference between the two methodsis not statistically significant, unless plotted standard deviations do not reflectstandard errors; in such case the authors should make this explicit.Finally, there are several parts in the paper that I was confused or struggled through excesswording e.g.,1) In the abstract: "In this paper, we ague that in the large-scale settingthe inductive principle of SA for risk minimization should be implemented in a very specific manner. ...". This sentence spans 3 lines and gives no information.2) In Lines #48-49 a description of SGD is given that is not typical. In plain SGD there is no averaging.3) Overall the use of "tuples" is very confusing (starting from the abstract.) "Tuple" can mean many different things. Because this term is heavily usedand only becomes more clear in Page 3, I would suggest a very early definition.4) Indexes in Line #106 of Def. 1 are non-overlapping?5) In Line #173 it says that the complete U-statistic is "far from optimal". I think this isevident by definition?6) Remark 1 is confusing and could be omitted, or added at a discussion section, orin conclusion.7) The "conditional performance analysis" is confusing as well. In Line #207 it says that the analysis is "conditional on the observed data samples". But from what follows it seems that we simply condition on the number of samples, not the data samples themselves, is that correct?8) In Line #215 it says "We point out that the present analysis can be extended to more generalcontexts" and then the authors cite the work of Moulines and Bach on non-asymptoticanalysis of stochastic approximations. This is a very confusing statement.What extension do the authors mean?Minor: * Line #184 D_B should have "mathcal" * Constant C in Line #295 is used earlier,