Paper:
A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization
We analyze stochastic gradient algorithms for optimizing nonconvex, nonsmooth finite-sum problems. In particular, the objective function is given by the summation of a differentiable (possibly nonconvex) component, together with a possibly nondifferentiable but convex component. We propose a proximal stochastic gradient algorithm based on variance reduction, called ProxSVRG+. Our main contribution lies in the analysis of ProxSVRG+. It recovers several existing convergence results and improves/generalizes them (in terms of the number of stochastic gradient oracle calls and proximal oracle calls). In particular, ProxSVRG+ generalizes the best results given by the SCSG algorithm, recently proposed by [Lei et al., 2017] for the smooth nonconvex case. ProxSVRG+ is also more straightforward than SCSG and yields simpler analysis. Moreover, ProxSVRG+ outperforms the deterministic proximal gradient descent (ProxGD) for a wide range of minibatch sizes, which partially solves an open problem proposed in . Also, ProxSVRG+ uses much less proximal oracle calls than ProxSVRG . Moreover, for nonconvex functions satisfied Polyak-Łojasiewicz condition, we prove that ProxSVRG+ achieves a global linear convergence rate without restart unlike ProxSVRG. Thus, it can automatically switch to the faster linear convergence in some regions as long as the objective function satisfies the PL condition locally in these regions. Finally, we conduct several experiments and the experimental results are consistent with the theoretical results.

Review:
This paper focuses on the optimization problem min f(x) + h(x), where f is of a finite sum structure (with n functions in the sum), with nonconvex but smooth components, and h is a convex but possibly nonsmooth function. So, this is a nonconvex finite sum problem with a convex regularizer. Function h is treated using a prox step.  The authors propose a small modification to ProxSVRG (called ProxSVRG+), and prove that this small modification has surprisingly interesting consequences. The modification consists in replacing the full gradient computation in the outer loop of ProxSVRG by an approximation thereof through subsampling/minibatch (batch size B). In the inner loop, minibatch steps are run (with minibatch size b).   Main theoretical results:  • The SFO complexity (i.e., number of calls of stochastic first order oracle) of ProxSVRG+ is better than the complexity of ProxSVRG, ProxSGD and ProxGD in all minibatch regimes (any b). The results are nicely summarized in Tables 1, 2 and Figure 1 on page 3. The complexity is O( n/(eps*sqrt{b}) + b/eps ) under no additional assumptions.  • Further improvement is obtained when one assumes bounded variance of the stochastic gradients, with  sigma = O(1). The improvement consist in replacing the “n” in the complexity by the minimum of n and 1/eps. • The SFO complexity is better than that of SCSG in the regimes when b is not too small and not too large.  • ProxSVRG results hold for b <= n^{2/3}, and ProxSGD results hold for b >= 1/eps. However, ProxSVRG+ result holds for all values of b. • The PO complexity (number of proximal steps) is O(1/\eps), which matches the best PO complexity of ProxGD and ProxSGD, and is better than the PO complexity of Natasha1.5 and SCSG.  Additional results of similar nature (but with logarithmic dependence on epsilon, as expected) are obtained under the PL (Polyak-Lojasiewicz) condition. A new variant of PL condition is proposed and used in the analysis. Unlike ProxSVRG, no restart is needed for ProxSVRG+.  Computational results are provided to support the theory.   Strengths:  The paper is well-written (there is only a small number of grammatical issues) and well-motivated. The fact that only a minor modification of ProxSVRG is needed to obtain all the improvements should not be seen negatively. The new analysis with the modification is simpler, more elegant, and tighter. The computational experiments clearly show that ProSVRG+ can obtain similar performance as ProxSVRG with much smaller minibatch sizes b (order of magnitude smaller). This is both interesting and important. This is a very nice paper that should be accepted.   Weaknesses and Questions:  1) Can the results be improved so that ProxSVRG+ becomes better than SCSG for all minibatch sizes b? 2) How does the PL condition you use compare with the PL conditions proposed in “Global Convergence of Arbitrary-Block Gradient Methods for Generalized Polyak-Łojasiewicz Functions”, arXiv:1709.03014 ? 3) Line 114: Is the assumption really necessary? Why? Or just sufficient? 4) I think the paper would benefit if some more experiments were included in the supplementary, on some other problems and with other datasets. Otherwise the robustness/generalization of the observations drawn from the included experiments is unclear.   Small issues:  1) Lines 15-16: The sentence starting with “Besides” is not grammatically correct. 2) Line 68: What is a “super constant”? 3) Line 75: “matches” -> “match” 4) Page 3: “orcale” – “oracle” (twice) 5) Caption of Table 1: “are defined” -> “are given” 6) Eq (3): use full stop 7) Line 122: The sentence is not grammatically correct. 8) Line 200: “restated” -> “restarted” 9) Paper [21]: accents missing for one author’s name  *****  I read the rebuttal and the other reviews, and am keeping my score.