Paper:
On Markov Chain Gradient Descent *
Stochastic gradient methods are the workhorse (algorithms) of large-scale optimization problems in machine learning, signal processing, and other computational sciences and engineering. This paper studies Markov chain gradient descent, a variant of stochastic gradient descent where the random samples are taken on the trajectory of a Markov chain. Existing results of this method assume convex objectives and a reversible Markov chain and thus have their limitations. We establish new non-ergodic convergence under wider step sizes, for nonconvex problems, and for non-reversible finite-state Markov chains. Nonconvexity makes our method applicable to broader problem classes. Non-reversible finite-state Markov chains, on the other hand, can mix substatially faster. To obtain these results, we introduce a new technique that varies the mixing levels of the Markov chains. The reported numerical results validate our contributions.

Review:
I'm happy with the authors' feedback, and will hence keep my "accept" decision.  ==============================================================  [Summary] This paper studies Markov chain gradient descent (MCGD) algorithm(s), an extension of stochastic gradient descent (SGD) in which the random samples are taken following a Markov chain. Built upon existing analysis of MCGD in [1,5,6], the paper generalizes the convergence result in various ways, answering a few open questions. In particular, the paper extends the ergodic result to non-ergodic ones, removes necessity for reversibility of the underlying Markov chain, and is able to deal with non-convex settings. The paper also provides a novel analysis based on keeping track of varying levels of mixing time corresponding to progressively better mixing, which may also shed light on future work. The numerical part of this paper is only for illustrative purposes (as the algorithm is not new), but the comparison between MCGD and SGD-T is informative and provides a good start and motivation for a more thorough study of the theoretical aspects of the algorithm.  [Quality, originality and significance] The paper deals with a few challenging open problems in the analysis of MCGD. Previously, all analysis are done for convex objectives and reversible Markov chains. But non-reversible Markov chains can have much better mixing/convergence behavior both in theory and practice [14], and non-convex settings are of great practical importance in e.g. deep learning. The paper elegantly deals with these issues by introducing a new varying mixing time level analysis, without adding artificial assumptions.   [Clarity] The paper is generally very well-written. The authors make notable efforts in motivating the necessity and challenges of the analysis for non-reversible and non-convex settings. In particular, the explanation of the assumptions between lines 117-128 is rather clear and frank. The example between lines 22-28 is also very good and showcase the advantage of MCGD over SGD in such a scenario. Nevertheless, a few things (mostly typos and wording) can still be slightly improved. Firstly, it might be slightly better to also mention the general MCMC in the example between lines 22-28. It may also help the readers to understand if a more explicit explanation of how the examples between lines 29-33 is related to the stochastic programming setting (1) is given. There are a few slight typos and wording issues, the removing of which may improve readability: 1. line 7: wider of -> wider range of;  2. line 19: expansive -> expensive;  3. line 25: If -> Even if; 4. line 29: natural -> naturally; 5. line 91: SG4 -> SGD4; 6. line 123: slower -> lower; 7. line 125: non-differential -> non-differentiable; 8. line 230: for the -> due to. Another suggestion is that the authors can explicitly state which results correspond to ergodic ones and which correspond to non-ergodic ones by e.g. directly stating that “Then we have the non-ergodic convergence” instead of just “Then we have” in line 193. This may help some readers who are not familiar with the definition of ergodicity in the optimization scenario to understand the results  [Some suggestions for improvement] Here I list a few things that I think can be further improved: 1. The example between lines 61-68 does not seem to guarantee a uniform stationary distribution for the Markov chain over the graph, and hence may not be completely consistent with (4). The authors may want to pay some attention to this potential issue. 2. It may help a lot if the authors can provide a non-reversible Markov chain for the numerical experiments (or just prove and state, if the example on page 3 is), and provide non-ergodic convergence curves. This will make the motivating example on page 3 even more inspiring, which will then illustrate the good performance of MCGD beyond ergodic convergence, reversible Markov chains, and convexity, which naturally ask for the new analysis proposed later in this paper. 3. The authors may want to either mention at the end of Section 1 or the beginning of Section 2 that the optimization problem in study for the finite state cases is always in the form of (4).   4. Give at least one sentence of explanation for the significance of obtaining non-ergodic convergence results, either in theory or in practice. Also explain the major hurdle of restricting to reversible Markov chains and convex objectives in the previous works. In particular, the authors may want to highlight the main idea of how varying mixing levels help solve the non-convexity and non-reversibility issue, and how it leads to non-ergodic results. 5. It looks a bit weird to add the reversibility assumption back in Section 5 when continuous state space is involved. The authors may want to make more explanations for this.  In general, this paper is well-written with insights and sufficient motivation, and it addresses a few challenging and open problems.