Original: Data Sampling Affects the Complexity of Online SGD over Dependent Data
Revised: Data Sampling Affects the Complexity of Online SGD over Dependent Data

Original: Abstract
Revised: Abstract

Original: Conventional machine learning applications typically assume that data samples are independently and identically distributed (i.i.d.).
Revised: Conventional machine learning applications typically assume that data samples are independently and identically distributed (i.i.d.).

Original: However, practical scenarios often involve a data-generating process that produces highly dependent data samples, which are known to heavily bias the stochastic optimization process and slow down the convergence of learning.
Revised: However, practical scenarios often involve a data-generating process that produces highly dependent data samples, which are known to heavily bias the stochastic optimization process and slow down the convergence of learning.

Original: In this paper, we conduct a fundamental study on how different stochastic data sampling schemes affect the sample complexity of online stochastic gradient descent (SGD) over highly dependent data.
Revised: In this paper, we conduct a fundamental study on how different stochastic data sampling schemes affect the sample complexity of online stochastic gradient descent (SGD) over highly dependent data.

Original: Specifically, with a ϕ -mixing process of data, we show that online SGD with proper periodic data-subsampling achieves an improved sample complexity over the standard online SGD in the full spectrum of the data dependence level.
Revised: Specifically, with a ϕ -mixing process of data, we show that online SGD with proper periodic data-subsampling achieves an improved sample complexity over the standard online SGD in the full spectrum of the data dependence level.

Original: Interestingly, even subsampling a subset of data samples can accelerate the convergence of online SGD over highly dependent data.
Revised: Interestingly, even subsampling a subset of data samples can accelerate the convergence of online SGD over highly dependent data.

Original: Moreover, we show that online SGD with mini-batch sampling can further substantially improve the sample complexity over online SGD with periodic data-subsampling over highly dependent data.
Revised: Moreover, we show that online SGD with mini-batch sampling can further substantially improve the sample complexity over online SGD with periodic data-subsampling over highly dependent data.

Original: Numerical experiments validate our theoretical results.
Revised: Numerical experiments validate our theoretical results.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Stochastic optimization algorithms have attracted great attention in the past decade due to its successful applications to a broad research areas, including deep learning [Goodfellow et al., 2016], reinforcement learning [Sutton and Barto, 2018], online learning [Bottou, 2010, Hazan, 2017], control [Marti, 2017], etc.
Revised: Stochastic optimization algorithms have attracted great attention in the past decade due to its successful applications to a broad research areas, including deep learning [Goodfellow et al., 2016], reinforcement learning [Sutton and Barto, 2018], online learning [Bottou, 2010, Hazan, 2017], control [Marti, 2017], etc.

Original: In the conventional analysis of stochastic optimization algorithms, it is usually assumed that all data samples are independently and identically distributed (i.i d.) and queried.
Revised: In the conventional analysis of stochastic optimization algorithms, it is usually assumed that all data samples are independently and identically distributed (i.i d.) and queried.

Original: For example, data samples in the traditional empirical risk minimization framework are assumed to be queried independently from the underlying data distribution, while data samples in reinforcement learning are assumed to be queried from the stationary distribution of the underlying
Revised: For example, data samples in the traditional empirical risk minimization framework are assumed to be queried independently from the underlying data distribution, while data samples in reinforcement learning are assumed to be queried from the stationary distribution of the underlying

Original: Markov chain.
Revised: Markov chain.

Original: Although the i.i.d.
Revised: Although the i.i.d.

Original: data assumption leads to a comprehensive understanding of the statistical limit and computation complexity of SGD, it violates the nature of many practical data-generating stochastic processes, which generate highly correlated samples that depend on the history.
Revised: data assumption leads to a comprehensive understanding of the statistical limit and computation complexity of SGD, it violates the nature of many practical data-generating stochastic processes, which generate highly correlated samples that depend on the history.

Original: In fact, dependent data can be found almost everywhere, e.g., daily stock price [Onalan, 2009, Fort and Roberts, 2005], weather/climate data, state transitions in Markov chains, etc.
Revised: In fact, dependent data can be found almost everywhere, e.g., daily stock price [Onalan, 2009, Fort and Roberts, 2005], weather/climate data, state transitions in Markov chains, etc.

Original: To understand the impact of data dependence on the convergence and complexity of stochastic algorithms, there is a growing number of recent works that introduce various definitions to quantify data dependence.
Revised: To understand the impact of data dependence on the convergence and complexity of stochastic algorithms, there is a growing number of recent works that introduce various definitions to quantify data dependence.

Original: Specifically, to analyze the finite-time convergence of various stochastic reinforcement learning algorithms, recent studies assume that the dependent samples queried from the Markov decision process satisfy a geometric mixing property
Revised: Specifically, to analyze the finite-time convergence of various stochastic reinforcement learning algorithms, recent studies assume that the dependent samples queried from the Markov decision process satisfy a geometric mixing property

Original: [Dalal et al., 2018, Zou et al., 2019, Xu and Gu, 2020, Qu and Wierman, 2020], which requires the underlying Markov chain to be uniformly ergodic or has a finite mixing time [Even-Dar et al., 2003].
Revised: [Dalal et al., 2018, Zou et al., 2019, Xu and Gu, 2020, Qu and Wierman, 2020], which requires the underlying Markov chain to be uniformly ergodic or has a finite mixing time [Even-Dar et al., 2003].

Original: On the other hand, to analyze the convergence of stochastic optimization algorithms over dependent data,
Revised: On the other hand, to analyze the convergence of stochastic optimization algorithms over dependent data,

Original: Karimi et al.
Revised: Karimi et al.

Original: [2019] assumed the existence of a solution to the Poisson equation associated with the underlying Markov chain, which is a weaker condition than the uniform ergodic condition [Glynn and Meyn, 1996].
Revised: [2019] assumed the existence of a solution to the Poisson equation associated with the underlying Markov chain, which is a weaker condition than the uniform ergodic condition [Glynn and Meyn, 1996].

Original: Moreover, Agarwal and
Revised: Moreover, Agarwal and

Original: Duchi [2012] introduced a ϕ -mixing process that quantifies how fast the distribution of future data samples (conditioned on a fixed filtration) converges to the underlying stationary data distribution.
Revised: Duchi [2012] introduced a ϕ -mixing process that quantifies how fast the distribution of future data samples (conditioned on a fixed filtration) converges to the underlying stationary data distribution.

Original: In particular, the ϕ -mixing process is more general than the previous two notions of data dependence [Douc et al., 2018].
Revised: In particular, the ϕ -mixing process is more general than the previous two notions of data dependence [Douc et al., 2018].

Original: While the aforementioned works leveraged the above notions of data dependence to characterize the sample complexity of various stochastic algorithms over dependent data, there still lacks theoretical understanding of how algorithm structure affects the sample complexity of stochastic algorithms under different levels of data dependence.
Revised: While the aforementioned works leveraged the above notions of data dependence to characterize the sample complexity of various stochastic algorithms over dependent data, there still lacks theoretical understanding of how algorithm structure affects the sample complexity of stochastic algorithms under different levels of data dependence.

Original: In particular, a key algorithm structure is the stochastic data sampling scheme, which critically affects the bias and variance of the stochastic learning process.
Revised: In particular, a key algorithm structure is the stochastic data sampling scheme, which critically affects the bias and variance of the stochastic learning process.

Original: In fact, under i.i.d.
Revised: In fact, under i.i.d.

Original: data and convex geometry, it is well known that SGD achieves the sample complexity lower bound under various stochastic data sampling schemes [Bottou, 2010], e.g., single-sample sampling and mini-batch sampling.
Revised: data and convex geometry, it is well known that SGD achieves the sample complexity lower bound under various stochastic data sampling schemes [Bottou, 2010], e.g., single-sample sampling and mini-batch sampling.

Original: However, these schemes may lead to substantially different convergence behaviors over highly dependent data, as they are no longer unbiased.
Revised: However, these schemes may lead to substantially different convergence behaviors over highly dependent data, as they are no longer unbiased.

Original: Therefore, it is of vital importance to understand the interplay among data dependence, stochastic data sampling and sample complexity of stochastic learning algorithms, and we want to ask the following fundamental question.
Revised: Therefore, it is of vital importance to understand the interplay among data dependence, stochastic data sampling and sample complexity of stochastic learning algorithms, and we want to ask the following fundamental question.

Original: • Q: How does stochastic data sampling affect the convergence rate and sample complexity of stochastic learning algorithms over dependent data?
Revised: • Q: How does stochastic data sampling affect the convergence rate and sample complexity of stochastic learning algorithms over dependent data?

Original: In this paper, we provide comprehensive answers to this fundamental question.
Revised: In this paper, we provide comprehensive answers to this fundamental question.

Original: Specifically, we conduct a comprehensive study of the convergence rate and sample complexity of the online SGD algorithm over a wide spectrum of data dependence levels under various stochastic data sampling schemes, including periodic subsampling and mini-batch sampling.
Revised: Specifically, we conduct a comprehensive study of the convergence rate and sample complexity of the online SGD algorithm over a wide spectrum of data dependence levels under various stochastic data sampling schemes, including periodic subsampling and mini-batch sampling.

Original: Our results show that online SGD with both data sampling schemes achieves a substantially improved sample complexity over the standard online SGD over highly dependent data.
Revised: Our results show that online SGD with both data sampling schemes achieves a substantially improved sample complexity over the standard online SGD over highly dependent data.

Original: We summarize our contributions as follows.
Revised: We summarize our contributions as follows.

Original: 1.1 OUR CONTRIBUTIONS
Revised: 1.1 OUR CONTRIBUTIONS

Original: We consider the following stochastic optimization problem.
Revised: We consider the following stochastic optimization problem.

Original: [Equation]
Revised: [Equation]

Original: where the objective function f is convex and Lipschitz continuous, and the expectation is taken over the stationary distribution µ of the underlying data-generating process P .
Revised: where the objective function f is convex and Lipschitz continuous, and the expectation is taken over the stationary distribution µ of the underlying data-generating process P .

Original: To perform online learning, we query a stream of dependent data samples from the underlying data-generating process.
Revised: To perform online learning, we query a stream of dependent data samples from the underlying data-generating process.

Original: Specifically, we adopt the ϕ -mixing process to quantify the data dependence via a decaying mixing coefficient function ϕ ξ ( k ) (see Definition 2.2)
Revised: Specifically, we adopt the ϕ -mixing process to quantify the data dependence via a decaying mixing coefficient function ϕ ξ ( k ) (see Definition 2.2)

Original: [Agarwal and Duchi, 2012].
Revised: [Agarwal and Duchi, 2012].

Original: We study the convergence of the online stochastic gradient descent (SGD) algorithm over a ϕ -mixing data stream under various stochastic data sampling schemes, including periodic subsampling and mini-batch sampling.
Revised: We study the convergence of the online stochastic gradient descent (SGD) algorithm over a ϕ -mixing data stream under various stochastic data sampling schemes, including periodic subsampling and mini-batch sampling.

Original: We summarize and compare the sample complexities of online SGD with these data sampling schemes under different ϕ -mixing data dependence models in Table 1.
Revised: We summarize and compare the sample complexities of online SGD with these data sampling schemes under different ϕ -mixing data dependence models in Table 1.

Original: We first study the convergence of online SGD over ϕ -mixing dependent data samples under the data subsampling scheme.
Revised: We first study the convergence of online SGD over ϕ -mixing dependent data samples under the data subsampling scheme.

Original: In particular, the data subsampling scheme utilizes only one data sample per r consecutive data samples by periodically skipping r − 1 samples.
Revised: In particular, the data subsampling scheme utilizes only one data sample per r consecutive data samples by periodically skipping r − 1 samples.

Original: With this data subsampling scheme, the subsampled data samples are less dependent for a larger subsampling period r .
Revised: With this data subsampling scheme, the subsampled data samples are less dependent for a larger subsampling period r .

Original: Also, the improvement is substantial when the data is highly dependent with an algebraic decaying ϕ -mixing coefficient.
Revised: Also, the improvement is substantial when the data is highly dependent with an algebraic decaying ϕ -mixing coefficient.

Original: Moreover, we study the sample complexity of online SGD over ϕ -mixing dependent data samples under the mini-batch sampling scheme.
Revised: Moreover, we study the sample complexity of online SGD over ϕ -mixing dependent data samples under the mini-batch sampling scheme.

Original: Compare to the data subsampling scheme, mini-batch sampling substantially reduces the mini-batch data dependence without skipping data samples.
Revised: Compare to the data subsampling scheme, mini-batch sampling substantially reduces the mini-batch data dependence without skipping data samples.

Original: Consequently, mini-batch update leverages the sample average over a mini batch of data samples to reduce both the bias (caused by the data dependence) and the variance (caused by stochastic sampling).
Revised: Consequently, mini-batch update leverages the sample average over a mini batch of data samples to reduce both the bias (caused by the data dependence) and the variance (caused by stochastic sampling).

Original: Specifically, we show that online SGD with mini-batch sampling achieves an orderwise lower sample complexity than both the standard online SGD and the online SGD with data subsampling in the full spectrum of the convergence rate of the ϕ -mixing coefficient.
Revised: Specifically, we show that online SGD with mini-batch sampling achieves an orderwise lower sample complexity than both the standard online SGD and the online SGD with data subsampling in the full spectrum of the convergence rate of the ϕ -mixing coefficient.

Original: Our study reveals that the widely used mini-batch sampling scheme can effectively reduce the bias caused by data dependence without sacrificing data efficiency.
Revised: Our study reveals that the widely used mini-batch sampling scheme can effectively reduce the bias caused by data dependence without sacrificing data efficiency.

Original: 1.2 RELATED WORK
Revised: 1.2 RELATED WORK

Original: Stochastic Algorithms over Dependent Data Steinwart and Christmann [2009] and Modha and Masry [1996] established the convergence analysis of online stochastic algorithms for streaming data with geometric ergodicity.
Revised: Stochastic Algorithms over Dependent Data Steinwart and Christmann [2009] and Modha and Masry [1996] established the convergence analysis of online stochastic algorithms for streaming data with geometric ergodicity.

Original: Duchi et al.
Revised: Duchi et al.

Original: [2011] proved that the stochastic subgradient method has strong convergence guarantee if the mixing time is uniformly bounded.
Revised: [2011] proved that the stochastic subgradient method has strong convergence guarantee if the mixing time is uniformly bounded.

Original: Agarwal and Duchi [2012] studied the convex/strongly convex stochastic optimization problem and proved high-probability convergence bounds for general stochastic algorithms under general stationary mixing processes.
Revised: Agarwal and Duchi [2012] studied the convex/strongly convex stochastic optimization problem and proved high-probability convergence bounds for general stochastic algorithms under general stationary mixing processes.

Original: Godichon-Baggioni et al.
Revised: Godichon-Baggioni et al.

Original: [2021] provided the nonasymptotic analysis of stochastic algorithms with strongly convex objective function over streaming mini-batch data.
Revised: [2021] provided the nonasymptotic analysis of stochastic algorithms with strongly convex objective function over streaming mini-batch data.

Original: In a more general setting, the stochastic approximation (SA) problem was studied in [Karimi et al., 2019] by assuming the existence of solution to a Poisson equation.
Revised: In a more general setting, the stochastic approximation (SA) problem was studied in [Karimi et al., 2019] by assuming the existence of solution to a Poisson equation.

Original: Recently,
Revised: Recently,

Original: Debavelaere et al.
Revised: Debavelaere et al.

Original: [2021] developed the asymptotic convergence analysis of SA problem for sub-geometric Markov dynamic noises.
Revised: [2021] developed the asymptotic convergence analysis of SA problem for sub-geometric Markov dynamic noises.

Original: Finite-time convergence of reinforcement learning Recently, a series of work studied the finite-time convergence of many stochastic reinforcement learning algorithms over Markovian dependent samples, including TD learning [Dalal et al., 2018, Xu et al., 2019, Kaledin et al., 2020],
Revised: Finite-time convergence of reinforcement learning Recently, a series of work studied the finite-time convergence of many stochastic reinforcement learning algorithms over Markovian dependent samples, including TD learning [Dalal et al., 2018, Xu et al., 2019, Kaledin et al., 2020],

Original: Q-learning [Qu and Wierman, 2020, Li et al., 2021, Melo et al., 2008, Chen et al., 2019, Xu and Gu, 2020], fitted
Revised: Q-learning [Qu and Wierman, 2020, Li et al., 2021, Melo et al., 2008, Chen et al., 2019, Xu and Gu, 2020], fitted

Original: Q-iteration
Revised: Q-iteration

Original: [Mnih et al., 2013, 2015, Agarwal et al., 2021],
Revised: [Mnih et al., 2013, 2015, Agarwal et al., 2021],

Original: [Table]
Revised: [Table]

Original: actor-critic algorithms [Wang et al., 2019, Yang et al., 2019,
Revised: actor-critic algorithms [Wang et al., 2019, Yang et al., 2019,

Original: Kumar et al., 2019, Qiu et al., 2019, Wu et al., 2020, Xu et al., 2020], etc.
Revised: Kumar et al., 2019, Qiu et al., 2019, Wu et al., 2020, Xu et al., 2020], etc.

Original: In these studies, the dependent Markovian samples are assumed to be generated from a geometric ϕ -mixing process, which is satisfied when the underlying
Revised: In these studies, the dependent Markovian samples are assumed to be generated from a geometric ϕ -mixing process, which is satisfied when the underlying

Original: Markov chain is uniformly ergodic or time-homogeneous with finite-states.
Revised: Markov chain is uniformly ergodic or time-homogeneous with finite-states.

Original: Regret of Stochastic Convex Optimization There have been many known regret bounds for online convex optimization problem.
Revised: Regret of Stochastic Convex Optimization There have been many known regret bounds for online convex optimization problem.

Original: Hazan [2017] has built the standard O ( √ T ) regret bound for online SGD algorithm with assuming the bounded gradient.
Revised: Hazan [2017] has built the standard O ( √ T ) regret bound for online SGD algorithm with assuming the bounded gradient.

Original: Xiao [2009] introduces the regret bound of online dual averaging method.
Revised: Xiao [2009] introduces the regret bound of online dual averaging method.

Original: To our best knowledge, there is no high-probability guaranteed regret bound for mini-batch SGD with considering the data dependence.
Revised: To our best knowledge, there is no high-probability guaranteed regret bound for mini-batch SGD with considering the data dependence.

Original:  FORMULATION AND ASSUMPTIONS
Revised:  FORMULATION AND ASSUMPTIONS

Original: In this section, we introduce the problem formulation and some basic assumptions.
Revised: In this section, we introduce the problem formulation and some basic assumptions.

Original: Consider a model with parameters w .
Revised: Consider a model with parameters w .

Original: For any data sample ξ , denote F ( w ; ξ ) ∈ R as the sample loss of this data sample under the model w .
Revised: For any data sample ξ , denote F ( w ; ξ ) ∈ R as the sample loss of this data sample under the model w .

Original: In this paper, we consider the following standard stochastic optimization problem that has broad applications in machine learning.
Revised: In this paper, we consider the following standard stochastic optimization problem that has broad applications in machine learning.

Original: [Equation]
Revised: [Equation]

Original: Here, the expectation is taken over the randomness of the data sample ξ , which is drawn from an underlying distribution µ .
Revised: Here, the expectation is taken over the randomness of the data sample ξ , which is drawn from an underlying distribution µ .

Original: We make the following standard assumptions regarding the problem (P)
Revised: We make the following standard assumptions regarding the problem (P)

Original: [Agarwal and Duchi, 2012].
Revised: [Agarwal and Duchi, 2012].

Original: Assumption 2.1.
Revised: Assumption 2.1.

Original: The optimization problem (P) satisfies 1.
Revised: The optimization problem (P) satisfies 1.

Original: For every ξ , function F ( · ; ξ ) is G -Lipschitz continuous over the domain W .
Revised: For every ξ , function F ( · ; ξ ) is G -Lipschitz continuous over the domain W .

Original: Function f ( · ) is convex and bounded below,
Revised: Function f ( · ) is convex and bounded below,

Original: i
Revised: i

Original: e., f ( w ∗ ) :
Revised: e., f ( w ∗ ) :

Original: = inf w ∈W f ( w ) >
Revised: = inf w ∈W f ( w ) >

Original: −∞ . 3.
Revised: −∞ . 3.

Original: W is convex and compact with bounded diameter R .
Revised: W is convex and compact with bounded diameter R .

Original: To solve this stochastic optimization problem, one often needs to query a stream of data samples from the distribution µ to perform optimization.
Revised: To solve this stochastic optimization problem, one often needs to query a stream of data samples from the distribution µ to perform optimization.

Original: Unlike traditional stochastic optimization that usually assumes that the data samples are i.i.d.
Revised: Unlike traditional stochastic optimization that usually assumes that the data samples are i.i.d.

Original: we consider a more general and practical dependent data-generating process as we elaborate below.
Revised: we consider a more general and practical dependent data-generating process as we elaborate below.

Original: Dependent data-generating process: We consider a stochastic process P that generates a stream of data samples { ξ 1 , ξ 2 , ..., } , which are not necessarily independent.
Revised: Dependent data-generating process: We consider a stochastic process P that generates a stream of data samples { ξ 1 , ξ 2 , ..., } , which are not necessarily independent.

Original: In particular, the stochastic process P has an underlying stationary distribution µ .
Revised: In particular, the stochastic process P has an underlying stationary distribution µ .

Original: To quantify the dependence of the data generation process, we introduce the following standard ϕ -mixing process [Agarwal and Duchi, 2012], where we denote {F t } t as the filtration generated by { ξ t } t .
Revised: To quantify the dependence of the data generation process, we introduce the following standard ϕ -mixing process [Agarwal and Duchi, 2012], where we denote {F t } t as the filtration generated by { ξ t } t .

Original: Definition 2.2 ( ϕ -mixing process) .
Revised: Definition 2.2 ( ϕ -mixing process) .

Original: Consider a stochastic process { ξ t } t with a stationary distribution µ .
Revised: Consider a stochastic process { ξ t } t with a stationary distribution µ .

Original: Let P ( ξ t + k ∈ ·|F t ) be the distribution of the ( t + k )
Revised: Let P ( ξ t + k ∈ ·|F t ) be the distribution of the ( t + k )

Original: -th sample conditioned on F t , and denote d TV as the total variation distance.
Revised: -th sample conditioned on F t , and denote d TV as the total variation distance.

Original: Then, the process { ξ t } t is called ϕ -mixing if the following mixing coefficient ϕ ξ ( · ) converges to  as k tends to infinity.
Revised: Then, the process { ξ t } t is called ϕ -mixing if the following mixing coefficient ϕ ξ ( · ) converges to 0 as k tends to infinity.

Original: [Equation]
Revised: [Equation]

Original: Intuitively the ϕ -mixing coefficient describes how fast the distribution of sample ξ t + k converges to the stationary distribution µ when conditioned on the filtration F t , as the time gap k → ∞ .
Revised: Intuitively, the ϕ -mixing coefficient describes how fast the distribution of sample ξ t + k converges to the stationary distribution µ when conditioned on the filtration F t , as the time gap k → ∞ .

Original: The ϕ -mixing process can be found in many applications, which involve mixing coefficients that converge to zero at different convergence rates.
Revised: The ϕ -mixing process can be found in many applications, which involve mixing coefficients that converge to zero at different convergence rates.

Original: Below we mention some representative examples.
Revised: Below we mention some representative examples.

Original: • Geometric ϕ -mixing process.
Revised: • Geometric ϕ -mixing process.

Original: Such a type of process has a geometrically diminishing mixing coefficient, i.e., ϕ ξ ( k ) ≤ ϕ 0 exp( − ck θ ) for some ϕ 0 , c, θ > 0 .
Revised: Such a type of process has a geometrically diminishing mixing coefficient, i.e., ϕ ξ ( k ) ≤ ϕ 0 exp( − ck θ ) for some ϕ 0 , c, θ > 0 .

Original: Examples include finite-state ergodic Markov chains and some aperiodic Harris-recurrent Markov processes [Modha and Masry, 1996, Agarwal and Duchi, 2012, Meyn and Tweedie, 2012]; • Algebraic ϕ -mixing process.
Revised: Examples include finite-state ergodic Markov chains and some aperiodic Harris-recurrent Markov processes [Modha and Masry, 1996, Agarwal and Duchi, 2012, Meyn and Tweedie, 2012]; • Algebraic ϕ -mixing process.

Original: Such a type of process has a polynomially diminishing mixing coefficient,[-i.e., ϕ ξ ( k ) ≤ ϕ 
-]
Revised: Such a type of process has a polynomially diminishing mixing coefficient,[-i.e., ϕ ξ ( k ) ≤ ϕ 
-]

Original:  k − θ for some ϕ 0 , θ > 0 .
Revised: i e., ϕ ξ ( k ) ≤ ϕ 0 k − θ for some ϕ 0 , θ > 0 .

Original: Examples include a large class of Metropolis-Hastings samplers [Jarner and Roberts, 2002] and some queuing systems [Agarwal and Duchi, 2012].
Revised: Examples include a large class of Metropolis-Hastings samplers [Jarner and Roberts, 2002] and some queuing systems [Agarwal and Duchi, 2012].

Original:  COMPLEXITY OF ONLINE SGD OVER DEPENDENT DATA
Revised:  COMPLEXITY OF ONLINE SGD OVER DEPENDENT DATA

Original: In this section, we recap the convergence results of online SGD over dependent data established in [Agarwal and
Revised: In this section, we recap the convergence results of online SGD over dependent data established in [Agarwal and

Original: Duchi, 2012].
Revised: Duchi, 2012].

Original: Throughout, we define the sample complexity as the total number of samples required for the algorithm to output a model w that achieves an ϵ convergence error with a certain probability, i.e., f ( w ) − f ( w ∗ ) ≤ ϵ with probability 1 − δ .
Revised: Throughout, we define the sample complexity as the total number of samples required for the algorithm to output a model w that achieves an ϵ convergence error with a certain probability, i.e., f ( w ) − f ( w ∗ ) ≤ ϵ with probability 1 − δ .

Original: Also, the standard regret of an online learning algorithm is defined as
Revised: Also, the standard regret of an online learning algorithm is defined as

Original: [Equation]
Revised: [Equation]

Original: where the models { w 1 , w 2 , ..., w n } are generated using the data samples { ξ 1 , ξ 2 , ..., ξ n } , respectively, and w ∗ is the minimizer of f ( w ) .
Revised: where the models { w 1 , w 2 , ..., w n } are generated using the data samples { ξ 1 , ξ 2 , ..., ξ n } , respectively, and w ∗ is the minimizer of f ( w ) .

Original: For this sequence of models { w 1 , w 2 , ..., w n } , we make the following mild assumption, which is satisfied by many SGD-type algorithms.
Revised: For this sequence of models { w 1 , w 2 , ..., w n } , we make the following mild assumption, which is satisfied by many SGD-type algorithms.

Original: Assumption 3.1.
Revised: Assumption 3.1.

Original: There is a non-increasing sequence { κ ( t ) } t such that ∥ w ( t + 1) − w ( t ) ∥
Revised: There is a non-increasing sequence { κ ( t ) } t such that ∥ w ( t + 1) − w ( t ) ∥

Original: ≤ κ ( t ) .
Revised: ≤ κ ( t ) .

Original: Online SGD is a popular and standard algorithm for solving the problem (P).
Revised: Online SGD is a popular and standard algorithm for solving the problem (P).

Original: In every iteration t , online SGD queries a sample ξ t from the data-generating process and performs the following SGD update.
Revised: In every iteration t , online SGD queries a sample ξ t from the data-generating process and performs the following SGD update.

Original: [Equation]
Revised: [Equation]

Original: where η t is the learning rate.
Revised: where η t is the learning rate.

Original: In Theorem 2 of [Agarwal and Duchi, 2012], the authors established a high probability convergence error bound for a generic class of stochastic algorithms.
Revised: In Theorem 2 of [Agarwal and Duchi, 2012], the authors established a high probability convergence error bound for a generic class of stochastic algorithms.

Original: Specifically, under the Assumptions 2.1 and 3.1, they showed that for any τ ∈ N with probability at least 1 − δ , the averaged predictor (cid:98) w n := 1 n (cid:80) nt =1 w ( t ) satisfies f ( (cid:98) w n ) − f ( 
Revised: Specifically, under the Assumptions 2.1 and 3.1, they showed that for any τ ∈ N with probability at least 1 − δ , the averaged predictor (cid:98) w n := 1 n (cid:80) nt =1 w ( t ) satisfies f ( (cid:98) w n ) − f ( w ∗ ) ≤ R n

Original: [Equation]
Revised: [Equation]

Original: Here, R n is the regret of the algorithm of interest, G is the
Revised: Here, R n is the regret of the algorithm of interest, G is the

Original: Lipschitz constant of the loss function F ( · ; ξ ) , and R is the diameter of the parameter domain, and τ ∈ N is an auxiliary parameter that is introduced to decouple the dependence of the data samples.
Revised: Lipschitz constant of the loss function F ( · ; ξ ) , and R is the diameter of the parameter domain, and τ ∈ N is an auxiliary parameter that is introduced to decouple the dependence of the data samples.

Original: From the above bound, one can see that the optimal choice of τ depends on the convergence rate of the mixing coefficient ϕ ξ ( τ ) .
Revised: From the above bound, one can see that the optimal choice of τ depends on the convergence rate of the mixing coefficient ϕ ξ ( τ ) .

Original: Specifically, consider the online
Revised: Specifically, consider the online

Original: SGD algorithm in (1).
Revised: SGD algorithm in (1).

Original: It can be shown that it achieves the regret R n = O ( √ n ) and satisfies κ ( t )
Revised: It can be shown that it achieves the regret R n = O ( √ n ) and satisfies κ ( t )

Original: =
Revised: =

Original: O (1 / √ t ) under a proper diminishing learning rate.
Revised: O (1 / √ t ) under a proper diminishing learning rate.

Original: Consequently, the above high-probability convergence bound for online SGD reduces to
Revised: Consequently, the above high-probability convergence bound for online SGD reduces to

Original: [Equation]
Revised: [Equation]

Original: Such a bound further implies the following sample complexity results of online SGD under different ϕ -mixing models.
Revised: Such a bound further implies the following sample complexity results of online SGD under different ϕ -mixing models.

Original: Corollary 3.2.
Revised: Corollary 3.2.

Original: The sample complexity of online SGD for achieving an ϵ convergence error over ϕ -mixing data is
Revised: The sample complexity of online SGD for achieving an ϵ convergence error over ϕ -mixing data is

Original: • If the data is geometric ϕ -mixing with parameter θ > 0 , then we set τ = O (cid:0) (log 1 ϵ ) 1 θ (cid:1) .
Revised: • If the data is geometric ϕ -mixing with parameter θ > 0 , then we set τ = O (cid:0) (log 1 ϵ ) 1 θ (cid:1) .

Original: The resulting sample complexity is in theorder of n =
Revised: The resulting sample complexity is in the

Original:  O (cid:0) ϵ − 2 (log 1 ϵ ) 2 θ (cid:1) .
Revised: order of n = O (cid:0) ϵ − 2 (log 1 ϵ ) 2 θ (cid:1) .

Original: • If the data is algebraic ϕ -mixing with parameter θ > 0 , then we set τ =
Revised: • If the data is algebraic ϕ -mixing with parameter θ > 0 , then we set τ =

Original: O ( ϵ − 1 θ ) .
Revised: O ( ϵ − 1 θ ) .

Original: The resulting sample complexity is in the order of n =
Revised: The resulting sample complexity is in the order of n =

Original: O ( ϵ − 2 − 2 θ ) .
Revised: O ( ϵ − 2 − 2 θ ) .

Original: It can be seen that if the data-generating process has a fast geometrically diminishingmixing coefficient, i.e., the data samples are close to being independent from each other, then the resulting sample complexity is almost the same as that of SGD with i.i.d.
Revised: It can be seen that if the data-generating process has a fast geometrically diminishing

Revised: mixing

Revised: coefficient,

Revised: i e.,

Revised: the

Revised: data samples

Revised: are

Revised: close to being independent from each other, then the resulting sample complexity is almost the same as that of SGD with i.i.d.

Original: samples.
Revised: samples.

Original: On the other hand, if the data-generating process mixes slowly with an algebraically diminishing mixing coefficient, i.e., the data samples are highly dependent, then the data dependence increases the sample complexity by a non-negligible factor of ϵ − 2 θ . 
Revised: On the other hand, if the data-generating process mixes slowly with an algebraically diminishing mixing coefficient, i.e., the data samples are highly dependent, then the data dependence increases the sample complexity by a non-negligible factor of ϵ − 2 θ . In

Original: In particular, such a factor is substantially large if the mixing rate parameter θ is close to zero.
Revised: particular, such a factor is substantially large if the mixing rate parameter θ is close to zero

Original:  COMPLEXITY OF ONLINE SGD WITH DATA SUBSAMPLING
Revised:  COMPLEXITY OF ONLINE SGD WITH DATA SUBSAMPLING

Original: When apply online SGD to solve stochastic optimization problems over dependent data, the key challenge is that the data dependence introduces non-negligible bias that slows down the convergence of the algorithm.
Revised: When apply online SGD to solve stochastic optimization problems over dependent data, the key challenge is that the data dependence introduces non-negligible bias that slows down the convergence of the algorithm.

Original: Hence, a straightforward solution is to reduce data dependence before performing stochastic optimization, and data subsampling is such a simple and effective approach [Nagaraj et al., 2020,
Revised: Hence, a straightforward solution is to reduce data dependence before performing stochastic optimization, and data subsampling is such a simple and effective approach [Nagaraj et al., 2020,

Original: Kotsalis et al., 2020].
Revised: Kotsalis et al., 2020].

Original: Specifically, consider a stream of ϕ -mixing data samples { ξ 1 , ξ 2 , ξ 3 , . . . } .
Revised: Specifically, consider a stream of ϕ -mixing data samples { ξ 1 , ξ 2 , ξ 3 , . . . } .

Original: Instead of utilizing the entire stream of data, we subsample a subset of this data stream with period r ∈ N and obtain the following subsampled data stream { ξ 1 , ξ r +1 , ξ 2 r +1 , . . . } .
Revised: Instead of utilizing the entire stream of data, we subsample a subset of this data stream with period r ∈ N and obtain the following subsampled data stream { ξ 1 , ξ r +1 , ξ 2 r +1 , . . . } .

Original: In particular, let {F t } t be the canonical filtration generated by { ξ tr +1 } t .
Revised: In particular, let {F t } t be the canonical filtration generated by { ξ tr +1 } t .

Original: Since the consecutive subsampled samples are r time steps away from each other, it is easy to verify that the subsampled data stream { ξ tr +1 } t is also a ϕ -mixing process with mixing coefficient given by ϕ rξ ( t ) =
Revised: Since the consecutive subsampled samples are r time steps away from each other, it is easy to verify that the subsampled data stream { ξ tr +1 } t is also a ϕ -mixing process with mixing coefficient given by ϕ rξ ( t ) =

Original: ϕ ξ ( rt ) , where ϕ r ξ denotes the mixing coefficient of the subsampled data stream { ξ tr +1 } t .
Revised: ϕ ξ ( rt ) , where ϕ rξ denotes the mixing coefficient of the subsampled data stream { ξ tr +1 } t .

Original: Therefore, by periodically subsampling the data stream, the resulting subsampled process has a faster-converging mixing coefficient.
Revised: Therefore, by periodically subsampling the data stream, the resulting subsampled process has a faster-converging mixing coefficient.

Original: Then, we can apply online SGD to such subsampled data, i.e., (SGD with subsampling):
Revised: Then, we can apply online SGD to such subsampled data, i.e., (SGD with subsampling):

Original: [Equation]
Revised: [Equation]

Original: In particular, the convergence error bound in eq.
Revised: In particular, the convergence error bound in eq.

Original: (2) still holds by replacing ϕ ξ ( τ ) with ϕ ξ ( rτ ) , and we obtain the following bound for online SGD with subsampling.
Revised: (2) still holds by replacing ϕ ξ ( τ ) with ϕ ξ ( rτ ) , and we obtain the following bound for online SGD with subsampling.

Original: [Equation]
Revised: [Equation]

Original: Such a bound implies the following sample complexity results of online SGD with subsampling under different convergence rates of the mixing coefficient ϕ ξ .
Revised: Such a bound implies the following sample complexity results of online SGD with subsampling under different convergence rates of the mixing coefficient ϕ ξ .

Original: Corollary 4.1.
Revised: Corollary 4.1.

Original: The sample complexity of online SGD with subsampling for achieving an ϵ convergence error over ϕmixing data process is.
Revised: The sample complexity of online SGD with subsampling for achieving an ϵ convergence error over ϕmixing data process is.

Original: • If the data is geometric ϕ -mixing with parameter θ > 0 , then we choose r = O (cid:0) (log 1 ϵ ) 1 θ (cid:1) and
Revised: • If the data is geometric ϕ -mixing with parameter θ > 0 , then we choose r = O (cid:0) (log 1 ϵ ) 1 θ (cid:1) and

Original: τ = O (1) .
Revised: τ = O (1) .

Original: The resulting sample complexity isin the order of rn =
Revised: The resulting sample complexity is rn =

Original: O (cid:0) ϵ − 2 (log 1 ϵ ) 1 θ (cid:1) .
Revised: O (cid:0) ϵ − 2 (log 1 ϵ ) 1 θ (cid:1) .

Original: • If the data is algebraic ϕ -mixing with parameter θ > 0 , then we choose r = O (cid:0) ϵ − 1 θ (cid:1) and τ = O (1) .
Revised: • If the data is algebraic ϕ -mixing with parameter θ > 0 , then we choose r = O (cid:0) ϵ − 1 θ (cid:1) and τ = O (1) .

Original: The resulting sample complexity isin the order of rn =
Revised: The resulting sample complexity is rn =

Original: O (cid:0) ϵ − 2 − 1 θ (cid:1) .
Revised: O (cid:0) [Table]

Original: Compare the above sample complexity results with those of the standard online SGD in Corollary 3.2, we conclude that data-subsampling can improve the sample complexity by a factor of (log 1 ϵ ) 1 θ and ϵ − 1 θ for geometric ϕ
Revised: Compare the above sample complexity results with those of the standard online SGD in Corollary 3.2, we conclude that data-subsampling can improve the sample complexity by a factor of (log 1 ϵ ) 1 θ and ϵ − 1 θ for geometric ϕ

Original: -mixing and algebraic ϕ -mixing data process, respectively.
Revised: -mixing and algebraic ϕ -mixing data process, respectively.

Original: Intuitively, this is because with data subsampling, we can choose a sufficiently large subsampling period r to decouple the data dependence in the term ϕ ξ ( rτ ) , as opposed to choosing a large τ in
Revised: Intuitively, this is because with data subsampling, we can choose a sufficiently large subsampling period r to decouple the data dependence in the term ϕ ξ ( rτ ) , as opposed to choosing a large τ in

Original: Corollary 3.2.
Revised: Corollary 3.2.

Original: In this way, the order of the dominant term (cid:112) τn log τδ is reduced.
Revised: In this way, the order of the dominant term (cid:112) τn log τδ is reduced.

Original: Therefore, when the data is highly dependent, it is beneficial to subsample the dependent data before performing SGD.
Revised: Therefore, when the data is highly dependent, it is beneficial to subsample the dependent data before performing SGD.

Original: We also note another advantage of using data-subsampling, i.e., it only requires computing the stochastic gradients of the subsampled data, and therefore can substantially reduce the computation complexity.
Revised: We also note another advantage of using data-subsampling, i.e., it only requires computing the stochastic gradients of the subsampled data, and therefore can substantially reduce the computation complexity.

Original:  COMPLEXITY OF ONLINE SGD WITH MINI-BATCH SAMPLING
Revised:  COMPLEXITY OF ONLINE SGD WITH MINI-BATCH SAMPLING

Original: Although the data-subsampling scheme studied in the previous section helps improve the sample complexity of online
Revised: Although the data-subsampling scheme studied in the previous section helps improve the sample complexity of online

Original: SGD, it does not leverage the full information of all the queried data.
Revised: SGD, it does not leverage the full information of all the queried data.

Original: In particular, when the data is highly dependent, we need to choose a large period r to reduce data dependence, and this will throw away a huge amount of valuable samples.
Revised: In particular, when the data is highly dependent, we need to choose a large period r to reduce data dependence, and this will throw away a huge amount of valuable samples.

Original: In this section, we study online SGD with another popular data sampling scheme that leverages the full information of all the sampled data, i.e., the mini-batch sampling scheme.
Revised: In this section, we study online SGD with another popular data sampling scheme that leverages the full information of all the sampled data, i.e., the mini-batch sampling scheme.

Original: We show that this simple and widely used scheme can effectively reduce data dependence without skipping data samples, and can achieve an improved sample complexity over online SGD with subsampling.
Revised: We show that this simple and widely used scheme can effectively reduce data dependence without skipping data samples, and can achieve an improved sample complexity over online SGD with subsampling.

Original: Specifically, consider a data stream { ξ t } t with ϕmixing dependent samples.
Revised: Specifically, consider a data stream { ξ t } t with ϕmixing dependent samples.

Original: We rearrange the data samples into a stream of mini-batches { x t } t , where each mini-batch x t contains B samples, i.e., x t = { ξ ( t − 1) B +1 , ξ ( t − 1) B +2 , . . .
Revised: We rearrange the data samples into a stream of mini-batches { x t } t , where each mini-batch x t contains B samples, i.e., x t = { ξ ( t − [Equation] ) B + [Equation]

Revised: [Equation]

Revised: ( t − 1) B +2 , . .

Original:  , ξ tB } .
Revised: . , ξ tB } .

Original: Then, we perform minibatch SGD update as follows.
Revised: Then, we perform minibatch SGD update as follows.

Original: (SGD with mini-batch sampling):
Revised: (SGD with mini-batch sampling):

Original: [Equation]
Revised: [Equation]

Original: Performing online learning with mini-batch sampling has several advantages.
Revised: Performing online learning with mini-batch sampling has several advantages.

Original: First, it substantially reduce the optimization variance and allows to use a large learning rate to facilitate the convergence of the algorithm.
Revised: First, it substantially reduce the optimization variance and allows to use a large learning rate to facilitate the convergence of the algorithm.

Original: As a comparison,
Revised: As a comparison,

Original: SGD with subsampling suffers from a large optimization variance.
Revised: SGD with subsampling suffers from a large optimization variance.

Original: Second, unlike subsampling, mini-batch sampling utilizes the information of all the queried data samples to improve the performance of the model.
Revised: Second, unlike subsampling, mini-batch sampling utilizes the information of all the queried data samples to improve the performance of the model.

Original: Moreover, as we show in the following lemma, mini-batch sampling substantially reduces the stochastic bias caused by the data dependence.
Revised: Moreover, as we show in the following lemma, mini-batch sampling substantially reduces the stochastic bias caused by the data dependence.

Original: In the sequel, we denote F ( w ; x ) :
Revised: In the sequel, we denote F ( w ; x ) :

Original: = 1 B (cid:80) ξ ∈ x F ( w ; ξ ) as the average loss on a mini-batch of samples.
Revised: = 1 B (cid:80) ξ ∈ x F ( w ; ξ ) as the average loss on a mini-batch of samples.

Original: With a bit abuse of notation, we also define {F t } t as the canonical filtration generated by the mini-batch samples { x t } t .
Revised: With a bit abuse of notation, we also define {F t } t as the canonical filtration generated by the mini-batch samples { x t } t .

Original: Lemma 5.1.
Revised: Lemma 5.1.

Original: Let Assumption 2.1 hold and consider the mini-batch data stream { x t } t .
Revised: Let Assumption 2.1 hold and consider the mini-batch data stream { x t } t .

Original: Then, for any w, v ∈ W measureable with regard to F t and any τ ∈ N , it holds that
Revised: Then, for any w, v ∈ W measureable with regard to F t and any τ ∈ N , it holds that

Original: [Equation]
Revised: [Equation]

Original: With dependent data, the above lemma shows that we can approximate the population risk f ( w ) by the conditional expectation E [ F ( w ; x t + τ ) |F t ] , which involves the mini-batch x t
Revised: With dependent data, the above lemma shows that we can approximate the population risk f ( w ) by the conditional expectation E [ F ( w ; x t + τ ) |F t ] , which involves the mini-batch x t

Original: + τ that is τ steps ahead of the filtration F t .
Revised: + τ that is τ steps ahead of the filtration F t .

Original: Intuitively, by the definition of ϕ -mixing process, as τ gets larger, the distribution of x t + τ conditional on F t gets closer to the stationary distribution µ .
Revised: Intuitively, by the definition of ϕ -mixing process, as τ gets larger, the distribution of x t + τ conditional on F t gets closer to the stationary distribution µ .

Original: In general, the estimation bias
Revised: In general, the estimation bias

Original: GRB (cid:80) Bi
Revised: GRB (cid:80) Bi

Original: =1 ϕ ξ ( τB + i ) depends on both the batch size and the accumulated mixing coefficient over the corresponding batch of samples.
Revised: =1 ϕ ξ ( τB + i ) depends on both the batch size and the accumulated mixing coefficient over the corresponding batch of samples.

Original: To provide a concrete understanding, below we calculate the estimation bias in eq.
Revised: To provide a concrete understanding, below we calculate the estimation bias in eq.

Original: (6) for various ϕ -mixing processes.
Revised: (6) for various ϕ -mixing processes.

Original: • Geometric ϕ
Revised: • Geometric ϕ

Original: -mixing:
Revised: -mixing:

Original: In this case, (cid:80) Bi =1 ϕ ξ ( τB + i ) ≤ (cid:80) ∞
Revised: In this case, (cid:80) Bi =1 ϕ ξ ( τB + i ) ≤ (cid:80) ∞

Original: i =1 ϕ ξ ( i ) =
Revised: i =1 ϕ ξ ( i ) =

Original: O (1) .
Revised: O (1) .

Original: Hence, the estimation bias is in the order of O ( GRB ) .
Revised: Hence, the estimation bias is in the order of O ( GRB ) .

Original: • Fast algebraic ϕ
Revised: • Fast algebraic ϕ

Original: -mixing
Revised: -mixing

Original: ( θ ≥ 1 ):
Revised: ( θ ≥ 1 ):

Original: In this case, (cid:80) Bi =1 ϕ ξ ( τB + i ) ≤ (cid:80)
Revised: In this case, (cid:80) Bi =1 ϕ ξ ( τB + i ) ≤ (cid:80)

Original: ∞
Revised: ∞

Original: i =1 ϕ ξ ( i ) =
Revised: i =1 ϕ ξ ( i ) =

Original: (cid:101) O (1) .
Revised: (cid:101) O (1) .

Original: Hence, the estimation bias is in the order of (cid:101) O
Revised: Hence, the estimation bias is in the order of (cid:101) O

Original: ( GRB ) , where (cid:101)
Revised: ( GRB ) , where (cid:101)

Original: O hides all logarithm factors.
Revised: O hides all logarithm factors.

Original: • Slow algebraic ϕ
Revised: • Slow algebraic ϕ

Original: -mixing
Revised: -mixing

Original: ( 0 < θ < 1 ):
Revised: ( 0 < θ < 1 ):

Original: In this case, (cid:80) Bi =1 ϕ ξ ( τB + i ) ≤
Revised: In this case, (cid:80) Bi =1 ϕ ξ ( τB + i ) ≤

Original: O (( τB ) 1 − θ ) .
Revised: O (( τB ) 1 − θ ) .

Original: Hence, the estimation bias is in the order of O ( GRτ 1 − θ B θ ) .
Revised: Hence, the estimation bias is in the order of O ( GRτ 1 − θ B θ ) .

Original: It can be seen that if the mixing coefficient converges fast, i.e., either geometrically or fast algebraically, then the data dependence has a negligible impact on the estimation error.
Revised: It can be seen that if the mixing coefficient converges fast, i.e., either geometrically or fast algebraically, then the data dependence has a negligible impact on the estimation error.

Original: On the other hand, when the mixing coefficient converges slow algebraically, it substantially increases the estimation bias, but it is still beneficial to use a large batch size.
Revised: On the other hand, when the mixing coefficient converges slow algebraically, it substantially increases the estimation bias, but it is still beneficial to use a large batch size.

Original: We obtain the following convergence error bound for online
Revised: We obtain the following convergence error bound for online

Original: SGD with mini-batch sampling over dependent data.
Revised: SGD with mini-batch sampling over dependent data.

Original: Theorem 5.2.
Revised: Theorem 5.2.

Original: Let Assumption 2.1 and 3.1 hold.
Revised: Let Assumption 2.1 and 3.1 hold.

Original: Apply SGD with mini-batch sampling to solve the stochastic optimization problem (P) over ϕ -mixing dependent data process and assume that it achieves regret R n .
Revised: Apply SGD with mini-batch sampling to solve the stochastic optimization problem (P) over ϕ -mixing dependent data process and assume that it achieves regret R n .

Original: Then, for any τ ∈ N and any minimizer w ∗ with probability at least 1 − δ , the averaged predictor (cid:98) w n := 1 n (cid:80) nt =1 w ( t ) satisfies f ( (cid:98) w n ) − f ( w ∗ ) ≤ R nn + G ( τ − 1) n n − τ +1 (cid:88) t = κ ( t ) +
Revised: Then, for any τ ∈ N and any minimizer w ∗ with probability at least 1 − δ , the averaged predictor (cid:98) w n := 1 n (cid:80) nt =1 w ( t ) satisfies f ( (cid:98) w n ) − f ( w ∗ ) ≤ R n

Original: [Equation]
Revised: [Equation]

Original: To further understand the order of the above bound, a standard regret analysis shows that mini-batch SGD achieves the regret R n n = (cid:101)
Revised: To further understand the order of the above bound, a standard regret analysis shows that mini-batch SGD achieves the regret R n n = (cid:101)

Original: O ( (cid:113) (cid:80) nj =1 ϕ ( j ) nB ) and κ ( t )
Revised: O ( (cid:113) (cid:80) nj =1 ϕ ( j ) nB ) and κ ( t )

Original: ≡ O ( (cid:113) Bn ) (see
Revised: ≡ O ( (cid:113) Bn ) (see

Original: Theorem C.3 for the proof).
Revised: Theorem C.3 for the proof).

Original: Consequently, the above convergence error bound reduces to the following bound.
Revised: Consequently, the above convergence error bound reduces to the following bound.

Original: [Equation]
Revised: [Equation]

Original: Such a bound further implies the following sample complexity results of online SGD with mini-batch sampling under different convergence rates of the mixing coefficient ϕ ξ .
Revised: Such a bound further implies the following sample complexity results of online SGD with mini-batch sampling under different convergence rates of the mixing coefficient ϕ ξ .

Original: Corollary 5.3.
Revised: Corollary 5.3.

Original: The sample complexity of online SGD with mini-batch sampling for achieving an ϵ convergence error over ϕ -mixing dependent data is
Revised: The sample complexity of online SGD with mini-batch sampling for achieving an ϵ convergence error over ϕ -mixing dependent data is

Original: • If the data is geometric ϕ -mixing with parameter θ > 0 , then we choose τ = 1 , B
Revised: • If the data is geometric ϕ -mixing with parameter θ > 0 , then we choose τ = 1 , B

Original: = O ( ϵ − 1 ) , n =
Revised: = O ( ϵ − 1 ) , n =

Original: O ( ϵ − 1 ) .
Revised: O ( ϵ − 1 ) .

Original: The overall sample complexity is nB
Revised: The overall sample complexity is nB

Original: =
Revised: =

Original: O ( ϵ − 2 ) .
Revised: O ( ϵ − 2 ) .

Original: • If the data is fast algebraic ϕ -mixing with parameter θ ≥ 1 , then we choose τ = 1 , B
Revised: • If the data is fast algebraic ϕ -mixing with parameter θ ≥ 1 , then we choose τ = 1 , B

Original: = O ( ϵ − 1 ) , n =
Revised: = O ( ϵ − 1 ) , n =

Original: O ( ϵ − 1 ) .
Revised: O ( ϵ − 1 ) .

Original: The overall sample complexity is nB =
Revised: The overall sample complexity is nB =

Original: (cid:101) O
Revised: (cid:101) O

Original: ( ϵ − 2 ) .
Revised: ( ϵ − 2 ) .

Original: • If the data is slow algebraic ϕ -mixing with param- eter 0 < θ < 1 , then we choose τ = 1 , B
Revised: • If the data is slow algebraic ϕ -mixing with parameter 0 < θ < 1 , then we choose τ = 1 , B

Original: = O
Revised: = O

Original: ( ϵ − 1 θ ) , n = O ( ϵ − 1 ) .
Revised: ( ϵ − 1 θ ) , n = O ( ϵ − 1 ) .

Original: The overall sample complexity is nB
Revised: The overall sample complexity is nB

Original: =
Revised: =

Original: O ( ϵ − 1 − 1 θ ) .
Revised: O ( ϵ − 1 − 1 θ ) .

Original: Remark.
Revised: Remark.

Original: This corollary provides a potential way to set the optimal batch size B with respect to the mixing rate θ .
Revised: This corollary provides a potential way to set the optimal batch size B with respect to the mixing rate θ .

Original: Specifically, we can leverage Lemma 5.1 to estimate the dependence parameter θ .
Revised: Specifically, we can leverage Lemma 5.1 to estimate the dependence parameter θ .

Original: Choosing batch size B = 1 , the upper bound of Lemma 5.1 becomes GRϕ ξ ( τ + 1) , which is proportional to the mixing coefficient ϕ ξ ( τ + 1) .
Revised: Choosing batch size B = 1 , the upper bound of Lemma 5.1 becomes GRϕ ξ ( τ + 1) , which is proportional to the mixing coefficient ϕ ξ ( τ + 1) .

Original: Therefore, the left-hand side E (cid:2) F ( w ; x t + τ ) − F ( v ; x t + τ ) |F
Revised: Therefore, the left-hand side E (cid:2) F ( w ; x t + τ ) − F ( v ; x t + τ ) |F

Original: t (cid:3) − (cid:0) f
Revised: t (cid:3) − (cid:0) f

Original: ( w ) − f ( v ) (cid:1) of Lemma 5.1 serves as an estimator, which can be estimated by (conditional) sample average queried at any fixed points ω, v .
Revised: ( w ) − f ( v ) (cid:1) of Lemma 5.1 serves as an estimator, which can be estimated by (conditional) sample average queried at any fixed points ω, v .

Original: Once we estimate this quantity with various values of τ , we can use regression to find out the type of convergence for ϕ ξ ( τ ) and estimate the parameter θ .
Revised: Once we estimate this quantity with various values of τ , we can use regression to find out the type of convergence for ϕ ξ ( τ ) and estimate the parameter θ .

Original: With the estimated θ , we then follow this corollary to choose the batch size.
Revised: With the estimated θ , we then follow this corollary to choose the batch size.

Original: It can be seen that online SGD with mini-batch sampling improves the sample complexity of online SGD with subsampling by a factor of O ((log 1 ϵ ) 1 θ ) , (cid:101) O
Revised: It can be seen that online SGD with mini-batch sampling improves the sample complexity of online SGD with subsampling by a factor of O ((log 1 ϵ ) 1 θ ) , (cid:101) O

Original: ( ϵ − 1 θ ) and O ( ϵ − 1 ) for geometric ϕ -mixing, fast algebraic ϕ -mixing and slow algebraic ϕ -mixing data samples, respectively.
Revised: ( ϵ − 1 θ ) and O ( ϵ − 1 ) for geometric ϕ -mixing, fast algebraic ϕ -mixing and slow algebraic ϕ -mixing data samples, respectively.

Original: This shows that mini-batch sampling can effectively reduce the bias caused by data dependence and leverage the full information of all the data samples to improve the learning performance.
Revised: This shows that mini-batch sampling can effectively reduce the bias caused by data dependence and leverage the full information of all the data samples to improve the learning performance.

Original: To provide an intuitive explanation, this is because with mini-batch sampling, we can choose a sufficiently large batch size B to reduce the bias caused by the data dependence and then choose a small auxiliary parameter τ = 1 .
Revised: To provide an intuitive explanation, this is because with mini-batch sampling, we can choose a sufficiently large batch size B to reduce the bias caused by the data dependence and then choose a small auxiliary parameter τ = 1 .

Original: As a comparison, to control the bias caused by data dependence, the standard online SGD needs to choose a very large τ and the online SGD with subsampling needs to choose a large subsampling period r that skips a huge amount of valuable data samples, especially when the mixing coefficient converges slowly.
Revised: As a comparison, to control the bias caused by data dependence, the standard online SGD needs to choose a very large τ and the online SGD with subsampling needs to choose a large subsampling period r that skips a huge amount of valuable data samples, especially when the mixing coefficient converges slowly.

Original: Therefore, our result proves that it is beneficial to use mini-batch data sampling when the data samples are highly dependent.
Revised: Therefore, our result proves that it is beneficial to use mini-batch data sampling when the data samples are highly dependent.

Original: Our proof of the high-probability bound in Theorem 5. for SGD with mini-batch sampling involves substantial new developments compared with the proof of [Agarwal and
Revised: Our proof of the high-probability bound in Theorem 5. for SGD with mini-batch sampling involves substantial new developments compared with the proof of [Agarwal and

Original: Duchi, 2012].
Revised: Duchi, 2012].

Original: Next, we elaborate on our technical novelty.
Revised: Next, we elaborate on our technical novelty.

Original: • In [Agarwal and Duchi, 2012], they defined the following random variable
Revised: • In [Agarwal and Duchi, 2012], they defined the following random variable

Original: [Equation]
Revised: [Equation]

Original: As this random variable involves only one sample ξ t + τ −  they bound the bias term X it − E
Revised: As this random variable involves only one sample ξ t + τ − 1 , they bound the bias term X it − E

Original: [ X it |F it − 1 ] as a universal constant.
Revised: [ X it |F it − 1 ] as a universal constant.

Original: As a comparison the random vari able X it would involve a mini-batch of samples x t + τ −in our analysis.With the
Revised: As a comparison, the random variable X it would involve a mini-batch of samples x t + τ −in our analysis.

Original: mini batch structure, the bias X it − E
Revised: With the mini-batch structure, the bias X it − E

Original: [ X it |F it − 1 ] can be written as an average of B zero-mean dependent random variables, which is close to zero with high probability due to the concentration phenomenon.
Revised: [ X it |F it − 1 ] can be written as an average of B zero-mean dependent random variables, which is close to zero with high probability due to the concentration phenomenon.

Original: Consequently, we are able to apply a Bernstein-type inequality developed in [Delyon et al., 2009] for dependent stochastic process to obtain an improved bias bound from O (1) to (cid:101)
Revised: Consequently, we are able to apply a Bernstein-type inequality developed in [Delyon et al., 2009] for dependent stochastic process to obtain an improved bias bound from O (1) to (cid:101)

Original: O
Revised: O

Original: (1 / √ B ) .
Revised: (1 / √ B ) .

Original: This is critical for obtaining the improved bound.
Revised: This is critical for obtaining the improved bound.

Original: • Second, with the improved high-probability bias bound mentioned above, the remaining proof of [Agarwal and Duchi, 2012] no longer holds.
Revised: • Second, with the improved high-probability bias bound mentioned above, the remaining proof of [Agarwal and Duchi, 2012] no longer holds.

Original: Specifically, we can no longer apply the Azuma’s inequality to bound the accumulated bias (cid:80) t ( X it − E
Revised: Specifically, we can no longer apply the Azuma’s inequality to bound the accumulated bias (cid:80) t ( X it − E

Original: [ X it |F it − 1 ]) , as each bias term is no longer bounded with probability one.
Revised: [ X it |F it − 1 ]) , as each bias term is no longer bounded with probability one.

Original: To address this issue, we developed a generalized Azuma’s inequality for martingale differences in Lemma B. based on Proposition 34 of [Tao et al., 2015] for independent zero-mean random variables.
Revised: To address this issue, we developed a generalized Azuma’s inequality for martingale differences in Lemma B. based on Proposition 34 of [Tao et al., 2015] for independent zero-mean random variables.

Original:  Third, we develop a high-probability regret bound for online SGD with mini-batch sampling over dependent data so that it can be integrated with the highprobability convergence bound in Theorem 5.2.
Revised: • Third, we develop a high-probability regret bound for online SGD with mini-batch sampling over dependent data so that it can be integrated with the highprobability convergence bound in Theorem 5.2.

Original: To our best knowledge, the regret of SGD over dependent data has not been studied before.
Revised: To our best knowledge, the regret of SGD over dependent data has not been studied before.

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: In this section, we examine our SGD theory via two experiments on stochastic quadratic programming and neural network training with dependent data.
Revised: In this section, we examine our SGD theory via two experiments on stochastic quadratic programming and neural network training with dependent data.

Original: 6.1 STOCHASTIC QUADRATIC PROGRAMMING
Revised: 6.1 STOCHASTIC QUADRATIC PROGRAMMING

Original: We consider the following stochastic convex quadratic optimization problem.
Revised: We consider the following stochastic convex quadratic optimization problem.

Original: [Equation]
Revised: [Equation]

Original: where A ⪰ 0 is a fixed positive semi-definite matrix and µ is the uniform distribution on [0 , 1] d .
Revised: where A ⪰ 0 is a fixed positive semi-definite matrix and µ is the uniform distribution on [0 , 1] d .

Original: Then, following the construction in [Jarner and Roberts, 2002], we generate an algebraic ϕ -mixing
Revised: Then, following the construction in [Jarner and Roberts, 2002], we generate an algebraic ϕ -mixing

Original: Markov chain that has the stationary distribution µ .
Revised: Markov chain that has the stationary distribution µ .

Original: In particular, its mixing coefficient ϕ ξ ( k ) converges at a sublinear convergence rate k − 1 r , where r >is a parameter that controls the speed of convergence.
Revised: In particular, its mixing coefficient ϕ ξ ( k ) converges at a sublinear convergence rate k − 1 r , where r >is a parameter that controls the speed of convergence.

Original: Please refer to Appendix D for more details of the experiment setup.
Revised: Please refer to Appendix D for more details of the experiment setup.

Original: We first estimate the following stochastic bias at the fixed origin point w = 0 d .
Revised: We first estimate the following stochastic bias at the fixed origin point w = 0 d .

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken over the randomness of the mini-batch of samples queried at time τ ∈ N .
Revised: where the expectation is taken over the randomness of the mini-batch of samples queried at time τ ∈ N .

Original: Such a bias is affected by several factors, including the time gap τ , the batch size B and the convergence rate parameter r of the mixing coefficient.
Revised: Such a bias is affected by several factors, including the time gap τ , the batch size B and the convergence rate parameter r of the mixing coefficient.

Original: In Figure 1, we investigate the impact of these factors on the stochastic bias, and we use 10k Monte Carlo samples to estimate the stochastic bias.
Revised: In Figure 1, we investigate the impact of these factors on the stochastic bias, and we use 10k Monte Carlo samples to estimate the stochastic bias.

Original: The top two figures fix the batch size, and it can be seen that the bias decreases as τ increases, which matches the definition of the ϕ -mixing process.
Revised: The top two figures fix the batch size, and it can be seen that the bias decreases as τ increases, which matches the definition of the ϕ -mixing process.

Original: Also, a faster-mixing Markov chain (i.e., smaller r ) leads to a smaller bias.
Revised: Also, a faster-mixing Markov chain (i.e., smaller r ) leads to a smaller bias.

Original: In particular, with batch size B =and a slow-mixing chain r = 2 , it takes an unacceptably large τ to achieve a relatively small bias.
Revised: In particular, with batch size B =and a slow-mixing chain r = 2 , it takes an unacceptably large τ to achieve a relatively small bias.

Original: This provides an empirical justification to Corollary 3.2 and explains why the standard SGD suffers from a high sample complexity over highly dependent data.
Revised: This provides an empirical justification to Corollary 3.2 and explains why the standard SGD suffers from a high sample complexity over highly dependent data.

Original: Moreover, as the batch size gets larger, one can achieve a numerically smaller bias, which matches our Lemma 5.1.
Revised: Moreover, as the batch size gets larger, one can achieve a numerically smaller bias, which matches our Lemma 5.1.

Original: The bottom two figures fix the convergence rate parameter of the mixing coefficient, and it can be seen that increasing the batch size significantly reduces the bias.
Revised: The bottom two figures fix the convergence rate parameter of the mixing coefficient, and it can be seen that increasing the batch size significantly reduces the bias.

Original: Consequently, instead of choosing a large τ to reduce the bias, one can simply choose a large batch size B = 100 and set τ = 1 .
Revised: Consequently, instead of choosing a large τ to reduce the bias, one can simply choose a large batch size B = 100 and set τ = 1 .

Original: This observation matches and justifies our theoretical results in Corollary 5.3.
Revised: This observation matches and justifies our theoretical results in Corollary 5.3.

Original: We further compare the convergence of SGD, SGD with subsampling and mini-batch SGD.
Revised: We further compare the convergence of SGD, SGD with subsampling and mini-batch SGD.

Original: Here, we set r = 2 to generate highly dependent data samples.
Revised: Here, we set r = 2 to generate highly dependent data samples.

Original: We set learning rate η = 0 .
Revised: We set learning rate η = 0 .

Original: 01 for both SGD and SGD subsampling
Revised: 01 for both SGD and SGD with subsampling, and set learning rate η = 0 .

Original: 01 × 
Revised: 01 × (cid:113)

Original: [Figure]
Revised: [Equation]

Original:  for mini-batch SGD with batch size B = 100 , as suggested by Theorem C.3 in the appendix.
Revised: 100 1 / 4 for mini-batch SGD with batch size B = 100 , as suggested by Theorem C.3 in the appendix.

Original: The results are plotted in Figure 2, where each curve corresponds to the mean of 100 independent trails.
Revised: The results are plotted in Figure 2, where each curve corresponds to the mean of 100 independent trails.

Original: It can be seen that SGD with subsampling achieves a lower loss than the standard SGD asymptotically, due to the use of less dependent data.
Revised: It can be seen that SGD with subsampling achieves a lower loss than the standard SGD asymptotically, due to the use of less dependent data.

Original: Moreover, mini-batch SGD achieves the smallest asymptotic loss.
Revised: Moreover, mini-batch SGD achieves the smallest asymptotic loss.

Original: All these observations are consistent with ourtheoretical results.
Revised: All these observations are consistent with our results.

Original: 6.2 NEURAL NETWORK TRAINING
Revised: 6.2 NEURAL NETWORK TRAINING

Original: We further apply these online SGD algorithms to train a convolutional neural network with the MNIST dataset [Lecun et al., 1998].
Revised: We further apply these online SGD algorithms to train a convolutional neural network with the MNIST dataset [Lecun et al., 1998].

Original: The network consists of two convolution blocks followed by two fully connected layers.
Revised: The network consists of two convolution blocks followed by two fully connected layers.

Original: Specifically, each convolution block contains a convolution layer, a maxpooling layer with stride step 2 , and a ReLU activation layer.
Revised: Specifically, each convolution block contains a convolution layer, a maxpooling layer with stride step 2 , and a ReLU activation layer.

Original: The convolution layers in the two blocks have input channel 1 , 10 and output channel 10 , 20 , respectively, and both of them have kernel size 5 , stride step 1 and with no padding.
Revised: The convolution layers in the two blocks have input channel 1 , 10 and output channel 10 , 20 , respectively, and both of them have kernel size 5 , stride step 1 and with no padding.

Original: The two fully connected layers have input dimensions 320 , 50 and output dimensions 50 , 10 , respectively.
Revised: The two fully connected layers have input dimensions 320 , 50 and output dimensions 50 , 10 , respectively.

Original: To generate a stream of dependent data, we first generate an algebraic ϕ -mixing
Revised: To generate a stream of dependent data, we first generate an algebraic ϕ -mixing

Original: Markov chain { X t } t with the construction provided in [Jarner and Roberts, 2002].
Revised: Markov chain { X t } t with the construction provided in [Jarner and Roberts, 2002].

Original: Then, we map each X t to a label of the MNIST dataset { 0 , 1 , 2 , . .
Revised: Then, we map each X t to a label of the MNIST dataset { 0 , 1 , 2 , . .

Original: , 9 } , and uniformly sample an image at random from the corresponding image class.
Revised: , 9 } , and uniformly sample an image at random from the corresponding image class.

Original: This data-generating process generates a dependent data stream with a ϕ ξ -mixing coefficient approximately k − 1 r .
Revised: This data-generating process generates a dependent data stream with a ϕ ξ -mixing coefficient approximately k − 1 r .

Original: We first test the performance of SGD with a fixed batch size and different correlation coefficients.
Revised: We first test the performance of SGD with a fixed batch size and different correlation coefficients.

Original: Specifically, we choose batch size B = 1000 and consider different correlation coefficients r ∈ { 1 . 0 , 1 . 25 , 1 . 5 , 1 . 75 , 2 . 0 } .
Revised: Specifically, we choose batch size B = 1000 and consider different correlation coefficients r ∈ { 1 . 0 , 1 . 25 , 1 . 5 , 1 . 75 , 2 . 0 } .

Original: Here, a larger r implies higher data dependency.
Revised: Here, a larger r implies higher data dependency.

Original: Figure 3 (left) plots the experiment results.
Revised: Figure 3 (left) plots the experiment results.

Original: It can be seen that with an increasing correlation coefficient, the convergence of SGD is slower.
Revised: It can be seen that with an increasing correlation coefficient, the convergence of SGD is slower.

Original: We further fix the correlation coefficient r = 2 . 0 and vary the batch size B ∈ { 8 , 16 , 32 , 64 , 128 } .
Revised: We further fix the correlation coefficient r = 2 . 0 and vary the batch size B ∈ { 8 , 16 , 32 , 64 , 128 } .

Original: Figure 3 (right) plots the experiment results.
Revised: Figure 3 (right) plots the experiment results.

Original: It can be seen that SGD with the largest batch size B = 128 achieves the smallest asymptotic loss among all choices of batch sizes.
Revised: It can be seen that SGD with the largest batch size B = 128 achieves the smallest asymptotic loss among all choices of batch sizes.

Original: In particular,
Revised: In particular,

Original: SGD with a larger batch size tends to converge faster over such dependent data.
Revised: SGD with a larger batch size tends to converge faster over such dependent data.

Original: This also matches our theoretical analysis and it implies that mini-batch SGD with a large batch size can benefit neural network training over dependent data.
Revised: This also matches our theoretical analysis and it implies that mini-batch SGD with a large batch size can benefit neural network training over dependent data.

Original:  CONCLUSION
Revised:  CONCLUSION

Original: In this study, we investigate the convergence property of
Revised: In this study, we investigate the convergence property of

Original: SGD under various popular stochastic update schemes over highly dependent data.
Revised: SGD under various popular stochastic update schemes over highly dependent data.

Original: Unlike the conventional i.i.d.
Revised: Unlike the conventional i.i.d.

Original: data setting in which the stochastic update schemes do not affect the sample complexity of SGD, the convergence of SGD in the data-dependent setting critically depends on the structure of the stochastic update scheme.
Revised: data setting in which the stochastic update schemes do not affect the sample complexity of SGD, the convergence of SGD in the data-dependent setting critically depends on the structure of the stochastic update scheme.

Original: In particular, we show that both data subsampling and mini-batch sampling can substantially improve the sample complexity of SGD over highly dependent data.
Revised: In particular, we show that both data subsampling and mini-batch sampling can substantially improve the sample complexity of SGD over highly dependent data.

Original: Our study takes one step forward toward understanding the theoretical limits of stochastic optimization over dependent data, and it opens many directions for future study.
Revised: Our study takes one step forward toward understanding the theoretical limits of stochastic optimization over dependent data, and it opens many directions for future study.

Original: For example, it is interesting to further explore the impact of algorithm structure on the sample complexity of stochastic reinforcement learning algorithms.
Revised: For example, it is interesting to further explore the impact of algorithm structure on the sample complexity of stochastic reinforcement learning algorithms.

Original: Also, it is important to develop advanced algorithm update schemes that can facilitate the convergence of learning over highly dependent data.
Revised: Also, it is important to develop advanced algorithm update schemes that can facilitate the convergence of learning over highly dependent data.

Original: Acknowledgements
Revised: Acknowledgements

Original: The work of Shaocong Ma, Ziyi Chen and Yi Zhou was supported in part by U.S. National Science Foundation under the Grants CCF-2106216 and DMS-2134223.The work of Y. Liang was supported in part by U.S. National Science Foundation under the grants CCF-1909291 and ECCS-2113860.
Revised: The work of Shaocong Ma, Ziyi Chen and Yi Zhou was supported in part by U.S. National Science Foundation under the Grants CCF-2106216 and DMS-2134223.

Original: A PROOF OF COROLLARY 4.

Original: In this section, we analyze the convergence error bound of the SGD with data-subsampling in (3).

Original: Given a ϕ ξ -mixing data stream { ξ 1 , ξ 2 , ξ 3 , . . .

Original: } , we consider the following subsampled data stream { ξ 1 , ξ r +1 , ξ 2 r +1 , . . . } .

Original: Let F be the canonical filtration generated by { x t } .

Original: Then the subsampled data stream { ξ tr +1 } t is ϕ

Original: rξ -mixing with the mixing coefficient given by

Original: [Equation]

Original: With this mixing coefficient, we can apply Theorem 2 of [Agarwal and Duchi, 2012] and obtain the following convergence error bound for any τ ∈ N .

Original: [Equation]

Original: Consider the standard SGD with a diminishing learning rate, we have κ ( t ) =

Original: O ( 1 √ t ) and R n = O ( √ n ) .

Original: Then, the convergence error bound becomes

Original: [Equation]

Original: Note that when n is sufficiently large and τ = O (1) , the term τn is dominated by the term 1 √ n .

Original: So we can omit this term in the above equation.

Original: Also note that only the right-hand side depends on τ , thus the inequality still holds by taking infimum of the right-hand side with respect to τ , and the following desired bound follows.

Original: [Equation]

Original: The above result further implies the following sample complexity results for different convergence rates of the mixing coefficient.

Original: • Geometric ϕ

Original: -mixing:

Original: In this case, ϕ ξ ( k ) ≤ O (exp( − k θ )) for some θ > 0 .

Original: Set the last term ϕ ξ ( rτ ) =

Original: O ( ϵ ) .

Original: We obtain that rτ

Original: = O

Original: ((log 1 ϵ ) 1 θ ) .

Original: Further set the second term τ − 1 √ n = O ( ϵ ) .

Original: We obtain that nτ − 2 =

Original: O ( ϵ − 2 ) .

Original: By choosing τ =

Original: O

Original: , the sample complexity

Original: is in the

Original: order of ϵ

Original: [Equation]

Original: • Algebraic ϕ

Original: -mixing:

Original: In this case, ϕ ξ ( k ) ≤ O ( k − θ ) for some θ > 0 .

Original: Set the last term ϕ ξ ( rτ ) =

Original: O ( ϵ ) .

Original: We obtain that τr = O ( ϵ − 1 θ ) .

Original: Set the second term τ − 1 √ n = O ( ϵ ) .

Original: We obtain that nτ − 2 =

Original: O ( ϵ − 2 ) .

Original: By setting τ = O (1) , the sample complexity is in the order of

Original: [Equation]

Original: B PROOF OF THEOREM 5.

Original: Define N := { 1 , 2 , 3 , . . . } .

Original: Also, recall that we are considering a data stream divided into small mini-batches.

Original: For convenience, we re-label the data stream { ξ 1 , ξ 2 , ξ 3 , . . . } as follows to explicitly indicate its mini-batch index.

Original: [Equation]

Original: The canonical filtration generated by the re-labeled data stream is denoted by (cid:98) F .

Original: Also, when the batch size is clear in the context, we denote the data in the specified mini-batch as x .

Original: For example, we use x t to represent the t -th mini-batch { ξ (1) t , ξ (2) t , . . .

Original: , ξ ( B ) t } .

Original: Then we can re-writhe the above data stream as { x 1 , x 2 , x 3 . . . . } .

Original: We denote the canonical filtration generated by the above sequence as F .

Original: Note that we have the following relation:

Original: [Equation]

Original: In summary, when we analyze the mini-batch SGD dynamics, we use the filtration F , and when we need to consider intra-batch samples, we use the filtration (cid:98) F .

Original: B.1 PROOF SKETCH

Original: Here listed the proof structure of Theorem 5.2 and the difference between our results and Agarwal and Duchi [2012].

Original: • Following Proposition B.2 (or Proposition 2 of Agarwal and Duchi [2012]), we obtain the convergence bound

Original: (cid:80) nt =

Original: [Equation]

Original: [Equation]

Original: where Z n = (cid:80) τi =1 (cid:80) t ∈I ( i )

Original: [ X it − E

Original: [ X it |F it − 1 ]] + (cid:80) τi

Original: =1 (cid:80) t ∈I ( i ) E

Original: [ X it |F it − 1 ] .

Original: In the next three bullets, we bound the two terms involved in Z n separately and bound the regret R n .

Original: • To bound the first term of Z n , Agarwal and Duchi [2012] choose to upper bound its summand as a constant,

Original: i

Original: e., X it − E

Original: [ X it |F it − 1 ] ≤ 2 GR almost surely, which further enables them to apply Azuma’s inequality to bound the summation (cid:80) t ∈I

Original: ( i ) [ X it − E

Original: [ X it |F it − 1 ]] with high probability.

Original: However, when X it involves a mini-batch of samples, bounding the summand X it − E

Original: [ X it |F it − 1 ] by constant will lead to a lose probabilistic bound (and hence a lose convergence rate & sample complexity).

Original: As a comparison, we leverage the mini-batch structure of the summand

Original: X

Original: it − E

Original: [ X it |F it − 1 ] and bound it using

Original: Bernstein’s inequality for dependent process (Lemma B.4) to obtain that | X it − E

Original: [ X it |F it − 1 ] | ≤ O (cid:16)(cid:113) (cid:80) Bi

Original: =1 ϕ ξ ( i ) B (cid:17) with high probability, which is tighter than the constant bound 2 GR for a large batch size.

Original: However, with this high probability bound of the summand, we can no longer apply the standard Azuma’s inequality to bound the summation (it requires almost sure boundedness).

Original: Therefore, we develop a generalized Azuma’s inequality (Lemma B.3) that relaxes the almost sure boundedness requirement.

Original: The resulting bound of this first term involves an additional probabilistic term (cid:80) τi =1 P ( (cid:80) t ∈I ( i )

Original: | X it − E

Original: [ X it |F it − 1 ] | ≥ α ) compared to (14) of Agarwal and Duchi [2012].

Original: • To bound the second term of Z n .

Original: Agarwal and Duchi [2012] bound its summand by a constant, i.e., E [ X it |F it − 1 ] ≤

Original: GRϕ ξ ( τ ) .

Original: As a comparison, in the mini-batch setting, we leverage the mini-batch structure of X it to bound the summand as E [ X it |F it − 1 ] ≤ GRB (cid:80) Bi

Original: =1 ϕ ξ ( τB + i ) , which is much smaller under a sufficiently large batch size B .

Original: This result is proved in our Lemma 5.1 as an improved version of Lemma 1 of Agarwal and Duchi [2012].

Original: • Lastly, we derive a high-probability regret bound R n for

Original: mini-batch SGD over dependent data, which is new to the existing literature.

Original: Here, the major challenge is to bound the mini-batch stochastic gradient variance over dependent data samples.

Original: In the conventional i.i.d.

Original: data case, the mini-batch stochastic gradient variance is typically bounded as

Original: E

Original: [Equation]

Original: However, the above bound has two limitations: 1) it only bounds the expectation (not with high-probability), therefore it cannot be combined with our previous high-probability bounds; 2) it does not consider the influence of data dependence.

Original: To address these issues, we propose the following decomposition:

Original: [Equation]

Original: ∇

Original: [Equation]

Original: ∥

Original: [Equation]

Original: The first term on the right hand side involves the conditional bias of the mini-batch stochastic gradient, and we bound it in the order of ˜ O

Original: ( (cid:80) Bi

Original: =1 ϕ ξ ( i ) B ) , by using Bernstein’s inequality for dependent data.

Original: The second term is the variance of E [ ∇ F ( w ( t ); x t )

Original: |F t − 1 ] , and we bound it using the ϕ -mixing property as O (( (cid:80) Bi =1 ϕ ξ ( i ) B ) 2 ) .

Original: In summary, we obtain the mini-batch stochastic gradient variance bound ∥ 1 B (cid:80)

Original: Bi

Original: =1 ∇ F ( w ; ξ i ) − ∇ f ( w ) ∥ 2 ≤ O ( (cid:80) Bi

Original: =1 ϕ ξ ( i ) B ) with high probability.

Original: B.2 KEY LEMMAS

Original: In this subsection, we present several useful preliminary results for proving Theorem 5.2.Throughout this subsection, we assume that Assumption 2.1 holds.

Original: The following lemma is a generalization of the Lemma 1 in [Agarwal and Duchi, 2012] by utilizing the batch structure.

Original: Lemma B.1.

Original: Let w, v be measurable with respect to F t .

Original: Then for any τ ∈ N ,

Original: [Equation]

Original: Proof.

Original: For any τ ∈ N , we consider the following decomposition.

Original: [Equation]

Original: We can further bound the term (A) using the mixing property of the dependent data stream.

Original: [Equation]

Original: where in the first inequality we use the facts that F ( · ; ξ ) is G -Lipschitz and the domain is bounded by R , and the second inequality is implied by the ϕ -mixing property.

Original: Substituting the above upper bound of (A) into the previous equation yields that

Original: [Equation]

Original: This completes the proof.

Original: The following proposition is directly adapted from Proposition 2 in Section 3, Agarwal and Duchi [2012].

Original: We include its proof here for completeness.

Original: Proposition B.2.

Original: Let { w ( t ) } t ∈ N be the model parameter sequence generated by (5).

Original: Also suppose that Assumption 3. holds.

Original: Then for any τ ∈ N , we have

Original: [Equation]

Original: Proof.

Original: For any τ ∈ N , we consider the following decomposition,

Original: [Equation]

Original: We will keep the first term and bound the term (B)

Original: [Equation]

Original: Recall that the term ( B 1 ) is the regret R n .

Original: We can bound the term ( B 2 ) by noting that

Original: [Equation]

Original: For the term ( B 3 ) , we can bound it using the G -Lipschitzness of F ( · ; ξ ) and the R -bounded domain.

Original: [Equation]

Original: ) (cid:105) ≤ G (cid:104) n (cid:88) t = n

Original: [Equation]

Original: +

Original: [Equation]

Original: τ

Original: [Equation]

Original: t

Original: [Equation]

Original: [Equation]

Original: t

Original: [Equation]

Original: ≤ 2 GR ( τ − 1) .

Original: Combining the above bounds of ( B 1 ) , ( B 2 ) , and ( B 3 ) , we obtain the upper bound of ( B ) as follows.

Original: [Equation]

Original: Then the proof is completed by substituting the upper bound of ( B ) into (9).

Original: The following generalized Azuma’s inequality generalizes the Proposition 34 of [Tao et al., 2015].

Original: The inequality can be used to bound sum of martingale difference random variables.

Original: Lemma B.3 (Generalized Azuma’s Inequality) .

Original: Let { X t } be a martingale difference sequence with respect to its canonical filtration F .

Original: Define Y =

Original: (cid:80)

Original: Ti =1 X

Original: i and assume E | Y | < ∞ .

Original: Then for any { α t } t >

Original: [Equation]

Original: Proof.

Original: Let T :=

Original: min { t : | X t | > α t } .

Original: Then the sets B t := { ω : T ( ω ) =

Original: t } are disjoint.

Original: Construct

Original: [Equation]

Original: By the above construction, the associated Doob martingale of Y ′ with respect to F is { Z t := (cid:80) t ∧ T

Original: i =

Original: 1 X i } .

Original: It satisfies the conditions of Azuma’s inequality, i.e.,

Original: • { Z t } forms a martingale with respect to F (because the stopped martingale is still a martingale).

Original: • | Z t − Z t − 1 | ≤ α t .

Original: Then we can apply Azuma’s inequality to Y ′ .

Original: [Equation]

Original: Now we can bound P (cid:18) | Y − E Y | ≥ λ

Original: (cid:113)(cid:80)

Original: Tt =1 α 2 t (cid:19) as follows.

Original: [Equation]

Original: Then the proof is completed.

Original: Here we notice the fact that E Y ′ =

Original: E Y by our construction.

Original: The following lemma is taken from (22), Theorem 4 of [Delyon et al., 2009].

Original: Lemma B.4 (Bernstein’s Inequality for Dependent Process) .

Original: Let { Z t } be a centered adaptive process with respect to F .

Original: Define the following quantities.

Original: [Equation]

Original: Then, it holds that

Original: [Equation]

Original: Application of Lemma B.4 to our proof.

Original: Here we make some comments about how to apply this inequality in our main proof.

Original: We define the following random variable in our proof.

Original: Throughout, we use the batch-level filtration F and the intra-batch level filtration (cid:98) F .

Original: The formal definition is given in Section B.3.

Original: [Equation]

Original: We also define the filtration F it :

Original: = F tτ + i − 1 for simplicity.

Original: Then, we have

Original: [Equation]

Original: Then

Original: [Equation]

Original: the bias can be rewritten as

Original: [Equation]

Original: where Y it is defined as

Original: [Equation]

Original: [Table]

Original: More specifically, we have

Original: [Equation]

Original: Recall that (cid:98) F is the canonical filtration generated from the data stream (8).

Original: Moreover, { Y it ( ξ ( j ) tτ

Original: + i − 1 ) } j =1 , 2 ,...,B is centered and adaptive with respect to this filtration.

Original: Then we can evaluate the quantities q , v , and m in Lemma B.4 as follows.

Original: • Bounding m is simple.

Original: By Assumption 2.1 we have ∥ Y it ( ξ ( j ) tτ

Original: + i

Original: −

Original: ∥

Original: ≤

Original: GR

Original: • The above bound of m leads to a simple bound for v ,

Original: i

Original: e., v ≤ 2 nG 2 R 2 .

Original: • The quantity q can be bounded as follows.

Original: [Equation]

Original: Then, by applying Lemma B.

Original: [Equation]

Original: we obtain the following high-probability bound.

Original: [Equation]

Original: Simplifying yields that

Original: [Equation]

Original: where C := 4 G 2 R 2 .

Original: B.3 PROOF OF THE MAIN RESULT

Original: Theorem B.5.

Original: Let { w ( t ) } t ∈ N be the model parameter sequence generated by (5).

Original: Suppose Assumptions 2.1 and 3.1 hold.

Original: Then, for any τ ∈ N , with probability at least 1 − δ , we have n (cid:88) t =

Original: [ f ( w ( t )) − f ( w ∗ )] ≤ GR n B

Original: B (cid:88) i = ϕ

Original: [Equation]

Original: ( τB + i )

Original: [Equation]

Original: B ·

Original: [Equation]

Original: GR

Original: [Equation]

Original: log

Original: [Equation]

Original: (cid:118)(cid:117)(cid:117)(cid:116)

Original: G

Original: [Equation]

Original: R

Original: B (log

Original: [Equation]

Original: n

Original: [Equation]

Original: ) 2 + (cid:0) 4 G 2 R 2 + 16 G 2 R

Original: B (cid:88) i =

Original: ϕ ξ ( i ) (cid:1) log 4 n δ (cid:17) · log 4 τ δ log 4 n δ + R n + G ( τ − 1) n − τ +1 (cid:88) t

Original: = κ ( t ) + 2 GR ( τ − 1) .

Original: In particular, if τ = 1 , then n (cid:88) t =

Original: [ f ( w ( t )) − f ( w ∗ )] ≤ R n + GR n B

Original: B (cid:88) i = ϕ ξ ( B + i ) + (cid:118)(cid:117)(cid:117)(cid:117)(cid:116) 2 n

Original: B · (cid:16)

Original: [Equation]

Original: GR

Original: [Equation]

Original: log

Original: [Equation]

Original: (cid:118)(cid:117)(cid:117)(cid:116)

Original: G

Original: [Equation]

Original: R

Original: B (log

Original: [Equation]

Original: n δ )

Original: [Equation]

Original: + (cid:0) 4 G 2 R 2 + 16 G 2 R

Original: B (cid:88) i =

Original: ϕ ξ ( i ) (cid:1) log 4 n δ (cid:17) · log 4 τ δ log 4 n δ .

Original: Proof.

Original: From Proposition B.2, we obtain the following bound.

Original: n (cid:88) t =

Original: [ f

Original: [Equation]

Original: w

Original: [Equation]

Original: t

Original: [Equation]

Original: − f

Original: [Equation]

Original: w

Original: [Equation]

Original: [Equation]

Original: ≤ n (cid:88) t =

Original: [Equation]

Original: To complete the proof, it suffices to bound the first term; we define this term as

Original: [Equation]

Original: We apply the same decomposition as the (13) of [Agarwal and Duchi, 2012].

Original: Define the index set I ( i ) as { 1 , . . .

Original: , ⌊ nτ ⌋ + 1 } for i ≤ n − τ

Original: ⌊ nτ ⌋ and { 1 , . . .

Original: , ⌊ nτ ⌋} otherwise.

Original: Then we have

Original: Z n

Original: [Equation]

Original: (cid:88)

Original: [Equation]

Original: (cid:88)

Original: [Equation]

Original: ( i

Original: [Equation]

Original: it − 1 ]]

Original: [Equation]

Original: τ (cid:88) i = (cid:88) t ∈I ( i

Original: [Equation]

Original: E [ X it |F it − 1 ] , where

Original: X

Original: [Equation]

Original: Note that by Lemma 5.1, we have that E

Original: [ X it |F it − 1 ] ≤ GRB

Original: [Equation]

Original: Bi =1 ϕ ξ ( τB + i ) .

Original: Then, we have

Original: [Equation]

Original: Define Y := (cid:80) t ∈I ( i )

Original: [ X it − E

Original: [ X it |F it − 1 ]] and α :=

Original: λ √ B .

Original: Notice that X it − E

Original: [ X it |F it − 1 ] is a centered random variable, that is, E

Original: [ X it − E

Original: [ X it |F it − 1 ]]

Original: = 0 .

Original: Unlike the corresponding step in [Agarwal and Duchi, 2012], | X it − E

Original: [ X it | is not bounded by a constant with probability 1 .

Original: We develop Azuma’s inequality (Lemma B.3) to deal with the situation where | X it − E

Original: [ X it | exceed the desired bound.

Original: [Equation]

Original: Also, bounding the additional concentration probability term (cid:80) nτ

Original: [Equation]

Original: =1 P ( | X it − E

Original: [ X it |F it − 1 ] | ≥ α ) with leveraging the minibatch structure requires the generalized Bernstein’s inequality.

Original: The detailed calculation can be found in the discussion after

Original: Lemma B.4.

Original: We obtain that

Original: [Equation]

Original: where C = 4 G 2 R 2 .

Original: In summary, the concentration bound for Z n is

Original: [Equation]

Original: Then, let δ 2 = 2 n exp (cid:0) −

Original: [Equation]

Original: , and we obtain that

Original: [Equation] It is a quadratic function of λ . Solving it yields that [Equation]
Revised: The work of Y. Liang was supported in part by U.S. National

Original: [-Also, let δ 
-]
Revised: [-Also, let δ 
-]

Original: = 2 τ exp

Original: [Equation]

Original: , we have that

Original: [Equation]

Original: Substituting (10) into the above equation, we obtain that

Original: [Equation]

Original: Then, we conclude that with probability at least 1 − δ , n (cid:88) t =

Original: [ f ( w ( t )) − f ( w ∗ )]

Original: [Equation]

Original: + R n + G ( τ − 1) n − τ +1 (cid:88) t

Original: = κ ( t ) + 2 GR ( τ − 1) .

Original: [Equation]

Original: The desired result follows by noting that (cid:80) nt =1 f ( w ( t )) ≥ nf ( (cid:98) w n ) .

Original: C REGRET ANALYSIS OF MINI-BATCH SGD

Original: In this section, we derive the regret bound of mini-batch SGD algorithm.

Original: Throughout, for each sample loss F ( w ; ξ ) , recall that its gradient ∥∇ F ( w ; ξ ) ∥ is uniformly bounded by G (see Assumption 2.1).

Original: In particular, we assume the k -th coordinate of ∇ F ( w ; ξ ) is uniformly bounded by G k , and we have G =

Original: (cid:112)(cid:80) k

Original: G 2 k .

Original: 1. Gradient Variance Bound under Dependent Data

Original: In the i.i.d.

Original: setting, the variance of stochastic gradient decreases as the batch size increases.

Original: Specifically, we have

Original: [Equation]

Original: Therefore, E ∥ 1 B (cid:80) Bi

Original: =1 ∇ F ( w ; ξ i ) − ∇ f ( w ) ∥ 2 = O ( 1 B ) .

Original: However, this bound no longer holds if the data samples are dependent.

Original: In the following lemma, we develop a similar result when the data is collected from a dependent stochastic process.

Original: Recall that ∇ F ( w ( t ); x t ) denotes the averaged gradient over the mini-batch x t , i.e.,

Original: [Equation]

Original: Lemma C.1.

Original: Let { w ( t ) } t ∈ N be the model parameter sequence generated by the mini-batch SGD in (5).

Original: Let Assumptions 2.1 and 3.1 hold.

Original: Then, with probability at least 1 − δ ,

Original: [Equation]

Original: Proof.

Original: Let x t = { ξ ( i ) t } Bi =1 be the t -th mini-batch samples.

Original: We consider the filtration within x t and denote it as { (cid:98) F ( i ) t } .

Original: Then, by the definition of canonical filtration,

Original: [Equation]

Original: is measurable with respect to (cid:98) F ( i ) t .

Original: Define

Original: [Equation]

Original: where ( · ) k denotes the k -th entry of the specified vector.

Original: And it is easy to see that { Y i,k } i is a centered process for any k ∈ { 1 , 2 , . . .

Original: , d } .

Original: With these construction, we start from the following decomposition.

Original: [Equation]

Original: Then we will bound the term (A) and (B), respectively.

Original: • Bounding (A):

Original: Note that

Original: [Equation]

Original: Then, we show that the process { Y i,k } i satisfies the conditions of Lemma B.4.

Original: – Since E [ Y i,k |F t −

Original: ] = 0 , we conclude that { Y i,k } i is a centered process.

Original: – Denote the k -th entry of X i as X i,k .

Original: We know that |

Original: X i,k | ≤ G k .

Original: Hence, we conclude that 0 ≤ | Y i,k | ≤ 2 G k .

Original: Then, we can set b i = 2 G k for all i .

Original: – Lastly, we can bound the quantity q defined in Lemma B.4 as follows.

Original: [Equation]

Original: Now, we can apply Lemma B.4 and obtain that

Original: [Equation]

Original: With a union bound

Original: [Equation]

Original: we obtain that

Original: [Equation]

Original: Further applying the union bound over k = 1 , 2 , . .

Original: , d , we obtain that

Original: [Equation]

Original: > λ 2 k (cid:111)(cid:33) ≤ 2 (cid:88) k exp (cid:32) − λ 2 k

Original: 1343 G 2 k B + 128 G 2 k B (cid:80)

Original: Bj =1 ϕ ξ ( j ) (cid:33) .

Original: Let δd = 2 exp (cid:18) − λ 2 k 1343 G 2 k B +128 G 2 k B (cid:80) Bj =1 ϕ ξ ( j ) (cid:19) , we obtain that λ 2 k = (cid:104) 134 3 G 2 k B + 128 G 2 k B

Original: B (cid:88)

Original: j = ϕ ξ ( j ) (cid:105) · log 2 d δ .

Original: Then we conclude that,

Original: [Equation]

Original: It implies that with the probability at least 1 − δ ,

Original: [Equation]

Original: By definition, G = (cid:112)(cid:80) k G 2 k .

Original: Finally, we have the following bound for term (A): with probability at least 1 − δ ,

Original: [Equation]

Original: • Bounding (B): Note that ∥

Original: [Equation]

Original: Then we bound the norm by triangle inequality,

Original: [Equation]

Original: Finally

Original: [Equation]

Original: we obtain the bound for the term (B) as

Original: [Equation]

Original: Combing the bounds of (A) and (B) yields that with probability at least 1 − δ ,

Original: [Equation]

Original: 2. High-Probability Regret Bound

Original: To derive the regret bound for the mini-batch SGD algorithm

Original: [Equation]

Original: we make the following additional mild assumption.

Original: Assumption C.2.

Original: The stochastic optimization problem (P) satisfies

Original: • Each sample loss F ( · ; ξ ) :

Original: W → R is convex.

Original: • The objective function f : W → R is L -smooth.

Original: Theorem C.3 (High-probability regret bound) .

Original: Let { w ( t ) } t ∈ N be the model parameter sequence generated by the mini-batch

Original: SGD in (5). Suppose Assumptions C.2, 3.1 and2.1 hold. Then, with probability at least 1 − δ , [Equation]
Revised: Science Foundation under the grants CCF-1909291 and

Original: Moreover, let η =

Original: O

Original: ( (cid:113)

Original: [Equation]

Original: , the optimized upper bound is in the order of

Original: [Equation]

Original: Proof.

Original: For convenience, we define g t = 1 B (cid:80) Bi

Original: =1 ∇ F ( w ( t ); ξ ( i ) t ) .

Original: By the algorithm update (5), we obtain that

Original: [Equation]

Original: Summing the above inequality over t yields that

Original: [Equation]

Original: By convexity of the function, we further obtain that

Original: [Equation] Then, we apply Lemma C.1 to bound the second term (cid:80) Tt = 1 ∥ g t − ∇ f ( w ( t ))
Revised: ECCS-2113860.

Original: ∥ 2 and then apply a union bound on over t .

Original: We conclude that, with probability at least 1 − δ ,

Original: T (cid:88)

Original: [Equation]

Original: The proof is completed.

Original: Lastly, we set the learning rate η .

Original: To minimize the obtained upper bound, it suffices to minimize the first two terms, as the last term can be combined with the left hand side of (11) when we apply this regret bound.

Original: The optimized learning rate is achieved when

Original: [Equation]

Original: Then, η is chosen as

Original: [Equation]

Original: D EXPERIMENT SETUP

Original: Recall that we consider the following convex quadratic optimization problem:

Original: [Equation]

Original: where A is a fixed positive semi-definite matrix and µ is the uniform distribution on [0 , 1] d .

Original: The data stream admitting such a stationary distribution µ can be generated by a certain Metropolis-Hastings sampler provided in [Jarner and Roberts, 2002].

Original: Specifically, it is described as follows

Original: • Let the “proposal" distribution q ( x ) have the density of Beta ( r + 1 , 1) ; that is,

Original: [Equation]

Original: Define the acceptance probability α ( x, y )

Original: =

Original: min { q ( x ) q

Original: y ) , 1 } .

Original: • If the current state is ξ t , then we sample ζ ∼ q .

Original: Define the next state ξ t +1 :

Original: [Equation]

Original: • Go back to Step 2 to generate the next state.

Original: [-We repeatedly generate d independent sequences starting from the same initial state s 
-]
Revised: [-We repeatedly generate d independent sequences starting from the same initial state s 
-]

Original: = 0 to obtain a d -dimension Markov chain.

Original: It has been shown that the above generated Markov chain converges to µ in distribution with an algebraic convergence rate ϕ ξ ( k ) ≤ O ( k − 1 /r ) in Proposition 5.2, [Jarner and Roberts, 2002].

Original: [-We consider the following bias term at the fixed point w = 
-]
Revised: [-We consider the following bias term at the fixed point w = 
-]

Original: d .

Original: [Equation]

Original: It can be used to approximate the left-hand side of Lemma 5.1.

Original: [-Since E (cid:2) F ( w ; x τ ) | s 0 = 
-]
Revised: [-Since E (cid:2) F ( w ; x τ ) | s 0 = 
-]

Original: d (cid:3) cannot be explicitly obtained, we use Monte Carlo method to estimate this conditional expectation.

Original: That is, we generate n = 10 , 000 independent trajectories starting from x 0 = 0 d .

Original: At the step τ , we estimate the expected value as 1 n (cid:80) ni

Original: =1 F ( w ; x ( i ) τ ) , where x ( i ) τ with the superscript ( i ) indicates that it is sampled from the i -th trajectory.

Original: Then we investigate the relation between the step τ and the mixing parameter r and the relation between the step τ and the batch size B .

Original: All the results are presented in Section 6.

Original: For the neural network experiments, we setup the hyper-parameters as described as follows: When comparing the correlation coefficients, we fix the batch size B = 1000 , and apply the scaling described in Section 6.1 η = (cid:113)

Original: [Equation]

Original: Bj =1 ϕ ξ ( j ) .

Original: [-When comparing the influence of batch size on a fixed correlated data stream, we fix the correlation coefficients r = 2 . 0 and set up the standard linear scaling η = 0 . 
-]
Revised: [-When comparing the influence of batch size on a fixed correlated data stream, we fix the correlation coefficients r = 2 . 0 and set up the standard linear scaling η = 0 . 
-]

Original: × B log B for the fair comparison among different batch sizes.

Original: E ADDITIONAL EXPERIMENTS

Original: We include a support vector machine (SVM) experiment here as a complement of empirical studies.

Original: The online data generator directly follows the construction used in Section 6.2; that is, we generate an algebraic ϕ -mixing Markov chain { X t } t then mapping each X t to a label of the MNIST dataset.

Original: For each image with size 28 × 28 , we flatten it as a 784 -dimensional vector.

Original: We train the SVM model by solving

Original: [Figure]

Original: where ( x, y ) is the image data and its corresponding label, D is the stationary distribution of data stream, and L Hinge is the multi-class Hinge loss.

Original: We verify the performance of SGD over different dependence coefficient r with a fixed batch size.

Original: More specifically, we set the batch size B = 128 , and adjust the correlation coefficients r from { 1 . 0 , 1 . 25 , 1 . 5 , 1 . 75 , 2 . 0 } .

Original: The learning rate is set as the scaling rule described in Section 6.1 η = 0 .

Original: 01 × (cid:113)

Original: [Figure]

Original: j ) .

Original: Figure 4 (left) gives the experiment result and it indicates that the convergence of SGD becomes slower when the data dependence is increasing.

Original: Furthermore, we fix the correlation coefficient r = 2 . 0 and compare the performance of SGD with different batch sizes.

Original: [-Here the learning rate follows the standard linear scaling η = 0 . 
-]
Revised: [-Here the learning rate follows the standard linear scaling η = 0 . 
-]

Original: × B log B for the fair comparison among different batch size.

Original: The experiment result in Figure 4 (right) shows increasing the batch size can reduce the convergence error.