Original: A Single-Timescale Analysis for Stochastic Approximation with Multiple Coupled Sequences
Revised: A Single-Timescale Analysis for Stochastic Approximation with Multiple Coupled Sequences

Original: Abstract
Revised: Abstract

Original: Stochastic approximation (SA) with multiple coupled sequences has found broadapplications in machine learning such as bilevel learning and reinforcement learning(RL).
Revised: Stochastic approximation (SA) with multiple coupled sequences has found broad applications in machine learning such as bilevel learning and reinforcement learning (RL).

Original: In this paper, we study the finite-time convergence of nonlinear SA withmultiple coupled sequences.
Revised: In this paper, we study the finite-time convergence of nonlinear SA with multiple coupled sequences.

Original: Different from existing multi-timescale analysis, weseek for scenarios where a fine-grained analysis can provide the tight performanceguarantee for single-timescale multi-sequence SA (STSA).
Revised: Different from existing multi-timescale analysis, we seek for scenarios where a fine-grained analysis can provide the tight performance guarantee for single-timescale multi-sequence SA (STSA).

Original: At the heart of ouranalysis is the smoothness property of the fixed points in multi-sequence SAthat holds in many applications.
Revised: At the heart of our analysis is the smoothness property of the fixed points in multi-sequence SA that holds in many applications.

Original: When all sequences have strongly monotoneincrements, we establish the iteration complexity of O ( ϵ − 1 ) to achieve ϵ -accuracy,which improves the existing O ( ϵ − 1 . 5 ) complexity for two coupled sequences.
Revised: When all sequences have strongly monotone increments, we establish the iteration complexity of O ( ϵ − 1 ) to achieve ϵ -accuracy, which improves the existing O ( ϵ − 1 . 5 ) complexity for two coupled sequences.

Original: When the main sequence does not have strongly monotone increment, we establishthe iteration complexity of O ( ϵ − 2 ) .
Revised: When the main sequence does not have strongly monotone increment, we establish the iteration complexity of O ( ϵ − 2 ) .

Original: We showcase the power of our result byapplying it to stochastic bilevel and compositional optimization problems, as wellas RL problems, all of which lead to improvements over their existing guarantees.
Revised: We showcase the power of our result by applying it to stochastic bilevel and compositional optimization problems, as well as RL problems, all of which lead to improvements over their existing guarantees.

Original:  Introduction
Revised:  Introduction

Original: Stochastic approximation (SA) is an iterative procedure used to find the zero of a function when onlythe noisy estimate of the function is observed.
Revised: Stochastic approximation (SA) is an iterative procedure used to find the zero of a function when only the noisy estimate of the function is observed.

Original: Specifically, with the mapping v : R d (cid:55)→ R d thesingle-sequence SA seeks to solve for v ( x ) = 0 with the following iterative update:
Revised: Specifically, with the mapping v : R d (cid:55)→ R d , the single-sequence SA seeks to solve for v ( x ) = 0 with the following iterative update:

Original: [Equation]
Revised: [Equation]

Original: where α k is the step size and ξ k is a random variable.
Revised: where α k is the step size and ξ k is a random variable.

Original: Since its introduction in [43] single-sequence
Revised: Since its introduction in [46], single-sequence

Original: SA has received great interests because of its broad range of applications to areas including stochasticoptimization and reinforcement learning (RL) [6, 49].
Revised: SA has received great interests because of its broad range of applications to areas including stochastic optimization and reinforcement learning (RL) [6, 53].

Original: The asymptotic convergence of single-sequence
Revised: The asymptotic convergence of single-sequence

Original: SA can be established by the ordinary differential equation method; see e.g., [4].
Revised: SA can be established by the ordinary differential equation method; see e.g., [4].

Original: To gain moreinsights into the performance difference of various stochastic optimization algorithms, the finite-timeconvergence of SA has been widely studied in recent years; see e.g., [41, 40, 29, 47, 39, 13].
Revised: To gain more insights into the performance difference of various stochastic optimization algorithms, the finite-time convergence of SA has been widely studied in recent years; see e.g., [43, 42, 30, 50, 54, 52, 41, 13].

Original: While most of the SA studies focus on the single-sequence case, the double-sequence SA wasintroduced in [3], which has been extensively applied to the RL methods involving a double-sequencestochastic update structure [49, 31, 10].
Revised: While most of the SA studies focus on the single-sequence case, the double-sequence SA was introduced in [3], which has been extensively applied to the RL methods involving a double-sequence stochastic update structure [53, 32, 10].

Original: With mappings v : R d 0 × R d 1 (cid:55)→ R d 0 and h : R d 0 × R d 1 (cid:55)→ R d 1 , the double-sequence SA seeks to solve v ( x, y )
Revised: With mappings v : R d 0 × R d 1 (cid:55)→ R d 0 and h : R d 0 × R d 1 (cid:55)→ R d 1 , the double-sequence SA seeks to solve v ( x, y )

Original: =
Revised: =

Original: h ( x, y )
Revised: h ( x, y )

Original: = 

Revised: =

Original: with the following update:
Revised: with the following update:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: where α k , β k are the step sizes, and ξ k , ψ k are random variables.
Revised: where α k , β k are the step sizes, and ξ k , ψ k are random variables.

Original: In (2), the update of x k andthat of y k depend on each other and thus the sequences are coupled .
Revised: In (2), the update of x k and that of y k depend on each other and thus the sequences are coupled .

Original: Due to this coupling, thedouble-sequence SA ismore challenging to analyze than its single-sequence counterpart.
Revised: To deal with the coupling, a naive thought is to stack ( x k , y k ) as one variable. However, it can be seen later that the convergence of [Table]

Original: Prior art on double-sequence SA.
Revised: Prior art on double-sequence SA.

Original: Many recent analyses of the double-sequence SA focus on thelinear case where v ( x, y ) and h ( x, y ) are linear mappings; see e.g., [33, 11, 24, 28].
Revised: Many recent analyses of the double-sequence SA focus on the linear case where v ( x, y ) and h ( x, y ) are linear mappings; see e.g., [34, 11, 25, 29].

Original: The key ideahere is to use the so-called two-time-scale (TTS) step sizes: One sequence is updated in the fastertime scale while the other is updated in the slower time scale; that is lim k →∞ α k /β k = 0 .
Revised: The key idea here is to use the so-called two-time-scale (TTS) step sizes: One sequence is updated in the faster time scale while the other is updated in the slower time scale; that is lim k →∞ α k /β k = 0 .

Original: By doingso, the two sequences are shown to decouple asymptotically, which allows us to leverage the analysisof the single-sequence SA.
Revised: By doing so, the two sequences are shown to decouple asymptotically, which allows us to leverage the analysis of the single-sequence SA.

Original: In particular, [28] proves an iteration complexity of O ( ϵ − 1 ) to achieveϵ -accuracy for the TTS linear SA, which is shown to be tight.
Revised: In particular, [29] proves an iteration complexity of O ( ϵ − 1 ) to achieve ϵ -accuracy for the TTS linear SA, which is shown to be tight.

Original: With similar choice of the step sizes, the
Revised: With similar choice of the step sizes, the

Original: TTS nonlinear SA was analyzed in [37, 12].
Revised: TTS nonlinear SA was analyzed in [39, 12].

Original: In [37], the finite-time convergence rate of TTS nonlinear
Revised: In [39], the finite-time convergence rate of TTS nonlinear

Original: SA was established under an assumption that the two sequences converge asymptotically.
Revised: SA was established under an assumption that the two sequences converge asymptotically.

Original: Later,[12] alleviates this assumption and shows that TTS nonlinear SA achieves an iteration complexity of
Revised: Later, [12] alleviates this assumption and shows that TTS nonlinear SA achieves an iteration complexity of

Original: O ( ϵ − 1 . 5 ) .
Revised: O ( ϵ − 1 . 5 ) .

Original: However, this iteration complexity is larger than O ( ϵ − 1 ) of the TTS linear SA.
Revised: However, this iteration complexity is larger than O ( ϵ − 1 ) of the TTS linear SA.

Original: The gap between the complexities of nonlinear and linear SA motivates an interesting question:
Revised: The gap between the complexities of nonlinear and linear SA motivates an interesting question:

Original: Q1:
Revised: Q1:

Original: Is it possible to prove a faster rate for the nonlinear SA with two coupled sequences?
Revised: Is it possible to prove a faster rate for the nonlinear SA with two coupled sequences?

Original: [Equation]
Revised: [Figure]

Original: k We first conduct an experiment to examine the possibility.
Revised: possibility

Original: Experiment.
Revised: Experiment.

Original: Figure 1 shows the performance of using thedouble-sequence SA (2) to solve the following problem
Revised: Figure 1 shows the performance of using the double-sequence SA (2) to solve the following problem

Original: [Equation]
Revised: [Equation]

Original: We use the double-sequence SA (2) to solve (3), wherev ( x,
Revised: We use the double-sequence SA (2) to solve (3), where

Original: [Equation]
Revised: [Equation]

Original: and ζ k , ξ k are independent Gaussian random variableswith zero mean and standard deviations of 0 .
Revised: and ζ k , ξ k are independent Gaussian random variables with zero mean and standard deviations of 0 .

Original: It is easyto check that (4) satisfies the assumptions in the existing
Revised: It is easy to check that (4) satisfies the assumptions in the existing

Original: TTS-SA analysis [12].
Revised: TTS-SA analysis [12].

Original: Therefore, we can use the twotime-scale step sizes and achieve the iteration complexityof O ( ϵ − 1 . 5 ) .
Revised: Therefore, we can use the two time-scale step sizes and achieve the iteration complexity of O ( ϵ − 1 . 5 ) .

Original: However, as suggested by Figure 1, the iterates still converge with step sizes in a singletime-scale ( α k = Θ( 1 k ) , β k = Θ( 1 k ) ).
Revised: However, as suggested by Figure 1, the iterates still converge with step sizes in a single time-scale ( α k = Θ( 1 k ) , β k = Θ( 1 k ) ).

Original: In this case, the iteration complexity is O
Revised: In this case, the iteration complexity is O

Original: ( ϵ − 1 ) , which isthe same as that of double-sequence linear SA [28].
Revised: ( ϵ − 1 ) , which is the same as that of double-sequence linear SA [29].

Original: This suggests that existing analysis of doublesequence SA might not be tight, at least for the class of updates similar to (4).
Revised: This suggests that existing analysis of double-sequence SA might not be tight, at least for the class of updates similar to (4).

Original: Indeed, as we willshow later, the iterates generated by (4) will converge with the iteration complexity of O ( ϵ − 1 ) .
Revised: Indeed, as we will show later, the iterates generated by (4) will converge with the iteration complexity of O ( ϵ − 1 ) .

Original: Furthermore, existing works on TTS SA mainly focus on the double-sequence case.
Revised: Furthermore, existing works on TTS SA mainly focus on the double-sequence case.

Original: While incases such as the multi-level stochastic optimization; see e.g., [56], more than two sequences are involved.
Revised: While in cases such as the multi-level stochastic optimization; see e.g., [61], more than two sequences are involved.

Original: This necessitates the use of the multi-sequence SA.
Revised: This necessitates the use of the multi-sequence SA.

Original: Specifically, with mappingsv :R d 0 × R d 1 · · · × R d N (cid:55)→ R [Equation] 0 , h n : R [Equation]
Revised: Specifically, with mappings v :

Original: n − 1 × R d n

Original: [Equation]

Original: R d n , we consider
Revised: R d n , we consider

Original: [Equation]
Revised: [Equation]

Original: where α k , β k, 1 , ..., β k,N are the step sizes, and ξ k , ψ 1 k , .
Revised: where α k , β k, 1 , ..., β k,N are the step sizes, and ξ k , ψ 1 k , .

Original: . . , ψ Nk are random variables.
Revised: . . , ψ Nk are random variables.

Original: For conciseness,we have used y 

Revised: For conciseness, we have used y

Original: k := x k here.
Revised: k := x k here.

Original: Our goal is to find the unique fixed-points x ∗ , y 1 , ∗ , . . .
Revised: Our goal is to find the unique fixed-points x ∗ , y 1 , ∗ , . . .

Original: , y N, ∗ such that
Revised: , y N, ∗ such that

Original: [Equation]
Revised: [Equation]

Original: Observing that in (5), for every n , the sequence of y nk is coupled with that of y n − 1 k and is ultimatelycoupled with the main sequence x k .
Revised: Observing that in (5), for every n , the sequence of y nk is coupled with that of y n − 1 k and is ultimately coupled with the main sequence x k .

Original: Meanwhile the update of x k also depends on { y nk } Nn =1 .
Revised: Meanwhile the update of x k also depends on { y nk } Nn =1 .

Original: Sinceall sequences in (5) are coupled, (5) is more challenging to analyze than the double-sequence SA.
Revised: Since all sequences in (5) are coupled, (5) is more challenging to analyze than the double-sequence SA.

Original: Prior art related to multi-sequence SA.
Revised: Prior art related to multi-sequence SA.

Original: The multilevel stochastic optimization problem [46] andthe multilevel SCO problem [1,53, 58, 44, 60] are closely related to the multi-sequence SA.
Revised: The multilevel stochastic optimization problem [49] and the multilevel SCO problem [1, 58, 63, 47, 65] are closely related to the multi-sequence SA.

Original: Totackle the multi-level structure, these recent methods have modified the vanilla multi-sequence SAupdate to achieve the state-of-the-art complexity and thus their updates are no longer in the form of(5).
Revised: To tackle the multi-level structure, these recent methods have modified the vanilla multi-sequence SA update to achieve the state-of-the-art complexity and thus their updates are no longer in the form of (5).

Original: In contrast, we focus on the multi-sequence SA update in (5).
Revised: In contrast, we focus on the multi-sequence SA update in (5).

Original: To the best of our knowledge, theonly analysis for (5) is [56] where the TTS technique is generalized to multi-time-scale.
Revised: To the best of our knowledge, the only analysis for (5) is [61] where the TTS technique is generalized to multi-time-scale.

Original: In [56], theiteration complexity will get worse as the number of sequences N increases.
Revised: In [61], the iteration complexity will get worse as the number of sequences N increases.

Original: This gives rise to another interesting question:
Revised: This gives rise to another interesting question:

Original: Q2:
Revised: Q2:

Original: Is it possible to establish convergence rate independent of the number of sequences?
Revised: Is it possible to establish convergence rate independent of the number of sequences?

Original: In this work, we give affirmative answers to both Questions Q1 and Q2 .
Revised: In this work, we give affirmative answers to both Questions Q1 and Q2 .

Original: Our contributions.
Revised: Our contributions.

Original: Specially, by exploiting the smooth assumption that can be satisfied in manyapplications, we show that the vanilla nonlinear SA can run in a single time scale !
Revised: Specially, by exploiting the smooth assumption that can be satisfied in many applications, we show that the vanilla nonlinear SA can run in a single time scale !

Original: We further provethat the order of the convergence rate is independent of the number of sequences N !
Revised: We further prove that the order of the convergence rate is independent of the number of sequences N !

Original: Intuitively, thisis possible because when the fixed point y n, ∗ is smooth in x , the y nk -update converges fast enoughsuch that its fixed-point residual after one-step update is at the same order as the drift of y n, ∗ .
Revised: Intuitively, this is possible because when the fixed point y n, ∗ is smooth in x , the y nk -update converges fast enough such that its fixed-point residual after one-step update is at the same order as the drift of y n, ∗ .

Original: In the context of prior art, our contributions can be summarized as follows (see Table 1)
Revised: In the context of prior art, our contributions can be summarized as follows (see Table 1).

Original: C1)
Revised: C1)

Original: Single-timescale analysis for multi-sequence SA.
Revised: Single-timescale analysis for multi-sequence SA.

Original: Different from existing two-timescale analysis[37, 5], we establish a unifying Single-Timescale analysis for SA with multiple coupled sequences thatwe term STSA .
Revised: Different from existing two-timescale analysis , 5], we establish a unifying Single-Timescale analysis for SA with multiple coupled sequences that we term STSA .

Original: When all the sequences have strongly-monotone increments, we improve the O ( ϵ − 1 . 5 )iteration complexity for multi-sequence TTS-SA in [12] to O
Revised: When all the sequences have strongly-monotone increments, we improve the O ( ϵ − 1 . 5 ) iteration complexity for multi-sequence TTS-SA in [12] to O

Original: ( ϵ − 1 ) .
Revised: ( ϵ − 1 ) .

Original: When the main sequence doesnot have the strongly-monotone increment, we provide the O ( ϵ − 2 ) iteration complexity.
Revised: When the main sequence does not have the strongly-monotone increment, we provide the O ( ϵ − 2 ) iteration complexity.

Revised: It is worth noting that though the single time-scale step sizes were also explored in [37, 44], the key enabler in those works is the decrease of variance which is a result of variance-reduction update or increasing batch size.

Revised: While this work and those in Table 1 focus on the case where the variance is non-decreasing.

Original: C2)
Revised: C2)

Original: STSA for stochastic bilevel optimization (SBO).
Revised: STSA for stochastic bilevel optimization (SBO).

Original: When applying our generic results to theSBO problem with double-sequence SA, for strongly-concave objective functions, we improve thebest-known sample complexity ˜ 
Revised: When applying our generic results to the SBO problem with double-sequence SA, for strongly-concave objective functions we improve the best-known sample complexity ˜ O

Original: O ( ϵ − 1 . 5 ) of TTSA in [25] to ˜ O
Revised: ( ϵ − 1 . 5 ) of TTSA in [26] to ˜ O

Original: ( ϵ − 1 ) .
Revised: ( ϵ − 1 ) .

Original: For the non-concave objectivefunction, we achieve the same sample complexity
Revised: For the non-concave objective function, we achieve the same sample complexity

Original: O ( ϵ − 2 ) of ALSET while relaxing the boundedupper-level gradient assumption made in [8].
Revised: O ( ϵ − 2 ) of ALSET while relaxing the bounded upper-level gradient assumption made in [8].

Original: C3)
Revised: C3)

Original: STSA for stochastic compositional optimization (SCO).
Revised: STSA for stochastic compositional optimization (SCO).

Original: When applying our results to the multilevel SCO problems, we improve the level-dependent sample complexities O ( ϵ − N +54 ) and O ( ϵ − N +84 )of multi-sequence SA based α -TSCGD method in [56] to the level-independent complexities ˜
Revised: When applying our results to the multilevel SCO problems, we improve the level-dependent sample complexities O ( ϵ − N +54 ) and O ( ϵ − N +84 ) of multi-sequence SA based α -TSCGD method in [61] to the level-independent complexities ˜

Original: O
Revised: O

Original: ( ϵ − 1 )and O ( ϵ − 2 ) , under the strongly-concave and non-concave objective functions, respectively.
Revised: ( ϵ − 1 ) and O ( ϵ − 2 ) , under the strongly-concave and non-concave objective functions, respectively.

Original: C4)
Revised: C4)

Original: STSA for policy optimization in RL problems.
Revised: STSA for policy optimization in RL problems.

Original: Moreover, applying our results to the actorcritic method achieves the same O ( ϵ − 2 ) sample complexity of ALSET-AC in [8] while relaxing theunverifiable assumption on the stationary distribution of Markov chains; applying our results to themeta policy gradient improves the O ( ϵ − 4 ) sample complexity of SG-MRL in [14] to O ( ϵ − 2 ) .
Revised: Moreover, applying our results to the actorcritic method achieves the same O ( ϵ − 2 ) sample complexity of ALSET-AC in [8] while relaxing the unverifiable assumption on the stationary distribution of Markov chains; applying our results to the meta policy gradient improves the O ( ϵ − 4 ) sample complexity of SG-MRL in [14] to O ( ϵ − 2 ) .

Original:  Main Results: Convergence of Single-timescale Multi-sequence SA
Revised:  Main Results: Convergence of Single-timescale Multi-sequence SA

Original: Before introducing the main results, we will first make some standard assumptions.
Revised: Before introducing the main results, we will first make some standard assumptions.

Original: Throughout thediscussion, we define [ N ] := { 1 , 2 , ..., N } , [ K ] := { 1 , 2 , ..., K } and y 0 :
Revised: Throughout the discussion, we define [ N ] := { 1 , 2 , ..., N } , [ K ] := { 1 , 2 , ..., K } and y 0 :

Original: = x for conciseness.
Revised: = x for conciseness.

Original: Assumption 1 (Smoothness of the fixed points)
Revised: Assumption 1 (Smoothness of the fixed points)

Original: For any n ∈ [ N ] and y n − 1 ∈ R d n − 1 , there existsa unique y n, ∗ ( y n − 1 ) ∈ R
Revised: For any n ∈ [ N ] and y n − 1 ∈ R d n − 1 , there exists a unique y n, ∗ ( y n − 1 ) ∈ R

Original: d n such that h n ( y n − 1 , y n, ∗ ( y n − 1 ))
Revised: d n such that h n ( y n − 1 , y n, ∗ ( y n − 1 ))

Original: = 0 .
Revised: = 0 .

Original: Moreover, there exist constants
Revised: Moreover, there exist constants

Original: L y,n and L y ′ ,n such that for any y n − 1 , ¯ y n − 1 ∈ R d n − 1 , the following inequalities hold
Revised: L y,n and L y ′ ,n such that for any y n − 1 , ¯ y n − 1 ∈ R d n − 1 , the following inequalities hold

Original: [Equation]
Revised: [Equation]

Original: Due to the change of y n − 1 k at each iteration, the solution of h n ( y n − 1 k , y n ) = 0 with respect to (w.r.t.)y n , that is y n, ∗ ( y n − 1 k ) , is drifting over consecutive iterations.
Revised: Due to the change of y n − 1 k at each iteration, the solution of h n ( y n − 1 k , y n ) = 0 with respect to (w.r.t.) y n , that is y n, ∗ ( y n − 1 k ) , is drifting over consecutive iterations.

Original: Given y n − 1 k , since only one-step of y nkupdate is performed at each iteration, one can only hope to establish convergence of y nk if the driftof its optimal solution is controlled in some sense.
Revised: Given y n − 1 k , since only one-step of y nk update is performed at each iteration, one can only hope to establish convergence of y nk if the drift of its optimal solution is controlled in some sense.

Original: Assumption 1 ensures both the zeroth-order andfirst-order drifts are controlled in the same scale of the change of y n − 1 k .
Revised: Assumption 1 ensures both the zeroth-order and first-order drifts are controlled in the same scale of the change of y n − 1 k .

Original: This assumption is satisfiedin linear SA [28] and other applications which will be shown later.
Revised: This assumption is satisfied in linear SA [29] and other applications which will be shown later.

Original: Define
Revised: Define

Original: [Equation]
Revised: [Equation]

Original: ( y 1 , ∗ ( x )) , . . .

Original:  , y N, ∗ ( . . .
Revised: . , y N, ∗ ( . . .

Original: y 2 , ∗ ( y 1 , ∗ ( x )) . . . )
Revised: y 2 , ∗ ( y 1 , ∗ ( x )) . . . )

Original: (cid:1) .
Revised: (cid:1) .

Original: With y 1: N as a concise notation for ( y 1 , ..., y N ) , we make the following assumption.
Revised: With y 1: N as a concise notation for ( y 1 , ..., y N ) , we make the following assumption.

Original: Assumption 2 (Lipschitz continuity of increments) For any n ∈ [ N ] , x, ¯ x ∈ R d 0 and y n , ¯ y n ∈
Revised: Assumption 2 (Lipschitz continuity of increments) For any n ∈ [ N ] , x, ¯ x ∈ R d 0 and y n , ¯ y n ∈

Original: R d n , there exist constants L v , L v,y and L h,n such that the following inequalities hold
Revised: R d n , there exist constants L v , L v,y and L h,n such that the following inequalities hold

Original: [Equation]
Revised: [Equation]

Original: Define F k as the σ -algebra generated by the random variables in { x i , y 1: N i } ki =1 and F nk as theσ -algebra generated by { x i , y 1: N i } ki =1 ∪ { y nk +1 } .
Revised: Define F k as the σ -algebra generated by the random variables in { x i , y 1: N i } ki =1 and F nk as the σ -algebra generated by { x i , y 1: N i } ki =1 ∪ { y nk +1 } .

Original: We make the following assumption on the noises.
Revised: We make the following assumption on the noises.

Original: Assumption 3 (Bias and variance)
Revised: Assumption 3 (Bias and variance)

Original: There exist constants { c n , σ n } Nn =0 such that ∀ k, n ,∥ E [ ξ k |F 1 k ] ∥ 2 ≤ c 20 α k and ∥ E [ ψ nk |F n +1 k ] ∥ 2 ≤ c 2 n β k,n ; E [ ∥ ξ k ∥ 2 |F 1 k ] ≤ σ 20 and E [ ∥ ψ nk ∥|F n +1 k ] ≤ σ 2 n .
Revised: There exist constants { c n , σ n } Nn =0 such that ∀ k, n , ∥ E [ ξ k |F 1 k ] ∥ 2 ≤ c 20 α k and ∥ E [ ψ nk |F n +1 k ] ∥ 2 ≤ c 2 n β k,n ; E [ ∥ ξ k ∥ 2 |F 1 k ] ≤ σ 20 and E [ ∥ ψ nk ∥|F n +1 k ] ≤ σ 2 n .

Original: Here we define F N +1 k :
Revised: Here we define F N +1 k :

Original: = F k .
Revised: = F k .

Original: Assumption 3 is a generalized version of the bias and varianceassumption in stochastic programming [18] or the noise assumption in single-sequence SA [29] tomulti-sequence case.
Revised: Assumption 3 is a generalized version of the bias and variance assumption in stochastic programming [19] or the noise assumption in single-sequence SA [30] to multi-sequence case.

Original: Similar assumption has also been made in the double-sequence SA [25].
Revised: Similar assumption has also been made in the double-sequence SA [26].

Original: Aswill be shown later, when applying STSA to the stochastic optimization problems, the conditionalindependence between samples of different levels given F k along with the small bias and boundedvariance condition will guarantee this assumption.
Revised: As will be shown later, when applying STSA to the stochastic optimization problems, the conditional independence between samples of different levels given F k along with the small bias and bounded variance condition will guarantee this assumption.

Original: Assumption 4 (Monotonicity of h ) For n ∈ [ N ] , h n ( y n − 1 , y n ) is one-point strongly monotone ony n, ∗ ( y n − 1 ) given any y n − 1 ; that is, there exists constant λ n > 0 such that (cf. h n ( y n 
Revised: Assumption 4 (Monotonicity of h ) For n ∈ [ N ] , h n ( y n − 1 , y n ) is one-point strongly monotone on y n, ∗ ( y n − 1 ) given any y n − 1 ; that is, there exists constant λ n > 0 such that (cf. h n ( y n − 1 , y n, ∗

Original: [Equation]
Revised: [Equation]

Original: 1 , y n, ∗ ) = 0 )
Revised: = 0 )

Original: [Equation]
Revised: [Equation]

Original: Assumption 4 is implied by the standard regularity assumptions in the previous works on TTS linear
Revised: Assumption 4 is implied by the standard regularity assumptions in the previous works on TTS linear

Original: SA [33, 28], and has also been exploited in the TTS nonlinear SA works; see e.g. [37, 12].
Revised: SA [34, 29], and has also been exploited in the TTS nonlinear SA works; see e.g. [39, 12].

Original: 2.1 The strongly-monotone case
Revised: 2.1 The strongly-monotone case

Original: We first consider the case when the main sequence x k has strongly-monotone increment.
Revised: We first consider the case when the main sequence x k has strongly-monotone increment.

Original: Assumption 5 (Monotonicity of v ) Suppose v ( x ) is one-point strongly monotone on x ∗ ; that is,there exists a positive constant λ 0 such that (cf.
Revised: Assumption 5 (Monotonicity of v ) Suppose v ( x ) is one-point strongly monotone on x ∗ ; that is, there exists a positive constant λ 0 such that (cf.

Original: v ( x ∗ ) = 0 )
Revised: v ( x ∗ ) = 0 )

Original: [Equation]
Revised: [Equation]

Original:  Same as Assumption 4, Assumption 5 is standard in the previous works on TTS SA [37, 12].
Revised: [Equation] Same as Assumption 4, Assumption 5 is standard in the previous works on TTS SA [39, 12].

Original: Thisassumption is a regularity assumption in the case of TTS linear SA; see e.g., [33, Assumption 2.3].
Revised: This assumption is a regularity assumption in the case of TTS linear SA; see e.g., [34, Assumption 2.3].

Original: Or in the case of bilevel optimization which will be discussed later, this assumption is satisfied whenthe objective function is strongly-concave.
Revised: Or in the case of bilevel optimization which will be discussed later, this assumption is satisfied when the objective function is strongly-concave.

Original: Due to space limitation, we directly present the result below and defer the proof to Appendix B.
Revised: Due to space limitation, we directly present the result below and defer the proof to Appendix B.

Original: Theorem 1 Consider the sequences generated by (5) .
Revised: Theorem 1 Consider the sequences generated by (5) .

Original: Suppose Assumptions 1–5 hold.
Revised: Suppose Assumptions 1–5 hold.

Original: Select stepsizes α k = Θ( 1 k ) and β k,n = Θ( 1 k ) .
Revised: Select step sizes α k = Θ( 1 k ) and β k,n = Θ( 1 k ) .

Original: It holds for any k that
Revised: It holds for any k that

Original: [Equation]
Revised: [Equation]

Original: where O ( · ) hides constants in the polynomial of N , and we have used y 0 k
Revised: where O ( · ) hides constants in the polynomial of N , and we have used y 0 k

Original: = x k for convenience.
Revised: = x k for convenience.

Original: Moreover, for any n ∈ [ N ] we have
Revised: Moreover, for any n ∈ [ N ] we have

Original: [Equation]
Revised: [Equation]

Original: almost surely (a.s.) ,
Revised: almost surely (a. [Equation]

Original: [Equation]
Revised: [Equation]

Original: It is worth noting that with (7a), Theorem 1 also implies the same convergence result for the errormetric ∥ x k − x [Equation] ∥[Equation]
Revised: It is worth noting that with (7a), Theorem 1 also implies the same convergence result for the error metric ∥ x k − x ∗ ∥

Original: + [Equation] Nn =1 ∥
Revised: + (cid:80) Nn =1 ∥

Original: y nk − y n, ∗ ∥ 2 , the formal justification of which is deferred to the proofof Theorem 1.
Revised: y nk − y n, ∗ ∥ 2 , the formal justification of which is deferred to the proof of Theorem 1.

Original: It is worth noting that the order of convergence in Theorem 1 is independent of N ,which is in contrast to the convergence rate that gets worse as N increases [12, 56].
Revised: It is worth noting that the order of convergence in Theorem 1 is independent of N , which is in contrast to the convergence rate that gets worse as N increases [12, 61].

Original: Remark 1 (Comparison with prior art in multi-sequence SA) Theorem 1 bridges the gap between the convergence rates of double-sequence linear and nonlinear SA by improving over the
Revised: Remark 1 (Comparison with prior art in multi-sequence SA) Theorem 1 bridges the gap between the convergence rates of double-sequence linear and nonlinear SA by improving over the

Original: O ( k − 23 ) rate shown in [12] with the additional assumption (7b).
Revised: O ( k − 23 ) rate shown in [12] with the additional assumption (7b).

Original: As will be shown later, this assumption is satisfied in various applications.
Revised: As will be shown later, this assumption is satisfied in various applications.

Original: Theorem 1 also generalizes the O ( 1 k ) convergence rate in thedouble-sequence linear SA analysis (e.g., [28]) to the multi-sequence nonlinear SA case.
Revised: Theorem 1 also generalizes the O ( 1 k ) convergence rate in the double-sequence linear SA analysis (e.g., [29]) to the multi-sequence nonlinear SA case.

Original: 2.2 The non-strongly-monotone case
Revised: 2.2 The non-strongly-monotone case

Original: Some applications of multi-sequence nonlinear SA such as the actor-critic method [31], Assumption5 does not hold.
Revised: Some applications of multi-sequence nonlinear SA such as the actor-critic method [32], Assumption 5 does not hold.

Original: This motivates us to consider a more general setting in this subsection where v ( x ) isnon-strongly-monotone.
Revised: This motivates us to consider a more general setting in this subsection where v ( x ) is non-strongly-monotone.

Original: Throughout this subsection, we make the following assumption.
Revised: Throughout this subsection, we make the following assumption.

Original: Assumption 6 Suppose there exists a mapping F : R d 0 (cid:55)→ R such that ∇ F ( x ) = v ( x ) .
Revised: Assumption 6 Suppose there exists a mapping F : R d 0 (cid:55)→ R such that ∇ F ( x ) = v ( x ) .

Original: Thesequence of { x k } is contained in an open set over which F ( x ) is upper bounded; e.g. F ( x ) ≤ C F .
Revised: The sequence of { x k } is contained in an open set over which F ( x ) is upper bounded; e.g. F ( x ) ≤ C F .

Original: As will be shown later, F ( x ) can be chosen as the objective function when applying SA to maximization problems.
Revised: As will be shown later, F ( x ) can be chosen as the objective function when applying SA to maximization problems.

Original: Assumption 6 is standard to ensure the convergence of x k ; see e.g. [6].
Revised: Then assumption 6 is standard to ensure the convergence of x k ; see e.g. [6].

Original: The following theorem gives the general finite-time convergence result of the nonlinear SA when themain sequence has the non-strongly-monotone increment.
Revised: The following theorem gives the general finite-time convergence result of the nonlinear SA when the main sequence has the non-strongly-monotone increment.

Original: The proof is deferred to Appendix C.
Revised: The proof is deferred to Appendix C.

Original: Theorem 2 Consider the sequences generated by (5) for k=[K].
Revised: Theorem 2 Consider the sequences generated by (5) for k=[K].

Original: Suppose Assumptions 1–4 & 6hold.
Revised: Suppose Assumptions 1–4 &hold.

Original: Select α k = Θ( 1 √ K ) , β k,n = Θ( 1 √ K ) with properly chosen initial step sizes, then it holds that
Revised: Select α k = Θ( 1 √ K ) , β k,n = Θ( 1 √ K ) with properly chosen initial step sizes, then it holds that

Original: [Equation]
Revised: [Equation]

Original: where O ( · ) hides problem dependent constants of a polynomial of N , and we have used y 0 k
Revised: where O ( · ) hides problem dependent constants of a polynomial of N , and we have used y 0 k

Original: = x k .
Revised: = x k .

Original: Theorem 2 implies a finite-time convergence rate of O ( ϵ − 2 ) , which is independent of the numberof sequences N .
Revised: Theorem 2 implies a finite-time convergence rate of O ( ϵ − 2 ) , which is independent of the number of sequences N .

Original: The error metric ∥∇ F ( x k ) ∥ used in Theorem 2 is of interest since it is a generalmeasure of the convergence of x k widely adopted in many applications of SA, especially when theincrement of x k is not strongly-monotone.
Revised: The error metric ∥∇ F ( x k ) ∥ used in Theorem 2 is of interest since it is a general measure of the convergence of x k widely adopted in many applications of SA, especially when the increment of x k is not strongly-monotone.

Original: Moreover, although we have assumed the existence anduniqueness of x ∗ in (6), the proof of Theorem 2 does not utilize this fact and thus the theorem appliesto the more general case where x ∗ is not unique or even does not exist. 
Revised: Moreover, although we have assumed the existence and uniqueness of x ∗ in (6), the proof of Theorem 2 does not utilize this fact and thus the theorem applies to the more general case where x ∗ is not unique or even does not exist. Remark 2 (Comments on stacking all the variables)

Revised: One naive way to establish the convergence of a multi-sequence update is to stack all the variables and view it as one sequence.

Revised: However, the stacked sequence requires stronger assumptions that are violated in the applications to converge.

Revised: For one, we would need v, h 1 , ..., h N to be jointly lipschitz continuous w.r.t.

Revised: the stacked variable ( x, y 1 , ..., y N ) .

Original:  Next we will showcase how the results can be applied to optimization and RL problems.
Revised: This condition is violated in, e.g., the application of actor-critic (Section 3.2). The upper-bounded function F can also be difficult to find. As it can be seen later in, e.g., Section 3 that such a F only exists for x and might not exist for the stacked variable ( x, y ) . Next we will showcase how the results can be applied to optimization and RL problems.

Original:  Applications to Stochastic Bilevel Optimization
Revised:  Applications to Stochastic Bilevel Optimization

Original: With mappings f : R d 0 × R d 1 (cid:55)→ R and g : R d 0 × R d 1 (cid:55)→ R , consider the following formulation ofthe bilevel optimization problem:
Revised: With mappings f : R d 0 × R d 1 (cid:55)→ R and g : R d 0 × R d 1 (cid:55)→ R , consider the following formulation of the bilevel optimization problem:

Original: [Equation]
Revised: [Equation]

Original: 14) where ζ and φ are two random variables.
Revised: (14) where ζ and φ are two random variables.

Original: 3.1 Reduction from the generic STSA results
Revised: 3.1 Reduction from the generic STSA results

Original: A popular approach to solving (14) is the gradient-based method [20, 25, 26,  8].
Revised: A popular approach to solving (14) is the gradient-based method [21, 26, 27, 8].

Original: Under someconditions that will be specified later, the gradient of F ( x ) takes the following form [20]:
Revised: Under some conditions that will be specified later, the gradient of F ( x ) takes the following form [21]:

Original: [Equation]
Revised: [Equation]

Original: Computing (15) requires y ∗ ( x ) , which is often unknown in practice.
Revised: Computing (15) requires y ∗ ( x ) , which is often unknown in practice.

Original: Instead, one can iterativelyupdate y k to approach y ∗ ( x k ) while using y k in place of y ∗ ( x k ) during the computation of (15)[25, 8].
Revised: Instead, one can iteratively update y k to approach y ∗ ( x k ) while using y k in place of y ∗ ( x k ) during the computation of (15) [26, 8].

Original: This leads to an update same as that in (5) with N =1 , where the mappings are defined as [Equation] 16a [Equation]
Revised: This leads to an update same as that in (5) with N =1 , where the mappings are defined as h ( x, y

Original: 16b

Original: [Equation]
Revised: [Equation]

Original: Since we only have two sequences, that is N =1 , we omit the index n to simplify notations.
Revised: Since we only have two sequences, that is N =1 , we omit the index n to simplify notations.

Original: In (16),ζ k is a random variable with the same distribution as that of ζ , and φ k , φ ′ k have the same distributionas that of φ .
Revised: In (16), ζ k is a random variable with the same distribution as that of ζ , and φ k , φ ′ k have the same distribution as that of φ .

Original: Here H yyk is a stochastic approximation of the Hessian inverse [ ∇ yy g ( x k , y k )] − 1 .
Revised: Here H yyk is a stochastic approximation of the Hessian inverse [ ∇ yy g ( x k , y k )] − 1 .

Original: Givenx k , when y k reaches the optimal solution y ∗ ( x k ) , it follows from (15) that v ( x k , y ∗ ( x k ))
Revised: Given x k , when y k reaches the optimal solution y ∗ ( x k ) , it follows from (15) that v ( x k , y ∗ ( x k ))

Original: =
Revised: =

Original: ∇ F ( x k ) .
Revised: ∇ F ( x k ) .

Original: As being discussed below Assumption 1, the lower-level optimal solution y ∗ ( x k ) is drifting at eachiteration.
Revised: As being discussed below Assumption 1, the lower-level optimal solution y ∗ ( x k ) is drifting at each iteration.

Original: Under the Lipschitz continuity assumption of y ∗ ( x ) , the drifting ∥
Revised: Under the Lipschitz continuity assumption of y ∗ ( x ) , the drifting ∥

Original: y ∗ ( x k +1 )
Revised: y ∗ ( x k +1 )

Original: − y ∗ ( x k ) ∥scales with ∥ x k +1 − x k ∥ which ultimately scales with ∥∇ F ( x k ) ∥ .
Revised: − y ∗ ( x k ) ∥ scales with ∥ x k +1 − x k ∥ which ultimately scales with ∥∇ F ( x k ) ∥ .

Original: To control the drift scale, formeranalysis heavily relies on the condition that ∥∇ F ( x k ) ∥ can be bounded for any k .
Revised: To control the drift scale, former analysis heavily relies on the condition that ∥∇ F ( x k ) ∥ can be bounded for any k .

Original: In SBO, this meansto either make a strong assumption on the Lipschitz continuity of f ( x, y ) w.r.t.
Revised: In SBO, this means to either make a strong assumption on the Lipschitz continuity of f ( x, y ) w.r.t.

Original: ( x, y ) , which leads tothe Lipschitz continuity of F ( x ) and the boundedness of ∥∇ F ( x k ) ∥ [8]; or to introduce projection in(16) to forcibly confine x k in a compact set [25], all of which greatly narrow the range of application.
Revised: ( x, y ) , which leads to the Lipschitz continuity of F ( x ) and the boundedness of ∥∇ F ( x k ) ∥ [8]; or to introduce projection in (16) to forcibly confine x k in a compact set [26], all of which greatly narrow the range of application.

Original: We will show that neither of these the conditions is needed by applying our generic results to SBO.
Revised: We will show that neither of these the conditions is needed by applying our generic results to SBO.

Original: Lemma 1 (Verifying assumptions of STSA) Consider the following conditions
Revised: Lemma 1 (Verifying assumptions of STSA) Consider the following conditions

Original: (a) For any x ∈ R d 1 , g ( x, y ) is strongly convex w.r.t.
Revised: (a) For any x ∈ R d 1 , g ( x, y ) is strongly convex w.r

Original:  y with modulus λ 1 > 0 .
Revised: t. y with modulus λ 1 > 0 .

Original: (b) There exist constants L xy , l xy , l yy such that ∇ y g
Revised: (b) There exist constants L xy , l xy , l yy such that ∇ y g

Original: ( x, y ) is L xy -Lipschitz continuous w.r.t.x; ∇ y g ( x, y ) is L h -Lipschitz continuous w.r.t.
Revised: ( x, y ) is L xy -Lipschitz continuous w.r.t. x; ∇ y g ( x, y ) is L h -Lipschitz continuous w.r.t.

Original: y .
Revised: y .

Original: ∇ xy g ( x, y ) , ∇ yy g ( x, y ) are respectivelyl xy -Lipschitz and l yy -Lipschitz continuous w.r.t.
Revised: ∇ xy g ( x, y ) , ∇ yy g ( x, y ) are respectively l xy -Lipschitz and l yy -Lipschitz continuous w.r.t.

Original: ( x, y ) .
Revised: ( x, y ) .

Original: (c) There exist constants l fx , l fy , l ′ fy , l y such that ∇ x f ( x, y ) and ∇ y f ( x, y ) are respectivelyl fx and l fy Lipschitz continuous w.r.t.
Revised: (c) There exist constants l fx , l fy , l ′ fy , l y such that ∇ x f ( x, y ) and ∇ y f ( x, y ) are respectively l fx and l fy Lipschitz continuous w.r.t.

Original: y ; ∇ y f ( x, y ) is l ′ fy -Lipschitz continuous w.r.t.
Revised: y ; ∇ y f ( x, y ) is l ′ fy -Lipschitz continuous w.r.t.

Original: x ;f ( x, y ) is l y -Lipschitz continuous w.r.t.
Revised: x ; f ( x, y ) is l y -Lipschitz continuous w.r.t.

Original: y .
Revised: y .

Original: (d) F ( x ) satisfies the restricted secant inequality: There exists a constant λ 0 > 0 such that⟨∇ F ( x ) , x − x ∗ ⟩ ≤ − λ 

Revised: (d) F ( x ) satisfies the restricted secant inequality: There exists a constant λ 0 > 0 such that ⟨∇ F ( x ) , x − x ∗ ⟩ ≤ − λ

Original: ∥ x − x ∗ ∥2 , where x ∗ := arg max x ∈ R d 1 F ( x ) .(e)
Revised: ∥ x − x ∗ ∥ , where x ∗ = arg max x ∈ R d 1 F ( x ) .

Original:  For any k , there exist constants c 0 , c 1 such that ∥ E [ ξ k |F 1 k ] ∥ 2 ≤ c 20 α k and ∥ E [ ψ k |F k ] ∥ 2 ≤c 21 β k ; there exist constants σ 0 , σ 1 such that E [ ∥ ξ k ∥ 2 |F 1 k ] ≤ σ 20 and E [
Revised: (e) For any k , there exist constants c 0 , c 1 such that ∥ E [ ξ k |F 1 k ] ∥ 2 ≤ c 20 α k and ∥ E [ ψ k |F k ] ∥ 2 ≤ c 21 β k ; there exist constants σ 0 , σ 1 such that E [ ∥ ξ k ∥ 2 |F 1 k ] ≤ σ 20 and E [

Original: ∥ ψ k ∥ 2 |F k ] ≤ σ 21 .
Revised: ∥ ψ k ∥ 2 |F k ] ≤ σ 21 .

Original: (f) There exists a constant C F such that F ( x ) ≤ C F .
Revised: (f) There exists a constant C F such that F ( x ) ≤ C F .

Original: We use a −→ b to indicate that a is a sufficient condition of b .
Revised: We use a ⇒ b to indicate that a is a sufficient condition of b .

Original: Then we have ( a ) & ( b ) −→ Assumption 1 ; ( a ) − ( c ) −→ Assumption 2 ; ( e ) −→ Assumption 3 ; ( a ) −→ Assumption 4 ; ( d ) −→ Assumption 5 ; ( f ) −→ Assumption 6 .
Revised: Then we have ( a ) & ( b ) ⇒ Assumption 1 ; ( a ) − ( c ) ⇒ Assumption 2 ; ( e ) ⇒ Assumption 3 ; ( a ) ⇒ Assumption 4 ; ( d ) ⇒ Assumption 5 ; ( f ) ⇒ Assumption 6 .

Original: The conditions listed above are commonly adopted in the literature [20, 25, 8].
Revised: The conditions listed above are commonly adopted in the literature [21, 26, 8].

Original: It is worth notingthat Lemma 1 does not need the L xy -Lipschitz continuity condition of f ( x, y ) w.r.t.
Revised: It is worth noting that Lemma 1 does not need the L xy -Lipschitz continuity condition of f ( x, y ) w.r.t.

Original: ( x, y ) .
Revised: ( x, y ) .

Original: This
Revised: This

Original: Lipschitz condition along with the L y -Lipschitz continuity of y ∗ ( x ) , which is implied by the standardconditions in Lemma 1, further leads to the Lipschitz continuity of F ( x ) :
Revised: Lipschitz condition along with the L y -Lipschitz continuity of y ∗ ( x ) , which is implied by the standard conditions in Lemma 1, further leads to the Lipschitz continuity of F ( x ) :

Original: [Equation]
Revised: [Equation]

Original: Although it is rather restrictive, this condition has been used in the previous work when F ( x ) is notstrongly-concave.
Revised: Although it is rather restrictive, this condition has been used in the previous work when F ( x ) is not strongly-concave.

Original: While our analysis does not need this condition.
Revised: While our analysis does not need this condition.

Original: Lastly, condition ( e ) is guaranteedby using independent samples in the upper and lower level along with [20, Algorithm 3] to obtain agood H yyk , which takes Ω( − log α k ) samples per iteration.
Revised: Lastly, condition ( e ) is guaranteed by using independent samples in the upper and lower level along with [21, Algorithm 3] to obtain a good H yyk , which takes Ω( − log α k ) samples per iteration.

Original: With Lemma 1, we have the followingcorollary regarding the convergence of (16).
Revised: With Lemma 1, we have the following corollary regarding the convergence of (16).

Original: Corollary 1 (STSA for SBO) Consider the STSA sequences with the update in (16) .
Revised: Corollary 1 (STSA for SBO) Consider the STSA sequences with the update in (16) .

Original: Under theconditions (a)–(e), Theorem 1 holds.
Revised: Under Conditions (a)–(e), Theorem 1 holds; that is, with α k = Θ( 1 k ) and β k = Θ ( 1 k ) we have [Equation]

Original:  Under Conditions (a)–(c), (e)-(f), Theorem 2 holds.
Revised: 18b) Under Conditions (a)–(c), (e) and (f), Theorem 2 holds; i.e., with α k =Θ( 1 √ K ) , β k =Θ( 1 √ K ) , we have [Equation]

Original:  Remark 2 (Comparison with prior art in SBO)
Revised: [Equation] Remark 3 (Comparison with prior art in SBO)

Original: When F ( x ) is strongly concave, Corollary 1 implies the sample complexity of O ( ϵ − 1 log ϵ − 1 ) , which improves over the best-known sample complexity O
Revised: When F ( x ) is strongly concave, Corollary 1 implies the sample complexity of O ( ϵ − 1 log ϵ − 1 ) , which improves over the best-known sample complexity O

Original: ( ϵ − 1 .
Revised: ( ϵ − 1 .

Original: 5 log ϵ − 1 ) in [25].
Revised: 5 log ϵ − 1 ) in [26].

Original: Different from [25], we do not need the projection of x k to a compactset.
Revised: Different from [26], we do not need the projection of x k to a compact set.

Original: When F ( x ) is non-concave, corollary 1 suggests a sample complexity of O ( ϵ − 2 log ϵ − 1 ) , whichis the same as the state-of-art complexity established in [8].
Revised: When F ( x ) is non-concave, corollary 1 suggests a sample complexity of O ( ϵ − 2 log ϵ − 1 ) , which is the same as the state-of-art complexity established in [8].

Original: Corollary 1 improves the result in [8] intwo major aspects: 1) it relaxes the Lipschitz continuity assumption on f ( x, y ) ; and, 2) an alternatingupdate is adopted in [8] to ensure stability, while some applications of SBO only allow simultaneousupdates.
Revised: Corollary 1 improves the result in [8] in two major aspects: 1) it relaxes the Lipschitz continuity assumption on f ( x, y ) ; and, 2) an alternating update is adopted in [8] to ensure stability, while some applications of SBO only allow simultaneous updates.

Original: Corollary 1 applies to those cases and thus has a broader range of application.
Revised: Corollary 1 applies to those cases and thus has a broader range of application.

Original: 3.2 Application to advantage actor-critic
Revised: 3.2 Application to advantage actor-critic

Original: RL problems are often modeled as a MDP described by M = {S , A , P , r, γ } , where S is the statespace, A is the action space; P ( s ′ | s, a ) is the probability of transitioning to s ′ ∈S given ( s, a ) ∈S×A ;r ( s, a ) ∈ [0 , 1] is the reward associated with ( s, a ) ; and γ ∈ (0 , 1) is a discount factor.
Revised: RL problems are often modeled as a MDP described by M = {S , A , P , r, γ } , where S is the state space, A is the action space; P ( s ′ | s, a ) is the probability of transitioning to s ′ ∈S given ( s, a ) ∈S×A ; r ( s, a ) ∈ [0 , 1] is the reward associated with ( s, a ) ; and γ ∈ (0 , 1) is a discount factor.

Original: A policy πmaps S to a distribution over A , and we use π ( a | s ) to denote the probability of choosing a under s .
Revised: A policy π maps S to a distribution over A , and we use π ( a | s ) to denote the probability of choosing a under s .

Original: Given a policy π , we define the value functions as V π ( s ) :
Revised: Given a policy π , we define the value functions as V π ( s ) :

Original: = E π (cid:104) (cid:80) ∞ t =0 γ t r ( s t , a t ) | s 

Revised: = E π (cid:104) (cid:80) ∞ t =0 γ t r ( s t , a t ) | s

Original: = s (cid:105) , where
Revised: = s (cid:105) , where

Original: E π is taken over the trajectory ( s 0 , a 0 , s 1 , a 1 , . . . )
Revised: E π is taken over the trajectory ( s 0 , a 0 , s 1 , a 1 , . . . )

Original: generated under policy π and transition kernel P .
Revised: generated under policy π and transition kernel P .

Original: With ρ denoting the initial state distribution, the discounted visitation distribution induced by policyπ is defined via d π ( s, a ) =
Revised: With ρ denoting the initial state distribution, the discounted visitation distribution induced by policy π is defined via d π ( s, a ) =

Original: (1 − γ ) (cid:80) ∞
Revised: (1 − γ ) (cid:80) ∞

Original: t =

Revised: t =

Original: γ t Pr π ( s t =
Revised: γ t Pr π ( s t =

Original: s | s 0 ∼ ρ ) π ( a | s ) .
Revised: s | s 0 ∼ ρ ) π ( a | s ) .

Original: To overcome the difficultyof learning a function, we parameterize the policy with x ∈ R d 0 , and solve
Revised: To overcome the difficulty of learning a function, we parameterize the policy with x ∈ R d 0 , and solve

Original: [Equation]
Revised: [Equation]

Original: To solve for (19), a popular method is the actor-critic (AC) method [31].
Revised: To solve for (20), a popular method is the actor-critic (AC) method [32].

Original: The actor-critic algorithmwith linear critic function is a special case of (5).
Revised: The actor-critic algorithm with linear critic function is a special case of (5).

Original: Specifically, the critic variable y is updated with
Revised: Specifically, the critic variable y is updated with

Original: [Equation]
Revised: [Equation]

Original: where µ π x is the stationary distribution of the Markov chain induced by π x , ϕ ( s ) ∈ R d 1 is the featurevector encoding state s and the sample ( s k , a k , s ′ k ) is returned by some sampling protocol.
Revised: where µ π x is the stationary distribution of the Markov chain induced by π x , ϕ ( s ) ∈ R d 1 is the feature vector encoding state s and the sample ( s k , a k , s ′ k ) is returned by some sampling protocol.

Original: Undersome regularity conditions, it is known that there exists a unique y ∗ ( x ) such that h ( x, y ∗ ( x ))
Revised: Under some regularity conditions, it is known that there exists a unique y ∗ ( x ) such that h ( x, y ∗ ( x ))

Original: = 

Revised: =

Original: The actor variable x is then updated with
Revised: The actor variable x is then updated with

Original: [Equation]
Revised: [Equation]

Original: For the AC update in (20) and (21), Assumption 2–4 and 6 or their sufficient conditions have beenexplored in the RL context by previous works [52].
Revised: For the AC update in (21) and (22), Assumption 2–4 and 6 or their sufficient conditions have been explored in the RL context by previous works [57].

Original: However, the smoothness of y ∗ ( x ) in Assumption1, which is the key condition leading to a faster convergence rate, has yet been verified.
Revised: However, the smoothness of y ∗ ( x ) in Assumption 1, which is the key condition leading to a faster convergence rate, has yet been verified.

Original: With thesame conditions as those adopted in [52], we prove that y ∗ ( x ) is indeed smooth.
Revised: With the same conditions as those adopted in [57], we prove that y ∗ ( x ) is indeed smooth.

Original: Lemma 2 Consider the AC update in(20) - (21)  .
Revised: Lemma 2 Consider the AC update in (21) - (22) .

Original: Under certain standard conditions, y ∗ ( x ) isdifferentiable and there existsa constant L y ′ {+> 
+}
Revised: Under the standard conditions specified in Appendix E, y ∗ ( x ) is differentiable and there exists L y ′ {+> 
+}

Original: such that ∥∇ y ∗ ( x ) − ∇ y ∗ ( x ′ )
Revised: such that ∥∇ y ∗ ( x ) − ∇ y ∗ ( x ′ )

Original: ∥ ≤ L y ′
Revised: ∥ ≤ L y ′

Original: ∥
Revised: ∥

Original: x − x ′ ∥ .
Revised: x − x ′ ∥ .

Original: As a comparison, the above condition was directly assumed in [8], while we provide a formaljustification for Lemma 2 in this work.
Revised: As a comparison, the above condition was directly assumed in [8], while we provide a formal justification for Lemma 2 in this work.

Original: With detailed verification of all assumptions deferred to
Revised: With detailed verification of all assumptions deferred to

Original: Appendix E, we then directly present the theorem regarding the convergence of AC.
Revised: Appendix E, we then directly present the theorem regarding the convergence of AC.

Original: Theorem 3 (Complexity of AC) Consider the AC update(20) - (21)  .
Revised: Theorem 3 (Complexity of AC) Consider the AC update (21) - (22) .

Original: Under some standard conditions specified in Appendix E, Theorem 2 holds.
Revised: Under the standard conditions specified in Appendix E, Theorem 2 holds; that is, with α k =Θ( 1 √ K ) and β k =Θ( 1 √ K ) , we have

Original:  In [8, 52], the projection step is adopted in the y k update to ensure that ∥ y k ∥ < ∞ , ∀ k .
Revised: [Equation] In [8, 57], the projection step is adopted in the y k update to ensure that ∥ y k ∥ < ∞ , ∀ k .

Original: Since theprojection radius is unknown in practice, adopting the projection is essentially assuming that ∥ y k ∥can be bounded for any k , which is quite strong.
Revised: Since the projection radius is unknown in practice, adopting the projection is essentially assuming that ∥ y k ∥ can be bounded for any k , which is quite strong.

Original: Theorem 3 holds without this projection.
Revised: Theorem 3 holds without this projection.

Original:  Applications to Stochastic Compositional Optimization
Revised:  Applications to Stochastic Compositional Optimization

Original: Define mappings f n : R d n (cid:55)→ R d n +1 for n = 0 , 1 , ..., N with d N +1 = 1 .
Revised: Define mappings f n : R d n (cid:55)→ R d n +1 for n = 0 , 1 , ..., N with d N +1 = 1 .

Original: The multi-level stochasticcompositional problem can be formulated as
Revised: The multi-level stochastic compositional problem can be formulated as

Original: [Equation]
Revised: [Equation]

Revised: f n ( x ):= E

Revised: [Equation]

Revised: [ f

Revised: [Equation]

Revised: ( x

Revised: [Equation]

Revised: ζ n )

Revised: [Equation]

Revised: n

Original:  where ζ 0 , ζ 1 , . . . , ζ N are random variables.
Revised: [Equation] [Equation] (24) where ζ 0 , ζ 1 , . . . , ζ N are random variables.

Original: Here we slightly overload the notation and use f n ( x ; ζ n )to represent the stochastic version of the mapping.
Revised: Here we slightly overload the notation and use f n ( x ; ζ n ) to represent the stochastic version of the mapping.

Original: 4.1 Reduction from the generic STSA results
Revised: 4.1 Reduction from the generic STSA results

Original: To solve the problem in (22), a natural scheme is to use the stochastic gradient descent method withthe gradient given by
Revised: To solve the problem in (24), a natural scheme is to use the stochastic gradient descent method with the gradient given by

Original: [Equation]
Revised: [Equation]

Original: 23) where we use ∇ f n ( f n − 1 ( . . .
Revised: where we use ∇ f n ( f n − 1 ( . . .

Original: f 0 ( x ) . . . ))
Revised: f 0 ( x ) . . . ))

Original: = ∇ f n ( x ) | x = f n − 1 ( ...f 0 ( x ) ... ) .
Revised: = ∇ f n ( x ) | x = f n − 1 ( ...f 0 ( x ) ... ) .

Original: To obtain a stochasticestimator of ∇ F ( x ) , we will need to obtain the stochastic estimators for ∇ f n ( f n − 1 ( ...f 0 ( x ) ... )) foreach n .
Revised: To obtain a stochastic estimator of ∇ F ( x ) , we will need to obtain the stochastic estimators for ∇ f n ( f n − 1 ( ...f 0 ( x ) ... )) for each n .

Original: For example, when n = 1 , one will need the estimator of ∇ f 1 ( E ζ 

Revised: For example, when n = 1 , one will need the estimator of ∇ f 1 ( E ζ

Original: [ f 0 ( x ; ζ 0 )]) .
Revised: [ f 0 ( x ; ζ 0 )]) .

Original: However,due to the possible non-linearity of ∇ f 1 ( · ) , the natural candidate ∇ f 1 ( f 0 ( x ; ζ 0 )) is not an unbiasedestimator of ∇ f 1 ( E ζ 

Revised: However, due to the possible non-linearity of ∇ f 1 ( · ) , the natural candidate ∇ f 1 ( f 0 ( x ; ζ 0 )) is not an unbiased estimator of ∇ f 1 ( E ζ

Original: [ f 0 ( x ; ζ 0 )]) .
Revised: [ f 0 ( x ; ζ 0 )]) .

Original: To tackle this issue, a popular method is to directly approximatethe mean E ζ n [ f n ( · ; ζ n )] with a tracking variable y n ∈ R d n for n = 0 , 1 , ..., N , see e.g., [56].
Revised: To tackle this issue, a popular method is to directly approximate the mean E ζ n [ f n ( · ; ζ n )] with a tracking variable y n ∈ R d n for n = 0 , 1 , ..., N , see e.g., [61].

Original: The update of y n is then a special case of the SA update in (5) with the generic mapping defined as
Revised: The update of y n is then a special case of the SA update in (5) with the generic mapping defined as

Original: [Equation]
Revised: [Equation]

Original: where ζ 0 k , . . .
Revised: where ζ 0 k , . . .

Original: , ζ Nk have the same distributions as that of ζ 0 , . . .
Revised: , ζ Nk have the same distributions as that of ζ 0 , . . .

Original: , ζ N respectively.
Revised: , ζ N respectively.

Original: It is then clear thateach y nk has a unique fixed-point y n, ∗ k = f n − 1 ( y n − 1 k ) , and thus y nk can be viewed as an approximationof f n ( y n − 1 k ) .
Revised: It is then clear that each y nk has a unique fixed-point y n, ∗ k = f n − 1 ( y n − 1 k ) , and thus y nk can be viewed as an approximation of f n ( y n − 1 k ) .

Original: With these approximations, variable x is updated in the form of (5) by definingv (
Revised: With these approximations, variable x is updated in the form of (5) by defining

Original: [Equation]
Revised: [Equation]

Original: , y

Original: [Equation]

Original: [Equation]

Original: where ˆ ζ 0 k has the same distribution as that of ζ 0 .
Revised: where ˆ ζ 0 k has the same distribution as that of ζ 0 .

Original: It is clear that when every y nk reaches its fixed-pointy n, ∗ k , it follows from (23) that v ( x k , y 1 , ∗ k , ..., y N, ∗ k )
Revised: It is clear that when every y nk reaches its fixed-point y n, ∗ k , it follows from (25) that v ( x k , y 1 , ∗ k , ..., y N, ∗ k )

Original: =
Revised: =

Original: ∇ F ( x k ) , which indicates that the expectedupdate direction of x k in (25) is ∇ F ( x k ) .
Revised: ∇ F ( x k ) , which indicates that the expected update direction of x k in (27) is ∇ F ( x k ) .

Original: Due to space limitation, we directly give the following corollary on the convergence of  the SCOupdate with the assumptions and proof deferred to Appendix F.
Revised: Next we provide a lemma that summarizes the sufficient conditions of Assumption 1–6. The listed conditions are standard in the literature [61, 7]. Lemma 3 (Verifying assumptions of STSA) Consider the following conditions (g)

Revised: Given any n ∈ { 0 , 1 , . . . , N } , there exist positive constants L y,n and L y ′ ,n such that the mapping f n ( · ) is L y,n -Lipschitz continuous and L y ′ ,n -smooth.

Revised: (h) Given F k , for any n ∈ [ N ] : f n ( y n − 1 k ; ζ nk ) and ∇ f n ( y n − 1 k ; ζ nk ) are respectively the unbiased estimators of f n ( y n − 1 k ) and ∇ f n ( y n − 1 k ) with bounded variance; f 0 ( x k ; ˆ ζ 0 k ) and ∇ f 0 ( x k ; ˆ ζ 0 k ) are respectively the unbiased estimators of f 0 ( x k ) and ∇ f 0 ( x k ) with bounded variance.

Revised: (i)

Revised: At each iteration k , ˆ ζ 0 k , ζ 0 k , ζ 1 k , . . .

Revised: , ζ Nk are conditionally independent of each other given F k .

Original: {+(j) Function F ( x ) satisfies the restricted secant inequality: There exists a constant λ 0 > 
+}
Revised: {+(j) Function F ( x ) satisfies the restricted secant inequality: There exists a constant λ 0 > 
+}

Original: {+such that ⟨∇ F ( x ) , x − x ∗ ⟩ ≤ − λ 
+}
Revised: {+such that ⟨∇ F ( x ) , x − x ∗ ⟩ ≤ − λ 
+}

Revised: ∥ x − x ∗ ∥ 2 , where x ∗ := arg max x ∈ R d 1 F ( x ) .

Revised: (k) There exists a constant C F such that F ( x ) ≤ C F .

Revised: We use a ⇒ b to indicate that a is a sufficient condition of b .

Revised: Then we have ( g ) ⇒ Assumption 1 and 2 ; ( h ) and ( i ) ⇒ Assumption 3 ; ( j ) ⇒ Assumption 5 ; ( k ) ⇒ Assumption 6 ; Assumption 4 holds for (26) .

Revised: With Lemma 3, we can directly arrive at the following corollary on the convergence of the stochastic compositional optimization method.

Original: Corollary 2 (STSA for multi-level SCO) Consider the STSA sequences generated with update(24) - (25) .
Revised: Corollary 2 (STSA for multi-level SCO) Consider the STSA sequences generated by (26) - (27) .

Original: Under the standard conditions specified in Appendix F, Theorem 1 or 2 holds.
Revised: Under Conditions (g)–(j), Theorem 1 holds. Under Conditions (g)–(i) and (k), Theorem 2 holds.

Original: Remark 3 (Comparison with prior art in SCO)
Revised: Remark 4 (Comparison with prior art in SCO)

Original: Corollary 2 establishes the sample complexity of
Revised: Corollary 2 establishes the sample complexity of

Original: O ( ϵ − 1 ) for the strongly monotone case and the complexity of O ( ϵ − 2 ) for the non-monotone case,which are both independent of N .
Revised: O ( ϵ − 1 ) for the strongly monotone case and the complexity of O ( ϵ − 2 ) for the non-monotone case, which are both independent of N .

Original: This improves over the O ( ϵ − N +54 ) complexity for the stronglyconcave case and the O ( ϵ − N +84 ) complexity for the non-concave case shown in [56].
Revised: This improves over the O ( ϵ − N +54 ) complexity for the strongly concave case and the O ( ϵ − N +84 ) complexity for the non-concave case shown in [61].

Original: There are otherworks that establish the same complexity as that in Corollary 2, but they require modification to thebasic SA update (24) and (25) to achieve acceleration; see e.g., [7, 1, 44].
Revised: There are other works that establish the same complexity as that in Corollary 2, but they require modification to the basic SA update (26) and (27) to achieve acceleration; see e.g., [7, 1, 47].

Original: 4.2 Application to model-agnostic meta policy gradient
Revised: 4.2 Application to model-agnostic meta policy gradient

Original: Consider a set of MDPs {M i } Mi =1 with M i = {S , A , P i , r i , γ } .
Revised: Consider a set of MDPs {M i } Mi =1 with M i = {S , A , P i , r i , γ } .

Original: The MDPs model a set of RLtasks that share the same state-action space while having different transition kernels P i and rewardfunctions r i .
Revised: The MDPs model a set of RL tasks that share the same state-action space while having different transition kernels P i and reward functions r i .

Original: To better compare with the previous work [14], we consider the finite-horizon objectivefunction with the policy π parametrized by x ∈ Rd 0 : F i ( x ) := E ζ ∼ π x (cid:104) (cid:80) Ht =
Revised: To better compare with the previous work [14], we consider the finite-horizon objective function with the policy π parametrized by x ∈ R

Original: [Equation]
Revised: [Equation]

Original: ( s

Original: [Equation]

Original: [Equation]

Original: ) | ρ

Original: [Equation]
Revised: [Equation]

Original: , P i (cid:105) ,where H ∈ N + is the horizon, and E ζ ∼ π x is taken over the trajectory ζ :=
Revised: (cid:105) , where H ∈ N + is the horizon, and E ζ ∼ π x is taken over the trajectory ζ :=

Original: ( s 0 , a 0 , s 1 , a 1 , ..., s H , a H )generated under policy π x , initial distribution
Revised: ( s 0 , a 0 , s 1 , a 1 , ..., s H , a H ) generated under policy π x , initial distribution

Original: ρ i and transition kernels P i .
Revised: ρ i and transition kernels P i .

Original: The goal of MAMPG is to find an initial policy π x that can achieve good performance in new tasksby performing a few policy gradient steps [15, 14].
Revised: The goal of MAMPG is to find an initial policy π x that can achieve good performance in new tasks by performing a few policy gradient steps [15, 14].

Original: In the case where N steps of gradient update areperformed, the problem of finding an initial policy parameter x can be formulated as
Revised: In the case where N steps of gradient update are performed, the problem of finding an initial policy parameter x can be formulated as

Original: [Equation]
Revised: [Equation]

Original: (26) where x is the shared initial policy parameter, i.e. ˜ x 0 i
Revised: where x is the shared initial policy parameter, i.e. ˜ x 0 i

Original: = x for any task i and ˜ x Ni ( x ) is the parameterafter running N steps of gradient ascent with respect to F i starting from x .
Revised: = x for any task i and ˜ x Ni ( x ) is the parameter after running N steps of gradient ascent with respect to F i starting from x .

Original: Solving (26) with SCO method.
Revised: Solving (28) with SCO method.

Original: The MAMPG problem in (26) can be solved by the stochastic compositional optimization method introduced before.
Revised: The MAMPG problem in (28) can be solved by the stochastic compositional optimization method introduced before.

Original: In order to get ∇ F ( x ) , one will need ∇ F i
Revised: In order to get ∇ F ( x ) , one will need ∇ F i

Original: (˜ x Ni ( x ))for each task i .
Revised: (˜ x Ni ( x )) for each task i .

Original: Observe that F i (˜ x Ni ( x )) can be written as a compositional function:F i (
Revised: Observe that F i (˜ x Ni ( x )) can be written as a compositional function:

Original: [Equation]
Revised: [Equation]

Original: x
Revised: f

Original: [Equation]
Revised: [Equation]

Original: = f Ni ( 
Revised: [Equation] ) with f ni ( x ) := x + η ∇ F i ( x ) , n = 0 , ..., N −

Original: [Equation]
Revised: [Equation]

Original: [-i ( . . . f 
 i ( x ) . . . )) with f ni ( x ) : = x + η ∇ F i ( x ) , n = 0 , ..., N − 1 , (27)-] where f Ni ( x ) =
Revised: [-i ( . . . f 
 i ( x ) . . . )) with f ni ( x ) : = x + η ∇ F i ( x ) , n = 0 , ..., N − 1 , (27)-]29) where f Ni ( x ) =

Original: F i ( x ) .
Revised: F i ( x ) .

Original: In order to approximate ∇ F i (˜ x Ni ( x )) , we can follow the discussion in
Revised: In order to approximate ∇ F i (˜ x Ni ( x )) , we can follow the discussion in

Original: Section 4 and introduce tracking variables y ni ∈ R d 0 for n ∈ [ N ] which are updated as follows
Revised: Section 4 and introduce tracking variables y ni ∈ R d 0 for n ∈ [ N ] which are updated as follows

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where we define f ni ( · ; ζ ) as a stochastic approximation of f ni ( · ) with random trajectory ζ .
Revised: where we define f ni ( · ; ζ ) as a stochastic approximation of f ni ( · ) with random trajectory ζ .

Original: Then weestimate ∇ F i (˜ x Ni ( x )) by ˆ ∇ F i,k :=
Revised: Then we estimate ∇ F i (˜ x Ni ( x )) by ˆ ∇ F i,k defined as

Original: ∇ f 0 i ( x ; ˆ ζ 0 k,i ) ∇ f 1 i ( y 1 k,i ; ζ 1 k,i ) · · · ∇ f Ni ( y Nk,i ; ζ Nk,i ) .
Revised: [Equation]

Original: To obtain an estimation of ∇ F ( x ) , we needto obtain ˆ ∇ F i,k for each i ∈ { 1 , 2 , ..., M } , that is, weneed to do update(28) for each i .
Revised: To obtain an estimation of ∇ F ( x ) , we need ˆ ∇ F i,k for each i ∈ { 1 , 2 , ..., M } . Thus we do (30) for each i .

Original: With { ˆ ∇ F i,k } Mi =1 ,we update the initial policy  as x k +

Revised: With { ˆ ∇ F i,k } Mi =1 , the initial policy is updated as x k +

Original: = x k + α k 1 M (cid:80) Mi
Revised: = x k + α k 1 M (cid:80) Mi

Original: =1 ˆ ∇
Revised: =1 ˆ ∇

Original: F i,k .
Revised: F i,k .

Original: Reduction from the generic results.
Revised: Reduction from the generic results.

Original: Let y n ∈ R d n M be a concatenation of y n i for i ∈ { 1 , 2 , ..., M } .
Revised: Let y n ∈ R d n M be a concatenation of y ni for i ∈ { 1 , 2 , ..., M } .

Original: With ζ nk := { ζ nk,i } Mi =1 , let f n ( y nk ; ζ nk ) be a concatenation of f ni ( y nk,i ; ζ nk,i ) for i ∈ { 1 , 2 , ..., M } .
Revised: With ζ nk := { ζ nk,i } Mi =1 , let f n ( y nk ; ζ nk ) be a concatenation of f ni ( y nk,i ; ζ nk,i ) for i ∈ { 1 , 2 , ..., M } .

Original: Thenwe can write the tracking variable update of all tasks jointly in the form of (5a), that is
Revised: Then we can write the tracking variable update of all tasks jointly in the form of (5a), that is

Original: [Equation]
Revised: [Equation]

Original: The initial policy update is a special case of (5b), that is
Revised: The initial policy update is a special case of (5b), that is

Original: [Equation]
Revised: [Equation]

Original: Due to space limitation, we directly give the result below and defer the proof to Appendix G.
Revised: Due to space limitation, we directly give the result below and defer the proof to Appendix G.

Original: Theorem 4 (Complexity of MAMPG) Consider the STSA sequences generated by the MAMPGupdate in (29) and (30) .
Revised: Theorem 4 (Complexity of MAMPG) Consider the STSA sequences generated by the MAMPG update in (32) and (33) .

Original: Under some standard conditions specified in Appendix G, Theorem 2 holds.
Revised: Under some standard conditions specified in Appendix G, Theorem 2 holds.

Original: Theorem 4 implies a sample complexity of O ( ϵ − 2 ) to achieve the ϵ -stationary initial policy, whichimproves over the O ( ϵ − 4 ) sample complexity in [14].
Revised: Theorem 4 implies a sample complexity of O ( ϵ − 2 ) to achieve the ϵ -stationary initial policy, which improves over the O ( ϵ − 4 ) sample complexity in [14].

Original: Moreover, Theorem 4 holds for any N ≥ 1 in(26), while the method in 
Revised: Moreover, Theorem 4 holds for any N ≥ 1 in (28), while the method in [14] only applies to the case N = 1 .  Conclusions

Revised: In this work, we consider the general nonlinear SA with multiple coupled sequences, and study its non-asymptotic performance.

Revised: Different from the dominating two-timescale SA analysis, we are particularly interested in under which conditions, single-timescale analysis can be applied to nonlinear

Original: Checklist
Revised: SA with multiple coupled sequences. When all the sequences have strongly monotone increments, we establish the iteration complexity of O ( ϵ − 1 ) . When the main sequence is not strongly-monotone, we establish the iteration complexity of O ( ϵ − 2 ) . We then apply our generic SA analysis to stochastic bilevel and compositional optimization and improve their existing results.

Revised: Specifically, we improve the state-of-the-art convergence rate of: 1) the SBO method and its application to the AC method; and, 2) the multi-level SCO method and its application to the MAMPG method.

Revised: Acknowledgments

Revised: This work was partially supported by National Science Foundation MoDL-SCALE Grantand the Rensselaer-IBM AI Research Collaboration ( http://airc.rpi.edu ), part of the IBM AI

Revised: Horizons Network ( http://ibm.biz/AIHorizons ).

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] This work only considers the strongly-monotone increments for the follower sequences, but it will be interesting to also consider the monotone increments with non-unique fixed point y ∗ .
Revised: [Yes] This work only considers the strongly-monotone increments for the follower sequences, but it will be interesting to also consider the monotone increments with non-unique fixed point y ∗ .

Original: Right now

Original:  the generic result in this work can only be applied to the unconstrained stochastic optimization, while it will be also interesting to consider whether it is possible to establish similar result that is applicable to the constrained stochastic optimization 
Revised: Right now the generic result in this work can only be applied to the unconstrained stochastic optimization, while it will be also interesting to consider whether it is possible to establish similar result that is applicable to the constrained stochastic optimization problems.

Original: problems.

Original: We will add discussion on the limitations in the revision.
Revised: We will add discussion on the limitations in the revision.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A] (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
Revised: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [N/A] (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
Revised: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A] 
Revised: [N/A] 4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: did

Original: you cite the creators

Original: [N/A] (b) Did you mention the license of the assets
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’reusing/curating
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/ A](e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A](b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A](c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]