Original: A Single-Timescale Analysis for Stochastic Approximation with Multiple Coupled Sequences
Revised: A Single-timescale Analysis for Stochastic Approximation with Multiple Coupled Sequences

Original: Abstract
Revised: Abstract

Original: Stochastic approximation (SA) with multiple coupled sequences has found broadapplications in machine learning such as bilevel learning and reinforcement learning(RL).
Revised: Stochastic approximation (SA) with multiple coupled sequences has found broadapplications in machine learning such as bilevel learning and reinforcement learning(RL).

Original: In this paper, we study the finite-time convergence of nonlinear SA withmultiple coupled sequences.
Revised: In this paper, we study the finite-time convergence of nonlinear SA withmultiple coupled sequences.

Original: Different from existing multi-timescale analysis, weseek for scenarios where a fine-grained analysis can provide the tight performanceguarantee for single-timescale multi-sequence SA (STSA).
Revised: Different from existing multi-timescale analysis, weseek for scenarios where a fine-grained analysis can provide the tight performanceguarantee for single-timescale multi-sequence SA (STSA).

Original: At the heart of ouranalysis is the smoothness property of the fixed points in multi-sequence SAthat holds in many applications.
Revised: At the heart of ouranalysis is the smoothness property of the fixed points in multi-sequence SAthat holds in many applications.

Original: When all sequences have strongly monotoneincrements, we establish the iteration complexity of O ( ϵ − 1 ) to achieve ϵ -accuracy,which improves the existing O ( ϵ − 1 . 5 ) complexity for two coupled sequences.
Revised: When all sequences have strongly monotoneincrements, we establish the iteration complexity of O ( ϵ − 1 ) to achieve ϵ -accuracy,which improves the existing O ( ϵ − 1 . 5 ) complexity for two coupled sequences.

Original: When the main sequence does not have strongly monotone increment, we establishthe iteration complexity of O ( ϵ − 2 ) .
Revised: When the main sequence does not have strongly monotone increment, we establishthe iteration complexity of O ( ϵ − 2 ) .

Original: We showcase the power of our result byapplying it to stochastic bilevel and compositional optimization problems, as wellas RL problems, all of which lead to improvements over their existing guarantees.
Revised: We showcase the power of our result byapplying it to stochastic bilevel and compositional optimization problems, as wellas RL problems, all of which lead to improvements over their existing guarantees.

Original:  Introduction
Revised:  Introduction

Original: Stochastic approximation (SA) is an iterative procedure used to find the zero of a function when onlythe noisy estimate of the function is observed.
Revised: Stochastic approximation (SA) is an iterative procedure used to find the zero of a function when onlythe noisy estimate of the function is observed.

Original: Specifically, with the mapping v : R d (cid:55)→ R d thesingle-sequence SA seeks to solve for v ( x ) = 0 with the following iterative update:
Revised: Specifically, with the mapping v : R d (cid:55)→ R d thesingle-sequence SA seeks to solve for v ( x ) = 0 with the following iterative update:

Original: [Equation]
Revised: [Equation]

Original: where α k is the step size and ξ k is a random variable.
Revised: where α k is the step size and ξ k is a random variable.

Original: Since its introduction in [43] single-sequence
Revised: Since its introduction in [39] single-sequence

Original: SA has received great interests because of its broad range of applications to areas including stochasticoptimization and reinforcement learning (RL) [6, 49].
Revised: SA has received great interests because of its broad range of applications to areas including stochasticoptimization and reinforcement learning (RL) [6, 45].

Original: The asymptotic convergence of single-sequence
Revised: The asymptotic convergence of single-sequence

Original: SA can be established by the ordinary differential equation method; see e.g., [4].
Revised: SA can be established by the ordinary differential equation method; see e.g., [4].

Original: To gain moreinsights into the performance difference of various stochastic optimization algorithms, the finite-timeconvergence of SA has been widely studied in recent years; see e.g., [41, 40, 29, 47, 39, 13].
Revised: To gain moreinsights into the performance difference of various stochastic optimization algorithms, the finite-timeconvergence of SA has been widely studied in recent years; see e.g., [37, 36, 26, 43, 35, 13].

Original: While most of the SA studies focus on the single-sequence case, the double-sequence SA wasintroduced in [3], which has been extensively applied to the RL methods involving a double-sequencestochastic update structure [49, 31, 10].
Revised: While most of the SA studies focus on the single-sequence case, the double-sequence SA wasintroduced in [3], which has been extensively applied to the RL methods involving a double-sequencestochastic update structure [45, 28, 10].

Original: With mappings v : R d 0 × R d 1 (cid:55)→ R d 0 and h : R d 0 × R d 1 (cid:55)→
Revised: With mappings v : R d 0 × R d 1 (cid:55)→ R d 0 and h : R d 0 × R d 1 (cid:55)→

Original: R d 1 , the double-sequence SA seeks to solve v ( x, y )
Revised: R d 1 , the double-sequence SA seeks to solve v ( x, y )

Original: =
Revised: =

Original: h ( x, y )
Revised: h ( x, y )

Original: = 

Revised: =

Original: with the following update:
Revised: with the following update:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: where α k , β k are the step sizes, and ξ k , ψ k are random variables.
Revised: where α k , β k are the step sizes, and ξ k , ψ k are random variables.

Original: In (2), the update of x k andthat of y k depend on each other and thus the sequences are coupled .
Revised: In (2), the update of x k andthat of y k depend on each other and thus the sequences are coupled .

Original: Due to this coupling, thedouble-sequence SA is more challenging to analyze than its single-sequence counterpart.
Revised: Due to this coupling, thedouble-sequence SA is more challenging to analyze than its single-sequence counterpart.

Original: Prior art on double-sequence SA.
Revised: Prior art on double-sequence SA.

Original: Many recent analyses of the double-sequence SA focus on thelinear case where v ( x, y ) and h ( x, y ) are linear mappings; see e.g., [33, 11, 24, 28].
Revised: Many recent analyses of the double-sequence SA focus on thelinear case where v ( x, y ) and h ( x, y ) are linear mappings; see e.g., [30, 11, 22, 25].

Original: The key ideahere is to use the so-called two-time-scale (TTS) step sizes: One sequence is updated in the fastertime scale while the other is updated in the slower time scale; that is lim k →∞ α k /β k = 0 .
Revised: The key ideahere is to use the so-called two-time-scale (TTS) step sizes: One sequence is updated in the fastertime scale while the other is updated in the slower time scale; that is lim k →∞ α k /β k = 0 .

Original: By doingso, the two sequences are shown to decouple asymptotically, which allows us to leverage the analysisof the single-sequence SA.
Revised: By doingso, the two sequences are shown to decouple asymptotically, which allows us to leverage the analysisof the single-sequence SA.

Original: In particular, [28] proves an iteration complexity of O ( ϵ − 1 ) to achieveϵ -accuracy for the TTS linear SA, which is shown to be tight.
Revised: In particular, [25] proves an iteration complexity of O ( ϵ − 1 ) to achieveϵ -accuracy for the TTS linear SA, which is shown to be tight.

Original: With similar choice of the step sizes, the
Revised: With similar choice of the step sizes, the

Original: TTS nonlinear SA was analyzed in [37, 12].
Revised: TTS nonlinear SA was analyzed in [33, 12].

Original: In [37], the finite-time convergence rate of TTS nonlinear
Revised: In [33], the finite-time convergence rate of TTS nonlinear

Original: SA was established under an assumption that the two sequences converge asymptotically.
Revised: SA was established under an assumption that the two sequences converge asymptotically.

Original: Later,[12] alleviates this assumption and shows that TTS nonlinear SA achieves an iteration complexity of
Revised: Later,[12] alleviates this assumption and shows that TTS nonlinear SA achieves an iteration complexity of

Original: O ( ϵ − 1 . 5 ) .
Revised: O ( ϵ − 1 . 5 ) .

Original: However, this iteration complexity is larger than O ( ϵ − 1 ) of the TTS linear SA.
Revised: However, this iteration complexity is larger than O ( ϵ − 1 ) of the TTS linear SA.

Original: The gap between the complexities of nonlinear and linear SA motivates an interesting question:
Revised: The gap between the complexities of nonlinear and linear SA motivates an interesting question:

Original: Q1:
Revised: Q1:

Original: Is it possible to prove a faster rate for the nonlinear SA with two coupled sequences?
Revised: Is it possible to prove a faster rate for the nonlinear SA with two coupled sequences?

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: We first conduct an experiment to examine the possibility.
Revised: We first conduct an experiment to examine the possibility.

Original: Experiment.
Revised: Experiment.

Original: Figure 1 shows the performance of using thedouble-sequence SA (2) to solve the following problem
Revised: Figure 1 shows the performance of using thedouble-sequence SA (2) to solve the following problem

Original: [Equation]
Revised: [Equation]

Original: We use the double-sequence SA (2) to solve (3), wherev ( x,
Revised: We use the double-sequence SA (2) to solve (3), wherev ( x,

Original: [Equation]
Revised: [Equation]

Original: and ζ k , ξ k are independent Gaussian random variableswith zero mean and standard deviations of 0 .
Revised: and ζ k , ξ k are independent Gaussian random variableswith zero mean and standard deviations of 0 .

Original: It is easyto check that (4) satisfies the assumptions in the existing
Revised: It is easyto check that (4) satisfies the assumptions in the existing

Original: TTS-SA analysis [12].
Revised: TTS-SA analysis [12].

Original: Therefore, we can use the twotime-scale step sizes and achieve the iteration complexityof O
Revised: Therefore, we can use the twotime-scale step sizes and achieve the iteration complexityof O

Original: ( ϵ − 1 . 5 ) .
Revised: ( ϵ − 1 . 5 ) .

Original: However, as suggested by Figure 1, the iterates still converge with step sizes in a singletime-scale ( α k = Θ( 1 k ) , β k = Θ( 1 k ) ).
Revised: However, as suggested by Figure 1, the iterates still converge with step sizes in a singletime-scale ( α k = Θ( 1 k ) , β k = Θ( 1 k ) ).

Original: In this case, the iteration complexity is O
Revised: In this case, the iteration complexity is O

Original: ( ϵ − 1 ) , which isthe same as that of double-sequence linear SA [28].
Revised: ( ϵ − 1 ) , which isthe same as that of double-sequence linear SA [25].

Original: This suggests that existing analysis of doublesequence SA might not be tight, at least for the class of updates similar to (4).
Revised: This suggests that existing analysis of doublesequence SA might not be tight, at least for the class of updates similar to (4).

Original: Indeed, as we willshow later, the iterates generated by (4) will converge with the iteration complexity of O ( ϵ − 1 ) .
Revised: Indeed, as we willshow later, the iterates generated by (4) will converge with the iteration complexity of O ( ϵ − 1 ) .

Original: Furthermore, existing works on TTS SA mainly focus on the double-sequence case.
Revised: Furthermore, existing works on TTS SA mainly focus on the double-sequence case.

Original: While incases such as the multi-level stochastic optimization; see e.g., [56], more than two sequences are involved.
Revised: While incases such as the multi-level stochastic optimization; see e.g., [52], more than two sequences are involved.

Original: This necessitates the use of the multi-sequence SA.
Revised: This necessitates the use of the multi-sequence SA.

Original: Specifically, with mappingsv : R d 0 × R d 1 · · · × R d N (cid:55)→ R [Equation] 0 , h n : R [Equation]
Revised: Specifically, with mappingsv : R d 0 × R d 1 · · · × R d N (cid:55)→ R d 0 , h n : R d n − 1 × R d n

Original: n − 1 × R d n

Original: [Equation]
Revised: [Equation]

Original: R d n , we consider
Revised: R d n , we consider

Original: [Equation]
Revised: [Equation]

Original: where α k , β k, 1 , ..., β k,N are the step sizes, and ξ k , ψ 1 k , .
Revised: where α k , β k, 1 , ..., β k,N are the step sizes, and ξ k , ψ 1 k , .

Original: . . , ψ Nk are random variables.
Revised: . . , ψ Nk are random variables.

Original: For conciseness,we have used y 

Revised: For conciseness,we have used y

Original: k := x k here.
Revised: k := x k here.

Original: Our goal is to find the unique fixed-points x ∗ , y 1 , ∗ , . . .
Revised: Our goal is to find the unique fixed-points x ∗ , y 1 , ∗ , . . .

Original: , y N, ∗ such that
Revised: , y N, ∗ such that

Original: [Equation]
Revised: [Equation]

Original: Observing that in (5), for every n , the sequence of y nk is coupled with that of y n − 1 k and is ultimatelycoupled with the main sequence x k .
Revised: Observing that in (5), for every n , the sequence of y nk is coupled with that of y n − 1 k and is ultimatelycoupled with the main sequence x k .

Original: Meanwhile the update of x k also depends on { y nk } Nn =1 .
Revised: Meanwhile the update of x k also depends on { y nk } Nn =1 .

Original: Sinceall sequences in (5) are coupled, (5) is more challenging to analyze than the double-sequence SA.
Revised: Sinceall sequences in (5) are coupled, (5) is more challenging to analyze than the double-sequence SA.

Original: Prior art related to multi-sequence SA.
Revised: Prior art related to multi-sequence SA.

Original: The multilevel stochastic optimization problem [46] andthe multilevel SCO problem [1, 53, 58, 44, 60] are closely related to the multi-sequence SA.
Revised: The multilevel stochastic optimization problem [42] andthe multilevel SCO problem [1, 49, 54, 40, 56] are closely related to the multi-sequence SA.

Original: Totackle the multi-level structure, these recent methods have modified the vanilla multi-sequence SAupdate to achieve the state-of-the-art complexity and thus their updates are no longer in the form of(5).
Revised: Totackle the multi-level structure, these recent methods have modified the vanilla multi-sequence SAupdate to achieve the state-of-the-art complexity and thus their updates are no longer in the form of(5) .

Original: In contrast, we focus on the multi-sequence SA update in (5).
Revised: In contrast, we focus on the multi-sequence SA update in (5).

Original: To the best of our knowledge, theonly analysis for (5) is [56] where the TTS technique is generalized to multi-time-scale.
Revised: To the best of our knowledge, theonly analysis for (5) is [52] where the TTS technique is generalized to multi-time-scale.

Original: In [56], theiteration complexity will get worse as the number of sequences N increases.
Revised: In [52], theiteration complexity will get worse as the number of sequences N increases.

Original: This gives rise to another interesting question:
Revised: This gives rise to another interesting question:

Original: Q2:
Revised: Q2:

Original: Is it possible to establish convergence rate independent of the number of sequences?
Revised: Is it possible to establish convergence rate independent of the number of sequences?

Original: In this work, we give affirmative answers to both Questions Q1 and Q2 .
Revised: In this work, we give affirmative answers to both Questions Q1 and Q2 .

Original: Our contributions.
Revised: Our contributions.

Original: Specially, by exploiting the smooth assumption that can be satisfied in manyapplications, we show that the vanilla nonlinear SA can run in a single time scale !
Revised: Specially, by exploiting the smooth assumption that can be satisfied in manyapplications, we show that the vanilla nonlinear SA can run in a single time scale !

Original: We further provethat the order of the convergence rate is independent of the number of sequences N !
Revised: We further provethat the order of the convergence rate is independent of the number of sequences N !

Original: Intuitively, thisis possible because when the fixed point y n, ∗ is smooth in x , the y nk -update converges fast enoughsuch that its fixed-point residual after one-step update is at the same order as the drift of y n, ∗ .
Revised: Intuitively, thisis possible because when the fixed point y n, ∗ is smooth in x , the y nk -update converges fast enoughsuch that its fixed-point residual after one-step update is at the same order as the drift of y n, ∗ .

Original: In the context of prior art, our contributions can be summarized as follows (see Table 1)
Revised: In the context of prior art, our contributions can be summarized as follows (see Table 1).

Original: C1)
Revised: C1)

Original: Single-timescale analysis for multi-sequence SA.
Revised: Single-timescale analysis for multi-sequence SA.

Original: Different from existing two-timescale analysis[37, 5], we establish a unifying Single-Timescale analysis for SA with multiple coupled sequences thatwe term STSA .
Revised: Different from existing two-timescale analysis[33, 5], we establish a unifying Single-Timescale analysis for SA with multiple coupled sequences thatwe term STSA .

Original: When all the sequences have strongly-monotone increments, we improve the O ( ϵ − 1 . 5 )iteration complexity for multi-sequence TTS-SA in [12] to O
Revised: When all the sequences have strongly-monotone increments, we improve the O ( ϵ − 1 . 5 )iteration complexity for multi-sequence TTS-SA in [12] to O

Original: ( ϵ − 1 ) .
Revised: ( ϵ − 1 ) .

Original: When the main sequence doesnot have the strongly-monotone increment, we provide the O ( ϵ − 2 ) iteration complexity.
Revised: When the main sequence doesnot have the strongly-monotone increment, we provide the O ( ϵ − 2 ) iteration complexity.

Original: C2)
Revised: C2)

Original: STSA for stochastic bilevel optimization (SBO).
Revised: STSA for stochastic bilevel optimization (SBO).

Original: When applying our generic results to theSBO problem with double-sequence SA, for strongly-concave objective functions, we improve thebest-known sample complexity ˜
Revised: When applying our generic results to theSBO problem with double-sequence SA, for strongly-concave objective functions, we improve thebest-known sample complexity ˜

Original: O ( ϵ − 1 . 5 ) of TTSA in [25] to ˜ O
Revised: O ( ϵ − 1 . 5 ) of TTSA in [23] to ˜ O

Original: ( ϵ − 1 ) .
Revised: ( ϵ − 1 ) .

Original: For the non-concave objectivefunction, we achieve the same sample complexity
Revised: For the non-concave objectivefunction, we achieve the same sample complexity

Original: O ( ϵ − 2 ) of ALSET while relaxing the boundedupper-level gradient assumption made in [8].
Revised: O ( ϵ − 2 ) of ALSET while relaxing the bounded upper-level gradient assumption made in [8].

Original: C3)
Revised: C3)

Original: STSA for stochastic compositional optimization (SCO).
Revised: STSA for stochastic compositional optimization (SCO).

Original: When applying our results to the multilevel SCO problems, we improve the level-dependent sample complexities O ( ϵ − N +54 ) and O ( ϵ − N +84 )of multi-sequence SA based α -TSCGD method in [56] to the level-independent complexities˜
Revised: When applying our results to the multi- level SCO problems, we improve the level-dependent sample complexities O ( ϵ − N +54 ) and O ( ϵ − N +84 ) of multi-sequence SA based α -TSCGD method in [52] to the level-independent complexities

Original:  O
Revised: ˜ O

Original: ( ϵ − 1 )and O ( ϵ − 2 ) , under the strongly-concave and non-concave objective functions, respectively.
Revised: ( ϵ − 1 )and O ( ϵ − 2 ) , under the strongly-concave and non-concave objective functions, respectively.

Original: C4)
Revised: C4)

Original: STSA for policy optimization in RL problems.
Revised: STSA for policy optimization in RL problems.

Original: Moreover, applying our results to the actorcritic method achieves the same O ( ϵ − 2 ) sample complexity of ALSET-AC in [8] while relaxing theunverifiable assumption on the stationary distribution of Markov chains; applying our results to themeta policy gradient improves the O ( ϵ − 4 ) sample complexity of SG-MRL in [14] to O ( ϵ − 2 ) .
Revised: Moreover, applying our results to the actor- critic method achieves the same O ( ϵ − 2 ) sample complexity of ALSET-AC in [8] while relaxing the unverifiable assumption on the stationary distribution of Markov chains; applying our results to the meta policy gradient improves the O ( ϵ − 4 ) sample complexity of SG-MRL in [14] to O ( ϵ − 2 ) .

Revised: Roadmap.

Revised: We organize the results as follows.

Revised: With the multi-sequence SA update introduced in thissection, we first present our main results in Section 2.

Revised: In Section 3, we apply our results to the SBO problem which encompasses the actor-critic method.

Revised: In Section 4, we apply our result to the SCOproblem which encompasses the model-agnostic meta policy gradient method.

Original:  Main Results: Convergence of Single-timescale Multi-sequence SA
Revised:  Main Results: Convergence of Single-timescale Multi-sequence SA

Original: Before introducing the main results, we will first make some standard assumptions.
Revised: Before introducing the main results, we will first make some standard assumptions.

Original: Throughout thediscussion, we define [ N ] := { 1 , 2 , ..., N } , [ K ] := { 1 , 2 , ..., K } and y 0 :
Revised: Throughout thediscussion, we define [ N ] := { 1 , 2 , ..., N } , [ K ] := { 1 , 2 , ..., K } and y 0 :

Original: = x for conciseness.
Revised: = x for conciseness.

Original: Assumption 1 (Smoothness of the fixed points)
Revised: Assumption 1 (Smoothness of the fixed points)

Original: For any n ∈ [ N ] and y n − 1 ∈ R d n − 1 , there existsa unique y n, ∗ ( y n − 1 ) ∈ R
Revised: For any n ∈ [ N ] and y n − 1 ∈ R d n − 1 , there existsa unique y n, ∗ ( y n − 1 ) ∈ R

Original: d n such that h n ( y n − 1 , y n, ∗ ( y n − 1 ))
Revised: d n such that h n ( y n − 1 , y n, ∗ ( y n − 1 ))

Original: = 0 .
Revised: = 0 .

Original: Moreover, there exist constants
Revised: Moreover, there exist constants

Original: L y,n and L y ′ ,n such that for any y n − 1 , ¯ y n − 1 ∈ R d n − 1 , the following inequalities hold
Revised: L y,n and L y ′ ,n such that for any y n − 1 , ¯ y n − 1 ∈ R d n − 1 , the following inequalities hold

Original: [Equation]
Revised: [Equation]

Original: Due to the change of y n − 1 k at each iteration, the solution of h n ( y n − 1 k , y n ) = 0 with respect to (w.r.t.)y n , that is y n, ∗ ( y n − 1 k ) , is drifting over consecutive iterations.
Revised: Due to the change of y n − 1 k at each iteration, the solution of h n ( y n − 1 k , y n ) = 0 with respect to (w.r.t.)y n , that is y n, ∗ ( y n − 1 k ) , is drifting over consecutive iterations.

Original: Given y n − 1 k , since only one-step of y nkupdate is performed at each iteration, one can only hope to establish convergence of y nk if the driftof its optimal solution is controlled in some sense.
Revised: Given y n − 1 k , since only one-step of y nkupdate is performed at each iteration, one can only hope to establish convergence of y nk if the driftof its optimal solution is controlled in some sense.

Original: Assumption 1 ensures both the zeroth-order andfirst-order drifts are controlled in the same scale of the change of y n − 1 k .
Revised: Assumption 1 ensures both the zeroth-order andfirst-order drifts are controlled in the same scale of the change of y n − 1 k .

Original: This assumption is satisfiedin linear SA [28] and other applications which will be shown later.
Revised: This assumption is satisfiedin linear SA [25] and other applications which will be shown later.

Original: Define [Equation]
Revised: Define v ( x ) :

Original:  ( y 1 , ∗ ( x )) , . . .
Revised: = v (cid:0) x, y 1 , ∗ ( x ) , y 2 , ∗ ( y 1 , ∗ ( x )) , . . .

Original: , y N, ∗ ( . . .
Revised: , y N, ∗ ( . . .

Original: y 2 , ∗ ( y 1 , ∗ ( x )) . . . )
Revised: y 2 , ∗ ( y 1 , ∗ ( x )) . . . )

Original: (cid:1) .
Revised: (cid:1) .

Original: With y 1: N as a concise notation for ( y 1 , ..., y N ) , we make the following assumption.
Revised: With y 1: N as a concise notation for ( y 1 , ..., y N ) , we make the following assumption.

Original: Assumption 2 (Lipschitz continuity of increments) For any n ∈ [ N ] , x, ¯ x ∈ R d 0 and y n , ¯ y n ∈
Revised: Assumption 2 (Lipschitz continuity of increments) For any n ∈ [ N ] , x, ¯ x ∈ R d 0 and y n , ¯ y n ∈

Original: R d n , there exist constants L v , L v,y and L h,n such that the following inequalities hold
Revised: R d n , there exist constants L v , L v,y and L h,n such that the following inequalities hold

Original: [Equation]
Revised: [Equation]

Original: Define F k as the σ -algebra generated by the random variables in { x i , y 1: N i } ki =1 and F nk as theσ -algebra generated by { x i , y 1: N i } ki =1 ∪ { y nk +1 } .
Revised: Define F k as the σ -algebra generated by the random variables in { x i , y 1: N i } ki =1 and F nk as theσ -algebra generated by { x i , y 1: N i } ki =1 ∪ { y nk +1 } .

Original: We make the following assumption on the noises.
Revised: We make the following assumption on the noises.

Original: Assumption 3 (Bias and variance)
Revised: Assumption 3 (Bias and variance)

Original: There exist constants { c n , σ n } Nn =0 such that ∀ k, n ,∥ E [ ξ k |F 1 k ] ∥ 2 ≤ c 20 α k and ∥ E [ ψ nk |F n +1 k ] ∥ 2 ≤ c 2 n β k,n ; E [ ∥ ξ k ∥ 2 |F 1 k ] ≤ σ 20 and E [ ∥ ψ nk ∥|F n +1 k ] ≤ σ 2 n .
Revised: There exist constants { c n , σ n } Nn =0 such that ∀ k, n ,∥ E [ ξ k |F 1 k ] ∥ 2 ≤ c 20 α k and ∥ E [ ψ nk |F n +1 k ] ∥ 2 ≤ c 2 n β k,n ; E [ ∥ ξ k ∥ 2 |F 1 k ] ≤ σ 20 and E [ ∥ ψ nk ∥|F n +1 k ] ≤ σ 2 n .

Original: Here we define F N +1 k :
Revised: Here we define F N +1 k :

Original: = F k .
Revised: = F k .

Original: Assumption 3 is a generalized version of the bias and varianceassumption in stochastic programming [18] or the noise assumption in single-sequence SA [29] tomulti-sequence case.
Revised: Assumption 3 is a generalized version of the standard zero-mean noisewith bounded moments assumption [30].

Original: Similar assumption has also been made in the double-sequence SA [25]. Aswill be shown later, when applying STSA to the stochastic optimization problems, the conditionalindependence between samples of different levels given F k along with the small bias and boundedvariance condition will guarantee this assumption.
Revised: Given F k , if ψ nk and ξ k are independent of each other andhave zero-mean, Assumption 3 holds with c n = 0 for any n .

Original: Assumption 4 (Monotonicity of h ) For n ∈ [ N ] , h n ( y n − 1 , y n ) is one-point strongly monotone ony n, ∗ ( y n − 1 ) given any y n − 1 ; that is, there exists constant λ n > 0 such that (cf.
Revised: Assumption 4 (Monotonicity of h ) For n ∈ [ N ] , h n ( y n − 1 , y n ) is one-point strongly monotone ony n, ∗ ( y n − 1 ) given any y n − 1 ; that is, there exists constant λ n > 0 such that (cf.

Original: h n ( y n [Equation] 1 , y n, ∗ ) = 0 )
Revised: h n ( y n − 1 , y n, ∗ ) = 0 )

Original: [Equation]
Revised: [Equation]

Original: Assumption 4 is implied by the standard regularity assumptions in the previous works on TTS linear
Revised: Assumption 4 is implied by the standard regularity assumptions in the previous works on TTS linear

Original: SA [33, 28], and has also been exploited in the TTS nonlinear SA works; see e.g. [37, 12].
Revised: SA [30, 25], and has also been exploited in the TTS nonlinear SA works; see e.g. [33, 12].

Original: 2.1 The strongly-monotone case
Revised: 2.1 The strongly-monotone case

Original: We first consider the case when the main sequence x k has strongly-monotone increment.
Revised: We first consider the case when the main sequence x k has strongly-monotone increment.

Original: Assumption 5 (Monotonicity of v ) Suppose v ( x ) is one-point strongly monotone on x ∗ ; that is,there exists a positive constant λ 0 such that (cf.
Revised: Assumption 5 (Monotonicity of v ) Suppose v ( x ) is one-point strongly monotone on x ∗ ; that is,there exists a positive constant λ 0 such that (cf.

Original: v ( x ∗ ) = 0 )
Revised: v ( x ∗ ) = 0 )

Original: [Equation]
Revised: [Equation]

Original: Same as Assumption 4, Assumption 5 is standard in the previous works on TTS SA [37, 12].
Revised: Same as Assumption 4, Assumption 5 is standard in the previous works on TTS SA [33, 12].

Original: Thisassumption is a regularity assumption in the case of TTS linear SA; see e.g., [33, Assumption 2.3].
Revised: Thisassumption is a regularity assumption in the case of TTS linear SA; see e.g., [30, Assumption 2.3].

Original: Or in the case of bilevel optimization which will be discussed later, this assumption is satisfied whenthe objective function is strongly-concave.
Revised: Or in the case of bilevel optimization which will be discussed later, this assumption is satisfied whenthe objective function is strongly-concave.

Original: Due to space limitation, we directly present the result below and defer the proof to Appendix B.
Revised: Due to space limitation, we directly present the result below and defer the proof to Appendix B.

Original: Theorem 1 Consider the sequences generated by (5) .
Revised: Theorem 1 Consider the sequences generated by (5) .

Original: Suppose Assumptions 1–5 hold.
Revised: Suppose Assumptions 1–5 hold.

Original: Select stepsizes α k = Θ( 1 k ) and β k,n =
Revised: Select stepsizes α k = Θ( 1 k ) and β k,n =

Original: Θ( 1 k ) .
Revised: Θ( 1 k ) .

Original: It holds for any k that
Revised: It holds for any k that

Original: [Equation]
Revised: [Equation]

Original: where O ( · ) hides constants in the polynomial of N , and we have used y 0 k
Revised: where O ( · ) hides constants in the polynomial of N , and we have used y 0 k

Original: = x k for convenience.
Revised: = x k for convenience.

Original: Moreover, for any n ∈ [ N ] we have
Revised: Moreover, for any n ∈ [ N ] we have

Original: [Equation]
Revised: [Equation]

Original: almost surely (a.s.) ,
Revised: almost surely (a.s.) ,

Original: [Equation]
Revised: [Equation]

Original: It is worth noting that with (7a), Theorem 1 also implies the same convergence result for the errormetric ∥ x k − x
Revised: It is worth noting that with (7a), Theorem 1 also implies the same convergence result for the errormetric ∥ x k − x

Original: [Equation]
Revised: [Equation]

Original: ∥
Revised: ∥

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: Nn =1 ∥
Revised: Nn =1 ∥

Original: y nk − y n, ∗ ∥ 2 , the formal justification of which is deferred to the proofof Theorem 1.
Revised: y nk − y n, ∗ ∥ 2 , the formal justification of which is deferred to the proofof Theorem 1.

Original: It is worth noting that the order of convergence in Theorem 1 is independent of N ,which is in contrast to the convergence rate that gets worse as N increases [12, 56].
Revised: It is worth noting that the order of convergence in Theorem 1 is independent of N ,which is in contrast to the convergence rate that gets worse as N increases [12, 52].

Original: Remark 1 (Comparison with prior art in multi-sequence SA) Theorem 1 bridges the gap between the convergence rates of double-sequence linear and nonlinear SA by improving over the
Revised: Remark 1 (Comparison with prior art in multi-sequence SA) Theorem 1 bridges the gap between the convergence rates of double-sequence linear and nonlinear SA by improving over the

Original: O ( k − 23 ) rate shown in [12] with the additional assumption (7b).
Revised: O ( k − 23 ) rate shown in [12] with the additional assumption (7b).

Original: As will be shown later, this assumption is satisfied in various applications.
Revised: As will be shown later, this assumption is satisfied in various applications.

Original: Theorem 1 also generalizes the O ( 1 k ) convergence rate in thedouble-sequence linear SA analysis (e.g., [28]) to the multi-sequence nonlinear SA case.
Revised: Theorem 1 also generalizes the O ( 1 k ) convergence rate in thedouble-sequence linear SA analysis (e.g., [25]) to the multi-sequence nonlinear SA case

Original: 2.2 The non-strongly-monotone case
Revised: 2.2 The non-strongly-monotone case

Original: Some applications of multi-sequence nonlinear SA such as the actor-critic method [31], Assumption5 does not hold.
Revised: Some applications of multi-sequence nonlinear SA such as the actor-critic method [28], Assumption5 does not hold.

Original: This motivates us to consider a more general setting in this subsection where v ( x ) isnon-strongly-monotone.
Revised: This motivates us to consider a more general setting in this subsection where v ( x ) isnon-strongly-monotone.

Original: Throughout this subsection, we make the following assumption.
Revised: Throughout this subsection, we make the following assumption.

Original: Assumption 6 Suppose there exists a mapping F : R d 0 (cid:55)→ R such that ∇ F ( x ) = v ( x ) .
Revised: Assumption 6 Suppose there exists a mapping F : R d 0 (cid:55)→ R such that ∇ F ( x ) = v ( x ) .

Original: Thesequence of { x k } is contained in an open set over which F ( x ) is upper bounded; e.g. F ( x ) ≤ C F .
Revised: Thesequence of { x k } is contained in an open set over which F ( x ) is upper bounded; e.g. F ( x ) ≤ C F .

Original: As will be shown later, F ( x ) can be chosen as the objective function when applying SA to maximization problems.
Revised: As will be shown later, F ( x ) can be chosen as the objective function when applying SA to maximization problems.

Original: Assumption 6 is standard to ensure the convergence of x k ; see e.g. [6].
Revised: Assumption 6 is standard to ensure the convergence of x k ; see e.g. [6].

Original: The following theorem gives the general finite-time convergence result of the nonlinear SA when themain sequence has the non-strongly-monotone increment.
Revised: The following theorem gives the general finite-time convergence result of the nonlinear SA when themain sequence has the non-strongly-monotone increment.

Original: The proof is deferred to Appendix C.
Revised: The proof is deferred to Appendix C.

Original: Theorem 2 Consider the sequences generated by (5) for k=[K].
Revised: Theorem 2 Consider the sequences generated by (5) for k=[K].

Original: Suppose Assumptions 1–4 & 6hold.
Revised: Suppose Assumptions 1–4 & 6hold.

Original: Select α k = Θ( 1 √ K ) , β k,n = Θ( 1 √ K ) with properly chosen initial step sizes, then it holds that
Revised: Select α k = Θ( 1 √ K ) , β k,n = Θ( 1 √ K ) with properly chosen initial step sizes, then it holds that

Original: [Equation]
Revised: [Equation]

Original: where O ( · ) hides problem dependent constants of a polynomial of N , and we have used y 0 k
Revised: where O ( · ) hides problem dependent constants of a polynomial of N , and we have used y 0 k

Original: = x k .
Revised: = x k .

Original: Theorem 2 implies a finite-time convergence rate of O ( ϵ − 2 ) , which is independent of the numberof sequences N .
Revised: Theorem 2 implies a finite-time convergence rate of O ( K − 2 ) , which is independent of the numberof sequences N .

Original: The error metric ∥∇ F ( x k ) ∥ used in Theorem 2 is of interest since it is a generalmeasure of the convergence of x k widely adopted in many applications of SA, especially when theincrement of x k is not strongly-monotone.
Revised: The error metric ∥∇ F ( x k ) ∥ used in Theorem 2 is of interest since it is a generalmeasure of the convergence of x k widely adopted in many applications of SA, especially when theincrement of x k is not strongly-monotone.

Original: Moreover, although we have assumed the existence anduniqueness of x ∗ in (6), the proof of Theorem 2 does not utilize this fact and thus the theorem appliesto the more general case where x ∗ is not unique or even does not exist.
Revised: Moreover, although we have assumed the existence anduniqueness of x ∗ in (6), the proof of Theorem 2 does not utilize this fact and thus the theorem appliesto the more general case where x ∗ is not unique or even does not exist.

Original: Next we will showcase how the results can be applied to optimization and RL problems.
Revised: Next we will showcase how the results can be applied to optimization and RL problems.

Original:  Applications to Stochastic Bilevel Optimization
Revised:  Applications to Stochastic Bilevel Optimization

Original: With mappings f : R d 0 × R d 1 (cid:55)→ R and g : R d 0 × R d 1 (cid:55)→ R , consider the following formulation ofthe bilevel optimization problem:
Revised: With mappings f : R d 0 × R d 1 (cid:55)→ R and g : R d 0 × R d 1 (cid:55)→ R , consider the following formulation ofthe bilevel optimization problem:

Original: [Equation]
Revised: [Equation]

Original: 14) where ζ and φ are two random variables.
Revised: 14) where ζ and φ are two random variables.

Original: 3.1 Reduction from the generic STSA results
Revised: 3.1 Reduction from the generic STSA results

Original: A popular approach to solving (14) is the gradient-based method [20, 25, 26, 8].
Revised: A popular approach to solving (14) is the gradient-based method [19, 23, 24, 8].

Original: Under someconditions that will be specified later, the gradient of F ( x ) takes the following form [20]:
Revised: Under someconditions that will be specified later, the gradient of F ( x ) takes the following form [19]:

Original: [Equation]
Revised: [Equation]

Original: Computing (15) requires y ∗ ( x ) , which is often unknown in practice.
Revised: Computing (15) requires y ∗ ( x ) , which is often unknown in practice.

Original: Instead, one can iterativelyupdate y k to approach y ∗ ( x k ) while using y k in place of y ∗ ( x k ) during the computation of (15)[25, 8].
Revised: Instead, one can iterativelyupdate y k to approach y ∗ ( x k ) while using y k in place of y ∗ ( x k ) during the computation of (15)[23, 8].

Original: This leads to an update same as that in (5) with N =1 , where the mappings are defined as
Revised: This leads to an update same as that in (5) with N =1 , where the mappings are defined as

Original: [Equation]
Revised: [Equation]

Original: 16a
Revised: 16a

Original: [Equation]
Revised: [Equation]

Original: 16b
Revised: 16b

Original: [Equation]
Revised: [Equation]

Original: Since we only have two sequences, that is N =1 , we omit the index n to simplify notations.
Revised: Since we only have two sequences, that is N =1 , we omit the index n to simplify notations.

Original: In (16),ζ k is a random variable with the same distribution as that of ζ , and φ k , φ ′ k have the same distributionas that of φ .
Revised: In (16),ζ k is a random variable with the same distribution as that of ζ , and φ k , φ ′ k have the same distributionas that of φ .

Original: Here H yyk is a stochastic approximation of the Hessian inverse [ ∇ yy g ( x k , y k )] − 1 .
Revised: Here H yyk is a stochastic approximation of the Hessian inverse [ ∇ yy g ( x k , y k )] − 1 .

Original: Givenx k , when y k reaches the optimal solution y ∗ ( x k ) , it follows from (15) that v ( x k , y ∗ ( x k ))
Revised: Givenx k , when y k reaches the optimal solution y ∗ ( x k ) , it follows from (15) that v ( x k , y ∗ ( x k ))

Original: =
Revised: =

Original: ∇ F ( x k ) .
Revised: ∇ F ( x k ) .

Original: As being discussed below Assumption 1, the lower-level optimal solution y ∗ ( x k ) is drifting at eachiteration.
Revised: As being discussed below Assumption 1, the lower-level optimal solution y ∗ ( x k ) is drifting at eachiteration.

Original: Under the Lipschitz continuity assumption of y ∗ ( x ) , the drifting ∥
Revised: Under the Lipschitz continuity assumption of y ∗ ( x ) , the drifting ∥

Original: y ∗ ( x k +1 )
Revised: y ∗ ( x k +1 )

Original: − y ∗ ( x k ) ∥scales with ∥ x k +1 − x k ∥ which ultimately scales with ∥∇ F ( x k ) ∥ .
Revised: − y ∗ ( x k ) ∥scales with ∥ x k +1 − x k ∥ which ultimately scales with ∥∇ F ( x k ) ∥ .

Original: To control the drift scale, formeranalysis heavily relies on the condition that ∥∇ F ( x k ) ∥ can be bounded for any k .
Revised: To control the drift scale, formeranalysis heavily relies on the condition that ∥∇ F ( x k ) ∥ is bounded for any k .

Original: In SBO, this meansto either make a strong assumption on the Lipschitz continuity of f ( x, y ) w.r.t.
Revised: In SBO, this means toeither make a strong assumption on the Lipschitz continuity of f ( x, y ) w.r.t.

Original: ( x, y ) , which leads tothe Lipschitz continuity of F ( x ) and the boundedness of ∥∇ F ( x k ) ∥ [8]; or to introduce projection in(16) to forcibly confine x k in a compact set [25], all of which greatly narrow the range of application.
Revised: ( x, y ) , which leads tothe Lipschitz continuity of F ( x ) and the boundedness of ∥∇ F ( x k ) ∥ [8]; or to introduce projection in(16) to forcibly confine x k in a compact set [23], all of which greatly narrow the range of application.

Original: We will show that neither of these the conditions is needed by applying our generic results to SBO.
Revised: We will show that neither of these the conditions is needed by applying our generic results to SBO.

Original: Lemma 1 (Verifying assumptions of STSA) Consider the following conditions
Revised: Lemma 1 (Verifying assumptions of STSA) Consider the following conditions

Original: (a) For any x ∈ R d 1 , g ( x, y ) is strongly convex w.r.t.
Revised: (a) For any x ∈ R d 1 , g ( x, y ) is strongly convex w.r.t.

Original: y with modulus λ 1 > 0 .
Revised: y with modulus λ 1 > 0 .

Original: (b) There exist constants L xy , l xy , l yy such that ∇ y g
Revised: (b) There exist constants L xy , l xy , l yy such that ∇ y g

Original: ( x, y ) is L xy -Lipschitz continuous w.r.t.x; ∇ y g ( x, y ) is L h -Lipschitz continuous w.r.t.
Revised: ( x, y ) is L xy -Lipschitz continuous w.r.t.x; ∇ y g ( x, y ) is L h -Lipschitz continuous w.r.t.

Original: y .
Revised: y .

Original: ∇ xy g ( x, y ) , ∇ yy g ( x, y ) are respectivelyl xy -Lipschitz and l yy -Lipschitz continuous w.r.t.
Revised: ∇ xy g ( x, y ) , ∇ yy g ( x, y ) are respectivelyl xy -Lipschitz and l yy -Lipschitz continuous w.r.t.

Original: ( x, y ) .
Revised: ( x, y ) .(c)

Original: (c) There exist constants l fx , l fy , l ′ fy , l y such that ∇ x f ( x, y ) and ∇ y f ( x, y ) are respectivelyl fx and l fy Lipschitz continuous w.r.t.
Revised: There exist constants l fx , l fy , l ′ fy , l y such that ∇ x f ( x, y ) and ∇ y f ( x, y ) are respectivelyl fx and l fy Lipschitz continuous w.r.t.

Original: y ; ∇ y f ( x, y ) is l ′ fy -Lipschitz continuous w.r.t.
Revised: y ; ∇ y f ( x, y ) is l ′ fy -Lipschitz continuous w.r.t.

Original: x ;f ( x, y ) is l y -Lipschitz continuous w.r.t.
Revised: x ;f ( x, y ) is l y -Lipschitz continuous w.r.t.

Original: y .
Revised: y .

Original: (d) F ( x ) satisfies the restricted secant inequality: There exists a constant λ 0 > 0 such that⟨∇ F ( x ) , x − x ∗ ⟩ ≤ − λ 

Revised: (d) F ( x ) satisfies the restricted secant inequality: There exists a constant λ 0 > 0 such that⟨∇ F ( x ) , x − x ∗ ⟩ ≤ − λ

Original: ∥ x − x ∗ ∥ 2 , where x ∗ := arg max x ∈ R d 1 F ( x ) .(e)
Revised: ∥ x − x ∗ ∥ 2 , where x ∗ := arg max x ∈ R d 1 F ( x ) .(e)

Original: For any k , there exist constants c 0 , c 1 such that ∥ E [ ξ k |F 1 k ] ∥ 2 ≤ c 20 α k and ∥ E [ ψ k |F k ] ∥ 2 ≤c 21 β k ; there exist constants σ 0 , σ 1 such that E [ ∥ ξ k ∥ 2 |F 1 k ] ≤ σ 20 and E [
Revised: For any k , there exist constants c 0 , c 1 such that ∥ E [ ξ k |F 1 k ] ∥ 2 ≤ c 20 α k and ∥ E [ ψ k |F k ] ∥ 2 ≤c 21 β k ; there exist constants σ 0 , σ 1 such that E [ ∥ ξ k ∥ 2 |F 1 k ] ≤ σ 20 and E [

Original: ∥ ψ k ∥ 2 |F k ] ≤ σ 21 .
Revised: ∥ ψ k ∥ 2 |F k ] ≤ σ 21 .(f)

Original: (f) There exists a constant C F such that F ( x ) ≤ C F .
Revised: There exists a constant C F such that F ( x ) ≤ C F .

Original: We use a −→ b to indicate that a is a sufficient condition of b .
Revised: We use a −→ b to indicate that a is a sufficient condition of b .

Original: Then we have
Revised: Then we have

Original: ( a ) & ( b ) −→ Assumption 1 ; ( a ) − ( c ) −→ Assumption 2 ; ( e ) −→ Assumption 3 ; ( a ) −→ Assumption 4 ; ( d ) −→ Assumption 5 ; ( f ) −→ Assumption 6 .
Revised: ( a ) & ( b ) −→ Assumption 1 ; ( a ) − ( c ) −→ Assumption 2 ; ( e ) −→ Assumption 3 ; ( a ) −→ Assumption 4 ; ( d ) −→ Assumption 5 ; ( f ) −→ Assumption 6 .

Original: The conditions listed above are commonly adopted in the literature [20, 25, 8].
Revised: The conditions listed above are commonly adopted in the literature [19, 23, 8].

Original: It is worth notingthat Lemma 1 does not need the L xy -Lipschitz continuity condition of f ( x, y ) w.r.t.
Revised: It is worth notingthat Lemma 1 does not need the L xy -Lipschitz continuity condition of f ( x, y ) w.r.t.

Original: ( x, y ) .
Revised: ( x, y ) .

Original: This
Revised: This

Original: Lipschitz condition along with the L y -Lipschitz continuity of y ∗ ( x ) , which is implied by the standardconditions in Lemma 1, further leads to the Lipschitz continuity of F ( x ) :
Revised: Lipschitz condition along with the L y -Lipschitz continuity of y ∗ ( x ) , which is implied by the standardconditions in Lemma 1, further leads to the Lipschitz continuity of F ( x ) :

Original: [Equation]
Revised: [Equation]

Original: Although it is rather restrictive, this condition has been used in the previous work when F ( x ) is notstrongly-concave.
Revised: Although it is rather restrictive, this condition has been used in the previous work when F ( x ) isnot strongly-concave.

Original: While our analysis does not need this condition.
Revised: While our analysis does not need this condition.

Original: Lastly, condition ( e ) is guaranteedby using independent samples in the upper and lower level along with [20, Algorithm 3] to obtain agood H yyk , which takes Ω( − log α k ) samples per iteration.
Revised: Lastly, condition ( e ) can beguaranteed by using [19, Algorithm 3] to obtain a good H yyk , which takes Ω( − log α k ) samples periteration.

Original: With Lemma 1, we have the followingcorollary regarding the convergence of (16).
Revised: With Lemma 1, we have the following corollary regarding the convergence of (16).

Original: Corollary 1 (STSA for SBO) Consider the STSA sequences with the update in (16) .
Revised: Corollary 1 (STSA for SBO) Consider the STSA sequences with the update in (16) .

Original: Under theconditions (a)–(e), Theorem 1 holds.
Revised: Under theconditions (a)–(e), Theorem 1 holds.

Original: Under Conditions (a)–(c), (e)-(f), Theorem 2 holds.
Revised: Under Conditions (a)–(c), (e)-(f), Theorem 2 holds.

Original: Remark 2 (Comparison with prior art in SBO)
Revised: Remark 2 (Comparison with prior art in SBO)

Original: When F ( x ) is strongly concave, Corollary 1 implies the sample complexity of O ( ϵ − 1 log ϵ − 1 ) , which improves over the best-known sample complexity O
Revised: When F ( x ) is strongly concave, Corollary 1 implies the sample complexity of O ( ϵ − 1 log ϵ − 1 ) , which improves over the best-known sample complexity O

Original: ( ϵ − 1 .
Revised: ( ϵ − 1 .

Original: 5 log ϵ − 1 ) in [25].
Revised: 5 log ϵ − 1 ) in [23].

Original: Different from [25], we do not need the projection of x k to a compactset.
Revised: Different from [23], we do not need the projection of x k to a compactset.

Original: When F ( x ) is non-concave, corollary 1 suggests a sample complexity of O ( ϵ − 2 log ϵ − 1 ) , whichis the same as the state-of-art complexity established in [8].
Revised: When F ( x ) is non-concave, corollary 1 suggests a sample complexity of O ( ϵ − 2 log ϵ − 1 ) , whichis the same as the state-of-art complexity established in [8].

Original: Corollary 1 improves the result in [8] intwo major aspects: 1) it relaxes the Lipschitz continuity assumption on f ( x, y ) ; and, 2) an alternatingupdate is adopted in [8] to ensure stability, while some applications of SBO only allow simultaneousupdates.
Revised: Corollary 1 improves the result in [8] intwo major aspects: 1) it relaxes the Lipschitz continuity assumption on f ( x, y ) ; and, 2) an alternatingupdate is adopted in [8] to ensure stability, while some applications of SBO only allow simultaneousupdates.

Original: Corollary 1 applies to those cases and thus has a broader range of application.
Revised: Corollary 1 applies to those cases and thus has a broader range of application.

Original: 3.2 Application to advantage actor-critic
Revised: 3.2 Application to advantage actor-critic

Original: RL problems are often modeled as a MDP described by M = {S , A , P , r, γ } , where S is the statespace, A is the action space; P ( s ′ | s, a ) is the probability of transitioning to s ′ ∈S given ( s, a ) ∈S×A ;r ( s, a ) ∈ [0 , 1] is the reward associated with ( s, a ) ; and γ ∈ (0 , 1) is a discount factor.
Revised: RL problems are often modeled as a MDP described by M = {S , A , P , r, γ } , where S is the statespace, A is the action space; P ( s ′ | s, a ) is the probability of transitioning to s ′ ∈S given ( s, a ) ∈S×A ;r ( s, a ) ∈ [0 , 1] is the reward associated with ( s, a ) ; and γ ∈ (0 , 1) is a discount factor.

Original: A policy πmaps S to a distribution over A , and we use π ( a | s ) to denote the probability of choosing a under s .
Revised: A policy πmaps S to a distribution over A , and we use π ( a | s ) to denote the probability of choosing a under s .

Original: Given a policy π , we define the value functions as V π ( s ) :
Revised: Given a policy π , we define the value functions as V π ( s ) :

Original: = E π (cid:104) (cid:80) ∞ t =0 γ t r ( s t , a t ) | s 

Revised: = E π (cid:104) (cid:80) ∞ t =0 γ t r ( s t , a t ) | s

Original: = s (cid:105) , where
Revised: = s (cid:105) , where

Original: E π is taken over the trajectory ( s 0 , a 0 , s 1 , a 1 , . . . )
Revised: E π is taken over the trajectory ( s 0 , a 0 , s 1 , a 1 , . . . )

Original: generated under policy π and transition kernel P .
Revised: generated under policy π and transition kernel P .

Original: With ρ denoting the initial state distribution, the discounted visitation distribution induced by policyπ is defined via d π ( s, a ) =
Revised: With ρ denoting the initial state distribution, the discounted visitation distribution induced by policyπ is defined via d π ( s, a ) =

Original: (1 − γ ) (cid:80) ∞
Revised: (1 − γ ) (cid:80) ∞

Original: t =

Revised: t =

Original: γ t Pr π ( s t =
Revised: γ t Pr π ( s t =

Original: s | s 0 ∼ ρ ) π ( a | s ) .
Revised: s | s 0 ∼ ρ ) π ( a | s ) .

Original: To overcome the difficultyof learning a function, we parameterize the policy with x ∈ R d 0 , and solve
Revised: To overcome the difficultyof learning a function, we parameterize the policy with x ∈ R d 0 , and solve

Original: [Equation]
Revised: [Equation]

Original: To solve for (19), a popular method is the actor-critic (AC) method [31].
Revised: To solve for (19), a popular method is the actor-critic (AC) method [28].

Original: The actor-critic algorithmwith linear critic function is a special case of (5).
Revised: The actor-critic algorithmwith linear critic function is a special case of (5).

Original: Specifically, the critic variable y is updated with
Revised: Specifically, the critic variable y is updated with

Original: [Equation]
Revised: [Equation]

Original: where µ π x is the stationary distribution of the Markov chain induced by π x , ϕ ( s ) ∈ R d 1 is the featurevector encoding state s and the sample ( s k , a k , s ′ k ) is returned by some sampling protocol.
Revised: where µ π x is the stationary distribution of the Markov chain induced by π x , ϕ ( s ) ∈ R d 1 is the featurevector encoding state s and the sample ( s k , a k , s ′ k ) is returned by some sampling protocol.

Original: Undersome regularity conditions, it is known that there exists a unique y ∗ ( x ) such that h ( x, y ∗ ( x ))
Revised: Undersome regularity conditions, it is known that there exists a unique y ∗ ( x ) such that h ( x, y ∗ ( x ))

Original: = 

Revised: =

Original: The actor variable x is then updated with
Revised: The actor variable x is then updated with

Original: [Equation]
Revised: [Equation]

Original: For the AC update in (20) and (21), Assumption 2–4 and 6 or their sufficient conditions have beenexplored in the RL context by previous works [52].
Revised: For the AC update in (20) and (21), Assumption 2–4 and 6 or their sufficient conditions have beenexplored in the RL context by previous works [48].

Original: However, the smoothness of y ∗ ( x ) in Assumption1, which is the key condition leading to a faster convergence rate, has yet been verified.
Revised: However, the smoothness of y ∗ ( x ) in Assumption1, which is the key condition leading to a faster convergence rate, has yet been verified.

Original: With thesame conditions as those adopted in [52], we prove that y ∗ ( x ) is indeed smooth.
Revised: With thesame conditions as those adopted in [48], we prove that y ∗ ( x ) is indeed smooth.

Original: Lemma 2 Consider the AC update in (20) - (21) .
Revised: Lemma 2 Consider the AC update in (20) - (21) .

Original: Under certain standard conditions, y ∗ ( x ) isdifferentiable and there exists a constant L y ′
Revised: Under certain standard conditions, y ∗ ( x ) isdifferentiable and there exists a constant L y ′

Original: such that ∥∇ y ∗ ( x ) − ∇ y ∗ ( x ′ )
Revised: such that ∥∇ y ∗ ( x ) − ∇ y ∗ ( x ′ )

Original: ∥ ≤ L y ′
Revised: ∥ ≤ L y ′

Original: ∥
Revised: ∥

Original: x − x ′ ∥ .
Revised: x − x ′ ∥ .

Original: As a comparison, the above condition was directly assumed in [8], while we provide a formaljustification for Lemma 2 in this work.
Revised: As a comparison, the above condition was directly assumed in [8], while we provide a formaljustification for Lemma 2 in this work.

Original: With detailed verification of all assumptions deferred to
Revised: With detailed verification of all assumptions deferred to

Original: Appendix E, we then directly present the theorem regarding the convergence of AC.
Revised: Appendix D, we then directly present the theorem regarding the convergence of AC.

Original: Theorem 3 (Complexity of AC) Consider the AC update (20) - (21) .
Revised: Theorem 3 (Complexity of AC) Consider the AC update (20) - (21) .

Original: Under some standard conditions specified in Appendix E, Theorem 2 holds.
Revised: Under some standard conditions specified in Appendix D, Theorem 2 holds.

Original: In [8, 52], the projection step is adopted in the y k update to ensure that ∥ y k ∥ < ∞ , ∀ k .
Revised: In [8, 48], the projection step is adopted in the y k update to ensure that ∥ y k ∥ < ∞ , ∀ k .

Original: Since theprojection radius is unknown in practice, adopting the projection is essentially assuming that ∥ y k ∥can be bounded for any k , which is quite strong.
Revised: Since theprojection radius is unknown in practice, adopting the projection is essentially assuming that ∥ y k ∥ isbounded for any k , which is quite strong.

Original: Theorem 3 holds without this projection.
Revised: Theorem 3 holds without this projection.

Original:  Applications to Stochastic Compositional Optimization
Revised:  Applications to Stochastic Compositional Optimization

Original: Define mappings f n : R d n (cid:55)→ R d n +1 for n = 0 , 1 , ..., N with d N +1 = 1 .
Revised: Define mappings f n : R d n (cid:55)→ R d n +1 for n = 0 , 1 , ..., N with d N +1 = 1 .

Original: The multi-level stochasticcompositional problem can be formulated as
Revised: The multi-level stochasticcompositional problem can be formulated as

Original: [Equation]
Revised: [Equation]

Original: where ζ 0 , ζ 1 , . . . , ζ N are random variables.
Revised: where ζ 0 , ζ 1 , . . . , ζ N are random variables.

Original: Here we slightly overload the notation and use f n ( x ; ζ n )to represent the stochastic version of the mapping.
Revised: Here we slightly overload the notation and use f n ( x ; ζ n )to represent the stochastic version of the mapping.

Original: 4.1 Reduction from the generic STSA results
Revised: 4.1 Reduction from the generic STSA results

Original: To solve the problem in (22), a natural scheme is to use the stochastic gradient descent method withthe gradient given by
Revised: To solve the problem in (22), a natural scheme is to use the stochastic gradient descent method withthe gradient given by

Original: [Equation]
Revised: [Equation]

Original: 23) where we use ∇ f n ( f n − 1 ( . . .
Revised: 23) where we use ∇ f n ( f n − 1 ( . . .

Original: f 0 ( x ) . . . ))
Revised: f 0 ( x ) . . . ))

Original: = ∇ f n ( x ) | x = f n − 1 ( ...f 0 ( x ) ... ) .
Revised: = ∇ f n ( x ) | x = f n − 1 ( ...f 0 ( x ) ... ) .

Original: To obtain a stochasticestimator of ∇ F ( x ) , we will need to obtain the stochastic estimators for ∇ f n ( f n − 1 ( ...f 0 ( x ) ... )) foreach n .
Revised: To obtain a stochasticestimator of ∇ F ( x ) , we will need to obtain the stochastic estimators for ∇ f n ( f n − 1 ( ...f 0 ( x ) ... )) foreach n .

Original: For example, when n = 1 , one will need the estimator of ∇ f 1 ( E ζ 

Revised: For example, when n = 1 , one will need the estimator of ∇ f 1 ( E ζ

Original: [ f 0 ( x ; ζ 0 )]) .
Revised: [ f 0 ( x ; ζ 0 )]) .

Original: However,due to the possible non-linearity of ∇ f 1 ( · ) , the natural candidate ∇ f 1 ( f 0 ( x ; ζ 0 )) is not an unbiasedestimator of ∇ f 1 ( E ζ 

Revised: However,due to the possible non-linearity of ∇ f 1 ( · ) , the natural candidate ∇ f 1 ( f 0 ( x ; ζ 0 )) is not an unbiasedestimator of ∇ f 1 ( E ζ

Original: [ f 0 ( x ; ζ 0 )]) .
Revised: [ f 0 ( x ; ζ 0 )]) .

Original: To tackle this issue, a popular method is to directly approximatethe mean E ζ n [ f n ( · ; ζ n )] with a tracking variable y n ∈ R d n for n = 0 , 1 , ..., N , see e.g., [56].
Revised: To tackle this issue, a popular method is to directly approximatethe mean E ζ n [ f n ( · ; ζ n )] with a tracking variable y n ∈ R d n for n = 0 , 1 , ..., N , see e.g., [52].

Original: The update of y n is then a special case of the SA update in (5) with the generic mapping defined as
Revised: The update of y n is then a special case of the SA update in (5) with the generic mapping defined as

Original: [Equation]
Revised: [Equation]

Original: where ζ 0 k , . . .
Revised: where ζ 0 k , . . .

Original: , ζ Nk have the same distributions as that of ζ 0 , . . .
Revised: , ζ Nk have the same distributions as that of ζ 0 , . . .

Original: , ζ N respectively.
Revised: , ζ N respectively.

Original: It is then clear thateach y nk has a unique fixed-point y n, ∗ k = f n − 1 ( y n − 1 k ) , and thus y nk can be viewed as an approximationof f n ( y n − 1 k ) .
Revised: It is then clear thateach y nk has a unique fixed-point y n, ∗ k = f n − 1 ( y n − 1 k ) , and thus y nk can be viewed as an approximationof f n ( y n − 1 k ) .

Original: With these approximations, variable x is updated in the form of (5) by definingv ( [Equation] , y [Equation]
Revised: With these approximations, variable x is updated in the form of (5) by definingv ( x, y 1 , . .

Original: [Equation]
Revised: [Equation]

Original: where ˆ ζ 0 k has the same distribution as that of ζ 0 .
Revised: where ˆ ζ 0 k has the same distribution as that of ζ 0 .

Original: It is clear that when every y nk reaches its fixed-pointy n, ∗ k , it follows from (23) that v ( x k , y 1 , ∗ k , ..., y N, ∗ k )
Revised: It is clear that when every y nk reaches its fixed-pointy n, ∗ k , it follows from (23) that v ( x k , y 1 , ∗ k , ..., y N, ∗ k )

Original: =
Revised: =

Original: ∇ F ( x k ) , which indicates that the expectedupdate direction of x k in (25) is ∇ F ( x k ) .
Revised: ∇ F ( x k ) , which indicates that the expectedupdate direction of x k in (25) is ∇ F ( x k ) .

Original: Due to space limitation, we directly give the following corollary on the convergence of the SCOupdate with the assumptions and proof deferred to Appendix F.
Revised: Due to space limitation, we directly give the following corollary on the convergence of the SCOupdate with the assumptions and proof deferred to Appendix E.

Original: Corollary 2 (STSA for multi-level SCO) Consider the STSA sequences generated with update(24) - (25) .
Revised: Corollary 2 (STSA for multi-level SCO) Consider the STSA sequences generated with update(24) - (25) .

Original: Under the standard conditions specified in Appendix F, Theorem 1 or 2 holds.
Revised: Under the standard conditions specified in Appendix E, Theorem 1 or 2 holds.

Original: Remark 3 (Comparison with prior art in SCO)
Revised: Remark 3 (Comparison with prior art in SCO)

Original: Corollary 2 establishes the sample complexity of
Revised: Corollary 2 establishes the sample complexity of

Original: O ( ϵ − 1 ) for the strongly monotone case and the complexity of O ( ϵ − 2 ) for the non-monotone case,which are both independent of N .
Revised: O ( ϵ − 1 ) for the strongly monotone case and the complexity of O ( ϵ − 2 ) for the non-monotone case,which are both independent of N .

Original: This improves over the O ( ϵ − N +54 ) complexity for the stronglyconcave case and the O ( ϵ − N +84 ) complexity for the non-concave case shown in [56].
Revised: This improves over the O ( ϵ − N +54 ) complexity for the stronglyconcave case and the O ( ϵ − N +84 ) complexity for the non-concave case shown in [52].

Original: There are otherworks that establish the same complexity as that in Corollary 2, but they require modification to thebasic SA update (24) and (25) to achieve acceleration; see e.g., [7, 1, 44].
Revised: There are otherworks that establish the same complexity as that in Corollary 2, but they require modification to thebasic SA update (24) and (25) to achieve acceleration; see e.g., [7, 1, 40].

Original: 4.2 Application to model-agnostic meta policy gradient
Revised: 4.2 Application to model-agnostic meta policy gradient

Original: Consider a set of MDPs {M i } Mi =1 with M i = {S , A , P i , r i , γ } .
Revised: Consider a set of MDPs {M i } Mi =1 with M i = {S , A , P i , r i , γ } .

Original: The MDPs model a set of RLtasks that share the same state-action space while having different transition kernels P i and rewardfunctions r i .
Revised: The MDPs model a set of RLtasks that share the same state-action space while having different transition kernels P i and rewardfunctions r i .

Original: To better compare with the previous work [14], we consider the finite-horizon objectivefunction with the policy π parametrized by x ∈ R d 0 : F i ( x ) :=
Revised: To better compare with the previous work [14], we consider the finite-horizon objectivefunction with the policy π parametrized by x ∈ R d 0 : F i ( x ) :=

Original: E ζ ∼ π x (cid:104) (cid:80)
Revised: E ζ ∼ π x (cid:104) (cid:80)

Original: Ht =
Revised: Ht =

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: γ

Original: [Equation]
Revised: [Equation]

Revised: ( s t , a

Original: [Equation]
Revised: [Equation]

Original: ) | ρ
Revised: ) | ρ

Original: [Equation]
Revised: [Equation]

Original: , P i (cid:105) ,where H ∈ N + is the horizon, and E ζ ∼ π x is taken over the trajectory ζ :=
Revised: , P i (cid:105) ,where H ∈ N + is the horizon, and E ζ ∼ π x is taken over the trajectory ζ :=

Original: ( s 0 , a 0 , s 1 , a 1 , ..., s H , a H )generated under policy π x , initial distribution
Revised: ( s 0 , a 0 , s 1 , a 1 , ..., s H , a H )generated under policy π x , initial distribution

Original: ρ i and transition kernels P i .
Revised: ρ i and transition kernels P i .

Original: The goal of MAMPG is to find an initial policy π x that can achieve good performance in new tasksby performing a few policy gradient steps [15, 14].
Revised: The goal of MAMPG is to find an initial policy π x that can achieve good performance in new tasksby performing a few policy gradient steps [15, 14].

Original: In the case where N steps of gradient update areperformed, the problem of finding an initial policy parameter x can be formulated as
Revised: In the case where N steps of gradient update areperformed, the problem of finding an initial policy parameter x can be formulated as

Original: [Equation]
Revised: [Equation]

Original: (26) where x is the shared initial policy parameter, i.e. ˜ x 0 i
Revised: (26) where x is the shared initial policy parameter, i.e. ˜ x 0 i

Original: = x for any task i and ˜ x Ni ( x ) is the parameterafter running N steps of gradient ascent with respect to F i starting from x .
Revised: = x for any task i and ˜ x Ni ( x ) is the parameterafter running N steps of gradient ascent with respect to F i starting from x .

Original: Solving (26) with SCO method.
Revised: Solving (26) with SCO method.

Original: The MAMPG problem in (26) can be solved by the stochastic compositional optimization method introduced before.
Revised: The MAMPG problem in (26) can be solved by the stochastic compositional optimization method introduced before.

Original: In order to get ∇ F ( x ) , one will need ∇ F i
Revised: In order to get ∇ F ( x ) , one will need ∇ F i

Original: (˜ x Ni ( x ))for each task i .
Revised: (˜ x Ni ( x ))for each task i .

Original: Observe that F i (˜ x Ni ( x )) can be written as a compositional function:
Revised: Observe that F i (˜ x Ni ( x )) can be written as a compositional function:

Original: F i (
Revised: F i (

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: = f Ni (
Revised: = f Ni (

Original: [Equation]
Revised: [Equation]

Original: i ( . . .
Revised: i ( . . .

Original: f 

Revised: f

Original: i ( x ) . . . )) with f ni ( x ) :
Revised: i ( x ) . . . )) with f ni ( x ) :

Original: = x + η ∇ F i ( x ) , n = 0 , ..., N − 1 , (27) where f Ni ( x ) =
Revised: = x + η ∇ F i ( x ) , n = 0 , ..., N − 1 , (27) where f Ni ( x ) =

Original: F i ( x ) .
Revised: F i ( x ) .

Original: In order to approximate ∇ F i (˜ x Ni ( x )) , we can follow the discussion in
Revised: In order to approximate ∇ F i (˜ x Ni ( x )) , we can follow the discussion in

Original: Section 4 and introduce tracking variables y ni ∈ R d 0 for n ∈ [ N ] which are updated as follows
Revised: Section 4 and introduce tracking variables y ni ∈ R d 0 for n ∈ [ N ] which are updated as follows

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where we define f ni ( · ; ζ ) as a stochastic approximation of f ni ( · ) with random trajectory ζ .
Revised: where we define f ni ( · ; ζ ) as a stochastic approximation of f ni ( · ) with random trajectory ζ .

Original: Then weestimate ∇ F i (˜ x Ni ( x )) by ˆ ∇ F i,k :=
Revised: Then weestimate ∇ F i (˜ x Ni ( x )) by ˆ ∇ F i,k :=

Original: ∇ f 0 i ( x ; ˆ ζ 0 k,i ) ∇ f 1 i ( y 1 k,i ; ζ 1 k,i ) · · · ∇ f Ni ( y Nk,i ; ζ Nk,i ) .
Revised: ∇ f 0 i ( x ; ˆ ζ 0 k,i ) ∇ f 1 i ( y 1 k,i ; ζ 1 k,i ) · · · ∇ f Ni ( y Nk,i ; ζ Nk,i ) .

Original: To obtain an estimation of ∇ F ( x ) , we need to obtain ˆ ∇ F i,k for each i ∈ { 1 , 2 , ..., M } , that is, we need to do update(28) for each i .
Revised: To obtain an estimation of ∇ F ( x ) , we need to obtain ˆ ∇ F i,k for each i ∈ { 1 , 2 , ..., M } , that is, we need to do update(28) for each i .

Original: With { ˆ ∇ F i,k } Mi =1 , we update the initial policy as x k +

Revised: With { ˆ ∇ F i,k } Mi =1 , we update the initial policy as x k +

Original: = x k + α k 1 M (cid:80) Mi
Revised: = x k + α k 1 M (cid:80) Mi

Original: =1 ˆ ∇
Revised: =1 ˆ ∇

Original: F i,k .
Revised: F i,k .

Original: Reduction from the generic results.
Revised: Reduction from the generic results.

Original: Let y n ∈ R d n M be a concatenation of y n i for i ∈ { 1 , 2 , ..., M } .
Revised: Let y n ∈ R d n M be a concatenation of y n i for i ∈ { 1 , 2 , ..., M } .

Original: With ζ nk := { ζ nk,i } Mi =1 , let f n ( y nk ; ζ nk ) be a concatenation of f ni ( y nk,i ; ζ nk,i ) for i ∈ { 1 , 2 , ..., M } .
Revised: With ζ nk := { ζ nk,i } Mi =1 , let f n ( y nk ; ζ nk ) be a concatenation of f ni ( y nk,i ; ζ nk,i ) for i ∈ { 1 , 2 , ..., M } .

Original: Thenwe can write the tracking variable update of all tasks jointly in the form of (5a), that is
Revised: Thenwe can write the tracking variable update of all tasks jointly in the form of (5a), that is

Original: [Equation]
Revised: [Equation]

Original: The initial policy update is a special case of (5b), that is
Revised: The initial policy update is a special case of (5b), that is

Original: [Equation]
Revised: [Equation]

Original: Due to space limitation, we directly give the result below and defer the proof to Appendix G.
Revised: Due to space limitation, we directly give the result below and defer the proof to Appendix F.

Original: Theorem 4 (Complexity of MAMPG) Consider the STSA sequences generated by the MAMPGupdate in (29) and (30) .
Revised: Theorem 4 (Complexity of MAMPG) Consider the STSA sequences generated by the MAMPGupdate in (29) and (30) .

Original: Under some standard conditions specified in Appendix G, Theorem 2 holds.
Revised: Under some standard conditions specified in Appendix F, Theorem 2 holds.

Original: Theorem 4 implies a sample complexity of O ( ϵ − 2 ) to achieve the ϵ -stationary initial policy, whichimproves over the O ( ϵ − 4 ) sample complexity in [14].
Revised: Theorem 4 implies a sample complexity of O ( ϵ − 2 ) to achieve the ϵ -stationary initial policy, whichimproves over the O ( ϵ − 4 ) sample complexity in [14].

Original: Moreover, Theorem 4 holds for any N ≥ 1 in(26), while the method in
Revised: Moreover, Theorem 4 holds for any N ≥ 1 in(26), while the method in

Original: Checklist
Revised: Checklist

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.1.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] This work only considers the strongly-monotone increments for the follower sequences, but it will be interesting to also consider the monotone increments with non-unique fixed point y ∗ . Right now
Revised: [TODO]

Original: the generic result in this work can only be applied to the unconstrained stochastic

Original: optimization, while it will be also interesting to consider whether it is possible

Original: to

Original: establish similar result that is applicable to the constrained stochastic optimization

Original: problems.

Original: We will add discussion on the limitations in the revision.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: of GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets did you cite the creators
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N
Revised: [N/A]

Original: A] (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/ A](e)
Revised: [N/A]

Original:  Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]5.
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A](b)
Revised: [N/A]

Original:  Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A](c)

Original: Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]

Revised: [Table]