F INITE - TIME ANALYSIS OF SINGLE - TIMESCALE ACTOR - CRITIC ON LINEAR QUADRATIC REGULATOR Anonymous authors
Paper under double-blind review
A BSTRACT
Actor-critic (AC) methods have achieved state-of-the-art performance in many challenging tasks.
However, their convergence in most practical applications are still poorly understood.
Existing works mostly consider the uncommon doubleloop or two-timescale stepsize variants for the ease of analysis.
We investigate the practical yet more challenging single-sample single-timescale natural AC for solving the canonical linear quadratic regulator problem.
Specifically, the actor and the critic update only once with a single sample in each iteration using proportional stepsizes.
We prove that the single-sample single-timescale natural AC(NAC) can attain an ϵ -optimal solution with a sample complexity of O ( ϵ − 2 ) , which elucidates on the practical efficiency of single-sample single-timescale NAC.
We develop a novel analysis framework that directly bounds the whole interconnected iteration system without the conservative decoupling commonly adopted in previous analysis of AC and NAC.
Our work presents the first finite-time analysis of single-sample single-timescale NAC with a global optimality guarantee.
I NTRODUCTION
Actor-critic (AC) methods achieved substantial success in solving many difficult reinforcement learning (RL) problems (LeCun et al., 2015; Mnih et al., 2016; Silver et al., 2017).
In addition to a policy update, AC methods employ a parallel critic update to bootstrap the Q-value for policy gradient estimation, which often enjoys reduced variance and fast convergence in training.
Despite the empirical success, theoretical analysis of AC in the most practical form remains challenging.
Most existing works focus on either the double-loop setting or the two-timescale setting, both of which are uncommon in practical implementations.
In double-loop AC, the actor is updated in the outer loop only after the critic takes sufficiently many steps to have an accurate estimation of the Q-value in the inner loop (Yang et al., 2019; Kumar et al., 2019; Wang et al., 2019).
Hence, the convergence of critic is decoupled from that of the actor.
The analysis is separated into a policy evaluation sub-problem in the inner loop and a perturbed gradient descent in the outer loop.
In two-timescale AC, the actor and the critic are updated simultaneously in each iteration using stepsizes of different timescales.
The actor stepsize (denotes by α t ) is typically smaller than that of the critic (denotes by β t ), with their ratio goes to zero as the iteration number goes to infinity (i.e., lim t →∞ α t /β t = 0 ).
The two-timescale allows the critic to approximate the correct Q-value in an asymptotic way.
This design essentially decouples the analysis of the actor and the critic.
The aforementioned AC variants are considered mainly for the ease of analysis.
In practice, the single-timescale AC, where the actor and the critic are updated simultaneously using constantly proportional stepsizes (i.e., with α t /β t = c α > 0 ), is more favorable due to its simplicity of implementation and empirical sample efficiency (Schulman et al., 2015; Mnih et al., 2016).
However, its analysis is significantly more difficult than the other variants.
To understand its finite-time convergence, some recent works (Fu et al., 2020; Zhou & Lu, 2022) consider multi-sample variants of single-timescale AC, where the critics are updated by the least square temporal difference (LSTD) estimator rather than the TD(0) update.
The idea is still to obtain an accurate policy gradient estimation at each iteration by using sufficient samples (LSTD), and then follows the common perturbed gradient analysis to guarantee the convergence of the actor, decoupling the convergence analysis of the actor and the critic.
In addition to the multi-sample settings, there are few attempts that analyzed the single-sample single-timescale AC(NAC), and they only attest local convergence (Chen et al.,
; Olshevsky & Gharesifard, 2022).
Besides, (Olshevsky & Gharesifard, 2022) only considers the simple tabular case.
We attempt to answer the more general yet more challenging question:
Can the single-sample single-timescale AC(NAC) find a global optimal policy, especially on the general unbounded continuous state-action space with unbounded reward?
To this end, we make the first step to consider the classic Linear Quadratic Regulation (LQR), a fundamental continuous state-action space control problem that are commonly employed to study the performance and the limits of RL algorithms (Fazel et al., 2018;
Yang et al., 2019; Tu & Recht, 2018;
Duan et al., 2021).
In particular, under the time-average cost, the single-sample single-timescale AC(NAC) algorithm for solving LQR consists of three parallel updates in each iteration: the cost estimator, the critic, and the actor.
Unlike the aforementioned double-loop, two-timescale, or multisample structures, there is no specialized design in single-sample single-timescale AC(NAC) that facilitates a decoupled analysis of its three interconnected updates.
In fact, it is both conservative and difficult to bound the three iterations separately.
Moreover, the existing perturbed gradient analysis can no longer be applied to establish the convergence of the actor either.
To tackle these challenges in analysis, we instead propose a novel framework to directly bound the overall interconnected iteration system altogether, without resorting to conservative decoupled analysis.
In particular, despite the inaccurate estimation in all three updates, we prove the estimation errors diminish to zero if the (constant) ratio of the stepsizes between the actor and the critic is below a threshold.
The identified threshold provides new insights into the practical choices of the stepsizes for single-timescale AC.
Overall, our contributions are summarized as follows:
• Our work furthers the theoretical understanding of AC(NAC) in its most practical form.
We for the first time show that the single-sample single-timescale NAC can provably find the ϵ -accurate global optimum with a sample complexity of O ( ϵ − 2 ) for tasks with unbounded continuous state-action space.
The previous works consider either specialized algorithm variants (Fu et al., 2020;
Zhou & Lu, 2022), or more restricted settings with only local convergence guarantee (Chen et al., 2021; Olshevsky & Gharesifard, 2022).
• We also contribute to the work of RL on continuous control task.
It is novel that even with actor updated by a roughly estimated gradient, the single-sample single-timescale NAC algorithm can still find the global optimal policy for LQR, under general assumptions.
Compared with all other modelfree RL algorithms for solving LQR (see related work 1.1), our work is the first one adopting the simplest single-sample single-loop structure, which may serve as the first step towards understanding the limits of AC(NAC) methods on continuous control task.
In addition, compared with the stateof-the-art double-loop AC for solving LQR (Yang et al., 2019), we improve the sample complexity from O ( ϵ − 5 ) to O
( ϵ − 2 ) .
We also show the algorithm is much more sample-efficient empirically compared to a few classic works in Section 5, which unveils the practical wisdom of AC(NAC) algorithm.
Technically, we provide a new proof framework that can establish the finite-time convergence for single-timescale AC.
In the finite-time analysis of double-loop AC (Yang et al., 2019) and twotimescale AC (Wu et al., 2020), the previous techniques hinge on decoupling the analysis of actor and critic, establishing the convergence of critic first and then the convergence of actor consequently.
The novelty of our proof framework is that we formulate the estimation errors of the time-average cost, the critic, and the natural policy gradient into an interconnected iteration system and establish the convergence for them simultaneously rather than separately.
This proof framework may provide new insights for finite-time analysis of other single-timescale algorithms.
1. R ELATED W ORK
In this section we review the existing works that are most relevant to ours.
Actor-Critic methods.
The first AC algorithm was proposed by Konda & Tsitsiklis (1999).
Kakade (2001) extended it to the natural AC algorithm.
The asymptotic convergence of AC algorithms has been well established in Kakade (2001); Bhatnagar et al.
; Castro & Meir (2010); Zhang et al.
Many recent works focused on the finite-time convergence of AC methods.
Under the double-loop setting, Yang et al.
(2019) established the global convergence of AC methods for solving LQR.
Wang et al.
(2019) studied the global convergence of AC methods with both the actor and the critic being parameterized by neural networks.
Kumar et al.
(2019) studied the finite-time local convergence of a few AC variants with linear function approximation.
Under the two-timescale
AC setting, Wu et al.
Xu et al.
(2020) established the finite-time convergence to a stationary point at a sample complexity of O ( ϵ − 2 . 5 ) .
Under the single-timescale setting, all the related works (Fu et al., 2020; Chen et al., 2021; Zhou & Lu, 2022; Olshevsky & Gharesifard, 2022) have been reviewed in the Introduction.
RL algorithms for LQR.
RL algorithms in the context of LQR have seen increased interest in the recent years.
These works can be mainly divided into two categories: model-based methods (Dean et al., 2018;
Mania et al., 2019; Cohen et al., 2019; Dean et al., 2020) and model-free methods.
Our main interest lies in the model-free methods.
Notably, Fazel et al.
(2018) established the first global convergence result for LQR under the policy gradient method using zeroth-order optimization.
Krauth et al.
(2019) studied the convergence and sample complexity of the LSTD policy iteration method under the LQR setting.
On the subject of adopting AC to solve LQR, Yang et al.
(2019) provided the first finite-time analysis with convergence guarantee and sample complexity under the double-loop setting.
Zhou & Lu (2022) considered the multi-sample (LSTD) and singletimescale setting.
For the more practical yet challenging single-sample single-timescale AC, there is no such theoretical guarantee so far, which is the focus of this paper.
Notation.
Without other specification, for two sequences { x n } and { y n } , we write x n =
O ( y n ) if there exists an constant C such that x n ≤ Cy n .
We use O ( · ) to further hide logarithm factors.
For any symmetric matrix M ∈ R n × n , let svec ( M ) ∈ R n ( n +1) /
2 denote the vectorization of the upper triangular part of M and smat ( · ) denote its inverse such that smat ( svec ( M ))
=
M .
Finally, we denote by A ⊗ s B the symmetric Kronecker product of two matrices A and B .
P RELIMINARIES
In this section, we introduce the AC algorithm and provide the theoretical background of LQR.
2. A CTOR -C RITIC A LGORITHMS
We consider the reinforcement learning for the standard Markov Decision Process (MDP) defined by ( X , U , P , c ) , where X is the state space, U is the action space, P ( x t +1 | x t , u t ) denotes the transition kernel that the agent transits to state x t
+1 after taking action u t at current state x t , and c ( x t , u t ) is the running cost.
A policy π θ ( u | x ) parameterized by θ is defined as a mapping from a given state to a probability distribution over actions.
In this paper, we aim to find a policy π θ that minimizes the infinite-horizon time-average cost, which is given by
[Equation]
where ρ θ denotes the stationary state distribution generated by policy π θ .
In the time-average cost setting, the state-action value (Q-value) of policy π θ is defined as
[Equation]
which describes the accumulated differences between running costs and average cost for selecting u in state x and thereafter following policy π θ (Sutton & Barto, 2018).
Based on this definition, we can use the policy gradient theorem (Sutton et al., 1999) to express the gradient of J ( θ ) with respect to θ as
[Equation]
One can also choose to update the policy using the natural policy gradient (Kakade, 2001), which is given by
[Equation]
where
[Equation]
Optimizing J ( θ ) in (1) with (3) requires evaluating the Q-value of the current policy π θ , which is usually unknown.
AC estimates both the Q-value and the policy.
The critic update approximates
Q-value towards the actual value of the current policy π θ using temporal difference (TD) learning (Sutton & Barto, 2018).
The actor improves the policy to reduce the time-average cost J ( θ ) via gradient descent.
Note that the AC with natural policy gradient is also known as natural AC.
2. N ATURAL A CTOR -C RITIC FOR L INEAR Q UADRATIC R EGULATOR
In this paper, we aim to demystify the convergence property of natural AC by focusing on the infinite-horizon time-average linear quadratic regulator (LQR) problem:
[Equation]
subject
[Equation]
where x t ∈ R d is a state and u t ∈ R k is a action; A ∈ R d × d and B ∈ R d × k are system matrices;
Q ∈ S d × d and R ∈ S
k × k are performance matrices; ϵ t ∼ N (0 , D 0 ) are i.i.d Gaussian random variables with D 0 > 0 .
From the optimal control theory (Anderson & Moore, 2007), the optimal policy of (4) is a linear feedback of the state
[Equation]
(6) where K ∗ ∈
R k × d is the optimal policy which can be uniquely found by solving an Algebraic
Riccati Equation (ARE) (Anderson & Moore, 2007) depending on A , B , Q , R .
This means that finding K ⋆ using ARE relies on the complete model knowledge.
In the sequel, we pursue finding the optimal policy in a model-free way by using the natural AC method, without knowing or estimating A, B, Q, R .
The structure of the optimal policy in (6) allows us to reformulate (4) as a static optimization problem over all feasible policy matrix K ∈ R k × d .
To encourage exploration, we parameterize the policy as
[Equation]
where σ > 0 is the standard deviation of the exploration noise.
In other words, given a state x t , the agent will take an action u t according to u t = − Kx t + σζ t , where ζ t ∼ N (0 , I k ) .
As a consequence, the closed-loop form of system (5) under policy (7) is given by
[Equation]
where ξ t =
ϵ t + σBζ t ∼ N (0 , D σ ) with D σ = D 0 + σ 2 BB ⊤ .
Note that optimizing over the set of stochastic policies (7) will lead to the same optimal K ∗ .
The set K of all stabilizing policies is given by
[Equation]
where ρ ( · ) denotes the spectral radius.
It is well known that if K ∈ K , the Markov chain in (8) yields a stationary state distribution N (0 , D K ) , where D K satisfies the following Lyapunov equation
[Equation]
Similarly, we define P K as the unique positive definite solution to
[Equation]
Based on D K and P K , the following lemma characterizes J ( K ) and its gradient ∇ K J ( K ) .
Lemma 2.1.
(Yang et al., 2019)
For any K ∈ K , the time-average cost J ( K ) and its gradient ∇ K J ( K ) take the following forms
[Equation]
where E K :
=
( R + B ⊤ P K B ) K − B ⊤ P K A .
Then, the natural gradient of J ( K ) can be calculated as (Fazel et al., 2018;
Yang et al., 2019)
[Equation] [Equation]
which eliminates the burden of estimating D K .
Note that we omit the constant coefficient since it can be absorbed by the stepsize.
Calculating the natural gradient ∇ NK J ( K ) requires estimating P K , which depends on A, B, Q, R .
To estimate the gradient without the knowledge of the model, we instead directly utilize the Q-value.
Lemma 2.2.
(Bradtke et al., 1994; Yang et al., 2019)
For any K ∈ K , the Q-value Q K ( x, u ) takes the following form
[Equation]
where
[Equation]
Clearly, if we can estimate Ω K , then E k in (13) can be readily estimated by using Ω 21 K and Ω 22 K .
S INGLE - SAMPLE S INGLE - TIMESCALE N ATURAL A CTOR -C RITIC
In this section, we describe the single-sample single-timescale natural AC algorithm for solving
LQR.
In view of the structure of the Q-value given in (14), we define the following feature function
[Equation]
Then, we can parameterize the Q-estimator (critic) by
[Equation]
Using the TD(0) learning, the critic update follows by
[Equation]
where β t is the stepsize of the critic and K denotes the policy under which the state-action pairs are sampled.
Note that the constant b is not required for updating the linear coefficient ω .
Taking the expectation of ω t +1 in (16) with respect to the stationary distribution, conditioned on ω t , the expected subsequent critic can be written as
[Equation]
where
[Equation]
Note that for ease of exposition, we denote ( x ′ , u ′ ) as the next state-action pair after ( x, u ) and abbreviate E x ∼ ρ K ,u ∼ π K ( ·| x ) as E ( x,u ) .
Given a policy π K , it is not hard to show that if the update in (17) has converged to some limiting point ω ∗ K , i.e., lim t →∞ ω t = ω ∗ K , ω ∗ K must be the solution of A K ω
=
b K .
Proposition 3.1.
Suppose K ∈ K .
Then the matrix A K defined in (18) is invertible and A K ω
=
b K has a unique solution ω ∗ K that satisfies
[Equation]
19) where Ω K is defined in (15).
Combining (13), (15), and (19), we can express the natural gradient of J ( K ) using ω ∗ K :
[Equation]
This allows us to estimate the natural policy gradient using the critic parameters ω t , and then update the actor in a model-free manner
[Equation]
where α t is the actor stepsize and
[Equation]
is the natural gradient estimation depending on ω t :
[Equation]
[Equation]
Furthermore, we introduce a cost estimator η t to estimate the time-average cost J ( K t ) .
Combining the critic update (16) and the actor update (20), the single-sample single-timescale natural AC for solving LQR is listed below.
error δ t =
c t − η t + ϕ ( x ′ t , u ′ t )
⊤ ω t − ϕ ( x t , u t )
⊤ ω t t + γ t
c t − η t )
1 =
Π ¯ ω ( ω t + β t δ t ϕ ( x t , u t )) 9:
Actor update: K t +1 =
K t − α t ( smat ( ω t ) 22 K t − smat ( ω t ) 21 ) 10: end for
Note that “single-sample” refers to the fact that only one sample is used to update the critic per actor step.
Line 3 of Algorithm 1 samples from the stationary distribution induced by the policy π K t , which is a mild requirement in the analysis of uniformly ergodic Markov chain, such as in the LQR problem (Yang et al., 2019).
It is only made to simplify the theoretical analysis.
Indeed, as shown in
Tu & Recht (2018), when K ∈ K , (8) is geometrically β -mixing and thus its distribution converges to the stationary distribution exponentially.
In practice, one can run the Markov chain in (8) a sufficient number of steps and sample one state from the last step.
In addition, “single-timescale” refers to the fact that the stepsizes for the critic and the actor updates are constantly proportional.
Since the update of the critic parameter in (16) requires the time-average cost J ( K t ) , Line 7 provides an estimation of it.
Besides, on top of (16), we additionally introduce a projection ( Π ¯ ω ) in Line 8 to keep the critic norm-bounded, which is common in the literature (Wu et al., 2020; Yang et al., 2019;
Xu et al., 2020).
In our analysis, the projection is relaxed using its nonexpansive property.
M AIN T HEORY
In this section, we establish the global convergence and analyze the finite-time performance of Algorithm 1.
All the proofs can be found in the Appendix A.
Before preceding, we make the following standard assumptions.
Assumption 4.1.
There exists a constant ¯ K > 0 such that ∥ K t ∥ ≤ ¯ K for all t .
The above assumes the uniform boundedness of the actor parameter (Konda & Tsitsiklis, 1999;
Karmakar & Bhatnagar, 2018; Barakat et al., 2022; Zhou & Lu, 2022).
As can be seen from our proof, it is only made to guarantee the boundedness of the feature functions, which is a standard assumption in the literature of analyzing AC with linear function approximation (Xu et al., 2020;
Wu et al., 2020; Zhou & Lu, 2022).
Assumption 4.2.
There exists a constant ρ ∈ (0 , 1) such that ρ ( A − BK t ) ≤ ρ for all t .
Assumption 4.2 is made to ensure the stability of the closed loop systems induced in each iteration and thus ensure the existence of the stationary distribution corresponding to policy π K t .
In the single-sample case, the estimation of the natural gradient of J ( K t ) can be highly noisy and biased.
In general, it is difficult to obtain a theoretical guarantee for this condition.
Nevertheless, we will present numerical examples to support this assumption.
Moreover, the assumption for the existence of stationary distribution is common and has been widely used in Chen et al.
; Zhou & Lu (2022); Olshevsky & Gharesifard (2022).
Under these two assumptions, we can now prove the convergence of Algorithm 1, which consists of three estimators: η t , ω t , and K t .
Theorem 4.3.
Suppose that Assumptions 4.1 and 4.2 hold and choose α t =
c α √ 1+ t , β t = γ
[Equation]
where c α is a small positive constant.
With probability at least 1 − 10 − 10 , we have
[Equation]
The theorem shows that the cost estimator, the critic, and the actor all converge at a sub-linear rate of O ( T − 12 ) .
Correspondingly, to obtain an ϵ -optimal policy, the required sample complexity is
O ( ϵ − 2 ) .
This order is consistent with the existing results on single-timescale AC (Fu et al., 2020;
Chen et al., 2021; Olshevsky & Gharesifard, 2022).
Nevertheless, our result is the first finite-time analysis of the single-sample single-timescale AC with a global optimality guarantee.
4. P ROOF S KETCH
The main challenge in the finite-time analysis lies in that the estimation errors of the time-average cost, the critic, and the natural policy gradient are strongly coupled.
To overcome this issue, we view the propagation of these errors as an interconnected system and analyze them comprehensively.
To see the merit of our analysis framework, we sketch the main proof steps of Theorem 4.3 in the following.
The supporting propositions and theorems mentioned below can be found in the
Appendix.
We define three measures A ( T ) , B ( T ) , C ( T ) which denote the average values of the cost estimation error, the critic error, and the square norm of the natural policy gradient, respectively:
[Equation]
where y t :=
η t − J ( K t ) is the cost estimation error and z t :=
ω t − ω ∗ t with ω ∗ t := ω ∗ K t is the critic error.
Note that E K t = ∇ NK t J ( K t ) is the natural policy gradient according to (13).
We first derive implicit (coupled) upper bounds for the cost estimation error y t , the critic error z t , and the natural gradient E K t , respectively.
After that, we solve an interconnected system of inequalities in terms of A ( T ) , B ( T ) , C ( T ) to establish the finite-time convergence.
Step 1: Cost estimation error analysis.
From the cost estimator update rule (Line 7 of Algorithm 1), we decompose the cost estimation error into:
[Equation]
The second term on the right hand side of (23) is a noise term introduced by random sampling of the state-action pairs, which reduces to 0 after taking the expectations.
The third term is the variation of the moving targets J ( K t ) tracked by cost estimator.
It is bounded by y t , z t , E K t utilizing the Lipschitz continuity of J ( K t ) (Proposition A.6), the actor update rule (21), and the CauchySchwartz inequality.
The last term reflects the variance in cost estimation, which is controlled by a high probability bound of c t (Proposition A.4).
Step 2: Critic error analysis .
By the critic update rule (Line 8 of Algorithm 1), we decompose the squared error by (neglecting the projection for the time being)
[Equation]
where the definitions of h, ¯ h, ∆ h, Λ , and O t can be found in (27) in the Appendix.
The second term on the right hand side of (24) is bounded by − µ ∥
z t ∥ 2 , where µ is a lower bound of σ min ( A K t ) proved in Proposition A.8.
The third term is a random noise introduced by sampling, which reduces to 0 after taking expectation.
The fourth term is caused by inaccurate cost and critic estimations, which can be bounded by the norm of y t and z t .
The fifth term tracks the difference between the drifting critic targets.
We control it by the Lipschitz continuity of the critic target established in Proposition A.9.
The last term reflects the variances of various estimations, which is bounded by the diminishing β t .
Step 3: Natural gradient norm analysis .
From the actor update rule (Line 9 of Algorithm 1) and the almost smoothness property of LQR (Lemma A.11), we derive
[Equation]
where ˆ E K t denotes the estimation of the natural gradient E K t .
The first term on the left hand side of (25) can be considered as the scaled square norm of the natural gradient.
The first term on the right hand side compares the actor’s performances between consecutive updates, which is bounded via Abel summation by parts.
The second term evaluates the inaccurate natural gradient estimation, which is then bounded by the critic error z t and the natural gradient E K t .
The last term can be considered as the variance of the perturbed natural gradient update, which is controlled by the diminishing stepsize.
Step 4:
Interconnected iteration system analysis.
Taking the expectation and summing (23), (24), and (25) from 0 to T − 1 , respectively, we obtain the following interconnected iteration system in terms of A ( T ) , B ( T ) , C ( T ) :
[Equation]
[Equation]
where b, d, e, g are positive constants.
By solving the above system of inequalities, we further prove that if bd 2 + bd 2 g 2 + 2 eg 2 < 1 , then A ( T ) , B ( T ) , C ( T ) converge at a rate of O ( T − 12 ) .
This condition can be easily satisfied by choosing the stepsize ratio c α to be smaller than a threshold defined in (52).
Step 5: Global convergence analysis.
To prove the global optimality, we utilize the gradient domination condition of LQR (Lemma A.12),
[Equation]
This property shows that the actor performance error can be bounded by the norm of the natural gradient (that is, Tr ( E ⊤ K E K ) ).
Since we have proved the average natural gradient norm C ( T ) converges to zero, summation over both sides of the above inequality yields
[Equation]
which is the convergence of the actor performance error.
We thus complete the proof of Theorem 4.3.
E XPERIMENTS
We provide two numerical examples to illustrate our theoretical results.
The first example is a twodimensional system and the second example is a four-dimensional system (See Appendix D for the system matrices and other settings).
The performance of Algorithm 1 is shown in Figure 1, where the left column corresponds to the first example and the right column to the second example.
The solid lines plot the mean values and the shaded regions denote the 95% confidence interval overindependent runs.
Consistent with our theorem, Figure 1(a) shows that the cost estimation error, the critic error, and the actor performance error all diminish at a rate of at least T − 12 .
The convergence also suggests that the intermediate closed-loop linear systems during iteration are uniformly stable.
We also compare Algorithm 1 with the zeroth-order method (Fazel et al., 2018) and the double-loop
AC algorithm proposed in (Yang et al., 2019)
(listed in Algorithm 2 and Algorithm 3, respectively, in Appendix D).
We plotted the relative errors of the actor parameters for all three methods in Figure 1(b).
Algorithm 1 demonstrates superior sample-efficiency compared to the other two algorithms, which is well supported by our theoretical analysis. [Equation]  [Equation]

[Equation]

[Equation]  [Equation] U &RVWHVWLPDWLRQHUURU
[Equation]
$FWRUSHUIRUPDQFHHUURU 
[Equation]

[Equation]

[Equation]

[Equation]
   
[Equation]

[Equation]

[Equation]
U &RVWHVWLPDWLRQHUURU &ULWLFHUURU $FWRUSHUIRUPDQFHHUURU
          6DPSOHFRPSOH[LW\      
K
K * F 6LQJOHWLPH6FDOH$& 7ZRWLPH6FDOH$& =HURWKRUGHU 'RXEOHORRS$&             6DPSOHFRPSOH[LW\    
K
K * F 6LQJOHWLPHVFDOH$& 7ZRWLPH6FDOH$& =HURWKRUGHU 'RXEOHORRS$& [Figure]
C ONCLUSION AND D ISCUSSION
In this paper, we establish the first finite-time global convergence analysis for the single-sample single-timescale natural actor-critic method under the Linear Quadratic Regulation (LQR) setting.
Our work is the first one adopting the simplest single-sample single-timescale structure for solving LQR, which may serve as the first step towards understanding the limits of the AC(NAC) on continuous control task.
We provide a novel analysis framework that systematically establishes the convergence of actor and critic simultaneously.
Our framework can be extended to analyze other single-timescale reinforcement learning algorithms.
A Proof of Main Theorems
A.1 Cost Estimate Error Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Critic Error Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 Natural Gradient Norm Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.4 Interconnected Iteration System Analysis . . . . . . . . . . . . . . . . . . . . . .
A.5 Global Convergence Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B Proof of Propositions
C Proof of Auxiliary Lemmas
D Experiment Details
A P ROOF OF M AIN T HEOREMS
We choose stepsizes α t =
c α √ 1+ t , β t = γ t = 1 √ 1+ t .
Additional constant multipliers c β , c γ can be considered in a similar way.
Before proceeding, we define the following notations for the ease of presentation:
[Equation]
[Equation]
In the sequel, we establish implicit (coupled) upper bounds for the cost estimator, the critic, and the actor in Theorem A.7, Theorem A.10, and Theorem A.13, respectively.
Then we prove the main
Theorem 4.3 by solving an interconnected system of inequalities in Appendix A.4.
Before start, we define two notations which are frequently used in our proof.
Definition A.1.
For any symmetric matrix M ∈ S n , we define the vector svec ( M ) ∈
R 12 ( n +1) as
[Equation] We further define its inverse smat ( · ) such that [Equation] A. COST ESTIMATION ERROR ANALYSIS
In this section, we establish an implicit upper bound for the cost estimator η t , in terms of the critic error and the natural gradient norm.
We first give an uniform upper bound for the covariance matrix D K t .
Proposition A.2.
(Upper bound for covariance matrix).
Suppose that Assumption 4.2 holds.
The covariance matrix of the stationary distribution N (0 , D K t ) induced by the Markov chain in (8) can be upper bounded by
[Equation]
where c 1 is a constant.
Note that the distribution of state-action pair is unbounded so that the feature function is also unbounded.
We can establish an upper bound for the tail probability of ( x, u ) by the Hansen-Wright inequality, the proof of which can be found in Rudelson & Vershynin (2013).
Lemma A.3.
(Hansen-Wright inequality).
For any integer m > 0 , let A be a matrix in R m × m and let η ∼ N (0 , I m ) be the standard Gaussian random variable in R m .
Then there exists an absolute constant ¯ c > 0 such that, for any θ ≥ 0 , we have
[Equation]
With this lemma, we can provide an uniform upper bound for the cost under high probability.
Proposition A.4.
(Upper bound for cost).
With probability at least 1 − 10 − 10 , for t = 0 , 1 , 2 , · · · , T − 1 , the cost satisfies
[Equation]
where ¯
[Equation]
) and c 2 is a constant.
Hereafter, we use ¯ U as an upper bound for all cost c ( x t , u t ) .
As a consequence, we choose η 0 ≤ ¯ U so that we have η t ≤ ¯ U for all t .
Lemma A.5.
(Perturbation of P K ).
Suppose K ′ is a small perturbation of K in the sense that
[Equation]
Then we have
[Equation]
Proof.
See Lemma 5.7 in Yang et al.
for a detailed proof.
With the perturbation of P K , we are ready to prove the Lipschitz continuous of J ( K ) .
Proposition A.6.
(Local Lipschitz continuity of J ( K ) )
Suppose Lemma A.5 holds, for any
K t , K t +1 , we have
[Equation]
where
[Equation]
[Equation]
Equipped with the above propositions and lemmas, we are able to bound the cost estimation error.
Theorem A.7.
Suppose that Assumptions 4.1 and 4.2 hold and choose α t =
[Equation]
1 √ 1+ t , where c α is a small positive constant.
With probability at least 1 − 10 − 10 , we have
[Equation]
Proof.
From line 5 of Algorithm 1, we have
[Equation]
Taking expectation up to ( x t , u t ) for both sides
[-2 + 2 γ 2 t ( c t − η t )=(1 − 2 γ t ) y 2 t + 2 γ t y t ( c t − J ( K t )) + 2 γ 2 t ( c t − η t ) 
-][Equation]
we have
[Equation]
To compute E [ y t ( c t − J ( K t ))]
[Equation]
we use the notation v t to denote the vector ( x t , u t ) and v 0: t to denote the sequence ( x 0 , u 0 ) , ( x 1 , u 1 ) , · · · , ( x t , u t ) .
Hence, we have
[Equation]
Once we know v 0: t − 1 , y t is not a random variable any more.
Thus we get
[Equation]
Hereafter
[Equation]
we need to verify Lemma A.5 first and use the local Lipschitz continuous property of
J ( K ) provided by Proposition A.6 to bound the cost estimation error.
Since we have
[Equation]
to satisfy (30) [Equation] we choose
[Equation]
Hence, according to the update rule
[Equation]
we have
[Equation]
where the last inequality comes from (28).
Thus Lemma A.5 holds for Algorithm 1.
As a consequence, Proposition A.6 is also guaranteed.
[Table]
[Equation]
where we use the fact that
[Equation]
Choose c α small enough such that
[Equation]
Then we get
[Equation]
Thus we have
E [ y
[Equation]
] ≤
[Equation]
K
[Equation]
−
[Equation]
Rearranging and summing from 0 to T − 1 , we have
T − 1 (cid:88)
[Equation]
In the sequel
[Equation]
we need to control I 1 , I 2 , I 3 respectively.
For I 1 , following Abel summation by parts, we have
[Equation]
where we use the fact that | y t | ≤ ¯ U .
For I 2 , we get
[Equation]
where the last inequality is due to
[Equation]
For I 3 , we have
[Equation]
where we use the fact 0 ≤ c t , η t ≤ ¯ U derived by Proposition A.4.
Combining all terms together, we get
T − 1 (cid:88) t =
[Equation]
Dividing by T , we have
[Equation]
(cid:88) t =
E
[Equation]
E K t
[Equation]
Thus we finish our proof.
A. C RITIC ERROR ANALYSIS
In this section, we derive an implicit bound for the critic error, in terms of the cost estimator error and the natural gradient norm.
First, we need the following propositions.
Proposition A.8.
For all the K t , there exists a constant µ > 0 such that
[Equation]
Proposition A.9.
(Lipschitz continuity of ω ∗ t )
For any ω ∗ t , ω ∗ t +1 , we have
[Equation]
[Equation]
where
[Equation]
[Equation]
Theorem A.10.
Suppose that Assumptions 4.1 and 4.2 hold and choose α t =
[Equation]
1 √ 1+ t , where c α is a small positive constant.
With probability at least 1 − 10 − 10 , we have
[Equation]
Proof.
Since we have A K t ω ∗ t = b K t , where b K t =
E ( x t ,u t ) [( c ( x t , u t ) − J ( K t )) ϕ
( x t , u t )] , we can further get
[Equation]
where in the last inequality, we use the fact that
[Equation]
Hence, we set
[Equation]
39) such that all ω ∗ t lie within this projection radius for all t .
From update rule of critic in Algorithm 1, we have
[Equation]
which further implies [Equation] [Equation] ω
[Equation]
∗
[Equation]
By applying 1-Lipschitz continuity of projection map, we have
[Equation]
This means
[Equation]
≤∥
[Equation]
t ⟨ z t , ¯ h ( ω t , K t ) ⟩ + 2 β t Λ( O t , ω t , K t ) + 2 β t ⟨ z t , ∆ h ( O t , η t , K t ) ⟩ + 2 ⟨ z t , ω ∗ t − ω ∗ t +1 ⟩ + 2 β 2 t ∥ h ( O t , ω t , K t ) + ∆ h O t , η t , K t ))
∥ 2 + 2 ∥ ω ∗ t − ω ∗ t +1 ∥ 2 .
From Proposition A.8, we know that σ min ( A K t ) ≥ µ for all K t .
Then we have
[Equation]
where we use the fact A K ω ∗ K t − b K t = 0 .
Hence
[Equation]
we have
[Equation]
where we use the fact that
[Equation]
Taking expectation up to ( x t , u t ) and noticing that
[Equation]
, we get
[Equation]
Therefore, using ∥ ∆ h ( O t , η t , K t ) ∥
≤ ¯ U | y t | [Equation] we can further rewrite (40) as
[Equation]
Based on (36)
[Equation]
we can rewrite the above inequality as
[Equation]
1 ∥
[Equation]
l
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
t
[Equation]
+
[Equation]
t
[Equation]
] +
[Equation]
Uβ t
[Equation]
t |
[Equation]
t
[Equation]
[Equation]
[Equation]
t
[Equation]
t [Equation] +1 ∥ [Equation]
[Equation]
∥
[Equation]
t ∥
[Equation]
2 ¯ Uβ t
[Equation]
|
[Equation]
t |∥
[Equation]
t
[Equation]
[Equation]
) l
[Equation]
t [Equation] z t ∥+ [Equation]
t
[Equation] t [Equation] where the second inequality is due to ∥ K t − K t +1 ∥ ≤
[Equation]
β t from (34), where
[Equation]
Choose c α small enough such that
[Equation]
Thus we can rewrite 41 as
[Equation]
Rearranging the inequality and summing from 0 to T − 1 yields µ
[Equation]
orc We need to control I
and I 2 , respectively.
For term I 1 , from Abel summation by parts, we have
[Equation]
For I 2 , from Cauchy-Schwartz inequality, we have
[Equation]
Combining the upper bound of the above two items, we can get
T − 1 (cid:88)
[Equation]
Dividing by T , we have
[Equation]
which concludes he convergence of critic.
A. N ATURAL GRADIENT NORM ANALYSIS
In this subsection, we derive an implicit bound for the natural gradient norm in terms of the the critic error.
Before proceeding, we need the following two lemmas, which characterize two important properties of LQR system.
Lemma A.11.
(Almost Smoothness).
For any two stable policies K and K ′ , J ( K ) and J ( K ′ ) satisfy:
[Equation]
Lemma A.12.
(Gradient Domination).
Let K ∗ be an optimal policy.
Suppose K has finite cost.
Then, it holds that
[Equation]
Theorem A.13.
Suppose that Assumptions 4.1 and 4.2 hold and choose α t =
[Equation]
1 √ 1+ t , where c α is a small positive constant.
With probability at least 1 − 10 − 10 , we have
[Equation]
Proof.
Combining the almost smoothness property [Equation] we get
J ( K t +
[Equation]
( K t
[Equation]
By the similar trick to the proof of Proposition A.
[Equation]
we can bound P K t by
[Equation]
where ˆ c 1 is a constant.
Hence we further have
[Equation]
where we use ∥ ˆ E K t ∥ F ≤ ( ¯ K + 1)¯ ω
[Equation]
Hence we define c 4 as follows
[Equation]
Then we get
[Equation]
where
[Equation]
Taking expectation up to ( x t , u t ) and rearranging the above inequality
[Equation]
we have
[Equation]
[Equation]
Summing over t from 0 to T − 1 gives
[Equation]
For term I 1 , using Abel summation by parts, we have
T [Equation] 1 (cid:88)
[Equation]
[Equation]
=
[Equation]
√ T
[Equation]
For term I 2 , by Cauchy-Schwartz inequality, we have
T − 1 (cid:88) t = E
[Equation]
T − 1 (cid:88) t =
E ∥ E K t ∥ 2 ) 12 (
T − 1 (cid:88) t =
E ∥ ˆ E K t − E K t ∥ 2 ) 12 .
Combining the results of I 1 and I 2 , we have
T
[Equation]
[Equation]
Dividing by T , we get
[Equation]
Thus we conclude our proof.
A. I NTERCONNECTED ITERATION SYSTEM ANALYSIS
From the definition in 22, we have
[Equation]
In the following, we give an interconnected iteration system analysis with respect to A ( T ) , B ( T ) and C ( T ) .
Theorem A.14.
Combining (32), (38) and (44), we have
[Equation]
Proof.
From (32), (38) and (44)
[Equation]
we have
[Equation]
For simplicity
[Equation]
we denote
[Equation]
Thus we further have
[Equation]
Then we have
[Equation]
For C ( T ) , we get
[Equation]
Combining 49, 50 and 51, we have
[Equation]
If bd 2 + bd 2 g 2 + 2 eg 2 < 1 , we have
[Equation]
Note that
[Equation]
Thus we can achieve bd 2 + bd 2 g 2 + 2 eg 2 < 1 by choosing the stepsize ratio smaller than the following threshold:
[Equation]
Therefore, we get
[Equation]
Thus we have
A
[Equation]
which concludes the proof
[Equation] A. G LOBAL CONVERGENCE ANALYSIS
Proof of Theorem 4.
Proof.
From gradient domination, we know that
[Equation]
From the convergence of C ( T ) , we know that
[Equation]
Hence, we have
[Equation]
Therefore, from 53 we get
[Equation]
Thus we conclude the proof of Theorem 4.3.
B P ROOF OF P ROPOSITIONS
To establish the Proposition 3.1, we need the following lemma, the proof of which can be found in
Nagar (1959); Magnus (1978).
Lemma B.1.
Let g ∼ N (0 , I n ) be the standard Gaussian random variable in R n and let M, N be two symmetric matrices.
Then we have
[Equation]
Proof of Proposition 3.1 :
Proof.
This proposition is a slight modification of lemma 3.2 in Yang et al.
and the proof is inspired by the proof of this lemma.
For any state-action pair ( x, u ) ∈ R d + k , we denote the successor state-action pair following policy π K by ( x ′ , u ′ ) .
With this notation, as we defined in (7), we have
[Equation]
where ϵ ∼ N (0 , D 0 ) and ζ ∼ N (0 , I k ) .
We further denote ( x, u ) and ( x ′ , u ′ ) by ϑ and ϑ ′ respectively.
Therefore, we have
[Equation]
(54) where
[Equation]
Therefore, by definition, we have ε ∼ N (0 , ˜ D 0 ) where
[Equation]
Since for any two matrices M and N , it holds that ρ ( MN )
=
ρ ( NM ) .
Then we get ρ ( L ) =
ρ ( A − BK ) < 1 .
Consequently, the Markov chain defined in (54) have a stationary distribution
N (0 , ˜ D K ) denoted by ˜ ρ K , where ˜ D K is the unique positive definite solution of the following Lyapunov equation
[Equation]
Meanwhile, from the fact that x ∼ N (0 , D K ) and u = − Kx + σζ , by direct computation we have
[Equation]
From the fact that ∥ AB ∥ F ≤ ∥ A ∥ F ∥ B ∥ and ∥ A ∥ ≤ ∥ A ∥ F , we have
[Equation]
Then we get
[Equation] Let M
[Equation]
N be any two symmetric matrices with appropriate dimension
[Equation]
we have
[Equation]
where ˜ D 1 / 2 K is the square root of ˜ D K .
By applying Lemma B. [Equation] we have
[Equation]
where the last equality follows from the fact that
[Equation]
for any two matrix M, N and a symmetric matrix S (Schacke, 2004).
Thus we have
E
[Equation]
Similarly
[Equation]
Since ϵ is independent of ϑ
[Equation]
we get
[Equation]
By the same argument, we have
[Equation]
where we make use of the Lyapunov equation (55).
Thus we get
[Equation]
[Equation]
Therefore, combining (56) and (57), we have
[Equation]
where in the last equality we use the fact that
[Equation]
for any matrices A, B, C, D .
Since ρ ( L ) < 1 , then I − L ⊤ ⊗ s L ⊤ is positive definite, which further implies A K is invertible.
From Bellman equation of Q K , we have
[Equation]
Multiply each side by ϕ ( x, u ) and take a expectation with respect to ( x, u ) , we get
[Equation]
We further have
[Equation]
where the first equality comes from the low of total expectation and
[Equation]
Therefore, we get
[Equation]
svec (Ω K ) =
b K , which implies ω ∗ K = svec (Ω K ) .
Thus we conclude our proof.
Proof of Proposition A.2 :
Proof.
Since D K t satisfies the Lyapunov equation defined in (10), we have
[Equation]
From Assumption 4.2, we know that ρ ( A − BK t ) ≤ ρ < 1 .
Thus for any ϵ > 0 , there exists a sub-multiplicative matrix norm ∥ · ∥ ∗ such that
[Equation]
Choose ϵ = 1 − ρ 2 , we get
[Equation]
Therefore, we can bound the norm of D K t by
[Equation]
Since all norms are equivalent on the finite dimensional Euclidean space, there exists a constant csatisfies
[Equation]
which concludes our proof.
Proof of Proposition A.4 :
Proof.
Since x t ∼ N (0 , D K t ) and u t = − K t x t + σζ t , we denote the joint distribution of ϑ t = ( x t , u t ) by ˜ ρ K t = N (0 , ˜ D K t ) where
[Equation]
Based on Lemma A.3, for ( x t , u t ) ∼ N (0 , ˜ D K t ) with ˜ D K t defined in (58), we obtain
[Equation]
Choose θ = c 2 log(10) ∥
˜ D K t ∥ with c 2 sufficiently large such that ¯ cc 2 > 12 and θ 2 ∥ ˜ D K t ∥ − 2 F > θ ∥ ˜ D K t ∥ − 1 , where we make use of the fact that ∥ ˜ D K t ∥ F is bounded.
Hence we have the following probability inequality
[Equation]
We define the following event
[Equation]
Then we have P ( A t ) ≥ 1 − 2 e − ¯ cc 2 log(10) ≥ 1 − 2 · 10 − 12 ≥ 1 − 10 − 11 .
We further define
[Equation]
Thus we get P ( ¯ A ) ≥ 1 − 10 − 10 .
In the sequel, we only consider the case when ¯ A holds.
That is, for any 0 ≤ t ≤ T − 1 , we have
[Equation]
where the third inequality holds since we choose c 2 large enough such that c 2 log(10) ≥ d + k and the last inequality is due to the fact
[Equation]
From Assumption 4.1, we know that ∥ K t ∥ ≤ ¯ K , so we have
[Equation]
[Equation]
From Proposition A.2, we know that
[Equation]
Substitute ∥ D K t ∥ into (59), we get
[Equation]
Notice that c ( x, u ) =
x ⊤ Qx + u ⊤ Ru , combine with (60), we get
[Equation]
Thus we can use ¯ U as an upper bound to both c ( x, u ) and ∥ x t ∥ 2 + ∥ u t ∥ [Equation] which concludes the proof.
Proof of Proposition A.6 :
Proof.
[Equation]
where the second inequality is due to the perturbation of P K in Lemma A.5 and
[Equation]
Thus we finish our proof.
Proof of Proposition A.8 :
Proof.
From Proposition 3.1, we know that
[Equation]
By Assumption 4.2, we have ρ ( L ) =
ρ ( A − BK t ) ≤ ρ < 1 .
Then we have
[Equation]
To bound σ min ( ˜ D K t ) , for any a ∈ R d and b ∈ R k , we have
[Equation]
≥ σ min
[Equation]
D K t )
∥ a − K ⊤ t b ∥ 2 + σ 2 ∥
b ∥
[Equation]
For ∥ a − K ⊤ t b ∥ 2 , we have
[Equation]
Hence we get
[Equation]
Thus we have
[Equation]
which further implies
[Equation]
We define
[Equation]
such that we get σ min
[Equation]
A
[Equation]
t ) ≥ µ, which concludes the proof.
Proof of Proposition A.9 :
Proof.
[Equation]
where
[Equation]
C P ROOF OF A UXILIARY L EMMAS
The following lemmas are well known and have been established in several papers (Yang et al., 2019; Fazel et al., 2018).
We include the proof here only for completeness.
Proof of Lemma 2.1 :
Proof.
Since we focus on the family of linear-Gaussian policies defined in (7), we have
[Equation]
Furthermore, for K ∈ R k × d such that ρ ( AB − K ) < 1 and positive definite matrix S ∈ R d × d , we define the following two operators
[Equation]
Hence, Γ K ( S ) and Γ ⊤ K ( S ) satisfy Lyapunov equations
[Equation]
respectively.
Therefore, for any positive definite matrices S 1 and S 2 , we get
[Equation]
Combining (10), (55), (64) and (65) [Equation] we know that [Equation]
[Equation]
Thus (62) implies
[Equation]
It remains to establish the gradient of J ( K ) .
Based on (62) [Equation] we have
[Equation]
where we use C to denote that we compute the gradient with respect to K and then substitute the expression of C .
Hence we get
[Equation]
Furthermore, we have
[Equation]
Then it reduces to compute ∇ K Tr ( C 1 Γ K {+( D σ )) | C 1 =( A − BK ) ⊤ C 
+}
( A − BK ) .
Applying this iteration for n times, we get
[Equation]
Meanwhile, by Lyapunov equation defined in (11), we have
[Equation]
Since ρ ( A − BK ) < 1 , we further get
[Equation]
Thus by letting n go to infinity in (68), we get
[Equation]
Hence, combining (67), we have
[Equation]
which concludes our proof
[Equation]
Proof of Lemma 2.2 :
Proof.
By definition, we have the state-value function as follows
[Equation]
Therefore, we have
[Equation]
Combining the linear dynamic system in (8) and the form of (70), we see that V K ( x ) is a quadratic function, which can be denoted by
[Equation]
where P K is defined in (11) and C K only depends on K .
Moreover, by definition, we know that E x ∼ ρ K
[ V K ( x )] = 0 , which implies
[Equation]
Thus we have C K = − Tr
( P K D K ) .
Hence, the expression of V K ( x ) is given by
[Equation]
= x ⊤ P K x − Tr ( P K D K ) .
Therefore, the action-value function Q K ( x, u ) can be written as Q ( x, u ) = c ( x, u ) − J ( K ) + E [ V K ( x ′ )
| x, u ]
=
c ( x, u ) − J ( K ) + ( Ax + Bu ) ⊤ P K ( Ax + Bu ) + Tr ( P K D 0 ) − Tr ( P K D K ) =
x ⊤ Qx + u ⊤ Ru
+ ( Ax + Bu ) ⊤ P K
( Ax + Bu ) − σ 2 Tr ( R + P K BB ⊤ ) − Tr
( P K Σ K ) .
Thus we finish the proof.
Proof of Lemma A.11 :
Proof.
By the definition of operator in (63) and (66), we have
[Equation]
Hereafter, we define ( A − BK ′ )
t x = x ′ t and u ′ t = − K ′ x ′ t .
Hence, we further have x ⊤ P K ′ x
= (cid:88) t ≥x ′⊤ t ( Q + K ′⊤ RK ′ )
x ′ t = (cid:88) t ≥( x ′⊤ t
Qx ′ t + u ′⊤ t
Ru ′ t ) . Therefore, we get
[Equation]
[Equation]
[Equation]
[Equation]
t
[Equation] [Equation]
x
[Equation]
x
[Equation]
Define
[Equation]
Then, from the expression of J ( K ) in (12a)
[Equation]
we have
[Equation]
K
[Equation]
where the last equation is due to the fact that
[Equation]
Hence, we finish our proof.
Proof of Lemma A.12 :
Proof.
By definition of A K,K ′ in (71), we have
[Equation]
where the equality is satisfied when K ′
=
K − ( R + B ⊤ P K B ) − 1 E K .
Therefore, we have
[Equation]
Thus we complete the proof of upper bound.
It remains to establish the lower bound.
Since the equality is attained at K ′ =
K − ( R +
B ⊤ P K B ) − 1 E K , we choose this K ′ such that [Equation]
[Equation]
Tr
[Equation]
K E K ) .
Overall, we have σ
[Equation]
≤
[Equation]
, which concludes our proof.
D E XPERIMENTAL DETAILS
Example D.1.
Consider a two-dimensional system with
A = (cid:20)
1 1 0 (cid:21) , B = (cid:20) 0 1 1 0 (cid:21) , Q = (cid:20) 9 2 2 1 (cid:21) , R = (cid:20)
2 2 8 (cid:21) .
Example D.2.
Consider a four-dimensional system with
A =   0 . 2 0 .
0 .
 , B =   0 .
1 0 .
 ,
Q =    0 .   0 . 0 .
0 . 
 , R = (cid:34)0 . 0 . 0 .
(cid:35) .
We compare our considered single-sample single-timescale AC with two other baseline algorithms that have been analyzed in the state-of-the-art theoretical works: the zeroth-order method (Fazel et al., 2018) (listed in Algorithm 2 on the next page) and the double loop AC (Yang et al., 2019) (listed in Algorithm 3 on the next page).
For the considered single-sample single-timescale AC, we set for both examples α t = 0 .
005 √ 1+ t , β t = 0 .
01 √ 1+ t , γ t = 0 .
1 √ 1+ t , σ = 1 , T = 10 6 .
Note that multiplying small constants to these stepsizes does not affect our theoretical results.
For the zeroth-order method proposed in Fazel et al.
(2018), we set z = 5000 , l = 20 , r = 0 .
1 , stepsize η = 0 .
01 and iteration number J = 1000 for the first numerical example; while in the second example, we set z = 20000 , l = 50 , r = 0 .
1 , η = 0 .
01 , J = 1000 .
We choose different parameters based on the trade-off between better performance and fewer sample complexity.
For the double loop AC proposed in Yang et al.
(2019), we set for both examples α t = 0 .
01 √ 1+ t , σ = 0 .
2 , η = 0 .
05 , inner-loop iteration number T = 500000 and
outer-loop iteration number J = 100 .
We note that the algorithm is fragile and sensitive to the practical choice of these parameters.
Moreover, we found that it is difficult for the algorithm to converge without an accurate critic estimation in the inner-loop.
In our implementation, we have to set the inner-loop iteration number to T =to barely get the algorithm converge to the global optimum.
This nevertheless demands a significant amount of computation.
Higher T iterations can yield more accurate critic estimation, and consequently more stable convergence, but at a price of even longer running time.
We run the outer-loop for 100 times for each run of the algorithm.
We run the whole algorithm 10 times independently to get the results shown in Figure.
With parallel computing implementation, it takes more than 2 weeks on our desktop workstation (Intel Xeon(R)
W-2225 CPU @
4.10GHz × 8) to finish the computation.
In comparison, it takes about 0.5 hour to run the single-sample single-timescale AC and 5 hours for the zeroth-order method.
Algorithm
Input: stabilizing policy gain K 0 such that ρ ( A − BK 0 ) < 1 , number of trajectories z , roll-out length l , perturbation amplitude r , stepsize η while updating current policy
do
Gradient Estimation: for i = 1 , · · · , z do
Sample x 0 from D
Simulate K j for l steps starting from x 0 and observe y 0 , · · · , y l − 1 and c 0 , · · · , c l − 1 .
Draw U i uniformly over matrices such that ∥ U i ∥
F = 1 , and generate a policy K j,U i =
K j + rU i .
Simulate K j,U i for l steps starting from x 0 and observe c ′ 0 , · · · , c ′ l − 1 .
Calculate empirical estimates: (cid:100) J iK j = l − 1 (cid:88) t
=
c t , (cid:100) L iK j = l − 1 (cid:88) t =
y t
y ⊤ t , (cid:92) J K j,Ui = l − 1 (cid:88) t
=
c ′ t .
end for
Return estimates: (cid:92) ∇ J ( K j ) =
z z (cid:88) i =
(cid:92) J K j,Ui − (cid:100) J iK j r
U i , (cid:100)
L K j = 1 z z (cid:88)
i = (cid:100)
L iK j .
Policy Update:
K j +1 =
K j − η (cid:92) ∇ J ( K j ) (cid:100) L K j − 1 .
j = j + 1 .
end while
Algorithm
Input: Initial policy π K 0 such that ρ ( A − BK 0 ) < 1 , stepsize γ for policy update.
while updating current policy do
Gradient Estimation:
Initialize the primal and dual variables by v 0 ∈ X Θ and ω 0 ∈ X Ω , respectively.
Sample the initial state x 0 ∈ R d from stationary distribution ρ K j .
Take action u 0 ∼ π K j ( ·| x 0 ) and obtain the reward c 0 and the next state x 1 .
for i = 1 , 2 , · · · , T do
Take action u t according to policy π K j , observe the reward c t and the next state x t +1 .
[-δ t = v 1 t − 1 − c t − 1 + [ ϕ ( x t − 1 , u t − 1 ) − ϕ ( x t , u t )] ⊤ v 2 t − 1 . v 1 t = v 1 t − 1 − α t [ ω 1 t − 
 + ϕ ( x t − 1 , u t − 1 )-][Equation]
[Equation]
[Equation]
Project v t and ω t to v 0 ∈ X Θ and ω 0 ∈ X Ω .
end for
Return estimates:
[Equation]
Policy Update:
[Equation]
j
[Equation]
end while