B OOSTING THE A CTOR WITH D UAL C RITIC
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper proposes a new actor-critic-style algorithm called Dual Actor-Critic or Dual-AC.
It is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation, which can be viewed as a two-player game between the actor and a critic-like function, which is named as dual critic.
Compared to its actor-critic relatives, Dual-AC has the desired property that the actor and dual critic are updated cooperatively to optimize the same objective function, providing a more transparent way for learning the critic that is directly related to the objective function of the actor.
We then provide a concrete algorithm that can effectively solve the minimax optimization problem, using techniques of multi-step bootstrapping, path regularization, and stochastic dual ascent algorithm.
We demonstrate that the proposed algorithm achieves the state-of-the-art performances across several benchmarks.
I NTRODUCTION
Reinforcement learning (RL) algorithms aim to learn a policy that maximizes the long-term return by sequentially interacting with an unknown environment.
Value-function-based algorithms ﬁrst approximate the optimal value function, which can then be used to derive a good policy.
These methods (Sutton, 1988; Watkins, 1989) often take advantage of the Bellman equation and use bootstrapping to make learning more sample efﬁcient than Monte Carlo estimation (Sutton & Barto, 1998).
However, the relation between the quality of the learned value function and the quality of the derived policy is fairly weak (Bertsekas & Tsitsiklis, 1996).
Policy-search-based algorithms such as REINFORCE (Williams, 1992) and others (Kakade, 2002; Schulman et al., 2015a), on the other hand, assume a ﬁxed space of parameterized policies and search for the optimal policy parameter based on unbiased Monte Carlo estimates.
The parameters are often updated incrementally along stochastic directions that on average are guaranteed to increase the policy quality.
Unfortunately, they often have a greater variance that results in a higher sample complexity.
Actor-critic methods combine the beneﬁts of these two classes, and have proved successful in a number of challenging problems such as robotics (Deisenroth et al., 2013), meta-learning (Bello et al., 2016), and games (Mnih et al., 2016).
An actor-critic algorithm has two components: the actor (policy) and the critic (value function).
As in policy-search methods, actor is updated towards the direction of policy improvement.
However, the update directions are computed with the help of the critic, which can be more efﬁciently learned as in value-function-based methods (Sutton et al., 2000; Konda & Tsitsiklis, 2003; Peters et al., 2005; Bhatnagar et al., 2009; Schulman et al., 2015b).
Although the use of a critic may introduce bias in learning the actor, its reduces variance and thus the sample complexity as well, compared to pure policy-search algorithms.
While the use of a critic is important for the efﬁciency of actor-critic algorithms, it is not entirely clear how the critic should be optimized to facilitate improvement of the actor.
For some parametric family of policies, it is known that a certain compatibility condition ensures the actor parameter update is an unbiased estimate of the true policy gradient (Sutton et al., 2000).
In practice, temporaldifference methods are perhaps the most popular choice to learn the critic, especially when nonlinear function approximation is used (e.g., Schulman et al.
(2015b)).
In this paper, we propose a new actor-critic-style algorithm where the actor and the critic-like function, which we named as dual critic, are trained cooperatively to optimize the same objective function.
The algorithm, called Dual Actor-Critic , is derived in a principled way by solving a dual form of the Bellman equation (Bertsekas & Tsitsiklis, 1996).
The algorithm can be viewed as a two-player game between the actor and the dual critic, and in principle can be solved by standard optimization algorithms like stochastic gradient descent (Section 2).
We emphasize the dual critic is not ﬁtting the value function for current policy , but that of the optimal policy .
We then show that, when function approximation is used, direct application of standard optimization techniques can result in instability in training, because of the lack of convex-concavity in the objective function (Section 3).
Inspired by the augmented Lagrangian method (Luenberger & Ye, 2015; Boyd et al., 2010), we propose path regularization for enhanced numerical stability.
We also generalize the two-player game formulation to the multi-step case to yield a better bias/variance tradeoff.
The full algorithm is derived and described in Section 4, and is compared to existing algorithms in Section 5.
Finally, our algorithm is evaluated on several locomotion tasks in the MuJoCo benchmark (Todorov et al., 2012), and compares favorably to state-of-the-art algorithms across the board.
Notation.
We denote a discounted MDP by M =
( S , A , P, R, γ ) , where S is the state space,
A the action space, P ( ·| s, a ) the transition probability kernel deﬁning the distribution over next-state upon taking action a in state x , R ( s, a ) the corresponding immediate rewards, and γ ∈ (0 , 1) the discount factor.
If there is no ambiguity, we will use (cid:80) a f ( a ) and (cid:82) f ( a ) da interchangeably.
D UALITY OF B ELLMAN O PTIMALITY E QUATION
In this section, we ﬁrst describe the linear programming formula of the Bellman optimality equation (Bertsekas et al., 1995; Puterman, 2014), paving the path for a duality view of reinforcement learning via Lagrangian duality.
In the main text, we focus on MDPs with ﬁnite state and action spaces for simplicity of exposition.
We extend the duality view to continuous state and action spaces in Appendix A.2.
Given an initial state distribution µ ( s ) , the reinforcement learning problem aims to ﬁnd a policy π ( ·| s ) :
S → P ( A ) that maximizes the total expected discounted reward with P ( A ) denoting all the probability measures over A , i.e. ,
[Equation]
1) where s i +1 ∼ P ( ·| s i , a i ) , a i ∼ π ( ·| s i ) .
Deﬁne V ∗ ( s ) :
= max π ∈P ( A ) E (cid:2)(cid:80) ∞ i =
γ i R ( s i , a i ) | s
= s (cid:3) , the Bellman optimality equation states that:
[Equation]
which can be formulated as a linear program (Puterman, 2014; Bertsekas et al., 1995):
[Equation] [Equation]
For completeness, we provide the derivation of the above equivalence in Appendix A. Without loss of generality, we assume there exists an optimal policy for the given MDP, namely, the linear programming is solvable.
The optimal policy can be obtained from the solution to the linear program (3) via
[Equation]
The dual form of the LP below is often easier to solve and yield more direct relations to the optimal policy.
[Equation]
Since the primal LP is solvable, the dual LP is also solvable, and P ∗ − D ∗ = 0 .
The optimal dual variables ρ ∗ ( s, a ) and optimal policy π ∗ ( a | s ) are closely related in the following manner:
Theorem 1 (Policy from dual variables) (cid:80) s,a
[Equation]
ρ ∗
[Equation]
s, a ) = 1 , and π
[Equation]
( a |
[Equation]
Since the goal of reinforcement learning task is to learn an optimal policy, it is appealing to deal with the Lagrangian dual which optimizes the policy directly, or its equivalent saddle point problem that jointly learns the optimal policy and value function.
Theorem 2 (Competition in one-step setting)
The optimal policy π ∗ , actor, and its corresponding value function V ∗ , dual critic, is the solution to the following saddle-point problem
[Equation]
[Equation]
where ∆[ V ]( s, a ) :
= R ( s, a ) + γ E s (cid:48) | s,a [ V ( s (cid:48) )] − V ( s ) .
The saddle point optimization (6) provides a game perspective in understanding the reinforcement learning problem (Goodfellow et al., 2014).
The learning procedure can be thought as a game between the dual critic, i.e. , value function for optimal policy, and the weighted actor, i.e. , α ( s ) π ( a | s ) : the dual critic V seeks the value function to satisfy the Bellman equation, while the actor π tries to generate state-action pairs that break the satisfaction.
Such a competition introduces new roles for the actor and the dual critic, and more importantly, bypasses the unnecessary separation of policy evaluation and policy improvement procedures needed in a traditional actor-critic framework.
S OURCES OF I NSTABILITY
To solve the dual problem in (6), a straightforward idea is to apply stochastic mirror prox (Nemirovski et al., 2009) or stochastic primal-dual algorithm
(Chen et al., 2014) to address the saddle point problem in (6).
Unfortunately, such algorithms have limited use beyond special cases.
For example, for an MDP with ﬁnite state and action spaces, the one-step saddle-point problem (6) with tabular parametrization is convex-concave, and ﬁnite-sample convergence rates can be established; see e.g., Chen & Wang (2016) and Wang (2017).
However, when the state/action spaces are large or continuous so that function approximation must be used, such convergence guarantees no longer hold due to lack of convex-concavity.
Consequently, directly solving (6) can suffer from severe bias and numerical issues, resulting in poor performance in practice (see, e.g. , Figure 1):
Large bias in one-step Bellman operator :
It is well-known that one-step bootstrapping in temporal difference algorithms has lower variance than Monte Carlo methods and often require much fewer samples to learn.
But it produces biased estimates, especially when function approximation is used.
Such a bias is especially troublesome in our case as it introduces substantial noise in the gradients to update the policy parameters.
Absence of local convexity and duality : Using nonlinear parametrization will easily break the local convexity and duality between the original LP and the saddle point problem, which are known as the necessary conditions for the success of applying primal-dual algorithm to constrained problems (Luenberger & Ye, 2015).
Thus none of the existing primal-dual type algorithms will remain stable and convergent when directly optimizing the saddle point problem without local convexity.
Biased stochastic gradient estimator with under-ﬁtted value function :
In the absence of local convexity, the stochastic gradient w.r
t. the policy π constructed from under-ﬁtted value function will presumably be biased and futile to provide any meaningful improvement of the policy.
Hence, naively extending the stochastic primal-dual algorithms in Chen & Wang (2016); Wang (2017) for the parametrized Lagrangian dual, will also lead to biased estimators and sample inefﬁciency.
D UAL A CTOR -C RITIC
In this section, we will introduce several techniques to bypass the three instability issues in the previous section: (1) generalization of the minimax game to the multi-step case to achieve a better bias-variance tradeoff; (2) use of path regularization in the objective function to promote local convexity and duality; and (
4. C OMPETITION IN MULTI - STEP SETTING
In this subsection, we will extend the minimax game between the actor and critic to the multi-step setting, which has been widely utilized in temporal-difference algorithms for better bias/variance tradeoffs (Sutton & Barto, 1998; Kearns & Singh, 2000).
By the deﬁnition of the optimal value function, it is easy to derive the k -step Bellman optimality equation as
[Equation]
Similar to the one-step case, we can reformulate the multi-step Bellman optimality equation into a form similar to the LP formulation, and then we establish the duality, which leads to the following mimimax problem:
Theorem 3 (Competition in multi-step setting)
The optimal policy π ∗ and its corresponding value function V ∗ is the solution to the following saddle point problem
[Equation]
(8) where δ (cid:16) { s i , a i } ki =0 , s k +1 (cid:17)
= (cid:80) ki
=
γ i R ( s i , a i ) + γ k +1 V ( s k +1 )
− V ( s ) and
E πα (cid:104) δ (cid:16) { s i , a i } ki =0 , s k +1 (cid:17)(cid:105) = (cid:88)
[Equation]
0 ) k (cid:89) i =
{+π ( a i | s i ) p ( s i +
+}
{+| s i , a i ) δ (cid:16) { s i , a i } ki =0 , s k +1 (cid:17) . The saddle-point problem (8) is similar to the one-step Lagrangian (6): the dual critic, V , and weighted k -step actor, α ( s 0 ) (cid:81) ki =
 π ( a i | s i ) , are competing for an equilibrium, in which critic and actor become the optimal value function and optimal policy.+}
However, it should be emphasized that due to the existence of max -operator over the space of distributions P ( A ) , rather than A , in the multi-step Bellman optimality equation (7), the establishment of the competition in multi-step setting in Theorem 3 is not straightforward: i) , its corresponding optimization is no longer a linear programming; ii) , the strong duality in (8) is not obvious because of the lack of the convex-concave structure. We ﬁrst generalize the duality to multi-step setting.
Due to space limit, detailed analyses for generalizing the competition to multi-step setting are provided in Appendix B.
4. P ATH R EGULARIZATION
When function approximation is used, the one-step or multi-step saddle-point problems (8) will no longer be convex in the primal parameters.
This could lead to instability and even divergence when solved by brute-force stochastic primal-dual algorithms.
One then desires to partially convexify the objectives without affecting the optimal solutions.
The augmented Lagrangian method (Boyd et al., 2010; Luenberger & Ye, 2015), also known as the method of multipliers , is designed and widely used for such purposes.
However, directly applying this method would require introducing penalty functions of the multi-step Bellman operator, which renders extra complexity and challenges in optimization.
Interested readers are referred to Appendix B.2 for further details.
Instead, we propose to use path regularization , as a stepping stone for promoting local convexity and computation efﬁciency.
The regularization term is motivated by the fact that the optimal value function satisﬁes the constraint V ( s ) =
E π ∗ (cid:2)(cid:80) ∞ i =
γ i R ( s i , a i ) | s (cid:3) .
In the same spirit as augmented Lagrangian, we will introduce to the objective the simple penalty function
E s ∼ µ ( s ) (cid:104)(cid:0) E π b (cid:2)(cid:80) ∞
i {+=
+}
γ i R ( s i , a i ) (cid:3) − V ( s ) (cid:1) 2 (cid:105) , leading to the following:
[Equation]
where η V (cid:62) 0 is a hyper-parameter controlling the strength of the regularization.
Note that in the penalty function above we use a behavior policy π b instead of an optimal policy, since the latter is unknown.
Adding such a regularization enables local duality in the primal parameters.
Indeed, this can be easily veriﬁed by showing the positive deﬁniteness of the Hessian at a local solution.
We call this approach path regularization , since it exploits the rewards in the sample path to regularize the solution path of value function V in the optimization procedure.
As a by-product, the regularization also provides a mechanism to utilize off-policy samples from behavior policy π b .
One can also see that the regularization indeed provides guidance and preference to search for the solution path.
Speciﬁcally, in each update of V the learning procedure, it tries to move towards the optimal value function while staying close to the value function of the behavior policy π b . Intuitively, such regularization restricts the feasible domain of candidates V to be a ball centered at V π b . Besides enhancing local convexity, such a penalty also avoids unboundedness of V in the learning procedure, and thus more numerical robust.
As long as the optimal value function is indeed in such a region, there will be little side-effect introduced.
Formally, we can show that with appropriate η V , the optimal solution ( V ∗ , α ∗ , π ∗ ) is not affected.
The main results of this subsection are summarized by the following theorem.
Theorem 4 (Property of path regularization)
The local duality holds for L r ( V, α, π ) .
Denote ( V ∗ , α ∗ , π ∗ ) as the solution to Bellman optimality equation, with some appropriate η V , ( V ∗ , α ∗ , π ∗ ) =
argmax α ∈P ( S ) ,π ∈P ( A ) argmin V L r ( V, α, π ) .
The proof of the theorem is given in Appendix B.3.
We emphasize that the theorem holds when V is given enough capacity, i.e. , in the nonparametric limit.
With parametrization introduced, deﬁnitely approximation error will be introduced, and the valid range of η V , which keeps optimal solution unchanged, will be affected.
However, the function approximation error is still an open problem for general class of parametrization, we omit such discussion here which is out of the range of this paper.
4. S TOCHASTIC D UAL A SCENT U PDATE
Rather than the primal form, i.e. , min V max α ∈P ( S ) ,π ∈P ( A ) L r ( V, α, π ) , we focus on optimizing the dual form max α ∈P ( S ) ,π ∈P ( A ) min V L r ( V, α, π ) .
The major reason is due to the sample efﬁciency consideration.
In the primal form, to apply the stochastic gradient descent algorithm at V t , one needs to solve max α ∈P ( S ) ,π ∈P ( A ) L r ( V t , α, π ) which involves sampling from each π and α during the solution path for the subproblem.
We deﬁne the regularized dual function (cid:96) r ( α, π ) :
= min V L r ( V, α, π ) .
We ﬁrst show the unbiased gradient estimator of (cid:96) r w.r.t.
θ ρ =
( θ α , θ π ) , which are parameters associated with α and π .
Then, we incorporate the stochastic update rule to the dual ascent algorithm (Boyd et al., 2010), resulting in the dual actor-critic (Dual-AC) algorithm.
The gradient estimators of the dual functions can be derived using chain rule and are provided below.
Theorem
The regularized dual function (cid:96) r ( α, π ) has gradients estimators
[Equation]
Therefore, we can apply stochastic mirror descent algorithm with the gradient estimator given in
Theorem 5 to the regularized dual function (cid:96) r ( α, π ) .
Since the dual variables are probabilistic distributions, it is natural to use KL -divergence as the prox-mapping to characterize the geometry in the family of parameters (Amari & Nagaoka, 1993; Nemirovski et al., 2009).
Speciﬁcally, in the t -th iteration,
[Equation]
where ˆ g t − 1 ρ = (cid:98) ∇
[Equation]
ρ [Equation] r (cid:0) θ t − 1 α , θ t − 1 π (cid:1) denotes the stochastic gradients estimated through (10) and (11) via given samples and KL ( q ( s, a ) || p
( s, a ))
= (cid:82) q ( s, a ) log q ( s,a ) p ( s,a ) dsda .
Intuitively, such update rule emphasizes a trade-off between the current policy and possible improvements based on samples.
The update of π shares some similarity to the TRPO, which is derived to ensure monotonic improvement of the new policy Schulman et al.
(2015a). We discuss the details in Section 4.4. Rather than just update V once via the stochastic gradient of ∇ V L r ( V, α, π ) in each iteration for solving saddle-point problem (Nemirovski et al., 2009), which is only valid in convexconcave setting, Dual-AC exploits the stochastic dual ascent algorithm which requires V t = argmin V L r α π
) in t -th iteration for estimating ∇
As we discussed, such operation will keep the gradient estimator of dual variables unbiased, which provides better direction for convergence.
In Algorithm 1, we update V t by solving optimization min V L r ( V, α t − 1 , π t − 1 ) .
In fact, the V function in the path-regularized Lagrangian L r ( V, α, π ) plays two roles: i) , inherited from the original
Lagrangian, the ﬁrst two terms in regularized Lagrangian (9) push the V towards the value function of the optimal policy with on-policy samples; ii) , on the other hand, the path regularization enforces
V to be close to the value function of behavior policy π b with off-policy samples.
Therefore, the V function in the Dual-AC algorithm can be understood as an interpolation between these two value functions learned from both on and off policy samples.
4. P RACTICAL I MPLEMENTATION
In above, we have introduced path regularization for recovering local duality property of the parametrized multi-step Lagrangian dual form and tailored stochastic mirror descent algorithm for optimizing the regularized dual function.
Here, we present several strategies for practical computation considerations.
Update rule of V t .
In each iteration, we need to solve V t = argmin θ V L r ( V, α t − 1 , π t − 1 ) , which depends on π b and η V , for estimating the gradient for dual variables.
In fact, the closer π b to π ∗ is, the smaller E s ∼ µ ( s ) (cid:104)(cid:0) E π b (cid:2)(cid:80) ∞
i =
γ i R ( s i , a i ) (cid:3) − V ∗ ( s ) (cid:1) 2 (cid:105) will be.
Therefore, we can set η V to be large for better local convexity and faster convergence.
Intuitively, the π t − 1 is approaching to π ∗ as the algorithm iterates.
Therefore, we can exploit the policy obtained in previous iteration, i.e. , π t − 1 , as the behavior policy.
The experience replay can also be used.
Furthermore, notice the L ( V, α t − 1 , π t − 1 ) is a expectation of functions of V , we will use stochastic gradient descent algorithm for the subproblem.
Other efﬁcient optimization algorithms can be used too.
Speciﬁcally, the unbiased gradient estimator for ∇ θ V L ( V, α t − 1 , π t − 1 ) is
[Equation]
We can use k -step Monte Carlo approximation for E π b µ (cid:2)(cid:80) ∞ i =
γ i R ( s i , a i ) (cid:3) in the gradient estimator.
As k is large enough, the truncate error is negligible (Sutton & Barto, 1998).
We will iterate via θ t,iV = θ t,i − 1 V
+ κ i (cid:98) ∇ θ t,i − 1 V
L r ( V, α t − 1 , π t − 1 ) until the algorithm converges.
It should be emphasized that in our algorithm, V t is not the estimation of the value function of π t .
Although V t eventually becomes the estimation of the optimal value function once the algorithm achieves the global optimum, in each update, the V t is one function which helps the current policy to be improved.
From this perspective, the Dual-AC bypasses the policy evaluation step.
Update rule of α t .
In practice, we may face with the situation that the initial sampling distribution is ﬁxed, e.g. , in MuJoCo tasks.
Therefore, we cannot obtain samples from α t ( s ) at each iteration.
We assume that ∃ η µ ∈ (0 , 1] , such that α ( s ) =
(1 − η µ ) β ( s ) + η µ µ ( s ) with β ( s ) ∈ P ( S ) .
Hence, we have
[Equation]
where ˜ α ( s ) =
(1 − η µ ) β ( s ) µ ( s ) .
Note that such an assumption is much weaker comparing with the requirement for popular policy gradient algorithms ( e.g. , Sutton et al.
; Silver et al.
(2014)) that assumes µ ( s ) to be a stationary distribution.
In fact, we can obtain a closed-form update for ˜ α
if a square-norm regularization term is introduced into the dual function.
Speciﬁcally,
Theorem 6 In t -th iteration, given V t and π t − 1 ,
[Equation]
[Equation]
Then, we can update ˜ α t through (14) with
Monte
Carlo approximation of
E
[Equation]
{ s i ,
[Equation]
} ki =
[Equation] , s k [Equation] 1 (cid:17)(cid:105) , avoiding the parametrization of ˜ α .
As we can see, the ˜ α t ( s ) reweights the samples based on the temporal differences and this offers a principled justiﬁcation for the heuristic prioritized reweighting trick used in (Schaul et al., 2015).
Update rule of θ tπ .
The parameters for dual function, θ ρ , are updated by the prox-mapping operator (12) following the stochastic mirror descent algorithm for the regularized dual function.
Speciﬁcally, in t -th iteration, given V t and α t , for θ π , the prox-mapping (12) reduces to
[Equation]
[Equation] where ˆ g tπ = (cid:98) ∇ θ π (cid:96) r ( θ tα , θ tπ ) .
Then, the update rule will become exactly the natural policy gradient (Kakade, 2002) with a principled way to compute the “policy gradient” ˆ g tπ .
This can be understood as the penalty version of the trust region policy optimization (Schulman et al., 2015a), in which the policy parameters conservative update in terms of KL -divergence is achieved by adding explicit constraints.
Exactly solving the prox-mapping for θ π requires another optimization, which may be expensive.
To further accelerate the prox-mapping, we approximate the KL-divergence with the second-order
Taylor expansion, and obtain an approximate closed-form update given by
[Equation]
where F t := E α t π t − 1 (cid:104) ∇ 2 log π θ t − 1 π (cid:105) denotes the Fisher information matrix.
Empirically, we may normalize the gradient by its norm (cid:113)
g tπ F − 1 t g tπ (Rajeswaran et al., 2017) for better performances.
Combining these practical tricks to the stochastic mirror descent update eventually gives rise to the dual actor-critic algorithm outlined in Algorithm 1.
R ELATED W ORK
The dual actor-critic algorithm includes both the learning of optimal value function and optimal policy in a uniﬁed framework based on the duality of the linear programming (LP) representation of
Bellman optimality equation.
The linear programming representation of Bellman optimality equation and its duality have been utilized for (approximate) planning problem (de Farias & Roy, 2004;
Wang et al., 2008; Pazis & Parr, 2011; O’Donoghue et al., 2011; Malek et al., 2014; Cogill, 2015), in which the transition probability of the MDP is known and the value function or policy are in tabular form.
Chen & Wang (2016); Wang (2017) apply stochastic ﬁrst-order algorithms (Nemirovski et al., 2009) for the one-step Lagrangian of the LP problem in reinforcement learning setting.
However, as we discussed in Section 3, their algorithm is restricted to tabular parametrization and are not applicable to MDPs with large or continuous state/action spaces.
The duality view has also been exploited in Neu et al.
Their algorithm is based on the duality of entropy-regularized Bellman equation (Todorov, 2007;
Rubin et al., 2012;
Fox et al., 2015; Haarnoja et al., 2017; Nachum et al., 2017), rather than the exact Bellman optimality equation used in our work.
Meanwhile, their algorithm is only derived and tested in tabular form.
Our dual actor-critic algorithm can be understood as a nontrivial extension of the (approximate) dual gradient method (Bertsekas, 1999, Chapter 6.3) using stochastic gradient and Bregman di- [Figure] vergence, which essentially parallels the view of (approximate) stochastic mirror descent algorithm
(Nemirovski et al., 2009) in the primal space.
As a result, the algorithm converges with diminishing stepsizes and decaying errors from solving subproblems.
Particularly, the update rules of α and π in the dual actor-critic are related to several existing algorithms.
As we see in the update of α , the algorithm reweighs the samples which are not ﬁtted well.
This is related to the heuristic prioritized experience replay (Schaul et al., 2015).
For the update in π , the proposed algorithm bears some similarities with trust region poicy gradient (TRPO) (Schulman et al., 2015a) and natural policy gradient (Kakade, 2002; Rajeswaran et al., 2017).
Indeed,
TRPO and NPR solve the same prox-mapping but are derived from different perspectives.
We emphasize that although the updating rules share some resemblance to several reinforcement learning algorithms in the literature, they are purely originated from a stochastic dual ascent algorithm for solving the two-play game derived from Bellman optimality equation.
E XPERIMENTS
We evaluated the dual actor-critic (Dual-AC) algorithm on several continuous control environments from the OpenAI Gym (Brockman et al., 2016) with MuJoCo physics simulator (Todorov et al., 2012).
We compared Dual-AC with several representative actor-critic algorithms, including trust region policy optimization (TRPO) (Schulman et al., 2015a) and proximal policy optimization (PPO) (Schulman et al., 2017) 1 .
We ran the algorithms with 5 random seeds and reported the average rewards with 50% conﬁdence interval.
Details of the tasks and setups of these experiments including the policy/value function architectures and the hyperparameters values, are provided in Appendix C.
6. A BLATION S TUDY
To justify our analysis in identifying the sources of instability in directly optimizing the parametrized one-step Lagrangian duality and the effect of the corresponding components in the dual actor-critic algorithm, we perform comprehensive Ablation study in InvertedDoublePendulum-v1, Swimmerv1, and Hopper-v1 environments.
We also considered the effect of k = { 10 , 50 } besides the one-step result in the study to demonstrate the beneﬁts of multi-step.
We conducted comparison between the Dual-AC and its variants, including Dual-AC w/o multi-step,
Dual-AC w/o path-regularization, Dual-AC w/o unbiased V , and the naive Dual-AC, for demonstrating the three instability sources in Section 3, respectively, as well as varying the k = { 10 , 50 } in Dual-AC.
Speciﬁcally, Dual-AC w/o path-regularization removes the path-regularization components; Dual-AC w/o multi-step removes the multi-step extension and the path-regularization; Dual-AC w/o unbiased V calculates the stochastic gradient without achieving the convergence of inner optimization on V ; and the naive Dual-AC is the one without all components. Moreover, Dual-AC with k = 10 and Dual-AC with k = 50 denote the length of steps set to be 10 and 50 , respectively.
The empirical performances on InvertedDoublePendulum-v1, Swimmer-v1, and Hopper-v1 tasks are shown in Figure 1.
The results are consistent across the tasks with the analysis.
The naive DualAC performs the worst.
The performances of the Dual-AC found the optimal policy which solves the problem much faster than the alternative variants.
The Dual-AC w/o unbiased V converges slower, showing its sample inefﬁciency caused by the bias in gradient calculation.
The Dual-AC w/o multistep and Dual-AC w/o path-regularization cannot converge to the optimal policy, indicating the importance of the path-regularization in recovering the local duality.
Meanwhile, the performance of Dual-AC w/o multi-step is worse than Dual-AC w/o path-regularization, showing the bias in onestep can be alleviated via multi-step trajectories. The performances of Dual-AC become better with the length of step k increasing on these three tasks. We conjecture that the main reason may be that in these three MuJoCo environments, the bias dominates the variance. Therefore, with the k increasing, the proposed Dual-AC obtains more accumulate rewards.
6. C OMPARISON IN C ONTINUOUS C ONTROL T ASKS
In this section, we evaluated the Dual-AC against TRPO and PPO across multiple tasks, including the InvertedDoublePendulum-v1, Hopper-v1, HalfCheetah-v1, Swimmer-v1 and Walker-v1.
These tasks have different dynamic properties, ranging from unstable to stable,
Therefore, they provide sufﬁcient benchmarks for testing the algorithms.
In Figure 2, we reported the average rewards across 5 runs of each algorithm with 50% conﬁdence interval during the training stage.
We also reported the average ﬁnal rewards in Table 1. [Table]
The proposed Dual-AC achieves the best performance in almost all environments, including Pendulum, InvertedDoublePendulum, Hopper, HalfCheetah and Walker.
These results demonstrate that Dual-AC is a viable and competitive RL algorithm for a wide spectrum of RL tasks with different dynamic properties.
A notable case is the InvertedDoublePendulum, where Dual-AC substantially outperforms TRPO and PPO in terms of the learning speed and sample efﬁciency, implying that Dual-AC is preferable to unstable dynamics.
We conjecture this advantage might come from the different meaning of V in our algorithm.
For unstable system, the failure will happen frequently, resulting the collected data are far away from the optimal trajectories.
Therefore, the policy improvement through the value function corresponding to current policy is slower, while our algorithm learns the optimal value function and enhances the sample efﬁciency.
C ONCLUSION
In this paper, we revisited the linear program formulation of the Bellman optimality equation, whose
Lagrangian dual form yields a game-theoretic view for the roles of the actor and the dual critic.
Although such a framework for actor and dual critic allows them to be optimized for the same objective function, parametering the actor and dual critic unfortunately induces instablity in optimization.
We analyze the sources of instability, which is corroborated by numerical experiments.
We then propose Dual Actor-Critic , which exploits stochastic dual ascent algorithm for the path regularized , multi-step bootstrapping two-player game, to bypass these issues.
The algorithm achieves the stateof-the-art performances on several MuJoCo benchmarks. [Figure]
Appendix A D ETAILS OF THE P ROOFS FOR S ECTIONA. D UALITY OF B ELLMAN O PTIMALITY E QUATION
Puterman (2014); Bertsekas et al.
(1995) provide details in deriving the linear programming form of the Bellman optimality equation.
We provide a brieﬂy proof here.
Proof We rewrite the linear programming 3 as
[Equation]
Recall the T is monotonic, i.e. , if V (cid:62) T V ⇒ T V (cid:62) T 2 V and V ∗ =
T ∞ V for arbitrary V , we have for ∀ V feasible, V (cid:62) T V (cid:62) T 2 V (cid:62) . . .
(cid:62) T ∞
V = V ∗ .
Theorem 1 (Optimal policy from occupancy)
[Equation]
=
[Equation]
Proof For the optimal occupancy measure, it must satisfy (cid:88)
[Equation]
where P denotes the transition distribution and I denotes a |S| × |SA| matrix where I ij =
if and only if j ∈ [( i − 1) |A| + 1 , . .
, i |A| ] .
Multiply both sides with 1 , due to µ and P are probabilities, we have (cid:104) 1 , ρ ∗ (cid:105)
= 1 .
Without loss of generality, we assume there is only one best action in each state.
Therefore, by the
KKT complementary conditions of (3), i.e. ,
[Equation]
which implies ρ ∗ ( s, a ) (cid:54)
=
if and only if a = a ∗ , therefore, the π ∗ by normalization.
Theorem 2 The optimal policy π ∗ and its corresponding value function V ∗ is the solution to the following saddle problem
[Equation]
where ∆[ V ]( s, a )
=
R ( s, a ) + γ E s (cid:48) | s,a [ V ( s (cid:48) )]
− V ( s ) .
Proof Due to the strong duality of the optimization (3), we have
[Equation]
Then, plugging the property of the optimum in Theorem 1, we achieve the ﬁnal optimization (6).
A. C ONTINUOUS S TATE AND A CTION MDP E XTENSION
In this section, we extend the linear programming and its duality to continuous state and action
MDP.
In general, the only weak duality holds for inﬁnite constraints, i.e. , P ∗ (cid:62) D ∗ .
With a mild assumption, we will recover the strong duality for continuous state and action MDP, and most of the conclusions in discrete state and action MDP still holds.
Speciﬁcally, without loss of generality, we consider the solvable MDP, i.e. , the optimal policy, π ∗ ( a | s ) , exists.
If (cid:107) R ( s, a ) (cid:107) ∞ (cid:54) C R , (cid:107) V ∗ (cid:107) ∞ (cid:54) C R 1 − γ .
Moreover,
[Equation]
where the ﬁrst inequality comes from 2 (cid:104)
{+f ( x ) , g ( x ) (cid:105) 2 (cid:54) (cid:107) f (cid:107) 
+}
+ (cid:107) g (cid:107) 22 .
[Equation]
∗ ( s [Equation] [Equation] ∞ , for some π b ∈ P that π b ( a | s ) > 0 for ∀
( s, a ) ∈ S × A .
Therefore, with the assumption that (cid:107) R ( s, a ) (cid:107) 2 µ (cid:54) C µR , ∀ a ∈ A , we have R ( s, a ) ∈
L 2 µπ b ( S × A ) and V ∗ ( s (cid:48) )
∈
L 2 µ ( S ) .
The constraints in the primal form of linear programming can be written as
[Equation]
where I − γ P : L 2 µ ( S )
→ L 2 µπ b ( S × A ) without any effect on the optimality.
For simplicity, we denote (cid:23) as (cid:23) L 2 µπb and (cid:104) f, g (cid:105) =
(cid:82) f
( s, a ) g
( s, a )
µ ( s ) π b ( a | s ) dsda .
Apply the Lagrangian multiplier for constraints in ordered Banach space in Burger (2003), we have
[Equation]
[Equation]
The solution ( V ∗ , (cid:37) ∗ ) also satisﬁes the KKT conditions,
[Equation] [Equation]
where (cid:62) denotes the conjugate operation.
By the KKT condition, we have
[Equation]
The strongly duality also holds,
i
[Equation]
e. ,
[Equation]
[Equation]
Proof We compute the duality gap
[Equation]
, which shows the strongly duality holds.
B D ETAILS OF T HE P ROOFS FOR S ECTIONB. C OMPETITION IN M ULTI -S TEP S ETTING
Once we establish the k -step Bellman optimality equation (7) [Equation] it is easy to derive the λ -Bellman optimality equation, i
[Equation]
e. ,
[Equation]
Proof Denote the optimal policy as π ∗ ( a | s ) , we have
[Equation]
holds for arbitrary ∀ k ∈ N .
Then
[Equation]
we conduct k ∼ G eo ( λ ) and take expectation over the countable inﬁnite many equation, resulting
[Equation]
Next, we investigate the equivalent optimization form of the k -step and λ -Bellman optimality equation, which requires the following monotonic property of T k and T λ .
Lemma 7 Both T k and T λ are monotonic.
Proof Assume U and V are the value functions corresponding to π 1 and π 2 , and U (cid:62) V , i.e. ,
U ( s ) (cid:62) V ( s ) , ∀ s ∈ S , apply the operator T k on U and V , we have
[Equation]
Due to U (cid:62) V , we have E π s k {++
+} | s [ U ( s k +1 )] (cid:62) E
π s k {++
+} | s [ V ( s k +1 )] , ∀ π ∈ P , which leads to the ﬁrst conclusion, T k U (cid:62) T k V .
Since T λ = (1 − λ ) (cid:80) ∞ k =1 T k
=
E k ∼G eo ( λ ) [ T k ] , therefore, T λ is also monotonic.
With the monotonicity of T k and T λ , we can rewrite the V ∗ as the solution to an optimization,
Theorem 8 The optimal value function V ∗ is the solution to the optimization
[Equation]
(28) where µ ( s ) is an arbitrary distribution over S .
Proof Recall the T k is monotonic, i.e. , V (cid:62) T k V ⇒ T k V (cid:62) T 2 k V and V ∗ =
T ∞ k V for arbitrary
V , we have for ∀ V , V (cid:62) T k V (cid:62) T 2 k V (cid:62) . . .
(cid:62) T ∞ k V
=
V ∗ , where the last equality comes from the Banach ﬁxed point theorem (Puterman, 2014).
Similarly, we can also show that ∀ V ,
V (cid:62) T ∞ λ V
= V ∗ .
By combining these two inequalities, we achieve the optimization.
We rewrite the optimization as
[Equation]
We emphasize that this optimization is no longer linear programming since the existence of maxoperator over distribution space in the constraints.
However, Theorem 1 still holds for the dual variables in (32) [Equation]
Proof Denote the optimal policy as
[Equation]
the KKT condition of the optimization (29) can be written as
[Equation]
Denote P
[Equation]
[Equation]
s
[Equation]
1 | s k
[Equation]
{+) (cid:81) k − 1 i =
+}
{+p ( s i +
+}
| s i , a i ) (cid:81) ki
=1 π ( a i | s i ) , we simplify the condition, i.e. ,
[Equation]
Due to the P π ∗ V k ( s (cid:48) | s, a ) is a conditional probability for ∀ V , with similar argument in Theorem 1, we have (cid:80) s,a ρ ∗ ( s, a )
= 1 .
By the KKT complementary condition, the primal and dual solutions, i.e. , V ∗ and ρ ∗ , satisfy
[Equation]
Recall V ∗ denotes the value function of the optimal policy, then, based on the deﬁnition, ˜ π ∗ V ∗ =
π ∗ which denotes the optimal policy.
Then, the condition (30) implies ρ ( s, a ) (cid:54)
=
if and only if a = a ∗ , therefore, we can decompose ρ ∗ ( s, a ) =
α ∗ ( s ) π ∗ ( a | s ) .
The corresponding Lagrangian of optimization (29) is
[Equation]
where ∆ πk [ V
[Equation]
s
[Equation]
a
[Equation]
We further simplify the optimization.
Since the dual variables are positive, we have
[Equation]
After clarifying these properties of the optimization corresponding to the multi-step Bellman optimality equation, we are ready to prove the Theorem 3.
Theorem
The optimal policy π ∗ and its corresponding value function V ∗ is the solution to the following saddle point problem
[Equation]
Proof By Theorem 1 in multi-step setting
[Equation]
we can decompose ρ ( s, a ) =
α ( s ) π ( a | s ) without any loss.
Plugging such decomposition into the Lagrangian 32 and realizing the equivalence among the optimal policies [Equation] we arrive the optimization as min V max α ∈P ( S ) ,π ∈P ( A )
L k
( V, α, π ) .
Then, because of the strong duality as we proved in Lemma 9, we can switch min and max operators in optimization 8 without any loss [Equation]
Lemma
The strong duality holds in optimization (8).
Proof Speciﬁcally, for every α ∈ P ( S ) , π ∈ P ( A ) ,
[Equation]
On the other hand
[Equation]
since L k ( V, α ∗ , π ∗ ) is convex w.r.t.
V , we have V ∗ ∈ argmin V L k ( V, α ∗ , π ∗ ) , by checking the ﬁrst-order optimality.
Therefore, we have
[Equation]
Combine these two conditions, we achieve the strong duality even without convex-concave property
[Equation]
B. T HE C OMPOSITION IN A PPLYING A UGMENTED L AGRANGIAN M ETHOD
We consider the one-step Lagrangian duality ﬁrst.
Following the vanilla augmented Lagrangian method, one can achieve the dual function as
[Equation]
where
[Equation]
( s ) π ( a | s ))
=
[Equation]
c (cid:110) [max (0 , α ( s ) π ( a | s ) +
[Equation]
∆[
[Equation]
]( s [Equation] a ))] 2 − α 2 ( s ) π 2 ( a | s ) (cid:111) . The computation of P c is in general intractable due to the composition of max and the condition expectation in ∆[ V ]( s, a ) , which makes the optimization for augmented Lagrangian method difﬁcult.
For the multi-step Lagrangian duality, the objective will become even more difﬁcult due to constraints are on distribution family P ( S ) and P ( A ) , rather than S × A .
B. P ATH R EGULARIZATION
Theorem
The local duality holds for L r ( V, α, π ) .
Denote ( V ∗ , α ∗ , π ∗ ) as the solution to Bellman optimality equation, with some appropriate η V , ( V ∗ , α ∗ , π ∗ ) =
[Equation]
Proof The local duality can be veriﬁed by checking the Hessian of L r ( θ V ∗ ) .
We apply the local duality theorem (Luenberger & Ye, 2015)[Chapter 14].
Suppose ( ˜ V ∗ , ˜ α ∗ , ˜ π ∗ ) is a local solution to min V max α ∈P ( S ) ,π ∈P ( A ) L r ( V, α, π ) , then, max α ∈P ( S ) ,π ∈P ( A ) min V L r ( V, α, π ) has a local solution ˜
V ∗ with corresponding ˜ α ∗ , ˜ π ∗ .
Next, we show that with some appropriate η V , the path regularization does not change the optimum.
Let U π ( s ) =
E π (cid:2)(cid:80) ∞ i =
γ i R ( s i , a i ) | s (cid:3) , and thus, U π ∗ =
V ∗ .
We ﬁrst show that for ∀ π b ∈ P ( A ) , we have
[Equation]
where the last second inequality comes from the fact that π b ( a i | s i ) p
( s i +
| s i , a i ) is distribution.
We then rewrite the optimization min V max α ∈P ( S ) ,π ∈P ( A ) L r ( V, α, π ) as min V max
[Equation]
A [Equation] − V ( s ) (cid:1) 2 (cid:105) (cid:54) (cid:15) (cid:111) , due to the well-known one-to-one correspondence between regularization η V and (cid:15) Nesterov (2005).
If we set η V with appropriate value so that its corresponding (cid:15) ( η V ) (cid:62) 21 − γ (cid:107) R ( s, a ) (cid:107) ∞ , we will have
V ∗ ∈ Ω (cid:15) ( η V ) , which means adding such constraint, or equivalently, adding the path regularization, does not affect the optimality.
Combine with the local duality, we achieve the conclusion.
In fact, based on the proof, the closer π b to π ∗ is, the smaller
E
[Equation]
will be.
Therefore, we can set η V bigger for better local convexity, which resulting faster convergence.
B. S TOCHASTIC D UAL A SCENT U PDATE
Corollary
The regularized dual function (cid:96) r ( α, π ) has gradients estimators
[Equation]
Proof We mainly focus on deriving ∇ θ π (cid:96) r
( θ α , θ π ) .
The derivation of ∇ θ α (cid:96) r
( θ α , θ π ) is similar.
By chain rule
[Equation]
we have
[Equation]
The ﬁrst term in RHS equals to zero due to the ﬁrst-order optimality condition for
V ( α, π ) =
argmin V L r ( V, α, π ) .
B. P RACTICAL A LGORITHM
Theorem 6 In t -th iteration, given V t and π t − 1 ,
[Equation]
Proof Recall the optimization w.r
[Equation]
t.
[Equation]
α is
[Equation]
denote τ ( s ) as the dual variables of the optimization
[Equation]
we have the KKT condition as
[Equation]

[Equation]
0 , s k +1 (cid:17)(cid:105)(cid:17)
= 0 ,
˜ α (cid:62) 0 , τ (cid:62) 0 , ⇒ τ ( s ) =
  − E π (cid:104) δ (cid:16) {
s i , a i } ki =0 , s k +1 (cid:17)(cid:105) E π (cid:104) δ (cid:16) { s i , a i } ki =0 , s k +1 (cid:17)(cid:105) <
E π (cid:104) δ (cid:16) { s i , a i } ki =0 , s k +1 (cid:17)(cid:105) (cid:62) 0 .
Therefore, in t -th iteration, ˜ α t ( s ) = 1 η α max (cid:16) 0 , E π (cid:104) δ (cid:16) {
s i , a i } ki =0 , s k +1 (cid:17)(cid:105)(cid:17) .
C E XPERIMENT D ETAILS
Policy and value function parametrization.
For fairness, we use the same parametrization across all the algorithms.
The parametrization of policy and value functions are largely based on the recent paper by Rajeswaran et al.
(2017), which shows the natural policy gradient with the RBF neural network achieves the state-of-the-art performances of TRPO on MuJoCo.
For the policy distribution, we parametrize it as π θ π ( a | s ) =
N ( µ θ π ( s ) , Σ θ π ) , where µ θ π ( s ) is a two-layer neural nets with the random features of RBF kernel as the hidden layer and the Σ
θ π is a diagonal matrix.
The
RBF kernel bandwidth is chosen via median trick (Dai et al., 2014; Rajeswaran et al., 2017).
The same as Rajeswaran et al.
(2017), we use 100 hidden nodes in Pendulum, InvertedDoublePendulum,
Swimmer, Hopper, and use 500 hidden nodes in HalfCheetah.
Since the TRPO and PPO uses
GAE (Schulman et al., 2015b) with linear baseline as V , we also use the parametrization for V in our algorithm.
However, the Dual-AC can adopt arbitrary function approximator without any change.
Training details.
We report the hyperparameters for each algorithms here.
We use the γ = 0 .for
all the algorithms.
We keep constant stepsize and tuned for TRPO, PPO and Dual-AC in { 0 . 001 , 0 . 01 , 0 . 1 } .
The batchsize are set to be 52 trajectories for comparison to the competitors in Section 6.2.
For the Ablation study, we set batchsize to be 24 trajectories for accelerating.
The
CG damping parameter for TRPO is set to be 10 − 4 .
We iterate 20 steps for the Fisher information matrix computation.
For the η V , η µ , 1 η α in Dual-AC from { 0 .