V ARIANCE R EDUCTION FOR P OLICY G RADIENT WITH A CTION -D EPENDENT F ACTORIZED B ASELINES
Anonymous authors
Paper under double-blind review
A BSTRACT
Policy gradient methods have enjoyed great success in deep reinforcement learning but suffer from high variance of gradient estimates.
The high variance problem is particularly exasperated in problems with long horizons or high-dimensional action spaces.
To mitigate this issue, we derive a bias-free action-dependent baseline for variance reduction which fully exploits the structural form of the stochastic policy itself and does not make any additional assumptions about the MDP.
We demonstrate and quantify the beneﬁt of the action-dependent baseline through both theoretical analysis as well as numerical results, including an analysis of the suboptimality of the optimal state-dependent baseline.
Our experimental results indicate that action-dependent baselines allow for faster learning on standard reinforcement learning benchmarks and high dimensional hand manipulation tasks. Finally, we show that the general idea of including additional information in baselines for improved variance reduction can be extended to partially observed and multi-agent tasks.
I NTRODUCTION
Deep reinforcement learning has achieved impressive results in recent years in domains such as video games from raw visual inputs (Mnih et al., 2015), board games (Silver et al., 2016), simulated control tasks (Schulman et al., 2016; Lillicrap et al., 2016; Rajeswaran et al., 2017a), and robotics (Levine et al., 2016).
An important class of methods behind many of these success stories are policy gradient methods (Williams, 1992; Sutton et al., 2000; Kakade, 2002; Schulman et al., 2015; Mnih et al., 2016), which directly optimize parameters of a stochastic policy through local gradient information obtained by interacting with the environment using the current policy.
Policy gradient methods operate by increasing the log probability of actions proportional to the future rewards inﬂuenced by these actions.
On average, actions which perform better will acquire higher probability, and the policy’s expected performance improves.
A critical challenge of policy gradient methods is the high variance of the gradient estimator.
This high variance is caused in part due to difﬁculty in credit assignment to the actions which affected the future rewards.
Such issues are further exacerbated in long horizon problems, where assigning credits properly becomes even more challenging.
To reduce variance, a “baseline” is often employed, which allows us to increase or decrease the log probability of actions based on whether they perform better or worse than the average performance when starting from the same state.
This is particularly useful in long horizon problems, since the baseline helps with temporal credit assignment by removing the inﬂuence of future actions from the total reward.
A better baseline, which predicts the average performance more accurately, will lead to lower variance of the gradient estimator.
The key insight of this paper is that when the individual actions produced by the policy can be decomposed into multiple factors, we can incorporate this additional information into the baseline to further reduce variance.
In particular, when these factors are conditionally independent given the current state, we can compute a separate baseline for each factor, whose value can depend on all quantities of interest except that factor.
This serves to further help credit assignment by removing the inﬂuence of other factors on the rewards, thereby reducing variance.
In other words, information about the other factors can provide a better evaluation of how well a speciﬁc factor performs.
Such factorized policies are very common, with some examples listed below.
• In continuous control and robotics tasks, multivariate Gaussian policies with a diagonal covariance matrix are often used.
In such cases, each action coordinate can be considered a factor.
Similarly, factorized categorical policies are used in game domains like board games and Atari.
• In multi-agent and distributed systems, each agent deploys its own policy, and thus the actions of each agent can be considered a factor of the union of all actions (by all agents).
This is particularly useful in the recent emerging paradigm of centralized learning and decentralized execution (Foerster et al., 2017; Lowe et al., 2017).
In contrast to the previous example, where factorized policies are a common design choice, in these problems they are dictated by the problem setting.
We demonstrate that action-dependent baselines consistently improve the performance compared to baselines that use only state information.
The relative performance gain is task-speciﬁc, but in certain tasks, we observe signiﬁcant speed-up in the learning process.
We evaluate our proposed method on standard benchmark continuous control tasks, as well as on a high dimensional door opening task with a ﬁve-ﬁngered hand, and on a blind peg insertion POMDP task.
We believe that our method will facilitate further applications of reinforcement learning methods in domains with extremely high-dimensional actions, including multi-agent systems.
Videos and additional results of the paper are available at https://sites.google.com/view/ad-baselines .
R ELATED WORKS
Three main classes of methods for reinforcement learning include value-based methods (Watkins &
Dayan, 1992), policy-based methods (Williams, 1992; Kakade, 2002; Schulman et al., 2015) and actor-critic methods (Konda & Tsitsiklis, 2000; Peters & Schaal, 2008; Mnih et al., 2016).
Valuebased and actor-critic methods usually compute a gradient of the objective through the use of critics, which are often biased, unless strict compatibility conditions are met (Sutton et al., 2000; Konda &
Tsitsiklis, 2000).
Such conditions are rarely satisﬁed in practice due to the use of stochastic gradient methods and powerful function approximators.
In comparison, policy gradient methods are able to compute an unbiased gradient, but suffer from high variance.
Policy gradient methods are therefore usually less sample efﬁcient, but can be more stable than critic-based methods (Duan et al., 2016).
A large body of work has investigated variance reduction techniques for policy gradient methods.
One effective method to reduce variance without introducing bias is through using a baseline, which has been widely studied (Sutton & Barto, 1998; Weaver & Tao, 2001;
Greensmith et al., 2004; Schulman et al., 2016).
However, fully exploiting the factorizability of the policy probability distribution to further reduce variance has not been studied.
Recently, methods like Q-Prop (Gu et al., 2017) make use of an action-dependent control variate, a technique commonly used in Monte Carlo methods and recently adopted for RL.
Since Q-Prop utilizes off-policy data, it has the potential to be more sample efﬁcient than pure on-policy methods.
However, Q-prop is signiﬁcantly more computationally expensive, since it needs to perform a large number of gradient updates on the critic using the off-policy data, thus not suitable with fast simulators.
In contrast, our formulation of action-dependent baselines has little computational overhead, and improves the sample efﬁciency compared to on-policy methods with state-only baseline.
The idea of using additional information in the baseline or critic has also been studied in other contexts.
Methods such as Guided Policy Search (Levine et al., 2016; Mordatch et al., 2015) and variants train policies that act on high dimensional observations like images, but use a low dimensional encoding of the problem like joint positions during the training process.
Recent efforts in multi-agent systems (Foerster et al., 2017; Lowe et al., 2017) also use additional information in the centralized training phase to speed-up learning.
However, using the structure in the policy parameterization itself to enhance the learning speed, as we do in this work, has not been explored.
P RELIMINARIES
In this section, we establish the notations used throughout this paper, as well as basic results for policy gradient methods, and variance reduction via baselines.
3. N OTATION
This paper assumes a discrete-time Markov decision process (MDP), deﬁned by ( S , A , P , r, ρ 0 , γ ) ,
in which S ⊆ R n is an n -dimensional state space,
A ⊆
R m an m -dimensional action space,
P : S ×
A × S → R + a transition probability function, r : S ×
A → R a bounded reward function, ρ 0 : S → R + an initial state distribution, and γ ∈ (0 , 1] a discount factor.
The presented models are based on the optimization of a stochastic policy π θ : S ×
A → R + parameterized by θ .
Let η ( π θ ) denote its expected return: η ( π θ ) =
E τ [ (cid:80) ∞ t =
γ t r ( s t , a t )] , where τ = ( s 0 , a 0 , . . . ) denotes the whole trajectory, s 0 ∼ ρ 0 ( s 0 ) , a t ∼ π θ ( a t
| s t ) , and s t +1 ∼ P ( s t +1 | s t , a t ) for all t .
Our goal is to ﬁnd the optimal policy arg max θ η ( π θ ) .
We will use ˆ Q ( s t , a t ) to describe samples of cumulative discounted return, and Q ( a t , s t ) to describe a function approximation of ˆ Q ( s t , a t ) .
We will use ”Q-function” when describing an abstract action-value function.
For a partially observable Markov decision process (POMDP), two more components are required, namely Ω , a set of observations, and O : S × Ω
→ R ≥ 0 , the observation probability distribution.
In the fully observable case, Ω ≡ S .
Though the analysis in this article is written for policies over states, the same analysis can be done for policies over observations.
3. T HE S CORE F UNCTION (SF) E STIMATOR
An important technique used in the derivation of the policy gradient is known as the score function (SF) estimator (Williams, 1992), which also comes up in the justiﬁcation of baselines.
Suppose that we want to estimate ∇ θ E x
[ f ( x )] where x ∼ p θ ( x ) , and the family of distributions { p θ ( x ) : θ ∈ Θ } has common support.
Further suppose that log p θ ( x ) is continuous in θ .
In this case we have
[Equation]
3. P OLICY G RADIENT
The Policy Gradient Theorem (Sutton et al., 2000) states that
[Equation]
For convenience, deﬁne ρ π ( s ) = (cid:80) ∞ {+t =
+}
γ t p ( s t = s ) as the state visitation frequency, and ˆ Q ( s t , a t ) = (cid:80) ∞ t (cid:48) =
t γ t (cid:48) − t r t (cid:48) .
We can rewrite the above equation (with abuse of notation) as
[Equation]
It is further shown that we can reduce the variance of this gradient estimator without introducing bias by subtracting off a quantity dependent on s t from ˆ Q ( s t , a t ) (Williams, 1992; Greensmith et al., 2004). See Appendix A for a derivation of the optimal state-dependent baseline.
[Equation]
This is valid because, applying the SF estimator in the opposite direction, we have
[Equation]
A CTION - DEPENDENT BASELINES
In practice there can be rich internal structure in the policy parameterization.
For example, for continuous control tasks, a very common parameterization is to make π θ ( a t
| s t ) a multivariate Gaussian with diagonal variance, in which case each dimension a it of the action a t is conditionally independent of other dimensions, given the current state s t .
Another example is when the policy outputs a tuple of discrete actions with factorized categorical distributions.
In the following subsections, we show that such structure can be exploited to further reduce the variance of the gradient estimator without introducing bias by changing the form of the baseline.
Then, we derive the optimal action-dependent baseline for a class of problems and analyze the suboptimality of non-optimal baselines in terms of variance reduction.
We then propose several practical baselines for implementation purposes.
We conclude the section with the overall policy gradient algorithm with action-dependent baselines for factorized policies. We provide an exposition for situations when the conditional independence assumption does not hold, such as for stochastic policies with general covariance structures, in Appendix E, and for compatibility with other variance reduction techniques in Appendix F.
4. B ASELINES FOR P OLICIES WITH C ONDITIONALLY I NDEPENDENT F ACTORS
In the following, we analyze action-dependent baselines for policies conditionally independent factors.
For example, multivariate Gaussian policies with a diagonal covariance structure are commonly used in continuous control tasks.
Assuming an m -dimensional action space, we have π θ ( a t | s t ) =
(cid:81) mi =1 π θ ( a it | s t ) .
Hence
[Equation]
In this case, we can set b
i , the baseline for the i th factor, to depend on all other actions in addition to the state.
Let a − i t denote all dimensions other than i in a t and denote the i th baseline by b i ( s t , a − i t ) .
Due to conditional independence, we have
[Equation]
Hence we can use the following gradient estimator
[Equation]
In Appendix E, we show that the methodology also applies to general policy structures (for example, a Gaussian policy with a general covariance structure), where the conditional independence assumption does not hold.
The result is bias-free albeit different baselines.
4. O PTIMAL ACTION - DEPENDENT BASELINE
In this section, we derive the optimal action-dependent baseline and show that it is better than the state-only baseline.
We seek the optimal baseline to minimize the variance of the policy gradient estimate.
First, we write out the variance of the policy gradient under any actiondependent baseline.
Let us deﬁne ∇ η
i
π θ ) :
=
E
[Equation]
[Equation]
and z i := ∇ θ log π θ ( a it | s t ) .
For simplicity of exposition, we make the following assumption:
[Equation]
which translates to meaning that different subsets of parameters strongly inﬂuence different action dimensions or factors
We note that this assumption is primarily for the theoretical analysis to be clean, and is not required to run the algorithm in practice.
In particular, even without this assumption, the proposed baseline is bias-free.
When the assumption holds, the optimal actiondependent baseline can be analyzed thoroughly.
Some examples where these assumptions do hold include multi-agent settings where the policies are conditionally independent by construction, cases where the policy acts based on independent components (Cao et al., 2007) of the observation space, and cases where different function approximators are used to control different actions or synergies (Todorov & Ghahramani, 2004; Todorov et al., 2005) without weight sharing.
The optimal action-dependent baseline is then derived to be:
[Equation]
[Equation]
[Equation]
See Appendix B for the full derivation.
Since the optimal action-dependent baseline is different for different action coordinates, it is outside the family of state-dependent baselines barring pathological cases.
4. S UBOPTIMALITY OF THE OPTIMAL STATE - DEPENDENT BASELINE
How much do we reduce variance over a traditional baseline that only depends on state?
Using
Equation (39) (Appendix C), we show the following improvement
[Equation]
See Appendices C and D for the full derivation.
We conclude that the optimal action-dependent baseline does not degenerate into the optimal state-dependent baseline. Equation 11 states that the variance difference is a weighted sum of the deviation of the per-component score-weighted marginalized Q (denoted Y i ) from the weighted average of all the component scored-weighted marginalized Q values.
This suggests that the difference is particularly large when the Q function is highly sensitive to the actions, especially along those directions that inﬂuence the gradient the most.
Our empirical results in Section 5 additionally demonstrate the beneﬁt of action-dependent over state-only baselines.
4. M ARGINALIZATION OF THE GLOBAL ACTION - VALUE FUNCTION
Using the previous theory, we now consider various baselines that could be used in practice and their associated computational cost.
Marginalized Q baseline Even though the optimal state-only baseline is known, it is rarely used in practice (Duan et al., 2016).
Rather, for both computational and conceptual beneﬁt, the choice of b ( s t ) =
E a t [ ˆ Q ( s t , a t )]
= V ( s t ) is often used.
Similarly, we propose to use b
i ( s t , a − i t ) =
E a it (cid:104)
ˆ Q π θ ( s t , a t ) (cid:105) which is the action-dependent analogue.
In particular, when log probability of each policy factor is loosely correlated with the action-value function, then the proposed baseline is close to the optimal baseline.
[Equation] [Equation]
when E a it (cid:104) z
Ti z i ˆ Q ( s t , a t ) (cid:105)
≈ E a it (cid:2)
z Ti z i (cid:3) E a it (cid:104)
ˆ Q ( s t , a t ) (cid:105) .
This has the added beneﬁt of requiring to learn only one function approximator, for estimating
Q ( s t , a t ) , and implicitly using it to obtain the baselines for each action coordinate.
That is, Q ( s t , a t ) is a function approximating samples ˆ Q ( s t , a t ) .
Monte Carlo marginalized Q baseline After ﬁtting Q π θ ( s t , a t ) we can obtain the baselines through Monte Carlo estimates: [Equation]
[Equation]
where α j ∼ π θ ( a it | s t ) are samples of the action coordinate i .
In general, any function may be used to aggregate the samples, so long as it does not depend on the sample value a it .
For instance, for discrete action dimensions, the sample max can be computed instead of the mean.
Mean marginalized Q baseline Though we reduced the computational burden from learning m functions to one function, the use of Monte Carlo samples can still be computationally expensive.
In particular, when using deep neural networks to approximate the Q-function, forward propagation through the network can be even more computationally expensive than stepping through a fast simulator (e.g. MuJoCo).
In such settings, we further propose the following more computationally practical baseline:
[Equation]
[Equation]
where ¯ a it =
E π θ (cid:2) a it (cid:3) is the average action for coordinate i .
4. F INAL ALGORITHM
The ﬁnal practical algorithm for fully factorized policies is as follows.
Algorithm
Require: number of iterations N , batch size B , initial policy parameters θ
Initialize action-value function estimate ˆ Q π θ ( s t , a t ) ≡ 0 and policy π θ for j in { 1 , . . .
, N } do
Collect samples: ( s t , a t ) t ∈{ 1 ,...,B }
Compute baseline: b i ( s t , a − i t ) =
E a it (cid:104)
ˆ Q π θ ( s t , ( a − i t , a it )) (cid:105) for i ∈ { 1 , . .
. , m } [Equation (14)]
Compute advantages: ˆ A iπ θ ( s t , a t ) = R t − b i ( s t , a − i t ) , ∀ t
Perform a policy update step on θ using ˆ A iπ θ ( s t , a t )
Update action-value function estimate: ˆ Q π θ ( s t , a t ) end for Computing the baseline can be done with either proposed technique in Section 4.4. A similar algorithm can be written for general policies (Appendix E), which makes no assumptions on the conditional independence across action dimensions.
E XPERIMENTS AND R ESULTS
Continuous control benchmarks Firstly, we present the results of the proposed action-dependent baselines on popular benchmark tasks.
These tasks have been widely studied in the deep reinforcement learning community (Duan et al., 2016; Gu et al., 2017; Lillicrap et al., 2016; Rajeswaran et al., 2017b).
The studied tasks include the hopper, half-cheetah, and ant locomotion tasks simulated in MuJoCo (E. Todorov, 2012).
In addition to these tasks, we also consider a door opening task with a high dimensional multi-ﬁngered hand, introduced in Rajeswaran et al. (2017a) to study the effectiveness of the proposed approach in high dimensional tasks.
Figure 1 presents the learning curves on these tasks.
We compare the action-dependent baseline with a baseline that uses only information about the states, which is the most common approach in literature.
We observe that the action-dependent baselines perform consistently better.
A popular baseline parameterization choice is a linear function on a small number of non-linear features of the state (Duan et al., 2016), especially for policy gradient methods.
In this work, to enable a fair comparison, we use a Random Fourier Feature representation for the baseline ( ? Rajeswaran et al., 2017b).
The features are constructed as: y ( x ) = sin( 1 ν Px + φ ) where P is a matrix with each element independently drawn from the standard normal distribution, φ is a random phase shift in [ − π, π ) and, and ν is a bandwidth parameter.
These features approximate the RKHS features under an RBF kernel.
Using these features, the baseline is parameterized as b = w T y ( x ) where x are the appropriate inputs to the baseline, and w are trainable parameters.
P and φ are not trained in this parameterization.
Such a representation was chosen for two reasons: (a) we wish to have the same number of trainable parameters for all the baseline architectures, and not have more parameters in the action-dependent case (which has a larger number of inputs to the baseline); (b) since the ﬁnal representation is linear, it is possible to accurately estimate the optimal parameters with a Newton step, thereby alleviating the results from confounding optimization issues.
For policy optimization, we use a variant of the natural policy gradient method as described in Rajeswaran et al.
(2017b).See
Appendix G for further experimental details. Choice of action-dependent baseline form Next, we study the inﬂuence of computing the baseline by using empirical averages sampled from the Q-function versus using the mean-action of the action-coordinate for computing the baseline (both described in 4.4).
In our experiments, as shown in Figure 2 we ﬁnd that the two variants perform comparably, with the latter performing slightly better towards the end of the learning process.
This suggests that though sampling from the Q-function [Figure] might provide a better estimate of the conditional expectation in theory, function approximation from ﬁnite samples injects errors that may degrade the quality of estimates.
In particular, sub-sampling from the Q-function is likely to produce better results if the learned Q-function is accurate for a large fraction of the action space
[Figure]
but getting such high quality approximations might be hard in practice
[Figure]
High-dimensional action spaces Intuitively, the beneﬁt of the action-dependent baseline can be greater for higher dimensional problems.
We show this effect on a simple synthetic example called m -DimTargetMatching.
The example is a one-step MDP comprising of a single state, S = { 0 } , an m -dimensional action space, A = R m , and a ﬁxed vector b ∈ R m .
The reward is given as the negative (cid:96) 2 loss of the action vector, r ( s, a ) = −(cid:107) a − b (cid:107) 2 .
The optimal action is thus to match the given vector by selecting a = b .
The results for the demonstrative example are shown in Figure 3, which shows that the action-dependent baseline successfully improves convergent more for higher dimensional problems than lower dimensional problems.
Due to the lack of state information, the linear baseline reduces to whitening the discounted returns.
The action-dependent baseline, on the other hand, allows the learning algorithm to assess the advantage of each individual action dimension by utilizing information from all other action dimensions.
Additionally, our algorithm scales well computationally to high-dimensional problems, as shown by our study of the 1000-dimensional action space problem in Figure 3.
[Figure]
Partially observable and multi-agent tasks Finally, we also consider the extension of the core idea of using global information, by studying a POMDP task and a multi-agent task.
We use the blind peg-insertion task which is widely studied in the robot learning literature (Montgomery & Levine, 2016).
The task requires the robot to insert the peg into the hole (slot) [Figure] but the robot is blind to the location of the hole.
Thus, we expect a searching behavior to emerge from the robot, where it learns that the hole is present on the table and performs appropriate sweeping motions till it is able to ﬁnd the hole.
In this case, we consider a baseline that knows the location of the hole.
We observe that a baseline with this additional information enables faster learning.
For the multi-agent setting, we analyze a two-agent particle environment task in which the goal is for each agent to reach their goal, where their goal is known by the other agent and they have a continuous communication channel. Similar training procedures have been employed in recent related works
Lowe et al.
; Levine et al.
Figure 4 shows that including the inclusion of information from other agents into the action-dependent baseline improves the training performance, indicating that variance reduction may be key for multi-agent reinforcement learning
[Figure] [Figure]
C ONCLUSION
An action-dependent baseline enables using additional signals beyond the state to achieve biasfree variance reduction.
In this work, we consider both conditionally independent policies and general policies, and derive an optimal action-dependent baseline for a wide class of problems.
We provide analysis of the variance reduction improvement over non-optimal baselines, including the traditional optimal baseline that only depends on state.
We additionally propose several practical action-dependent baselines which perform well on a variety of continuous control tasks and are demonstrated to give greater improvement for synthetic high-dimensional action problems.
The use of additional signals beyond the local state generalizes to other problem settings, for instance in
POMDP and multi-agent tasks.
In future work, we propose to investigate related methods in such settings on large scale problems. A D ERIVATION OF THE OPTIMAL STATE - DEPENDENT BASELINE
We provide a derivation of the optimal state-dependent baseline, which minimizes the variance of the policy gradient estimate and is based in Greensmith et al.
[Equation]
[Equation]
Note that the ﬁnal term does not depend on b ( · ) by the score function argument and thus does not affect the minimizer
[Equation]
[Equation]
[Equation]
B D ERIVATION OF THE OPTIMAL ACTION - DEPENDENT BASELINE
We derive the optimal action-dependent baseline, which minimizes the variance of the policy gradient estimate.
First, we write out the variance of the policy gradient under any action-dependent baseline.
Recall the following notations: we deﬁne ∇ η
i
π θ ) :=
E
[Equation]
( a it | s
[Equation]
(cid:16) ˆ
[Equation]
( s t , a t )
[Equation]
i ( s t , a − i t ) (cid:17)(cid:105) and z
i := ∇ θ log π θ ( a it | s t ) .
Recall the following assumption:
[Equation]
which translates to meaning that different subsets of parameters strongly inﬂuence different action dimensions or factors
This is true in case of distributed systems by construction, and also true in a single agent system if different action coordinates are strongly inﬂuenced by different policy network channels.
Under this assumption, we have:
[Equation]
where we denote the mean correction term M ij :
=
E ρ π ,π (cid:104)
z i ˆ Q ( s t , a t ) (cid:105) T E ρ π ,π (cid:104)
z
j ˆ Q ( s t , a t ) (cid:105) .
Also let M = (cid:80) i (cid:80) j M ij .
Note that M does not depend on b i ( · ) , and thus does not affect the optimal value.
The step from Equation (21) to Equation (22) additionally results from the score function argument, whereby E ρ π ,π (cid:2)
{+z i b i ( s t , a − i t ) (cid:3) = 
+}
(see Equation (7)).
The overall variance is minimized when each component variance is minimized.
We now derive the optimal baselines b ∗ i ( s t , a − i t ) which minimize each respective component.
[Equation]
M ii
Having written down the expression for variance under any action-dependent baseline
[Equation]
we seek the optimal baseline that would minimize this variance.
[Equation]
The optimal action-dependent baseline is:
[Equation]
C D ERIVATION OF VARIANCE REDUCTION IMPROVEMENT
We now turn to quantifying the reduction in variance of the policy gradient estimate under the optimal baseline derived above.
Let Var ∗ ( ∇ θ η ( π θ )) denote the variance resulting from the optimal action-dependent baseline
[Equation]
and let Var ( ∇ θ η ( π θ )) denote the variance resulting from another baseline b
= ( b i ( s t , a it ))
i ∈ [ m ] , which may be suboptimal or action-independent.
We use the notation:
[Equation]
[Equation]
Finally
[Equation]
deﬁne the variance improvement I b :=
Var ( ∇ θ η ( π θ ))
− Var ∗ ( ∇ θ η i ( π θ )) .
Using these deﬁnitions, the variance can be re-written as:
[Equation]
Furthermore, the variance of the gradient with the optimal baseline can be written as
[Equation]
The difference in variance can be calculated as:
[Equation]
D D ERIVATION OF SUBOPTIMALITY OF THE OPTIMAL STATE - DEPENDENT BASELINE
Using the notation from Appendix C and working off of Equation (39), we have:
[Equation]
E B ASELINES FOR G ENERAL A CTIONS
In the preceding derivations, we have assumed policy actions are conditionally independent across dimensions.
In the more general case, we only assume that there are m factors a 1 t through a mt which altogether forms the action a t .
Conditioned on s t , the different factors form a certain directed acyclic graphical model (including the fully dependent case).
Without loss of generality, we assume that the following factorization holds:
[Equation]
(44) where f ( i ) denotes the indices of the parents of the i th factor.
Let D ( i ) denote the indices of descendants of i in the graphical model (including i itself).
In this case, we can set the i th baseline to be b i ( s t , a [ m ] \ D ( i ) t ) , where [ m ] = { 1 , 2 , . . .
, m } .
In other words, the i th baseline can depend on all other factors which the i th factor does not inﬂuence.
The overall gradient estimator is given by
[Equation]
In the most general case without any conditional independence assumptions, we have f ( i ) =
, i − 1 } , and D ( i ) =
{ i, i + 1 , . . .
, m } .
The above equation reduces to
[Equation]
The above analysis for optimal baselines and variance suboptimality transfers also to the case of general actions.
The applicability of our techniques to general action spaces may be of crucial importance for many application domains where the conditional independence assumption does not hold up, such as language tasks and other compositional domains.
Even in continuous control tasks, such as hand manipulation, and many other tasks where it is common practice to use conditionally independent factorized policies, it is reasonable to expect training improvement from policies without a full conditionally independence structure.
Computing action-dependent baselines for general actions The marginalization presented in
Section 4.4 does not apply for the general action setting.
Instead, m individual baselines can be trained according to the factorization, and each of them can be ﬁtted from data collected from the previous iteration.
In the general case, this means ﬁtting m functions
b i ( s t , a 1 t , . . .
, a i t ) , for i ∈ { 1 , . . .
, m } .
There may also exist special cases like conditional independent actions, for which more efﬁcient training procedures exist.
A closely related example is block diagonal covariance structures (e.g. in multi-agent settings), where we may wish to instead learn an overall Q function and marginalize over block factors.
Another interesting example to explore is sparse covariance structures.
F C OMPATIBILITY WITH GAE
Temporal Difference (TD) learning methods such as GAE (Schulman et al., 2016) allow us to smoothly interpolate between high-bias, low-variance estimates and low-bias, high-variance estimates of the policy gradient.
These methods are based on the idea of being able to predict future returns, thereby bootstrapping the learning procedure.
In particular, when using the value function as a baseline, we have A ( s t , a t ) =
E [ r t + γV
( s t +1 )
− V ( s t )] =
[ r t + γb ( s t +1 ) − b ( s t )]] if b ( s ) is an unbiased estimator for V ( s ) .
GAE uses an exponential averaging of such temporal difference terms over a trajectory to signiﬁcantly reduce the variance of the advantage at the cost of a small bias (it allows us to pick where we want to be on the bias-variance curve).
Similarly, if we use b i ( s t , a − i t ) as an unbiased estimator for E a it
[ ˆ Q ( s t , a t )] , we have:
[Equation]
Thus, the temporal difference error with the action dependent baselines is an unbiased estimator for the advantage function as well.
This allows us to use the GAE procedure to further reduce variance at the cost of some bias.
The following study shows that action-dependent baselines are consistent with TD procedures with their temporal differences being estimates of the advantage function.
Our results summarized in
Figure 5 suggests that slightly biasing the gradient to reduce variance produces the best results, while high-bias estimates perform poorly.
Prior work with baselines that utilize global information (Foerster et al., 2017) employ the high-bias variant.
The results here suggest that there is potential to further improve upon those results by carefully studying the bias-variance trade-off.
[Figure]
corresponding
( λ = 1) has trouble learning initially.
Allowing for a small bias to reduce the variance, corresponding to the intermediate λ = 0 .
97 produces the best overall result consistent with the
G E XPERIMENT DETAILS
Parameters : Unless otherwise stated, the following parameters are used in the experiments in this work: γ = 0 .
995 , λ GAE = 0 .
97 , kl desired = 0 .
Policies : The policies used are 2-layer fully connected networks with hidden sizes=(32, 32).
Initialization : the policy is initialized with Xavier initialization except ﬁnal layer weights are scaled down (by a factor of 100x).
Note that since the baseline is linear (with RBF features) and estimated with a Newton step, the initialization is inconsequential.
Per-experiment conﬁguration :
The following parameters are for both state-only and actiondependent versions of the experiments.
The m -DimTargetMatching experiments use a linear feature baseline.
Tasks
Benchmarks
Hand task
Peg Insertion
CommunicateTarget m -DimTargetMatching
Trajectories
Horizon
RBF features
N/A