On All-Action Policy Gradients
Abstract
In this paper, we analyze the variance of stochastic policy gradient with many action samples per state (all-action SPG).
We decompose the variance of SPG and derive an optimality condition for all-action SPG.
The optimality condition shows when all-action SPG should be preferred over single-action counterpart and allows to determine a variance-minimizing sampling scheme in SPG estimation.
Furthermore, we propose dynamics-all-action (DAA) module, an augmentation that allows for all-action sampling without manipulation of the environment.
DAA addresses the problems associated with using a Q-network for all-action sampling and can be readily applied to any on-policy SPG algorithm.
We find that using DAA with a canonical on-policy algorithm (PPO) yields better sample efficiency and higher policy returns on a variety of continuous action environments.
Introduction
Stochastic policy gradient (SPG) is a method of optimizing stochastic policy through gradient ascent in the context of reinforcement learning (RL) [43, 39, 28, 37].
When paired with powerful function approximators, SPG-based algorithms have proven to be one of the most effective methods for achieving optimal performance in Markov Decision Processes (MDPs) with unknown transition dynamics [35, 32].
Unfortunately, exact calculation of the gradient is unfeasible and thus the objective has to be estimated [39].
Resulting variance is known to impact the learning speed, as well as performance of the trained agent [19, 41]
On-policy sample efficiency (ie.
the amount of required interactions with the environment to achieve certain performance) is particularly affected by variance, as the gradient has to be evaluated over long trajectories to achieve sufficient quality of the SPG estimate [25].
As such, a variety of methods for on-policy SPG variance reduction have been proposed.
Method of most prominence, baseline variance reduction, has been shown to improve algorithms stability and became indispensable to contemporary SPG implementations [28, 34].
Alternative approaches include Q-value bootstrapping [10], reducing the effect of long-horizon stochasticity via small discount [3], increasing number of samples via parallel agents [25] or using all-action estimators [1, 21, 29, 6].
In all-action SPG (AA-SPG), the gradient is estimated using more than one action sample per state, without including the follow-up states of those additional actions in the gradient calculation.
The method builds upon conditional Monte-Carlo and yields variance that is smaller or equal to that of single-action SPG given fixed trajectory length [4].
Such additional action samples can be drawn with replacement [6] or without [21].
Furthermore, they can be generated by rewinding the environment [33] or by using a parametrized Q-value approximator [1].
Whereas using Q-value approximator does not incur environment interaction cost, it biases the gradient estimate [29].
In contrast to that, drawing additional action samples from the environment does not bias the gradient, but creates an interaction cost which is unacceptable in a lot of settings.
This leads to a question: given a trajectory of length T and some fixed cost of sampling actions, is SPG variance more favourable when spending the resources on sampling additional actions or extending the trajectory ?
The contributions of this paper are twofold.
Firstly, we analyze SPG variance theoretically.
We quantify the variance reduction stemming from adopting AA estimation strategy as compared to extending the trajectory of a single-action agent.
We prove that AA-SPG is a viable scheme for estimating gradients in RL problems.
Secondly, we propose an implementation of AA, which we refer to as dynamics-all-action module (DAA).
DAA leverages a learned dynamics model to sample state-action gradients and can be used in conjunction with any on-policy SPG algorithm.
We show that augmenting SPG algorithm with DAA yields better sample efficiency and higher reward sums than Q-network all-action SPG, as well as single-action SPG baseline.
We validate empirically our approach by augmenting state of the art on-policy SPG algorithm [35] with DAA and testing its performance on a variety of continuous control tasks.
Background
A Markov Decision Process (MDP) [30] is a tuple ( S, A, R, p, γ ) , where S is a countable set of states, A is a countable set of actions, R ( s, a ) is the state-action reward, p ( s ′ | s, a ) is a transition kernel (with the initial state distribution denoted as p 0 ) and γ ∈ (0 , 1] is a discount factor.
A policy π ( a | s ) is a state-conditioned action distribution.
Given a policy π , MDP becomes a Markov reward process with a transition kernel p π ( s ′ | s ) =
(cid:82) a π ( a | s ) p ( s ′ | s, a ) da , which we refer to as the underlying Markov chain.
The underlying Markov chain is assumed to have finite variance, an unique stationary distribution denoted as p π 0 [31, 19] and an unique discounted stationary distribution denoted as p
π ∗ .
Interactions with the MDP according to some policy π are called trajectories and are denoted as τ πT ( s t ) =
(( s t , a t , r t ) , ( s t +1 , a t +1 , r t +1 ) , ..., ( s t + T , a t + T , r t + T )) , where a t ∼ π ( a t | s t ) , r t ∼ R ( s, a ) and s t +1 ∼ p ( s t +1 | s t , a t ) .
If T < ∞ , then the trajectory is referred to as finite.
The value function V π ( s ) =
E τ π ∞ ( s ) [ (cid:80) ∞ t =
γ t R ( s t , a t )] and Q-value function Q π ( s, a ) =
E τ π ∞ ( s | a ) [ (cid:80) ∞ t =
γ t R ( s t , a t )]
=
R ( s, a ) + γ E s ′ ∼ p
( s ′ | s,a )
[ V π ( s ′ )] sample a t according to some fixed policy π .
State-action advantage is defined as A π ( s, a ) =
Q π ( s, a ) − V π ( s ) .
An optimal policy is a policy that maximizes discounted total return J = (cid:82) s {+
+} V π
( s 0 )
ds 0 .
Given a policy parametrized by θ , the values of θ can be updated via SPG [39] θ ← θ + ∇ θ J ∗ which is is given by [38]:
[Equation]
As such, SPG is proportional to a double expectation of Q π ( s, a ) ∇ θ log π ( a | s ) , with the outer expectation taken wrt.
the discounted stationary distribution p π ∗ and the inner expectation taken wrt. policy π . By assuming ergodicity of the underlying Markov chain, policy gradient can be estimated via a trajectory generated according to the policy [28, 33, 27, 44]:
[Equation]
Where ∇ J denotes the SPG estimator, ∇ J ( s t , a t ) =
γ t Q π ( s t , a t ) ∇ log π ( a t | s t ) and s t , a t ∼ τ πT .
In the setup above, the outer expectation of Equation 1 is estimated via Monte-Carlo [23] with T state samples drawn from the non-discounted stationary distribution p π 0 ; and the inner expectation is estimated with a single action per state drawn from the policy π ( a | s ) .
The resulting variance can be reduced to a certain degree by a control variate, with state value being a popular choice for such baseline [34].
Then, the Q-value from Equation 1 is replaced by A π ( s t , a t ) .
If state value is learned by a parametrized approximator, it is referred to as the critic .
Critic bootstrapping [10] is defined as Q π ( s, a ) =
R ( s, a ) + V π ( s ′ ) with s ′ ∼ p ( s ′ | s, a ) and can be used to balance the bias-variance tradeoff of Q-value approximations.
Given a control variate, the variance of policy gradient can be further reduced by approximating the inner integral of Equation 2 with a quadrature of N > 1 action samples. Then ∇ J is equal to:
[Equation]
Where a nt denotes the n th action sampled at state s t .
Furthermore, MDP transitions are conditioned only on the first action performed (ie.
p π ( s t +1 | s t , a nt ) ̸
= p π ( s t +1 | s t ) ⇐⇒ n = 0 ). Such approach is referred to as all-action policy gradient or expected policy gradient [1, 29, 6].
As follows from law of iterated expectations, all-action (AA) estimator is unbiased and yields lower or equal variance as compared to single-action SPG with equal trajectory length [29].
Since the policy log-probabilities are known, using AA estimation requires approximating the Q-values of additional action samples. As such, AA is often implemented by performing rollouts in a rewinded environment [33, 20, 21] or by leveraging a Q-network at the cost of bias [1, 29, 6].
The variance reduction stemming from using AA-SPG has been shown to increase both performance and sample efficiency of SPG algorithms [6].  Variance of Stochastic Policy Gradient
Throughout the section, we assume no stochasticity induced by learning Q-values and we treat
Q-values as known.
Furthermore, when referring to SPG variance, we refer to the diagonal of policy parameter variance-covariance matrix.
As shown by [29, 6], given fixed trajectory length T , AA-SPG variance is smaller or equal to the variance of regular SPG.
However, approximating the inner expectation of SPG always uses some resources (ie.
compute or environment interactions).
Such resources could be used to reduce the SPG variance through other means, for example by extending the trajectory length.
To this end, we build upon the result of [29, 6] by comparing the variance reduction effect stemming from employing AA-SPG as opposed to using regular single-action SPG with an extended trajectory length.
The variance of SPG, denoted as V , is given by the variance of empirical mean of T dependent random variables sampled from the MDP [17, 5]:
[Equation]
As follows from the ergodic theorem [26], conditional probability of visiting state s t given starting in state s 0 with action a 10 approaches the non-discounted stationary distribution p π 0 exponentially fast as t grows lim t →∞ p ( s t | s 0 , a 1 0 )
=
p π 0 ( s t ) .
Therefore, Cov t ≥ Cov t +1 , as well as lim t →∞ Cov t = 0 .
Equation 4 shows the well known result that increasing the trajectory length
T decreases V .
This result is key to the success of parallel actor-critic implementations [25, 45].
Unfortunately, the form above does not allow us to inspect the effect of using all-action SPG (ie.
sampling many actions per state in the trajectory).
To quantify such effect, we decompose V into sub-components.
Lemma 1 - SPG variance decomposition Given a finite trajectory τ πT and SPG estimation using T state samples and N action samples per state, V can be decomposed into :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The components can be grouped according to dependency on N :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
For derivation see Appendix A. Given N = 1 , the variance simplifies to a single-action SPG case.
The statement shows that SPG variance can be decomposed into: variance of the underlying Markov chain, which marginalizes policy stochasticity and is decreased only by increasing the trajectory length ( T ); and starting state-conditioned variance of the policy and covariance correction due to policy stochasticity, which indeed is reduced by both state and action samples ( T and N respectively).
[Table]
component
Cartpole
Ball
Reacher
Finger
Cheetah
Walker
[Equation]
[Equation]
[Equation]
[Table]
Table 1 shows sample variance components of SPG estimator.
In particular, the table shows that with Q-values marginalized, the policy is responsible for around 90% of SPG variance in tested environments.
We proceed with analytical analysis of the variance reduction stemming increasing N and T .
For derivation of the Lemma below, see Appendix A.
Lemma 2 - SPG variance reduction wrt.
N and T Given a finite trajectory τ πT and SPG estimation using T state samples and N action samples per state, the SPG variance reduction stemming from increasing N by 1 is (denoted as ∆ N ) and the SPG variance reduction stemming from increasing the length of trajectory to T + δT with δ ∈ (0 , ∞ ) (denoted as ∆ T ) are equal to : ∆ N = −
[Equation]
In Equations 9 and 10, Cov t =
[-E s ∼ p π 
-] Cov s t ,a t ∼ τ πT [ ∇ θ J ( s 0 , a 0 ) , ∇ θ J ( s t , a t )] .
Equations above show the diminishing variance reduction stemming from increasing N by 1 or T by δT .
Incorporating δ captures the notion of relative costs of increasing N and T .
If δ = 1 , then the cost of increasing N by 1 (sampling one more action per state in trajectory) is equal to doubling the trajectory length.
Now, it follows that AA-SPG is an optimal strategy only if ∆ N ≤ ∆ T for given values of N , T and δ .
Theorem 1 - AA-SPG optimality condition for MDPs Given a finite trajectory τ πT and SPG estimation using T state samples and N action samples per state, the SPG variance reduction stemming from increasing N by 1 is bigger than SPG variance reduction stemming from increasing T by δT for δ = 1 and N = 1 when :
[Equation]
For derivation with N ≥ 1 and δ ∈ (0 , ∞ ) see
Appendix A.
The theorem represents an optimality condition under which it is optimal to switch from regular SPG (AA-SPG with N = 1 ) to AA-SPG with N = 2 .
Surprisingly, the optimality condition for δ = 1 and N = 1 is dependent solely on the covariance structure of the data.
As follows from Theorem 1, AA-SPG is optimal when the weighted sum of MDP covariances exceeds the variance of the Markov Chain underlying the MDP.
As follows,
AA-SPG is most effective in problems where action-dependent covariance constitutes a sizeable portion of the total SPG variance (ie.
problems where future outcomes largely depend on actions taken in the past and consequently, ∇ θ J ( s t + k , a t + k ) largely depends upon a t ).
Figure below shows the potential of AA-SPG estimators on complex continuous action environments.
(a) Cartpole, swingup (b) Cup, catch (c) Reacher, easy (d) Finger, spin (e ) Cheetah, run
[Figure]
Corollary 1 - AA-SPG optimality condition for contextual bandits Given a finite trajectory τ πT and SPG estimation using T state samples and N action samples per state, the SPG variance reduction from increasing ∆ N = 1 is bigger than SPG variance reduction from ∆ T = δT when :
[Equation]
Corollary above is a specific case of Theorem 1.
By assuming a contextual bandit problem, the covariances are equal to zero and the optimality condition is vastly simplified.
As follows from the definition of variance, the LHS of Equation 11 is greater or equal to 0 .
However, the RHS becomes negative when δN > 1 .
Since N ≥ 1 , it follows that AA-SPG is never optimal for bandits if δ ≥(ie.
the cost of acquiring an additional action sample is equal or greater than to the cost of acquiring an additional state sample).
Whereas the efficiency of AA-SPG for contextual bandits is restricted,
Theorem 1 shows that AA-SPG can be a preferable strategy for gradient estimation in MDPs.
We leave determining the optimality condition with sampled Q-values for future work.
Dynamics-all-action SPG
One of the conclusions of our analysis is that AA-SPG estimators are guaranteed to be useful when δ ≈ 0 .
This is reflected in the existing AA-SPG methods, which utilize a Q-network (QAA-SPG) to approximate additional samples [1, 29, 6].
By using the interactions with environment as the point of reference, QAA-SPG methods assume δ = 0 .
However, using a Q-network to approximate additional action samples yields bias.
Whereas the bias can theoretically be reduced to zero, the conditions required for such bias annihilation are unrealistic [29].
Being fully dependent on policy, Q-network has to learn a constantly changing target [42].
Furthermore, single-action supervision challenges generating informative samples for multiple actions from Q-network.
This results in unstable training when Q-network is used to bootstrap the policy gradient [24, 42, 10, 12].
Inspired by the success of model-based RL, we propose dynamics-all-action SPG (DAA-SPG) - an AA module that, contrary to existing all-action methods, does not require a Q-network for SPG approximation.
Instead, DAA leverages a learned dynamics model for evaluation of the inner expectation from SPG.
In DAA, the Q-value of the additional actions are approximated with a critic-bootstrapped finite trajectory generated by a latent dynamics model, consisting of transition and reward networks [11, 13, 18, 9, 32].
The main advantage of such approach when compared to QAA-SPG is that both reward and transition networks learn stationary targets throughout training, thus offering better convergence properties and lower bias.
DAA does not assume an exact form of gradient calculation.
As such, it can be used in conjunction with any on-policy SPG algorithm (eg.
A2C, TRPO or PPO).
[Figure]
yields The proposed method builds upon two bodies of work: all-action SPG and model-based RL.
Previous work on all-action SPG used either a Q-network [1, 29, 6] or rollouts achieved by environment rewinding [33, 21].
In comparison, DAA-SPG leverages a learned dynamics model for sample simulation.
In contrast to model-based RL methods, we do not learn the policy inside the dynamics model [11, 18, 32, 14].
Instead, similarly to actor-critic methods, the policy gradient approximation is anchored on the trajectory stemming from the real environment, with the dynamics model used only to refine the estimator (ie. reduce its variance through Monte-Carlo conditioning).
Experiments
Our implementations are slight modifications of the PPO implementation provided in [15].
Furthermore, in all experiments and across all algorithms, we use the optimized hyperparameters provided in [15].
There are only two differences between our implementation and the baseline implementation.
Firstly, we do not perform advantage normalization.
Although it is known to sometimes increase the performance of SPG agents, it has no grounding in SPG theory and impacts the variance structure of the SPG estimate.
We point the reader to Appendix B for results that use advantage normalization.
Finally, we slightly increase the hidden layer sizes and switch activation functions from Tanh to ReLUs.
All architecture details are provided in Appendix C. To simplify our DAA-PPO implementation, we do not use the state of the art transition/reward network learning schemes proposed in Dreamer [13] or SimPLe [18].
Instead, we use a simple action-conditioned two-layer MLP architecture trained using L2 loss on off-policy data sampled from an experience buffer.
While using more advanced dynamics models would likely further refine the performance of DAA-SPG, we fill that it would give an unnecessary advantage over the baseline QAA-PPO.
In QAA-PPO implementation, the Q-network is a two-layer MLP trained using L2 loss with TD( λ ) advantages as targets.
5.1 Performance of DAA-PPO as compared to QAA-PPO and PPO
We investigate the performance of DAA-SPG on continuous action tasks from DeepMind Control
Suite [40].
Whereas learning a latent dynamics model from images was shown to work [11, 13, 32], it is known to offer performance benefits over model-free counterparts stemming from backpropagation of additional non-sparse loss functions [16, 36, 47].
To mitigate such performance benefits for DAA-SPG, we use proprioceptive state representations given by the environment, with transition and reward network working directly on such state representations
[Figure]
Figure 3 shows the performance of regular PPO, QAA-PPO and DAA-PPO, with all algorithms using consistent trajectory length of T = 1024 and number of action samples N = 4 (except PPO for which N = 1 ).
In the experimental setup, PPO gradient is unbiased, but has strictly greater variance that QAA-PPO and DAA-PPO (as follows from Lemma 2).
Furthermore, QAA-PPO and DAA-PPO gradients have proportional variance, but differ only with bias (QAA-PPO bias stems from Q-network; DAA-PPO bias stems from dynamics model).
Figure 3 points towards more favourable bias-variance mix of DAA-PPO as compared to QAA-PPO.
Furthermore, the bias of DAA-PPO can be further reduced by employing more advanced schemes for dynamics model learning.
In Appendix B we conduct variety of ablation studies, where we compare the performance of DAA-PPO for various amounts of simulated action samples, as well as different simulation depth for Q-value estimation.
5.2 Simulating additional actions as compared to simulating additional states
When using a dynamics model to simulate actions, the bias of reward model, as well as compounded errors of transitions translate only to the bias of Q-values associated with the simulated action samples.
Since the exact log-probabilities and underlying state representations are known, the resulting gradient is still largely approximated using unbiased, exact values.
In contrast, using a dynamics model to simulate state samples requires not only the approximation of Q-values, but also prediction of the state representations.
The inevitable error of state prediction thus anchors the gradient upon log-probability values which in fact should be associated with a different state.
Such perspective is supported by the Lipschitz continuity analysis of approximate MDP models [2, 9].
Note that using dynamics model for simulating additional states in policy gradient approximation is the dominant approach in leveraging dynamics models for policy learning [18, 13, 7].
We therefore hypothesize that using dynamics model for AA estimation might yield more favourable bias-variance tradeoff as compared to using dynamics model to sample additional states given a fixed simulation budget.
To this end, we compare performance of DAA-PPO with N = 4 to an agent that uses the dynamics model to simulate additional states (as compared to DAA-PPO that uses the dynamics model to simulate additional actions).
As such, we compare DAA-PPO with T = 1024 and N = 4 to PPO with T = 1024 + T ′ , where T ′ is simulated by the dynamics model and T ′ =
N ∗ 1024 = 4096 .
[Figure]
Related work
All-action SPG
The idea of AA-SPG was proposed in an unfinished preprint 1 by Sutton et al.
Later, the topic was expanded upon by several authors.
TRPO [33] ’vine procedure’ uses multiple without-replacement action samples per state generated via environment ’rewinding’.
The without-replacement PG estimator was further refined by [21, 20], who also proposed to use the without-replacement samples as a free baseline.
MAC [1] calculates the inner integral of SPG exactly (ie.
sample the entire action space for given states) using Q-network, with the scheme applicable only to discrete action spaces and tested on simple environments.
Similarly, [29] propose to estimate the inner integral with a quadrature of N samples given by a Q-network.
The authors also derive basic theoretical properties of AA-SPG.
Besides expanding on the theoretical framework, [6] propose an off-policy algorithm that, given a Gaussian actor and quadratic critic, can compute the inner integral analytically.
Model-based RL ME-TRPO [22] leverages ensemble of environment models to increase the sample efficiency of TRPO.
WM [11] uses environment interactions to learn the dynamics model, with the policy learning done via evolutionary strategies inside the dynamics model.
Similarly,
SimPLe [18] learns the policy by simulating states via the dynamics model.
Dreamer [13, 14] refines the dynamics model learning by proposing a sophisticated joint learning scheme for recurrent transition and discrete state representation models, but the policy learning is still done by simulating states inside the dynamics model.
Notably, Dreamer was shown to solve notoriously hard Humanoid task [46].
MuZero [32] leverages the dynamics model to perform Monte-Carlo tree search inside the latent model.
Perhaps the closest to the proposed approach is MBVE [8].
There, an off-policy DPG agent uses the dynamics model to estimate n -step Q-values and thus refine the approximation.
Conclusions
In this paper, we analyzed variance of the SPG estimator mathematically.
We showed that it can be disaggregated into sub-components dependent on policy stochasticity, as well as the components which are dependent solely on the structure of the Markov process underlying the policy-embedded
MDP.
By optimizing such components with respect to the number of state and action samples, we derived an optimality condition which shows when AA-SPG is a preferable strategy as compared to traditional, single-action SPG.
We used the result to show the difficult conditions AA-SPG has to meet to be an optimal choice for the case of contextual bandit problems.
We hope that those theoretical results will reinvigorate research into AA estimation.
Furthermore, we proposed DAA-SPG module - an approach that leverages dynamics models for AA estimation.
We evaluated its performance against QAA-SPG and SPG baselines. Our experiments showed that despite the proposed method simplicity, it compares favourably in terms of both sample efficiency and final performance on most of the tested environments and for a wide range of hyperparameter settings.
Finally, we discussed the bias-variance trade-off induced by using dynamics model to simulate action samples (DAA-SPG), as compared to using dynamics model to simulate state samples, as is done in variety of the recent model-based RL approaches.
We release the code used for experiments under this url.
Acknowledgements
We would like to thank Witold Bednorz, Piotr Miło´s and Łukasz Kuci´nski for valuable discussions and notes.
Marek Cygan is cofinanced by National Centre for Research and Development as a part of EU supported Smart Growth Operational Programme 2014-2020 (POIR.01.01.01-00-0392/17-00). The experiments were performed using the Entropy cluster funded by NVIDIA, Intel, the Polish National Science Center grant UMO-2017/26/E/ST6/00622 and ERC Starting Grant TOTAL.
A Appendix A - derivations
To simplify the notation, we define:
[Equation]
Furthermore, we write:
[Equation]
With:
[Equation]
For convenience
[Equation]
throughout the Appendix we will assume finite state and action spaces.
However, same reasoning works for continuous spaces.
For the sake of completeness [Equation] we include derivation of expected value of the AA-SPG estimator defined in Equation 3:
[Equation]
A.1 Derivation of Lemma
Following the AA-SPG definition outlined in Equation [Equation] V 1 is equal to:
[Equation]
Thus:
[Equation] [Equation]
The above result for N = 1 is reported in [29], noting it as stemming from law of total variance.
However, we could not find the proof in existing literature.
We proceed with calculation of V 2 for N ≥ 1 :
[Table]
[Equation]
Where p πt ( s t | s 0 , a 10 ) denotes the t step transition kernel conditioned on s 0 and a 10 (ie.
the first sampled action in s 0 )
[Equation]
Thus
[Equation]
the t th covariance of AA-SPG is equal to:
[Equation]
Combining Equation 17 with Equation 19 and Equation 13 concludes the derivation of Lemma
[Equation]
A.2 Derivation of Lemma
Since N is defined to be a natural number, we calculate the variance reduction effect stemming from increasing N via the forward difference operator:
[Equation]
We also use the shorthand notation:
[Equation]
Thus:
[Equation]
We proceed with calculation of the forward difference:
[Equation]
Var
[Equation]
Similarly
[Equation]
we calculate ∆ T V :
[Equation]
Now
[Equation]
we assume that the trajectory length guarantees reaching a regenerative state [Equation] and thus (cid:80) T + δT −k = T
[Equation]
[Equation]
Combining Equation 27 with Equation 17 concludes derivation of Lemma 2.
A.3 Derivation of Theorem
We start the derivation by stating that AA-SPG is advantageous in terms of variance reduction as compared to increased trajectory length SPG when − ∆ N ≥ − ∆ T .
As such:
[Equation]
We use Equations 17 and 19 to expand the RHS:
[Equation]
Now [Equation] we use Equations 28 and 29.
Furthermore [Equation] we move all terms dependent on the policy to the LHS:
[Equation]
Now, in order to recover the Corollary [Equation] we assume a contextual bandit setup (ie.
p π ( s ′ | s )
=
p π ( s ′ ) ).
Then:
[Equation]
Which is equivalent to:
[Equation]
We proceed with the derivation for the MDP setup, where p π ( s ′ | s ) ̸
= p π ( s ′ ) .
We write N = 1 , which implies that we start in the regular single-action SPG setup.
Furthermore, we assume δ = 1 , which according to the setup implies equal cost of sampling additional action and state samples.
Thus, Equation 30 simplifies to:
[Equation]
Which concludes the derivation of Theorem [Equation]
B Appendix B - Ablation studies
We start with analysis of the effect of increase of N .
Since simulating action samples with a dynamics model induces bias, we hypothesize that there exists some value of N after which further increase of N will deteriorate the learning.
Figure 5 shows DAA-PPO learning curves for different values of N .
Interestingly, we see that N = 4 chosen for the experiment is an extremely conservative value and further significant performance boost is possible via the increase of N .
[Figure]
As follows from Figure 5, increasing N in DAA-PPO yields performance benefits due to further decrease in variance.
Such improvement is not monotonic, as at some point the bias of dynamics model might peturb the learning process (ie.
N = 64 in Reacher, easy).
Furthermore, we analyse the effect of the simulation horizon (depth of simulation for Q-value estimation) on DAA-PPO learning curves.
The results are shown in Figure
[Figure]
As can be seen in Figure 6, DAA-PPO performance degenerates for larger simulation horizons for Q-value estimation.
For larger simulation horizons, the Q-value estimations are largely dependent on the dynamics model, which might result in extreme levels of bias early in the training.
This effect might perhaps be alleviated by using more advanced dynamics model architectures or some sort of annealing scheme, which would increase the simulation horizon as the dynamics model becomes more stable.
We leave testing those hypotheses for future work.
Results for the above ablations were generated with 20 random seeds.
Finally, we investigate the effect of using advantage normalization in conjunction with DAA.
Figure below shows the learning curves for PPO and DAA-PPO when using advantage normalization.
[Figure]
As follows from Figure 7, using advantage normalization with DAA-PPO further boosts the performance of the proposed algorithm.
C Appendix C - implementation details
[Table]