A C LOSER L OOK AT D EEP P OLICY G RADIENTS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study how the behavior of deep policy gradient algorithms reﬂects the conceptual framework motivating their development.
To this end, we propose a ﬁnegrained analysis of state-of-the-art methods based on key elements of this framework: gradient estimation, value prediction, and optimization landscapes.
Our results show that the behavior of deep policy gradient algorithms often deviates from what their motivating framework would predict: the surrogate objective does not match the true reward landscape, learned value estimators fail to ﬁt the true value function, and gradient estimates poorly correlate with the “true” gradient.
The mismatch between predicted and empirical behavior we uncover highlights our poor understanding of current methods, and indicates the need to move beyond current benchmark-centric evaluation methods
I NTRODUCTION
Deep reinforcement learning (RL) is behind some of the most publicized achievements of modern machine learning (Silver et al., 2017; OpenAI, 2018;
Dayarathna et al., 2016; OpenAI et al., 2018).
In fact, to many, this framework embodies the promise of the real-world impact of machine learning.
However, the deep RL toolkit has not yet attained the same level of engineering stability as, for example, the current deep (supervised) learning framework.
Indeed, recent studies demonstrate that state-of-the-art deep RL algorithms suffer from oversensitivity to hyperparameter choices, lack of consistency, and poor reproducibility (Henderson et al., 2017).
This state of affairs suggests that it might be necessary to re-examine the conceptual underpinnings of deep RL methodology.
More precisely, the overarching question that motivates this work is:
To what degree does current practice in deep RL reﬂect the principles informing its development?
Our speciﬁc focus is on deep policy gradient methods, a widely used class of deep RL algorithms.
Our goal is to explore the extent to which state-of-the-art implementations of these methods succeed at realizing the key primitives of the general policy gradient framework.
Our contributions.
To this end, we take a broader look at policy gradient algorithms and their relation to their underlying framework.
With this perspective in mind, we perform a ﬁne-grained examination of key RL primitives as they manifest in practice.
Concretely, we study:
• Gradient Estimation: we ﬁnd that even when agents improve in reward, their gradient estimates used in parameter updates poorly correlate with the “true” gradient.
We additionally show that gradient estimate quality decays with training progress and task complexity.
Finally, we demonstrate that varying the sample regime yields training dynamics that are unexplained by the motivating framework and run contrary to supervised learning intuition.
• Value Prediction: our experiments indicate that value networks successfully solve the supervised learning task they are trained on, but do not ﬁt the true value function.
Additionally, employing a value network as a baseline function only marginally decreases the variance of gradient estimates compared to using true value as a baseline (but still dramatically increases agent’s performance compared to using no baseline at all).
• Optimization Landscapes: we show that the optimization landscape induced by modern policy gradient algorithms is often not reﬂective of the underlying true reward landscape, and that the latter is frequently poorly behaved in the relevant sample regime.
Overall, our results demonstrate that the motivating theoretical framework for deep RL algorithms is often unpredictive of phenomena arising in practice.
This suggests that building reliable deep RL algorithms requires moving past benchmark-centric evaluations to a multi-faceted understanding of their often unintuitive behavior.
We conclude (in Section 3) by discussing several areas where such understanding is most critically needed.
E XAMINING THE P RIMITIVES OF D EEP P OLICY G RADIENT A LGORITHMS
In this section, we investigate the degree to which our theoretical understanding of RL applies to modern methods.
We consider key primitives of policy gradient algorithms: gradient estimation, value prediction and reward ﬁtting.
In what follows, we perform a ﬁne-grained analysis of state-ofthe-art policy gradient algorithms (PPO and TRPO) through the lens of these primitives—detailed preliminaries, background, and notation can be found in Appendix A.1.
2. G RADIENT ESTIMATE QUALITY
A central premise of policy gradient methods is that stochastic gradient ascent on a suitable objective function yields a good policy.
These algorithms use as a primitive the gradient of the surrogate objective function:
[Equation]
where in the above we use standard RL notation (see Appendix A.1 for more details).
An underlying assumption behind these methods is that we have access to a reasonable estimate of this quantity.
This assumption effectively translates into an assumption that we can accurately estimate the expectation above using an empirical mean of ﬁnite (typically ∼ 10 3 ) samples.
Evidently (since the agent attains a high reward) these estimates are sufﬁcient to consistently improve reward—we are thus interested in the relative quality of these gradient estimates in practice, and the affect of gradient quality on optimization.
[Figure]
The vertical line (at x = 2 K) indicates the sample regime used for gradient estimation in standard implementations of policy gradient methods.
In general, it seems that obtaining tightly concentrated gradient estimates would require signiﬁcantly more samples than are used in practice, particularly after the ﬁrst few timesteps.
For other tasks – such as Walker2d-v2 and Hopper-v2 – the plots (seen in Appendix Figure 10) have similar trends, except that gradient variance is slightly lower.
How accurate are the gradient estimates we compute?
To answer this question, we examine two of the most natural measures of estimate quality: the empirical variance and the convergence to the “true” gradient.
To evaluate the former, we measure the average pairwise cosine similarity between estimates of the gradient computed from the same policy with independent rollouts (Figure 1).
We evaluate the latter by ﬁrst forming an estimate of the true gradient with a large number of state-action pairs.
We then examine the convergence of gradient estimates to this “true” gradient (which we once again measure using cosine similarity) as we increase the number of samples (Figure 2).
[Figure]
For other tasks – such as Walker2d-v2 and Hopper-v2 – the plots (seen in Appendix Figure 11) have similar trends, except that gradient estimation is slightly better.
We observe that deep policy gradient methods operate with relatively poor estimates of the gradient , especially as task complexity increases and as training progresses (contrast Humanoid-v2, a “hard” task, to other tasks and contrast successive checkpoints in Figures 1 and 2).
This is in spite of the fact that our agents continually improve throughout training, and attain nowhere near the maximum reward possible on each task.
In fact, we sometimes observe a zero or even negative correlation in the relevant sample regime 1 .
While these results might be reminiscent of the well-understood “noisy gradients” problem in supervised learning (Robbins & Monro, 1951; d’Aspremont, 2008; Kawaguchi, 2016; Safran & Shamir, 2018; Livni et al., 2014; Keskar et al., 2016; Hochreiter & Schmidhuber, 1997), we have very little understanding of how gradient quality affects optimization in the substantially different reinforcement learning setting.
For example:
• The sample regime in which RL algorithms operate seems to have a profound impact on the robustness and stability of agent training —in particular, many of the sensitivity issues reported by Henderson et.
al.
(Henderson et al., 2017) are claimed to disappear (Sutskever, 2018) in higher-sample regimes.
Understanding the implications of working in this sample regime, and more generally the impact of sample complexity on training stability remains to be precisely understood.
• Agent policy networks are trained concurrently with value networks (discussed more in the following section) meant to reduce the variance of gradient estimates.
Under our conceptual framework, we might expect these networks to help gradient estimates more as training progresses, contrary to what we observe in Figure 1.
The value network also makes the now two-player optimization landscape and training dynamics even more difﬁcult to grasp, as such interactions are poorly understood.
• The relevant measure of sample complexity for many settings (number of state-action pairs) can differ drastically from the number of independent samples used at each training iteration (the number of complete trajectories).
The latter quantity (a) tends to be much lower than the number of state-action pairs, and (b) decreases across iterations during training
All the above factors make it unclear to what degree our intuition from classical settings transfer to the deep RL regime.
And the policy gradient framework, as of now, provides little predictive power regarding the variance of gradient estimates and its impact on reward optimization.
[Figure]
Optimization in higher-sample regimes.
What happens to agent training dynamics when we sample more state-action pairs at each step?
In Figure 3, we show the training curves of Humanoidv2 agents with proximal policy optimization (PPO).
Across agents, we vary the learning rate between 5 · 10 − 5 and 1 . 25 · 10 − 4 , and the number of state-action pairs sampled per step between 2 · 10 3 and 2 · 10 4 .
The resulting agent behavior is rather unintuitive in a variety of ways: (a) the optimal learning rate for the small-sample regime is higher than the optimal highest sample regime learning rate; in fact, (b) the “inverse linear scaling rule” Smith et al.
Keskar et al.
(2017) between learning rate and batch size observed in supervised deep learning is not at all observed in this regime; and, remarkably, (c) at even slightly higher learning rates than optimal, the agents in the higher sample regime peak and then immediately consistently deteriorate in reward for the rest of training.
Our results indicate that despite having a rigorous theoretical framework for RL, we lack a crisp understanding of the structure of the reward landscape and optimization process.
2. V ALUE PREDICTION
Our ﬁndings from the previous section motivate a deeper look into gradient estimation.
After all, the policy gradient in its original formulation (Sutton et al., 1999) is known to be hard to estimate, and thus algorithms employ a variety of variance reduction methods.
The most popular of these techniques is a baseline function.
Concretely, an equivalent form of the policy gradient is given by:
[Equation]
where b ( s t ) is some ﬁxed function of the state s t .
A canonical choice of baseline function is the value function V π ( s ) , the expected return from a given state (more details and motivation in A.1):
[Equation]
Indeed, ﬁtting a value-estimating function (Schulman et al., 2015c; Sutton & Barto, 2018) (a neural network, in the deep RL setting) and using it as a baseline function is precisely the approach taken by most deep policy gradient methods.
Concretely, one trains a value network V πθ t such that:
[Equation]
where V π θ t − 1 ( s t ) are estimates given by the last value function, and A t is the advantage of the policy, i.e. the returns minus the estimated values.
(Typically, A t is estimated using generalized advantage estimation, as described in (Schulman et al., 2015c).)
Our ﬁndings in the previous section prompt us to take a closer look at the value network and its impact on the variance of gradient estimates.
Value prediction as a supervised learning problem.
We ﬁrst analyze the value network through the lens of the supervised learning problem it solves.
After all, (4) describes an empirical risk
[Figure]
G A
[Figure]
M
R E
TRPO PPO
PPO-M (a)    400 # Iterations
[Figure]
R e t u r n s M R E
TRPO PPO
PPO-M (b)
[Figure]
Efﬁcacy of the value network as a variance reducing baseline for Walker2d-v2 agents.
We measure the empirical variance of the gradient (c.f. (1)) as a function of the number of state-action pairs used in estimation, for different choices of baseline functions: the value network (used by the agent in training), the “true” value function (ﬁt to the returns using 500 K state-action pairs sampled from the current policy) and the “zero” value function (i.e. replacing advantages with returns).
We observe that using the true value function leads to a signiﬁcantly lower-variance estimate of the gradient compared to the value network.
In turn, employing the value network yields a noticeable variance reduction compared to the zero baseline function, even though this difference may appear rather small in the small-sample regime ( 2 K).
Additional plots are in Appendix Figure 15.
minimization, where a loss is minimized over a set of sampled ( s t , a t ) .
So, how does V πθ perform as a solution to (4)?
And in turn, how does (4) perform as a proxy for learning the true value function?
Our results (Figure 4a) show that the value network does succeed at both ﬁtting the given loss function and generalizing to unseen data, showing low and stable mean relative error (MRE).
However, the signiﬁcant drop in performance as shown in Figure 4b indicates that the supervised learning problem induced by (4) does not lead to V πθ learning the underlying true value function.
Does the value network lead to a reduction in variance?
Though evaluating the V πθ baseline function as a value predictor as we did above is informative, in the end the sole purpose of the value function is to reduce variance.
So: how does using our value function actually impact the variance of our gradient estimates?
To answer this question, we compare the variance reduction that results from employing our value network against both a “true” value function and a trivial “zero” baseline function (i.e. simply replacing advantages with returns).
Our results, captured in
Figure 5, show that the “true” value function yields a much lower-variance estimate of the gradient.
This is especially true in the sample regime in which we operate.
We note, however, that despite not effectively predicting the true value function or inducing the same degree of variance reduction, the value network does help to some degree (compared to the “zero” baseline).
Additionally, the seemingly marginal increase in gradient correlation provided by the value network (compared to the “true” baseline function) turns out to result in a signiﬁcant improvement in agent performance.
(Indeed, agents trained without a baseline attain almost an order of magnitude worse ﬁnal reward.)
Our ﬁndings here suggest that we still need a better understanding of the role of the value network in agent training and raise several questions that we discuss in Section 3.
2. E XPLORING THE OPTIMIZATION LANDSCAPE
Another key assumption of policy gradient algorithms is that ﬁrst-order updates (w.r.t.
policy parameters) actually yield better policies.
It is thus natural to examine how valid this assumption is.
The true rewards landscape.
We begin by examining the landscape of agent reward with respect to the policy parameters.
Indeed, even if deep policy gradient methods do not optimize for the true reward directly (e.g. if they use a surrogate objective), the ultimate goal of any policy gradient algorithm is to navigate this landscape.
First, Figure 6 shows that while estimating the true reward landscape with a high number of samples yields a relatively smooth reward landscape (perhaps suggesting viability of direct reward optimization), estimating the true reward landscape in the typical, low sample regime results in a landscape that appears jagged and poorly-behaved.
The low-sample regime thus gives rise to a certain kind of barrier to direct reward optimization.
Indeed, applying our algorithms in this regime makes it impossible to distinguish between good and bad points in the landscape, even though the true underlying landscape is fairly well-behaved.
The surrogate objective landscape.
The untamed nature of the rewards landscape has led to the development of alternate approaches to reward maximization.
Recall that an important element of many modern policy gradient methods is the maximization of a surrogate objective function in place of the true rewards (the exact mechanism behind the surrogate objective is detailed in Appendix A.1, and particularly in (14)).
The surrogate objective, based on relaxing the policy improvement theorem of Kakade and Langford (Kakade & Langford, 2002), can be viewed as a simpliﬁcation of the reward maximization objective.
As a purported approximation of the true returns, one would expect that the surrogate objective landscape approximates the true reward landscape fairly well.
That is, parameters corresponding to good surrogate objective will also correspond to good true reward.
Figure 7 shows that in the early stages of training, the optimization landscapes of the true reward and surrogate objective are indeed approximately aligned.
However, as training progresses, the surrogate objective becomes much less predictive of the true reward in the relevant sample regime.
In particular, we often observe that directions that increase the surrogate objective lead to a decrease of the true reward (see Figures 7, 8).
In a higher-sample regime (using several orders of magnitude more samples), we ﬁnd that PPO and TRPO turn out to behave rather differently.
In the case of TRPO, the update direction following the surrogate objective matches the true reward much more closely.
[Figure]
“step direction” axis corresponds to moving one full step in parameter space.
In the random direction one unit corresponds to moving along a random norm 2 Gaussian vector in the parameter space.
In practice, the norm of the step is typically an order of magnitude lower than the random direction.
While the landscape is very noisy in the low-sample regime, large numbers of samples reveal a well-behaved underlying landscape.
See Figures 26, 25 of the Appendix for additional plots.
[Table]
e p d
i r e c t i o n 0.0 0.5 1.0 1.5 2.0 2.5 3.
R a n d o m d i r e c t i o n 0.00.51.01.52.02.
S t e p d
i r e c t i o n 0.0 0.5 1.0 1.5 2.0 2.5 3.
R a n d o m d i r e c t i o n 0.00.51.01.52.02.
S t e p d
i r e c t i o n 0.0 0.5 1.0 1.5 2.0 2.5 3.
R a n d o m d i r e c t i o n 0.00.51.01.52.02.
in training the and landscapes fairly well regimes but However, for PPO we consistently observe landscapes where the step direction leads to lower true reward, even in the high-sample regime.
This suggests that even when estimated accurately enough, the surrogate objective might not be an accurate proxy for the true reward.
(Recall from Section 2.
that this is a sample regime where we are able to estimate the true gradient of the reward fairly well.)
T OWARDS S TRONGER F OUNDATIONS FOR D EEP RL
Deep reinforcement learning (RL) algorithms have shown great practical promise, and are rooted in a well-grounded theoretical framework.
However, our results indicate that this framework often fails to provide insight into the practical performance of these algorithms.
This disconnect impedes our understanding of why these algorithms succeed (or fail), and is a major barrier to addressing key challenges facing deep RL such as brittleness and poor reproducibility.
To close this gap, we need to either develop methods that adhere more closely to theory, or build theory that can capture what makes existing policy gradient methods successful.
In both cases, the ﬁrst step is to precisely pinpoint where theory and practice diverge.
To this end, we analyze and consolidate our ﬁndings from the previous section.
Gradient estimation.
Our analysis in Section 2.1 shows that the quality of gradient estimates that deep policy gradient algorithms use is rather poor.
Indeed, even when agents improve, such gradient estimates often poorly correlate with the true gradient (c.f. Figure 2).
We also note that gradient correlation decreases as training progresses and task complexity increases.
While this certainly does not preclude the estimates from conveying useful signal, the exact underpinnings of this phenomenon in deep RL still elude us.
In particular, in Section 2.1 we outline a few keys ways in which the deep
RL setting is quite unique and difﬁcult to understand from an optimization perspective, both theoretically and in practice (e.g. Figure 3).
In addition, varying the number of samples used in gradient estimates highlights further unexplained behavior—in Figure 3 we see that after reward peaks, further learning consistently reduces average attained reward in later iterations.
Overall, understanding the impact of gradient estimate quality on deep RL algorithms is challenging and largely unexplored.
Value prediction.
The ﬁndings presented in Section 2.2 identify two key issues.
First, while the value network successfully solves the supervised learning task it is trained on, it does not accurately model the “true” value function.
Second, employing the value network as a baseline does decrease the gradient variance (compared to the trivial (“zero”) baseline).
However, this decrease is rather marginal compared to the variance reduction offered by the “true” [Table]
i r e c t i o n 0.0 0.5 1.0 1.5 2.0 2.5 3.
R a n d o m d i r e c t i o n 0.00.51.01.52.02. S t e p d i r e c t i o n 0.0 0.5 1.0 1.5 2.0 2.5 3. R a n d o m d i r e c t i o n 0.00.51.01.52.02.
S t e p d
i r e c t i o n 0.0 0.5 1.0 1.5 2.0 2.5 3.
R a n d o m d i r e c t i o n 0.00.51.01.52.02.
See Figure
It is natural to wonder whether this failure in modeling the value function is inevitable. For example, how does the loss function used to train the value network impact value prediction and variance reduction?
More broadly, we lack an understanding of the precise role of the value network in training.
Can we empirically quantify the relationship between variance reduction and performance? And does the value network play a broader role than just variance reduction?
Optimization landscape.
We have also seen, in Section 2.3, that the optimization landscape induced by modern policy gradient algorithms, the surrogate objective, is often not reﬂective of the underlying true reward landscape.
We thus need a deeper understanding of why current methods succeed despite these issues, and, more broadly, how to better navigate the true reward landscape.
R ELATED W ORK
The idea of using gradient estimates to update neural network-based RL agents dates back at least to REINFORCE (Williams, 1992).
Later, Sutton (Sutton et al., 1999) established a unifying framework casting these algorithms as instances of the policy gradient method.
Our work focuses on proximal policy optimization (PPO) (Schulman et al., 2017) and trust region policy optimization (TRPO) (Schulman et al., 2015a), which are two of the most prominent policy gradient algorithms used in deep RL, drawing inspiration from works such as (Peters et al., 2010) and Kakade (2001).
Many recent works document the brittleness of deep RL algorithms (Henderson et al., 2018; 2017;
Islam et al., 2017).
(Rajeswaran et al., 2017) and (Mania et al., 2018) demonstrate that on many benchmark tasks, state-of-the-art performance can be attained by augmented randomized search approaches.
McCandlish et al.
(2018) investigates gradient noise in large-batch settings, and Ahmed et al.
(2018) investigates the role of entropy regularization (which we do not study) on optimization.
C ONCLUSION
In this work, we analyze the degree to which key primitives of deep policy gradient algorithms follow their conceptual underpinnings.
Our experiments show that these primitives often do not conform to the expected behavior: gradient estimates poorly correlate with the true gradient, better gradient estimates can require lower learning rates and can induce degenerate agent behavior, value networks reduce gradient estimation variance to a signiﬁcantly smaller extent than the true value, and the underlying optimization landscape can be misleading.
This demonstrates that there is a signiﬁcant gap between the theory inspiring current algorithms and the actual mechanisms driving their performance.
Overall, our ﬁndings suggest that developing a deep RL toolkit that is truly robust and reliable will require moving beyond the current benchmarkdriven evaluation model to a more ﬁne-grained understanding of deep RL algorithms.
A A PPENDIX A. B ACKGROUND
In the reinforcement learning (RL) setting, an agent interacts with a stateful environment with the goal of maximizing cumulative reward.
Formally, we model the environment as a (possibly randomized) function mapping its current state s and an action a supplied by the agent to a new state s (cid:48) and a resulting reward r .
The choice of actions of the agent is governed by the its policy π .
This policy is a function mapping environment states to a distribution over the actions to take.
The objective of an RL algorithm is to ﬁnd a policy π which maximizes the expected cumulative reward, where the expectation is taken over both environment randomness and the (randomized) action choices.
Preliminaries and notation.
For a given policy π , we denote by π ( a | s ) the probability that this policy assigns to taking action a when the environment is in the state s .
We use r ( s, a ) to denote the reward that the agent earns for playing action a in response to the state s .
A trajectory τ = { ( a t , s t ) : t ∈ { 1 . . .
T }} is a sequence of state-action pairs that constitutes a valid transcript of interactions of the agent with the environment.
(Here, a t (resp.
s t ) corresponds to the action taken by the agent (resp.
state of the environment) in the t -th round of interaction.)
We then deﬁne π ( τ ) to be the probability that the trajectory τ is executed if the agent follows policy π (provided the initial state of the environment is s 1 ).
Similarly, r ( τ ) = (cid:80) t r ( s t , a t ) denotes the cumulative reward earned by the agent when following this trajectory, where s t (resp.
a t ) denote the t -th state (resp. action) in the trajectory τ .
In the RL setting, however, we often choose to maximize the discounted cumulative reward of a policy R :
= R 1 , where R t is deﬁned as
[Equation]
and 0 < γ < 1 is a “discount factor”.
The discount factor ensures that the cumulative reward of a policy is well-deﬁned even for an inﬁnite time horizon, and it also incentivizes achieving reward earlier.
Policy gradient methods.
A widely used class of RL algorithms that will be the focus of our analysis is the class of so-called policy gradient methods .
The central idea behind these algorithms is to ﬁrst parameterize the policy π θ using a parameter vector θ .
(In the deep RL context, π θ is expressed by a neural network with weights θ .)
Then, we perform stochastic gradient ascent on the cumulative reward with respect to θ .
In other words, we want to apply the stochastic ascent approach to our problem:
[Equation]
where τ ∼ π θ represents trajectories (rollouts) sampled from the distribution induced by the policy π θ .
This approach relies on the key observation (Sutton et al., 1999) that under mild conditions, the gradient of our objective can be written as:
[Equation]
and the latter quantity can be estimated directly by sampling trajectories according to the policy π θ .
When we use the discounted variant of the cumulative reward and note that the action of the policy at time t cannot affect its performance at earlier times, we can express our gradient estimate as:
[Equation]
where Q π θ ( s t , a t ) represents the expected returns after taking action a t from state s t :
[Equation]
Value estimation and advantage.
Unfortunately, the variance of the expectation in (7) can be (and often is) very large, which makes getting an accurate estimate of this expectation quite challenging.
To alleviate this issue, a number of variance reduction techniques have been developed.
One of the most popular such techniques is the use of a so-called baseline function, wherein a state-dependent value is subtracted from Q π θ .
Thus, instead of estimating (7) directly, we use:
[Equation]
where b ( · ) is a baseline function of our choice.
A natural choice of the baseline function is the value function, i.e.
[Equation]
When we use the value function as our baseline, the resulting gradient estimation problem becomes:
[Equation]
where
[Equation]
is referred to as the advantage of performing action a t .
Different methods of estimating V π θ have been proposed, with techniques ranging from moving averages to the use of neural network predictors Schulman et al.
(2015b).
Surrogate Objective.
So far, our focus has been on extracting a good estimate of the gradient with respect to the policy parameters θ .
However, it turns out that directly optimizing the cumulative rewards can be challenging.
Thus, a modiﬁcation used by modern policy gradient algorithms is to optimize a “surrogate objective” instead.
We will focus on maximizing the following local approximation of the true reward Schulman
[Equation] [Equation]
or the normalized advantage variant proposed to reduce variance Schulman et al
[Equation] [Equation]
where
[Equation]
and π is the current policy.
Trust region methods.
The surrogate objective function, although easier to optimize, comes at a cost: the gradient of the surrogate objective is only predictive of the policy gradient locally (at the current policy).
Thus, to ensure that our update steps we derive based on the surrogate objective are predictive, they need to be conﬁned to a “trust region” around the current policy.
The resulting trust region methods (Kakade, 2001; Schulman et al., 2015a; 2017) try to constrain the local variation of the parameters in policy-space by restricting the distributional distance between successive policies.
A popular method in this class is trust region policy optimization (TRPO) Schulman et al.
(2015a), which constrains the KL divergence between successive policies on the optimization trajectory, leading to the following problem:
[Equation]
[Equation] In practice, this objective is maximized using a second-order approximation of the KL divergence and natural gradient descent, while replacing the worst-case KL constraints over all possible states with an approximation of the mean KL based on the states observed in the current trajectory.
Proximal policy optimization.
In practice, the TRPO algorithm can be computationally costly— the step direction is estimated with nonlinear conjugate gradients, which requires the computation of multiple Hessian-vector products.
To address this issue, Schulman et al.
Schulman et al.
(2017) propose proximal policy optimization (PPO), which utilizes a different objective and does not compute a projection.
Concretely, PPO proposes replacing the KL-constrained objective (16) of TRPO by clipping the objective function directly as:
[Equation] [Equation]
where
[Equation]
In addition to being simpler, PPO is intended to be faster and more sample-efﬁcient than
TRPO (Schulman et al., 2017).
A. E XPERIMENTAL S ETUP
We use the following parameters for PPO and TRPO based on a hyperparameter grid search:
[Table]
error bars
A.
S TANDARD R EWARD P LOTS
[Figure]
A. Q UALITY OF G RADIENT E STIMATION
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
function for agents trained on the Hopper and Walker2d-v2 MuJoCo task.
We measure the empirical variance of the gradient (c.f. (1)) as a function of the number of state-action pairs used in estimation, for different choices of baseline functions.
Speciﬁcally, we compare the baseline coming from: the value network (used by the agent during training), the “true” value function (ﬁt directly to the returns using a large number of state-action pairs ( 500 K) sampled from the current policy) and the “zero” value function (i.e. simply replacing advantages with returns).
We observe that the usage of the true value function leads to a signiﬁcantly lower-variance estimate of the gradient as compared to the value network.
In turn, employing the value network yields a noticeably larger variance reduction compared to the zero baseline function, even though this difference may appear rather small in the most relevant, small-sample regime ( 2 K).
A. O PTIMIZATION L ANDSCAPE
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] t
[Figure]
p
[Figure]
a n d o m d i r e c t i o n
[Figure] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
e p
[Figure]
– PPO-M reward landscapes.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
R a n d o m d i r e c t i o
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
S
[Figure]
p
[Figure]
-v2 – TRPO reward landscapes.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
S t e p
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
[Figure]
[Figure]
n d
[Figure]
d i
[Figure]
i
[Figure]
[Figure]
n
[Figure]
o
[Figure]
d i r
[Figure]
c t i o n
[Figure]
p
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
p
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Table]
[Figure]
[Table]
[Figure]
R a n d o m d i r e c t i o n
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
t
[Table]
p d i r e c t i o n
[Table]
[Figure]
[Table]
[Table]
R a n d o m d i r e c t i o n
[Table]
[Table]
[Table]
[Figure]
[Figure]
[Table]
[Figure]
[Figure]
R a n d o
[Figure]
d i r
[Figure]
c t i
[Figure]
t e p
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
a n d o m d i r e c t
i
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
t
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
S t p
[Figure]
e p
[Figure]
[Table]
R
[Table]
n
[Table]
m d i r e c t i o n
[Table]
[Table]
R a n d o m d i r e c t i o n 0.0 0.
[Table]
[Table] [Table]
[Figure]
S t e p d
i r e c t i o n 0.0 0.5 1.0 1.5 2.0 2.5 3.
R a n d o m d i r e c t i o n 0.0 0.5 1.
( ∼ 21 trajectories) ( ∼ 232 trajectories) ( ∼ 1,179 trajectories)
S t e p
S t e p d
i r e c t i o n 0.0 0.5 1.0 1.5 2.0 2.5 3.
R a n d o m d i r e c t i o n 0.0 0.5 1.
S t e p d
i r e c t i o n 0.0 0.5 1.0 1.5 2.0 2.5 3.
R a n d o m d i r e c t i o n 0.0 0.5 1.
S t e p d
i r e c t i o n 0.0 0.5 1.0 1.5 2.0 2.5 3.
R a n d o m d i r e c t i o n 0.0 0.5 1.
( ∼ 19 trajectories)
( ∼ 198 trajectories) ( ∼ 1,068 trajectories)
S t e p
S t e p d
i r e c t i o n 0.0 0.5 1.0 1.5 2.0 2.5 3.
R a n d o m d i r e c t i o n 0.0 0.5 1.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
R
[Figure]
n
[Figure]
d i r e c t i o n
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
R a n d o m d i r e c t i o n
[Figure]
a n d o m d i r
[Figure]
t i o n
[Figure]
( ∼ 18 trajectories) ( ∼ 242 trajectories) ( ∼ 1,433 trajectories)
S t e p
[Figure]
R a n d o m d i r e c t i o n
[Figure]
t
[Figure]
( ∼ 12 trajectories) ( ∼ 162 trajectories) ( ∼ 912 trajectories)
S t e p
S t e p d
i r e c t i o n 0.0 0.5 1.0 1.5 2.0 2.5 3.
R a n d o m d i r e c t i o n 0.0 0.5 1.
[Figure]
S t e p d i r
[Figure]
n
[Figure]