I MPLEMENTATION M ATTERS IN D EEP P OLICY G RADIENTS : A C ASE S TUDY ON PPO AND TRPO
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms, Proximal Policy Optimization and Trust Region Policy Optimization.
We investigate the consequences of “codelevel optimizations:” algorithm augmentations found only in implementations or described as auxiliary details to the core algorithm.
Seemingly of secondary importance, such optimizations have a major impact on agent behavior.
Our results show that they (a) are responsible for most of PPO’s gain in cumulative reward over TRPO, and (b) fundamentally change how RL methods function.
These insights show the difﬁculty, and importance, of attributing performance gains in deep reinforcement learning
I NTRODUCTION
Deep reinforcement learning (RL) algorithms have fueled many of the most publicized achievements in modern machine learning (Silver et al., 2017; OpenAI, 2018; Abbeel & Schulman, 2016;
Mnih et al., 2013).
However, despite these accomplishments, deep RL methods still are not nearly as reliable as their (deep) supervised counterparts.
Indeed, recent research found the existing deep
RL methods to be brittle (Henderson et al., 2017; Zhang et al., 2018), hard to reproduce (Henderson et al., 2017; Tucker et al., 2018), unreliable across runs (Henderson et al., 2017; 2018), and sometimes outperformed by simple baselines (Mania et al., 2018).
The prevalence of these issues points to a broader problem: we do not understand how the parts comprising deep RL algorithms impact agent training, either separately or as a whole.
This unsatisfactory understanding suggests that we should re-evaluate the inner workings of our algorithms.
Indeed, the overall question motivating our work is: how do the multitude of mechanisms used in deep RL training algorithms impact agent behavior?
Our contributions.
We will speciﬁcally analyze the underpinnings of agent behavior—both cumulative reward, as well as more ﬁne-grained algorithmic properties.
As a ﬁrst step towards tackling this question, we conduct a case study of two of the most popular deep policy-gradient methods:
Trust Region Policy Optimization (TRPO) (Schulman et al., 2015a) and Proximal Policy Optimization (PPO) (Schulman et al., 2017).
These algorithms are closely related: PPO was originally developed as a reﬁnement of TRPO.
We ﬁnd that much of the method’s performance comes from various code-level or implementation optimizations not present in TRPO: seemingly small modiﬁcations to the core algorithm either found only in a paper’s original implementation, or described as auxiliary details.
1 We pinpoint these optimizations, and run an ablation study demonstrating that they are instrumental to the PPO’s performance.
This observations prompt us to study how such code-level optimizations change agent training dynamics, and whether we can truly think of them as merely auxiliary improvements.
Our results indicate that code-level optimizations fundamentally change algorithms’ operation, going beyond improvements in agent reward.
Concretely, we ﬁnd that these optimizations are in fact essential for satisfying a key motivating principle behind TRPO and PPO’s operations: trust region enforcement.
Additionally, we ﬁnd that these optimizations are both necessary and sufﬁcient to maintain a trust region, regardless of whether or not the clipping algorithm—typically thought to be the central algorithm of PPO—is employed.
Ultimately, we ﬁnd that the PPO code-optimizations are signiﬁcantly more important in terms of ﬁnal reward achieved than the choice of general training algorithm (TRPO vs. PPO).
This result is in stark contrast to the previous view that the central PPO clipping method drives the gains seen in
Schulman et al.
In doing so, we demonstrate that the algorithmic changes imposed by such optimizations make rigorous comparisons of algorithms difﬁcult.
Without a rigorous understanding of the full impact of code-level optimizations, we cannot hope to gain any reliable insight from comparing algorithms on benchmark tasks.
Our results emphasize the importance of building RL methods in a modular manner.
To progress towards more performant and reliable algorithms, we need to understand each component’s impact on agent behavior and performance—both individually, and as part of a whole.
R ELATED W ORK
The idea of using gradient estimates to update neural network–based RL agents dates back at least to the work of Williams (1992), who proposed the REINFORCE algorithm.
Later, Sutton et al.
(1999) established a unifying framework that casts the previous algorithms as instances of the policy gradient method.
Our work focuses on proximal policy optimization (PPO) (Schulman et al., 2017) and trust region policy optimization (TRPO)
(Schulman et al., 2015a), which are two of the most prominent policy gradient algorithms used in deep RL.
Much of the original inspiration for the usage of the trust regions stems from the conservative policy update of Kakade (2001).
This policy update, similarly to TRPO, uses a natural gradient descent-based greedy policy update.
TRPO also bears similarity to the relative policy entropy search method of Peters et al.
(2010), which constrains the distance between marginal action distributions (whereas TRPO constrains the conditionals of such action distributions).
Notably, Henderson et al.
(2017) points out a number of brittleness, reproducibility, and experimental practice issues in deep RL algorithms.
Importantly, we build on the observation of Henderson et al.
(2017) that ﬁnal reward for a given algorithm is greatly inﬂuenced depending on the code base used.
Rajeswaran et al.
(2017) and Mania et al.
(2018) also demonstrate that on many of the benchmark tasks, the performance of PPO and TRPO can be matched by fairly elementary randomized search approaches.
Additionally, Tucker et al.
(2018) showed that one of the recently proposed extensions of the policy gradient framework, i.e., the usage of baseline functions that are also actiondependent (in addition to being state-dependent), might not lead to better policies after all.
A TTRIBUTING S UCCESS IN P ROXIMAL P OLICY O PTIMIZATION
Our overarching goal is to better understand the underpinnings of the behavior of deep policy gradient methods.
We thus perform a careful study of two tightly linked algorithms: TRPO and PPO (recall that PPO is motivated as TRPO with a different trust region enforcement mechanism).
To better understand these methods, we start by thoroughly investigating their implementations in practice.
We ﬁnd that in comparison to TRPO, the PPO implementation contains many non-trivial optimizations that are not (or only barely) described in its corresponding paper.
Indeed, the standard implementation of PPO 2 contains the following additional optimizations:
Value function clipping: Schulman et al.
(2017) originally suggest ﬁtting the value network via regression to target values:
[Equation]
but the standard implementation instead ﬁts the value network with a PPO-like objective:
[Equation]
2 From the
baselines
where V θ is clipped around the previous value estimates (and ε is ﬁxed to the same value as the value used in (2) to clip the probability ratios).
Reward scaling: Rather than feeding the rewards directly from the environment into the objective, the PPO implementation performs a certain discount-based scaling scheme.
In this scheme, the rewards are divided through by the standard deviation of a rolling discounted sum of the rewards (without subtracting and re-adding the mean)—see Algorithmin Appendix A.2.
Orthogonal initialization and layer scaling: Instead of using the default weight initialization scheme for the policy and value networks, the implementation uses an orthogonal initialization scheme with scaling that varies from layer to layer.
Adam learning rate annealing: Depending on the task, the implementation sometimes anneals the learning rate of Adam (Kingma & Ba, 2014)
(an already adaptive method) for optimization.
Reward Clipping :
The implementation also clips the rewards within a preset range (usually [ − 5 , 5] or [ − 10 , 10] ).
Observation Normalization : In a similar manner to the rewards, the raw states are also not fed into the optimizer.
Instead, the states are ﬁrst normalized to mean-zero, variance-one vectors.
Observation Clipping : Analagously to rewards, the observations are also clipped within a range, usually [ − 10 , 10] . 8.
Hyperbolic tan activations : As also observed by Henderson et al.
(2017), implementations of policy gradient algorithms also also use hyperbolic tangent function activations between layers in the policy and value networks.
Global Gradient Clipping :
After computing the gradient with respect to the policy and the value networks, the implementation clips the gradients such the “global (cid:96) 2 norm” (i.e. the norm of the concatenated gradients of all parameters) does not exceed 0 .
[Figure]
of 3 × 2 4 agents)
and plot histograms in which agents are partitioned based on whether each optimization is on or off.
Our results show that reward normalization, Adam annealing, and network initialization are crucial to obtaining the best average reward with PPO.
We detail our experimental setup in Appendix A.1.
[Table]
These optimizations may appear as merely surface-level or insigniﬁcant algorithmic changes to the core policy gradient method at hand.
However, we ﬁnd that they dramatically affect the performance of PPO.
To demonstrate this, we start by performing a full ablation study on the four optimizations mentioned above 3 .
Figure 1 shows a histogram of the ﬁnal rewards of agents trained with every possible conﬁguration of the above optimizations.
Our ﬁndings suggest that these optimizations are necessary for PPO to attain its claimed performance.
The above ﬁndings show that our ability to understand PPO from an algorithmic perspective hinges on the ability to distill out its fundamental principles from such algorithm-independent (in the sense that these optimizations can be implemented for any policy gradient method) optimizations.
We thus consider a variant of PPO called PPO-M INIMAL (PPO-M) which implements only the core of the algorithm.
PPO-M uses the standard value network loss, no reward scaling, the default network initialization, and Adam with a ﬁxed learning rate.
Importantly, PPO-M ignores all the code-level optimizations listed above in the beginning of Section 3.
We then explore PPO-M alongside PPO and TRPO.
We list all the algorithms we study and their deﬁning properties in Table 1.
Overall, our results on the importance of these optimizations both corroborate results demonstrating the brittleness of deep policy gradient methods, and demonstrate that even beyond environmental brittleness, the algorithms themselves exhibit high sensitivity to implementation choices 4 .
C ODE -L EVEL O PTIMIZATIONS HAVE A LGORITHMIC E FFECTS
In the previous section, we found that canonical implementations of PPO contain many code-level optimizations : implementation choices that are not motivated as core to a method but profoundly impact performance.
This mismatch leads us to ask: how do these seemingly superﬁcial code-level optimizations impact underlying agent behavior?
In this section, we demonstrate that the code-level optimizations fundamentally alter agent behavior.
Rather than merely improving ultimate cumulative award, such optimizations directly impact the principles motivating the core algorithms.
Trust Region Optimization.
A key property of policy gradient algorithms is that update steps computed at any speciﬁc policy π θ t are only guaranteed predictiveness in a neighborhood around θ t .
Thus, to ensure that the update steps we derive remain predictive many policy gradient algorithms ensure that these steps stay in the vicinity of the current policy.
The resulting “trust region” methods (Kakade, 2001; Schulman et al., 2015a; 2017) try to constrain the local variation of the parameters in policy-space by restricting the distributional distance between successive policies.
A popular method in this class is trust region policy optimization (TRPO) Schulman et al.
(2015a).
TRPO constrains the KL divergence between successive policies on the optimization trajectory, leading to the following problem:
[Equation]
In practice, we maximize this objective with a second-order approximation of the KL divergence and natural gradient descent, and replace the worst-case KL constraints over all possible states with an approximation of the mean KL based on the states observed in the current trajectory.
One disadvantage of the TRPO algorithm is that it can be computationally costly—the step direction is estimated with nonlinear conjugate gradients, which requires the computation of multiple Hessianvector products.
To address this issue, Schulman et al.
(2017) propose proximal policy optimization (PPO), which tries to enforce a trust region with a different objective that does not require computing a projection.
Concretely, PPO proposes replacing the KL-constrained objective (1) of TRPO by clipping the objective function directly as:
[Equation]
where
[Equation]
In addition to its simplicity, PPO is intended to be faster and more sample-efﬁcient than
TRPO (Schulman et al
[Equation]
Trust regions in TRPO and PPO.
Enforcing a trust region is a core algorithmic property of different policy gradient methods.
However, whether or not a trust region is enforced is not directly observable from the ﬁnal rewards.
So, how does this algorithmic property vary across state-of-the-art policy gradient methods?
In Figure 2 we measure the mean KL divergence between successive policies in a training run of both TRPO and PPO-M (PPO without code-level optimizations).
Recall that TRPO is designed speciﬁcally to constrain this KL-based trust region, while the clipping mechanism of PPO attempts to approximate it.
Indeed, while TRPO precisely enforcing this trust region, the successive KL divergence between policies in PPO-M grows exponentially as training progresses.
While this may seem surprising at ﬁrst, we ﬁnd that the unbounded nature of the trust region actually follows naturally from the clipping mechanism of PPO.
In particular, the contribution of a single state-action pair to the gradient of the PPO objective is given by:
[Equation]
where
[Equation]
are respectively the standard and clipped versions of the surrogate objective.
As a result, since we initialize π θ as π (and thus the ratios start all equal to one) the ﬁrst step we take is identical to a maximization step over the unclipped surrogate objective.
Therefore, the size of step we take is determined solely be the steepness of the surrogate landscape (i.e. Lipschitz constant of the optimization problem we solve), and we can end up moving arbitrarily far from the trust region.
In fact, observe in Figure 2 that PPO-M fails at even maintaining a trust region based on the maximum ratio (i.e., the exact quantity that PPO tries to control via clipping).
Remarkably, despite that the core mechanism of PPO (which is captured in PPO-M) fails to maintain a trust region, we ﬁnd that PPO with optimizations actually does seem to maintain a KL-based trust region.
This demonstrates that perhaps the key to PPO’s success even from an algorithmic viewpoint comes from auxiliary optimizations, rather than the core methodology.
Enforcing a trust region without projecting or clipping.
It turns out that code-level optimizations alone enforce a trust region without clipping the objective function or explicitly bounding the distance between successive policies.
Indeed, Figure 3 demonstrates that PPO-N O C LIP (PPO without clipping), when accompanied by the original PPO code-level optimizations, can actually maintain reasonable trust regions on benchmark tasks (see Table 1 for a summary of the algorithms we study).
The trust region for PPO-N O C LIP bounds KL more tightly than the TRPO KL bound (this is represented by the horizontal, black dotted line in the mean KL plot) and we do not observe the same exponentially increasing trend we found in PPO-M in Figure 2.
Overall our results indicate that so-called code-level optimizations do not merely increase performance: they fundamentally change algorithms’ operation in unexpected ways.
I DENTIFYING R OOTS OF A LGORITHMIC P ROGRESS
State-of-the-art deep policy gradient methods are comprised of many interacting components.
At what is generally described as their core, these methods incorporate mechanisms like trust regionenforcing steps, time-dependent value predictors, and advantage estimation methods for controlling the exploitation/exploration trade-off (Schulman et al., 2015b).
However, these algorithms also incorporate many less oft-discussed optimizations (cf.
Section 3) that ultimately dictate much of agent behavior (cf.
Section 4).
Given the need to improve on these algorithms, the fact that such optimizations are so important begs the question: how do we identify the true roots of algorithmic progress in deep policy gradient methods?
Unfortunately, we ﬁnd that answering this question is not easy.
Going back to our study of PPO and TRPO, it is widely believed (and claimed) that the key innovation of PPO responsible for its improved performance over the baseline of TRPO is the ratio clipping mechanism (cf.
Section 4).
However, we have already shown that this clipping mechanism does not enforce the KL region it is supposed to.
Where is PPO’s improved performance coming from, then?
To address this question, we set out to disentangle the impact of PPO’s core clipping mechanism from its code-level optimizations.
Speciﬁcally, we examine how employing the core PPO and TRPO steps changes model performance while controlling for the effect of code-level optimizations iden-
[Figure]
the agents maintain a bounded trust region (indeed, more bounded than the trust region of the best TRPO agent, shown as the black line on the right), and achieve high reward, despite not using the PPO clipping step at all.
We give the same plots corresponding to other benchmark tasks in Appendix 6—in the same way, the clipping-free agents often succeed at maintaining a very reasonable trust region without using the core clipping mechanism.
As previously, we detail our experimental setup in Appendix A.1.
tiﬁed in standard implementations of PPO (cf.
Section 3).
(Note that these code-level optimizations are algorithm independent: they can be straightforwardly applied to any policy gradient method.)
To account for the effects of these optimizations, we study an additional algorithm which we denote as TRPO+ , consisting of the core algorithmic contribution of TRPO in combination with PPO’s code-level optimizations as identiﬁed in Section 3.
We note that the four algorithms we study (listed in Table 1: PPO, PPO-M, TRPO, and TRPO+) now capture all combinations of core algorithms and code-level optimizations, allowing us to study the impact of each in a more ﬁne-grained manner.
As our results show in Table 2, it turns out that code-level optimizations contribute to algorithms’ increased performance often signiﬁcantly more than the choice of algorithm (i.e., using PPO vs.
TRPO).
For example, on Walker2d-v2, PPO and TRPO see 28% and 39% improvements (respectively) when equipped with code-level optimizations.
At the same time, after ﬁxing the choice to use or not use optimizations, employing the core TRPO algorithm yields improvements of 20% and 10% (for the cases of with and without-optimizations respectively).
In fact, our results suggest that much of PPO’s improved performance (over TRPO) actually stems from code-level optimizations—on three of the four tasks, TRPO+ actually outperforms PPO.
Given the relative insigniﬁcance of the step mechanism compared to the use of code-level optimizations, we are prompted to ask: to what extent is the clipping mechanism of PPO even necessary?
In
Table 3, we assess this by considering a PPO-N O C LIP algorithm which uses code-level optimizations but no clipping mechanism (this is the same algorithm we studied in Section 4 in the context of trust region enforcement).
Recall that we list all the algorithms studied in Table 1.
It turns out that the clipping mechanism is not at all necessary to achieve high performance— we ﬁnd that PPON O C LIP achieves similar results to PPO in terms of benchmark performance without any clipping at all.
[Table] [Table]
Our results suggest that it is difﬁcult to attribute success to different aspects of policy gradient algorithms without careful analysis.
C ONCLUSION
In this work, we take a ﬁrst step in examining how the mechanisms powering deep policy gradient methods impact agents both in terms of achieved reward and underlying algorithmic behavior.
Wanting to understand agent operation from the ground up, we take a deep dive into the operation of two of the most popular deep policy gradient methods: TRPO and PPO.
In doing so, we identify a number of “code-level optimizations”—algorithm augmentations found only in algorithms’ implementations or described as auxiliary details in their presentation—and ﬁnd that these optimizations have a drastic effect on agent performance.
In fact, these seemingly unimportant optimizations fundamentally change algorithm operation in ways unpredicted by the conceptual policy gradient framework.
Indeed, the optimizations prove necessary for enforcing trust regions regardless of whether we optimize with the PPO step or just unconstrained stochastic gradient descent.
We go on to test the importance of these code-level optimizations in agent performance, and ﬁnd that PPO’s marked improvement over TRPO (and even stochastic gradient descent) is largely due to these optimizations.
Overall, our results highlight the necessity of designing deep RL methods in a modular manner.
When building algorithms, we should understand precisely how each component impacts agent training—both in terms of overall performance and underlying algorithmic behavior.
It is impossible to properly attribute successes and failures in the complicated systems that make up deep RL methods without such diligence.
More broadly, our ﬁndings suggest that developing an RL toolkit will require moving beyond the current benchmark-driven evaluation model to a more ﬁne-grained understanding of deep RL methods.
A A PPENDIX A. E XPERIMENTAL S ETUP
We use the following parameters for PPO, PPO-M, and TRPO based on a hyperparameter grid search:
[Table]
All error bars we plot are 95% conﬁdence intervals, obtained via bootstrapped sampling.
A. PPO C ODE -L EVEL O PTIMIZATIONS
DD
A. T RUST R EGION O PTIMIZATION
[Figure]
[Figure] [Figure]