A R ISK -S ENSITIVE P OLICY G RADIENT M ETHOD
Anonymous authors
Paper under double-blind review
A BSTRACT
Standard deep reinforcement learning (DRL) agents aim to maximize expected reward, considering collected experiences equally in formulating a policy.
This differs from human decision-making, where gains and losses are valued differently and outlying outcomes are given increased consideration.
It also wastes an opportunity for the agent to modulate behavior based on distributional context.
Several approaches to distributional DRL have been investigated, with one popular strategy being to evaluate the projected distribution of returns for possible actions.
We propose a more direct approach, whereby the distribution of full-episode outcomes is optimized to maximize a chosen function of its cumulative distribution function (CDF).
This technique allows for outcomes to be weighed based on relative quality, does not require modiﬁcation of the reward function to modulate agent behavior, and may be used for both continuous and discrete action spaces.
We show how to achieve an asymptotically consistent estimate of the policy gradient for a broad class of CDF-based objectives via sampling, subsequently incorporating variance reduction measures to facilitate effective on-policy learning.
We use the resulting algorithm to train agents with different “risk proﬁles” in penalty-based formulations of six OpenAI Safety Gym environments, observing that moderate emphasis on improvement in training scenarios where the agent performs poorly both increases the accumulation of positive rewards and decreases the frequency of incurred penalties.
We found that, in all environments tested, the same risk proﬁle can be used to produce both stronger overall performance than standard
Proximal Policy Optimization (PPO) and higher levels of positive reward than PPO constrained by Lagrangians to maintain the same cost levels.
I NTRODUCTION
While deep reinforcement learning (DRL) has been used to master an impressive array of simulated tasks in controlled settings, it has not yet been widely adopted for high-stakes, real-world applications.
One reason for this gap is the lack of distributional perspective in standard artiﬁcial agents.
Endowing agents with such perspective could make their decision-making more robust, potentially leading to increased safety, increased trust from humans, and more widespread real-world adoption.
In reinforcement learning (RL), risk arises due to uncertainty around the possible outcomes of an agent’s future actions.
It is a result of randomness in the operating environment, mismatch between training and test conditions, and the inherent randomness of a stochastic policy.
Risk-sensitive policies, or those that consider more than a mean over the distribution of possible outcomes, offer the potential for added robustness under uncertain and dynamic conditions.
There is an evolving landscape of algorithmic paradigms for handling risk in RL, from constraint-based approaches adapted from optimal control (Achiam et al., 2017; Chow et al., 2019; Ray et al., 2019; Zhong et al., 2020) to adversarial approaches emerging from AI Safety (Garc´ıa & Fern´andez, 2015; Amodei et al., 2016).
Within this landscape, learning approaches that optimize distributional measures offer the ability to express design preferences over the full distribution of potential outcomes, through the speciﬁcation of a risk-sensitivity criterion.
Distributional RL has been studied for value-based methods, with a popular strategy being to use the distributional Bellman equation to estimate the distribution of Q-values for each member of a discrete set of potential actions (Bellemare et al., 2017; Dabney et al., 2018a;b).
However, distributional RL has not been widely explored for policy gradient methods, which could permit direct optimization of risk-sensitive measures and naturally accommodate both discrete and continuous action spaces.
In the following, we introduce a novel framework for risk-sensitive learning using policy gradients.
Our approach allows agents to be trained with different risk proﬁles through design-time speciﬁcation of both utility and weight functions, with the latter being deﬁned over the estimated distribution of full-episode rewards.
This framework enables agent-based learning that captures aspects of human decision-making, such as overemphasis of rare occurrences and diminishing marginal utility relative to a reference outcome (Kahneman & Tversky, 1979).
It also allows implementation of another key strategy of human learning: emphasizing improvement on tasks where one is deﬁcient.
We demonstrate the ability of our algorithm to use this strategy to improve performance relative to both unconstrained and constrained methods in six OpenAI Safety Gym environments (Ray et al., 2019).
R ELATED W ORK
Constrained RL offers a set of approaches to safe exploration (Garc´ıa & Fern´andez (2015))
that aim to enforce explicit constraints throughout the learning process via methods including Lagrangian constraints (Ray et al.
and constraint coefﬁcients (Achiam et al.
Differing from the safe exploration scenario, we here consider problems with distinct training and test phases where agent performance is to be evaluated.
Our experiments indicate that risk-sensitive learning can offer performance improvements over constrained learning in scenarios where safety constraints need not be enforced during training.
Distributional approaches to risk-sensitive RL have primarily been explored in the value-based setting.
Therein, the value distribution has been explicitly modeled through categorical techniques (Bellemare et al., 2017) or quantile regression (Dabney et al., 2018a) and used to improve both value predictions and overall performance.
Recent works utilize distributional modeling in the actor-critic setting to enable application to continuous action spaces, again demonstrating improved performance over baseline approaches (Ma et al., 2020;
Zhang et al., 2021; Duan et al., 2021).
In value-based approaches, risk-sensitivity criteria are applied at run time as a nonlinear warping of the estimated value distribution.
Policy gradient approaches offer additional promise for risk-sensitive RL, but require direct optimization of a parameterized policy with respect to a distributional objective.
Some existing methods are limited to a speciﬁc class of learning objective, such as the set of concave risk measures that allow a globally-optimal solution (Tamar et al., 2015; Zhong et al., 2020).
Others allow a broader class of measures but are more restrictive in the class of policies that can be represented (Prashanth et al., 2016).
We aim for a risk-sensitive policy gradient approach that both offers signiﬁcant ﬂexibility in the choice of learning objective and can learn policies parameterized by a deep neural network.
Various measures have been considered in the context of risk-sensitive RL, including exponential utility (Pratt, 1964), percentile performance criteria (Wu & Lin, 1999), value-at-risk (Leavens, 1945), conditional value-at-risk (Rockafellar & Uryasev, 2000), and prospect theory (Kahneman & Tversky, 1979).
In this work, we consider a class of risk-sensitivity measures motivated by Cumulative
Prospect Theory (CPT) (Tversky & Kahneman, 1992).
CPT uniquely models two key aspects of human decision-making: (1) a utility function u , computed relative to a reference point that induces more risk-averse behavior in the presence of gains than losses and (2) a weight function w that prioritizes outlying events.
Speciﬁc forms of u and w are given in Tversky & Kahneman (1992) (and in Appendix A.4), but the general form of CPT admits a wide variety of risk-sensitive objectives.
Here we show how to train agents to optimize this class of objectives through sampling-based estimation of their policy gradients and requisite variance reduction.
The ﬁnal algorithm resembles well-known on-policy approaches such as Proximal Policy Optimization (Schulman et al., 2017b) and is similarly widely applicable.
Although we do not explore it here, the incorporation of an appropriate risk-sensitivity criterion could additionally enable risk-aware exploration and adversarial training for increased robustness (Pinto et al., 2017; Parisi et al., 2019; Zhang et al., 2020).
R ISK -S ENSITIVE P OLICY O PTIMIZATION
In this section we formalize the class of distributional objectives to be considered, derive a samplingbased approximation of its policy gradient, enact variance reduction on this estimate, and use the result to produce a practical learning algorithm.
3. P RELIMINARIES : P ROBLEM AND N OTATION
Standard deep reinforcement learning seeks to maximize the expected reward of an agent over encountered trajectories; that is, it maximizes the objective
[Equation]
Here p θ ( τ ) is the distribution over trajectories τ ≡ s 1 , a 1 , . . .
, s T , a T induced by a policy parameterized by θ ; s t , a t , and r ( s t , a t ) denote the state, action, and reward at time t , respectively.
To enable the incorporation of distributional context, we instead consider the objective
[Equation]
where u ( r ( τ )) is the utility associated with full-trajectory reward r ( τ ) ≡ (cid:80) t r ( s t , a t ) and w is a piecewise differentiable weighting function of the CDF of trajectory reward P θ ( r ( τ ))
= (cid:82) r ( τ )
−∞ p θ ( r
[Equation]
dr (cid:48) .
Equation 2 is inspired by CPT (Tversky & Kahneman, 1992), which includes a pair of integrals of this form.
It was chosen for its generality; by using different utility functions u and weight functions w one may represent all of the risk measures mentioned in Section 2 and all of the risk measures evaluated by Dabney et al.
(2018a).
The form (2) reduces to (1) when u and w are both the identity mapping.
While designed for the episodic setting, the objective (2) may be considered for inﬁnite horizons through the choice of appropriately long windows.
3. R ISK -S ENSITIVE P OLICY G RADIENT
To optimize the objective (2), we ﬁrst derive an approximation to its gradient with respect to the policy parameters θ .
Working toward a representation that can be sampled, we assert the independence of the reward on θ and use the chain rule to write
[Equation]
where w (cid:48) is the derivative of w with respect to P θ ( r ( τ )) .
The gradient of the CDF may be written as follows:
[Equation]
Here the ﬁrst equality is the integral representation of P θ ( r ( τ )) , the second uses the Heaviside step function to select all trajectories with total reward ≤ r ( τ ) , the third follows from the independence of reward on θ , and the fourth follows from the expression for the derivative of the natural logarithm.
In the following, we also use the complementary expression
[Equation]
Either form, or a combination of the two
[Equation]
may be substituted into (3) and the result sampled over N trajectories by ﬁrst ordering trajectories i = 1 . . .
N by increasing reward r ( τ ) .
Then
[Equation]
where the term w (cid:48) (0) ∇ θ P θ ( r ( τ 0 ))
≡ 0 .
This ordering scheme produces an asymptotically consistent estimate, as shown in the context of CPT value estimation by Prashanth et al.
∇ θ P θ ( r ( τ i )) may be sampled in one of two ways, based on either (4) or (5):
[Equation]
The expression (6) may be used to train a policy that optimizes the distributional objective (2) in a manner similar to REINFORCE (Williams, 1992).
3. V ARIANCE R EDUCTION
Reducing the variance of sample-based gradient estimates enables faster learning.
Here we take several steps to reduce the variance of (6), as has been done with the policy gradient estimate of
REINFORCE (Williams, 1992).
First, note that cross-trajectory terms of the form f
( τ i , a j,t , s j,t ) =
u ( r ( τ i )) ∇ θ log π θ ( a j,t | s j,t ) , while nonzero, do not contribute to the gradient estimate in expectation when i (cid:54)
=
j .
A proof of this assertion is given in Appendix A.1.
Using (4) for the ﬁrst term of (6) and (5) for the second allows us to write
[Equation]
Removing cross-trajectory terms gives
[Equation]
Note that the weight coefﬁcients ( w (cid:48) ( iN ) + w (cid:48)
( i − 1 N )) should be normalized over each batch.
The expression (9) is equivalent to (6) in expectation, but with reduced variance (see Appendix A.1 for justiﬁcation).
It has a clear intuition – trajectories are assigned utilities based on their rewards and their contributions to the gradient are scaled by the derivative of the weight function, just as they are in Cumulative Prospect Theory (Tversky & Kahneman, 1992).
Standard variance reduction techniques may be applied to this simpliﬁed form.
Without further assumption or introduction of additional bias, a static baseline b can be employed:
[Equation]
Justiﬁcation for this assertion is given in Appendix A.2.
Learning may be further improved if we additionally assume that utility may be allocated on a per-step basis.
In this case, per-step utilities are computed as the difference between what the full-episode utility would be if the episode were to end at a given time step and what it would have been had the episode ended at the previous time step.
While not applicable in cases where episode utility is adjusted based on ﬁnal outcome, this assumption has the signiﬁcant beneﬁt of modeling the temporal allocation of rewards and aligns with the standard formulation of RL.
With it, the variance of (9) may be further reduced through the incorporation of utility-to-go and a state-dependent baseline V φ ( s i,t ) :
[Equation]
Here u ( s i,t (cid:48) , a i,t (cid:48) ) is the per-step utility.
The value function V φ ( s i,t ) is parameterized by φ and trained via regression to minimize
[Equation]
A standard argument, similar to the approach taken in (Achiam, 2018), can be used to show that the incorporation of utility-to-go does not change the expected value of (9).
The addition of a state-dependent baseline also does not introduce additional bias, as justiﬁed in Appendix A.2.
Finally, discount factors, bootstrapping, and trust regions may be used to provide additional variance reduction, just as they are in conventional on-policy learning (Appendix A.2).
These measures may introduce additional bias to the policy gradient estimate, but typically lead to more sampleefﬁcient learning.
In our experiments, we evaluate the use of generalized advantage estimation (GAE; (Schulman et al., 2016)) based on the utility-to-go as well as the clipping-based trust regions of
Proximal Policy Optimization (PPO; (Schulman et al., 2017b)).
Incorporating these in the policy gradient yields
[Equation]
where A πu ( s i,t , a i,t ) is the standard GAE except with per-step utilities in place of rewards.
Trust regions are implemented similarly to PPO, pessimistically clipping policy updates to be within a multiplicative factor of 1 ± (cid:15) of the existing policy:
[Equation]
They are used to perform multiple policy updates using the same batch of data, providing learning that is no longer strictly on-policy but that can be signiﬁcantly more sample efﬁcient.
When following this route, we apply the same early stopping mechanism, based on the Kullback-Leibler divergence ( D KL ) between old and new policies, as was used by Ray et al.
3. L EARNING A LGORITHM
The above sample-based estimate of the policy gradient may be used to train agents to maximize distributional objectives of the form (2).
The resulting method, Cumulative Prospect Proximal Policy
Optimization (C3PO), is given in Algorithm 1 and mirrors standard on-policy learning.
Algorithm
Require: Policy: initial parameters θ 0 , learning rate α θ , updates per batch M θ
Require:
Value: initial parameters φ 0 , learning rate α φ , updates per batch M φ
Require:
Early stopping threshold D KL, stop for k = 1 , 2 , . . .
do
Collect set of episodes D k = { τ i } by running policy π ( θ k ) in the environment
Compute per-step utilities u
( s i,t , a i,t )
Fit value function by regression: for m = 1 , . . .
M φ do φ ← φ + α φ ∇ φ
[Equation]
V
[Equation]
(cid:19)
[Equation]
end for
Update utility-based advantage estimates A πu ( s , a ) , using new V φ ( s )
Compute weight coefﬁcients based on ordered episode outcomes and normalize
Update policy, using KL-based early stopping: for m = 1 , . . .
M θ do if D KL ( π θ ||
π θ old ) <
D KL, stop then θ ←
[Equation] [Equation]
,t
[Equation]
) else break end if end for end for
Beyond the utility and weight components, Algorithm 1 differs from conventional methods in the requirement to collect full episodes of data in each batch.
This requirement can be removed if outcomes can be deﬁned over partial rather than full episodes, an assumption that is often viable and matches human decision-making.
For instance, while out of scope for this work, our approach could be applied to the Atari suite (Bellemare et al., 2013) by considering the outcomes of ﬁxed-length windows and restructuring Algorithm 1 to mimic minibatch PPO (Schulman et al., 2017b).
E XPERIMENTS
To evaluate our approach, we sought to both establish that it can effectively optimize different distributional objectives and explore the impact of using different objectives on agent outcomes.
We found the OpenAI Safety Gym (Ray et al., 2019) to be suitable for these purposes.
Safety Gym is a conﬁgurable suite of continuous, multidimensional control tasks wherein different types of robots must navigate through obstacles with different dynamics to perform different tasks.
By including both positive and negative events in each training scenario, it allowed us to evaluate how our various agents handled risk.
Safety Gym is also highly stochastic: the locations of the goals and obstacles are randomized, leading to outcome variability and forcing the agent to learn a generalized navigation strategy.
Safety Gym logs adverse events but does not incorporate them into the reward function.
As our method relies solely on the training signal from the reward, we assigned each logged adverse event a ﬁxed, negative reward contribution in experiments using it or other unconstrained agents.
Our initial experiments were conducted with a reward contribution of − 0 .
025 , which was found to allow agents to prioritize reaching goals but deter them from collisions with obstacles.
To further emphasize obstacle avoidance, we doubled this contribution to − 0 .
in our experiments using cautious weightings.
These choices and the role they play are further discussed in Section 5.
To highlight distributional differences, we focused on the publicly available, obstacle-rich levelenvironments.
1 Avoiding the longer compute time of the “Doggo” robot, we evaluated the “Point” and “Car” robots on each task (“Goal”, “Button”, and “Push”).
Further details on these environments and our rationale for choosing them are given in Appendix A.3.
In all experiments, we evaluated ﬁve random seeds and matched the hyperparameters used in the baselines accompanying Safety Gym (Ray et al., 2019) as closely as possible.
The neural networks used to model both policy and value were multilayer perceptrons (MLPs), with two hidden layers of 256 units each and tanh activations.
As in Ray et al.
(2019), the policy network outputs the mean values of a multivariate gaussian with diagonal covariance.
The control variances are optimized but independent of state.
The full complement of variance reduction measures were used throughout; see
Appendix A.4 for experimental justiﬁcation of this choice.
4. D IFFERING O BJECTIVES
Agent performance was explored under four different distributional objectives.
In addition to expected reward and CPT (conﬁgured to match the original form of Tversky & Kahneman (1992) and as given in Appendix A.4), we optimized for cautious ( η = 0 . 75 ) and aggressive ( η = − 0 . 75 ) versions of the distortion risk measure proposed in Wang (2000).
This measure is deﬁned as w ( p ) =
Φ(Φ − 1 ( p )+ η ) , where Φ and Φ − 1 are the standard normal cumulative distribution function and its inverse.
While we found this form to be convenient, the “Pow” metric in Dabney et al.
(2018a) or any other set of similarly shaped w curves should achieve a similar effect.
In experiments using the objective from
Tversky & Kahneman (1992), the reference point was taken to be the mean episode reward of the current batch, matching the tendency of humans to change their standards over time.
The four weight functions and their resulting coefﬁcients in (9) are shown in Figure 1.
Plots of the total rewards (including penalties) in training, average cost events per episode in training, and outcome distributions in testing are shown for one environment in Figure 2 and for two additional environments in Appendix A.5.
The trends were fairly consistent over the three environments evaluated in this manner.
While no explicit effort was made to handle cost (agents were given only the sum of positive rewards and penalties), the cautious and aggressive weightings consistently accumulated relatively low and high costs, respectively.
The cautious (Wang( η = 0 . 75 )) agent typically also generated the highest positive and total rewards after 10 millions steps of training.
To generate the histograms in Figures 2, 5, 8, 9, and 13 as well as the numbers in Table 1, the trained agents were deployed on a set of 5000 test episodes – 1000 for each of the 5 networks learned using different random seeds in training.
The resulting distributions therefore include contributions from both aleatoric and epistemic uncertainty.
Sampling was turned off, allowing the agents to choose their perceived optimal action at each time step.
In this context the beneﬁt of emphasizing the lower part of the outcome distribution (i.e., cautious weighting) became more pronounced, in part because the methods that emphasize poor outcomes tended to maintain higher policy entropy (Appendix A.5).
4. C AUTIOUS W EIGHTINGS
To further explore the apparent beneﬁts of cautious weightings in Safety Gym, we trained a series of variably cautious agents by tuning η in the risk-averse weight function proposed by Wang (2000).
Histograms of their episode rewards in testing are given in Appendix A.5 and summarized in Table 1.
In these environments, agent performance – both in terms of improving the lower end of the reward distribution and on average – was seen to generally improve with increasing η until the range η ∈ [0 .
75 , 1 . 25] , subsequently degrading.
Additional comparisons were made with PPO (Schulman et al., 2017b), which unsurprisingly was found to closely track performance of the “Uniform” agent.
We found that naively incorporating cautious weightings into PPO improved its performance (row
PPO + Wang(0.75) in Table 1), though not to the level of the full C3PO method with η = 0 .
We then pursued a set of longer runs to compare C3PO with the cautious objective from Wang (2000) to both unconstrained and constrained benchmarks.
Here we did not tune η , keeping it ﬁxed at 0 .for
all experiments.
Comparisons with unconstrained methods for three environments are given in
Figures 3, 4, and 5 and for the remaining three environments in Appendix A.6.
In addition to PPO, we compared performance with Trust Region Policy Optimization (TRPO; Schulman et al. (2017a)) as conﬁgured in Ray et al.
Since this TRPO conﬁguration generally outperformed the PPO conﬁguration in Ray et al.
(2019) from which we derived the hyperparameters for C3PO, we would expect C3PO to be at a disadvantage compared to TRPO.
However, we found C3PO had the highest
[Table]
average reward (including penalty) in ﬁve of the six environments and lowest average cost in ﬁve of the six environments.
In addition, agents that used the cautious weightings tended to have more stable and repeatable training, as evidenced by the tight distribution of their learning curves.
This tightness was found to reﬂect a lack of negative outlier episodes and potentially lower epistemic uncertainty throughout training.
Finally, note that the use of a nonzero penalty for cost events resulted in signiﬁcantly lower incurred costs than were observed with unconstrained agents trained without a penalty (Ray et al., 2019).
PPO and TRPO were seen to reach similar cost levels without a penalty; these levels are indicated by red dashed lines in the cost ﬁgures.
Comparisons with versions of PPO and TRPO that use Lagrangian constraints (PPO-Lagrangian and TRPO-Lagrangian; Ray et al.
to match the cost level of C3PO are shown in Figureand Appendix A.7 .
We see that, given the same level of cost incurred per episode, agents trained using C3PO consistently achieve higher levels of reward than those trained with PPO-Lagrangian and TRPO-Lagrangian.
As above, training is seen to be more stable and repeatable using our risksensitive method.
Additional comparisons were generated with Constrained Policy Optimization (CPO; Achiam et al.
but are not shown in Figure 6 because they failed to maintain the cost levels of the other methods.
For completeness, they are given in Appendix A.7.
D ISCUSSION
The analysis above allows for sample-based policy gradient estimates of a broad class of distributional objectives.
Variance reduction measures were shown to enable efﬁcient optimization based on these estimates (Appendix A.4).
However, it was not seen to be the case that a given distributional objective could be most effectively optimized directly.
Instead, the best results were generally obtained through moderate emphasis on improving negative training outcomes (Table 1).
To understand this behavior, consider the interplay of optimization and exploration in the training of cautious and aggressive agents.
Cautious weightings continually emphasize the lower part of the outcome distribution, pushing that part of the distribution upward and adjusting behavior the most where it is most necessary.
Once a part of the state space where the agent is deﬁcient is adequately addressed, a different part of the state space takes its place.
Policy entropy remains high because of the emphasis on problematic situations, ensuring adequate exploration.
This trend continues with increasing η , until the point where the agent begins to ignore high quality training outcomes too much.
Conversely, aggressive weightings continually emphasize the best outcomes in the distribution.
When an already strong outcome is given increased attention, it is likely to stay at the top.
Hence agents trained with aggressive weightings tend to become myopic, obsessing over a fraction of the state space while neglecting the rest of it.
They tend to explore inadequately and ironically fail to attain better top-end performance than more cautious weightings.
Given the consistent performance gains observed using our method, we propose that it represents a useful option for improving the performance and stability of on-policy learners.
This should be particularly true in the presence of a meaningful trade-off between positive and negative reward terms and when there is signiﬁcant stochasticity.
While our approach does add an additional hyperparameter – the shaping constant η – one value for that hyperparameter was seen to provide gains across all environments tested.
While our approach does not provide for a direct choice of cost limit as constrained methods do, it is simpler to implement and was consistently seen to be more performant for the cost level it reached.
It is also likely possible to use cautious weightings in conjunction with constrained RL, though this has not yet been investigated.
C ONCLUSIONS
In this work, we proposed a risk-sensitive learning algorithm based on a policy gradient estimate for a broad class of distributional objectives.
When conﬁgured to emphasize improvement in scenarios where the agent performs poorly, we found our method to compare favorably with existing unconstrained and constrained on-policy learners.
A A PPENDIX A. E VALUATION OF CROSS - TRAJECTORY TERMS IN POLICY GRADIENT
In this section, we ﬁrst show that the cross-trajectory terms in our policy gradient estimate (6) have an expectation value of 0 .
We then argue that their removal leads to a policy gradient estimate with reduced variance.
Lemma 1.
Cross-trajectory terms of the form f ( τ i , a j,t , s j,t ) =
u ( r ( τ i )) ∇ θ log π θ ( a j,t | s j,t ) ,
where i (cid:54) =
j , do not contribute to the gradient estimate (6) in expectation.
Proof.
First, note that
[Equation]
Then consider the innermost expectation:
[Equation]
To see why removal of cross-trajectory terms leads to reduced variance, consider that the full expression (6) may be written as the sum of terms of the form f
( τ i , a j,t , s j,t ) =
u ( r ( τ i )) ∇ θ log π θ ( a j,t , s j,t ) .
Its variance is the sum of the total variance from terms
where i =
j , the total variance from terms where i (cid:54) =
j , and a term proportional to the covariance of these two totals.
However, because each term in the covariance contains at least one trajectory that differs from the rest, the above reasoning may be applied to argue that the covariance is 0.
Hence, the removal of the cross-trajectory terms lowers the variance of the policy gradient estimate by the variance of the cross-trajectory terms.
A. I NTRODUCTION OF S TATIC AND S TATE -D EPENDENT B ASELINES
Lemma 2.
A static baseline of the utility may be added to the policy gradient estimate (9) without introduction of bias.
Proof.
The additional term is 0 in expectation as
[Equation]
i n (cid:19)
[Equation]
w (cid:48) (cid:18)
[Equation]
−n (cid:19)(cid:19) (cid:90) s i,t p ( s i,t | π θ ) ∇ θ (cid:90) a i,t π θ ( a i,t | s i,t )
d a i,t d s i,t = b (cid:18)
[Equation]
i n
[Equation]
n (cid:19)(cid:19) (cid:90)
[Equation]
i
[Equation]
The contribution of the weight terms ( w (cid:48) ( in ) + w (cid:48)
( i − 1 n )) may be pulled out of the integral between the ﬁrst and second line because of its independence on both state and action.
This term is ﬁxed for a given trajectory by the rank of its reward amongst the rewards accumulated on all trajectories in the current batch.
In our variance reduction experiments (Appendix A.4), the “Base” agent uses b equal to the mean of full-episode utility in the current batch.
As described in Section 3.3, we may further adjust the policy gradient estimate through introduction of per-step utilities.
In this case, we may justify the use of a state-dependent baseline through the following.
Lemma 3.
A state-dependent baseline V φ ( s i,t ) may be added to the policy gradient estimate (9) without introduction of bias, if per-step utilities are assumed
[Equation]
Proof.
The additional term is 0 in expectation as
[Equation]
s i,t ) ∇ θ (cid:90) a i,t π θ ( a i,t | s i,t )
d a i,t d s i,t = (cid:18) w (cid:48) (cid:18) i n (cid:19) + w (cid:48) (cid:18)
i
[Equation]
n (cid:19)(cid:19) (cid:90) s i
[Equation]
t p (
[Equation]
t |
[Equation]
[Equation] [Equation]
= 0 .
The rationale for pulling the w (cid:48) terms out of the integral is the same as in Lemma 2.
Finally, we note that the ability to pull the contribution of the weight terms ( w (cid:48) ( in ) + w (cid:48) ( i − 1 n )) to the front of Equation 11 allows us to formulate advantage estimates based on per-step utility.
Bootstrap estimates of the value function V φ ( s i,t ) and Generalized Advantage Estimation as in Schulman et al.
(2016) can be conducted exactly as they are in standard on-policy learning, if rewards are replaced by per-step utilities.
A. A DDITIONAL I NFORMATION ON S AFETY G YM
As mentioned in Section 4, we chose to evaluate our approach using the OpenAI Safety Gym (Ray et al., 2019).
The choice was governed by our desire to test in conditions with clear cost-beneﬁt trade-offs, signiﬁcant stochasticity, adequate complexity, and available benchmarks.
While our methods are not limited to particular task types or observation/action spaces, we found Safety Gym to be suitable for exploring their potential.
The six environments chosen were the most obstacle-rich of the publicly available environments that used the “Point” and “Car” robots.
The Point robot is constrained to the 2D plane and has two control dimensions: one for moving forward/backward and one for turning.
The Car robot also has two control dimensions, corresponding to independently actuated parallel wheels.
It has a freely rotating wheel and, while it is not constrained to the 2D plane, typically remains in it.
While we expect our results to extend to the remaining default robot, “Doggo”, we did not experiment with it because of the order of magnitude longer training times it exhibited in Ray et al.
Several types of obstacles and tasks were present in the environments we evaluated.
In all cases, the robot is given a ﬁxed amount of time (1000 steps) to complete the prescribed task as many times as possible and is motivated by both sparse and dense reward contributions.
In the “Goal” environments, the robot must navigate to a series of randomly-assigned goal positions, with a new target being assigned as soon as a goal is reached.
In the “Button” environments, the robot must reach and press a sequence of goal buttons while avoiding other buttons.
In the “Push” task, the robot must push a box to a series of goal positions.
The set of obstacles are different for each task; among the three environments there are a total of ﬁve different constraint elements (hazards, vases, incorrect buttons, pillars, and gremlins), each with different dynamics.
See Ray et al.
for further details.
A. E MPIRICAL P ERFORMANCE OF V ARIANCE R EDUCTION M EASURES
To gauge the impact of the variance reduction techniques outlined in Section 3.3, we evaluated their performance in maximizing the value function of Cumulative Prospect Theory (Tversky & Kahneman, 1992).
As mentioned in Section 3.1, this function has two integrals of the form (2):
[Equation]
In Tversky & Kahneman (1992), the utility functions are computed relative to a reference point and reﬂect the tendency of humans to be more risk-averse in the presence of gains than in the presence of losses.
The weight functions { w + , w − } model our inclination to emphasize the best and worst possible outcomes in our decision-making.
More speciﬁcally, in these experiments we used the piecewise utility functions u + ( r )
=
H ( r − r 0 )(
r − r 0 ) σ and u − = λH ( r 0 − r )
( r 0 − r ) σ with static reference r 0 = 10 , σ = 0 .
88 , and λ = 2 .
The weight function w ( p ) =
p η ( p η +(1 − p ) η ) 1 η was used, where η = 0 .
61 for r < r 0 and η = 0 .
69 for r ≥ r 0 .
Four methods were evaluated, incorporating progressive amounts of variance reduction:
• Base : Risk-sensitive policy gradient with a static baseline (10) • UTG : Base with utility-to-go and a neural network baseline (11) • GAE : UTG with generalized advantage estimation ((13) without clipping) • TR : GAE with trust regions ((13)
with clipping (14))
As shown in Figure 7, the incorporation of these techniques increased the sample efﬁciency of the
CPT value optimization signiﬁcantly.
Consequently, we used the full complement (TR) in all other experiments.
A. D IFFERING O BJECTIVES : A DDITIONAL R ESULTS
Below we include results for all environments for the experiments described in Section 4.1.
In addition we note the trend of policy entropies with different distributional objectives.
In general, more cautious weightings maintain higher entropy for longer than more aggressive weightings.
Note that these plots represent an upper bound because they do not account for action clipping by the environment; see Ray et al.
for details.
A. A DDITIONAL C OMPARISONS WITH U NCONSTRAINED M ETHODS
Below are plots of average episode reward and average number of episode cost events throughout training for the remainder of the environments on which we conducted long runs (Section 4.2).
Also included are histograms of testing performance for those runs.
A. A DDITIONAL C OMPARISONS WITH C ONSTRAINED M ETHODS
As mentioned in Section 4.
we compared the performance of C3PO with constrained methods by setting the cost limit of the constrained methods to match the cost level attained by C3PO.
Here we provide
• the positive reward plots for the remaining three Safety Gym environments studied, • the cost plots for each of the six environments, and
The intent of the cost plots of Figure 15 is to show rough consistency between the cost levels of our approach and Lagrangian methods conﬁgured to have the same cost limit.
This is veriﬁed, but other trends should be noted.
First, while the Lagrangian-based methods typically follow the cost constraint well, they cannot satisfy it in each batch.
Second, our approach tends to have comparable or lower cost rates throughout training.
This safe exploration metric, deﬁned in Ray et al. (2019), refers to the average cost per episode over all of training up to a given point. Results related to Constrained Policy Optimization (CPO; Achiam et al. (2017)) are included here but not in the main text because, consistent with (Ray et al., 2019), we were not able to conﬁgure CPO to respect the cost levels of the other constrained methods.
Here we show the cost levels reached by CPO compared with C3PO (Figure 16) as well as a comparison of the average episode rewards of the two (Figure 17).
For the latter, we employed the penalty scaling used by C3PO to enable a fair comparison.
A. S UPPLEMENTARY M ATERIALS
The code used to produce these results is included in our Supplementary Materials.