R ISK -A VERSE O FFLINE R EINFORCEMENT L EARNING
A BSTRACT
Training Reinforcement Learning (RL) agents online in high-stakes applications is often prohibitive due to the risk associated with exploration.
Thus, the agent can only use data previously collected by safe policies.
While previous work considers optimizing the average performance using ofﬂine data, we focus on optimizing a risk-averse criterion. In particular, we present the Ofﬂine RiskAverse Actor Critic (O
RAAC)
, a model -free RL algorithm that is able to learn risk -averse policies in a fully ofﬂine setting
We show that O RAAC learns poli- cies with higher risk -averse performance than risk-neutral approaches in different robot control tasks.
Furthermore, considering risk-averse criteria guarantees distributional robustness of the average performance with respect to particular distribution shifts.
We demonstrate empirically that in the presence of natural distribution-shifts, O-RAAC learns policies with good average performance.
I NTRODUCTION
In high-stakes applications, the deployment of highly-performing Reinforcement Learning (RL) agents is limited by prohibitively large costs at early exploration stages (Dulac-Arnold et al., 2019).
To address this issue, the ofﬂine (or batch) RL setting considers learning a policy from a limited batch of pre-collected data.
However, high-stakes decision-making is typically also risk-averse : we assign more weight to adverse events than to positive ones (Pratt, 1978).
Although several algorithms for risk-sensitive RL exist (Howard & Matheson, 1972; Mihatsch & Neuneier, 2002), none of them addresses the ofﬂine setting.
On the other hand, existing ofﬂine RL algorithms consider the average performance criterion and are risk-neutral (Ernst et al., 2005; Lange et al., 2012).
Main contributions
We present the ﬁrst approach towards learning a risk-averse RL policy for high-stakes applications using only ofﬂine data : the O fﬂine R isk-
A verse A ctor- C ritic (O-RAAC).
The algorithm has three components: a distributional critic that learns the full value distribution (Section 3.1), a risk-averse actor that optimizes a risk averse criteria (Section 3.2) and an imitation learner implemented with a variational auto-encoder (VAE) that reduces the bootstrapping error due to the ofﬂine nature of the algorithm (Section 3.3).
In Figure 1, we show how these components interact with each other.
Finally, in Section 4 we demonstrate the empirical performance of O-RAAC.
Our implementation is freely available at Github: https://github.com/nuria95/O-RAAC .
1. R ELATED W ORK Risk-Averse RL
The most common risk-averse measure in the literature is the Conditional Valueat-Risk (CVaR) (Rockafellar & Uryasev, 2002), which corresponds to the family of Coherent RiskMeasures (Artzner et al., 1999), and we focus mainly on these risk-measures.
Nevertheless, other risk criteria such as Cumulative Prospect Theory (Tversky & Kahneman, 1992) or Exponential Utility (Rabin, 2013) can also be used with the algorithm we propose.
In the context of RL, Petrik &
Subramanian (2012); Chow & Ghavamzadeh (2014);
Chow et al.
(2015) propose dynamic programming algorithms for solving the CVaR of the return distribution with known tabular Markov Decision
Processes (MDPs).
For unknown models, Morimura et al.
(2010) propose a SARSA algorithm for (CVaR) optimization but it is limited to the on-policy setting and small action spaces.
To scale to larger systems, Tamar et al.
(2012; 2015) propose on-policy Actor-Critic algorithms for Coherent
Risk-Measures.
However, they are extremely sample inefﬁcient due to sample discarding to compute [Figure] the risk-criteria and the high-variance of the gradient estimate.
While Prashanth et al.
(2016) address sample efﬁciency by considering Cumulative Prospect Theory instead of Coherent Risk-Measures, their algorithm is limited to tabular MDPs and is also on-policy.
Instead, Tang et al.
(2020) propose an off-policy algorithm that approximates the return distribution with a Gaussian distribution and learns its moments using the Bellman equation for the mean and the variance of the distribution.
Instead, we learn the full return distribution without making the Gaussianity assumption (Bellemare et al., 2017).
Perhaps most closely related is the work of Singh et al.
(2020), who consider also a distributional critic but their algorithm is limited to the CVaR and they do not address the ofﬂine RL setting.
Furthermore, they use a sample-based distributional critic, which makes the computation of the CVaR inefﬁcient.
Instead, we modify Implicit Quantile Networks (Dabney et al., 2018) in order to compute different risk criteria efﬁciently.
Although (Dabney et al., 2018) already investigated risk-related criteria, their scope is limited to discrete action spaces (e.g., the Atari domain) in an off-policy setting whereas we consider continuous actions in an ofﬂine setting.
Ofﬂine RL
The biggest challenge in ofﬂine RL is the Bootstrapping Error: a Q-function is evaluated at state-action pairs where there is little or no data and these get propagated through the Bellman equation (Kumar et al., 2019).
In turn, a policy optimized with ofﬂine data induces a state-action distribution that is shifted from the original data (Ross et al., 2011).
To address this, Fujimoto et al.
(2019) propose to express the actor as the sum between an imitation learning component and a perturbation model to control the deviation of the behavior policy.
Other approaches to control the difference between the data-collection policy and the optimized policy include regularizing the policies with the behavior policy using the MMD distance (Kumar et al., 2019) or f-divergences (Wu et al., 2020; Jaques et al., 2019), or using the behavior policy as a prior (Siegel et al., 2020).
An alternative strategy in ofﬂine RL is to be pessimistic with respect to the epistemic uncertainty that arises due to data scarcity.
Yu et al.
(2020) take a model-based approach and penalize the per-step rewards with the epistemic uncertainty of their dynamical model.
Using a model-free approach Kumar et al.
; Buckman et al.
(2020) propose to learn a lower bound of the Q-function using an estimate of the uncertainty as penalty in the target of the equation.
Our work uses ideas from both strategies to address the ofﬂine risk-averse problem.
First, we use an imitiation learner to control the bootstrapping error.
However, by considering a risk-averse criterion, we are also optimizing over a pessimistic distribution compatible with the empirical distribution in the data set.
The connections between risk-aversion and distributional robustness are well studied in supervised learning (Shapiro et al., 2014; Namkoong & Duchi, 2017; Curi et al., 2020; Levy et al., 2020) and in reinforcement learning (Chow et al., 2015; Pan et al., 2019).
P ROBLEM S TATEMENT
We consider a Markov Decision Process (MDP) with possibly continuous s ∈ S and possibly continuous actions a ∈ A , transition kernel P ( ·| s, a ) , reward kernel R ( ·| s, a ) and discount factor γ .
We denote by π a stationary policy, i.e., a mapping from states to distribution over actions.
We have access to a ﬁxed batch data set collected with an unknown behaviour policy π β .
We call d β the joint state, action, reward, next-state distribution induced by the behaviour policy and ρ β the marginal state distribution.
We access this distribution by sampling from the ﬁxed data set.
Similarly, for any policy π , we call d π the joint state, action, reward, next-state distribution induced by π on the MDP.
In risk-neutral RL, the goal is to ﬁnd a policy that maximizes the expected discounted sum of returns E d π (cid:2)(cid:80) ∞ t =1 γ t − 1 R ( ·| S t , A t ) (cid:3) , where the expectation is taken with respect to the stochasticity introduced by the reward kernel, the transition kernel, and the policy.
We deﬁne as
Z π ( s, a ) =
D (cid:80) ∞
t =1 γ t − 1 R ( ·| S t , A t ) as the return distribution conditioned on ( S 1 = s, A 1 = a ) and following π thereafter.
Here =
D denotes equality in distribution.
The risk-neutral RL objective is the expectation of the distribution of Z π .
In risk-averse settings, we replace the expectation with a distortion operator D that is a mapping from the distribution over the returns to the reals.
Thus, the goal is to ﬁnd a policy π that maximizes
[Equation]
With this framework, we address many different risk-averse distortions.
For example, this includes probability weighting functions (Gonzalez & Wu, 1999; Tversky & Kahneman, 1992), the CVaR (Rockafellar & Uryasev, 2002), the mean-variance criteria (Namkoong & Duchi, 2017) or the Wang criteria (Wang, 1996; Rabin, 2013).
O FFLINE R ISK -A VERSE A CTOR -C RITIC (O-RAAC)
We now present our algorithm O-RAAC for ofﬂine-risk averse RL.
One of the main technical challenges in going beyond expected rewards is to ﬁnd an analogue of the Q-function for the particular distortion operator we want to optimize.
Unfortunately, the Bellman target of most risk-analogues does not have a closed-form expression.
Therefore, we instead learn the full distribution of returns as proposed by Bellemare et al.
In Section 3.1, we describe the training procedure for the distributional critic.
Next, in Section 3.2 we deﬁne the actor loss as the risk-distortion operator on the learned return distribution and optimize it using a gradient-based approach.
Up to this point, the actor-critic template is enough to optimize a risk-averse criteria.
However, as we focus on the ofﬂine setting, we need to control the bootstrapping error.
To this end, we use a variational auto-encoder
VAE to learn a generative model of the behavior policy in Section 3.3.
Finally, in Section 3.4 we bring all the pieces together and instantiate our algorithm for different risk distortions D .
3. D ISTRIBUTIONAL C RITIC L EARNING
To learn the distributional critic, we exploit the distributional Bellman equation of returns
Z π ( s, a ) =
D R ( s, a ) + γZ π ( S (cid:48) , A (cid:48) ) for policy evaluation.
The random variables S (cid:48) , A (cid:48) are distributed according to s (cid:48) ∼ P ( ·| s, a ) and A (cid:48)
∼ π ( ·| s (cid:48) ) .
In particular, we represent the return distribution implicitly through its quantile function as proposed by Dabney et al.
We use this representation because many risk distortion operators can be efﬁciently computed using the quantile function of the underlying random variable.
We parameterize the quantile function through a neural network with learnable parameters w .
We express such implicit quantile function as Z πw ( s, a ; τ ) , where τ ∈ [0 , 1] is the quantile level.
Whereas the neural network architecture proposed by Dabney et al.
(2018) is for discrete actions only, we extend it to continuous actions by considering all s , a , and τ as the inputs and only the quantile value as the output.
To learn the parameters w , we use the distributional variant of ﬁtted value-iteration (Bellemare et al., 2017; Munos & Szepesv´ari, 2008) using a quantile Huber-loss (Huber, 1964) as a surrogate of the Wasserstein-distance as proposed by Dabney et al.
To this end, we use a target network with parameters w (cid:48) and compute the temporal difference (TD) error at a sample ( s, a, r, s (cid:48) ) as
[Equation]
) for τ, τ (cid:48) independently sampled from the uniform distribution, i.e., τ, τ (cid:48) ∼ U (0 , 1) and a (cid:48) ∼ π ( ·| s (cid:48) ) .
The τ -quantile Huber-loss is
[Equation]
We prefer the Huber loss over the L 2 or L 1 loss as it is better behaved due to smooth gradientclipping (Mnih et al., 2015).
Finally, we approximate the quantile loss for all levels τ by sampling
N independent quantiles τ and N (cid:48) independent target quantiles τ (cid:48) .
The critic loss is
[Equation]
3. L EARNING A R ISK -A VERSE A CTOR
In risk-averse applications, we generally prefer deterministic policies over stochastic ones because introducing extra randomness is against a risk-averse behavior (Pratt, 1978) and there is no beneﬁt of exploration often associated with stochastic policies in the ofﬂine setting.
Hence, we consider parameterized deterministic policies π θ ( s ) :
S → A .
Given a learned distributional critic, we deﬁne the actor loss as
[Equation]
where D is the operator that models risk aversion.
We consider Markovian policies because these contain the optimal policy for many common risk distortions in the discounted inﬁnite horizon setting (Ruszczy´nski, 2010, Theorem 4).
To minimize the actor loss (5), we use pathwise derivatives of the objective (Mohamed et al., 2020), computed by backpropagating the actor through the learned critic at states sampled from the ofﬂine data set.
Minimizing the actor loss is equivalent to maximizing the risk-averse performance.
We leverage the quantile representation of the critic (c.f. Section 3.1) to compute the risk distortion operator inside the actor loss (5), Given a quantile representation of a distribution, common risk distortions D can be efﬁciently approximated using a sampling-based scheme.
In particular, there exists a quantile sampling distribution P D associated to D such that
[Equation]
For cumulative prospect theory (Tversky & Kahneman, 1992) and coherent risk measures (Artzner et al., 1999), the associated quantile sampling distributions are well-known.
In the particular case of the CVaR, this is known as the Acerbi’s formula (Acerbi & Tasche, 2002)
[Equation]
When other representations are used for the critic, computing the risk-distortion D becomes computationally expensive.
In some particular cases, there are variational formulas to compute the risk-distortion D that require to solve an inner optimization problem.
For example, Singh et al.
(2020) use the common Rockafellar truncated optimization procedure (Rockafellar & Uryasev, 2002) to compute the CVaR, which is sample inefﬁcient and has high variance (Curi et al., 2020).
3. O FF - POLICY TO O FFLINE : C ONTROLLING THE B OOTSTRAPING E RROR .
Up to this point, the actor-critic procedure we describe in Sections 3.1 and 3.2 is theoretically sufﬁcient to learn a risk-averse RL agent.
However, in the ofﬂine setting the bootstrapping error appears: when evaluating the TD-error (2), the Z -value target will be evaluated at actions where there is no data (Kumar et al., 2019) and propagated through the Bellman equation.
To address this issue, (Kumar et al., 2019; Wu et al., 2020; Siegel et al., 2020) propose stochastic policies and penalize deviations from the behaviour policy while optimizing the actor.
As discussed in Section 3.2, on one hand we prefer deterministic policies over stochastic ones for risk-averse optimization and on the other hand, we prefer stochastic policies to avoid overﬁtting to the ﬁxed batch of data.
To this end, we use a similar parameterization to Fujimoto et al.
and decompose the actor in two components: an imitation learning component π IL and a perturbation model ξ θ such that the policy is expressed as:
[Equation]
That is, b is an action sampled from the imitation learning component, ξ θ is a conditionally deterministic perturbation model that is optimized maximizing the actor loss (5) and λ is a hyperparameter that scales the perturbation magnitude.
Thus, all the randomness in our policy arises from the behaviour policy and not from the subsequent optimization.
Furthermore, if we would have access to the behaviour policy π β we could directly replace the imitation learning module by π β .
To learn a generative model of the π IL from state-action pairs from the behaviour distribution d β we use a conditional variational autoencoder (VAE) (Kingma & Welling, 2014), which is also done in Fujimoto et al.
The main advantage of the VAE compared to behavioral cloning (Bain &
Sammut, 1995) is that it does not suffer from mode-collapse which hinders the actor optimization, and compared to inverse imitation learning (Ziebart et al., 2008;
Abbeel & Ng, 2004)
it does not assume that the policy is optimal.
We chose the VAE over Generative Adversarial Networks (Ho &
Ermon, 2016) due to easiness of training (Arjovsky & Bottou, 2017).
The conditional variational autoencoder is a probabilistic model that samples an action b ∼ VAE
[Equation]
according to the generative model
[Equation]
where E φ 1 is the neural network encoder, D φ 2 is the neural network decoder, and we sample the code z using the re-parameterization trick (Kingma & Welling, 2014).
To learn the φ = { φ 1 , φ 2 } parameters we place a prior N (0 , I ) on the code z and minimize the variational lower-bound
[Equation]
[Equation]
Upon a new state s , the generative model of the VAE φ ( s ) is z ∼ N (0 , I ) , b
= D φ 2 ( s, z ) .
3. F INAL A LGORITHM
input Data set, Critic Z w and critic-target Z w (cid:48) , VAE φ = { E φ 1 , D φ 2 } , Perturbation model ξ θ and target ξ θ (cid:48) , modulation parameter λ , Distortion operator D or distortion sampling distribution P D , critic-loss parameters N, N (cid:48) , κ , mini-batch size B , learning rate η , soft update parameter µ .
for t = 1 , . . .
do
Sample B transitions ( Sample N quantiles and quantiles and compute
Compute policy π θ =
b
+ λξ θ ( s, b ) , s.t.
b ∼ VAE φ ( s, a ) as in (9).
Compute critic loss L critic ( w ) in ( ; actor loss L actor ( θ in ( ; VAE loss L VAE ( φ in (
Gradient step w ← w − η ∇L critic ( w ) ; θ ← θ − η ∇L actor ( θ ) ; φ ← φ − η ∇L VAE ( φ ) .
Perform soft-update on w (cid:48) ← µw + ( [Equation] µ
[Equation]
w (cid:48) ; θ (cid:48)
← µθ + (1 − µ ) θ (cid:48) .
end for
We now combine the critic in Section 3.1, the actor in Section 3.2 and the VAE in Section 3.3 and show the pseudo-code of O-RAAC in Algorithm 1.
We replace the expectations in the critic loss (4), actor loss (5) and VAE loss (10) with empirical averages of samples from the data set.
As an ablation, we also propose the RAAC algorithm, in which the actor is parameterized with a neural network and there is no imitation learning component.
The critic’s goal is to learn the reward distribution, the VAE goal is to learn a baseline action for the actor and the goal of the perturbation model is to be risk-averse.
Although Santara et al.
(2018) and Lacotte et al.
(2019) propose risk-averse imitation learning algorithms, these interact with the environment in an on-policy way.
Furthermore, the goal of the imitation learning component in
O-RAAC is not to be risk-averse, but to provide a baseline to the risk-averse perturbation.
O-RAAC requires a distortion metric D as an input.
For different D , it generalizes existing distributional RL algorithms and extends them to the ofﬂine setting.
For example, when D is the expectation operator, then the agent is risk neutral and O-RAAC is an ofﬂine version of D4PG (Barth-Maron et al., 2018).
Likewise, when D is the Rockafellar-truncation operator (Rockafellar & Uryasev, 2002) we recover the algorithm by Singh et al.
(2020) for optimizing the CVaR.
[Table]
E XPERIMENTAL E VALUATION
In this section, we test the performance of O-RAAC using D = CVaR
α =0 .
1 as risk distortion.
We use Acerbi’s formula (7) to compute the risk distortion.
In particular, we ask:
How does RAAC perform as a risk-averse agent in the off-policy setting?
(Section 4.1) 2.
How does O-RAAC perform as a risk-averse agent in the ofﬂine setting?
(Section 4.2) 3.
How does O-RAAC perform as a risk-neutral agent in the ofﬂine setting?
(Section 4.3)
For further details such as hyperparameter selection and extended results please refer to Appendix A.
4. O FF - POLICY S ETTING : R ISK -A VERSE PERFORMANCE
In this experiment, we intend to demonstrate the effectiveness of our algorithm as a risk-averse learner without introducing the extra layer of complexity of the ofﬂine setting.
4.1. E XPERIMENTAL S ETUP
As a toy example, we chose a 1-D car with state s = ( x, v ) , for position and velocity.
The agent controls the car with an acceleration a ∈ [ − 1 , 1] .
The car dynamics with a time step ∆ t = 0 .
1 is x t +1 = x t + v
[Equation]
The control objective is to move the car to x g = 2 . 5 as fast as possible, starting from rest.
To model the risk of crashing or of getting a speed ﬁne, we introduce a penalization when the car exceeds a speed limit ( v > 1 ).
Hence, we use a random reward function given by
[Equation]
where I is an indicator function and B 0 .
2 is a Bernoulli Random Variable with probability p
=
The episode terminates after 400 steps or when the agent reaches the goal.
Benchmarks
We compare RAAC with a risk-neutral algorithm (D4PG algorithm by Barth-Maron et al.
(2018) with IQN and 1-step returns) and WCPG by Tang et al.
(2020), a competing risk-averse algorithm.
Both RAAC and WCPG optimize the 0.1-CVaR of the returns.
4.1. R ESULT D ISCUSSION
In Table 1, we show the results of the experiment.
As expected, D4PG ignores the low probability penalties and learns to accelerate the car with maximum power.
Thus it has the largest expected return but the lowest CVaR. WCPG also fails to maximize the CVaR as it assumes that the return distribution is Gaussian.
In turn, it under-estimates the variance of the return distribution of the maximum acceleration policy.
Consequently, it over-estimates the CVaR of the returns and prefers the latter policy over the maximum-CVaR policy.
In contrast, RAAC learns the full value distribution
Z and computes the CVaR reliably.
Thus, it learns to saturate the velocity below the v = 1 threshold and ﬁnds the highest CVaR policy.
See Appendix A.1 for more experiments in this setting.
Qualitative Evaluation
In Figure 3 in Appendix A.1 we show the different trajectories as an illustration of the resulting risk-averse behavior.
4. O FFLINE S ETTING : R ISK -A VERSE PERFORMANCE
In this experiment, we intend to demonstrate the effectiveness of our algorithm as a risk-averse learner in the ofﬂine setting and in high-dimensional environments.
4.2. E XPERIMENTAL S ETUP
We test the algorithm on a variety of continuous control benchmark tasks on the data provided in the D4RL dataset (Fu et al., 2020).
We use three MuJoCo tasks: HalfCheetah, Walker and Hopper (Todorov et al., 2012).
In particular, we chose the medium (M) and expert (E) variants of these datasets.
Since the tasks are deterministic, we incorporate stochasticity into the original rewards to have a meaningful assessment of risk and to showcase a practical example of when the risk-averse optimization makes sense.
As a risk-averse performance metric, we chose the 0.1-CVaR of the episode returns.
We use the following reward functions:
Half-Cheetah: R t ( s, a ) =
¯ r t ( s, a ) − 70 I v> ¯ v · B 0 .
1 , where ¯ r t ( s, a ) is the original environment reward, v the forward velocity, and ¯ v is a threshold velocity ( ¯ v = 4 for the (M) variant and ¯ v =for the (E) variant).
As with the car example, this high-velocity penalization models a penalty to the rare but catastrophic event of the robot breaking – we want to be risk-averse to it.
We evaluate the
Half-Cheetah for 200 time steps.
Walker2D/Hopper: R t ( s, a ) =
¯ r t ( s, a ) − p
I | θ | > ¯ θ · B 0 . 1 , where ¯ r t ( s, a ) is the original environment reward, θ is the pitch angle, ¯ θ is a threshold angle ( ¯ θ = 0 .
5 for the Walker2d-M/E and ¯ θ = 0 .for
the Hopper-M/E) and p = 30 for the Walker2d-M/E and p = 50 for the Hopper-M/E.
When | θ | > 2¯ θ the robot falls, the episode terminates, and we stop collecting such rewards.
To avoid such situation, we shape the rewards with the stochastic event at θ > ¯ θ .
The maximum duration of the
Walker2D and the Hopper is 500 time steps.
Benchmarks
We optimize O-RAAC for the 0.1-CVaR. To demonstrate its effectiveness optimizing other risk distortions, we optimize O-RAAC using the 0.25-CVaR and the CPW distortion proposed by Tversky & Kahneman (1992); Gonzalez & Wu (1999).
As a competing risk-averse algorithm, we augment WCPG with a VAE to yield the O-WCPG algorithm.
As state-of-the-art risk-neutral agents, we use BEAR (Kumar et al., 2019) and O-D4PG, which is equivalent to BCQ (Fujimoto et al., 2019) with a distributional critic.
As ablations, we compare RAAC (i.e. O-RAAC without the VAE) and the performance of the VAE as a pure imitation learning algorithm.
Finally, we estimate the returns of the behavior policy by evaluating the stochastic reward functions on the data set and estimate its variance by bootstrapping batches of transitions.
4.2. R ESULT D ISCUSSION
In Table 2, we show the results of the experiments.
In all environments, O-RAAC has a higher 0.1-CVaR than the benchmarks.
In environments that terminate, it also has longer duration than competitors.
When optimizing for other risk-averse criteria, O-RAAC still has good 0.1-CVaR performance.
The behavior policy has poor risk-averse performance and the VAE ablation imitates such policy, hence performs poorly too.
On the other side of the spectrum, RAAC performs poorly in all categories.
This indicates that the ofﬂine version enhancement of RAAC is crucial for ofﬂine problems.
O-WCPG performs poorly in terms of the CVaR.
This might be due to the return distribution not being Gaussian.
Finally, BEAR and O-D4PG usually have good risk-neutral performance, particularly in the medium version of these datasets, but poor risk-averse performance.
This is expected as these algorithms are designed to optimize the risk-neutral performance.
Qualitative Evaluation
We visualize the risk-averse performance by looking at the support of the risk-event in Figure 2.
Most of the support of the data set induced by the behavior policy lies in the risky region.
O-RAAC learns to shift the distribution towards the risk-free region (green shaded area).
On the other hand, O-D4PG (risk-neutral) struggles to shift the distribution as it ignores the rare penalties in the risky region.
Only in the Half Cheetah-E experiment O-D4PG manages to shift the distribution towards the safe region.
This is because most of the behavior policy is on the risky region and the average performance of the behavior policy under the new stochastic rewards is low. [Figure]
F r e qu e n c y
Velocity (Half Cheetah-M)
[Figure]
BehaviorSafeRegion
[Figure] Angle (Walker2D-M) [Figure]
Angle (Hopper-M)
[Figure]
F r e qu e n c y
Velocity (Half Cheetah-E)
[Figure] Angle (Walker2D-E) [Figure]
Angle (Hopper-E)
4. O FFLINE SETTING : R ISK -N EUTRAL PERFORMANCE
It is well-known that coherent risk-related criteria have a dual distributional robust criterion formulation (Shapiro et al., 2014; Iyengar, 2005; Osogami, 2012).
In particular, the following holds:
[Equation]
(11) where D π is a dual set of distribution that is induced by the distortion measure D and the distribution d π .
When the distortion set is the expectation, the dual set collapses to a singleton D π = { d π } .
For the CVaR, Chow & Ghavamzadeh (2014) expresses the dual set for MDPs in Proposition 1.
Given this dual result, it is straightforward to show that E d π [ Z π ( x, a )] ≥ D
[ Z π ( x, a )] by deﬁnition of the minimum.
In this sense, optimizing D [ Z π ( x, a )] is equivalent to optimizing a pessimistic estimate of the risk-neutral performance, in a similarly way to Buckman et al.
(2020) and Kumar et al.
4.3. E XPERIMENTAL S ETUP
Despite the goal being to maximize a risk-neutral objective , we evaluate whether it is beneﬁcial to optimize a risk-averse criterion in the ofﬂine setting.
We test this hypothesis using the same setup as in Section 4.2.
Namely, we train O-RAAC using a risk-averse metric, but we evaluate on the average value, a risk-neutral metric.
Benchmarks
We use the same benchmarks as in Section 4.2.
4.3. R ESULT D ISCUSSION
We show the risk-neutral performance in the “Mean” columns in Table 2.
In all data sets, O-RAAC performs better or similar to the benchmarks.
A reason for this is that the episodes of risk-averse agents last longer and thus collect rewards for more time steps.
However, this is not the only reason.
For example, BEAR in the Walker-expert environment has longer episodes and still lower mean returns than O-RAAC.
In the Hopper-expert, RAAC and VAE have similar duration to O-RAAC.
Yet, O-RAAC has a higher average return than each of the ablations.
This indicates that optimizing a risk-averse performance is beneﬁcial when comparing the risk-neutral performance, specially in data sets where the data is not very diverse (e.g., in the “expert” data sets).
Qualitative Evaluation
In Figure
[Table]
we see that in most cases there is a distribution shift between the behaviour distribution and both O-RAAC and O-D4PG.
As the shift increases, we see the beneﬁts of learning using distributionally robust objectives.
As a particular example we take the Hopper-E distribution.
The behavior policy is safe as it does not terminate before the 500 time steps but much of the data is on the risky region.
O-RAAC learns to center the distribution in the safe region and yet hops forward efﬁciently.
On the other hand, O-D4PG also learns to shift away of the risky region.
However, it is not risk averse and it overshoots towards the other end of the risky region, where there is not sufﬁcient data to have good critic estimates.
[Table]
C ONCLUSION
In high-stakes applications
[Table]
decision-making is usually risk-averse and no interactions with the environment are allowed.
For this practical setting, we introduce O-RAAC, the ﬁrst fully ofﬂine riskaverse algorithm.
O-RAAC is compatible with many common risk-averse criteria such as coherentrisk measures or cumulative prospect theory.
Due to the distributional-robust properties of risksensitive criteria, it also optimizes risk-neutral criteria under natural distribution shift that occur in the ofﬂine setting.
Empirically, O-RAAC outperforms other algorithms in terms of risk-averse performance and is competitive with risk-neutral algorithms in terms of risk-neutral performance.
Particularly, in cases where there is not much data diversity, such as in expert data sets, optimizing risk-averse metrics is beneﬁcial due to inherent robustness properties. A E XTENDED E XPERIMENTAL R ESULTS A. C AR
We ran the Car environment for RAAC, D4PG and WCPG using 5 independent random seeds.
We evaluate ﬁnal policies for 1000 interactions and report the averaged results with corresponding standard deviation in Table 1.
In Figure 3, we show the trajectories when following aforementioned policies.
Reward function design
We use the reward function given by
[Equation]
where I is an indicator function and B 0 .
2 is a Bernoulli Random Variable with probability p
=
That is, r f = 370 is a sparse reward that the agent gets at the goal and r d = − 10 is a negative reward that penalizes delays on reaching the goal.
Finally, the agent receives a negative reward of r v = − 25 with probability 0.2 when it exceeds the v > 1 threshold.
As the returns is a sum of bernoulli R.V. we know that it will be a Binomial distribution.
For this particular case, we expect that if the number of steps is large enough, the Gaussianity assumption that WCPG does is good as Binomial distributions are asymptotically Gaussian (Vershynin, 2018).
However, the episode terminates after at most thirteen risky steps and the approximation is not good.
We show in Figure 3 the trajectories for RAAC, D4PG and WCPG.
P o s iti on
RAAC
D4PG
WCPG
Goal
V e l o c it y
Timesteps
A cce l e r a ti on
A. M U J O C O E NVIRONMENTS
We ran 5 independent random seeds and evaluate for 20 episodes the policy every 100 gradient steps for HalfCheetah and 500 gradient steps for Hopper and Walker2d .
We plot the learning curves of the medium variants in Figure 4 and expert variant in Figure 5.
To report the tests in Table 2, we earlystop the policy that outputs the best CVaR and evaluate on 100 episodes with 5 different random seeds.
Behavior policies
For sake of reference, we evaluate the stochastic reward function on the stateaction pairs in the behavior data set.
Unfortunately, the data sets do not distinguish between episodes.
Hence, to estimate the returns, we use the state-action distribution in the data set and split it into chunks of 200 time steps for the Half-Cheetah and 500 time steps for the Walker2D and the Hopper.
We then compute the return of every chunk by sampling a realization from its stochastic reward function.
Finally, we bootstrap the resulting chunks into 10 datasets by sampling uniformly at random with replacement and estimate the mean and CVaR 0 .
1 of the returns in each batch.
We report the average of the bootstrap splits together with the standard deviation in Table 2.
[Figure]
Hopper-M
[Figure]
A. A DDITIONAL E XPERIMENTAL D ETAILS A.3. A RCHITECTURES
We use neural networks as function approximators for all the elements in the architecture.
Critic architecture:
For the critic architecture, we build on the IQN network Dabney et al.
but we extend it to the continuous action setting by adding an additional action input to the critic network, resulting in the function:
[Equation]
12) where ψ s : X → R d , ψ a : A → R d , m sa : R d × d → R n , m saτ : R n → R n , ψ τ : R → R n and f : R n → R
[Figure]
Half
[Figure]
-E
[Figure]
Walker2D
[Figure]
i s ky
S t e p s
[Figure]
For the embedding ψ τ we use a linear function of n cosine basis functions of the form cos( πiτ ) i = 1 .., n , with n = 16 as proposed in Dabney et al.
For ψ s , ψ a we use a multi-layer perceptron (MLP) with a single hidden layer with d = 64 units for the Car experiment and with d = 256 units for all MuJoCo experiments.
For the merging function m sa , which takes as an input the concatenation of ψ s ( s ) and ψ a ( a ) , we use a single hidden layer with n = 16 units.
For the merging function m saτ , we force interaction between its two inputs via a multiplicative function m saτ ( u 1 , u 2 ) =
u 1 (cid:12) u 2 , where u 1 = m sa ( ψ s ( s ) , ψ a ( a )) and u 2 = ψ τ ( τ ) and (cid:12) denotes the element-wise product of two vectors.
For f we use a MLP with a single hidden layer with 32 units
We used ReLU non-linearities for all the layers.
Actor architecture:
The architecture of the actor model is
[Equation]
(13) where ξ : A → R (cid:107)A(cid:107) and b is the output of the imitation learning component.
For the RAAC algorithm we remove b and set λ = 1 .
For the Car experiments, we used a MLP with 2 hidden layers of size 64.
For the MuJoCo experiments, based on Fujimoto et al.
(2019), we used a MLP embedding with 3 hidden layers of sizes 400, 300 and 300.
We used ReLU non-linearities for all the hidden layers and we saturate the output with a Tanh non-linearity.
VAE architecture:
The architecture of the conditional VAE φ is also based on Fujimoto et al.
It is deﬁned by two networks, an encoder E φ 1 ( s, a ) and decoder D φ 2 ( s, z ) .
Each network has two hidden layers of size 750 and it uses ReLU non-linearities.
A.3. H YPERPARAMETERS
All the network parameters are updated using Adam (Kingma & Ba, 2015) with learning rates η = 0 .
001 for the critic and the VAE, and η = 0 .
for the actor model, as in Fujimoto et al.
The target networks for the critic and the perturbation models are updated softly with µ = 0 .
For the critic loss (4) we use N = N (cid:48)
=
32 quantile samples, whereas to approximate the CVaR to compute the actor loss (5) (7) we use 8 samples from the uniform distribution between [0 , 0 . 1] .
In Figure 6, we show an ablation on the effect of the hyper-parameter lambda.
As we can see, a correct selection of lambda is of crucial performance as it trades-off pure imitation learning with pure reinforcement learning.
As λ → 0 , the policy imitates the behavior policy has poor risk-averse performance.
As λ → 1 , the policy suffers from the bootstrapping error and the performance is also low.
We ﬁnd values of λ ∈ [0 . 05 , 0 . 5] to be the best, although the speciﬁc λ is environment dependent.
This observation coincides with those in Fujimoto et al.
(2019, Appendix D.1).
For all MuJoCo experiments, the λ parameter which modulates the action perturbation level was experimentally set to 0.25, except for the HalfCheetah-medium experiment for which it was set to 0.5.
As we can see from Figure 6, this is not the best value of λ , but rather a value that performs well across most environments.
[Figure]