CO PTI DICE:
O FFLINE
C ONSTRAINED
R EINFORCEMENT
L EARNING
VIA
S TATIONARY
D ISTRIBUTION
C ORRECTION E STIMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider the ofﬂine constrained reinforcement learning (RL) problem, in which the agent aims to compute a policy that maximizes expected return while satisfying given cost constraints, learning only from a pre-collected dataset.
This problem setting is appealing in many real-world scenarios, where direct interaction with the environment is costly or risky, and where the resulting policy should comply with safety constraints.
However, it is challenging to compute a policy that guarantees satisfying the cost constraints in the ofﬂine RL setting, since the offpolicy evaluation inherently has an estimation error.
In this paper, we present an ofﬂine constrained RL algorithm that optimizes the policy in the space of the stationary distribution.
Our algorithm, COptiDICE, directly estimates the stationary distribution corrections of the optimal policy with respect to returns, while constraining the cost upper bound, with the goal of yielding a cost-conservative policy for actual constraint satisfaction.
Experimental results show that COptiDICE attains better policies in terms of constraint satisfaction and return-maximization, outperforming baseline algorithms.
I NTRODUCTION
Reinforcement learning (RL) has shown great promise in a wide range of domains, such as complex games (Mnih et al., 2015; Silver et al., 2017) and robotic control (Lillicrap et al., 2016;
Haarnoja et al., 2018).
However, the need to interact with the environment during learning hinders its widespread application to many real-world problems for which executing exploratory behavior in the environment is costly or dangerous.
Ofﬂine RL (also known as batch RL) (Lange et al., 2012;
Levine et al., 2020) algorithms sidestep this problem and perform policy optimization solely from a set of pre-collected data.
The use of existing (ofﬂine) data can make ofﬂine reinforcement learning applicable to real world systems and and has led to a sharp increase in interest in this paradigm.
Recent works on ofﬂine RL, however, mostly assume that the environment is modeled as a Markov decision process (MDP), and standard ofﬂine RL algorithms focus on reward-maximization only (Fujimoto et al., 2019; Wu et al., 2019; Kumar et al., 2019; Siegel et al., 2020; Wang et al., 2020;
Kumar et al., 2020).
In contrast, in real-world domains it is common that the behavior of the agent is subject to additional constraints beyond the reward.
Consider, for example, autonomous driving or an industrial robot in a factory.
Some behaviors may damage the agent itself or its surroundings, and safety constraints should thus be considered as part of the objective of a suitable reinforcement learning system.
One of the ways to mathematically characterize a constrained RL problem is through the formalism of constrained Markov Decision Processes (CMDP) (Altman, 1999).
In
CMDPs taking an action incurs a cost as well as a reward, and the goal is to maximize the expected long-term reward while satisfying a bound on the expected long-term cost.
In this work, we aim to solve the constrained decision making problem in the ofﬂine RL setting, to enable deployment in various safety-critical domains where direct learning interactions are infeasible.
Ofﬂine constrained RL inherits the difﬁculties of ofﬂine unconstrained RL, while introducing additional challenges.
First, since the target policy being optimized deviates from the data-collection policy with no further data collection, distribution shift becomes the central difﬁculty.
To mitigate the distributional shift, existing ofﬂine RL methods frequently adopt the pessimism principle: either by explicit policy (and critic) regularization that penalizes deviation from the data-collection policy (Jaques et al., 2019;
Wu et al., 2019; Kumar et al., 2019; Siegel et al., 2020; Wang et al., 2020; Lee et al., 2020; Kostrikov et al., 2021) or by reward penalty to the uncertain state-action regions (Kumar et al., 2020; Kidambi et al., 2020; Yu et al., 2020).
Second, in ofﬂine constrained RL, the computed policy should satisfy the given cost constraints when it is deployed to the real environment.
Unfortunately off-policy policy evaluation inherently has estimation errors, and it is therefore difﬁcult to ensure that a policy estimated from a ﬁnite dataset will satisfy the constraint when executed in the environment.
In addition, constrained policy optimization usually involves an additional optimization for the Lagrange multiplier associated with the cost constraints.
Actor-critic-based constrained RL algorithms thus have to solve triple (i.e. critic, actor, Lagrange multiplier) intertwined optimization problems (Borkar, 2005;
Tessler et al., 2019), which can be very unstable in practice.
In this paper, we present an ofﬂine constrained RL algorithm that optimizes the state-action stationary distribution directly, rather than the Q-function or the policy.
We show that such treatment obviates the need for multiple estimators for the value and the policy, yielding a single optimization objective that is practically solvable.
Still, naively constraining the cost value may result in severe constraint violation in the real environment, as we demonstrate empirically.
We thus propose a method to constrain the upper bound of the cost value, aiming to compute a policy more robust in constraint violation, where the upper bound is computed in a way motivated by a recent advance in off-policy conﬁdence interval estimation (Dai et al., 2020).
Our algorithm, Ofﬂine C onstrained
Policy Opti mization via stationary DI stribution C orrection E stimation (COptiDICE), estimates the stationary distribution corrections of the optimal policy that maximizes rewards while constraining the cost upper bound, with the goal of yielding a cost-conservative policy for better actual constraint satisfaction.
COptiDICE computes the upper bound of the cost efﬁciently by solving an additional minimization problem.
Experimental results show that COptiDICE attains a better policy in terms of constraint satisfaction and reward-maximization, outperforming several baseline algorithms.
B ACKGROUND
A Constrained Markov Decision Process (CMDP) (Altman, 1999) is an extension of an MDP, formally deﬁned by a tuple M = (cid:104) S, A, T, R,
C
=
{ C k } 1 ..K , ˆ c = { ˆ c k } 1 ..K , p 0 , γ (cid:105) , where S is the set of states, A is the set of actions, T : S ×
A → ∆( S ) is a transition probability, R : S ×
A → R is the reward function, C k : S ×
A → R is the k -th cost function with its corresponding threshold ˆ c k ∈ R , p 0 ∈ ∆( S ) is the initial state distribution, and γ ∈ (0 , 1] is the discount factor.
A policy π : S → ∆( A ) is a mapping from state to distribution over actions.
We will express solving CMDPs in terms of stationary distribution.
For a given policy π , the stationary distribution d π is deﬁned by:
[Equation]
(1) where s 0 ∼ p 0 , a t ∼ π ( s t ) , s t +1 ∼ T ( s t , a t ) for all timesteps t ≥ 0 .
We deﬁne the value of the policy as V R ( π ) :=
E ( s,a ) ∼ d π
[ R ( s, a )] ∈ R and V C ( π ) :
= E ( s,a ) ∼ d π [ C
[Equation]
s, a )] ∈ R K .
Constrained
RL aims to learn an optimal policy that maximizes the reward while bounding the costs up to the thresholds by interactions with the environment:
[Equation]
Lagrangian relaxation is typically employed to solve Eq.
(2), leading to the unconstrained problem:
[Equation]
where λ ∈ R K is the Lagrange multiplier and ˆ c ∈ R K is the vector-valued cost threshold.
The inner maximization in Eq.
(3) corresponds to computing an optimal policy that maximizes scalarized rewards R ( s, a ) − λ
(cid:62) C ( s, a ) , due to the linearity of the value function with respect to the reward function.
The outer minimization corresponds to balancing the cost penalty in the scalarized reward function: if the current policy is violating the k -th cost constraint, λ k increases so that the cost is penalized more in the scalarized reward, and vice versa.
In the ofﬂine RL setting, online interaction with the environment is not allowed, and the policy is optimized using the ﬁxed ofﬂine dataset D
= { ( s 0 , s, a, r, c, s (cid:48) )
i } Ni =1 collected with one or more (unknown) data-collection policies.
The empirical distribution of the dataset is denoted as d D , and we will abuse the notation d D for s ∼ d D , ( s, a ) ∼ d D , ( s, a, s (cid:48) ) ∼ d D .
We abuse ( s 0 , s, a, s (cid:48) ) ∼ d D for s 0 ∼ p 0 , ( s, a, s (cid:48) ) ∼ d D .
We denote the space of data samples ( s 0 , s, a, s (cid:48) ) as X .
A naive way to solve (3) in an ofﬂine manner is to adopt an actor-critic based ofﬂine RL algorithm for max π V R − λ (cid:62) C ( π ) while jointly optimizing λ .
However, the intertwined training procedure of offpolicy actor-critic algorithms often suffers from instability due to the compounding error incurred by bootstrapping out-of-distribution action values in an ofﬂine RL setting (Kumar et al., 2019).
The instability would be exacerbated when the nested optimization for λ is added.
O FFLINE C ONSTRAINED RL VIA S TATIONARY D ISTRIBUTION C ORRECTION E STIMATION
In this section, we present our ofﬂine constrained RL algorithm, C onstrained Policy Opti mization via stationary DI stribution C orrection E stimation (COptiDICE).
The derivation of our algorithm starts by augmenting the standard linear program for CMDP (Altman, 1999) with an additional f -divergence regularization 1 :
[Equation]
where D f ( d || d D ) :=
E ( s,a ) ∼ d D (cid:2) f
(cid:0) d ( s,a ) d D ( s,a ) (cid:1)(cid:3) is the f -divergence between the distribution d and the dataset distribution d D , and α > 0 is the hyperparameter that controls the degree of pessimism, i.e. how much we penalize the distribution shift, a commonly adopted principle for ofﬂine RL (Nachum et al., 2019b;
Kidambi et al., 2020; Yu et al., 2020; Lee et al., 2021).
We assume that d D >and f is a strictly convex and continuously differentiable function with f (1)
= 0 .
Note that when α = 0 , the optimization (4-7) reduces to the standard linear program for CMDPs.
The Bellman-ﬂow constraints (6-7) ensure that d is the stationary distribution of a some policy, where d ( s, a ) can be interpreted as a normalized discounted occupancy measure of ( s, a ) .
Thus, we seek the stationary distribution of an optimal policy that maximizes the reward value (4) while bounding the cost values (5).
Once the optimal solution d ∗ has been estimated, its corresponding optimal policy is obtained by π ∗ ( a | s ) =
[Equation]
Now, consider the Lagrangian for the constrained optimization problem (4-7):
[Equation]
where λ ∈ R K + is the Lagrange multiplier for the cost constraints (5)
[Equation]
and ν ( s ) ∈ R is the Lagrange multiplier for the Bellman ﬂow constraints (6).
Solving (8) in its current form requires evaluation of
T ( s (cid:48) | s, a ) for ( s, a ) ∼ d , which is not accessible in the ofﬂine RL setting.
To make the optimization tractable, we rearrange the terms so that the direct dependence on d is eliminated, while introducing new optimization variables w that represent stationary distribution corrections:
[Equation]
where e λ,ν ( s, a ) :
= R ( s, a ) − λ (cid:62) C ( s, a ) + γ E s (cid:48) ∼ T ( s,a )
[ ν ( s (cid:48) )] − ν ( s ) is the advantage function by regarding ν as a state value function, and w ( s, a ) :
= d ( s,a ) d D ( s,a ) is the stationary distribution correction.
Every term in Eq.
(9) can be estimated from samples in the ofﬂine dataset D :
[Equation]
where ˆ e λ,ν ( s, a, s (cid:48) ) :
=
R ( s, a ) − λ (cid:62) C ( s, a )+ γν ( s (cid:48) ) − ν ( s ) is the advantage estimate using a single sample.
As a consequence, (10) can be optimized in a fully ofﬂine manner.
Moreover, exploiting the strict convexity of f , we can further derive a closed-form solution for the inner maximization in (9) as follows.
All the proofs can be found in Appendix B.
Proposition 1.
For any ν and λ , the closed-form solution of the inner maximization problem in (9) is given by:
[Equation]
Finally, by plugging the closed-form solution (11) into (9), we obtain the following convex minimization problem:
[Equation]
To sum up, by operating in the space of stationary distributions, constrained (ofﬂine) RL can in principle be solved by solving a single convex minimization (12) problem.
This is in contrast to existing constrained RL algorithms that manipulate both Q-function and policy, and thus require solving triple optimization problems for the actor, the critic, and the cost Lagrange multiplier with three different objective functions.
Note also that when λ is ﬁxed and treated as a constant, (12) reduces to OptiDICE (Lee et al., 2021) for unconstrained RL with the scalarized rewards R ( s, a ) − λ (cid:62) C ( s, a ) , without considering the cost constraints.
In order to meet the constraints, λ should also be optimized, and the procedure of (12) can be understood as joint optimization of:
[Equation]
[Equation]
Once the optimal solution of (12), ( λ ∗ , ν ∗ ) , is computed, w ∗ λ ∗ ,ν ∗ ( s, a ) =
d π ∗ ( s,a ) d D ( s,a ) is also derived by (11), which is the stationary distribution correction between the stationary distribution of the optimal policy for the CMDP and the dataset distribution.
3. C OST - CONSERVATIVE C ONSTRAINED P OLICY O PTIMIZATION
Our ﬁrst method based on (12) relies on off-policy evaluation (OPE) using DICE to ensure cost constraint satisfaction, i.e. E ( s,a ) ∼ d D [ w λ,ν ( s, a ) C ( s, a )]
≈ E ( s,a ) ∼ d π [ C ( s, a )] ≤ ˆ c .
However, as we will see later, constraining the cost value estimate naively can result in constraint violation when deployed to the real environment.
This is due to the fact that an off-policy value estimate based on a ﬁnite dataset inevitably has estimation error.
Reward estimation error may be tolerated as long as the value estimates are useful as policy improvement signals: it may be sufﬁcient to maintain the relative order of action values, while the absolute values matter less.
For the cost value constraint, we instead rely on the estimated value directly.
To make a policy robust against cost constraint violation in an ofﬂine setting, we consider the constrained policy optimization scheme that exploits the upper bound of the cost value estimate:
[Equation]
Then, the key question is how to estimate the upper bound of the policy value.
One natural way is to exploit bootstrap conﬁdence interval (Efron & Tibshirani, 1993; Hanna et al., 2017).
We can construct bootstrap datasets D i by resampling from D and run an OPE algorithm on each D i , which yields population statistics for conﬁdence interval estimation { ˆ V C ( π ) i } mi =1 .
However, this procedure is computationally very expensive since it requires solving m OPE tasks.
Instead, we take a different approach in a more computationally efﬁcient way motivated by CoinDICE (Dai et al., 2020), a recently proposed DICE-family algorithm for off-policy conﬁdence interval estimation.
Speciﬁcally, given that our method estimates the stationary distribution corrections w ( s, a ) ≈ d
π ( s,a ) d D ( s,a ) of the target policy π , we consider the following optimization problem for each cost function C k :
[Equation]
where ˜ p ( s 0 , s, a, s (cid:48) )
=
˜ p 0 ( s 0 )˜
p ( s, a )˜ p ( s (cid:48) | s, a ) is the distribution over data samples ( s 0 , s, a, s (cid:48) )
∈ X which lies in the simplex ∆( X ) , and (cid:15) > 0 is the hyperparameter.
In essence, we want to adversarially optimize the distribution over data samples ˜ p
so that it overestimates the cost value by (15).
At the same time, we enforce that the distribution ˜ p should not be perturbed too much from the empirical data distribution d D by the KL constraint (16).
Lastly, the perturbation of distribution should be done in a way that maintains compatibility with the Bellman ﬂow constraint.
The constraint (17) is analogous to the Bellman ﬂow constraint (6) by noting that ˜ p ( s, a ) w ( s, a ) =
˜ p
( s, a )
d π ( s,a ) d D ( s,a ) ≈
d π ( s, a ) .
In this optimization, when (cid:15) = 0 , the optimal solution is simply given by ˜ p ∗ =
d D , which yields the vanilla OPE result via DICE, i.e. E ( s,a ) ∼ d D [ w ( s, a ) C k ( s, a )] .
For (cid:15) > 0 , the cost value will be overestimated more as (cid:15) increases.
Through a derivation similar to that for obtaining (12), we can simplify the constrained optimization into a single unconstrained minimization problem as follows.
We denote ( s 0 , s, a, s (cid:48) ) as x for notational brevity.
Proposition 2.
The constrained optimization problem (15-17) can be reduced to solving the follow- ing unconstrained minimization problem:
[Equation]
(18) where τ ∈ R + corresponds to the Lagrange multiplier for the constraint (16) , and χ ( s ) ∈ R corresponds to the Lagrange multiplier for the constraint (17) .
In other words, min τ ≥ 0 ,χ (cid:96) ( τ, χ ) =
E ( s,a ) ∼ ˜ p ∗ [ w ( s, a ) C k
( s, a )] where ˜ p ∗ is the optimal perturbed distribution of the problem (15-17).
Also, for the optimal solution ( τ ∗ , χ ∗ ) , ˜ p ∗ is given by:
[Equation]
Note that every term in (18) can be estimated only using samples of the ofﬂine dataset D , thus it can be optimized in a fully ofﬂine manner.
This procedure can be understood as computing the weights for each sample while adopting reweighting in the DICE-based OPE, i.e. UpperBound( ˆ V C ( π )) =
E x ∼ d D [˜ ω ∗ ( x ) w ( s, a ) C ( s, a )] where ˜ ω ∗ ( x ) =
(normalized ω ∗ ( x ) of (19) ) .
The weights are given non-uniformly so that the cost value is overestimated to the extent controlled by (cid:15) .
Remark.
CoinDICE (Dai et al., 2020) solves the similar optimization problem to estimate an upper cost value of the target policy π as follows:
[Equation]
It is proven that (20) provides an asymptotic (1 − α ) -upper-conﬁdence-interval of the policy value if (cid:15) :
= ξ α N where ξ α is the (1 − α ) -quantile of the χ 2 -distribution with 1 degree of freedom (Dai et al., 2020).
Compared to our optimization problem (18) , CoinDICE’s (20) involves the additional outer maximization, which is for estimating w ( s, a ) =
˜ d π ( s,a ) d D ( s,a ) , the stationary distribution corrections of the target policy π .
In contrast, we consider the case when w is given, thus solving the inner minimization alone is enough.
Finally, we are ready to present our ﬁnal algorithm COptiDICE, an ofﬂine constrained RL algorithm that maximizes rewards while bounding the upper cost value, with the goal of computing a policy robust against cost violation.
COptiDICE addresses (14) by solving the following joint optimization.
[Equation]
Compared to (13), the additional minimization for ( τ, χ ) is introduced to estimate the upper bound of cost value.
3. P OLICY E XTRACTION
Our algorithm estimates the stationary distribution corrections of the optimal policy, rather than directly obtaining the policy itself.
Since the stationary distribution corrections do not provide a direct way to sample an action, we need to extract the optimal policy π ∗ from w ∗ ( s, a ) =
d π ∗ ( s,a ) d D ( s,a ) , in order to select actions when deployed.
For ﬁnite CMDPs, it is straightforward to obtain π ∗ by π ∗ ( a | s ) =
[Equation] [Equation]
s
[Equation]
However, the same method cannot directly be applied to continuous CMDPs due to the intractability of computing the normalization constant.
For continuous CMDPs, we instead extract the policy using importance-weighted behavioral cloning:
[Equation]
which maximizes the log-likelihood of actions to be selected by the optimal policy π ∗ .
3. P RACTICAL A LGORITHM WITH F UNCTION A PPROXIMATION
For continuous or large CMDPs, we represent our optimization variables using neural networks.
The Lagrange multipliers ν and χ are networks parameterized by θ and φ respectively:
ν θ : S → R is a feedforward neural network that takes a state as an input and outputs a scalar value, and χ φ :
S → R K is deﬁned similarly.
λ ∈ R K
+ and τ ∈ R K
+ are represented by K -dimensional vectors.
For the policy π ψ , we use a mixture density network (Bishop, 1994) where the parameters of a Gaussian mixture model are output by the neural network.
The parameters of the ν θ network are trained by minimizing the loss:
[Equation] [Equation]
where ˆ w ( s, a, s (cid:48) ) :
= ( f (cid:48) ) − 1 (cid:0)
α ( R ( s, a ) − λ (cid:62) C ( s, a ) + γν θ ( s (cid:48) ) − ν θ ( s ))
(cid:1)
+ .
While J λν can be a biased estimate of L ( λ, ν ) in (12) in general due to ( f (cid:48) ) − 1 ( E [ · ]) (cid:54)
=
E [( f (cid:48) ) − 1 ( · )] , we can show that
J λν is an upper bound of L ( ν, λ ) (i.e. we minimize the upper bound), and J λν =
L ( ν, λ ) holds if transition dynamics are deterministic (e.g. Mujoco control tasks).
The parameters of the χ φ network and τ can be trained by:
[Equation]
This involves a logarithm outside of the expectation, which implies that mini-batch approximations would introduce a bias.
Still, we adopt the simple mini-batch approximation for computational efﬁciency, with a moderately large batch size (e.g. 1024), which worked well in practice.
The empirical form of the loss we use is given by:
[Equation] [Equation]
Lastly, λ and the policy parameter ψ are optimized by:
[Equation] [Equation]
The complete pseudo-code is described in Appendix C, where every parameter is optimized jointly.
E XPERIMENTS 4. T ABULAR CMDP S (R ANDOMLY GENERATED CMDP S )
We ﬁrst probe how COptiDICE can improve the reward performance beyond the data-collection policy while satisfying the given cost constraint via repeated experiments.
We follow a protocol similar to the random MDP experiment in (Laroche et al., 2019; Lee et al., 2021) but with cost as an additional consideration.
We conduct repeated experiments for 10K runs, and for each run, a CMDP M is generated randomly with the cost threshold ˆ c = 0 .
We test with two types of data-collection policy π D , a constraint-satisfying policy (i.e. V C ( π D ) = 0 . 09 ) and a constraint-violating policy (i.e. V C ( π D )
= 0 .
Then, a varying number of trajectories N ∈ { 10 , 20 , 50 , 100 , 200 , 500 , 1000 , 2000 } are collected from the sampled CMDP using the constructed data-collection policy π D , which constitutes the ofﬂine dataset D .
Finally, the ofﬂine dataset D is given to each ofﬂine constrained RL algorithm, and its reward and cost performance is evaluated.
For the reward, we evaluate the normalized performance of the policy π by V
[Equation]
∈ ( −∞ , 1] to see the performance improvement of π over π D intuitively, where π ∗ is the optimal policy of the underlying CMDP.
More details can be found in Appendix D.
Ofﬂine constrained RL has been mostly unexplored, thus lacks published baseline algorithms.
We consider the following three baselines.
First, BC denotes the simple behavior cloning algorithm to see whether the proposed method is just remembering the dataset.
Second, Baseline denotes the algorithm that constructs a maximum-likelihood estimation (MLE) CMDP ˆ M using D and then solves the MLE CMDP using a tabular CMDP solver (LP solver) (Altman, 1999).
Third, C-SPIBB is the variant of SPIBB (an ofﬂine RL method for tabular MDPs)
(Laroche et al., 2019), where we modiﬁed SPIBB to deal with the cost constraint by Lagrange relaxation (Appendix D).
Figure 1a presents the result when the data-collection policy is constraint-satisfying.
The performance of BC approaches the performance of the π D as the size of the dataset increases, as expected.
When the size of the ofﬂine dataset is very small, Baseline severely violates the cost constraint when its computed policy is deployed to the real environment (cost red curve in Figure 1a), and it even fails to improve the reward performance over π D (reward red curve in Figure 1a).
This result is expected since Baseline overﬁts to the MLE CMDP, exploiting the highly uncertain state-actions.
This can cause a signiﬁcant gap between the stationary distribution of the optimized policy computed in M and the one computed in ˆ M , leading to failure in both reward performance improvement and cost constraint satisfaction.
To prevent such distributional shift, ofﬂine RL algorithms commonly adopt the pessimism principle, encouraging staying close to the data support.
We can observe that such pessimism principle is also effective in ofﬂine constrained RL: both C-SPIBB (orange) and the naive version of COptiDICE (green) show consistent policy improvement over the data-collection policy while showing better constraint satisfaction.
Still, the pessimism principle alone is not sufﬁcient to ensure constraint satisfaction, raising the need for additional treatment for cost-conservativeness.
Finally, our COptiDICE (blue) shows much stricter cost satisfaction than other baseline algorithms while outperforming Baseline and C-SPIBB in terms of reward performance.
Figure 1b presents the result when the data-collection policy is constraint-violating, where all the baseline algorithms exhibit severe cost violation.
This result shows that if the agent is encouraged to stay close to the constraint-violating policy, it may negatively affect the constraint satisfaction, although the pessimism principle was beneﬁcial in terms of reward maximization.
In this situation, only COPtiDICE (blue) could meet the given constraint in general, which demonstrates the effectiveness of our proposed method of constraining the upper bound of cost value.
Although COptiDICE (blue) shows reward performance degradation when the size of dataset is very small, this is natural in that it sacriﬁces the reward performance to lower the cost value to meet the constraint conservatively, and it still outperforms the baseline in this low-data regime.
[Figure]
4. C ONTINUOUS CONTROL TASKS (RWRL S UITE )
We also evaluate COptiDICE on domains from the Real-World RL (RWRL) suite (Dulac-Arnold et al., 2020), where the safety constraints are employed.
The cost of 1 is given if the task-speciﬁc safety constraint is violated at each time step, and the goal is to compute a policy that maximizes rewards while bounding the average cost up to ˆ c .
Per-domain safety constraints and the cost constraint thresholds are given in Appendix D. Due to lack of an algorithm that addresses ofﬂine constrained RL in continuous action space, we compare COptiDICE with simple baseline algorithms, i.e. BC : A simple behavior-cloning agent, CRR (Wang et al., 2020): a state-of-the-art (unconstrained) ofﬂine RL algorithm, and C-CRR
: the constrained variant of CRR with Lagrangian relaxation where a cost critic and a Lagrange multiplier for the cost constraint are introduced (
Appendix D Since there is no standard dataset for ofﬂine constrained RL, we collected data using online constrained RL agents (C-DMPO; the constrained variant of Distributional MPO) (Abdolmaleki et al., 2018; Mankowitz et al., 2021).
We trained the online C-DMPO with various cost thresholds ˆ c and saved checkpoints at regular intervals, which constitutes the pool of policy checkpoints.
Then, we generated datasets, where each of them consists of β × 100% constraint-satisfying trajectories and the rest constraint-violating trajectories.
The trajectories were sampled by policies in the pool of policy checkpoints.
Since there is a trade-off between reward and cost in general, the dataset is a mixture of low-reward-low-cost trajectories and high-reward-high-cost trajectories.
Figure 2 presents our results in RWRL tasks, where the dataset contains mostly constraint-satisfying trajectories ( β = 0 .
8) , with some cost-violating trajectories.
This type of dataset is representative of many practical scenarios: the data-collecting agent behaves safely in most cases, but sometimes exhibit exploratory behaviors which can be leveraged for potential performance improvement.
Due to the characteristics of these datasets, BC (orange) generally yields constraint-satisfying policy, but its reward performance is also very low.
CRR (red) signiﬁcantly improves reward performance over
BC, but it does not ensure that the constraints are satisﬁed.
C-CRR (brown) incurs relatively lower cost value than BC in Walker, Quadruped, and Humanoid, but its reward performance is clearly worse than BC.
The naive COptiDICE algorithm (green) takes the cost constraint into account but nevertheless frequently violates the constraint due to limitations of OPE.
Finally, COptiDICE (blue) computes a policy that is more robust to cost violation than other algorithms, while signiﬁcantly outperforming BC in terms of reward performance.
We observe that the hyperparameter (cid:15) in (16) controls the degree of cost-conservativeness as expected: larger values of (cid:15) lead to overestimates of the cost value, yielding a more conservative policy.
To study the dependence on the characteristics of the dataset we experiment with different values of β .
Figure 3a show the result for β = 0 .
8 (low-reward-low-cost data), Figure 3b for β = 0 .
5 , and Figure 3c for β = 0 .
2 (high-reward-high-cost data).
These results show the expected trend: more high-reward-high-cost data leads to a joint increase in rewards and costs of all agents.
A simple modiﬁcation of the unconstrained CRR to the constrained one (brown) was not effective enough to satisfy the constraint.
Our vanilla ofﬂine constrained RL algorithm (green), encouraged to stay close to the data, also suffers from severe constraint violation when most of the trajectories are given as the constraint-violating ones, which is similar to the result of Figure 1c-1d.
Finally,
[Figure] [Figure]
(blue) demonstrates more robust behavior to avoid constraint violations across dataset conﬁgurations, highlighting the effectiveness of our method constraining the cost upper bound.
D ISCUSSION AND C ONCLUSION
The notion of safety in RL has been captured in various forms such as risk-sensitivity (Chow et al., 2015; Urp ´ ı et al., 2021;
Yang et al., 2021), Robust MDP (Iyengar, 2005; Tamar et al., 2014), and
Constrained MDP (Altman, 1999), among which we focus on CMDP as it provides a natural formalism to encode safety speciﬁcations (Ray et al., 2019).
Most of the existing constrained RL algorithms (Achiam et al., 2017; Tessler et al., 2019; Satija et al., 2020) are on-policy algorithms, which cannot be applied to the ofﬂine setting directly.
A recent exception is the work by Le et al.
(2019) that also aims to solve constrained RL in an ofﬂine setting, though its method is limited to discrete action spaces and relies on solving an MDP completely as an inner optimization, which is inefﬁcient.
It also relies on the vanilla OPE estimate of the policy cost, which could result in severe constraint violation when deployed.
Lastly, in a work done concurrently to ours, Xu et al.
(2021) also exploits overestimated cost value to deal with the cost constraint, but their approach relies on an actor-critic algorithm, while ours relies on stationary distribution optimization.
In this work, we have presented a DICE-based ofﬂine constrained RL algorithm, COptiDICE.
DICEfamily algorithms have been proposed for off-policy evaluation (Nachum et al., 2019a;
Zhang et al., 2020a;b; Yang et al., 2020b; Dai et al., 2020), imitation learning (Kostrikov et al., 2019), ofﬂine policy selection (Yang et al., 2020a), and RL (Nachum et al., 2019b; Lee et al., 2021), but none of them is for constrained RL.
Our ﬁrst contribution was a derivation that constrained ofﬂine RL can be tackled by solving a single minimization problem.
We demonstrated that such approach, in its simplest form, suffers from constraint violation in practice.
To mitigate the issue, COptiDICE instead constrains the cost upper bound, which is estimated in a way that exploits the distribution correction w obtained by solving the RL problem.
Such reuse of w eliminates the nested optimization in
CoinDICE (Dai et al., 2020), and COptiDICE can be optimized efﬁciently as a result.
Experimental results demonstrated that our algorithm achieved better trade-off between reward maximization and constraint satisfaction than several baselines, across domains and conditions.
A A LGORITHM FOR U NDISCOUNTED CMDP
For γ = 1 , the optimization problem (4-7) should be modiﬁed by adding an additional normalization constraint (cid:80)
[Equation]
Then
[Equation]
we consider the Lagrangian:
[Equation]
where µ ∈ R is the Lagrange multiplier for the normalization constraint (32).
Then, we rearrange the terms so that the direct dependence on d is eliminated, while introducing new optimization variables w ( s, a ) =
d ( s,a ) d D ( s,a ) that represent stationary distribution corrections:
[Equation]
which yields the minmax optimization problem that can be optimized in a fully ofﬂine manner
Due to the strict convexity of f , we can also derive the closed form solution of the inner maximization problem:
[Equation]
(35) which simplify the the minmax problem (34) into the following single minimization problem:
[Equation]
Once the optimal solution ( λ ∗ , ν ∗ , µ ∗ ) are obtained, we can also compute the stationary distribution corrections of the optimal policy using (35).
Still, naively using (36) would similarly result in cost violation when the resulting policy is deployed to the real environment.
We therefore adopt constraining the upper bound of cost value:
[Equation] [Equation]
which completes the brief description of COptiDICE for γ = 1 .
B P ROOFS
Proposition 1.
For any ν and λ , the closed-form solution of
[Equation]
is given by:
[Equation]
Proof.
For a ﬁxed λ and ν , we write the dual of the maximization problem max w ≥
L ( w, λ, ν
[Equation] [Equation]
By the strong duality, it is sufﬁcient to consider KKT condition for ( w ∗ , κ ∗ ) .
Condition 1 (Primal feasibility)
w ∗ ( s, a ) ≥ 0 ∀ s, a
Condition 2 (Dual feasibility) κ ∗ ( s, a ) ≥ 0 ∀ s, a
Condition 3 (Stationarity)
d D ( s, a )( e λ,ν ( s, a ) − αf (cid:48) ( w ∗ ( s, a ))
+ κ ∗ ( s, a ))
= 0 ∀ s, a
Condition 4 (Complementary slackness) w ∗ ( s, a ) κ ∗ ( s, a ) =
∀ s, a
By Stationarity and d D > 0 ,
[Equation]
) (cid:1) ∀ s, a.
For each ( s, a ) , consider either w ∗ ( s, a ) > 0 or w ∗ ( s, a ) = 0 .
(When w ∗ ( s, a ) > 0 ): κ ∗ ( s, a ) = 0 holds by Complementary slackness , which means:
[Equation]
Note that Dual feasibility holds.
Since f (cid:48) is a strictly increasing function, e ν ( s, a ) > αf (cid:48) (0) .
(When w ∗ ( s, a ) = 0 ): By Stationarity and Dual feasibility ,
[Equation]
This concludes w ∗ ν
( s, a ) =
( f (cid:48) ) − 1 (cid:0) 1 α e λ,ν
( s, a ) (cid:1) + .
Proposition 2.
The constrained optimization problem (15-17) can be reduced to solving the follow- ing unconstrained minimization problem:
[Equation]
[Equation]
where τ ∈ R + corresponds to the Lagrange multiplier for the constraint (16) , and χ ( s ) ∈ R corresponds to the Lagrange multiplier for the constraint (17) .
In other words, min τ ≥ 0 ,χ (cid:96) ( τ, χ ) =
E ( s,a ) ∼ ˜ p ∗ [ w ( s, a ) C k
( s, a )] where ˜ p ∗ is the optimal perturbed distribution of the problem (15-17).
Also, for the optimal solution ( τ ∗ , χ ∗ ) , ˜ p ∗ is given by:
[Equation]
Proof.
For the given constrained optimization problem:
[Equation]
We consider the Lagrangian:
[Equation]
where τ ∈ R + is the Lagrange multiplier for KL constraint (43)
[Equation]
χ ( s (cid:48) )
∈ R is the Lagrange multiplier for (44)
[Equation]
and ζ ∈ R is the Lagrange multiplier for the normalization constraint that ensures (cid:80)
[Equation]
˜ p ( s 0 , s, a, s (cid:48) )
= 1 .
Then, we rearrange (45) by:
[Equation] [Equation]
Then
[Equation]
we can compute the closed form solution for the inner-maximization problem.
Consider ∂g ( τ,χ, ˜ p ) ∂
˜ p ( x ) = 0 should hold for each x .
Then,
[Equation]
s 0 ) (cid:1)(cid:17) (48) with some normalization constant described in terms of ζ .
Then, by plugging (48) into (46) with eliminating ζ , we obtain the result.
(41) is also by direct application of (48).
C P SEUDOCODE OF CO PTI DICE Algorithm
COptiDICE
Compute gradients and perform SGD update: 5:
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
(26)) 7: ψ ←
[Equation]
η ∇
[Equation]
J
[Equation]
( ψ ) (Eq. (27))
D E XPERIMENTAL S ETTINGS D. R ANDOM CMDP S
For random CMDP experiments, We follow a similar experimental protocol as (Laroche et al., 2019;
Lee et al., 2021) with additional consideration of cost constraint.
Random CMDP generation For each run, we constructed a random CMDP with | S | = 50 , | A | = 4 , γ = 0 .
95 , and a ﬁxed initial state s 0 .
The transition probability is constructed randomly with connectivity of 4, i.e. for each ( s, a ) , we sample 4 states uniformly, and then, the transition probabilities to those states are determined by Dirichlet(1 , 1 , 1 , 1) .
The reward of 1 is given to a single state that minimizes the optimal policy’s reward value at s 0 , and 0 is given anywhere else.
This reward design can be roughly understood as choosing a goal state that is most difﬁcult to reach from s 0 .
The cost function is generated randomly, i.e. C ( s, a ) ∼ Beta(0 . 2 , 0 . 2) for ∀ s ∈ S, a ∈ { a 2 , a 3 , a 4 } and C ( s, a 1 ) = 0 to ensure existence of a feasible policy of the CMDP.
Lastly, the cost threshold ˆ c = 0 . 1 is used.
Data-collection policy generation To construct the data-collection policy, we consider the following entropy-regularized CMDP:
[Equation]
[Equation]
(50) where ˆ c D ∈ { 0 . 09 , 0 . 11 } is the hyperparameter that determines the cost value of π D .
Starting from σ = 0 , we gradually increase σ (i.e. adding stochasticity to policy, which adversarially affect the reward performance) until the reward performance of the computed policy reaches 0 .
9 × V R ( π ∗ ) where π ∗ is the optimal policy of the given CMDP.
After the data-collection policy π D is generated, we sample trajectories using π D , which constitutes the ofﬂine dataset D .
We conducted experiments for a varying number of trajectories, i.e. (the number of trajectories) ∈ { 10 , 20 , 50 , 100 , 200 , 500 , 1000 , 2000 } .
Each episode is terminated either when 50 time steps have reached or the agent reached the goal state that yields a non-zero reward.
Hyperparameters For C-SPIBB, we used N ∧ = 5 .
For COptiDICE, we used α = 1 N and (cid:15) = 0 .
N , where N denotes the number of trajectories in D .
We also used f ( x ) = 12 ( x − 1) 2 , which corresponds to χ 2 -divergence.
C-SPIBB: Constrained variant of SPIBB C-SPIBB solves the ofﬂine RL problem with respect to R − λC via SPIBB while updating λ in the direction of ( ˆ V C ( π SPIBB )
− ˆ c ) , where ˆ V C ( π SPIBB ) is evaluated using the MLE CMDP.
D. RWRL C ONTROL T ASKS
Network architecture and hyperparameters We used the ACME framework (Hoffman et al., 2020).
For the ν θ network and the χ φ network, we used LayerNormMLP with hidden sizes of [512 , 512 , 256] .
For the policy network π ψ , we used the network architecture used in CRR (Wang et al., 2020).
Speciﬁcally, we the π ψ network consists of 4 ResidualMLP blocks with hidden size of 1024 and a mixture of Gaussians policy head with 5 mixture components.
We used the batch size of 1024.
We use Adam optimizer with learning rate 3e-4.
Similar to ValueDICE (Kostrikov et al., 2021), we additionally adopt gradient penalties (Gulrajani et al., 2017) for the ν θ network and the χ φ network with coefﬁcient 10e-5.
We performed grid search for α ∈ { 0 . 01 , 0 . 05 , 0 . 1 } for each domain.
We used the following f as in OptiDICE (Lee et al., 2021):
[Equation]
Task speciﬁcation We conduct experiments on domains using the RWRL suite with safety-spec.
The safety coefﬁcient is a ﬂag in the RWRL suite with safety-spec, and its value can be between 0. and 1.0.
Lowering the value of the ﬂag incurs more safety-constraint violation (cost of 1).
Originally, each domain has multiple types of safety constraints, but we use only one of them, which was the hardest safety constraint to be satisﬁed by an online constrained RL agent.
In summary, we used the following task speciﬁcations (safety coefﬁcients, name of the used safety constraint, cost threshold ˆ c ).
• Cartpole (realworld-swingup): safety-coeff=0.3, slider pos , ˆ c = 0 .
1 , • Walker (realworld-walk): safety-coeff=0.3, joint velocity constraint ˆ c = 0 .
• Quadruped (realworld-walk): safety-coeff=0.5, joint angle constraint ˆ c = 0 .
Humanoid (realworld-walk): safety-coeff=0.5, joint angle constraint , ˆ c = 0 .
The ofﬂine dataset consists trajectories of 1000 episodes for Cartpole and Walker, and 5000 episodes for Quadruped and Humanoid.
C-CRR: Constrained variant of CRR C-CRR additionally introduces the cost critic Q C and the
Lagrange multiplier λ for the cost constraint.
Then, the reward critic and the cost critic are trained by minimizing TD-error for reward and cost respectively.
The actor is trained by weighted behaviorcloning: max π E ( s,a ) ∼ d D [exp( ˆ A ( s, a ) /β ) log π ( a | s )] where ˆ A ( s, a ) =
( Q R ( s, a ) − λQ C ( s, a )) −m (cid:80) mj
=1 ( Q R ( s, a j ) − λQ C ( s, a j )) , with a j ∼ π ( ·| s ) .
This corresponds to optimizing the policy with respect to the scalarized reward R − λC .
The Lagrange multiplier λ is updated in the direction of ( E ( s,a ) ∼ d D [ Q C ( s, a )] − ˆ c ) .
E A BLATION E XPERIMENTS ON DIFFERENT COST THRESHOLDS
In order to see the sensitivity of the proposed method to different cost thresholds
[Figure]
we conduct ablation experiments on different cost thresholds using randomly generated CMDPs.
The experimental setup is identical to the one described in Section 4.
[Figure]
On the same data-collection policy (whose cost value is either 0.09 or 0.11) and the same ofﬂine dataset as in Figure
[Figure]
we tested each algorithm against different target cost thresholds ˆ c ∈ { 0 . 08 , 0 . 09 , 0 . 10 , 0 .
Each row in Figure 4 presents the result for each target cost threshold, ranging from ˆ c = 0 .
08 to ˆ c = 0 .
Our COptiDICE (blue) shows consistent robustness to constraint violation on varying cost thresholds
[Figure]
while other algorithms fail to meet the constraints especially when the threshold is set to low values (e.g. rows 1 and 2).
F A DDITIONAL E XPERIMENTS USING M IXTURE D ATASET
In Section 4.
[Figure]
we demonstrated the results when the ofﬂine dataset was collected by a single datacollection policy.
However, in real-world situations, it would be common that data-collecting agents act safely in most cases but have some unsafe attempts.
To simulate this scenario
[Figure]
we conduct additional experiments on the use of a mixture dataset, where the dataset is collected by both constraintsatisfying and constraint-violating policy
[Figure]
Figure 5 presents the result, where the dataset is collected by two policies, the cost-satisfying one (i.e. its cost value is 0.09) and the cost-violating one (i.e. its cost value is 0.11).
The overall trend remains the same as in Figure 1 (Figure 5a
≈ Figure 1a, Figure 5b ≈ Figure 1b): when the dataset consists of mostly cost-satisfying trajectories, baseline algorithms exhibits less constraint violation, while they show more constraint violation when the dataset consists of mostly cost-violating ones.
Our COptiDICE (blue) still shows much stricter cost satisfaction than other baseline algorithms, demonstrating a better trade-off between reward maximization and constraint satisfaction than baselines.
G D ISCUSSION ON THE D ATASET C OVERAGE A SSUMPTION
Although we have made an assumption d D > 0 in Section 3, this is not strictly required for the ofﬂine RL algorithm to work in practice.
We adopted this assumption same as OptiDICE (Lee et al., 2021) for the simplicity of describing the algorithm derivation from Eq (4-7) to Eq (10): the assumption makes Eq.
(10) correspond to solving the underlying true CMDP of Eq.
If we take this assumption off, Eq.
(10) then becomes equivalent to solving the reduced CMDP where the state and action spaces are limited to the support of d D .
To see this, we consider the following constrained optimization problem, where the optimization variables
d ( s, a ) are deﬁned only for the ( s, a ) who are within the support of d D .
We denote ˆ p 0 and ˆ T as the empirical initial state distribution and the empirical transition function respectively.
[Equation]
[Equation]
Then, by considering the Lagrangian for (51) and following the similar derivation of Eq.
(8-9), we can ﬁnally arrive at Eq.
(10) without any assumption on the coverage of d D , due to the fact that
[Equation]
always holds.
In other words, our ofﬂine RL algorithms that rely on Eq.
(10) essentially solve the reduced CMDP that is limited to the support of d D .
Furthermore, solving the reduced CMDP is a valid method in the ofﬂine RL setting since we may want to optimize the policy only within the dataset support in order to prevent unexpected performance degradation by out-of-distribution actions.