E XPLICIT P ARETO F RONT O PTIMIZATION FOR C ONSTRAINED R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Many real-world problems require that reinforcement learning (RL) agents learn policies that not only maximize a scalar reward, but do so while meeting constraints, such as remaining below an energy consumption threshold.
Typical approaches for solving constrained RL problems rely on Lagrangian relaxation, but these suffer from several limitations.
We draw a connection between multi-objective RL and constrained RL, based on the key insight that the constraint-satisfying optimal policy must be Pareto optimal.
This leads to a novel, multi-objective perspective for constrained RL.
We propose a framework that uses a multi-objective RL algorithm to ﬁnd a Pareto front of policies that trades off between the reward and constraint(s), and simultaneously searches along this front for constraint-satisfying policies.
We show that in practice, an instantiation of our framework outperforms existing approaches on several challenging continuous control domains, both in terms of solution quality and sample efﬁciency, and enables ﬂexibility in recovering a portion of the Pareto front rather than a single constraint-satisfying policy.
I NTRODUCTION
Deep reinforcement learning (RL) has shown great potential for training policies that optimize a single scalar reward.
Recent approaches have exceeded human-level performance on Atari (Mnih et al., 2015) and Go (Silver et al., 2016), and have also achieved impressive results in continuous control tasks, including robot locomotion (Lillicrap et al., 2016; Schulman et al., 2017), acrobatics (Peng et al., 2018), and real-world robot manipulation (Levine et al., 2016; Zeng et al., 2019).
However, many problems, especially in the real world, require that policies meet certain constraints.
For instance, we might want a factory robot to optimize task throughput while keeping actuator forces below a threshold, to limit wear-and-tear.
Or, we might want to minimize energy usage for cooling a data center while ensuring that temperatures remain below some level (Lazic et al., 2018).
Such problems are often encoded as constrained Markov Decision Processes (CMDPs) (Altman, 1999), where the goal is to maximize task return while meeting the constraint(s).
Typical approaches for solving CMDPs use Lagrangian relaxation (Bertsekas, 1999) to transform the constrained optimization problem into an unconstrained one.
However, existing Lagrangian-based approaches suffer from several limitations.
First, because the relaxed objective is a weighted-sum of the task return and constraint violation, this assumes a convex Pareto front (Das & Dennis, 1997).
In addition, when the constraint is difﬁcult to satisfy, in practice policies can struggle to obtain task reward.
Finally, such approaches typically produce a single policy that satisﬁes a speciﬁc constraint threshold.
However, the exact constraint threshold may not be known in advance, or one may prefer to choose from a set of policies across a range of acceptable thresholds.
We aim to achieve the goal of CMDPs (i.e., ﬁnding a constraint-satisfying policy that maximizes task return) while avoiding these limitations, by introducing a novel, general framework based on multi-objective MDPs (MO-MDP).
A MO-MDP can be seen as a CMDP where the constrained objectives are instead unconstrained.
Our key insight is that if we have access to the Pareto front, then we can ﬁnd the optimal constraint-satisfying policy by simply searching along this front.
However, ﬁnding the entire Pareto front is unnecessary if only a relatively small portion of the policies along the Pareto front meet the constraints.
Therefore, we propose to also simultaneously prioritize learning for the preferences (i.e., trade-offs between reward and cost) that are most likely to produce policies that satisfy the constraints, and thus cover the relevant part of the Pareto front.
To our knowledge, there is no existing framework for applying multi-objective RL algorithms to constrained RL problems.
Our main contribution is a general framework that enables this, by learning which preferences produce constraint-satisfying policies.
This framework can be combined with any multi-objective RL algorithm that learns an approximate Pareto front of policies.
Our second contribution is to extend a state-of-the-art multi-objective RL algorithm, multi-objective maximum a posteriori policy optimization (MO-MPO) (Abdolmaleki et al., 2020), to learn the Pareto front in a single training run.
We use this extension of MO-MPO within our framework, and call the resulting algorithm constrained MO-MPO .
We show in practice, constrained MO-MPO outperforms existing approaches on challenging continuous control tasks with constraints.
R ELATED WORK
Constrained reinforcement learning.
Constrained RL algorithms seek policies that meet the desired constraints at deployment time.
Most approaches use Lagrangian relaxation (Bertsekas, 1999).
Recent Lagrangian-based approaches claim convergence to constraint-satisfying (Tessler et al., 2019) or optimal (Paternain et al., 2019) solutions, but this is debatable (Szepesvari, 2020).
Recent works seek to stabilize this optimization by approximating the reward and cost functions with convex relaxations (Yu et al., 2019) or by utilizing derivatives of the constraint function (Stooke et al., 2020).
Other works have applied Lagrangian relaxation to mean-value constraints (Tessler et al., 2019), convex set constraints (Miryooseﬁ et al., 2019), and local constraint satisfaction (Bohez et al., 2019).
Existing Lagrangian approaches involve linear scalarization, however, and thus cannot ﬁnd solutions that lie on concave portions of the true Pareto front (Das & Dennis, 1997).
In contrast, we build on a multi-objective RL algorithm that does not rely on scalarization, and in practice, our approach can indeed ﬁnd constraint-satisfying solutions on a concave Pareto front (see Sec.
5.1, humanoid walk ).
Safe reinforcement learning.
In safe RL, the aim is to achieve constraint satisfaction not only during deployment, but also during learning (Garc ´ ıa & Fern ´ andez, 2015).
Recent works for deep RL modify the policy improvement step to guarantee that the policy will never violate constraints during training (Achiam et al., 2017; Berkenkamp et al., 2017;
Chow et al., 2018; 2019;
Yang et al., 2020;
Zhang et al., 2020).
These approaches require, however, that the initial policy meets (or almost meets) the constraints.
Otherwise, performance degrades substantially and constrained policy optimization (CPO) (Achiam et al., 2017), for example, performs worse than Lagrangian-based approaches (Ray et al., 2019).
The aim of our work is to ﬁnd better solutions for constrained RL, rather than safe RL.
We discuss in Sec.
6 how our method can be extended to reduce constraint violation during training.
Multi-objective reinforcement learning.
Our approach is built on ideas from multi-objective RL (MORL), which consists of single-policy and multi-policy approaches.
Single policy methods learn a policy that is optimal for a given setting of reward preferences.
Most rely on linear scalarization (Roijers et al., 2013), which restricts solutions to the convex portions of the Pareto front and can be sensitive to reward scales.
Non-linear scalarizations have been proposed (Tesauro et al., 2008;
Van Moffaert et al., 2013; Golovin & Zhang, 2020), but these are harder to combine with value-based
RL and have seen limited use in deep RL.
Recently, Abdolmaleki et al.
(2020) introduced MO-MPO, where the preferences represent per-objective constraints on the policy improvement step.
MO-MPO does not rely on scalarization and is thus invariant to reward scales.
Multi-policy MORL aims to ﬁnd a set of policies that covers the whole Pareto front.
Recent approaches learn a manifold in parameter space that optimizes the hypervolume of the Pareto front (Parisi et al., 2016; 2017).
While such approaches could be combined with our framework, their scalability to deep RL remains to be shown.
Other works combine single policy approaches with a general objective to optimize hypervolume (Xu et al., 2020).
The instantiation of our framework is similar in spirit to such two-level methods, but applied to a different problem setting.
B ACKGROUND AND NOTATION 3. C ONSTRAINED M ARKOV D ECISION P ROCESSES
A constrained Markov Decision Process (CMDP) consists of states s ∈ S , actions a ∈ A , an initial state distribution p 0 ( s ) , a transition function p ( s (cid:48) | s, a ) , reward functions { r k ( s, a ) } K k =0 , constraint thresholds { c k } Kk =1 , and a discount factor γ ∈ [0 , 1] .
The ( K + 1 ) reward functions consist of a task reward r 0 and constrained rewards r 1: K ; we will refer to these together as objectives .
A policy π ( a | s ) maps from a given state to a distribution over actions.
The optimal solution to a constrained MDP is a policy that maximizes the expected return for the task reward r 0 , while ensuring that the expected return for all constrained rewards r 1: K satisfy their respective thresholds:
[Equation]
where E π is shorthand for the expectation over trajectories when following the policy π , given a ﬁxed initial state distribution.
Here the constraint thresholds are lower bounds, without loss of generality.
The value function V πk ( s ) of a policy π is its expected return for objective r k when starting from state s .
The action-value function Q πk ( s, a ) denotes the expected return for objective r k after taking action a in state s and thereafter acting according to the policy: Q πk ( s, a )
=
r k ( s, a ) + γ E s (cid:48) ∼ p ( s (cid:48) | s,a )
[ V πk ( s (cid:48) )] .
Most approaches for solving CMDPs are based on Lagrangian relaxation, where the constrained problem is turned into the following unconstrained optimization problem:
[Equation]
These approaches alternate between optimizing for the Lagrange multipliers λ k and the policy π .
While our multi-objective perspective on constrained RL aims to achieve the goal of CMDPs—ﬁnding a constraint-satisfying policy that maximizes task return—it does not directly solve (1).
Nonetheless, we will show our approach outperforms those that directly solve (1) via Lagrangian relaxation.
3. M ULTI - OBJECTIVE REINFORCEMENT LEARNING
In order to achieve the goal of CMDPs, we propose to leverage advances in multi-objective RL.
A multi-objective MDP (MO-MDP) is deﬁned in the same way as a CMDP, except without constraint thresholds.
Because there are multiple unconstrained objectives, there is not a single optimal policy.
Instead, there is a set of optimal policies, called the Pareto front .
A policy is Pareto optimal if there is no other policy that improves its return for one objective without decreasing return for another.
Preferences.
Each policy on the Pareto front is the optimal policy for a particular setting of preferences (i.e., desired trade-off over objectives).
1 This is typically encoded via a preference vector (cid:15) , in which each element (cid:15) k represents the relative importance of the corresponding objective r k .
Connection to CMDPs.
Given a CMDP {S , A , p 0 , p, r 0: K , c 1: K , γ } , consider the corresponding MO-MDP {S , A , p 0 , p, r 0: K , γ } .
Regardless of the thresholds c 1: K , the optimal policy for the CMDP must lie on the Pareto front for this corresponding MO-MDP.
Thus, if we knew the Pareto front, we could simply search along it to ﬁnd the constraint-satisfying policy that maximizes task return.
Multi-objective Maximum a Posteriori Policy Optimization (MO-MPO).
Whereas most MORL algorithms are based on linear scalarization, MO-MPO takes a distributional approach to training policies.
For each policy improvement step, MO-MPO ﬁrst computes a non-parametric policy for each objective r k that improves the (parametric) policy with respect to that objective, subject to a non-negative constraint (cid:15) k on the KL-divergence between the improved and old policies.
Then, the policy is updated via supervised learning on the sum of these non-parametric policies.
Intuitively, (cid:15) k deﬁnes the inﬂuence of objective r k on the ﬁnal policy; an objective with larger (cid:15) k has more inﬂuence.
However, MO-MPO can only train policies for a single preference setting (cid:15) .
Sec. 4. explains how we extend MO-MPO to train preference-conditioned policies π ( a | s, (cid:15) ) , in order to use it in our framework.
We choose to extend MO-MPO because it does not suffer from the limitations of linear scalarization, and has been shown to outperform such approaches (Abdolmaleki et al., 2020).
A PPROACH
Problem Statement.
For a given CMDP, we seek to ﬁnd the preference vectors (cid:15) in the corresponding MO-MDP that produce constraint-satisfying action policies π ( a | s, (cid:15) ) .
To do this, we want to ﬁnd the distribution π ( (cid:15) ) that maximizes the probability that the action policies satisfy the constraints.
Overview of Framework.
Our framework enables applying multi-objective RL algorithms to solve constrained RL problems.
It is inspired by the observation that for any CMDP, there exist preference settings (cid:15) for the objectives that lead to the desired constraint-satisfying solutions.
We propose to ﬁnd these preference settings by optimizing for a hierarchical policy π ψ ( (cid:15) )
π θ ( a | s, (cid:15) ) that ﬁrst selects a preference setting (cid:15) and then takes an action conditioned on (cid:15) .
We also maintain a policy evaluation function Q πk ( a, s, (cid:15) ) for each objective.
To train this policy, we decouple learning the preference policy π ψ ( (cid:15) ) from learning the action policy π θ ( a | s, (cid:15) ) .
Formally, we alternate between optimizing two sub-problems:
Learn a new action policy π new ( a | s, (cid:15) ) given the current distribution of preferences π old ( (cid:15) ) .
Learn a new preference policy π new ( (cid:15) ) given the current action policy π old ( a | s, (cid:15) ) .
We aim to converge to an optimal policy π ∗ ( (cid:15) )
π ∗ ( a | s, (cid:15) ) that not only satisﬁes the constraints but is also Pareto optimal.
Note that we can use different time scales for alternating between Steps 1 and 2.
One extreme is to ﬁrst fully optimize for a Pareto front curve given an initial preference distribution, and then fully optimize for solutions that satisfy the constraints.
In our experiments we take the other extreme: we alternate between taking one learning step on π ψ ( (cid:15) ) and π θ ( a | s, (cid:15) ) independently.
For Step 1, any multi-objective RL algorithm capable of training preference-conditioned policies can be used.
Sec. 4.1 describes how we use MO-MPO for Step 1, which requires non-trivially extending it to train preference-conditioned policies.
For Step 2, the notion of learning a preference policy is the core novel component of our framework: this is what enables us to apply multi-objective RL algorithms to constrained RL problems.
Sec.
4.2 describes how to do this.
4. L EARNING PREFERENCE - CONDITIONED ACTION POLICIES (S TEP 1)
In this section, we explain how we extend MO-MPO (Abdolmaleki et al., 2020) to learn preferenceconditioned policies for the current preference distribution π old ( (cid:15) ) .
MO-MPO can only train policies π ( a | s ) for a single preference (cid:15) , so it cannot be directly used.
This section describes how we extend MO-MPO to train a single policy
π θ ( a | s, (cid:15) ) conditioned on preference parameters (cid:15) ∼ π old ( (cid:15) ) , that represents the entire Pareto front.
This is a non-trivial extension, that requires making both the action policy and Q-functions preference-conditioned, and replacing the scalar temperature with a preference-conditioned temperature network, as well as modifying the underlying MO-MPO optimization principles.
We also introduce hindsight relabeling of preferences to stabilize off-policy learning and improve sample efﬁciency.
Our extended version is a policy iteration algorithm with two steps:
• Policy evaluation: learn preference-conditioned Q-functions for all objectives.
• Policy improvement: improve the preference-conditioned policy according to Q-functions.
Preference-conditioned policy evaluation.
In this step, we train a separate Q-function per objective (coined Q-decomposition by Russell & Zimdars 2003), to evaluate the current policy π old ( (cid:15) )
π old ( a | s, (cid:15) ) under the state distribution µ ( s ) .
To learn the Q-function, for each learning step we sample L transitions from the replay buffer { s i , a i , { r ki } Kk , s (cid:48) i } Li .
Since our Q-functions Q πk ( a, s, (cid:15) ) are also conditioned on preferences, we use hindsight relabeling to augment the states with preference parameters sampled from the current preference policy (cid:15) i ∼ π old ( (cid:15) ) , resulting in transitions { [ s i , (cid:15) i ] , a i , { r ki } Kk , [ s (cid:48)
i , (cid:15) i ] } Li .
Any policy evaluation algorithm can be used to learn these
Q-functions.
We use distributional policy evaluation with 5-step return (Barth-Maron et al., 2018).
Preference
-conditioned policy improvement
The policy improvement step assumes a state distribution µ ( s ) , per-objective Q functions Q old k ( s, a, (cid:15) ) , the current action policy π old ( a | s, (cid:15) ) , and the current preference policy π old ( (cid:15) )
It consists of two sub-steps: 1) ﬁnding per-objective improved action distributions, and 2) distilling these distributions into a new preference-conditioned action policy via supervised learning.
Finding per-objective distributions: To ﬁnd per-objective improved action distributions q k ( a | s, (cid:15) ) , we optimize the following RL optimization problem for each objective:
[Equation]
We can solve this problem in closed form to obtain
[Equation]
where η ω k ( (cid:15) k ) is a preference-dependent temperature function for objective r k , and is parameterized by ω k .
This temperature function is obtained by minimizing the following dual function:
[Equation]
In practice, we maintain a single function η ω ( (cid:15) ) with shared parameters ω for all objectives.
To approximate the expectations over the preference distribution and state distribution, we draw L states from the replay buffer and L preferences (cid:15) from the preference policy π old ( (cid:15) ) .
To approximate the integrals over a , for each ( s, (cid:15) ) pair we sample M actions from the current policy π old ( a | s, (cid:15) ) .
Learning a new parameterized action policy: After obtaining per-objective improved policies, we use supervised learning to distill these distributions into a new parameterized policy:
[Equation]
subject to a trust region with bound β > 0 for more stable learning.
To solve this optimization, we use Lagrangian relaxation as described in Abdolmaleki et al.
4. L EARNING PREFERENCE POLICIES (S TEP 2)
We will now explain how to optimize the preference distribution π ψ ( (cid:15) ) for selecting action policies π old ( a | s, (cid:15) ) with better constraint satisfaction.
To achieve this, we deﬁne a ﬁtness function f
k that evaluates satisfaction of constraint threshold c k , given the current action policy π old ( a | s, (cid:15) ) and
Q-function Q old k ( s, a, (cid:15) ) .
The ﬁtness function should be chosen based on the problem and form of the constraint.
In our empirical evaluation, for equality constraints we use a ﬁtness function that penalizes the difference between the expected Q-values and constraint threshold, and for inequality constraints it penalizes the amount the expected Q-values fall under the threshold:
[Equation]
Given a chosen ﬁtness function, we now can use any off-the-shelf RL algorithm to optimize the preference policy π ψ ( (cid:15) ) , initialized by π old ( (cid:15) ) , to maximize the ﬁtness function.
See Appendix D for more details on the general procedure underlying our algorithm (Appendix, Algorithm 1).
E XPERIMENTS
We ﬁnd that our approach of Pareto front optimization applied to MO-MPO (i.e., constrained MOMPO), ﬁnds solutions that are on par with those found by MORL algorithms (Appendix C.1).
In our experiments, we compare constrained MO-MPO against existing RL algorithms for constrained
MDPs: for a number of challenging continuous control tasks, we analyze the quality of policies found for a range of constraint thresholds.
Unless otherwise mentioned, we use (6) as the ﬁtness function.
For action policies we use Gaussian distributions parameterized by neural networks, and for the preference policy in constrained MO-MPO, we use a discrete distribution.
Architecture details and learning hyperparameters are described in Appendix A.
Domains.
We evaluate our approach in two continuous control domains.
First, we use the humanoid run and walk tasks from DeepMind Control Suite (Tassa et al., 2018).
This domain has a challenging 21-dimensional action space.
The agent receives a shaped reward for running at 10 m/s or walking at 1 m/s.
The constraint is imposed on the expected negative control norm (i.e., −(cid:107) a (cid:107) 2 ), which roughly captures the “energy” expended by the agent.
This is relevant for real-world settings (e.g., in robotics or control tasks), where energy consumption often needs to be taken into account.
We also evaluate on the level-two point mass tasks from the Safety Gym suite (Ray et al., 2019), which we refer to as point goal , button , and push .
The agent receives a sparse reward for either reaching a goal location or pushing a box to a goal.
The constraint is imposed on the expected per-episode cost, which is incurred by running into or over objects.
There are several types of objects, each with its own cost.
Unless otherwise mentioned, the constraint is with respect to the total cost.
Pareto plots.
When we plot learned Pareto fronts for the tasks (e.g., in Fig.
1, top), the x-coordinate is average per-episode negative cumulative cost and the y-coordinate is average per-episode task reward.
For both axes, higher values are better.
All trained policies for the same task are evaluated on the same set of randomly-initialized environments.
For constrained MO-MPO policies, we sample a new preference at the beginning of each episode, rather than at every timestep.
Baselines.
As a baseline we consider Lagrangian relaxation; because of its simplicity and effectiveness it is a common choice (Ray et al., 2019; Stooke et al., 2020).
To ensure a fair comparison with our approach, we use MPO to train policies to optimize the Lagrangian dual objective (2).
We obtain state-of-the-art results on Safety Gym tasks with this MPO-Lagrangian baseline (Appendix C.2).
5. Q UALITY OF SOLUTIONS
We evaluated our approach and the baseline for constraint thresholds linearly spaced in the range [ − 4 , − 1] for humanoid run , [ − 2 , − 0 .
5] for humanoid walk , and [ − 15 , − 2] for point tasks.
For constrained MO-MPO, the preference policy π ( (cid:15) ) is initialized to uniform and is a discrete distribution over 100 linearly-spaced values starting at 10 − 5 and up 0 .
15 for humanoid run , 0 .
3 for humanoid walk , and 0 .
2 for the point tasks.
Results are shown in Fig.
We observe the biggest difference in performance for humanoid walk : the Lagrangian baseline only ﬁnds policies at the extremes, whereas our approach ﬁnds constraint-satisfying solutions that achieve non-zero task reward.
This may be because the ground-truth Pareto front for this task is concave (Sec.
On humanoid run , point goal , and point button , the baseline and our approach ﬁnd similar-quality solutions for easier constraint thresholds, but as the constraint threshold becomes more difﬁcult to satisfy (i.e., smaller in magnitude), our approach dominates (Fig. 1, top row; Fig. 2a).
The only exception is point push , for which both approaches ﬁnd similar-quality solutions for all constraint thresholds.
We hypothesize that this is because in this particular task, the cost does not conﬂict much with the task objective, so it is easier to ﬁnd high-task-reward solutions that incur small cost.
Our approach always ﬁnds policies that satisfy the constraints for humanoid, whereas the Lagrangian baseline occasionally violates the constraint.
6 For Safety Gym tasks, the policies found by both our approach and the baseline slightly violate the constraints (Fig. 1, bottom row); this is due to Q-function underestimates for the cost, rather than a limitation of the policy optimization (Appendix C.4)
[Figure]
Our approach is also signiﬁcantly more sample-efﬁcient than the baseline.
For instance on humanoid run , constrained MO-MPO learns constraint-satisfying policies that reach reasonably high task reward mid-way through training, whereas the majority of MPO-Lagrangian policies obtain zero task reward at the same point (Fig. 3a, 3b).
This could be because with two conﬂicting objectives,
MPO-Lagrangian cannot optimize for both objectives at once, whereas our approach can (Fig. 2b).
Multiple random seeds.
We additionally ran experiments with ﬁve random policy initializations per constraint threshold, for a subset of four thresholds.
The results support the conclusions drawn above.
In particular, our approach signiﬁcantly outperforms the baseline for the harder-to-satisfy constraint thresholds, in four out of the ﬁve tasks (see Appendix C.3 for details).
Scaling to multiple constraints.
We also investigated how well constrained MO-MPO scales to multiple constraints, for a more difﬁcult version of point goal (see Appendix B.2 for details).
There are two constraints: one on the cost of crossing over hazards, and one on bumping into vases.
For each type of cost, we ﬁx its threshold at − 2 and vary the other between − 10 and 0 .
The baseline fails to obtain task reward when the constraint for hazards is ﬁxed at the challenging threshold of − 2 , whereas constrained MO-MPO ﬁnds a range of solutions (Fig.
3c, right).
5. F LEXIBILITY
Since our approach decouples Pareto front optimization from Pareto exploration, by choosing the appropriate ﬁtness function, one can in theory recover any portion of the Pareto front.
This is useful because in practice, an RL practitioner may not know the true constraint, and will be better able
to
[Figure] [Figure]
choose which policy to deploy after observing a couple of options along the Pareto front.
Typical approaches to constrained RL ﬁnd a single policy that meets the constraint exactly.
On humanoid run , we evaluate constrained MO-MPO using (7) as the ﬁtness function for several inequality constraint thresholds.
We ﬁnd that our approach indeed learns a portion of the Pareto front that satisﬁes the constraint (Fig. 4a).
These plots are obtained by taking the preferences at regular percentiles between 5 th and 95 th percentiles of the learned preference distribution, and evaluating the policy conditioned on each of these preferences, on the same set of randomly-initialized environments.
5. S TATE - DEPENDENT PREFERENCES
By using a state-dependent preference policy π ( (cid:15) | s ) , we can extend our approach to satisfy constraints per-initial-state , rather than in expectation.
8 To do this, we train a neural network that maps the state to a discrete distribution over preference settings.
We implement this for the MPO-Lagrangian baseline by making the Lagrange multiplier state-dependent as well, as introduced by Bohez et al.
(2019); this trains policies that satisfy the constraint per-state —there is no equivalent for the Lagrangian approach that satisﬁes constraints per-initial-state.
We use both approaches to train policies for humanoid run , for the same settings of constraint thresholds as in Sec. 5.1.
Our approach ﬁnds constraint-satisfying policies that achieve higher task reward than the baseline (Fig. 4b).
The constraint violation is computed by summing the per-episode
[Figure]
constraint violation, obtained by comparing the average per-timestep action norm cost for that episode against the constraint threshold.
C ONCLUSION AND FUTURE WORK
In this paper, we introduced a novel framework for constrained RL, by leveraging the ability of multiobjective RL algorithms to ﬁnd Pareto-optimal solutions.
This framework can be combined with any algorithm that ﬁnds a Pareto front of solutions.
Our empirical results show that an instantiation of this framework, constrained MO-MPO, outperforms the commonly-used Lagrangian relaxation approach in terms of solution quality, stability, and sample-efﬁciency.
In particular, the Lagrangian approach struggles to train policies that obtain task reward when the constraints are difﬁcult to meet, or when the ground-truth Pareto front is concave.
Our framework also enables ﬂexibility via the choice of ﬁtness function and can be extended to meet constraints per-initial-state, rather than in expectation.
One limitation of this work is that the support of the preference policy’s discrete distribution must include preference setting(s) that satisfy the constraint threshold.
In practice, this is straightforward to do for MO-MPO, because the (cid:15) k encode preferences in a way that is independent of the objectives’ reward scales.
In future work, it is worth exploring other distributions for the preference policy that would overcome this limitation, for instance a mixture of Gaussians.
Another limitation is this work cannot be applied directly to safe RL.
We plan to extend our approach to reduce the cost incurred during training, perhaps by initializing with a conservative distribution over preferences, that prioritizes minimizing cost.
Finally, we also plan to exploit the ﬂexibility of our framework to solve constrained RL problems with more than one unconstrained objective
A I MPLEMENTATION D ETAILS
We use an asynchronous off-policy actor-learner setup.
In this setup, actors fetch policy parameters from the learner and act in the environment, writing transitions to the replay buffer.
The learner uses the transitions in the replay buffer to update the policies and Q-functions (and optionally the temperature network and Lagrange multiplier network, depending on the approach).
We implement this setup in Acme (Hoffman et al., 2020), a framework for distributed RL.
To make learning more stable, we maintain a target network for each trained network; we use these target networks for computing gradients, and update them after every 200 gradient steps.
We use 32 actors for training.
We use Adam (Kingma & Ba, 2015) for optimization, with a learning rate of 10 − 4 unless otherwise speciﬁed.
All policies and Q-functions are implemented as feed-forward neural networks.
Our action policies π θ output a Gaussian distribution with a diagonal covariance matrix, and our preference policies π ψ output a categorical distribution.
We found that adding layer normalization followed by a hyperbolic tangent ( tanh ) to the ﬁrst layer of the policy and Q-function networks improves stability of learning.
For the Safety Gym point mass tasks, we train a separate Q-function per objective, with a shared ﬁrst hidden layer.
For humanoid tasks, the Q-values for action norm cost are instead computed exactly from the actions; thus these Q-values have a discount factor of zero.
The default hyperparameters we used for our experiments are reported in Table 1, and task-speciﬁc hyperparameters are reported in Table 2.
In our experiments, we trained policies with constrained
MO-MPO and the MPO Lagrangian baseline for a range of different constraint thresholds, these thresholds are reported in Table 3.
We consider the “harder” constraint thresholds (as plotted in Fig.
1, top row) to be the 15 lowest-magnitude thresholds for the humanoid tasks, and the 7 lowest-magnitude thresholds for the point mass tasks.
Gathering data.
We gather data for the replay buffer via the actors.
When the action policy π θ is preference-conditioned, at the start of each episode, each actor ﬁrst samples a preference (cid:15) (cid:48) from the preference policy π ψ , and then acts according to π θ ( a | s, (cid:15) (cid:48) ) until the end of the episode.
For the next episode, the actor repeats this, sampling a new preference.
Evaluation.
For humanoid tasks, we evaluate each trained policy on the same 100 randomlyinitialized environments.
For Safety Gym tasks, we do the same on the same 500 randomly-initialized environments.
Table 4 reports how many actor steps policies are trained for, before evaluation.
We evaluate with a deterministic action policy π θ by using the mean of the Gaussian distribution over actions.
When the action policy is conditioned on preferences, at the beginning of each episode during evaluation, we ﬁrst sample a preference (cid:15) from the preference policy, and then execute the mean of π θ ( a | s, (cid:15) ) , sticking with the same sampled preference for the entire episode.
However, when the preference policy is state-dependent, we instead condition the action policy on the median preference from π ψ ( (cid:15) | s ) , because we ﬁnd that this leads to better satisfaction of per-episode constraints.
B E XPERIMENTAL DOMAINS B. H UMANOID
We use the humanoid run and walk tasks from DeepMind Control Suite (Tassa et al., 2018).
The observation space is 67 -dimensional and the action space is 21 -dimensional.
Observations consist of joint angles, joint velocities, center-of-mass velocity, head height, torso orientation, and hand and feet positions.
Actions correspond to joint accelerations; the minimum and maximum action limits are − 1 and 1 , respectively.
Each episode is 1000 timesteps.
The unconstrained task reward is given by the environment: there is a shaped reward for maintaining a horizontal speed (in any direction) of 10 m/s for humanoid run and 1 m/s for humanoid walk .
The shaped reward is equal to min( h/h ∗ , 1) , where h is the speed of the agent and h ∗ is the target speed, both in m/s.
The constrained reward is the negative l 2 -norm of the action vector: r cost ( s, a ) = −(cid:107) a
[Equation]
This can be thought of as limiting the energy usage of the agent.
[Table] [Table]
[Table] [Table]
yes
[Table]
ELU
Lagrange
[Table]
or g (
[Table]
| s
[Table]
| s
[Table] [Table]
[Table] [Table]
yes
[Table] [Table]
[Table]
MPO / MO
[Table]
π
[Table] [Table]
(cid:15) k 0 .
KL
[Table]
on
[Table]
mean, β µ
[Table]
KL
[Table]
on
[Table]
covariance, β Σ 10 −
[Table]
temperature
[Table] [Table]
rate (for
[Table]
variables)
[Table]
MPO
[Table]
MO
[Table]
MPO
[Table]
π ψ
[Table]
per state default (cid:15) k 0 .
KL
[Table]
on policy, β 10 −
[Table]
η
[Table]
rate
[Table]
variables)
[Table]
trainingbatch
[Table] [Table]
[Figure]
[Table]
[Figure]
B. S AFETY G YM
We use the level-two point mass tasks in Safety Gym: goal , button , and push (Ray et al., 2019).
The action space is 2-dimensional, with minimum and maximum action limits of − 1 and 1 , respectively.
The observation space is 60-dimensional for point goal , and 76-dimensional for point button and push .
There are 1000 timesteps per episode.
At the start of each episode, the entities (i.e., goal, agent, obstacles) are randomly initialized.
[Table]
MPO
[Table]
KL
[Table] [Table]
We use the default task conﬁgurations as used in Ray et al.
(2019), for instance how many obstacles are spawned, the regions in which entities can be spawned, the size of entities
[Table]
and the type of observations.
The only difference is that we use sparse task reward instead of shaped task reward, since the former is more realistic
[Table]
A sparse task reward of 1 is given in goal when the agent enters the goal area, in button when the agent touches the target button, and in push when the agent pushes a box into the goal area.
The constrained reward is the negative cumulative cost.
In goal the agent incurs cost by being in hazardous regions or bumping into vases; in button the agent incurs cost by being in hazardous regions, bumping into gremlins, or touching non-target buttons; in push the agent incurs cost being in hazardous regions or bumping into pillars.
All obstacles are static, except for gremlins, which move in a ﬁxed circular pattern; the agent moves vases when it bumps into them.
In our experiments with two constraints (Sec. 5.1), we use a more difﬁcult variant of the point goal task, with two changes.
First, the length of the square area in which entities are spawned is three-quarters of the original length, which makes the density of obstacles higher and thus more difﬁcult for the agent to navigate around.
In addition, the radius of the goal region is two-thirds the original radius, which allows for it to be placed in more difﬁcult-to-reach locations: because the random initialization of the scene enforces that there are no collisions between the entities, the smaller the goal is, the more potential open areas it could be placed in.
C A DDITIONAL EXPERIMENTS AND ANALYSIS C. C OMPARISON TO MULTI - OBJECTIVE RL ALGORITHMS
We ﬁrst evaluate whether constrained MO-MPO is able to ﬁnd solutions that lie on the Pareto front.
Since the ground-truth Pareto front is not available in general, we approximate this for humanoid run
[Figure] [Figure]
by training policies with MO-MPO for a range of preference
We trained preference-conditioned policies with ﬁve random initializations, for the same (cid:15) task and (cid:15) norm sampled uniformly from the same range; we call this approach controllable MO-MPO.
This produces a Pareto front that is comparable to MO-MPO’s, both in terms of coverage and solution quality (Fig. 5, middle right).
It is also possible to train preference-conditioned policies where the preferences correspond to weights for linear scalarization, rather than KL-divergence bounds as in MO-MPO.
So as a baseline, we train weight-conditioned policies with MPO (Abdolmaleki et al., 2018); we call this baseline controllable
MPO.
These policies were conditioned on a w task of 1 . 0 and w norm sampled uniformly between 0 and 0 .
Compared to controllable MO-MPO, we had to train controllable MPO for longer (900M actor steps instead of 500M) in order to reach reasonable task performance, and controllable MPO was still unable to ﬁnd solutions with low action norm (Fig. 5, middle left).
This is as expected: Abdolmaleki et al.
(2020) show that MO-MPO outperforms MPO with linear scalarization on humanoid run , so one would expect controllable MO-MPO to also outperform controllable MPO.
We tried running controllable MPO with higher weights, but these did not show any learning progress within 500M actor steps.
Policies trained with constrained MO-MPO for equality constraint thresholds from − 4 to − 1 also lie on the MO-MPO Pareto front (Fig. 5, right).
We train a separate preference-conditioned action policy and preference policy for each unique constraint threshold.
This supports that learning the preference distribution simultaneously while training the policy and Q-functions does not negatively impact the ﬁnal solutions that are found.
C. MPO L AGRANGIAN BASELINE
The baseline we use in our empirical evaluation is MPO with Lagrangian relaxation.
We ran this baseline on all Safety Gym point mass tasks, using the default settings (including dense reward) so that
[Figure]
[Figure]
0 . 8 ), point goal ( − 4 and − 2 ), and point button ( − 4 and − 2 ). For the other constraint thresholds, both approaches achieve similar task reward. In terms of satisfying constraint thresholds, our approach performs on-par with or better than the baseline.
Each bar shows either the average task reward (solid bars) or cost (clear bars) per episode, averaged over ﬁve seeds. For both, higher is better. The error bars refer to standard deviation. The red lines denote the constraint thresholds: clear bars above the constraint threshold indicate that the constraint is met.
we can directly compare against the constraint-satisfying policies obtained in Ray et al.
Our
MPO-Lagrangian baseline compares favorably with those in Ray et al.
(2019), that instead combine
Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) with Lagrangian relaxation (Fig. 6).
This is with the caveat that policies were trained with TRPO-Lagrangian and
PPO-Lagrangian for 10 million actor steps, whereas our training curves for MPO-Lagrangian go up to 100 million actor steps.
C. M ULTIPLE RANDOM SEEDS
To verify that the empirical results we observed in Sec. 5.1 are signiﬁcant, we used our approach and the baseline to train policies for the same constraint threshold, starting from ﬁve random initializations.
We evaluated this for four constraint thresholds per task, selected from the ranges in Table 3.
Results are shown in Fig.
Constrained MO-MPO achieves signiﬁcantly higher task reward than the baseline for the harder-tosatisfy constraint thresholds in four out of the ﬁve tasks.
For the other constraint thresholds, both
[Figure]
[Figure]
approaches achieve similar task reward.
terms of
As mentioned, the slight violation of constraint thresholds for the Safety Gym point tasks is due to underestimation of Q-values, rather than a limitation of the policy improvement method; this is described in the following subsection.
C. Q- FUNCTION ESTIMATES
We noticed that for the Safety Gym point mass tasks, both our approach and the Lagrangian baseline train policies that slightly exceed the constraints (Fig. 1, bottom row).
After digging deeper, we realized that this results from suboptimal Q-function learning, rather than suboptimal policy optimization.
The learned Q-values for the cost consistently underestimate the actual cost incurred (Fig. 8a).
The policy optimization ﬁnds a policy that meets the constraints, assuming that the learned
Q-values for the cost are accurate (Fig. 8b).
Thus, better constraint satisfaction can be obtained by improving policy evaluation, so that estimated
Q-values are more accurate—this is orthogonal to our proposed approach, which is focused on policy improvement.
D A LGORITHMIC DETAILS
In this section, we give details on how to learn the preference policy and provide an algorithm box for the policy improvement step.
We also describe the overall procedure we use to collect data and learn from the data.
D. L EARNING PREFERENCE - CONDITIONED POLICIES
As discussed in the main paper, we also learn a new preference policy π ψ ( (cid:15) ) given the current preference policy π old ( (cid:15) ) and ﬁtness functions f k ( (cid:15) )
(that evaluate (cid:15) in terms of constraint satisfaction).
One could add the ﬁtness functions together to obtain (cid:80)
k f k ( (cid:15) ) , and use any off-the-shelf RL algorithm to optimize for a new preference policy.
On the other hand, this problem can be been as a multi-objective problem where the ﬁtness for each constraint k is an objective.
To this end, we use
MO-MPO for learning π ψ ( (cid:15) ) .
Note that in this paper we have at most two constraints.
In the case of a problem with one constraint (i.e., one “objective”), MO-MPO (Abdolmaleki et al., 2020) reduces to
MPO (Abdolmaleki et al., 2018).
More formally, following the MO-MPO algorithm, we optimize the following constrained optimization problem, that can be solved via Lagrangian relaxation (Abdolmaleki et al., 2020):
[Equation]
where δ deﬁnes a trust region for more stable learning and p k ( (cid:15) ) is a non-parametric improved policy for each objective, i.e,
[Equation]
ϕ k is a temperature variable that we maintain for each objective (or ﬁtness function) and is obtained by optimizing the convex dual function
[Equation]
where α k for each objective k deﬁnes a desired KL-divergence bound between the new improved policy p k ( (cid:15) ) and current policy π old ( ψ ) .
We use the same value of α k = 0 .
1 for all ﬁtness functions
k throughout the paper.
For more details on MO-MPO, please refer to (Abdolmaleki et al., 2020).
Note that this section assumes that the preference policy is state-independent, i.e., π ( (cid:15) ) .
The optimization is analogous for a state-dependent preference policy π ( (cid:15) | s ) , which we consider in Sec.
D. G ENERAL A LGORITHM
We maintain one online network and one target network for each Q-function, action policy and preference policy.
We also maintain one online network for the temperature function.
Target networks are updated every ﬁxed number of steps by copying parameters from the online network. Online networks are updated using gradient descent in each learning iteration. We refer to the target networks by using the subscript/superscript “old” throughout the paper. We use an asynchronous actor-learner setup.
In this setup actors fetch policy parameters from the learner and act in the environment and write transitions to the replay buffer.
Note that at the beginning of each episode, we ﬁrst sample one preference parameter (cid:15) from the preference policy, which remains ﬁxed until the end of the episode. The learner uses the transitions in the replay buffer to update the Q-functions, policies and temperature functions. Algorithm 1 describes one step of policy improvement for constrained MO-MPO. [Table]
Li
[Table]
k
[Table]
[Table]
, where [k] is the index of the vector
[Table]
, using optimizer O 14: for k = 0, ..., K do 15: q ijk ∝ exp( Q ijk η ω ( (cid:15) i )
[ k ] ) 16: end for 17:18: // Update action policy 19: δ θ ← −∇ θ (cid:80) Li (cid:80) Mj (cid:80)
Kk =
q ijk log π θ ( a ij | s i , (cid:15) i ) 20: (subject to additional KL regularization) 21:
Update π θ based on δ π , using optimizer O 22: 23:24: // Collect dataset { (cid:15) i , f ik } M,Ki,k , where 25: // M preferences (cid:15) i ∼ π old ( (cid:15) ) and f ik = f k
( (cid:15) i ) 26:27: //
Compute epsilon distribution for each constrained objective 28: for k = 1, ..., K do 29: δ ϕ k ← ∇ ϕ k ϕ k
α k + ϕ k log
(cid:16)(cid:80)
i 1 M exp (cid:16) f
ik ϕ k (cid:17)(cid:17) 30:
Update ϕ k based on δ ϕ k , using optimizer O 31: p ik ∝ exp( f ik ϕ k ) 32: end for 33:34: // Update preference policy 35: δ ψ ← −∇ ψ (cid:80) Mi (cid:80) Kk =1 p
ik log π ψ ( (cid:15) i ) 36: (subject to additional KL regularization) 37:
Update π ψ based on δ ψ , using optimizer O 38: 39: until ﬁxed number of steps 40: return π old ( a | s, (cid:15) )
=
π θ ( a | s, (cid:15) ) and π old ( (cid:15) ) =
π ψ ( (cid:15) )