Original: A Constrained Multi-Objective Reinforcement Learning Framework
Revised: A Constrained Multi-Objective Reinforcement Learning Framework

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract: Many real-world problems, especially in robotics, require that reinforcement learning (RL) agents learn policies that not only maximize an environment reward, but also satisfy constraints.
Revised: Abstract: Many real-world problems, especially in robotics, require that reinforcement learning (RL) agents learn policies that not only maximize an environment reward, but also satisfy constraints.

Original: We propose a high-level framework forsolving such problems, that treats the environment reward and costs as separateobjectives, and learns what preference over objectives the policy should optimizefor in order to meet the constraints.
Revised: We propose a high-level framework forsolving such problems, that treats the environment reward and costs as separateobjectives, and learns what preference over objectives the policy should optimizefor in order to meet the constraints.

Original: We call this Learning Preferences and Policies in Parallel (LP3).
Revised: We call this Learning Preferences and Policies in Parallel (LP3).

Original: By making different choices for how to learn the preferenceand how to optimize for the policy given the preference, we can obtain existingapproaches (e.g., Lagrangian relaxation) and derive novel approaches that lead tobetter performance.
Revised: By making different choices for how to learn the preferenceand how to optimize for the policy given the preference, we can obtain existingapproaches (e.g., Lagrangian relaxation) and derive novel approaches that lead tobetter performance.

Original: One of these is an algorithm that learns a set of constraintsatisfying policies, useful for when we do not know the exact constraint a priori.
Revised: One of these is an algorithm that learns a set of constraintsatisfying policies, useful for when we do not know the exact constraint a priori.

Original:  Introduction
Revised:  Introduction

Original: Many problems, especially in the real world and in robot applications, require that policies meetconstraints [1].
Revised: Many problems, especially in the real world and in robot applications, require that policies meetconstraints [1].

Original: For instance, we might want a factory robot to optimize task throughput whilekeeping actuator forces below a threshold, to limit wear-and-tear.
Revised: For instance, we might want a factory robot to optimize task throughput whilekeeping actuator forces below a threshold, to limit wear-and-tear.

Original: Or, we might want to minimizeenergy usage for cooling a data center while ensuring temperatures remain below some level [2].
Revised: Or, we might want to minimizeenergy usage for cooling a data center while ensuring temperatures remain below some level [2].

Original: Existing approaches for constrained RL typically formulate the problem as a constrained Markov
Revised: Existing approaches for constrained RL typically formulate the problem as a constrained Markov

Original: Decision Process (CMDP) [3], and many rely on Lagrangian relaxation [4] to solve it.
Revised: Decision Process (CMDP) [3], and many rely on Lagrangian relaxation [4] to solve it.

Original: Lagrangianrelaxation transforms the constrained optimization into an unconstrained one, where the objectiveis a sum of the expected task return and weighted constraint functions.
Revised: Lagrangianrelaxation transforms the constrained optimization into an unconstrained one, where the objectiveis a sum of the expected task return and weighted constraint functions.

Original: The weights (i.e., Lagrangemultipliers) are updated to ensure constraint satisfaction.
Revised: The weights (i.e., Lagrangemultipliers) are updated to ensure constraint satisfaction.

Original: However, some constrained RL problems do not ﬁt the CMDP formulation.
Revised: However, some constrained RL problems do not ﬁt the CMDP formulation.

Original: For instance, when wedo not have an exact desired constraint threshold in mind, we may want to choose from a set ofpolicies that have costs within a viable range.
Revised: For instance, when wedo not have an exact desired constraint threshold in mind, we may want to choose from a set ofpolicies that have costs within a viable range.

Original: Or, in some settings we have soft constraints [5],where violating the constraint is undesirable but not catastrophic.
Revised: Or, in some settings we have soft constraints [5],where violating the constraint is undesirable but not catastrophic.

Original: These situations arise because ofthe inherent trade-off between maximizing task reward and satisfying constraints.
Revised: These situations arise because ofthe inherent trade-off between maximizing task reward and satisfying constraints.

Original: Our key insight is to view constrained RL from a multi-objective perspective, where the task rewardand each constraint are separate objectives.
Revised: Our key insight is to view constrained RL from a multi-objective perspective, where the task rewardand each constraint are separate objectives.

Original: To train a policy, we must tell it what preference overobjectives to optimize for—i.e., how much the policy should prioritize one objective over another.
Revised: To train a policy, we must tell it what preference overobjectives to optimize for—i.e., how much the policy should prioritize one objective over another.

Original: At the extremes, we could tell the policy to either focus entirely on maximizing task reward whiledisregarding constraint satisfaction, or focus entirely on satisfying the constraints without caringabout the task.
Revised: At the extremes, we could tell the policy to either focus entirely on maximizing task reward whiledisregarding constraint satisfaction, or focus entirely on satisfying the constraints without caringabout the task.

Original: Some preference settings will lead to policies that satisfy the constraints, but we donot know in advance which ones these are.
Revised: Some preference settings will lead to policies that satisfy the constraints, but we donot know in advance which ones these are.

Original: Thus we propose to learn which preferences lead to constraint-satisfying policies, while simultaneously training policies for those preferences.
Revised: Thus we propose to learn which preferences lead to constraint-satisfying policies, while simultaneously training policies for those preferences.

Original: We call this framework Learning Preferences and
Revised: We call this framework Learning Preferences and

Original: Policies in Parallel (LP3) , shown in Fig.
Revised: Policies in Parallel (LP3) , shown in Fig.

Original: In LP3 there are two main design choices: 1) whichmulti-objective RL algorithm is used to train the policy, and 2) whether to learn a single preferenceor a distribution over preferences.
Revised: In LP3 there are two main design choices: 1) whichmulti-objective RL algorithm is used to train the policy, and 2) whether to learn a single preferenceor a distribution over preferences.

Original: Lagrangian relaxation is one possible instantiation—it uses linearscalarization as the multi-objective RL algorithm and learns a single preference (Sec. 4.2).
Revised: Lagrangian relaxation is one possible instantiation—it uses linearscalarization as the multi-objective RL algorithm and learns a single preference (Sec. 4.2).

Original: learn policy, for the current preferences learn preferences to satisfy constraints e.g. with linear scalarization or distributional single vs. distribution
Revised: learn policy, for the current preferences learn preferences to satisfy constraints e.g. with linear scalarization or distributional single vs. distribution

Original: LP3 Framework
Revised: LP3 Framework

Original: Module
Revised: Module

Original: [Figure]
Revised: [Figure]

Original: Module
Revised: Module

Original: [Figure]
Revised: [Figure]

Original: We explore two other instantiations of our framework.
Revised: We explore two other instantiations of our framework.

Original: In the ﬁrst, to train the policy we use a stateof-the-art multi-objective RL algorithm, Multi-Objective Maximum a Posteriori Optimization (MOMPO) [6].
Revised: In the ﬁrst, to train the policy we use a stateof-the-art multi-objective RL algorithm, Multi-Objective Maximum a Posteriori Optimization (MOMPO) [6].

Original: In the second, we learn a probability distribution over preferences, based on how likelythey are to lead to constraint-satisfying policies.
Revised: In the second, we learn a probability distribution over preferences, based on how likelythey are to lead to constraint-satisfying policies.

Original: This enables ﬁnding a set of constraint-satisfyingpolicies in a single training run, that differ in terms of task return and constraint performance.
Revised: This enables ﬁnding a set of constraint-satisfyingpolicies in a single training run, that differ in terms of task return and constraint performance.

Original: Our main contributions, among others, are:
Revised: Our main contributions, among others, are:

Original: • We propose a multi-objective framework for constrained RL, called LP3 (Sec. 4).
Revised: • We propose a multi-objective framework for constrained RL, called LP3 (Sec. 4).

Original: Thisis a general framework, that includes Lagrangian relaxation as one possible instantiation.
Revised: Thisis a general framework, that includes Lagrangian relaxation as one possible instantiation.

Original: Although the connection between constrained and multi-objective problems is intuitive
Revised: Although the connection between constrained and multi-objective problems is intuitive

Original: our work is the ﬁrst to leverage this to create novel algorithms for constrained RL.
Revised: our work is the ﬁrst to leverage this to create novel algorithms for constrained RL.

Original: • Based on this framework, we derive two novel algorithms for constrained RL.
Revised: • Based on this framework, we derive two novel algorithms for constrained RL.

Original: One of theseis the ﬁrst algorithm, to the best of our knowledge, for ﬁnding a set of constraint-satisfying
Revised: One of theseis the ﬁrst algorithm, to the best of our knowledge, for ﬁnding a set of constraint-satisfying

Original: policies—this is important when the constraint is soft or not exactly known a priori.
Revised: policies—this is important when the constraint is soft or not exactly known a priori.

Original: • We run extensive experiments that compare these two new approaches with Lagrangian-
Revised: • We run extensive experiments that compare these two new approaches with Lagrangian-

Original: based approaches on continuous control tasks across three task suites, including high-
Revised: based approaches on continuous control tasks across three task suites, including high-

Original: dimensional locomotion tasks.
Revised: dimensional locomotion tasks.

Original: Videos are at http://sites.google.com/view/cmorl .
Revised: Videos are at http://sites.google.com/view/cmorl .

Original:  Related Work
Revised:  Related Work

Original: Constrained reinforcement learning.
Revised: Constrained reinforcement learning.

Original: Constrained RL algorithms learn policies that satisfy constraints while maximizing task return.
Revised: Constrained RL algorithms learn policies that satisfy constraints while maximizing task return.

Original: Many approaches formulate this problem as a CMDP anduse Lagrangian relaxation [4].
Revised: Many approaches formulate this problem as a CMDP anduse Lagrangian relaxation [4].

Original: Recent works claim this converges to constraint-satisfying [9] or optimal [10] solutions.
Revised: Recent works claim this converges to constraint-satisfying [9] or optimal [10] solutions.

Original: Other works seek to stabilize this optimization by incorporating convex relaxations [11] or derivatives of the constraint function [12].
Revised: Other works seek to stabilize this optimization by incorporating convex relaxations [11] or derivatives of the constraint function [12].

Original: Others have applied Lagrangian relaxationto mean-value constraints [9], convex set constraints [13], and local constraint satisfaction [14].
Revised: Others have applied Lagrangian relaxationto mean-value constraints [9], convex set constraints [13], and local constraint satisfaction [14].

Original: For soft constraints, MetaL [5] uses meta-gradients to adjust the Lagrangian learning rate, to ﬁnda single policy that achieves high task reward without overly violating the constraints.
Revised: For soft constraints, MetaL [5] uses meta-gradients to adjust the Lagrangian learning rate, to ﬁnda single policy that achieves high task reward without overly violating the constraints.

Original: However,this particular trade-off between the reward and cost may not be desirable.
Revised: However,this particular trade-off between the reward and cost may not be desirable.

Original: Instead, we propose anapproach that returns a set of policies to choose from, that make different trade-offs.
Revised: Instead, we propose anapproach that returns a set of policies to choose from, that make different trade-offs.

Original: Safe reinforcement learning.
Revised: Safe reinforcement learning.

Original: In safe RL, the aim is to achieve constraint satisfaction not only at deployment, but also during learning [15].
Revised: In safe RL, the aim is to achieve constraint satisfaction not only at deployment, but also during learning [15].

Original: Recent works for deep RL modify the policy improvementstep to guarantee that the policy will never violate constraints during training [16, 17, 18, 19, 20, 21].
Revised: Recent works for deep RL modify the policy improvementstep to guarantee that the policy will never violate constraints during training [16, 17, 18, 19, 20, 21].

Original: These approaches require, however, that the initial policy (nearly) satisﬁes the constraints; otherwise,performance may decrease signiﬁcantly [22].
Revised: These approaches require, however, that the initial policy (nearly) satisﬁes the constraints; otherwise,performance may decrease signiﬁcantly [22].

Original: We focus on constrained RL, rather than safe RL, butwe discuss how to extend our approach to reduce constraint violation during training (Sec. 6).
Revised: We focus on constrained RL, rather than safe RL, butwe discuss how to extend our approach to reduce constraint violation during training (Sec. 6).

Original: Multi-objective reinforcement learning.
Revised: Multi-objective reinforcement learning.

Original: In our framework, the policy is trained with multiobjective RL (MORL).
Revised: In our framework, the policy is trained with multiobjective RL (MORL).

Original: MORL consists of single-policy and multi-policy approaches.
Revised: MORL consists of single-policy and multi-policy approaches.

Original: Single policymethods learn a policy that is optimal for a given trade-off.
Revised: Single policymethods learn a policy that is optimal for a given trade-off.

Original: Most rely on linear scalarization [23],which restricts solutions to the convex portions of the Pareto front and can be sensitive to rewardscales.
Revised: Most rely on linear scalarization [23],which restricts solutions to the convex portions of the Pareto front and can be sensitive to rewardscales.

Original: Non-linear scalarizations exist [24, 25, 26], but these are harder to combine with value-based
Revised: Non-linear scalarizations exist [24, 25, 26], but these are harder to combine with value-based

Original: RL and have seen limited use in deep RL.
Revised: RL and have seen limited use in deep RL.

Original: Recently, Abdolmaleki et al.
Revised: Recently, Abdolmaleki et al.

Original: [6] introduced MO-MPO, asingle-policy approach that does not rely on scalarization and is invariant to reward scales.
Revised: [6] introduced MO-MPO, asingle-policy approach that does not rely on scalarization and is invariant to reward scales.

Original: Multi-policy MORL aims to ﬁnd a set of policies that covers the whole Pareto front.
Revised: Multi-policy MORL aims to ﬁnd a set of policies that covers the whole Pareto front.

Original: Recent workslearn a manifold in parameter space [27, 28] or combine single policy approaches with a generalobjective [29], to directly optimize the hypervolume of the Pareto front.
Revised: Recent workslearn a manifold in parameter space [27, 28] or combine single policy approaches with a generalobjective [29], to directly optimize the hypervolume of the Pareto front.

Original:  Background and Notation3.1 Markov Decision Processes
Revised:  Background and Notation3.1 Markov Decision Processes

Original: A Markov Decision Process (MDP) is deﬁned by {S , A , p, r 0 , γ } : the state space S , action space A , 86 transition function p ( s (cid:48) | s, a ) , bounded task reward function r 0 ( s, a ) , and discount factor γ ∈ [0 , 1] .
Revised: A Markov Decision Process (MDP) is deﬁned by {S , A , p, r 0 , γ } : the state space S , action space A , 86 transition function p ( s (cid:48) | s, a ) , bounded task reward function r 0 ( s, a ) , and discount factor γ ∈ [0 , 1] .

Original: Constrained MDP.
Revised: Constrained MDP.

Original: Constrained RL problems are typically formulated as constrained MDPs(CMDP).
Revised: Constrained RL problems are typically formulated as constrained MDPs(CMDP).

Original: A CMDP is deﬁned by {S , A , p, r 0: K , β 1: K , γ } .
Revised: A CMDP is deﬁned by {S , A , p, r 0: K , β 1: K , γ } .

Original: Compared to a regular MDP, a CMDP in- 89 cludes additional constraint-speciﬁc reward functions { r k ( s, a ) } Kk =1 with corresponding thresholds{ β k } Kk =1 .
Revised: Compared to a regular MDP, a CMDP in- 89 cludes additional constraint-speciﬁc reward functions { r k ( s, a ) } Kk =1 with corresponding thresholds{ β k } Kk =1 .

Original: All reward functions r k return scalar rewards; we will refer to these as separate objectives .
Revised: All reward functions r k return scalar rewards; we will refer to these as separate objectives .

Original: A policy π ( a | s ) maps from a state to a distribution over actions.
Revised: A policy π ( a | s ) maps from a state to a distribution over actions.

Original: Let J πk
Revised: Let J πk

Original: =
Revised: =

Original: E π
Revised: E π

Original: [ (cid:80) t γ t r k ( s t , a t )]denote the expected return for objective k when following policy π , given a ﬁxed initial state distribution.
Revised: [ (cid:80) t γ t r k ( s t , a t )]denote the expected return for objective k when following policy π , given a ﬁxed initial state distribution.

Original: The action-value function Q πk ( s, a ) is the expected return for objective k after taking actiona in state s , and then acting based on policy π .
Revised: The action-value function Q πk ( s, a ) is the expected return for objective k after taking actiona in state s , and then acting based on policy π .

Original: The optimal solution to a CMDP is a policy that maximizes expected task return, while ensuring allother expected returns satisfy their thresholds:
Revised: The optimal solution to a CMDP is a policy that maximizes expected task return, while ensuring allother expected returns satisfy their thresholds:

Original: [Equation]
Revised: [Equation]

Original: Many approaches for solving CMDPs are based on Lagrangian relaxation, which turns the constrained problem into the following unconstrained optimization problem:
Revised: Many approaches for solving CMDPs are based on Lagrangian relaxation, which turns the constrained problem into the following unconstrained optimization problem:

Original: [Equation]
Revised: [Equation]

Original: These approaches alternate between optimizing the Lagrange multipliers λ k and the policy π .
Revised: These approaches alternate between optimizing the Lagrange multipliers λ k and the policy π .

Original: Multi-Objective MDP.
Revised: Multi-Objective MDP.

Original: A multi-objective MDP (MO-MDP) is deﬁned by {S , A , p, r 0: K , γ } .
Revised: A multi-objective MDP (MO-MDP) is deﬁned by {S , A , p, r 0: K , γ } .

Original: In 101 contrast to an (C)MDP, there are multiple rewards to optimize for, and no constraints.
Revised: In 101 contrast to an (C)MDP, there are multiple rewards to optimize for, and no constraints.

Original: The optimalpolicy depends on the desired preference (cid:15) across objectives.
Revised: The optimalpolicy depends on the desired preference (cid:15) across objectives.

Original: In linear scalarization , the preference(cid:15) deﬁnes a weighting across objectives; the optimal policy is the one that maximizes (cid:80)
Revised: In linear scalarization , the preference(cid:15) deﬁnes a weighting across objectives; the optimal policy is the one that maximizes (cid:80)

Original: Kk =0 (cid:15)
Revised: Kk =0 (cid:15)

Original: k J πk .
Revised: k J πk .

Original: When no preferences are given, the solution to a MO-MDP is the set of all non-dominated policies,also known as the Pareto front .
Revised: When no preferences are given, the solution to a MO-MDP is the set of all non-dominated policies,also known as the Pareto front .

Original: A policy is non-dominated if there is no other policy that improvesits return for one objective without decreasing return for another.
Revised: A policy is non-dominated if there is no other policy that improvesits return for one objective without decreasing return for another.

Original: 3.2 Multi-Objective MPO (MO-MPO)
Revised: 3.2 Multi-Objective MPO (MO-MPO)

Original: Whereas most multi-objective RL algorithms use linear scalarization, MO-MPO [6] takes a distributional approach to combining objectives.
Revised: Whereas most multi-objective RL algorithms use linear scalarization, MO-MPO [6] takes a distributional approach to combining objectives.

Original: MO-MPO extends MPO [30] to the multi-objective setting.
Revised: MO-MPO extends MPO [30] to the multi-objective setting.

Original: MO-MPO is a policy iteration algorithm with two steps: policy evaluation and policy improvement.
Revised: MO-MPO is a policy iteration algorithm with two steps: policy evaluation and policy improvement.

Original: Policy evaluation.
Revised: Policy evaluation.

Original: In this step, MO-MPO evaluates the current policy π old ( a | s ) by training a
Revised: In this step, MO-MPO evaluates the current policy π old ( a | s ) by training a

Original: Q-function per objective, as according to Q-decomposition [31].
Revised: Q-function per objective, as according to Q-decomposition [31].

Original: Policy improvement.
Revised: Policy improvement.

Original: Given the previous policy π old ( a | s ) and Q-functions { Q old k ( s, a ) } Kk =0 , in 114 this step MO-MPO improves the policy for a given state visitation distribution µ ( s ) .
Revised: Given the previous policy π old ( a | s ) and Q-functions { Q old k ( s, a ) } Kk =0 , in 114 this step MO-MPO improves the policy for a given state visitation distribution µ ( s ) .

Original: MO-MPOdecomposes policy improvement into two steps, similar to the view proposed in Ghosh et al.
Revised: MO-MPOdecomposes policy improvement into two steps, similar to the view proposed in Ghosh et al.

Original: [32]:ﬁrst ﬁnd per-objective improved action distributions, and then distill these into a new policy.
Revised: [32]:ﬁrst ﬁnd per-objective improved action distributions, and then distill these into a new policy.

Original: Finding per-objective distributions: For each objective k , MO-MPO computes a nonparametricpolicy q k ( a | s ) that improves the old policy with respect to that objective, subject to a non-negativeconstraint (cid:15) k on the KL-divergence between the improved and old policies:
Revised: Finding per-objective distributions: For each objective k , MO-MPO computes a nonparametricpolicy q k ( a | s ) that improves the old policy with respect to that objective, subject to a non-negativeconstraint (cid:15) k on the KL-divergence between the improved and old policies:

Original: [Equation]
Revised: [Equation]

Original: Learning a new parameterized policy: MO-MPO then updates the policy via supervised learning onthese nonparametric policies, subject to a trust region with bound β > 0 for more stable learning:
Revised: Learning a new parameterized policy: MO-MPO then updates the policy via supervised learning onthese nonparametric policies, subject to a trust region with bound β > 0 for more stable learning:

Original: [Equation]
Revised: [Equation]

Original: The KL-constraints { (cid:15) k } Kk =0 encode preferences: the larger (cid:15) k , the more inﬂuence objective k has.
Revised: The KL-constraints { (cid:15) k } Kk =0 encode preferences: the larger (cid:15) k , the more inﬂuence objective k has.

Original:  Learning Preferences and Policies in Parallel (LP3)
Revised:  Learning Preferences and Policies in Parallel (LP3)

Original: In this section we introduce LP3, our constrained multi-objective RL framework.
Revised: In this section we introduce LP3, our constrained multi-objective RL framework.

Original: We ﬁrst formulatethe constrained RL problem as a Constrained Multi-Objective MDP (CMO-MDP).
Revised: We ﬁrst formulatethe constrained RL problem as a Constrained Multi-Objective MDP (CMO-MDP).

Original: Following this,we describe how LP3 solves this MDP, and detail how Lagrangian relaxation algorithms are aninstantiation of this framework.
Revised: Following this,we describe how LP3 solves this MDP, and detail how Lagrangian relaxation algorithms are aninstantiation of this framework.

Original: We then derive two novel constrained RL algorithms based on LP3.
Revised: We then derive two novel constrained RL algorithms based on LP3.

Original: 4.1 Constrained Multi-Objective MDP (CMO-MDP)
Revised: 4.1 Constrained Multi-Objective MDP (CMO-MDP)

Original: The CMDP formulation implies a single optimal solution.
Revised: The CMDP formulation implies a single optimal solution.

Original: But in real-world settings, one may beconsidering a range of viable constraint thresholds, which correspond to different trade-offs betweenthe task reward and constraints.
Revised: But in real-world settings, one may beconsidering a range of viable constraint thresholds, which correspond to different trade-offs betweenthe task reward and constraints.

Original: This connects to the multi-objective setting.
Revised: This connects to the multi-objective setting.

Original: We deﬁne a constrained multi-objective MDP (CMO-MDP) as {S , A , p, r 0: K , β N : K , γ } , where 133 N ≥ 1 .
Revised: We deﬁne a constrained multi-objective MDP (CMO-MDP) as {S , A , p, r 0: K , β N : K , γ } , where 133 N ≥ 1 .

Original: In a CMO-MDP, the optimization is over K reward functions, where N of these aretask rewards, and the remaining K − N rewards have constraints on them.
Revised: In a CMO-MDP, the optimization is over K reward functions, where N of these aretask rewards, and the remaining K − N rewards have constraints on them.

Original: Thus a CMO-MDP differs from a CMDP in two key ways: 1) the optimization objective is over all rewards, including theconstraint-speciﬁc ones; and 2) there may be multiple task rewards r 0: N .
Revised: Thus a CMO-MDP differs from a CMDP in two key ways: 1) the optimization objective is over all rewards, including theconstraint-speciﬁc ones; and 2) there may be multiple task rewards r 0: N .

Original: A CMO-MDP compares constraint-satisfying policies in terms of their returns for the task rewardand constraint functions—its solution is the set of all constraint-satisfying Pareto optimal policies.
Revised: A CMO-MDP compares constraint-satisfying policies in terms of their returns for the task rewardand constraint functions—its solution is the set of all constraint-satisfying Pareto optimal policies.

Original: Incontrast, a CMDP ranks policies according to only task return—its solution is the single constraintsatisfying policy with the highest task return.
Revised: Incontrast, a CMDP ranks policies according to only task return—its solution is the single constraintsatisfying policy with the highest task return.

Original: 4.2 The LP3 Framework for Solving CMO-MDPs
Revised: 4.2 The LP3 Framework for Solving CMO-MDPs

Original: We now describe LP3, a unifying framework (Fig. 1) for deriving algorithms that solve CMDPsand CMO-MDPs.
Revised: We now describe LP3, a unifying framework (Fig. 1) for deriving algorithms that solve CMDPsand CMO-MDPs.

Original: This framework has two modules: learning the policy, and learning which preference(s) (over the task and constraints) the policy should optimize for to meet the constraints.
Revised: This framework has two modules: learning the policy, and learning which preference(s) (over the task and constraints) the policy should optimize for to meet the constraints.

Original: For the ﬁrst module, the key choice is which multi-objective RL algorithm (such as linear scalarization or MO-MPO) is used to train the policy π θ to optimize for a given preference.
Revised: For the ﬁrst module, the key choice is which multi-objective RL algorithm (such as linear scalarization or MO-MPO) is used to train the policy π θ to optimize for a given preference.

Original: For the second module, the key choice is whether to learn a single preference (cid:15) or a preferencedistribution p ( (cid:15) ) .
Revised: For the second module, the key choice is whether to learn a single preference (cid:15) or a preferencedistribution p ( (cid:15) ) .

Original: The former corresponds to learning a policy π θ ( a | s ) .
Revised: The former corresponds to learning a policy π θ ( a | s ) .

Original: Learning a distribution overpreferences, however, requires training a preference-conditioned policy π θ ( a | s, (cid:15) ) , that learns to optimize for different preference settings.
Revised: Learning a distribution overpreferences, however, requires training a preference-conditioned policy π θ ( a | s, (cid:15) ) , that learns to optimize for different preference settings.

Original: This enables ﬁnding a set of constraint-satisfying policies,produced by sampling preferences from the ﬁnal p ( (cid:15) ) and rolling out the fully-trained π θ ( a | s, (cid:15) ) ,conditioned on that preference.
Revised: This enables ﬁnding a set of constraint-satisfying policies,produced by sampling preferences from the ﬁnal p ( (cid:15) ) and rolling out the fully-trained π θ ( a | s, (cid:15) ) ,conditioned on that preference.

Original: In practice, using a preference distribution can also improve learning, because training over multiple preferences enables more diverse data collection and exploration.
Revised: In practice, using a preference distribution can also improve learning, because training over multiple preferences enables more diverse data collection and exploration.

Original: Note that we can use different time scales for training the two modules in this framework.
Revised: Note that we can use different time scales for training the two modules in this framework.

Original: In ourexperiments, we alternate between taking one learning step on each.
Revised: In ourexperiments, we alternate between taking one learning step on each.

Original: Connection to Lagrangian Relaxation.
Revised: Connection to Lagrangian Relaxation.

Original: As mentioned, Lagrangian relaxation methods are equivalent to LP3 with linear scalarization and a single preference optimized using gradient descent.
Revised: As mentioned, Lagrangian relaxation methods are equivalent to LP3 with linear scalarization and a single preference optimized using gradient descent.

Original: Toshow this, deﬁne the preference (cid:15) =
Revised: Toshow this, deﬁne the preference (cid:15) =

Original: [1 , λ 1 , λ 2 , . . .
Revised: [1 , λ 1 , λ 2 , . . .

Original: , λ K ] (cid:62) , where λ 1: K are the Lagrange multipliers.
Revised: , λ K ] (cid:62) , where λ 1: K are the Lagrange multipliers.

Original: In the ﬁrst module, we train the policy to maximize a linear combination of Q-values, weighted bythe current preferences (cid:15) old : (cid:80) Kk =0 (cid:15) old k Q k ( a, s ) .
Revised: In the ﬁrst module, we train the policy to maximize a linear combination of Q-values, weighted bythe current preferences (cid:15) old : (cid:80) Kk =0 (cid:15) old k Q k ( a, s ) .

Original: We can use any standard RL algorithm for this.
Revised: We can use any standard RL algorithm for this.

Original: In the second module, given the current policy π old ( a | s ) , we use gradient descent to optimize for asingle preference setting (cid:15) with respect to the loss L ( (cid:15) > 0)
Revised: In the second module, given the current policy π old ( a | s ) , we use gradient descent to optimize for asingle preference setting (cid:15) with respect to the loss L ( (cid:15) > 0)

Original: = (cid:80)
Revised: = (cid:80)

Original: Kk =1 (cid:15) k f k .
Revised: Kk =1 (cid:15) k f k .

Original: Here, f k is the ﬁtnessscore corresponding to whether the policy respects threshold β k :
Revised: Here, f k is the ﬁtnessscore corresponding to whether the policy respects threshold β k :

Original: [Equation]
Revised: [Equation]

Original: 4.3 Novel Instantiations of LP 
Revised: 4.3 Novel Instantiations of LP

Original: Making different design choices in the LP3 framework leads to two novel algorithms for constrained
Revised: Making different design choices in the LP3 framework leads to two novel algorithms for constrained

Original: RL.
Revised: RL.

Original: The ﬁrst replaces linear scalarization with a state-of-the-art multi-objective RL algorithm, MOMPO.
Revised: The ﬁrst replaces linear scalarization with a state-of-the-art multi-objective RL algorithm, MOMPO.

Original: The motivation is that MO-MPO may be better at training policies for a given preference, since it has been shown to outperform linear scalarization on a variety of control problems [6].
Revised: The motivation is that MO-MPO may be better at training policies for a given preference, since it has been shown to outperform linear scalarization on a variety of control problems [6].

Original: We call this LP3 [MO-MPO] .
Revised: We call this LP3 [MO-MPO] .

Original: It uses the same second module (for learning the preference) as Lagrangian relaxation.
Revised: It uses the same second module (for learning the preference) as Lagrangian relaxation.

Original: The second algorithm, in addition to using MO-MPO, learns a distribution overpreferences.
Revised: The second algorithm, in addition to using MO-MPO, learns a distribution overpreferences.

Original: We call this LP3 [MO-MPO-D] and describe it below; full details are in Appendix E.
Revised: We call this LP3 [MO-MPO-D] and describe it below; full details are in Appendix E.

Original: 4.3.1 Module 1: Preference-Conditioned MO-MPO
Revised: 4.3.1 Module 1: Preference-Conditioned MO-MPO

Original: Learning a distribution over preferences requires training a preference-conditioned policy π θ ( a | s, (cid:15) ) ,but MO-MPO can only train a policy π θ ( a | s ) for a single preference (cid:15) .
Revised: Learning a distribution over preferences requires training a preference-conditioned policy π θ ( a | s, (cid:15) ) ,but MO-MPO can only train a policy π θ ( a | s ) for a single preference (cid:15) .

Original: We extend MO-MPO totrain a single policy π θ ( a | s, (cid:15) ) conditioned on preferences (cid:15) ∼ p old ( (cid:15) ) , by making both the policyevaluation and policy improvement steps preference-conditioned as follows.
Revised: We extend MO-MPO totrain a single policy π θ ( a | s, (cid:15) ) conditioned on preferences (cid:15) ∼ p old ( (cid:15) ) , by making both the policyevaluation and policy improvement steps preference-conditioned as follows.

Original: Preference-conditioned policy evaluation.
Revised: Preference-conditioned policy evaluation.

Original: We evaluate the current policy p old ( (cid:15) )
Revised: We evaluate the current policy p old ( (cid:15) )

Original: π old ( a | s, (cid:15) )via off-policy learning of per-objective Q-functions Q πk ( a, s, (cid:15) ) .
Revised: π old ( a | s, (cid:15) )via off-policy learning of per-objective Q-functions Q πk ( a, s, (cid:15) ) .

Original: Note that these Q-functions mustbe preference-conditioned because the policy is.
Revised: Note that these Q-functions mustbe preference-conditioned because the policy is.

Original: For each learning step we sample L transitionsfrom the replay buffer, { s i , a i , { r ki } Kk , s (cid:48) i } Li .
Revised: For each learning step we sample L transitionsfrom the replay buffer, { s i , a i , { r ki } Kk , s (cid:48) i } Li .

Original: Motivated by prior work on learning from off-policy 181 data [33], we use relabeling to augment the states with preferences sampled from the current preference distribution (cid:15) i ∼ p old ( (cid:15) ) , resulting in transitions { [ s i , (cid:15) i ] , a i , { r ki } Kk , [ s (cid:48) i , (cid:15) i ] } Li .
Revised: Motivated by prior work on learning from off-policy 181 data [33], we use relabeling to augment the states with preferences sampled from the current preference distribution (cid:15) i ∼ p old ( (cid:15) ) , resulting in transitions { [ s i , (cid:15) i ] , a i , { r ki } Kk , [ s (cid:48) i , (cid:15) i ] } Li .

Original: Any policy 183 evaluation algorithm can be used.
Revised: Any policy 183 evaluation algorithm can be used.

Original: We use distributional policy evaluation with 5-step return [34].
Revised: We use distributional policy evaluation with 5-step return [34].

Original: Preference-conditioned policy improvement.
Revised: Preference-conditioned policy improvement.

Original: This step improves the current policy π old ( a | s, (cid:15) ) .
Revised: This step improves the current policy π old ( a | s, (cid:15) ) .

Original: Finding per-objective distributions: To ﬁnd preference-conditioned per-objective improved actiondistributions q k ( a | s, (cid:15) ) , we optimize the following policy optimization problem for each objective:
Revised: Finding per-objective distributions: To ﬁnd preference-conditioned per-objective improved actiondistributions q k ( a | s, (cid:15) ) , we optimize the following policy optimization problem for each objective:

Original: [Equation]
Revised: [Equation]

Original: This generalizes (3) from MO-MPO, which assumes ﬁxed KL-constraints (cid:15) , to a distribution p old ( (cid:15) )over KL-constraints.
Revised: This generalizes (3) from MO-MPO, which assumes ﬁxed KL-constraints (cid:15) , to a distribution p old ( (cid:15) )over KL-constraints.

Original: We use a closed-form known nonparametric solution for q k [35, 30].
Revised: We use a closed-form known nonparametric solution for q k [35, 30].

Original: Learning a new parameterized policy: After obtaining per-objective improved policies, we use supervised learning to distill these into a new parameterized preference-conditioned policy:
Revised: Learning a new parameterized policy: After obtaining per-objective improved policies, we use supervised learning to distill these into a new parameterized preference-conditioned policy:

Original: [Equation]
Revised: [Equation]

Original: 4.3.2 Module 2: Learning Preferences
Revised: 4.3.2 Module 2: Learning Preferences

Original: The second module optimizes the preference (cid:15) or preference distribution p ( (cid:15) ) to lead to better constraint satisfaction.
Revised: The second module optimizes the preference (cid:15) or preference distribution p ( (cid:15) ) to lead to better constraint satisfaction.

Original: This assumes ﬁtness functions
Revised: This assumes ﬁtness functions

Original: f k that evaluate satisfaction of the constraintthreshold c k , given the current policy π old and Q-function Q old k . 

Revised: f k that evaluate satisfaction of the constraintthreshold c k , given the current policy π old and Q-function Q old k .

Original: We can use any gradient-basedalgorithm to optimize the preferences, initialized by (cid:15) old or p old ( (cid:15) ) , to maximize the ﬁtness.
Revised: We can use any gradient-basedalgorithm to optimize the preferences, initialized by (cid:15) old or p old ( (cid:15) ) , to maximize the ﬁtness.

Original: In our empirical evaluation, we use ﬁtness functions that give a score of zero to preferences that leadto constraint satisfaction, and negative scores to all others.
Revised: In our empirical evaluation, we use ﬁtness functions that give a score of zero to preferences that leadto constraint satisfaction, and negative scores to all others.

Original: For equality constraints we use a ﬁtnessfunction that penalizes the difference between the expected Q-values and constraint threshold, andfor inequality constraints it penalizes the amount the expected Q-values violate the threshold:
Revised: For equality constraints we use a ﬁtnessfunction that penalizes the difference between the expected Q-values and constraint threshold, andfor inequality constraints it penalizes the amount the expected Q-values violate the threshold:

Original: [Equation]
Revised: [Equation]

Original:  Experiments
Revised:  Experiments

Original: To recap, in the LP3 framework, Lagrangian relaxation corresponds to using linear scalarization(LS) and learning a single preference (Fig. 1).
Revised: To recap, in the LP3 framework, Lagrangian relaxation corresponds to using linear scalarization(LS) and learning a single preference (Fig. 1).

Original: Lagrangian relaxation is a widely-used approach for
Revised: Lagrangian relaxation is a widely-used approach for

Original: [Figure]
Revised: [Figure]

Original: constrained RL because of its simplicity and effectiveness [22, 12].
Revised: constrained RL because of its simplicity and effectiveness [22, 12].

Original: We proposed two novel changes:1) using MO-MPO instead of LS in Module 1, and 2) learning a distribution over preferences ratherthan a single preference in Module 2.
Revised: We proposed two novel changes:1) using MO-MPO instead of LS in Module 1, and 2) learning a distribution over preferences ratherthan a single preference in Module 2.

Original: In our empirical evaluation, we ﬁrst run ablation studies thatstudy the impact of each of these changes.
Revised: In our empirical evaluation, we ﬁrst run ablation studies thatstudy the impact of each of these changes.

Original: Then, we run further experiments with LP3 [MO-MPOD], that combines both improvements.
Revised: Then, we run further experiments with LP3 [MO-MPOD], that combines both improvements.

Original: The main baseline we compare against is LP3 [LS] , whichuses MPO [30] to optimize the Lagrangian dual objective (2).
Revised: The main baseline we compare against is LP3 [LS] , whichuses MPO [30] to optimize the Lagrangian dual objective (2).

Original: This is a strong Lagrangian-basedbaseline that obtains state-of-the-art results on benchmark constrained RL tasks (Appendix C.2).
Revised: This is a strong Lagrangian-basedbaseline that obtains state-of-the-art results on benchmark constrained RL tasks (Appendix C.2).

Original: Implementation.
Revised: Implementation.

Original: We train Gaussian policies parameterized by neural networks.
Revised: We train Gaussian policies parameterized by neural networks.

Original: Our implementations are based on the open-sourced MPO and MO-MPO in Acme [37].
Revised: Our implementations are based on the open-sourced MPO and MO-MPO in Acme [37].

Original: For LP3 [MO-MPO-D],we learn a separate distribution per preference (cid:15) k , initialized to uniform.
Revised: For LP3 [MO-MPO-D],we learn a separate distribution per preference (cid:15) k , initialized to uniform.

Original: We choose to use a discrete distribution because it is ﬂexible and interpretable.
Revised: We choose to use a discrete distribution because it is ﬂexible and interpretable.

Original: Unless otherwise mentioned, we use (9)as the ﬁtness function for LP3 [MO-MPO] and use (8) for LP3 [MO-MPO-D]—so that it convergesto a single policy, rather than a set of policies, for easier comparison with the single-preferenceapproaches.
Revised: Unless otherwise mentioned, we use (9)as the ﬁtness function for LP3 [MO-MPO] and use (8) for LP3 [MO-MPO-D]—so that it convergesto a single policy, rather than a set of policies, for easier comparison with the single-preferenceapproaches.

Original: We sweep over hyperparameters for each approach; details are in Appendix A.
Revised: We sweep over hyperparameters for each approach; details are in Appendix A.

Original: 5.1 Module 1: Linear Scalarization vs. MO-MPO
Revised: 5.1 Module 1: Linear Scalarization vs. MO-MPO

Original: To ﬁrst assess the impact of using MO-MPO instead of linear scalarization, we compare LP3 [MOMPO] against LP3 [LS] across several domains.
Revised: To ﬁrst assess the impact of using MO-MPO instead of linear scalarization, we compare LP3 [MOMPO] against LP3 [LS] across several domains.

Original: LP3 [LS] is an ablation of LP3 [MO-MPO] withrespect to Module 1; both learn a single preference in the same way for Module 2.
Revised: LP3 [LS] is an ablation of LP3 [MO-MPO] withrespect to Module 1; both learn a single preference in the same way for Module 2.

Original: Toy Domain.
Revised: Toy Domain.

Original: First, we consider a toy domain with a continuous action ( a ∈ R ) and two objectives( r ( a ) ∈ R 2 ).
Revised: First, we consider a toy domain with a continuous action ( a ∈ R ) and two objectives( r ( a ) ∈ R 2 ).

Original: The constraint is on r 2 ( a ) .
Revised: The constraint is on r 2 ( a ) .

Original: For the reward, we use either the Schaffer or FonsecaFleming function, which are standard test functions in multi-objective optimization [38].
Revised: For the reward, we use either the Schaffer or FonsecaFleming function, which are standard test functions in multi-objective optimization [38].

Original: For eachfunction, we train policies for four constraint thresholds.
Revised: For eachfunction, we train policies for four constraint thresholds.

Original: For Schaffer, both approaches ﬁnd theoptimal solutions.
Revised: For Schaffer, both approaches ﬁnd theoptimal solutions.

Original: However, for Fonseca-Fleming, which has a concave Pareto front, only LP3[MO-MPO] can ﬁnd all optimal solutions (Fig. 2, right).
Revised: However, for Fonseca-Fleming, which has a concave Pareto front, only LP3[MO-MPO] can ﬁnd all optimal solutions (Fig. 2, right).

Original: LP3 [LS] only ﬁnds solutions at the twoextremes, even after tuning the learning rate.
Revised: LP3 [LS] only ﬁnds solutions at the twoextremes, even after tuning the learning rate.

Original: This is likely because linear scalarization cannot ﬁndsolutions on the concave portions of a Pareto front [36].
Revised: This is likely because linear scalarization cannot ﬁndsolutions on the concave portions of a Pareto front [36].

Original: Even in this simple setting, we can see thepotential advantages of using MO-MPO over linear scalarization, in Module 1 of LP3.
Revised: Even in this simple setting, we can see thepotential advantages of using MO-MPO over linear scalarization, in Module 1 of LP3.

Original: Humanoid.
Revised: Humanoid.

Original: We next consider two high-dimensional locomotion tasks, humanoid run and walkfrom the DeepMind Control Suite [39].
Revised: We next consider two high-dimensional locomotion tasks, humanoid run and walkfrom the DeepMind Control Suite [39].

Original: The constraint is imposed on the expected negative controlnorm (i.e., −(cid:107) a (cid:107) 2 ), which roughly captures the “energy” expended by the agent.
Revised: The constraint is imposed on the expected negative controlnorm (i.e., −(cid:107) a (cid:107) 2 ), which roughly captures the “energy” expended by the agent.

Original: We train policies using LP3 [LS] and LP3 [MO-MPO] for a variety of constraint thresholds.
Revised: We train policies using LP3 [LS] and LP3 [MO-MPO] for a variety of constraint thresholds.

Original: Weplot the average task reward and cost for the fully-trained policies in Fig.
Revised: Weplot the average task reward and cost for the fully-trained policies in Fig.

Original: LP3 [MO-MPO] ﬁndshigher-quality solutions for both tasks, especially for walk, which may be because its ground-truth
Revised: LP3 [MO-MPO] ﬁndshigher-quality solutions for both tasks, especially for walk, which may be because its ground-truth

Original: [Table]
Revised: [Table]

Original: Pareto front is concave.
Revised: Pareto front is concave.

Original: LP3 [MO-MPO] is also more sample efﬁcient: for humanoid run, mid-waythrough training LP3 [MO-MPO] policies obtain moderate task reward, while LP3 [LS] policieshave not learned the task at all (Fig. 3, “run, mid-training”).
Revised: LP3 [MO-MPO] is also more sample efﬁcient: for humanoid run, mid-waythrough training LP3 [MO-MPO] policies obtain moderate task reward, while LP3 [LS] policieshave not learned the task at all (Fig. 3, “run, mid-training”).

Original: This could be because LP3 [MO-MPO]can optimize for both objectives at once, whereas LP3 [LS] cannot (see analysis in Appendix C.4).
Revised: This could be because LP3 [MO-MPO]can optimize for both objectives at once, whereas LP3 [LS] cannot (see analysis in Appendix C.4).

Original: 5.2 Module 2: Single Preference vs. Distribution over Preferences
Revised: 5.2 Module 2: Single Preference vs. Distribution over Preferences

Original: We combine MO-MPO, because it outperforms linear scalarization, with learning a distribution overpreferences to obtain LP3 [MO-MPO-D].
Revised: We combine MO-MPO, because it outperforms linear scalarization, with learning a distribution overpreferences to obtain LP3 [MO-MPO-D].

Original: LP3 [MO-MPO] is an ablation of LP3 [MO-MPO-D] withrespect to Module 2.
Revised: LP3 [MO-MPO] is an ablation of LP3 [MO-MPO-D] withrespect to Module 2.

Original: For humanoid, both approaches perform similarly (Fig. 3).
Revised: For humanoid, both approaches perform similarly (Fig. 3).

Original: A key advantage of using LP3 [MO-MPO-D], as opposed to single-preference approaches, is theability to solve a CMO-MDP and discover a set of constraint-satisfying non-dominated solutions,rather than ﬁnding a single solution.
Revised: A key advantage of using LP3 [MO-MPO-D], as opposed to single-preference approaches, is theability to solve a CMO-MDP and discover a set of constraint-satisfying non-dominated solutions,rather than ﬁnding a single solution.

Original: This is useful because in practice, we may not know the trueconstraint ahead of time, and would thus prefer to see several options along the Pareto front.
Revised: This is useful because in practice, we may not know the trueconstraint ahead of time, and would thus prefer to see several options along the Pareto front.

Original: Weevaluate this in the following section on a task with three objectives, two of which are constrained.
Revised: Weevaluate this in the following section on a task with three objectives, two of which are constrained.

Original: 5.3 Evaluation of Full Algorithm
Revised: 5.3 Evaluation of Full Algorithm

Original: Finally, we focus on comparing LP3 [MO-MPO-D], which combines both improvements, againstexisting approaches in the literature, as well as our LP3 [LS], a strong Lagrangian-based approach.
Revised: Finally, we focus on comparing LP3 [MO-MPO-D], which combines both improvements, againstexisting approaches in the literature, as well as our LP3 [LS], a strong Lagrangian-based approach.

Original: RWRL.
Revised: RWRL.

Original: We use the same four Real World RL (RWRL)
Revised: We use the same four Real World RL (RWRL)

Original: [40] tasks and constraints used by Calianet al.
Revised: [40] tasks and constraints used by Calianet al.

Original: [5], and we compare against the algorithms evaluated in that work (described in Appendix B.5).
Revised: [5], and we compare against the algorithms evaluated in that work (described in Appendix B.5).

Original: The cost is binary and depends on the task, e.g., for humanoid walk, the cost is − 1 if the joint anglesare outside of a speciﬁed range, and 0 otherwise.
Revised: The cost is binary and depends on the task, e.g., for humanoid walk, the cost is − 1 if the joint anglesare outside of a speciﬁed range, and 0 otherwise.

Original: For each task, we used LP3 [MO-MPO-D] and LP3[LS] to train ﬁve policies with different random initializations.
Revised: For each task, we used LP3 [MO-MPO-D] and LP3[LS] to train ﬁve policies with different random initializations.

Original: LP3[MO-MPO-D] performs best onthree out of four tasks, incurring less cost while achieving on-par or better task reward (Table 1).
Revised: LP3[MO-MPO-D] performs best onthree out of four tasks, incurring less cost while achieving on-par or better task reward (Table 1).

Original: Safety Gym.
Revised: Safety Gym.

Original: From OpenAI Safety Gym [22], we use the Level 2 point mass tasks: goal, button,and push.
Revised: From OpenAI Safety Gym [22], we use the Level 2 point mass tasks: goal, button,and push.

Original: There is a sparse reward for either reaching a goal location or pushing a box to a goal.
Revised: There is a sparse reward for either reaching a goal location or pushing a box to a goal.

Original: There is a cost for running into or over objects.
Revised: There is a cost for running into or over objects.

Original: On two out of the three tasks, LP3 [MO-MPO-D]ﬁnds better solutions than LP3 [LS], on harder-to-satisfy constraint thresholds (Fig. 3).
Revised: On two out of the three tasks, LP3 [MO-MPO-D]ﬁnds better solutions than LP3 [LS], on harder-to-satisfy constraint thresholds (Fig. 3).

Original: For pointpush and for easier-to-satisfy constraint thresholds, LP3 [LS] performs on par with LP3 [MO-MPOD]
Revised: For pointpush and for easier-to-satisfy constraint thresholds, LP3 [LS] performs on par with LP3 [MO-MPOD]

Original: (see Appendix C).
Revised: (see Appendix C).

Original: We hypothesize this is because when meeting the constraint does not conﬂictmuch with the task objective, it is easier to ﬁnd high-task-reward solutions that incur small cost.
Revised: We hypothesize this is because when meeting the constraint does not conﬂictmuch with the task objective, it is easier to ﬁnd high-task-reward solutions that incur small cost.

Original: Robot.
Revised: Robot.

Original: We created a simulated robot manipulation task with the Sawyer arm, with a sparse rewardfor reaching a green block and pushing it to a target position.
Revised: We created a simulated robot manipulation task with the Sawyer arm, with a sparse rewardfor reaching a green block and pushing it to a target position.

Original: There is a cost for either the gripperor green block colliding with the yellow block (Fig. 4, left).
Revised: There is a cost for either the gripperor green block colliding with the yellow block (Fig. 4, left).

Original: On this task, LP3 [MO-MPO-D] and
Revised: On this task, LP3 [MO-MPO-D] and

Original: LP3 [LS] converge to similar task performance and constraint satisfaction, but LP3 [LS] takes abouttwice as many actor steps to do so (Fig. 4).
Revised: LP3 [LS] converge to similar task performance and constraint satisfaction, but LP3 [LS] takes abouttwice as many actor steps to do so (Fig. 4).

Original: This is likely due to the use of MO-MPO in Module1—recall that in the ﬁrst ablation study, this improved sample efﬁciency in humanoid run (Sec. 5.1).
Revised: This is likely due to the use of MO-MPO in Module1—recall that in the ﬁrst ablation study, this improved sample efﬁciency in humanoid run (Sec. 5.1).

Original: Multiple Constraints.
Revised: Multiple Constraints.

Original: We also evaluate on tasks with threeobjectives: a more difﬁcult version ofpoint goal from Safety Gym (see Appendix B.3) and  walker run  from the Control Suite.
Revised: We also evaluate on tasks with three and four objectives: walker run andhumanoid walk from the Control Suite.

Original: Inpointgoal, there is a sparse reward for reaching the goal and separate costs for crossing over hazards andbumping into vases. In walker run,  adense reward is given for torso height and moving forward,and there is a cost for energy usage (captured by negative control norm). [Figure]
Revised: In walker run, there are three objectives: a torso heightreward, a moving forward reward, and a cost for energy usage (captured by negative control norm). In humanoid walk, there are four objectives: the task reward (from the standard Control Suite task),a moving forward reward, a moving left reward, and a cost for energy usage.

Original: [Figure]
Revised: [Figure]

Original: For point goal

Original: [Figure]
Revised: [Figure]

Original: we train policies for several constraint thresholds: for each type of cost, we ﬁx itsthreshold to − 2 and vary the other between − 10 and 0 . LP3 [LS] cannot learn the task when thehazards constraint is challenging (at − 2 ), whereas LP3 [MO-MPO-D] ﬁnds arange of solutions(Fig.
Revised: we can only plot at a

Original: 5, left).
Revised: , we use

Original: For walker run, we set inequality constraints on movingforward and energy usage.
Revised: For both tasks, we set inequality constraints on moving and energy usage.

Original: We use LP3 [MO-MPO-D] to train preference-conditioned policies using the ﬁtness function (9).
Revised: We use LP3 [MO-MPOD] to train preference-conditioned policies using the ﬁtness function (9).

Original: These policies learn a portion of the Pareto front that satisﬁes the constraints (Fig. 5, right). This isuseful when the constraint is soft or notexactly known  a piori.
Revised: These policies learn aportion of the Pareto front that satisﬁes the constraints (Fig. 5), which is useful when the constraintis soft or not known exactly a piori.

Revised: For humanoid walk, the agent learns to smoothly interpolatebetween walking left and walking forward (Fig. 5, right-most plot).

Revised: In this task, walking left is amore challenging gait and would not likely emerge naturally without a constraint on this objective.

Original: Appendix C contains further experiments and analysis, including on constraint satisfaction (Appendix C.5) and evaluation with multiple random seeds per constraint threshold (Appendix C.7).
Revised: Appendix C contains further experiments and analysis, including on constraint satisfaction (Appendix C.5) and evaluation with multiple random seeds per constraint threshold (Appendix C.7).

Original:  Conclusion and Future Work
Revised:  Conclusion and Future Work

Original: In this paper, we proposed to formulate constrained RL as a constrained multi-objective MDP.
Revised: In this paper, we proposed to formulate constrained RL as a constrained multi-objective MDP.

Original: Basedon this multi-objective perspective, we described a general framework, LP3, for solving constrained
Revised: Basedon this multi-objective perspective, we described a general framework, LP3, for solving constrained

Original: RL problems.
Revised: RL problems.

Original: We used this perspective to introduce two new algorithms: 1) using a better multiobjective RL algorithm, MO-MPO, instead of linear scalarization; and 2) learning a distributionover preferences instead of a single preference, to ﬁnd a set of constraint-satisfying non-dominatedpolicies.
Revised: We used this perspective to introduce two new algorithms: 1) using a better multiobjective RL algorithm, MO-MPO, instead of linear scalarization; and 2) learning a distributionover preferences instead of a single preference, to ﬁnd a set of constraint-satisfying non-dominatedpolicies.

Original: For safe RL, one could initialize the preference distribution to prioritize minimizing cost.
Revised: For safe RL, one could initialize the preference distribution to prioritize minimizing cost.

Original: This work has several limitations.
Revised: This work has several limitations.

Original: One is that LP3 [MO-MPO(-D)] has not been validated yet ona real robot platform.
Revised: One is that LP3 [MO-MPO(-D)] has not been validated yet ona real robot platform.

Original: However, MPO with Lagrangian relaxation has been successfully run onrobots [e.g. 14], and our ablation study suggests LP3 [MO-MPO] should perform at least as well.
Revised: However, MPO with Lagrangian relaxation has been successfully run onrobots [e.g. 14], and our ablation study suggests LP3 [MO-MPO] should perform at least as well.

Original: Another limitation is that LP3 depends on accurate policy evaluation to ensure constraint satisfaction, because it relies on Q-value estimates to learn preferences.
Revised: Another limitation is that LP3 depends on accurate policy evaluation to ensure constraint satisfaction, because it relies on Q-value estimates to learn preferences.

Original: The preference learning rate cannotbe too high, otherwise the distribution will converge based on Q-values estimates for a partiallytrained policy—which may not accurately reﬂect which preferences will lead to constraint satisfaction for the fully-trained policy.
Revised: The preference learning rate cannotbe too high, otherwise the distribution will converge based on Q-values estimates for a partiallytrained policy—which may not accurately reﬂect which preferences will lead to constraint satisfaction for the fully-trained policy.

Original: In addition, using a discrete distribution for the preferences requiresthat its support includes preferences that satisfy the constraint threshold.
Revised: In addition, using a discrete distribution for the preferences requiresthat its support includes preferences that satisfy the constraint threshold.

Original: Finally, a limitation is thatwe focus on problems with up to three objectives.
Revised: Finally, a limitation is thatwe focus on problems with up to four objectives.

Original: In principle, LP3 [MO-MPO-D] can be scaled upto more objectives with minimal additional computational cost, and this can be easily parallelized.
Revised: In principle, LP3 [MO-MPO-D] can be scaled upto more objectives with minimal additional computational cost, and this can be easily parallelized.

Original: However, the more objectives there are that are in conﬂict with each other, the more difﬁcult theproblem is.
Revised: However, the more objectives there are that are in conﬂict with each other, the more difﬁcult theproblem is.

Original: It would be valuable to investigate performance on these problems in future work.
Revised: It would be valuable to investigate performance on these problems in future work.