Original: P ARETO P OLICY A DAPTATION
Revised: P ARETO P OLICY A DAPTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a policy gradient method for Multi-Objective Reinforcement Learning under unknown, linear preferences.
Revised: We present a policy gradient method for Multi-Objective Reinforcement Learning under unknown, linear preferences.

Original: By enforcing Pareto stationarity, a ﬁrst-order condition for Pareto optimality, we are able to design a simple policy gradient algorithm that approximates the Pareto front and infers the unknown preferences.
Revised: By enforcing Pareto stationarity, a first-order condition for Pareto optimality, we are able to design a simple policy gradient algorithm that approximates the Pareto front and infers the unknown preferences.

Original: Our method relies on a projected gradient descent solver that identiﬁes common ascent directions for all objectives.
Revised: Our method relies on a projected gradient descent solver that identifies common ascent directions for all objectives.

Original: Leveraging the solution of that solver, we introduce Pareto Policy Adaptation ( PPA ), a loss function that adapts the policy to be optimal with respect to any distribution over preferences.
Revised: Leveraging the solution of that solver, we introduce Pareto Policy Adaptation ( PPA ), a loss function that adapts the policy to be optimal with respect to any distribution over preferences.

Original: PPA uses implicit differentiation to back-propagate the loss gradient bypassing the operations of the projected gradient descent solver.
Revised: PPA uses implicit differentiation to back-propagate the loss gradient bypassing the operations of the projected gradient descent solver.

Original: Our approach is straightforward, easy to implement and can be used with all existing policy gradient and actor-critic methods.
Revised: Our approach is straightforward, easy to implement and can be used with all existing policy gradient and actor-critic methods.

Original: We evaluate our method in a series of reinforcement learning tasks.
Revised: We evaluate our method in a series of reinforcement learning tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep reinforcement learning has a pivotal role in solving several control problems of practical interest that would otherwise be intractable, such as robotic locomotion (1), the Atari (2) and Go games (3) to name a few.
Revised: Deep reinforcement learning has a pivotal role in solving several control problems of practical interest that would otherwise be intractable, such as robotic locomotion (1), the Atari (2) and Go games (3) to name a few.

Original: While such approaches have focused on the scalar reward setting, there exist many real-world problems that have multiple, conﬂicting objectives and, therefore, cannot be addressed by the current reinforcement learning tools.
Revised: While such approaches have focused on the scalar reward setting, there exist many real-world problems that have multiple, conflicting objectives and, therefore, cannot be addressed by the current reinforcement learning tools.

Original: In such scenario, it is challenging to ﬁnd an optimal control policy as the trade-offs ( preferences ) among the objectives may not be precisely known at training time or may differ from user to user.
Revised: In such scenario, it is challenging to find an optimal control policy as the trade-offs ( preferences ) among the objectives may not be precisely known at training time or may differ from user to user.

Original: The paradigm of Multi-Objective Reinforcement Learning (MORL) provides a generalized framework for dealing with multi-dimensional rewards signals and has been identiﬁed as one of the main challenges of real-world reinforcement learning (4).
Revised: The paradigm of Multi-Objective Reinforcement Learning (MORL) provides a generalized framework for dealing with multi-dimensional rewards signals and has been identified as one of the main challenges of real-world reinforcement learning (4).

Original: However, learning optimal policies for MORL has been proven to be quite challenging, because most strategies require either to have access to or be able to infer a quantiﬁcation of the relative importance of the objectives, or they perform sophisticated searches in the value or policy space aiming to ﬁnd an ensemble of policies that are non-inferior to each other.
Revised: However, learning optimal policies for MORL has been proven to be quite challenging, because most strategies require either to have access to or be able to infer a quantification of the relative importance of the objectives, or they perform sophisticated searches in the value or policy space aiming to find an ensemble of policies that are non-inferior to each other.

Original: The former methods lack adaptability and generalizability, as their performance is tied to the given or inferred preferences, while the later ones suffer from scalability issues, because they learn multiple policies, and are usually complicated to implement.
Revised: The former methods lack adaptability and generalizability, as their performance is tied to the given or inferred preferences, while the later ones suffer from scalability issues, because they learn multiple policies, and are usually complicated to implement.

Original: In this work, we aim to strike a balance between those two families of methods.
Revised: In this work, we aim to strike a balance between those two families of methods.

Original: Related Work : Multi-Objective Optimization (MOO) provides the fundamental tools for MORL.
Revised: Related Work : Multi-Objective Optimization (MOO) provides the fundamental tools for MORL.

Original: In MOO, there are two common solution concepts: scalarization and Pareto optimality .
Revised: In MOO, there are two common solution concepts: scalarization and Pareto optimality .

Original: The former one derives a scalar objective and uses standard single-objective optimization techniques (5; 6; 7; 8; 9).
Revised: The former one derives a scalar objective and uses standard single-objective optimization techniques (5; 6; 7; 8; 9).

Original: The later method is based on the concept of Pareto dominance and considers the set of all noninferior solutions (10; 11).
Revised: The later method is based on the concept of Pareto dominance and considers the set of all noninferior solutions (10; 11).

Original: Multiple gradient methods that leverage ﬁrst-order, necessary conditions for Pareto optimality have also been developed (12; 13; 14).
Revised: Multiple gradient methods that leverage first-order, necessary conditions for Pareto optimality have also been developed (12; 13; 14).

Original: Similarly to MOO, existing work in MORL can be roughly divided into two main categories: Singlepolicy methods aim to maximize a single, scalarized reward.
Revised: Similarly to MOO, existing work in MORL can be roughly divided into two main categories: Singlepolicy methods aim to maximize a single, scalarized reward.

Original: These methods essentially transform the problem into a single-objective MDP and differ mostly in the way they determine and express the preferences.
Revised: These methods essentially transform the problem into a single-objective MDP and differ mostly in the way they determine and express the preferences.

Original: Scalarization is usually performed using a weighted sum of the reward vector (15; 16; 17) or, less commonly, using linear mappings (18).
Revised: Scalarization is usually performed using a weighted sum of the reward vector (15; 16; 17) or, less commonly, using linear mappings (18).

Original: Different single-policy methods are based on thresholds or lexicographical ordering (19) or different classes of preferences (20; 21).
Revised: Different single-policy methods are based on thresholds or lexicographical ordering (19) or different classes of preferences (20; 21).

Original: More recently, a scalarized Q-learning algorithm has been developed (22) which uses the concept of corner weight to infer and optimize over preferences.
Revised: More recently, a scalarized Q-learning algorithm has been developed (22) which uses the concept of corner weight to infer and optimize over preferences.

Original: It was extended to handle dynamic preferences in (23) and to utilize the convex envelop of the Pareto front in (24).
Revised: It was extended to handle dynamic preferences in (23) and to utilize the convex envelop of the Pareto front in (24).

Original: Finally, a scale-invariant supervised learning approach for encoding the preferences was developed in (25).
Revised: Finally, a scale-invariant supervised learning approach for encoding the preferences was developed in (25).

Original: On the other hand, multiple-policy methods use the vector-valued reward signal and learn a set of policies that approximate the Pareto front.
Revised: On the other hand, multiple-policy methods use the vector-valued reward signal and learn a set of policies that approximate the Pareto front.

Original: The Pareto optimal solutions are the subset of non-inferior and equally good alternative polices among all distributions in the policy space and multiple-policy methods mainly differ on the approximation scheme for the Pareto front.
Revised: The Pareto optimal solutions are the subset of non-inferior and equally good alternative polices among all distributions in the policy space and multiple-policy methods mainly differ on the approximation scheme for the Pareto front.

Original: One common approach is to perform multiple runs (16; 15) of a single scalarized reward function over a set of preferences.
Revised: One common approach is to perform multiple runs (16; 15) of a single scalarized reward function over a set of preferences.

Original: Unfortunately, such methods lack scalability to high-dimensional rewards.
Revised: Unfortunately, such methods lack scalability to high-dimensional rewards.

Original: Other approaches leverage convex approximations of the Pareto front (26) or linear approximations of the value function (27; 28) to learn optimal deterministic policies.
Revised: Other approaches leverage convex approximations of the Pareto front (26) or linear approximations of the value function (27; 28) to learn optimal deterministic policies.

Original: Multi-objective ﬁtted Q-iteration (29; 30) enables us to learn policies for all linear preferences by encapsulating the preference vector as an input to the Q-function approximator.
Revised: Multi-objective fitted Q-iteration (29; 30) enables us to learn policies for all linear preferences by encapsulating the preference vector as an input to the Q-function approximator.

Original: A policy gradient method based on discrete approximations of the Pareto front was proposed in (31) and enhanced to utilize continuous approximations in (32).
Revised: A policy gradient method based on discrete approximations of the Pareto front was proposed in (31) and enhanced to utilize continuous approximations in (32).

Revised: A gradient-based method that learns a manifold on the policy parameter space leveraging episodic exploration and importance sampling was developed in (33).

Original: Finally, a prediction-guided evolutionary algorithm which is able to ﬁnd dense approximation to the Pareto front is proposed in (33).
Revised: Finally, a prediction-guided evolutionary algorithm which is able to find dense approximation to the Pareto front was proposed in (34).

Original: Single-policy methods have the advantage of learning a single policy and being simple to implement.
Revised: Single-policy methods have the advantage of learning a single policy and being simple to implement.

Original: However, they suffer from instability issues, as a small change on the preferences may lead to performance collapse (34), and also rely on heuristics to infer the preferences when they are not known.
Revised: However, they suffer from instability issues, as a small change on the preferences may lead to performance collapse (35), and also rely on heuristics to infer the preferences when they are not known.

Original: On the other hand, multiple-policy methods enjoy the advantage of being able to adapt to changing preferences because the solution (being an approximation to the Pareto front) encapsulates all trade-offs between the objectives.
Revised: On the other hand, multiple-policy methods enjoy the advantage of being able to adapt to changing preferences because the solution (being an approximation to the Pareto front) encapsulates all trade-offs between the objectives.

Original: However, this beneﬁt comes at a higher computational cost as we typically have to learn and store multiple policies.
Revised: However, this benefit comes at a higher computational cost as we typically have to learn and store multiple policies.

Original: In this paper, we aim to bridge the gab between single and multiple policy MORL methods: Our method learns a single policy along with the underlying preference vectors and is able to adapt the inferred preference to any preference distribution.
Revised: In this paper, we aim to bridge the gab between single and multiple policy MORL methods: Our method learns a single policy along with the underlying preference vectors and is able to adapt the inferred preference to any preference distribution.

Original: Our preference vectors are inferred, not by using heuristics, but by approximating the
Revised: Our preference vectors are inferred, not by using heuristics, but by approximating the

Original: Pareto front via a ﬁrst-order condition (see Fig.
Revised: Pareto front via a first-order condition (see Fig.

Original: 1 for an example).
Revised: 1 for an example).

Original: Our method is inspired by the multiple gradient descent algorithm for MOO introduced in (13) as well as its application in the ﬁeld of multi-task supervised learning presented in (35).
Revised: Our method is inspired by the multiple gradient descent algorithm for MOO introduced in (13) as well as its application in the field of multi-task supervised learning presented in (36).

Original: [Figure]
Revised: [Figure]

Original: Contributions :
Revised: Contributions :

Original: We propose a policy gradient method for multi-objective reinforcement learning under unknown, linear preferences.
Revised: We propose a policy gradient method for multi-objective reinforcement learning under unknown, linear preferences.

Original: Initially, we present Pareto stationarity as a necessary, ﬁrstorder condition for Pareto optimality and develop a multi-objective policy gradient algorithm which uses a projected gradient descent solver to search for and take steps in a common ascent direction for all objectives.
Revised: Initially, we present Pareto stationarity as a necessary, firstorder condition for Pareto optimality and develop a multi-objective policy gradient algorithm which uses a projected gradient descent solver to search for and take steps in a common ascent direction for all objectives.

Original: Following that, we tackle the problem of adapting the policy gradient to any given preference distribution.
Revised: Following that, we tackle the problem of adapting the policy gradient to any given preference distribution.

Original: We utilize our method from the ﬁrst part and introduce the Pareto Policy Adaptation (PPA), a loss function that penalizes deviations between the given preference distribution and the recovered preferences.
Revised: We utilize our method from the first part and introduce the Pareto Policy Adaptation (PPA), a loss function that penalizes deviations between the given preference distribution and the recovered preferences.

Original: Using implicit differentiation, we are able to back-propagate the gradient of PPA bypassing the operations of the projected gradient descent solver, which makes our method applicable to real-world problems.
Revised: Using implicit differentiation, we are able to back-propagate the gradient of PPA bypassing the operations of the projected gradient descent solver, which makes our method applicable to real-world problems.

Original: We evaluate our method in a series of synthetic and real-world domains.
Revised: We evaluate our method in a series reinforcement learning tasks.

Original: P RELIMINARIES 2. M ARKOV D ECISION P ROCESSES
Revised: The most closely related work to ours is presented in (31). The authors discuss the existence of common ascent directions, introduce a quadratic optimization problem for their computation and give two gradient-based learning algorithms. Even though our method identifies the common ascent directions using a similar optimization problem, we leverage these directions to define the PPA loss function, which is a simple and efficient way 1) to incorporate the ascent directions into any learning algorithm and 2) to account for a given preference distribution. Leveraging implicit differentiation,

Revised: PPA enables us to incorporate multiple rewards and preference distributions into any modern policy gradient algorithm by a mere modification of the loss function and by implementing a projected gradient descent optimizer.

Original: We consider the framework of  Markov Decision Processes (MDP) M := ( S , A , R , P  ) , where S and A denote the (ﬁnite) state and(continuous) action spaces, P : S×A×S → [0 , 1] is the transition kernel representing the environment dynamics,i.e., P ( s | s (cid:48) , a ) is the probability of transitioning from state s (cid:48) to state s taking action a , r ( s, a ) ≡ R :S ×
Revised: We consider the framework of Multi-Objective Markov Decision Process (MOMDP) M := ( S , A , R , P , γ, Ω , f Ω ) , where S and A denote the (discrete) state and action spaces, P : S × A × S → [0 , 1] is the transition kernel representing the environment dynamics, r ( s, a ) ≡ R :

Revised: S ×

Original: A → R  is a function mapping state-action tuples to rewards.
Revised: A → R M is a function mapping state-action tuples to reward vectors, Ω is the space of preferences and f ω : R M → R a preference function which produces a scalar utility using a preference ω ∈ Ω . Do declutter notation, we assume that there exists a given initial state. We consider linear preferences f ω ( r ( s, a ))

Revised: =

Original:  A stationary control policy π : S ×
Revised: ω T r ( s, a ) . A fixed preference vector ω ∈ Ω allows to directly compare the vectorized reward and value function by comparing their scalarized utilities, and, therefore, the MOMDP collapses to a standard MDP. A stationary control policy π : S ×

Original: A → P ( A ) , where P ( A ) is the Borel set of A , is a function that assigns a probability distribution over action for all states.
Revised: A → P ( A ) , where P ( A ) is the Borel set of A , is a function that assigns a probability distribution over actions for all states.

Original: We denote with Π the space of all stationary, stochastic policies.

Original: The value v π : S → R |S| of a policy π is deﬁned at state s ∈ S as v π ( s ) :
Revised: The value v π : S → R M ×|S| of a policy π is defined as v π ( s ) :

Original: =
Revised: =

Original: E π [ (cid:80) ∞ t =0 γ t r ( s t , a t ) | s 0 =
Revised: E π [ (cid:80) ∞ t =0 γ t r ( s t , a t ) | s 0 =

Original: s ] , where γ ∈ (0 , 1) is a discount factor and the expectation is taken under the distribution over trajectories τ
Revised: s ] , where γ ∈ (0 , 1) is a discount factor and the expectation is taken under the distribution over trajectories τ

Original: = ( s 0 , a 0 , s 1 , a , s 2 , . . . )
Revised: = ( s 0 , a 0 , s 1 , a , s 2 , . . . )

Original: obtained by starting from s and following a t ∼ π( ·| s t ) thereafter.
Revised: obtained by starting from s and following policy π thereafter.

Original: The goal of reinforcement learning is to ﬁnd a policy π ∗ ∈ Π that maximizes an objective, typically taken to be the expectation under the initial state distribution of the value function, i.e., π ∗ ∈ arg max π ∈P (Π) E s ∼ β

Original: [ v π ( s )] , where β is the initial state distribution.

Original: The discounted state-action occupation measure under policy π is deﬁned as d π ( s, a ) :
Revised: The discounted state-action occupation measure under policy π is defined as d π ( s, a ) :

Original: = (cid:80) ∞ t =0 γ t P ( s t = s, a t = a | π ) for all states s ∈ S and allows us to write the value function compactly as v π ( s ) =
Revised: = (cid:80) ∞ t =0 γ t P ( s t = s, a t = a | π ) for all states s ∈ S and allows us to write the value function compactly as v π ( s ) =

Original: E d π [ r ( s, a ) | s 

Revised: E d π [ r ( s, a ) | s

Original: = s ] .
Revised: = s ] .

Original: A Multi-Objective Markov Decision Process (MOMDP) M := ( S , A , R , P , β, γ, Ω , f Ω ) is the same as an MDP with the reward r ( s, a ) ≡ R : S × A → R M being a vector-valued function.
Revised: Definition 1.

Original: Additionally, the MDP tuple is augmented with a space of preferences Ω and a preference function f ω : R M

Original: → R which produces a scalar utility using a preference ω ∈ Ω . In this work, we consider linear preferences f ω ( r ( s, a )) = ω T r
Revised: Considering all possible returns ˜ r := (cid:80) ∞ t =0 γ t r ( s t , a t ) , we define the Reward Space

Original: ( s, a ) . Analogously to MDPs, the value v π : S → R M ×|S| of a policy π is deﬁned as v π ( s ) : =
Revised: Pareto Coverage Set (RS-PCS) or Pareto frontier as F ∗ r := { ˜ r : ˜ r ∈ R M , ˜ r ′ ≻

Original: E π [ (cid:80) ∞ t =0 γ t r ( s t , a t ) | s 0 = s ] . We observe that a ﬁxed preference vector ω ∈ Ω allows to directly compare the vectorized reward and value function by comparing their scalarized utilities, and, therefore, the MOMDP collapses to a standard MDP. On the other hand, if we consider all possible returns ˜ r [-:= (cid:80) ∞ t =
 γ t r ( s t , a t )-] , we deﬁne the Reward
Revised: P ˜ r [-:= (cid:80) ∞ t =
 γ t r ( s t , a t )-]} , where the symbol ≻

Original: Space Pareto Coverage Set ( RS-PCS ) or Pareto frontier as F ∗ r := { ˜ r : ˜ r ∈ R M , ˜ r (cid:48) (cid:31) P ˜ r } , where the symbol (cid:31) P denotes Pareto dominance : greater or equal in all objectives and strictly greater in at least one objective.
Revised: P denotes Pareto dominance: greater or equal in all objectives and strictly greater in at least one objective.

Original: For all possible preferences in Ω and under the linear preference assumption, we deﬁne the Reward Space Convex Coverage Set ( RS-CCS ) as
Revised: For all possible preferences in Ω and under the linear preference assumption, we define the Reward Space Convex Coverage Set (RS-CCS) as

Original: [Equation]
Revised: [Equation]

Original: The set C ∗ R , Ω contains all discounted reward vectors that could be optimal for some preference vector in Ω .
Revised: The set C ∗ R , Ω contains all discounted reward vectors that could be optimal for some preference vector in Ω .

Original: Anything in F ∗ r but not in C ∗ R , Ω cannot be useful under linear preferences.
Revised: Anything in F ∗ r but not in C ∗ R , Ω cannot be useful under linear preferences.

Original: In that case, it sufﬁces to restrict our analysis to C ∗ R , Ω .
Revised: In that case, it suffices to restrict our analysis to C ∗ R , Ω .

Original: Also, we assume, without loss of generality, that (cid:80) m ω m = 1 , ω m ≥ 0 , m = 1 , 2 . . .
Revised: Also, we assume, without loss of generality, that [Equation] m [Equation]

Original:  M for all ω ∈ Ω , i.e., the preference vector is a convex combination of the objectives.
Revised: , ω [Equation] ≥ 0 , m = 1 , 2 . . . M for all ω ∈ Ω , i.e., the preference vector is a convex combination of the objectives.

Original: L EARNING WITH U NKNOWN P REFERENCES

Original: In this section, we propose a new policy gradient algorithm for MORL under unknown preferences.

Original: The key idea of our approach is to infer the preference vector ω , use that ω as scalarization coefﬁcient for the multi-objective criterion and ascent on the gradient of the scalarized value.

Original: We start by extending the deﬁnition of Pareto optimality from the reward space(given in Sec. 2.2) to the policy space.
Revised: We extend the definition of Pareto optimality from the reward space to the policy space.

Original: Deﬁnition 1.
Revised: Definition 2.

Original: A policy π dominates a policy π (cid:48) if and only if v π (cid:31) P v π (cid:48) .
Revised: A policy π dominates a policy π ′ if and only if v π ≻ P v π ′ .

Original: A policy π ∗ is called
Revised: A policy π ∗ is called

Original: Pareto optimal if there exists no other policy π that dominates π ∗ .
Revised: Pareto optimal if there exists no other policy π that dominates π ∗ .

Original: The set Π ∗ P of all Pareto optimal policies is called Pareto policy set and its image F ∗ π := { v π } π ∈ Π ∗ P is called Pareto policy frontier.
Revised: The set Π ∗ P of all Pareto optimal policies is called Pareto policy set and its image F ∗ π := { v π } π ∈ Π ∗ P is called Pareto policy frontier.

Original: The set
Revised: The set

Original: [Equation]
Revised: [Equation]

Original: is called the Policy Space Convex Coverage Set (PS-CCS).
Revised: is called the Policy Space Convex Coverage Set (PS-CCS).

Original: Lemma 1.
Revised: Lemma 1.

Original: Let π ∈ C ∗ Π , Ω be a policy.
Revised: Let π ∈ C ∗ Π , Ω be a policy.

Original: Then, we have E π [ ˜ r ] ∈ C ∗ R , Ω , i.e., the expected return vector under policy π belongs in the RS-CCS.
Revised: Then, we have E π [ ˜ r ] ∈ C ∗ R , Ω , i.e., the expected return vector under policy π belongs in the RS-CCS.

Original: Proof.

Original: Consider a policy π ∈ C ∗ Π , Ω and a policy π (cid:48)

Original: ∈ Π .

Original: Then, for each ω ∈ Ω , we can write

Original: [Equation]

Original: where we used the importance sampling identity and deﬁned r (cid:48) ( s, a ) :

Original: = d π (cid:48) ( s, a ) /d

Original: π ( s, a ) r ( s, a ) .

Original: Per Def. 2, there exists ω ∈ Ω such that ω T v π ≥ ω T v π (cid:48) .

Original: Leveraging the fact that the value function can be expressed as v π =

Original: E π

Original: [ ˜ r ] as well as Eq. 3, we have ω T E π

Original: [ ˜ r ] ≥ ω T E π [ ˜ r (cid:48) ] . This implies that E π [ ˜ r ] ∈ C ∗ R , Ω and concludes the proof. Lemma 1  connects the PS-CCS with the RS-CCS and allows us to use the former one in place of the later.
Revised: Lemma 1 (proof given in the Appendix) connects the PS-CCS with the RS-CCS and allows us to use the former one in place of the later.

Original: This is very convenient because searching for Pareto optimal vectors in the reward space is not practically efﬁcient (e.g., due to the sparsity of reward signal) or useful (e.g., because they cannot be easily mapped to values or actions).
Revised: This is very convenient because searching for Pareto optimal vectors in the reward space is not practically efficient (e.g., due to the sparsity of reward signal) or useful (e.g., because they cannot be easily mapped to values or actions).

Original: By ﬁnding a policy π ∈ C ∗ Π , Ω , Lemma 1 guarantees that the expected return vector under π belongs in C ∗ R , Ω and, therefore, maps the problem of reward space exploration to policy space exploration.
Revised: By finding a policy π ∈ C ∗ Π , Ω , Lemma 1 guarantees that the expected return vector under π belongs in C ∗ R , Ω and, therefore, maps the problem of reward space exploration to policy space exploration.

Original: Additionally, Lemma 1 imposes no restrictions on the class of policies and allows us to adopt the common practice of parameterizing the policy bya high-dimensional vector
Revised: Additionally, Lemma 1 imposes no restrictions on the class of policies and allows us to adopt the common practice of parameterizing the policy by high-dimensional vector

Original: θ ∈ Θ
Revised: θ ∈ Θ

Original: (e.g., a neural network).
Revised: (e.g., a neural network).

Original: The deﬁnition of PS-CCS given in Eq.carries over to this parametric class as is, i.e., we can deﬁne the PS-CCS in the policy parameter space as follows:
Revised: The definition of PS-CCS given Eq. equation 3 carries over to this parametric class as is, i.e., we can define the PS-CCS in the policy parameter space as follows

Original: [Equation]
Revised: [Equation]

Revised:  L EARNING WITH U NKNOWN P REFERENCES

Revised: In this section, we propose a new policy gradient algorithm for MORL under unknown preferences.

Revised: The key idea of our approach is to infer the preference vector ω , use that ω as scalarization coefficient for the multi-objective criterion and ascent on the gradient of the scalarized value.

Original: To search for a Pareto optimal policy, we leverage a ﬁst-order condition known as Pareto stationarity.
Revised: To search for a Pareto optimal policy, we leverage a fist-order condition known as Pareto stationarity.

Original: Deﬁnition 2 (Pareto stationarity) .
Revised: Definition 3 (Pareto stationarity) .

Original: A policy π θ is called Pareto stationary if and only if 
Revised: A policy π θ is called Pareto stationary if and only if (cid:80)

Original: [Equation]
Revised: [Equation]

Original: for some
Revised: for some

Original: [Equation]
Revised: [Equation]

Original: Pareto stationarity implies that there exists a vanishing convex combination of the objectives’ gradients and is a necessary condition for Pareto optimality (13).
Revised: Pareto stationarity implies that there exists a vanishing convex combination of the objectives’ gradients and is a necessary condition for Pareto optimality (13).

Original: The following theorem connects the
Revised: The following theorem connect the

Original: PS-CCS with Pareto stationarity.
Revised: PS-CCS with Pareto stationarity.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let π θ ∈ C ∗ Θ , Ω be a policy in the PS-CCS.
Revised: Let π θ ∈ C ∗ Θ , Ω be a policy in the PS-CCS.

Original: Then, π θ is Pareto stationary.
Revised: Then, π θ is Pareto stationary.

Original: Proof.
Revised: Proof.

Original: Let π θ ∈ C ∗ Θ , Ω be a policy and ω the corresponding preferences, i.e., we have ω T v π θ ≥ ω T v π θ (cid:48) , ∀ θ (cid:48) ∈ Θ .
Revised: Let π θ ∈ C ∗ Θ , Ω be a policy and ω the corresponding preferences, i.e., we have ω T v π θ ≥ ω T v π θ ′ , ∀ θ ′ ∈ Θ .

Original: The last relation can be written as ω T E π θ [ ˜ r ] ≥ ω T E π θ (cid:48)
Revised: The last relation can be written as ω T E π θ [ ˜ r ] ≥ ω T E π θ ′

Original: [ ˜ r ] , ∀ θ (cid:48) ∈ Θ .
Revised: [ ˜ r ] , ∀ θ ′ ∈ Θ .

Original: This implies that θ is a global maximizer of ω T E π θ [ ˜ r ] or, equivalently, that its gradient at θ vanishes.
Revised: This implies that θ is a global maximizer of ω T E π θ [ ˜ r ] or, equivalently, that its gradient at θ vanishes.

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: which corresponds to the necessary condition for Pareto stationarity as per Def. 2.
Revised: which corresponds to the necessary condition for Pareto stationarity as per Def. 3.

Original: Theorem 1 essentially states that Pareto stationarity is a necessary condition for a policy to belong to PS-CCS.
Revised: Theorem 1 essentially states that Pareto stationarity is a necessary condition for a policy to belong to PS-CCS.

Original: Additionally, as stated in the proof, the convex combination of the objectives that leads to vanishing gradients coincides with the preferences that make the underlying policy part of the
Revised: Additionally, as stated in the proof, the convex combination of the objectives that leads to vanishing gradients coincides with the preferences that make the underlying policy part of the

Original: PS-CCS.
Revised: PS-CCS.

Original: This presents us with a strong necessary condition for recovering the unknown preferences and for identifying an optimal policy.
Revised: This presents us with a strong necessary condition for recovering the unknown preferences and for identifying an optimal policy.

Original: This is because the problem of ﬁnding a Pareto stationary is tractable via gradient methods.
Revised: This is because the problem of finding a Pareto stationary is tractable via gradient methods.

Original: In fact, if a policy is not Pareto stationary then there exists a common ascent direction for all objectives.
Revised: In fact, if a policy is not Pareto stationary then there exists a common ascent direction for all objectives.

Original: To ﬁnd this descent direction, we use the following theorem.
Revised: To find this descent direction, we use the following theorem.

Original: Theorem 2 ((13), Theorem 1) .
Revised: Theorem 2 ((13), Theorem 1) .

Original: Consider the following quadratic optimization problem
Revised: Consider the following quadratic optimization problem

Original: [Equation]
Revised: [Equation]

Original: Let ω ∗ be the optimal solution of Problem P A and µ ∗ the underlying minimal-norm objective.
Revised: Let ω ∗ be the optimal solution of Problem P A and µ ∗ the underlying minimal-norm objective.

Original: Then:
Revised: Then:

Original: either µ ∗ = 0 and π θ ∗ is a Pareto stationary policy 2.
Revised: either µ ∗ = 0 and π θ ∗ is a Pareto stationary policy 2.

Original: or (cid:80) Mm =1 ω ∗ m ∇ θ E d πθ [ r m ( s, a )] is a common ascent direction for all objectives.
Revised: or (cid:80) M m =1 ω ∗ m ∇ θ E d πθ [ r m ( s, a )] is a common ascent direction for all objectives.

Original: Problem P A is essentially equivalent to the formulation of the famous Minimum-Norm Point problem, which arisesin many ﬁelds such as optimal control (36), submodular minimization (37) and portfolio optimization (38).
Revised: Problem P A is essentially equivalent to the formulation of the famous Minimum-Norm Point problem, which arises many fields such as optimal control (37), submodular minimization (38) and portfolio optimization (39).

Original: The MNP problem has been well studied and several combinatorial and recursive algorithms have been proposed (39; 40; 41).
Revised: The MNP problem has been well studied and several combinatorial and recursive algorithms have been proposed (40; 41; 42).

Original: Such methods search for the exact solution and typically have exponential run-time (42), which prevents their application in high-dimensional control policies.
Revised: Such methods search for the exact solution and typically have exponential run-time (43), which prevents their application in high-dimensional control policies.

Original: Therefore, we leverage convex optimization methods for Problem 1.
Revised: Therefore, we adopt convex optimization approach for Problem 2.

Original: We start by noting that the constraint set E = { ω ∈ R M : (cid:80) Mm =1 ω m = 1 , ω m ≥ 0 , ∀ m = 1 , 2 . . .
Revised: We start by noting that the constrain set E = { ω ∈ R M : (cid:80) Mm =1 ω m = 1 , ω m ≥ 0 , ∀ m = 1 , 2 . . .

Original: M } is essentially the unit simplex and the projection problem, i.e., Π E ( ω ) =
Revised: M } is essentially the unit simplex and the projection problem, i.e., Π E ( ω ) =

Original: arg min δ ∈E (cid:107) ω − δ (cid:107) , can be efﬁciently solved.
Revised: arg min δ ∈E ∥ ω − δ ∥ , can be efficiently solved.

Original: In fact, there exists a unique dual variable τ ∈ R such thatΠ E ( ω ) =
Revised: In fact, there exists a unique dual variable τ ∈ R such that

Original: ( ω − τ ) + , where ( x ) + = max( x, 0) .
Revised: [Equation] where ( x ) + = max( x, 0) .

Original: To ﬁnd the dual variable, we enforce the equality constraint, i.e., (cid:80) m
Revised: To find the dual variable, we enforce the equality constraint, i.e., (cid:80) m

Original: ( ω m − τ ) + = 1 .
Revised: ( ω m − τ ) + = 1 .

Original: This non-linear equation of a single variable can be solved very efﬁciently using the Newton method.
Revised: This non-linear equation of a single variable can be solved very efficiently using the Newton method.

Original: This suggests a projected gradient descent solver for solving Problem
Revised: This suggests a projected gradient descent solver for solving Problem

Original: P A and recovering the unknown preferences.
Revised: P A and recovering the unknown preferences.

Original: Inputs: Initial parameters θ 0 , learning rate α 1 foreach k = 0 , 1 , . . .
Revised: Inputs: Initial parameters θ 0 , learning rate α 1 foreach k = 0 , 1 , . . .

Original: do
Revised: do

Original: Collect trajectories D
Revised: Collect trajectories D

Original: = { τ i } , τ i ∼ π θ k
Revised: = { τ i } , τ i ∼ π θ k

Original: Compute rewards-to-go r t g m ← 1 |D| (cid:80) τ,t r t ∇ θ log π θ k
Revised: Compute rewards-to-go r t g m ← 1 |D| (cid:80) τ,t r t ∇ θ log π θ k

Original: ( a t | s t ) ( ω 1 , . . .
Revised: ( a t | s t ) ( ω 1 , . . .

Original: ω M )
Revised: ω M )

Original: ←− PGDSolver ( g 1 , . . .
Revised: ←− PGDSolver ( g 1 , . . .

Original: g M , α ) θ k +1 ← θ k + (cid:80) Mm =1 ω m g m 7 end 9 procedure PGDSolver( g 1 , . .
Revised: g M , α ) θ k +1 ← θ k + (cid:80) Mm =1 ω m g m 7 end 9 procedure PGDSolver( g 1 , . .

Original: , g M , α ) :
Revised: , g M , α ) :

Original: Initialize feasible ω = ( ω 1 , . . .
Revised: Initialize feasible ω = ( ω 1 , . . .

Original: ω M ) while not converged do
Revised: ω M ) while not converged do

Original: Find τ solving (cid:80) m
Revised: Find τ solving (cid:80) m

Original: ( ω m − τ ) + = ω m ← ω m + αg Tm (cid:80) n e n g n , m = 1 . . .
Revised: ( ω m − τ ) + = ω m ← ω m + αg Tm (cid:80) n e n g n , m = 1 . . .

Original: M ω ← ( ω − τ )
Revised: M ω ← ( ω − τ )

Original: + end return ( ω 1 , . . .
Revised: + end return ( ω 1 , . . .

Original: ω M )
Revised: ω M )

Original: Our method (Algorithm 1) extends the classical REINFORCE algorithm (43) to take gradient steps in the direction given by the weighted combination of the multi-objective gradient, with the weights being the preferences recovered by the projected gradient descent solver.
Revised: Our method (Algorithm 1) extends the classical REINFORCE algorithm (44) to take gradient steps in the direction given by the weighted combination of the multi-objective gradient, with the weights being the preferences recovered by the projected gradient descent solver.

Original: Lemma 1, Theorem 1 and 2 essentially state that Algorithm 1 learns arbitrary policies in the Convex Coverage Set along with their underlying preference vectors and provide theoretical justiﬁcation for our method.
Revised: Lemma 1, Theorem 1 and 2 essentially state that Algorithmlearns arbitrary policies in the Convex Coverage Set along with their underlying preference vectors and provide theoretical justification for our method.

Original: Algorithmis simple to implement, compatible with all policy gradient and actorcritic methods and provides a practical algorithm for learning Pareto optimal policies under unknown preferences.
Revised: Algorithm 1 is simple to implement, compatible with all policy gradient and actor-critic methods and provides a practical algorithm for learning Pareto optimal policies under unknown preferences.

Original: However, it suffers from a subtle drawback: we have no control over the identiﬁed preferences and they cannot be optimized to account for any given preference distribution.
Revised: However, it suffers from a subtle drawback: we have no control over the identified preferences and they cannot be optimized to account for any given preference distribution.

Original: This is the problem of policy adaptation which we address in the next section.
Revised: This is the problem of policy adaptation which we address in the next section.

Original: It is worth comparing our projected gradient descent solver with (35), where the authors used the
Revised: It is worth comparing our projected gradient descent solver with (36), where the authors used the

Original: Frank-Wolfe method for Problem P A , albeit in the context of supervised learning.
Revised: Frank-Wolfe method for Problem P A , albeit in the context of supervised learning.

Original: Both algorithms have the same convergence rate of O (1 /N ) and since the projection to the constraint set can be easily calculated, we opted for the projected gradient descent optimizer.
Revised: Both algorithms have the same convergence rate of O (1 /N ) and since the projection to the constraint set can be easily calculated, we opted for the projected gradient descent optimizer.

Original: The overhead added by solving for the dual variable τ is minimal because the Newton’s method is known to enjoy quadratic convergence rate and is very efﬁcient in practice.
Revised: The overhead added by solving for the dual variable τ is minimal because the Newton’s method is known to enjoy quadratic convergence rate and is very efficient in practice.

Original: The trade-off of using Frank-Wolfe is thatit requires to obtain the feasible minimizer of the linear approximation of the objective around the current iterant, which is typically solved approximately and worsens the convergence rate by a multiplicative constant (44).
Revised: The trade-off of using Frank-Wolfe is that requires to obtain the feasible minimizer of the linear approximation of the objective around the current iterant, which is typically solved approximately and worsens the convergence rate by a multiplicative constant (45).

Original: Nonetheless, we expect that both methods can be used interchangeably in practice.
Revised: Nonetheless, we expect that both methods can be used interchangeably in practice.

Original:  P ARETO P OLICY A DAPTATION
Revised:  P ARETO P OLICY A DAPTATION

Original: The policy gradient algorithm presented in the previous section ﬁnds policies that are optimal (i.e., Pareto stationary) for the inferred preferences.
Revised: The policy gradient algorithm presented in the previous section finds policies that are optimal (i.e., Pareto stationary) for the inferred preferences.

Original: In this section, we show how to address the problem of adapting the policy to be optimal for a given distribution D ω over preferences.
Revised: In this section, we show how to address the problem of adapting the policy to be optimal for a given distribution D ω over preferences.

Original: We start by interpreting the optimal solution of Problem P A as a function of the policy parameters ω ∗ := ω ∗ ( θ ) = 
Revised: We start by expressing the optimal solution of Problem P A as a function of the policy parameters ω ∗ := [Equation] = [Equation]

Original: ( ω 1 ( θ ) , ω 2 ( θ ) , . . .
Revised: [Equation] ) , ω [Equation] [Equation]

Original: ω M ( θ )) .
Revised: ω M ( θ )) .

Original: We introduce the Pareto Policy Adaptation( PPA ) 1 loss function as follows:
Revised: We introduce the Pareto Policy Adaptation loss function as follows:

Original: [Equation]
Revised: [Equation]

Original: where λ ∈ [0 , 1] is a weight that trades off between the two losses.
Revised: where λ ∈ [0 , 1] is a weight that trades off between the two losses.

Original: The ﬁrst term of PPA is the scalarized reward, with the scalarization coefﬁcient being the inferred preferences.
Revised: The first term of PPA is the scalarized reward, with the scalarization coefficient being the inferred preferences.

Original: The second term penalizes the expected deviation between the given and inferred preferences.
Revised: The second term penalizes the expected deviation between the given and inferred preferences.

Original: To further motivate the PPA loss, consider the special case of a single, given preference vector ω 0 .
Revised: To further motivate the PPA loss, consider the special case of a single, given preference vector ω 0 .

Original: In this case, the MOMDP collapses to a standard MDP, with ω 0 being the scalarization coefﬁcient for the reward [-ω ∗ v πv π(a) 0 .0 .0 .
-]
Revised: In this case, the MOMDP collapses to a standard MDP, with ω 0 being the scalarization coefficient for the reward [-ω ∗ v πv π(a) 0 .0 .0 .
-]and value functions.

Original: [-ωω ∗ ( θ ) v πv π(b) 0 .0 .0 .
-]
Revised: [-ωω ∗ ( θ ) v πv π(b) 0 .0 .0 .
-]

Original: ωω ∗ ( θ ) ωv πv π(c) and value functions.

Original: Additionally, assume that the policy π θ has been adapted to that preference vector, i e., it achieves ω ∗ ( θ )
Revised: Additionally, assume that the policy π θ has been adapted to that preference vector, i.e., it achieves ω ∗ ( θ )

Original: = ω 0 for some parameter vector θ .
Revised: = ω 0 for some parameter vector θ .

Original: Then, our loss function in Eq.reduces to
Revised: Then, Eq. 6 reduces to

Original: [Equation]
Revised: L ( θ ) =

Original:  Maximization of Eq. 7 corresponds to taking ascent steps along (cid:80) m ω 0 m ∇ θ E d πθ [ r m ( s, a )] , which, per Theorem 2, is an ascent direction guaranteed to increase all objectives.
Revised: E d πθ (cid:2) (cid:80) Mm =1 ω 0 m r m ( s, a ) (cid:3) . Maximization of this loss corresponds to taking ascent steps along (cid:80) m ω 0 m ∇ θ E d πθ [ r m ( s, a )] , which, per Theorem 2, is an ascent direction guaranteed to increase all objectives.

Original: This simpliﬁed scenario essentially states the if we exactly match the preferences learned from Problem P A to the given preference vector ω 0 , then not only the PPA loss function reduces to the loss function for the scalarized reward, but also the ascent steps along its gradient lead to concurrent improvements of all objectives.{+ω ∗ v πv π(a) 0 .0 .0 .
 ωω ∗ ( θ ) v πv π(b) 0 .0 .0 .
 ωω ∗ ( θ ) ωv πv π(c)+}
Revised: This simplified scenario essentially states the if we exactly match the preferences learned from Problem P A to the given preference vector ω 0 , then not only the PPA loss function reduces to the loss function for the scalarized reward, but also the ascent steps along its {+ω ∗ v πv π(a) 0 .0 .0 .
 ωω ∗ ( θ ) v πv π(b) 0 .0 .0 .
 ωω ∗ ( θ ) ωv πv π(c)+}

Revised: gradient lead to concurrent improvements of all objectives.

Original: We show a geometric illustration of the PPA loss function in Fig.
Revised: We show a geometric illustration of the PPA loss function in Fig.

Original: Despite its simplicity, one important obstacle in using the PPA loss to train agents in practice is that the optimal solution to the Problem P A is analytically known only for M = 2 (solution is presented in the Appendix).
Revised: Despite its simplicity, one important obstacle in using the PPA loss to train agents in practice is that the optimal solution to the Problem P A is analytically known only for M = 2 (solution is presented in the Appendix).

Original: For M > 2 , it can rather only be approximated by the iterative projected gradient descent solver introduced in Algorithm 1.
Revised: For M > 2 , it can rather only be approximated by the iterative projected gradient descent solver introduced in Algorithm 1.

Original: This makes the backward pass challenging because the gradients need to be back-propagated through the (perhaps thousands) iterations of the projected gradient descent solver.
Revised: This makes the backward pass challenging because the gradients need to be back-propagated through the (perhaps thousands) iterations of the projected gradient descent solver.

Original: Since back-propagation is known to be a very expensive operation compared to the forward pass, this could make our approach inefﬁcient for practical problems.
Revised: Since back-propagation is known to be a very expensive operation compared to the forward pass, this could make our approach inefficient for practical problems.

Original: The problem of back-propagating through solutions of numerical solvers arises in several deep learning architectures such as deep equilibrium models (45) and optimization-based neural network layers (46).
Revised: The problem of back-propagating through solutions of numerical solvers arises in several deep learning architectures such as deep equilibrium models (46) and optimization-based neural network layers (47).

Original: Luckily, the implicit value theorem allows us to back-propagate the gradient bypassing the operations of the projected gradient descent solver.
Revised: Luckily, the implicit value theorem allows us to back-propagate the gradient bypassing the operations of the projected gradient descent solver.

Original: Theorem 3.
Revised: Theorem 3.

Original: Let l := l ( ω ∗ ( θ )) be a generic loss function that depends explicitly on the the optimal solution ω ∗ ( θ ) of Problem P A .
Revised: Let l := l ( ω ∗ ( θ )) be a generic loss function that depends explicitly on the the optimal solution ω ∗ ( θ ) of Problem P A .

Original: Then, the gradient of l with respect to the policy parameters is
Revised: Then, the gradient of l with respect to the policy parameters is

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  where [ G ( θ )] m,n = 2 g Tm ( θ ) g n ( θ ) , [ H ( θ )] m, · = − (cid:80) n ω ∗ n ( θ ) ∇ Tθ ( g Tn ( θ ) g m ( θ )) ,  µ ∗ ( θ ) is the optimal dual variable, e and 0 are the identity and zero vectors of appropriate dimensions, I M is the identity matrix of dimension M and D ( · ) create a diagonal matrix from a vector.
Revised: [Equation] where [ G ( θ )] m,n = 2 g Tm ( θ ) g n ( θ ) , [ H ( θ )] m, · = − (cid:80) n ω ∗ n ( θ ) ∇ Tθ ( g Tn ( θ ) g m ( θ )) , g m ( θ ) = ∇ θ E d π θ [ r m ( s, a )] , µ ∗ ( θ ) is the optimal dual variable, e and 0 are the identity and zero vectors of appropriate dimensions, I M is the identity matrix of dimension M and D ( · ) create a diagonal matrix from a vector.

Original: When applied on a vector quantity, the operator ∇ θ denotes the Jacobian matrix and the operator ∇ Tθ denotes the transpose of the underlying gradient or Jacobian.
Revised: When applied on a vector quantity, the operator ∇ θ denotes the Jacobian matrix and the operator ∇ Tθ denotes the transpose of the underlying gradient or Jacobian.

Original: The proof of Theorem 3 (given in the Appendix) leverages the implicit value theorem which enables us to differentiate the KKT optimality conditions of Problem P A with respect to the policy parameters.
Revised: The proof of Theorem 3 (given in the Appendix) leverages the implicit value theorem which enables us to differentiate the KKT optimality conditions of Problem P A with respect to the policy parameters.

Original: Observe that matrix H ( θ ) is essentially of dimension M ×
Revised: Observe that matrix H ( θ ) is essentially of dimension M ×

Original: d θ , d θ being the dimension of θ .
Revised: d θ , d θ being the dimension of θ .

Original: This dependence on d θ makes the solution of eq:nmatrix inefﬁcient, as we need to compute and store a matrix of size (2 M + 1) d θ .
Revised: This dependence on d θ makes the solution of eq:nmatrix inefficient, as we need to compute and store a matrix of size (2 M + 1) d θ .

Original: However, in practice, we are not interested in knowing the full gradient of ω ∗ ( θ ) , but rather in back-propagating the gradient of the loss function; Eq. 8 has an appealing form which allows us to use vector-Jacobian products (VJPs) to evaluate the gradient ∇ θ l one row at a time.
Revised: However, in practice, we are not interested in knowing the full gradient of ω ∗ ( θ ) , but rather in back-propagating the gradient of the loss function; Eq. 7 has an appealing form which allows us to use vector-Jacobian products (VJPs) to evaluate the gradient ∇ θ l one row at a time.

Original: This frees us from having to store matrix H ( θ ) and compute its product with another matrix.
Revised: This frees us from having to store matrix H ( θ ) and compute its product with another matrix.

Original: We observe that the solution in Eq.
Revised: We observe that the solution in Eq.

Original: 8 requires knowledge of the optimal dual variable µ ∗ ( θ ) , which is not directly available.
Revised: 7 requires knowledge of the optimal dual variable µ ∗ ( θ ) , which is not directly available.

Original: However, the optimal primal solution ω ∗ ( θ ) given by the projected gradient descent solver (Algorithm 1) can be utilized to obtain µ ∗ ( θ ) by substituting it into the KKT optimality conditions of Problem P A (details in the Appendix) and solving the resulting linear system.
Revised: However, the optimal primal solution ω ∗ ( θ ) given by the projected gradient descent solver (Algorithm 1) can be utilized to obtain µ ∗ ( θ ) by substituting it into the KKT optimality conditions of Problem P A (details in the Appendix) and solving the resulting linear system.

Original: That system has M + 1 variables, therefore its solution adds minimal computational overhead.
Revised: That system has M + 1 variables, therefore its solution adds minimal computational overhead.

Original: E XPERIMENTAL E VALUATION
Revised: It is worth discussing similarities and differences between PPA and EQL (24). PPA retrieves and adapts to preferences during the training phase, whereas EQL first learns a preference-dependant policy and then retrieves preferences using a combination of stochastic search and policy gradient. Also, the loss functions are partially similar: the penalty term in PPA is the MSE loss between the given and inferred preferences, whereas the EQL loss function is the MSE loss of vector Q-values. Additionally, given that PPA is a policy method and EQL is a value method, they can be combined in an actor-critic algorithm.

Revised: This is a very interesting direction for future work.

Revised: One limitation of our work is that it relies on Pareto stationarity, which is a necessary but not sufficient condition for Pareto optimality.

Revised: i.e., there is no guarantee for Pareto optimality only stationarity.

Revised: Nonetheless, Pareto optimality is hard to be achieved and we often need to resort to first-order approximations.

Revised: Finally, the optimal policy obtained under a given distribution is not, in general, optimal under a different distribution and the model has to be fine-tuned.

Revised: We expect that we will not need to re-train the network entirely and that we can leverage the previous optimal policy to find the new one.

Revised: The technical details of how this would be achieved are a direction for future work.

Revised:  E XPERIMENTAL E VALUATION

Original: In this section, we evaluate the performance of our proposed method.
Revised: In this section, we evaluate the performance of our proposed method.

Original: In our implementation, we extended the popular Proximal Policy Optimization (PPO) (47) algorithm so that it can handle
Revised: In our implementation, we extended the popular Proximal Policy Optimization (PPO) (48) algorithm so that it can handle

Original: our
Revised: our

Original: PPA loss function.
Revised: PPA loss function.

Original: We implemented all experiments on PyTorch using modiﬁed versions of OpenAI
Revised: We implemented all experiments on PyTorch using modified versions of OpenAI

Original: Gym environments.
Revised: Gym environments.

Original: We give the details of our experimental setup in the Appendix.
Revised: We give the details of our experimental setup in the Appendix.

Original: Domains : We evaluate on 4 environments (details given in the Appendix): (a) Multi-Objective
Revised: Domains : We evaluate on 4 environments (details given in the Appendix): (a) Multi-Objective

Original: GridWorld (MOWG): a variant of the classical gridworld, (b) Deep Sea Treasure (DST): the classical multi-objective reinforcement learning enviroment, (c) Mulit-Objective SuperMario (MOSM): modiﬁed, multi-objective variant of the popular video game that has a 5-dimensional reward signal and (d) Multi-Objective MuJoCo (MOMU): a modiﬁed version of the MuJoCo physics simulator, focusing on locomotion tasks.
Revised: GridWorld (MOWG): a variant of the classical gridworld, (b) Deep Sea Treasure (DST): a slightly modified version of classical multi-objective reinforcement learning enviroment (49), (c) MulitObjective SuperMario (MOSM): modified, multi-objective variant of the popular video game that has a 5-dimensional reward signal and (d) Multi-Objective MuJoCo (MOMU): a modified version of the MuJoCo physics simulator, focusing on locomotion tasks.

Original: The ﬁrst two are simpliﬁed control tasks where the Pareto frontier is easy to identify (see example in Fig. 1) and act as proof-of-concept for our method.

Original: With the remaining two environments we aim to show that our method is effective for handling vector reward signals in real-world problems.

Original: Ablation Study and Baselines : To highlight the extent to which our results are driven by the PPA loss function, we perform an ablation study.
Revised: Ablation Study and Baselines : To highlight the extent to which our results are driven by the PPA loss function, we perform an ablation study.

Original: In more details, we consider the following three agents: a Pareto Policy Adaptation ( PPA ):
Revised: In more details, we consider the following three agents: (a) Pareto Policy Adaptation ( PPA ):

Original: This is our original loss function as introduced in Sec. 4.
Revised: This is our original loss function as introduced in Sec. 4.

Original: (b) non- adaptive PPA (na-PPA):
Revised: (b) non-adaptive PPA (na-PPA):

Original: Same as PPA but the preferences are static, i.e., the policy is not adapted to the preference distribution.
Revised: Same as PPA but the preferences are static, i.e., the policy is not adapted to the preference distribution.

Original: This agent is essentially Algorithm 1 without the PPA loss function.
Revised: This agent is essentially Algorithm 1 without the PPA loss function.

Original: c ) Fixed Preferences:
Revised: (c) Fixed Preferences:

Original: Same as na-PPA but the preferences are given.
Revised: Same as na-PPA but the preferences are given.

Original: This corresponds to optimizing the scalarized objective.
Revised: This corresponds to optimizing the scalarized objective.

Original: Additionally, we compare the performance of our method with a range of state-of-the-art baselines.
Revised: Additionally, we compare the performance of our method against a range of state-of-the-art baselines.

Original: In particular, we compare against: the Multi-Objective Fitted Q-iteration (MOFTQI) (30), the Conditional Neural network with Optimistic Linear Support (CN-OLS)(22; 23), the Conditioned Network with Diverse Experience Replay (CN-DER) (23), the Envelope Q-Learning (EQL) (24).
Revised: In particular, we compare against: the Pareto Following Algorithm (PFA) (31), the Conditional Neural network with Optimistic Linear Support (CNOLS)(22; 23), the Conditioned Network with Diverse Experience Replay (CN-DER) (23), the Envelope Q-Learning (EQL) (24) and the Prediction-Guided Multi-Objective Reinforcement Learning (PGMORL) (34).

Original: Evaluation Metrics : To evaluate the performance of our method and compare against the baselines, we use the following metrics:
Revised: Evaluation Metrics : To evaluate the performance of our method and compare against the baselines, we use the following metrics:

Revised: HyperVolume ( HV ): Let F be a reward space Pareto front approximation and r 0 be a reference vector.

Revised: Then, we define the HyperVolume as HV :

Revised: = (cid:82) 1 H ( P ) ( z ) dz , where H ( P ) = { z ∈ R M : ∃ r ∈ F : r 0 ≺ P z ≺ P r } and 1 ( · ) is the indicator function.

Original: Pareto Dominance ( PD ): Let r 1 t , r 2 t be the reward vectors of trajectories sampled from agent 1 and 2, respectively.
Revised: Pareto Dominance ( PD ): Let r 1 t , r 2 t be the reward vectors of trajectories sampled from agent 1 and 2, respectively.

Original: We deﬁne the Pareto Dominance metric asPD 1 , 2 :=
Revised: We define the Pareto Dominance metric as

Original:  1 T (cid:80) Tt =0 1 ( r 1 t (cid:31) P r 2 t ) , where 1 ( · ) is the indicator function.
Revised: PD 1 , 2 := 1 T (cid:80) T t =0 1 ( r 1 t ≻ P r 2 t ) , where 1 ( · ) is the indicator function.

Original: This is essentially the fraction of time when the return vector under agent 1 Pareto dominates the return vector under agent 2.
Revised: This is essentially the fraction of time when the reward vector under agent 1 Pareto dominates the reward vector under agent 2.

Original: Note that PD 1 , 2 + PD 2 , 1 ≤ 1 . 2.
Revised: Note that PD 1 , 2 + PD 2 , 1 ≤ 1 . 3.

Original: Utility ( UT ): For a reward vector r t sampled from any algorithm, we deﬁne the Utility metric as UT :=  1 T (cid:80) Tt [-=
-]
Revised: Utility ( UT ): For a reward vector r t sampled from any agent, we define the Utility metric as UT := E ω [ 1 T (cid:80) Tt [-=
-]=

Original: E ω [ ω T r t ] , where the expected value is evaluated by sampling uniformly from the preference space Ω or from the preference distribution D ω .
Revised: 0 ω T r t ] , where the expected value is evaluated by sampling uniformly from the preference space Ω or from the preference distribution D ω .

Original: The PD metric is useful for comparing agents pair-wise and is simple to calculate. On the other hand, UT can compare multiple agents at the expense of computing an expectation over preferences, which may become challenging for higher-dimensional rewards.
Revised: [Figure]

Original: Main Results : Figure 3 shows the results of the ablation study conducted on the MOGW and DST domains.
Revised: Main Results : Figure 3 shows the results of the ablation study conducted on the MOGW and DST domains.

Original: We compare against the non-adaptive and the ﬁxed-preferences agents.
Revised: We compare against the non-adaptive and the fixedpreferences agents.

Original: For the later case, we pick three different preference vectors: ( ω 1 , ω 2 ) =
Revised: For the later case, we pick three different preference vectors: ( ω 1 , ω 2 ) =

Original: (0 . 5 , 0 . 5) , ( ω 1 , ω 2 ) =
Revised: (0 . 5 , 0 . 5) , ( ω 1 , ω 2 ) =

Original: 75 , 0 . 25) and ( ω 1 , ω 2 ) =
Revised: 75 , 0 . 25) and ( ω 1 , ω 2 ) =

Original: These plots facilitate the visual identiﬁcation of Pareto dominant agents.
Revised: These plots facilitate the visual identification of Pareto dominant agents.

Original: Focusing on the MOGW (upper row), we observe that our PPA agent converges to a policy that yields a higher reward and lower time-penalty than any other agent, i..
Revised: Focusing on the MOGW (upper row), we observe that our PPA agent converges to a policy that yields a higher reward and lower timepenalty than any other agent, i..

Original: e, PPA Pareto dominates all other agents.
Revised: e, PPA Pareto dominates all other agents.

Original: While a similar statement cannot be made for the DST domain, we note that PPA is not Pareto dominated by any other agent and the only agent that it does not dominate is the ﬁxedpreference one for ( ω 1 , ω 2 ) =
Revised: While a similar statement cannot be made for the DST domain, we note that PPA is not Pareto dominated by any other agent and the only agent that it does not dominate is the fixed-preference one for ( ω 1 , ω 2 ) =

Original: On the other hand, the non-adaptive PPA ( na-PPA ) seems to perform relatively poor.
Revised: On the other hand, the non-adaptive PPA ( na-PPA ) seems to perform relatively poor.

Original: However, there exists some merit in using it because, in contrast to the ﬁxed-preference agents, na-PPA does not rely on being given a preference vector.
Revised: However, there exists some merit in using it because, in contrast to the fixed-preference agents, na-PPA does not rely on being given a preference vector.

Original: It is also worth noting that in the DST domain the preference vector ( ω 1 , ω 2 ) =
Revised: It is also worth noting that in the DST domain the preference vector ( ω 1 , ω 2 ) =

Original: (0 . 25 , 0 . 75) gives very poor results, which implies that handpicking the preferences might lead to performance collapse.
Revised: (0 . 25 , 0 . 75) gives very poor results, which implies that handpicking the preferences might lead to performance collapse.

Original: Overall, these results suggest that PPA steadily achieves better performance and is superior than the other agents.
Revised: Overall, these results suggest that PPA steadily achieves better performance and is superior than the other agents.

Original: It is interesting to observe that the reward and time penalty curves are quite similar in the MOWG domain.
Revised: It is interesting to observe that the reward and time penalty curves are quite similar in the MOWG domain.

Original: This agrees with our intuition because in the MOGW domain the reward and time penalty have an ”inverse” relationship (i.e., the sooner we reach the bottom right, the higher the reward).[Table] [Figure] t i m e p e n a l t y
Revised: This agrees with our intuition because in the MOGW domain the reward and time penalty have an ”inverse” relationship (i.e., the sooner we reach the bottom right, the higher the reward).

Original: [Figure]
Revised: [Figure]

Original: na [Figure] PPA 1 = [Figure]
Revised: In the next line of experiments

Original: =

Original: [Figure]
Revised: [Figure]

Original: In the next series of experiments, we compare our method with state-of-the-art

Original: MORL algorithms (Table 1).

Original: In this comparison, we use both the Pareto Dominance ( PD ) and Average Utility ( UT ) metrics. Since PD is a pair-wise metric, we ﬁrst compute the pair-wise PD i,j values for all agents and then average-out the j component to obtain a single value for each agent. Note that this metric is speciﬁc to the ensemble of agents that we consider and needs to be re-calculated if we compare a different set of agents. The UT metric is calculated by sampling preferences from the preference space of each domain.
Revised: we evaluate the performance of PPA in continuous control tasks.

Original: We assumed no prior knowledge on the preferences, so we performed uniform sampling of 1000 points.

Original: In the

Original: MOGW and DST cases, where the preference space is one-dimensional (recall that preferences are assumed to lie in the unit simplex), this sampling posses no computational challenge. In the MOSM case, where the preferences are 4-dimensional, we sample 100 preferences along each preference dimension. Additionally, given that the preferences lie in the unit simplex, we scale the rewardsin the range [ − 1 , 1] across each dimension. This scaling is important when calculating the average utility but has no inﬂuence on the
Revised: We modified the MuJoCo Gym interface to output 2-dimensional rewards on each step.

Original: Pareto dominance metric.

Original: Observe that some agents have negative utility, which isindeed possible given the aforementioned reward scaling. Our method systematically outperforms all other baselines across all domains, which is another strong empirical ﬁnding for its superior performance. [Figure] [Figure]
Revised: The first one is the commonly used reward in such environments, i

Original: [Figure]

Original: na

Original: [Figure]
Revised: [Figure]

Original: In the last line of experiments, we evaluate the performance of PPA in continuous control tasks. We modiﬁed the MuJoCo Gym interface to output 2-dimensional rewards on each step. The ﬁrst one is the commonly used reward in such environments, i.e., the difference between two sequential positions of the robot.
Revised: e., the difference between two sequential positions of the robot.

Original: The second one is the negative power consumption of the robot.
Revised: The second one is the energy consumed by the robot.

Original: We show the results on Fig.
Revised: We show the results on Fig.

Original: We compare against the ablated agents and show the Pareto Dominance ( PD ) and 
Revised: We compare against the ablated agents and show the Pareto Dominance ( PD ) and Average

Original: Average Utility ( UT ) metrics.
Revised: [Table] Utility ( UT ) metrics.

Original: The only case where our method does not attain the best performance is the Hopper-v3 environment under the UT metric, where the performance of the ﬁxed-preferences agent ( ω 1 , ω 2 ) =
Revised: The only case where our method does not attain the best performance is the Hopper-v3 environment under the UT metric, where the performance of the fixed-preferences agent ( ω 1 , ω 2 ) =

Original: 25) slightly surpasses the performance of PPA.
Revised: 25) slightly surpasses the performance of PPA.

Original: However, observe that the same agent attains very poor performance in the HalfCheetah-v3 enviroment.
Revised: However, observe that the same agent attains very poor performance in the HalfCheetah-v3 enviroment.

Original: This suggests that, given the complex inter-dependence between the two rewards, simply hand-crafting a preference vector is far from optimal and a preference vector that yields good performance in one environment may perform poorly in a different environment or judged under a different metric.
Revised: This suggests that, given the complex inter-dependence between the two rewards, simply hand-crafting a preference vector is far from optimal and a preference vector that yields good performance in one environment may perform poorly in a different environment or judged under a different metric [Table] In the last line of experiments [Table]

Original: However, PPA shows a consistently better performance across all environments and under both metrics.
Revised: we compare our method against state-of-the-art MORL algorithms (Table 1). Since PD is a pair-wise metric, we first compute the pair-wise PD i,j values for all agents and then average-out the j component to obtain a single value for each agent. Note that this metric is specific to the ensemble of agents that we consider and needs to be re-calculated if we compare a different set of agents. The UT metric is calculated by sampling preferences from the preference space of each domain.

Revised: We assume assume a uniform preference distribution and sample 1000 points for the 2-dimensional preference case (MOGW, DST, MUMO) and 100 across each dimension for the MOSM case, where the preference space has 4 dimensions.

Revised: Additionally, we scale the rewards in the range [ − 1 , 1] across each dimension.

Original: S UMMARY AND O UTLOOK
Revised: This scaling is important when calculating the average utility but has no particular influence on the Pareto dominance or hypervolume. Observe that some agents have negative utility, which is indeed possible given the aforementioned reward scaling. Our method systematically outperforms all other baselines across all domains, which is another strong empirical finding for its performance.  C ONCLUSION

Original: We introduced a policy gradient algorithm for Multi-Objective Reinforcement Learning (MORL) under linear preferences.
Revised: We introduced a policy gradient algorithm for Multi-Objective Reinforcement Learning (MORL) under linear preferences.

Original: By approximating the Pareto front via a ﬁrst-order necessary condition, our method learns a single policy that can be adapted to any preference distribution.
Revised: By approximating the Pareto front via a first-order necessary condition, our method learns a single policy that can be adapted to any preference distribution.

Original: The two pillars of our method is a projected gradient descent solver that searches for common ascent direction for all objectives and a novel Pareto Policy Adaptation ( PPA ) loss function that leverages the solutions of that solver to adapt the policy to any preference distribution.
Revised: The two pillars of our method is a projected gradient descent solver that searches for common ascent direction for all objectives and a novel Pareto Policy Adaptation ( PPA ) loss function that leverages the solutions of that solver to adapt the policy to any preference distribution.

Original: Our method enjoys the advantages of both single and multiple policy MORL algorithms while being simple to implement and compatible with all policy gradient and actor-critic methods.
Revised: Our method enjoys the advantages of both single and multiple policy MORL algorithms while being simple to implement and compatible with all policy gradient and actor-critic methods.

Original: We demonstrated the effectiveness of our method in several reinforcement learning tasks.
Revised: We demonstrated the effectiveness of our method in several reinforcement learning tasks.

Original: One limitation of our current work is the linear preference assumption, which may be over-simplifying for some problems.

Original: In the case of non-linear preferences, our method can still be applied but should be seen as a heuristic.

Original: Furthermore, the development of our method relies on Pareto stationarity, which is a necessary but not sufﬁcient condition for Pareto optimality.

Original: i.e., there is no guarantee for Pareto optimality only stationarity.

Original: Nonetheless, Pareto optimality is hard to be achieved in general and we often need to resort to ﬁrst-order approximations.

Original: A A PPENDIX A T HEORETICAL F RAMEWORK
Revised: A A PPENDIX A T HEORETICAL F RAMEWORK

Original: In the ﬁrst part of the Appendix, we give the proof omitted in Section 4 as well as the analytical solution of Problem P A for the special case of 2-dimensional reward signals.
Revised: In the first part of the Appendix, we give the proofs omitted in Sections 2 and 4 as well as the analytical solution of Problem P A for the special case of 2-dimensional reward signals.

Original: For the ease of reference, we re-state the underlying theorem.
Revised: For the ease of reference, we re-state the underlying theorem.

Original: A. P ROOF
Revised: A. P ROOFS

Original: Theorem 3.
Revised: Lemma 1. Let π ∈ C ∗ Π , Ω be a policy. Then, we have E π [ ˜ r ] ∈ C ∗ R , Ω , i.e., the expected return vector under policy π belongs in the RS-CCS. Proof.

Revised: Consider a policy π ∈ C ∗ Π , Ω and a policy π ′ ∈ Π such that d π ′ ( s, a ) = 0 wherever d π ( s, a ) = 0 .

Revised: Then, we can write

Revised: [Equation]

Revised: where we used the importance sampling identity and defined r ′ ( s, a ) :

Revised: = d π ′ ( s, a ) /d

Revised: π ( s, a ) r ( s, a ) .

Revised: Per Def. 2, there exists ω ∈ Ω such that ω T v π ≥ ω T v π ′ .

Revised: Leveraging the fact that the value function can be expressed as v π =

Revised: E π

Revised: [ ˜ r ] as well as Eq.

Original: Let l := l ( ω ∗ ( θ )) be a generic loss function that depends explicitly on the the optimal solution ω ∗ ( θ ) of Problem P A . 
Revised: equation 8, we have ω T E π [ ˜ r ] ≥ ω T E π [ ˜ r ′ ] . This implies that E π [ ˜ r ] ∈ C ∗ R , Ω and concludes the proof. Remark 1.

Original: Then, the gradient of l with respect to the policy parameters is
Revised: Under the additional assumption of normalized rewards, we can prove the converse of

Original: [Equation]
Revised: Lemma 1.

Original: e
Revised: However, since we are are concerned with approximating C ∗ R , Ω by C ∗ Π , Ω (and not the other way around), we refrain from making that additional assumption to prove the converse of Lemma 1.

Original: [Equation]
Revised: Theorem 3.

Original:   [Equation] T   dld ω ∗ 
Revised: Let l := l ( ω ∗ ( θ )) be a generic loss function that depends explicitly on the the optimal solution ω ∗ ( θ ) of Problem P A .

Original:  ,
Revised: Then, the gradient of l with respect to the policy parameters is

Original: [Equation]
Revised: [Equation]

Original: 9) where [ G ( θ )] m,n = 2 g Tm ( θ ) g n ( θ ) , [ H ( θ )] m, · = − (cid:80) n ω ∗ n ( θ ) ∇ T θ ( g T n ( θ ) g m ( θ )) , e and 0 are the identity and zero vectors of appropriate dimensions, I M is the identity matrix of dimension M and
Revised: where [ G ( θ )] m,n = 2 g Tm ( θ ) g n ( θ ) , [ H ( θ )] m, · = − (cid:80) n ω ∗ n ( θ ) ∇ Tθ ( g Tn ( θ ) g m ( θ )) , e and 0 are the identity and zero vectors of appropriate dimensions, I M is the identity matrix of dimension M and

Original: D ( · ) create a diagonal matrix from a vector.
Revised: D ( · ) create a diagonal matrix from a vector.

Original: Proof.
Revised: Proof.

Original: We start by writing the Lagrangian of Problem P A
Revised: We start by writing the Lagrangian of Problem P A

Original: [Equation]
Revised: [Equation]

Original: where g m ( θ ) = ∇ θ E d πθ [ r m ( s, a )] .
Revised: where g m ( θ ) = ∇ θ E d πθ [ r m ( s, a )] .

Original: Let ( ω ∗ ( θ ) , λ ∗ ( θ ) , µ ∗ ( θ )) be the policy-dependant optimal primal and dual variables.
Revised: Let ( ω ∗ ( θ ) , λ ∗ ( θ ) , µ ∗ ( θ )) be the policy-dependant optimal primal and dual variables.

Original: The KKT conditions for stationarity, primal feasibility and complementary slackness are 
Revised: The KKT conditions for stationarity, primal feasibility and complementary slackness are ∂L

Original: [Equation]
Revised: [Equation]

Original: By taking the gradient of the above equations with respect to the policy parameters [Equation] we obtain
Revised: By taking the gradient of the above equations with respect to the policy parameters, we obtain

Original: M (cid:88)
Revised: M (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Using matrix notation [Equation] we can rewrite the above system of equations as follows:[Equation]
Revised: Using matrix notation, we can rewrite the above system of equations as follows:

Original: [Equation]
Revised: [Equation]

Original: By inverting Eq.
Revised: By inverting Eq.

Original: equation 13 and using the chain rule ∇ θ l ( ω ( θ ))
Revised: equation 13 and using the chain rule ∇ θ l ( ω ( θ ))

Original: =
Revised: =

Original: ∂l∂ω ∇ θ l we obtain the claimed result.
Revised: ∂l∂ω ∇ θ l we obtain the claimed result.

Original: [Figure]

Original: We observe that the solution of Eq.
Revised: We observe that the solution of Eq.

Original: equation 13 requires knowledge of the optimal dual variable µ ∗ ( θ ) , which is not directly available when we use the projected gradient descent solver of Algorithm 1; that solver only gives us the optimal primal variable ω ∗ ( θ ) .
Revised: equation 13 requires knowledge of the optimal dual variable µ ∗ ( θ ) , which is not directly available when we use the projected gradient descent solver of Algorithm 1; that solver only gives us the optimal primal variable ω ∗ ( θ ) .

Original: However, we can substitute that optimal ω ∗ ( θ ) into the KKT conditions given by Eq.
Revised: However, we can substitute that optimal ω ∗ ( θ ) into the KKT conditions given by Eq.

Original: (11a–11c)
Revised: (11a–11c)

Original: and solve the resulting linear system of M +1 variables to ﬁnd the optimal dual variables.
Revised: and solve the resulting linear system of M +1 variables to find the optimal dual variables.

Original: The number of variables in that system depends only on the dimension of the reward signal M and not on the dimension of θ , therefore it adds minimal computational overhead.
Revised: The number of variables in that system depends only on the dimension of the reward signal M and not on the dimension of θ , therefore it adds minimal computational overhead.

Original: This step would not be necessary if we chose a primal-dual method for solving Problem P A .
Revised: This step would not be necessary if we chose a primal-dual method for solving Problem P A .

Original: However, primal-dual methods may add to the complexity of the implementation, which may not be worthy given the simplicity of Problem P A .
Revised: However, primal-dual methods may add to the complexity of the implementation, which may not be worthy given the simplicity of Problem P A .

Original: Exploring such a trade-off is an interesting direction for future work.
Revised: Exploring such a trade-off is an interesting direction for future work.

Original: A. P ARETO S TATIONARITY : 2- DIMENSIONAL CASE
Revised: A. P ARETO S TATIONARITY : 2- DIMENSIONAL CASE

Original: In general, the problem of ﬁnding common ascent directions (Problem P A ) can be solved either exactly using the methods from the minimum norm points literature (as discussed in Sec. 3) or approximately using convex optimization methods.
Revised: In general, the problem of finding common ascent directions (Problem P A ) can be solved either exactly using the methods from the minimum norm points literature (as discussed in Sec. 3) or approximately using convex optimization methods.

Original: Due to the dimensionality of the gradient vector, the former class of methods are not applicable in our case.
Revised: Due to the dimensionality of the gradient vector, the former class of methods are not applicable in our case.

Original: Therefore, we resorted to the projected gradient descent solver presented in Algorithm 1.
Revised: Therefore, we resorted to the projected gradient descent solver presented in Algorithm 1.

Original: This solver is essentially an internal optimizer which we run before every parameter update.
Revised: This solver is essentially an internal optimizer which we run before every parameter update.

Original: Luckily, Problem P A is convex and the gradient descent optimizer converges rapidly in practice with little tuning.
Revised: Luckily, Problem P A is convex and the gradient descent optimizer converges rapidly in practice with little tuning.

Original: Additionally, for the special case of two objectives, we can leverage the analytical solution to achieve further performance improvements.
Revised: Additionally, for the special case of two objectives, we can leverage the analytical solution to achieve further performance improvements.

Original: We re-state Problem P A for convenience bellow:
Revised: We re-state Problem P A for convenience bellow:

Original: [Equation]
Revised: [Equation]

Original: ..(cid:15) M (cid:110) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) M (cid:88) m = (cid:15) m

Original: [Equation]

Original: d

Original: [Equation]

Original: [ r

Original: [Equation]

Original: ( s, a )] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  s.t

Original: M (cid:88) m

Original: = (cid:15) m

Original: = 1 , (cid:15) m ≥ 0 , ∀

Original: [Equation]

Original: (cid:111) .

Original: ( P A ) For M = 2 , the problem reduces to min (cid:15) ∈ [0 , 1] (cid:107) (cid:15) ∇ [Equation] E d 
Revised: For M = 2 , the problem reduces to min ϵ ∈ [0 , 1] ∥ ϵ ∇ θ E d πθ [ r 1 ( s, a )]

Original: [Equation]

Original: [Equation]

Original: s, a

Original: [Equation]

Original: ] + (

Original: [Equation] (cid:15) [Equation] θ E d πθ [ r 2 ( s, a )] (cid:107) 22 , which is a one-dimensional quadratic optimization problem of a single variable and has the following analytical solution:
Revised: + (1 − ϵ ) ∇ θ E d πθ [ r 2 ( s, a )] ∥ 22 , which is a one-dimensional quadratic optimization problem of a single variable and has the following analytical solution:

Original: [Equation]
Revised: [Equation]

Original: where [ · ] + represents the clipping operator, i.e., [ x ] + = max (min ( x, 1) , 0) .
Revised: where [ · ] + represents the clipping operator, i.e., [ x ] + = max (min ( x, 1) , 0) .

Original: Geometrically,
Revised: Geometrically,

Original: Eq.
Revised: Eq.

Original: equation 14 implies that minimal-norm point lies either on the boundary on the convex hull ( (cid:15) = 0 or (cid:15) = 1 ) and the desired descent direction is parallel to corresponding gradient or is equal to the vector perpendicular to the difference of the gradients.
Revised: equation 14 implies that minimal-norm point lies either on the boundary on the convex hull ( ϵ = 0 or ϵ = 1 ) and the desired descent direction is parallel to corresponding gradient or is equal to the vector perpendicular to the difference of the gradients.

Original: The former one is a corner case and occurs when the ascent direction for one of the objectives is a non-decreasing direction for the other one.
Revised: The former one is a corner case and occurs when the ascent direction for one of the objectives is a non-decreasing direction for the other one.

Original: While this is only applicable to M = 2 , it enables us to replace the inner projected gradient descent optimizer in Algorithm 1 with the analytical solution in Eq.
Revised: While this is only applicable to M = 2 , it enables us to replace the inner projected gradient descent optimizer in Algorithm 1 with the analytical solution in Eq.

Original: equation 14.
Revised: equation 14.

Original: Additionally, this special case is applicable to several real-world cases since there are several problems that have the reward-cost structure. 
Revised: Additionally, this special case is applicable to several real-world cases since there are several problems that have the reward-cost structure. [Figure] SuperMarioBros Ant-v

Revised: HalfCheetah-v

Revised: Hopper-v

Original:  B E XPERIMENTAL D ETAILS B. E NVIRONMENTS
Revised: Humanoid-v B E XPERIMENTAL D ETAILS B. E NVIRONMENTS

Original: Our implementation uses modiﬁed versions of 4 OpenAI Gym enviroments (Fig. 5).
Revised: Our implementation uses modified versions of 4 OpenAI Gym enviroments (Fig. 5).

Original: The ﬁrst two are synthetic domains and act as a proof-of-concept for our method, while the other two are complex, real domains.
Revised: The first two are synthetic domains and act as a proof-of-concept for our method, while the other two are complex, real domains.

Original: The details are listed below:
Revised: The details are listed below:

Original: Multi-Objective GridWorld (MOGW) :
Revised: Multi-Objective GridWorld (MOGW) :

Original: A multi-objective variant of the classic gridworld environment.
Revised: A multi-objective variant of the classic gridworld environment.

Original: The agent starts from the top-left corner, moves in a 16 × 16 -grid and receives a reward equal to 1 when it reaches the bottom-right corner.
Revised: The agent starts from the top-left corner, moves in a 16 × 16 -grid and receives a reward equal to 1 when it reaches the bottom-right corner.

Original: Each action results in a small time penalty of − 0 .
Revised: Each action results in a small time penalty of − 0 .

Original: Any policy that results in a trajectory that minimizes the taxicab distance between the agent’s starting position and the goal is Pareto optimal.
Revised: Any policy that results in a trajectory that minimizes the taxicab distance between the agent’s starting position and the goal is Pareto optimal.

Original: Our implementation uses the gym-minigridenviroment, which we modify to output the aforementioned 2-dimensional reward.
Revised: Our implementation uses the gym-minigridenviroment, which we modify to output the aforementioned 2-dimensional reward.

Original: Deep Sea Treasure (DST) :
Revised: Deep Sea Treasure (DST) :

Original: A classic multi-objective reinforcement learning environment.
Revised: A classic multi-objective reinforcement learning environment.

Original: The agent is located in a 10 × 11 -grid and controls a submarine searching for treasures.
Revised: The agent is located in a 10 × 11 -grid and controls a submarine searching for treasures.

Original: There are multiple treasures the value of which increases as their distance increases.
Revised: There are multiple treasures the value of which increases as their distance increases.

Original: The agent receives a time penalty of − 0 .
Revised: The agent receives a time penalty of − 0 .

Original: 1 for each action.
Revised: 1 for each action.

Original: The Pareto front of non-dominated policies can be easily extracted 48) and is also shown on Fig.
Revised: The Pareto front of non-dominated policies can be easily extracted (49) and is also shown on Fig.

Original: As in the MOGW case, we use the gym-minigrid package and extend it to the reward map shown in Fig.
Revised: As in the MOGW case, we use the gym-minigrid package and extend it to the reward map shown in Fig.

Original: Multi-Objective SuperMario ( MOSM ): We use a modiﬁed multi objective variant of the popular 
Revised: Multi-Objective SuperMario ( MOSM ): We use a modified, multi-objective variant of the popular video game.

Original: video game.

Original: There are  scalar rewards: x-pos : the difference in Mario’s horizontal position between current and last time point, time : a small 
Revised: There are 5 scalar rewards: x-pos : the difference in Mario’s horizontal position between current and last time point, time : a small time penalty of − 0 .

Original: time

Original: penalty of

Original: −

Original:  , deaths : a large negative penalty of − 100 each time Mario dies
Revised: 1 , deaths : a large negative penalty of − 100 each time Mario dies, coin : rewards for collecting coins, and enemy : rewards for eliminating an enemy.

Original: , coin :

Original: rewards for

Original: collecting

Original: coins and enemy : rewards for

Original: eliminating an enemy.

Original: We use the gym-super-mario-bros 3 package and the multi-objective extension developed in (
Revised: We use the gym-super-mario-bros 2 package and the multi-objective extension developed in (24).

Original: Multi-Objective MuJoCo (MOMU) :
Revised: Multi-Objective MuJoCo (MOMU) :

Original: We focus on locomotion tasks and modify the environments provided by the MuJoCo physics simulator to output vector rewards.
Revised: We focus on locomotion tasks and modify the environments provided by the MuJoCo physics simulator to output vector rewards.

Original: To do so, we create a custom wrapper for the built-in OpenAI Gym interface.
Revised: To do so, we create a custom wrapper for the built-in OpenAI Gym interface.

Original: There are 2 scalar rewards: x-pos : the difference in the robot’s current and last horizontal position, pow : a negative reward for that represents the energy consumption.
Revised: There are 2 scalar rewards: x-pos : the difference in the robot’s current and last horizontal position, r-pos : a positive reward for moving in circular trajectories.

Original: B. I MPLEMENTATION D ETAILS
Revised: B. I MPLEMENTATION D ETAILS

Original: We implement our algorithm in PyTorch building on top of the torch-ac package 4 .
Revised: We implement our algorithm in PyTorch building on top of the torch-ac package 3 .

Original: Our implementation extends the Proximal Policy Optimization (PPO) algorithm to handle our PPA loss function (Eq.
Revised: Our implementation extends the Proximal Policy Optimization (PPO) algorithm to handle our PPA loss function (Eq.

Original: equation 6).
Revised: equation 6).

Original: We use the clipped version of PPO with the clip ratio threshold set to 0.2.
Revised: We use the clipped version of PPO with the clip ratio threshold set to 0.2.

Original: We set the GAE parameter to λ = 0 .
Revised: We set the GAE parameter to λ = 0 .

Original: 95 and the discount factor to γ = 0 . 99 .
Revised: 95 and the discount factor to γ = 0 . 99 .

Original: Network Architecture : We use a consistent network architecture across all our simulation for both the actor and the critic: Two hidden layers of 64 and 64 neurons each and a ReLU non-linearity.
Revised: Network Architecture : We use a consistent network architecture across all our simulation for both the actor and the critic: Two hidden layers of 64 and 64 neurons each and a ReLU non-linearity.

Original: In the discreet action case (MOGW, DST, MOSM), we append the actor network with an output layer of dimension equal to the number of actions.
Revised: In the discreet action case (MOGW, DST, MOSM), we append the actor network with an output layer of dimension equal to the number of actions.

Revised: In the MuJoCo enviroments, where the actions are 1 https://

Original: In the MuJoCo enviroments, where the actions are continous, we use that output layer to parametrize a multi-dimensional Gaussian distribution, via its mean and variance, assuming uncorrelated actions.
Revised: https:// -super-mario-broshttps:// .com/ /torch-ac continous, we use that output layer to parametrize a multi-dimensional Gaussian distribution, via its mean and variance, assuming uncorrelated actions.

Original: Each non-terminal layer has a ReLU nonlinearity.
Revised: Each non-terminal layer has a ReLU nonlinearity.

Original: Even though the hyperbolic tangent is known to be preferred for reinforcement learning applications, we experimentally observed that it leads to vanishing gradients and slows down learning.
Revised: Even though the hyperbolic tangent is known to be preferred for reinforcement learning applications, we experimentally observed that it leads to vanishing gradients and slows down learning.

Original: This may be due to the fact that the PPA loss function requires second-differentiation.
Revised: This may be due to the fact that the PPA loss function requires second-differentiation.

Original: In the
Revised: In the

Original: SuperMario enviroment, we worked on the pixel domain and two convolutional layer of 32 andlayers as feature extractors, which were shared between the actor and the critic.
Revised: SuperMario enviroment, we worked on the pixel domain and two convolutional layer of 32 andlayers as feature extractors, which were shared between the actor and the critic.

Original: Hyperparameters and Training : To train the neural network, we use the Adam optimizer ( β 1 = 0 .
Revised: Hyperparameters and Training : To train the neural network, we use the Adam optimizer ( β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: with a learning rate of 0.0001.
Revised: with a learning rate of 0.0001.

Original: In the MOGW, DST and MOMU domains, we optimize the parameters of the actor and critic separately.
Revised: In the MOGW, DST and MOMU domains, we optimize the parameters of the actor and critic separately.

Original: In the MOSM domain, given that the actor and critic share the convolutional layers, we sum their losses (using weights 1 and 0.5, respectively) and use a common optimizer.
Revised: In the MOSM domain, given that the actor and critic share the convolutional layers, we sum their losses (using weights 1 and 0.5, respectively) and use a common optimizer.

Original: We run all of our simulations in the Google Cloud Platorm usingvCores and one NVIDIA Tesla T4 GPU.
Revised: We run all of our simulations in the Google Cloud Platorm usingvCores and one NVIDIA Tesla T4 GPU.

Original: For the MOGW and DST environments, we train the agent for 0 .
Revised: For the MOGW and DST environments, we train the agent for 0 .

Original: 5 M frames and for the MOSM and MOMU for 1 M frames.
Revised: 5 M frames and for the MOSM and MOMU for 1 M frames.

Original: For each 512 frames we perform one update of the network parameters iterating over 10 epochs of the collected data and using a mini-batch size of 64.
Revised: For each 512 frames we perform one update of the network parameters iterating over 10 epochs of the collected data and using a mini-batch size of 64.