Original: P ARETO -E FFICIENT D ECISION A GENTS FOR O FFLINE M ULTI -O BJECTIVE R EINFORCEMENT L EARNING
Revised: P ARETO -E FFICIENT D ECISION A GENTS FOR O FFLINE M ULTI -O BJECTIVE R EINFORCEMENT L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The goal of multi-objective reinforcement learning (MORL) is to learn policies that simultaneously optimize multiple competing objectives.
Revised: The goal of multi-objective reinforcement learning (MORL) is to learn policies that simultaneously optimize multiple competing objectives.

Original: In practice, an agent’s preferences over the objectives may not be known apriori, and hence, we require policies that can generalize to arbitrary preferences at test time.
Revised: In practice, an agent’s preferences over the objectives may not be known apriori, and hence, we require policies that can generalize to arbitrary preferences at test time.

Original: In this work, we propose a new data-driven setup for offline MORL, where we wish to learn a preference-agnostic policy agent using only a finite dataset of offline demonstrations of other agents and their preferences.
Revised: In this work, we propose a new data-driven setup for offline MORL, where we wish to learn a preference-agnostic policy agent using only a finite dataset of offline demonstrations of other agents and their preferences.

Original: The key contributions of this work are two-fold.
Revised: The key contributions of this work are two-fold.

Original: First, we introduce D4MORL, (D)atasets for MORL that are specifically designed for offline settings.
Revised: First, we introduce D4MORL, (D)atasets for MORL that are specifically designed for offline settings.

Original: It contains 1.8 million annotated demonstrations obtained by rolling out reference policies that optimize for randomly sampled preferences on 6 MuJoCo environments with 2-3 objectives each.
Revised: It contains 1.8 million annotated demonstrations obtained by rolling out reference policies that optimize for randomly sampled preferences on 6 MuJoCo environments with 2-3 objectives each.

Original: Second, we propose Pareto-Efficient Decision Agents (PEDA), a family of offline
Revised: Second, we propose Pareto-Efficient Decision Agents (PEDA), a family of offline

Original: MORL algorithms that builds and extends  Decision Transformers (Chen et al., 2021)  via a novel preference-and-return conditioned policy.
Revised: MORL algorithms that builds and extends return-conditioned offline methods including Decision Transformers (Chen et al., 2021) and RvS (Emmons et al., 2022) via a novel preference-and-return conditioned policy.

Original: Empirically, we show that PEDA closely approximates the behavioral policy on the D4MORL benchmark and provides an excellent approximation of the Pareto-front with appropriate conditioning, as measured by the hypervolume and sparsity metrics.
Revised: Empirically, we show that PEDA closely approximates the behavioral policy on the D4MORL benchmark and provides an excellent approximation of the Pareto-front with appropriate conditioning, as measured by the hypervolume and sparsity metrics.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We are interested in learning agents for multi-objective reinforcement learning (MORL) that optimize for one or more competing objectives.
Revised: We are interested in learning agents for multi-objective reinforcement learning (MORL) that optimize for one or more competing objectives.

Original: This setting is commonly observed in many real-world scenarios.
Revised: This setting is commonly observed in many real-world scenarios.

Original: For instance, an autonomous driving car might trade off high speed and energy savings depending on the user’s preferences.
Revised: For instance, an autonomous driving car might trade off high speed and energy savings depending on the user’s preferences.

Original: The car will move fast for users prefer speed, or keep a  steady speed for uses who try to save energy.
Revised: If the user has a relatively high preference for speed, the agent will move fast regardless of power usage; on the other hand, if the user tries to save energy, the agent will keep a more steady speed.

Original: One key challenge with MORL is that different users might have different preferences on the objectives and systematically exploring policies for each preference might be expensive or even impossible.
Revised: One key challenge with MORL is that different users might have different preferences on the objectives and systematically exploring policies for each preference might be expensive, or even impossible.

Original: In the online setting, prior work considers several approximations based on  learning an ensemble of policies based on enumerating preferences (Mossalam et al., 2016a, Xu et al., 2020), or extensions of single-objective algorithms such as Q-learning to vectorized value functions (Yang et al., 2019).
Revised: In the online setting, prior work considers several approximations based on scalarizing the vector-valued rewards of different objectives based on a single preference (Lin, 2005), learning an ensemble of policies based on enumerating preferences (Mossalam et al., 2016a, Xu et al., 2020), or extensions of single-objective algorithms such as Q-learning to vectorized value functions (Yang et al., 2019).

Original: We introduce the setting of offline multi-objective reinforcement learning for high-dimensional state and action spaces, where our goal is to train an MORL policy agent using an offline dataset of demonstrations  with known preferences.
Revised: We introduce the setting of offline multi-objective reinforcement learning for high-dimensional state and action spaces, where our goal is to train an MORL policy agent using an offline dataset of demonstrations from multiple agents with known preferences.

Original: Similar to the single-task setting, offline MORL can utilize auxiliary logged datasets to minimize interactions when deploying agents, thus improving data efficiency and minimizing interactions when deploying agents in high-risk settings.
Revised: Similar to the single-task setting, offline MORL can utilize auxiliary logged datasets to minimize interactions when deploying agents, thus improving data efficiency and minimizing interactions when deploying agents in high-risk settings.

Original: In addition to its practical utility, offline RL (Levine et al., 2020) has enjoyed major successes in the last few years (Kumar et al., 2020, Kostrikov et al., 2021, Chen et al., 2021) on challenging high-dimensional environments for continuous control and game-playing.
Revised: In addition to its practical utility, offline RL (Levine et al., 2020) has enjoyed major successes in the last few years (Kumar et al., 2020, Kostrikov et al., 2021, Chen et al., 2021) on challenging high-dimensional environments for continuous control.

Original: Our contributions in this work are two-fold in introducing benchmarking datasets and a new family of MORL, as described below.
Revised: Our contributions in this work are two-fold in introducing benchmarking datasets and a new family of MORL algorithms, as described below.

Original: We introduce Datasets for Multi-Objective Reinforcement Learning (D4MORL), a collection of 1. million trajectories on 6 multi-objective MuJoCo environments (Xu et al., 2020) with two or three objectives each.
Revised: We introduce Datasets for Multi-Objective Reinforcement Learning (D4MORL), a collection of 1. million trajectories on 6 multi-objective MuJoCo environments (Xu et al., 2020). Here, 5 environ- ments consist of 2 objectives and 1 environment consists of 3 objectives.

Original: For each environment in D4MORL, we collect demonstrations from 2 pretrained behavioral agents: expert and amateur , where the relative expertize is  measured empirically via their hypervolumes.
Revised: For each environment in D4MORL, we collect demonstrations from 2 pretrained behavioral agents: expert and amateur , where the relative expertize is defined in terms of the Pareto-efficiency of the agents and measured empirically via their hypervolumes.

Original: Furthermore, we also include 3 kinds of preference distributions with varying entropies to expose additional data-centric aspects for downstream benchmarking.
Revised: Furthermore, we also include 3 kinds of preference distributions with varying entropies to expose additional data-centric aspects for downstream benchmarking.

Revised: Lack of MORL datasets and large-scale benchmarking has been a major challenge for basic research (Hayes et al., 2021), and we hope that D4MORL can aid future research in the field.

Original: Next, we propose Pareto-Efficient Decision Agents (PEDA), a family of offline MORL algorithms that extends  Decision Transformer (DT) (Chen et al., 2021)  to the multi-objective setting.
Revised: Next, we propose Pareto-Efficient Decision Agents (PEDA), a family of offline MORL algorithms that extends return-conditioned methods including Decision Transformer (DT) (Chen et al., 2021) and RvS (Emmons et al., 2022) to the multi-objective setting.

Original: For{+These methods learn a returnconditioned policy via a supervised loss on the predicted actions. In recent work, these methods have successfully scaled to agents that demonstrate broad capabilities in multi-task settings (Lee et al., 
 Reed et al., 2022).+}
Revised: {+These methods learn a returnconditioned policy via a supervised loss on the predicted actions. In recent work, these methods have successfully scaled to agents that demonstrate broad capabilities in multi-task settings (Lee et al., 
 Reed et al., 2022).+}

Original:  MORL, we introduce a novel preference and return conditioned policy network and train it via a supervised learning loss.
Revised: For MORL, we introduce a novel preference and return conditioned policy network and train it via a supervised learning loss.

Revised: At test time, naively conditioning on the default preferences and maximum possible returns leads to out-of-distribution behavior for the model, as neither has it seen maximum returns for all objectives in the training data nor is it possible to simultaneously maximize all objectives under competition.

Revised: We address this issue by learning to map preferences to appropriate returns and hence, enabling predictable generalization at test-time.

Original: We find PEDA performs exceedingly well on D4MORL and closely approximates the reference Pareto-frontier of the behavioral policy used for data generation.
Revised: Empirically, we find PEDA performs exceedingly well on D4MORL and closely approximates the reference Pareto-frontier of the behavioral policy used for data generation.

Original: In the multi-objective HalfCheetah environment, compared with an  upper bound on the hypervolume of 5 .
Revised: In the multi-objective HalfCheetah environment, compared with an average upper bound on the hypervolume of 5 .

Original: 79 ˆ 10 6 achieved by the behavioral policy, PEDA achieves a hypervolume of 5 .
Revised: 79 ˆ 10achieved by the behavioral policy, PEDA achieves an average hypervolume of 5 .

Original: 77 ˆ 10 6 on the
Revised: 77 ˆ 10 6 on the

Original: Expert and 5 .
Revised: Expert and 5 .

Original: 76 ˆ 10 6 on the Amateur datasets.
Revised: 76 ˆ 10 6 on the Amateur datasets.

Original:  P RELIMINARIES
Revised:  R ELATED W ORK

Revised: Multi-Objective Reinforcement Learning Predominant works in MORL focus on the online setting where the goal is to train agents that can generalize to arbitrary preferences.

Revised: This can be achieved by training a single preference-conditioned policy (Yang et al., 2019; Parisi et al., 2016)), or an ensemble of single-objective policies for a finite set of preferences (Mossalam et al., 2016b;

Revised: Xu et al., 2020; Zhang & Li, 2007)).

Revised: Many of these algorithms consider vectorized variants of standard algorithms such as Q-learning (Mossalam et al., 2016b;

Revised: Yang et al., 2019), often augmented with strategies to guide the policy ensemble towards the Pareto front using evolutionary or incrementally updated algorithms (Xu et al., 2020; Zhang & Li, 2007; Mossalam et al., 2016b; Roijers et al., 2014; Huang et al., 2022)).

Revised: Other approaches have also been studied, such as framing MORL as a meta-learning problem (Chen et al., 2018), learning the action distribution for each objective (Abdolmaleki et al., 2020), and learning the relationship between objectives (Zhan & Cao, 2019) among others.

Revised: In contrast to these online MORL works, our focus is on learning a single policy that works for all preferences using only offline datasets.

Revised: There are also a few works that study decision making with multiple objectives in the offline setting and sidestep any interaction with the environments.

Revised: Wu et al., 2021 propose a provably efficient offline MORL algorithm for tabular MDPs based on dual gradient ascent.

Revised: Satija et al., 2021 study learning of safe policies by extending the approach of Laroche et al., 2017 to the offline MORL setting.

Revised: Their proposed algorithm assumes knowledge of the behavioral policy used to collect the offline data and is demonstrated primarily on tabular MDPs with finite state and action spaces.

Revised: In contrast, we are interested in developing dataset benchmarks and algorithms for scalable offline policy optimization in high-dimensional MDPs with continuous states and actions.

Revised: Multi-Task Reinforcement Learning MORL is also closely related to multi-task reinforcement learning, where every task can be interpreted as a distinct objective.

Revised: There is an extensive body of work in learning multi-task policies both in the online and offline setups (Wilson et al., 2007; Lazaric & Ghavamzadeh, 2010; Teh et al., 2017) inter alia.

Revised: However, the key difference is that typical MTRL benchmarks and algorithms do not consider solving multiple tasks that involve inherent trade-offs.

Revised: Consequently, there is no notion of Pareto efficiency and an agent can simultaneously excel in all the tasks without accounting for user preferences.

Revised: Reinforcement Learning Via Supervised Learning A body of recent works have formulated offline reinforcement learning as an autoregressive sequence modeling problem using Decision Transformers (DT) or Trajectory Transformers ( Chen et al., 2021, Janner et al., 2021)

Revised: The key idea in

Revised: DT is to learn a transformer-based policy that conditions on the past history and a dynamic estimate of the returns (a.k.a.

Revised: returns-to-go).

Revised: Follow-up works consider online learning (Zheng et al., 2022) as well as simpler variants that rely only on multi-layer perceptrons (Emmons et al., 2022).

Revised: Such agents are generally more stable and robust to optimize due to the simplicity of loss function and easier to scale to more complex settings such as environments with high-dimensional actions or states, as shown in recent works in multi-task RL (Lee et al., 2022;

Revised: Reed et al., 2022).

Revised:  P RELIMINARIES

Original: Setup and Notation.
Revised: Setup and Notation.

Original: We operate in the general framework of multi-objective Markov decision process (MOMDP) with linear preferences (Wakuta, 1995).
Revised: We operate in the general framework of multi-objective Markov decision process (MOMDP) with linear preferences (Wakuta, 1995).

Original: An MOMDP is represented by the tuple x S , A , P , R , Ω , f, γ y .
Revised: An MOMDP is represented by the tuple x S , A , P , R , Ω , f, γ y .

Original: At each timestep t , the agent with a current state s t P S takes an action a t P A to transition into a new state s t ` 1 with probability P p s t ` 1 | s t , a t q and observes a reward vector r t “ R p s t , a t q P R n .
Revised: At each timestep t , the agent with a current state s t P S takes an action a t P A to transition into a new state s t ` 1 with probability P p s t ` 1 | s t , a t q and observes a reward vector r t “ R p s t , a t q P R n .

Original: Here, n is the number of objectives.
Revised: Here, n is the number of objectives.

Original: The vector-valued return R P R n of an agent is given by the discounted sum of reward vectors over a time horizon, R “ ř t γ t r t .
Revised: The vector-valued return R P R n of an agent is given by the discounted sum of reward vectors over a time horizon, R “ ř t γ t r t .

Original: We also assume that there exists a linear utility function f and a space of preferences Ω that can map the reward vector r t and a preference vector ω P Ω to a scalar reward r t , i.e., r t “ f p r t , ω q “ ω ⊺ r t .
Revised: We also assume that there exists a linear utility function f and a space of preferences Ω that can map the reward vector r t and a preference vector ω P Ω to a scalar reward r t , i.e., r t “ f p r t , ω q “ ω ⊺ r t .

Original: The expected vector return of a policy π is given an G π “ r G π 1 , G π 2 , . . .
Revised: The expected vector return of a policy π is given an G π “ r G π 1 , G π 2 , . . .

Original: , G πn s ⊺ where the expected return of the i th objective is given as G πi “ E a t ` 1 „ π p¨| s t ,
Revised: , G πn s ⊺ where the expected return of the i th objective is given as G πi “ E a t ` 1 „ π p¨| s t ,

Original: ω q r ř t R p s t , a t q i s for some predefined time horizon and preference vector ω .
Revised: ω q r ř t R p s t , a t q i s for some predefined time horizon and preference vector ω .

Original: The goal is to train a multi-objective policy π p a | s , ω
Revised: The goal is to train a multi-objective policy π p a | s , ω

Original: q such that the expected scalarized return ω ⊺ G π “ E r ω ⊺ ř t R p s t , a t qs is maximized.
Revised: q such that the expected scalarized return ω ⊺ G π “ E r ω ⊺ ř t R p s t , a t qs is maximized.

Original: Pareto Optimality.
Revised: Pareto Optimality.

Original: In MORL, one cannot optimize all objectives simultaneously, so policies are evaluated based on the Pareto set of their vector-valued expected returns.
Revised: In MORL, one cannot optimize all objectives simultaneously, so policies are evaluated based on the Pareto set of their vector-valued expected returns.

Original: Consider a preferenceconditioned policy π p a | s , ω q that is evaluated for m distinct preferences ω 1 , . .
Revised: Consider a preference-conditioned policy π p a | s , ω q that is evaluated for m distinct preferences ω 1 , . .

Original: ω m , and let the resulting policy set be represented as t π p u p “ 1 ,...,m , where π p “ π p a | s , ω “ ω p q , and G π p is the corresponding unweighted expected return.
Revised: ω m , and let the resulting policy set be represented as t π p u p “ 1 ,...,m , where π p “ π p a | s , ω “ ω p q , and G π p is the corresponding unweighted expected return.

Original: We say the solution G π p is dominated by G π q when there is no objective for which π q is worse than π p , i.e., G π p
Revised: We say the solution G π p is dominated by G π q when there is no objective for which π q is worse than π p , i.e., G π p

Original: i
Revised: i

Original: ă G π q i for @
Revised: ă G π q i for @

Original: i P r 1 , 2 , . .
Revised: i P r 1 , 2 , . .

Original: . , n s .
Revised: . , n s .

Original: If a solution is not dominated, it is part of the Pareto set denoted as P .
Revised: If a solution is not dominated, it is part of the Pareto set denoted as P .

Original: The curve traced by the solutions in a Pareto set is also known as the Pareto front.
Revised: The curve traced by the solutions in a Pareto set is also known as the Pareto front.

Original: In MORL, our goal is to define a policy such that its empirical Pareto set is a good approximation of the true Pareto front.
Revised: In MORL, our goal is to define a policy such that its empirical Pareto set is a good approximation of the true Pareto front.

Original: While we do not know the true Pareto front for many problems, we can  evaluate the Pareto set P based on two metrics, hypervolume and sparsity  . 
Revised: While we do not know the true Pareto front for many problems, we can define metrics for relative comparisons between different algorithms. Specifically, we evaluate a Pareto set P based on two metrics, hypervolume and sparsity that we describe next. Definition 1 (Hypervolume) . Hypervolume H p P q measures the space or volume enclosed by the solutions in the Pareto set P :

Original: These two metrics are used by Xu et al., 2020.
Revised: [Equation] where H p P q “ t z P Z |D i : 1 ď i ď | P | , r ĺ z ĺ P p i qu . P p i q is the i th solution in P , ĺ is the dominance relation operator, and

Revised: 1 H p P

Revised: q p z q equals 1 if z P H p P q and 0 otherwise.

Revised: Higher hypervolumes are better.

Original: We define and discuss them in Appendix F.
Revised: Definition 2 (Sparsity) . Sparsity S p P q measures the density of the Pareto front covered by a Pareto set P : [Equation] where ˜ P i represents a list sorted as per the values of the i th objective in P and ˜ P i p k q is the k th value in the sorted list.

Revised: Lower sparsity is better.

Revised: See Figure 1 for an illustration and Appendix E for discussion on other possible metrics.

Original:  D4MORL: D ATASETS FOR O FFLINE M ULTI -O BJECTIVE R EINFORCEMENT L EARNING
Revised:  D4MORL: D ATASETS FOR O FFLINE M ULTI -O BJECTIVE R EINFORCEMENT L EARNING

Original: In offline RL, the goal of an RL agent is to learn the optimal policy using a fixed dataset without any interactions with the environment (Levine et al., 2020).
Revised: In offline RL, the goal of an RL agent is to learn the optimal policy using a fixed dataset without any interactions with the environment (Levine et al., 2020).

Original: This perspective brings RL closer to supervised learning, where the presence of large-scale datasets has been foundational for further progress in the field.
Revised: This perspective brings RL closer to supervised learning, where the presence of large-scale datasets has been foundational for further progress in the field.

Original: Many such data benchmarks exist for offline RL as well; a notable one is the D4RL (Fu et al., 2020) benchmark for continuous control which has led to the development of several state-of-the-art offline RL algorithms (Kostrikov et al., 2021; Kumar et al., 2020; Chen et al., 2021) that can scale favorably even in high dimensions.
Revised: Many such data benchmarks exist for offline RL as well; a notable one is the D4RL (Fu et al., 2020) benchmark for continuous control which has led to the development of several state-of-the-art offline RL algorithms (Kostrikov et al., 2021; Kumar et al., 2020; Chen et al., 2021) that can scale favorably even in high dimensions.

Original: To the best of our knowledge, there are no such existing benchmarks for offline MORL.
Revised: To the best of our knowledge, there are no such existing benchmarks for offline MORL.

Revised: Even for the online setting, most works in MORL conduct evaluations on toy MDPs (e.g., gridworld) with a few exceptions that include continuous control benchmarks, e.g., Chen et al.

Revised: Xu et al.

Revised: This calls for a much needed push towards more challenging benchmarks for reliable evaluation of MORL, especially in the offline setting.

Original: In this paper, we introduce Datasets for Multi-Objective Reinforcement Learning (D4MORL), a large-scale benchmark for offline MORL.
Revised: In this paper, we introduce Datasets for Multi-Objective Reinforcement Learning (D4MORL), a large-scale benchmark for offline MORL.

Original: Our benchmark consists of offline trajectories frommulti-objective MuJoCo environments including 5 environments with 2 objectives each viz.
Revised: Our benchmark consists of offline trajectories frommulti-objective MuJoCo environments including 5 environments with 2 objectives each viz.

Original: MOAnt, MO-HalfCheetah, MO-Hopper, MO-Swimmer, MO-Walker2d, and one environment with three objectives: MO-Hopper-3obj.
Revised: MOAnt, MO-HalfCheetah, MO-Hopper, MO-Swimmer, MO-Walker2d, and one environment with three objectives: MO-Hopper-3obj.

Revised: The objectives are conflicting for each environment; for instance, the two objectives in MO-Hopper correspond to jumping and running; in MO-HalfCheetah, MOSwimmer, and MO-Walker2d, they correspond to the speed and energy savings of the agent.

Original: The objectives are conflicting for each environment, please see Appendix A for more details on the semantics of the target objectives.
Revised: See Appendix A for more details on the semantics of the target objectives for each environment.

Original: These environments were first introduced in Xu et al.
Revised: These environments were first introduced in Xu et al.

Original: (2020) for online MORL. [Figure]
Revised: (2020) for online MORL, and as such, we use their pretrained ensemble policies as building blocks for defining new behavioral polices for dataset collection, which we discuss next.

Original: 3. T RAJECTORY S AMPLING
Revised: 4. T RAJECTORY S AMPLING

Original: The quality of the behavioral policy used for sampling trajectories in the offline dataset is a key factor for benchmarking downstream offline RL algorithms. 
Revised: The quality of the behavioral policy used for sampling trajectories in the offline dataset is a key factor for benchmarking downstream offline RL algorithms. In existing benchmarks for single-objective

Revised: RL such as D4RL (Fu et al., 2020), the quality of a behavioral policy can be ascertained and varied based on its closeness to a single expert policy, as measured by their scalar-valued returns.

Original: For a MOMDP,  a reference expert policy (or set of policies) should reflect the optimal returns for all possible preferences in the preference space.
Revised: For a MOMDP, we do not have the notion of a scalar return and hence, a reference expert policy (or set of policies) should reflect the optimal returns for all possible preferences in the preference space.

Original: We use Prediction-Guided Multi-Objective Reinforcement Learning (PGMORL), a state-of-the-art
Revised: We use Prediction-Guided Multi-Objective Reinforcement Learning (PGMORL), a state-of-the-art

Original: MORL algorithm for defining reference expert policies.
Revised: MORL algorithm for defining reference expert policies.

Original: PGMORL (Xu et al., 2020) uses evolutionary algorithms to train an ensemble of policies to approximate the Pareto set.
Revised: PGMORL (Xu et al., 2020) uses evolutionary algorithms to train an ensemble of policies to approximate the Pareto set.

Original: Each reference policy in the ensemble is associated with a unique preference; for any new preference, it is mapped to the closest preference in the reference set.
Revised: Each reference policy in the ensemble is associated with a unique preference; for any new preference, it is mapped to the closest preference in the reference set.

Revised: The number of policies in the ensemble can vary significantly; for instance, we have roughly 70 reference policies for MO-Antand 2445 policies for harder environments such as MO-Hopper-3obj.

Original: Given a desired preference, we define two sets of behavioral policies:
Revised: Given a desired preference, we define two sets of behavioral policies:

Original: Expert Dataset: We find the best reference policy in the policy ensemble, and always follow the action taken by the reference policy.
Revised: Expert Dataset: We find the best reference policy in the policy ensemble, and always follow the action taken by the reference policy.

Original: Amateur Dataset: As before, we first find the best reference policy in the policy ensemble.
Revised: Amateur Dataset: As before, we first find the best reference policy in the policy ensemble.

Original: With a fixed probability p , we randomly perturb the actions of the reference policies.
Revised: With a fixed probability p , we randomly perturb the actions of the reference policies.

Original: Otherwise, with probability 1 ´ p , we take the same action as the reference policy.
Revised: Otherwise, with probability 1 ´ p , we take the same action as the reference policy.

Original: In D4MORL, we set p “ 0 . 65 .
Revised: In D4MORL, we set p “ 0 . 65 .

Original: Further details are described in Appendix C. In Figure 1, we show the returns of the trajectories rolled out from the expert and amateur policies for the 2 objective environments evaluated for a uniform sampling of preferences.
Revised: Further details are described in Appendix C. In Figure 2, we show the returns of the trajectories rolled out from the expert and amateur policies for the 2 objective environments evaluated for a uniform sampling of preferences.

Original: We can see that the expert trajectories typically dominate the amateur trajectories, as desired.
Revised: We can see that the expert trajectories typically dominate the amateur trajectories, as desired.

Original: For the amateur trajectories, we see more diversity in the empirical returns for both objectives under consideration.
Revised: For the amateur trajectories, we see more diversity in the empirical returns for both objectives under consideration.

Original: The return patterns for the amateur trajectories vary across different environments providing a diverse suite of datasets in our benchmark. 
Revised: The return patterns for the amateur trajectories vary across different environments providing a diverse suite of datasets in our benchmark. [Figure] [Figure]

Original: 3. P REFERENCE S AMPLING
Revised: 4. P REFERENCE S AMPLING

Original: The coverage of any offline dataset is an important factor in dictating the performance of downstream offline RL algorithms (Levine et al., 2020).
Revised: The coverage of any offline dataset is an important factor in dictating the performance of downstream offline RL algorithms (Levine et al., 2020).

Original: For MORL, the coverage depends on both the behavioral
Revised: For MORL, the coverage depends on both the behavioral

Original: MORL policy as well as the distribution of preferences over which this policy is evaluated.
Revised: MORL policy as well as the distribution of preferences over which this policy is evaluated.

Original: We use the following protocols for sampling from the preference space Ω .
Revised: We use the following protocols for sampling from the preference space Ω .

Original: First, we restrict our samples to lie within a physically plausible preference space Ω ˚ Ď Ω covered by the behavioral policy π β .
Revised: First, we restrict our samples to lie within a physically plausible preference space Ω ˚ Ď Ω covered by the behavioral policy π β .

Original: For instance,the MO-Hopper  agent can never gain running rewards without leaving the floor.
Revised: For instance, MO-Hopper has two objectives: jumping and running. Since the agent can never gain running rewards without leaving the floor.

Original: Thus, the preference of 100% running and 0% jumping is  excluded from our preference sampling distribution.
Revised: Thus, the preference of 100% running and 0% jumping is not achievable and excluded from our preference sampling distribution.

Original: Second, we are primarily interested in offline trajectories that emphasize competition between multiple objectives rather than focusing on a singular objective.
Revised: Second, we are primarily interested in offline trajectories that emphasize competition between multiple objectives rather than focusing on a singular objective.

Original: To enforce this criteria, we define 3 sampling distributions concentrated around the centroid of the preference simplex.
Revised: To enforce this criteria, we define 3 sampling distributions concentrated around the centroid of the preference simplex.

Original: The largest spread distribution samples uniformly from Ω ˚ and is denoted as High-Entropy ( High-H ).
Revised: The largest spread distribution samples uniformly from Ω ˚ and is denoted as High-Entropy ( High-H ).

Original: Next, we have a Medium-Entropy ( Med-H ) distribution specified via samples of Dirichlet distributions with large values of their concentration hyperparameters (aka α ).
Revised: Next, we have a Medium-Entropy ( Med-H ) distribution specified via samples of Dirichlet distributions with large values of their concentration hyperparameters (aka α ).

Original: Finally, we have a Low-Entropy ( Low-H ) distribution that is again specified via samples of Dirichlet distributions, but with low values of their concentration hyperparameters.
Revised: Finally, we have a Low-Entropy ( Low-H ) distribution that is again specified via samples of Dirichlet distributions, but with low values of their concentration hyperparameters.

Original: We illustrate the samples for each of the preference distributions along with their empirical entropiesand other details in Figure 2 from Appendix B.
Revised: We illustrate the samples for each of the preference distributions along with their empirical entropies in Figure 3.

Original:  By ensuring different levels of coverage, we can test the generalizability of an MORL policy to preferences unseen during training.
Revised: Further details on the sampling distributions are deferred to Appendix B. By ensuring different levels of coverage, we can test the generalizability of an MORL policy to preferences unseen during training.

Original: In general, we expect Low-H to be the hardest of the three distributions due to its restricted coverage, followed by Med-H and High-H . 
Revised: In general, we expect Low-H to be the hardest of the three distributions due to its restricted coverage, followed by Med-H and High-H . Overall Data Generation Pipeline.

Revised: The pseudocode for generating the dataset is described in

Revised: Algorithm 1.

Original: P ARETO -E FFICIENT D ECISION A GENTS (PEDA)
Revised: Given a preference distribution, we first sample a preference ω and query the closest behavioral policy in either the amateur/expert ensemble matching ω . We rollout this policy for T time steps (or until the end of an episode if sooner) and record the state, action, and reward information. Each trajectory in our dataset is represented as:

Revised: [Equation]

Revised: Algorithm procedure C OLLECT (prefDist, nTraj, env, pretrainedAgents, T) agents =

Revised: pretrainedAgents prefs =

Revised: prefDist(nTraj) all trajs =

Revised: [] for ω in prefs do agent

Revised: = closestAgent(agents, ω ) s = env.reset() done =

Revised: False τ = [ ω ] t =while (NOT done) AND (t ă T) do a = agent.get action( s ) s 1 , done, r = env.step( a ) append s , a , s 1 , r to τ s = st = t

Revised: +append τ to all trajs return all trajs

Revised: For every environment in D4MORL, we collect 50K trajectories of length T “ 500 for both expert and amateur trajectory distributions under each of the 3 preference distributions.

Revised: Overall, this results in a total of 1.8M trajectories over all 6 environments, which corresponds to roughly 867M time steps.

Revised: We refer the reader to Table 5 in Appendix

Revised: B for additional statistics on the dataset.

Original: In this section, we propose Pareto-Efficient Decision Agents (PEDA) , a family of offline multiobjective RL agents that aims to achieve Pareto-efficiency by extending Decision Transformers (Chen et al., 2021) into multi-objective setting.
Revised: In this section, we propose Pareto-Efficient Decision Agents (PEDA) , a family of offline multiobjective RL agents that aims to achieve Paretoefficiency by extending Decision Transformers (Chen et al., 2021) into multi-objective setting.

Original: Our models extend from the architecture of Decision Transformers (DT) and its variant, Reinforcement Learning Via Supervised Learning (RvS).
Revised: We first introduce the architecture of Decision Transformers (DT) and its variant, Reinforcement Learning Via Supervised Learning (RvS), followed by our modifications extending them to the multi-objective setting. DT casts offline RL as a conditional sequence modeling problem that predicts the next action by conditioning a transformer on past states, actions, and desired returns. The desired returns are defined as returns-to-go (RTG) g t “ ř Tt 1 “ t r t 1 , the future returns that this action is intended to achieve. Therefore, the trajectory is represented by τ “ă s 1 , a 1 , g 1 , . . .

Original: 4. M ULTI -O BJECTIVE R EINFORCEMENT L EARNING V IA S UPERVISED L EARNING
Revised: , s T , a T , g T ą . In practice, we use a causally masked transformer architecture such as GPT (Radford et al., 2019) to process this sequence and predict the actions by observing the past K timesteps consisting of 3 K tokens. DT and its variants have been shown to be more stable and robust to optimize due to the simplicity of loss function; easier to scale to more complex settings such as environments with high-dimensional actions or states, and agents with broad capabilities such as multitask settings (Lee et al., 2022). Hence, we adopt Decision Transformers (Chen et al., 2021) as the representative base algorithm on which we build our work.

Revised: In follow-up work, Emmons et al.

Revised: (2022) extend DT and shows that even multi-layer perceptrons conditioned on the average returns-to-go can achieve similar performance without the use of transformers.

Revised: They call their model as Reinforcement Learning Via Supervised Learning (RvS).

Revised: However, RvS is generally not very stable when conditioned on very large returns, unlike DT.

Revised: 5. M ULTI -O BJECTIVE R EINFORCEMENT L EARNING V IA S UPERVISED L EARNING

Original: In PEDA, our goal is to train a single preference-conditioned agent for offline MORL.
Revised: In PEDA, our goal is to train a single preference-conditioned agent for offline MORL.

Original: By including preference conditioning, we enable the policy to be trained on arbitrary offline data, including trajectories collected from behavioral policies that are associated with alternate preferences.
Revised: By including preference conditioning, we enable the policy to be trained on arbitrary offline data, including trajectories collected from behavioral policies that are associated with alternate preferences.

Original: To parameterize our policy agents, we extend the DT and RvS architectures to include preference tokens and vector-valued returns.
Revised: To parameterize our policy agents, we extend the DT and RvS architectures to include preference tokens and vector-valued returns.

Original: We refer to such preference-conditioned extensions of these architectures as MODT(P) and MOR V S(P) respectively, which we describe next.
Revised: We refer to such preference-conditioned extensions of these architectures as MODT(P) and MOR V S(P) respectively, which we describe next.

Original: Preference Conditioning.
Revised: Preference Conditioning.

Original: Naively, we can easily incorporate the preference ω into DT by adding this token for each timestep and feeding it a separate embedding layer.
Revised: Naively, we can easily incorporate the preference ω into DT by adding this token for each timestep and feeding it a separate embedding layer.

Original: However, empirically we find that such a model design tends to ignore ω and the correlation between the preferences and predicted actions is weak.
Revised: However, empirically we find that such a model design tends to ignore ω and the correlation between the preferences and predicted actions is weak.

Original: Therefore, we propose to concatenate ω to other tokens before any layers in MODT(P).
Revised: Therefore, we propose to concatenate ω to other tokens before any layers in MODT(P).

Original: Concretely, we define s ˚ “ s À ω , a ˚ “ a À ω , and g ˚ “ g À ω where À denotes the concatenation operator.
Revised: Concretely, we define s ˚ “ s À ω , a ˚ “ a À ω , and g ˚ “ g À ω where À denotes the concatenation operator.

Original: Hence, triples of s ˚ , a ˚ , g ˚ form the new trajectory.
Revised: Hence, triples of s ˚ , a ˚ , g ˚ form the new trajectory.

Original: As for
Revised: As for

Original: MOR V S(P), we concatenate the preference with the states and the average RTGs by default and the network interprets everything as one single input.
Revised: MOR V S(P), we concatenate the preference with the states and the average RTGs by default and the network interprets everything as one single input.

Original: Multi-Objective Returns-to-Go.
Revised: Multi-Objective Returns-to-Go.

Original: Similar to RTG for the single objective case, we can define vector-valued RTG as g t “ ř Tt 1 “ t r t 1 Given a preference vector ω , we can scalarize the total returnsto-go as ˆ g t “ ω T g t .
Revised: Similar to RTG for the single objective case, we can define vector-valued RTG as g t “ ř Tt 1 “ t r t 1 Given a preference vector ω , we can scalarize the total returns-to-go as ˆ g t “ ω T g t .

Original: In principle, the scalarized RTG ˆ g t and preference vector ω can recover the vector-valued RTG g t .
Revised: In principle, the scalarized RTG ˆ g t can be recovered given the preference vector ω and the vector-valued RTG g t .

Original: However, empirically we find that directly feeding MODT/MOR V S with the preference-weighted RTG vector g t d ω is slightly preferable for stable training, where d denotes the elementwise product operator.
Revised: However, empirically we find that directly feeding MODT/MOR V S with the preference-weighted RTG vector g t d ω is slightly preferable for stable training, where d denotes the elementwise product operator.

Original: Another unique challenge in the MORL setting concerns the scale of different objectives.
Revised: Another unique challenge in the MORL setting concerns the scale of different objectives.

Original: Since different objectives can signify different physical quantities (e.g., energy and speed), the choice of scaling can influence policy optimization.
Revised: Since different objectives can signify different physical quantities (e.g., energy and speed), the choice of scaling can influence policy optimization.

Original: We adopt a simple normalization scheme, where the returns for each objective are normalized by subtracting the minimum observed value for that objective and dividing it by the range of values (max-min).
Revised: We adopt a simple normalization scheme, where the returns for each objective are normalized by subtracting the minimum observed value for that objective and dividing it by the range of values (max-min).

Original: Note that the maximum and minimum are computed based on the offline dataset and hence, they are not necessarily the true min/max objective values.
Revised: Note that the maximum and minimum are computed based on the offline dataset and hence, they are not necessarily the true min/max objective values.

Revised: Post this normalization, the values for every objective in the trajectory are on the same scale between 0 and 1.

Original: For evaluating the hypervolume and sparsity, we use the unnormalized values so that we can make comparisons across different datasets that may have different min/max boundaries. 
Revised: For evaluating the hypervolume and sparsity, we use the unnormalized values so that we can make comparisons across different datasets that may have different min/max boundaries. Training.

Revised: We follow a simple supervised training procedure where we train the policies on randomly sampled mini-batches with MSE loss (for continuous actions).

Revised: In MODT and MODT(P), the input states, actions, and returns-to-go (with concatenated preferences) are treated as tokens and embedded through one layer of MLP.

Revised: We apply a layer of MLP and Tanh on the last hidden state of GPT-2 transformer to predict next action.

Revised: In MOR V S and MOR V S(P), we use only information from the current timestep and MLP layers to predict the next action.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate the performance of PEDA on D4MORL benchmark.
Revised: In this section, we evaluate the performance of PEDA on D4MORL benchmark.

Original: First, we investigate the benefits of preference conditioning by evaluating on MODT and MOR V S where no preference information is available while still using multi-objective return-to-go.
Revised: First, we investigate the benefits of preference conditioning by evaluating on decision transformers (DT) and RvS (MOR V S) where no preference information is available and we scalarize multi-objective vector returns into weighted sums.

Original: We denote our methods with preference conditioning as MODT(P) and MOR V S(P).
Revised: We denote our methods with preference conditioning as MODT(P) and MOR V S(P).

Original: Then, we compare our methods with classic imitation learning that uses supervised loss to train a mapping from states (w/ or w/o concatenating preferences) to actions.
Revised: Second, we compare our methods with classic imitation learning and temporal difference learning algorithms with preference conditioning. Imitation learning. Imitation learning simply uses supervised loss to train a mapping from states (w/ or w/o concatenating preferences) to actions.

Original: We use behavioral cloning (BC) here and train multi-layer MLPs as models named BC (w/o preference) and BC(P) (w/ preference). 
Revised: We use behavioral cloning (BC) here and train multi-layer MLPs as models named BC (w/o preference) and BC(P) (w/ preference). Temporal difference learning.

Original: Finally, we modify the network architecture of Conservative Q-Learning (CQL) (Kumar et al., 2020), a state-of-the-art standard offline 
Revised: Conservative Q-Learning (CQL) (Kumar et al., 2020) is the stateof-the-art standard offline RL method, which learns a conservative Q-function f : S ˆ A Ñ R through neural networks.

Original: RL method, such that it also takes preference vectors as inputs to learn a preference-conditioned Q-function f ˚ : S ˆ A ˆ Ω Ñ R .
Revised: We modify the network architecture such that it also takes preference vectors as inputs to learn a preference-conditioned Q-function f ˚ : S ˆ A ˆ Ω Ñ R .

Original: We denote this method as CQL(P).[Table]
Revised: We denote this method as CQL(P).

Original:  M ULTI - OBJECTIVE O FFLINE B ENCHMARK
Revised: 6. M ULTI - OBJECTIVE O FFLINE B ENCHMARK

Original: [Table]
Revised: [Table]

Original: Hypervolume.
Revised: Hypervolume.

Original: We compare hypervolume of our methods with all baselines on Expert datasets in Table 1.
Revised: We compare hypervolume of our methods with all baselines on expert datasets in Table 1 as well as amateur dataset in Table 2.

Original: For the two-objective environments, we evaluate the models on 501 and 325 equally spaced preference points for two-objective and three-objective environments respectively.
Revised: For the two-objective environments [Table] we evaluate the [Table]

Revised: Ó ) performance on High-H-Expert dataset.

Revised: MODT(P)

Revised: and MOR V S(P) have a lower density.

Revised: BC(P) also has a competitive sparsity in smaller environments such as Swimmer.

Revised: [Table]

Revised: models on 501 equally spaced preference points in the range [0, 1]; on the three-objective environment MO-Hopper-3obj, models are evaluated on 325 equally spaced points.

Original: Each point is evaluated 5 times with random environment re-initialization, and the median value is recorded.
Revised: Each point is evaluated 5 times with random environment re-initialization, and the median value is recorded.

Original: Finally, all the results are based onaverage of 3 random seeds  along with the standard error.
Revised: Finally, all the results are based on 3 random seeds and we report the mean performance along with the standard error.

Original: In Table 1, we can see that MODT(P) and MOR V S(P) outperform other baselines with higher mean, lower standard errors.
Revised: In Table 1 and Table 2, we can see that MODT(P) and MOR V S(P) outperform other baselines and has a relatively very low standard error.

Original: They also approach the behavioral policy upper-bound.
Revised: Also, PEDA variants including MODT(P) and MOR V S(P) approaches the behavioral policy upper-bound.

Original: Sparsity.
Revised: Sparsity.

Original: We also evaluate sparsity performance.
Revised: We also evaluate sparsity performance.

Original: We only show results for preference-conditioned models since sparsity  is only meaningful when models  are sensitive to preference and are similarin hypervolume performance.
Revised: Since sparsity comparison is only meaningful between models that are sensitive to preference and have a relatively similar hypervolume performance, we only show results for models that concatenate preference.

Original: Overall, MOR V S(P) has the lowest sparsity in most environments, while at the same time featuring an outstanding hypervolume. 
Revised: Overall, MOR V S(P) has the lowest sparsity in most environments, while at the same time featuring an outstanding hypervolume. 6. A BLATION S TUDY

Revised: Pareto front approximation.

Revised: We ablate how well the MODT(P) and MOR V S(P) can approximate the Pareto front through conditioning on different preference points.

Revised: We show the results in

Revised: Figure 4, where we can see that the models can approximate the Pareto front, while having some dominated points colored in pink mostly in the MO-Hopper and MO-Walker2d environments.

Original: We include the the results and analysis on  the Amateur datasets in Table 4 and Table 3 from
Revised: The results are based on average of 3 seeds, and the full plot can be found in Appendix F. [Table] In addition, MO-Hopper and MO Walker2d

Revised: PEDA

Revised: The distribution of returns.

Original: Appendix G. These tables lead to the same conclusion.
Revised: We ablate how well MODT(P) and MOR V S(P) follow their given target return, based on a normalized and weighted value. We present the results in Figure 5 for MOR V S(P) under High-H-Expert datasets and refer to Appendix G for full settings.

Revised: Here, we see that the models follow the oracle line nicely when conditioned on target within the dataset distribution, and generalize to targets outside of the dataset distribution as well.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduced a novel dataset benchmark and algorithms for offline Multi-Objective
Revised: In this paper, we introduced a novel dataset benchmark and algorithms for offline Multi-Objective

Original: Reinforcement Learning.
Revised: Reinforcement Learning.

Original: We first proposed D4MORL, a dataset benchmark consisting of offline datasets generated from behavioral policies of different fidelities (expert/amateur) and rolled out under preference distributions with varying entropies (high/medium/low).
Revised: We first proposed D4MORL, a dataset benchmark consisting of offline datasets generated from behavioral policies of different fidelities (expert/amateur) and rolled out under preference distributions with varying entropies (high/medium/low).

Original: Then, we propose PEDA, a family of offline MORL policy optimization algorithms based on decision transformers and show that by concatenating and embedding preference together with other inputs, our policies can effectively approximate the Pareto front of the underlying behavioral policy as measured by the hypervolume and sparsity metrics.
Revised: Then, we propose PEDA, a family of offline MORL policy optimization algorithms based on decision transformers and show that by concatenating and embedding preference together with other inputs, our policies can effectively approximate the Pareto front of the underlying behavioral policy as measured by the hypervolume and sparsity metrics.

Original: Our proposed family includes MLP and transformer based variants, viz.
Revised: Our proposed family includes MLP and transformer based variants, viz.

Original: the MOR V S(P) and MODT(P), with MOR V S(P) performing the best overall.
Revised: the MOR V S(P) and MODT(P), with MOR V S(P) performing the best overall.

Original: In some scenarios, the learned policies can also generalize to higher target rewards that exceeds the data distribution.
Revised: In some scenarios, the learned policies can also generalize to higher target rewards that exceeds the data distribution.

Original: To our knowledge, the PEDA variants are the first offline MORL policies that supports both the continuous action control and continuous preference space.
Revised: To our knowledge, the PEDA variants are the first offline MORL policies that supports both the continuous action control and continuous preference space.

Original: Ó ) performance on High-H-Expert dataset.
Revised: In this paper we make sure the experiment results are reproducible through training the models on multiple seeds and evaluating each preference point for multiple times. We report the mean value with standard error in all of our experiment result tables.

Original: MODT(P)
Revised: Furthermore, we upload our model code along with our submission, and sample data can be found in this anonymous Google Drive folder: https://drive.google.com/drive/folders/ 1FiF5xmCSJ2vL_frLYmeZNc_nUrhRfUXC?usp=sharing .

Original: and MOR V S(P) have a lower density. BC(P) also has a competitive sparsity in smaller environments such as Swimmer [Table]
Revised: Due to storage limits, we currently release only a subset of our data that can be used directly for training; we will release our full datasets in the final version.

Original: A E NVIRONMENT D ESCRIPTION
Revised: A E NVIRONMENT D ESCRIPTION

Original: All environments are the same as in Xu et al., 2020, except for when resetting the environment, each parameter are uniformly sampled from the r x ´ 10 ´ 3 , x ` 10 ´ 3 s with x being the default value.
Revised: All environments are the same as in Xu et al., 2020, except for when resetting the environment, each parameter are uniformly sampled from the r x ´ 10 ´ 3 , x ` 10 ´ 3 s with x being the default value.

Original: The only exception is that MO-Hopper and MO-Hopper-3objalways reset height the same way as 1 . 25 , since this parameter directly relates to the reward function.
Revised: The only exception is that MO-Hopper and MO-Hopper-3objalways reset height the same way as 1 . 25 , since this parameter directly relates to the reward function.

Original: All environments have a max episode length of 500 steps per trajectory, but the agent may also die before reaching the maximum length.
Revised: All environments have a max episode length of 500 steps per trajectory, but the agent may also die before reaching the maximum length.

Original: A. MO-A NT
Revised: A. MO-A NT

Original: The two objectives in MO-Ant are achieved distance in x and y axes respectively, denoted as r “ r r vxt , r vyt s ⊺ .
Revised: The two objectives in MO-Ant are achieved distance in x and y axes respectively, denoted as r “ r r vxt , r vyt s ⊺ .

Original: Consider the position of the agent is represented as p x t , y t q at time t and takes the action a t .
Revised: Consider the position of the agent is represented as p x t , y t q at time t and takes the action a t .

Original: The agent has a fixed survival reward r s “ 1 . 0 , dt “ 0 . 05 , and an action cost of r a “ 12 ř k a 2 k .
Revised: The agent has a fixed survival reward r s “ 1 . 0 , dt “ 0 . 05 , and an action cost of r a “ 12 ř k a 2 k .

Original: The rewards are calculated as:
Revised: The rewards are calculated as:

Original: [Equation]
Revised: [Equation]

Original: A. MO-H ALF C HEETAH
Revised: A. MO-H ALF C HEETAH

Original: The two objectives in MO-HalfCheetah are running speed, and energy saving, denoted as r “ r r vt , r et s ⊺ .
Revised: The two objectives in MO-HalfCheetah are running speed, and energy saving, denoted as r “ r r vt , r et s ⊺ .

Original: Consider the position of the agent is represented as p x t , y t q at time t and takes the action a t .
Revised: Consider the position of the agent is represented as p x t , y t q at time t and takes the action a t .

Original: The agent has a fixed survival reward r s “ 1 . 0 , fixed dt “ 0 . 05 , and an action cost of r a “ ř k a 2 k .
Revised: The agent has a fixed survival reward r s “ 1 . 0 , fixed dt “ 0 . 05 , and an action cost of r a “ ř k a 2 k .

Original: The rewards are calculated as:
Revised: The rewards are calculated as:

Original: [Equation]
Revised: [Equation]

Original: A. MO-H OPPER
Revised: A. MO-H OPPER

Original: The two objectives in MO-Hopper are running and jumping, denoted as r “ r r r , r j s ⊺ .
Revised: The two objectives in MO-Hopper are running and jumping, denoted as r “ r r r , r j s ⊺ .

Original: Consider the position of the agent is represented as p x t , h t q at time t and takes the action a t .
Revised: Consider the position of the agent is represented as p x t , h t q at time t and takes the action a t .

Original: The agent has a fixed survival reward r s “ 1 . 0 , a fixed initial height as h init “ 1 . 25 , a fixed dt “ 0 . 01 , and an action cost of r a “ 2 ˆ 10 ´ 4 ř k a 2 k .
Revised: The agent has a fixed survival reward r s “ 1 . 0 , a fixed initial height as h init “ 1 . 25 , a fixed dt “ 0 . 01 , and an action cost of r a “ 2 ˆ 10 ´ 4 ř k a 2 k .

Original: The rewards are calculated as:
Revised: The rewards are calculated as:

Original: [Equation]
Revised: [Equation]

Original: A. MO-H OPPER -3 OBJ
Revised: A. MO-H OPPER -3 OBJ

Original: The physical dynamics are the same in MO-Hopper and MO-Hopper-3obj, while this environment has 3 objectives: running, jumping, and energy saving.
Revised: The physical dynamics are the same in MO-Hopper and MO-Hopper-3obj, while this environment has 3 objectives: running, jumping, and energy saving.

Original: The rewards are denoted as r “ r r r , r j , r e s ⊺ .
Revised: The rewards are denoted as r “ r r r , r j , r e s ⊺ .

Original: Consider the position of the agent is represented as p x t , h t q at time t and takes the action a t .
Revised: Consider the position of the agent is represented as p x t , h t q at time t and takes the action a t .

Original: The agent has a fixed survival reward r s “ 1 . 0 , a fixed initial height as h init “ 1 . 25 , a fixed dt “ 0 . 01 , and an action cost of r a “ ř k a 2 k .
Revised: The agent has a fixed survival reward r s “ 1 . 0 , a fixed initial height as h init “ 1 . 25 , a fixed dt “ 0 . 01 , and an action cost of r a “ ř k a 2 k .

Original: The rewards are calculated as:
Revised: The rewards are calculated as:

Original: [Equation]
Revised: [Equation]

Original: A. MO-S WIMMER
Revised: A. MO-S WIMMER

Original: The two objectives inMO-Swimmer are speed and energy saving, denoted as r “ r r v , r e s ⊺ .
Revised: The two objectives inMO-Swimmer are speed and energy saving, denoted as r “ r r v , r e s ⊺ .

Original: Consider the position of the agent is represented as p x t , y t q at time t and takes the action a t .
Revised: Consider the position of the agent is represented as p x t , y t q at time t and takes the action a t .

Original: The agent has a fixed dt “ 0 . 05 , and an action cost of r a “ ř k a 2 k .
Revised: The agent has a fixed dt “ 0 . 05 , and an action cost of r a “ ř k a 2 k .

Original: The rewards are calculated as:
Revised: The rewards are calculated as:

Original: [Equation]
Revised: [Equation]

Original: A. MO-W ALKER 2 D
Revised: A. MO-W ALKER 2 D

Original: The objectives in MO-Walker2d are speed and energy saving, denoted as r “ r r v , r e s ⊺ .
Revised: The objectives in MO-Walker2d are speed and energy saving, denoted as r “ r r v , r e s ⊺ .

Original: Consider the position of the agent is represented as p x t , y t q at time t and takes the action a t .
Revised: Consider the position of the agent is represented as p x t , y t q at time t and takes the action a t .

Original: The agent has a fixed survival reward r s “ 1 . 0 , a fixed dt “ 0 . 008 , and an action cost of r a “ ř k a 2 k .
Revised: The agent has a fixed survival reward r s “ 1 . 0 , a fixed dt “ 0 . 008 , and an action cost of r a “ ř k a 2 k .

Original: The rewards are calculated as:
Revised: The rewards are calculated as:

Original: [Equation]
Revised: [Equation]

Original: To uniformly sample the High-H data from the entire preference space, the problem is equivalent to sampling from a n -dimensional simplex, where n is the number of objectives.
Revised: To uniformly sample the High-H data from the entire preference space, the problem is equivalent to sampling from a n -dimensional simplex, where n is the number of objectives.

Original: The resulting sampling is:
Revised: The resulting sampling is:

Original: [Equation]
Revised: [Equation]

Original: We take the 1-norm following the exponential distribution to make sure each preference add up to 1.
Revised: We take the 1-norm following the exponential distribution to make sure each preference add up to 1.

Original: When Ω ˚ ‰ Ω , we perform rejection sampling to restrict the range.
Revised: When Ω ˚ ‰ Ω , we perform rejection sampling to restrict the range.

Original: To sample the Med-H and Low-H data, we first sample α from a non-negative uniform distribution, then sample the corresponding Dirichlet preference.
Revised: To sample the Med-H and Low-H data, we first sample α from a non-negative uniform distribution, then sample the corresponding Dirichlet preference.

Original: Here, we sample a different alpha to make sure the center of the Dirichlet changes and thus allows more variation.
Revised: Here, we sample a different alpha to make sure the center of the Dirichlet changes and thus allows more variation.

Original: [Equation]
Revised: [Equation]

Original: where α „ Unif p {+0 , 
+}
Revised: where α „ Unif p {+0 , 
+}

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: q ; where α „ Unif p 1 {
Revised: q ; where α „ Unif p 1 {

Original: [Equation]
Revised: [Equation]

Original: 10 6 , 2 { 3 ˆ 10 6 q
Revised: 10 6 , 2 { 3 ˆ 10 6 q

Original: [Equation]
Revised: [Equation]

Original: For sampling from behavioral policy consists of a group of single-objective policies
Revised: For sampling from behavioral policy consists of a group of single-objective policies

Original: π β “ t π 1 , . .
Revised: π β “ t π 1 , . .

Original: , π B u with B being the total number of candidate policies, we recommend first find the expected unweighted raw rewards G π 1 , . . .
Revised: , π B u with B being the total number of candidate policies, we recommend first find the expected unweighted raw rewards G π 1 , . . .

Original: , G π B .
Revised: , G π B .

Original: Then, find the estimated ˆ ω π 1 , . . .
Revised: Then, find the estimated ˆ ω π 1 , . . .

Original: , ˆ ω π B by letting ˆ ω π
Revised: , ˆ ω π B by letting ˆ ω π

Original: b i “
Revised: b i “

Original: G πbi ř nj
Revised: G πbi ř nj

Original: “ 1 G πbj , which represents the estimated preference on i th objective of b th candidate policy.
Revised: “ 1 G πbj , which represents the estimated preference on i th objective of b th candidate policy.

Original: For a sampled preference ω „ Ω ˚ , use the policy that provides the smallest euclidean distance d p ω ,
Revised: For a sampled preference ω „ Ω ˚ , use the policy that provides the smallest euclidean distance d p ω ,

Original: ˆ ω π b q .
Revised: ˆ ω π b q .

Original: Empirically, this means picking the candidate policy that has the expected reward ratio closest to ω .
Revised: Empirically, this means picking the candidate policy that has the expected reward ratio closest to ω .

Original: B P REFERENCE S AMPLING D ETAILS
Revised: B D ATASET D ETAILS

Original: To uniformly sample the High-H data from the entire preference space, the problem is equivalent to sampling from a n -dimensional simplex, where n is the number of objectives.
Revised: To uniformly sample the High-H data from the entire preference space, the problem is equivalent to sampling from a n -dimensional simplex, where n is the number of objectives.

Original: The resulting sampling is:
Revised: The resulting sampling is:

Original: [Equation]
Revised: [Equation]

Original: We take the 1-norm following the exponential distribution to make sure each preference add up to 1.
Revised: We take the 1-norm following the exponential distribution to make sure each preference add up to 1.

Original: When Ω ˚ ‰ Ω , we perform rejection sampling to restrict the range.
Revised: When Ω ˚ ‰ Ω , we perform rejection sampling to restrict the range.

Original: [Figure]
Revised: [Table]

Original: To sample the Med-H and Low-H data, we first sample α from a non-negative uniform distribution, then sample the corresponding Dirichlet preference.
Revised: To sample the Med-H and Low-H data, we first sample α from a non-negative uniform distribution, then sample the corresponding Dirichlet preference.

Original: Here, we sample a different alpha to make sure the center of the Dirichlet changes and thus allows more variation.
Revised: Here, we sample a different alpha to make sure the center of the Dirichlet changes and thus allows more variation.

Original: [Equation]
Revised: [Equation]

Original:  where α 
Revised: ; where α „ Unif p

Original: [Equation] 
Revised: [Equation] [Equation]

Original:  ; where α „ 
Revised: f Dirichlet p α q ; where α „ Unif p 1 { 3 ˆ 10 6 ,

Original: [Equation]
Revised: [Equation]

Revised: { 3 ˆ 10 6 q (10)

Original: For sampling from behavioral policy consists of a group of single-objective policies
Revised: For sampling from behavioral policy consists of a group of single-objective policies

Original: π β “ t π 1 , . .
Revised: π β “ t π 1 , . .

Original: , π B u with B being the total number of candidate policies, we recommend first find the expected unweighted raw rewards G π 1 , . . .
Revised: , π B u with B being the total number of candidate policies, we recommend first find the expected unweighted raw rewards G π 1 , . . .

Original: , G π B .
Revised: , G π B .

Original: Then, find the estimated ˆ ω π 1 , . . .
Revised: Then, find the estimated ˆ ω π 1 , . . .

Original: , ˆ ω π B by letting ˆ ω π
Revised: , ˆ ω π B by letting ˆ ω π

Original: b i “
Revised: b i “

Original: [Equation] nj[Equation]
Revised: G πbi ř nj

Original:  G πbj , which represents the estimated preference on i th objective of b th candidate policy.
Revised: “ 1 G πbj , which represents the estimated preference on i th objective of b th candidate policy.

Original: For a sampled preference ω „ Ω ˚ , use the policy that provides the smallest euclidean distance d p ω ,
Revised: For a sampled preference ω „ Ω ˚ , use the policy that provides the smallest euclidean distance d p ω ,

Original: ˆ ω π b q .
Revised: ˆ ω π b q .

Original: Empirically, this means picking the candidate policy that has the expected reward ratio closest to ω .
Revised: Empirically, this means picking the candidate policy that has the expected reward ratio closest to ω .

Original: C E XPERT & A MATEUR D ETAILS
Revised: C E XPERT & A MATEUR D ATASETS

Original: In Expert collection , we sample trajectories using the fully-trained behavioral policy π β .
Revised: In Expert collection , we sample trajectories using the fully-trained behavioral policy π β .

Original: In this paper, we use PGMORL by Xu et al., 2020 as π β
Revised: In this paper, we use PGMORL by Xu et al., 2020 as π β

Original: [Equation]
Revised: [Equation]

Original: In the Amateur collection , the policies has a 35% chance being stochastic on top of the expert collection.
Revised: In the Amateur collection , the policies has a 35% chance being stochastic on top of the expert collection.

Original: Actions has a chance being stochastic, during which it is scaled from the expert action, as following:
Revised: Actions has a chance being stochastic, during which it is scaled from the expert action, as following:

Original: [Equation]
Revised: [Equation]

Original: In the MO-Swimmer environment only, we let actions has a 35% chance to be a uniform random sample from the entire action space rather than being the same as expert to increase variance and achieve a performance similar to amateur.
Revised: In the MO-Swimmer environment only, we let actions has a 35% chance to be a uniform random sample from the entire action space rather than being the same as expert to increase variance and achieve a performance similar to amateur.

Original: The resulting strategy for MO-Swimmer is:
Revised: The resulting strategy for MO-Swimmer is:

Original: [Equation]
Revised: [Equation]

Original: D R ELATED W ORK

Original: Multi-Objective Reinforcement Learning Predominant works in MORL focus on the online setting where the goal is to train agents that can generalize to arbitrary preferences.

Original: This can be achieved by training a single preference-conditioned policy (Yang et al., 2019; Parisi et al., 2016), or an ensemble of single-objective policies for a finite set of preferences (Mossalam et al., 2016b; Xu et al., 2020; Zhang & Li, 2007).

Original: Many of these algorithms consider vectorized variants of standard algorithms such as Q-learning (Mossalam et al., 2016b;

Original: Yang et al., 2019), often augmented with strategies to guide the policy ensemble towards the Pareto front using evolutionary or incrementally updated algorithms (Xu et al., 2020; Zhang & Li, 2007; Mossalam et al., 2016b; Roijers et al., 2014;

Original: Huang et al., 2022).

Original: In contrast to these online MORL works, our focus is on learning a single policy that works for all preferences using only offline datasets.

Original: There are also a few works that study offline MORL.

Original: Wu et al., 2021 propose a provably efficient algorithm based on dual gradient ascent.

Original: Satija et al., 2021 study learning of safe policies by extending the approach of Laroche et al., 2017 to the offline MORL setting.

Original: However, these algorithms assume the environments are primarily tabular MDPs with finite state and action spaces, while we are interested in high-dimensional MDPs with continuous states and actions.

Original: Multi-Task Reinforcement Learning MORL is also closely related to multi-task reinforcement learning, where every task can be interpreted as a distinct objective.

Original: There is an extensive body of work in learning multi-task policies both in the online and offline setups (Wilson et al., 2007; Lazaric & Ghavamzadeh, 2010; Teh et al., 2017) inter alia.

Original: However, the key difference is that typical MTRL benchmarks and algorithms do not consider solving multiple tasks that involve inherent trade-offs.

Original: Consequently, there is no notion of Pareto efficiency and an agent can simultaneously excel in all the tasks without accounting for user preferences.

Original: Reinforcement Learning Via Supervised Learning A body of recent works have formulated offline reinforcement learning as an autoregressive sequence modeling problem using Decision Transformers (DT)

Original: (Chen et al., 2021).

Original: The key idea in DT is to learn a transformer-based policy that conditions on the past history and a dynamic estimate of the returns (a.k.a.

Original: returns-to-go).

Original: Followup works consider simpler variants that rely only on multi-layer perceptrons (Emmons et al., 2022).

Original: E T RANING D ETAILS
Revised: D T RANING D ETAILS

Original: Common hyper-parameters have the same values across all models, except for the learning rate scheduler and warmup steps.
Revised: Common hyper-parameters have the same values across all models, except for the learning rate scheduler and warmup steps.

Original: In MODT family, inputs are embedded by a 1-layer MLP into Hidden
Revised: In MODT family, inputs are embedded by a 1-layer MLP into Hidden

Original: Size, and n layer represents the number of transformer blocks; in BC family, n layer represents the number of MLP layers to embedding each input; in MOR V S and MOR V S(P) family, the default embedding strategy are used, please check Emmons et al., 2022.
Revised: Size, and n layer represents the number of transformer blocks; in BC family, n layer represents the number of MLP layers to embedding each input; in MOR V S and MOR V S(P) family, the default embedding strategy are used, please check Emmons et al., 2022.

Original: Here, we consider MOR V S and
Revised: Here, we consider MOR V S and

Original: MOR V S(P) both have context length of 1 because they only use the current state to predict the next action, whereas MODT and BC use the past 20.
Revised: MOR V S(P) both have context length of 1 because they only use the current state to predict the next action, whereas MODT and BC use the past 20.

Original: E. P ARAMETERS
Revised: D. P ARAMETERS

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: Optimizer

Original: AdamW

Original: [Table]
Revised: [Table]

Original: lambda None lambda [Table] N/A
Revised: D. T RAINING S TEPS

Original: [Table]
Revised: [Table]

Original: ReLU E.  [Table] The exception is that we train MO-Swimmer and MO-Hopper-3obj under Low-H-Amateur for 40K and 120K steps respectively using MODT.
Revised: The exception is that we train MO-Swimmer and MO-Hopper-3obj under Low-H-Amateur for 40K and 120K steps respectively using MODT.

Original: E. A TTEMPTED MODT AND MODT(P) A RCHITECTURES
Revised: D. A TTEMPTED MODT AND MODT(P) A RCHITECTURES

Original: Here are all the MODT architectures we tried in our experiments.
Revised: Here are all the MODT architectures we tried in our experiments.

Original: We followed Case 4 in all of the experiment results as we find that it makes the model follows closely to the given preference.
Revised: We followed Case 4 in all of the experiment results as we find that it makes the model follows closely to the given preference.

Original: Consider ω as an independent token of the causal transformer.
Revised: Consider ω as an independent token of the causal transformer.

Original: Train a separate embedding for ω , concatenate the embeddings to get f ϕ s p s q À f ϕ ω p ω q , f ϕ a p a q À f ϕ ω p ω q , and f ϕ g p g q À f ϕ ω p ω q then pass into the transformer.
Revised: Train a separate embedding for ω , concatenate the embeddings to get f ϕ s p s q À f ϕ ω p ω q , f ϕ a p a q À f ϕ ω p ω q , and f ϕ g p g q À f ϕ ω p ω q then pass into the transformer.

Original: Add another MLP layer on top of the Case 2 after concatenation, then pass output into the transformer.
Revised: Add another MLP layer on top of the Case 2 after concatenation, then pass output into the transformer.

Original: Concatenate ω to other tokens before any layers.
Revised: Concatenate ω to other tokens before any layers.

Original: This means we have s ˚ “ s À ω , a ˚ “ a À ω , and g ˚ “ g À ω .
Revised: This means we have s ˚ “ s À ω , a ˚ “ a À ω , and g ˚ “ g À ω .

Original: F E VALUATION M ETRICS
Revised: E O THER E VALUATION M ETRICS

Original: Hypervolums & Sparsity We introduce the hypervolume and sparsity metric in this section with a visualization.

Original: Definition 1 (Hypervolume) .

Original: Hypervolume H p P q measures the space or volume enclosed by the solutions in the Pareto set P :

Original: [Equation]

Original: where H p P q “ t z P Z |D i : 1 ď i ď | P | , r ĺ z ĺ P p i qu .

Original: P p i q is the i th solution in P , ĺ is the dominance relation operator, and

Original: 1 H p P

Original: q p z q equals 1 if z P H p P q and 0 otherwise.

Original: Higher hypervolumes are better.

Original: Definition 2 (Sparsity) .

Original: Sparsity S p P q measures the density of the Pareto front covered by a Pareto set P :

Original: [Equation]

Original: where ˜ P i represents a list sorted as per the values of the i th objective in P and ˜ P i p k q is the k th value in the sorted list.

Original: Lower sparsity is better.

Original: See Figure 3 for an illustration and Appendix F for discussion on other possible metrics.

Original: Other Metrics.

Original: Among a variety of metrics for MORL, we use Hypervolume and Sparsity (SP) to benchmark models in this paper for several reasons.
Revised: Among a variety of metrics for MORL, we use Hypervolume and Sparsity (SP) to benchmark models in this paper for several reasons.

Original: First, metrics such as the ϵ -metric require prior knowledge of the true Pareto Fronts, which are not available for our MuJoCo Environments.
Revised: First, metrics such as the ϵ -metric require prior knowledge of the true Pareto Fronts, which are not available for our MuJoCo Environments.

Original: Second, due to limited resources, we only assume linear reward function and cannot collect real-time user feedback, thus utility-based metrics such as expected utility metric (EUM) are not applicable.
Revised: Second, due to limited resources, we only assume linear reward function and cannot collect real-time user feedback, thus utility-based metrics such as expected utility metric (EUM) are not applicable.

Original: Finally, it is easier to benchmark both our offline agents and behavioral policy on these metrics, allowing for informative comparisons.
Revised: Finally, it is easier to benchmark both our offline agents and behavioral policy on these metrics, allowing for informative comparisons.

Original: G A MATEUR R ESULTS

Original: Due to space limit

Original: [Table]

Original: we present the results for Amateur datasets with High-H entropy for all environments here in Table 3 and Table 4.

Original: We can achieve the same conclusion that PEDA variants including MODT(P) and MOR V S(P) have the best hypervolume and sparsity performance against other baselines.

Original: They also can approach or exceed the Amateur behavioral policy

Original: [Table]

Original: which shows that our algorithms are able to learn strong policies from considerable amount of suboptimal data rather than merely imitation learning agents

Original: [Table]

Original: [Table]

Original: H P ARETO F RONT & R ETURN D ISTRIBUTION
Revised: F P ARETO S ET V ISUALIZATIONS

Original: Pareto front approximation.

Original: We ablate how well the PEDA variants and other baselines can approximate the Pareto front through conditioning on different preference points.

Original: We show the Table 4: Sparsity ( Ó ) performance on High-H-Amateur dataset. We can see that all  modelsstill have a similar or stronger sparsity performance when trained on amateur datasets. Furthermore,
Revised: We present the Pareto Set visualizations for all of our models trained under each High-H dataset in

Original: MOR V S(P) still presents the strongest performance.

Original: While BC(P) has strong performance in MOHopper-3obj and MO-Swimmer, it also fails to give a dense solution in other environments and has a higher standard error.

Original: [Table] results in Figure 5, where we can see that the MODT(P) and MOR V S(P) can approximate the Pareto front, while having some dominated points colored in pink mostly in the MO-Hopper and
Revised: Figure 6.

Original: MO-Walker2d environments.

Original: The results are based on average of 3 seeds.

Original: The distribution of returns.

Original: We ablate how well MODT(P) and MOR V S(P) follow their given target return, based on a normalized and weighted value.

Original: We present the results in Figure 4 for

Original: MOR V S(P)

Original: under High-H-Expert datasets.

Original: Here, we see that the models follow the oracle line nicely when conditioned on target within the dataset distribution, and generalize to targets outside of the dataset distribution as well.

Original: We present the Pareto Set visualizations for all of our models trained under each High-H dataset in Figure 5.

Original: All subplots are based on 1 seed, in which we evaluate the model using 501 equally spaced preference points in 2 objective environment andequally spaced preference in the 3 objective environment from the full preference space.
Revised: All subplots are based on 1 seed, in which we evaluate the model using 501 equally spaced preference points in 2 objective environment and 351 equally spaced preference in the 3 objective environment from the full preference space.

Original: Since the environments are stochastically initialized, we evaluate 5 times at each preference point and take the median value.
Revised: Since the environments are stochastically initialized, we evaluate 5 times at each preference point and take the median value.

Original: We here allow a small tolerance on coloring the dominated points.
Revised: We here allow a small tolerance on coloring the dominated points.

Original: If a preference point is within the achievable preference Ω ˚ but the solution is dominated, we color it in red.
Revised: If a preference point is within the achievable preference Ω ˚ but the solution is dominated, we color it in red.

Original: Since our models condition on continuous preference points and environments are initializaed stochastically, we give a small tolerance (3%-8%) for points to be colored in blue.
Revised: Since our models condition on continuous preference points and environments are initializaed stochastically, we give a small tolerance (3%-8%) for points to be colored in blue.

Original: The hypervolume and sparsity metric, on the other hand, can based on strictly undominated solutions without tolerance. 
Revised: The hypervolume and sparsity metric, on the other hand, can based on strictly undominated solutions without tolerance. [Table] [Table]

Original: I M EDIUM & L OW E NTROPY D ATASET T RAINING
Revised: G M EDIUM & L OW E NTROPY D ATASET T RAINING

Original: We train on the Medium-Entropy and Low-Entropy datasets for the MO-HalfCheetah environment.
Revised: We train on the Medium-Entropy and Low-Entropy datasets for the MO-HalfCheetah environment.

Original: Overall, models have a similar performance under Med-H and High-H datasets, but suffers when only trained on Low-H .
Revised: Overall, models have a similar performance under Med-H and High-H datasets, but suffers when only trained on Low-H .

Original: We present the results in Table 5, in which we illustrate that the Low-H dataset has a worse expert and amateur performance due to reduced variability on preference.
Revised: We present the results in Table 6, in which we illustrate that the Low-H dataset has a worse expert and amateur performance due to reduced variability on preference.

Original: However, MODT(P) and MOR V S(P) are still able to get close or exceed in hypervolume on all datasets, which showcase the effectiveness of PEDA as an efficient MORL policy.
Revised: However, MODT(P) and MOR V S(P) are still able to get close or exceed in hypervolume on all datasets, which showcase the effectiveness of PEDA as an efficient MORL policy.

Original: Results are based on average of 3 seeds with the standard error given.
Revised: Results are based on average of 3 seeds with the standard error given.

Original: J T RAINING WITH 1- DIM RTG
Revised: H T RAINING WITH 1- DIM RTG

Original: We attempted to train MODT and RvS with 1-dim return-to-go rather than a separate rtg for each objective.
Revised: We attempted to train MODT and RvS with 1-dim return-to-go rather than a separate rtg for each objective.

Original: According to results on MO-HalfCheetah and the High-H datasets in 6, using multidimensional rtg enhances the performance of MODT(P), and are about the same for MOR V S(P) when preference are concatenated to states.
Revised: According to results on MO-HalfCheetah and the High-H datasets in 7, using multidimensional rtg enhances the performance of MODT(P), and are about the same for MOR V S(P) when preference are concatenated to states.

Original: However, it reduces standard error significantly in both
Revised: However, it reduces standard error significantly in both

Original: [Table]

Original: [Table]

Original: MODT(P) and MOR V S(P).
Revised: MODT(P) and MOR V S(P).

Original: In the naive models when preference are not concatenated to states, using a multi-dimensional rtg helps to achieve a much more competitive hypervolume.
Revised: In the naive models when preference are not concatenated to states, using a multi-dimensional rtg helps to achieve a much more competitive hypervolume.

Original: We thus believe multi-dimensional rtg conveys important preference information when the model doesn’t directly take preference as an input.
Revised: We thus believe multi-dimensional rtg conveys important preference information when the model doesn’t directly take preference as an input.

Original: Results are based on average of 3 seeds with the standard error given.
Revised: Results are based on average of 3 seeds with the standard error given.

Original: K O VERALL D ATA G ENERATION P IPELINE

Original: The pseudocode for generating the dataset is described in Algorithm 1.

Original: Given a preference distribution, we first sample a preference ω and query the closest behavioral policy in either the amateur/expert ensemble matching ω .

Original: We rollout this policy for T time steps (or until the end of an episode if sooner) and record the state, action, and reward information.

Original: Each trajectory in our dataset is represented as:

Original: [Equation]

Original: For every environment in D4MORL, we collect 50K trajectories of length T “ 500 for both expert and amateur trajectory distributions under each of the 3 preference distributions.

Original: Overall, this results in a total of 1.8M trajectories over all 6 environments, which corresponds to roughly 867M time steps.

Original: Please refer to Table 7 in for additional statistics on the dataset.

Original: procedure C OLLECT (prefDist, nTraj, env, pretrainedAgents, T) agents = pretrainedAgents prefs =

Original: prefDist(nTraj) all trajs =

Original: [] for ω in prefs do agent

Original: = closestAgent(agents, ω ) s = env.reset() done =

Original: False τ = [ ω ] t =while (NOT done) AND (t ă T) do a = agent.get action( s ) s 1 , done, r = env.step( a ) append s , a , s 1 , r to τ s = st = t

Original: +append τ to all trajs return all trajs

Original: [Table]