P ARETO -E FFICIENT D ECISION A GENTS FOR O FFLINE M ULTI -O BJECTIVE R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
The goal of multi-objective reinforcement learning (MORL) is to learn policies that simultaneously optimize multiple competing objectives.
In practice, an agent’s preferences over the objectives may not be known apriori, and hence, we require policies that can generalize to arbitrary preferences at test time.
In this work, we propose a new data-driven setup for offline MORL, where we wish to learn a preference-agnostic policy agent using only a finite dataset of offline demonstrations of other agents and their preferences.
The key contributions of this work are two-fold.
First, we introduce D4MORL, (D)atasets for MORL that are specifically designed for offline settings.
It contains 1.8 million annotated demonstrations obtained by rolling out reference policies that optimize for randomly sampled preferences on 6 MuJoCo environments with 2-3 objectives each.
Second, we propose Pareto-Efficient Decision Agents (PEDA), a family of offline
MORL algorithms that builds and extends return-conditioned offline methods including Decision Transformers (Chen et al., 2021) and RvS (Emmons et al., 2022) via a novel preference-and-return conditioned policy.
Empirically, we show that PEDA closely approximates the behavioral policy on the D4MORL benchmark and provides an excellent approximation of the Pareto-front with appropriate conditioning, as measured by the hypervolume and sparsity metrics.
I NTRODUCTION
We are interested in learning agents for multi-objective reinforcement learning (MORL) that optimize for one or more competing objectives.
This setting is commonly observed in many real-world scenarios.
For instance, an autonomous driving car might trade off high speed and energy savings depending on the user’s preferences.
If the user has a relatively high preference for speed, the agent will move fast regardless of power usage; on the other hand, if the user tries to save energy, the agent will keep a more steady speed.
One key challenge with MORL is that different users might have different preferences on the objectives and systematically exploring policies for each preference might be expensive, or even impossible.
In the online setting, prior work considers several approximations based on scalarizing the vector-valued rewards of different objectives based on a single preference (Lin, 2005), learning an ensemble of policies based on enumerating preferences (Mossalam et al., 2016a, Xu et al., 2020), or extensions of single-objective algorithms such as Q-learning to vectorized value functions (Yang et al., 2019).
We introduce the setting of offline multi-objective reinforcement learning for high-dimensional state and action spaces, where our goal is to train an MORL policy agent using an offline dataset of demonstrations from multiple agents with known preferences.
Similar to the single-task setting, offline MORL can utilize auxiliary logged datasets to minimize interactions when deploying agents, thus improving data efficiency and minimizing interactions when deploying agents in high-risk settings.
In addition to its practical utility, offline RL (Levine et al., 2020) has enjoyed major successes in the last few years (Kumar et al., 2020, Kostrikov et al., 2021, Chen et al., 2021) on challenging high-dimensional environments for continuous control and game-playing.
Our contributions in this work are two-fold in introducing benchmarking datasets and a new family of MORL, as described below.
We introduce Datasets for Multi-Objective Reinforcement Learning (D4MORL), a collection of 1. million trajectories on 6 multi-objective MuJoCo environments (Xu et al., 2020).
Here, 5 environ- ments consist of 2 objectives and 1 environment consists of 3 objectives.
For each environment in D4MORL, we collect demonstrations from 2 pretrained behavioral agents: expert and amateur , where the relative expertize is defined in terms of the Pareto-efficiency of the agents and measured empirically via their hypervolumes.
Furthermore, we also include 3 kinds of preference distributions with varying entropies to expose additional data-centric aspects for downstream benchmarking.
Lack of MORL datasets and large-scale benchmarking has been a major challenge for basic research (Hayes et al., 2021), and we hope that D4MORL can aid future research in the field.
Next, we propose Pareto-Efficient Decision Agents (PEDA), a family of offline MORL algorithms that extends return-conditioned methods including Decision Transformer (DT) (Chen et al., 2021) and RvS (Emmons et al., 2022) to the multi-objective setting.
These methods learn a returnconditioned policy via a supervised loss on the predicted actions.
In recent work, these methods have successfully scaled to agents that demonstrate broad capabilities in multi-task settings (Lee et al.,
Reed et al., 2022).
For MORL, we introduce a novel preference and return conditioned policy network and train it via a supervised learning loss.
At test time, naively conditioning on the default preferences and maximum possible returns leads to out-of-distribution behavior for the model, as neither has it seen maximum returns for all objectives in the training data nor is it possible to simultaneously maximize all objectives under competition.
We address this issue by learning to map preferences to appropriate returns and hence, enabling predictable generalization at test-time.
Empirically, we find PEDA performs exceedingly well on D4MORL and closely approximates the reference Pareto-frontier of the behavioral policy used for data generation.
In the multi-objective
HalfCheetah environment, compared with an average upper bound on the hypervolume of 5 .
79 ˆ 10achieved by the behavioral policy, PEDA achieves an average hypervolume of 5 .
77 ˆ 10 6 on the
Expert and 5 .
76 ˆ 10 6 on the Amateur datasets.
R ELATED W ORK
Multi-Objective Reinforcement Learning Predominant works in MORL focus on the online setting where the goal is to train agents that can generalize to arbitrary preferences.
This can be achieved by training a single preference-conditioned policy (Yang et al., 2019; Parisi et al., 2016)), or an ensemble of single-objective policies for a finite set of preferences (Mossalam et al., 2016b;
Xu et al., 2020; Zhang & Li, 2007)).
Many of these algorithms consider vectorized variants of standard algorithms such as Q-learning (Mossalam et al., 2016b;
Yang et al., 2019), often augmented with strategies to guide the policy ensemble towards the Pareto front using evolutionary or incrementally updated algorithms (Xu et al., 2020; Zhang & Li, 2007; Mossalam et al., 2016b; Roijers et al., 2014; Huang et al., 2022)).
Other approaches have also been studied, such as framing MORL as a meta-learning problem (Chen et al., 2018), learning the action distribution for each objective (Abdolmaleki et al., 2020), and learning the relationship between objectives (Zhan & Cao, 2019) among others.
In contrast to these online MORL works, our focus is on learning a single policy that works for all preferences using only offline datasets.
There are also a few works that study decision making with multiple objectives in the offline setting and sidestep any interaction with the environments.
Wu et al., 2021 propose a provably efficient offline MORL algorithm for tabular MDPs based on dual gradient ascent.
Satija et al., 2021 study learning of safe policies by extending the approach of Laroche et al., 2017 to the offline MORL setting.
Their proposed algorithm assumes knowledge of the behavioral policy used to collect the offline data and is demonstrated primarily on tabular MDPs with finite state and action spaces.
In contrast, we are interested in developing dataset benchmarks and algorithms for scalable offline policy optimization in high-dimensional MDPs with continuous states and actions.
Multi-Task Reinforcement Learning MORL is also closely related to multi-task reinforcement learning, where every task can be interpreted as a distinct objective.
There is an extensive body of work in learning multi-task policies both in the online and offline setups (Wilson et al., 2007; Lazaric & Ghavamzadeh, 2010; Teh et al., 2017) inter alia.
However, the key difference is that typical MTRL benchmarks and algorithms do not consider solving multiple tasks that involve inherent trade-offs.
Consequently, there is no notion of Pareto efficiency and an agent can simultaneously excel in all the tasks without accounting for user preferences.
Reinforcement Learning Via Supervised Learning A body of recent works have formulated offline reinforcement learning as an autoregressive sequence modeling problem using Decision Transformers (DT) or Trajectory Transformers ( Chen et al., 2021, Janner et al., 2021)
The key idea in
DT is to learn a transformer-based policy that conditions on the past history and a dynamic estimate of the returns (a.k.a.
returns-to-go).
Follow-up works consider online learning (Zheng et al., 2022) as well as simpler variants that rely only on multi-layer perceptrons (Emmons et al., 2022).
Such agents are generally more stable and robust to optimize due to the simplicity of loss function and easier to scale to more complex settings such as environments with high-dimensional actions or states, as shown in recent works in multi-task RL (Lee et al., 2022;
Reed et al., 2022).
P RELIMINARIES
Setup and Notation.
We operate in the general framework of multi-objective Markov decision process (MOMDP) with linear preferences (Wakuta, 1995).
An MOMDP is represented by the tuple x S , A , P , R , Ω , f, γ y .
At each timestep t , the agent with a current state s t P S takes an action a t P A to transition into a new state s t ` 1 with probability P p s t ` 1 | s t , a t q and observes a reward vector r t “ R p s t , a t q P R n .
Here, n is the number of objectives.
The vector-valued return R P R n of an agent is given by the discounted sum of reward vectors over a time horizon, R “ ř t γ t r t .
We also assume that there exists a linear utility function f and a space of preferences Ω that can map the reward vector r t and a preference vector ω P Ω to a scalar reward r t , i.e., r t “ f p r t , ω q “ ω ⊺ r t .
The expected vector return of a policy π is given an G π “ r G π 1 , G π 2 , . . .
, G πn s ⊺ where the expected return of the i th objective is given as G πi “ E a t ` 1 „ π p¨| s t ,
ω q r ř t R p s t , a t q i s for some predefined time horizon and preference vector ω .
The goal is to train a multi-objective policy π p a | s , ω
q such that the expected scalarized return ω ⊺ G π “ E r ω ⊺ ř t R p s t , a t qs is maximized.
Pareto Optimality.
In MORL, one cannot optimize all objectives simultaneously, so policies are evaluated based on the Pareto set of their vector-valued expected returns.
Consider a preference-conditioned policy π p a | s , ω q that is evaluated for m distinct preferences ω 1 , . .
ω m , and let the resulting policy set be represented as t π p u p “ 1 ,...,m , where π p “ π p a | s , ω “ ω p q , and G π p is the corresponding unweighted expected return.
We say the solution G π p is dominated by G π q when there is no objective for which π q is worse than π p , i.e., G π p
i
ă G π q i for @
i P r 1 , 2 , . .
. , n s .
If a solution is not dominated, it is part of the Pareto set denoted as
P .
The curve traced by the solutions in a Pareto set is also known as the Pareto front.
In
MORL, our goal is to define a policy such that its empirical Pareto set is a good approximation of the true Pareto front.
While we do not know the true Pareto front for many problems, we can define metrics for relative comparisons between different algorithms.
Specifically, we evaluate a Pareto set P based on two metrics, hypervolume and sparsity that we describe next.
Definition 1 (Hypervolume) .
Hypervolume H p P q measures the space or volume enclosed by the solutions in the Pareto set P :
[Equation]
where H p P q “ t z P Z |D i : 1 ď i ď | P | , r ĺ z ĺ
P p i qu .
P p i q is the i th solution in P , ĺ is the dominance relation operator, and
1 H p P
q p z q equals 1 if z P H p P q and 0 otherwise.
Higher hypervolumes are better.
Definition 2 (Sparsity) .
Sparsity S p P q measures the density of the Pareto front covered by a Pareto set P :
[Equation]
where ˜ P i represents a list sorted as per the values of the i th objective in P and ˜ P i p k q is the k th value in the sorted list.
Lower sparsity is better.
See Figure 1 for an illustration and Appendix E for discussion on other possible metrics.
D4MORL: D ATASETS FOR O FFLINE M ULTI -O BJECTIVE R EINFORCEMENT L EARNING
In offline RL, the goal of an RL agent is to learn the optimal policy using a fixed dataset without any interactions with the environment (Levine et al., 2020).
This perspective brings RL closer to supervised learning, where the presence of large-scale datasets has been foundational for further progress in the field.
Many such data benchmarks exist for offline RL as well; a notable one is the D4RL (Fu et al., 2020) benchmark for continuous control which has led to the development of several state-of-the-art offline RL algorithms (Kostrikov et al., 2021; Kumar et al., 2020; Chen et al., 2021) that can scale favorably even in high dimensions.
To the best of our knowledge, there are no such existing benchmarks for offline MORL.
Even for the online setting, most works in MORL conduct evaluations on toy MDPs (e.g., gridworld) with a few exceptions that include continuous control benchmarks, e.g., Chen et al.
Xu et al.
This calls for a much needed push towards more challenging benchmarks for reliable evaluation of MORL, especially in the offline setting.
In this paper, we introduce Datasets for Multi-Objective Reinforcement Learning (D4MORL), a large-scale benchmark for offline MORL.
Our benchmark consists of offline trajectories frommulti-objective MuJoCo environments including 5 environments with 2 objectives each viz.
MOAnt, MO-HalfCheetah, MO-Hopper, MO-Swimmer, MO-Walker2d, and one environment with three objectives: MO-Hopper-3obj.
The objectives are conflicting for each environment; for instance, the two objectives in MO-Hopper correspond to jumping and running; in MO-HalfCheetah, MOSwimmer, and MO-Walker2d, they correspond to the speed and energy savings of the agent.
See
Appendix A for more details on the semantics of the target objectives for each environment.
These environments were first introduced in Xu et al.
(2020) for online MORL, and as such, we use their pretrained ensemble policies as building blocks for defining new behavioral polices for dataset collection, which we discuss next.
4. T RAJECTORY S AMPLING
The quality of the behavioral policy used for sampling trajectories in the offline dataset is a key factor for benchmarking downstream offline RL algorithms.
In existing benchmarks for single-objective
RL such as D4RL (Fu et al., 2020), the quality of a behavioral policy can be ascertained and varied based on its closeness to a single expert policy, as measured by their scalar-valued returns.
For a
MOMDP, we do not have the notion of a scalar return and hence, a reference expert policy (or set of policies) should reflect the optimal returns for all possible preferences in the preference space.
We use Prediction-Guided Multi-Objective Reinforcement Learning (PGMORL), a state-of-the-art
MORL algorithm for defining reference expert policies.
PGMORL (Xu et al., 2020) uses evolutionary algorithms to train an ensemble of policies to approximate the Pareto set.
Each reference policy in the ensemble is associated with a unique preference; for any new preference, it is mapped to the closest preference in the reference set.
The number of policies in the ensemble can vary significantly; for instance, we have roughly 70 reference policies for MO-Antand 2445 policies for harder environments such as MO-Hopper-3obj.
Given a desired preference, we define two sets of behavioral policies:
Expert Dataset: We find the best reference policy in the policy ensemble, and always follow the action taken by the reference policy.
Amateur Dataset: As before, we first find the best reference policy in the policy ensemble.
With a fixed probability p , we randomly perturb the actions of the reference policies.
Otherwise, with probability 1 ´ p , we take the same action as the reference policy.
In D4MORL, we set p “ 0 . 65 .
Further details are described in Appendix C. In Figure 2, we show the returns of the trajectories rolled out from the expert and amateur policies for the 2 objective environments evaluated for a uniform sampling of preferences.
We can see that the expert trajectories typically dominate the amateur trajectories, as desired.
For the amateur trajectories, we see more diversity in the empirical returns for both objectives under consideration.
The return patterns for the amateur trajectories vary across different environments providing a diverse suite of datasets in our benchmark.
[Figure]
[Figure]
4. P REFERENCE S AMPLING
The coverage of any offline dataset is an important factor in dictating the performance of downstream offline RL algorithms (Levine et al., 2020).
For MORL, the coverage depends on both the behavioral
MORL policy as well as the distribution of preferences over which this policy is evaluated.
We use the following protocols for sampling from the preference space Ω .
First, we restrict our samples to lie within a physically plausible preference space Ω ˚ Ď Ω covered by the behavioral policy π β .
For instance, MO-Hopper has two objectives: jumping and running.
Since the agent can never gain running rewards without leaving the floor.
Thus, the preference of 100% running and 0% jumping is not achievable and excluded from our preference sampling distribution.
Second, we are primarily interested in offline trajectories that emphasize competition between multiple objectives rather than focusing on a singular objective.
To enforce this criteria, we define 3 sampling distributions concentrated around the centroid of the preference simplex.
The largest spread distribution samples uniformly from Ω ˚ and is denoted as High-Entropy ( High-H ).
Next, we have a Medium-Entropy ( Med-H ) distribution specified via samples of Dirichlet distributions with large values of their concentration hyperparameters (aka α ).
Finally, we have a Low-Entropy ( Low-H ) distribution that is again specified via samples of Dirichlet distributions, but with low values of their concentration hyperparameters.
We illustrate the samples for each of the preference distributions along with their empirical entropies in Figure 3.
Further details on the sampling distributions are deferred to Appendix B. By ensuring different levels of coverage, we can test the generalizability of an MORL policy to preferences unseen during training.
In general, we expect Low-H to be the hardest of the three distributions due to its restricted coverage, followed by Med-H and High-H .
Overall Data Generation Pipeline.
The pseudocode for generating the dataset is described in
Algorithm 1.
Given a preference distribution, we first sample a preference ω and query the closest behavioral policy in either the amateur/expert ensemble matching ω .
We rollout this policy for
T time steps (or until the end of an episode if sooner) and record the state, action, and reward information.
Each trajectory in our dataset is represented as:
[Equation]
Algorithm procedure C OLLECT (prefDist, nTraj, env, pretrainedAgents, T) agents =
pretrainedAgents prefs =
prefDist(nTraj) all trajs =
[] for ω in prefs do agent
= closestAgent(agents, ω ) s = env.reset() done =
False τ = [ ω ] t =while (NOT done) AND (t ă T) do a = agent.get action( s ) s 1 , done, r = env.step( a ) append s , a , s 1 , r to τ s = st = t
+append τ to all trajs return all trajs
For every environment in D4MORL, we collect 50K trajectories of length T “ 500 for both expert and amateur trajectory distributions under each of the 3 preference distributions.
Overall, this results in a total of 1.8M trajectories over all 6 environments, which corresponds to roughly 867M time steps.
We refer the reader to Table 5 in Appendix
B for additional statistics on the dataset.
P ARETO -E FFICIENT D ECISION A GENTS (PEDA)
In this section, we propose Pareto-Efficient Decision Agents (PEDA) , a family of offline multiobjective RL agents that aims to achieve Paretoefficiency by extending Decision Transformers (Chen et al., 2021) into multi-objective setting.
We first introduce the architecture of Decision Transformers (DT) and its variant, Reinforcement Learning Via Supervised Learning (RvS), followed by our modifications extending them to the multi-objective setting.
DT casts offline RL as a conditional sequence modeling problem that predicts the next action by conditioning a transformer on past states, actions, and desired returns.
The desired returns are defined as returns-to-go (RTG) g t “ ř Tt 1 “ t r t 1 , the future returns that this action is intended to achieve.
Therefore, the trajectory is represented by τ “ă s 1 , a 1 , g 1 , . . .
, s T , a T , g T ą .
In practice, we use a causally masked transformer architecture such as GPT (Radford et al., 2019) to process this sequence and predict the actions by observing the past K timesteps consisting of 3 K tokens.
DT and its variants have been shown to be more stable and robust to optimize due to the simplicity of loss function; easier to scale to more complex settings such as environments with high-dimensional actions or states, and agents with broad capabilities such as multitask settings (Lee et al., 2022).
Hence, we adopt Decision Transformers (Chen et al., 2021) as the representative base algorithm on which we build our work.
In follow-up work, Emmons et al.
(2022) extend DT and shows that even multi-layer perceptrons conditioned on the average returns-to-go can achieve similar performance without the use of transformers.
They call their model as Reinforcement Learning Via Supervised Learning (RvS).
However, RvS is generally not very stable when conditioned on very large returns, unlike DT.
5. M ULTI -O BJECTIVE R EINFORCEMENT L EARNING V IA S UPERVISED L EARNING
In PEDA, our goal is to train a single preference-conditioned agent for offline MORL.
By including preference conditioning, we enable the policy to be trained on arbitrary offline data, including trajectories collected from behavioral policies that are associated with alternate preferences.
To parameterize our policy agents, we extend the DT and RvS architectures to include preference tokens and vector-valued returns.
We refer to such preference-conditioned extensions of these architectures as MODT(P) and MOR V S(P) respectively, which we describe next.
Preference Conditioning.
Naively, we can easily incorporate the preference ω into DT by adding this token for each timestep and feeding it a separate embedding layer.
However, empirically we find that such a model design tends to ignore ω and the correlation between the preferences and predicted actions is weak.
Therefore, we propose to concatenate ω to other tokens before any layers in MODT(P).
Concretely, we define s ˚ “ s À ω , a ˚ “ a À ω , and g ˚ “ g À ω where À denotes the concatenation operator.
Hence, triples of s ˚ , a ˚ , g ˚ form the new trajectory.
As for
MOR V S(P), we concatenate the preference with the states and the average RTGs by default and the network interprets everything as one single input.
Multi-Objective Returns-to-Go.
Similar to RTG for the single objective case, we can define vector-valued RTG as g t “ ř Tt 1 “ t r t 1 Given a preference vector ω , we can scalarize the total returns-to-go as ˆ g t “ ω T g t .
In principle, the scalarized RTG ˆ g t can be recovered given the preference vector ω and the vector-valued RTG g t .
However, empirically we find that directly feeding MODT/MOR V S with the preference-weighted RTG vector g t d ω is slightly preferable for stable training, where d denotes the elementwise product operator.
Another unique challenge in the MORL setting concerns the scale of different objectives.
Since different objectives can signify different physical quantities (e.g., energy and speed), the choice of scaling can influence policy optimization.
We adopt a simple normalization scheme, where the returns for each objective are normalized by subtracting the minimum observed value for that objective and dividing it by the range of values (max-min).
Note that the maximum and minimum are computed based on the offline dataset and hence, they are not necessarily the true min/max objective values.
Post this normalization, the values for every objective in the trajectory are on the same scale between 0 and 1.
For evaluating the hypervolume and sparsity, we use the unnormalized values so that we can make comparisons across different datasets that may have different min/max boundaries.
Training.
We follow a simple supervised training procedure where we train the policies on randomly sampled mini-batches with MSE loss (for continuous actions).
In MODT and MODT(P), the input states, actions, and returns-to-go (with concatenated preferences) are treated as tokens and embedded through one layer of MLP.
We apply a layer of MLP and Tanh on the last hidden state of GPT-2 transformer to predict next action.
In MOR V S and MOR V S(P), we use only information from the current timestep and MLP layers to predict the next action.
E XPERIMENTS
In this section, we evaluate the performance of PEDA on D4MORL benchmark.
First, we investigate the benefits of preference conditioning by evaluating on decision transformers (DT) and RvS (MOR V S) where no preference information is available and we scalarize multi-objective vector returns into weighted sums.
We denote our methods with preference conditioning as MODT(P) and MOR V S(P).
Second, we compare our methods with classic imitation learning and temporal difference learning algorithms with preference conditioning.
Imitation learning.
Imitation learning simply uses supervised loss to train a mapping from states (w/ or w/o concatenating preferences) to actions.
We use behavioral cloning (BC) here and train multi-layer MLPs as models named BC (w/o preference) and BC(P) (w/ preference).
Temporal difference learning.
Conservative Q-Learning (CQL) (Kumar et al., 2020) is the stateof-the-art standard offline RL method, which learns a conservative Q-function f : S ˆ A Ñ R through neural networks.
We modify the network architecture such that it also takes preference vectors as inputs to learn a preference-conditioned Q-function f ˚ : S ˆ A ˆ Ω Ñ R .
We denote this method as CQL(P).
6. M ULTI - OBJECTIVE O FFLINE B ENCHMARK
[Table]
Hypervolume.
We compare hypervolume of our methods with all baselines on expert datasets in
Table 1 as well as amateur dataset in Table 2.
For the two-objective environments [Table] we evaluate the
[Table]
performance on High-H-Expert dataset.
MODT(P)
and MOR V S(P) have a lower density.
BC(P) also has a competitive sparsity in smaller environments such as Swimmer.
[Table]
models on 501 equally spaced preference points in the range [0, 1]; on the three-objective environment MO-Hopper-3obj, models are evaluated on 325 equally spaced points.
Each point is evaluated 5 times with random environment re-initialization, and the median value is recorded.
Finally, all the results are based on 3 random seeds and we report the mean performance along with the standard error.
In Table 1 and Table 2, we can see that MODT(P) and MOR V S(P) outperform other baselines and has a relatively very low standard error.
Also, PEDA variants including MODT(P) and
MOR V S(P) approaches the behavioral policy upper-bound.
Sparsity.
We also evaluate sparsity performance.
Since sparsity comparison is only meaningful between models that are sensitive to preference and have a relatively similar hypervolume performance, we only show results for models that concatenate preference.
Overall, MOR V S(P) has the lowest sparsity in most environments, while at the same time featuring an outstanding hypervolume.
6. A BLATION S TUDY
Pareto front approximation.
We ablate how well the MODT(P) and MOR V S(P) can approximate the Pareto front through conditioning on different preference points.
We show the results in
Figure 4, where we can see that the models can approximate the Pareto front, while having some dominated points colored in pink mostly in the MO-Hopper and MO-Walker2d environments.
The results are based on average of 3 seeds, and the full plot can be found in Appendix F.
[Table]
In addition, MO-Hopper and MO-Walker2d present to be the most challenging environments for
PEDA variants, featuring more dominated solutions than other environments.
The distribution of returns.
We ablate how well MODT(P) and MOR V S(P) follow their given target return, based on a normalized and weighted value.
We present the results in Figure 5 for
MOR V S(P)
under High-H-Expert datasets and refer to Appendix G for full settings.
Here, we see that the models follow the oracle line nicely when conditioned on target within the dataset distribution, and generalize to targets outside of the dataset distribution as well.
C ONCLUSION
In this paper, we introduced a novel dataset benchmark and algorithms for offline Multi-Objective
Reinforcement Learning.
We first proposed D4MORL, a dataset benchmark consisting of offline datasets generated from behavioral policies of different fidelities (expert/amateur) and rolled out under preference distributions with varying entropies (high/medium/low).
Then, we propose PEDA, a family of offline MORL policy optimization algorithms based on decision transformers and show that by concatenating and embedding preference together with other inputs, our policies can effectively approximate the Pareto front of the underlying behavioral policy as measured by the hypervolume and sparsity metrics.
Our proposed family includes MLP and transformer based variants, viz.
the MOR V S(P) and MODT(P), with MOR V S(P) performing the best overall.
In some scenarios, the learned policies can also generalize to higher target rewards that exceeds the data distribution.
To our knowledge, the PEDA variants are the first offline MORL policies that supports both the continuous action control and continuous preference space.
R EPRODUCIBILITY S TATEMENT
In this paper we make sure the experiment results are reproducible through training the models on multiple seeds and evaluating each preference point for multiple times.
We report the mean value with standard error in all of our experiment result tables.
Furthermore, we upload our model code along with our submission, and sample data can be found in this anonymous Google Drive folder: https://drive.google.com/drive/folders/ 1FiF5xmCSJ2vL_frLYmeZNc_nUrhRfUXC?usp=sharing .
Due to storage limits, we currently release only a subset of our data that can be used directly for training; we will release our full datasets in the final version.
A E NVIRONMENT D ESCRIPTION
All environments are the same as in Xu et al., 2020, except for when resetting the environment, each parameter are uniformly sampled from the r x ´ 10 ´ 3 , x ` 10 ´ 3 s with x being the default value.
The only exception is that MO-Hopper and MO-Hopper-3objalways reset height the same way as 1 . 25 , since this parameter directly relates to the reward function.
All environments have a max episode length of 500 steps per trajectory, but the agent may also die before reaching the maximum length.
A. MO-A NT
The two objectives in MO-Ant are achieved distance in x and y axes respectively, denoted as r “ r r vxt , r vyt s ⊺ .
Consider the position of the agent is represented as p x t , y t q at time t and takes the action a t .
The agent has a fixed survival reward r s “ 1 . 0 , dt “ 0 . 05 , and an action cost of r a “ 12 ř k a 2 k .
The rewards are calculated as:
[Equation]
A. MO-H ALF C HEETAH
The two objectives in MO-HalfCheetah are running speed, and energy saving, denoted as r “ r r vt , r et s ⊺ .
Consider the position of the agent is represented as p x t , y t q at time t and takes the action a t .
The agent has a fixed survival reward r s “ 1 . 0 , fixed dt “ 0 . 05 , and an action cost of r a “ ř k a 2 k .
The rewards are calculated as:
[Equation]
A. MO-H OPPER
The two objectives in MO-Hopper are running and jumping, denoted as r “ r r r , r j s ⊺ .
Consider the position of the agent is represented as p x t , h t q at time t and takes the action a t .
The agent has a fixed survival reward r s “ 1 . 0 , a fixed initial height as h init “ 1 . 25 , a fixed dt “ 0 . 01 , and an action cost of r a “ 2 ˆ 10 ´ 4 ř k a 2 k .
The rewards are calculated as:
[Equation]
A. MO-H OPPER -3 OBJ
The physical dynamics are the same in MO-Hopper and MO-Hopper-3obj, while this environment has 3 objectives: running, jumping, and energy saving.
The rewards are denoted as r “ r r r , r j , r e s ⊺ .
Consider the position of the agent is represented as p x t , h t q at time t and takes the action a t .
The agent has a fixed survival reward r s “ 1 . 0 , a fixed initial height as h init “ 1 . 25 , a fixed dt “ 0 . 01 , and an action cost of r a “ ř k a 2 k .
The rewards are calculated as:
[Equation]
A. MO-S WIMMER
The two objectives inMO-Swimmer are speed and energy saving, denoted as r “ r r v , r e s ⊺ .
Consider the position of the agent is represented as p x t , y t q at time t and takes the action a t .
The agent has a fixed dt “ 0 . 05 , and an action cost of r a “ ř k a 2 k .
The rewards are calculated as:
[Equation]
x
[Equation]
A. MO-W ALKER 2 D
The objectives in MO-Walker2d are speed and energy saving, denoted as r “ r r v , r e s ⊺ .
Consider the position of the agent is represented as p x t , y t q at time t and takes the action a t .
The agent has a fixed survival reward r s “ 1 . 0 , a fixed dt “ 0 . 008 , and an action cost of r a “ ř k a 2 k .
The rewards are calculated as:
[Equation]
To uniformly sample the High-H data from the entire preference space, the problem is equivalent to sampling from a n -dimensional simplex, where n is the number of objectives.
The resulting sampling is:
[Equation]
We take the 1-norm following the exponential distribution to make sure each preference add up to 1.
When Ω ˚ ‰ Ω , we perform rejection sampling to restrict the range.
To sample the Med-H and Low-H data, we first sample α from a non-negative uniform distribution, then sample the corresponding Dirichlet preference.
Here, we sample a different alpha to make sure the center of the Dirichlet changes and thus allows more variation.
[Equation]
where α
[Equation]
Unif
[Equation]
q ; where α „ Unif p 1 {
[Equation] [Equation]
[Equation]
For sampling from behavioral policy consists of a group of single-objective policies
π β “ t π 1 , . .
, π B u with B being the total number of candidate policies, we recommend first find the expected unweighted raw rewards G π 1 , . . .
, G π B .
Then, find the estimated ˆ ω π 1 , . . .
, ˆ ω π B by letting ˆ ω π
b i “
G πbi ř nj
“ 1 G πbj , which represents the estimated preference on i th objective of b th candidate policy.
For a sampled preference ω „ Ω ˚ , use the policy that provides the smallest euclidean distance d p ω ,
ˆ ω π b q .
Empirically, this means picking the candidate policy that has the expected reward ratio closest to ω .
B D ATASET D ETAILS
To uniformly sample the High-H data from the entire preference space, the problem is equivalent to sampling from a n -dimensional simplex, where n is the number of objectives.
The resulting sampling is:
[Equation]
We take the 1-norm following the exponential distribution to make sure each preference add up to 1.
When Ω ˚ ‰ Ω , we perform rejection sampling to restrict the range.
[Table]
To sample the Med-H and Low-H data, we first sample α from a non-negative uniform distribution, then sample the corresponding Dirichlet preference.
Here, we sample a different alpha to make sure the center of the Dirichlet changes and thus allows more variation.
[Equation]
where α „ Unif p
[Equation]
f Dirichlet p α q ; where α „ Unif p 1 { 3 ˆ 10 6 ,
[Equation]
{ 3 ˆ 10 6 q (10)
For sampling from behavioral policy consists of a group of single-objective policies
π β “ t π 1 , . .
, π B u with B being the total number of candidate policies, we recommend first find the expected unweighted raw rewards G π 1 , . . .
, G π B .
Then, find the estimated ˆ ω π 1 , . . .
, ˆ ω π B by letting ˆ ω π
b i “
G πbi ř nj
“ 1 G πbj , which represents the estimated preference on i th objective of b th candidate policy.
For a sampled preference ω „ Ω ˚ , use the policy that provides the smallest euclidean distance d p ω ,
ˆ ω π b q .
Empirically, this means picking the candidate policy that has the expected reward ratio closest to ω .
C E XPERT & A MATEUR D ATASETS
In Expert collection , we sample trajectories using the fully-trained behavioral policy π β .
In this paper, we use PGMORL by Xu et al., 2020 as π β
[Equation]
In the Amateur collection , the policies has a 35% chance being stochastic on top of the expert collection.
Actions has a chance being stochastic, during which it is scaled from the expert action, as following:
[Equation]
In the MO-Swimmer environment only, we let actions has a 35% chance to be a uniform random sample from the entire action space rather than being the same as expert to increase variance and achieve a performance similar to amateur.
The resulting strategy for MO-Swimmer is:
[Equation]
D T RANING D ETAILS
Common hyper-parameters have the same values across all models, except for the learning rate scheduler and warmup steps.
In MODT family, inputs are embedded by a 1-layer MLP into Hidden
Size, and n layer represents the number of transformer blocks; in BC family, n layer represents the number of MLP layers to embedding each input; in MOR V S and MOR V S(P) family, the default embedding strategy are used, please check Emmons et al., 2022.
Here, we consider MOR V S and
MOR V S(P) both have context length of 1 because they only use the current state to predict the next action, whereas MODT and BC use the past 20.
D. P ARAMETERS
[Table]
D. T RAINING S TEPS
[Table]
The exception is that we train MO-Swimmer and MO-Hopper-3obj under Low-H-Amateur for 40K and 120K steps respectively using MODT.
D. A TTEMPTED MODT AND MODT(P) A RCHITECTURES
Here are all the MODT architectures we tried in our experiments.
We followed Case 4 in all of the experiment results as we find that it makes the model follows closely to the given preference.
Consider ω as an independent token of the causal transformer.
Train a separate embedding for ω , concatenate the embeddings to get f ϕ s p s q À f ϕ ω p ω q , f ϕ a p a q À f ϕ ω p ω q , and f ϕ g p g q À f ϕ ω p ω q then pass into the transformer.
Add another MLP layer on top of the Case 2 after concatenation, then pass output into the transformer.
Concatenate ω to other tokens before any layers.
This means we have s ˚ “ s À ω , a ˚ “ a À ω , and g ˚ “ g À ω .
E O THER E VALUATION M ETRICS
Among a variety of metrics for MORL, we use Hypervolume and Sparsity (SP) to benchmark models in this paper for several reasons.
First, metrics such as the ϵ -metric require prior knowledge of the true Pareto Fronts, which are not available for our MuJoCo Environments.
Second, due to limited resources, we only assume linear reward function and cannot collect real-time user feedback, thus utility-based metrics such as expected utility metric (EUM) are not applicable.
Finally, it is easier to benchmark both our offline agents and behavioral policy on these metrics, allowing for informative comparisons.
F P ARETO S ET V ISUALIZATIONS
We present the Pareto Set visualizations for all of our models trained under each High-H dataset in
Figure 6.
All subplots are based on 1 seed, in which we evaluate the model using 501 equally spaced preference points in 2 objective environment and 351 equally spaced preference in the 3 objective environment from the full preference space.
Since the environments are stochastically initialized, we evaluate 5 times at each preference point and take the median value.
We here allow a small tolerance on coloring the dominated points.
If a preference point is within the achievable preference Ω ˚ but the solution is dominated, we color it in red.
Since our models condition on continuous preference points and environments are initializaed stochastically, we give a small tolerance (3%-8%) for points to be colored in blue.
The hypervolume and sparsity metric, on the other hand, can based on strictly undominated solutions without tolerance.
[Table] [Table]
G M EDIUM & L OW E NTROPY D ATASET T RAINING
We train on the Medium-Entropy and Low-Entropy datasets for the MO-HalfCheetah environment.
Overall, models have a similar performance under Med-H and High-H datasets, but suffers when only trained on Low-H .
We present the results in Table 6, in which we illustrate that the Low-H dataset has a worse expert and amateur performance due to reduced variability on preference.
However, MODT(P) and MOR V S(P) are still able to get close or exceed in hypervolume on all datasets, which showcase the effectiveness of PEDA as an efficient MORL policy.
Results are based on average of 3 seeds with the standard error given.
H T RAINING WITH 1- DIM RTG
We attempted to train MODT and RvS with 1-dim return-to-go rather than a separate rtg for each objective.
According to results on MO-HalfCheetah and the High-H datasets in 7, using multidimensional rtg enhances the performance of MODT(P), and are about the same for MOR V S(P) when preference are concatenated to states.
However, it reduces standard error significantly in both
MODT(P) and MOR V S(P).
In the naive models when preference are not concatenated to states, using a multi-dimensional rtg helps to achieve a much more competitive hypervolume.
We thus believe multi-dimensional rtg conveys important preference information when the model doesn’t directly take preference as an input.
Results are based on average of 3 seeds with the standard error given.