O N M ULTI - OBJECTIVE P OLICY O PTIMIZATION AS A T OOL FOR R EINFORCEMENT L EARNING : C ASE S TUDIES IN O FFLINE RL AND F INETUNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Many advances that have improved the robustness and efﬁciency of deep reinforcement learning (RL) algorithms can, in one way or another, be understood as introducing additional objectives or constraints in the policy optimization step.
This includes ideas as far ranging as exploration bonuses, entropy regularization, and regularization toward teachers or data priors.
Often, the task reward and auxiliary objectives are in conﬂict, and in this paper we argue that this makes it natural to treat these cases as instances of multi-objective (MO) optimization problems.
We demonstrate how this perspective allows us to develop novel and more effective RL algorithms.
In particular, we focus on ofﬂine RL and ﬁnetuning as case studies, and show that existing approaches can be understood as MO algorithms relying on linear scalarization.
We hypothesize that replacing linear scalarization with a better algorithm can improve performance.
We introduce Distillation of a Mixture of
Experts (DiME), a new MORL algorithm that outperforms linear scalarization and can be applied to these non-standard MO problems.
We demonstrate that for ofﬂine
RL, DiME leads to a simple new algorithm that outperforms state-of-the-art.
For ﬁnetuning, we derive new algorithms that learn to outperform the teacher policy.
I NTRODUCTION
Deep reinforcement learning (RL) algorithms have solved a number of challenging problems, including in games (Mnih et al., 2015; Silver et al., 2016), simulated continuous control (Heess et al., 2017;
Peng et al., 2018), and robotics (OpenAI et al., 2018).
The standard RL setting appeals through its simplicity: an agent acts in the environment and can discover complex solutions simply by maximizing cumulative discounted reward.
In practice, however, the situation is often more complicated.
For instance, without a carefully crafted reward function or sophisticated exploration strategy, learning may require hundreds of millions of environment interactions, or may not be possible at all.
A number of strategies have been developed to mitigate the shortcomings of the pure RL paradigm.
These include strategies that regularize the ﬁnal solution, for instance by maximizing auxiliary rewards (Jaderberg et al., 2017) or the entropy of the policy (Mnih et al., 2016; Haarnoja et al., 2018).
Others facilitate exploration, for instance via demonstrations (Brys et al., 2015), shaping rewards (Ng et al., 1999), exploration bonuses (Bellemare et al., 2016; Pathak et al., 2017), or guidance from a pre-trained teacher policy, known as policy ﬁnetuning (Xie et al., 2021).
Strategies for reusing old environment interactions can in the extreme case enable agents to learn solely from a ﬁxed dataset, known as ofﬂine RL (Fu et al., 2020; Gulcehre et al., 2020).
In this case regularization is needed to ensure that the learned policy is well supported by the data distribution.
These approaches may seem quite different at ﬁrst glance, but they can all be understood as introducing additional objectives, or constraints, in the policy optimization step.
Since reward maximization and the additional objective(s) are usually in conﬂict, these algorithmic approaches can also be understood as solving a multi-objective RL (MORL) problem.
In practice, the solution to this MORL problem is usually implemented via a form of linear scalarization (LS), i.e., by taking a weighted sum of the objectives.
However, LS suffers from a number of limitations: it is sensitive to the scale of the objectives (Abdolmaleki et al., 2020) and is limited in the solutions it can ﬁnd (Das & Dennis, 1997).
Our key insight is that an interpretation of algorithmic approaches, such as the above, in terms of
MORL provides us with novel tools to solve fundamental challenges in RL.
To demonstrate this, we focus on ofﬂine RL and ﬁnetuning as case studies.
We show recent ofﬂine RL algorithms (e.g.,
CRR (Wang et al., 2020), AWAC (Nair et al., 2020)) can be understood as using LS to solve the MO problem.
We hypothesize that by replacing LS with a more appropriate approach, we can obtain betterperforming algorithms.
We thus propose a new MORL approach, called Distillation of a Mixture of
Experts (DiME), that can replace LS in this setting.
We derive novel algorithms for ﬁnetuning and ofﬂine RL, based on DiME.
In ofﬂine RL, this new algorithm outperforms state-of-the-art (Sec. 6).
Our main contributions are as follows:
• We propose that MORL can be used as a tool for tackling fundamental challenges in RL.
We show that existing algorithms (e.g., for ofﬂine RL) rely on LS to solve the MO problem.
Based on this insight, we explore replacing LS with a new, improved MO algorithm.
• By replacing LS, we derive a novel ofﬂine RL algorithm from the multi-objective perspective (Sec. 5.2), that is simple, intuitive and outperforms the state-of-the-art (Sec. 6.2).
• In a similar manner, we also derive new ﬁnetuning algorithms (Sec. 5.1), and show that these algorithms can train policies that exceed the teacher’s performance (Sec. 6.3).
• To replace LS, we introduce a new MORL approach, DiME.
We support DiME by theoretical derivations (Sec. 4.1) and show it outperforms LS on standard MORL tasks (Sec. 6.1).
R ELATED W ORK
Ofﬂine RL and Finetuning.
Our case studies in this work will involve two related learning problems: ofﬂine (or batch) RL (Levine et al., 2020; Lange et al., 2012) and policy ﬁnetuning (Xie et al., 2021).
Both leverage existing information, either in the form of a dataset or a teacher policy.
In this work, we use proximity to the dataset or policy as an additional objective in a MORL problem.
In ofﬂine RL, the goal is to learn a policy from a given, ﬁxed dataset of transitions.
While classical off-policy RL algorithms can be used, they perform worse than bespoke algorithms (Fujimoto et al., 2019).
This is likely due to the extrapolation errors incurred when learning a Q-function—naïve policy optimization could optimistically exploit such errors.
Therefore, many existing approaches attempt to constrain the policy optimization to stay close to the samples present in the available dataset (Peng et al., 2019; Nair et al., 2020; Siegel et al., 2020; Wang et al., 2020).
Finetuning seeks to improve upon an expert (or teacher) policy, which can be speciﬁed either nonparametrically via a dataset of transitions or as a parametric model.
In contrast to ofﬂine RL, ﬁnetuning happens in an online setting, where the agent is able to interact with the environment.
Existing approaches for ﬁnetuning minimize the cross-entropy between the teacher and agent policies (Schmitt et al., 2018), based on distillation or mimicking (Rusu et al., 2015; Parisotto et al., 2015).
Multi-Objective RL.
MORL algorithms train policies for environments with multiple sources of reward.
There are single-policy and multi-policy approaches.
The former ﬁnds a policy that is optimal for a single preference trade-off across rewards.
One common technique is to use the trade-off to combine the rewards into a single scalar reward, and then use standard RL to optimize this (Roijers et al., 2013).
Typically linear scalarization is used, but it is sensitive to reward scales and cannot always ﬁnd optimal solutions (Das & Dennis, 1997).
Non-linear scalarizations exist (Van Moffaert et al., 2013; Golovin & Zhang, 2020), but are hard to combine with value-based RL.
Instead of scalarization, recent work on MO-MPO (Abdolmaleki et al., 2020) relies on constrained optimization.
Multi-policy approaches seek to ﬁnd a set of policies that cover the Pareto front.
Various techniques exist, for instance repeatedly calling a single-policy approach with strategically-chosen trade-off settings (Roijers et al., 2014; Mossalam et al., 2016; Zuluaga et al., 2016), simultaneously learning a set of policies by using a multi-objective variant of Q-learning (Moffaert & Nowé, 2014; Reymond & Nowé, 2019;
Yang et al., 2019), learning a manifold of policies in parameter space (Parisi et al., 2016; 2017), or combining single-policy approaches with an overarching objective (Xu et al., 2020).
In contrast, here we consider objectives that are not only environment rewards, but also learning- or regularization-focused such as staying close to a behavioral prior.
State-of-the-art MORL approaches, including MO-MPO, cannot be readily applied.
We thus introduce a new algorithm, DiME, which is on-par with state-of-the-art and can handle these non-standard objectives (Sec. 4).
We use DiME as a vehicle to demonstrate the beneﬁts of taking a multi-objective perspective toward challenges in RL.
[Table]
B ACKGROUND
We consider the RL problem deﬁned by a Markov Decision Process (MDP).
An MDP consists of states s ∈ S , actions a ∈ A , an initial state distribution p ( s 0 ) , transition probabilities p ( s t +1 | s t , a t ) , a reward function r ( s, a ) ∈ R
[Table]
and a discount γ ∈ [0 , 1) .
A policy π θ ( a | s ) is a state-conditional distribution over actions, parametrized by θ .
Together with the transition probabilities, this gives rise to a state visitation distribution µ ( s ) .
The action-value function is the expected return from choosing action a in state s and then following policy π : Q π ( s, a ) =
E π [ (cid:80) ∞ t =
γ t r ( s t , a t ) | s 0 =
s, a 0 = a ] .
This can also be written as Q π ( s, a ) =
E s (cid:48) ∼ p [ r ( s, a ) + γV π
( s (cid:48) )] , where V π ( s ) =
E a ∼ π [ Q π ( s, a )] .
In this paper, both the LS and DiME approaches to multi-objective RL follow a policy iteration scheme, repeatedly alternating between policy evaluation and policy improvement.
In the policy evaluation step, at iteration i the current policy π θ i is evaluated by training a separate value function
Q ik for every objective k .
This requires bootstrapping 1 according to Q-decomposition (Russell &
Zimdars, 2003).
Any policy evaluation algorithm can be used to learn these Q-functions.
In the policy improvement step, given the previous policy π i (short for π θ i when clear from context) and associated
Q-functions { Q k } Kk =1 , the aim is to improve the policy for a given state visitation distribution µ .
3. P OLICY I MPROVEMENT : S INGLE O BJECTIVE
DiME introduces a new way to perform policy improvement with respect to multiple objectives, so we will now describe the policy improvement step, ﬁrst for the single-objective setting.
Policy improvement can itself be decomposed into the following two alternating steps (Ghosh et al., 2020).
Improvement.
This step ﬁnds a (nonparametric) policy q that improves with respect to the Qfunction, while staying close to the current iterate π i , leading to the Lagrangian objective function
[Equation]
with the known analytic solution q ( a | s ) ∝ π i ( a | s ) exp( Q [Equation] s, a ) /η )
(Peters et al., 2010).
Projection.
This step projects the improved policy to the space of parametric policies by maximizing the following (which simpliﬁes to policy gradient if one uses a parametric q in the improvement step):
[Equation]
3. P OLICY I MPROVEMENT : M ULTIPLE O BJECTIVES
Suppose the policy must improve with respect to K objectives instead.
In multi-objective problems, a solution is Pareto-optimal if there is no other solution with better performance for one objective without worse performance for another.
The set of all Pareto-optimal solutions is the Pareto front .
To ﬁnd a speciﬁc Pareto-optimal policy, let α = { α k } Kk =1 deﬁne a preference trade-off across objectives, such that all α k ≥ 0 .
We can further enforce (cid:80) k
α k
= 1 .
A larger α k means the k -th objective is more important, and should lead to increased inﬂuence of objective k on the policy optimization.
Linear Scalarization.
Here the trade-off is used to produce a single scalar reward via a weighted sum of the objectives.
This leads to the nonparametric solution for q in Table 1, while the projection step is the same as for a single objective.
A drawback of LS is that not only the trade-offs, but also the scales of the Q k , inﬂuence the contribution of each objective.
MO-MPO.
In contrast, Abdolmaleki et al.
(2020) optimizes (1) separately for each objective, to obtain K improved policies { q k } Kk =1 .
The trade-offs { α k } specify the constraint threshold on the KL-divergence between q k and π i .
As seen in Table 1, this leads to nonparametric solutions with temperatures η k , that are each a solution to a convex optimization problem involving α k .
A main advantage of MO-MPO is that it combines objectives in a scale-invariant way 3 , unlike LS.
However, because the trade-offs in MO-MPO deﬁne KL-constraints, the improvement operator used for q k must be able to meet this constraint exactly.
This means MO-MPO cannot be easily applied to non-standard settings like ofﬂine RL, as we will discuss later in Sec. 5.2.
Thus we propose DiME, which can be applied to non-standard settings and maintains the beneﬁt of scale-invariance.
D ISTILLED M IXTURE OF E XPERTS
Like MO-MPO, DiME computes a separate q k per objective.
As seen in Table 1, the nonparametric solution is almost identical, with the crucial exception that the temperature η k no longer depends on α k .
Unlike LS and MO-MPO, DiME only incorporates the trade-offs in the projection step objective
[Equation] [Equation]
so any improvement operator can be used to obtain q k .
This allows us to apply DiME to ofﬂine RL and other non-standard settings.
In addition, the trade-offs in DiME are more interpretable than those in MO-MPO, since they are weights on local experts q k , rather than KL-divergence constraints.
Appendix B describes DiME in more detail, including how the temperature η k is computed.
However, we may not a priori know what the preference trade-off across objectives should be.
In this case we can extend DiME to learn a conditional policy, thereby distilling the entire Pareto front of optimal policies into a single π θ ( ·| s, α ) .
We train this policy by augmenting states with normalized trade-offs α sampled from some distribution ν , and then optimizing the following objective:
[Equation]
where q k ( a | s, α ) ∝ π i ( a | s, α ) exp( Q k ( s, a, α ) /η k ) .
Note that in order to evaluate this policy, we must also learn an α -conditioned value function Q k ( s, a, α ) .
Appendix A contains more details.
After training an α -conditioned policy, one can use Q k ( α ) =
E s ∼ µ (cid:82) π ( a | s, α
) Q k ( s, a, α ) d a fully ofﬂine to pick the best trade-off α with respect to objective k , using any hyperparameter optimization (HPO) algorithm, including a simple search.
If the learned Q k are inaccurate, which is often true in ofﬂine RL, we can instead deploy the policy to search for the best trade-off.
This is cheap; in our case studies α is between 0 and 1 .
In this paper we aim to learn high-quality α -conditioned policies, which we will show is possible using DiME.
Investigating other ways of picking α is out of this scope.
4. M ULTI -O BJECTIVE RL AS P ROBABILISTIC I NFERENCE
We will now derive LS and DiME from the RL as inference perspective for MORL, showing that both optimize for the same high-level problem, but differ in their choice of variational distribution.
Consider a binary random variable R indicating a policy improvement event: R = 1 indicates the policy has improved, while R = 0 indicates that it has not.
Policy optimization then corresponds to maximizing the marginal likelihood E µ log p θ ( R | s ) with respect to θ .
Employing the expectationmaximization (EM) algorithm
and assuming a conditional probability p ( R | s, a ) ∝ exp( Q ( s, a ) /η ) , one can recover many algorithms in the literature (Levine, 2018).
Let us instead deﬁne a separate indicator R k for each objective, with similarly deﬁned conditional probabilities, and consider the following weighted log-likelihood objective:
[Equation]
where E µ is a shorthand for E s ∼ µ and α ∈ R K + specify a preference trade-off across the K objectives.
This generalization has a few nice properties.
First, when all objectives are equally preferred, the α k are all equal and it reduces to a straightforward probabilistic factorization of independent events: p ( { R k } Kk =1 )
= (cid:81)
k p
R k ) .
Second, any vanishing α k leads to an objective being ignored.
Linear Scalarization.
To apply EM, we could introduce a variational distribution q ( a | s ) and use it to decompose and lower-bound (5).
This choice of variational distribution results in LS: the E-step corresponds to (1) with Q ( s, a ) replaced by (cid:80) k
α k Q k ( s, a ) , and the M-step is identical to (2).
DiME.
We could instead introduce a separate variational distribution q k for each log p θ ( R k | s ) , which results in DiME.
The E-step now corresponds to K independent copies of the improvement step in (1), each with its own Q k ( s, a ) .
The M-step is the projection step introduced in (3).
This reveals that both standard LS and DiME optimize the same information-theoretic objective (see
Appendix B.1 for a complete derivation).
Both employ EM, but differ in their choice of variational decomposition.
We hypothesize that ﬁtting a separate variational distribution per objective allows
DiME to ﬁnd a tighter variational lower bound, which may lead to better optimization.
C ASE S TUDIES
Our primary aim in this work is to showcase MORL as a tool for tackling fundamental challenges in deep RL.
We therefore demonstrate its beneﬁts in two active areas of research: ﬁnetuning and ofﬂine
RL.
Finetuning is concerned with improving upon an expert (or teacher) policy, whether it is available in the form of a parametric model or a dataset of transitions.
Meanwhile in ofﬂine RL, one must learn a policy from a given ﬁxed dataset of transitions.
In contrast to ofﬂine RL, ﬁnetuning happens in an online fashion so the agent interacts with the environment to collect additional experience.
These two settings share a fundamental similarity—they both use behavioral priors to make learning more efﬁcient.
Existing algorithms in both settings constrain the policy optimization, either explicitly or implicitly, to stay close to the given priors (Levine et al., 2020;
Monier et al., 2020;
Schmitt et al., 2018).
Instead, we propose to explicitly formulate these in the MORL framework with two objectives:
one for the task return via the task Q-function Q ( s, a ) learned via bootstrapping,
and one for staying close to the behavioral prior via the log-density 4 ratio log π b ( a | s ) π i ( a | s ) , where π i is the current policy and π b is the behavioral prior.
When we have direct access to the density π b , we can evaluate the log-density ratio pointwise.
However, in many real-world applications, we only have access to samples from π b via a dataset of historical interactions.
In the following subsections, we dive into each of these cases, deriving the corresponding policy losses for both LS and DiME.
We use a scalar α ∈ [0 , 1] to specify the preference trade-off between the two objectives.
5. F INETUNING
Equipped with the MORL perspective, we introduce two novel ﬁnetuning algorithms by substituting the objectives above in the LS and DiME rows of Table 1, yielding the policy optimization functions:
[Equation]
and
[Equation]
All functions Z ∗ ( s ) normalize the product of π i and the corresponding tempered and exponentiated objective; though analytically intractable, they can be estimated from sampled actions for each state.
Note that both of the equations above are equivalent at the extreme values of α = 0 and 1 .
Relation to existing approaches.
We can recover existing algorithms with slight modiﬁcations to these policy objectives.
In particular, setting η 2 = 1 in the DiME row of Table 1 yields q 2 = π b and the following policy objective function:
[Equation]
which is similar to the ﬁnetuning approach in (Schmitt et al., 2018)
Similarly, taking instead the LS approach to MORL and employing the policy gradient (which uses a parametric q in Sec. 3.1) as the underlying policy optimization, we get:
[Equation]
which is similar to (Galashov et al., 2019)
Both simply add a weighted KL regularization to a standard RL objective, but they differ in the direction of the KL.
5. O FFLINE RL
Recall that in ofﬂine RL, we only have access to the behavioral prior π b via a dataset D π b of sampled transitions ( s, a, r, s (cid:48) ) ; we cannot evaluate its density and thus cannot directly compute the objective function for staying close to the prior.
Instead, we must simplify both J LS and J DiME from the ﬁnetuning subsection so that they only require samples from π b and µ , for which we can rely on D π b .
Linear Scalarization.
By tying the preference and scale parameters so that α = η , we can change the expectation over actions in (6) analytically to be with respect to the behavior policy π b
[Equation]
where β = α/ (1 − α ) .
There are close similarities between this LS-based policy objective and other policy losses used in recent ofﬂine RL algorithms.
For example, β corresponds to the temperature in CRR (Wang et al., 2020) and AWAC (Nair et al., 2020), and we can recover them exactly by replacing the task return objective with the advantage function.
As an other example, BEAR (Kumar et al., 2019) and BCQ (Fujimoto et al., 2019) can both be formulated as using LS, but with the policy gradient as their underlying policy optimizer as in (9).
However, unlike the objective function in (10), which only requires samples from π b , these methods have the limitation that they require modeling π b to evaluate the second term in (9) (Levine et al., 2020;
Monier et al., 2020).
DiME.
Setting η 2 = 1 and renaming η 1 = η in (7), we obtain
[Equation]
Notice that the ﬁrst term is the loss used by policy optimization methods such as REPS (Peters et al., 2010) and MPO (Abdolmaleki et al., 2018), and the second term is exactly a behavioral cloning loss.
We call this new algorithm DiME (BC).
This is a simple way of adapting online algorithms for ofﬂine
RL, while being theoretically grounded in a multi-objective perspective.
Intuitively, since both terms are independent of the scale of the Q-values, optimizing the weighted sum is a reasonable thing to do.
This is in contrast to recent concurrent work that adds a behavioral cloning loss to a policy gradient loss, which depends on the scale of the Q-values (Fujimoto & Gu, 2021).
Remark on drawbacks of LS approaches.
First, the preference α and scale η can no longer be set independently; they are coupled together in the temperature β .
This means we cannot efﬁciently train a trade-off-conditioned policy using the LS-based loss in (10), because the normalization constant Z 0 = (cid:82) ( s,a ) ∼D πb exp( Q ( s, a ) /β ) depends on the choice of the trade-off—this would require estimating (10) via samples separately per trade-off.
Second, since J LS only involves an expectation with respect to the expert behavior policy, the policy cannot evaluate and exploit the
Q -value estimates for actions beyond those taken by the expert.
One could argue that this is by design, in order to avoid ﬁtting extrapolation errors in the Q-function.
However, with J DiME this is a choice we can make by setting the preference parameter independently (and perhaps adaptively).
Remark on using other MORL approaches.
Although there exist many MORL algorithms that outperform LS on standard MORL tasks, they typically require that the rewards for all objectives are given.
Thus it is not straightforward to apply these algorithms for ofﬂine RL, where we cannot directly compute the reward for staying close to the prior.
We tried getting around this in order to use
MO-MPO for ofﬂine RL, but it was very complicated to implement (for details see Appendix B.4).
[Figure]
E XPERIMENTS
We ﬁrst investigate how DiME performs on standard MORL tasks, compared to LS (Sec. 6.1). Since our focus is on applying MORL for general challenges in RL, we do not consider MORL algorithms that cannot be applied to ofﬂine RL.
Then we investigate the application of DiME in two case studies, ofﬂine RL (Sec. 6.2) and ﬁnetuning (Sec. 6.3), where our aim is to evaluate whether replacing LS with DiME leads to better-performing algorithms.
Implementation details are in Appendix C.
6. M ULTI -O BJECTIVE RL
Toy Domain.
Our bandit task has action a ∈ R and reward r ( a ) ∈ R 2 , with the reward deﬁned by either the Schaffer or Fonseca-Fleming function (Okabe et al., 2004).
The Pareto front is the set of all points of the function.
By varying the trade-off, DiME ﬁnds solutions along the entire Pareto front for both functions.
However, no matter what the trade-off is, LS only ﬁnds solutions at the extremes of Fonseca-Fleming, because this Pareto front is concave (Fig. 1a).
This is a fundamental limitation of LS—it cannot ﬁnd solutions on the concave portions of a Pareto front (Das & Dennis, 1997).
Humanoid.
We also evaluate on two humanoid tasks from the DeepMind Control Suite (Tassa et al., 2018).
For each, the original task reward objective is augmented with an “energy”-expenditure penalty on actions: −(cid:107) a (cid:107) 2 .
In both tasks, the Pareto front found by DiME is superior to that found by
LS, both qualitatively (Fig. 1b)
and in terms of hypervolume, which is a standard metric for capturing the overall quality of a Pareto front (Table 1c).
Additional Experiments.
We also ﬁnd that DiME is scale-invariant—it obtains similar performance when the energy cost is scaled up by ten times, whereas LS no longer can learn the task.
DiME also performs on-par with MO-MPO, which is state-of-the-art.
Please refer to Appendix D for details.
Recall that DiME can be applied to non-standard settings like ofﬂine RL, whereas MO-MPO cannot.
6. O FFLINE RL
Baselines.
For ofﬂine RL, our main baseline uses LS to trade off between the task return and staying close to the behavioral prior, with the advantage function as the task return objective.
This optimizes for E ( s,a ) ∼D πb exp( (1 − α ) /
α A ( s, a )) log π θ ( a | s ) .
This is the same policy loss as CRR-exp (Wang et al., 2020) and AWAC (Nair et al., 2020), two state-of-the-art ofﬂine RL algorithms.
Our behavioral cloning (BC) baseline optimizes for E ( s,a ) ∼D πb log π θ ( a | s ) , which is LS or DiME with α = 1 .
Variations of DiME.
For ofﬂine RL, our DiME-based algorithm optimizes for (11), where the second term is a behavior cloning loss; we call this DiME (BC).
For this second term, we could instead use an advantage-weighted behavior cloning loss, E π b ( a | s ) [exp( A ( s, a )) log π θ ( a | s )] , which is exactly the CRR policy loss with a temperature of 1 ; we call this DiME (AWBC).
Since training a separate policy per trade-off is computationally expensive, we also experiment with training a single trade-off-conditioned policy π θ ( a | s, α ) , that can optimize for all trade-offs in the range [0 , 1] .
We refer to this as DiME multi, and combine it with both BC and AWBC.
We choose to train a single policy that can optimize for all trade-offs, rather than learning the best trade-off during training, because it is a well-known problem that policy evaluation is inaccurate in ofﬂine RL.
[Table]
Evaluation.
We evaluate on tasks from two ofﬂine RL benchmarks, RL Unplugged (Gulcehre et al., 2020) and D4RL (Fu et al., 2020), with results in Table 2.
For LS, DiME (BC), and DiME (AWBC), we train one policy per trade-off α , linearly spaced from 0 .
05 to 1 .
For DiME (BC) multi and DiME (AWBC) multi, we train ten policies with different random seeds.
We train all policies for one million learner steps.
After training, we evaluate the policy in the test-time environment once per trade-off.
From RL Unplugged, we evaluate on the nine Control Suite tasks.
For both LS and DiME, the setting of α has a signiﬁcant impact on performance, and the optimal setting is different per task (Fig. 2).
Comparing the best performance obtained for each task (across all trade-offs), all DiME variants outperform LS.
In addition, training a single tradeoff-conditioned policy (with DiME multi) is not only more efﬁcient, but also leads to better performance.
This may be because conditioning on α regularizes learning.
With LS though, we cannot efﬁciently train α -conditioned policies because α controls both the trade-off and scaling.
To make a more fair comparison against prior work, we also ﬁnd the best ﬁxed trade-off across all tasks for LS and DiME (BC/AWBC), and report these results in the middle columns of Table 2.
Both DiME variants still outperform existing approaches.
From D4RL, we evaluate on the six Ant Maze tasks and three Franka Kitchen tasks.
We only evaluate the DiME multi variants, with one random seed, since they perform best on the Control Suite tasks and have low variance.
LS and DiME perform on par, and both outperform state-of-the-art approaches.
6. F INETUNING
Finally, we will evaluate whether our new ﬁnetuning algorithms, derived from the multi-objective perspective, can improve upon the performance of a teacher policy.
We evaluate both the LS-based and DiME-based algorithm with either a ﬁxed or learned tradeoff.
To learn the tradeoff, we update α based on the loss J ( α )
=
α ( E s ∼ µ,a ∼ π i Q ( s, a ) − c ) .
This is analogous to the update for the
Lagrange multiplier in Lagrangian relaxation for constrained optimization.
The resulting strategy is intuitive: stay close to the behavioral prior while the expected return is below the threshold c , and otherwise optimize for the bootstrapped Q-function.
We pick c based on the expected return from fully imitating the prior for the given task (
[Figure] [Figure]
We evaluate on the humanoid and manipulator tasks from the DeepMind Control Suite.
For the three humanoid tasks, we start with a suboptimal humanoid stand policy as the behavioral prior.
For the two manipulator tasks, the behavioral prior is a policy trained for that task.
The choice of ﬁxed tradeoff impacts the result signiﬁcantly, and the optimal choice depends on the task, so either searching over or learning the tradeoff is necessary.
Across all ﬁve tasks, for both LS and DiME, the learned tradeoff converges to better performance than fully imitating the prior ( α = 1 ) and learns much more quickly than learning from scratch ( α = 0 ).
This has the advantage of training only a single policy, as opposed to training a separate policy per ﬁxed tradeoff.
Adapting the trade-off helps because in ﬁnetuning, the agent is able to gather online interactions.
An adaptive scheduling for the trade-off allows the agent to initially copy the teacher and then eventually deviate from the teacher to achieve better performance based on its new experiences.
In contrast, in the ofﬂine RL setting, we needed to use ﬁxed trade-offs because such online interactions are not allowed.
C ONCLUSION AND F UTURE W ORK
In this work, we propose using multi-objective optimization as a tool for tackling challenges in RL.
We explored the beneﬁts of this in ofﬂine RL and ﬁnetuning, both of which involve staying close to a behavioral prior.
Our derivations revealed that existing approaches optimize for a linear scalarization of two objectives, task return and a log-density ratio.
We proposed a new MORL algorithm, DiME, as an alternative to LS.
Applying DiME to ofﬂine RL leads to state-of-the-art performance.
A key takeaway is the impact of the trade-off α on performance, and the need to either search over or optimize for it.
As a ﬁrst solution we explored either training trade-off-conditioned policies or adapting the trade-off during training; it is worth exploring other approaches in future work.
In ofﬂine RL, a limitation of optimizing for the trade-off is that it requires being able to accurately evaluate the policy for multiple trade-offs.
We chose the simplest option in our experiments: deploy the fully-trained policy for each trade-off in the test-time environment, and pick the best-performing trade-off.
But this may not be feasible in real-world applications, for instance if it is expensive or dangerous to deploy a policy.
One option is to use recent ofﬂine policy evaluation algorithms to accurately estimate a policy’s test-time performance (for all trade-offs) without requiring deployment.
R EPRODUCIBILITY S TATEMENT
In Appendix C we provide full implementation details for DiME, our novel algorithms for ﬁnetuning and ofﬂine RL, and the baselines.
We report hyperparameter settings and network architectures in
Tables 3 and 4, also in Appendix C.
Our code is based on the open-sourced implementations of
MO-MPO and MPO in Acme (Hoffman et al., 2020), which is an open-source RL framework.
We are in the process of open-sourcing the code
A T RAINING T RADE - OFF - CONDITIONED P OLICIES
In this section we provide details on training a trade-off-conditioned policy π θ ( a | s, α ) conditioned on preference parameters
α ∼ ν
( α ) .
(Note that here α corresponds exactly to α from the main body of this manuscript; we had to distinguish it from the single scalar that appears in our case studies, but we drop the underline here to lighten notation.)
Conditioning the policy on the trade-off α requires that the Q-functions are also trade-off-conditioned.
We also use hindsight relabeling of trade-offs to perform off-policy learning.
Preference-conditioned policy evaluation.
We evaluate the current policy π i ( a | s, α ) via off-policy learning of per-objective Q-functions Q ik ( s, a, α ) .
When training the Q-network, every sampled transition ( s, a, { r k } Kk =1 , s (cid:48) ) is augmented such that the states include a sampled trade-off parameter α ∼ ν , resulting in transitions ([ s, α ] , a, { r k } Kk =1 , [ s (cid:48) , α ]) .
From here, any critic learning algorithm can be used; we use distributional Q-learning (Bellemare et al., 2017; Barth-Maron et al., 2018).
See
Section C for more details.
Preference-conditioned policy improvement.
The policy improvement step produces a trade-offconditioned policy π i +1 ( a | s, α ) that improves upon its predecessor π i .
Improvement: To ﬁnd per-objective improved action distributions q k ( a | s, α ) , we optimize the following policy optimization problem for each objective:
[Equation] [Equation]
Similarly, from here we can follow steps analogous to those in Section B.1, but with trade-offconditioned policies and Q-functions.
This means that this set of trade-off-conditioned problems have solutions that are analogous to (38) for each objective.
Projection: After obtaining per-objective improved policies q k , we can use supervised learning to distill these distributions into a new parameterized policy:
[Equation]
subject to a trust region with bound β > 0 for more stable learning.
To solve this optimization, we use Lagrangian relaxation as described in Abdolmaleki et al.
In practice.
To approximate the expectations over the state and trade-off distributions, we draw states from a dataset (or replay buffer in online RL) and augment each with a sampled trade-off α ∼ ν .
To approximate the integrals over actions a , for each ( s, α ) pair we sample N actions from the current policy π i ( a | s, α ) .
B A LGORITHMIC D ETAILS AND D ERIVATIONS B. M ULTI -O BJECTIVE RL AS P ROBABILISTIC I NFERENCE
In this section we present further details of our derivation, which uniﬁes both linear scalarization and DiME, relying on the RL as inference framework.
We begin by recalling some of the motivation behind the objective.
Consider a set of binary random variables R k indicating a policy improvement event with respect to objective k ; R k = 1 indicates that our policy has improved for objective k , while R k = 0 indicates that it has not.
We seek a policy that improves with respect to all objectives given some preferences over objectives.
Concretely, we seek policy parameters θ that maximize the following marginal likelihood, given preference trade-off coefﬁcients { α k }:
[Equation]
[Equation]
Assuming independence between improvement events R k we use the following model to enforce trade-offs between objectives,
i
[Equation]
e,
[Equation]
This generalization has a few nice properties.
First, when all objectives are equally preferred, the α k are all equal and it reduces to a straightforward probabilistic factorization of independent events: p ( { R k } Kk =1 )
= (cid:81)
k p
R k ) .
Second, any vanishing α k leads to an objective being ignored
[Equation]
Consider the intractable marginal likelihood L that we wish to maximize
[Equation]
then introduce variational distributions q k and expand as follows
[Equation]
Let us denote the second term as F , deﬁned as follows
[Equation]
Then one can rearrange terms in the expansion of the marginal likelihood F to yield
[Equation]
Since the KL divergence is a non-negative quantity and the α k are all positive, this last equation shows that L lower-bounds the quantity we wish to maximize, namely F .
Whether we call it expectation-maximization or improvement-projection, the steps are as follows:
1. tighten the gap between the lower bound L and F at the current iterate θ i , by maximizing L with respect to q k while keeping θ = θ i ﬁxed, then 2.
maximize the lower bound L with respect to θ , holding q k ﬁxed to the solution of step 1.
Formally, these two steps result in the following optimization problems
[Equation]
where in the second step we removed terms that do not depend on θ .
These steps are respectively referred to as the E- and M-step in the statistics literature, and improvement and projection in policy optimization.
While these two optimization problems seem decoupled from the original goal of maximizing F , the connection with the EM algorithm allows us to beneﬁt from the guarantee that iterates θ i obtained in this way lead to improvements in F .
B. I MPROVEMENT OR E- STEP
In the next sections we detail the two steps in turn, beginning with the improvement E-step, where we ﬁrst notice that the objective is a weighted sum of K independent terms that can all be minimized separately
[Equation]
These separate problems can be solved analytically by making the KL vanish with the following solution
[Equation]
This nonparametric distribution reweights the actions based on the improvement likelihood p
( R k | s, a ) .
This likelihood is free for us to model as follows and as is standard in the RL as inference literature
[Equation]
where η k is a objective-dependent temperature parameter that controls how greedy the solution q k ( a | s ) is with respect to its associated objective Q k ( s, a ) .
Substituting this choice of p ( R k | s, a ) into the independent problems (28) we obtain the following objective
[Equation]
After multiplying this objective by − η k
[Equation]
we obtain the following equivalent maximization problem
[Equation]
where we recognize a KL regularization term that materializes naturally from this derivation.
Indeed this last optimization problem can be seen as a Lagrangian, imposing a soft constraint on the KL between the improved policy q k and the current iterate θ i .
If we instead impose a hard such constraint, we obtain the following problem:
[Equation]
This problem’s Lagrangian form is as follows, and is nearly identical to (35)—albeit with a different dual variable instead of η k ; nevertheless we reuse the notation η k instead of introducing a new dual variable:
[Equation]
[Equation]
Putting everything together.
Since the original problem is equivalent to the Lagrangian relaxation of this one, we can simply substitute our improvement likelihood (30) into the solution to the original problem (29) to obtain the solution:
[Equation]
η k , where
[Equation] [Equation]
Adaptively varying η k .
Finally, this solution can in turn be substituted back into (37) to obtain the following convex dual function:
[Equation]
where in the ﬁnal step we substitute (39) to reveal the hidden dual variable.
In practice
[Equation]
this function is optimized alongside of θ in order to adapt the dual variable η k and mitigate overﬁtting to Q k
[Equation]
B. P ROJECTION OR M- STEP
Let us now look at the projection step (M-step in EM terminology), which produces the next iterate θ i +1 for our parametric policy.
Recall the optimization problem involved in the M-step is the following
[Equation]
where q k is ﬁxed to the solution from the improvement step, namely (38).
This problem corresponds to ﬁnding the policy π θ that acts as a barycentre between the nonparametric policies q k , each weighted by its speciﬁed preference trade-off coefﬁcient α k .
Other than this important difference
[Equation]
the multi-objective follows from the single-objective case in a straightforward way.
Substituting the solution q k from (38) into the expression for L ( θ ; q k ) yields
[Equation]
This is the objective function that is optimized to obtain the next iterate in practice.
At this stage we can further impose a trust-region or soft KL constraint on this optimization problem.
In practice.
We compute this policy objective via Monte Carlo, sampling states s from a replay buffer, then sampling actions for each state from our previous iterate π i .
We use these samples to compute weights exp( Q k ( s, a ) /η k ) and normalize them across states, which corresponds to using a sample-based approximation to Z k ( s ) as well.
Finally, we take the convex combination of these weights, according to the preference trade-offs, which yields the ultimate weights—seen in brackets in (51).
B. O N HOW D I ME IS BETTER SUITED THAN MO-MPO FOR O FFLINE RL.
In this section we expand on precisely why MO-MPO in its current published form is not readily suitable for ofﬂine RL and how our proposed method improves on it in this aspect.
Let us begin by revisiting the MO-MPO objective function (Eq. 1; Abdolmaleki et al., 2020) and substituting in the ofﬂine RL objectives discussed in the paper.
[Equation]
where we deliberatly omit the normalization constants to lighten the notation.
Recall that MO-MPO enforces the preference trade-off across objectives by setting the KL-constraint thresholds (cid:15) k between the improved policies and the current policy iterate as in (45). Therefore the preference trade-offs are implicit in the varying temperatures η k , which are continuously adapted throughout training to ensure the satisfaction of the corresponding constraint (cid:15) k . As a result, when applying the same simplifying trick as used for DiME—setting η 2 = 1 and renaming η 1 = η —we forfeit control of the trade-off completely. Concretely, the second term above reduces to an expectation with respect to the behavior dataset D π b as follows:
[Equation]
where we recover the now common form of an RL objective combined with an additional BC regularization term.
Notice that not only is there no trade-off α
but the temperature η 2 , which implicitly controls it, has also disappeared.
Let us now compare this expression to that of DiME:
[Equation]
where we highlight the key difference between the two losses, namely that the preference parameter α allows us to explicitly control the relative importance of the task’s Q-value objective and the objective that keeps the policy close to the one that generated the ofﬂine dataset.
In contrast, MO-MPO can no longer satisfy the preference setting speciﬁed by the choice of (cid:15) k , because of the simplifying assumption that η 2 = 1 .
An alternative approach would be to use the dataset D π b to learn a distilled behavior policy ˆ π b to model π b , effectively performing behavioral cloning as a preliminary subroutine.
This would allow us to compute the log-density ratio directly, to use it as an objective in the original expression (54).
This strategy introduces approximation errors in the form of assumptions made on the form of π b and generalization to states that are not present in D π b .
While we tried both this and the MO-MPO objective in (55) in preliminary experiments, neither proved to be competitive with the LS / CRR baseline and were thus omitted from further investigations.
C I MPLEMENTATION D ETAILS
This section describes how we implemented our algorithm, DiME, and the linear scalarization (LS) and behavioral cloning (BC) baselines.
To ensure a fair comparison, we use the same hyperparameters and network architectures for DiME and the baselines, wherever possible.
All algorithms are implemented in Acme (Hoffman et al., 2020), an open-source framework for distributed RL.
Training Setup.
For the multi-objective RL and ﬁnetuning settings, we use an asynchronous actor-learner setup, with multiple actors.
In this setup, actors fetch policy parameters from the learner and act in the environment, storing those transitions in the replay buffer.
The learner samples batches of transitions from the replay buffer and uses these to update the policy and Q-function networks.
For the ofﬂine RL setting, the dataset of transitions is given and ﬁxed (i.e., there are no actors) and the learner samples batches of transitions from that dataset.
To stabilize learning, we use the common technique of maintaining a target network for each trained network.
These target networks are used for computing gradients.
Every ﬁxed number of steps, the target network’s weights are updated to match the online network’s weights.
For optimization, we use Adam (Kingma & Ba, 2015).
For the ofﬂine RL experiments, we use Adam with weight decay to stabilize learning.
Gathering Data
The actors gather data and store it in the replay buffer.
When the policy is conditioned on trade-offs, the trade-off is ﬁxed for each episode.
In other words, at the start of each episode, the actor samples a trade-off α (cid:48) from the distribution ν , and acts based on π ( a | s, α (cid:48) ) for the remainder of the episode.
At the start of the next episode, the actor samples a different trade-off, and repeats.
Hyperparameters and Architecture.
The policy and Q-function networks are feed-forward networks, with an ELU activation after each hidden layer.
For both, after the ﬁrst hidden layer, we add layer normalization followed by a hyperbolic tangent ( tanh ); we found that this improves stability of learning.
The policy outputs a Gaussian distribution with a diagonal covariance matrix.
The default hyperparameters we use in our experiments are reported in Table 3; setting-speciﬁc hyperparameters are reported in Table 4.
Distributional Q-learning.
For our policy evaluation we use a distributional Q-network, C51 (Bellemare et al., 2017).
It has proven to be a very effective critic in actor-critic methods for continuous control (Barth-Maron et al., 2018; Hoffman et al., 2020).
We use the exact same training procedure as published and in open-source implementations (Barth-Maron et al., 2018; Hoffman et al., 2020), i.e., n -step return bootstrap targets and a projected cross-entropy loss.
See Table 3 under Q-learning for our chosen hyperparameters.
Evaluation.
We evaluate a deterministic policy, by using the mean of the output Gaussian distribution rather than sampling from it.
When we evaluate a trade-off-conditioned policy, we condition it on trade-offs linearly spaced from 0 .
05 to 1 . 0 .
For each trade-off α (cid:48) , we perform several rollouts where we execute the mean action from π ( a | s, α (cid:48) ) .
Recall that this trade-off speciﬁes a convex combination for two objectives.
For the multi-objective RL experiments, all policies are evaluated after 500 million actor steps.
For ofﬂine RL, all policies are evaluated after 1 million learner steps.
Learned Trade-off for Finetuning.
In our ﬁnetuning experiments, the learned trade-off is passed through the sigmoid function (i.e., 1 / (1 + exp( − x )) ) to ensure it is bounded between 0 and 1 .
Compute Resources.
To train a single policy for the multi-objective RL experiments (humanoid walk and run), we used an NVIDIA v
GPU for about 24 hours.
To train a single policy for ﬁnetuning, we used a NVIDIA V100 GPU for 9-12 hours.
To train a single policy for ofﬂine RL, we used a v2-32 TPU for about 6 hours for the RL Unplugged tasks, and for about 3 hours for the D4RL tasks.
These computational costs were the same for all algorithms we ran: LS and the DiME variants.
D E XPERIMENTS : D ETAILS AND A DDITIONAL R ESULTS
This section provides details regarding our experiment domains, as well as additional experiments and plots.
D. M ULTI -O BJECTIVE RL D.1. T OY D OMAIN
Our toy domain is a bandit with a continuous action a ∈ R .
The reward r ( a ) ∈ R 2 is speciﬁed by either the Schaffer function or the Fonseca-Fleming function, f : R (cid:55)→ R 2 .
Each point along this function is optimal for a particular tradeoff between the two reward objectives, so the function can be seen as a Pareto front.
The Schaffer function corresponds to a convex Pareto front, and is deﬁned as
[Equation]
) = aand f 2 ( a ) =
( a − 2) 2 .
[Table]
[Table]
[Figure]
The Fonseca-Fleming function corresponds to a concave Pareto front, and is deﬁned as f
[Equation]
and f 2 ( a [Equation] = 1 − exp( − ( a + 1) 2 ) . These are standard test functions in multi-objective optimization, where the aim is to minimize them (Okabe et al., 2004).
Since in RL we want to maximize reward, we deﬁne the reward to be the negative of these functions.
D.1. H UMANOID
We also evaluate on the humanoid walk and run tasks from the open-sourced DeepMind Control
Suite (Tassa et al., 2018), with the two objectives proposed in Abdolmaleki et al.
The ﬁrst objective is the original task reward, which is a shaped reward that is given for moving at a target speed ( 1 meters per second for walking and 10 m/s for running), in any direction.
The second objective is the negative (cid:96) 2 -norm of the actions, −(cid:107) a (cid:107) 2 , which can be seen as encouraging the agent to be more energy-efﬁcient.
The observations are 67 -dimensional, consisting of joint angles, joint velocities, center-of-mass velocity, head height, torso orientation, and hand and feet positions.
The actions are 21 -dimensional and in [ − 1 , 1] ; these correspond to setting joint accelerations.
Each episode has 1000 timesteps.
At the start of every episode, the conﬁguration of the humanoid is randomly initialized.
Additional Results.
When we plot the per-episode performance (rather than an average across episodes), we see that linear scalarization only ﬁnds solutions at the two extremes for the humanoid walk task, i.e. policies that achieve high task reward but with high action norm cost, or incur zero cost but fail to obtain any reward (Fig. 4a).
We hypothesize that this may be because the humanoid walk task has a concave Pareto front of solutions; recall that linear scalarization is fundamentally unable to ﬁnd solutions on a concave Pareto front (Das & Dennis, 1997).
In contrast, DiME is able to ﬁnd solutions that trade off between the two objectives.
On both tasks, DiME ﬁnds a similar Pareto front as MO-MPO, which is a state-of-the-art approach for multi-objective RL (Fig. 4b).
DiME requires training a separate policy per trade-off, which can be computationally expensive.
In contrast, our trade-off-conditioned version of DiME, which we call DiME multi, trains a single policy for a range of trade-offs.
This obtains a similar Pareto front compared to that found by DiME (Fig.
5), after the same number of actor steps (500 million).
We also ran an experiment to test the scale-invariance of DiME.
We modiﬁed the humanoid run task such that the action norm cost is now multiplied by ten.
We used DiME and linear scalarization to train policies for three random seeds, for each of two trade-off settings.
For both the original task and the task with scaled-up action norm costs, DiME trains policies with similar performance (Fig. 6, right).
This is the case for both trade-off settings—note the two clusters of differently-colored points in the plot.
In contrast, linear scalarization is not scale-invariant: task performance suffers when the cost is scaled up (Fig. 6, left).
With a trade-off of 0 .
05 , although this obtains better performance than [Figure] [Figure] [Figure]
[Figure]
t a s k r e t u r n L S D i M E a c t i o n n o r m the trade-off of 0 . 01 with the original objectives (i.e., gets higher reward with lower cost), with the scaled-up action norm cost, policies do not obtain any task reward at all with a trade-off of 0 .
Finally, we delve into why DiME outperforms LS. Our hypothesis is that this is because DiME computes an improved policy for each objective (before projecting them), and is thus more likely to ensure improvement with respect to all objectives. In contrast, since LS combines rewards directly, it cannot guarantee improvement with respect to all objectives. To test out this theory, we plotted how policies trained by LS and DiME perform with respect to each objective over the course of training, for humanoid run.
We found that policies trained with DiME indeed exhibit improvement
[Figure]
[Figure]
with respect to both objectives simultaneously over the course of training, whereas policies trained with LS typically suffer from worsening performance for one objective while improving for the other.
In our ofﬂine RL experiments, we evaluate on two open-sourced benchmarks for ofﬂine RL: RL
Unplugged (Gulcehre et al., 2020) and D4RL (Fu et al., 2020).
We use the nine Control Suite tasks from RL Unplugged, the six Ant Maze tasks from D4RL, and the three Franka Kitchen tasks from
D4RL.
For each task, we train policies on the dataset of transitions provided by the benchmark.
These are continuous control tasks, that include particularly challenging ones that state-of-the-art approaches perform poorly on (e.g., Ant Maze large play and large diverse).
For a comprehensive description of the environments, please refer to Gulcehre et al.
(2020) and Fu et al.
[Table]
Additional Results.
Here we show plots for per-tradeoff performance for all nine Control Suite tasks from RL Unplugged (Fig. 8) and for the Ant Maze and Franka Kitchen tasks from D4RL (Fig.
We also provide the full table of results for the D4RL tasks (Table 5).
Across all 18 tasks, the best tradeoff found by DiME obtains either higher or on-par performance compared to the best tradeoff found by LS.
Recall that LS is equivalent to CRR, a state-of-the-art approach for ofﬂine RL, in terms of the policy loss it optimizes for.
D. F INETUNING
For ﬁnetuning, we evaluate on three humanoid tasks (stand, walk, and run) and two manipulator tasks (insert ball and insert peg) from the DeepMind Control Suite.
There are two main differences between ﬁnetuning and ofﬂine RL: 1) in ﬁnetuning the policy is able to interact with the environment, whereas in ofﬂine RL it cannot, and 2) in ﬁnetuning the behavioral prior may be in the form of a teacher policy that we can query to obtain π b ( a | s ) , whereas in ofﬂine RL the behavior prior is in the form of a dataset of transitions.
For the humanoid tasks, we trained a policy for humanoid stand with standard deep RL (using
MPO (Abdolmaleki et al., 2018)) and stopped it midway through training, when it reached an average task reward of about 400 .
We then used this teacher policy for ﬁnetuning in all three humanoid tasks.
However, for the manipulator tasks, training a policy from scratch is highly sample-inefﬁcient, as can be see by the learning curves for learning from scratch (i.e., α = 0 ) in Fig.
So for manipulator insert ball and insert peg, we took the best policy trained using DiME in our ofﬂine RL experiments, and used that as the teacher policy for ﬁnetuning in that task.
Fig.
10 shows learning curves for all ﬁve tasks.
When the tradeoff is learned (in orange), DiME and linear scalarization perform on-par.
However, when the tradeoff is ﬁxed, for linear scalarization it is more difﬁcult to pick an appropriate tradeoff, because the appropriate tradeoff depends on the relative scales of the rewards.
In fact, for the humanoid tasks, none of the ﬁxed (non-zero) tradeoffs for LS perform better than learning from scratch.
[Figure]