D ATA S HARING WITHOUT R EWARDS IN M ULTI -T ASK O FFLINE R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Ofﬂine reinforcement learning (RL) bears the promise to learn effective control policies from static datasets but is thus far unable to learn from large databases of heterogeneous experience.
The multi-task version of ofﬂine RL enables the possibility of learning a single policy that can tackle multiple tasks and allows the algorithm to share ofﬂine data across tasks.
Recent works indicate that sharing data between tasks can be highly beneﬁcial in multi-task learning.
However, these beneﬁts come at a cost – for data to be shared between tasks, each transition must be annotated with reward labels corresponding to other tasks.
This is particularly expensive and unscalable, since the manual effort in annotating reward grows quadratically with the number of tasks.
Can we retain the beneﬁts of data sharing without requiring reward relabeling for every task pair?
In this paper, we show that, perhaps surprisingly, under a binary-reward assumption, simply utilizing data from other tasks with constant reward labels can not only provide substantial improvement over only using the single-task data and previously proposed success classiﬁers, but it can also reach comparable performance to baselines that take advantage of the oracle multi-task reward information.
We also show that this performance can be further improved by selectively deciding which transitions to share, again without introducing any additional models or classiﬁers.
We discuss how these approaches relate to each other and baseline strategies under various assumptions on the dataset.
Our empirical results show that it leads to improved performance across a range of different multi-task ofﬂine RL scenarios, including robotic manipulation from visual inputs and ant-maze navigation.
I NTRODUCTION
Ofﬂine reinforcement learning (RL) provides the promise of a fully data-driven framework for learning performant policies.
To avoid costly active data collection and exploration, ofﬂine RL methods utilize a previously collected dataset to extract the best possible behavior, making it feasible to use RL to solve real-world problems where active exploration is expensive, dangerous, or otherwise infeasible (Zhan et al., 2021; de Lima & Krohling, 2021;
Wang et al., 2018; Kalashnikov et al., 2018).
However, this concept is only viable when a signiﬁcant amount of data for the target task is available in advance.
A more realistic scenario might allow for a much smaller amount of task-speciﬁc data, combined with a large amount of task-agnostic data, that is not labeled with task rewards and some of which may not be relevant.
For example, if our goal is to train a robot to perform a new manipulation task (e.g., cutting an onion), we might have some data of the robot (suboptimally) attempting that task, perhaps collected under human teleoperation and manually labeled with rewards, combined with plentiful data of other tasks, some of which might be structurally related (e.g., picking up an onion, or cutting a carrot).
This scenario presents several questions: How do we decide which prior data should be included when learning the new task?
And how do we determine which reward labels to use for this prior data?
Prior methods have offered several potential answers to these two questions, typically in isolation.
For the ﬁrst question, it has been recently observed that a na¨ıve sharing strategy of sharing data from all tasks can be highly suboptimal (Kalashnikov et al., 2021), and some works have proposed both manual (Kalashnikov et al., 2021) and automated (Yu et al., 2021a;
Eysenbach et al., 2020) data-sharing strategies that prioritize the most structurally similar prior data.
Most such methods assume that this shared data can be automatically relabeled with the reward function for the new task (Kalashnikov et al., 2021; Yu et al., 2021a; Eysenbach et al., 2020), but the assumption that we have access to the functional form of this reward is a strong one: for example, in many real-world settings, the reward might require human labeling or human-provided examples (Cabi et al., 2019;
Finn et al., 2016b).
To this end, some prior works have proposed learning classiﬁers for reward labeling (Fu et al., 2018b;
Xie et al., 2018; Singh et al., 2019), or other automated mechanisms (Konyushkova et al., 2020).
But these mechanisms themselves add complexity and potential brittleness to the pipeline.
Thus, we aim to devise a simple uniﬁed method that determines which data to share and which rewards to use, with minimal supervision and no additional modeling and learning.
In this paper, we make the potentially surprising observation that data from other tasks can be utilized with na¨ıve constant reward labels, when the MDP consists of binary rewards.
We show that this simple method, which does not involve learning any additional models or classiﬁers, can outperform more sophisticated techniques in practice.
Our approach simply utilizes data from other tasks with a constant reward label (e.g., r = 0 ), and uses a value-aware strategy to decide which prior transitions to include for the new task.
This strategy, based on the conservative data sharing (CDS) technique proposed in prior work (which assumes oracle reward access) (Yu et al., 2021a), also does not require learning any additional model and simply uses the Q-function that is already learned as part of the
RL process.
Our main contribution, which we call conservative unsupervised data sharing (CUDS), is a technique for sharing data in multi-task ofﬂine RL that does not require any reward labels or reward function access for the task-agnostic data, and requires no additional model or classiﬁer.
To achieve that, our method assumes a particular form of the MDP that consists of binary rewards.
We discuss the behaviors of our methods, showing that, even without ground truth reward labels, our simple data sharing scheme achieves Q-values that are lower-bounded by the Q-values obtained with sharing all data with the ground-truth rewards and can be combined CDS to selectively ﬁlter out potentially irrelevant data under different assumptions on the structure of the dataset.
Our empirical evaluation conducted over various multi-task ofﬂine RL scenarios such as robotic manipulation from visual inputs and ant-maze navigation shows that this approach improves over the performance of more sophisticated techniques that either learn the reward function explicitly, or utilize other methods to propagate reward labels.
In addition, we show that the proposed approach is comparable to an oracle baseline that has access to true multi-task rewards.
R ELATED W ORK
Ofﬂine RL.
Ofﬂine RL (Ernst et al., 2005; Riedmiller, 2005; Lange et al., 2012; Levine et al., 2020) considers the problem of learning a policy from a static dataset without interacting with the environment, which has shown promises in many practical applications such as robotic control (Kalashnikov et al., 2018; Mandlekar et al., 2020; Rafailov et al., 2021), NLP (Jaques et al., 2019), healthcare (Shortreed et al., 2011; Wang et al., 2018), education (de Lima & Krohling, 2021), electricity supply (Zhan et al., 2021) and UI design (Apostolopoulos et al., 2021).
The main challenge of ofﬂine RL is the distributional shift between the learned policy and the behavior policy (Fujimoto et al., 2018; Kumar et al., 2019), which can cause erroneous value backups due to out-of-distribution actions generated by the learned policy.
To address this issue, prior methods have constrained the learned policy to not deviate much from the behavior policy via policy regularization (Liu et al., 2020;
Jaques et al., 2019; Wu et al., 2019;
Zhou et al., 2020; Kumar et al., 2019; Siegel et al., 2020; Peng et al., 2019;
Zhou et al., 2020; Kostrikov et al., 2021; Ghasemipour et al., 2021), conservative value functions (Kumar et al., 2020; Sinha & Garg, 2021), an auxiliary behavioral cloning loss (Fujimoto &
Gu, 2021) and model-based training with conservative penalties (Yu et al., 2020d; Kidambi et al.,
[Table] [Table]
[Table] [Table]
, 2021; Yu et al., 2021b).
Multi-Task RL and data sharing.
Multi-task RL (Wilson et al., 2007; Parisotto et al., 2015;
Teh et al., 2017; Espeholt et al., 2018; Hessel et al., 2019; Yu et al., 2020a; Xu et al., 2020; Yang et al., 2020; Kalashnikov et al., 2021; Sodhani et al., 2021; Stooke et al., 2021) enables the goal of learning a single policy that solves multiple skills efﬁciently.
Despite the promising results, multitask RL suffers from three main challenges, optimization difﬁculties (Schaul et al., 2019; Hessel et al., 2019; Yu et al., 2020a), effective weight sharing for learning shared representations (Parisotto et al., 2015;
Teh et al., 2017; Espeholt et al., 2018; Xu et al., 2020; D’Eramo et al., 2019; Sodhani et al., 2021; Stooke et al., 2021), and sharing data across different tasks (Eysenbach et al., 2020;
Kalashnikov et al., 2021; Yu et al., 2021a).
We consider the multi-task ofﬂine RL setting and focus on the challenge of sharing data across different tasks.
Prior works share data across tasks based on metrics such as learned Q-values (Eysenbach et al., 2020;
Li et al., 2020; Yu et al., 2021a), human domain knowledge (Kalashnikov et al., 2021), the distance to the target goals in goal-conditioned settings (Andrychowicz et al., 2017; Pong et al., 2018; Nair et al., 2018; Liu et al., 2019; Sun et al., 2019; Lin et al., 2019; Huang et al., 2019; Lynch & Sermanet, 2020; Yang et al., 2021; Chebotar et al., 2021), and the learned distance with robust inference in the ofﬂine meta-RL setting (Li et al., 2019).
However, all of these either require access to the functional form of the reward functions of each task in order to relabel the rewards or are limited to goal-conditioned settings.
Therefore, they are not applicable to the multi-task ofﬂine RL problem that we consider where only the reward label of the originally-executed task is provided.
Our work addresses this issue via simply relabeling the data shared from other tasks with a constant value and uses the conservative data sharing strategy (Yu et al., 2021a) to further improve the performance.
RL with unlabeled data.
Prior works tackle the problem of learning from data without reward labels via either directly imitating expert trajectories (Pomerleau, 1988; Ross & Bagnell, 2012; Ho &
Ermon, 2016), learning reward functions from expert data using inverse RL (Abbeel & Ng, 2004; Ng & Russell, 2000; Ziebart et al., 2008; Finn et al., 2016a; Fu et al., 2018a;b; Konyushkova et al., 2020), or learning a reward / value classiﬁer that discriminates successes and failures (Xie et al., 2018;
Singh et al., 2019;
Eysenbach et al., 2021).
These algorithms require online data collection and do not consider the ofﬂine RL setting.
Singh et al.
(2020) considers the single-task ofﬂine setting with both task-speciﬁc datasets and task-agnostic prior datasets and relabel the unlabeled prior data as failures since these prior transitions cannot solve the task.
Our method is not limited to such single-task settings and instead considers the more general multi-task ofﬂine RL with data-sharing problem.
P RELIMINARIES
Multi-task RL.
Standard multi-task RL considers a multi-task Markov decision process (MDP),
M = ( S , A , P, γ, { R i , i } Ni =1 ) , where S and A denote the state and action spaces respectively,
P ( s (cid:48) | s , a ) denotes the dynamics, γ ∈ [0 , 1) is the discount factor, and R 1 , · · · , R N correspond to reward functions of different tasks i ∈
[ N ] for total number of N tasks where [ N ] is the shorthand for { 1 , 2 , . .
. , N } .
In our setting, we assume a binary per-task R i ∈ { 0 , 1 } , where 1 denotes success of the task and 0 otherwise.
Note that the dynamics are assumed to be the same across all tasks, which is not entirely general but is indeed practical in many problem settings as noted in Yu et al.
(2021a)
and stands as a common assumption in prior data sharing works (Yu et al., 2021a;
Kalashnikov et al., 2021;
Eysenbach et al., 2020).
Regardless, there are many practical scenarios with changing rewards and invariant dynamics such as various object manipulation objectives (Xie et al., 2018), different goal navigation tasks (Fu et al., 2020), and distinct user preferences (Christiano et al., 2017).
The goal of multi-task RL is to ﬁnd a task-conditioned policy π ( a | s , i ) that expected return in a multi-task
MDP: π ∗ ( a | s , · ) :=
arg max π E i ∼ [ N ] E π ( ·|· ,i ) [ (cid:80) t γ t R i ( s t , a t )] .
Note that it is possible to model the policies for each task independently as { π 1 ( a | s ) , · · · , π N ( a | s ) } without any weight sharing.
In our work, we use the single task-conditioned policy to study data sharing and do not consider the weight sharing aspect, which is orthogonal to the focus of the paper, which is also noted in Yu et al.
(2021a).
Multi-task ofﬂine RL and data sharing.
Multi-task ofﬂine RL considers the problem of learning the multi-task policy π ( a | s , i ) from a static multi-task dataset with D = ∪ Ni =1 D i where D (cid:105) =
{ ( s j , a j , s (cid:48) j , r j ) } Mj =1 is the per-task dataset.
D i is generated by a behavior policy π β ( a | s ) , without any interaction with the environment.
The most straightforward approach to learn π ( a | s , i ) would be train it for task i only using D i .
However, sharing data from different tasks to task i has been shown to be conducive in the multi-task ofﬂine RL setting (Kalashnikov et al., 2021; Yu et al., 2021a).
To do so, prior works (Eysenbach et al., 2021;
Kalashnikov et al., 2021; Yu et al., 2021a) assume access to the functional form of the reward r i , which is a rather strong assumption that is usually impractical to specify in practical applications due to the challenge of reward speciﬁcation.
The next straightforward approach is to na¨ıvely sharing data across all tasks, denoted as Sharing All.
Formally,
Sharing All deﬁnes the dataset of transitions relabeled from task j to task i as D j → i and the method can be then deﬁned as D eﬀ i :
= D i ∪ ( ∪ j (cid:54) =
i D j
→ i ) , where D eﬀ i denotes the effective dataset for task i .
While Sharing All improves over not sharing data, as shown in Yu et al.
(2021a), Sharing All leads to distributional shift that could degrade performance in certain situations (Yu et al., 2021a).
In our work, we focus on the CDS (Yu et al., 2021a), which relabels data that aims to mitigate the distributional shift introduced by sharing other task data.
CDS addresses such an issue by proposing a conservative data sharing strategy as follows:
[Equation]
where s j , a j , s (cid:48)
j denote the transition from D j , r i denotes the reward of s j , a j , s (cid:48)
j relabeled for task i , π denotes the task-conditioned policy π ( ·|· , i ) , ∆ π ( s j , a j ) is the condition that shares data only if the expected Q-value of the relabeled transition exceeds the top k -percentile of the Q-values of the original task data, i.e.
[Equation]
Beyond controlling the distributional shift introduced in data sharing, multi-task ofﬂine RL also needs to address the main challenge in standard ofﬂine RL, which is the distributional shift between the learned policy π and the behavior policy π β .
To handle both types of distributional shifts, CDS (Yu et al., 2021a) combines the conservative data sharing and the constrained policy optimization problem and arrives at the following objective:
[Equation] [Equation]
where π eﬀ β ( a | s , i ) is the effective behavior policy for task i denoted as π eﬀ β ( a | s , i ) :
=
|D
eﬀ i ( s , a ) | / |D
eﬀ i ( s ) | , J D eff i ( π ) denotes the average return of policy π in the empirical MDP induced by the effective dataset, and D ( π, π eﬀ β ) denotes a divergence measure (e.g., KL-divergence (Jaques et al., 2019; Wu et al., 2019), ﬁsher divergence (Kostrikov et al., 2021), MMD distance (Kumar et al., 2019) or D CQL from conservative Q-learning (Kumar et al., 2020)) between the learned policy π and the effective behavior policy π eﬀ β .
While optimizing Eq.
3 with Eq.
1 as the data sharing scheme is able to mitigate distributional shift and improve over multi-task ofﬂine RL without sharing data and na ¨ ıvely sharing data across all tasks as shown in Yu et al.
(2021a), it requires the assumption of the access to the functional form of the reward functions, which is rather strong and make application of data sharing to real-world applications impractical.
We will instead present a simple yet effective data sharing and relabeling scheme in the setting where we do not make such an assumption and instead, only have the reward labels for originally commanded task in the following section.
D ATA S HARING WITHOUT R EWARDS IN M ULTI -T ASK O FFLINE RL
The goal of our method is to enable effective data sharing across different tasks without access to the functional form of the reward functions for each task.
Data from each task is only labeled for that particular task, and we do not know a priori which data is relevant to each task.
Effective data sharing therefore requires resolving two questions: (i) which data from other tasks should we use for a given task?
and (ii) how do we label this data with rewards?
One simple approach is to annotate all available data from other tasks with some “proxy” reward signal, and treat it no differently from data that is already labeled.
That is, after relabeling with the proxy reward, we can simply put these transitions into the replay buffer of a value-based ofﬂine RL method.
But how can we obtain a reliable proxy reward signal?
Next, we will discuss two variants of our method in Section 4.1, understanding of both variants in Section 4.2, and practical implementations in Section 4.3.
4. C ONSERVATIVE
Prior work assumes that it is necessary to relabel prior data with some estimate of the true reward function so that the proxy reward closely reﬂects the true reward.
We take a different approach, and instead argue that, under some assumptions, we can obtain many of the beneﬁts of data sharing simply by labeling the multi-task data with the lowest possible reward, which we assume to be 0 without loss of generality in the binary-reward setting.
We refer to this simple strategy as unsupervised data sharing (UDS).
Na¨ıve UDS prescribes sharing data from every task to every other task, and labels the shared data with a reward value of 0.
Formally, for each task i ∈ [ N ] , we deﬁne the UDS procedure as follows:
[Equation]
Intuitively, UDS relabels data shared from other tasks with the lowest possible reward, hence making the learned Q-functions more conservative that the data sharing scheme with the oracle rewards.
We will show that UDS learns Q-values that are lower-bounded by the Q-values learned by the na¨ıve
Sharing All scheme with true reward relabeling, and can be information-theoretically optimal in ofﬂine RL thanks to such conservatism.
Our empirical results in Section 5 also suggests that the beneﬁts from data sharing outweigh the downsides of reward bias in practice.
Next, we move on from the choice of proxy reward, to the choice of which data should be shared.
While UDS is simple yet effective in multi-task data sharing without reward relabing, na¨ıvely sharing data from all other tasks with zero rewards in the ofﬂine RL setting can result in overly conservative
Q-functions and policies.
To further reﬁne this strategy, we can adapt the CDS algorithm (Yu et al., 2021a)
detailed in Section 3 to ﬁlter out irrelevant transitions from other tasks, and only share those transitions that are likely to be informative.
We call this strategy conservative unsupervised data sharing (CUDS).
We deﬁne the CUDS strategy as follows:
[Equation]
As we will discuss in the next subsection, CUDS is able to select potentially useful transitions under certain structural assumptions on the multi-task ofﬂine dataset, and therefore produce Q-values that are not as excessively conservative as those produced by UDS.
As shown in Section 5, our empirical evaluation further validates that CUDS improves over UDS and prior approaches.
Algorithm 1 (Conservative)
Unsupervised Data Sharing
Require: Multi-task ofﬂine datasets ∪ Ni =1 D i .
1: Randomly initialize policy π θ ( a | s , i ) .
2: for k = 1 , 2 , 3 , ·
, do 3: Initialize
D eﬀ ← {} 4: for i = 1 , · · · , N do 5:
D eﬀ i
= D i ∪{ ( s j , a j , s (cid:48)
j , 0) ∈ D
j → i ∀ j ∈ [ N ] \{ i }} (UDS) or D eﬀ i
= D i ∪{ ( s j , a j , s (cid:48)
j , 0) ∈ D
j → i : ∆ π ( s , a ) ≥ 0 ∀ j ∈ [ N ] \ { i }} using Eq.
2 (CUDS).
Perform policy improvement by solving Eq. 3 by sampling data from D eﬀ .
4. U NDERSTANDING THE B EHAVIOR OF UDS AND CUDS
In this section, we aim to understand the behavior of the UDS and CUDS.
We ﬁrst consider the
UDS scheme, which simply shares all available data from other tasks, and labels the reward for each transition from other tasks as 0 .
When instantiated with CQL as the ofﬂine RL method, the Q-values of a given policy learned by UDS for each task i are the ﬁxed point of the recursion:
[Equation]
where (cid:98) r ( s , a , i ) =
for all ( s , a ) ∈ D
j → i .j
(cid:54)
=
i , and (cid:98) r ( s , a , i ) is equivalent to the empirical reward observed otherwise (Kumar et al., 2020).
We will now try to understand how UDS compares to the No Sharing strategy, which only uses the labeled data for training.
Note that this comparison is non-trivial since, while UDS utilizes a larger dataset, it can induce signiﬁcant reward bias during training.
However, by assumption, 0 is the lowest possible reward, we would intuitively expect that
UDS should be more conservative, compared to Sharing All that relabels with the true reward.
While we may surmise that being too conservative on unlabeled data may be suboptimal, conservatism has been shown to be information-theoretically optimal (Jin et al., 2021; Rashidinejad et al., 2021) in ofﬂine RL and bandit problems.
Even though an unlabeled dataset provides us with information about environment dynamics, it does not provide information about rewards, and any optimistic estimate of reward on this data may lead to poor performance in the worst case.
We formally derive the performance guarantee for UDS in Proposition F.1 using the framework of safe-policy improvement and discuss cases where it can perform better than No Sharing.
We discuss in Appendix F.2.3 that UDS can perform better than No Sharing in long horizon tasks as well as in cases where the unlabeled dataset consists of similar proportions of various state-action pairs as the labeled dataset.
Please refer to this section for the theoretical results.
Our bounds utilize a new technique that allows us to prove tighther, non-trivial bounds for UDS.
despite the pessimism which is discussed in Appendix F.2.1.
To understand the behavior of CUDS, we will consider a simple abstract model of CUDS-style relabeling.
In the tabular setting, this model updates the Q-function to match the target (conservative)
Q-value if the transition is selected for the update, and retains the old table entry otherwise.
Formally, consider a binary vector w ∈ R |S|×|A| that indicates whether a corresponding state-action pair ( s , a ) is utilized for the backup or not.
Then, our weighted scheme performs the following backups:
[Equation]
Equation 7 can be intuitively understood as performing a conservative backup from the actual transition observed in the dataset when the binary weight w ( s , a ) = 1 , and simply truncating the Bellman backup and retaining the previous Q-values (cid:98) Q k ( s , a , i ) , otherwise.
For example,
CUDS performs a conservative backup with (cid:98) r ( s , a , i ) ≤ (cid:98)
r Sharing All ( s , a , i ) only on transitions where w ( s , a ) =
I [∆ π ( s , a ) ≥ 0] .
To understand how this affects the resulting Q-function, we consider two structural conditions on the ofﬂine dataset: (1) a scenario where no trajectory in the relabeled dataset for a given target task
D j → i actually visits state-action tuples that were observed in D i , and (2) when trajectories in D j → i overlap with at least a fraction of state-action tuples in the original labeled data for this task D i .
We will abstract CUDS as utilizing w k ( s , a ) =
I [ (cid:98) Q k ( s , a , i ) ≥ ι ] for some threshold ι (see Eqn. 2).
Remark 4.1 (CUDS reduces to no sharing under condition (1) ) .
When the trajectories in the unlabeled, relabeled dataset do not overlap with any trajectory in the labeled dataset for a given task, any backup performed by CUDS on an unlabeled transition will eventually drive its Q-value to 0 as k → ∞ .
Thus, CUDS weights w k ( s , a ) will eventually take on 0 values for such transitions, and will not be selected by the future weights, i.e., w j ( s , a ) =
∀ j ≥ k + 1 .
Perhaps unsurprisingly, when the unlabeled data has no overlap with the labeled data, CUDS reduces to no sharing.
However, the more practically relevant case is when the unlabeled data overlaps with the labeled data.
We consider the scenario when UDS has been run initially to obtain a starting set of
Q-values, (cid:98) Q 0 ( s , a , i ) , which deﬁnes the initial weight vector.
Remark 4.2 (CUDS selects more useful unlabeled transitions) .
Imagine a transitions ( s , a , s (cid:48) , 0) ∈
D j → i for which the next state (and the next policy action) ( s (cid:48) , a (cid:48) ) are observed in the labeled dataset (denoted D i ).
This transition will will attain large initial Q-values (cid:98) Q 0 ( s , a , i ) if executing the policy after ( s (cid:48) , a (cid:48) ) eventually reaches the state that corresponds to a high reward of 1.0, due to the Bellman backup component of CUDS.
However, on the ﬂip side, these backups performed by CUDS are conservative, and performing more backups can reduce the Q-value.
Two scenarios might then arise: (i) the Q-values eventually decrease and CUDS is deactivated, i.e., ∃ k, w k ( s , a ) = 0 , in which case this transition is discarded and not used for learning anymore as the backup in Equation 7 preserves the Q-value (the second term) when w k ( s , a ) = 0 , or (ii) the learning process reaches an equilibrium where w k ( s , a ) = 1 ∀ k , meaning that this relabeled transition is used for learning.
We have now provided a theoretical analysis of CUDS and a comparison between CUDS and UDS in
Appendix F.2.
Additionally, we provide several new experiments to build insight into why UDS and
CUDS work in Appendix G.
4. P RACTICAL I MPLEMENTATIONS
We present pseudocode for UDS and CUDS in Algorithm 1.
We train the Q-values with CQL to obtain conservative Q-values, and use the conservative Q-values to compute ∆ π ( s , a ) deﬁned in Eq. 2.
For CUDS, in practice, instead of computing the hard threshold of ∆ π ( s , a ) ≥ 0 to determine data sharing, we follow Yu et al.
(2021a)
and transform the condition ∆ π ( s , a ) ≥ 0 into a soft weighting scheme, with weights given by w CUDS ( s , a ; j → i ) :
= σ (cid:16) ∆( s , a ; j → i ) τ (cid:17) , where τ is a hyperparameter for the temperature of the sigmoid term in w CUDS that is automatically selected by the running average of ∆( s , a ; j → i ) .
These weights are applied to both critic and actor training.
For both UDS and CUDS, we train a policy π ( a | s , i ) where π ( a | s , i ) could either be a single task-conditioned task with weight sharing or separate policies for each task without weight sharing.
For more details of the practice implementations, see Appendix B.
E XPERIMENTS
In this section, we present our empirical evaluation, which aims to answer the following questions: (1)
Can our simple approach outperform prior methods for utilizing unlabeled ofﬂine data on multi-task ofﬂine datasets?
(2) Can the conservative data sharing strategy further improve the results achieved by our method?
(3) Is our approach able to attain competitive result compared to the prior multi-task ofﬂine RL algorithm that have access to the true rewards?
(4) How does CUDS compare to prior ofﬂine RL methods that directly learn representations from the multi-task ofﬂine dataset and run ofﬂine training on top of the representation?
Comparisons.
We compare UDS and CUDS to a number of prior methods.
We ﬁrst evaluate: No
Sharing , which performs applies standard ofﬂine RL algorithm to the multi-task setting without sharing data across tasks, Reward Predictor , which learns a classiﬁer that directly predicts the reward using supervised learning, VICE (Fu et al., 2018c), an inverse RL method that learns a reward classiﬁer from the labeled data and then annotates the unlabeled data with the learned classiﬁer, and
RCE (Eysenbach et al., 2021), a method similar to VICE except that RCE represents the Q-function as a classiﬁer and learns the reward for unlabeled data implicitly.
We adapt VICE and RCE to the multi-task ofﬂine RL setting by extracting transitions with reward labels equal to 1 and treating these datapoints as positives to learn the classiﬁer for each task.
We also train VICE and RCE , but adapt them to the ofﬂine setting using CQL, i.e. the same base ofﬂine RL method as in UDS and
CUDS .
Finally, to answer question (4), we conduct empirical evaluations on ACL (Yang & Nachum, 2021), which is a recent ofﬂine RL algorithm that performs representation learning on the ofﬂine dataset and trains the policy on top of the representation.
For more details for experimental set-up and hyperparameter settings, please see Appendix B. We also include evaluations of our methods under different quality of the relabeled data in Appendix C, results of UDS and CUDS in dense-reward settings in Appendix D, comparisons to model-based ofﬂine RL approaches in Appendix E, and empirical analysis of the reasons that UDS and CUDS work in Appendix G.
5. M AIN E VALUATION
To answer questions (1), (2), and (3), we perform empirical evaluations on two state-based multi-task robotic manipulation and navigation datasets and one image-based multi-task manipulation dataset introduced in prior work (Yu et al., 2021a), which we will discuss below.
Tasks and Datasets.
Following the experimental setup in prior work (Yu et al., 2021a), we consider three domains shown in Fig.
1: (i) the Meta-World (Yu et al., 2020b) domain, which consists of four tasks of opening and closing doors and drawers; (ii) the Antmaze (Fu et al., 2020) domain, which consists of two sizes of mazes (medium and large) with 3 and 7 tasks respectively; and (iii) the multi-task visual manipulation domain, which consists of 10 tasks with different combinations of object-oriented grasping, with 7 objects (banana, bottle, sausage, milk box, food box, can and carrot), and placing the picked objects onto one of three ﬁxtures (bowl, plate and divider plate).
For all domains, we use binary rewards, where 1 denotes the successful completion of the task andcorresponds to failure.
Note that for Meta-World, we use a ﬁxed 200 timesteps for each episode and do not terminate the episode when receiving a reward of 1 at an intermediate timestep.
In
Antmaze, we terminate the episode upon seeing a reward of 1 with the maximum
[Table] [Table]
we use large datasets with wide coverage of the state space and 152K transitions for the door open and drawer close tasks and datasets with limited (2K transitions)
[Table]
but optimal demonstrations for the door close and drawer open tasks.
For AntMaze, following Yu et al.
(2021a), we modify the datasets introduced by Fu et al.
(2020) by equally dividing the large dataset into different parts for different tasks, where each task corresponds to a different goal position.
For image-based manipulation, we directly use the dataset collected by Yu et al.
(2021a), which contains a total of 100K RL episodes with 25 transitions for each episode, where the success rate is 40% and 80% for the picking and placing tasks, respectively.
Note that the success rate of placing is higher because the robot is already holding the object at the start of the placing tasks, making the placing easier to solve.
Results of Question (1).
The main results are in Table 1.
UDS achieves better performance than vanilla multi-task ofﬂine RL without data sharing and compared to reward learning methods, suggesting that our simple relabeling method is effective in both multi-task manipulation and navigation domains.
Since the reward learning approaches obtain similar or worse results compared to no sharing, we only compare our methods to No Sharing and the oracle methods in the image-based experiments.
As shown in Table 2.
UDS outperforms No Sharing in 7 out of 10 tasks as well as the average task performance by a signiﬁcant margin.
Therefore, UDS is able to effectively leverage unlabeled data shared from other tasks and achieves potentially surprisingly strong results compared to more sophisticated methods that handle unlabeled ofﬂine data, answering question (1).
Results of Question (2).
In both the state-based and vision-based experiments shown in Tableand Table 2, we ﬁnd that CUDS further improves upon the performance of UDS , which empirically indicates that the less conservative policy learned from CUDS’s selective ﬁltering scheme is more performant in practice.
Additionally, we measure the success rates of the relabeled data in Table
Appendix C, measured by the oracle multi-task reward function on the Meta-World and AntMaze domain.
We see that the success rates of the relabeled data are above 0% by a signiﬁcant margin in most of the tasks.
This suggests that UDS and CUDS are not simply relabeling with the true reward, since the relabeled data does not entirely consist of failures but rather has a signiﬁcant number of successful transitions.
Results of Question (3).
We also show results for oracle methods that receive true reward labels:
Sharing All , which shares all data with ground truth rewards, and CDS , which uses the CDS strategy (Yu et al., 2021a) with ground truth reward relabeling.
We present the results in Table 3 for state-based experiments and the last two columns on the right in Table 2 for the vision-based multi-
[Table]
[Table]
task robotic manipulation problem.
Both CUDS and UDS achieves competitive results compared to
CDS and Sharing All , indicating that our simple relabeling scheme is able to remove the dependence of functional form of reward functions without much loss of performance due to lacking ground-truth reward access.
This addresses question (3).
Results of Question (4).
Finally, to answer question (4), on the Meta-World environment, we compare UDS and CUDS to ACL (Yang & Nachum, 2021).
We use the version of ACL without inputting reward labels.
ACL can be viewed as an alternative to our unlabeled sharing data scheme, which leverages unlabeled data for representation learning rather than sharing it directly.
We show the comparison to ACL in Table 4.
UDS and CUDS outperform ACL in the average task performance while ACL is only proﬁcient on drawer-open and drawer-close, and it cannot solve door-open or door-close.
This indicates that sharing the unlabeled data conservatively across all tasks is important in multi-task ofﬂine RL while pretraining representations on the whole multi-task ofﬂine dataset might have limited beneﬁt.
We note that UDS / CUDS are complementary to ACL and these approaches can be combined together to further improve performance, which we leave as future work.
C ONCLUSION
In the paper, we present two new algorithms, UDS and CUDS, that handle the problem of how to share data across tasks without access to the functional form of the multi-task reward function in the multi-task ofﬂine RL setting.
UDS lifts the strong assumption of having access to the reward of all tasks at each transition in previous works in multi-task data sharing via simply sharing data across all tasks and relabeling the reward of data from other tasks to the minimum reward in the
MDP, which indicates failure of the task.
CUDS further improves over UDS via applying a more sophisticated data sharing scheme (Yu et al., 2021a) that shares data only if the relabeled Q-values improve over the expected Q-values of the original task data.
We justify that UDS obtains Q-values that are lower-bounded by the Q-values learned by data sharing with true reward labels and then discuss that under certain structures of ofﬂine datasets, CUDS can selectively apply conservative policy evaluation on only transitions with high Q-values, resulting in a less conservative algorithm.
Empirically, we show that both CUDS and UDS signiﬁcantly outperform vanilla multi-task ofﬂine RL without data sharing as well as more complex methods that learns the reward function either explicitly or implicitly on a range of robotic manipulation and navigation domains.
CUDS also improves over
UDS on all of the domains.
Furthermore, CUDS and UDS achieve competitive results compared to data sharing methods with access to the oracle rewards.
While our method removes the strong assumption on reward functions in data sharing for multi-task ofﬂine RL and enjoys both theoretical guarantees and good empirical results, it does have a few limitations.
For example, UDS and CUDS are evaluated in MDPs with binary rewards.
Exploring their effects in MDPs with continuous rewards will be an exciting future avenue.
R EPRODUCIBILITY S TATEMENT
We provided the code, data, and instructions needed to reproduce the main experimental results of the experiments with low-dimensional inputs in the supplementary material.
We include all assumptions and derivations of all claims in Section 4.2 and Appendix A. For datasets used in our experiments, we discuss the details of processing the datasets in Appendix B.
A Q-V ALUES L EARNED VIA UDS
Proposition A.1.
For a policy π , let (cid:98) Q π UDS ( · , · , i ) denote the ﬁxed point of Eqn. 6.
Then, (cid:98) Q π UDS ( s , a ) lower-bounds the Q-function (cid:98) Q π Sharing All that would be obtained had we used the true rewards.
Proof.
Note that (cid:98) Q π UDS ( s , a ) at iteration k +1 is deﬁned in Eq.
6, which we restate it for convenience:
[Equation]
whereas (cid:98) Q π Sharing All ( s , a ) at iteration k + 1 is deﬁned as
[Equation]
Assume (cid:98)
[Equation]
, i.e. same Q-value initialization and (cid:98) Q k Sharing All ≥ (cid:98) Q k UDS .
Using such induction hypothesis and the fact that (cid:98) r ( s , a , i ) ≤ r ( s , a , i ) for all s , a , we can conclude that (cid:98) Q π UDS ( s , a ) ≤ (cid:98) Q π Sharing All ( s , a ) .
Therefore, UDS learns the Q-value that lower-bounds the Q-values learned data sharing all tasks with the ground-truth rewards.
B D ETAILS OF UDS AND CUDS
In this section, we include the details of training UDS and CUDS in Appendix B.1 as well as details on the environment and datasets used in our experiments in Appendix B.2.
Finally, we discuss the compute information of UDS and CUDS in Appendix B.3.
For additional details, please see our anonymous website: https://sites.google.com/view/uds-cuds/ .
B. D ETAILS ON THE TRAINING PROCEDURE
Our practical implementation of UDS optimizes the following objectives for the Q-functions and the policy:
[Equation]
and
[Equation]
Similarly, CUDS optimizes the following objectives for training the critic and the policy with a soft weight:
[Equation]
and
[Equation]
←
[Equation]
where β is the coefﬁcient of the CQL penalty on distribution shift, µ is an action sampling distribution that covers the action bound as in CQL.
We follow all the CQL hyperparameters used in Yu et al.
(2021a).
To compute the weight w CUDS ( s , a ; j → i ) :
= σ (cid:16) ∆( s , a ; j → i ) τ (cid:17) , we pick τ , i.e. the temperature term, using the exponential running average of ∆( s , a ; j → i ) with decay 0 .
995 for each task following Yu et al.
(2021a).
Following Yu et al.
(2021a) again, we clip the automatically chosen τ with a minimum and maximum threshold, which we directly use the values from Yu et al.
(2021a).
We use [1 , 50] and [10 , ∞ ] as the minimum and maximum threshold for the multi-task Meta-World and AntMaze domains respectively whereas the vision-based robotic manipulation domain does not require such clipping.
Following the training protocol in Yu et al.
(2021a), for experiments with low-dimensional inputs, we use a stratiﬁed batch with 128 transitions for each task to train the Q-functions and the policy.
We also balance the numbers of transitions sampled from the original task and the number of transitions drawn from other task data.
Speciﬁcally, for each task i , we sample 64 transitions from D i and the remaining 64 transitions from ∪ j (cid:54) =
i D j → i .
In CUDS, for each task i ∈
[ N ] , we only apply w CUDS to data shared from other tasks on multi-task Meta-World environments and multi-task vision-based robotic manipulation tasks while we also apply the relabeling weight to transitions sampled from the original task dataset D i with 50% probability in the multi-task AntMaze domain.
Regarding the choices of the architectures, for state-based domains, we use 3-layer feedforward neural networks with 256 hidden units for both the Q-networks and the policy.
We condition the policy on a one-hot task ID, which is appended to the input state.
In domains with high-dimensional image inputs, we adopt the multi-headed convolutional neural networks used in Kalashnikov et al.
Yu et al.
(2021a).
We use images with dimension 472 × 472 × 3 , extra state features ( g robot status , g height ) and the one-hot task vector as the observations similar Kalashnikov et al.
Yu et al.
(2021a).
Following the set-up in Kalashnikov et al.
(2018; 2021); Yu et al.
(2021a), we use Cartesian space control of the end-effector of the robot in 4D space (3D position and azimuth angle) along with two binary actions to open/close the gripper and terminate the episode respectively to represent the actions.
For more details, see Kalashnikov et al.
B. D ETAILS ON THE ENVIRONMENT AND THE DATASETS
In this subsection, we include the discussion of the details the environment and datasets used for evaluating UDS and CUDS.
Note that all of our environment and ofﬂine datasets are from prior work (Yu et al., 2021a).
We will nonetheless discuss the details to make our work self-contained.
We acknowledge that all datasets with low-dimensional inputs are under the MIT License.
Multi-task Meta-World domains.
We use the door open , door close , drawer open and drawer close environments introduced in Yu et al.
(2021a)
from the public Meta-World (Yu et al., 2020c) repo 1 .
In this multi-task Meta-World environment, a door and a drawer are put on the same scene, which ensures that all four tasks share the same state space.
The environment uses binary rewards for each task, which are adapted from the success condition deﬁned in the Meta-World public repo.
In this case, the robot gets a reward of 1 if it solves the target task and 0 otherwise.
We direct use the ofﬂine datasets constructed in Yu et al.
(2021a), which are generated by training online SAC policies for each task with the dense reward deﬁned in the Meta-World repo forepochs.
The medium-replay datasets use the whole replay buffer of the online SAC agent untilepochs while the expert datasets are collected by the ﬁnal online SAC policy.
Multi-task AntMaze domains.
Following Yu et al.
(2021a), we use the antmaze-medium-play and antmaze-large-play datasets from D4RL (Fu et al., 2020) and partitioning the datasets into multi-task datasets in an undirected way deﬁned in Yu et al.
(2021a).
Speciﬁcally, the dataset is randomly splitted into chunks with equal size, and then each chunk is assigned to a randomly chosen task.
Therefore, under such a task construction scheme, the task data for each task is of low success rate for the particular task it is assigned to and it is imperative for the multi-task ofﬂine RL algorithm to leverage effective data sharing strategy to achieve good performance.
In AntMaze, we also use a binary reward, which provides the agent a reward of +1 when the ant reaches a position within a 0.
radius of the task goal, which is also the reward used default by Fu et al.
The terminal of an episode is set to be true when a reward of +1 is observed.
[Table]
Multi-task image-based robotic picking and placing domains.
Following Kalashnikov et al.
(2021); Yu et al.
(2021a), we use sparse rewards for each task.
That is, reward 1 is assigned to episodes that meet the success conditions and 0 otherwise.
The success conditions are deﬁned in (Kalashnikov et al., 2021).
We directly use the dataset used in Yu et al.
(2021a).
Such a dataset is collected by ﬁrst training a policy for each individual task using QT-Opt (Kalashnikov et al., 2018) until the success rate reaches 40% and 80% for picking tasks and placing tasks respectively and then combine the replay buffers of all tasks as the multi-task ofﬂine dataset.
The dataset consists of a total number of 100K episodes with 25 transitions for each episode.
B. C OMPUTATION C OMPLEXITY
We train UDS and CUDS on a single NVIDIA GeForce RTX 2080 Ti for one day on the state-based domains.
For the vision-based robotic picking and placing experiments, it takes 3 days to train it on 16 TPUs.
C A DDITIONAL DETAILS ON THE QUALITY OF DATA SHARED FROM OTHER TASKS
We present the success rate of the data shared from other tasks to the target task computed by the oracle multi-task reward function in both the multi-task Meta-World and AntMaze domains in Table 5.
Note that the success rate of drawer close and door close are particularly high since for other tasks, the drawer / door is initialized to be closed and therefore the success rate of other task data for these two tasks are almost 100% as deﬁned by the success condition in the public Meta-World repo.
Apart from these two particularly high success rates, the success rates of the shared data are consistently above 0% across all tasks in both domains.
This fact suggests that UDS and CUDS are not relabeling with the ground truth reward where the relabeled data are actually all failures but rather performs the conservative bellman backups on relabeled data that is shown to be effective empirically.
To better understand the performance of UDS under different relabeled data quality, we evaluate the
UDS under different success rates of the data relabeled from other tasks in the multi-task Meta-World domain.
Speciﬁcally, we ﬁlter out data shared from other tasks to ensure that the success rates of the relabeled data are 5%, 50% and 90% respectively.
We compare the results of UDS on such data compositions to the performance of UDS in Table 1 where the success rate of relabeled data is 59.6% as shown in Table 5.
The full results are in Table 6.
UDS on relabeled data with 50% and 90% success rates achieves similar results compared to original UDS whereas UDS on relabel data with 5% success rate is signiﬁcantly worse.
Hence, UDS can obtain good results in settings where the relabeled data is of high quality despite incurring high reward bias, but is not helpful in settings where the shared data is of low quality and does not offer much information about solving the target task.
D E MPIRICAL R ESULTS OF UDS AND CUDS IN MORE GENERAL DENSE REWARD SETTINGS
In this section, we evaluate UDS and CUDS in the dense reward setting in order to test if UDS and
CUDS work in more general reward settings and are not limited to binary rewards.
We pick the multi-task walker environment as used in prior work (Yu et al., 2021a), which consists of three tasks, run forward , run backward and jump .
The reward functions of the three tasks are r ( s, a ) =
[Table] [Table]
[Table] [Table]
[Table] [Table]
∗(cid:107)
[Table]
and r ( s, a ) = −(cid:107)
[Table]
001 ∗(cid:107) a (cid:107)
[Table]
10 ∗ ( z − init z ) respectively where v x denotes the velocity along the x-axis and z denotes the z-position of the halfcheetah and init z denotes the initial z-position.
In UDS and CUDS, we relabel the rewards routed from other tasks with the minimum reward value in the ofﬂine dataset of the target task.
As shown in
Table 7, CUDS and UDS outperform No Sharing by a large margin while also performing comparably to CDS and Sharing All.
Therefore, CUDS and UDS are not limited to settings with binary rewards but are able to be applied to more general cases, in particular, environments with dense rewards.
E C OMPARISONS OF CUDS AND UDS TO M ULTI -T ASK M ODEL -B ASED O FFLINE RL A PPROACHES
In this section, we compare CUDS and UDS to a recent, state-of-the-art model-based ofﬂine RL method COMBO (Yu et al., 2021b) in the Meta-World domain.
We adapt COMBO to the multi-task ofﬂine setting by learning the dynamics model on data of all tasks combined and and performing vanilla multi-task ofﬂine training without data sharing using the model learned with all of the data.
As shown in Table 8, CUDS and UDS indeed outperform COMBO in the average task success rate.
The intuition behind this is that COMBO is unable to learn an accurate dynamics model for tasks with limited data as in our Meta-World setting.
F T HEORETICAL A NALYSIS OF UDS AND CUDS
In this section, we will theoretically analyze UDS and CUDS to understand when these approaches can perform well.
We will ﬁrst discuss our notation, then present our theoretical results, then discuss the intuitive explanations of these results, and ﬁnally, provide proofs of the theoretical results.
F. N OTATION AND A SSUMPTIONS
Let π β ( a | s ) denote the behavior policy for task
i (note that index i was dropped from π β ( a | s ; i ) for brevity).
The dataset, D i is generated from the marginal state-action distribution of π β , i.e.,
D ∼ d π β ( s ) π β ( a | s ) .
We deﬁne d π D as the state marginal distribution introduced by the dataset D under π .
For our analysis, we will abstract ofﬂine RL algorithms into a generic constrained policy optimization problem (Kumar et al., 2020):
[Equation] [Equation]
J D ( π ) denotes the average return of policy π in the empirical MDP induced by the transitions in the dataset, and D ( π, π β ) denotes a divergence measure (e.g., KL-divergence (Jaques et al., 2019;
Wu et al., 2019), MMD distance (Kumar et al., 2019) or D CQL (Kumar et al., 2020)) between the learned policy π and the behavior policy π β .
Let D CQL ( p, q ) denote the following distance between two distributions p ( x ) and q ( x ) with equal support X :
[Equation]
Unless otherwise mentioned, we will drop the subscript “CQL” from D CQL and use D and D CQL interchangeably.
Prior works (Kumar et al., 2020; Yu et al., 2021a) have shown that the optimal policy π ∗ i that optimizes Equation 11 attains a high probability safe-policy improvement guarantee, i.e., J ( π ∗ i ) ≥ J ( π β ) − ζ i , where ζ i is:
[Equation]
The ﬁrst term in Equation 12 corresponds to the decrease in performance due to sampling error and this term is high when the single-task optimal policy π ∗ i visits rarely observed states in the dataset
D i and/or when the divergence from the behavior policy π β is higher under the states visited by the single-task policy s ∼ d π ∗
i D i .
We will show that UDS and CUDS enjoy safe policy improvement.
In our analysis, we assume r ( s , a ) ∈ [0 , 1] .
Finally, as discussed in Section 3, let D eﬀ i denote the relabeled dataset for task i , which includes both D i and the transitions from other tasks relabeled with a 0 reward.
Assumptions.
To prove our theoretical results, following prior work (Kumar et al., 2020; Yu et al., 2021a)
we assume that the empirical rewards and dynamics concentrate towards their mean.
Assumption F.1.
∀ s , a , the following relationships hold with high probability, ≥ 1 − δ
[Equation]
Similar to prior work (Kumar et al., 2020; Yu et al., 2021a), we also make a coverage assumption, i.e., we assume that each state-action pair is observed in the dataset D i , but the rewards and transition dynamics are stochastic, so, the occurrence of each state-action pair does not trivially imply good performance.
To relax this assumption, we can extend our analysis to function approximation (e.g., linear function approximation (Duan et al., 2020)), where such a coverage assumption is only required on all directions of the feature space, and not all state-action pairs.
This would not signiﬁcantly change the analysis, and hence we opt for the simple but illustrative analysis in a tabular setting here.
F. T HEORETICAL R ESULTS
We ﬁrst provide a performance guarantee for UDS which is then used to show that under certain conditions on the sizes of the labeled D i and the effective dataset, D eﬀ i , UDS attains a better policy improvement guarantee than na¨ıve no sharing.
We ﬁrst brieﬂy discuss a novel component of our proof technique, then present the theoretical results, and then interpret it.
F.2. O UR P ROOF T ECHNIQUE
While there are several techniques to provide guarantees for ofﬂine RL algorithms, we will build on the line of safe-policy improvement bounds, previously used in Kumar et al.
Yu et al.
(2021a).
However, na¨ıvely applying these guarantees to our UDS setting will give rise to very weak bounds, since a number of these guarantees utilize a bound on the value difference of the policy in the empirical MDP and the actual MDP (term (i)) as shown below:
[Equation]
Typically, term (i) depends on the sampling error on states that are visited by the learned policy π , and decays to 0 with inﬁnite samples, but UDS can learn quite pessimistic Q-values due to the reward labeling procedure.
However, this may not affect the policy performance since the relative ordering of actions might still be the same.
This is not accounted for in any prior analysis we are aware of.
Therefore, we introduce a novel analysis tool that, rather than decomposing J ( π ) − J ( π β ) na¨ıvely using the return in the empirical MDP, decomposes it using the return of the policy π in the best empirical MDP that still produces the policy π as its optimal policy.
One simple way to obtain this best empirical MDP is via afﬁne transformations on the reward function that preserve optimality.
So, for strengthening our bound, we shall compute the bound similar to the above equation for different afﬁne transformations of the reward and pick the one that gives the tightest bound.
Formally, let g ( · ) be an afﬁne function: g ( x )
= u · x + v for some u > 0 , u ∈ R and v ∈ R .
Then our decomposition looks like:
[Equation]
Then, to obtain a strong lower bound on J ( π ) − J ( π β ) , we can ﬁrst bound each of the terms for a given choice of g = ( u, v ) , and then take the supremum over u and v .
This is reﬂected in the performance guarantee we present next.
F.2. P ERFORMANCE G UARANTEE FOR UDS
Proposition F.1 (Policy improvement guarantee for UDS) .
Let π ∗ UDS denote the policy learned by
UDS for a given task i , and let π eﬀ β ( a | s , i ) denote the behavior policy for the combined dataset for task i ,
D eﬀ i .
Then with high probability ≥ 1 − δ , π ∗ UDS is a ζ -safe policy improvement over π eﬀ β , i.e.,
J ( π ∗ UDS ) ≥ J ( π eﬀ β ) − ζ , where ζ is:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
, where we use the notation f ( s , a ) :
= |D i ( s , a ) | |D eff i ( s , a ) | .
A proof of Proposition F.1 is provided in Appendix F.3.
To intuitively interpret the various terms that appear, we note that term (b) corresponds to the standard policy improvement that arises as a result of using an ofﬂine RL algorithm, term (c) corresponds to sampling error that arises as a result of performing ofﬂine RL on the dynamics induced by a ﬁnite dataset, but note that this term depends on the size of the effective dataset, D eﬀ i and not only the labeled dataset D i for the task.
Term (a) corresponds to the bias incurred as a result of labeling various transitions with a 0 reward in the data, and term (d) corresponds to the sampling error in the reward function, under the assumption of a stochastic reward function.
F.2. H OW DOES UDS COMPARE TO N O S HARING ?
In the setting when no data is shared across tasks, we attain the guarantee shown in Equation 12.
Comparing Proposition F.1 to this guarantee, we note that under some scenarios, UDS yields a tighter bound compared to No Sharing.
Two such scenarios are given by:
Long-horizon tasks: Consider a scenario where tasks have a long horizon H = 11 − γ and |D eﬀ i ( s ) | =
H
|D i ( s ) | .
In this case, dynamics sampling error term (term (c)) consists of one less factor of H when UDS is utilized, compared to when it is not.
Since the dynamics sampling error grows quadratically in the horizon, whereas other terms grow linearly, a reduction in this term by increasing sample size (i.e., denominator) can lead to a stronger guarantee for UDS than No Sharing.
This reasoning does not even consider term (d), which can be trivially upper-bounded by the corresponding term for No Sharing, even though UDS reduces this term as well.
The fraction f ( s , a ) is identical for all state-action pairs in the labeled D i ,
i
e., the unlabeled dataset consists of equal proportions state-action pairs as the labeled dataset.
Consider an extreme case when the unlabeled dataset consists of the trajectories in the labeled dataset
such that f ( s , a ) =
c 0 for all state-action tuples, just not annotated with rewards.
In this case, reward bias takes on a constant value across all the transitions in the dataset, and by virtue of utilizing u and v in our bound in Proposition F.1, we note that the overall effect of this reward bias disappears, since v can compensate for this bias.
F.2. E XTENSION TO CUDS
Finally, we extend Proposition F.1 to a performance guarantee for CUDS by integrating the technique above with the analysis from Yu et al.
(2021a).
To analyze CUDS, we consider the abstract model of the conservative data sharing scheme developed by Yu et al.
(2021a).
This model suggests that CUDS approximates the following optimization in the empirical MDP generated by the relabeled dataset:
[Equation]
Now
[Equation]
utilizing Proposition F.1 and Proposition 5.1 from Yu et al.
(2021a)
[Equation]
we obtain the following guarantee for CUDS:
Corollary F.1.
Let π ∗ CUDS ( a | s , i ) be the optimal policy found by CUDS (Equation 13) and let π ∗ β ( a | s , i ) denote the behavior policy that optimizes Equation 13 for task
i ∈
[ N ] .
Then, with high probability ≥ 1 − δ , π ∗ CUDS is a ζ -safe policy improvement over π ∗ β ,
i
[Equation]
e., J ( π ∗ CUDS ) ≥ J ( π ∗ β ) − ζ CUDS , where ζ CUDS is given by:
[Equation]
[Table]
[Equation]
Proof.
The proof of this proposition follows directly from the proof of Proposition F.1 with the exception that this argument must be applied against the optimized behavior policy π ∗ β .
Comparing the bounds for CUDS and UDS.
We now interpret the bound in Corollary F.1 comparatively against the bound in Proposition F.1.
First note that since the abstract model of CUDS (Equation 13) optimizes the behavior policy π ∗ β , we ﬁrst note from Equation 14 of Yu et al.
(2021a)
that for any other behavior policy π (cid:48) ,
[Equation]
This means that the numerator of the sampling error term (term (c)) is smaller when CUDS is utilized as compared to when UDS is utilized.
In addition, since CUDS relabels unlabeled data from Equation 13, this scheme also increases the dataset size, increasing the denominator of term (c).
On the other hand, note that while UDS increases the denominator |D eﬀ i | , it may also increase the distributional shift D ( π ∗ , π eﬀ β ) appearing in the numerator of the sampling error term.
Our practical version of CUDS (Equation 2), which approximates Equation 13 by relabeling only the top k percentile of the unlabeled data based on the objective in Equation 2, gives us a control over the effective dataset size after relabeling |D
eﬀ i | , while still ensuring a reduced value of D ( π ∗ , π ∗ β ) , and is thus expected to reduce ζ compared to UDS.
Intuitively , note that the bounds in Proposition F.1 and Corollary F.1, guarantee safe-policy improvement over different base policies π eﬀ β vs π ∗ β .
Intuitively, we would expect that J ( π ∗ β ) ≥ J ( π eﬀ β ) in practice, especially for a large α , since CUDS optimizes the behavior policy towards high return, compared to simply relabeling all unlabeled transitions.
Therefore, CUDS not only reduces ζ compared to UDS, but also, in practice, is expected to improve over π ∗ β , which performs better than π eﬀ β .
Thus, we would expect CUDS to be better in practice compared to UDS.
F. P ROOF OF P ROPOSITION F.
As mentioned in the beginning of Section F.2.1, to strengthen the conventional safe policy improvement bound, we instead utilize a different form of loss decomposition of the improvement of the learned policy relative to the behavior policy with the afﬁne transformation g :
[Equation]
Now we will discuss how to bound each of the terms: terms (i) and (ii) correspond to the divergence between a transformed empirical policy return and the actual return.
While usually, this difference depends on the sampling error and distributional shift, in our case, it additionally depends on the reward bias induced on the unlabeled data and the transformation g .
We ﬁrst discuss the terms that contribute to this reward bias.
Bounding the reward bias.
Denote the effective reward of a particular transition ( s , a , r, s (cid:48) )
∈ D eﬀ i , as (cid:98) r eﬀ i , which considers contributions from both the reward (cid:98) r
( s , a ) observed in dataset D i , and the contribution of 0 reward from the relabeled dataset:
[Equation] [Equation]
Deﬁne f
( s , a ) :
= |D i ( s , a ) | |D eff i ( s , a ) | for notation compactness.
Equation 15 and the form of the reward transformation g ( x ) = u · x + v can then be used to derive the following difference against the true rewards:
[Equation]
where the last step follows from the fact that the ground-truth reward r ( s , a ) ∈ [0 , 1] and the fact that v will be chosen to minimize this upper bound.
Now, we lower bound the reward bias as follows:
[Equation]
where the last step follows from the fact that r ( s , a ) ≤ 1 .
To highlight the signiﬁcance of this reward transformation, note that in the last step, if ∀ s , a , f ( s , a ) =
c 0 , then the best reward transformation would choose v = 1 − c 0 , and that completely eliminates the excess bias induced in the bound.
Upper bounding g (cid:16) (cid:98) J i ( π ) (cid:17) − J i ( π ) .
Next, using the upper and lower bounds on the reward bias, we now derive an upper bound on the difference between the value of a policy computed under the empirical MDP and the actual MDP.
To compute this difference, we follow the following steps
[Equation]
Following Kumar et al.
(Theorem 3.6)
[Equation]
we can bound the second term ∆ 2 using:
[Equation]
To upper bound ∆
[Equation]
we utilize the reward upper bound from Equation 16:
[Equation]
Combining the results so far
[Equation]
we obtain
[Equation]
for any policy π :
[Equation]
Lower bounding g (cid:16) (cid:98) J i ( π ) (cid:17) − J i ( π ) .
To lower bound this quantity, we follow the step shown in
Equation
[Equation]
and lower bound the term ∆ 2 by using the negative of the RHS of Equation 20, and lower bound ∆ 1 by upper bounding its absolute value as shown below:
[Equation]
This gives rise to the complete lower bound:
[Equation]
Policy improvement term (ii).
Finally
[Equation]
the missing piece that needs to be bounded is the policy improvement term (ii) in the decomposition of g ( J ( π ))
− g ( J ( π β )) .
Utilizing the abstract form of ofﬂine RL (Equation 11, we note that term (ii) is lower bounded as:
[Equation]
Putting it all together.
To obtain the ﬁnal expression of Proposition F.1, we put all the parts together, and include some simpliﬁcations to obtain the ﬁnal expression.
The bound we show is relative to the effective behavior policy π eﬀ β .
Applying Equation 26 for term (i) on policy π , Equation 27 for term (ii), and Equation 23 for the behavior policy π eﬀ β , we obtain the following:
[Equation]
Note that in the second step above, we upper bound the quantities ∆ (cid:48) 1 and ∆ 2 corresponding to π eﬀ β with twice the expression for policy π .
This is because the effective behavior policy π eﬀ β consists of a mixture of the original behavior policy ˆ π β with the additional data, and thus the new effective dataset consists of the original dataset D i as its part.
Upper bounding it with twice the corresponding term for π is a valid bound, though a bit looser, but this bound sufﬁces for our interpretations.
Finally to ﬁnish the proof, we can take the supremum over the best choice of ( u, v ) .
Thus, we obtain the desired bound in Proposition F.1.
G E MPIRICAL ANALYSIS OF THE REASON THAT CUDS AND UDS WORK
In this section, we perform an empirical study on the Meta-World domain to better understand the reason that UDS and CUDS work well.
Our theoretical analysis suggests that UDS will help the most on domains with limited data or narrow coverage or low data quality.
To test these conditions in practice, we perform empirical analysis on two domains as follows.
We ﬁrst choose the door open task with three different combinations of dataset size and data quality of the task-speciﬁc data with reward labels: • 2k transitions with the expert-level performance (i.e. high-quality data with limited sample size and narrow coverage ) • 2k transitions with medium-level performance (i e. medium-quality data with limited sample size and narrow coverage ) • a medium-replay dataset with 152k transitions (i.e. medium-quality data with sufﬁcient sample size and broad coverage ).
We share the same data from the other three tasks, door close , drawer open and drawer close as in Table 1, which are .
As shown in Table 9, both UDS and CUDS are able to outperform No Sharing in the three settings, suggesting that increasing the coverage of the ofﬂine data as suggested by our theory does lead to performance boost in wherever we have limited good-quality data (expert), limited medium-quality data (medium) and abundant medium-quality data (mediumreplay).
It’s worth noting that UDS and CUDS signiﬁcantly outperform No Sharing in the limited expert and medium data setting whereas in the medium-replay setting with broader coverage, CUDS outperforms No sharing but UDS fails to achieve non-zero success rate.
Such results suggest that
UDS and CUDS can yield greater beneﬁt when the target task doesn’t have sufﬁcient data and the number of relabeled data is large. The fact that UDS is unable to learn on medium-replay datasets also suggests that data sharing without rewards is less useful in settings where the coverage of the labeled ofﬂine data is already quite broad.
To further understand the sensitivity of UDS to the data coverage and the data quality of both target task data (i.e. with reward labels) and relabeled data (i.e. without reward labels), we perform another empirical study using the hopper environment from the D4RL (Fu et al., 2020) benchmark. We consider the following 6 different combinations varying the quality and amount of the labeled and unlabeled datasets: [Table]
10k labeled data from hopper-expert + unlabeled 1M data hopper-random (i [Table] e., high-quality + narrow labeled data, low-quality + broad unlabeled data) 2.
10k labeled data from hopper-expert + unlabeled 1M data from hopper-medium (i
[Table]
e., high-quality + narrow labeled data, medium-quality + narrow unlabeled data) 3.
10k labeled data from hopper-medium + unlabeled 1M data from hopper-random
(i
[Table]
e., medium-quality + narrow labeled data, low-quality + broad unlabeled data) 4.
10k labeled data from hopper-medium + unlabeled 1M data from hopper-expert (i
[Table]
e., medium-quality + narrow labeled data, high-quality + narrow unlabeled data) 5.
10k labeled data from hopper-random + unlabeled 1M data from hopper-medium (i.e., low-quality + broad labeled data, medium-quality + narrow unlabeled data) 6.
10k labeled
data from hopper-random + unlabeled 1M
data
from hopper-expert (i.e., low-quality + broad labeled data, high-quality + narrow unlabeled data)
Results.
In cases (1) and (2), adding the unlabeled random or medium data, should increase coverage, since the labeled data only consists of expert transitions.
Moreover, the induced reward bias due to incorrect labeling of rewards on the medium unlabeled data should not hurt, since the 10k expert transitions retain their correct labels, and the medium/random data should only serve as negatives.
Therefore, we expect the beneﬁts of coverage to outweigh any reward bias, and as shown in Table 10, we ﬁnd that UDS does help.
In cases (4), (5) and (6), when the relabeled data is better compared to the labeled data (i.e., expert or medium), we ﬁnd that even if the rewards on these transitions are incorrect, behavior regularization properties induced by ofﬂine RL algorithms allow UDS to attain better performance than no sharing by utilizing the unlabeled data.
In case (3), we ﬁnd that UDS hurts compared to No Sharing.
This is because the target task data as well as unlabeled data are both low-medium quality and medium data already provides decent coverage (not as high as random data, but not as low as expert data).
Therefore, in this case, we believe that the addition of unlabeled data neither provides trajectories of good quality that can help improve performance, nor does it signiﬁcantly improve coverage, and only hurts by incurring reward bias.
We therefore believe that UDS may not help in such cases where the coverage does not improve, and added data is not so high quality.
G. S UMMARY OF EMPIRICAL ANALYSIS
Given our results in Table 9 and Table 10, we summarize the applicability of UDS under different scenarios in Table 11 below.
[Table]
[Table]