Data Sharing without Rewards in Multi-Task Ofﬂine Reinforcement Learning
Abstract
Ofﬂine reinforcement learning (RL) bears the promise to learn effective control policies from static datasets but is thus far unable to learn from large databases of heterogeneous experience.
The multi-task version of ofﬂine RL enables the possibility of learning a single policy that can tackle multiple tasks and allows the algorithm to share ofﬂine data across tasks.
Recent works indicate that sharing data between tasks can be highly beneﬁcial in multi-task learning.
However, these beneﬁts come at a cost – for data to be shared between tasks, each transition must be annotated with reward labels corresponding to other tasks.
This is particularly expensive and unscalable, since the manual effort in annotating reward grows quadratically with the number of tasks.
Can we retain the beneﬁts of data sharing without requiring reward relabeling for every task pair?
In this paper, we show that, perhaps surprisingly, under a binary-reward assumption, simply utilizing data from other tasks with constant reward labels can not only provide substantial improvement over only using the single-task data and previously proposed success classiﬁers, but it can also reach comparable performance to baselines that take advantage of the oracle multi-task reward information.
We also show that this performance can be further improved by selectively deciding which transitions to share, again without introducing any additional models or classiﬁers.
We discuss how these approaches relate to each other and baseline strategies under various assumptions on the dataset.
Our empirical results show that it leads to improved performance across a range of different multi-task ofﬂine RL scenarios, including robotic manipulation from visual inputs and ant-maze navigation.
Introduction
Ofﬂine reinforcement learning (RL) provides the promise of a fully data-driven framework for learning performant policies.
To avoid costly active data collection and exploration, ofﬂine RL methods utilize a previously collected dataset to extract the best possible behavior, making it feasible to use RL to solve real-world problems where active exploration is expensive, dangerous, or otherwise infeasible [82, 8, 68, 29].
However, this concept is only viable when a signiﬁcant amount of data for the target task is available in advance.
A more realistic scenario might allow for a much smaller amount of task-speciﬁc data, combined with a large amount of task-agnostic data, that is not labeled with task rewards and some of which may not be relevant.
For example, if our goal is to train a robot to perform a new manipulation task (e.g., cutting an onion), we might have some data of the robot (suboptimally) attempting that task, perhaps collected under human teleoperation and manually labeled with rewards, combined with plentiful data of other tasks, some of which might be structurally related (e.g., picking up an onion, or cutting a carrot).
This scenario presents several questions: How Deep Reinforcement Learning Workshop at 35th Conference on Neural Information Processing Systems (NeurIPS 2021). do we decide which prior data should be included when learning the new task?
And how do we determine which reward labels to use for this prior data?
Prior methods have offered several potential answers to these two questions, typically in isolation.
For the ﬁrst question, it has been recently observed that a naïve sharing strategy of sharing data from all tasks can be highly suboptimal [30], and some works have proposed both manual [30] and automated [80, 13] data-sharing strategies that prioritize the most structurally similar prior data.
Most such methods assume that this shared data can be automatically relabeled with the reward function for the new task [30, 80, 13], but the assumption that we have access to the functional form of this reward is a strong one: for example, in many real-world settings, the reward might require human labeling or human-provided examples [5, 16].
To this end, some prior works have proposed learning classiﬁers for reward labeling [18, 71, 60], or other automated mechanisms [32].
But these mechanisms themselves add complexity and potential brittleness to the pipeline.
Thus, we aim to devise a simple uniﬁed method that determines which data to share and which rewards to use, with minimal supervision and no additional modeling and learning.
In this paper, we make the potentially surprising observation that data from other tasks can be utilized with naïve constant reward labels, when the MDP consists of binary rewards.
We show that this simple method, which does not involve learning any additional models or classiﬁers, can outperform more sophisticated techniques in practice.
Our approach simply utilizes data from other tasks with a constant reward label (e.g., r = 0 ), and uses a value-aware strategy to decide which prior transitions to include for the new task.
This strategy, based on the conservative data sharing (CDS) technique proposed in prior work (which assumes oracle reward access) [80], also does not require learning any additional model and simply uses the Q-function that is already learned as part of the RL process.
Our main contribution, which we call conservative unsupervised data sharing (CUDS), is a technique for sharing data in multi-task ofﬂine RL that does not require any reward labels or reward function access for the task-agnostic data, and requires no additional model or classiﬁer.
To achieve that, our method assumes a particular form of the MDP that consists of binary rewards.
We discuss the behaviors of our methods, showing that, even without ground truth reward labels, our simple data sharing scheme achieves Q-values that are lower-bounded by the Q-values obtained with sharing all data with the ground-truth rewards and can be combined CDS to selectively ﬁlter out potentially irrelevant data under different assumptions on the structure of the dataset.
Our empirical evaluation conducted over various multi-task ofﬂine RL scenarios such as robotic manipulation from visual inputs and ant-maze navigation shows that this approach improves over the performance of more sophisticated techniques that either learn the reward function explicitly, or utilize other methods to propagate reward labels.
In addition, we show that the proposed approach is comparable to an oracle baseline that has access to true multi-task rewards.  Related Work
Ofﬂine RL.
Ofﬂine RL [11, 55, 36, 38] considers the problem of learning a policy from a static dataset without interacting with the environment, which has shown promises in many practical applications such as robotic control [29, 45, 53], NLP [27], healthcare [58, 68], education [8], electricity supply [82] and UI design [3].
The main challenge of ofﬂine RL is the distributional shift between the learned policy and the behavior policy [22, 34], which can cause erroneous value backups due to out-of-distribution actions generated by the learned policy.
To address this issue, prior methods have constrained the learned policy to not deviate much from the behavior policy via policy regularization [43, 27, 70, 83, 34, 59, 50, 83, 33, 23], conservative value functions [35, 62], an auxiliary behavioral cloning loss [21] and model-based training with conservative penalties [79, 31, 4, 66, 46, 37, 81].
Multi-Task RL and data sharing.
Multi-task RL [69, 49, 67, 12, 24, 76, 72, 75, 30, 63, 64] enables the goal of learning a single policy that solves multiple skills efﬁciently.
Despite the promising results, multi-task RL suffers from three main challenges, optimization difﬁculties [57, 24, 76], effective weight sharing for learning shared representations [49, 67, 12, 72, 9, 63, 64], and sharing data across different tasks [13, 30, 80].
We consider the multi-task ofﬂine RL setting and focus on the challenge of sharing data across different tasks.
Prior works share data across tasks based on metrics such as learned Q-values [13, 39, 80], human domain knowledge [30], the distance to the target goals in goal-conditioned settings [2, 52, 47, 42, 65, 41, 26, 44, 74, 6], and the learned distance with robust inference in the ofﬂine meta-RL setting [40].
However, all of these either require access to the functional form of the reward functions of each task in order to relabel the rewards or are limited to goal-conditioned settings.
Therefore, they are not applicable to the multi-task ofﬂine RL problem that we consider where only the reward label of the originally-executed task is provided.
Our work addresses this issue via simply relabeling the data shared from other tasks with a constant value and uses the conservative data sharing strategy [80] to further improve the performance.
RL with unlabeled data.
Prior works tackle the problem of learning from data without reward labels via either directly imitating expert trajectories [51, 56, 25], learning reward functions from expert data using inverse RL [1, 48, 84, 15, 17, 18, 32], or learning a reward / value classiﬁer that discriminates successes and failures [71, 60, 14].
These algorithms require online data collection and do not consider the ofﬂine RL setting.
[61] considers the single-task ofﬂine setting with both task-speciﬁc datasets and task-agnostic prior datasets and relabel the unlabeled prior data as failures since these prior transitions cannot solve the task.
Our method is not limited to such single-task settings and instead considers the more general multi-task ofﬂine RL with data-sharing problem.
Preliminaries
Multi-task RL.
Standard multi-task RL considers a multi-task Markov decision process (MDP),
M = ( S , A , P, γ, { R i , i } N i =1 ) , where S and A denote the state and action spaces respectively,
P ( s (cid:48) | s , a ) denotes the dynamics, γ ∈ [0 , 1) is the discount factor, and R 1 , · · · , R N correspond to reward functions of different tasks i ∈
[ N ] for total number of N tasks where [ N ] is the shorthand for { 1 , 2 , . . . , N } .
In our setting, we assume a binary per-task R i ∈ { 0 , 1 } , where 1 denotes success of the task and 0 otherwise.
Note that the dynamics are assumed to be the same across all tasks, which is not entirely general but is indeed practical in many problem settings as noted in [80] and stands as a common assumption in prior data sharing works [80, 30, 13].
Regardless, there are many practical scenarios with changing rewards and invariant dynamics such as various object manipulation objectives [71], different goal navigation tasks [20], and distinct user preferences [7].
The goal of multi-task RL is to ﬁnd a task-conditioned policy π ( a | s , i ) that expected return in a multi-task
MDP: π ∗ ( a | s , · ) :=
arg max π E i ∼ [ N ] E π ( ·|· ,i ) [ (cid:80) t γ t R i ( s t , a t )] .
Note that it is possible to model the policies for each task independently as { π 1 ( a | s ) , · · · , π N ( a | s ) } without any weight sharing.
In our work, we use the single task-conditioned policy to study data sharing and do not consider the weight sharing aspect, which is orthogonal to the focus of the paper, which is also noted in [80].
Multi-task ofﬂine RL and data sharing.
Multi-task ofﬂine RL considers the problem of learning the multi-task policy π ( a | s , i ) from a static multi-task dataset with D = ∪ Ni =1 D i where D (cid:105) =
{ ( s j , a j , s (cid:48) j , r j ) } Mj =1 is the per-task dataset.
D i is generated by a behavior policy π β ( a | s ) , without any interaction with the environment.
The most straightforward approach to learn π ( a | s , i ) would be train it for task i only using D i .
However, sharing data from different tasks to task i has been shown to be conducive in the multi-task ofﬂine RL setting [30, 80].
To do so, prior works [14, 30, 80] assume access to the functional form of the reward r i , which is a rather strong assumption that is usually impractical to specify in practical applications due to the challenge of reward speciﬁcation.
The next straightforward approach is to naïvely sharing data across all tasks, denoted as Sharing
All.
Formally, Sharing All deﬁnes the dataset of transitions relabeled from task j to task i as D j → i and the method can be then deﬁned as D eﬀ i := D i ∪ ( ∪ j (cid:54) =
i D j
→ i ) , where D eﬀ i denotes the effective dataset for task i .
While Sharing All improves over not sharing data, as shown in [80], Sharing All leads to distributional shift that could degrade performance in certain situations [80].
In our work, we focus on the CDS [80], which relabels data that aims to mitigate the distributional shift introduced by sharing other task data.
CDS addresses such an issue by proposing a conservative data sharing strategy as follows:
[Equation]
where s j , a j , s (cid:48)
j denote the transition from D j , r i denotes the reward of s j , a j , s (cid:48)
j relabeled for task i , π denotes the task-conditioned policy π ( ·|· , i ) , ∆ π ( s j , a j ) is the condition that shares data only if the expected Q-value of the relabeled transition exceeds the top k -percentile of the Q-values of the original task data, i.e.
[Equation]
Beyond controlling the distributional shift introduced in data sharing, multi-task ofﬂine RL also needs to address the main challenge in standard ofﬂine RL, which is the distributional shift between the learned policy π and the behavior policy π β .
To handle both types of distributional shifts, CDS [80] combines the conservative data sharing and the constrained policy optimization problem and arrives at the following objective:
[Equation]
where π eﬀ β ( a | s , i ) is the effective behavior policy for task i denoted as π eﬀ β ( a | s , i ) :
=
|D
eﬀ i ( s , a ) | / |D
eﬀ i ( s ) | , J D eff i ( π ) denotes the average return of policy π in the empirical MDP induced by the effective dataset, and D ( π, π eﬀ β ) denotes a divergence measure (e.g., KL-divergence [27, 70], ﬁsher divergence [33], MMD distance [34] or D CQL from conservative Q-learning [35]) between the learned policy π and the effective behavior policy π eﬀ β .
While optimizing Eq.
3 with Eq.
1 as the data sharing scheme is able to mitigate distributional shift and improve over multi-task ofﬂine RL without sharing data and naïvely sharing data across all tasks as shown in [80], it requires the assumption of the access to the functional form of the reward functions, which is rather strong and make application of data sharing to real-world applications impractical.
We will instead present a simple yet effective data sharing and relabeling scheme in the setting where we do not make such an assumption and instead, only have the reward labels for originally commanded task in the following section.
Data Sharing without Rewards in Multi-Task Ofﬂine RL
The goal of our method is to enable effective data sharing across different tasks without access to the functional form of the reward functions for each task.
Data from each task is only labeled for that particular task, and we do not know a priori which data is relevant to each task.
Effective data sharing therefore requires resolving two questions: (i) which data from other tasks should we use for a given task?
and (ii) how do we label this data with rewards?
One simple approach is to annotate all available data from other tasks with some “proxy” reward signal, and treat it no differently from data that is already labeled.
That is, after relabeling with the proxy reward, we can simply put these transitions into the replay buffer of a value-based ofﬂine RL method.
But how can we obtain a reliable proxy reward signal?
Next, we will discuss two variants of our method in Section 4.1, understanding of both variants in Section 4.2, and practical implementations in Section 4.3.
4.1 Conservative Unsupervised Data Sharing
Prior work assumes that it is necessary to relabel prior data with some estimate of the true reward function so that the proxy reward closely reﬂects the true reward.
We take a different approach, and instead argue that, under some assumptions, we can obtain many of the beneﬁts of data sharing simply by labeling the multi-task data with the lowest possible reward, which we assume to be 0 without loss of generality in the binary-reward setting.
We refer to this simple strategy as unsupervised data sharing (UDS).
Naïve UDS prescribes sharing data from every task to every other task, and labels the shared data with a reward value of 0.
Formally, for each task i ∈ [ N ] , we deﬁne the UDS procedure as follows:
[Equation]
Intuitively, UDS relabels data shared from other tasks with the lowest possible reward, hence making the learned Q-functions more conservative that the data sharing scheme with the oracle rewards.
We will show that UDS learns Q-values that are lower-bounded by the Q-values learned by the naïve
Sharing All scheme with true reward relabeling, and can be information-theoretically optimal in ofﬂine RL thanks to such conservatism.
Our empirical results in Section 5 also suggests that the beneﬁts from data sharing outweigh the downsides of reward bias in practice.
Next, we move on from the choice of proxy reward, to the choice of which data should be shared.
While UDS is simple yet effective in multi-task data sharing without reward relabing, naïvely sharing data from all other tasks with zero rewards in the ofﬂine RL setting can result in overly conservative
Q-functions and policies.
To further reﬁne this strategy, we can adapt the CDS algorithm [80] detailed in Section 3 to ﬁlter out irrelevant transitions from other tasks, and only share those transitions that are likely to be informative.
We call this strategy conservative unsupervised data sharing (CUDS).
We deﬁne the CUDS strategy as follows:
[Equation]
[Equation] As we will discuss in the next subsection, CUDS is able to select potentially useful transitions under certain structural assumptions on the multi-task ofﬂine dataset, and therefore produce Q-values that are not as excessively conservative as those produced by UDS.
As shown in Section 5, our empirical evaluation further validates that CUDS improves over UDS and prior approaches.
Algorithm 1 (Conservative)
Unsupervised Data Sharing
[Table]
[Table]
[Table]
[Table]
4.2 Understanding the Behavior of UDS and CUDS
In this section, we aim to understand the behavior of the UDS and CUDS.
We ﬁrst consider the
UDS scheme, which simply shares all available data from other tasks, and labels the reward for each transition from other tasks as 0 .
When instantiated with CQL as the ofﬂine RL method, the Q-values of a given policy learned by UDS for each task i are the ﬁxed point of the recursion:
[Equation]
where (cid:98) r ( s , a , i ) =
for all ( s , a ) ∈ D
j → i .j
(cid:54)
=
i , and (cid:98) r ( s , a , i ) is equivalent to the empirical reward observed otherwise [35].
We will now try to understand how UDS compares to the No Sharing strategy, which only uses the labeled data for training.
Note that this comparison is non-trivial since, while UDS utilizes a larger dataset, it can induce signiﬁcant reward bias during training.
However, by assumption, 0 is the lowest possible reward, we would intuitively expect that UDS should be more conservative, compared to Sharing All that relabels with the true reward.
While we may surmise that being too conservative on unlabeled data may be suboptimal, conservatism has been shown to be information-theoretically optimal [28, 54] in ofﬂine RL and bandit problems.
Even though an unlabeled dataset provides us with information about environment dynamics, it does not provide information about rewards, and any optimistic estimate of reward on this data may lead to poor performance in the worst case.
We formally derive the performance guarantee for UDS in Proposition F.1 using the framework of safe-policy improvement and discuss cases where it can perform better than No Sharing. We discuss in Appendix F.2.3 that UDS can perform better than No Sharing in long horizon tasks as well as in cases where the unlabeled dataset consists of similar proportions of various state-action pairs as the labeled dataset. Please refer to this section for the theoretical results. Our bounds utilize a new technique that allows us to prove tighther, non-trivial bounds for UDS.
despite the pessimism which is discussed in Appendix F.2.1.
To understand the behavior of CUDS, we will consider a simple abstract model of CUDS-style relabeling.
In the tabular setting, this model updates the Q-function to match the target (conservative)
Q-value if the transition is selected for the update, and retains the old table entry otherwise.
Formally, consider a binary vector w ∈ R |S|×|A| that indicates whether a corresponding state-action pair ( s , a ) is utilized for the backup or not.
Then, our weighted scheme performs the following backups:
[Equation]
Equation 7 can be intuitively understood as performing a conservative backup from the actual transition observed in the dataset when the binary weight w ( s , a ) = 1 , and simply truncating the Bellman backup and retaining the previous Q-values (cid:98) Q k ( s , a , i ) , otherwise.
For example,
CUDS performs a conservative backup with (cid:98) r ( s , a , i ) ≤ (cid:98)
r Sharing All ( s , a , i ) only on transitions where w ( s , a ) =
I [∆ π ( s , a ) ≥ 0] .
To understand how this affects the resulting Q-function, we consider two structural conditions on the ofﬂine dataset: (1) a scenario where no trajectory in the relabeled dataset for a given target task
D j → i actually visits state-action tuples that were observed in D i , and (2) when trajectories in D j → i overlap with at least a fraction of state-action tuples in the original labeled data for this task D i .
We will abstract CUDS as utilizing w k ( s , a ) =
I [ (cid:98) Q k ( s , a , i ) ≥ ι ] for some threshold ι (see Eqn. 2).
Remark 4.1 (CUDS reduces to no sharing under condition (1) ) .
When the trajectories in the unlabeled, relabeled dataset do not overlap with any trajectory in the labeled dataset for a given task, any backup performed by CUDS on an unlabeled transition will eventually drive its Q-value to 0 as k → ∞ .
Thus, CUDS weights w k ( s , a ) will eventually take on 0 values for such transitions, and will not be selected by the future weights, i.e., w j ( s , a ) =
∀ j ≥ k + 1 .
Perhaps unsurprisingly, when the unlabeled data has no overlap with the labeled data, CUDS reduces to no sharing.
However, the more practically relevant case is when the unlabeled data overlaps with the labeled data.
We consider the scenario when UDS has been run initially to obtain a starting set of
Q-values, (cid:98) Q 0 ( s , a , i ) , which deﬁnes the initial weight vector.
Remark 4.2 (CUDS selects more useful unlabeled transitions) .
Imagine a transitions ( s , a , s (cid:48) , 0) ∈
D j → i for which the next state (and the next policy action) ( s (cid:48) , a (cid:48) ) are observed in the labeled dataset (denoted D i ). This transition will will attain large initial Q-values (cid:98) Q 0 ( s , a , i ) if executing the policy after ( s (cid:48) , a (cid:48) ) eventually reaches the state that corresponds to a high reward of 1.0, due to the Bellman backup component of CUDS.
However, on the ﬂip side, these backups performed by CUDS are conservative, and performing more backups can reduce the Q-value.
Two scenarios might then arise: (i) the Q-values eventually decrease and CUDS is deactivated, i.e., ∃ k, w k ( s , a ) = 0 , in which case this transition is discarded and not used for learning anymore as the backup in Equation 7 preserves the Q-value (the second term) when w k ( s , a ) = 0 , or (ii) the learning process reaches an equilibrium where w k ( s , a ) = 1 ∀ k , meaning that this relabeled transition is used for learning.
We have now provided a theoretical analysis of CUDS and a comparison between CUDS and UDS in Appendix F.2. Additionally, we provide several new experiments to build insight into why UDS and CUDS work in Appendix G.
4.3 Practical Implementations
We present pseudocode for UDS and CUDS in Algorithm 1.
We train the Q-values with CQL to obtain conservative Q-values, and use the conservative Q-values to compute ∆ π ( s , a ) deﬁned in Eq. 2.
For CUDS, in practice, instead of computing the hard threshold of ∆ π ( s , a ) ≥ 0 to determine data sharing, we follow Yu et al.
[80] and transform the condition ∆ π ( s , a ) ≥ 0 into a soft weighting scheme, with weights given by w CUDS ( s , a ; j → i ) :
= σ (cid:16) ∆( s , a ; j → i ) τ (cid:17) , where τ is a hyperparameter for the temperature of the sigmoid term in w CUDS that is automatically selected by the running average of ∆( s , a ; j → i ) .
These weights are applied to both critic and actor training.
For both UDS and CUDS, we train a policy π ( a | s , i ) where π ( a | s , i ) could either be a single task-conditioned task with weight sharing or separate policies for each task without weight sharing.
For more details of the practice implementations, see Appendix B.
Experiments
In this section, we present our empirical evaluation, which aims to answer the following questions: (1)
Can our simple approach outperform prior methods for utilizing unlabeled ofﬂine data on multi-task ofﬂine datasets?
(2) Can the conservative data sharing strategy further improve the results achieved by our method?
(3) Is our approach able to attain competitive result compared to the prior multi-task ofﬂine RL algorithm that have access to the true rewards?
(4) How does CUDS compare to prior ofﬂine RL methods that directly learn representations from the multi-task ofﬂine dataset and run ofﬂine training on top of the representation?
Comparisons.
We compare UDS and CUDS to a number of prior methods.
We ﬁrst evaluate: No
Sharing , which performs applies standard ofﬂine RL algorithm to the multi-task setting without sharing data across tasks, Reward Predictor , which learns a classiﬁer that directly predicts the reward using supervised learning, VICE [19], an inverse RL method that learns a reward classiﬁer from the labeled data and then annotates the unlabeled data with the learned classiﬁer, and RCE [14], a method similar to VICE except that RCE represents the Q-function as a classiﬁer and learns the reward for unlabeled data implicitly.
We adapt VICE and RCE to the multi-task ofﬂine RL setting by extracting transitions with reward labels equal to 1 and treating these datapoints as positives to learn the classiﬁer for each task.
We also train VICE and RCE , but adapt them to the ofﬂine setting using CQL, i.e. the same base ofﬂine RL method as in UDS and CUDS .
Finally, to answer question (4), we conduct empirical evaluations on ACL [73], which is a recent ofﬂine RL algorithm that performs representation learning on the ofﬂine dataset and trains the policy on top of the representation.
For more details for experimental set-up and hyperparameter settings, please see Appendix B. We also include evaluations of our methods under different quality of the relabeled data in Appendix C, results of UDS and CUDS in dense-reward settings in Appendix D, comparisons to model-based ofﬂine RL approaches in Appendix E, and empirical analysis of the reasons that UDS and CUDS work in Appendix G.
5.1 Main Evaluation
To answer questions (1), (2), and (3), we perform empirical evaluations on two state-based multi-task robotic manipulation and navigation datasets and one image-based multi-task manipulation dataset introduced in prior work [80], which we will discuss below.
Tasks and Datasets.
Following the experimental setup in prior work [80], we consider three domains shown in Fig.
1: (i) the Meta-World [77] domain, which consists of four tasks of opening and closing doors and drawers; (ii) the Antmaze [20] domain, which consists of two sizes of mazes (medium and large) with 3 and 7 tasks respectively; and (iii) the multi-task visual manipulation domain, which consists of 10 tasks with different combinations of object-oriented grasping, with 7 objects (banana, bottle, sausage, milk box, food box, can and carrot), and placing the picked objects onto one of three ﬁxtures (bowl, plate and divider plate).
For all domains, we use binary rewards, where 1 denotes the successful completion of the task and 0 corresponds to failure.
Note that for Meta-World, we use a ﬁxed 200 timesteps for each episode and do not terminate the episode when receiving a reward ofat an intermediate timestep.
In Antmaze, we terminate the episode upon seeing a reward of 1 with the maximum possible 1000 transitions per episode.
We use the same datasets as prior work [80].
For Meta-World, we use large datasets with wide coverage of the state space and 152K transitions for the door open and drawer close tasks and datasets with limited (2K transitions), but optimal demonstrations for the door close and drawer open tasks.
For AntMaze, following [80], we modify the datasets introduced by Fu et al.
[20] by equally dividing the large dataset into different parts for different tasks, where each task corresponds to a different goal position.
For image-based manipulation, we directly use the dataset collected by Yu et al.
[80], which contains a total of 100K RL episodes with 25 transitions for each episode, where the success rate is 40% and 80% for the picking and placing tasks, respectively.
Note that the success rate of placing is higher because the robot is already holding the object at the start of the placing tasks, making the placing easier to solve.
Results of Question (1).
The main results are in Table 1.
UDS achieves better performance than vanilla multi-task ofﬂine RL without data sharing and compared to reward learning methods, suggesting that our simple relabeling method is effective in both multi-task manipulation and navigation domains.
Since the reward learning approaches obtain similar or worse results compared to no sharing, we only compare our methods to No Sharing and the oracle methods in the image-based experiments.
As shown in Table 2.
UDS outperforms No Sharing in 7 out of 10 tasks as well as the average task performance by a signiﬁcant margin.
Therefore, UDS is able to effectively leverage unlabeled data shared from other tasks and achieves potentially surprisingly strong results compared to more sophisticated methods that handle unlabeled ofﬂine data, answering question (1). [Table] [Table] [Table]
Results of Question (2).
In both the state-based and vision-based experiments shown in Tableand Table 2, we ﬁnd that CUDS further improves upon the performance of UDS , which empirically indicates that the less conservative policy learned from CUDS’s selective ﬁltering scheme is more performant in practice.
Additionally, we measure the success rates of the relabeled data in Table
Appendix C, measured by the oracle multi-task reward function on the Meta-World and AntMaze domain.
We see that the success rates of the relabeled data are above 0% by a signiﬁcant margin in most of the tasks.
This suggests that UDS and CUDS are not simply relabeling with the true reward, since the relabeled data does not entirely consist of failures but rather has a signiﬁcant number of successful transitions.
Results of Question (3).
We also show results for oracle methods that receive true reward labels:
Sharing All , which shares all data with ground truth rewards, and CDS , which uses the CDS strategy [80] with ground truth reward relabeling.
We present the results in Table 3 for state-based experiments and the last two columns on the right in Table 2 for the vision-based multi-task robotic manipulation problem.
Both CUDS and UDS achieves competitive results compared to CDS and
Sharing All , indicating that our simple relabeling scheme is able to remove the dependence of functional form of reward functions without much loss of performance due to lacking ground-truth reward access.
This addresses question (3).
Results of Question (4).
Finally, to answer question (4), on the Meta-World environment, we compare UDS and CUDS to ACL [73].
We use the version of ACL without inputting reward labels.
ACL can be viewed as an alternative to our unlabeled sharing data scheme, which leverages unlabeled
[Table]
data for representation learning rather than sharing it directly.
We show the comparison to ACL in Table 4.
UDS and CUDS outperform ACL in the average task performance while ACL is only proﬁcient on drawer-open and drawer-close, and it cannot solve door-open or door-close.
This indicates that sharing the unlabeled data conservatively across all tasks is important in multi-task ofﬂine RL while pretraining representations on the whole multi-task ofﬂine dataset might have limited beneﬁt.
We note that UDS / CUDS are complementary to ACL and these approaches can be combined together to further improve performance, which we leave as future work.
Conclusion
In the paper, we present two new algorithms, UDS and CUDS, that handle the problem of how to share data across tasks without access to the functional form of the multi-task reward function in the multi-task ofﬂine RL setting.
UDS lifts the strong assumption of having access to the reward of all tasks at each transition in previous works in multi-task data sharing via simply sharing data across all tasks and relabeling the reward of data from other tasks to the minimum reward in the MDP, which indicates failure of the task.
CUDS further improves over UDS via applying a more sophisticated data sharing scheme [80] that shares data only if the relabeled Q-values improve over the expected Q-values of the original task data.
We justify that UDS obtains Q-values that are lower-bounded by the Q-values learned by data sharing with true reward labels and then discuss that under certain structures of ofﬂine datasets, CUDS can selectively apply conservative policy evaluation on only transitions with high Q-values, resulting in a less conservative algorithm.
Empirically, we show that both CUDS and UDS signiﬁcantly outperform vanilla multi-task ofﬂine RL without data sharing as well as more complex methods that learns the reward function either explicitly or implicitly on a range of robotic manipulation and navigation domains.
CUDS also improves over UDS on all of the domains.
Furthermore, CUDS and UDS achieve competitive results compared to data sharing methods with access to the oracle rewards.
While our method removes the strong assumption on reward functions in data sharing for multi-task ofﬂine RL and enjoys both theoretical guarantees and good empirical results, it does have a few limitations.
For example, UDS and CUDS are evaluated in MDPs with binary rewards.
Exploring their effects in MDPs with continuous rewards will be an exciting future avenue.