Skills Regularized Task Decomposition for Multi-task Offline Reinforcement Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Reinforcement learning (RL) with diverse offline datasets can have the advantage ofleveraging the relation of multiple tasks and the common skills learned across thosetasks, hence allowing us to deal with real-world complex problems efficiently in adata-driven way.
In offline RL where only offline data is used and online interactionwith the environment is restricted, it is yet difficult to achieve the optimal policyfor multiple tasks, especially when the data quality varies for the tasks.
In thispaper, we present a skill-based multi-task RL technique on heterogeneous datasetsthat are generated by behavior policies of different quality.
To learn the shareableknowledge across those datasets effectively, we employ a task decompositionmethod for which common skills are jointly learned and used as guidance toreformulate a task in shared and achievable subtasks.
In this joint learning, we use
Wasserstein auto-encoder (WAE) to represent both skills and tasks on the samelatent space and use the quality-weighted loss as a regularization term to inducetasks to be decomposed into subtasks that are more consistent with high-qualityskills than others.
To improve the performance of offline RL agents learned onthe latent space, we also augment datasets with imaginary trajectories relevant tohigh-quality skills for each task.
Through experiments, we show that our multitask offline RL approach is robust to the mixed configurations of different-qualitydatasets and it outperforms other state-of-the-art algorithms for several roboticmanipulation tasks and drone navigation tasks.
Introduction
In the reinforcement learning (RL) field, the offline RL research has recently gained much attention, asnumerous works showed that it is effective for various problems of sequential decision making to adopta data-driven learning mechanism using previously collected data of experiences and trajectories [1,2, 3].
In the meanwhile, multi-task RL is considered promising to enhance the generality of RLpolicies and improve the learning efficiency [4, 5, 6, 7, 8, 9, 10].
Recently, a data sharing methodfor multi-task learning was introduced to address the issue of limited data for real-world controlapplications [11].
Yet, multi-task RL has not been fully investigated in offline settings.
In the offline RL context, we present a novel multi-task model by which a single policy for multipletasks can be data-efficiently achieved and its learning procedure is robust to heterogeneous datasetsof different quality.
In offline RL where interaction with the environment is not allowed and arbitraryor low-performance behavior policies might be involved in data collection, it is important to maintainthe robustness in learning on different-quality data.
To this end, we devise a joint learning mechanismof skill (short-term action sequences from the datasets) and task representation, which enables thetask decomposition into achievable subtasks via quality-aware skill regularization.
We also employdata augmentation based on high-quality skills, thus creating plausible trajectories and alleviating the limited quality and scale issues of offline datasets.
Through experiments, we demonstrate thatour model achieves robust performance for multi-task robotic manipulation and drone navigation,without requiring additional interaction with the environment.
The main contributions of this paper are summarized as follows.
We first present a novel multi-taskoffline RL model that enables the task decomposition into achievable subtasks through the qualityaware joint learning on skills and tasks.
The model ensures the robustness of learned policies uponthe mixed configurations of different-quality datasets.
In doing so, we devise the data augmentationscheme specific to multi-task RL on limited offline datasets, aiming at creating imaginary trajectoriesthat are likely to be generated by expert policies.
We then evaluate our model under multi-task robotand drone scenarios and demonstrate its benefit particularly upon heterogeneous data conditions.
Overall Approach
In this section, we describe the problem of making optimal decisions upon a multi-task Markovdecision process (MDP) and briefly present our data-driven approach to the problem.
2.1 Preliminary
Reinforcement learning (RL) offers an active learning framework based on MDPs for tacklingsequential decision problems.
In conventional RL formulation, a learning environment is representedas an MDP M with ( S , A , P , R, γ ) where S is a state space, A is an action space, P ( s t +1 | s t , a t ) isa transition probability for states s t , s t +1 ∈ S and an action a t ∈ A , R ( s t , a t ) is a reward function,and γ ∈ [0 , 1] is a discount factor.
The objective of an RL agent is to find an optimal policy π ∗ ( a | s )by which the cumulative discounted reward J ( π ) =
E π [ (cid:80) ∞ t =
γ t r t ] [12] is maximized.
In general,the policy π is learned via continual interaction with a learning environment defined in an MDP.
Offline RL Offline RL aims at maximizing the cumulative discounted reward J ( π ) as explainedabove within the same MDP formulation; however, unlike conventional RL, it is assumed to use onlystatic datasets of previously collected trajectories D
= { ( s t , a t , r t , s t +1 ) } t for training.
Likewise,it rarely considers direct interaction with the environment.
Offline RL algorithms can increase theusability of previously collected data in the domain of making sequential decisions where temporalcredit assignment with long time horizons is important.
Multi-task RL Multi-task RL considers more than a single task when achieving the optimal policyπ ∗ .
It is normally formulated as a family of MDPs {T i =
( S , A , P i , R i , γ ) } i where each individualtask T i is associated with its respective MDP
and it is sampled according to a task distribution p ( T ) .
Hidden Parameter MDP To represent the implicit temporal dynamics properties in a multi-taskenvironment, which are relevant to the Markovian properties of each task, we introduce a hiddenlatent variable v t [13].
That is, we have R ( s t , v t , a t ) :
= R v t ( s t , a t ) and P ( s t +1 , v t +1 | a t , s t , v k ) :=
P v t ( s t +1 | a t , s t ) for s t , s t +1 ∈ S and a t ∈ A , where the actual state space is extended to S × Vand V is the collection of latent variables v t .
Then, we obtain the respective partially observable
MDP (POMDP) that is formulated as a tuple ( S × V , A , Ω , P V , R V , O , γ ) where Ω = S and
O ( s t , v t ) (cid:55)→ s t denote the observation space and the observation function, respectively.
2.2 Overall Approach For Multi-task Offline RL
In principle, offline RL algorithms train a policy π by optimizing the objective argmax π J D ( π ) −α · c ( π, π D ) which tends to mitigate the extrapolation problem [1].
J D is the average return of thelearned policy π in the empirical MDP ˜ M derived from static datasets D , which are generated bya behavior policy π D .
Some cost function c ( · ) is used to regulate the distance of the policies π andπ D
and α is a hyperparameter.
With this regularization by the behavior policy, offline RL algorithmsare often vulnerable to low-quality datasets.
Overfitting problems can occur such that the maximumaverage return max π J D ( π ) of ˜ M is much lower than that of its respective true MDP M , when alow-performance or arbitrary policy is used for data generation.
In multi-task offline RL, we reformulate a family of MDPs {T i } i as a hidden parameter MDP inthat multiple MDPs are combined into a single POMDP based on hidden parameters that specifytemporal Markovian properties of the environment.
While the overfitting issue of offline RL can bealleviated by exploring the relation of multiple tasks and inducing the shareable knowledge from theirii
[Figure]
regularized
[Figure]
Imaginary demonstrations
[Figure]
augmentation
[Figure]
imaginary demonstrations
[Figure]
augmentation.
In (a), sub-trajectories from static datasets are converted into skill embeddings and together enable the decomposition tasks into
In b), for training offline RL agents, imaginary trajectories similar to expert demonstrations are sampled from the space and added to the datasets.
datasets in a multi-task setting, it is not guaranteed that inferring the hidden parameters fully enablesthe well-structured representation of related tasks.
It is because the behavior policy heterogeneity andstate-action pair disparity of tasks can prevent the sub-trajectories of common-knowledge tasks frombeing closely mapped on the latent space [14, 15].
To address the issue of different-quality and heterogeneous datasets in the context of multi-task offline
RL, we take a novel task embedding approach that combines (a) skill-regularized task decompositionand (b) data augmentation by imaginary demonstrations, as illustrated in Figure 1.
(a)
The formerenables the decomposition and reformulation of an individual task in the latent space of achievablesubtasks by jointly learning the common skills and adapting the subtasks in more achievable representation via quality-aware skill regularization (in Section 3).
This task decomposition establishessubtask embeddings on the same latent space in which agents can be efficiently trained.
(b) The latterimproves the performance of offline RL agents trained on the latent space by augmenting datasetswith imaginary demonstrations (in Section 4).
For each task, its well-matched skills are consequentlyused to generate additional trajectories similar to expert data without online interaction.
This canimprove the adaptability of offline RL algorithms for such tasks that have only low-quality data or donot exist in the static datasets.
Then, using the subtask decomposition and augmented datasets withimaginary demonstrations, a multi-task policy is learned via offline RL algorithms.
Task Decomposition with Quality-aware Skill Regularization
In this section, we describe the task decomposition model with quality-aware skill regularization onmulti-task offline datasets.
The model includes skill embedding and task embedding networks thatare jointly learned with a regularization term based on the behavior quality, as shown in Figure 2.
Specifically, we define short-term action sequences as skills and embed them in a latent space.
Byjointly learning skill embeddings and task embeddings, we induce tasks to be transformed intosubtasks that are achievable by and aligned with skills.
In doing so, we incorporate the behaviorquality of each sub-trajectory into the joint learning procedure as part of skill regularization, thusmitigating the adverse effect by low-quality data.
3.1 Learning Skill Embeddings
To represent the agent’s behavior to a latent space Z , we use an auto-encoder mechanism.
Anencoder q ϕ takes as input a sequence of state-action pairs d t =
( s, a ) t − n : t + n − 1 for time intervaliii
[Figure]
[ t − n, t + n − 1] , mapping it to a latent vector b t ∈ Z , while a decoder p ϕ reconstructs the inputaction sequence a t − n : t + n − 1 from the pair of b t and s t − n : t + n − 1 .
We term the latent vector b t skill 117 embeddings , considering that action sequences on short-term horizons capture agent’s behaviors for aspecific task.
For maintaining the learning stability on skill embeddings b t ∈ Z , we use Wassersteinauto-encoder (WAE) [16] with the maximum mean discrepancy (MMD)-based penalty and a priordistribution on b t .
Then, the loss function is defined as
[Equation]
where { ˜ b i } mi =1 ∼ P B is sampled by a prior of skill embeddings, λ > 0 is a prior distribution-basedregulation hyperparameter, and { s t i , a t i , d t i } mi =1 ∼ D .
Here, L PR is used to restrict skill embeddings,
[Equation]
where m is the size of { b } , { ˜ b } and k :
Z × Z → R is the positive-definite reproducing kernel.
3.2 Skill-regularized Task Decomposition
To decompose an individual task into a set of shared and achievable subtasks, we use skill embeddingsas guidance for the analysis of relation between tasks.
We first view each task as a compositionof subtasks which can be modeled as a hidden parameter MDP.
For task embeddings, we thenuse the WAE-based model architecture similar to skill embeddings previously described.
For subtrajectories τ t =
( s t − n : t , a t − n − 1: t − 1 , r t − n − 1: t − 1 ) of n -length transitions each, we have an encoder 131 q θ : τ t (cid:55)−→ z t ∈ Z to yield task embeddings and a decoder p θ : ( s t , a t , z t ) (cid:55)−→ ( s t +1 , r t ) to expressthe transition probability P and reward function R .
Then, for { s t i , a t i , τ t i } mi =1 ∼ D , the loss functionfor task embeddings is defined as
[Equation]
We also add a skill regularization term.
Since in offline RL datasets, trajectories are not necessarilygenerated by experts or optimal policies for all tasks, we incorporate the quality of trajectories inthe regularization term.
Once sub-trajectories are transformed in skill embeddings, their quality isestimated by the episodic returns ˜ R ( s, a ) .
The quality-aware skill regularization loss is defined as
[Equation]
iv
Offline dataset D , subtask embedding parameter θ , skill embedding parameter ϕ
Regulation hyperparameter λ , batch size m , learning rate η loop
Sample { d t i ,τ t i ,s t i − n : t i + n ,a t i − n : t i + n ,r t i − n : t i } mi =1 ∼ D { b t 1 ,b t 2 ,...,b t m } =
q ϕ ( { d t 1 ,d t 2 ,...,d t m } ) , { z t 1 ,z t 2 ,...,z t m } =
q θ ( { τ t 1 ,τ t 2 ,...,τ t m } ) { ˜ b 0 , ˜ b
1 ,..., ˜ b n } ∼ P B = N (0 , 1) , { ˜ z 0 , ˜ z 1 ,..., ˜ z n } ∼ P Z = N (0 , 1)
L SE ( ϕ ) = 1 m (cid:80) mi =1 (cid:80) n − 1 j = − n ∥ a t i + j − p ϕ
( s t i + j ,b t i )
∥ 2 + L PR ( { b t i } mi =1 , { ˜ b i } mi =1 ) using (1)
L TE ( θ ) = 1 m (cid:80) mi =1 (cid:80)
j = − n ∥ ( s t i + j +1 ,r t
i + j ) − p θ ( s t
i + j ,a t i + j ,z t i )
∥ 2 using (3)
L SR ( θ ) = 1 m (cid:80) mi =1 ˜ R ( s t
i ,a t i ) ·
∥ q θ ( τ t i ) − q ϕ ( d t i ) ∥ 2 using (4)
L SRTD ( θ ) =
L TE ( θ ) + L PR ( { z t i } mi =1 , { ˜ z i } mi =1 ))
+ L SR ( θ ) using (5) ϕ ← ϕ + η · ∇ L SE , θ ← θ + η · ∇ L SRTD end loop return θ,ϕ
The overall loss is defined as
[Equation]
where P Z is a prior of task embeddings.
This allows the encoder q θ to generate the embeddings atsubtask-level (or subtask embeddings) upon a sequence of sub-trajectories which are agnostic totasks in multi-task settings.
In particular, each task is represented within close proximity to somehigh-quality skills learned from the trajectories with large episodic returns.
By increasing the use ofhigh-quality skills task-agnostically, this task decomposition reduces the adverse effect of low-qualitydata and induces the task decomposition into more achievable subtasks.
When training offline RLagents, we use the output of q θ as part of the state input to RL algorithms.
Algorithm 1 implementsthe learning procedure explained in (1)-(5).
Here we provide the analysis of skill regularization effects in (4).
Let q and p be a skill encoderand decoder obtained by minimizing the loss in (1), and consider p as part of the environmentsimilarly in [17, 18].
Then, the decoder p follows the MDP M p =
( S , A
= Z , P p , R p , γ ) in whicha high-level (skill) action z t ∈ Z is converted to such a low-level (primitive) action a t ∼ p ( ·| s t , z t )that directly interacts with the environment.
Furthermore, assume that sub-trajectories τ in (3) anda sequence of state-action pairs d in (1) are restricted to the current state.
Then, we obtain suchhigh-level policies q θ and q which are trained for the MDP M p .
As the output of q θ is containedin the input states of M p , our objective is to maximize the performance gap between q θ and q ;i.e., maximize
[Equation]
q ) where J p is the average return in M p .
Following [19], weobtain that η ( θ ) =
E s ∼ d qθ ,z ∼ q θ
[ R qs,z − V q ( s )] where d q θ is a state visit distribution induced byq θ , R qs,z is an episodic return induced by q , and V q is a value function of q .
However, in offline
RL, it is difficult to approximate q θ precisely, so we rather want to use the distribution of q forthe state visitation distribution of q θ without much propagation error.
To do this, we optimize˜ η ( θ )
=
E s ∼ q,z ∼ q θ
[ R qs,z − V q ( s )] under the restriction that q and q θ remain in close proximity [20],maximize θ ˜ η ( θ ) :
=
E s ∼ d q ,z ∼ q θ
[ R qs,z − V q ( s )] subject to KL ( q θ ( ·| s ) || q ( ·| s )) ≤ ϵ.
Under the KL-divergence constraint in (6), we obtain that the state distribution of d q corresponds tothat of the dataset D , implying s ∼ d q ∼ = D .
Using the classical Lagrangian argument, we can simplify the optimization problem in (6) into anonconstraint optimization problem,
[Equation]
where β is a Lagrangian multiplier.
By differentiating the right-hand side of (7) with respect to q θand following the optimal policy derivation procedure in [21, 22], we obtain the closed form solutionthat satisfies the return-weighted condition below.
[Equation]
When omitting the baseline term V q ( s ) and up to the constant, we also obtain that the weighted skillregularized loss in (4) renders subtask embeddings to be matched with high-quality skills for a giventask, thereby facilitating the task decomposition into shareable and achievable subtasks.v
[Figure]
Data Augmentation by Imaginary Demonstrations
In offline RL, since a given static dataset might not fully represent its respective true MDP andfurther exploration is not allowed, it is common for RL agents to experience sub-optimal performance.
Generative models and noise are used to generate additional trajectories and enable local explorationfor offline RL algorithms without interaction.
In this section, we introduce a data augmentationmethod specific to the aforementioned task decomposition with quality-aware skill regularization,so that we can tackle the overfitting and limited performance issue.
While existing works aim atreducing the adverse effect of unseen states by exploiting state augmentation methods [23, 24, 25],we focus on augmenting such trajectories (imaginary demonstrations) that are likely to be generatedby a high-quality skill-based learned policy.
Specifically, we integrate the task decoder p θ and skill decoder p ϕ into a generative model, asillustrated in Figure 3.
Note that both p θ and p ϕ are established through the skill-regularized taskdecomposition.
Then, we obtain
[Equation]
where q θ is the task embedding model and τ t ∼ D .
Note that in this generative model, p θ performsthe same role of the world model in conventional model-based RL approaches.
The skill-regularized task decomposition enables q θ to be an optimal policy in the MDP M p ϕ asexplained in (8).
Accordingly, it turns out that the augmentation procedure in (9) yields a plausibletrajectory similar to expert demonstrations, given that the high-quality skill corresponding to thetrajectory is incorporated into p ϕ .
In consequence, the procedure in (9) can alleviate the negativeeffect of low-quality datasets.
Figure 4 depicts the effect of our data augmentation by imaginarydemonstrations, where (a) and (b) show the difference of low-quality and high-quality datasets interms of the state-action pair distribution.
Our imaginary demonstrations in (c) generated from (a) thesource dataset share similarity with (b) the expert dataset, while (d) the dataset generated by someconventional augmentation using Gaussian noise does not.
The table shows the average reward ofdatasets calculated by reward relabeling, indicating that our skill-based augmentation by imaginarydemonstrations produces higher quality data compared to the Gaussian noise-based augmentation.
Experiments
In this section, we evaluate the performance of our model under various configurations of multi-taskoffline datasets.
Experiment settings For evaluation, we use several robotic manipulation tasks and drone navigationtasks using the Meta-world environment [26] and the Airsim drone simulator [27].
The detailedsettings including hyperparameters and environment conditions can be found in Appendix.
Comparison methods We implement several multi-task RL methods for comparison including:
TD3+BC [2] is a state-of-the art offline RL algorithm, which incorporates a behavior cloning regularization term into the update steps of TD3, inducing the learned policy towards the actions found in thedataset.
For multi-task learning, TD3+BC is extended to include one-hot encoded task representationas part of the state.
We use this baseline to compare our approach with conventional multi-taskvi
[Figure]
RL algorithms without considering subtask decomposition.
PCGrad [7] is a gradient surgery-basedmulti-task RL algorithm, which uses a projection function to remove the directional conflicts betweengradients.
This is implemented based on the hypothesis such that performance degradation can beexacerbated by gradient conflicts when training uncorrelated tasks.
Soft modularization (SoftMod) [8]is a modular deep neural network architecture tailored for multi-task RL.
To mitigate the negativeimpact of learning different tasks on a single policy, it leverages the softly weighted routing path on aset of modules that are specifically trained on multiple tasks.
It also employs a loss-balancing strategyto rapidly adapt to the different learning progress of the tasks.
Offline datasets We create and use three types of datasets according to specific behavior policies atdifferent quality levels.
Medium-Replay (MR) denotes the datasets sampled by a learning processfrom the initial to partially-trained medium policies, Replay (RP) denotes the datasets sampled duringa whole learning process, and Medium-Expert (ME) denotes the datasets sampled by a learningprocess from the medium to expert policies.
Note that each dataset in MR, RP, and ME for a taskcontains episodic trajectories of 150, 100, and 50, respectively, unless otherwise stated.
5.1 Meta-world Tests
Here, we evaluate our model using the MT10 benchmark (i.e., 10 different control tasks) in Metaworld [26], where each task is given a specific manipulation objective such as opening a door orclosing a window.
The tasks share common primitive functions such as grasp and moving, so they canbe seen as general multi-tasks with shared subtasks, which are consistent with our task decompositionstrategy.
The implementation detail can be found in Appendix.
Performance on MT10 benchmark Table 1 shows the performance in the average success rate on
MT10 under mixed configurations of different datasets (MR, RP, ME).
We compare the performanceachieved by our model (SRTD, SRTD+ID) and other methods (TD3+BC, PCGrad, SoftMod).
Notethat Skill Regularized Task Decomposition (SRTD) is trained as in Section 3, and SRTD with
Imaginary Demonstrations (SRTD+ID) is trained by our whole model introduced in Sections 3and 4.
Our model (SRTD, SRTD+ID) yields the best performance consistently for all configurations,demonstrating its superiority with 8.67% ∼ 17.67% higher success rates, compared to the mostcompetitive comparison method, SoftMod.
SRTD consistently shows robust performance, while thecomparison methods do not.
TD3+BC and PCGrad show better performance for the configurations oflow-quality datasets, e.g., the row of (MR 10, RP 0, ME 0), but SoftMod shows better performancefor the configurations of high-quality datasets e.g., the row of (MR 0, RP 0, ME 10).
TD3+BC and PCGrad explore the orthogonality of tasks by accumulating task-specific knowledgeseparately without much interference when learning different tasks, and SoftMod rather exploits thecommonality of the tasks by learning shared skills and dynamically extracting task-specific knowledgeby the combination of its modules [7, 8]. Specifically, our TD3+BC implementation with one-hottask encoding tends to learn individual tasks separately, considering that the task encoding does notrepresent the semantic relation of different tasks explicitly. PCGrad intends for geometric separationof model updates for minimizing both the individual loss of each task and the distance betweenthe weights of different tasks.
SoftMod utilizes the shared weights of a modular network for jointlyvii [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
optimizing the multiple loss functions for the tasks [28].
Our SRTD tends to adjust both orthogonality and commonality using the quality-aware joint learning, so it can achieve robust performance for different mixed configurations
SRTD
ID improves the performance over SRTD by 2.97% at average, clarifying the benefit of imaginary demonstrations in offline datasets
Ablation study Figure 5(a) shows the effect of our skill regularization, where TE denotes thetask embedding without skill regularization (i.e., only L TE in (3) is used), and SRTD-Q denotes
SRTD without the quality weighted term ˜ R ( s, a ) in (4).
SRTD shows better performance thanthe others consistently for all configurations.
However, SRTD-Q shows worse performance than
TE, which specifies the benefit of quality-aware regularization.
Figure 5(b) shows the effect ofour data augmentation method, where SRTD+N denotes SRTD with the Gaussian noise-baseddata augmentation commonly used in offline RL [23].
We observe that SRTD+N rarely achievesperformance gains, compared to SRTD.
This clarifies the advantage of SRTD+ID that leverageshigh-quality skills to generate trajectories.
5.2 A Case Study for Airsim-based Drone Navigation
Here, we verify the applicability of our model in real-world problem scenarios by conducting acase study with autonomous quad-copter drones in the Airsim simulator [27].
We configure variousrealistic maps in PEDRA [29] and diverse wind patterns to build a multi-task drone flying environment.
Regarding the RL formulation, the drone agent is set to observe its lidar data, position, speed, and angleof rotation, while it is set not to observe any data about wind patterns.
With continuous observations,the drone agent conducts control actions by manipulating the 3-dimensional acceleration, and receivesrewards based on the goal distance.
For evaluation, we measure the normalized episodic return basedon the maximum episodic return obtained by a fully trained online RL agent.
Table 2 compares theperformance by our model and the others.
For all mixed configurations, our model shows the bestperformance, outperforming the competitive SoftMod by 5.01% ∼ 11 .
37% .viii
[Table]
Related Work
Multi-task RL Multi-task RL has been investigated for sample-efficiently dealing with complexcontrol problems in real-world settings [30, 6, 8, 31].
By training a deep neural network with multipletasks jointly, multi-task RL algorithms drive agents to learn how to share, reuse, and combine theknowledge across correlated tasks.
Yang et al.
[8] presented an explicit modular architecture witha soft routing network for training an integrated multi-task policy.
This, called soft modularization,addresses the issue of unclear task relation in a single network such that which shared parameters arerelated to which tasks.
Yu et al.
[7] proposed a gradient surgery methodology that directly removesthe negative effects of multi-task learning in a single policy, identifying and adjusting geometricconflicts of calculated gradients when learning different tasks.
Task and skill embeddings in multi-task RL Several approaches using task embeddings have beenintroduced in the context of meta RL [32, 33, 34], multi-task RL [6, 35], imitation learning [36,37], and non-stationary RL [38, 39].
Pertsch et al.
[40] demonstrated that with a pretrained low-levelpolicy that readily achieves given skills, a high-level policy yielding appropriate skills can facilitatethe learning efficiency, where skills are embedded in the latent space with expert data.
Sodhaniet al.
[6] used additional metadata for learning a multi-task policy, exploiting task descriptions innatural language to represent the semantics and relation of tasks in the latent space.
While theseprior works rely on online interaction and they rarely consider heterogeneous datasets and differentbehavior polices, which are common in multi-task offline RL, our model employs the quality-awareregularization to handle the mixed configurations of multi-task datasets.
We also devise a jointlearning mechanism for skill and task representation in offline settings.
Data augmentation in offline RL To alleviate the issue of limited datasets and unseen states,several works exploited data augmentation [23, 41], data sharing [15, 11, 42], and model-basedapproach [43, 41] in offline RL.
For example, Sinha et al.
[23] tested several data augmentationschemes, demonstrating the possible performance gain with offline RL algorithms.
Yu et al.
[11]presented the conservative Q-function that can judge which transitions are relevant for learning aspecific task, thus establishing a conditional data sharing strategy upon data scarcity situations.
Ourdata augmentation by imaginary demonstrations is in the same vein, but it focuses on exploitingcommon skills to generate trajectories that are likely to be generated by expert policies.
Conclusion
In this study, we proposed a novel multi-task offline RL model to tackle the problem of heterogeneousdatasets of different behavior quality across tasks.
In the model, skills and tasks are jointly learnedwith quality-aware regularization so that achievable subtasks are found and aligned with high-qualityskills.
Our model consistently yields robust performance upon various mixed configurations ofdifferent-quality datasets without requiring additional interaction with the environment.
The directionof our future works is to investigate the hierarchy of skill representation with different temporalabstraction levels in multi-task offline RL.
This will tackle the limitation of our model that considersonly fixed-length sub-trajectories for task and skill embeddings.ix
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
xii