ASPiRe: Adaptive Skill Priors for Reinforcement Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
We introduce ASPiRe (Adaptive Skill Prior for RL), a new approach that leveragesprior experience to accelerate reinforcement learning.
Unlike existing methods thatlearn a single skill prior from a large and diverse dataset, our framework learns alibrary of different distinction skill priors (i.e., behavior priors) from a collectionof specialized datasets, and learns how to combine them to solve a new task.
Thisformulation allows the algorithm to acquire a set of specialized skill priors thatare more reusable for downstream tasks; however, it also brings up additionalchallenges of how to effectively combine these unstructured sets of skill priors toform a new prior for new tasks.
Specifically, it requires the agent not only to identifywhich skill prior(s) to use but also how to combine them (either sequentially orconcurrently) to form a new prior.
To achieve this goal, ASPiRe includes Adaptive
Weight Module (AWM) that learns to infer an adaptive weight assignment betweendifferent skill priors and uses them to guide policy learning for downstream tasksvia weighted Kullback-Leibler divergences.
Our experiments demonstrate that
ASPiRe can significantly accelerate the learning of new downstream tasks in thepresence of multiple priors and show improvement on competitive baselines.
Introduction
Transferring prior experience to new tasks is central to an agent’s adaptability.
In this work, weaim to accelerate online reinforcement learning by leveraging prior experience from large offline data.
Previous works [1, 2, 3] have proposed to extract temporally extended actions into skills fromoffline datasets and learn a single behavior prior over the skill space, which we refer to as skill prior .
The skill prior is then used to guide the learning for a new downstream task by constraining thelearned policy to stay close to the prior distribution.
However, regularizing with a single skill priormight be insufficient for more complex downstream tasks.
We conclude two major limitations forusing a single skill prior: (1) The datasets could be generated by multiple policies for diverse tasks.
As a consequence, the action distribution from behavior trajectories might be multi-modal, and theresulting learned prior might learn an "average" behavior [4] (2)
Complex tasks not only requirereusing previous skills but also composing them.
Single skill prior might only facilitate the learningfor tasks requiring specific skills but not the composition.
Consider a task that requires the agent totraverse a maze with obstacles inside.
The agent needs to compose navigation skills and avoid skillsto complete the task (See Fig. 1).
Guided with only navigation skill prior or avoid skill prior mightlead to inefficient learning.
This observation suggests guiding the learning with multiple skill priors, with each prior specializedin one task.
During the learning of new downstream tasks, the most relevant priors should be selectedto regularize the learned policy based on the task and specific states.
Specifically, it requires the agentto:
(1) Identify which skill prior(s) to use.
Since not all skill priors are relevant to the new task athand, imposing irrelevant skill priors to regularize can mislead the learning process.
(2) Learn howto combine multiple skill priors.
Many complex tasks require the agent to identify and combinemultiple priors to solve them effectively.
The integration of skill priors can be either sequential(active one prior at a time) or concurrent (active multiple priors at the same time), which need to bedetermined by the agent automatically.
To address the above challenges, we propose ASPiRe , a new method to accelerate the learning of newdownstream tasks by guiding the learned policy with an adaptive skill prior.
To implement this idea,we first extract the multiple primitive skill priors from the separated and labeled offline data.
Eachprimitive skill prior is specialized in solving one specific task (e.g., navigation prior and avoid prior).
For a new downstream task, the learned policy is trained with a new task reward and regularized bythe adaptive skill prior.
The adaptive skill prior can be one of the primitive skill priors or constructedby composing primitive skill priors.
Our method includes an Adaptive Weight Module (AWM) that infers an adaptive weight assignmentover learned primitive skill priors and uses these inferred weights to guide policy learning throughweighted Kullback-Leibler (KL) divergence.
The weight applied to the KL divergence between thelearned policy and a given primitive skill prior determines its impact on the learned policy.
Thisimplicitly composes multiple primitive skill priors into a composite skill prior and guides the learnedpolicy.
Our formulation allows the algorithm to combine the primitive skill priors both sequentiallyand concurrently as well as adaptively adjust them during the online learning phase.
We evaluate our method in challenging tasks with sparse rewards.
We demonstrate that our methodcan accelerate the learning of new downstream tasks in the presence of multiple priors and showimprovement over competitive baselines on both learning efficiency and task success rate.
Related work
Offline reinforcement learning.
Offline reinforcement learning [4] exploits an existing offlinedataset and learns the best policy from it without interacting with environments.
This is valuablewhen interacting with the environment for a task is expensive.
Many algorithms have been developed,both in model-free [5, 6, 7, 8, 9, 10, 11, 12, 13] and model-based [14, 15, 16, 17, 18] settings.
Offline RL has also demonstrated its ability to accelerate online learning [2, 3, 19].
While most ofthe previous works only consider a behavior prior for a single task, the learned policy is guided bythe behavior prior to achieve efficient exploration.
Rao et al.
[20] proposes to use a mixture latentvariable model to compose multiple behavior priors, but this framework only activates one behaviorprior at a time.
How to leverage multiple behavior priors during online learning both sequentiallyand concurrently remains an open question.
Our method provides a solution to adaptively select abehavior prior (i.e., skill prior) or compose a new prior to guide the learned policy.
Reusing and Composing skills.
It is crucial for intelligent agents to reuse and compose learnedskills to solve new tasks.
A wide range of works have been proposed to solve this longstandingproblem [21, 22].
One way to compose and reuse skills is using value functions [23, 24, 25] orcumulants [26, 27, 28, 29].
Numerous works also explore the hierarchical structure in which ahigh-level policy integrates a collection of low-level controllers (primitives) [30, 31, 32, 33, 34] andeach of low-level controllers contains a distinct skill.
However, most of the hierarchical methodsonly sequentially activate the primitives.
The other line of work for reusing and composing skills isto operate on a latent space, which can be mapped to action space [35, 36, 37, 38, 39, 40].
Amongthose works, several approaches employ the idea of mixture-of-experts [41] by explicitly modeling alibrary of primitive skills and learning a weight gate function to compose the skills [42, 43, 44, 45].
These works show that this additional structure is beneficial for transferring skills to complex tasks.
Our work can also be viewed through the mixture-of-experts lens, i.e., solve downstream tasks bycomposing a library of primitives.
However, our work treats the composition as a regularization toguide the learned policy, instead of executing the composition as a policy directly.
Approach
[Figure]
The goal for ASPiRe is to acceleratedownstream task learning using a library of primitive skill priors.
Thereare two main challenges for this task:1)
Decide which primitive skill priorshould be activated given a new downstream task, where using unrelatedprimitive skill prior could mislead thelearning process.
2) How to best combine multiple primitive skill priors, either sequentially or concurrently.
Inparticular, to the best of our knowledge, how to simultaneously constrainthe policy learning with multiple skillpriors is an open challenge.
To address these challenges, we propose a method that regularizes the policy learning using an “adaptive” skillprior, which can be either one of theprimitive skill priors or a newly created skill prior by the composition of multiple primitive skill priors (i.e., the composite skill prior).
Our approach consists of two learning stages: offline and online (See Fig 1).
In the offline stage, thesystem learns multiple primitive skill priors from a set of labeled datasets.
Each dataset containsexpert demonstrations of one specific primitive task.
In the online stage (for downstream tasks), thealgorithm creates an adaptive skill prior by assigning weight over primitive skill priors.
The weightsare generated by the adaptive weight module (AWM).
Then they are used to regularize the policytraining.
Problem Formulation:
In the offline learning stage, the algorithm has access to K datasets{D 1 , ..., D K } with task labelling {T 0 , .., T K } .
Each dataset D i contains action-state trajectoriesτ = { ( s 0 , a 0 ) , ..., ( s T , a T ) } for task T
i (i.e. push).
The datasets are generated by expert behaviorpolicy µ 1 , .., µ K .
We further assume that none of the single behaviors can individually solve thetypically more complex downstream learning tasks.
The online learning stage for the downstream task is formulated as a reinforcement learning problem,modelled as an infinite-horizon Markov Decision Process (MDP) M
= ( S , A , P , R , ρ, γ ) , where Sis the continuous state space, A is the continuous action space, P is the dynamic p ( s ′ t | s t , a t ) , R isthe scalar reward r t , ρ is the initial state distribution and γ is the discount factor.
The goal is to learna policy π ( a t | s t ) to maximize the expected discounted sum of rewards (cid:80)
Tt =
E π
[ γ t r ( s t , a t ) ].
3.1 Extracting multiple primitive skill Priors
We denote an action sequence { a t , ..., a t + H − 1 } with fixed horizon H as a skill a t .
In the offlinelearning phase, we first learn a shared low-dimensional skill embedding space Z from aggregateddatasets {D i } Ki =0 .
We then construct primitive skill priors on top of the skill embedding
To learn the skill embedding, following the previous works [1, 2, 46], we fit a variational autoencoderand maximize the following evidence lower bound (ELBO):
[Equation]
Here, β is a parameter that controls the regularization term.
The skill encoder q ( z | a t ) embeds theskill a t into latent space Z and the skill decoder p ( a t | z ) reconstructs the latent vectors back to skills.
We set p ( z ) as a unit Gaussian prior to regularize the encoder learning.
Note that action sequencesa are randomly sampled from all offline dataset {D i } Ki =0 such that diverse action sequences can beembedded in a shared skill embedding space with the same encoder and decoder.
We freeze the skill encoder and decoder parameters after sufficient training.
Next, we extract multipleprimitive skill priors on top of the shared skill embedding.
For each offline dataset D i , we extract aprimitive skill prior P ia
( z | s t ) , which is parameterized as a Gaussian distribution.
To train primitives,we sample state and skill tuples ( s t , a t ) from D i and minimize the KL divergence between the skillposterior and the predicted prior E ( s t , a t )
∼D i D KL (cid:0) q ( z | a t ) , p ia ( z | s t ) (cid:1) , similar to previous work [2].
3.2 Regularized reinforcement learning via adaptive skill prior
For downstream task learning, instead of learning a policy on action space A , we learn a policyπ θ ( z | s t ) which acts on high-level skill embedding space Z .
Previous work [2] has demonstratedthat learning on the skill embedding space is beneficial for complex long-horizon tasks.
The skillembedding z can be decoded into an action sequence a via the skill decoder p ( a | z ) for H stepsaction execution.
We further replace the dynamic in our problem formulation with H-step dynamicp ′
( s t
+ H | s t , z t ) and rewards with H-step rewards r ′ t = (cid:80)
Ht =1 r t .
We guide the learned policy via an adaptive skill prior over the course of online learning for downstream tasks.
Ideally, the policy should be regularized by all relevant primitive skill priors at eachstate.
We consider two scenarios: 1) only one primitive prior is relevant, 2) multiple primitive priorsare relevant.
Previous work [2] considers the first scenario and regularizes the learning policy byproviding an additional negated KL divergence between the learned policy and the offline skill prior.
In states where multiple primitive skills are involved, the agent needs to achieve several goalssimultaneously, which are implicitly encoded in distinct primitive skill priors.
It is therefore natural toguide the learned policy with multiple primitive skill priors.
To implement this idea, we can augmentthe standard RL objective with a negated weighted KL divergence :
[Equation]
Here, we refer to ω ( s ) as the weighting function , which specifies the weights w = ( w 1 , w 2 , ..., w K )to apply on KL divergence between the learned policy and primitive skill priors.
We consider weightsare bounded { w i } Ki =1 ∈ [0 , 1] and are normalized (cid:80) Ki
=
w i = 1 .
Assigning weights to the primitiveskill priors implicitly forms a composite skill prior .
This can be interpreted as the learned policy willbe constrained by a skill distribution p c ( z t ) that can minimize the weighted KL divergence givenweights w at state s t :
[Equation]
We refer to it as composite skill prior generated by weight w at state s t .
Weights at a given state canbe viewed as a measure of how much one primitive skill prior should contribute to the compositeskill prior.
The state-dependent weights can be learned simultaneously with the learned policy, and we discuss it in section 3.3.
For now, we assume the weighting function ω ( s t ) is given, and the goalis to maximize the Equation 2.
We follow the policy constraints formulation [5, 6, 10] to optimize the objective.
We consider a
Q-function Q ϕ and a policy π θ parameterized by the network with parameters ϕ and θ , respectively.
The policy parameters can be optimized by minimizing the weighted KL divergence penalty:
[Equation]
Here α ≥ 0 is a temperature parameter to constrain the weighted KL-divergence, which can betuned automatically by providing a target divergence parameter δ [2] (See appendix A.3 for details).
Q-function parameters can be optimized by minimizing the Bellman residual:
[Equation]
3.3 Adaptive Weight Module
In this section, we introduce Adaptive Weight Module (AWM), a component to infer the optimalweights adaptively over the course of tasks.
Our goal is to learn a parameterized weighting functionω σ to compose the primitive skill priors via weighted KL divergence, and the resulting compositeskill prior can guide the policy learning.
Learning skill prior generator.
To efficiently evaluate the skills under composite skill priors, weimplement skill prior generator G η ( z | s t , w ) , a generative model which outputs the composite skillprior generated by the weights w given the state s t .
This can be optimized by minimizing the weighted
KL divergence between output distribution G η ( ·| s t , w ) and primitive skill priors:
[Equation]
We parameterize the skill prior generator as a Gaussian distribution by the network with parametersη .
During the learning, we randomly sample weights w from the unit simplex as the weight input.
This helps the skill prior generator to learn all possible compositions across states.
Learning adaptive skill prior.
Our goal is to construct an adaptive skill prior that is biased towardsthe high reward trajectories, similar to Siegel et al.
The key difference is that our approach aimsto construct such adaptive prior for a novel downstream task by composing multiple primitive skillpriors.
In effect, we would like to find weights assignment such that the corresponding compositeskill prior can contain the skills that are more likely to lead to task success.
We formulate this as:
[Equation]
where Q ϕ is the policy critic defined in section 3.2.
The Q-value in the objective measures theusefulness of skills for the task at hand.
The inner expectation can be viewed as a measurement of theusefulness of the composite skill prior generated by weight w .
The objective prefers a composite skillprior, in which skills have higher Q-value and are more promising to be explored.
We approximate theinner expectation with M samples from G η , i.e, E z ∼ G η ( ·| s t ,ω σ ( s t ))
[ Q ϕ ( s t , z )]
= 1 M [ Q ϕ ( s t , z )
| z ∼
G η ( ·| s t , ω σ ( s t ))] .
We find M = 20 is sufficient to provide a stable and accurate estimation.
We parameterize the weighting function ω σ by the network with parameter σ , and use softmax asthe output layer.
To learn the optimal weight, we set the Equation 7 as the loss function.
We add anadditional regularization term to penalize weights for deviating from uniform weight 1 K in the earlystage of the training. This encourages the learned policy to explore more skills in diverse primitiveskill priors and prevents weights from collapsing to a singleton due to the noise in Q-value in the
Algorithm
1: Inputs: K primitive skill priors { P ia ( z t | s t } Ki = 2: Initialize: replay buffer D , policy π θ ( z t | s t ) , critic Q ϕ ( s t , z t ) , target critic Q ¯ ϕ
( s t , z t ) , weighting function ω σ ( s t ) , skill
prior generator G η
( z t | s t , w t ) 3: for each iteration do 4: for every H environment steps do 5: z t ∼ π θ ( z t | s t )
▷ Sample skill from policy 6: s ′ t ∼ p
( s t
+ H | s t , z t ) .
▷ Execute skill 7:
D ← D ∪ { s t , z t , r t , s ′ t } ▷ Store experience in replay buffer 8: end for 9: for each gradient step do 10:
Sample a batch of transition tuples { s t , z t , r t , s ′ t } from D 11: w t = ω σ ( s t )
▷ Get weights from weighting function 12: θ ← θ − λ θ ∇ θ J π ( θ ) ▷ Update policy parameters via weighted KL divergence 13: ϕ ← ϕ − λ ϕ ∇ ϕ J Q ( ϕ )
▷ Update critic parameters 14: η ← η − λ η ∇ σ J ( η ) ▷ Update generator parameters 15: σ ← σ − λ σ ∇ σ J ( σ ) ▷ Update weighting parameters 16:
¯ ϕ ← τϕ + (1 − τ )¯ ϕ
▷ Update target critic 17: end for 18: end for early phase.
Here, β is the annealing coefficient, and it gradually decreases.
The weighting functionparameters can be trained to minimize the following:
[Equation]
For each gradient step, we optimize both the learned policy and the AWM.
The full Adaptive Skill Prior algorithm is summarized in Algorithm 1.
For implementation details, see appendix, A.3.
Experiments
We would like to answer the following questions through our experiments: a) Can leveraging multipleprimitive skill priors accelerate downstream task learning?
b) What’s the advantage of using thecomposite skills as regularization compared to directly using them as policy?
c)
Can AWM optimallyassign proper weights over different primitive skill priors based on the downstream tasks, even whenirrelevant primitive skill priors are presented?
4.1 Experiment setup
We evaluate our method in three modified environments from D4RL [47].
ASPiRe is given twoprimitive skill priors in the first two experiments.
For the last experiment, we test ASPiRe’s abilityto compose three primitive skill priors.
For each environment, we first collect datasets for distinctskills from expert demonstrations, which allows us to extract distinct primitive skill priors.
We thenevaluate our method with extracted primitives in downstream tasks.
Point Maze.
We modified the Point Maze environment by adding multiple obstacles in the maze.
The task is to navigate a point mass agent in the maze and reach the goal.
During this process, theagent must avoid the obstacles randomly placed in the maze.
The original point maze environment isalready challenging because of the sparse reward feedback.
Randomly placing obstacles in the mazemakes this problem even more difficult.
For each episode, the maze layout is fixed.
However, theagent start, goal, and obstacles locations are randomly specified.
Our method will be equipped withtwo primitive skill priors for this task: navigation and avoid.
Ant Push.
The ant needs to push the box to the goal while avoiding an obstacle in this task.
ASPiRewill carry two primitive skill priors: push and avoid.
The ant, box, obstacle, and goal locations are randomly generated at the beginning of every episode.
The agent will receive a sparse reward uponthe box reaching the goal location.
Ant Maze.
We modified the Ant Maze by adding an obstacle and a box in the maze.
The agent’sgoal is to traverse the maze while pushing the box to the goal position without hitting the obstacle.
The agent always starts in the lower right corner of the maze.
However, the box, goal, and obstaclepositions are randomly selected from a pre-defined set.
Ant Maze itself poses a complex explorationproblem due to sparse reward feedback.
We make this problem even more complex by changing thesuccess criteria.
In this case, ASPiRe will carry three primitives: push, navigation, and avoid.
See appendix A.4 for details on environment, data collection process and training.
[Figure]
ASPiRe
BC+SAC
MCP
Scratch
SPiRL
Point Maze
Ant
[Figure]
Alternative approaches.
We compare ASPiRe with several prior works for downstream task learningperformance.
To fairly evaluate the performance, we let all methods (besides SPiRL, as the methodlearns its own skill embedding) act on skill embedding space Z learned by the ASPiRe.
• Scratch.
Train a Soft Actor-Critic (SAC, [48]) agent from scratch.
This comparison tests thebenefit of using the learned skill priors to guide downstream task learning.
• Behavioral Cloning with finetuning.
A behavior cloning policy is first trained with offline dataand then finetuned using SAC for the downstream tasks.
• MCP [42] .
The method composes primitive policies through multiplicative composition of
Gaussians with a learnable weighting function.
MCP then samples the actions from the resulting multiplicative composite distribution.
For a fair evaluation, MCP is directly given the same set of primitives that ASPiRe is equipped with.
This comparison tests the benefit of treating compositeskills as prior, instead of directly executing them.
• SPiRL [2] .
Guide the learned policy with a single skill prior.
We train the skill prior by combiningall primitive datasets into one and learning a single prior from it.
This comparison tests thebenefit of guiding the learning policy with multiple primitive skill priors.
4.2 Experiment results
As shown in Fig.
2, our experiments demonstrate that ASPiRe is able to learn efficiently and convergeto high success rates in all challenging environments.
Benefits of multiple primitive skill priors.
We first compare ASPiRe with methods without skillprior guiding, namely training from scratch and behavior cloning with finetuning.
They fail to solveall three tasks (Fig. 2).
This confirms the finding in Pertsch et al.
[2] that skill prior is beneficialfor learning complex and long-horizon tasks.
Next, we compare our method with SPiRL [2], whichregularizes the learned policy with a single skill prior extracted from the aggregated dataset.
By leveraging the skill prior, it can achieve more efficient exploration but still relatively low learningspeed and final success rates compared to ASPiRe (Fig. 2).
We hypothesize that the performancegap between ASPiRe and SPiRL comes from the prior they deploy to guide the learned policy.
Wefind that SPiRL learns an "average" behavior over all primitives.
It prevents SPiRL from guidingstates where the specific primitive skills are required.
In the case of ASPiRe, it can construct anadaptive skill prior from all primitive skills priors.
Therefore, the learned policy can receive sufficientguidance to explore and learn the tasks.
[Figure] [Figure]
Benefits of using composite skills as regularization.
We compare ASPiRe with MCP, whichdirectly executes the composite skills as a policy with a learnable weight.
As shown in Figure 2,
MCP struggles to learn a meaningful policy for all three tasks.
This finding demonstrates that usingcomposite skills as regularization provides the necessary flexibility for ASPiRe to learn the policiesthat can specifically solve the downstream task.
We find the final learned policy has lower entropythan the skill priors in most cases.
In fact, skill priors provide guidance on the valid skills to explorethe environment, but the learned policy needs to further search in this valid skill space to solve thespecific problem.
We hypothesize that the resulting policy from MCP might be overly broad as themultiplicative composition of Gaussian inherent the relative high entropy from the primitives.
Learning optimal weights.
To demonstrate ASPiRe’s ability to learn optimal weights, we visualizethe weights generated by the weighting function for the primtive skills priors at different states for point maze and ant maze environments.
From Fig.
3 and 4, we find that ASPiRe can compose theprimitive skill priors both sequentially and concurrently throughout the tasks.
More importantly,
ASPiRe regularizes the learned policy with relevant priors, e.g., activates the avoid primitive whenobstacles are near the agent.
See appendix A.1 for more trajectories.
[Figure]
Training with irrelevant primitive.
To test ASPiRe’s ablity in handling irrelevant primitive skillprior, we modify the task of the Ant Maze environment, where the ant needs to reach the goal whileavoiding the obstacle along the way without pushingthe box.
We place the box in the lower right cornerof the maze, which is not on the path to the goal.
Inthis case, the push primitive is irrelevant, and regularizing with it will yield low learning efficiency.
From Fig.
5, we find that ASPiRe still achieves goodperformance even with irrelevant primitive.
We plotaverage weights over the states during the training.
ASPiRe weights the navigation primitive heavily andassigns a low weight for the irrelevant primitive – push.
The weight for avoid is also relatively lowsince there are fewer states where avoiding is needed (i.e., only when the obstacle is in view).
[Figure]
Guiding with composite or single primitive.
In thisexperiment, we test the advantage of using composite skill prior to guiding policy learning comparedto guiding with a single primitive skill prior.
To answer the question, we guide the learned policy withone of the primitive skill priors (i.e., push, navigation and avoid respectively) in ant maze environment.
From Fig.
6, we find that the learned policy guidedwith only the avoid primitive or navigation primitivefails to learn the task.
Guided with the push primitive achieves relatively good performance but fails tocomplete the setting requiring traverse in the maze.
This experiment suggests that composing multipleskill priors is beneficial for the complex multi-taskdownstream tasks.
Limitation and future work.
When extracting primitive skill priors, our algorithm requires separated andlabeled datasets, each containing a specialized skill.
However, such dataset may not always be easyto generate, and manually separating or defining different primitive skill priors can be challengingand sometimes ambiguous.
Therefore, an interesting future direction will be allowing the system toprocess unlabeled dataset and automatically extract a set of distinct primitive skill priors.
Conclusion
We present ASPiRe, an approach to transfer a spectrum of skills from offline data to accelerate thelearning of unseen downstream tasks.
We propose to guide the learned policy with multiple primitiveskill priors and transfer the diverse skills to RL via weighed KL divergence.
We model an Adaptive
Weight Module to learn optimal weights assignment and construct an adaptive skill prior.
We evaluate
ASPiRe in challenging, sparse reward environments and demonstrate the benefits of guiding thelearning with multiple skill priors.
We also show that ASPiRe is able to select the most relevant skillpriors even when unrelated skill priors are presented.
Checklist
For all authors...(
a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes
b) Did you describe the limitations of your work
[Yes] See Section {+
 (c)+}
Did you
discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines
and ensured that
your paper conforms
to them? [Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results
[N/A]
(b) Did you include
complete proofs of all theoretical results
[N/A]
If you ran experiments
a) Did you include the code, data, and
instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)
[Yes] The codeis in the supplement material We
provide the codes to regenerate the datasets anddownstream task learning and all instructions are inside readme. md
(b) Did you specify all the training details (e.g , data splits, hyperparameters, how they
were
chosen)?
[Yes] See appendix A.
(c) Did you report error bars (e.g., with respect to the random seed after running ex-
periments multiple times)?
[Yes] The shaded area in Fig.
represents the standard
deviation across 8 seeds
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See appendix A.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use the D4RLbenchmark and cite in the section 4.
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]