Bootstrapped Transformer for Offline Reinforcement Learning
Anonymous Author(s)
AffiliationAddress
Abstract
Offline reinforcement learning (RL) aims at learning policies from previouslycollected static trajectory data without interacting with the real environment.
Recentworks provide a novel perspective by viewing offline RL as a generic sequencegeneration problem, adopting sequence models such as Transformer architecture tomodel distributions over trajectories, and repurposing beam search as a planningalgorithm.
However, the training datasets utilized in general offline RL tasks arequite limited and often suffer from insufficient distribution coverage, which couldbe harmful to training sequence generation models yet has not drawn enoughattention in the previous works.
In this paper, we propose a novel algorithmnamed Bootstrapped Transformer, which incorporates the idea of bootstrappingand leverages the learned model to self-generate more offline data to further boostthe sequence model training.
We conduct extensive experiments on two offline RLbenchmarks and demonstrate that our model can largely remedy the existing offline
RL training limitations and beat other strong baseline methods.
We also analyzethe generated pseudo data and the revealed characteristics may shed some light onoffline RL training.
Introduction
Online reinforcement learning (RL) [42, 33, 8] requires environments and interactions to train policies.
However, such a requirement obstructs RL from being applied to many real-world problems wheresimulation environments are unavailable or interactions are incredibly costly.
Offline RL tries to getrid of the requirement and learn policies from a previously collected offline dataset without interactingwith the real environment.
This setting enables more practical RL training in the real world and thusattracts lots of attention [27].
Unlike conventional approaches that focus on learning value functions [41] or computing policygradients [34] recent works [32, 3] view the offline RL problem from a novel perspective where it isconsidered a generic sequence generation problem.
Specifically, these works utilize a mainstreamsequence modeling neural network, i e., Transformer [47] to model the joint distribution of the offlinedecision-making trajectory data, after discretizing each term vector within the transitions (states,rewards, actions), in a supervised learning (SL) manner.
In such a sequence modeling framework, atrajectory consisting of states, actions, and rewards is viewed as a stream of tokens, and the learned
Transformer can generate tokens in an autoregressive way and subsequently be utilized for executionin the real environments.
This framework unifies multiple components in offline RL in one bigsequence model, such as estimation of behavior policy and modeling for predictive dynamics [32],thus achieving superior performance.
However, there exist two major data limitations in offline RL tasks, which might be harmful totraining sequence generation models yet have not been well addressed in the previous works.
First, the offline data coverage is limited.
The training data in offline RL tasks could be collected byarbitrary policies like human experts or random policies, which cannot guarantee sufficient coverageof the whole state and action spaces [11, 36].
The second issue lies in the amount of training data.
Specifically, the largest dataset in the widely-used offline RL benchmark D4RL [9] contains onlyabout 4,000 trajectories, which is quite limited especially considering those in natural languagesequence modeling such as WikiText [31] which contains about 3,800,000 sentence sequences.
Suchlimited datasets may deteriorate the sequence generation model, since the common belief in theliterature is that training sequence generation models from scratch often requires a large amount ofdata with rich information [24].
Therefore, handling limited datasets should be carefully consideredin the sequence modeling methods in offline RL tasks.
Some previous works try to tackle the above data limitations via data augmentation to expand thedata coverage, either through image augmentation [25, 51, 50] such as cropping, or by applyingperturbations on the state inputs [26, 43].
Nevertheless, these works can neither be applicable forother types of observations except image-based input, nor ensure to be consistent with the underlying
Markov decision process (MDP).
Another way is to train an additional model learning environmentdynamics [18, 20, 30, 52].
Yet it requires extra efforts, and the goals of model learning and policyoptimization have not been aligned, which may derive suboptimal solutions [53].
Moreover, all thesemethods have not considered the advanced sequence modeling framework.
In this paper, we try to relieve these problems under the sequence modeling paradigm for offline RL.
Specifically, we consider using the learned model itself to generate data and remedy the limitationsin training datasets .
We further propose a novel algorithm, Bootstrapped Transformer (BooT) ,to implement the above idea.
Generally, BooT uses Transformer to generate data and then uses thegenerated sequence data of high confidence to further train the model for bootstrapping.
To make ourstudy comprehensive, we investigate two generation schemes.
The first one performs autoregressivegeneration, and the other uses the teacher-forcing strategy [48].
We also test two bootstrappingschemes.
One trains the model further on the generated data and discards them once they have beenused; the other uses the generated data repeatedly during the later training process.
In this way, the above issues in previous works can be addressed simultaneously.
Specifically, thesequence generation model unifies predictive dynamics models and policies [32]; thus, we do notneed to train extra environment models for generating data, and such a generation-based method canalso be applied to proprioceptive observations (e.g., positions & velocities information).
Moreover,since the sequence model learns the distribution of the dataset and models the sequence generationprobability, the data generated by the learned model itself will be more consistent with the originaloffline data [29].
Thus, we expect higher consistency of the pseudo trajectory data generated by themodel than those of adding random noises, which has been empirically verified in our experiment.
We further conduct comprehensive experiments on multiple offline RL benchmarks, and BooTachieves significant performance improvements.
For Gym domain in D4RL benchmark, BooT evenreaches a much higher average score than one of the baselines with a pre-trained 80 times larger model[39], which strongly demonstrates the superiority of BooT.
We also provide an in-depth analysis ofdata generated by BooT with both qualitative and quantitative results.
Visualization demonstratesthat the generated data act as an expansion of the original training dataset.
Besides, we quantitativelymeasure the difference between the original and generated trajectories, and empirically conclude thatthe generated pseudo trajectories from BooT generally lie in a neighborhood of the correspondingoriginal trajectories while keeping consistent with the underlying MDP in RL tasks, providing strongsupport for our motivation.
Related Work
Offline Reinforcement Learning.
Offline RL algorithms learn policies from a static dataset withoutinteracting with the real environment [27].
Deploying off-policy RL algorithms directly into the offlinesetting suffers from the distributional shift problem, which could cause a significant performancedrop, as revealed in [11].
To address this issue, model-free algorithms try to constrain the policyclosed to the behavior policy [11, 10] or penalize the values of out-of-distribution state-action pairs[23, 21].
On the other side, model-based algorithms [52, 20] simulate the real environment andgenerate more data for policy training.
All these methods tend to require the policy to be pessimistic,while our algorithm does not involve this additional constraint.
Besides, our algorithm is built uponthe sequence generation framework, which is different from the above methods.
Sequence Modeling.
Sequence modeling is a long-developed and popular area, and various methodshave been proposed in natural language processing [16, 44], speech recognition [6], and informationsystem [40].
Recently, Transformer [47] has illustrated powerful model performance and has beenwidely used in sequence modeling and generation tasks [5].
Some recent papers adapt Transformerinto RL by viewing RL as a sequence modeling problem.
Decision Transformer (DT) [3] learnsthe distribution of trajectories, and only predicts actions by feeding target rewards and previousstates, rewards, and actions.
Trajectory Transformer (TT) [32] further brings out the capability ofthe sequence model by predicting states, actions, and rewards and employing beam search.
Weleverage the paradigm of TT and provide a generation-based data augmentation methodology toimprove the data coverage and further improve the performance.
Recent works also try to usepre-trained Transformers on other domains and finetune on offline RL [39] or other decision-making[17, 28] tasks.
We focus on the data augmentation method, which is different and compatible withthe pre-training techniques.
Data Augmentation (DA).
DA is a common approach to promoting data richness and enhancingmodeling efficacy in supervised learning scenarios.
They adopt data augmentation either throughpseudo labeling [49] , input perturbation [4], or generative model [15] to produce additional data.
Although DA has not been widely used in RL tasks, recent works have shown the promisingperformance of DA in RL methods.
Some of them target the data inefficiency issue.
Specifically,
RAD [26] incorporates common perturbation-based approaches, while DrQ [50], DrQ-v2 [51], and
DrAC [37] also introduce regularization terms with respect to the augmented data in actor and critic.
Some other works [14, 7, 13] try to achieve high stability for online RL via data augmentation.
Foroffline RL, S4RL [43] provides various raw-state-based DA methods and takes similar regularizationto DrQ. Different from the previous works, we augment the offline data distribution through sequencegeneration by the learned Transformer model to remedy the limitations in training datasets for betterfacilitating offline policy learning in this paper.
Preliminary
3.1 Online and Offline Reinforcement Learning
Generally, RL models the sequential decision making problem as a Markov Decision Process
M = ( S , A , T , r, γ ) , where S is the state space and A is the action space.
Given state s , s ′ ∈ S andaction a ∈ A , T ( s ′ | s , a ): S ×
A × S → [0 , 1] is the transition probability and r ( s , a ):
S × A → Rdefines the reward function.
γ ∈ (0 , 1] is the discount factor.
The policy π : S ×
A → [0 , 1] takesaction a at state s with probability π ( a | s ) .
At time step t ∈ [1 , T ] , the accumulative discountedreward in the future, named reward-to-go, is R t =
(cid:80)
Tt ′
=
t γ t ′ − t r t ′ .
The goal of online RL is to find apolicy π that maximizes
J = E a t ∼ π ( ·| s t ) , s t +
∼T ( ·| s t , a t ) [ (cid:80) Tt =
1 γ t − 1 r t ( s t , a t )] by learning from thetransitions ( s , a , r, s ′ ) through interacting with the environment in an online manner [45].
Offline RL, instead of interacting with the real environment, makes use of a fixed dataset D collectedby behavior policy π b , to learn a policy π that maximizes J for subsequent execution in the realenvironment.
Here, π b can either be a single policy or a mixture of policies and is inaccessible.
Weassume that the collected data is trajectory-wise, i.e., D = { τ i } Di =1 , where τ = { ( s i , a i , r i , s ′ i ) }
Ti =1 .
3.2 Offline Reinforcement Learning as Sequence Modeling
In this paper, we take the offline RL problem as a sequence modeling task, following the previouswork [32].
We introduce the details of the sequence modeling paradigm in this subsection.
After that,we illustrate our data augmentation and bootstrapping strategy to remedy the limitations in trainingdatasets and further improve the learning performance in Sec. 4.
First, we treat each input trajectory τ as a sequence and add reward-to-go R t = (cid:80)
Tt ′
=
t γ t ′ − t r t ′after reward r t as auxiliary information at each timestep t , which acts as future heuristics for furtherexecution planning,
[Equation]
Second, a discretization approach is adopted to transform the continuous states and actions intodiscrete token spaces, each of which contains N and M tokens, respectively.
This turns τ aux into a sequence of discrete tokens with a total length of ( N + M + 2) T as
[Equation]
Without causing confusion, we will keep using τ to indicate discretized augmented trajectory τ dis inthe following.
Finally, we model τ with the Trajectory Transformer (TT)
[32] from the perspective of sequencemodeling.
TT regards each trajectory as a sentence and is trained with the standard teacher-forcingprocedure.
Let θ be the parameters of TT and P θ be its induced conditional probabilities.
Definelog P θ ( τ t | τ <t ) =
N
[Equation]
log P θ ( s it | s <it , τ <t ) +
M (cid:88)
j = log P θ ( a jt | a <
jt , s t , τ <t ) + log P θ ( r t | a t , s t , τ <t ) + log P θ
( R t | r t , a t , s t , τ <t ) , (3) the training objective is to maximize
L ( τ ) =
T (cid:88) t = log P θ ( τ t | τ <t ) , (4) where τ <t is the trajectory from timestep 1 to ( t − 1) , s <
it is dimension 1 to ( i − 1) of state s t , andthe similar for a <jt .
As for model inference, we adopt beam search as the planning algorithm to maximize cumulativediscounted reward plus reward-to-go estimates, similar to [32].
Bootstrapped Transformer
In this section, we describe our Bootstrapped Transformer (BooT) algorithm, utilizing self-generatedtrajectories as auxiliary data to further train the model, which is the general idea of bootstrapping.
Bootstrapping for supervised learning, such as pseudo labeling [19] and self-training [49], has beenwidely used.
Apart from those using pseudo label generation, in the RL domain, we directly utilizethe learned sequence model to generate novel trajectory data, which has been an expansion of theoriginal offline data (as shown in Sec. 5.3), and we further augment the original data distributionusing the generated data to better approximate the real distribution following the underlying MDP.
The self-generated trajectories are fed as additional training data to further boost sequence modelingand decision making.
Overall, we divide BooT into two main parts, (i) generating trajectories with the learned sequencemodel and (ii) using the generated trajectories to augment the original offline dataset for bootstrappingthe sequence model learning.
The overall training algorithm of BooT has been described in Algorithm1.
Truncated Original Trajectory 𝑠 𝑡 𝑠 𝑡𝑁 𝑎 𝑡 𝑎 𝑡𝑀 𝑟 𝑡 𝑅 𝑡 ⋯ ⋯ ⋯
Ƹ𝑠 𝑡ො𝑎
𝑡ො𝑎
𝑡Ƹ𝑟 𝑡 ෠𝑅 𝑡 Ƹ𝑠 𝑡+11 ⋯ ⋯ ⋯
Trajectory Transformer
Teacher-forcing Generation Ƹ𝑠 𝑡+1ො𝑎 𝑡+1ො𝑎 𝑡+1Ƹ𝑟 𝑡+1 ෠𝑅 𝑡+1 Ƹ𝑠 𝑡+21 ⋯ ⋯ ⋯
𝑠 𝑡 𝑠 𝑡𝑁 𝑎 𝑡+ 𝑎 𝑡+1𝑀 𝑟 𝑡+1 𝑅 𝑡+1 ⋯ ⋯ ⋯
Trajectory Transformer
Autoregressive Generation Ƹ𝑠 𝑡+ ො𝑎 𝑡+11 ො𝑎 𝑡+
Ƹ𝑟 𝑡+1 ෠𝑅 𝑡+1 Ƹ𝑠 𝑡+2⋯ ⋯ ⋯
Trajectory Transformer Ƹ𝑠 𝑡+11 ⋯ Ƹ𝑠 𝑡+1𝑁 ො𝑎 𝑡+11 ⋯ ො𝑎 𝑡+1𝑀 ⋯
Ƹ𝑟 𝑡+1 ෠𝑅 𝑡+
Concatenate
Truncated Original Trajectory
Predicted Tokens 𝑠 𝑡 𝑠 𝑡𝑁 𝑎 𝑡 𝑎 𝑡𝑀 𝑟 𝑡 𝑅 𝑡 ⋯ ⋯
Ƹ𝑠 𝑡+ Ƹ𝑠 𝑡+1𝑁 ො𝑎 𝑡+ ො𝑎 𝑡+1𝑀 ⋯ ⋯ ⋯
⋯
Ƹ𝑟 𝑡+1 ෠𝑅 𝑡+
4.1 Trajectory Generation
In this section, we demonstrate the details of trajectory generation.
For an original trajectory τ ∈
Din the original dataset, we resample the last T ′ timesteps ( T ′ < T ) as
[Equation]
[Equation]
where τ ≤ T − T ′ represents the original trajectory truncated at timestep ( T − T ′ )
Then, we concatenatethe generated token sequence to the truncated original trajectory and obtain a new trajectory
[Equation]
where ◦ denotes concatenation operation.
We investigate two different trajectory generation schemes below, both of which are demonstrated in
Figure 1.
We briefly compare these two methods in both efficiency and generation quality here, andperform detailed comparison experiments in Sec.
Autoregressive generation is to produce each predicted token ˜ y n by sampling from the probability
conditioned on the previously generated tokens as
[Equation]
Here we denote the n th generated token as y n and the sequence
of
all
the
previously generatedtokens as ˜ y <n .
The
autoregressive
generation
scheme
requires recurrent predictions based on the
previously generated tokens.
Teacher-forcing generation adopts a similar procedure as teacher-forcing training by condition-
ing on the original token ground truth from the previous timesteps
[Equation]
Here, y <n denotes tokens from the original trajectory with position previous to ˜ y n .
Teacherforcing generation only relies on ground truth tokens.
Due to error accumulation in the sequential generation process, autoregressive generation tendsto produce trajectories that deviate more from the original data distribution than teacher-forcinggeneration.
As has been found in Sec.
5.3, autoregressive generation could expand the coverage ofthe training data more effectively and derive better learning efficacy.
Compared to autoregressivegeneration, teacher-forcing requires only one forward pass and thus consumes less time.
However,since teacher-forcing generation is not able to expand dataset coverage as effectively as autoregressivegeneration, it will lead to less improvement in performance.
4.2 Training on the Augmented Dataset
After generating new trajectories, we use these trajectories to train our sequence model in addition tothe original trajectory data.
In BooT, we conduct two different bootstrapping schemes: Bootstrap-once, denoted as BooTo , and Bootstrap-repeat, denoted as BooT-r .
As shown in Algorithm 1, both methods will firsttrain the sequence model over the original offline trajectories and then utilize it to generate noveltrajectories.
After that, BooT-o will train the model again on the generated trajectories and discardthem immediately once they have been used.
In contrast, BooT-r will append the generated trajectories
G into the dataset D , and the generated trajectories will be used in every epoch after they are appendedto the dataset.
In order to prevent accumulating learning bias caused by training with inaccurately generated data,we choose a part of the trajectories with the highest confidence scores in each batch according tothe generation percentage η % .
Here, we define confidence as the average log probability of all thegenerated tokens as
[Equation]
where log P θ ( τ t | τ <t ) is defined in Eq.
Intuitively, since the confidence score of trajectory isdefined by its prediction probability that models the sequence generation mechanism, by filtering thegenerated trajectories with their confidence score, we can keep only the generated trajectory data of ahigh quality to further help the sequence model training through feeding these quality data back forbootstrapping.
Moreover, to stabilize bootstrapped training, we leverage the idea of curriculum learning described in[2] but adopt a more straightforward method here.
The sequence model is first trained over the originaloffline dataset and then gradually learns from the self-generated trajectories.
To achieve this, we onlyuse the generated trajectories to train our model after training k epochs over the original trajectories(without generated data).
Though we could use more sophisticated scheduled sampling strategiessuch as [2], they may incorporate additional hyperparameters, resulting in complex parameter tuning.
It turns out that our simple proposed version has shown promising effectiveness in the experiments.
Experiments
In this section, we mainly (i) evaluate the performance of BooT and (ii) investigate the characteristicsof the generated trajectories of BooT. We first compare the evaluation results of our algorithm toother offline RL algorithms, and conduct a comparison among different bootstrapping and sequencegeneration schemes.
Then we study the characteristics of generated data with both qualitative andquantitative analysis to illustrate more insights of our method.
Finally, we introduce the ablationstudy on hyperparameters and analyze their influence on BooT. Reproducible codes are provided insupplementary materials.
5.1 Evaluation Settings
Benchmark and Compared Baselines.
We evaluate our BooT algorithm on the dataset of continuous control tasks from the D4RL offline dataset [9].
We first perform experiments on Gym domain,which includes three environments (halfcheetah, hopper, walker2d) with three levels (medium,medium-replay, medium-expert) for each environment.
We compare BooT to the following variousbaselines: (1) behavior cloning ( BC ), a pure imitation learning method; (2) model-based offline planning ( MBOP [1]), a current strong baseline with model-based offline RL algorithm; (3) conservative
Q-learning ( CQL [23]), a current strong baseline of model-free offline RL algorithm; (4) Decision
Transformer ( DT Original [3]) with its variant using GPT2 pre-trained model ( DT GPT2 , [39]),and (5) Trajectory Transformer ( TT [32]), two recent works applying Transformer to offline RL;and multiple variants derived from our proposed method BooT for ablation studies.
In addition, weperform experiments on Adroit domain [38] in D4RL dataset, including four environments (pen,hammer, door, relocate) with also three levels (human,
[Table]
We report the normalized scores according to the protocol in [9], where a normalized score of 0corresponds to the average return of random policies and 100 corresponds to that of a domain-specificexpert.
The performance of BC is taken from [3], and the performance of CQL is taken from [22].
The results of MBOP, DT Original, DT GPT2, and TT are reported from their original papers.
Besides, we reproduce the TT algorithm, namely TT (Reproduce) .
We also perform an additionalexperiment with a slightly modified setting as TT (Re-train) : after the k th original training epoch, foreach training batch, we randomly choose ⌊ η % · K ⌋ original trajectories and train the model for thesecond time.
This setting is to ensure the numbers of training steps are the same for TT and BooT.
To compare with other offline data augmentation methods, we perform two additional experimentsusing TT with S4RL [43] techniques, which adds random noise to the input trajectories as augmenteddata.
Specifically, TT+S4RL (All) adds random noise to all states in a trajectory, while TT+S4RL(Last) adds random noise to the last T ′ timesteps of a trajectory.
The former setting resembles addingrandom noise to raw states in S4RL, while the latter is to compare with BooT where generatedtrajectories only differ from the original trajectories at the last T ′ timestep.
For a fair comparison, we adopt the same training hyperparameters with TT in all the above experiments.
We also use the same model structure and model size as TT for all BooT experiments.
Finally, we compare the performances of different bootstrapping schemes for BooT, using BooT-o and
BooT-r, with autoregressive and teacher-forcing trajectories, respectively.
All results in this sectioncorrespond to 15 random seeds, including 5 training seeds for independently trained Transformermodels and 3 evaluation seeds for each model, if not explicitly mentioned.
5.2 Detailed Results
In this section, we first present the overall result of BooT compared with other baselines in Table 1,and then show the detailed results of BooT with different bootstrapping schemes in Table 2.
As isshown in Table 1, BooT achieves improvements from TT and performs on par or better than otheroffline RL algorithms.
Compared to our reproduced results of TT, BooT obtains improvements by16 .
8% in average score, and achieves better performance on all 9 datasets.
[Table] [Table]
Comparison among TT (Reproduce),
TT (Re-train), and BooT suggest theimprovement in the performance of
BooT does not result from the extratraining steps of the algorithm.
Furthermore, we observe that directlyadding Gaussian noise to the trainingdata does not help improve the performance effectively as BooT
fromthe results of TT+S4RL experiments.
We argue that data generated by themodel itself are more consistent withthe underlying MDP than by addingnoise; thus, BooT obtains larger performance improvements, which alsoechos to our motivation.
Experiment results in Table 3 indicatethat BooT can also help improve theperformance on more complicated tasks in Adroit domain, where BooT achieves better performancethan TT baseline with an average improvement of 17 . 9% .
Results show that teacher-forcing generationhas already obtained noticeable improvement compared to the baselines, and autoregressive generationconsumes too much time in Adroit domain as the number of dimensions of the trajectories is toolarge.
As a result, we only perform experiments with teacher-forcing generation on Adroit domain.
5.3 Further Analysis of Bootstrapping
[Figure] [Figure]
[Table] [Table]
To better understand how BooT helpsimprove the performance, we performnumerical analysis on the generatedtrajectories in Table 4.
We calculatethe distance from the generated trajectories of BooT to their correspondingoriginal offline trajectories, denotedas Dataset , and the distance from thegenerated trajectories to real trajectories from the environment in the evaluation phase, denoted as Environment , compared to those from TT+S4RL method.
As we areinvestigating generation results but not training schemes here, we calculate the distance using only
BooT-o algorithm, which is sufficient for analysis.
RMSE stands for the Root Mean Squared Errorvalue of the trajectory difference, and MMD stands for the Maximum Mean Discrepancy of twotrajectory sets.
More details can be found in Appendix C.1.
In Table 4, the results of Dataset show that BooT produces trajectories with smaller RMSE comparedto adding random noise, which demonstrates that generated trajectories generally lie in a neighborhoodof the corresponding original offline trajectories in Euclidean space.
Also, trajectories generatedby BooT have larger MMD compared with the baseline, showing that generated data are differentfrom original offline trajectories w.r.t.
the data distribution.
This indicates that BooT expands theoriginal offline data, and the visualization in Figure 2 also verifies this claim.
But is such expansion reasonable?
The answer is yes and is supported by Environment results, which show that the distancebetween the generated trajectories and real trajectories from the evaluation environment is relativelysmaller compared with the baseline, demonstrating that the generated trajectories are more consistentwith the real distribution of the underlying MDP in RL tasks.
Thus, BooT provides a more reasonableexpansion to the original offline data, and subsequently improves the performance of the model.
Furthermore, in Figure 2, we visualize the distribution of (i) transitions ( s , a , r, R ) in the originaldata, (ii) data from teacher-forcing generation, and (iii) data from autoregressive generation in BooT.
We excerpt the last T ′ transitions from these trajectories, reduce them to 2-dimension via t-SNE [46]altogether, and plot them separately.
We also provide more visualization in Appendix B.
It is clearly illustrated that a large portion of data generated by the teacher-forcing method overlapswith the support of the original dataset, while those generated autoregressively lie out of the originaldata distribution.
The overall results demonstrate that generated trajectories from BooT expand thedata coverage while still keeping consistency with the underlying MDP of the corresponding RL task,thus resulting in better performance compared to the baselines.
This also suggests why BooT doesnot require extra mechanisms such as conservatism [1] to work well.
5.4 Ablation on Hyperparameters
[Figure] [Figure]
When to start to perform bootstrapping?
We performed an ablation study on generation thresholdk , which determines the threshold to start performing bootstrapping.
For k = 0 .
2 , BooT will performbootstrapping after the model is trained without generating sequences for 20% of training epochs.
Asis shown in Figure 3, the performance of BooT will first increase and then decrease as k increases.
Itis reasonable that if bootstrapping starts too early, the model will generate sequences that differ fromthe training dataset, and accumulate more error in the following training procedure.
If bootstrappingstarts too late, the model will learn little from bootstrapping as the model may have converged.
Thus,an appropriate value of k could be selected.
How much new trajectories to generate in bootstrapping?
We conduct an ablation study ongeneration percentage η % , which is used to control the ratio of the generated trajectories to theoriginal offline trajectories.
As shown in Figure 4, the performance of BooT first increases and thendecreases as η % increases.
For a small η % value, the number of total training steps on generated datamight be too small to effectively improve the performance of the model.
However, a large η % valuemay involve too much generated data with a low confidence score.
As explained in Sec.
4.2, thiscould cause accumulating learning bias and results in a performance drop.
Conclusion and Future Work
In this paper, we consider the data coverage issues encountered in offline RL scenarios and propose abootstrapping algorithm with a sequence modeling paradigm to self-generate novel trajectory data andin turn feed them back to boost sequence model training.
We adopt two sequence generation schemes,and each of them generates trajectories with unique characteristics compared to the original offlinedata distribution.
Compared to the other strong baselines, the experiments over the public offline RLbenchmarks have illustrated the great effectiveness of our proposed method, and the analysis alsoprovides some insightful evidence about how it works.
One drawback of our method is the relatively larger training time consumption compared to othermethods because of online pseudo training data generation.
In the future, we plan to improve theefficiency by incorporating more advanced techniques such as non-autoregressive sequence generation[12, 35], to improve the training efficiency.
Checklist
For all authors...(
a
) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b)
Did you describe the limitations of your work
[Yes
We
described it in Sec. 6.
c) Did you discuss any
potential negative societal impacts of your work
[N/A
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] We provide
reproducible code for our work in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] They are specified in Appendix A.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Our experiment results are provided with mean and std.
values.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] They are included in Appendix
A.2.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] They are specifiedand cited in Sec. 5.
(b) Did you mention the license of the assets?
[Yes] They are specified in Appendix B.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We provide reproducible code for our work in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]