Bootstrapped Transformer for Offline Reinforcement Learning
Abstract
Offline reinforcement learning (RL) aims at learning policies from previously collected static trajectory data without interacting with the real environment.
Recent works provide a novel perspective by viewing offline RL as a generic sequence generation problem, adopting sequence models such as Transformer architecture to model distributions over trajectories, and repurposing beam search as a planning algorithm.
However, the training datasets utilized in general offline RL tasks are quite limited and often suffer from insufficient distribution coverage, which could be harmful to training sequence generation In this paper, we propose a novel algorithm named Bootstrapped Transformer, which incorporates the idea of bootstrapping and leverages the learned model to self-generate more offline data to further boost the sequence model training.
We conduct extensive experiments on two offline RL benchmarks and demonstrate that our model can largely remedy the existing offline RL training limitations and beat other strong baseline methods.
We also analyze the generated pseudo data and the revealed characteristics may shed some light on offline RL training.
The codes and supplementary materials are available at https://seqml.github.io/bootorl .
Introduction
Online reinforcement learning (RL) [42, 33, 8, 50] requires environments and interactions to train policies.
However, such a requirement obstructs RL from being applied to many real-world problems where simulation environments are unavailable or interactions are incredibly costly.
Offline RL tries to get rid of the requirement and learn policies from a previously collected offline dataset without interacting with the real environment.
This setting enables more practical RL training in the real world and thus attracts lots of attention [27].
Unlike conventional approaches that focus on learning value functions [41] or computing policy gradients [34] recent works [32, 3] view the offline RL problem from a novel perspective where it is considered a generic sequence generation problem.
Specifically these works utilize a mainstream sequence modeling neural network, i.e., Transformer [47], to model the joint distribution of the offline decision-making trajectory data, after discretizing each term vector within the transitions (states, rewards, actions), in a supervised learning (SL) manner.
In such a sequence modeling framework, a trajectory consisting of states, actions, and rewards is viewed as a stream of tokens, and the learned
Transformer can generate tokens in an autoregressive way and subsequently be utilized for execution in the real environments.
This framework unifies multiple components in offline RL in one big sequence model, such as estimation of behavior policy and modeling for predictive dynamics [32], thus achieving superior performance.
However, there exist two major data limitations in offline RL tasks, which might be harmful to training sequence generation models yet have not been well addressed in the previous works.
First, the offline data coverage is limited.
The training data in offline RL tasks could be collected by arbitrary policies like human experts or random policies, which cannot guarantee sufficient coverage of the whole state and action spaces [11, 36].
The second issue lies in the amount of training data.
Specifically, the largest dataset in the widely-used offline RL benchmark D4RL [9] contains only about 4,000 trajectories, which is quite limited especially considering those in natural language sequence modeling such as WikiText [31] which contains about 3,800,000 sentence sequences.
Such limited datasets may deteriorate the sequence generation model, since the common belief in the literature is that training sequence generation models from scratch often requires a large amount of data with rich information [24].
Therefore, handling limited datasets should be carefully considered in the sequence modeling methods in offline RL tasks.
Some previous works try to tackle the above data limitations via data augmentation to expand the data coverage, either through image augmentation [25, 52, 51] such as cropping, or by applying perturbations on the state inputs [26, 43].
Nevertheless, these works can neither be applicable for other types of observations except image-based input, nor ensure to be consistent with the underlying
Markov decision process (MDP).
Another way is to train an additional model learning environment dynamics [18, 20, 30, 53].
Yet it requires extra efforts, and the goals of model learning and policy optimization have not been aligned, which may derive suboptimal solutions [54].
Moreover, all these methods have not considered the advanced sequence modeling framework.
In this paper, we try to relieve these problems under the sequence modeling paradigm for offline RL.
Specifically, we consider using the learned model itself to generate data and remedy the limitations in training datasets .
We further propose a novel algorithm, Bootstrapped Transformer (BooT) , to implement the above idea.
Generally, BooT uses Transformer to generate data and then uses the generated sequence data of high confidence to further train the model for bootstrapping.
To make our study comprehensive, we investigate two generation schemes.
The first one performs autoregressive generation, and the other uses the teacher-forcing strategy [48].
We also test two bootstrapping schemes.
One trains the model further on the generated data and discards them once they have been used; the other uses the generated data repeatedly during the later training process.
In this way, the above issues in previous works can be addressed simultaneously.
Specifically, the sequence generation model unifies predictive dynamics models and policies [32]; thus, we do not need to train extra environment models for generating data, and such a generation-based method can also be applied to proprioceptive observations (e.g., positions & velocities information).
Moreover, since the sequence model learns the distribution of the dataset and models the sequence generation probability, the data generated by the learned model itself will be more consistent with the original offline data [29].
Thus, we expect higher consistency of the pseudo trajectory data generated by the model than those of adding random noises, which has been empirically verified in our experiment.
We further conduct comprehensive experiments on multiple offline RL benchmarks, and BooT achieves significant performance improvements.
For Gym domain in D4RL benchmark, BooT even reaches a much higher average score than one of the baselines with a pre-trained 80 times larger model [39], which strongly demonstrates the superiority of BooT. We also provide an in-depth analysis of data generated by BooT with both qualitative and quantitative results.
Visualization demonstrates that the generated data act as an expansion of the original training dataset.
Besides, we quantitatively measure the difference between the original and generated trajectories, and empirically conclude that the generated pseudo trajectories from BooT generally lie in a neighborhood of the corresponding original trajectories while keeping consistent with the underlying MDP in RL tasks, providing strong support for our motivation.
Related Work
Offline Reinforcement Learning.
Offline RL algorithms learn policies from a static dataset without interacting with the real environment [27].
Deploying off-policy RL algorithms directly into the offline setting suffers from the distributional shift problem, which could cause a significant performance drop, as revealed in [11].
To address this issue, model-free algorithms try to constrain the policy closed to the behavior policy [11, 10] or penalize the values of out-of-distribution state-action pairs [23, 21].
On the other side, model-based algorithms [53, 20] simulate the real environment and generate more data for policy training.
All these methods tend to require the policy to be pessimistic, while our algorithm does not involve this additional constraint.
Besides, our algorithm is built upon the sequence generation framework, which is different from the above methods.
Sequence Modeling.
Sequence modeling is a long-developed and popular area, and various methods have been proposed in natural language processing [16, 44], speech recognition [6], and information system [40].
Recently, Transformer [47] has illustrated powerful model performance and has been widely used in sequence modeling and generation tasks [5].
Some recent papers adapt Transformer into RL by viewing RL as a sequence modeling problem.
Decision Transformer (DT) [3] learns the distribution of trajectories, and only predicts actions by feeding target rewards and previous states, rewards, and actions.
Trajectory Transformer (TT) [32] further brings out the capability of the sequence model by predicting states, actions, and rewards and employing beam search.
We leverage the paradigm of TT and provide a generation-based data augmentation methodology to improve the data coverage and further improve the performance.
Recent works also try to use pre-trained Transformers on other domains and finetune on offline RL [39] or other decision-making [17, 28] tasks.
We focus on the data augmentation method, which is different and compatible with the pre-training techniques.
Data Augmentation (DA).
DA is a common approach to promoting data richness and enhancing modeling efficacy in supervised learning scenarios.
They adopt data augmentation either through pseudo labeling [49], input perturbation [4], or generative model [15] to produce additional data.
Although DA has not been widely used in RL tasks, recent works have shown the promising performance of DA in RL methods.
Some of them target the data inefficiency issue.
Specifically,
RAD [26] incorporates common perturbation-based approaches, while DrQ [51], DrQ-v2 [52], and
DrAC [37] also introduce regularization terms with respect to the augmented data in actor and critic.
Some other works [14, 7, 13] try to achieve high stability for online RL via data augmentation.
For offline RL, S4RL [43] provides various raw-state-based DA methods and takes similar regularization to DrQ. Different from the previous works, we augment the offline data distribution through sequence generation by the learned Transformer model to remedy the limitations in training datasets for better facilitating offline policy learning in this paper.
Preliminary 3.1 Online and Offline Reinforcement Learning
Generally, RL models the sequential decision making problem as a Markov Decision Process
M = ( S , A , T , r, γ ) , where S is the state space and A is the action space.
Given state s , s ′ ∈ S and action a ∈ A , T ( s ′ | s , a ): S ×
A × S → [0 , 1] is the transition probability and r ( s , a ): S ×
A → R defines the reward function.
γ ∈ (0 , 1] is the discount factor.
The policy π : S ×
A → [0 , 1] takes action a at state s with probability π ( a | s ) .
At time step t ∈ [1 , T ] , the accumulative discounted reward in the future, named reward-to-go, is R t =
(cid:80)
Tt ′
=
t γ t ′ − t r t ′ .
The goal of online RL is to find a policy π that maximizes J = E a t ∼ π ( ·| s t ) , s t +
∼T ( ·| s t , a t ) [ (cid:80) Tt =
1 γ t − 1 r t ( s t , a t )] by learning from the transitions ( s , a , r, s ′ ) through interacting with the environment in an online manner [45].
Offline RL, instead of interacting with the real environment, makes use of a fixed dataset D collected by behavior policy π b , to learn a policy π that maximizes J for subsequent execution in the real environment.
Here, π b can either be a single policy or a mixture of policies and is inaccessible.
We assume that the collected data is trajectory-wise, i.e., D = { τ i } Di =1 , where τ = { ( s i , a i , r i , s ′ i ) }
Ti =1 .
3.2 Offline Reinforcement Learning as Sequence Modeling
In this paper, we take the offline RL problem as a sequence modeling task, following the previous work [32].
We introduce the details of the sequence modeling paradigm in this subsection.
After that, we illustrate our data augmentation and bootstrapping strategy to remedy the limitations in training datasets and further improve the learning performance in Sec. 4.
Input :
D Parameter : Number of epochs E , bootstrap epoch threshold k , generation percentage η % and generation length T ′ .
Note : Either BooT-o or BooT-r is executed.
1: for i = 1 , . . .
, E do 2:
Initialize the set of generated trajectories G ∅ 3: for each batch of trajectories τ ⊂ D do 4: Train model with as Eq (4) 5: if i > k then # (perform bootstrapping) 6:
Generate K new trajectories G ′ = { τ ′ } K 1 from τ as Eq. (5) and Eq. Select ⌊ η % · K ⌋ trajectories with the highest confidence calculated as Eq.
G ′ ← argmax S ⊆G ′ , | S | = ⌊ η % · K ⌋ (cid:0)(cid:80) τ ′ ∈ S c τ ′ (cid:1) 8: ( -o) Train the model with τ ′ ∈ G ′ as Eq.
(4) 9: (BooT-r) G ← G ∪ G ′ 10: end if 11: end for 12: ( r) D ← D ∪ G 13: end for
First, we treat each input trajectory τ as a sequence and add reward-to-go R t = (cid:80)
Tt ′
=
t γ t ′ − t r t ′ after reward r t as auxiliary information at each timestep t , which acts as future heuristics for further execution planning,
[Equation]
Second, a discretization approach is adopted to transform the continuous states and actions into discrete token spaces, each of which contains N and M tokens, respectively.
This turns τ aux into a sequence of discrete tokens with a total length of ( N + M + 2) T as
[Equation]
Without causing confusion, we will keep using τ to indicate discretized augmented trajectory τ dis in the following.
Finally, we model τ with the Trajectory Transformer (TT)
[32] from the perspective of sequence modeling.
TT regards each trajectory as a sentence and is trained with the standard teacher-forcing procedure.
Let θ be the parameters of TT and P θ be its induced conditional probabilities.
Define
[Equation]
the training objective is to maximize
[Equation]
where τ <t is the trajectory from timestep 1 to ( t − 1) , s <
it is dimension 1 to ( i − 1) of state s t , and the similar for a <jt .
As for model inference, we adopt beam search as the planning algorithm to maximize cumulative discounted reward plus reward-to-go estimates, similar to [32].
Bootstrapped Transformer
In this section, we describe our Bootstrapped Transformer (BooT) algorithm, utilizing self-generated trajectories as auxiliary data to further train the model, which is the general idea of bootstrapping.
Bootstrapping for supervised learning, such as pseudo labeling [19] and self-training [49], has been
Truncated Original Trajectory 𝑠 𝑡 𝑠 𝑡𝑁 𝑎 𝑡 𝑎 𝑡𝑀 𝑟 𝑡 𝑅 𝑡 ⋯ ⋯ ⋯
Ƹ𝑠 𝑡ො𝑎
𝑡ො𝑎
𝑡Ƹ𝑟 𝑡 ෠𝑅 𝑡 Ƹ𝑠 𝑡+11 ⋯ ⋯ ⋯
Trajectory Transformer
Teacher-forcing Generation Ƹ𝑠 𝑡+1ො𝑎 𝑡+1ො𝑎 𝑡+1Ƹ𝑟 𝑡+1 ෠𝑅 𝑡+1 Ƹ𝑠 𝑡+21 ⋯ ⋯ ⋯
𝑠 𝑡 𝑠 𝑡𝑁 𝑎 𝑡+ 𝑎 𝑡+1𝑀 𝑟 𝑡+1 𝑅 𝑡+1 ⋯ ⋯ ⋯
Trajectory Transformer
Autoregressive Generation Ƹ𝑠 𝑡+ ො𝑎 𝑡+11 ො𝑎 𝑡+
Ƹ𝑟 𝑡+1 ෠𝑅 𝑡+1 Ƹ𝑠 𝑡+2⋯ ⋯ ⋯
Trajectory Transformer Ƹ𝑠 𝑡+11 ⋯ Ƹ𝑠 𝑡+1𝑁 ො𝑎 𝑡+11 ⋯ ො𝑎 𝑡+1𝑀 ⋯
Ƹ𝑟 𝑡+1 ෠𝑅 𝑡+
Concatenate
Truncated Original Trajectory
Predicted Tokens 𝑠 𝑡 𝑠 𝑡𝑁 𝑎 𝑡 𝑎 𝑡𝑀 𝑟 𝑡 𝑅 𝑡 ⋯ ⋯
Ƹ𝑠 𝑡+ Ƹ𝑠 𝑡+1𝑁 ො𝑎 𝑡+ ො𝑎 𝑡+1𝑀 ⋯ ⋯ ⋯
⋯
Ƹ𝑟 𝑡+1 ෠𝑅 𝑡+ widely used.
Apart from those using pseudo label generation, in the RL domain, we directly utilize the learned sequence model to generate novel trajectory data, which has been an expansion of the original offline data (as shown in Sec. 5.3), and we further augment the original data distribution using the generated data to better approximate the real distribution following the underlying MDP.
The self-generated trajectories are fed as additional training data to further boost sequence modeling and decision making.
Overall, we divide BooT into two main parts, (i) generating trajectories with the learned sequence model and (ii) using the generated trajectories to augment the original offline dataset for bootstrapping the sequence model learning.
The overall training algorithm of BooT has been described in Algorithm 1.
4.1 Trajectory Generation
In this section, we demonstrate the details of trajectory generation.
For an original trajectory τ ∈ D in the original dataset, we resample the last T ′ timesteps ( T ′ < T ) as
[Equation]
5) where τ ≤ T − T ′ represents the original trajectory truncated at timestep ( T − T ′ ) .
Then, we concatenate the generated token sequence to the truncated original trajectory and obtain a new trajectory
[Equation] [Equation]
where ◦ denotes concatenation operation.
We investigate two different trajectory generation schemes below, both of which are demonstrated in
Figure 1.
We briefly compare these two methods in both efficiency and generation quality here, and perform detailed comparison experiments in Sec.
Autoregressive generation is to produce each predicted token ˜ y n by sampling from the probability conditioned on the previously generated tokens as
[Equation]
Here we denote the n th generated token as y n and the sequence of all the previously generated tokens as ˜ y <n .
The autoregressive generation scheme requires recurrent predictions based on the previously generated tokens.
Teacher-forcing generation adopts a similar procedure as teacher-forcing training by conditioning on the original token ground truth from the previous timesteps
[Equation]
Here,
y <
n denotes tokens from the original trajectory with position previous to
˜
y n .
Teacherforcing generation only relies on ground truth tokens.
Due to error accumulation in the sequential generation process, autoregressive generation tends to produce trajectories that deviate more from the original data distribution than teacher-forcing generation.
As has been found in Sec.
5.3, autoregressive generation could expand the coverage of the training data more effectively and might derive better learning efficacy.
Compared to autoregressive generation, teacher-forcing requires only one forward pass and thus consumes less time.
However, since teacher-forcing generation is not able to expand dataset coverage as effectively as autoregressive generation, it might lead to slightly less improvement in performance.
4.2 Training on the Augmented Dataset
After generating new trajectories, we use these trajectories to train our sequence model in addition to the original trajectory data.
In BooT, we conduct two different bootstrapping schemes: Bootstrap-once, denoted as BooTo , and Bootstrap-repeat, denoted as BooT-r .
As shown in Algorithm 1, both methods will first train the sequence model over the original offline trajectories and then utilize it to generate novel trajectories.
After that, BooT-o will train the model again on the generated trajectories and discard them immediately once they have been used.
In contrast, BooT-r will append the generated trajectories
G into the dataset D , and the generated trajectories will be used in every epoch after they are appended to the dataset.
In order to prevent accumulating learning bias caused by training with inaccurately generated data, we choose a part of the trajectories with the highest confidence scores in each batch according to the generation percentage
η % .
Here, we define confidence as the average log probability of all the generated tokens as
[Equation]
9) where log P θ ( τ t | τ <t ) is defined in Eq.
Intuitively, since the confidence score of trajectory is defined by its prediction probability that models the sequence generation mechanism, by filtering the generated trajectories with their confidence score, we can keep only the generated trajectory data of a high quality to further help the sequence model training through feeding these quality data back for bootstrapping.
Moreover, to stabilize bootstrapped training, we leverage the idea of curriculum learning described in [2] but adopt a more straightforward method here.
The sequence model is first trained over the original offline dataset and then gradually learns from the self-generated trajectories.
To achieve this, we only use the generated trajectories to train our model after training k epochs over the original trajectories (without generated data).
Though we could use more sophisticated scheduled sampling strategies such as [2], they may incorporate additional hyperparameters, resulting in complex parameter tuning.
It turns out that our simple proposed version has shown promising effectiveness in the experiments.
Experiments
In this section, we mainly (i) evaluate the performance of BooT and (ii) investigate the characteristics of the generated trajectories of BooT. We first compare the evaluation results of our algorithm to other offline RL algorithms, and conduct a comparison among different bootstrapping and sequence generation schemes.
Then we study the characteristics of generated data with both qualitative and quantitative analysis to illustrate more insights of our method.
Finally, we introduce the ablation study on hyperparameters and analyze their influence on BooT.
[Table]
5.1 Evaluation Settings
Benchmark and Compared Baselines.
We evaluate our BooT algorithm on the dataset of continuous control tasks from the D4RL offline dataset [9].
We first perform experiments on Gym domain, which includes three environments (halfcheetah, hopper, walker2d) with three levels (medium, medium-replay, medium-expert) for each environment.
We compare BooT to the following various baselines: (1) behavior cloning ( BC ), a pure imitation learning method; (2) model-based offline planning ( MBOP [1]), a current strong baseline with model-based offline RL algorithm; (3) conservative
Q-learning ( CQL [23]), a current strong baseline of model-free offline RL algorithm; (4) Decision
Transformer ( DT Original [3]) with its variant using GPT2 pre-trained model ( DT GPT2 , [39]), and (5) Trajectory Transformer ( TT [32]), two recent works applying Transformer to offline RL; and multiple variants derived from our proposed method BooT for ablation studies.
In addition, we perform experiments on Adroit domain [38] in D4RL dataset, including four environments (pen, hammer, door, relocate) with also three levels (human, cloned, expert) for each environment to better compare our method with TT and CQL baselines.
We report the normalized scores according to the protocol in [9], where a normalized score ofcorresponds to the average return of random policies and 100 corresponds to that of a domain-specific expert.
The performance of BC is taken from [3], and the performance of CQL is taken from [22].
The results of MBOP, DT Original, DT GPT2, and TT are reported from their original papers.
Besides, we reproduce the TT algorithm, namely TT (Reproduce) .
We also perform an additional experiment with a slightly modified setting as TT (Re-train) : after the k th original training epoch, for each training batch, we randomly choose ⌊ η % · K ⌋ original trajectories and train the model for the second time.
This setting is to ensure the numbers of training steps are the same for TT and BooT.
To compare with other offline data augmentation methods, we perform two additional experiments using TT with S4RL [43] techniques, which adds random noise to the input trajectories as augmented data.
Specifically, TT+S4RL (All) adds random noise to all states in a trajectory, while TT+S4RL (Last) adds random noise to the last T ′ timesteps of a trajectory.
The former setting resembles adding random noise to raw states in S4RL, while the latter is to compare with BooT where generated trajectories only differ from the original trajectories at the last T ′ timestep.
For a fair comparison, we adopt the same training hyperparameters with TT in all the above experiments.
We also use the same model structure and model size as TT for all BooT experiments.
Finally, we compare the performances of different bootstrapping schemes for BooT, using BooT-o and
BooT-r, with autoregressive and teacher-forcing trajectories, respectively.
All results in this section correspond to 15 random seeds, including 5 training seeds for independently trained Transformer models and 3 evaluation seeds for each model, if not explicitly mentioned.
5.2 Detailed Results
In this section, we first present the overall result of BooT compared with other baselines in Table 1, and then show the detailed results of BooT with different bootstrapping schemes in Table 2.
As is shown in Table 1, BooT achieves improvements from TT and performs on par or better than other
[Table]
offline RL algorithms.
Compared to our reproduced results of TT, BooT obtains improvements by 16 .
8% in average score
[Table]
and achieves better performance on all 9 datasets
[Table] [Table]
Comparison among TT (Reproduce),
TT (Re-train)
[Table]
and BooT suggest the improvement in the performance of
BooT does not result from the extra training steps of the algorithm.
Furthermore, we observe that directly adding Gaussian noise to the training data does not help improve the performance effectively as BooT from the results of TT+S4RL experiments.
We argue that data generated by the model itself are more consistent with the underlying MDP than by adding noise; thus, BooT obtains larger performance improvements, which also echos to our motivation.
Experiment results in Table 3 indicate that BooT can also help improve the performance on more complicated tasks in Adroit domain, where BooT achieves better performance than TT baseline with an average improvement of 17 . 9% .
Results show that teacher-forcing generation has already obtained noticeable improvement compared to the baselines
[Figure]
and autoregressive generation consumes too much time in Adroit domain as the number of dimensions of the trajectories is too large.
As a result, we only perform experiments with teacher-forcing generation on Adroit domain
[Figure]
5.3 Further Analysis of Bootstrapping
[Figure]
[Table]
To better understand how BooT helps improve the performance, we perform numerical analysis on the generated trajectories in Table 4.
We calculate the distance from the generated trajectories of BooT to their corresponding original offline trajectories, denoted as Dataset , and the distance from the generated trajectories to real trajectories from the environment in the evaluation phase, denoted as Environment , compared to those from TT+S4RL method.
As we are investigating generation results but not training schemes here, we calculate the distance using only
BooT-o algorithm, which is sufficient for analysis.
RMSE stands for the Root Mean Squared Error value of the trajectory difference, and MMD stands for the Maximum Mean Discrepancy of two trajectory sets.
More details can be found in Appendix D.1.
In Table 4, the results of Dataset show that BooT produces trajectories with smaller RMSE compared to adding random noise, which demonstrates that generated trajectories generally lie in a neighborhood of the corresponding original offline trajectories in Euclidean space.
Also, trajectories generated by BooT have larger MMD compared with the baseline, showing that generated data are different from original offline trajectories w.r.t.
the data distribution.
This indicates that BooT expands the original offline data, and the visualization in Figure 2 also verifies this claim.
But is such expansion reasonable?
The answer is yes and is supported by Environment results, which show that the distance between the generated trajectories and real trajectories from the evaluation environment is relatively smaller compared with the baseline, demonstrating that the generated trajectories are more consistent with the real distribution of the underlying MDP in RL tasks.
Thus, BooT provides a more reasonable expansion to the original offline data, and subsequently improves the performance of the model.
Furthermore, in Figure 2, we visualize the distribution of (i) transitions ( s , a , r, R ) in the original data, (ii) data from teacher-forcing generation, and (iii) data from autoregressive generation in BooT.
We excerpt the last T ′ transitions from these trajectories, reduce them to 2-dimension via t-SNE [46] altogether, and plot them separately.
We also provide more visualization in Appendix D.2.
It is clearly illustrated that a large portion of data generated by the teacher-forcing method overlaps with the support of the original dataset, while those generated autoregressively lie out of the original data distribution.
The overall results demonstrate that generated trajectories from BooT expand the data coverage while still keeping consistency with the underlying MDP of the corresponding RL task, thus resulting in better performance compared to the baselines.
This also suggests why BooT does not require extra mechanisms such as conservatism [1] to work well.
5.4 Ablation on Hyperparameters
[Figure] [Figure]
When to start to perform bootstrapping?
We performed an ablation study on generation threshold k , which determines the threshold to start performing bootstrapping.
For k = 0 .
2 , BooT will perform bootstrapping after the model is trained without generating sequences for 20% of training epochs.
As is shown in Figure 3, the performance of BooT will first increase and then decrease as k increases.
It is reasonable that if bootstrapping starts too early, the model will generate sequences that differ from the training dataset, and accumulate more error in the following training procedure.
If bootstrapping starts too late, the model will learn little from bootstrapping as the model may have converged.
Thus, an appropriate value of k could be selected.
How many new trajectories to generate in bootstrapping?
We conduct an ablation study on generation percentage η % , which is used to control the ratio of the generated trajectories to the original offline trajectories.
As shown in Figure 4, the performance of BooT first increases and then decreases as η % increases.
For a small η % value, the number of total training steps on generated data might be too small to effectively improve the performance of the model.
However, a large η % value may involve too much generated data with a low confidence score.
As explained in Sec.
4.2, this could cause accumulating learning bias and results in a performance drop.
How long should new trajectories be generated?
We perform an additional ablation study on generation length T ′ , which is used to control the length of generated sequences.
As shown in Figure 5, the performance of BooT with TF generation generally decreases as T ′ increases, and most results are better than the baselines.
Furthermore, the performance of BooT-r is relatively robust to the changes of T ′ .
As the number of tokens per timestep differs among different environments, it is difficult to determine the most suitable T ′ for all environments.
Besides, T ′ =
1 performs the best in experiments, so we think it is sufficient to simply set T ′
= 1 for most cases.
[Figure] [Figure]
5.5 Experiments of TT Retrained with Loss-weighted Data
We implement an additional TT baseline with loss-weighted retraining, to experiment whether the generated data are better for bootstrapping.
In the experiment, we add a simple loss-weighted re-train component for TT-retrain, choosing the data with the lowest / largest η % loss in each batch to re-train the model, to better compare with TT and TT-retrain baselines.
The results are shown in Figure 6.
For supplementary, the numerical results are provided in Appendix C.4.
The results have illustrated that using a loss-weighted re-train does help to improve the performance of the model, but BooT still performs much better than both schemes of loss-weighted re-train.
Compared to simple random re-training scheme, re-training the model with data corresponding to the lowest loss helps improve the performance of the model, while using data corresponding to the largest loss has less improvement or even possibly degrades the performance.
Conclusion and Future Work
In this paper, we consider the data coverage issues encountered in offline RL scenarios and propose a bootstrapping algorithm with a sequence modeling paradigm to self-generate novel trajectory data and in turn feed them back to boost sequence model training.
We adopt two sequence generation schemes, and each of them generates trajectories with unique characteristics compared to the original offline data distribution.
Compared to the other strong baselines, the experiments over the public offline RL benchmarks have illustrated the great effectiveness of our proposed method, and the analysis also provides some insightful evidence about how it works.
One drawback of our method is the relatively larger training time consumption compared to other methods because of online pseudo training data generation.
In the future, we plan to improve the efficiency by incorporating more advanced techniques such as non-autoregressive sequence generation [12, 35], to improve the training efficiency.