Decision Transformer: Reinforcement Learning via Sequence Modeling
Abstract
We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem.
This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT.
In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling.
Unlike prior approaches to RL that ﬁt value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked
Transformer.
By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return.
Despite the simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free ofﬂine RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.
1. Introduction
Recent work has shown transformers (Vaswani et al., 2017) can model high-dimensional distributions of semantic concepts at scale, including effective zero-shot generalization in language (Brown et al., 2020) and out-of-distribution image generation (Ramesh et al., 2021).
Given the diversity of successful applications of such models, we seek to examine their application to sequential decision making problems formalized as reinforcement learning (RL).
In contrast to prior work using transformers as an architectural choice for components within traditional RL algorithms (Parisotto et al. , author(s). 2020; Zambaldi et al., 2018), we seek to study if generative trajectory modeling – i.e. modeling the joint distribution of the sequence of states, actions, and rewards – can serve as a replacement for conventional RL algorithms.
We consider the following shift in paradigm: instead of training a policy through conventional RL algorithms like temporal difference (TD) learning (Sutton & Barto, 2018), we will train transformer models on collected experience using a sequence modeling objective.
This will allow us to bypass the need for bootstrapping for long term credit assignment – thereby avoiding one of the “deadly triad” (Sutton & Barto, 2018) known to destabilize RL.
It also avoids the need for discounting future rewards, as typically done in TD learning, which can induce undesirable short-sighted behaviors.
Additionally, we can make use of existing transformer frameworks widely used in language and vision that are easy to scale, utilizing a large body of work studying stable training of transformer models.
In addition to their demonstrated ability to model long sequences, transformers have other advantages.
Transformers can perform credit assignment directly via self-attention, in contrast to Bellman backups which slowly propagate rewards and are prone to “distractor” signals (Hung et al., 2019).
This can enable transformers to still work effectively in the presence of sparse or distracting rewards.
Finally, empirical evidence suggest that a transformer modeling approach can model a wide distribution of behaviors, enabling better generalization and transfer (Ramesh et al., 2021).
We explore our hypothesis by considering ofﬂine RL, where we will task agents with learning policies from suboptimal data – producing maximally effective behavior from ﬁxed, limited experience.
This task is traditionally challenging due to error propagation and value overestimation (Levine et al., 2020).
However, it is a natural task when training with a sequence modeling objective.
By training an autoregressive model on sequences of states, actions, and returns, we reduce policy sampling to autoregressive generative modeling.
We can specify the expertise of the policy – which “skill” to query – by selecting the desired return tokens acting
as a prompt for
generation
R s a a s a a causal transformer emb.
+ pos.
enc.
linear decoder
[Figure]
[Figure] [Figure] Illustrative example. To get an intuition for our proposal, consider the task of ﬁnding the shortest path on a directed graph, which can be posed as an RL problem.
The reward is 0 when the agent is at the goal node and − 1 otherwise. We train a GPT (Radford et al., 2018) model to predict next token in a sequence of returns-to-go (sum of future rewards), states, and actions.
Training only on random walk data – with no expert demonstrations – we can generate optimal trajectories at test time by adding a prior to generate highest possible returns (see more details and empirical results in the Appendix) and subsequently generate the corresponding sequence of actions via conditioning.
Thus, by combining the tools of sequence modeling with hindsight return information, we achieve policy improvement without the need for dynamic programming.
Motivated by this observation, we propose Decision Transformer, where we use the GPT architecture to autoregressively model trajectories (shown in Figure 1).
We study whether sequence modeling can perform policy optimization by evaluating Decision Transformer on ofﬂine RL benchmarks in Atari (Bellemare et al., 2013), OpenAI
Gym (Brockman et al., 2016), and Key-to-Door (Mesnard et al., 2020) environments.
We show that – without using dynamic programming – Decision Transformer matches or exceeds the performance of state-of-the-art model-free ofﬂine
RL algorithms (Agarwal et al., 2020; Kumar et al., 2020).
Furthermore, in tasks where long-term credit assignment is required, Decision Transformer capably outperforms the
RL baselines.
With this work, we aim to bridge sequence modeling and transformers with RL, and hope that sequence modeling serves as a strong algorithmic paradigm for RL.
2. Preliminaries 2.1. Ofﬂine reinforcement learning
We consider learning in a Markov decision process (MDP) described by the tuple ( S , A , P , R ).
The MDP tuple consists of states s ∈ S , actions a ∈ A , transition dynamics P ( s (cid:48) | s, a ) , and a reward function r =
R ( s, a ) .
We use s t , a t , and r t =
R ( s t , a t ) to denote the state, action, and reward at timestep t , respectively.
A trajectory is made up of a sequence of states, actions, and rewards: τ = ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , . . .
, s T , a T , r T ) .
The return of a trajectory at timestep t , R t = (cid:80) T t (cid:48)
=
t r t (cid:48) , is the sum of future rewards from that timestep.
The goal in reinforcement learning is to learn a policy which maximizes the expected return
E (cid:104)(cid:80) Tt =1 r t (cid:105) in an MDP.
In ofﬂine reinforcement learning, instead of obtaining data via environment interactions, we only have access to some ﬁxed limited dataset consisting of trajectory rollouts of arbitrary policies.
This setting is harder as it removes the ability for agents to explore the environment and collect additional feedback.
2.2. Transformers
Transformers were proposed by Vaswani et al.
(2017) as an architecture to efﬁciently model sequential data.
These models consist of stacked self-attention layers with residual connections.
Each self-attention layer receives n embeddings { x i } ni =1 corresponding to unique input tokens, and outputs n embeddings { z i } ni =1 , preserving the input dimensions.
The i -th token is mapped via linear transformations to a key k i , query q i , and value v i .
The i -th output of the self-attention layer is given by weighting the values v j by the normalized dot product between the query q i and other keys k j :
[Equation]
As we shall see later, this allows the layer to assign “credit” by implicitly forming state-return associations via similarity of the query and key vectors (maximizing the dot product).
In this work, we use the GPT architecture (Radford et al., 2018), which modiﬁes the transformer architecture with a causal self-attention mask to enable autoregressive generation, replacing the summation/softmax over the n tokens with only the previous tokens in the sequence ( j ∈ [1 , i ] ).
We defer the other architecture details to the original papers.
3. Method
We present Decision Transformer, which models trajectories autoregressively with minimal modiﬁcation to the transformer architecture, as summarized in Figure 1.
Trajectory representation.
The key desiderata in our choice of trajectory representation are that it should enable transformers to learn meaningful patterns and we should be able to conditionally generate actions at test time.
It is nontrivial to model rewards since we would like the model to generate actions based on future desired returns, rather than past rewards.
As a result, instead of feeding the rewards directly, we feed the model with the returns-to-go (cid:98) R t =
(cid:80) Tt (cid:48)
=
t r t (cid:48) .
This leads to the following trajectory representation which is amenable to autoregressive training and generation:
[Equation]
At test time, we can specify the desired performance (e.g.
1 for success or 0 for failure), as well as the environment starting state, as the conditioning information to initiate generation.
After executing the generated action for the current state, we decrement the target return by the achieved reward and repeat until episode termination.
Architecture.
We feed the last K timesteps into Decision
Transformer, for a total of 3 K tokens (one for each modality: return-to-go, state, or action).
To obtain token embeddings, we learn a linear layer for each modality, which projects raw inputs to the embedding dimension, followed by layer normalization (Ba et al., 2016).
For environments with visual inputs, the state is fed into a convolutional encoder instead of a linear layer.
Additionally, an embedding for each timestep is learned and added to each token – note this is different than the standard positional embedding used by transformers, as one timestep corresponds to three tokens.
The tokens are then processed by a GPT (Radford et al., 2018) model, which predicts future action tokens via autoregressive modeling.
Training.
We are given a dataset of ofﬂine trajectories.
We sample minibatches of sequence length K from the dataset.
The prediction head corresponding to the input token s t is trained to predict a t – either with cross-entropy loss for discrete actions or mean-squared error for continuous actions – and the losses for each timestep are averaged.
We did not ﬁnd predicting the states or returns-to-go to improve performance, although it is easily permissible within our framework (as shown in Section 5.3) and would be an interesting study for future work.
4. Evaluations on Ofﬂine RL Benchmarks
In this section, we investigate the performance of Decision
Transformer relative to dedicated ofﬂine RL and imitation learning algorithms.
In particular, our primary points of comparison are model-free ofﬂine RL algorithms based on
TD-learning, since our Decision Transformer architecture is fundamentally model-free in nature as well.
Furthermore,
TD-learning is the dominant paradigm in RL for sample efﬁciency, and also features prominently as a sub-routine in many model-based RL algorithms (Sutton, 1990; Janner et al., 2019).
We also compare with behavior cloning and variants, since it also involves a likelihood based policy learning formulation similar to ours.
The exact algorithms depend on the environment but our motivations are as follows:
• TD learning : most of these methods use an action-space constraint or value pessimism and will be the most faithful comparison to Decision Transformer, representing standard RL methods.
A state-of-the-art model-free method is Conservative Q-Learning (CQL) (Kumar et al., 2020) which serves as our primary comparison.
In addition, we also compare against other prior model-free RL algorithms like BEAR (Kumar et al., 2019b) and
[Figure]
BRAC (Wu et al., 2019).
• Imitation learning : this regime similarly uses supervised losses for training, rather than Bellman backups.
We use behavior cloning here, and include a more detailed discussion in Section 5.
We evaluate on both discrete (Atari (Bellemare et al., 2013)) and continuous (OpenAI Gym (Brockman et al., 2016)) control tasks.
The former involves high-dimensional observation spaces and requires long-term credit assignment, while the latter requires ﬁne-grained continuous control, representing a diverse set of tasks.
Our main results are summarized in Figure 3, where we show averaged normalized performance for each domain.
4.1. Atari
The Atari benchmark (Bellemare et al., 2013) is challenging due to its high-dimensional visual inputs and difﬁculty of credit assignment arising from the delay between actions and resulting rewards.
We evaluate our method on 1% of all samples in the DQN-replay dataset as per Agarwal et al.
(2020), representing 500 thousand of the 50 million transitions observed by an online DQN agent (Mnih et al., 2015) during training; we report the mean and standard deviation of 3 seeds.
We normalize scores based on a professional gamer, following the protocol of Hafner et al.
(2020), where 100 represents the professional gamer score and 0 represents a random policy.
We compare to CQL (Kumar et al., 2020), REM (Agarwal et al., 2020), and QR-DQN (Dabney et al., 2018) on four
Atari tasks (Breakout, Qbert, Pong, and Seaquest) that are evaluated in Agarwal et al.
We use context lengths of K = 30 for Decision Transformer (except K =for Pong).
We also report the performance of behavior cloning (BC), which utilizes the same network architecture and hyperparameters as Decision Transformer but does not have return-to-go conditioning 2 .
For CQL, REM, and QR- DQN baselines, we report numbers directly from the CQL and REM papers.
We show results in Table 1.
Our method is competitive with CQL in 3 out of 4 games and outperforms or matches REM, QR-DQN, and BC on all 4 games.
4.2. OpenAI Gym
In this section, we consider the continuous control tasks from the D4RL benchmark (Fu et al., 2020).
We also consider a 2D reacher environment that is not part of the benchmark, and generate the datasets using a similar methodology to the D4RL benchmark.
Reacher is a goal-conditioned task and has sparse rewards, so it represents a different setting than the standard locomotion environments (HalfCheetah,
Hopper, and Walker).
The different dataset settings are described below.
Medium: 1 million timesteps generated by a “medium” policy that achieves approximately one-third the score of an expert policy.
Medium-Replay: the replay buffer of an agent trained to the performance of a medium policy (approximately 25k-400k timesteps in our environments).
Medium-Expert: 1 million timesteps generated by the medium policy concatenated with 1 million timesteps generated by an expert policy.
We compare to CQL (Kumar et al., 2020), BEAR (Kumar et al., 2019b), BRAC (Wu et al., 2019), and AWR (Peng et al., 2019).
CQL represents the state-of-the-art in modelfree ofﬂine RL, an instantiation of TD learning with value pessimism.
Score are normalized so that 100 represents an expert policy, as per Fu et al.
CQL numbers are reported from the original paper; BC numbers are run by us; and the other methods are reported from the D4RL paper.
Our results are shown in Table 2.
Decision Transformer achieves the highest scores in a majority of the tasks and is competitive with the state of the art in the remaining tasks.
[Table]
[Table]
5. Discussion 5.1. Does Decision Transformer perform behavior cloning on a subset of the data?
In this section, we seek to gain insight into whether Decision Transformer can be thought of as performing imitation learning on a subset of the data with a certain return.
To investigate this, we propose a new method, Percentile Behavior Cloning (%BC), where we run behavior cloning on only the top X % of timesteps in the dataset, ordered by episode returns.
The percentile X % interpolates between standard
BC ( X = 100% ) that trains on the entire dataset and only cloning the best observed trajectory ( X → 0% ), trading off between better generalization by training on more data with training a specialized model that focuses on a desirable subset of the data.
We show full results comparing %BC to Decision Transformer and CQL in Table 3, sweeping over X ∈ [10% , 25% , 40% , 100%] .
Note that the only way to choose the optimal subset for cloning is to evaluate using rollouts from the environment, so %BC is not a realistic approach; rather, it serves to provide insight into the behavior of Decision Transformer.
When data is plentiful – as in the D4RL regime – we ﬁnd %BC can match or beat other ofﬂine RL methods.
On most environments, Decision Transformer is competitive with the performance of the best %BC, indicating it can hone in on a particular subset after training on the entire dataset distribution.
In contrast, when we study low data regimes – such as Atari, where we use 1% of a replay buffer as the dataset – %BC is weak (shown in Table 4).
This suggests that in scenarios with relatively low amounts of data, Decision Transformer can outperform %BC by using all trajectories in the dataset to improve generalization, even if those trajectories are dissimilar from the return conditioning target.
Our results indicate that Decision Transformer can be more effective than simply performing imitation learning on a subset of the dataset.
On the tasks we considered, Decision Transformer either outperforms or is competitive to %BC, without the confound of having to select the optimal subset.
5.2. How well does Decision Transformer model the distribution of returns?
We evaluate the ability of Decision Transformer to understand return-to-go tokens by varying the desired target return over a wide range – evaluating the multi-task distribution modeling capability of transformers.
Figure 5 shows the average sampled return accumulated by the agent over the course of the evaluation episode for varying values of target return.
On every task, the desired target returns and the true observed returns are highly correlated.
On some tasks like
[Table]
[Table]
Pong, HalfCheetah and Walker, Decision Transformer generates trajectories that almost perfectly match the desired returns (as indicated by the overlap with the oracle line).
Furthermore, on some Atari tasks like Seaquest, we can prompt the Decision Transformer with higher returns than the maximum episode return available in the dataset, demonstrating that Decision Transformer is sometimes capable of extrapolation.
[Figure]
5.3. Does Decision Transformer perform effective long-term credit assignment?
To evaluate long-term credit assignment capabilities of our model, we consider a variant of the Key-to-Door environment proposed in Mesnard et al.
This is a grid-based environment with a sequence of three phases: (1) in the ﬁrst phase, the agent is placed in a room with a key; (2) then, the agent is placed in an empty room; (3) and ﬁnally, the agent is placed in a room with a door.
The agent receives a binary reward when reaching the door in the third phase, but only if it picked up the key in the ﬁrst phase.
We train on datasets of trajectories generated by applying random actions and report success rates in Table 5.
Methods that use highsight return information: our Decision Transformer model and %BC (trained only on successful episodes) are able to learn effective policies – producing near-optimal paths, despite only training on random walks.
TD learning (CQL) cannot effectively propagate Q-values over the long horizons involved and gets poor performance.
5.4. Can transformers be accurate critics in sparse reward settings?
In previous sections, we established that decision transformer can produce effective policies (actors).
We now evaluate whether transformer models can also be effective critics.
We modify Decision Transformer to output return tokens in addition to action tokens on the Key-to-Door environment.
We ﬁnd that the transformer continuously updates reward probability based on events during the episode, shown in
Figure 4 (Left).
Furthermore, we ﬁnd the transformer attends to critical events in the episode (picking up the key or reaching the door), shown in Figure 4 (Right), indicating formation of state-reward associations as discussed in
Raposo et al.
and enabling accurate value prediction.
5.5. Does Decision Transformer perform well in sparse reward settings?
A known weakness of TD learning algorithms is that they require densely populated rewards, which can be unrealistic and/or expensive.
In contrast, Decision Transformer can improve robustness in these settings since it makes minimal
[Figure]
[Table]
assumptions on the density of the reward.
To evaluate this, we consider a delayed return version of the D4RL benchmarks where the agent does not receive any rewards along the trajectory, and instead receives the cumulative reward of the trajectory in the ﬁnal timestep.
Our results for delayed returns are shown in Table 6.
Delayed returns minimally affect
Decision Transformer; and due to the nature of the training process, while imitation learning methods are reward agnostic.
While TD learning collapses, Decision Transformer and %BC perform well, indicating that Decision Transformer can be more robust to delayed rewards.
See Appendix D for additional discussions.
6. Related Work
Ofﬂine reinforcement learning.
To mitigate the impact of distribution shift in ofﬂine RL, prior algorithms either (a) constrain the policy action space (Fujimoto et al., 2019;
Kumar et al., 2019a; Siegel et al., 2020) or (b) incorporate value pessimism (Fujimoto et al., 2019; Kumar et al., 2020), or (c) incorporate pessimism into learned dynamics models (Kidambi et al., 2020; Yu et al., 2020).
Since we do not use Decision Transformers to explicitly learn the dynamics model, we primarily compare against model-free algorithms in our work; in particular, adding a dynamics model tends to improve the performance of model-free algorithms.
Another line of work explores learning wide behavior distribution from an ofﬂine dataset by learning a task-agnostic set of skills, either with likelihood-based approaches (Ajay et al., 2020; Campos et al., 2020; Pertsch et al., 2020; Singh et al., 2021) or by maximizing mutual information (Eysenbach et al., 2019; Lu et al., 2020; Sharma et al., 2020).
Our work is similar to the likelihood-based approaches, which do not use iterative Bellman updates – although we use a simpler sequence modeling objective instead of a variational method, and use rewards for conditional generation of behaviors. Supervised learning in reinforcement learning settings. Some prior methods for reinforcement learning bear more resemblance to static supervised learning, such as Q-learning (Watkins, 1989; Mnih et al., 2013), which still uses iterative backups, or likelihood-based methods such as behavior cloning, which do not (discussed in previous section). Recent work (Srivastava et al., 2019; Kumar et al., 2019c; ogm, 2019) studies “upside-down” reinforcement learning (UDRL), which are similar to our method in seeking to model behaviors with a supervised loss conditioned on the target return.
A key difference in our work is the shift of motivation to sequence modeling rather than supervised learning: while the practical methods differ primarily in the context length and architecture, sequence modeling enables behavior modeling even without access to the reward, in a similar style to language (Radford et al., 2018) or images (Chen et al., 2020), and is known to scale well (Brown et al., 2020).
The method proposed by Kumar et al. (2019c) is most similar to our method with K = 1 , which we ﬁnd sequence modeling/long contexts to outperform (see Section D).
Ghosh et al.
(2019) extends prior UDRL methods to use state goal conditioning, rather than rewards, and Paster et al. (2020) further use an LSTM with state goal conditioning for goal-conditoned online RL settings. Concurrent to our [Table]
work, Janner et al. (2021) propose Trajectory Transformer, which is similar to Decision Transformer but additionally uses state and return prediction, as well as discretization, which incorporates model-based components. We believe that their experiments, in addition to our results, highlight the potential for sequence modeling to be a generally applicable idea for reinforcement learning.
Credit assignment.
Many works have studied better credit assignment via state-association, learning an architecture which decomposes the reward function such that certain “important” states comprise most of the credit (Ferret et al., 2019;
Harutyunyan et al., 2019; Mesnard et al., 2020).
They use the learned reward function to change the reward of an actor-critic algorithm to help propagate signal over long horizons.
Similar to our long-term setting, some works have speciﬁcally shown such state-associative architectures can perform better in delayed reward settings (Arjona-Medina et al., 2018; Hung et al., 2019; Liu et al., 2019; Raposo et al., 2021).
In contrast, we allow these properties to naturally emerge in a transformer architecture, without having to explicitly learn a reward function or a critic.
Conditional language generation.
Various works have studied guided generation for images (Karras et al., 2019) and language (Ghazvininejad et al., 2017; Weng, 2021).
Several works (Ficler & Goldberg, 2017; Hu et al., 2017; Rajani et al., 2019; Yu et al., 2017; Ziegler et al., 2019; Keskar et al., 2019) have explored training or ﬁne-tuning of models for controllable text generation.
Class-conditional language models can also be used to learn disciminators to guide generation (Dathathri et al., 2019;
Ghazvininejad et al., 2017;
Holtzman et al., 2018; Krause et al., 2020).
However, these approaches mostly assume constant “classes”, while in reinforcement learning the reward signal is time-varying.
Furthermore, it is more natural to prompt the model desired target return and continuously decrease it by the observed rewards over time, since the transformer model and environment jointly generate the trajectory.
Attention and transformer models.
Transformers (Vaswani et al., 2017) have been applied successfully to many tasks in natural language processing (Devlin et al., 2018;
Radford et al., 2018) and computer vision (Carion et al., 2020; Dosovitskiy et al., 2020).
However, transformers are relatively unstudied in RL, mostly due to differing nature of the problem, such as higher variance in training.
Zambaldi et al.
(2018) showed that augmenting transformers with relational reasoning improve performance in combinatorial environments and Ritter et al.
(2020) showed iterative self-attention allowed for RL agents to better utilize episodic memories.
Parisotto et al.
(2020) discussed design decisions for more stable training of transformers in the high-variance
RL setting.
Unlike our work, these still use actor-critic algorithms for optimization, focusing on novelty in architecture.
Additionally, in imitation learning, some works have studied transformers as a replacement for LSTMs: Dasari & Gupta (2020) study one-shot imitation learning, and Abramson et al.
(2020) combine language and image modalities for text-conditioned behavior generation.
7. Conclusion
We proposed Decision Transformer, seeking to unify ideas in language/sequence modeling and reinforcement learning.
On standard ofﬂine RL benchmarks, we showed Decision
Transformer can match or outperform strong algorithms designed explicitly for ofﬂine RL with minimal modiﬁcations from standard language modeling architectures.
We hope this work inspires investigation into using large transformer models for RL.
We used a simple supervised loss that was effective in our experiments, but applications to large-scale datasets could beneﬁt from self-supervised pretraining tasks.
In addition, one could consider more sophisticated embeddings for returns, states, and actions – for instance, conditioning on return distributions to model stochastic settings instead of deterministic returns.
Transformer models can also be used to model the state evolution of trajectory, potentially serving as an alternative to modelbased RL, and we hope to explore this in future work.
Successful real-world deployment of AI systems requires strong generalization to unseen situations.
While we do not directly evaluate generalization ability, we utilize a method known to generalize well in domains such as language and vision. Given our promising results on ofﬂine RL benchmarks, we are excited about the potential of larger RL systems built upon the Decision Transformer framework. For real-world applications, it will also be important to analyze the errors transformers make in MDP settings and biases in the datasets used for training.
A. Experimental Details A.1. Atari
We build our
Decision
Transformer implementation for Atari games off of minGPT ( https: //github.com/karpathy/minGPT ), a publicly available re-implementation of GPT.
We use most of the default hyperparameters from their character-level
GPT example ( https://github.com/karpathy/ minGPT/blob/master/play_char.ipynb ).
We reduce the batch size (except in Pong), block size, number of layers, attention heads, and embedding dimension for faster training.
For processing the observations, we use the
DQN encoder from Mnih et al.
(2015) with an additional linear layer to project to the embedding dimension.
For return-to-go conditioning, we use either 1 × or 5 × the maximum return in the dataset, but more possibilities exist for principled return-to-go conditioning.
In Atari experiments, we use Tanh instead of LayerNorm (as described in
Section 3) after embedding each modality, but did this does not make a signiﬁcant difference in performance.
The full list of hyperparameters can be found in Table 7.
A.2. OpenAI Gym A.2.1. D ECISION T RANSFORMER
Our code is based on the Huggingface Transformers library (Wolf et al., 2020).
Our hyperparameters on all OpenAI
Gym tasks are shown below in Table 8.
Heuristically, we ﬁnd using larger models helps to model the distribution of returns, compared to standard RL model sizes (which learn one policy).
For reacher we use a smaller context length than the other environments, which we ﬁnd to be helpful as the environment is goal-conditioned and the episodes are shorter.
We choose return targets based on expert performance for each environment, except for HalfCheetah where we ﬁnd 50% performance to be better due to the datasets containing lower relative returns to the other environments.
Models were trained for 10 5 gradient steps using the AdamW optimizer (Loshchilov & Hutter, 2017) following PyTorch defaults.
A.2.2. B EHAVIOR C LONING
As brieﬂy mentioned in Section 4.2, we found previously reported behavior cloning baselines to be weak, and so run them ourselves using a similar setup as Decision Transformer.
We tried using a transformer architecture, but found using an MLP (as in previous work) to be stronger.
We train for 2 .
5 × 10 4 gradient steps; training more did not improve performance.
Other hyperparameters are shown in Table 9.
The percentile behavior cloning experiments use the same hyperparameters.
B. Graph Shortest Path
We give details of the illustrative example discussed in the introduction.
The task is to ﬁnd the shortest path on a ﬁxed directed graph, which can be formulated as an MDP where reward is 0 when the agent is at the goal node and − 1 otherwise.
The observation is the integer index of the graph node the agent is in.
The action is the integer index of the graph node to move to next.
The transition dynamics transport the agent to the action’s node index if there is an edge in the graph, while the agent remains at the past node otherwise.
The returns-to-go in this problem correspond to negative path lengths and maximizing them corresponds to generating shortest paths.
In this environment, we use the GPT model as described in Section 3 to generate both actions and return-to-go tokens.
This makes it possible for the model it generate its own (realizable) returns-to-go ˆ R .
Since we require a return prompt to generate actions and we do assume knowledge of the optimal path length upfront, we use a simple prior over returns that favors shorter paths: P prior ( ˆ R = k ) ∝ T + 1 − k , where T is the maximum trajectory length.
Then, it is combined with the return probabilities generated by the GPT model: P ( ˆ R t | s 0: t , a 0: t − 1 , ˆ R 0: t − 1 ) =
P GPT ( ˆ R t | s 0: t , a 0: t − 1 , ˆ R 0: t − 1 ) × P prior ( ˆ R t ) 10 .
Note that the prior and return-to-go predictions are entirely computable by the model, and thus avoids the need for any external or oracle information like the optimal path length.
Adjustment of generation by a prior has also been used for similar purposes in controllable text generation in prior work (Krause et al., 2020).
We train on a dataset of 1 , 000 graph random walk trajectories of T = 10 steps each with a random graph of 20 nodes and edge sparsity coefﬁcient of 0 .
We report the results in Figure 6, where we ﬁnd that transformer model is able to signiﬁcantly improve upon the number of steps required to reach the goal, closely matching performance of optimal paths.
There are two reasons for the favorable performance on this task.
In one case, the training dataset of random walk trajectories may contain a segment that directly corresponds to the desired shortest path, in which case it will be generated by the model.
In the second case, generated paths are entirely original and are not subsets of trajectories in the training dataset - they are generated from stitching sub-optimal segments.
We ﬁnd this case accounts for 15 .
8% of generated paths in the experiment.
While this is a simple example and uses a prior on generation that we do not use in other experiments for simplicity, it illustrates how hindsight return information can be used with generation priors to avoid the need for explicit dynamic programming.
[Table]
C. Atari Task Scores
Table 10 shows the normalized scores used for normalization used in Hafner et al.
Tables 11 and 12 show the raw scores corresponding to Tables 1 and 4, respectively.
For %BC scores, we use the same hyperparameters as Decision
Transformer for fair comparison.
For REM and QR-DQN, there is a slight discrepancy between Agarwal et al.
(2020) and Kumar et al.
; we report raw data provided to us by REM authors.
D. Additional Discussions D.1. Context Length Ablation
To assess the importance of access to previous states, actions, and returns, we ablate on the context length K .
This is interesting since it is generally considered that the previous state (i.e. K = 1 ) is enough for reinforcement learning algorithms when frame stacking is used, as we do.
Tableshows that performance of Decision Transformer is signiﬁcantly worse when K = 1 , indicating that past information is useful for Atari games.
One hypothesis is that when we are representing a distribution of policies – like with sequence modeling – the context allows the transformer to identify which policy generated the actions, enabling better learning and/or improving the training dynamics.
One key difference between Decision Transformer and prior ofﬂine RL algorithms is that we do not require policy regularization or conservatism to achieve good performance. Our conjecture is that TD-learning based algorithms learn an approximate value function and improve the policy by optimizing this value function. This act of optimizing a learned function can exacerbate and exploit any inaccuracies in the value function approximation, causing failures in policy improvement.
Since Decision Transformer does not require explicit optimization using learned functions as objectives, it avoids the need for regularization or conservatism. D.3. How can Decision Transformer beneﬁt online RL? Ofﬂine RL and the ability to model behaviors has the potential to enable sample-efﬁcient online RL for downstream tasks.
Works studying the transition from ofﬂine to online generally ﬁnd that likelihood-based approaches, like our sequence modeling objective, are more successful (Gao et al., 2018; Nair et al., 2020). As a result, although we studied ofﬂine RL in this work, we believe Decision Transformer can meaningfully improve online RL methods by serving as a strong model for behavior generation. For instance, Decision Transformer can serve as a powerful “memorization engine” and in conjunction with powerful exploration
[Table] [Table]
algorithms like Go-Explore (Ecoffet et al., 2019)
[Table]
has the potential to simultaneously model and generative a diverse set of behaviors
[Table]
# of steps to goal 0. p r o p o r t i o n o f p a t h s shortest path transformer random walk
[Table] [Table]
[Table] [Table]