Original: Multi-Game Decision Transformers
Revised: Multi-Game Decision Transformers

Original: Abstract
Revised: Abstract

Original: A longstanding goal of the ﬁeld of AI is a method for learning a highly capable, generalist agent from diverse experience.
Revised: A longstanding goal of the ﬁeld of AI is a method for learning a highly capable, generalist agent from diverse experience.

Original: In the subﬁelds of vision and language, this was largely achieved by scaling up transformer-based models and training them on large, diverse datasets.
Revised: In the subﬁelds of vision and language, this was largely achieved by scaling up transformer-based models and training them on large, diverse datasets.

Original: Motivated by this progress, we investigate whether the same strategy can be used to produce generalist reinforcement learning agents.
Revised: Motivated by this progress, we investigate whether the same strategy can be used to produce generalist reinforcement learning agents.

Original: Speciﬁcally, we show that a single transformer-based model – with a single set of weights – trained purely ofﬂine can play a suite of up to 46 Atari games simultaneously at close-to-human performance.
Revised: Speciﬁcally, we show that a single transformer-based model – with a single set of weights – trained purely ofﬂine can play a suite of up to 46 Atari games simultaneously at close-to-human performance.

Original: When trained and evaluated appropriately, we ﬁnd that the same trends observed in language and vision hold, including scaling of performance with model size and rapid adaptation to new games via ﬁne-tuning.
Revised: When trained and evaluated appropriately, we ﬁnd that the same trends observed in language and vision hold, including scaling of performance with model size and rapid adaptation to new games via ﬁne-tuning.

Original: We compare several approaches in this multi-game setting, such as online and ofﬂine RL methods and behavioral cloning, and ﬁnd that our Multi-Game Decision
Revised: We compare several approaches in this multi-game setting, such as online and ofﬂine RL methods and behavioral cloning, and ﬁnd that our Multi-Game Decision

Original: Transformer models offer the best scalability and performance.
Revised: Transformer models offer the best scalability and performance.

Original: We release the pre-trained models and code to encourage further research in this direction.
Revised: We release the pre-trained models and code to encourage further research in this direction.

Original:  Introduction
Revised:  Introduction

Original: Building large-scale generalist models that solve many tasks by training on massive task-agnostic datasets has emerged as a dominant approach in natural language processing [18, 12], computer vision [19, 6], and their intersection [61, 4].
Revised: Building large-scale generalist models that solve many tasks by training on massive task-agnostic datasets has emerged as a dominant approach in natural language processing [18, 12], computer vision [19, 6], and their intersection [61, 4].

Original: These models can adapt to new tasks (such as translation [63, 78]), make use of unrelated data (such as using high-resource language to improve translations of low-resource languages [17]) or even incorporate new modalities by projecting images into language space [46, 75].
Revised: These models can adapt to new tasks (such as translation [63, 78]), make use of unrelated data (such as using high-resource language to improve translations of low-resource languages [17]) or even incorporate new modalities by projecting images into language space [46, 75].

Original: The success of these methods largely derives from a combination of scalable model architectures [77], an abundance of unlabeled task-agnostic data and continuous improvements in high performance computing infrastructure.
Revised: The success of these methods largely derives from a combination of scalable model architectures [77], an abundance of unlabeled task-agnostic data and continuous improvements in high performance computing infrastructure.

Original: Crucially, scaling laws [38, 31] indicate that performance gains due to scale have not yet reached a saturation point.
Revised: Crucially, scaling laws [38, 31] indicate that performance gains due to scale have not yet reached a saturation point.

Original: In this work, we argue that a similar progression is possible in the ﬁeld of reinforcement learning, and take initial steps toward scalable methods that produce highly capable generalist agents.
Revised: In this work, we argue that a similar progression is possible in the ﬁeld of reinforcement learning, and take initial steps toward scalable methods that produce highly capable generalist agents.

Original: In contrast to vision and language domains, reinforcement learning has seen advocacy for the use of smaller models [16, 49, 8] and is usually either used to solve single tasks, or multiple tasks within the same environment.
Revised: In contrast to vision and language domains, reinforcement learning has seen advocacy for the use of smaller models [16, 49, 8] and is usually either used to solve single tasks, or multiple tasks within the same environment.

Original: Importantly, training across multiple environments – with very different dynamics, rewards, visuals, and agent embodiments – has been studied less signiﬁcantly.
Revised: Importantly, training across multiple environments – with very different dynamics, rewards, visuals, and agent embodiments – has been studied less signiﬁcantly.

Original: [Figure]
Revised: [Figure]

Original: Speciﬁcally, we investigate whether a single model – with a single set of parameters – can be trained to act in multiple environments from large amounts of expert and non-expert experience.
Revised: Speciﬁcally, we investigate whether a single model – with a single set of parameters – can be trained to act in multiple environments from large amounts of expert and non-expert experience.

Original: We consider training on a suite of 41 Atari games [9, 25] for their diversity, informally asking “Can models learn something universal from playing many video games?”.
Revised: We consider training on a suite of 41 Atari games [9, 25] for their diversity, informally asking “Can models learn something universal from playing many video games?”.

Original: To train this model, we use only the previously-collected trajectories from Agarwal et al.
Revised: To train this model, we use only the previously-collected trajectories from Agarwal et al.

Original: [1], but we evaluate our agent interactively.
Revised: [1], but we evaluate our agent interactively.

Original: We are not striving for mastery or efﬁciency that game-speciﬁc agents can offer, as we believe we are still in early stages of this research agenda.
Revised: We are not striving for mastery or efﬁciency that game-speciﬁc agents can offer, as we believe we are still in early stages of this research agenda.

Original: Rather, we investigate whether the same trends observed in language and vision hold for large-scale generalist reinforcement learning agents.
Revised: Rather, we investigate whether the same trends observed in language and vision hold for large-scale generalist reinforcement learning agents.

Original: We ﬁnd that we can train a single agent that achieves 126% of human-level performance simultaneously across all games after training on ofﬂine expert and non-expert datasets (see Figure 1).
Revised: We ﬁnd that we can train a single agent that achieves 126% of human-level performance simultaneously across all games after training on ofﬂine expert and non-expert datasets (see Figure 1).

Original: Furthermore, we see similar trends that mirror those observed in language and vision: rapid ﬁnetuning to never-before-seen games with very little data (Section 4.5), a scaling relationship between performance and model size (Section 4.4), and faster training progress for larger models (Appendix G).
Revised: Furthermore, we see similar trends that mirror those observed in language and vision: rapid ﬁnetuning to never-before-seen games with very little data (Section 4.5), a scaling relationship between performance and model size (Section 4.4), and faster training progress for larger models (Appendix G).

Original: Notably, not all existing approaches to multi-environment training work well.
Revised: Notably, not all existing approaches to multi-environment training work well.

Original: We investigate several approaches, including treating the problem as ofﬂine decision transformer-based sequence modeling [14, 35], online RL [53], ofﬂine temporal difference methods [42], contrastive representations [56], and behavior cloning [60].
Revised: We investigate several approaches, including treating the problem as ofﬂine decision transformer-based sequence modeling [14, 35], online RL [53], ofﬂine temporal difference methods [42], contrastive representations [56], and behavior cloning [60].

Original: We ﬁnd that decision transformer based models offer the best performance and scaling properties in the multi-environment regime.
Revised: We ﬁnd that decision transformer based models offer the best performance and scaling properties in the multi-environment regime.

Original: However, to permit training on both expert and non-expert trajectories, we ﬁnd it is necessary to use a guided generation technique from language modeling to generate expert-level actions, which is an important departure from standard decision transformers.
Revised: However, to permit training on both expert and non-expert trajectories, we ﬁnd it is necessary to use a guided generation technique from language modeling to generate expert-level actions, which is an important departure from standard decision transformers.

Original: Our contributions are threefold: First, we show that it is possible to train a single high-performing generalist agent to act across multiple environments from ofﬂine data alone.
Revised: Our contributions are threefold: First, we show that it is possible to train a single high-performing generalist agent to act across multiple environments from ofﬂine data alone.

Original: Second, we show that scaling trends observed in language and vision hold.
Revised: Second, we show that scaling trends observed in language and vision hold.

Original: And third, we compare multiple approaches for achieving this goal, ﬁnding that decision transformers combined with guided generation perform the best.
Revised: And third, we compare multiple approaches for achieving this goal, ﬁnding that decision transformers combined with guided generation perform the best.

Original: It is our hope this study can inspire further research in generalist agents.
Revised: It is our hope this study can inspire further research in generalist agents.

Original: To aid this, we make our pre-trained models and code publicly available.
Revised: To aid this, we make our pre-trained models and code publicly available.

Original: [Figure]
Revised: [Figure]

Original:  Related Work
Revised:  Related Work

Original: A generalist agent for solving a variety of environments has been a goal for artiﬁcial intelligence (AI) researchers since the inception of AI as a ﬁeld of study [50].
Revised: A generalist agent for solving a variety of environments has been a goal for artiﬁcial intelligence (AI) researchers since the inception of AI as a ﬁeld of study [50].

Original: This same reason motivated the introduction of the Atari suite (the Arcade Learning Environment, or ALE) as a testbed for learning algorithms [10]; in their own words, the ALE is for “empirically assessing agents designed for general competency.”
Revised: This same reason motivated the introduction of the Atari suite (the Arcade Learning Environment, or ALE) as a testbed for learning algorithms [10]; in their own words, the ALE is for “empirically assessing agents designed for general competency.”

Original: While the celebrated deep Q -learning [52] and actor critic [54] agents were among the ﬁrst to use a single algorithm for all games, they nevertheless required separate training and hyperparameters for each game agent.
Revised: While the celebrated deep Q -learning [52] and actor critic [54] agents were among the ﬁrst to use a single algorithm for all games, they nevertheless required separate training and hyperparameters for each game agent.

Original: Later works have demonstrated the ability to learn a single neural network agent on multiple Atari games simultaneously, either online [20] or via policy distillation [59, 67].
Revised: Later works have demonstrated the ability to learn a single neural network agent on multiple Atari games simultaneously, either online [20] or via policy distillation [59, 67].

Original: The aim of our work is similar – to learn a single agent for playing multiple
Revised: The aim of our work is similar – to learn a single agent for playing multiple

Original: Atari games – with a focus on ofﬂine learning.
Revised: Atari games – with a focus on ofﬂine learning.

Original: We demonstrate results with human-level competency on up to 46 games, which is unseen in the literature.
Revised: We demonstrate results with human-level competency on up to 46 games, which is unseen in the literature.

Original: A closely related setting is learning to solve multiple tasks within the same or similar environments.
Revised: A closely related setting is learning to solve multiple tasks within the same or similar environments.

Original: For example in the robotics ﬁeld, existing works propose to use language-conditioned tasks [48, 3, 34], while others posit goal-reaching as a way to learn general skills [51], among other proposals [37, 82].
Revised: For example in the robotics ﬁeld, existing works propose to use language-conditioned tasks [48, 3, 34], while others posit goal-reaching as a way to learn general skills [51], among other proposals [37, 82].

Original: In this work, we tackle the problem of learning to act in a large collection of environments with distinctively different dynamics, rewards, and agent embodiments.
Revised: In this work, we tackle the problem of learning to act in a large collection of environments with distinctively different dynamics, rewards, and agent embodiments.

Original: This complicated but important setting requires a different type of generalization that has been studied signiﬁcantly less.
Revised: This complicated but important setting requires a different type of generalization that has been studied signiﬁcantly less.

Original: A concurrent work [65] also aims to train a transformer-based generalist agent based on ofﬂine data including for the ALE.
Revised: A concurrent work [65] also aims to train a transformer-based generalist agent based on ofﬂine data including for the ALE.

Original: This work differs from ours in that the ofﬂine training data is exclusively near-optimal and it requires prompting by expert trajectories at inference time.
Revised: This work differs from ours in that the ofﬂine training data is exclusively near-optimal and it requires prompting by expert trajectories at inference time.

Original: In contrast, we extend decision transformers [14] from the Upside-Down RL family [71, 68] to learn from a diverse dataset (expert and non-expert data), predict returns, and pick optimality-conditioned returns.
Revised: In contrast, we extend decision transformers [14] from the Upside-Down RL family [71, 68] to learn from a diverse dataset (expert and non-expert data), predict returns, and pick optimality-conditioned returns.

Original: Furthermore, we provide comparisons against existing behavioral cloning, online and ofﬂine RL methods, and contrastive representations [80, 56].
Revised: Furthermore, we provide comparisons against existing behavioral cloning, online and ofﬂine RL methods, and contrastive representations [80, 56].

Original: Other works that also consider LLM-like sequence modeling for a variety of single control tasks include [66, 84, 35, 23, 57].
Revised: Other works that also consider LLM-like sequence modeling for a variety of single control tasks include [66, 84, 35, 23, 57].

Original:  Method
Revised:  Method

Original: We consider a decision-making agent that at every time t receives an observation of the world o t , chooses an action a t , and receives a scalar reward r t .
Revised: We consider a decision-making agent that at every time t receives an observation of the world o t , chooses an action a t , and receives a scalar reward r t .

Original: Our goal is to learn a single optimal policy distribution P ∗ θ ( a t | o ≤ t , a <t , r <t ) with parameters θ that maximizes the agent’s total future return
Revised: Our goal is to learn a single optimal policy distribution P ∗ θ ( a t | o ≤ t , a <t , r <t ) with parameters θ that maximizes the agent’s total future return

Original: R t = (cid:80)
Revised: R t = (cid:80)

Original: k>t r k on all the environments we consider.
Revised: k>t r k on all the environments we consider.

Original: 3.1 Reinforcement Learning as Sequence Modeling
Revised: 3.1 Reinforcement Learning as Sequence Modeling

Original: Following [14], we pose the problem of ofﬂine reinforcement learning as a sequence modeling problem where we model the probability of the next sequence token x i conditioned on all tokens prior to it: P θ ( x i | x <i ) , similar to contemporary decoder-only sequence models [12, 15, 62].
Revised: Following [14], we pose the problem of ofﬂine reinforcement learning as a sequence modeling problem where we model the probability of the next sequence token x i conditioned on all tokens prior to it: P θ ( x i | x <i ) , similar to contemporary decoder-only sequence models [12, 15, 62].

Original: The sequences we consider have the form:
Revised: The sequences we consider have the form:

Original: [Equation]
Revised: [Equation]

Original: where t represents a time-step, M is the number of image patches per observation (which we further discuss in Section 3.2), and ˆ R t is the agent’s target return for the rest of the sequence.
Revised: where t represents a time-step, M is the number of image patches per observation (which we further discuss in Section 3.2), and ˆ R t is the agent’s target return for the rest of the sequence.

Original: Such a sequence order respects the causal structure of the environment decision process.
Revised: Such a sequence order respects the causal structure of the environment decision process.

Original: Figure 3 presents an overview of our model architecture.
Revised: Figure 3 presents an overview of our model architecture.

Original: Returns, actions, and rewards are tokenized (See Section 3.2 for details), and we train the model to predict the next return, action, and reward discrete token in a sequence via standard cross-entropy loss.
Revised: Returns, actions, and rewards are tokenized (See Section 3.2 for details), and we train the model to predict the next return, action, and reward discrete token in a sequence via standard cross-entropy loss.

Original: The sequence we consider is different from Chen et al.
Revised: The sequence we consider is different from Chen et al.

Original: [14], which has (cid:104) ...
Revised: [14], which has (cid:104) ...

Original: , ˆ R t , o t , a t , ... (cid:105) .
Revised: , ˆ R t , o t , a t , ... (cid:105) .

Original: Our design allows predicting the return distribution and sampling from it, instead of relying on a user to manually select an expert-level return at inference time (See Section 3.4).
Revised: Our design allows predicting the return distribution and sampling from it, instead of relying on a user to manually select an expert-level return at inference time (See Section 3.4).

Original: Predicting future value and rewards have been shown to be useful objectives for learning better representations in artiﬁcial reinforcement learning agents [47, 69, 44] and important signals for representation learning in humans [5].
Revised: Predicting future value and rewards have been shown to be useful objectives for learning better representations in artiﬁcial reinforcement learning agents [47, 69, 44] and important signals for representation learning in humans [5].

Original: Thus, while we may not directly use all of the predicted quantities, the task of predicting them encourages structure and representation learning of our o a a causal transformer decoder t t t return action
Revised: Thus, while we may not directly use all of the predicted quantities, the task of predicting them encourages structure and representation learning of our o a a causal transformer decoder t t t return action

Original: R t
Revised: R t

Original: R ... t o M t . . .
Revised: R ... t o M t . . .

Original: r t r t observation k>t Σ r k . . .
Revised: r t r t observation k>t Σ r k . . .

Original: reward ...
Revised: reward ...

Original: + environments.
Revised: + environments.

Original: In this work, we do not attempt to predict future observations due to their non-discrete nature and the additional model capacity that would be required to generate images.
Revised: In this work, we do not attempt to predict future observations due to their non-discrete nature and the additional model capacity that would be required to generate images.

Original: However, building image-based forward prediction models of the environment has been shown to be a useful representation objective for RL [28, 27, 29].
Revised: However, building image-based forward prediction models of the environment has been shown to be a useful representation objective for RL [28, 27, 29].

Original: We leave it for future investigation.
Revised: We leave it for future investigation.

Original: 3.2 Tokenization
Revised: 3.2 Tokenization

Original: To generate returns, actions, and rewards via multinomial distributions similarly to language generation, we convert these quantities to discrete tokens.
Revised: To generate returns, actions, and rewards via multinomial distributions similarly to language generation, we convert these quantities to discrete tokens.

Original: Actions a are already discrete quantities in the environments we consider.
Revised: Actions a are already discrete quantities in the environments we consider.

Original: We convert scalar rewards to ternary quantities {− 1 , 0 , +1 } , and uniformly quantize returns into a discrete range shared by all our environments.
Revised: We convert scalar rewards to ternary quantities {− 1 , 0 , +1 } , and uniformly quantize returns into a discrete range shared by all our environments.

Original: Inspired by the simplicity and effectiveness of transformer architectures for processing images [19], we divide each observation image into a collection of M patches 3 (see Figure 3).
Revised: Inspired by the simplicity and effectiveness of transformer architectures for processing images [19], we divide each observation image into a collection of M patches 3 (see Figure 3).

Original: Each patch is additively combined with a trainable position encoding and linearly projected into the input token embedding space.
Revised: Each patch is additively combined with a trainable position encoding and linearly projected into the input token embedding space.

Original: We experimented with using image tokenizations coming from a convolutional network, but did not ﬁnd it to have a signiﬁcant beneﬁt and omitted it for simplicity.
Revised: We experimented with using image tokenizations coming from a convolutional network, but did not ﬁnd it to have a signiﬁcant beneﬁt and omitted it for simplicity.

Original: We chose our tokenization scheme with simplicity in mind, but many other schemes are possible.
Revised: We chose our tokenization scheme with simplicity in mind, but many other schemes are possible.

Original: While all our environments use a shared action space, varying action spaces when controlling different agent morphologies can still be tokenized using methods of [33, 43, 26].
Revised: While all our environments use a shared action space, varying action spaces when controlling different agent morphologies can still be tokenized using methods of [33, 43, 26].

Original: And while we used uniform quantization to discretize continuous quantities, more sophisticated methods such as VQ-VAE [76] can be used to learn more effective discretizations.
Revised: And while we used uniform quantization to discretize continuous quantities, more sophisticated methods such as VQ-VAE [76] can be used to learn more effective discretizations.

Original: 3.3 Training Dataset
Revised: 3.3 Training Dataset

Original: To train the model, we use an existing dataset of Atari trajectories (with quantized returns) introduced in [1].
Revised: To train the model, we use an existing dataset of Atari trajectories (with quantized returns) introduced in [1].

Original: The dataset contains trajectories collected from the training progress of a DQN agent [53].
Revised: The dataset contains trajectories collected from the training progress of a DQN agent [53].

Original: Following [25], we select 46 games where DQN signiﬁcantly outperforms a random agent.
Revised: Following [25], we select 46 games where DQN signiﬁcantly outperforms a random agent.

Original: 41 games are used for training and 5 games are held out for out-of-distribution generalization experiments.
Revised: 41 games are used for training and 5 games are held out for out-of-distribution generalization experiments.

Original: We chose 5 held-out games representing different categories including Alien and MsPacman (maze based), Pong (ball tracking), SpaceInvaders (shoot vertically), and StarGunner (shoot horizontally), to ensure out-of-distribution generalization can be evaluated on different types of games.
Revised: We chose 5 held-out games representing different categories including Alien and MsPacman (maze based), Pong (ball tracking), SpaceInvaders (shoot vertically), and StarGunner (shoot horizontally), to ensure out-of-distribution generalization can be evaluated on different types of games.

Original: For each of 41 games, we use data from 2 training runs, each containing roll-outs from 50 policy checkpoints, in turn each containing 1 million environment steps.
Revised: For each of 41 games, we use data from 2 training runs, each containing roll-outs from 50 policy checkpoints, in turn each containing 1 million environment steps.

Original: This totals 4.1 billion steps.
Revised: This totals 4.1 billion steps.

Original: Using the tokenization scheme in previous sections, the dataset contains almost 160 billion tokens.
Revised: Using the tokenization scheme in previous sections, the dataset contains almost 160 billion tokens.

Original: As the dataset contains agent behaviors at all stages of learning, it contains both expert and non-expert behaviors.
Revised: As the dataset contains agent behaviors at all stages of learning, it contains both expert and non-expert behaviors.

Original: We do not perform any special ﬁltering, curation, or balancing of the dataset.
Revised: We do not perform any special ﬁltering, curation, or balancing of the dataset.

Original: The motivation to train on such data instead of expert-only behaviors is twofold: Firstly, sub-optimal behaviors are more diverse than optimal behaviors and may still be useful for learning representations of the environment and consequences of poor decisions.
Revised: The motivation to train on such data instead of expert-only behaviors is twofold: Firstly, sub-optimal behaviors are more diverse than optimal behaviors and may still be useful for learning representations of the environment and consequences of poor decisions.

Original: Secondly, it may be difﬁcult to create a single binary criteria for optimality as it is typically a graded quantity.
Revised: Secondly, it may be difﬁcult to create a single binary criteria for optimality as it is typically a graded quantity.

Original: Thus, instead of assuming only task-relevant expert behaviors, we train our model on all available behaviors, yet generate expert behavior at inference time as described in the next section.
Revised: Thus, instead of assuming only task-relevant expert behaviors, we train our model on all available behaviors, yet generate expert behavior at inference time as described in the next section.

Original: 3.4 Expert Action Inference
Revised: 3.4 Expert Action Inference

Original: As described above, our training datasets contain a mix of expert and non-expert behaviors, thus directly generating actions from the model imitating the data is unlikely to consistently produce expert behavior (as we conﬁrm in Section 4.7).
Revised: As described above, our training datasets contain a mix of expert and non-expert behaviors, thus directly generating actions from the model imitating the data is unlikely to consistently produce expert behavior (as we conﬁrm in Section 4.7).

Original: Instead, we want to control action generation to consistently produce actions of highly-rewarding behavior.
Revised: Instead, we want to control action generation to consistently produce actions of highly-rewarding behavior.

Original: This mirrors the problem of discriminator-guided generation in language models, for which a variety of methods have been proposed [40, 79, 58].
Revised: This mirrors the problem of discriminator-guided generation in language models, for which a variety of methods have been proposed [40, 79, 58].

Original: We propose an inference-time method inspired by [40] and assume a binary classiﬁer P ( expert t | ... ) that identiﬁes whether or not the behavior is expert-level before taking an action at time t .
Revised: We propose an inference-time method inspired by [40] and assume a binary classiﬁer P ( expert t | ... ) that identiﬁes whether or not the behavior is expert-level before taking an action at time t .

Original: Following
Revised: Following

Original: Bayes’ rule, the distribution of expert-level returns at time t is then:
Revised: Bayes’ rule, the distribution of expert-level returns at time t is then:

Original: [Equation]
Revised: [Equation]

Original: where R low is the return lower bound and R high is the return upper bound.
Revised: where R low is the return lower bound and R high is the return upper bound.

Original: Similarly to [70, 73, 74, 39], we deﬁne a binary classiﬁer to be proportional to future return with inverse temperature κ 4 :
Revised: Similarly to [70, 73, 74, 39], we deﬁne a binary classiﬁer to be proportional to future return with inverse temperature κ 4 :

Original: [Equation]
Revised: [Equation]

Original: This results in a simple auto-regressive procedure where we ﬁrst sample high-but-plausible target returns R t according to log-probability log P θ ( R t | ... )
Revised: This results in a simple auto-regressive procedure where we ﬁrst sample high-but-plausible target returns R t according to log-probability log P θ ( R t | ... )

Original: + κ
Revised: + κ

Original: ( R t − R low ) /
Revised: ( R t − R low ) /

Original: ( R high − R low ) , and then sample actions according to P
Revised: ( R high − R low ) , and then sample actions according to P

Original: θ ( a t | R t , ... ) .
Revised: θ ( a t | R t , ... ) .

Original: See Figure 4 for an illustration of this procedure and
Revised: See Figure 4 for an illustration of this procedure and

Original: Appendix B.3 for implementation details.
Revised: Appendix B.3 for implementation details.

Original: It can be seen as a variation of return-conditioned policies [41, 71, 14] that automatically generates expert-level (but likely) returns at every timestep, instead of manually ﬁxing them for the duration of the episode.
Revised: It can be seen as a variation of return-conditioned policies [41, 71, 14] that automatically generates expert-level (but likely) returns at every timestep, instead of manually ﬁxing them for the duration of the episode.

Original: [Figure]
Revised: [Figure]

Original: Importantly, this formulation only affects the inference procedure of the model – training is entirely unaffected and can rely on standard next-token prediction frameworks and infrastructure.
Revised: Importantly, this formulation only affects the inference procedure of the model – training is entirely unaffected and can rely on standard next-token prediction frameworks and infrastructure.

Original: While we chose this formulation for its simplicity, controllable generation is an active area of study and we expect other more effective methods to be introduced in the future.
Revised: While we chose this formulation for its simplicity, controllable generation is an active area of study and we expect other more effective methods to be introduced in the future.

Original: As such, our contribution is to point out a connection between problems of controllable generation in language modeling and optimality conditioning in control.
Revised: As such, our contribution is to point out a connection between problems of controllable generation in language modeling and optimality conditioning in control.

Original:  Experiments
Revised:  Experiments

Original: We formulate our experiments to answer a number of questions that are addressed in following sections:
Revised: We formulate our experiments to answer a number of questions that are addressed in following sections:

Original: • How do different online and ofﬂine methods perform in the multi-game regime?
Revised: • How do different online and ofﬂine methods perform in the multi-game regime?

Original: • How do different methods scale with model size ?
Revised: • How do different methods scale with model size ?

Original: • How effective are different methods at transfer to novel games ?
Revised: • How effective are different methods at transfer to novel games ?

Original: • Does multi-game decision transformer improve upon training data?
Revised: • Does multi-game decision transformer improve upon training data?

Original: • Does expert action inference (Section 3.4) improve upon behavioral cloning?
Revised: • Does expert action inference (Section 3.4) improve upon behavioral cloning?

Original: • Does training on expert and non-expert data bring beneﬁts over expert-only training?
Revised: • Does training on expert and non-expert data bring beneﬁts over expert-only training?

Original: We also consider whether there are beneﬁts to speciﬁcally using the transformer architecture in
Revised: We also consider whether there are beneﬁts to speciﬁcally using the transformer architecture in

Original: Appendix D, and qualitatively explore the attention behavior of these models in Appendix H.
Revised: Appendix D, and qualitatively explore the attention behavior of these models in Appendix H.

Original: 4.1 Setup
Revised: 4.1 Setup

Original: Model Variants and Scaling.
Revised: Model Variants and Scaling.

Original: We base our decision transformer (DT) conﬁguration on GPT-2 [12] as summarized in Appendix B.1.
Revised: We base our decision transformer (DT) conﬁguration on GPT-2 [12] as summarized in Appendix B.1.

Original: We report results for DT-200M (a Multi-Game DT with 200M parameters) if not speciﬁed otherwise.
Revised: We report results for DT-200M (a Multi-Game DT with 200M parameters) if not speciﬁed otherwise.

Original: Other smaller variants are DT-40M and DT-10M. We set sequence length to 4 game frames for all experiments, which results in sequences of 156 tokens.
Revised: Other smaller variants are DT-40M and DT-10M. We set sequence length to 4 game frames for all experiments, which results in sequences of 156 tokens.

Original: Training and Fine-tuning.
Revised: Training and Fine-tuning.

Original: We train all Multi-Game DT models on TPUv4 hardware and the
Revised: We train all Multi-Game DT models on TPUv4 hardware and the

Original: Jaxline (Babuschkin et al.
Revised: Jaxline (Babuschkin et al.

Original: [7]) framework for 10M steps using the LAMB optimizer [81] with a 3 · 10 − 4 learning rate, 4000 steps linear warm-up, no weight decay, gradient clip 1.0, β 1 = 0 .
Revised: [7]) framework for 10M steps using the LAMB optimizer [81] with a 3 · 10 − 4 learning rate, 4000 steps linear warm-up, no weight decay, gradient clip 1.0, β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: 999 , and batch size 2048.
Revised: 999 , and batch size 2048.

Original: For ﬁne-tuning on novel games, we train for 100k steps with a 10 − 4 learning rate, 10 − 2 weight decay and batch size of 256 instead.
Revised: For ﬁne-tuning on novel games, we train for 100k steps with a 10 − 4 learning rate, 10 − 2 weight decay and batch size of 256 instead.

Original: Both regimes used image augmentations as described in Appendix B.5.
Revised: Both regimes used image augmentations as described in Appendix B.5.

Original: Metrics.
Revised: Metrics.

Original: We measure performance on individual Atari games by human normalized scores (HNS) [53], i.e. ( score − score random ) /
Revised: We measure performance on individual Atari games by human normalized scores (HNS) [53], i.e. ( score − score random ) /

Original: ( score human − score random ) , or DQN-normalized scores, i.e. normalizing by the best DQN scores seen in the training dataset instead of using human scores.
Revised: ( score human − score random ) , or DQN-normalized scores, i.e. normalizing by the best DQN scores seen in the training dataset instead of using human scores.

Original: To create an aggregate comparison metric across all games, we use inter-quartile mean (IQM) of humannormalized scores across all games, following evaluation best practices proposed in [2].
Revised: To create an aggregate comparison metric across all games, we use inter-quartile mean (IQM) of humannormalized scores across all games, following evaluation best practices proposed in [2].

Original: Due to the prohibitively long training times, we only evaluated one training seed.
Revised: Due to the prohibitively long training times, we only evaluated one training seed.

Original: We additionally report median aggregate metric in Appendix E.
Revised: We additionally report median aggregate metric in Appendix E.

Original: 4.2 Baseline Methods
Revised: 4.2 Baseline Methods

Original: BC Our Decision Transformer (Section 3.1) can be reduced to a transformer-based Behavioral
Revised: BC Our Decision Transformer (Section 3.1) can be reduced to a transformer-based Behavioral

Original: Cloning (BC) [60] agent by removing the target return condition and return token prediction.
Revised: Cloning (BC) [60] agent by removing the target return condition and return token prediction.

Original: Similar to what we do for Decision Transformer, we also learn BC models at different scales (10M, 40M, 200M parameters) while keeping other conﬁgurations unchanged.
Revised: Similar to what we do for Decision Transformer, we also learn BC models at different scales (10M, 40M, 200M parameters) while keeping other conﬁgurations unchanged.

Original: C51 DQN
Revised: C51 DQN

Original: As a point of comparison for online performance, we use the C51 algorithm [11] which is a variant of deep Q -learning (DQN) but with a categorical loss for minimizing the temporal difference (TD) errors.
Revised: As a point of comparison for online performance, we use the C51 algorithm [11] which is a variant of deep Q -learning (DQN) but with a categorical loss for minimizing the temporal difference (TD) errors.

Original: Following improvements suggested in Hessel et al.
Revised: Following improvements suggested in Hessel et al.

Original: [30] as well as our own empirical observations, we use multi-step learning with n = 4 .
Revised: [30] as well as our own empirical observations, we use multi-step learning with n = 4 .

Original: For the single-game experiments, we use the standard convolutional neural network (CNN) used in the implementation of C51 [13].
Revised: For the single-game experiments, we use the standard convolutional neural network (CNN) used in the implementation of C51 [13].

Original: For the multi-game experiments, we modify the C51 implementation based on a hyperparameter search to use an Impala neural network architecture [20] with three blocks using 64 , 128 , and 128 channels respectively with a batch size of 128 and update period of 256 .
Revised: For the multi-game experiments, we modify the C51 implementation based on a hyperparameter search to use an Impala neural network architecture [20] with three blocks using 64 , 128 , and 128 channels respectively with a batch size of 128 and update period of 256 .

Original: CQL For an ofﬂine TD-based learning algorithm we use conservative Q -learning (CQL) [42].
Revised: CQL For an ofﬂine TD-based learning algorithm we use conservative Q -learning (CQL) [42].

Original: Namely, we augment the categorical loss of C51 with a behavioral cloning loss minimizing − log π Q ( a | s ) , where ( s, a ) is a state-action pair sampled from the ofﬂine dataset and π Q ( ·| s )
Revised: Namely, we augment the categorical loss of C51 with a behavioral cloning loss minimizing − log π Q ( a | s ) , where ( s, a ) is a state-action pair sampled from the ofﬂine dataset and π Q ( ·| s )

Original: =
Revised: =

Original: softmax( Q ( s, · )) .
Revised: softmax( Q ( s, · )) .

Original: Following the recommendations in Kumar et al.
Revised: Following the recommendations in Kumar et al.

Original: [42] we weight the contribution of the BC loss by 1 when using 100% of the ofﬂine data (multi-game training) and 4 when using 1% (single-game ﬁnetuning).
Revised: [42] we weight the contribution of the BC loss by 1 when using 100% of the ofﬂine data (multi-game training) and 4 when using 1% (single-game ﬁnetuning).

Original: For scaling experiments, we vary the number of blocks and channels in each block of the Impala: the number of blocks and channels is one of (5 blocks , 128 channels)
Revised: For scaling experiments, we vary the number of blocks and channels in each block of the Impala: the number of blocks and channels is one of (5 blocks , 128 channels)

Original: ≈ 5M params , (10 blocks , 256 channels)
Revised: ≈ 5M params , (10 blocks , 256 channels)

Original: ≈ 30M params , (5 blocks , 512 channels)
Revised: ≈ 30M params , (5 blocks , 512 channels)

Original: ≈ 60M params , (10 blocks , 512 channels)
Revised: ≈ 60M params , (10 blocks , 512 channels)

Original: ≈ 120M params .
Revised: ≈ 120M params .

Original: CPC, BERT, and ACL
Revised: CPC, BERT, and ACL

Original: For rapid adaptation to new games via ﬁne-tuning, we consider representation learning baselines including contrastive predictive coding (CPC) [56], BERT pretraining [18], and attentive contrastive learning (ACL) [80].
Revised: For rapid adaptation to new games via ﬁne-tuning, we consider representation learning baselines including contrastive predictive coding (CPC) [56], BERT pretraining [18], and attentive contrastive learning (ACL) [80].

Original: All state representation networks are implemented as additional multi-layer perceptrons (MLPs) or transformer layers on top of the Impala CNN used in C51 and CQL baselines.
Revised: All state representation networks are implemented as additional multi-layer perceptrons (MLPs) or transformer layers on top of the Impala CNN used in C51 and CQL baselines.

Original: CPC uses two additional MLP layers with 512 units each interleaved with ReLU activation to represent φ ( s ) , which is optimized by maximizing φ ( s ) (cid:62)
Revised: CPC uses two additional MLP layers with 512 units each interleaved with ReLU activation to represent φ ( s ) , which is optimized by maximizing φ ( s ) (cid:62)

Original: Wφ ( s (cid:48) ) of true transitions ( s, s (cid:48) ) and minimizing φ ( s ) (cid:62)
Revised: Wφ ( s (cid:48) ) of true transitions ( s, s (cid:48) ) and minimizing φ ( s ) (cid:62)

Original: Wφ (˜ s ) where ˜ s is a state randomly sampled from the batch (including states from other games).
Revised: Wφ (˜ s ) where ˜ s is a state randomly sampled from the batch (including states from other games).

Original: For BERT pretraining, we use 2 self-attention layers withattention heads of 256 units each and feed-forward dimension 512 , and train φ ( s ) using BERT’s masked self-prediction loss on a trajectory of sequence length 16 .
Revised: For BERT pretraining, we use 2 self-attention layers withattention heads of 256 units each and feed-forward dimension 512 , and train φ ( s ) using BERT’s masked self-prediction loss on a trajectory of sequence length 16 .

Original: ACL shares the same model parametrization as BERT, with the inclusion of action prediction in the pretraining objective.
Revised: ACL shares the same model parametrization as BERT, with the inclusion of action prediction in the pretraining objective.

Original: 4.3 How do different online and ofﬂine methods perform in the multi-game regime?
Revised: 4.3 How do different online and ofﬂine methods perform in the multi-game regime?

Original: We compare different online and ofﬂine algorithms in the multi-game regime and their single-game counterparts in Figure 1.
Revised: We compare different online and ofﬂine algorithms in the multi-game regime and their single-game counterparts in Figure 1.

Original: We ﬁnd that single-game specialists are still most performant.
Revised: We ﬁnd that single-game specialists are still most performant.

Original: Among multigame generalist models, our Multi-Game Decision Transformer model comes closest to specialist performance.
Revised: Among multigame generalist models, our Multi-Game Decision Transformer model comes closest to specialist performance.

Original: Multi-game online RL with non-transformer models comes second, while we struggled to get good performance with ofﬂine non-transformer models.
Revised: Multi-game online RL with non-transformer models comes second, while we struggled to get good performance with ofﬂine non-transformer models.

Original: We note that our multi-game online
Revised: We note that our multi-game online

Original: C51 DQN median score of 68% (see Appendix E) which compares similarly to multi-game median
Revised: C51 DQN median score of 68% (see Appendix E) which compares similarly to multi-game median

Original: Impala score of 70%, which we calculated from results reported by [20] for our suite of games.
Revised: Impala score of 70%, which we calculated from results reported by [20] for our suite of games.

Original: We believe the apparent advantage of ofﬂine DT compared to online multi-game methods like C may be explained in part through classical differences between online and ofﬂine settings in RL [45].
Revised: We believe the apparent advantage of ofﬂine DT compared to online multi-game methods like C may be explained in part through classical differences between online and ofﬂine settings in RL [45].

Original: Online methods must balance exploration with the ability to learn and generalize from experience, which could be challenging in the multi-game setting, whereas ofﬂine DT only needs to learn to distill and generalize from the ﬁxed multi-game experience given to it (collected by specialist DQN agents [1]).
Revised: Online methods must balance exploration with the ability to learn and generalize from experience, which could be challenging in the multi-game setting, whereas ofﬂine DT only needs to learn to distill and generalize from the ﬁxed multi-game experience given to it (collected by specialist DQN agents [1]).

Original: Beyond the difference between online and ofﬂine, one could also argue that C51 suffers from more training instability than DT due to the use of a temporal difference (TD) loss, which we discuss in the next paragraph.
Revised: Beyond the difference between online and ofﬂine, one could also argue that C51 suffers from more training instability than DT due to the use of a temporal difference (TD) loss, which we discuss in the next paragraph.

Original: 4.4 How do different methods scale with model size?
Revised: 4.4 How do different methods scale with model size?

Original: In large language and vision models, lowest-achievable training loss typically decreases predictably with increasing model size.
Revised: In large language and vision models, lowest-achievable training loss typically decreases predictably with increasing model size.

Original: Kaplan et al.
Revised: Kaplan et al.

Original: [38] demonstrated an empirical scaling relationship between the capacity of a language model (NLP terminology for a next-token autoregressive generative model) and its performance (negative log likelihood on held-out data).
Revised: [38] demonstrated an empirical scaling relationship between the capacity of a language model (NLP terminology for a next-token autoregressive generative model) and its performance (negative log likelihood on held-out data).

Original: These trends were veriﬁed over many orders of magnitude of model size, ranging from few-million parameter models to hundreds of billion parameter models.
Revised: These trends were veriﬁed over many orders of magnitude of model size, ranging from few-million parameter models to hundreds of billion parameter models.

Original: [Figure]
Revised: [Figure]

Original: We investigate whether similar trends hold for interactive in-game performance – not just training loss – and show a similar performance scaling trend in Figure 5a.
Revised: We investigate whether similar trends hold for interactive in-game performance – not just training loss – and show a similar performance scaling trend in Figure 5a.

Original: Multi-Game Decision Transformer performance reliably increases over more than an order of magnitude of parameter scaling, whereas the other methods either saturate, or have much slower performance growth.
Revised: Multi-Game Decision Transformer performance reliably increases over more than an order of magnitude of parameter scaling, whereas the other methods either saturate, or have much slower performance growth.

Original: In contrast, in Figures 5a and 5b, we ﬁnd that CQL does not improve with increased model size, and actually shows a sharp drop in the performance of larger models on the ﬁne-tuning tasks.
Revised: In contrast, in Figures 5a and 5b, we ﬁnd that CQL does not improve with increased model size, and actually shows a sharp drop in the performance of larger models on the ﬁne-tuning tasks.

Original: Temporal
Revised: Temporal

Original: Difference (TD) methods suffer greater instability with larger model size in the multi-game setting, leading to this “inverse” scaling.
Revised: Difference (TD) methods suffer greater instability with larger model size in the multi-game setting, leading to this “inverse” scaling.

Original: Indeed, our attempts at other objectives closer to pure TD (C51,
Revised: Indeed, our attempts at other objectives closer to pure TD (C51,

Original: DQN, DDQN) led to even worse results (which we do not report).
Revised: DQN, DDQN) led to even worse results (which we do not report).

Original: We note that similar conclusions about instability with respect to network size have been made by other work [22].
Revised: We note that similar conclusions about instability with respect to network size have been made by other work [22].

Original: We also ﬁnd that larger models train faster, in the sense of reaching higher in-game performance after observing the same number of tokens.
Revised: We also ﬁnd that larger models train faster, in the sense of reaching higher in-game performance after observing the same number of tokens.

Original: We discuss these results in Appendix G.
Revised: We discuss these results in Appendix G.

Original: 4.5 How effective are different methods at transfer to novel games?
Revised: 4.5 How effective are different methods at transfer to novel games?

Original: Pretraining for rapid adaptation to new games has not been explored widely on Atari games despite being a natural and well-motivated task due to its relevance to how humans transfer knowledge to new games.
Revised: Pretraining for rapid adaptation to new games has not been explored widely on Atari games despite being a natural and well-motivated task due to its relevance to how humans transfer knowledge to new games.

Original: Nachum and Yang [55] employed pretraining on large ofﬂine data and ﬁne-tunining on small expert data for Atari and compared to a set of state representation learning objectives based on bisimulation [24, 83], but their pretraining and ﬁne-tuning use the same game.
Revised: Nachum and Yang [55] employed pretraining on large ofﬂine data and ﬁne-tunining on small expert data for Atari and compared to a set of state representation learning objectives based on bisimulation [24, 83], but their pretraining and ﬁne-tuning use the same game.

Original: We are instead interested in the transfer ability of pretrained agents to new games.
Revised: We are instead interested in the transfer ability of pretrained agents to new games.

Original: We hence devise our own evaluation setup by pretraining DT, CQL, CPC, BERT, and ACL on the full datasets of the 41 training games with 100M steps each, and ﬁne-tuning one model per held-out game using 1% (1M steps) from each game.
Revised: We hence devise our own evaluation setup by pretraining DT, CQL, CPC, BERT, and ACL on the full datasets of the 41 training games with 100M steps each, and ﬁne-tuning one model per held-out game using 1% (1M steps) from each game.

Original: The 1% ﬁne-tuning data is uniformly sampled from the 50M step dataset without quality ﬁltering.
Revised: The 1% ﬁne-tuning data is uniformly sampled from the 50M step dataset without quality ﬁltering.

Original: DT and CQL use the same objective for pretraining and ﬁne-tuning, whereas CPC, BERT, and ACL each use their own pretraining objective and are ﬁne-tuned using the BC objective.
Revised: DT and CQL use the same objective for pretraining and ﬁne-tuning, whereas CPC, BERT, and ACL each use their own pretraining objective and are ﬁne-tuned using the BC objective.

Original: All methods are ﬁne-tuned for 100,000 steps, which is much shorter than training any agent from scratch.
Revised: All methods are ﬁne-tuned for 100,000 steps, which is much shorter than training any agent from scratch.

Original: We additionally include training CQL from scratch on the 1% held-out data to highlight the beneﬁt of rapid ﬁne-tuning.
Revised: We additionally include training CQL from scratch on the 1% held-out data to highlight the beneﬁt of rapid ﬁne-tuning.

Original: Fine-tuning performance on the held-out games is shown in Figure 6.
Revised: Fine-tuning performance on the held-out games is shown in Figure 6.

Original: Pretraining with the DT objective performs the best across all games.
Revised: Pretraining with the DT objective performs the best across all games.

Original: All methods with pretraining outperform training CQL from scratch, which veriﬁes our hypothesis that pretraining on other games should indeed help with rapid learning of a new game.
Revised: All methods with pretraining outperform training CQL from scratch, which veriﬁes our hypothesis that pretraining on other games should indeed help with rapid learning of a new game.

Original: CPC and BERT underperform DT, suggesting that learning state representations alone is not sufﬁcient for desirable transfer performance.
Revised: CPC and BERT underperform DT, suggesting that learning state representations alone is not sufﬁcient for desirable transfer performance.

Original: While ACL adds an action prediction auxiliary loss to BERT, it showed little effect, suggesting that modeling the actions in the right way on the ofﬂine data is important for good transfer performance.
Revised: While ACL adds an action prediction auxiliary loss to BERT, it showed little effect, suggesting that modeling the actions in the right way on the ofﬂine data is important for good transfer performance.

Original: Furthermore, we ﬁnd that ﬁne-tuning performance improves as the DT model becomes larger, while CQL ﬁne-tuning performance is inconsistent with model size (see Figure 5b).
Revised: Furthermore, we ﬁnd that ﬁne-tuning performance improves as the DT model becomes larger, while CQL ﬁne-tuning performance is inconsistent with model size (see Figure 5b).

Original: [Figure]
Revised: [Figure]

Original: 4.6 Does multi-game decision transformer improve upon training data?
Revised: 4.6 Does multi-game decision transformer improve upon training data?

Original: We want to evaluate whether decision transformer with expert action inference is capable of acting better than the best demonstrations seen during training.
Revised: We want to evaluate whether decision transformer with expert action inference is capable of acting better than the best demonstrations seen during training.

Original: To do this, we look at the top 3 performing decision transformer model rollouts.
Revised: To do this, we look at the top 3 performing decision transformer model rollouts.

Original: We use top 3 rollouts instead of the mean across all rollouts to more fairly compare to the best demonstration, rather than an average expert demonstration.
Revised: We use top 3 rollouts instead of the mean across all rollouts to more fairly compare to the best demonstration, rather than an average expert demonstration.

Original: We show percentage improvement over best demonstration score for individual games in Figure 7.
Revised: We show percentage improvement over best demonstration score for individual games in Figure 7.

Original: We see signiﬁcant improvement over the training data in a number of games.
Revised: We see signiﬁcant improvement over the training data in a number of games.

Original: I
Revised: I

Original: [Figure]
Revised: [Figure]

Original: p r o v e m en t o v e r be s t sc
Revised: p r o v e m en t o v e r be s t sc

Original: o r e i n da t
Revised: o r e i n da t

Original: [Figure]
Revised: [Figure]

Original: s e t
Revised: s e t

Original: [Figure]
Revised: [Figure]

Original: ss au l t A s t
Revised: ss au l t A s t

Original: [Figure]
Revised: [Figure]

Original: r i
Revised: r i

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: tt
Revised: tt

Original: [Figure]
Revised: [Figure]

Original: one
Revised: one

Original: [Figure]
Revised: [Figure]

Original: ea
Revised: ea

Original: [Figure]
Revised: [Figure]

Original: de
Revised: de

Original: [Figure]
Revised: [Figure]

Original: ou
Revised: ou

Original: [Figure]
Revised: [Figure]

Original: r n i v
Revised: r n i v

Original: [Figure]
Revised: [Figure]

Original: l
Revised: l

Original: C
Revised: C

Original: [Figure]
Revised: [Figure]

Original: pede
Revised: pede

Original: [Figure]
Revised: [Figure]

Original: hoppe r C o
Revised: hoppe r C o

Original: C
Revised: C

Original: [Figure]
Revised: [Figure]

Original: zy
Revised: zy

Original: C
Revised: C

Original: [Figure]
Revised: [Figure]

Original: D e m
Revised: D e m

Original: [Figure]
Revised: [Figure]

Original: tt a
Revised: tt a

Original: [Figure]
Revised: [Figure]

Original: D oub l e D un
Revised: D oub l e D un

Original: [Figure]
Revised: [Figure]

Original: ndu
Revised: ndu

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: D
Revised: D

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: ee
Revised: ee

Original: [Figure]
Revised: [Figure]

Original: s t
Revised: s t

Original: [Figure]
Revised: [Figure]

Original: i t
Revised: i t

Original: [Figure]
Revised: [Figure]

Original: ophe
Revised: ophe

Original: [Figure]
Revised: [Figure]

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original: i t a
Revised: i t a

Original: [Figure]
Revised: [Figure]

Original: H
Revised: H

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: I c
Revised: I c

Original: [Figure]
Revised: [Figure]

Original: ck
Revised: ck

Original: [Figure]
Revised: [Figure]

Original: J
Revised: J

Original: [Figure]
Revised: [Figure]

Original: m e s bond
Revised: m e s bond

Original: K anga r oo K r
Revised: K anga r oo K r

Original: [Figure]
Revised: [Figure]

Original: ll
Revised: ll

Original: [Figure]
Revised: [Figure]

Original: ung F u
Revised: ung F u

Original: M
Revised: M

Original: [Figure]
Revised: [Figure]

Original: h
Revised: h

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: G
Revised: G

Original: P hoen
Revised: P hoen

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: P oo
Revised: P oo

Original: [Figure]
Revised: [Figure]

Original: Q be r t
Revised: Q be r t

Original: R i v e rr a i d
Revised: R i v e rr a i d

Original: [Figure]
Revised: [Figure]

Original: oad
Revised: oad

Original: [Figure]
Revised: [Figure]

Original: unne
Revised: unne

Original: [Figure]
Revised: [Figure]

Original: obo
Revised: obo

Original: [Figure]
Revised: [Figure]

Original: an k
Revised: an k

Original: S eaque s t
Revised: S eaque s t

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: e P
Revised: e P

Original: [Figure]
Revised: [Figure]

Original: o t
Revised: o t

Original: U p ND
Revised: U p ND

Original: [Figure]
Revised: [Figure]

Original: deo
Revised: deo

Original: [Figure]
Revised: [Figure]

Original: 4.7 Does optimal action inference improve upon behavior cloning?
Revised: 4.7 Does optimal action inference improve upon behavior cloning?

Original: [Figure]
Revised: [Figure]

Original: m i d a r A ss a u l t A s t e r i x
Revised: m i d a r A ss a u l t A s t e r i x

Original: A t l a n t i s
Revised: A t l a n t i s

Original: B a n k H
Revised: B a n k H

Original: e
Revised: e

Original: i s t
Revised: i s t

Original: B a tt l e Z o n e
Revised: B a tt l e Z o n e

Original: B e a m
Revised: B e a m

Original: R i d e r B o x
Revised: R i d e r B o x

Original: i n g
Revised: i n g

Original: B r e a k o u t
Revised: B r e a k o u t

Original: C a r n i v a l
Revised: C a r n i v a l

Original: C e n t
Revised: C e n t

Original: i p e d e
Revised: i p e d e

Original: C h
Revised: C h

Original: o
Revised: o

Original: pp e r C o mm a n d
Revised: pp e r C o mm a n d

Original: C r a z y C li m b e r
Revised: C r a z y C li m b e r

Original: D e m o n
Revised: D e m o n

Original: A tt a c k
Revised: A tt a c k

Original: D o u b l e
Revised: D o u b l e

Original: D un k
Revised: D un k

Original: E n d u r o
Revised: E n d u r o

Original: F i s h
Revised: F i s h

Original: i n g
Revised: i n g

Original: D e r b y
Revised: D e r b y

Original: F r ee w a y
Revised: F r ee w a y

Original: F r o s t b i t e
Revised: F r o s t b i t e

Original: G o p h
Revised: G o p h

Original: e r
Revised: e r

Original: G r a v i t a r H e r o
Revised: G r a v i t a r H e r o

Original: I c e H o c k e y
Revised: I c e H o c k e y

Original: J a m e s b o n d
Revised: J a m e s b o n d

Original: K a n g a r oo K r u ll
Revised: K a n g a r oo K r u ll

Original: K un g F u
Revised: K un g F u

Original: M a s t e r
Revised: M a s t e r

Original: N a m e T h i s G a m e
Revised: N a m e T h i s G a m e

Original: P h
Revised: P h

Original: o e n i x
Revised: o e n i x

Original: P oo y a n Q b e r t
Revised: P oo y a n Q b e r t

Original: R i v e rr a i d
Revised: R i v e rr a i d

Original: R o a d R unn e r
Revised: R o a d R unn e r

Original: R o b o t a n k
Revised: R o b o t a n k

Original: S e a q u e s t
Revised: S e a q u e s t

Original: T i m e P il o t
Revised: T i m e P il o t

Original: U p N D
Revised: U p N D

Original: o w n
Revised: o w n

Original: V i
Revised: V i

Original: d e o P i n b a ll
Revised: d e o P i n b a ll

Original: W i z a r d
Revised: W i z a r d

Original: O f W o r
Revised: O f W o r

Original: Y a r s R e v e n g e
Revised: Y a r s R e v e n g e

Original: Z a xx o n
Revised: Z a xx o n

Original: [Figure]
Revised: [Figure]

Original: In Figure 1 we see that IQM performance across all games is indeed signiﬁcantly improved by generating optimality-conditioned actions.
Revised: In Figure 1 we see that IQM performance across all games is indeed signiﬁcantly improved by generating optimality-conditioned actions.

Original: Figure 8 shows the mean and standard deviation of scores across all games.
Revised: Figure 8 shows the mean and standard deviation of scores across all games.

Original: While behavior cloning may sometimes produce highly-rewarding episodes, it is less likely to do so.
Revised: While behavior cloning may sometimes produce highly-rewarding episodes, it is less likely to do so.

Original: We ﬁnd decision transformer outperforms behavioral cloning in 31 out ofgames.
Revised: We ﬁnd decision transformer outperforms behavioral cloning in 31 out ofgames.

Original: 4.8 Does training on expert and non-expert data bring beneﬁts over expert-only training?
Revised: 4.8 Does training on expert and non-expert data bring beneﬁts over expert-only training?

Original: We believe that, comparing to learning from expert demonstrations, learning from large, diverse datasets that include some expert data but primarily non-expert data help learning and improve performance.
Revised: We believe that, comparing to learning from expert demonstrations, learning from large, diverse datasets that include some expert data but primarily non-expert data help learning and improve performance.

Original: To verify this hypothesis, we ﬁlter our training data [1] from each game by episodic returns and only preserve top 10% trajectories to produce an expert dataset (see Appendix F for details).
Revised: To verify this hypothesis, we ﬁlter our training data [1] from each game by episodic returns and only preserve top 10% trajectories to produce an expert dataset (see Appendix F for details).

Original: We use this expert dataset to train our multi-game decision transformer (DT-40M) and the transformer-based behavioral cloning model (BC-40M).
Revised: We use this expert dataset to train our multi-game decision transformer (DT-40M) and the transformer-based behavioral cloning model (BC-40M).

Original: Figure 9 compares these models trained on expert data and our DT-40M trained on all data.
Revised: Figure 9 compares these models trained on expert data and our DT-40M trained on all data.

Original: We observe that (1) Training only on expert data improves behavioral cloning; (2) Training on full data, including expert and non-expert data, improves Decision Transformer; (3) Decision Transformer with full data outperforms behavioral cloning trained on expert data.
Revised: We observe that (1) Training only on expert data improves behavioral cloning; (2) Training on full data, including expert and non-expert data, improves Decision Transformer; (3) Decision Transformer with full data outperforms behavioral cloning trained on expert data.

Original: [Figure]
Revised: [Figure]

Original:  Conclusion
Revised:  Conclusion

Original: In the quest to develop highly capable and generalist agents, we have made important and measurable progress.
Revised: In the quest to develop highly capable and generalist agents, we have made important and measurable progress.

Original: Namely, our results exhibit a clear beneﬁt of using large transformer-based models in multi-game domains, and the general trends in these results – performance improvements with larger models and the ability to rapidly ﬁne-tune to new tasks – mirror the successes observed for large-scale vision and language models.
Revised: Namely, our results exhibit a clear beneﬁt of using large transformer-based models in multi-game domains, and the general trends in these results – performance improvements with larger models and the ability to rapidly ﬁne-tune to new tasks – mirror the successes observed for large-scale vision and language models.

Original: Our results also highlight difﬁculties of online RL algorithms in handling the complexity of multi-game training on Atari.
Revised: Our results also highlight difﬁculties of online RL algorithms in handling the complexity of multi-game training on Atari.

Original: It is interesting to note that our best results are achieved by decision transformers, which essentially learn via supervised learning on sequence data, compared to alternative approaches such as temporal difference learning (more typical in reinforcement learning), policy gradients, and contrastive representation learning.
Revised: It is interesting to note that our best results are achieved by decision transformers, which essentially learn via supervised learning on sequence data, compared to alternative approaches such as temporal difference learning (more typical in reinforcement learning), policy gradients, and contrastive representation learning.

Original: This begs the question of whether online learning algorithms can be modiﬁed to be as “data-absorbent” as DT-like methods.
Revised: This begs the question of whether online learning algorithms can be modiﬁed to be as “data-absorbent” as DT-like methods.

Original: While even our best generalist agents at times fall short of performance achieved by agents trained on a single task, this is broadly consistent with related works that have trained single models on many tasks [36, 65].
Revised: While even our best generalist agents at times fall short of performance achieved by agents trained on a single task, this is broadly consistent with related works that have trained single models on many tasks [36, 65].

Original: Still, our best generalist agents are already capable of outperforming the data they are trained on.
Revised: Still, our best generalist agents are already capable of outperforming the data they are trained on.

Original: We believe the trends suggest clear paths for future work – that, with larger models and larger suites of tasks, performance is likely to scale up commensurately.
Revised: We believe the trends suggest clear paths for future work – that, with larger models and larger suites of tasks, performance is likely to scale up commensurately.

Original: Limitations.
Revised: Limitations.

Original: We acknowledge reasons for caution in over-generalizing our conclusions.
Revised: We acknowledge reasons for caution in over-generalizing our conclusions.

Original: Our results are based largely on performance in the Atari suite, where action and observation spaces are aligned across different games.
Revised: Our results are based largely on performance in the Atari suite, where action and observation spaces are aligned across different games.

Original: It is unclear whether ofﬂine RL datasets such as Atari are of sufﬁcient scale and diversity that we would see similar performance scaling as observed in NLP and vision benchmarks.
Revised: It is unclear whether ofﬂine RL datasets such as Atari are of sufﬁcient scale and diversity that we would see similar performance scaling as observed in NLP and vision benchmarks.

Original: Whether we can observe other forms of generalization, such as zero-shot adaptation, as well as whether our conclusions hold for other settings, remains unclear.
Revised: Whether we can observe other forms of generalization, such as zero-shot adaptation, as well as whether our conclusions hold for other settings, remains unclear.

Original: Societal Impacts.
Revised: Societal Impacts.

Original: In the current setting, we do not foresee signiﬁcant societal impact as the models are limited to playing simple video games.
Revised: In the current setting, we do not foresee signiﬁcant societal impact as the models are limited to playing simple video games.

Original: We emphasize that our current agents are not intended to interact with humans or be used outside of self-contained game-playing domains.
Revised: We emphasize that our current agents are not intended to interact with humans or be used outside of self-contained game-playing domains.

Original: One should exercise increased caution if extending our algorithms and methods to such situations in order to ensure any safety and ethical concerns are appropriately addressed.
Revised: One should exercise increased caution if extending our algorithms and methods to such situations in order to ensure any safety and ethical concerns are appropriately addressed.

Original: At the same time, the capability of decision making based on reward feedback – rather than purely imitation of the data – has the potential to be easier to align with human values and goals.
Revised: At the same time, the capability of decision making based on reward feedback – rather than purely imitation of the data – has the potential to be easier to align with human values and goals.

Original: Acknowledgements
Revised: Acknowledgements

Original: We would like to thank Oscar Ramirez, Roopali Vij, Sabela Ramos, Rishabh Agarwal, Shixiang (Shane) Gu, Aleksandra Faust, Noah Fiedel, Chelsea Finn, Sergey Levine, John Canny, Kimin Lee,Hao Liu, Ed Chi, and Luke Metz for their valuable contributions and support for this work. A Contribution Statement
Revised: We would like to thank Oscar Ramirez, Roopali Vij, Sabela Ramos, Rishabh Agarwal, Shixiang (Shane) Gu, Aleksandra Faust, Noah Fiedel, Chelsea Finn, Sergey Levine, John Canny, Kimin Lee,

Original: Kuang-Huei Lee Proposed project direction.

Original: Contributed to the JAX and TF Decision Transformer (DT) and BC code.

Original: Ran BC and DT experiments.

Original: Contributed to paper writing.

Original: Oﬁr Nachum Proposed project direction.

Original: Contributed to TF code for DQN, CQL, and representation learning algorithms.

Original: Ran experiments for DQN and CQL.

Original: Contributed to paper writing.

Original: Mengjiao Yang Contributed code in representation learning algorithms.

Original: Ran experiments for CPC, BERT, ACL pretraining + OOD ﬁnetuning. Contributed to paper writing. Lisa Lee Contributed to TF codebase.
Revised: Hao Liu, Ed Chi, and Luke Metz for their valuable contributions and support for this work.

Original: Ran alternative environment experiments.

Original: Helped with paper editing.

Original: Daniel Freeman Contributed to JAX training code and dataset generation pipelines.

Original: Generated datasets.

Original: Helped with paper writing.

Original: Winnie Xu Contributed to JAX training infrastructure, beginner / expert dataset generation, augmentation and metrics pipelines.

Original: Ran experiments for alternative DT variants.

Original: Helped with paper writing.

Original: Work done while a Research Intern.

Original: Sergio Guadarrama Helped with project direction and experiment discussions.

Original: Worked on paper editing.

Original: Ian Fischer Helped with project direction and experiment discussions.

Original: Worked on paper editing.

Original: Eric Jang Helped with project direction, contributed to building JAX infrastructure.

Original: Helped with paper writing.

Original: Henryk Michalewski Contributed to building jaxline training and data processing infrastructure.

Original: Ran ﬁne-tuning experiments.

Original: Helped with project direction and experiment discussions.

Original: Contributed to paper writing.

Original: Igor Mordatch Proposed project direction.

Original: Contributed to building JAX training and data processing infrastructure.

Original: Ran DT experiments.

Original: Contributed to paper writing and visualizations.

Original: B Implementation Details B.1 Transformer network architecture

Original: The input consists of a sequence of observations, returns, actions and rewards.

Original: Observations are images in the format B × T × W × H × C .

Original: We use 84 × 84 grayscale images ( i.e. , W = 84 , H = 84 , C = 1 ).

Original: Similar to ViT [19], we extract M non-overlapping image patches, perform a linear projection and then rasterise them into d model -dimensional 1D tokens.

Original: We deﬁne each patch to be 14 × 14 pixels ( i.e. , M = 6 × 6 = 36 ).

Original: A learned positional embedding is added to each of the patch tokens o 1 , ...,

Original: o M to retain positional information as in ViT. As described in Section 3.2, returns are discretized into 120 buckets in {− 20 , ..., 100 } , and rewards are converted to ternary quantities {− 1 , 0 , +1 } .

Original: For the whole sequence (cid:104) ..., o t 1 , ..., o tM , ˆ R t , a t , r t , ... (cid:105) , we learn another positional embedding at each position and add to each token embedding.

Original: We experimented with rotary position embedding [72], but did not ﬁnd a signiﬁcant beneﬁt from them in our setting.

Original: On top of the token embeddings, our transformer models use a standard transformer decoder architecture.

Original: A standard transformer implementation for sequence modeling would employ a sequential causal attention masking to prevent positions from attending to subsequent positions [77].

Original: However, for the sequence (cid:104) ..., o t 1 , ..., o tM , ˆ R t , a t , r t , ... (cid:105) that we consider, we do not want to prevent the position corresponding to observation token o tm from accessing subsequent observation tokens { o tm (cid:48) : m (cid:48)

Original: > m } within the same timestep, since there is no clear sequential causal relation between image patches.

Original: Therefore, we change the sequential causal masking to allow observation tokens within the same timestep to access each other, but not subsequent positions after o tM , i.e. ˆ R t , a t , r t , o t +11 , ..., o t +1 M , ...

Original: Table 1 summarizes the transformer conﬁgurations we use for each model size.

Original: We train these models on an internal cluster, each with 64 TPUv4.

Original: Due to prohibitively long training times, we only evaluated one training seed.

Original: For the 40M DT model, we perform four additional runs with different seeds to investigate the sensitivity to random seeds.

Original: The mean and standard deviation of the median score across 5 random seeds are 0.79 and 0.06; the mean and standard deviation of IQM scores are 0.95 and 0.05.

Original: Compared to the differences to the baseline results and the results of other DT model sizes, we consider the variance to be relatively small.

Original: [Table]

Original: We performed hyperparameter search on DT-40M, and directly applied to DT-200M. With the set of hyperparameters working the best for smaller models, it still manifests a clear scaling trend of multi-game and transfer learning performance.

Original: B.2 Fine-tuning protocol for Atari games

Original: In the ﬁne-tuning experiments, we reserved ﬁve games (Alien, MsPacman, Pong, Space Invaders and

Original: Star Gunner) to be used only for ﬁne-tuning.

Original: These games were selected due to their varied gameplay characteristics.

Original: Each game was ﬁne-tuned separately to measure the model’s transfer performance for a ﬁxed game.

Original: We use 1% of the original dataset (corresponding to roughly 500 000 transitions) to speciﬁcally test ﬁne-tuning in low-data regimes.

Original: B.3 Action and return sampling during in-game evaluation

Original: We sample actions from the model with a temperature of 1.

Original: Inspired by Nucleus sampling (Holtzman et al.

Original: [32]), we only sample from the top 85th percentile action logits for all Decision Transformer models and Behavioral Cloning models (this parameter was selected to give highest performance for both models).

Original: While we train the model to predict actions for all timesteps in the sequence, during in-game evaluation, we execute the last predicted action in the sequence (conditioned on all past observations, and past generated actions, rewards, and target returns).

Original: To generate target returns as discussed in Section 3.4, we sample them from the model with the temperature of 1 and the top 85th percentile logits.

Original: We use κ = 10 in all our experiments.

Original: To avoid storing the history of previously generated target returns (which may be difﬁcult to incorporate into some RL frameworks), we experimented with autoregressively regenerating all target returns in the sequence, and found that to work well without requiring any special recurrent state maintenance outside of the model.

Original: Algorithm 1 has pseudocode for our expert return and action inference.

Original: As an alternative way to generate expert-but-likely returns, we also experimented with simply generating N return samples from the model according to log-probability log P θ ( R t | ... ) , and picking the highest one.

Original: We then generate the action conditioned on this largest picked return as before.

Original: This avoids needing the hyperparameter κ .

Original: In this setting, we found N = 128 , inverse temperature of 0.75 for return sampling, no percentile cutoff for return sampling, and sampling from the top 50th percentile action logits with a temperature of 1 to work similarly well.

Original: B.4 Evaluation protocol and Atari environment details

Original: Our environment is the Atari 2600 Gym environment with pre-processing performed as in Agarwal et al.

Original: Our Atari observations are 84 × 84 grayscale images.

Original: We compress observation images to jpeg in the dataset (to keep dataset size small) and during in-game evaluation.

Original: All games use the same shared set of 18 discrete actions.

Original: For all methods, each game score is calculated by averaging over 16 model rollout episode trials.

Original: To reduce inter-trial variability, we do not use sticky actions during evaluation.

Original: B.5 Image augmentation

Original: All models were trained with image augmentations.

Original: We investigate training with the following augmentation methods: random cropping, random channel permutation, random pixel permutation,

Original: Algorithm

Original: Given an environment E , the current time step t .

Original: κ = 10 , the return upper bound R high = 100 , the return lower bound R low = −

Original: Initialize a context window (cid:104) ..., r t − 2 , o t − 1 1 , ..., o t − 1 M , ˆ R t − 1 , a t − 1 , r t − 1 , o t 1 , ..., o tM (cid:105) , abbreviated as . . .

Original: in the following.

Original: # Autoregressive return and action generation while terminal state not reached yet do

Original: Compute 121 logits ( R t = − 20 , . .

Original: . , 100) for the categorical return distribution P

Original: ( R t | ... )

Original: # Increase logits proportionally to return magnitudes to prefer high magnitude

Original: Deﬁne log P ( R t | expert t , ... )

Original: = log P

Original: ( R t | ... )

Original: + κ

Original: ( R t − R low ) /

Original: ( R high − R low )

Original: # Sample a return

Original: R t ∼ P ( R t | expert t , ... )

Original: Compute logits for the categorical action distribution P ( a t | R t , ... )

Original: # Sample an action a t ∼ P ( a t | R t , ... )

Original: # Interact with the environment o t +11 , ..., o t +1 M , r t ∼ E step ( a t ) t = t + 1 and shift the context window accordingly end while horizontal ﬂip, vertical ﬂip, and random rotations.

Original: We found random cropping and random rotations to work the best.

Original: (In our random cropping implementation, images of size 84 × 84 are padded on each side with 4 zero-value pixels, and then randomly cropped to 84 × 84 .)

Original: In general, we aim to expand the domain of problems solved during training to similar kinds that we hope to generalize to by encoding useful inductive biases.

Original: We maintain the same random augmentation parameters for each window sequence.

Original: We apply data augmentation in both pre-training and ﬁne-tuning.

Original: C Baseline Implementation Details BC

Original: Our BC model is effectively the same as our DT model but removing the return token ˆ R t from the training sequence:

Original: [Equation]

Original: Instead of predicting a return token (distribution) given observation tokens o t 1 , ..., o tM and the previous part of the sequence, we directly predict an action token (distribution), which also means that we remove return conditioning for the BC model.

Original: During evaluation, we sample actions with a temperature of 1, and sample from the top 85th percentile logits (as discussed in Appendix B.3).

Original: All other implementation details and conﬁgurations are identical to DT.

Original: C51 DQN For single-game experiments, our implementation and training followed the details in [11] except for using multi-step learning with n = 4 .

Original: For multi-game experiments we trained using the details provided in the main text; we ran the algorithm for 15M gradient steps ( ≈ 4 B environment steps ≈ 16 B Atari frames).

Original: CQL For CQL we use the same optimizer and learning rate as for C51 DQN.

Original: We use a per-replica batch size of 32 and run for 1M gradient steps on a TPU pod with 32 cores, yielding a global batch size of 256.

Original: During ﬁnetuning for each game, we copy the entire Q -network trained with CQL, and apply an additional 100k gradient steps of batch size 32 on a single CPU, where each batch is sampled exclusively from the ofﬂine dataset of the ﬁnetuned game.

Original: We also experimented with smaller learning rates ( 0 . 00003 instead of the default 0 . 00025 ) and larger batch sizes ( 1024 , 4096 ) but found the results largely unchanged.

Original: We also tried using ofﬂine C51 and double DQN as opposed to CQL, and found performance to be worse.

Original: CPC For the CPC baseline [56], we apply a contrastive loss between φ ( o t ) , φ ( o t +1 ) using the objective function

Original: [Equation]

Original: where W is a trainable matrix and ρ is a non-trainable prior distribution; for mini-batch training we set ρ to be the distribution of states in the mini-batch.

Original: The state representations φ ( o ) is parametrized by CNNs followed by two MLP layers with 512 units each interleaved with ReLU activation.

Original: For the CNN architecture, we used the C51 implementation with an Impala neural network architecture of three blocks using 16, 32, and 32 channels respectively, and trained with a batch size ofand learning rate of 0.00025 both during pretraining and downstream BC adaptation.

Original: We conduct representation learning for a total of 1M gradient steps, and ﬁnetune on 1% data for 100k steps every 50k steps of representation learning and report the best ﬁnetuning results.

Original: BERT and ACL Our BERT and ACL baselines are based on the representation learning objectives described in [80].

Original: For the BERT [18] state representation learning baseline, we (1) take a subtrajectory o t : t + k , a t : t + k , r t : t + k from the dataset (without special tokenization as in DT), (2) randomly mask a subset of these, (3) pass the masked sequence into a transformer, and then (4) for each masked input state o t + i , apply a contrastive loss between its representation φ ( o t + i ) and the transformer output Transformer[ i ] at the corresponding sequence position:

Original: [Equation]

Original: where ρ is the distribution over states in the mini-batch.

Original: For attentive contrastive learning (ACL) [80], we apply an additional action prediction loss to the output of BERT at the sequence positions of the action inputs.

Original: To parameterize φ , we use the same CNN architecture as in CPC, while the transformer is parameterized by two self-attention layers with 4 attention heads of 256 units each and feed-forward dimension 512.

Original: The transformer does not apply any additional directional masking to its inputs.

Original: We used k = 16 .

Original: Pretraining and ﬁnetuning is analogous to CPC.

Original: Namely, when ﬁnetuning we take the pretrained representation φ and use a BC objective for learning a neural network (two MLP layers withunits each) policy on top of this representation.

Original: D Comparisons between transformers and convolution networks

Original: [Figure]

Original: Decision Transformer is an Upside-Down RL (UDRL) [68, 71] implementation that uses the transformer architecture and considers RL as a sequence modeling problem.

Original: To understand the beneﬁt of the transformer architecture, we compare to a UDRL implementation that uses feed-forward, convolutional

Original: Impala networks [20].

Original: Speciﬁcally, we use the same return, action, and reward tokenizers as in

Original: DT, and only replace the observation (four consecutive Atari frames stacked together) encoding to use the Impala architecture.

Original: Similar to what we do for CQL, we also experiment with different sizes of the Impala architecture by varying the number of blocks and channels in each block of the Impala network: the number of blocks and channels is one of (5 blocks , 128 channels)

Original: ≈ 5M params , (10 blocks , 256 channels)

Original: ≈ 30M params , (5 blocks , 512 channels)

Original: ≈ 60M params .

Original: We use a (768 , 768) 2 -layer fully-connected head to predict the next return token from observation embedding; another (768 , 768) head to predict the next action token from a concatenation of observation embedding and return token embedding; and another (768 , 768) head to predict the next reward token from a concatenation of observation embedding, return token embedding, and action token embedding.

Original: The input to the model is slightly different from what we have for DT: Instead of considering a

Original: T -timestep sub-trajectory ( T = 4 ) where each timestep contains o t , R t , a t , r t , we stack T image frames (as common in [53]), and only consider R t , a t , r t from the last timestep.

Original: All other design choices and evaluation protocols are the same as DT.

Original: Figure 10 shows clear advantages of Decision Transformer over UDRL with the Impala architecture.

Original: In the comparison between UDRL (Impala) and CQL that uses the same Impala network at each model size we evaluated, we observe that UDRL (Impala) outperforms CQL.

Original: The results show that the beneﬁts of our method come not only from using network architectures, but also from the UDRL formulation.

Original: While the reasons for the beneﬁts of transformer architecture over other neural networks are still an open question in general (see e.g., [64]), our hypothesis is that transformers allow for easier discovery of correlations between components of the input and output, due to the fact that transformers process the input as a ﬂat sequence with attention allowed between any patch, action, or return token.

Original: Although it is not feasible to compare transformer with all possible convolutional architectures due to the broad design space, we believe these empirical results still show a clear trend favoring both UDRL and transformer architectures.

Original: E Comparisons between methods using median human normalized scores

Original: We used inter-quartile mean (IQM) to aggregate performance over individual games in Figure 1.

Original: Median is another metric commonly used to aggregate scores (although it has issues as discussed in [2]: it has high variability, and in the most extreme case, the median is unaffected by zero performance on nearly half of the tasks.).

Original: For completeness, we report the median scores for all methods:

Original: H u m an - N o r m a li z ed

Original: M ed i an

Original: S c

Original: o r e

Original: [Figure]

Original: blue are generalists.

Original: Single-game BCQ results are from Gulcehre et al.

Original: For expert-ﬁltering experiments in Section 4.8, we also provide the plot of expert ﬁltering effects with median human-normalized scores in Figure 12.

Original: We note that ranking of various conﬁgurations do not change across aggregate metrics.

Original: [Figure]

Original: For Upside-Down RL comparison experiments Appendix D, we also provide median humannormalized scores in Figure 13.

Original: [Figure]

Original: F Details of Expert Dataset Generation

Original: To generate the expert dataset for experiments in Section 4.

Original: [Figure]

Original: we we ﬁlter our training data [1] from each game by episodic returns and only preserve top 10% trajectories to produce an expert dataset.

Original: We plot of return histograms for reference in Figure

Original: [Figure]

Original: G Effect of Model Size on Training Speed

Original: It is believed that large transformer-based language models train faster than smaller models, in the sense that they reach higher performance after observing a similar number of tokens [38, 15].

Original: We ﬁnd this trend to hold in our setting as well.

Original: Figure 15 shows an example of performance on two example games as multi-game training progresses.

Original: We see that larger models reach higher scores per number of training steps taken (thus tokens observed).

Original: [Figure]

Original: Asterix

Original: [Figure]

Original: H Qualitative Attention Analysis

Original: We ﬁnd that the Decision Transformer model consistently attends to observation image patches that contain meaningful game entities.

Original: Figure 16 visualizes selected attention heads and layers for various games.

Original: We ﬁnd heads consistently attend to entities such as player character, player’s free movement space, non-player objects, and environment features.

Original: [Figure]

Original: I Raw Atari Scores

Original: We report full raw scores of 41 training Atari games for best performing sizes of multi-game models in Table

Original: [Table]

Original: Frostbite 2,435.

Original: [Table]

Original: [Table]

Original: Hero 20,408.

Original: IceHockey -10.

Original: Jamesbond 700.

Original: Kangaroo 12,700.

Original: Krull 8,685.

Original: KungFuMaster 15,562.

Original: NameThisGame 9,056.

Original: Phoenix 5,295.

Original: Pooyan 2,859.

Original: [Table]

Original: Riverraid 14,755.

Original: RoadRunner 54,568.

Original: Robotank 63.

Original: Seaquest 5,173.

Original: TimePilot 2,743.

Original: UpNDown 16,291.

Original: [Table]