StarCraft II Unplugged: Large Scale Ofﬂine Reinforcement Learning
Ewalds Mahyar Bordbar
Abstract
StarCraft II is one of the most challenging reinforcement learning (RL) environments; it is partially observable, stochastic, and multi-agent, and mastering
StarCraft II requires strategic planning over long-time horizons with real-time low-level execution.
It also has an active human competitive scene.
StarCraft II is uniquely suited for advancing ofﬂine RL algorithms, both because of its challenging nature and because Blizzard has released a massive dataset of millions of StarCraft II games played by human players.
This paper leverages that and establishes a benchmark, which we call StarCraft II Unplugged , that introduces unprecedented challenges for ofﬂine reinforcement learning.
We deﬁne a dataset (a subset of Blizzard’s release), tools standardising an API for ML methods, and an evaluation protocol.
We also present baseline agents, including behaviour cloning, and ofﬂine variants of V-trace actor-critic and MuZero .
We ﬁnd that the variants of those algorithms with behaviour value estimation and single-step policy improvement work best and exceed 90% win rate against previously published AlphaStar behaviour cloning agents.
Introduction
Deep Reinforcement Learning (RL) is dominated by online RL algorithms, where agents must interact with the environment to explore and learn.
This online RL paradigm achieved considerable success on Atari [33], Go [43], StarCraft II [50], DOTA 2 [7], and robotics [2].
However, the requirements of extensive interaction and exploration make these algorithms unsuitable and unsafe for many real-world applications.
In contrast, in the ofﬂine setting [14, 15, 17], agents learn from a ﬁxed dataset previously logged by humans or other agents.
While the ofﬂine setting would enable RL in real-world applications, most ofﬂine RL benchmarks such as D4RL [14] and RL Unplugged [17] have mostly focused on simple environments with data produced by RL agents.
More challenging benchmarks are needed to make progress towards more ambitious real-world applications.
To rise to this challenge, we introduce StarCraft II Unplugged , an ofﬂine RL benchmark, which uses a dataset derived from replays of millions of humans playing the multi-player competitive game of
StarCraft II.
StarCraft II continues to be one of the most complex simulated environments, with partial observability, stochasticity, large action and observation spaces, delayed rewards, and multi-agent dynamics.
Additionally, mastering the game requires strategic planning over long time horizons, and real-time low-level execution.
Given these difﬁculties, breakthroughs in StarCraft II Unplugged will likely translate to many other ofﬂine RL settings, potentially transforming the ﬁeld.
Additionally, unlike most RL domains, StarCraft II has an independent leaderboard of competitive human players over a wide range of skills.
It also constitutes a rich and abundant source of data to train and evaluate ofﬂine RL agents.
With this paper, we release the most challenging large-scale ofﬂine RL benchmark to date, including the code of a canonical agent and data processing software.
We note that removing the environment interactions from the training loop signiﬁcantly lowers the compute demands of StarCraft II, making this environment accessible to far more researchers in the AI community.
The paper introduces several ofﬂine RL agents that can learn competitive policies purely from human replays.
This has been made possible by innovations in architectures and algorithms.
Extensive experimentation and evaluations, with proposed metrics, have taught us that many existing ofﬂine
RL algorithms fail in this benchmark.
However, they have also provided us with insights on how to design successful agents.
Chieﬂy among these insights is the following recipe for constructing successful agents: ﬁrst train a policy and value function network to estimate the behavior policy and value function.
Then, during evaluation, either perform a single-step of policy improvement or improve the policy with the ﬁxed and pretrained behavior value function.
We believe sharing these insights will be valuable to anyone interested in ofﬂine RL, especially at large scale.
Background
The underlying system dynamics of StarCraft II can be described by a Markov decision process (MDP; [6]).
An MDP, M def = ( S , A , P, r, d ) , consists of ﬁnite sets of states S and actions A , a transition distribution P ( s 0 | s, a ) , s, s 0 2 S , a 2 A , a reward function r : S ⇥ A !
R , and an initial state distribution d : S !
In the ofﬂine setting, the agent does not interact with the MDP but learns only from a dataset D containing sequences ( s t , a t , r t +1 )
Nt =0 .
The dataset D is assumed to have been generated by following an unknown behaviour policy µ , a distribution over actions conditioned on the state: µ ( a | s ) .
The goal of ofﬂine RL is to ﬁnd a policy ⇡ that maximizes the expected discounted return
[Equation]
where s 0 ⇠ d and  t is a discount factor at time step t .
We refer to the policy ⇡ as the target policy .
We use V µ and V ⇡ to denote the estimated value function for the behaviour and target policies µ and ⇡ , respectively.
StarCraft II Unplugged: An Ofﬂine RL Benchmark
This work builds on top of the StarCraft II Learning Environment and associated replay dataset [51], and the agents described in [50], by providing a few key components necessary for an ofﬂine RL benchmark:
• Evaluation metric.
Built-in rule-based agents provide a measure of performance, but this metric can quickly saturate.
In this work we present alternative metrics.
• Baseline agents.
We provide a number of well tuned baseline agents.
• Open source code.
Building an agent that performs well on StarCraft II is a large engineering endeavor.
We provide a well tuned behavior cloning agent which forms the backbone for all agents presented in this paper 3 .
3.1 Challenging Properties for Ofﬂine RL
When learning from ofﬂine data, the performance of algorithms depends greatly on the availability of different state-action pairs in the data.
We call this coverage — the more state-action pairs are absent, i.e. the lower the coverage, the more challenging the problem is.
Here we highlight a few properties of StarCraft II that make it particularly challenging from a coverage perspective.
Action space. StarCraft II has a highly structured action space.
As discussed in [50], the agent must select an action type, select a subset of its units to apply the action to, select a target for the action (either a map location or a visible unit), and decide when to observe and act next.
Expanding, there are approximately 10 26 possible actions per step.
For comparison, Atari has only 18 possible actions per step.
This makes it much more challenging to attain high state-action coverage for StarCraft II.
Stochastic environment. Stochastic environments may need many more trajectories to obtain high state-action coverage.
The main source of stochasticity is an unknown opponent policy, which itself may be stochastic.
To a small extent, the dynamics of the game engine are stochastic, for example some units have random movement, or random spawn points, and some commands can have a random delay.
For comparison, in the Atari environment stochasticity arises only from sticky actions [32].
Partial Observability. StarCraft II is an imperfect information game.
Players only have information about opponent units that are within the ﬁeld of view of the player’s own units.
As a result, players need to scout, i.e. send their units around the map to gather information about the current state of the game, and may need it at a later point in the game.
In comparison, a memory of the 3 most recent frames is usually considered sufﬁcient for Atari.
Data source. For StarCraft II, we have access to a dataset of millions of human replays.
These replays display a wide and diverse range of exploration and exploitation strategies.
In contrast, the existing benchmarks [17, 1] have a bias toward datasets generated by RL agents.
Replays are also labelled with the MMR of the player, a valuable data about their skill.
These properties create challenges for existing algorithms, and are likely to emphasize any differences between behaviour cloning, online RL algorithms, and ofﬂine RL algorithms.
3.2 Dataset
About 20 million StarCraft II games are publicly available through the replay packs 4 .
For technical reasons, we restrict the data to StarCraft II versions 4.8.2 to 4.9.2 which leaves nearly 5 million games.
They come from the StarCraft II ladder , the ofﬁcial matchmaking mechanism.
Each player is rated by their MMR , a ranking mechanism similar to Elo [12].
The MMR ranges from 0 to around 7000.
Figure 1 shows the distribution of MMR among the episodes.
In order to get quality training data, we only use games played by players with MMR greater than 3500, which corresponds to the top 22% of players.
This leaves us with approximately 1.4 million games.
Since we consider two-player games, from a machine learning point of view each game forms two episodes, one for each side, so there are 2.8 million episodes in the dataset.
This represents a total of more thanyears of game played.
Episodes can span many steps.
StarCraft II is a real-time strategy game with a fast internal game clock.
In order to make trajectories shorter, we only observe the steps when the player took an action.
Consequently, each time step t of the dataset contain a pair ( s t , a t ), but also the delay , the number of internal game steps since the last action a t  1 .
The delay is encoded as part of s t .
Each episode also replay-api contains metadata, the most important ones being the outcome R which can be 1 for a victory, for a defeat and 0 for a draw (draws are extremely rare in StarCraft II), as well as the MMR of each player.
Since the games were played online using Blizzard’s matchmaking system, both players have similar MMR in the vast majority of games.
Methodology 4.1 Training
StarCraft II Unplugged is a large-scale benchmark for data-driven approaches.
Algorithms may only use data from the dataset described in Appendix A.2 to learn to play.
Algorithms may not collect more data by interacting with the environment.
However, we allow for online policy evaluation, i.e. policies can be run in the environment to measure how well they perform.
This evaluation may be useful for hyperparameter tuning.
Unlike the original AlphaStar agents [50], the agents here are trained to play all three races of StarCraft 2.
This is more challenging, as agents are typically better when they are trained on a single race.
They are also trained to play on all 10 maps available in the dataset.
We typically train the agent on rollouts , i.e. sequences of K consecutive timesteps ( s 0 , a 0 , r 1 , . . .
s K  1 , a K  1 , r K ) , assembled in a minibatch of M independent rollouts.
Unless speciﬁed otherwise, the minibatches are independent from each other, such that two consecutive minibatches are not correlated.
The rollout length K differs between methods, as explained in Section 5.
4.2 Reference Agents
In this paper, we present six agents trained on this dataset as reference agents.
We propose to use them for evaluation of future works, by measuring the win-rate of the new agent against them.
They fall into three categories: behaviour cloning , actor-critic and
Monte-Carlo tree search (MCTS).
Details about these agents, their implementations, and ablations are given in Section 5.
Their performances can be found in Table 1.
Behaviour Cloning (BC).
Our behaviour cloning baseline [36, 44] is an imitation learning method that trains an agent to predict what would be the next action in the dataset given the observation.
As such it uses a supervised loss which is the cross-entropy for StarCraft II.
A BC agent learns an estimate of the behavior policy ˆ µ using only states and actions from the dataset.
Fine-Tuned Behaviour Cloning (FT-BC).
The FT-BC agent restarts training from a pretrained BC agent, and ﬁne-tunes the pretrained parameters on the winning games with the highest MMR only, similar to the ﬁne-tuning used in [50].
Ofﬂine Actor-Critic (OAC).
The OAC agent adapts the V-trace algorithm from Impala [13] to ofﬂine RL in the i.i.d.
setting.
To compute importance sampling ratios, it uses the behaviour policy ˆ µ estimated by the BC agent.
The actor-critic method starts from ˆ µ and learns an improved policy ⇡ .
Unlike the online version of actor-critic, this agent uses V µ , the value function of the behaviour policy, as the critic (instead of V ⇡ ).
Emphatic Ofﬂine Actor-Critic (E-OAC).
The emphatic method is a theoretical tool to improve stability in off-policy learning [46, 24].
This agent applies N-steps Emphatic Traces (NETD) [25] to the Ofﬂine Actor-Critic agent, however using consecutive data sequences.
MuZero Supervised MCTS (MZS-MCTS).
We apply MuZero Unplugged [39] for StarCraft II but with a key difference.
While MuZero Unplugged uses Monte Carlo tree search (MCTS; [9]) at training time, MuZero Supervised is trained with supervised learning and thus does estimation of [Table] the behvaioural policy ˆ µ and behavioural value V µ .
At inference time, the policy, value [Table] and latent model is used to perform MCTS for policy improvement and action selection.
MuZero Supervised Policy (MZS-P). For completeness, we also evaluate the performance of the same MuZero Supervised agent, but without MCTS at inference time, and instead directly using the policy network.
This is similar to a behaviour cloning agent [Table] but can often perform better because of the additional regularisation effects from training a value function and the latent model which are not used during inference (as also noted in [18, 39, 22]) [Table]
4.3 Evaluation and Metrics
Numerous metrics can be used to evaluate the agents.
On one hand, the easiest to compute (and least informative) is simply to look at the value of the loss functions.
On the other hand, perhaps the most informative (and most difﬁcult to compute) metric would be to evaluate the agent against a wide panel of human players, including professional players.
In this paper, we propose a compromise between these two extremes.
We evaluate our agents by playing repeated games against a ﬁxed selection ofopponents: the very_hard built-in bot 5 , as well as the 6 reference agents presented above.
During training, we only evaluate the agents against the very_hard bot, since it is signiﬁcantly less expensive, and we mostly use that as a validation metric, to tune hyper-parameters and discard non-promising runs.
Fully trained agents are evaluated against the full set of opponents presented above.
We combine these win-rates into two aggregated metrics while uniformly sampling the races of any pair of these agents: robustness , computed as one minus the minimum win-rate over the set of reference agents, and Elo rating [12].
See details of the metrics computation in Appendix A.5.
These metrics are presented in Table 1.
Deep Dive into Reference Agents
The performance of our six reference agents is shown in Table 1.
In the rest of this section, we provide details of these agents, as well as ablation studies.
Results in Table 1 are evaluated on agents with a softmax temperature of 0 . 8 .
This means that at evaluation time, the softmax logits are multiplied by 0 .
8 before being converted to probabilities.
This makes the agents more deterministic, but also stronger.
Scalar action logits
MLP function action logits
MLP merge MLP merge
MLP encoder vectors delay action logits
MLP merge queued action logits
MLP merge repeat
Units action logits
PointerNets merge unit tags action logits
PointerNet target unit tag
Feature planes action logits
ConvNet merge world stride conv up
ResNet aggregate embed
ResNet stride conv down merge scatter Transformer feature planes encoder units encoder Inputs
Trainable module
Fixed operation
Outputs
Actions
Legend
Vectors
Units
Feature planes e m bed e m bed e m bed e m bed merge merge e m bed e m bed
5.1 Architecture
We used the same architecture for all the agents presented in this paper.
It is derived from the model used in [50], with some improvements.
Inputs and outputs of the StarCraft II API are structured around three types of data: vectors , units and feature planes (more details can be found in Sections A. and A.3). We structure the architecture around them, as shown on Figure 3. Unlike the model in [50], we do not use an LSTM module or any form of memory, which is discussed in Section 5.2.
More details on the architecture can be found in Appendix A.4.
5.2 Behaviour Cloning
The BC and FT-BC agents are trained using a supervised loss, where given a trajectory ( s 0 , a 0 , ..., s t  1 , a t  1 , s t ) , the model predicts a t .
This is similar to training a language model.
The procedure is detailed in Algorithm 1 in the Appendix.
It is the same procedure that was used by the
AlphaStar Supervised agent to create the subsequent Grandmaster level AlphaStar Final agent [50], with the difference that we train a single agent on all three races of StarCraft, instead of three separate agents.
This makes the problem more challenging.
The input of the agent is the observation s t , the past action a t  1 , the MMR of the player and the previous LSTM state (for the model using a LSTM, see below).
During inference, we can then control the quality of the game played by the agent by changing the MMR input.
In practice, we set the MMR to the highest value to ensure the agent plays its best.
We used the Adam optimizer [31] with a cosine learning rate schedule [29].
Implementation details can be found in Appendix B.1.1.
Memory. The AlphaStar agent of [50] uses a LSTM module.
Surprisingly, our experiments show that removing the LSTM entirely result in a better performing model, although the ﬁnal values of the losses are higher.
The LSTM-based agent reaches a win-rate of 70% against the very_hard bot (vs. 84% for the memory-less agent).
Therefore all agents presented in this paper are memory-less.
Data ﬁltering. Since behaviour cloning just mimics the training data, a higher quality data can lead to better performance.
We ran experiments to ﬁlter the data to include only top-tier MMR and/or only winning games.
Unfortunately, since ﬁltering the data also decreases the number of episodes, the overall performance of the agent is deteriorated.
The number of episodes after ﬁltering and the ﬁnal performance are shown in Table 3.
Minibatch size and rollout length.
The minibatch size M and rollout size K inﬂuence the ﬁnal behaviour cloning performance, as shown in Table 2.
We found that larger total amount of data per
Fine-tuning. The BC agent achieves better results when a secondary training phase is added.
We call the resulting method FT-BC.
We ﬁrst train the agent on all the data (with MMR>3500), then, after reducing the learning rate, we further train the agent on the very best data (MMR>6200) and only winning games.
This scheme leads to the best behaviour cloning results, as shown in Table 3.
Implementation details of the FT-BC agent can be found in Appendix B.1.1.
5.3 Ofﬂine Actor-Critic
Actor-critic [54, 5] algorithms learn a policy ⇡ and a value function V ⇡ .
In off-policy settings, where ⇡ differs from the behaviour policy µ , it computes Importance Sampling (IS) ratios ⇢ t ( a t | s t ) def = ⇡ ( a t | s t ) /µ
( a t | s t ) where ( s t , a t ) are sampled from the behaviour policy, i.e. come from the data.
We use V-trace [13] to reduce variance.
Our setup differs from the classical off-policy actor-critic.
Indeed, we do not have access to the behaviour policy µ used by the players, we can only observe their actions.
Instead, we use the BC agent to provide an estimate of the behaviour policy ˆ µ and use the estimated ˆ ⇢ def = ⇡ / ˆ µ in place of the ground truth IS ratios.
The BC agent is also used as the starting point for OAC ( i.e. it is used to initialize the weights).
Learning V µ .
We train our network to produce the estimated behaviour value V µ using a secondary output, via a MLP conditioned on the features.
It is learned using a Mean-Squared Error (MSE) loss: || V µ ( s t )  R || 22 where s t comes from a trajectory which leads to the game outcome R (-1 for a loss, 1 for a win).
We have tried both training it at the same time as behaviour cloning, or afterwards on the pre-trained model, and have observed no signiﬁcant difference in performance.
The value function trained for the OAC and E-OAC models is trained afterwards.
It reaches 72% accuracy, which is computed as the fraction of steps where the sign of V µ is the same as R .
Divergence of V ⇡ .
The standard actor-critic method learns ⇡ , an improvement of µ , as well as
V ⇡ , the corresponding value function.
This is usually done with temporal differences (TD) learning.
We observe that when doing so, using the value function V ⇡ leads to divergence during training, as shown on Figure 5 in the Appendix.
We explored several ways to mitigate this problem:
• Early stopping.
The policy ⇡ improves at ﬁrst, before deteriorating.
Therefore we could stop training early to obtain an improved policy ⇡ (shown in Figure 5).
We do not use this method since it requires using the environment to detect when to stop.
• Alternatively, we can use V µ as a critic, and keep it ﬁxed, instead of estimating V ⇡ iteratively.
This is similar to the behavior value estimation of [16].
It is implemented for the OAC agent.
• The Emphatic Ofﬂine Actor-Critic (E-OAC) agent uses emphatic traces to weight each value and policy update in OAC.
N-step Emphatic Traces (NETD) [25] avoids deadly triads in off-policy learning with linear value function approximation.
Using estimated importance sampling ratios ˆ ⇢ , the NETD trace F can be computed as in [25].
Training parameters. Since actor-critic uses temporal differences, the rollout length K has to be larger than 1 , we use K = 64 .
Because of the way emphatic traces are computed, the E-OAC [Table] [Table] agent requires learning from consecutive minibatches 6 .
Details can be found in Appendices B.1.
and B.1.3.
As explained in Appendix A.3, we only apply policy improvement to the function and delay arguments of the action for simplicity.
5.4 MuZero
MuZero Unplugged [39] is a recently introduced ofﬂine RL approach which achieves state-of-the-art performance on a variety of ofﬂine RL benchmarks [17, 11].
Inspired by recent extensions to handle stochastic environments [34], we evaluate MuZero as an ofﬂine RL approach for StarCraft II.
We use the approach introduced in Sampled MuZero [23] to handle the large action space of StarCraft
II.
Sampled MuZero samples multiple actions from the policy and then restricts the search to only the sampled actions.
This allows us to scale to the large action space of StarCraft II.
We provide further implementation details in Appendix B.1.4.
MuZero Supervised .
Our ﬁrst key result is that MCTS is a strong policy improvement for StarCraft
II when training the model with supervised learning.
Throughout the training, and thus even for relatively weak policies and value functions, we found that MCTS almost always performs better than the neural network policy when training the policy and value function via supervised learning.
MuZero Unplugged .
Our preliminary experiments on using the full MuZero Unplugged algorithm, i.e. training with MCTS targets, were not successful.
We found that the policy would collapse quickly to a few actions with high (over-)estimated value.
While MCTS at inference time improves performance, using MCTS at training time leads to a collapsed policy.
To investigate this further, we evaluate the performance of repeated applications of MCTS policy improvement on the behavioural policy ˆ µ and value V µ .
We do this by training a new MuZero model using MCTS actions of a behavioural policy, i.e. ˆ ⌫ = MCTS (ˆ µ, V µ ) .
We found that the MCTS performance of this policy
MCTS (ˆ ⌫ , V µ ) is worse than the performance of ˆ ⌫ or MCTS (ˆ µ, V µ ) .
Thus, repeated applications of MCTS do not continue to improve the policy.
We believe this is likely due to MCTS policy distribution generating out of distribution action samples with over-estimated value estimates.
Discussion
The behaviour cloning agent is the base of all agents in this work.
It is conditioned on the MMR during training, which helps ﬁlter out lower quality trajectories.
Nevertheless, the behavior cloning agent is still fundamentally limited to estimating the behavior policy, ignorant about rewards.
As a result, the policy it learns is a smoothed version of all the policies that generated the dataset.
We experimented with training the BC agent conditioning on rewards, and evaluated it conditioning on the winning game outcomes, without observing any gain in performance.
In contrast, ofﬂine
RL methods use rewards to improve learned policies in different ways.
MZS-P uses rewards in auxiliary losses, FT-BC uses rewards when episodes for ﬁnetuning, and OAC, E-OAC, and
MZS-MCTS use rewards to perform reinforcement learning.
Final results show that agents that use rewards via reinforcement learning perform the best.
We have observed that algorithms originally designed for online learning — even with off-policy corrections — do not work well when applied directly to the full ofﬂine RL setting.
We attribute this in part to the problem of Deadly Triads [48, 45, 49].
However, many recent works have found these algorithms can be made more effective simply by making modiﬁcations that ensure the learned policy stays close to the behavior policy µ , that the learned value stays close to V µ , or both.
Our results with Actor-Critic and MuZero are in accordance with these ﬁndings.
The reference agents are the methods we attempted that resulted improving the policy.
However, we have tried several other methods without success, including Advantage-Weighted Regression (AWR) [35], PPO [41], and state-action value based methods like SARSA [37], CRR [53] and
R-BVE [16].
To this day, we do not know if there is a fundamental reason for this, or if we simply failed to ﬁnd the necessary changes these methods would require to perform well on this dataset.
Related Work
Online RL has been very impactful for building agents to play computer games.
RL agents can outperform professional human players in many games such as StarCraft II [50], DOTA [7] or
Atari [33, 4].
Similar levels of progression have been observed on board games, including chess and Go [42, 43].
Although ofﬂine RL approaches have shown promising results on Atari recently [40], it has not been previously explored how they would perform on complex partially observable games using data derived from human experts.
RL Unplugged [17] introduces a suite of benchmarks for Ofﬂine RL with a diverse set of task domains with a uniﬁed API and evaluation protocol.
D4RL [14] is an ofﬂine RL benchmark suite focusing only on mixed data sources.
However, both RL Unplugged and D4RL lack high-dimensional, partially observable tasks.
This paper ﬁlls that gap by introducing a benchmark for StarCraft II.
Ofﬂine RL has become an active research area, as it enables us to leverage ﬁxed datasets to learn policies to deploy in the real-world.
Ofﬂine RL methods include 1) policy-constraint approaches that regularize the learned policy to stay close to the behaviour policy [53, 15], 2) value-based approaches that encourage more conservative value estimates, either through a pessimistic regularization or uncertainty [28, 16], 3) model-based approaches [55, 26, 40], and 4) adaptations of standard offpolicy RL methods such as DQN [1] or D4PG [53].
Recently methods using only one-step of policy improvement has been proven to be very effective on ofﬂine reinforcement learning [16, 8].
Conclusions
Ofﬂine RL has enabled the deployment of RL ideas to the real-world.
Academic interest in this area has grown and several benchmarks have been proposed, including RL-Unplugged [17], D4RL [14], and RWRL [11].
However, because of the relatively small-scale and synthetic nature of these benchmarks, they don’t capture the challenges of real-world ofﬂine RL.
In this paper, we introduced StarCraft II Unplugged, a benchmark to evaluate agents which play
StarCraft II by learning only from ofﬂine data.
This data is comprised of over a million games games mostly played by amateur human StarCraft II players on Blizzard’s Battle.
Net.
Thus, the benchmark more accurately captures the challenges of ofﬂine RL where an agent must learn from logged data, generated by a diverse group of weak experts, and where the data doesn’t exhaust the full state and action space of the environment.
We showed that ofﬂine RL algorithms can exceed 90% win-rate against the previously published
AlphaStar Supervised agent (trained using behaviour cloning).
However, the gap between online and ofﬂine methods still exists and we hope the benchmark will serve as a testbed to advance the state of art in ofﬂine RL algorithms.
Acknowledgments and Disclosure of Funding
We would like to thank Scott Reed and David Silver for reviewing the manuscript, the Alphastar team [50] for sharing their knowledge and experience about the game.
We would like to thank the authors of MuZero Unplugged [39] and Sampled MuZero [23] for advising on the development of the MuZero
Supervised agent.
We also thank the wider DeepMind research, engineering and environment teams for the technical and intellectual infrastructure upon which this work is built.
We are grateful to the developers of tools like Jax [3] and Haiku [21] that enabled this research.