An Investigation into the Open World Survival Game Crafter
Abstract
We share our experience with the recently released Crafter benchmark, a 2D open world survival game.
Crafter allows tractable investigation of novel agents and their generalization, exploration and long-term
reasoning capabilities.
We evaluate agents on the original Crafter environment, as well as on a newly introduced set of generalization environments, suitable for evaluating agents’ robustness to unseen objects and fastadaptation (meta-learning) capabilities.
Through several experiments we provide a couple of critical insights that are of general interest for future work on Crafter.
We find that: (1) Simple agents with tuned hyper-parameters outperform all previous agents.
(2) Feedforward agents can unlock almost all achievements by relying on the inventory display.
(3) Recurrent agents improve on feedforward ones, also without the inventory information.
(4) Baseline agents fail to generalize to OOD objects, object-centric agents improve over them.
We will open-source our code.
1. Introduction
Common benchmarks and solid baselines are essential for developing new models and correctly measuring progress in machine learning.
Datasets in supervised learning such as ImageNet (Deng et al., 2009) and environments in reinforcement learning (RL) such as Atari (Bellemare et al., 2013), ProcGen (Cobbe et al., 2020) and MineRL (Guss et al., 2021) have played a crucial role in improving the existing and creating novel models.
Equally importantly, it is critical to compare with solid, yet simple baselines as the ones we introduce in this paper, to judge the improvements offered by novel methods.
This is particularly the case in deep RL, where reproducing existing work is often hard due to a plethora of difficulties (Henderson et al., 2018).
to:
´
Early benchmarks were an important milestone to show that deep RL methods can learn control from high-dimensional images (Koutn´ık et al., 2013; Mnih et al., 2015).
However, they focused on a narrow set of tasks and IID setting between training and evaluation environments, where the agents could memorize a sequence of actions that leads to high rewards, without understanding the mechanics of the world.
For better evaluations, the focus has been recently shifted to closely study the agents’ behaviors through their performance on carefully designed benchmarks (Osband et al., 2020) and to learning agents that can generalize to environments with distributions beyond what they are trained on (e.g. ProcGen or Crafter).
In this work we focus on Crafter (Hafner, 2021), a recently introduced open world survival game, that allows tractable investigation of new agents and their generalization, exploration and long-term reasoning capabilities.
Compared to other benchmarks, Crafter has a set of advantages that make it suitable for RL research such as fast iteration speed (agents can be trained in a few hours on a standard GPU and a single CPU core), model evaluation by inspecting semantically meaningful achievements the agents unlocked, and controllable environment objects that facilitates systematic studies.
On the original Crafter environment our experiments show important, previously unpublished insights into baselines and environment workings.
Crucially we establish solid baselines by showing that simple agents with tuned hyper-parameters outperform all previous agents.
We also introduce a set of new environments, CrafterOOD, that test agents’ generalization and robustness to unseen objects, setting the stage for development of fast-adaptation algorithms such as meta-learning.
Our main findings can be summarised as follows: (1) Simple agents with tuned hyper-parameters outperform all previous agents.
(2) Feedforward (FF) agents can unlock almost all achievements by relying on the inventory display (bottom part of the image in Figure 1a) as a form of a “scratchpad”.
(3) Recurrent agents improve on FF ones, even without the inventory information.
(4) Baseline agents fail to generalize to out-of-distribution (OOD) objects, object-centric agents improve over them.
Researchers should be aware of these baseline and environment insights when testing new ideas on the Crafter environment.
[Figure]
2. Environments
Here we briefly summarize essential properties of the
Crafter environment and the newly introduced CrafterOOD variant.
For additional details we refer to (Hafner, 2021).
Crafter.
Crafter is an open world survival game for RL research, whose game dynamics are inspired by popular game Minecraft.
The benchmark is designed to facilitate existing research challenges, such as strong generalization via procedural generation, deep exploration via achievements conditioned on one another, learning from high-dimensional image observations and sparse rewards that require longterm reasoning and credit assignment.
It facilitates evaluation by unlocking semantically meaningful achievements and fast iteration speed.
In Crafter a unique terrain is generated for every episode with grasslands, lakes and mountains, that contain forests, caves, ores and lava.
The agent needs to collect food and water, protect against enemies and collect resources to craft tools to then unlock all achievements, with collecting the diamond being the last and the most difficult one (see example gameplay in Figure 1a).
There areachievements and the main evaluation metric is the crafter score S computed by averaging unlocked achievements in the log space (to account for differences in their difficulties):
S = exp( 1 N (cid:80) N i =1 ln(1 + s i )) , where s i ∈ [0 , 100] is percentage of episodes in which achievement i was unlocked.
CrafterOOD.
To stay alive in Crafter, the player eats food (plants or cows), drinks water and protects itself from enemies.
To craft tools the player needs to collect resources such as wood, stone, coal and iron.
All these agents and resources are represented as objects in the environment.
We introduce a set of new environments, CrafterOOD, where an agent is trained on one distribution of such objects and then evaluated on another one, possibly one which contains objects never seen during training.
In this way CrafterOOD facilitates testing agents’ generalization and robustness to unseen objects and sets the stage for development of fastadaptation algorithms such as meta-learning.
We introduce new variants for objects the agent most frequently interacts with: trees, cows, zombies, stones, coal and skeletons.
For a detailed overview of newly introduced objects see Figure 3.
3. Methods
We base all our methods on the PPO (Schulman et al., 2017) implementation in the stable baselines (Raffin et al., 2021).
3.1. Linear and Recurrent PPO
PPO learns to map images to actions via policy gradients.
Two FF variants are investigated, that differ by the CNN policy, and a recurrent version based on LSTM (Wierstra et al., 2007; Hochreiter & Schmidhuber, 1997) (see Appendix G).
PPO with NatureCNN (PPO-CNN).
This baseline is architecturally identical to the one used in (Hafner, 2021), with the CNN policy from the DQN paper (Mnih et al., 2015).
PPO with size-preserving CNN (PPO-SPCNN).
Sizepreserving CNN (SPCNN) differs from the PPO-CNN baseline by the CNN architecture.
We introduce it as a baseline for attention-based agents that use SPCNN for “feature mixing”, inspired by SlotAttention (Locatello et al., 2020).
SPCNN does not have pooling layers, so the resulting output tensor is of the same height and width as the input image.
The flattened tensor that is fed into a linear layer is much larger (64x64x64 instead of 8x8x64 for CNN).
Therefore, the policy is also very large (134M parameters).
Recurrent PPO (RecPPO).
Crafter is a partially observable environment.
The agent observes only a part of the world (see Figure 1a).
To perform well it needs to remember the location of resources, e.g. food, mining materials and where it placed objects for crafting new tools, e.g. table or furnace.
Additionally, some achievements require the agent to perform a long chain of reasoning (see Figurein (Hafner, 2021) for the complete overview).
For these reasons, we introduce recurrent agents (RecPPO-CNN and
RecPPO-SPCNN) where we use LSTMs as the critic and the actor networks.
These networks in theory entitle the agent to have memories and can help unlock achievements.
3.2. Attention-based PPO
Our experiments show that CNN agents fail to generalize to OOD environments.
Recently, object-centric methods have successfully been used for OOD generalization in supervised and unsupervised learning (Greff et al., 2017; van Steenkiste et al., 2018; Kosiorek et al., 2018; Stani´c & Schmidhuber, 2019; Greff et al., 2019;
Locatello et al., 2020; Stani´c et al., 2020; Kipf et al., 2021) and in RL (Watters et al., 2019; Veerapaneni et al., 2020; Kipf et al., 2020;
Carvalho et al., 2021) though previous work did not consider procedurally generated open world games.
Since Crafter is composed of objects, we expected these methods to facilitate learning and show stronger generalization capabilities.
In this vein, we designed two attention-based agents and investigated them on CrafterOOD.
They take as input either patches extracted from the image through a size-preserving
CNN (Figure 5) or learn their own representation of an object (by attending over the whole input tensor Figure 9).
In
Crafter, it is possible to crop patches corresponding to individual objects, which is convenient for complexity control when developing new methods (see Appendix G for details).
PPO with self-attention (PPO-SA) (Figure 1b) learns a policy via a dot-product self-attention between the input patches and a learned “CLS” token (a vector initialized to unit Gaussian parameters and optimized via backprop).
This is similar to using the CLS token in BERT (Devlin et al., 2018).
Let ( x 1 , ..., x k , CLS ) ∈
R ( k +1)
× d in be the input sequence, where k is the number of input patches and d in is their dimensionality.
Dot-product self-attention is defined as Attention ( Q, K, V ) = softmax (cid:16) QK T √ d (cid:17) V , where Q ∈ R ( k +1)
× d , K ∈ R ( k +1)
× d , V ∈ R
( k +1)
× d , are the query, key and value matrices, resulting from linear mapping of the input sequence onto a space of dimension d .
We add positional embeddings to the input, which enables learning relative patch positions, e.g. if an enemy is nearby.
PPO with cross-attention networks (PPO-CA) (Figure 1c) learns a set of vectors, which we refer to as ’slots’.
This idea was proposed in SetTransformer (Lee et al., 2019) and successfully used for unsupervised learning of objects (Locatello et al., 2020), learning permutation-invariant agents (Tang et al., 2020; Tang & Ha, 2021) and general perception modules (Jaegle et al.
, 2021b;a; Alayrac et al., 2022).
PPOCA computes attention between queries, keys and values as in self-attention, with the queries coming from the n learned slots, so we have Q ∈ R n × d , K ∈ R k × d , V ∈ R k × d , where k is the number of input patches.
In the extreme case, the patch size is 1, which allows higher expressiveness as each slot can attend to variable-size or further apart image regions.
After cross-attention, slots are pooled by a mean operation and then fed into the policy.
[Table]
4. Experiments and Core Findings
Improved baselines.
By tuning a few hyper-parameters, we find that the simple PPO-CNN baseline outperforms the model-based DreamerV2 (Hafner et al., 2020)
(Table 1).
Surprisingly, even though PPO-SPCNN model has 134M parameters, PPO is not only able to train it, but it also outperforms all other variants in both IID and OOD settings (Tables 1 and 2).
This holds for both FF (PPO-SPCNN) and the recurrent (RecPPO-SPCNN) models.
We tried tuning
DreamerV2, but did not improve on results in (Hafner, 2021) (see Appendix E for hyper-parameters we investigated).
Agents use inventory display as a “scratchpad.”
Although purely FF, PPO-SPCNN is able to unlock almost all achievements (19/22).
This essentially requires the agent to remember the unlocked achievements (e.g I have an iron pick-axe now, need to dig a diamond next).
However, the fact that the memory-less FF models unlocked most achievements leads us to hypothesize that it is using the inventory on the input image as a “scratchpad.”
We test this by training agents with the inventory removed.
The results speak in favor of our hypothesis, as the performance significantly drops (compare
PPO-CNN and “PPO-CNN (no inventory)” in Table 1).
Recurrent improve over FF agents, even without inventory information.
Agent’s reliance on inventory display as a “scratchpad” lead us to investigating whether recurrent agents could improve on FF ones by memorizing actions, map, or unlocked achievements.
With observable inventory,
RecPPO-CNN does not improve upon FF variant PPO-CNN (10.3 vs 10.4 in Table 1).
However, when the inventory is not observable RecPPO-CNN outperforms PPO-CNN (7. vs 6.9).
This might indicate that the recurrent agents learn to store achievements in the memory, although not perfectly as we observe a drop from the case with observable inventory.
Note also that RecPPO-SPCNN outperforms its FF variant
PPO-SPCNN in both IID and OOD settings (Table 2).
Here the inventory is observed, so the largest benefit must arise from memorizing the map layout.
[Table]
CrafterOOD generalization.
We generate a collection of increasingly difficult adaptation scenarios.
Agents are trained and evaluated on progressively more distinct environments.
Environments contain four different object variants for trees, cows, iron, stones, zombies and skeletons.
The settings differ in object distributions in training and evaluation environments.
For example, the IID case corresponds to observing only the first object variant O 1 during training and evaluation (first two rows in Table 2).
Starting from training with uniformly distributed objects ( O 1 − 4 = 25% ) we make generalization progressively more difficult by skewing the training distribution towards the first object.
The evaluation environment always contains only the last three objects uniformly distributed.
In Table 2 we see that up to the point of observing evaluation objects 16% of the time agents generalize fairly well (see also Figure 4 in Appendix B).
Decreasing the percentage of evaluation objects further, the performance of all agents consistently drops.
Finally, when trained only on the first object ( O 1 = 100% ) the agent relies on pure chance or interacting with objects that do not change (e.g. water, iron).
Our experiments show that the object-centric agents PPO-SA and PPO-CA match the vanilla PPO agents on the IID and easy OOD generalization cases (the last two columns in Table 2).
In the most difficult OOD generalization cases though (the last three pairs of rows in Table 2), PPO-SA and PPO-CA show better generalization compared to the PPO-CNN and
PPO-SPCNN.
To the best of our knowledge, this is the first time cross-attention-based methods (e.g. SlotAttention- and
Perceiver-like) are applied to an open-world RL survival game, and we found their vanilla versions not to work out-ofthe-box.
To achieve good level of performance we analyzed their components and present our findings in Appendix F.
This analysis led us to converge to an architecture with a single cross-attention over the input, no latent self-attention, no layer normalization, no residual connections and (unlike
SlotAttention)
no competition between the slots via softmax over queries.
Additionally, these methods let us inspect their attention, making them interpretable and potentially easier to build on in future work (see Appendix C and D).
5. Conclusion and Discussion
For research on new machine learning methods and proper progress measuring, it is crucial to start from strong baselines.
We reported a couple of important observations from our experiments with the Crafter baselines.
These insights indicate that a systematic study of baselines and environment workings (e.g. inventory acting as a “scratchpad”) is needed.
Furthermore, we introduced CrafterOOD, an environment variant that is OOD in objects appearance and show that when the evaluation environment contains unseen objects the baseline agents fail to adapt, but the object-centric ones improve over them.
This might be a fruitful environment for developing fast-adaptation and meta-learning agents.
Also it would be interesting to consider other OOD variants of Crafter, such as ones with objects of varying sizes or frequencies of appearance.
We see Crafter as a good candidate to explore all these direction as one has control over the environment configuration, can inspect models visually and have fast iteration cycles with our solid baselines.
We hope our findings will be of interest and taken into consideration for future work on Crafter.
Acknowledgements
We thank Danijar Hafner, Shixiang Shane Gu and Yingtao
Tian for their useful comments and suggestions on an earlier version of this paper.
This research was partially funded by the ERC Advanced grant no: 742870, project AlgoRNN
A. OOD environment textures
Player
Plant
Cow
Zombie
Skeleton
Arrow
Water
Sand
Grass
Tree
Path
Stone
Coal
Iron
Diamond
Lava
Table
Furnace
TreeV
CowV
ZombieV
StoneV
CoalV
SkeletonV
TreeV
CowV
ZombieV
StoneV
CoalV
SkeletonV
TreeV
CowV
ZombieV
StoneV
CoalV
SkeletonV
TreeV
CowV
ZombieV
StoneV
CoalV
SkeletonV
B. CrafterOOD performance
In Figure 4 we provide heatmap of the CrafterOOD scores previously reported in ?? .
P P OC N N
P P OS P C N
N
R e c PP O - C
N N
R e c PP O - S P C N
N
P P OS
A
P P OC A
Agents train: 100,0,0,
IID eval: 100,0,0, train: 25,25,25,
OOD eval: 0,33,33, train: 52,16,16,
OOD eval: 0,33,33, train: 76,8,8,
OOD eval: 0,33,33, train: 88,4,4,
OOD eval: 0,33,33, train: 94,2,2,
OOD eval: 0,33,33, train: 97,1,1,
OOD eval: 0,33,33, train: 100,0,0,
OOD eval: 0,33,33,
E n v i r o n m e n t s
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
CrafterOODapp score
C. Attention Visualization in Self-Attention (SA)
In this section we visualize what the CLS token attends.
Note that all the below visualizations stem from a single agent.
They are representative to what we observed in most episodes of a trained agent.
collects
It can later use
D. Learned Attention Visualization in Cross-Attention (CA)
In Figure 9 we visualize the learned attention patterns (not fixed to patches, but any slot can attend over the entire input image).
We observe more numerous, but smaller attention patterns compared to patch-based attention that is typically focused on 2-5 regions in the image.
Although in theory more powerful (because they can choose to ignore most parts of the image, or even the object at hand), in practice learned attention models underperform the patch-based ones.
We suspect the reason could be that the learned attention agents first need to find out what an object actually is, whereas patch-based ones have a more “guided” learning process.
We speculate that greater gains of learned attention would be in scenarios with objects of varying sizes, in which case a fixed grid is too rigid representation.
E. Hyper-parameter Configurations
[Table] [Table]
F. Cross-Attention Network Ablation
In this section we present an ablation study on the object-centric PPO-CA agent in Table 5.
We found vanilla versions of methods based on cross-attention (SlotAttention (Locatello et al., 2020) and Perceiver (Jaegle et al., 2021b)) not to work out of the box.
For example, these methods typically employ LayerNorm and residual MLPs, but in Table 5 we can see that the variant using these “CA + Residual MLP + LayerNorm” underperforms.
Also any of these two elements included individually (“CA + Residual MLP” and “CA + LayerNorm”) did not improve performance.
Moreover, introducing competition over slots (via a softmax over queries) “CA + Slot Competition” (akin to SlotAttention) also hurt the downstream performance.
This analysis led us to converge to an architecture that uses a single cross-attention over the input image, with no latent self-attention, no layer normalization, no residual connections and (unlike SlotAttention)
no competition between the slots via softmax over queries.
We also observed that there is a sweet spot of number of heads and number of slots where both have value 8.
Although we cannot say what the direct relationship between these network parameters and the final RL performance is, we can speculate that the higher number of slots lets each of them to specialize in parts (objects) of the input image, and higher number of heads allows for operation specialization for each of the heads.
Finally, we observe that larger patch size (with the appropriate stride) improves performance.
The learned attention over the whole input image “CA, Patch Size=1, Stride=1” does not perform as well as attention with larger patches.
Although in theory more powerful as it can attend to varying object sizes, this variant needs first to learn what parts of the image belong together into a single object.
In its current form,
Crafter objects are all of equal size, so the patches can correspond to these.
We suspect attention over the whole image (patch size of 1) would be beneficial if we would have objects with varying sizes.
[Table]
10.0 ± 0.
CA + R
[Table]
MLP 7.3 ± 0.
CA + L AYER N ORM 4.1 ± 0.
CA + R
[Table]
MLP
[Table]
N ORM 3.1 ± 0.
CA
[Table]
S LOT
[Table]
7.1 ± 0.
CA, N UMBER OF S LOTS = 8.2 ± 0.
CA, N UMBER OF S LOTS = 7.8 ± 0. CA, N UMBER OF S LOTS = 8.7 ± 0.
CA, N UMBER OF S LOTS = 10.0 ± 0.
CA, N UMBER OF S LOTS = 9.1 ± 0.
CA, N UMBER OF H EADS = 6.6 ± 0.
CA, N UMBER OF H EADS = 7.3 ± 0.
CA, N UMBER OF H EADS = 8.0 ± 0. CA, N UMBER OF H EADS = 10.0 ± 0.
CA, N UMBER OF H EADS = 7.2 ± 0.
CA, P ATCH S IZE =1, S TRIDE = 6.9 ± 0.
CA, P ATCH S IZE =8, S TRIDE = 7.2 ± 0. CA, P ATCH S IZE =12, S TRIDE = 8.7 ± 0.
CA, P ATCH S IZE =12, S TRIDE = 8.9 ± 0.
CA, P ATCH S IZE =16, S TRIDE = 6.2 ± 0.
CA, P ATCH S IZE =16, S TRIDE = 10.0 ± 0.
G. Network Configurations
[Table] [Table]
[Table] [Table]
[Table] [Table]