P ROCEDURAL GENERALIZATION BY PLANNING WITH SELF - SUPERVISED WORLD MODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
One of the key promises of model-based reinforcement learning is the ability to generalize using an internal model of the world to make predictions in novel environments and tasks.
However, the generalization ability of model-based agents is not well understood because existing work has focused on model-free agents when benchmarking generalization.
Here, we explicitly measure the generalization ability of model-based agents in comparison to their model-free counterparts.
We focus our analysis on MuZero [59], a powerful model-based agent, and evaluate its performance on both procedural and task generalization.
We identify three factors of procedural generalization—planning, self-supervised representation learning, and procedural data diversity—and show that by combining these techniques, we achieve state-of-the art generalization performance and data efﬁciency on Procgen [8].
However, we ﬁnd that these factors do not always provide the same beneﬁts for the task generalization benchmarks in Meta-World [73], indicating that transfer remains a challenge and may require different approaches than procedural generalization.
Overall, we suggest that building generalizable agents requires moving beyond the single-task, model-free paradigm and towards self-supervised model-based agents that are trained in rich, procedural, multi-task environments.
I NTRODUCTION
The ability to generalize to previously unseen situations or tasks using an internal model of the world is a hallmark capability of human general intelligence [11, 40] and is thought by many to be of central importance in machine intelligence as well [13, 25, 58, 65].
Although signiﬁcant strides have been made in model-based systems in recent years [27], the most popular model-based benchmarks consist of identical training and testing environments [e.g. 26, 68] and do not measure or optimize for for generalization at all.
While plenty of other work in model-based RL does measure generalization [e.g. 17, 47, 69, 75], each approach is typically evaluated on a bespoke task, making it difﬁcult to ascertain the state of generalization in model-based RL more broadly.
Model-free RL, like model-based RL, has also suffered from both the “train=test” paradigm and a lack of standardization around how to measure generalization.
In response, recent papers have discussed what generalization in RL means and how to measure it [6, 7, 35, 48, 70], and others have proposed new environments such as Procgen [8] and Meta-World [73] as benchmarks focusing on measuring generalization.
While popular in the model-free community [e.g. 46, 72, 77], these benchmarks have not yet been widely adopted in the model-based setting.
It is therefore unclear whether model-based methods outperform model-free approaches when it comes to generalization, how well model-based methods perform on standardized benchmarks, and whether popular modelfree algorithmic improvements such as self-supervision [14, 43] or procedural data diversity [66, 76] yield the same beneﬁts for generalization in model-based agents.
In this paper, we investigate three factors of generalization in model-based RL: planning, selfsupervised representation learning, and procedural data diversity.
We analyze these methods through a variety of modiﬁcations and ablations to MuZero Reanalyse [59, 60], a state-of-the-art modelbased algorithm.
To assess generalization performance, we test our variations of MuZero on two types of generalization (See Figure 1): procedural and task .
Procedural generalization involves evaluating agents on unseen conﬁgurations of an environment (e.g., changes in observation rendering, map or terrain changes, or new goal locations) while keeping the reward function largely the same.
Task generalization, in contrast, involves evaluating agents’ adaptability to unseen reward functions (“tasks”) within the same environment.
We focus on two benchmarks designed for both types of generalization, Procgen [8] and Meta-World [73].
[Figure]
Our results broadly indicate that self-supervised, model-based agents hold promise in making progress towards better generalization.
We ﬁnd that (1) MuZero achieves state-ofthe-art performance on Procgen and the procedural and multi-task
MetaWorld benchmarks (ML-1 and ML45 train), outperforming a controlled model-free baseline; (2) MuZero’s performance and data efﬁciency can be improved with the incorporation of self-supervised representation learning; and (3) that with selfsupervision, less data diversity is required to achieve good performance.
However, (4) these ideas help less for task generalization on the ML-45 test set from Meta-World, suggesting that different forms of generalization may require different approaches.
M OTIVATION AND B ACKGROUND 2. G ENERALIZATION IN RL
We are interested in the setting where an agent is trained on a set of n train MDPs drawn IID from the same distribution, M train =
{M i } n train i =1 , where M i ∼ p ( M ) .
The agent is then tested on another set of n test MDPs drawn IID 1 from p , but disjoint from the training set: M test =
{M j } n test j =1 where
M j ∼ p ( M ) such that M j (cid:54)∈ M train ∀ M j .
Generalization , then, is how well the agent performs in expectation on the test MDPs after training (analogous to how well a supervised model performs on the test set).
Qualitatively, generalization difﬁculty can be roughly seen as a function of the number ( n train ) of distinct MDPs seen during training (what we will refer to as data diversity ), as well as the breadth or amount of variation in p ( M ) itself.
Intuitively, if the amount of diversity in the training set is small (i.e. low values of n train ) relative to the breadth of p ( M ) , then this poses a more difﬁcult generalization challenge.
As n train → ∞ , we will eventually enter a regime where p ( M ) is densely sampled, and generalization should become much easier.
Given the above deﬁnition of generalization, we distinguish between two qualitative types of distributions over MDPs.
In procedural generalization, all MDPs with non-zero probability under p ( M ) share the same underlying logic to their dynamics (e.g., that walls are impassable) and rewards (e.g., that coins are rewarding), but differ in how the environment is laid out (e.g., different mazes) and in how it is rendered (e.g., color or background).
In task generalization, MDPs under p ( M ) share the same dynamics and rendering, but differ in the reward function (e.g., picking an object up vs. pushing it), which may be parameterized (e.g. specifying a goal location).
Procedural generalization Many recent works attempt to improve procedural generalization in different ways.
For example, techniques that have been successful in supervised learning have also been shown to help procedural generalization in RL, including regularization [8, 32, 41] and auxiliary self-supervised objectives [42, 43].
Other approaches including better hyperparameter tuning [46], curriculum learning strategies [34], and stronger architectural inductive biases [4, 22, 74] may also be effective.
However, these various ideas are often only explored in the model-free setting, and are not usually evaluated in combination, making it challenging to know which are the most beneﬁcial for model-based RL.
Here we focus explicitly on model-based agents, and evaluate three factors in combination: planning, self-supervision, and data diversity.
Task generalization Generalizing to new tasks or reward functions has been a major focus of meta-RL [18, 56], in which an agent is trained on a dense distribution of tasks during training and a meta-objective encourages few-shot generalization to new tasks.
However, meta-RL works have primarily focused on model-free algorithms [though see 47], and the distribution from which train/test MDPs are drawn is typically quite narrow and low dimensional.
Another approach to task generalization has been to ﬁrst train task-agnostic representations [71] or dynamics [63] in an exploratory pre-training phase, and then use them for transfer.
Among these, Sekar et al.
[63] focus on the model-based setting but require access to the reward function during evaluation.
In
Section 4.2, we take a similar approach of pre-training representations and using them for task transfer, with the goal of evaluating whether planning and self-supervised representation learning might assist in this.
2. F ACTORS OF G ENERALIZATION
Planning Model-based RL is an active area of research [27, 45] with the majority of work focusing on gains in data efﬁciency [25, 26, 33, 36, 60], though it is often motivated by a desire for better zero- or few-shot generalization as well [17, 28, 47, 63, 69].
In particular, model-based techniques may beneﬁt data efﬁciency and generalization in three distinct ways.
First, model learning can act as an auxiliary task and thus aid in learning representations that better capture the structure of the environment and enable faster learning [19].
Second, the learned model can be used to select actions on-the-ﬂy via MPC [17], enabling faster adaptation in the face of novelty.
Third, the model can also be used to train a policy or value function by simulating training data [65] or constructing more informative losses [20, 29], again enabling faster learning (possibly entirely in simulation).
A recent state-of-the-art agent, MuZero [59], combines all of these techniques: model learning, MPC, simulated training data, and model-based losses.
2 We focus our analysis on MuZero for this reason, and compare it to baselines that do not incorporate model-based components.
Self-supervision Model learning is itself a form of self-supervision, leveraging an assumption about the structure of MDPs in order to extract further learning signal from collected data than is possible via rewards alone.
However, recent work has argued that this is unnecessary for modelbased RL: all that should be required is for models to learn the task dynamics, not necessarily environment dynamics [21].
Yet even in the context of model-free RL, exploiting the structure of the dynamics has been shown to manifest in better learning efﬁciency, generalization, and representation learning [43, 62, 71].
Here, we aim to test the impact of self-supervision in model-based agents by focusing on three popular classes of self-supervised losses: reconstruction, contrastive, and selfpredictive.
Reconstruction losses involve directly predicting future observations [e.g. 17, 26, 36, 69].
Contrastive objectives set up a classiﬁcation task to determine whether a future frame could result from the current observation [3, 37, 51].
Finally, self-predictive losses involve having agents predict their own future latent states [24, 62].
Data diversity Several works have shown that exposing a deep neural network to a diverse training distribution can help its representations better generalize to unseen situations [15, 53].
A dense sampling over a diverse training distribution can also act as a way to sidestep the out-of-distribution generalization problem [53].
In robotics, collecting and training over a diverse range of data has been found to be critical particularly in the sim2real literature where the goal is to transfer a policy learned in simulation to the real world [2, 66].
Guez et al.
[22] also showed that in the context of zero-shot generalization, the amount of data diversity can have interesting interactions with other architectural choices such as model size.
Here, we take a cue from these ﬁndings and explore how important procedural data diversity is in the context of self-supervised, model-based agents.
2. M U Z ERO
We evaluate generalization with respect to a state-of-the-art model-based agent, MuZero [59].
MuZero is an appealing candidate for investigating generalization because it already incorporates many ideas that are thought to be useful for generalization and transfer, including replay [60], planning [64], and model-based representation learning [21, 50].
However, while these components contribute to strong performance across a wide range of domains, other work has suggested that MuZero does not necessarily achieve perfect generalization on its own [28].
It therefore serves as a strong baseline but with clear room for improvement.
MuZero learns an implicit (or value-equivalent, see Grimm et al.
[21]) world model by simply learning to predict future rewards, values and actions.
It then plans with Monte-Carlo Tree Search (MCTS) [39, 10] over this learned model to select actions for the environment.
Data collected from the environment, as well as the results of planning, are then further used to improve the learned reward function, value function, and policy.
More speciﬁcally, MuZero optimizes the following loss at every time-step t , applied to a model that is unrolled 0 . . .
K steps into the future: l t ( θ )
[Equation]
[Equation]
CE
[Equation]
ˆ
[Equation]
k , r k )) , where ˆ π k , ˆ v k and ˆ r k are respectively the policy, value and reward prediction produced by the k -step unrolled model.
The targets for these predictions are drawn from the corresponding time-step t + k of the real trajectory:
π k is the improved policy generated by the search tree, v k is an n -step return bootstrapped by a target network, and r k is the true reward.
As MuZero uses a distributional approach for training the value and reward functions [12], their losses involve computing the cross-entropy ( CE ); this also typically results in better representation learning.
For all our experiments, we speciﬁcally parameterize v and r as categorical distributions similar to the Atari experiments in Schrittwieser et al.
To enable better sample reuse and improved data efﬁciency, we use the Reanalyse version of MuZero [60].
Reanalyse works by continuously re-running MCTS on existing data points, thus computing new improved training targets for the policy and value function.
It does not change the loss function described in Section 2.3.
In domains that use continuous actions (such as Meta-World), we use
Sampled MuZero [31] that modiﬁes MuZero to plan over sampled actions instead.
See Section A. for more details on MuZero.
E XPERIMENTAL D ESIGN
We analyze three potential drivers of generalization (planning, self-supervision, and data diversity) across two different environments.
For each algorithmic choice, we ask: to what extent does it improve procedural generalization, and to what extent does it improve task generalization?
3. E NVIRONMENTS
Procgen Procgen [8] is a suite of 16 different Atari-like games with procedural environments (e.g. game maps, terrain, and backgrounds), and was explicitly designed as a setting in which to test procedural generalization.
It has also been extensively benchmarked [8, 9, 34, 54].
For each game,
Procgen allows choosing between two difﬁculty settings (easy or hard) as well as the number of levels seen during training.
In our experiments, we use the “hard” difﬁculty setting and vary the numbers of training levels to test data diversity (see below).
For each Procgen game, we train an agent for 30M environment frames.
We use the same network architecture and hyper-parameters that Schrittwieser et al.
[60] used for Atari and perform no environment speciﬁc tuning.
We report mean normalized scores across all games as in Cobbe et al.
The normalized score for each game is computed as ( score − min ) ( max − min ) , where min and max are the minimum and maximum scores possible per game as reported in [8].
We then average the normalized scores across all games to report the mean normalized score.
Meta-World Meta-World [73] is a suite of 50 different tasks on a robotic SAWYER arm, making it more suitable to test task generalization.
Meta-World has three benchmarks focused on generalization: ML-1, ML-10, and ML-45. ML-
consists of 3 goal-conditioned tasks where the objective is to generalize to unseen goals during test time.
ML-10 and ML-45 require generalization to completely new tasks (reward functions) at test time after training on either 10 or 45 tasks, respectively.
[Figure]
We that both planning and supervision
Comparing with existing state-of-the-art methods which were trained for 200M frames on the right (PPO [61], PLR [34], and UCB-DrAC+PLR [55, 34], data from [34]), we note that MuZero itself exceeds state-of-the-art performance after being trained on only 30M frames.
For all plots, dark lines indicate median performance across 3 seeds and the shaded regions denote the min and max performance across seeds.
For training curves see Figure A.3.
Meta-World exposes both state and pixel observations, as well as dense and sparse versions of the reward functions.
In our experiments, we use the v2 version of Meta-World, dense rewards, and the corner3 camera angle for pixel observations.
For Meta-World, we trained Sampled MuZero [31] for 50M environment frames.
We measure performance in terms of average episodic success rate across task(s).
Note that this measure is different from task rewards, which are dense.
3. F ACTORS OF G ENERALIZATION
Planning To evaluate the contribution of planning (see Section 2.2), we compared the performance of a vanilla MuZero Reanalyse agent with a Q-Learning agent.
We designed the Q-Learning agent to be as similar to MuZero as possible: for example, it shares the same codebase, network architecture, and replay strategy.
The primary difference is that the Q-Learning agent uses a Q-learning loss instead of the MuZero loss in Section 2.3 to train the agent, and uses (cid:15) -greedy instead of MCTS to act in the environment.
See Section A.2 for further details.
Self-supervision We looked at three self-supervised methods as auxiliary losses on top of MuZero: image reconstruction, contrastive learning [23], and self-predictive representations [62].
We do not leverage any domain-speciﬁc data-augmentations for these self-supervised methods.
Reconstruction.
Our approach for image reconstruction differs slightly from the typical use of mean reconstruction error over all pixels.
In this typical setting, the use of averaging implies that the decoder can focus on the easy to model parts of the observation and still perform well.
To encourage the decoder to model all including the hardest to reconstruct pixels, we add an additional loss term which corresponds to the max reconstruction error over all pixels.
See Section A.3 for details.
Contrastive.
We also experiment with a temporal contrastive objective which treats pairs of observations close in time as positive examples and un-correlated timestamps as negative examples [1, 51, 22].
Our implementation of the contrastive objective is action-conditioned and uses the
MuZero dynamics model to predict future embeddings.
Similar to ReLIC [44], we also use a KL regularization term in addition to the contrastive loss.
The contrastive loss operates entirely in the
[Figure]
latent space and does not need decoding to the pixel space at each prediction step, unlike the reconstruction loss.
See Section A.4 for details.
Self-predictive.
Finally, we experiment with self-predictive representations (SPR) [62] which are trained by predicting future representations of the agent itself from a target network using the
MuZero dynamics model.
Similar to the contrastive loss, this objective operates in the latent space but does not need negative examples.
See Section A.5 for details.
Data diversity To evaluate the contribution of data diversity (see Section 2.1), we ran experiments on Procgen in which we varied the number of levels seen during training (either 10, 100, 500, or ∞ ).
In all cases, we evaluated on the inﬁnite test split.
Meta-World does not expose a way to modify the amount of data diversity, therefore we did not analyze this factor in our Meta-World experiments.
R ESULTS
We ran all experiments according to the design described in Section 3.
Unless otherwise speciﬁed, all reported results are on the test environments of Procgen and Meta-World (results on the training environments are also reported in the Appendix) and are computed as medians across seeds.
4. P ROCEDURAL G ENERALIZATION
Overall results on Procgen are shown in Figure 2 and Table A.2.
MuZero achieves a meannormalized test score of 0.50, which is slightly higher than earlier state-of-the-art methods like
UCB-DrAC+PLR which was speciﬁcally designed for procedural generalization tasks [34], while also being much more data efﬁcient (30M frames vs. 200M frames).
MuZero also outperforms our Q-Learning baseline which gets a score of 0.36.
Performance is further improved by the addition of self-supervision, indicating that both planning and self-supervised model learning are important for generalization.
Effect of planning While not reaching the same level of performance of MuZero, the Q-Learning baseline performs quite well and achieves a mean normalized score of 0.36, matching performance of other specialized model-free methods such as PLR [34].
By modifying the Q-Learning baseline to learn a 5-step value equivalent model (similar to MuZero), we ﬁnd its performance further improves to 0.45, though does not quite catch up to MuZero itself (see Figure 3, left).
This suggests that while
[Figure] [Figure]
O b s e r v a t i o n
D e c o d i n g simply learning a value-equivalent model can bring representational beneﬁts, the best results come from also using this model for action selection and/or policy optimization.
We also tested the effect of planning in the Meta-World ML-1 benchmark from states.
Table A.
and
Figure A.6 show the results.
We ﬁnd that both the Q-Learning and MuZero agents achieve perfect or near-perfect generalization performance on this task, although MuZero is somewhat more stable and data-efﬁcient.
The improvement of MuZero in stability or data efﬁciency again suggests that model-learning and planning can play a role in improving performance.
Effect of self-supervision Next, we looked at how well various self-supervised auxiliary losses improve over MuZero’s value equivalent model on Procgen (Figure 3, right).
We ﬁnd that contrastive learning and self-predictive representations both substantially improve over MuZero’s normalized score of 0.50 to 0.63 (contrastive) or 0.64 (SPR), which are new state-of-the-art scores on Procgen.
The reconstruction loss also provides beneﬁts but of a lesser magnitude, improving MuZero’s performance from 0.50 to 0.57.
All three self-supervised losses also improve the data efﬁciency of generalization.
Of note is the fact that a MuZero agent with the contrastive loss can match the ﬁnal performance of the baseline MuZero agent using only a third of the data (10M environment frames).
To further tease apart the difference between MuZero with and without self-supervision, we performed a qualitative comparison between reconstructed observations of MuZero with and without image reconstruction.
The vanilla MuZero agent was modiﬁed to include image reconstruction, but with a stop gradient on the embedding so that the reconstructions could not inﬂuence the learned representations.
We trained each agent on two games, Chaser and Climber.
As can be seen in Figure 4, it appears that the primary beneﬁt brought by self-supervision is in learning a more accurate world model and capturing more ﬁne-grained details such as the position of the characters and enemies.
In Chaser, for example, the model augmented with a reconstruction loss is able to to predict the position of the character across multiple time steps, while MuZero’s embedding only retains precise information about the character at the current time step.
This is somewhat surprising in light of the arguments made for value-equivalent models [21], where the idea is that world models do not need to capture full environment dynamics—only what is needed for the task.
Our results suggest that in more complex, procedurally generated environments, it may be challenging to learn even the task dynamics from reward alone without leveraging environment dynamics, too.
Further gains in model quality might be gained by also properly handling stochasticity [52] and causality [57].
Effect of data diversity Overall, we ﬁnd that increased data diversity improves procedural generalization in Procgen, as shown in Figure 5.
Speciﬁcally, at 10 levels, self-supervision barely improves the test performance on Procgen over MuZero.
But as we increase the number of levels to 100, 500, or ∞ , we observe a substantial improvement when using self-supervision.
This is an interesting ﬁnding which suggests that methods that improve generalization might show promise only when we evaluate them on environments with a lot of inherent diversity, such as procedurally gener-
[Figure]
ated environments.
Consequently, evaluating methods solely on single task settings (as is common in RL) might lead us to overlook innovations which might have a big impact on generalization.
Not only do we ﬁnd that MuZero with self-supervision achieves better ﬁnal generalization performance than the baselines, we also observe that self-supervision improves generalization performance even when controlling for training performance .
To see this, we visualized generalization performance as a function of training performance (see Figure A.2).
The ﬁgure shows that even when two agents are equally strong (according to the rewards achieved during training), they differ at test time, with those trained with self-supervision generally achieving stronger generalization performance.
This suggests that in addition to improving data efﬁciency, self-supervision leads to more robust representations—a feature that again might be overlooked if not measuring generalization.
4. T ASK G ENERALIZATION
[Figure]
[Table]
Note that MAML and RL 2 were trained for around 400M environment steps from states, whereas MuZero was trained from pixels for 50M on
10M for ﬁne-
As we have shown, planning, selfsupervision, and data diversity all play an important role in procedural generalization.
We next investigated whether this trend holds for task generalization, too.
To test this, we pre-trained the Q-Learning and
MuZero agents with and without selfsupervision on the ML-10 and MLtraining sets of Meta-World and then evaluated zero-shot test performance
[Table]
In all experiments
[Table]
we trained agents from pixels using the dense setting of the reward functions.
We report the agent’s success rate computed the same way as in Yu et al.
As shown in Table 1, we can observe that MuZero reaches better training performance on ML10 (97.6%) and ML-45 (77.2%) compared to existing state-of-the-art agents [73].
These existing approaches are meta-learning methods, and are therefore not directly comparable in terms of test performance due to different data budgets allowed at test time; although MuZero does not reach the same level of performance as these agents at test, we ﬁnd it compelling that it succeeds as often as it does, especially after being trained for only 50M environment steps (compared to 400M for the baselines).
MuZero also outperforms Q-Learning in terms of zero-shot test performance, indicating a positive beneﬁt of planning for task generalization.
However, reconstruction and other forms of self-supervision do not improve performance and may even decrease it (see also Figure A.7).
[Figure]
We also looked at whether the pre-trained representations or dynamics would assist in more dataefﬁcient task transfer by ﬁne-tuning the agents on the test tasks for 10M environment steps.
Figureshows the results, measured as cumulative regret (see also Figure A.7 for learning curves).
Using pre-trained representations or dynamics does enable both Q-Learning and MuZero to outperform corresponding agents trained from scratch—evidence for weak positive transfer to unseen tasks for these agents.
Additionally, MuZero exhibits better data efﬁciency than Q-Learning, again showing a beneﬁt for planning.
However, self-supervision again does not yield improvements in ﬁne-tuning, and as before may hurt in some cases (Figure A.7).
This indicates that while MuZero (with or without self-supervision) excels at representing variation across tasks seen during training, there is room for improvement to better transfer this knowledge to unseen reward functions.
We hypothesize that MuZero only exhibits weak positive transfer to unseen tasks due to a combination of factors.
First, the data that its model is trained on is biased towards the training tasks, and thus may not be sufﬁcient for learning a globally-accurate world model that is suitable for planning in different tasks.
Incorporating a more exploratory pre-training phase [e.g. 63] might help to alleviate this problem.
Second, because the model relies on task-speciﬁc gradients, the model may over-represent features that are important to the training tasks (perhaps suggesting that valueequivalent models [21] may be poorly suited to task generalization).
Third, during ﬁnetuning, the agent must still discover what the reward function is, even if it already knows the dynamics.
It is possible that the exploration required to do this is the primary bottleneck for task transfer, rather than the model representation itself.
Meta-learning methods like those benchmarked by Yu et al.
[73] may be implicitly learning better exploration policies [67], suggesting a possible reason for their stronger performance in few-shot task generalization.
Finally, MuZero only gets 15 frames of history, and increasing the size of its memory might also facilitate better task inference.
C ONCLUSION
In this paper, we systematically investigate how well modern model-based methods perform at hard generalization problems, and whether self-supervision can improve the generalization performance of such methods.
We ﬁnd that in the case of procedural generalization in Procgen, both model-based learning and self-supervision have additive beneﬁts and result in state-of-the-art performance on test levels with remarkable data efﬁciency.
In the case of task generalization in Meta-World, we ﬁnd that while a model-based agent does exhibit weak positive transfer to unseen tasks, auxiliary selfsupervision does not provide any additional beneﬁt, suggesting that having access to a good world model is not always sufﬁcient for good generalization [28].
Indeed, we suspect that to succeed at task generalization, model-based methods must be supplemented with more sophisticated online exploration strategies, such as those learned via meta-learning [16, 38, 67] or by leveraging world models in other ways [63].
Overall, we conclude that self-supervised model-based methods are a promising starting point for developing agents that generalize better, particularly when trained in rich, procedural, and multi-task environments.
R EPRODUCIBILITY S TATEMENT
In this paper we have taken multiple steps to ensure that the algorithms, experiments, and results are as reproducible as possible.
Both environments used in our experiments, Procgen and MetaWorld, are publicly available.
In order to produce reliable results, we ran multiple seeds in our key experiments.
Our choice of agent is based on MuZero, which is published work [31, 59, 60].
In addition, we have included an extensive appendix (Appendix A) which describes the architectural details of MuZero, the exact hyperparameters used for all experiments and a detailed description of the Q-Learning agents and the self-supervised losses.
A A GENT D ETAILS A. M U Z ERO
Network Architecture: We follow the same network architectures follow the ones used in
MuZero ReAnalyse [60].
For pixel based inputs, the images are ﬁrst sent through a convolutional stack that downsamples the 96 × 96 image down to an 8 × 8 tensor (for Procgen) or a 16 ×tensor (for Meta-World).
This tensor then serves as input to the encoder.
Both the encoder and the dynamics model were implemented by a ResNet with 10 blocks, each block containing 2 layers.
For pixel-based inputs (Procgen and ML-45) each layer of the residual stack was convolutional with a kernel size of 3x3 and 256 planes.
For state based inputs (ML-1) each layer was fully-connected with a hidden size of
[Table]
Hyperparameters We list major hyper-parameters used in this work in Table A.
[Table]
[Table]
Additional Implementation Details: For Meta-World experiments, we provide the agent with past reward history as well.
We found this to be particularly helpful when training on Meta-World since implicit task inference becomes easier.
In both Procgen and Meta-World, the agent is given a history of the 15 last observations.
The images are concatenated by channel and then input as one tensor to the encoder.
For each game, we trained our model using 2 TPUv3-8 machines.
A separate actor gathered environment trajectories with 1 TPUv3-8 machine.
A. C ONTROLLED M ODEL - FREE B ASELINE
Our controlled Q-Learning baseline begins with the same setup as the MuZero agent (Section A.1) and modiﬁes it in a few key ways to make it model-free rather than model-based.
The Q-Learning baseline uses n -step targets for action value function.
Given a trajectory { s t , a t , r t } Tt =0 , the target action value is computed as follow
[Equation]
Where Q ξ is the target network whose parameter ξ is updated every 100 training steps.
In order to make the model architecture most similar to what is used in the MuZero agent, we decompose the action value function into two parts: a reward prediction ˆ r and a value prediction
V , and model these two parts separately.
The total loss function is, therefore, L total =
L reward +
L value .
The reward loss is exactly the same as that of MuZero.
For the value loss, we can decompose
Equation 1 in the same way:
[Equation]
Since the reward prediction should be taken care of by L reward and it usually converges fast, we assume ˆ r t = r t and the target is simpliﬁed to Equation 2.
We can then use this value target to compute the value loss L value =
CE ( V target ( s ) , V ( s )) .
In Meta-World, since it has a continuous action space, maximizing over the entire action space is infeasible.
We follow the Sampled Muzero approach [31] and maximize only over the sampled actions.
The Q-Learning baseline uses 5-step targets for computing action values.
This is the same as in
MuZero, but unlike MuZero, the Q-Learning baseline only trains the dynamics for a single step.
However, we also provide results for a 5-step dynamics function which we call QL+Model and
QL+Model+Recon which further adds an auxiliary for reconstruction.
A. M U Z ERO + R ECONSTRUCTION
The decoder architecture mirrors the ResNet encoder with 10 blocks, but with each convolutional layer replaced by a de-convolutional layer [49].
In addition to the regular mean reconstruction loss, we add an additional max-loss term which computes the maximum reconstruction loss across all pixels.
This incentives the reconstruction objective to not neglect the hardest to reconstruct pixels, and in turn makes the reconstructions sharper.
The reconstruction loss can be used as follows:
[Equation]
where (cid:96) ( x, y )
=
( x − y ) 2 is the element-wise squared distance between each feature (i.e. RGB) value in the image array, where i and j are the pixel indices, where f is the feature index, and where | X | is the size of the array.
A. M U Z ERO + C ONTRASTIVE
Our implementation of the contrastive loss does not rely on any image augmentations but focuses on the task of predicting future embeddings of the image encoder [3, 51].
Howerver, in this case, the dynamics function produces a predicted vector x n at each step n in the unroll, making the contrastive task action-conditional similar to CPC | Action [23].
We utilize the same target network used in train temporal difference in MuZero to compute our target vectors.
We then attempt to match x n with the corresponding target vector y n directly computed from the observation at that timestep.
We compute the target vector y n with the same target network weights used to compute the temporal difference loss in MuZero.
Since the contrastive loss is ultimately a classiﬁcation problem, we also need negative examples.
For each pair x n and y n , the sets { x i } Ni =0 ,i (cid:54) = n and { y i } Ni =0 ,i (cid:54) = n serve as the negative examples with N the set of randomly sampled examples in the minibatch.
We randomly sample 10% of the examples to constitute N .
As in previous work [3, 5, 44] we use a cosine distance between x i and y i and feed this distance metric through a softmax function.
We use the loss function in ReLIC [44] and CoBERL [3] and use a temperature of 0.1.
The 256-dimensional vector x n is derived from a two layer critic function which is ﬁrst a convolutional layer with stride 1 and kernel size 3 on top of the state of the dynamics function.
This is then fed into a fully connected layer to produce x n .
For each step n , the target vector y n , which serves as a positive example, is derived from the ﬁnal convolutional layer ( 8 ) that is fed into the residual stack of the encoder.
The vector is computed by inputting the encoder, using the target network weights, with the corresponding image at that future step n .
The encoding is then fed into the critic network to compute y n .
The encoder of MuZero uses 15 past images as history.
However, when we compute the target vectors, our treatment of the encoding history is different from that of the agent; instead of stacking all of the historical images ( n − 15 ...n − 1 ) up to the corresponding step n , we simply replace the history stack with 15 copies of the image at the current step n .
We found that this technique enhanced the performance of contrastive learning.
A. M U Z ERO + SPR
The implementation of SPR is similar to the contrastive objective except that it does not use negative examples.
Thus it only uses a mean squared error instead of InfoNCE.
The architecture for SPR is nearly identical to what is described in Section A.4.
The same layers mentioned compute predicted and target vectors x n and y n respectively.
However, SPR relies on an additional projection layer for learning.
We thus add a projection layer p n of 256 dimensions on top of the critic function mentioned in Section A.4.
Instead of computing a loss between x n and y n , we follow the original formulation of SPR and compute mean squared error between L2-normalized p n and L2-normalized y n .
B A DDITIONAL R ESULTS B. C OMPARISON OF Q-L EARNING AND M U Z ERO ON P ROCGEN
As discussed in the main text, we trained a 5-step value-equivalent model on top of the Q-Learning agent.
The performance of this agent with and without supervision, as well as the baseline QLearning agent, is reported in Table A.2 and visualized in Figure A.1.
Both incorporating the 5-step model and adding reconstruction loss helps the Q-Learning agent’s performance.
When using model-based planning, it’s important to ensure that the model is “correct”.
Otherwise, planning on an “incorrect” model could hurts the agent performance, as we can see for the last few games in Figure A.1.
Speciﬁcally, we can see that on these games, MuZero performs far worse than Q-Learning; yet, its performance is dramatically improved by the addition of the reconstruction loss.
This suggests that without self-supervision, MuZero’s model is poor, resulting in compounding model errors during planning.
Figure 4 provides a qualitative analysis of MuZero’s model in two of these games, conﬁrming this hypothesis.
[Figure]
[Table]
train
[Table]
train
[Table] [Table]
[Table]
dodgeball train
[Table]
fruitbot train
[Table]
train
[Table]
[Table]
jumper train
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure] [Figure]
i s
[Figure]
a v e f l y e r l e a p e
[Figure]
m
[Figure]
e h e
i s t
c o i n r un m i n e r f r u i t
[Figure]
o t c li m
[Figure]
e r j u m p e
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Table]
Q-
[Figure] [Table]