O N THE ROLE OF PLANNING IN MODEL - BASED DEEP REINFORCEMENT LEARNING
A BSTRACT
Model-based planning is often thought to be necessary for deep, careful reasoning and generalization in artiﬁcial agents.
While recent successes of model-based reinforcement learning (MBRL) with deep function approximation have strengthened this hypothesis, the resulting diversity of model-based methods has also made it difﬁcult to track which components drive success and why.
In this paper, we seek to disentangle the contributions of recent methods by focusing on three questions: (1) How does planning beneﬁt MBRL agents?
(2) Within planning, what choices drive performance?
(3) To what extent does planning improve generalization?
To answer these questions, we study the performance of MuZero [58], a state-of-the-art MBRL algorithm with strong connections and overlapping components with many other MBRL algorithms.
We perform a number of interventions and ablations of MuZero across a wide range of environments, including control tasks, Atari, and 9x9 Go.
Our results suggest the following: (1) Planning is most useful in the learning process, both for policy updates and for providing a more useful data distribution.
(2) Using shallow trees with simple Monte-Carlo rollouts is as performant as more complex methods, except in the most difﬁcult reasoning tasks.
(3) Planning alone is insufﬁcient to drive strong generalization.
These results indicate where and how to utilize planning in reinforcement learning settings, and highlight a number of open questions for future MBRL research.
Model-based reinforcement learning (MBRL) has seen much interest in recent years, with advances yielding impressive gains over model-free methods in data efﬁciency [12, 15, 25, 76], zero- and few-shot learning [16, 37, 60], and strategic thinking [3, 62,
These methods combine planning and learning in a variety of ways, with planning speciﬁcally referring to the process of using a learned or given model of the world to construct imagined future trajectories or plans.
Some have suggested that models will play a key role in generally intelligent artiﬁcial agents [14, 50, 55, 56, 57, 67], with such arguments often appealing to model-based aspects of human cognition as proof of their importance [24, 26, 28, 41].
While the recent successes of MBRL methods lend evidence to this hypothesis, there is huge variance in the algorithmic choices made to support such advances.
For example, planning can be used to select actions at evaluation time [e.g., 12] and/or for policy learning [e.g., 34]; models can be used within discrete search [e.g., 58] or gradient-based planning [e.g., 25, 29]; and models can be given [e.g., 45] or learned [e.g., 12].
Worryingly, some works even come to contradictory conclusions, such as that long rollouts can hurt performance due to compounding model errors in some settings [e.g., 34], while performance continues to increase with search depth in others [58].
Given the inconsistencies and non-overlapping choices across the literature, it can be hard to get a clear picture of the full MBRL space.
This in turn makes it difﬁcult for practitioners to decide which form of MBRL is best for a given problem (if any).
The aim of this paper is to assess the strengths and weaknesses of recent advances in MBRL to help clarify the state of the ﬁeld.
We systematically study the role of planning and its algorithmic design choices in a recent state-of-the-art MBRL algorithm, MuZero [58].
Beyond its strong performance,
MuZero’s use of multiple canonical MBRL components (e.g., search-based planning, a learned model, value estimation, and policy optimization) make it a good candidate for building intuition about the roles of these components and other methods that use them.
Moreover, as discussed in the next section, MuZero has direct connections with many other MBRL methods, including Dyna [67],
MPC [11], and policy iteration [33].
To study the role of planning, we evaluate overall reward obtained by MuZero across a wide range standard MBRL environments: the DeepMind Control Suite [70], Atari [8], Sokoban [51], Minipacman [22], and 9x9 Go [42].
Across these environments, we consider three questions.
(1) For what purposes is planning most useful?
Our results show that planning—which can be used separately for policy improvement, generating the distribution of experience to learn from, and acting at testtime—is most useful in the learning process for computing learning targets and generating data.
What design choices in the search procedure contribute most to the learning process?
We show that deep, precise planning is often unnecessary to achieve high reward in many domains, with two-step planning exhibiting surprisingly strong performance even in Go.
(3) Does planning assist in generalization across variations of the environment—a common motivation for model-based reasoning?
We ﬁnd that while planning can help make up for small amounts of distribution shift given a good enough model, it is not capable of inducing strong zero-shot generalization on its own.
B ACKGROUND AND R ELATED W ORK
Model-based reinforcement learning (MBRL)
[9, 26, 47, 49, 74] involves both learning and planning.
For our purposes, learning refers to deep learning of a model, policy, and/or value function.
Planning refers to using a learned or given model to construct trajectories or plans.
In most MBRL agents, learning and planning interact in complex ways, with better learning usually resulting in better planning, and better planning resulting in better learning.
Here, we are interested in understanding how differences in planning affect both the learning process and test-time behavior.
act update observe plan
MBRL classiﬁed - time planning, which use the model to select actions, and background planning, which use the model to update a policy [68].
For example, model-predictive control (MPC)
[11] is a classic decision-time planning method that uses the model to optimize a sequence of actions starting from the current environment state.
Decisiontime planning methods often feature robustness to uncertainty and fast adaptation to new scenarios [e.g., 76], though may be insufﬁcient in settings which require longterm reasoning such as in sparse reward tasks or strategic games like Go.
Conversely, Dyna [67] is a classic background planning method which uses the the model to simulate data on which to train a policy via standard modelfree methods like Q-learning or policy gradient.
Background planning methods often feature improved data efﬁciency over model-free methods [e.g., 34], but exhibit the same drawbacks as model-free approaches such as brittleness to out-of-distribution experience at test time.
A number of works have adopted hybrid approaches combining both decision-time and background planning.
For example, Guo et al.
[23], Mordatch et al.
[48] distill the results of a decision-time planner into a policy.
Silver et al.
[61], Tesauro & Galperin [71] do the opposite, allowing a policy to guide the behavior of a decision-time planner.
Other works do both, incorporating the distillation or imitation step into the learning loop by allowing the distilled policy from the previous iteration to guide planning on the next iteration, illustrated by the “update” arrow in Figure 1.
This results in a form of approximate policy iteration which can be implemented both using single-step [40, 54] or multi-step [17, 18] updates, the latter of which is also referred to as expert iteration [3] or dual policy iteration [66].
Such algorithms have succeeded in board games [3, 4, 63, 64], discrete-action
MDPs [27, 51, 58] and continuous control [43, 45, 65].
In this paper, we focus our investigation on MuZero [58], a state-of-the-art member of the approximate policy iteration family.
MuZero is a useful testbed for our analysis not just because of its strong performance, but also because it exhibits important connections to many other works in the
MBRL literature.
For example, MuZero implements a form of approximate policy iteration [18] and has close ties to MPC in that it uses MPC as a subroutine for acting.
There is also an interesting connection to recent Dyna-style MBRL algorithms via regularized policy optimization: speciﬁcally, these methods simulate data from the model and then update the policy on this data using TRPO [34, 39, 46, 52].
Recent work by Grill et al.
[20] showed that the MuZero policy update approximates
TRPO, making the learning algorithm implemented by these Dyna-style algorithms quite similar to that implemented by MuZero.
Finally, the use of value gradients [10, 19, 25, 29] leverages model gradients to compute a policy gradient estimate; this estimate is similar (without regularization) to the estimate that MCTS converges to when used as a policy iteration operator [20, 75].
Thus, our results with MuZero have implications not just for its immediate family but to MBRL methods more broadly.
In contrast to other work in MBRL, which focuses primarily on data efﬁciency [e.g., 34, 36] or model learning [e.g., 12], our primary concern in this paper is in characterizing the role of planning with respect to reward after a large but ﬁxed number of learning steps, as well as to zero-shot generalization performance (however, we have also found our results to hold when performing the same experiments and measuring approximate regret).
We note that MuZero can also be used in a more data-efﬁcient manner (“MuZero Reanalyze”) by using the search to re-compute targets on the same data multiple times [58], but leave exploration of its behavior in this regime to future work.
Our analysis joins a number of other recent works that seek to better understand the landscape of
MBRL methods and the implications of their design choices.
For example, Chua et al.
[12] perform a careful analysis of methods for uncertainty quantiﬁcation in learned models.
Other research has investigated the effect of deep versus shallow planning [32, 35], the utility of parametric models in
Dyna over replay [73], and benchmark performance of a large number of popular MBRL algorithms in continuous control tasks [74].
Our work is complementary to these prior works and focuses instead on the different ways that planning may be used both during training and at evaluation.
P RELIMINARIES : O VERVIEW OF M U Z ERO
MuZero uses a learned policy and learned value, transition and reward models within Monte-Carlo tree search (MCTS) [13, 38] both to select actions and to generate targets for policy learning (Figure 1).
We provide a brief overview of MuZero here and refer readers to Appendix A and Schrittwieser et al.
[58] for further details.
Algorithm 1 and 2 present pseudocode for MuZero and MCTS, respectively.
Model MuZero plans in a hidden state space using a learned model µ θ parameterized by θ and comprised of three functions.
At timestep t , the encoder embeds past observations into a hidden state, s 0 t
= h θ ( o 1 , . . .
, o t ) .
Given a hidden state and an action in the original action space, the (deterministic) recurrent dynamics function predicts rewards and next states, r kθ,t , s kt =
g θ ( s k − 1 t , a k − 1 t ) , where k is the number of imagined steps into the future starting from a real observation at time t .
In addition, the prior (not used in the classic Bayesian sense of the term) predicts a policy and value for a given hidden state, π k θ,t , v k θ,t = f θ ( s k t ) , and is used to guide the tree search.
Search Beginning at the root node s 0 t , each simulation traverses the search tree according to a search policy until a previously unexplored action is reached.
The search policy is a variation on the pUCT rule [53, 38] that balances exploitation and exploration, and incorporates the prior to guide this (see Equation 1 in Section A.3).
After selecting an unexplored action a (cid:96) at state s (cid:96)t , the tree is expanded by adding a new node with reward r (cid:96) +1 θ,t , state s (cid:96) +1 t , policy π (cid:96) +1 θ,t , and value v (cid:96) +1 θ,t predicted by the model.
The value and reward are used to form a bootstrapped estimate of the cumulative discounted reward, which is backed up to the root, updating the estimated return Q and visit count N of each node on the path.
After B simulations, MCTS returns a value v MCTS t (the average cumulative discounted reward at the root) and policy π MCTS t (a function of the normalized count of the actions taken at the root during search).
Acting After search, an action is sampled from the MCTS policy, a t ∼ π MCTS t , and is executed in the environment to obtain reward r env t .
Data from the search and environment are then added to a replay buffer for use in learning: { o t , a t , r env t , π MCTS t , v MCTS t } .
Learning The model is jointly trained to predict the reward, policy, and value for each future timestep k = 0 . . .
K .
The reward target is the observed environment reward, r env t + k .
The policy target is the MCTS-constructed policy π MCTS t + k .
The value target is the n -step bootstrapped discounted return z t = r env t +1 + γr env t +2 + · · · + γ n − 1 r env t + n + γ n v MCTS t + n .
For reward, value, and policy losses (cid:96) r , (cid:96) v , and (cid:96) p , respectively, the overall loss is then (cid:96) t ( θ ) =
(cid:80) Kk =
(cid:96) r
( r kθ,t , r env t + k ) + (cid:96) v ( v kθ,t , z t + k ) + (cid:96) p ( π kθ,t , π MCTS t + k ) .
Note that MuZero is not trained to predict future observations or hidden states: the learning signal for the dynamics comes solely from predicting future rewards, values, and policies.
We do, however, perform additional experiments in Section D.2 in which we train MuZero to also predict observations via an additional reconstruction loss; our results indicate that the observationbased model is slightly more accurate but does not qualitatively change MuZero’s behavior.
H YPOTHESES AND E XPERIMENTAL M ETHODS
Our investigation focuses on three key questions: (1) How does planning drive performance in
MuZero? (2) How do different design choices in the planner affect performance?
(3) To what extent does planning support generalization?
To answer these questions, we manipulated a number of different variables across our experiments (Figure 2, see also Section B.1): the maximum depth we search within the tree ( D tree ), the maximum depth we optimize the exploration-exploitation tradeoff via pUCT
( D UCT ), the search budget ( B ), the model (learned or environment simulator), and the planning algorithm itself (MCTS or breadth-ﬁrst search).
(1) Overall contributions of planning Performance in many model-free RL algorithms is driven by computing useful policy improvement targets.
We hypothesized that, similarly, using the search for policy improvement (as opposed to exploration or acting) is a primary driver of MuZero’s performance, with the ability to compare the outcome of different actions via search enabling even ﬁner-grained and therefore more powerful credit assignment.
To test this hypothesis, we implemented several variants of MuZero which use search either for learning, for acting, or both.
When learning, we compute π MCTS either using full tree search ( D tree = ∞ ) or using only one step of lookahead ( D tree = 1 ).
When acting, we sample actions either from the policy prior π θ,t or from π MCTS computed using D tree = ∞ .
These choices allow us to deﬁne the following variants (see table in Figure 3): “One-Step”, which trains and uses the model in a one-step way to generate learning targets, similar to having a Q-function; “Learn”, which uses the search only for learning; “Data”, which uses the search only to select actions during training while using the model in a one-step way to generate targets; “Learn+Data”, which uses the search both for learning and for action selection during training; and “Learn+Data+Eval”, which corresponds to full MuZero.
See Section B.2 for more details about these variants.
[Figure]
(2) Planning for learning One feature of MCTS is its ability to perform “precise and sophisticated lookahead” [58].
To what extent does this lookahead support learning stronger policies?
We hypothesized that more complex planning like tree search—as opposed to simpler planning, like random shooting—and deeper search is most helpful for learning in games like Go and Sokoban, but less helpful for the other environments.
To test this, we manipulated the tree depth ( D tree ), UCT depth ( D UCT ), and search budget 1 ( B ).
Note that our aim with varying
D UCT is to evaluate the effect of simpler versus more complex planning, rather than the effect of exploration.
(3) Generalization in planning Model-based reasoning is often invoked as a way to support generalization and ﬂexible reasoning [e.g., 26, 41].
We similarly hypothesized that given a good model, planning can help improve zero-shot generalization.
First, we evaluated the ability of the individual model components to generalize to new usage patterns.
Speciﬁcally, we evaluated pre-trained agents using: larger search budgets than seen during training, either the learned model or the environment simulator, and either MCTS or breadth-ﬁrst search (see Section B.4).
Second, we tested generalization to unseen scenarios by evaluating pre-trained Minipacman agents of varying quality (assessed by the number of unique mazes seen during training) on novel mazes drawn from the same or different distributions as in training.
R ESULTS
We evaluated MuZero on eight tasks across ﬁve domains, selected to include popular MBRL environments with a wide range of characteristics including episode length, reward sparsity, and variation of initial conditions.
First, we included two Atari games [8] which are commonly thought to require long-term coordinated behavior: Ms. Pacman and Hero .
We additionally included Minipacman [51], a toy version of Ms. Pacman which supports procedural generation of mazes.
We also included two strategic games that are thought to heavily rely on planning: Sokoban [22, 51] and 9x9 Go [42].
Finally, because much work in MBRL focuses on continuous control [e.g., 74], we also included three tasks from the DeepMind Control Suite [70]:
Acrobot (Sparse Swingup),
Cheetah (Run), and Humanoid (Stand).
We discretized the action space of the control tasks as in
Tang & Agrawal [69], Grill et al.
Three of these environments also exhibit some amount of stochasticity and partial observability: the movement of ghosts in Minipacman is stochastic;
Go is a two-player game and thus stochastic from the point of view of each player independently; and using a limited number of observation frames in Atari makes it partially observable (e.g., it is not possible to predict when the ghosts in Ms. Pacman will change from edible to dangerous).
Further details of all environments are available in Appendix C.
4. B ASELINES
1S
L
D
L+D L+D+E 0.
N o r m a li z ed
R e [Figure]
Sokoban
[Figure]
Act
Test Act 1S
D tree =
π θ,t π θ,t L
D tree = ∞
π θ,t π θ,t D
D tree = D tree =
∞
π θ,t L+D
D tree = ∞ D tree = ∞
π θ,t L+D+E
D tree = ∞
D tree = ∞ D tree = ∞
Before beginning our analysis, we tuned
MuZero for each domain and ran baseline experiments with a search budget of B = 10 simulations in Minipacman, B = 25 in Sokoban,
B = 150 in 9x9 Go, and B = 50 in all other environments.
Additional hyperparameters for each environment are available in Appendix C and learning curves in Section D.3; unless otherwise speciﬁed, all further experiments used the same hyperparameters as the baselines.
We obtained the following median ﬁnal scores, computed using the last 10% of steps during training (median across ten seeds): 620.07 on Acrobot, 885.57 on Cheetah, 787.
on Humanoid, 29916.97 on Hero, 45346.71 on
Ms. Pacman, 310.1 on Minipacman, 0.97 on
Sokoban (proportion solved), and 0.72 on 9x
Go (proportion games won against Pachi 10k [7], a bot with strong amateur play).
These baselines are all very strong, with the Atari and
Control Suite results being competitive with state-of-the-art [22, 31, 58].
4. O VERALL C ONTRIBUTIONS OF P LANNING
We ﬁrst compared vanilla MuZero to different variants which use search in varying ways, as described in Section 3.
To facilitate comparisons across environments, we normalized scores to lie between the performance attained by a randomly initialized policy (Table 8) and the full version of MuZero (Table 7).
Figure 3 shows the results.
Across environments, the “One-Step” variant has a median strength of 46.7% ( N = 80 ).
[Figure]
Strategy
[Figure]
Although this variant is not entirely model-free, it does remove much of the dependence on the model, thus establishing a useful minimal-planning baseline to compare against.
Using a deeper search solely to compute policy updates (“Learn”) improves performance to 68.5% ( N = 80 ).
The “Data” variant—where search is only used to select actions—similarly improves over “One-Step” to 66.7% ( N = 80 ).
These results indicate both the utility in training a multi-step model, and that search may also drive performance by enabling the agent to learn from a different state distribution resulting from better actions (echoing other recent work leveraging planning for exploration [45, 60]).
Allowing the agent to both learn and act via search during training (“Learn+Data”) further improves performance to a median strength of 90.3% ( N = 80 ).
Finally, search at evaluation (“Learn+Data+Eval”) brings the total up to 100%.
Using the search for learning, for acting during training, and for acting at test time appears to provide complementary beneﬁts (see also Table 12).
This conﬁrms our hypothesis that an important beneﬁt of search is for policy learning, but also highlights that search can have positive impacts in other ways, too.
Indeed, while some environments beneﬁt more from a model-based learning signal (Cheetah, Sokoban, Go) others beneﬁt more from model-based data (Hero, Minipacman, Humanoid, Ms. Pacman, Acrobot).
Combining these two uses enables MuZero to perform well across most environments (though Hero and Minipacman exhibit their best performance with “Data”; this effect is examined further in Section 4.3).
Search at evaluation time only provides a small boost in most environments, though it occasionally proves to be crucial, as in Acrobot and Go.
Overall, using search during training (“Learn+Data”) is the primary driver of MuZero’s performance over the “One-Step” baseline, leveraging both better policy targets and an improved data distribution.
As discussed in Section 1, MuZero has important connections to a number of other approaches in MBRL; consequently, our results here suggest implications for some of these other methods.
For example, the “Learn” variant of MuZero is similar to Dyna-style methods that perform policy updates with TRPO [34, 39, 46, 52].
Given the improved performance of “Learn+Data” over “Learn”, these methods may similarly beneﬁt from also using planning to select actions, and not just for learning.
The “Data” variant of MuZero is similar in spirit to Dyna-2 [61]; our results suggest that works which implement this approach [5, 6] may similarly beneﬁt from incorporating planning into policy and value updates.
“Learn+Data+Eval” shows the contribution of MPC over “Learn+Data”,
[Figure]
and combined with our results in Section 4.4, highlights the importance of guiding MPC with robust policy priors and/or value functions.
Of course, one major difference between MuZero and all these other methods is that MuZero uses a value equivalent model which is not grounded in the original observation space [21].
We therefore reran some of our experiments using a model that is trained to reconstruct observations in order to measure any potential implications of this difference, but did not ﬁnd that such models change the overall pattern of results (see Section D.2 and Figure 13-14).
4. P LANNING FOR L EARNING
Tree depth Figure 4a shows the result of varying tree depth D tree ∈ { 1 , 2 , 3 , 5 , ∞} while keeping
D UCT
= D tree and the search budget constant.
Scores are normalized by the “Learn+Data” agent from Section 4.2.
Strikingly, D tree does not make much of a difference in most environments.
Even in Sokoban and Go, we can recover reasonable performance using D tree = 2 , suggesting that deep tree search may not be necessary for learning a strong policy prior, even in the most difﬁcult reasoning domains.
Looking at individual effects within each domain, we ﬁnd that deep trees have a negative impact in Minipacman and an overall positive impact in Ms. Pacman, Acrobot, Sokoban, and Go (see Table 13).
While we did not detect a quantitative effect in the other environments, qualitatively it appears as though very deep trees may cause worse performance in Hero.
Exploration vs. exploitation depth Figure 4b shows the strength of the policy prior as a result of manipulating the pUCT depth D UCT ∈ { 1 , 2 , 3 , 6 , ∞} while keeping D tree = ∞ and the search budget constant.
Note that D UCT = 1 corresponds to only exploring with pUCT at the root node and performing pure Monte-Carlo sampling thereafter.
We ﬁnd D UCT to have no effect in any environment except 9x9 Go (Table 14); Anthony et al.
[4] also found larger values of D UCT to be important for Hex.
Thus, exploration-exploitation deep within the search tree does not seem to matter at all except in the most challenging settings.
Search budget Figure 4c shows the strength of the policy prior after training with different numbers of simulations, with D tree = ∞ and D UCT = 1 (except in Go, where D UCT = ∞ ), corresponding to exploring different actions at the root node and then performing Monte-Carlo rollouts thereafter.
We opted for these values as they correspond to a simpler form of planning, and our previous experiments showed that larger settings of D UCT made little difference.
We ﬁnd an overall strong effect of the number of simulations on performance in all environments (Table 15).
However, despite the overall positive effect of the search budget, too many simulations have a detrimental effect in many
[Figure]
In-distribution generalization is shown in Figure 12 (Appendix) as the behavior is similar.
environments, replicating work showing that some amount of planning can be beneﬁcial, but too much can harm performance [e.g., 34].
Additionally, the results with Ms. Pacman suggest that two simulations provide enough signal to learn well in some settings.
It is possible that with further tuning, other environments might also learn effectively with smaller search budgets.
4. G ENERALIZATION IN P LANNING
Model generalization to new search budgets Figure 5a shows the results of evaluating the baseline agents (Section 4.1) using up to 625 simulations.
As before, we ﬁnd a small but signiﬁcant improvement in performance of 7.4 percentage points between full MuZero and agents which do not use search at all ( t = − 5 .
84 , p < 0 .
001 , N = 70 ).
Both Acrobot and Sokoban exhibit slightly better performance with more simulations, and although we did not perform experiments here with Go,
Schrittwieser et al.
[58] did and found a positive impact.
However, Minipacman exhibits worse performance, and other environments show no overall effect (Table 16).
The median reward obtained across environments at 625 simulations is also less than the baseline by a median of 4.7 percentage points ( t = − 5 .
71 , p < 0 .
001 , N = 70 ), possibly indicating an effect of compounding model errors.
This suggests that for identical training and testing environments, additional search may not always be the best use of computation; we speculate that it might be more worthwhile simply to perform additional Dyna-like training on already-observed data [see “MuZero Reanalyze”, 58].
Policy and value generalization with a better model Planning with the simulator yields somewhat better results than planning with the learned model (Figure 5b), with all environments except Hero exhibiting positive rank correlations with the number of simulations (Table 17).
Ms. Pacman, in particular, more than doubles in performance after 625 simulations 2 .
However, across environments, 25, 125, and 625 simulations only increased performance over the baseline by a median of aboutpercentage points.
Thus, planning with a better model may only produce small gains.
Model generalization to new planners
We ﬁnd dramatic differences between MCTS and BFS, with BFS exhibiting a catastrophic drop in performance with any amount of search.
This is true both when using the learned model (Figure 11, Appendix) and the simulator (Figure 5c), in which case the only learned component that is relied on is the value function.
Consider the example of Sokoban, where the branching factor is ﬁve; therefore, ﬁve steps of BFS search means that all actions at the root node are expanded, and the action will be selected as the one with the highest value.
Yet, the performance of this agent is substantially worse than just relying on the policy prior.
This suggests a mismatch between the value function and the policy prior, where low-probability (off-policy) actions are more likely to have high value errors, thus causing problems when expanded by BFS.
Moreover, this problem is not speciﬁc to BFS: in sparse reward environments, any planner (such as random shooting) that relies on the value function without the policy prior will suffer from this effect.
This result highlights that in complex agent architectures involving multiple learned components, compounding error in the transition model is not the only source of error to be concerned about.
Generalizing to new mazes We trained Minipacman agents on 5, 10, or 100 unique mazes and then tested them on new mazes drawn either from the same distribution or a different distribution.
Figure 6 shows the out-of-distribution results and Figure 12 the in-distribution results.
Using the learned model, we see very slight gains in performance up to 125 simulations on both in-distribution and out-of-distribution mazes, with a sharp drop-off in performance after that reﬂecting compounding model errors in longer trajectories.
The simulator allows for somewhat better performance, with greater improvements for small numbers of train mazes ( t = − 10 . 43 , p < 0 .
001 , N = 360 , see also Table 18 and 19), indicating the ability of search to help with some amount of distribution shift when using an accurate model.
However, as can be seen in the ﬁgure, this performance plateaus at a much lower value than what would be obtained by training the agent directly on the task.
Moreover, reward using the simulator begins to decrease at 3125 simulations compared to at 125 simulations using the learned model ( t = − 5 .
59 , p < 0 .
001 , N 1 = 20 , N 2 = 20 ), again indicating a sensitivity to errors in the value function and policy prior.
In fact, this is the same effect as can be seen in Figure 5c; the only reason the drop-off in performance is less drastic than with BFS is because the policy prior used by MCTS keeps the search more on-policy.
D ISCUSSION
In this work, we explored the role of planning in MuZero [58] through a number of ablations and modiﬁcations.
We sought to answer three questions: (1) In what ways does planning contribute to ﬁnal performance?
(2) What design choices within the planner contribute to stronger policy learning?
(3) How well does planning support zero-shot generalization?
In most environments, we ﬁnd that (1) search is most useful in constructing targets for policy learning and for generating a more informative data distribution; (2) simpler and shallower planning is often as performant as more complex planning; and (3) search at evaluation time only slightly improves zero-shot generalization, and even then only if the model is highly accurate.
Although all our experiments were performed with MuZero, these results have implications for other MBRL algorithms due to the many connections between MuZero and other approaches (Section 1).
A major takeaway from this work is that while search is useful for learning, simple and shallow forms of planning may be sufﬁcient.
This has important implications in terms of computational efﬁciency: the algorithm with D UCT
= 1 can be implemented without trees and is thus far easier to parallelize than MCTS, and the algorithm with D tree = 1 can be implemented via model-free techniques [e.g., 1], suggesting that MBRL may not be necessary at all for strong ﬁnal performance in some domains.
Moreover, given that search seems to provide minimal improvements at evaluation in many standard RL environments, it may be computationally prudent to avoid using search altogether at test time.
The result that deep or complex planning is not always needed suggests that many popular environments used in MBRL may not be fully testing the ability of model-based agents (or RL agents in general) to perform sophisticated reasoning.
This may be true even for environments which seem intuitively to require reasoning, such as Sokoban.
Indeed, out of all our environments, only Acrobot and 9x9 Go strongly beneﬁted from search at evaluation time.
We therefore emphasize that for work which aims to build ﬂexible and generalizable model-based agents, it is important to evaluate on a diverse range of settings that stress different types of reasoning.
Our generalization experiments pose a further puzzle for research on model-based reasoning.
Even given a model with good generalization (e.g., the simulator), search in challenging environments is ineffective without a strong value function or policy to guide it.
Indeed, our experiments demonstrate that if the value function and policy themselves do not generalize, then generalization to new settings will also suffer.
But, if the value function and policy do generalize, then it is unclear whether a model is even needed for generalization.
We suggest that identifying good inductive biases for policies which capture something about the world dynamics [e.g., 6, 22], as well as learning appropriate abstractions [30], may be as or more important than learning better models in driving generalization.
Overall, this work provides a new perspective on the contributions of search and planning in integrated MBRL agents like MuZero.
We note that our analysis has been limited to single-task and (mostly) fully-observable, deterministic environments, and see similar studies focusing on multi-task and more strongly partially-observed and stochastic environments as important areas for future work.
A CKNOWLEDGMENTS
We are grateful to Ivo Danihelka, Michal Valko, Jean-bastien Grill, Eszter V´ertes, Matt Overlan, Tobias Pfaff, David Silver, Nate Kushman, Yuval Tassa, Greg Farquhar, Loic Matthey, Andre Saraiva,
Florent Altch´e and many others for helpful comments and feedback on this project
A. M ODEL DETAILS
In MuZero, the model µ θ is trained to directly predict three quantities for each future timestep k = 1 . . .
K .
These are the policy π kθ,t ≈ π MCTS ( a t + k | o 1 , . . .
, o t , a t , . .
. , a t + k − 1 ) , the value function v kθ,t ≈ E (cid:2) r env t + k +
+ γr env t
+ k +
+ . . .
| o 1 , . .
, o t , a t , . .
. , a t + k − 1 (cid:3) , and the immediate reward r kθ,t ≈ r env t
+ k , where π MCTS is the policy used to select actions in the environment, r env is the observed reward, and γ is the environment discount factor.
A. MCTS DETAILS
Pseudocode for MCTS is presented in Algorithm 2.
MuZero uses the pUCT rule [53] for the search policy.
The pUCT rule maximizes an upper conﬁdence bound [38] to balance exploration and exploitation during search.
Speciﬁcally, the pUCT rule selects action
[Equation]
where N is the number of times a has been selected during search, Q is the average cumulative discounted reward of a , and c 1 , c 2 are constants that control the relative inﬂuence of Q and π θ .
Following Schrittwieser et al.
[58], we set c 1 = 1 . 25 and c 2 = 19652 in our experiments.
At the root node alone, a small amount of Dirichlet noise is added to the policy prior π θ to encourage additional exploration at the root.
1: function MCTS(root state s 0 , model µ θ , number of simulations B ) 2: initialize edge statistics { ( N ( s 0 , a ) , Q ( s 0 , a )) } a to3: for k = 1 . . .
B do 4: r l , s l , v l ← S EARCH A ND E XPAND ( s 0 ) 5:
B ACKUP ( r l , s l , v l ) 6: end for 7: return π MCTS and v MCTS (Equation 6 and 7) 8: end function 9: function S EARCH A ND E XPAND ( s ) 10: while true do 11: a ← P UCT ( s ) (cid:46) choose action according to pUCT 12: r (cid:48) , s (cid:48) ← T RANSITION ( s, a ) (cid:46) use cached values if possible 13:
if N ( s, a ) = 0 then 14: add node s (cid:48) as child of s on edge a 15: initialize edge statistics { ( N ( s (cid:48) , a (cid:48) ) , Q ( s (cid:48) , a (cid:48) ))
} a (cid:48) to16: compute π θ , v θ ← f θ ( s (cid:48) ) 17: return r (cid:48) , s (cid:48) , v θ 18: end if 19: s ← s (cid:48) 20: end while 21: end function 22: function B ACKUP ( r (cid:48) , s (cid:48) , v (cid:48) ) 23: for each edge on the path from s (cid:48) to the root s 0 do 24: update statistics N, Q with Equation 3 and25: end for 26: end function
Within the search tree, each node has an associated hidden state s .
For each action a from s there is an edge ( s, a ) on which the number of visits N
( s, a ) and the current value estimate Q ( s, a ) are stored.
When a new node with state s is created in the expansion step, the statistics for each edge are initialized as { N ( s, a ) = 0 , Q ( s, a ) = 0 } .
The estimated policy prior π θ ( s, a ) , reward r θ , and state transition are also stored after being computed on expansion since they are deterministic and can be cached.
Thus, the model only needs to be evaluated once per simulation when the new leaf is added.
When backing up after expansion, the statistics on all of the edges from the leaf to the root are updated.
Speciﬁcally, for k = l . . .
0 , a bootstrapped l − k -step cumulative discounted reward estimate
[Equation]
is computed.
The statistics for each edge ( s k , a k ) for k = 0 . . .
l − 1 in the simulation path are updated as
[Equation]
To keep Q estimates bounded within [0 , 1] , the Q estimates are ﬁrst normalized as ¯ Q ∈ [0 , 1] before passing them to the pUCT rule.
The normalized estimates are computed as
[Equation] [Equation]
where Q min = min ( s,a ) ∈
Tree Q ( s, a ) and Q max = max ( s,a
) ∈
Tree Q ( s, a ) are the minimum and maximum Q values observed in the search tree so far.
After all simulations are complete, the policy π MCTS returned by MCTS is the visit count distribution at the root s 0 parameterized by a temperature T
[Equation]
During training, the temperature is set as a function of the number of learner update steps.
Speciﬁcally, the temperature is set to 1 and then decayed by a factor of 0 .
after every 5000 steps.
The value v MCTS returned by MCTS is the average discounted return over all simulations
[Equation]
A. T RAINING DETAILS
The MCTS policy is used to select an action a t ∼ π MCTS t , which is then executed in the environment and a reward r env t observed.
The model is jointly trained to match targets constructed from the observed rewards and the MCTS policy and value for each future timestep k .
The policy targets are simply the MCTS policies, while the value targets are the n -step bootstrapped discounted returns
[Equation]
t
[Equation]
n .
For reward, value, and policy losses (cid:96) r , (cid:96) v , and (cid:96) p , respectively, the overall loss is then
[Equation]
where c ||
θ
|| 2 is an L2 regularization term.
For the rewards, values, and policies, a cross-entropy loss is used for each of (cid:96) r , (cid:96) v , and (cid:96) p .
B F URTHER I MPLEMENTATION D ETAILS B. D EPTH - LIMITED MCTS
The depth-limited MCTS algorithm is implemented by replacing the regular MCTS search and expand subroutines with a modiﬁed subroutine as follows.
The backup subroutine remains unchanged.
Algorithm
MCTS.
1: function S EARCH A ND E XPAND (root state s 0 , max UCT depth D UCT , max tree depth D tree ) 2: k ←3: while k < D tree and s k is not a leaf do (cid:46) Search for leaf node 4: a ← S ELECT A CTION ( s k , k, D UCT ) 5:
s k +1 ← T RANSITION ( s k , a ) 6:
k ← k +7: end while 8: if k < D tree then (cid:46)
Expand unless maximum depth is reached 9: a ← S ELECT A CTION ( s k , k, D UCT ) 10: s k +1 ← TRANSITION ( s k , a ) 11:
Add s k +1 to tree 12: end if 13: return s k + 14: end function 15: function S ELECT A CTION ( s, k, D UCT ) (cid:46)
The search policy 16: if k < D UCT then 17: a ← P UCT ( s ) (cid:46) Choose action according to pUCT 18: else 19: a ∼ π θ ( ·| s ) (cid:46)
Sample from prior 20: end if 21: return a 22: end function
We initially tried varying D tree and D UCT together on Ms. Pacman and Minipacman.
However, as we did not see any effect, we varied these variables separately for the remainder of our experiments in order to limit computation.
We did not run any experiments with D UCT = 0 , which corresponds to pure Monte-Carlo search.
This is because this variant introduces a confound: with D
UCT = 0 , the visit counts are no longer informative and unsuitable for use as policy learning targets.
To test D UCT
= 0 would therefore also require modifying the learning target.
Future work could test this by using the same MPO update described in the next section.
Additionally, we note that depth-limited MCTS will converge to the BFS solution (Section B.4) in the limit of inﬁnite simulations.
For ﬁnite simulations, depth-limited MCTS interpolates between the policy prior and the BFS policy.
B. M U Z ERO V ARIANTS
As described in Section 3, we implemented the following variants of MuZero:
• One-Step .
This variant uses D tree = 1 for learning, and acts by sampling from the policy prior.
We also only train the model to predict one time step into the future (in all other variants, we train it to predict ﬁve time steps into the future).
This variant is therefore as close as possible to a model-free version of MuZero, and could in principle be implemented solely with a Q-function.
• Learn .
This variant uses D tree = ∞ for learning, and acts by sampling from the policy prior.
It therefore gets the beneﬁts of a deep tree search for learning, but not for acting.
• Data .
This variant uses D tree = 1 for learning, acts during training by sampling from π MCTS (computed using D tree = ∞ ), and acts at test time from the policy prior.
This allows us to measure the impact of deep search on the distribution of data experienced during learning.
• Learn+Data .
This variant using D tree = ∞ both for learning and for acting during training; actions at test time are sampled from the policy prior.
This variant thus beneﬁts from search in two ways during learning, but uses no search at test time.
• Learn+Data+Eval .
This variant is equivalent to the original version of MuZero, using search (with D tree = ∞ ) for learning and for all action selection.
Another way to test the impact of search on learning and the data distribution would be to add a secondary model-free loss function to MuZero, making it easier to fully ablate the model-based loss function in the One-Step and Data variants.
While we leave this exploration to future work, we mention that Hamrick et al.
[27] investigated another member of the approximate policy iteration family which relies solely on Q-functions and which combines a model-free loss with a model-based loss.
Thus, they were able to test the effect of a purely model-free variant, as well as a “Data”-like variant, both of which underperformed the full version of their model.
B. R EGULARIZED POLICY UPDATES (MPO) WITH MCTS
Past work has demonstrated that MuZero’s policy targets suffer from degeneracies at low visit counts [27, 20].
To account for this, we modiﬁed the policy targets to use an MPO-style update [1] rather than the visit count distribution, similar to [20]: π MPO t + k ∝ π kθ,t · exp (cid:0) q MCTS /τ (cid:1) .
Here, q MCTS are the Q-values at the root node of the search tree; τ = 0 .
1 is a temperature parameter; and we use π MPO t + k in place of π MCTS t + k in Equation 8.
Note that the Q-values for unvisited actions are set to zero; while it is in general a poor estimate for the true Q-function, we found this choice to outperform setting the Q-function for unvisited actions to the value function.
This is likely because unvisited actions are unlikely under the prior and perhaps ought not be reinforced unless good estimates are obtained through exploration.
However, this choice leads to a biased MPO update; how to unbias it will be a topic of further research.
Similarly, we chose the MPO update for its ease of implementation, but it is likely other forms of regularized policy gradient (e.g. TRPO [59], or more generally natural or mirror policy optimization [72, 2]) would result in quantitively similar ﬁndings.
We also did not tune τ for different environments; it is likely that properly tuning it could further improve performance of the agent at small search budgets.
B. B READTH - FIRST SEARCH
In our generalization experiments, we replaced the MCTS search with a breadth-ﬁrst search algorithm.
BFS explores all children at a particular depth of the tree in an arbitrary order before progressing deeper in the tree.
Our implementation of BFS does not use the policy prior.
Additionally, when performing backups, we compute the maximum over all values seen rather than averaging.
Final actions are selected based on the highest Q-value after search, rather than highest visit count.
Thus, this implementation of BFS is (1) maximally exploratory and (2) relies mostly on the value estimates v θ,t (especially when the simulator is used instead of the learned model).
C E NVIRONMENT AND A RCHITECTURE D ETAILS
We evaluate on the following environments.
• Minipacman : a toy version of Ms. Pacman with ghosts that move stochastically.
We modiﬁed the version introduced by [22] such that the maze is procedurally generated on each episode.
See Section C.1 for details.
• Hero (Atari): a sparse reward, visually complex video game.
The goal is to navigate a character through a mine, clearing cave-ins, destroying enemies, and rescuing trapped miners.
• Ms. Pacman (Atari): a fast-paced, visually complex video game.
The goal is to control Pacman to eat all the “food” in a maze, while avoiding being eaten by ghosts.
• Acrobot Sparse Swingup (Control Suite): a low-dimensional, yet challenging control task with sparse rewards.
The task is to balance upright an under-actuated double pendulum.
• Cheetah Run (Control Suite): a six-dimensional control task, where the goal is to control the joints of a “cheetah” character to make it run forward in a 2D plane.
• Humanoid Stand (Control Suite): a 21-dimensional control task, where the goal is to control the joints of a humanoid character to make it stand up.
• Sokoban : a difﬁcult puzzle game that involves pushing boxes onto targets and in which incorrect moves can be unrecoverable [51].
• 9x9 Go
: an easier version of Go than the full 19x19 game, provided by [42].
Evaluation is reported against Pachi [7], with 10 4 evaluations per move (strong amateur play).
We now provide speciﬁc details on the network architectures and hyperparameters used for each environment.
Unless speciﬁed and for layers where it is appropriate, all layers in the networks use padding ‘SAME’ and stride 1 , and convolutions are 2-D with 3 × 3 kernels.
All networks consist of an encoder h θ , a recurrently-applied dynamics function g θ , and a prior function f θ .
For most environments, the encoder h θ is a resnet composed of a number of segments.
Each segment consists of a convolution, a layer norm, a number of residual blocks, and a ReLU.
Each residual block contains a layer norm, a ReLU, a convolution, a layer norm, a ReLU, and a ﬁnal convolution.
The input of the residual block is then added to the output of its ﬁnal convolution.
For most environments, the dynamics network has the same structure as the encoder, with a different number of segments and residual blocks.
Finally, the prior function predicts three quantities and is composed of three separate networks: a policy head, a value head, and a reward head.
The policy, value, and reward heads each consist of a 1 × 1 convolution followed by a number of linear layers, with a ReLU between each pair of layers.
The hyperparameters in Table 1 are shared across all environments except Go.
C. M INIPACMAN
Minipacman is a toy version of Ms. Pacman.
Unlike the Atari game, Minipacman also exhibits a small amount of stochasticity in that the ghosts move using an epsilon-greedy policy towards the agent.
[Table]
C.1. M AZES
For Minipacman, we altered the environment to support the use of both procedurally generated mazes (“in-distribution” mazes) and the standard maze (“out-of-distribution” mazes), both of size 15x19.
Figure 6 shows example mazes of both types.
In all experiments except generalization, we trained agents on an unlimited number of the “in-distribution” mazes, and tested on other mazes also drawn from this set.
For the generalization experiments, we trained agents on a ﬁxed set of either 5, 10, or 100 “in-distribution” mazes.
Then, at evaluation time, we either tested on mazes drawn from the full in-distribution set or from the out-of-distribution set.
To generate the procedural mazes, we ﬁrst used Prim’s algorithm to generate corridors.
To make the maze more navigable, we then randomly removed walls with a probability of p = 0 .
The number of initial ghosts was sampled as g 0 ∼ 1 + Poisson(1) and this number increased by g ∆ ∼ 0 .
+
U (0 , 1) ghosts per level, such that the total number of ghosts at level l was g l = (cid:98) g
+ ( l − 1) g ∆ (cid:99) .
The number of pills was always set to 4.
The default Minipacman maze was hand crafted to be similar to the Ms. Pacman maze.
In the default maze, there is always one initial ghost ( g 0 = 1 ) and this number increases by 1 every two levels ( g ∆ = 0 . 5 ).
We similarly set the number of pills to 4.
In all mazes, the initial locations of the ghosts, pills, and Pacman is randomly chosen at the beginning of every episode.
C.1. N ETWORK ARCHITECTURE
For Minipacman, the encoder has 2 segments, each with 64 channels and 2 residual blocks.
The dynamics function has 1 segment with 5 residual blocks, all with 64 channels.
The policy head has a convolution with 4 channels and one linear layer with a channel per action (in Minipacman, this is 5 ).
The value head has a convolution with 32 channels and two linear layers with 64 andchannels.
The reward network has the same structure as the value head.
All Minipacman experiments were run using 400 CPU-based actors and 1 NVIDIA V100 for the learner.
[Table]
[Figure]
The Atari Learning Environment [8] is a challenging benchmark of 57 classic Atari 2600 games played from pixel observations.
We evaluate on Ms. Pacman and Hero.
Each observation consists of the 4 previous frames and action repeats is 4 .
Note that, as MuZero does not incorporate recurrence, this limited number of frames makes some aspects of the the environment partially observable (such as when the ghosts turn from being edible to inedible).
For Atari, the encoder consists of 4 segments with (64 , 128 , 128 , 128) channels, each with 2 residual blocks, followed by 1 segment with 5 residual blocks with 128 channels.
The dynamics network has 1 segment with 5 residual blocks with 128 channels.
The heads are the same as in Minipacman.
All Atari experiments were run using 1024 CPU-based actors and 4 NVIDIA V100s for the learner.
[Table]
[Figure]
The DeepMind Control Suite [70] is a widely used benchmark for control tasks in MuJoCo.
We select 3 high-dimensional environments Cheetah (Run), Acrobot (Swing-up Sparse) and Humanoid (Stand).
We use the raw state observation as input and follow the same procedure as [20, 69] to discretize the continuous action space.
Each action dimension is discretized into 5 bins evenly spaced between [ − 1 , 1] .
For Control Suite environments, we use a model based on that of [44].
The encoder consists of a linear layer with 300 channels, a layer norm, a tanh, a linear layer with 200 channels, and an exponential linear unit (ELU).
The dynamics function is simply a linear layer with 200 channels followed by an ELU.
The weights in the encoder and dynamics function are initialized uniformly.
The policy head is a factored policy head composed of a linear layer with # dimensions × # bins channels.
The factored policy head independently chooses an action for each dimension.
The value and reward heads are both composed of a linear layer with 64 channels followed by a ReLU and then a linear layer with 2001 channels.
All Control Suite experiments were run using 1024 CPU-based actors and 2 second-generation (v2)
Tensor Processing Units (TPUs) for the learner.
[Table]
Sokoban [51] is a classic puzzle problem, where the agent’s task is to push a number of boxes onto target locations.
In this environment many moves are irreversible as the boxes can only be pushed forward and hence the puzzle can become unsolvable if wrong moves are made.
For Sokoban, the encoder is the same as that used for Minipacman with an additional 256 -channel 1 × 1 convolutional layer at the end.
Instead of a resnet, for the dynamics network for Sokoban we used a DRC(3, 1) convolutional LSTM [22].
For the heads, we use the same network as in
Minipacman except that the convolutional layers in the policy, value, and reward heads have 32 , 32 , and 16 channels, respectively.
All Sokoban experiments were run using 2048 CPU-based actors and 4 NVIDIA V100s for the learner.
C. 9 X 9 G O
9x
Go is a smaller version of the full 19x19 game.
While the board is fully observed, the actions of the other player cannot be fully predicted, thus making the game stochastic from each players’ perspective.
For the Go experiments, we used a different implementation of the MuZero algorithm due to easier interfacing with the Go environment.
The main difference between the implementation used for other environments and the one for Go is the data pipeline.
In the ﬁrst implementation, actors and
[Table]
learner communicate asynchronously through a replay buffer.
In the one used for Go, there is no replay buffer; instead, actors add their data to a queue which the learner then consumes.
Additionally, while the implementation of MuZero used in the other experiments trained the value function using n -step returns, z t = r env t +1 + γr env t +2 + · · · + γ n − 1 r env t + n + γ n v MCTS t + n (see Section 2), the one used here uses lambda returns, z λt = (1 − λ ) (cid:80) ∞ n =1 λ n − 1 z t : t + n .
The two player aspect of the game is handled entirely by having a single player playing both moves, but using a discount of − 1 .
The input to the agent is the last two states of the Go board and one color plane, where each state is encoded relative to color of the player with 3 planes, own stones, opponent’s stones and empty stones.
The encoder consists of a single convolution layer with 128 channels, followed by 6 residual 3 ×convolutional blocks with 128 channels (each block consists of two convolutional layers with a skip connection).
The network is size-preserving so hidden states are of size 9x9.
The transition model takes as input the last hidden state and the action encoded as a one-hot plane, and also consists in 6 residual convolutional blocks with 128 channels.
The representation model consists in one 1 ×convolutional layer with 2 channels, followed by an MLP with a single layer of 256 units.
[Table]
D A DDITIONAL R ESULTS AND A NALYSIS D. E XTENSIONS TO FIGURES IN MAIN PAPER
Figure 10 shows the same information as Figure 3 (contributions of search to performance) but split into separate groups and with error bars shown.
Figure 11 shows the same information as Figure(effect of search at evaluation as a function of the number of simulations) but using breadth-ﬁrst search with a learned model.
Figure 12 shows the same information as Figure 6 (effect of search on generalization to new mazes) but for the in-distribution mazes instead of the out-of-distribution mazes.
Figure 22 presents learning curves for Go for different values of D UCT and numbers of simulations.
[Figure] [Figure]
[Figure]
D. M U Z ERO WITH OBSERVATION - RECONSTRUCTION LOSS
As discussed above, planning in MuZero is performed entirely in a hidden space.
This hidden space has no direct semantic association with the environment state as its sole purpose is to enable prediction of future rewards, policies, and values.
This is in contrast, however, to much other work on model-based reinforcement learning, for which the model explicitly predicts environment states or observations.
To investigate the effect of tying the model to the environment, we added an observation-reconstruction loss to our MuZero implementation and re-ran a subset of the experiments from Section 4.2 using this loss.
D.2. O VERALL CONTRIBUTIONS OF PLANNING WITH RECONSTRUCTION LOSS
For two pixel-based environments (Minipacman and Sokoban), we added a binary cross-entropy loss between the true pixel observations and predicted future observations.
Similarly, for the three statebased environments (Acrobot, Cheetah, and Humanoid), we added an L 2 loss between the true state observations and the predicted future states.
We evaluated the effect of the reconstruction losses on these environments, for each of the “Learn,” “Learn+Data,” and “Learn+Data+Eval“ MuZero variants of Section 4.2.
Final performance results are shown in Figure 13 and learning curves are shown in Figure 14.
[Figure]
Variants without the reconstruction loss are the same as those shown in Figure 10.
As shown by these ﬁgures, the performance of MuZero is relatively unchanged when using a reconstruction loss.
The two main effects of the reconstruction loss, as shown by these ﬁgures, are (1) that learning consistently takes off faster when using the reconstruction loss (except in Humanoid) and (2) that the reconstruction loss allows the “Learn” variant of Acrobot to obtain reward.
This latter point indicates that the model learned by “Learn” without the reconstruction loss is impoverished in a way that causes compounding errors with forward credit assignment, as discussed in van Hasselt et al.
Overall, our results indicate that learning a model in observation space does not fundamentally alter the role of planning in MuZero but it does improve the ﬁdelity of the model, which can beneﬁt planning.
Given the connections between MuZero and MBRL in general, we postulate that this conclusion holds true for other MBRL methods as well.
[Figure]
L
L+obs
L+D
L+D+obs
L+D+E
L+D+E+obs
[Figure]
Cheetah
L
L+obs
L+D
L+D+obs
L+D+E
L+D+E+obs
[Figure]
Humanoid
L
L+obs
L+D
L+D+obs
L+D+E
L+D+E+obs
[Figure]
Learner
[Figure]
[Figure]
Mini PM
L
L+obs
L+D
L+D+obs
L+D+E
L+D+E+obs
[Figure]
D.
O BSERVATION - RECONSTRUCTION LOSS IMPLEMENTATION DETAILS
To implement the reconstruction loss, we added a reconstruction function ˆ o kθ,t = ρ θ ( s kt ) to the model µ θ that takes in the hidden state s t and generates a predicted observation ˆ o kθ,t for the k th imagined step after timestep t .
The predicted observation is then fed into an additional term (cid:96) ρ (ˆ o kt , o t + k ) in the MuZero loss (Equation 8).
The parameters and architectures were lightly tuned on a subset of these environments.
For the pixel-based environments (i.e., Minipacman and Sokoban), ρ θ is a simple 3 -layer convolutional network with 3 × 3 kernels, 128 channels per layer, and ReLU activations feeding into a linear layer with 3 channels and a sigmoid activation.
The loss (cid:96) ρ is the binary cross-entropy loss.
Similarly, for the state-based environments (i.e., Acrobot, Cheetah, and Humanoid), ρ θ is a simple feed-forward network with 3 linear layers of size [200 , 100 , | O | ] and ELU activations, where | O | is the size of the state vector.
The output of this is fed into (cid:96) ρ , which is an L 2 loss.
D. B ASELINE LEARNING CURVES
Figures 15, 16 17, 18, 19, 20, 21, 22 show the learning curves for each environment for each experiment in the main body.
In all plots, shaded regions indicate minimum and maximum seeds, while lines show the median across seeds.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
D. B ASELINE VALUES
[Table] [Table]
[Table] [Table]
Sokoban 0.
D. O VERALL CONTRIBUTIONS OF PLANNING
[Table]
[Table]
TrainAct (MB - MF) t = 5 .
11 , p < 0 .
001 , N {+1 = 240 , N 
+}
=
TestAct (MB - MF) t = 6 . 83 , p < 0 .
001 , N 1 = 80 , N 2 =
[Table]
individual Spearman rank correlations between reward and log( D UCT ) for each environment.
p -values are adjusted for multiple comparisons using the Bonferroni correction.
The main effects are primarily driven by Go
[Table]
[Table] [Table] [Table]
F (7 , 433) = 45 . 45 , p < 0 .
*** log
[Table]
B )
F (1 , 433) = 593 . 90 , p < 0 .
*** log( B )
F (1 , 433) = 121 .
70 , p < 0 .
Environment: log( B )
F (7 , 433) = 11 .
64 , p < 0 .
Acrobot ρ = 0 .
76 , p < 0 .
001 , N =
[Table]
ρ = 0 .
76 , p < 0 .
001 , N =
Humanoid ρ = 0 . 88 , p < 0 .
001 , N =
Hero ρ = 0 .
75 , p < 0 .
001 , N =
Ms. Pacman ρ = 0 .
61 , p < 0 .
001 , N =
Minipacman ρ = 0 .
70 , p < 0 .
001 , N =
Sokoban ρ = 0 . 87 , p < 0 .
001 , N =
[Table]
ρ = 0 .
48 , p = 0 .
05 , N =.
D. P LANNING FOR GENERALIZATION
[Table] [Table]
[Table] [Table]