Structure and randomness in planning and reinforcement learning
Abstract
Planning in large state spaces inevitably needs to balance depth and breadth of the search.
It has a crucial impact on planners performance and most manage this interplay implicitly.
We present a novel method Shoot Tree Search (STS) , which makes it possible to control this trade-off more explicitly.
Our algorithm can be understood as an interpolation between two celebrated search mechanisms: MCTS and random shooting.
It also lets the user control the bias-variance trade-off, akin to TD ( n ) , but in the tree search context.
In experiments on challenging domains, we show that STS can get the best of both worlds consistently achieving higher scores.
Introduction
Classically, reinforcement learning is split into model-free and model-based methods.
Each of these approaches has its strengths and weaknesses: the former often achieves state-of-the-art performance, while the latter holds the promise of better sample efﬁciency and adaptability to new situations.
Interestingly, in both paradigms, there exists a non-trivial interplay between structure and randomness.
In the model-free approach, Temporal Difference (TD) prediction leverages the structure of function approximators, while Monte Carlo (MC) prediction relies on random rollouts.
Model-based methods often employ planning, which counterfactually evaluates future scenarios.
The design of a planner can lean either towards randomness, with random rollouts used for state evaluation (e.g. random shooting), or towards structure, where a data-structure, typically a tree or a graph, forms a backbone of the search (e.g. Monte Carlo Tree Search).
We present a novel method: Shoot Tree Search (STS).
The development of the algorithm was motivated by the aforementioned observations concerning structure, randomness, and dilemma between breadth and depth of the search.
It lets the user control the depth and breadth of the search more explicitly and can be viewed as a bias-variance control method.
STS itself can be understood as an interpolation between MCTS and random shooting.
We show experimentally that, on a diverse set equal of environments, STS can get the best of both worlds.
We also provide some toy environments, to get an insight into why STS can be expected to perform well.
The critical element of STS, multi-step expansion , can be easily implemented on top of many algorithms from the MCTS family.
As such, it can be viewed as one of the extensions in the MCTS toolbox.
Background and related work
The introduction to reinforcement learning can be found in Sutton and Barto [16].
In contemporary research, the line between model-free and model-based methods is often blurred.
An early example is Guo et al.
[6], where MCTS plays the role of an ‘expert’ in DAgger (Ross and Bagnell [12]), a policy learning algorithm.
In the series of papers Silver et al.
[14, 15], culminating in AlphaZero, the authors developed a system combining elements of combinatorial algorithms and reinforcement learning methods that master the game of Go (and others).
Similar ideas were also studied in Anthony et al.
In Miło´s et al.
[10], planning and model-free learning were brought together to solve combinatorial environments.
As most of these works, we use the model-based reinforcement learning paradigm, in which the agent has access to a true model of the environment.
Methods
A Generic Planner, presented in Algorithm 1, gives a uniﬁed view on all methods analyzed in the paper: Random Shooting, MCTS and, our novel approach, STS.
By a suitable choice of functions
SELECT , EXPAND , UPDATE and CHOOSE _ ACTION , we can recover each of these methods.
Typically, a planner is a part of a training process, see Algorithm 2.
In a positive feedback loop, the planner improves the quality of data used for training of the value function V θ and a policy π φ .
Conversely, the policy and value function might further improve planning.
Implementation details of
Algorithm 2 are provided in Appendix A.1.
In the Appendixes A.4 and A.5 we give a detailed description of the Random Shooting and MCTS planning methods.
Below we present our novel planner.
Algorithm 1 Generic Planner, deﬁnes required constants, variables and objects used in further algorithms
Require:
C planning passes H planning horizon γ discount factor
Use:
N ( s, a ) visit count
W ( s, a ) total action-value
Q ( s, a ) mean action-value
V θ value function π φ policy model environment simulator # Initialize N, W, Q to zero function PLANNER ( state ) for 1 . . .
C do path , leaf ← SELECT ( state ) rollout , leaf ← EXPAND ( leaf )
UPDATE ( path , rollout , leaf ) return CHOOSE _ ACTION ( state )
Algorithm 2 Training loop, additionally environment
# Initialize parameters of V θ , π φ # Initialize replay _ buffer repeat
episode ← COLLECT _ EPISODE replay _ buffer.
ADD ( episode )
B ← replay _ buffer.
BATCH
Update V θ , π φ using B and SGD until convergence function COLLECT _ EPISODE s ← env.
RESET episode ← [] repeat a ← PLANNER (s) s (cid:48) , r ← env.
STEP ( a ) episode .
APPEND (( s, a, r, s (cid:48) )) s ← s (cid:48) until episode is done return CALCULATE _ TARGET ( episode )
Shoot Tree Search Shoot Tree Search (STS) extends MCTS in a novel way, by redesigning the expansion phase, see Algorithm 3.
Given a leaf and a planning horizon H the method expands H consecutive vertices starting from the leaf.
Each new node is chosen according to the in-tree policy and is added to the search tree.
Note a crucial difference between STS and vanilla MCTS using random rollouts: in contrast to the latter, STS adds visited nodes to the tree, so the explored paths can easily be branched out during later planning passes.
We call this mechanism multi-step expansion .
Algorithm function EXPAND ( leaf ) s ← leaf rollout ← [] for 1 . . .
H do
MCTS .
EXPAND ( s ) a ← CHOOSE _ ACTION ( s ) s (cid:48) , r ← tree [ s ][ a ] rollout .
APPEND (( s, a, r )) s ← s (cid:48) return rollout , s function SELECT ( state )
The same as in Algorithm 7.
function CHOOSE _
ACTION ( s )
The same as in Algorithm 7.
function UPDATE ( path , rollout , leaf ) s (cid:48) ← leaf c ←quality ←for s, a, r ← reversed ( path + rollout )
do if s (cid:48) ∈ path then v ←else v ← V θ ( s (cid:48) )
c ← c +quality ← c ∗ r + γ ∗ quality + v )
W ( s, a ) ← W ( s, a ) + quality
N ( s, a ) ← N ( s, a ) + c
Q ( s, a ) ← W ( s,a ) N ( s,a ) s (cid:48) ← s
STS can be viewed as a sophisticated version of Random Shooting applied to MCTS.
In this interpretation, STS interpolates between the two methods.
We demonstrate empirically that the change introduced by STS is essential to solving challenging RL domains; see Section 4.
We note that
H = 1 corresponds to MCTS.
Experiments
We tested the spectrum of algorithms presented in Section 3 on the Sokoban and Google Research
Football domains.
Those tasks present numerous challenges, which evaluate various properties of planning algorithms.
The training details, a list of hyper-parameters, network architectures and multi-step expansion analysis are presented in appendices A.1, A.2, A.3 and A.9 respectively.
4.1 Sokoban
Scenario C
H
S. rate N p N t
N g av.
loops  95.2%     96.5%     95.7%
[Table]
In the ﬁrst experiment, we evaluated the planning capabilities of STS.
We used a pre-trained value function and varied the number of passes C and the depth of multi-step expansion H , such that H · C remains constant.
In Tablewe present quantities ( N p , N t , N g ), which measure planning costs.
In two presented scenarios, there is a sweet spot for the choice of H .
For this choice, the number of tree nodes, N t , which is the most important metric, is the smallest.
Interestingly, we observe an increase in the solved rate.
This may possibly be explained by the fact that the number of distinct visited game states, N g , grows.
This suggests that STS explores more aggressively and efﬁciently.
In the second line of experiments, we analyzed the training performance (see Algorithm 2).
For
MCTS we used C = 50 passes per step, while for STS we considered C = 10 passes with multi-step expansion H = 5 .
The learning curve for STS dominates the learning curve for MCTS, which persists throughout training, see Figure 1.
Since the difﬁculty of Sokoban levels increases progressively, the achieved improvement is substantial, even though in absolute terms, it may seem small.
Random Shooting methods perform poorly for Sokoban: we evaluated Bandit Shooting (Algorithm 5), which struggled to exceed 5% solved rate.
For a description of Google Research
Football see Appendix A.8.
STS and MCTS STS achieves state-of-the-art results on the GRF
Academy and signiﬁcantly outperforms other methods.
For STS we used C = 30 passes with H =and for MCTS we set corresponding
C = 300 .
STS Conv.
completely solves 8 out of 11 academy environments, see full results in Table 5, and is the best or close to the best on the remaining 3 .
One can observe that in Corner, Counterattack easy and hard, Pass and shoot with keeper, Run to score with keeper, and Single goal vs. lazy academies the difference between STS and MCTS, as well as most of the other methods, is substantial.
See Table 2 for exact results.
We stress that STS Conv.
easily beats any other method in one-to-one comparison across all academies.
STS MLP achieves a close second place.
These results provide further evidence that STS gives a boost in environments requiring long-horizon planning.
This stands in sharp contrast with MCTS, which was not able to achieve impressive results in the considered time budget.
In the STS experiments we used approx.
8 M training samples (median).
More details can be found in Appendix A.8, including ablations.
[Table]
k ee p e r
S i ng l e go a l v e r s u s l az y
PPO 0 .0 .0 .0 .0 .0 .0 .1.
R a ndo m
S hoo ti ng ﬂat 0 .0 .0 .0 .0 .0 .0 .0 .0 .
PPO 0 .0 .0 .0 .0 .0 .0 .0 .0 .
MLP 0 .0.
B a nd it
S ho o ti ng ﬂat 0 .0 .0 .0 .0 .0 .0 .0 .0 .
PPO 1.
MLP 0 .0 .0 .0 .0 .
Conv.
MCTS Conv.
S T S
MLP 1.
Conv.
Conclusions
In this paper, we introduced a new algorithm, Shoot Tree Search.
STS aims to explicitly address the dilemma between depth and breadth search in large state spaces.
Having empirically veriﬁed the efﬁciency of this extension in many challenging scenarios, we argue that it could be included in a standard MCTS toolbox.
A. Training details
We provide the code of our methods and hyper-parameters conﬁguration ﬁles in https://github.
com/shoot-tree-search/sts .
The training loop follows the logic of Algorithm 2.
We use a distributed setup with 30 workers and a replay buffer of size 30000 .
We perform 1000 optimizer updates on batches of transitions whenever all workers collect and store one full episode.
During batch sampling, we ensured an equal amount of examples from solved and unsolved episodes.
In GRF and Sokoban experiments, each episode was limited to 100 and 200 time steps, respectively.
A value function approximator, V θ , is trained via the MSE loss and "reward-to-go" targets (cid:80)
Ti = t +1 γ
i − t − 1 r i , where T is the terminal time-step in an episode.
Q -function approximator, used by MCTS and STS in GRF experiments (see Section A.5 for details), is trained via the MSE loss and tree action-values targets, similar to the one used in Hamrick et al.
[7], Miło´s et al.
Policy, π φ , is trained using the cross-entropy loss.
As targets, we use one-hot encoded actions chosen in the environment for Random Shooting and the empirical distribution of actions chosen in the root during the planning for Bandit Shooting, MCTS, and STS.
The total loss is a weighted sum of the value function (or the Q -function) loss, the policy loss (weighted by 1e − 2 in Random Shooting and Bandit Shooting, and 1e − 3 in MCTS and STS), and a regularizing, l 2 term (weighted by 1e − 6 ).
A pre-trained PPO policy in Shooting methods was obtained using a script included in the Google
Research Football repository (see Kurach et al.
[9]) and the OpenAI Baselines (Dhariwal et al.
PPO2 implementation.
A. Hyper-parameters
[Table]
Discounting γ 0 .0 .0 .0 .
Exploration weight c puct 10 .
Policy π
[Table]
temperature2 .- - 2 .1 .1 . Action sampling temp. τ - - - 0 . 30 .0 . Dirichlet parameter α - - - 0 . 030 .0 .
Noise
[Table]
noise - - - 0 .
[Table] [Table]
[Table] no no no no
yes yes
[Table] [Table]
[Table] [Table]
Table 3 presents hyper-parameters used in our experiments.
These were based on hyper-parameters previously proposed in the literature, e.g., Miło´s et al.
[10], and a certain amount of tuning.
A. Network architectures
In GRF experiments we use two different state representations: ’simple115’ and ’extended’ (see
Section A.8).
In the former case, we use an MLP architecture with two hidden layers of 64 neurons, while in the latter case, we use 4 convolutional layers with 16 , 3x3, ﬁlters, zero-padding and stride 2 , followed by a dense layer of 64 neurons.
In both cases, two heads, corresponding to a value function (or Q -function for MCTS and STS) and policy, follow.
In Sokoban experiments, we use 5 convolutional layers of 64 , 3x3, ﬁlters with zero-padding and stride 1 , followed by a dense layer of 128 neurons and heads corresponding to a value function and policy (policy is used only for Shooting methods).
In all the cases, we use the ReLU non-linearity.
We use the standard Keras initialization schemes, except for MCTS and STS in GRF experiments, see Section A.8.2.
A. Random Shooting
In this section we present two instantiations of Algorithm 1, which use Monte Carlo rollouts to evaluate state-actions pairs: Random Shooting and Bandit Shooting, see Algorithm 4 and Algorithm 5, respectively.
Algorithm function SELECT ( state ) s ← state a ∼ π φ ( s, · ) s (cid:48) , r ← model.
STEP ( s, a ) return ( s, a, r ) , s (cid:48) function EXPAND ( leaf ) s 0 ← leaf rollout ← ( s k , a k , r k +1 )
H − 1 k =0 where s k +1 , r k +1 ← model.
STEP ( s k , a k ) and a k ∼ π φ ( s k , · ) return rollout , s H function UPDATE ( path , rollout , leaf ) ˆ
G ← (cid:80) Hk =1 γ k − 1 r k + γ H V θ ( leaf ) where r k ∈ rollout s, a, r ← path quality ← r + γ ∗ ˆ G
W ( s, a ) ← W ( s, a ) + quality N ( s, a ) ← N ( s, a ) +
[Equation] Q
[Equation] function CHOOSE _ ACTION ( s ) return arg max a Q ( s, a )
The simplest version of Algorithm 4, the so-called ﬂat Monte Carlo [5, 13], does not use a policy π φ (instead rollouts are uniformly sampled) nor a value function V θ (just truncated sum of rewards ˆ G = (cid:80)
Hk
=1 γ k − 1 r k ).
Bandit Shooting, presented in Algorithm 5, is a Multi-armed Bandits variant of
Random Shooting and uses PUCT [15] rule to improve exploration and thus achieve more reliable evaluations of actions.
Algorithm 5 Bandit Shooting Planner, additionally requires exploration weight c puct
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Table]
Bandit Shooting with additional exploration mechanisms Algorithm 6 describes Bandit Shooting with additional exploration mechanisms: mixing the policy with Dirichlet noise (as in [15]) and action sampling with temperature τ in CHOOSE _ ACTION ( s ) .
The noise variable D is sampled from the Dirchlet distribution Dir ( α ) each time when P LANNER is called (see also Algorithm 2).
A. MCTS
Monte Carlo Tree Search (MCTS) is a family of methods, that iteratively and explicitly build a search tree, see Browne et al.
It follows the schema of Algorithm 1.
SELECT traverses down the tree, according to an in-tree policy, until a leaf is encountered.
EXPAND grows the tree by adding the leaf’s children.
The values of these new nodes are estimated, usually with the help of a rollout policy in a similar vein as Random Shooting Planner, or via the value network V θ (see Silver et al.
In this work, we refer to the latter version, using value networks, as MCTS.
Finally, UPDATE backpropagates these values from the leaf up the tree.
After planning, CHOOSE_ACTION chooses an action to take according to the number of visits in each child of the root node.
This is consistent with
AlphaZero [15].
A basic variant of MCTS is presented in Algorithm 7.
More details are provided in
Appendix A.5.
Algorithm 7 MCTS, additionally uses tree structure tree .
function SELECT ( state ) s ← state path ← [] while s belongs to tree do a ← CHOOSE _ ACTION ( s ) s (cid:48) , r ← tree [ s ][ a ] path .
APPEND (( s, a, r )) s ← s (cid:48) return path , s function EXPAND ( leaf ) for a ∈ A do s (cid:48) , r ← model.
STEP ( leaf , a ) tree [ leaf ][ a ] ← ( s (cid:48) , r ) W ( leaf , a ) ← r + γ ∗ V θ ( s (cid:48) )
N ( leaf , a ) ← 1 Q ( leaf , a ) ← W ( leaf , a ) return [] , leaf function UPDATE ( path , rollout , leaf ) quality ← V θ ( leaf ) for s, a, r ← reversed ( path ) do quality ← r + γ ∗ quality W ( s, a ) ← W ( s, a )
+ quality
N ( s, a ) ← N ( s, a ) +
Q ( s, a ) ← W ( s,a ) N ( s,a ) function [Table] ) return arg [Table]
( s, a )
In our experiments, we used various implementations of MCTS.
The reasons were two-fold.
First, some implementation details ﬁt better Sokoban and some GRF.
Second, we wanted to check in various cases that the multi-step expansion is beneﬁcial, see Section A.6.
In Sokoban experiments, we used the MCTS implementation similar to the one in Miło ´ s et al.
[10], containing a loop avoidance mechanism and transposition tables.
The loop avoidance mechanism alters SELECT and CHOOSE_ACTION (see Algorithm 7) so that the selected path does not contain repetitions of states.
The transposition tables are a rather standard technique, which proposes to accumulate search statistics (i.e., W, N, Q ) for states of the environment
In GRF, we used our custom implementation of MCTS based on the one in Silver et al.
It uses leaf evaluation with Q -function and policy networks.
The Q -function is used to evaluate all children of a given node at once (instead of separately invoking value function V θ in UPDATE ).
The policy network is considered to be ’prior’ for choosing actions, similarly as in SELECT in Algorithm 6.
Dirichlet noise, parameterized by α and c noise , is mixed with the prior in the root and action sampling with temperature τ is used to choose action on the real environment, similarly as in Bandit Shooting with additional exploration mechanisms in Section A.4.
Additionally, we put a limit, depth limit , on the maximum number of nodes visited in a single STS pass.
A. STS
We tested STS with two MCTS setups described in Section A.5.
In both the cases we observed substantial experimental improvements as reported in Section A.7 and Section A.8.
This alone, in our view, provides enough evidence that the multi-step expansion is a useful method.
Apart from this, STS offers practical computational beneﬁts, which are analyzed below.
A.6.1 Computational beneﬁts of STS
We distinguish three types of computational costs in MCTS (see Algorithm 7):
Traversing down the search tree (performed in SELECT and EXPAND ).
Backpropagation of values and counts update (handled by UPDATE ).
Evaluation of heuristics (value network V θ , or Q -function and policy as described in Section A.5)
In large GRF experiments, we found that it was the ﬁrst cost that dominated the remaining two.
The reason is that the cost of building a search tree is quadratic to its depth.
The use of multi-step expansion signiﬁcantly reduces this cost as several nodes are added during single tree traversal.
In our case, these beneﬁts allowed for much smoother experimenting with GRF and are, arguably, a step towards developing more efﬁcient planners.
We expect this might be practically useful (i.e., costs 1 and 2 are dominant) when the search size is large, or the heuristic evaluation is relatively cheap compared to the environment step.
This is the case in some of our GRF experiments.
The GRF simulator is rather complex and slower than small MLP networks.
The following simple lemma offers some theoretical analysis.
Lemma A.6.1.
Assume that STS and MCTS build the same tree T , starting from the root state s 0 .
Denote the number of nodes in T as C and the number of nodes to be added at a single multi-step expansion of STS as H .
Then the number of steps in T performed by STS will be lower compared to
MCTS by a factor in [ h − 12 , h ] .
Proof.
Lets consider h consecutive nodes s 1 , . . .
, s h in the search tree added in a single EXPAND step during STS search.
In STS, the number of steps, C STS , in the tree during SELECT and EXPAND is equal to h + d , where d is distance between s 0 and s 1 in T .
To add the same set of nodes during MCTS search, one need h separate calls to SELECT and EXPAND .
The total number of steps performed is
[Equation]
Clearly,
[Equation]
Similar calculation hold for the costs of backpropagation.
A. Sokoban experiments
Sokoban is a well-known combinatorial puzzle, where the agent’s goal is to push all boxes (marked as yellow, crossed squares) to the designed spots (marked as squares with a red dot in the middle), see Figure 2.
Additionally, to the navigational challenge, Sokoban’s difﬁculty is attributed to the irreversibility of certain actions.
A typical example is pushing a box into a corner, though there are H multiple less apparent cases.
The environment’s complexity is formalized by the fact that, deciding whether a level of Sokoban is solvable or not, is PSPACE-complete, see e.g. Dor and Zwick [4].
Due to these challenges, the game is often used to test reinforcement learning and planning methods.
Sokoban is an environment known for its combinatorial complexity.
The agent’s goal is to push all boxes (marked as yellow, crossed squares) to the designed spots (marked as squares with a red dot in the middle), see Figure 2.
Apart from the navigational challenge, the difﬁculty of this game is greatly increased by the fact that some actions are irreversible.
A canonical example of such an action is pushing a box into a corner, though there are multiple less obvious cases.
Formally, this difﬁculty manifests itself in the fact that deciding whether a level of Sokoban is solvable or not, is NP-hard, see e.g. Dor and Zwick [4].
Due to these challenges the game has been considered as a testbed for reinforcement learning and planning methods.
For a description of Sokoban see Section 4.1.
In our experiments, we used inputs of dimension ( x, x, 7) , where ( x, x ) is the size of the board ( (10 , 10) in most cases) and 7 is one-hot encoding of the state of a given cell (enumerated as follows: wall, empty, target, box_target, box, player, player_target).
In most experiments, we used 4 boxes and the limit of 200 steps.
The agent is rewarded with 1 by putting a box into a designated spot and additionally withwhen all the boxes are in place 2 .
The action space consists of four
A.7.1 Evaluation experiments
In Table 4 we show full details of the evaluation experiment (which complements Table 1).
Recall that in this experiment, we evaluated the planning capabilities of STS in isolation from training.
To this end, we used a pre-trained value function and varied the number of passes C and the depth of multi-step expansion H , such that H · C remains constant.
In Table 4, we present quantities ( N p , N t , N g ), which measure planning costs for ﬁnding a solution (the average number of passes,
[Table]
no avoid loops
[Table]
tree nodes and game states observed, respectively, until the solution is found).
We run experiments with and without the loop avoidance mechanism (see Section A.5).
We observe that there is a sweet spot for the choice of H .
It is evident for the ’no avoid loop’ case, C = 32 , H = 8 .
For this choice, the number of tree nodes, N t , which is the most important metric, is the smallest.
Interestingly, we observe a signiﬁcant increase in the solved rate.
This may be explained by the fact that the number of distinct visited game states, N g , grows.
This suggests that STS explores more aggressively and efﬁciently.
For bigger H , we observe a further increase of the solved rate until some point, though at the cost of much bigger N t .
In experiments with the avoid loop mechanism, there is a similar effect for C = 64 , H = 4 , though more subtle, probably because results are already quite strong.
Moreover, we observe a more signiﬁcant drop in performance as H increases (when planning resembles more shooting methods).
The values presented in Table 4 are averages over more than 5000 boards.
A.7.2 MCTS and Shooting on simpler boards
We found the Bandit Shooting method underperforming on Sokoban.
As a sanity test, we tested a simpler setting with smaller boards of size (6 , 6) and two boxes.
Learning curves are presented in
Figure 3.
MCTS and STS experiments quickly learn to solve over 99% of boards.
Bandit Shooting experiment showed stable but much slower progress.
We also evaluated the version of Bandit
Shooting, with additional loop avoidance, see Section A.5.
This mechanism was beneﬁcial for MCTS and STS but failed to bring improvements for the shooting algorithms.
A.7.3 Correcting biased value functions estimates with deep search
To generate value function heatmaps we evaluated the pre-trained MCTS value function for each possible agent position in a room.
Figures 5 and 6 present two chosen rooms with their corresponding
VF heatmaps.
Speciﬁcally, the room in Figure 6 was solved by the STS with 10 passes andsteps of multi-step expansion and wasn’t solved by the MCTS with 50 passes, both without the avoid loops mechanism.
We include movies of both agents in this room in the code repository: https://github.com/shoot-tree-search/sts/tree/master/movies .
Because the value function is biased, it makes MCTS stuck in states with overestimated value.
See
Figure 6, in this room MCTS gets stuck in the bottom-left region.
However, with a deeper tree, STS can get unstuck quicker
and still ﬁnd a solution.
Remember that the search statistics (i.e., W, N, Q ) are accumulated for states of the environment (see Appendix A.5).
As this overestimated region gets searched deeper the bias in the value function gets discounted more and the agent ﬁgures out there are no rewards in reality.
At some point, other actions will have a higher value and the agent has a chance to get unstuck and explore other parts of the room.
That being said, it should be noted that this "potential well" will still attract the agent, make its planning paths distracted, even when it gets unstuck.
STS is less vulnerable to this effect and is able to solve this room despite high bias in the value function.
A. Google Research Football experiments
For a description of Google Research Football see Appendix A.8.
Google Research Football (GRF) is an environment recently introduced in Kurach et al.
It is an advanced, physics-based simulator of the game of football.
A part of GRF is the Football Academy consisting of 11 scenarios highlighting various tactical difﬁculties, see Kurach et al.
[8, Table 10] for description.
A GRF Academy episode is considered ﬁnished after 100 steps or when the goal is scored by the agent.
Reported results correspond to solved rates over 20 episodes in case of Shooting methods with an uniform and a pre-trained policy and around 30 episodes in case of all other methods.
Results for
MCTS, STS, and Shooting methods with the trained policy are medians of at least three training runs.
During evaluations we disabled Dirichlet noise and action sampling (in Bandit Shooting Expl.,
MCTS and STS).
Google Research Football offers two major mode of observations: ’simple115’ and ’extracted’ (also called the super mini-map).
The simple115 state representation is consists of coordinates of players, players’ movement directions, the ball position, a ball movement direction, a one-hot encoding of ball ownership, a one-hot encoding of which player is active.
This totals in a vector of length 115 .
The extracted state representation consists 4 stacked layers of size (72 , 96) .
Layers contain one-hot encoding of spatial positions of game entities.
These are (on the subsequent layers): players on the left team, players on the right team, the ball, and the active player.
We note that even though the extracted representation contains ’less information’ than simple115, it has been reported in [8] to generate better results.
In our experiments, we use the so-called checkpoint rewards, which provide an additional signal for approaching the goal area.
Details can be found in [8], where they were introduced and used in large-scale experiments.
The action space in GRF consists of 19 actions representing high-level football behaviors (e.g. "Short
Pass"), see Kurach et al.
[8, Table 1].
Figure 7 shows selected training curves on Google Research Football, the best from each family of our methods: Shooting, MCTS and STS.
Figure 8 shows all training curves for our methods on Google
Research Football.
On the y-axis is the solved rate calculated as described above in Section A.8.
On the x-axis is the number of real steps in the environment (planning steps in the simulator are not added).
Curves are mean over 3 training runs with different seeds and shaded regions represent 95% conﬁdence intervals (exceptionally for Bandit Shooting we report just one run).
Moreover, to smooth the curves, data points are averaged in the windows of 10000 steps.
A.8.1 Shooting methods
Table 5 presents our methods performance in all Google Research Football academies.
Tuning c puct turned out to be the most important one to make Bandit Shooting work, see Algorithm 5.
In a nutshell, it needs to be adjusted to scale of rewards (value function) in a given environment.
In our experiments we found c puct = 2 .
5 to work best.
Using additional Dirichlet noise, c noise > 0 , and action sampling on the real environment, τ >(see Algorithm 6) resulted in inferior results with an exception of the "Counterattack hard" scenario.
A.8.2 MCTS and STS experiments
Apart from multi-step expansion we introduced another simple novel method, which might be of interest to the general public.
Namely, before starting training, we set the weights of the last layer of the Q -value neural network to 0 (see Section A.3 for a detailed description of architectures).
We observed that this signiﬁcantly improved the training stability due to better exploration (and avoiding suboptimal strategies at the early stages of training).
See ’No zero initialization’ on Figure 10.
H
[Figure]
Method 3 v s .
1 w it
h k ee p e r
C o r n e r
C oun t e r a tt ac k
ea s y
C oun t e r a tt ac k h a r d
E m p
t
y go a l
E m p
t
y go a l c l o s e
P a ss a nd s hoo t w it h k ee p e r R un p a ss a nd s hoo t w it h k ee p e r
R un t o s c
o r e
R un t o s c
o r e w it
h k ee p e r
S i ng l e go a l v e r s u s l az y
[Table]
A.8.3 Ablations
The ablations were performed on three environments from GRF Academy: corner , counterattack hard and empty goal , see Figure 10.
The ﬁrst two environments are difﬁcult, while the last one is easy.
The following parameters or settings were subject to analysis (they correspond to the labels in
Figure 10):
• prior noise weight : a weight in the mixture of Dirichlet noise and the prior.
• depth limit : the maximum number of nodes visited in a single STS pass.
• sampling temperature : temperature for sampling the actions on the real environment.
• MCTS n _ passes 300 : this corresponds the standard MCTS setting with H = 1 (MCTS) and C =• Value network n _ passes : value network is used instead of Q -function.
Note that n _ passes = 2 matches roughly the Q -function version in terms of visited states (recall, see Section 7, that Q -function evaluates all children at once and that number of actions in GRF is 19 ).
• No policy : instead of a learned policy network, a uniform policy is used.
• No zero initialization : the last layer of the value function neural network was not initialized to 0 (see description at the beginning of Section A.8.2).
The default setup (denoted as Prior noise weight 0 . 1 ) is always positioned at the top in Figure 10.
It uses parameters described in Table 3 in the Google Research Football STS column.
A. Multi-step expansion analysis on toy problems
First, consider an MDP presented at the top of Figure 9.
It showcases the situation when the errors are systematic: in the vicinity of the starting state s 0 , the estimates of the value function are biased (for simplicity set to 0 and shown as white vertices), while the values in the area surrounding terminal states are accurate (shown as color vertices).
This example is an exaggeration.
However, something similar can happen in practice, when information is propagated with TD -like methods or the environment has an “easy” region, which is hard to ﬁnd.
Under these circumstances, STS, given large enough H , will be able to reach accurate values (color vertices) within a few passes.
On the contrary, MCTS would explore the whole uncertain area (white vertices) in a breadth-ﬁrst fashion.
Second, consider an MDP shown at the bottom of Figure 9.
It illustrates the case when the errors are “pseudo-random”.
In this MDP all rewards are 0 except the marked edges, where they are − a, a > 0 .
Starting from s 0 , the agent can move only to the right.
The perfect value function is 0 in every state, however we assume that the current noisy value estimates equal to (cid:15) i on the “tail” part of the diagram.
In this example, we assume that the errors arise in interactions of many factors, thus can be modeled as i.i.d.
centered random variables (cid:15) i such that E | (cid:15) i | < + ∞ .
The optimal path, going over the green edge and later over the tail, is accompanied by several ’decoy’ paths (marked in orange).
They will not be entered unless errors on the tail have accumulated below − a .
We denote the probability of such an event by p H , where H is the number of steps in the multi-step expansion ( H = 1 corresponds to MCTS).
In Lemma A.9.1, we show that p 1 > p H for
H ≥ 2 , and in fact p H [-! 
-]→ 0 when H → + ∞ .
Lemma A.9.1.
Under the above assumptions p 1 > p H and p H → 0 .
Proof.
Assume that for the ﬁrst (cid:96) ≥ 2 steps of the search tree was unfolded via the middle (green) edge and further via the tail.
The state-action value estimated by the MCTS/STS is thus q (cid:96) = ( {+(cid:15) 
+}
+ . . . + (cid:15) (cid:96) − 2 ) /(cid:96) .
Consequently,
[Equation]
The claims follow from the fact q [-` ! 
-](cid:96) → 0 a.s., which itself is the consequence of the strong law of large numbers.
As the lemma serves mainly the illustrative purpose we used the i.i.d.
assumption, which can be easily weakened.
As a test we simulate the case (cid:15) i ∼ N (0 , 1) and a = 0 . 3 .
In this case p 1 = 0 .
56 , p 2 = 0 .
46 , p 4 = 0 .
35 , p 8 = 0 .
41 , p 16 = 0 .
Note that p 1 /p H is as high as 3 for
H = 16 and quite natural choice of a and (cid:15) i .
A. Infrastructure used
We ran our experiments on clusters with servers typically equipped with 24 or 28 CPU cores and 64GB of memory.
A typical experiment was 72 hours long (the timeout set on the clusters), which was enough for most experiments.
Experiments that did not converge during this time were resumed.
During the project, we run more than 10 k experiments.