A
(cid:63)
MCTS: S EARCH WITH T HEORETICAL G UARANTEE USING P OLICY AND V ALUE F UNCTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Combined with policy and value neural networks, Monte Carlos Tree Search (MCTS) is a critical component of the recent success of AI agents in learning to play board games like Chess and Go (Silver et al., 2017).
However, the theoretical foundations of MCTS with policy and value networks remains open.
Inspired by
MCTS, we propose A (cid:63) MCTS , a novel search algorithm that uses both the policy and value predictors to guide search and enjoys theoretical guarantees.
Speciﬁcally, assuming that value and policy networks give reasonably accurate signals of the values of each state and action, the sample complexity (number of calls to the value network) to estimate the value of the current state, as well as the optimal one-step action to take from the current state, can be bounded.
We apply our theoretical framework to different models for the noise distribution of the policy and value network as well as the distribution of rewards, and show that for these general models, the sample complexity is polynomial in D , where D is the depth of the search tree.
Empirically, our method outperforms MCTS in these models.
I NTRODUCTION
Monte Carlo Tree Search (MCTS) is an online heuristic search algorithm commonly used to ﬁnd optimal policies for problems in reinforcement learning.
It is a vital component of recent breakthroughs for AI in game play, including AlphaGo and AlphaZero (Silver et al., 2016; 2017).
An interesting difference between the MCTS used in AlphaGo/AlphaZero and traditional MCTS is the adoption of neural networks to predict the value of the current state ( value network ), and to prioritize the next action to search ( policy network ).
The value network replaces random rollouts used to evaluate a non-terminal state, which saves both online computation time and substantially reduces the variance of the estimate.
The policy network (coupled with PUCT (Rosin, 2011)) prioritizes promising actions for more fruitful exploration.
Both networks are trained on pre-existing datasets (either from human replays or self-play) to incorporate prior knowledge.
This strategy has been applied in other domains including Neural Architecture Search (Wang et al., 2018; Wistuba, 2017;
Negrinho & Gordon, 2017; Wang et al., 2019b).
Despite the empirical success and recent popularity, theoretical foundations are lacking.
(Kocsis &
Szepesv´ari, 2006; Coquelin & Munos, 2007) analyze MCTS with rollouts using a multi-armed bandit framework and gives asymptotic results without ﬁnite sample complexity bounds.
Powering MCTS with value/policy neural networks instead of rollouts is even more poorly understood.
To better understand how to efﬁciently learn optimal policies in decision trees using value and policy networks, we propose a novel algorithm, A (cid:63) MCTS , which is a combination of A (cid:63) (Delling et al., 2009; Kanoulas et al., 2006) and MCTS.
Like A (cid:63) , it uses a priority queue to store all leaves currently being explored and picks the most optimistic one to expand, according to an upper conﬁdence bound heuristic function.
Like MCTS, it uses policy and value networks to prioritize the next state to be explored.
To facilitate the analysis, the policy and value networks are treated as black-box functions and a statistical model is built the accuracy of the predictions.
In particular, we assume the standard deviation of the noisy estimates of intermediate state values decays (either polynomially or exponentially) as a function of the depth of the tree, so that the values of near-terminal states are estimated to a greater degree of accuracy.
With this statistical model, we provide theoretical guarantees for the sample complexity (expected number of state expansions / rollouts) to determine optimal actions.
We apply our theoretical framework to simple models for rewards and show that our algorithm enjoys a sample complexity that is polynomial in depth D .
Our experiments validate the theoretical analysis and demonstrate the effectiveness of A (cid:63) MCTS over benchmark MCTS algorithms with value and policy networks.
To our knowledge, our work is the ﬁrst that studies tree search for optimal actions in the presence of pre-trained value and policy networks, and we hope that it inspires further progress on this important problem.
B ACKGROUND AND R ELATED W ORK
Monte Carlo Tree Search (Browne et al., 2012) is an algorithm that tries to ﬁnd optimal decisions in
Markov Decision Processes by taking random samples in the decision space and building a tree based on the results.
In each iteration, the agent traverses the tree by acting according to the tree policy (which usually involves some exploration) for the set of known states and then determines the value of an unexpanded state by performing random rollouts.
MCTS is commonly used to search for a one-step optimal actions from the root.
The agent then takes this action, observes the result, transitions to a new state, and repeats MCTS starting from this new state, with the beneﬁt of the additional observation.
The main theoretical question of interest in MCTS is how quickly and efﬁciently one can ﬁnd this optimal one-step action, in terms of the number of random rollouts, or the number of nodes in the tree that should be expanded.
On the theoretical side, there is a small body of literature that addresses the theoretical foundations of MCTS with random rollouts (Kocsis & Szepesv ´ ari (2006), Veness et al.
The algorithm, called
UCT (UCB applied to trees) models the search as a complex multi-armed bandit problem where the rewards are allowed to drift over time.
The analysis shows that after a period of time T , only the optimal branch will be followed.
Veness et al.
(2011) gives various ways of improving UCT with variance reduction techniques.
Later followup work in Coquelin & Munos (2007) highlights some hard instances of UCT and offers some alternatives, including a modiﬁed UCT (with a different exploration bonus), Flat UCB, and
BAST (smooth tree models).
On the experimental side, MCTS was popularized by the story of AlphaGo (Silver et al.
(2016)), which used MCTS using a weighted combination of the value network’s output and random rollouts to defeat the leading champion in Go.
Follow-up work in AlphaZero (Silver et al.
(2017)) used MCTS with only policy and value neural networks, without random rollouts and reports faster training with better performance.
However, there is no theoretical foundation for the search algorithm that
AlphaZero uses.
A (cid:63) (Nosrati et al., 2012) uses a priority queue to keep track of lowest cost actions.
A (cid:63) combined with hand-designed heuristics, has been used in many path ﬁnding and routing problems (AlShawi et al., 2012;
Rana & Zaveri, 2011; Ghaffari, 2014).
Recent work (Wang et al., 2019a; Chen & Wei, 2011;
Sigurdson & Bulitko, 2017) trains neural-based heuristic functions for A (cid:63) for speciﬁc problems.
However, to the best of our knowledge, we are not aware of any prior work that performs theoretical analysis of such heuristic based A (cid:63) algorithms.
P RELIMINARIES AND S UMMARY OF C ONTRIBUTIONS
In this work we consider a deterministic, ﬁnite D -horizon Markov decision process (MDP) with no intermediate rewards.
With the set of states denoted by S , there is one deterministic root initial state s 0 and the process terminates at horizon D with a termination reward that depends on the end state.
We assume that there are exactly K possible actions at every intermediate state s and for simplicity that the process does not backtrack, i.e., each action leads to a new and unique state.
This is a Markov decision tree.
The set of leaf states of the Markov decision tree are denoted as L .
We shall denote an arbitrary state at depth d as s d and also a leaf state as l ∈ L .
There is a value function V : S → R that takes as input a state s and returns the value of s , deﬁned in the following way.
If s ∈ L , then V s is the termination reward of that particular leaf.
For an intermediate node s , V s = max l ∈L ( s ) V l , where [Table]
random variable for the noise of the value prediction U s d at depth d .
V s d − U s
d ∼ X d .
[Table] standard deviation of X d , ﬁxed and known as part of the problem model. We assume that σ d decays with depth. [Table]
exploration bonus, determined based on σ d .
| X d | ≤ c d with high probability
U s d + c d priority value (value used to determine the ordering of which to nodes expand in the priority queue).
[Table]
the
[Table]
L ( s ) is the set of leaf nodes that share s as an ancestor, i.e., the set of leaf nodes in the subtree rooted at s .
We assume that the highest value V ∗ ≡ max l ∈L V l = max s ∈S V s is achieved at a unique leaf node l ∗ = arg max l ∈L V l .
In this work, we focus on the problem of efﬁciently computing V ∗ exactly or approximately with a controlled accuracy without direct access to the value function V : S → R , as a stepping stone to ﬁnding the optimal one-step action from s 0 .
Instead, there is a pre-trained black-box resource, value network U : S → R that takes as input a state and returns a noisy value estimate, i.e., for a state s d at depth d , U s d ≡ V s d + X s d , where X s d are i.i.d.
copies of a random variable X d that depends only on the depth d .
We assume that at depth D , which is the level with all the leaf nodes, U l = V l (i.e., the value network returns the ground truth values without any noise).
The efﬁciency is deﬁned in terms of sample complexity, which is the number of queries made to the value network for the value of a state.
In the worst case, all leaves need to be queried and the sample complexity is K D .
Another type of available pre-trained black box resource is a policy network .
Let C ( s ) =
{ r 1 , . . .
r K } denote the set of K children of state s , each corresponding to the transition state of one of the K actions { a 1 , . . .
a K } .
The policy network outputs a probability estimate of the form p r
i
[Equation] where each U πr
i is a perturbed estimate of V r i .
Here the superscript π indicates that this is the approximation from the policy network.
We assume that if the child r i is at depth d then the estimate
U πr
i
=
V r i + X
πr i , where X πr i are i.i.d.
copies of a random variable X πd that depends only on the depth d .
We assume that the distribution of X d and X πd are known a priori for each depth d .
Further, the variances of X d and X π d decrease sufﬁciently rapidly as the depth d increases (i.e., as one traverses deeper in the tree, the noise level is smaller).
This assumption is motivated by basic approximate dynamic programming principles, which is that near-terminal states require a smaller sample complexity to estimate well, since the backwards induction will propagate this noise to earlier states, making it harder to estimate the values of initial states well. 3. M AIN C ONTRIBUTIONS
We develop and analyze (giving speciﬁc sample complexity bounds) algorithms to estimate V ∗ exactly and approximately using value and policy networks.
Since our value and policy networks are trained functions that give ﬁxed, deterministic noisy outputs for each state (no matter how many times one calls the function on a particular state), where the noise variance is smaller at deeper levels of the search tree, we use our value and policy network estimates as signals for which are the promising states to expand.
We rank the states based on our optimistic estimates U s d + c d in a priority queue, and we expand (query the value network for the value estimate) the children of the top nodes of the priority queue to get increasingly more accurate value estimates and determine the value of the optimal leaf V ∗ .
The main contributions are organized in the following sections:
• Section 4 introduces and analyzes A (cid:63) MCTS-
V ∗ and A (cid:63) MCTS-
π ∗ V ∗ that are our core techniques for estimating V ∗ exactly.
A (cid:63) MCTS-
V ∗ uses only a value network to determine V ∗ exactly and outputs the optimal one-step from s 0 that leads to l ∗ .
We give an expected sample complexity in Theorem 1.
A (cid:63) MCTS-
π ∗ V ∗ builds on A (cid:63) MCTS-
V ∗ by using a policy network to further reduce the sample complexity, which we bound in Theorem 2.
• Section 5 extends A (cid:63) MCTS-
V ∗ and A (cid:63) MCTS-
π ∗ V ∗ to the case when we tolerate δ -additive approximations of V ∗ .
• Section 6 introduces some sample distribution models for the value functions and the noise and applies our results to those models.
We show that for some reasonable models, our algorithms can achieve sample complexity that is polynomial in depth D .
We also provide some experimental comparisons against benchmark MCTS implementations.
F INDING V ∗ E XACTLY USING V ALUE AND P OLICY N ETWORKS
This section presents our main techniques, A (cid:63) MCTS-
V ∗ and A (cid:63) MCTS-
π ∗ V ∗ , which ﬁnd V ∗ exactly using a value network only in the former and a value network and a policy network in the latter.
A (cid:63) MCTS-
V ∗ , Algorithm 1, uses only a value network to determine V ∗ exactly and outputs the optimal one-step action from s 0 that takes the agent to an ancestor of l ∗ .
The idea behind the algorithm is very simple and similar to A (cid:63) search.
We ﬁrst expand (i.e., query the value network) the children of the root node s 0 , compute their value network estimate plus the exploration bonus, and then insert these nodes to a priority queue Q .
The exploration bonus, denoted by c d , is chosen such that P ( | V s d − U s d | ≤ c d ) ≥ 1 − βDK d .
In each subsequent iteration, the algorithm picks the top element in the priority queue Q , which is the most optimistic state, and expands all the children of that state, and adds those children to Q .
We terminate at the ﬁrst time when the most optimistic element is a leaf node.
Since the value network gives the values of leaf nodes exactly, this leaf node must be l ∗ .
We present the complexity bound in Theorem 1.
The complexity bound is based on one very simple observation, which is that we end up choosing a sub-optimal node if the value network estimate of that node plus the upper bound on the possible error (optimism) is higher than the true optimal value.
Otherwise we will not choose to expand that node, since we are sure that it will not be the optimal node.
Therefore a sub-optimal internal node s d is chosen if V ∗ ≤ U s d + c d = V s d + X s d + c d , where U s d is the value network estimate for state s d , and c d is the upper bound on the possible error for the value estimate. We should also account for the ancestors of s d , if we are lucky enough to be able to rule out one of the ancestors of s d , we would never expand beyond that ancestor to reach s d , and s d would not be in our priority queue.
Theorem 1 states this complexity bound.
Theorem 1.
With probability 1 − β , Algorithm 1, A (cid:63) MCTS-
V ∗ , returns V ∗ .
The expected sample complexity (number of calls to the value network) is
E [ N ] = KD + (cid:88)
[Equation]
[Figure] where A ( l ∗ ) is the set of ancestor states of the optimal leaf, l ∗ , and ∆ s d = V ∗ − V s d .
Before we give the proof, we brieﬂy note that getting an exact expression for the expected sample complexity depends on the problem model (the distribution for the gaps ∆ s d and the errors X s d ).
We give two examples in Section 6 of different models and we apply this theorem to those models to derive the sample complexity, which we show is polynomial in depth D .
Please check Appendix for the proof.
4. C OMBINING V ALUE N ETWORK WITH P OLICY N ETWORK
Suppose that in addition to the value network U , one has access to a policy network U π that outputs, for each child of s , r 1 , . . .
r K (each corresponding to the transition state of a different action a 1 , . . .
a K ) a probability estimate of the form:
[Equation]
where each U πr i
= V r i + X
πr i is a perturbed estimate of V r
i .
If r i is at depth d , the noise X πr i is an i.i.d.
copy of the random variable X πd that depends only on the depth d .
Since all the probabilities are known, we assume without loss of generality that states r 1 , . . .
r K are ordered so that U πr 1 ≥
U πr 2 ≥
U πr 3 . .
≥ U πr K , or equivalently p r 1 ≥ p r 2 ≥ p r 3 . . .
≥ p r K .
The following algorithm A (cid:63) MCTS-
π ∗ V ∗ uses both the value and the policy networks to determine
V ∗ exactly.
When we expand a given state s d , instead of querying for the value estimate for every child of s d and adding each child to the priority queue, we add the top 2 children (ordered by the probabilities given by the policy network) to the queue by default.
For the k -th child node, where k > 2 , we add this child if there is a small gap between the policy network probability of the k − 1 -th child and the probability of the top child, where the threshold for the gap is given by the noise model.
Intuitively, we are saying that if this gap is sufﬁciently big, even after accounting for the noise in the policy network, there is no way that this k -th child is part of the optimal policy, so we do not have to add it to the queue.
The key insight behind this algorithm is that, if there is a large gap between the probabilities of the children as given by the policy network, one can infer that the children states associated with the smaller probabilities are not worth expanding because they have small values.
This is made rigorous in the statement and proof of Theorem 2.
Theorem 2.
With probability 1 − β , Algorithm 2, A (cid:63) MCTS-
π ∗ V ∗ , returns V ∗ .
The expected sample complexity (number of calls to the value network) is
[Equation]
where ∆ s d = V ∗ − V s d , and X π ( d +1) ,k is the random variable associated with r k for the policy network,
i.e., U πr k
= V r k + X π
( d +1) ,k .
As with Theorem 1, getting an exact expression for the expected sample complexity depends on the problem model (the distribution for the gaps ∆ s d and the errors X s d ).
[-Moreover, V r 1 ≥ U πr 
-]
We give two examples in Section 6 of different models and we apply this theorem to those models to derive the sample complexity, which we show is polynomial in depth D .
Please check Appendix for the proof.
A PPROXIMATING V ∗ U SING V ALUE AND P OLICY N ETWORKS
This section extends the algorithms to the problem setting where we are willing to tolerate an additive error δ in the estimate for V ∗
We ﬁrst consider the extension of A (cid:63) MCTS-
V ∗ , Algorithm 1.
The key idea is that it sufﬁces to stop searching at a depth (cid:101) D whenever c (cid:101) D ≤ δ .
As long as the most optimistic state at depth (cid:101) D is identiﬁed, the value of this state approximates V ∗ up to an additive error δ .
The main result is stated in Theorem 3 and the algorithm is Algorithm A (cid:63) MCTS-
ˆ V , presented in Appendix 8.3.
Theorem 3.
With probability 1 − β , Algorithm 3, A (cid:63) MCTS-
ˆ V , returns V ∗ up to additive error δ .
The expected sample complexity (number of calls to the value network) is
[Equation]

 where A ( l ∗ ) is the set of ancestor states of the optimal leaf, l ∗ , and ∆ s d = V ∗ − V s d , and (cid:101) D is the minimum depth at which c (cid:101)
D ≤ δ
Using the same reasoning as the one for A (cid:63) MCTS-
ˆ V , we can extend A (cid:63) MCTS-
π ∗ V ∗ to the following result.
Theorem 4.
With probability 1 − β , one can ﬁnd V ∗ up to additive approximation error δ using expected sample complexity (number of calls to the value network):
[Equation]
where (cid:101) D is the minimum depth at which c (cid:101)
D ≥ δ .
M ODELS AND E XPERIMENTS
In this section, we discuss two simple yet representative models for the Markov decision tree and show that the sample complexity is polynomially bounded under reasonable noise models.
6. C HOOSING c d FOR G AUSSIAN NOISE
The noises X d , X πd are taken to be Gaussian N (0 , σ 2 d ) random variables.
Two noise models are tested.
The ﬁrst is a Gaussian with an exponentially decaying standard deviation, i.e., X d , X πd ∼ N (0 , 1 α 2 d ) for some α > 1 .
The second is a Gaussian with a polynomially decaying standard deviation, i.e.,
X d , X πd ∼ N (0 , 1 d 2 γ ) , for some γ > 1 .
The experiments are carried out for α = 1 . 3 and 1 . 5 for the exponential case and γ = 1 . 3 and 1 . 5 for the polynomial case.
The assumption of decaying standard deviation as a function of depth is motivated by basic approximate dynamic programming principles, which we elaborate on in Section 3.
The choice of c d is essential to the performance of the algorithms.
Standard concentration inequalities for sub-Gaussian random variables state that P ( | X d | ≥ c d ) ≤ 2 e − c 2 d 2 σ 2 d , and this probability expression is required to be less than βDK d .
After substituting it in, we arrive at c d = O ( √ dσ d ) .
6. C ONSTANT G AP M ODEL
In the constant gap model, V ∗ − V l ∗ =
η for a ﬁxed constant η > 0 and V l = 0 for any other leaf l (cid:54) =
l ∗ .
Therefore, the gap ∆( l ) =
η for any l (cid:54) =
l ∗ .
The noise variables X d , X πd are mean-zero
Gaussian random variables.
Using value network.
The expected sample complexity for Algorithm 1 is
[Equation]
This translates to a sample complexity of E [ N ] ≤ KD + D 2 ( K − 1) K c , for the smallest constant c that satisﬁes ησ c − √ c ≥ √ 2 log K .
The analysis for this result, as well as similar analyses and sample complexity results for Algorithmand the approximate counterparts are in Appendix 8.4.
Experiments.
We compare the performance of A (cid:63) MCTS-
V ∗ with the MCTS algorithm that uses the
UCB estimate Q s,
[Equation]
a and queries the value network U s instead of expanding state s with random rollouts, where c is a constant, and N s,b are visitation counts for choosing action a from state s .
We also compare A (cid:63) MCTS-
π ∗ V ∗ with the PUCT algorithm described in Silver et al.
(2017) that uses the UCB estimate Q s,a + c · p s,a · (cid:113) (cid:80) b N s,b N s,a , where p s,a is the probability given by the policy network U π .
The experiments are performed on a tree of depth D = 10 with K = 5 children per state.
As deﬁned in the constant gap model, the optimal leaf has reward η and all other leaves have reward 0 , and we experiment with η = 1 and η = 0 .
In our experiments, we use c = 1 in the the bonus UCB expression for benchmark MCTS and PUCT algorithms.
Our algorithms set c d = 5 · √ dσ d , where σ d is the variance of the Gaussian noise.
This choice is justiﬁed earlier in Section 6.1.
Since the main use case for these algorithms is to ﬁnd the optimal one-step action from s 0 , we collect data on the one-step action that the algorithm would output at every 1 , 000 expansions (or queries to the value network U ).
MCTS algorithms would output the one-step action from s 0 with the highest visitation count.
Our algorithms would output the action from s 0 that leads to s , where s is the highest-valued element in the current priority queue Q based on the value network estimate U s .
We give an overall budget of 20,000 expansions and note whether or not (so the data is 1 or 0) the algorithm chooses the right one-step action.
Each of our experiments average over 200 trials.
Tables 1 and 2 give the proportion of trials that return the correct optimal one-step action at the 20,000-th expansion.
To give an idea about the performance gain, Figures 3 and 4 show the success proportion over time, at successive 1000 expansion intervals for Algorithm 2 and PUCT for η = 0 .
5 in both noise models (polynomially and exponentially decaying).
[Table]
=
[Table]
=
[Table]
α
[Table]
[Table]
6. G ENERATIVE M ODEL
The model generates the value function of the nodes in the Markov decision tree in a recursive way.
This model ﬁrst chooses the optimal value
V s
= V ∗ at the root s 0 .
Given the value function of a parent node p , the value functions for the children r 1 , . . .
, r K are generated as follows: one of the children (without loss of generality, say r 1 ) has the same value function as the parent V p ; each of the remaining children r i has a value function V p − Y with Y sampled independently from ∼ U [0 , η ] , the uniform distribution over the interval [0 , η ] .
At level d , there are exactly (cid:0) dt (cid:1)
( K − 1) t states with value function distributed according to V ∗ − ( Y 1 + . . .
+ Y t ) .
When t is non-negligible, Y 1 + . . .
+ Y t concentrates at constant value tη/ 2 by the concentration of measure phenomenon.
Using value networks.
Following the estimate in Theorem 1, the overall complexity for Algorithmcan be bounded up to an extra KD term by
[Equation]
where Z is a standard normal distribution.
The goal is to bound this by a quantity polynomial (rather than exponential) in D when η , K , and α are considered ﬁxed.
To proceed, we introduce
T ( d ) =
[Equation]
For any t ≥ T ( d ) ,
[Equation]
Therefore, for a ﬁxed d ,
[Equation]
Therefore, up to a term linear in D , the sample complexity can be bounded by
[Equation]
[Equation]
Notice ﬁrst that P ( Z ≥ ηt 2 σ d − √ d ) is always bounded by 1 .
Second
, since in both the exponential and polynomial noise models σ d decays faster than 1 / √ d asymptotically, T ( d ) = 2( √ 2log K +1) √
dσ d η is uniformly bounded by a constant C ( K, η, α ) due to the exponential decay in d .
Therefore, the term (4) can be bounded by ( DK ) C
( K,η,α ) , which is polynomial (rather than exponential) in the depth D .
Similar analysis and sample complexity statements can be made for the performance of Algorithmand the approximate counterparts.
We defer this treatment to Appendix 8.5.
Experiments.
Our experimental results for the Generative Model follow the same setup and parameters as the experiments for the Constant Gap Model in Section 6.2.
Tables 3 and 4 give the proportion of trials that return the correct optimal one-step action at the 20,000-th expansion.
Figures 6 andshow the success proportion over time, at successive 1000 expansion intervals for Algorithm 2 and
PUCT for η = 1 in both noise models (polynomially and exponentially decaying).
[Table] [Table]
search in the Generative Model with polynomially decaying noise and η = 0 .
C ONCLUSION
We introduce A (cid:63) MCTS, a search algorithm that uses pre-trained value/policy networks to guide exploration and learn to make optimal decisions.
We provide expected sample complexity bounds and demonstrate theoretically and experimentally that our algorithms work efﬁciently for reasonable models of reward distributions and noise (in value/policy network estimates) distribution.
Future work includes more rigorous experimental analysis, improved search algorithms that automatically adapts to the noise level of the value/policy networks, and applications to real scenarios.  A PPENDIX 8. P ROOF OF T HEOREM Proof.
Firstly, the event E that all states s d satisfy | X s d | = | V s d − U s d | ≤ c d happens with probability 1 − β , where X s d are i.i.d. copies of X d . This is because that c d is chosen so that P ( | V s d − U s d | ≤ c d ) ≥ 1 − βDK d for each s d .
Therefore, taking union bound over K d states at depth d and then over all D levels ensures that the event E happens with probability 1 − β .
A (cid:63) MCTS-
V ∗ chooses iteratively the state s d with the highest UCB estimate, U s d + c d and queries the value network for the values of each of the K children of s d .
Under event E , the UCB estimate
U s d + c d is always an over-estimate of V s d .
Since the algorithm always take from the queue the state s d with the highest UCB estimate U s d + c d
and it only stops when the queue Q of possible candidates has only one leaf element, the algorithm is guaranteed to ﬁnd l ∗ .
Suppose otherwise, i.e., | Q | =and Q contains a leaf state l where V l < V ∗ .
Then for an intermediate state s d that is an ancestor to l ∗ , V l < V s d = V ∗ ≤ U s d
+ c d under the event E .
Therefore, as Q is sorted in descending order according to the UCB estimates, s d must be ahead of l in Q , which is a contradiction.
To estimate the complexity, one needs to think about how many non-optimal internal nodes are queried using the value network.
Notice that a sub-optimal internal node s d is chosen if V ∗ ≤
U s d + c d = V s d + X s d + c d .
Then it is necessary to understand the probability of the event that ∆ s d − X s d − c d ≥ 0 , because under that event the algorithm does not need to query the value network for s d .
Therefore the sample complexity N (i,e. the number of states for which the value network is queried) is the given by
[Equation]
where K is the number of children of each node, and A ( s d ) is the set of ancestors of state s d .
Since V s d = V ∗ , ∆ s d = 0 whenever s d ∈ A ( l ∗ ) .
Moreover, because | X s d | ≤ c d under event E , it follows that the expected sample complexity is
[Equation]
8. P ROOF OF T HEOREM
Proof.
Firstly, the event E that all states s d satisfy | V s d − U s d | ≤ c d and | V s d − U πs d | ≤ c π d happens with probability 1 − β , following the same union bound argument as in Theorem 1.
The correctness of A (cid:63) MCTS-
π ∗ V ∗ follows largely from the proof of correctness for A (cid:63) MCTS-
V ∗ .
The main difference in the two algorithms is in the new sampling condition for r i for i > 2 , which is that ln (cid:16) p r 1 p ri − 1 (cid:17) ≤ 2 c πd +1 .
To show why this is relevant, let us elaborate on the case of i = 3 .
Under the event E , V r 3 ≤ U πr 3 + c πd +1 , where c πd +1 depends on the noise distribution X πd +1 .
{+Moreover, V r 1 ≥ U πr 
+}
− c πd +1 .
Hence, one should sample r 3 if U πr 2 + c πd +1 ≥
U πr 1 − c πd +1 , i.e. U πr 1 − U πr 2 ≤ 2 c πd +1 .
{+This is because if U πr 2 + c πd +
+}
<
U πr 1 − c
πd
{++
+}
then it is impossible for V r 3 > V r 1 .
Since the probabilities are given in a softmax form, the condition U πr 1 − U πr 2 ≤ 2 c πd +1 is equivalent to
[Equation]
p r(cid:19)
[Equation]
+ (cid:19) .
When combined with the result from Theorem 1, it gives the following sample complexity bound:
[Equation]
[Equation]
This is equivalent to:
[Equation]
where X π ( d +1) ,k is the noise random variable associated with r k for the policy network,
i
[Equation]
e., U πr k
= V
[Equation]
+ X π ( d
[Equation]
) ,k .
The events 1 (cid:16)
[Equation]
and
[Equation]
are independent, since ∆ s d is the gap between V ∗ and V s d , V r 1 − V r k is the difference in value between two children of s d , and these two quantities are uncorrelated.
Therefore our total expected sample complexity is:
[Equation]
8. A PPROXIMATING V ∗ Algorithm 3 A (cid:63) MCTS- ˆ V
Require: Value Network, additive approximation tolerance δ , failure tolerance β end
10: end while 11: ( 12: Return leads s 8.3. P ROOF OF T HEOREM 3 FOR A LGORITHM
Proof.
Under event E , the correctness for Algorithm 3, A (cid:63) MCTS-
ˆ V largely follows the proof in
Theorem 1, with the exception that the algorithm stops when the most optimistic state in the queue Q is at a special intermediate depth (cid:101) D .
Let ˜ l ∗ be the state at depth
∈ Q d c d
For every node s (cid:101) D at depth (cid:101) D , Q contains an ancestor of s (cid:101) D .
Under the event E , every estimate
U s d + c d is a valid upper bound of V s d and the true value of s d ’s descendants.
It follows that
U ˜ l ∗ − c (cid:101)
D ≤ V ∗ ≤ U ˜
l ∗ + c (cid:101) D .
Therefore, U ˜ l ∗ approximates V ∗ up to additive error δ when c (cid:101) D ≤ δ .
In order to reason about the sample complexity, notice that we query the value network U for a state s d if U ˜ l ∗ < U s d + c d ≤ V s d + X s d + c d with the additional requirement that d ≤ (cid:101) D .
Therefore, it is sufﬁcient to consider the event where
U ˜ l ∗ − V s d ≤ X s d + c d .
Since U ˜ l ∗ is lower-bounded by V ∗ − c
(cid:101) D , it sufﬁces to consider the event of V ∗ −
[Equation]
Therefore the sample complexity N of states queried using the value network is upper-bounded by
N ≤
[Equation]
Since ∆ s d = 0 whenever s d ∈ A ( l ∗ ) and | X s d | ≤ c d under event E , it follows that the expected sample complexity is
[Equation]
8. C ONSTANT G AP M ODEL
The ﬁrst example we will consider is the constant gap model, where ∆( l ) =
η for any l (cid:54) =
l ∗ , and the noise variables X d , X πd are mean-0 Gaussian random variables.
8.4. U SING V ALUE N ETWORK
Lemma 1.
In the Constant Gap Model, the expected sample complexity for Algorithm 1 is:
[Equation]
This translates to a sample complexity of E [ N ] ≤ KD + D 2 ( K − 1) K c , for some constant c that satisﬁes ησ c − √ c ≥ √ 2 log K .
Proof.
For A (cid:63) MCTS-
V ∗ , Algorithm 1, Theorem 1, the sample complexity
E [ N ] = KD + (cid:88)
[Equation]
is easy to evaluate, because the individual events ∆ s d − X s d − c d ≤ 0 and ∆ s d (cid:48) − X s d
[Equation]
− c d
[Equation]
≤are independent.
We claim that
[Equation]
d /
[Equation]
A
[Equation]
l
[Equation]
is equivalent to
[Equation]
To see why this is the case, we can reason about the ﬁrst few levels.
At depth d = 1 , we choose to expand a state with probability P ( η − X 1 − c 1 ≤ 0) .
There are K − 1 states s 1 where s 1 / ∈ A ( l ∗ ) .
If we expand a particular node, the sample complexity is K .
Therefore the expression would be ( K − 1)( K ) · P ( η − X 1 − c 1 ≤ 0) .
At depth d = 2 , there are ( K − 1)( K ) states whose depth 1 ancestor is not in A ( l ∗ ) .
For these states, we choose to expand each with probability P ( η − X 2 − c 2 ≤ 0) P ( η − X 1 − c 1 ≤ 0) .
The sample complexity for each expansion is K , so in expectation, we query ( K − 1)( K 2 ) P ( η − X 2 − c 2 ≤ 0)
P ( η − X 1 − c 1 ≤ 0) states.
There are also K − 1 states who are not in A ( l ∗ ) but their parent in depth 1 are in A ( l ∗ ) , and we expand them with probability P ( η − X 2 − c 2 ≤ 0) .
For these states, we query ( K − 1)( K ) P ( η − X 2 − c 2 ≤ 0) states in expectation.
Continuing this reasoning through for depths 3 , . . .
D − 1 yields the ﬁnal expression in Equation 2.
Therefore the expected sample complexity is:
[Equation]
We can rewrite the above as:
[Equation]
which is equivalent to:
[Equation]
where Z is the standard normal distribution.
Standard tail inequalities gives us that P ( Z ≥ t ) ≤ 1 t 1 √ 2 π e − t 2 2 ≤ e − t 2 2 , where the last inequality holds for t > 1 .
We are interested in a value for t such that e − t 2 2 ≤ e − log K .
This is achieved when t = √ 2 log K .
For exponentially decaying standard deviation as a function of depth, i.e., σ d = 1 α d for α > 1 , when
[Equation]
Since α , and K are constants, there exists a constant c such that
[Equation]
whenever d − j ≥ c .
Therefore, the sample complexity is bounded by:
[Equation]
which is polynomial in the depth D .
For polynomially decaying standard deviation as a function of depth, i.e., σ d = 1 d α for α > 1 , when
[Equation]
Since α , and K are constants, there exists a constant c (cid:48) such that
[Equation]
whenever d − j ≥ c (cid:48) , and a similar sample complexity to Equation 3 holds.
8.4. δ - APPROXIMATE VALUE ESTIMATION
The above reasoning generalizes naturally to a δ -approximate value estimation using A (cid:63) MCTS-
ˆ V ,
Algorithm
[Equation]
and the expected sample complexity is:
[Equation]
8.4. U SING V ALUE AND P OLICY N ETWORK
Lemma 2.
In the Constant Gap Model, the expected sample complexity for Algorithm 2 is:
[Equation]
Proof.
The policy network for this constant gap problem model is not extremely useful, because for suboptimal states, the immediate children have the same optimality gap, and therefore, the probabilities given by the policy network will be fairly evenly distributed among the immediate children nodes.
The policy network can give more differentiated probabilities for the immediate children of a parent state that is an ancestor of the optimal leaf.
First, the sample complexity KD for the optimal path can be improved to
[Equation]
For the ancestor leaf at level d , the number of child leaves that we would expand in expectation using the information from the policy network is
[Equation]
The second expression can be improved to:
[Equation]
To see why this is the case, we can reason about the ﬁrst few levels.
At depth d = 1 , we choose to expand a state with probability P ( η − X 1 − c 1 ≤ 0) .
There are K − 1 states s 1 where s 1 / ∈ A ( l ∗ ) , but in expectation we only consider 2 + K − 1 (cid:80) k
=
[Equation]
P
[Equation]
η + X
[Equation]
1) , 1 + X
[Equation] [Equation]
2 ≤ 2 c π 1 (cid:1) of these states based on the information from the policy network.
If we expand a particular state, the sample complexity is K , since all K children nodes have the same value in this constant gap model, so the sampling criteria using probabilities from the policy network will always be met.
Therefore the expression would be (cid:20) 2 +
[Equation]
At depth d = 2 , there are in expectation
[Equation] [Equation]
) states that we would consider expanding from whose depth 1 ancestor is not in A ( l ∗ ) .
For these states, we choose to expand each with probability P ( η − X 2 − c 2 ≤ 0) P ( η − X 1 − c 1 ≤ 0) .
The sample complexity for each expansion is K , so in expectation, we query (cid:20)
+ K
[Equation]
There are also K − 1 states who are not in A ( l ∗ ) but their parent in depth 1 are in A ( l ∗ ) , and we expand them with probability P ( η − X 2 − c 2 ≤ 0) .
For these K − 1 states, in expectation, we have expanded only
[Equation]
of them, therefore, we expect to query
[Equation]
Continuing this reasoning through for depths 3 , . . .
D − 1 yields the ﬁnal expression:
[Equation] [Equation]
For a δ -approximate value estimate, similar arguments apply to give a sample complexity of:
[Equation]
8. G ENERATIVE M ODEL
The model generates the value function of the nodes in the Markov decision tree in a recursive way.
This model ﬁrst chooses the optimal value
V s
= V ∗ at the root s 0 .
Given the value function of a parent node p , the value functions for the children r 1 , . . .
, r K are generated as follows: one of the children (without loss of generality, say r 1 ) has the same value function as the parent V p ; each of the remaining children r i has a value function V p − Y with Y sampled independently from ∼ U [0 , η ] , the uniform distribution over the interval [0 , η ] .
At level d , there are exactly (cid:0) dt (cid:1)
( K − 1) t states with value function distributed according to V ∗ − ( Y 1 + . . .
+ Y t ) .
When t is non-negligible, Y 1 + . . .
+ Y t concentrates at constant value tη/ 2 by the concentration of measure phenomenon.
Using value networks.
Let us consider the expected complexity of Algorithm 1 for this model.
In the algorithms, c d = (cid:101) C √ dσ d for some constant (cid:101) C .
Following the estimate in Theorem 1, the overall complexity can be bounded up to an extra KD term by
[Equation]
where Z is a standard normal distribution.
The goal is to bound this by a quantity polynomial (rather than exponential) in D when η , K , and α are considered ﬁxed.
To proceed, we introduce
T ( d ) =
[Equation]
For any t ≥ T ( d ) ,
[Equation]
Therefore, for a ﬁxed d ,
[Equation]
Therefore, up to a term linear in D , the sample complexity can be bounded by
[Equation]
Notice ﬁrst that P ( Z ≥ ηt 2 σ d − √ d ) is always bounded by 1 .
Second
, since in both the exponential and polynomial noise models σ d decays faster than 1 / √ d asymptotically, T ( d ) = 2( √ 2log K +1) √
dσ d η is uniformly bounded by a constant C ( K, η, α ) due to the exponential decay in d .
Therefore, the term (4) can be bounded by ( DK ) C
( K,η,α ) , which is polynomial (rather than exponential) in the depth D .
Using value and policy networks.
Let us consider now the expected complexity of Algorithm 2.
Following the argument of the constant gap model, it can be bounded by
[Equation]
which is also polynomial in D as it improves over the complexity of using just the value network.
Approximating V ∗ using value and policy networks.
For a δ -approximate value estimate using a value network, one can stop early at level (cid:101) D instead and the complexity analysis gives a bound
[Equation]
For a δ -approximate value estimate using both policy and value networks, one can stop early at level (cid:101) D instead and the complexity analysis gives a bound
[Equation]