Spending Thinking Time Wisely: Accelerating MCTS with Virtual Expansions
Anonymous Author(s)
AffiliationAddress
Abstract
One of the most important AI research questions is to trade off computation versusperformance since “perfect rationality" exists in theory but is impossible to achievein practice.
Recently, Monte-Carlo tree search (MCTS) has attracted considerableattention due to the significant performance improvement in various challengingdomains.
However, the expensive time cost during search severely restricts itsscope for applications.
This paper proposes the Virtual MCTS (V-MCTS), a variantof MCTS that spends more search time on harder states and less search time onsimpler states adaptively.
We give theoretical bounds of the proposed method andevaluate the performance and computations on 9 × 9 Go board games and Atarigames.
Experiments show that our method can achieve comparable performancesto the original search algorithm while requiring less than 50% search time onaverage.
We believe that this approach is a viable alternative for tasks under limitedtime and resources.
Introduction
When artificial intelligence was first studied in the 1950s, researchers have sought to find the solutionto the question “How to build an agent with perfect rationality".
The term “perfect rationality"[7, 24, 26] here refers to the decision made with infinite amounts of computations.
However, one canonly solve small-scale problems without considering the practical computation time since classicalsearch algorithms usually exhibit exponential running time.
Therefore, recent AI research would nolonger seek to achieve “perfect rationality", but instead carefully trade-off computation versus thelevel of rationality.
People have developed computational models like “bounded optimality" to modelthese settings [26].
The increasing level of rationality under the same computational budget has givenus a lot of AI successes.
Algorithms include the Monte-Carlo sampling algorithms, the variationalinference algorithms, and using DNNs as universal function approximators [9, 8, 13, 30, 17].
Recently, MCTS-based RL algorithms have achieved much success, mainly on board games.
Themost notable achievement is that AlphaGo beats Hui Fan in 2015 [30].
It is the first time a computerprogram beat a human professional Go player.
Afterward, AlphaGo beats two top-ranking humanplayers, Lee Sedol in 2016 and Jie Ke in 2017, the latter of which ranked first worldwide at thetime.
Later, MCTS-based RL algorithms were further extended to other board games and Atarigames [27].
EfficientZero [34] significantly improves the sample efficiency of MCTS-based RLalgorithms, shedding light on its future applications in the real world like robotics and self-driving.
Despite the impressive performance of MCTS-based RL algorithms, they require massive amounts ofcomputation to train and evaluate.
For example, MuZero [27] used 1000 TPUs trained for 12 hoursto learn the game of Go, and for a single Atari game, it needs 40 TPUs to train 12 hours.
Comparedto previous algorithms on the Atari games benchmark, it needs around two orders of magnitude more compute.
This prohibitively large computational requirement has slowed down both the furtherdevelopment of MCTS-based RL algorithms as well as its practical use.
Under the hood, MCTS-based RL algorithms imagine the futures when taking different future actionsequences.
However, this imaging process for the current method is not computationally efficient.
For example, AlphaGo needs to look ahead 1600 game states to place a single stone.
On the contrary,top human professional players can only think through around 100-200 game states per minute [30].
Apart from the inefficiency, the current MCTS algorithm deals with easy and challenging cases withthe same computational budget.
However, human knows to use their time when it is most needed.
In this paper, we aim to design new algorithms that save the computational time of the MCTSbased RL methods.
We make three key contributions : (1) We present Virtual MCTS, a variant of
MCTS, to approximate the vanilla MCTS search policies with less computation.
Moreover, unlikeprevious pruning-based methods that focus on the selection or evaluation stage in MCTS, our methodimproves the search loop.
It terminates the search iterations earlier adaptively when current states aresimpler; (2) Theoretically, we provide some error bounds of the proposed method.
Furthermore, thevisualization results indicate that Virtual MCTS has a better computation and performance trade-offthan vanilla MCTS; (3) Empirically, our method can save more than 50% of search times on thechallenging game Go 9 × 9 and more than 60% on the visually complex Atari games while keepingcomparable performances to those of vanilla MCTS.
Related Work
2.1 Reinforcement Learning with MCTS
For a long time, Computer Go has been regarded as a remarkably challenging game [3, 6].
Researchersattempt to use Monte-Carlo techniques that evaluate the value of the node state through randomplayouts [4, 11, 12, 30].
Afterward, UCT algorithms have generally been applied in Monte-Carlotree search (MCTS) algorithms, which use UCB1 to select action at each node of the tree [20].
Recently, MCTS-based RL methods [30, 32, 31, 27] have become increasingly popular and achievedsuper-human performances on board games because of their strong ability to search.
Modern MCTS-based RL algorithms include four stages in the search loop : selection, expansion,evaluation, and backpropagation.
The computation bottlenecks in vanilla MCTS come from thesearch loop, especially for the evaluation stage and the selection stage of each iteration.
The selectionstage is time-consuming when the search tree becomes wider and deeper.
The evaluation stage isquite expensive because people attempt to evaluate the node value by random playouts to the end inprevious researches.
Due to the search loop, MCTS-based algorithms have multiple model inferencescompared to other model-free RL methods like PPO [28] and SAC [16].
2.2 Acceleration of MCTS
MCTS-based methods have proved their strong capability of solving complex games or tasks.
However, the high computational cost of MCTS hinders its application to some real-time and moregeneral scenarios.
Therefore, numerous works are devoted to accelerating MCTS.
For example, tomake the selection stage more effective, some heuristic pruning methods [14, 33, 29, 1, 2] aim toreduce the width and depth of the search tree with some heuristic functions.
Furthermore, for moreefficient evaluations, Lorentz [22] proposed early playout termination of MCTS (MCTS-EPT) to stopthe random playouts early and use an evaluation function to assess win or loss.
Moreover, Hsueh et al.[18] applied MCTS-EPT to the Chinese dark chess and proved its effectiveness.
Afterward, similarideas have been applied in the evaluation stage of AlphaGoZero [32] and later MCTS-based methods[31, 27, 34].
They evaluate the Q -values through a learnable evaluation network instead of runningplayouts to the end.
Grill et al.
[15] propose a novel regularized policy optimization method basedon AlphaZero to decrease the search budget of MCTS, which is from the optimization perspective.
Danihelka et al.
[10] propose a policy improvement algorithm based on sampling actions withoutreplacement, named Gumbel trick to achieve better performance when planning with few simulations.
However, these methods mentioned above focus on the specific stage of the search iteration or reducethe total budget through pruning and optimization methods, which are orthogonal to us.
And fewworks targets at the search loop.
Lan et al.
[21] propose DS-MCTS, which defines the uncertainty of
MCTS and approximates it by extra DNNs with specific features for board games in training.
Duringthe evaluation, DS-MCTS will check periodically and stop the search if the state is certain.
Background
The AlphaGo series of work [30, 32, 31, 27] are all MCTS-based reinforcement learning algorithms.
Those algorithms assume the environment transition dynamics are known or learn the environmentdynamics.
Based on the dynamics, they use the Monte-Carlo tree search (MCTS) as the policyimprovement operator.
I.e., taking in the current policy, MCTS returns a better policy with the searchalgorithm.
The systematic search allows the MCTS-based RL algorithm to quickly improve thepolicy and perform much better in the setting where heavy reasoning is required.
3.1 MCTS
This part briefly introduces the MCTS method implemented in reinforcement learning applications.
As mentioned in the related works, modern MCTS-based RL algorithms include four stages in thesearch loop, namely selection, expansion, evaluation, and backpropagation.
MCTS takes in the current states and generates a policy after the search loop of N iterations.
Here Nis a constant number of iterations set by the designer, regarded as the total budget.
In the selectionstage of each iteration, an action will be selected by maximizing over UCB.
Specifically, AlphaZero[31] and MuZero [27] are developed based on a variant of UCB, P-UCT [25] and have achieved greatsuccess on board games and Atari games.
The formula of P-UCT is the Eq (1):
[Equation]
where k is the index of iteration, A is the action set, Q ( s, a ) is the estimated Q-value, P ( s, a ) isthe policy prior obtained from neural networks, N ( s, a ) is the visitations to select the action a fromthe state s
and c 1 , c 2 are hyper-parameters.
The output of MCTS is the visitation of each actionof the root node.
After N search iterations, the final policy π ( s ) is defined as the normalized rootvisitation distribution π N ( s ) , where π k ( s, a ) =
N ( s, a ) / (cid:80) b ∈A N ( s, b ) =
N ( s, a ) /k, a ∈ A .
Forsimplification, we use π k in place of π k ( s ) sometimes.
And the detailed procedure of MCTS isintroduced in Appendix.
In our method, we propose to approximate the final policy π N ( s ) with ˆ π k ( s ) ,which we name as a virtual expanded policy, through a new expansion method and a termination rule.
In this way, the number of iterations in MCTS can be reduced from N to k .
3.2 Computation Requirement
Most of the computations in MCTS-based RL are in the MCTS procedure.
Each action taken by
MCTS requires N times neural network evaluations, where N is a constant number of iterations inthe search loop.
Traditional RL algorithms, such as PPO [28] or DQN [23], only need a single neuralnetwork evaluation per action.
Thus, MCTS-based RL is roughly N times computationally moreexpensive than traditional RL algorithms.
In practice, training a single Atari game needs 12 hours ofcomputation time on 40 TPUs [27].
The computation need is roughly two orders of magnitude morethan traditional RL algorithms [28], although the final performance of MuZero is much better.
Method
We aim to spend more search time on harder states and less on easier states.
Intuitively, human knowswhen to make a quick decision or a slow decision under different circumstances.
Unfortunately,this situation-aware behavior is absent in current MCTS algorithms.
Therefore, we propose an MCTS variant that terminates the search iteration adaptively.
It consists of two components: a novelexpansion method named virtual expansion to estimate the final visitation based on the current partialtree; a termination rule that decides when to terminate based on the hardness of the current scenario.
And we will display the adaptive mechanism through visualizations in Section 5.5.
4.1 Termination Rule
We propose to terminate the search loop earlier based on the current tree statistics.
Intuitively, we nolonger need to search further if we find that recent searches have little changes on the root visitationdistribution.
With this intuition in mind, we propose a simple modification to the MCTS searchalgorithm.
As mentioned in 3.1, π k ( s ) is the policy defined by the visitations of the root state atiteration k .
Let ∆ s ( i, j ) be the L1 difference of π i ( s ) , π j ( s ) , namely ∆ s ( i, j )
=
||
π i ( s )
− π j ( s ) || 1 .
Then we terminate the search loop when we have searched at least rN iterations and ∆ s ( k, k/ 2) <
ϵ ,where ϵ is a tolerance hyper-parameter, r ∈ (0 , 1) is the ratio of the minimum search budget and Nis the full search iterations.
We show that under certain conditions, a bound on ∆ s ( k, k/ 2) impliesa bound on ∆ s ( k, N ) .
∆ s ( k, N ) measures the distance between the current policy π k ( s ) and theoracle policy π N ( s ) .
In this way, ∆ s ( k, k/ 2) reflects the hardness of the state s .
Consequently, oncethe gap is small enough, it is unnecessary for more search iterations.
4.2 Virtual Expansion in MCTS
[Equation]
[Equation]
[Equation]
[Equation]
a )+ 11:
N k +1 ( s, a ) =
N k
( s, a ) +12:
end for 13: Return Q k +1 ( s, a ) ,
N k +1 ( s, a )
1: Current k -th iteration step: 2: Input: A , P, Q k ( s, a ) , N k
( s, a ) , ˆ N k
( s, a ) 3: 4: if Not init ˆ N k ( s, a ) then 5:
Init: ˆ N k
( s, a ) ← N k
( s, a ) 6: end
if 7: 8: s ← s root 9: a ∗ ← UCB1 ( Q, P, ˆ N ) 10: ˆ N k
( s, a )
← ˆ N k ( s, a ) +11: 12: Return ˆ N k
( s, a )
For the termination rule ∆ s ( k, k/ 2) <
ϵ , we assume π i and π j are directly comparable.
However,they are not directly comparable because the tree is expanded with UCT.
As the number of visitsincreases, the upper bound would be tighter, and the latter visits are more focused on the promisingparts.
Thus earlier visitation distributions (smaller iteration number) can exhibit more exploratorydistribution, while latter ones (larger iteration number) are more exploitative on promising parts.
To compare π i and π j properly, we propose a method called virtual expansion in place of the vanillaexpansion.
Briefly, it aligns two distributions by virtual UCT expansions until the constant budget N .
When the tree is expanded at iteration k , it has N − k iterations to go.
A normal expansion wouldrequire evaluating neural network N − k times for a more accurate Q ( s, a ) estimate for each arm atthe root node.
Our proposed virtual expansion still expands N − k times according to UCT, but itignores the N − k neural network evaluations and assumes that each arm’s Q ( s, a ) does not change.
We denote the virtual expanded distribution from π i as a virtual expanded policy ˆ π i .
By doingvirtual expansions on both π i and π j , we will obtain the corresponding virtual expanded policiesˆ π i , ˆ π j .
Here we effectively remove the different levels of exploration/exploitation in the two policies.
Then the termination condition becomes the difference of virtual expanded policies.
We name therule as VET-Rule (Virtual Expanded Termination Rule):
[Equation]
The comparisons between vanilla expansion and virtual expansion are illustrated in Algorithm 1and 2.
The time-consuming computations are highlighted in Algorithm 1.
Line 4 to 7 in Algorithm1 target at searching with UCT to reach an unvisited state for exploration.
Then it evaluates thestate and backpropagates along the search path to better estimate Q -values.
After total N iterations,the visitation distribution of the root node π N ( s ) is considered as the final policy π ( s ) .
However,in virtual expansion, listed in Algorithm 2, it only searches one step from the root node.
Andit selects actions based on the current estimations without changing any properties of the search tree.
Furthermore, the virtual visited counts ˆ N k
( s, a ) are changed after virtual visits to balance theexploitation and the exploration issue.
After N − k times virtual expansion, the virtual expandedpolicy becomes ˆ π k ( s, a ) =
ˆ N k
( s, a ) /N
instead of N k ( s, a )
/k .
When k = N , further searches afterthe root have no effects on the final policy.
So ˆ π N ( s, a ) =
π N ( s, a ) .
1: Input: budget N , state s , conservativeness r , error ϵ 2: Notice: π k ( s ) , ˆ π k ( s ) are policy distributions.
3: Notice: π k ( s, a ) , ˆ π k ( s, a ) are probabilities for action a .
4: for k ∈ N do 5:
Selection with UCB 6:
Expansion for the new node 7:
Evaluation with Neural Networks 8:
Backpropagation for updating Q and visitations 9: π k ( s, a ) ← N k ( s, a ) /k 10:
Virtual expand N − k nodes and update ˆ N ( s, a ) 11: ˆ π k ( s, a ) ← ˆ N k ( s, a ) /N
if k ≥ rN ∧ (cid:12)(cid:12)(cid:12)(cid:12)
ˆ π k ( s ) − ˆ π
k/ 2 ( s ) (cid:12)(cid:12)(cid:12)(cid:12) 1 <
ϵ then 13: π ( s ) ← ˆ π k ( s ) 14:
Break 15: end if 16: π ( s ) ← π k ( s ) 17: end for 18: Return π ( s )
The procedure of MCTS with VETRule is listed in Algorithm 3.
Wename our method Virtual MCTS (VMCTS), a variant of MCTS with VETRule.
Compared with the original
MCTS, lines 8-13 are the pseudo-codefor the rule.
In each iteration, we dosome calculations with little cost tojudge whether the VET-Rule is satisfied.
If it is, then the search processis terminated and returns the currentvirtual expanded policy ˆ π k ( s ) .
Thus,it skips the next N − k model predictions from neural networks in theevaluation stage highlighted in line 7.
In this way, we can approximate theoracle distribution π N by ˆ π k while reducing the budget of N simulationsto k .
Here, k ≥ rN and r is a hyperparameter of the minimum budgetrN .
We can reduce the tree size by1 /r times at most.
4.4 Theoretical Justifications
Furthermore, we will give some theoretical bounds on the Q -values and ˆ∆ s ( k, N ) of V-MCTS.
Before this, we define some notations first: k is the index of the current search iteration, and N is thenumber of total search iterations.
A is the action set and |A| > 1 .
Each action a ∈ A is associatedwith a value, which is a random variable bounded in the interval [0 , 1] with expectation Q a .
At step k ,the empirical mean value over the k trails is Q k
( s, a ) .
For simplification, we denote Q k ( s, a ) as ¯ Q ka .
We denote the empirical mean value after N − k virtual expansion as ˆ Q Na .
Since we only deal withthe visitation distribution of the root, we omit the state subscript for the root state.
For convenience,we assume that different actions are ordered by their corresponding expected values, which meansthat 1 ≥ Q 1 ≥ Q 2 ≥ · · · ≥ Q a ≥ · · · ≥ Q |A| ≥ 0 .
Theorem 4.1.
Given r ∈ (0 , 1) , confidence δ ∈ (0 , 1) , finite action set A .
∃ N 0 > 0 , ∀ N >
N 0 , k ≥ rN , let ϵ k = (cid:113)
k
ln 100 k 2 δ , after k times vanilla expansion and N − k times virtualexpansion, we have (a) Value Consistency in Virtual Expansion : Pr { (cid:84) a ∈A (cid:12)(cid:12)(cid:12) ˆ Q Na − Q a (cid:12)(cid:12)(cid:12)
<
ϵ k } >(1 − eδ |A| 50 r 2 N 2 )
Expansion : Pr { (cid:12)(cid:12)(cid:12) ˆ Q N ∗ − ¯ Q N 1 (cid:12)(cid:12)(cid:12) <
ϵ k + ϵ N } >1 − 2( δ 50 k 2 exp ( 1 1 .
√ k )
+ δ 50 N 2 exp ( 1 N )) ,
where ∗ := arg max a ∈A ¯ Q ka , e is the Euler’s number.
Theorem 4.1 (a) gives a bound of the distance between the empirical mean values after virtualexpansions and the expected values.
Noticed that lim N →∞ ϵ rN =
and lim N →∞ eδ |A| 50 r 2 N 2 =0 .
It tells that, after enough trails, the expected Q -values of all actions can be estimated by thecorresponding empirical Q -values after virtual expansion.
Furthermore, when the Q -values haveconverged, the effect of virtual expansion is the same as that of vanilla expansion.
Denote the bestempirical action as ∗ , and the best expected action is 1 because Q 1 ≥ Q a .
Theorem 4.1 (b) notes thatthe Q -value of the best empirical action with virtual expansion is of high probability to be close tothe Q-value of the best expected action with vanilla expansion.
Intuitively, it tells that whether or not we successfully find the best expected action, the best empirical action has similar effects to the bestexpected action.
The proof is attached in Appendix.
Theorem 4.2.
( Error Bound of V-MCTS ): Given r ∈ (0 , 1) , confidence δ ∈ (0 , 1) , finite actionset A .
Suppose the virtual expanded policy ˆ π k is generated from Algorithm 3 (V-MCTS), ∃ N 0 > 0 ,∀ N > N 0 , k ≥ rN , ∀ ϵ ∈
(0 , 1] , if (cid:12)(cid:12)(cid:12)(cid:12)
ˆ π k ( s ) − ˆ π
k/ 2 ( s ) (cid:12)(cid:12)(cid:12)(cid:12) 1 <
ϵ , we have Pr {
|| π N ( s ) − ˆ π k ( s ) ||
1 <3 ϵ } > 1 − eδ |A| 50 N 2 (1 + 4 r 2 ) , where e is the Euler’s number.
Theorem 4.2 tells that a bound of ˆ∆ s ( k, k/ 2) implies a bound of ˆ∆ s ( k, N ) with high probability.
Noticed that lim N →∞ eδ |A| 50 N 2 (1 + 4 r 2 )
= 0 .
Therefore, the oracle policy π N ( s ) can be approximatedby ˆ π k ( s ) after enough trails.
The proof of this theorem is attached in Appendix.
Given the minimum distance ϵ , for easier states, the rule ˆ∆ s ( k, k/ 2) <
ϵ is easier to satisfy.
That’sbecause the Q-values of the tree nodes keep in a small range even with more search iterations.
Thus,the virtual expanded policy generated by V-MCTS is close to the oracle policy, and the search loopwill be terminated earlier if the state is easier.
In the next section, we do ablations to investigate theeffects of the hyper-parameters and show visualizations to verify the adaptive behavior.
Experiments
In this section, the goal of the experiments is to prove the effectiveness and efficiency of V-MCTS.
First, we compare the performance and the cost between the vanilla MCTS and our method.
Specifically, we evaluate the board game Go 9 × 9 , and a few Atari games.
In addition, we do some ablationsto examine the virtual expansion’s effectiveness and evaluate the sensitiveness of hyper-parameters.
Finally, we show the adaptive mechanism through visualizations and performance analysis.
5.1 Setup
Models and Environments Recently, Ye et al.
[34] proposed EfficientZero, a variant of MuZero [27]with three extra components to improve the sample efficiency, which only requires 8 GPUs in training,and thus it is more affordable.
Here we choose the board game Go 9 × 9 and a few Atari games asour benchmark environments.
The game of Go tests how the algorithm performs in a challengingplanning problem.
And Atari games feature visual complexity.
Hyper-parameters As for the Go 9 × 9 , we choose Tromp-Taylor rules.
The environment of Gois built based on an open-source codebase, GymGo [19].
We evaluate the performance of the agentagainst GNU Go v3.8 at level 10 [5] for 200 games.
We include 100 games as the black player and100 games as the white one with different seeds.
We set the komi to 6.5, as most papers do.
As forthe Atari games, we choose 5 games with 100k environment steps.
In each setting, we use 3 trainingseeds and 100 evaluation seeds for each trained model.
More details are attached in Appendix.
Baselines We compare our method to EfficientZero with vanilla MCTS, on Go 9 × 9 and some Atarigames.
Moreover, DS-MCTS [21] also terminates the MCTS adaptively through trained uncertaintynetworks.
But it requires specific features designed for Go games, and it only works in the evaluationstage.
Therefore, we also compare the final performance for Go games with the DS-MCTS.
5.2 Results on Go
Figure 1(a) illustrates the computation and performance trade-off on Go against the same GnuGo(level 10) agent.
The x-axis is the training speed, and the y-axis is the winning rate.
Therefore, thecurve which lies to the top-left has better performance than the bottom-right in terms of the trade-off.
We train the baseline method with constant budgets N , which is noted blue points.
Besides, wealso train with hyperparameters r
We evaluate trained
Firstly, for all the methods, more search iterations (larger N or smaller eps ) lead to higher winningrates but result in more response time.
Secondly, V-MCTS ( ϵ = 0 . 1 ) achieves 71% winning rateagainst the GnuGo level 10, which is close to 75% from MCTS ( N = 150 ).
And the time costof V-MCTS ( ϵ = 0 . 1 ) for a one-step move is 0.12s while the GnuGo engine is 0.18s and MCTS
[Figure]
( N = 150 ) is more than 0.2s.
Therefore, such termination rule can keep strong performances withless budget.
For a more detailed breakdown of the time consumption of V-MCTS, please see the
Appendix.
Finally, we can find that the red dashed line lies to the right of the blue one.
It indicatesthat V-MCTS is better than the vanilla MCTS considering the computation and performance trade-off.
Figure 1(b) illustrates the changes of winning rates and the average tree size over the training stage.
Here, as the red dashed line shows, the tree size of V-MCTS varies over training and keeps smallerthan the maximum size ( N = 150 ) while the winning rate keeps comparable to the MCTS ( N = 150 ).
Consequently, V-MCTS can work well not only in evaluation but also in training.
[Table]
We also compare ourmethod to DS-MCTS [21],which terminates the searchwhen the state is predictedto be certain with DNNs.
To make fair comparisons,we implement the DSMCTS and follow theirdesign of features for Gogames.
We set N max = 150 , c
= { 30 , 75 , 120 } , thr = { . 1 , .
1 } in DS-MCTS.
Then we comparethe winning rate and the average budget among the vanilla MCTS, DS-MCTS, and V-MCTS.
Experiments show that V-MCTS outperforms the DS-MCTS in both aspects, listed in Table 1.
Weattribute the better performance of V-MCTS to the virtual expanded policy.
DS-MCTS chooses π k ( s )as the policy after the termination of the search, while V-MCTS chooses ˆ π k ( s ) , which has theoreticalguarantees to approximate π N ( s )
[Table] [Table]
[Figure]
5.3 Results on Atari
Apart from the results of Go, we also evaluate our method on some visually complex games.
Sincethe search space of Atari games is much smaller than that of Go and the Atari games are easier, wechoose a few Atari games to study how the proposed method impacts the performance.
We follow thesetting of EffcientZero, 100k Atari benchmark, which contains only 400k frames data.
The results areshown in Table 2.
Generally, we find that our method works on Atari games.
The tree size is adaptive,and the performance of V-MCTS is still comparable to the MCTS with full search trails.
It has betterperformance than the MCTS( N = 30 ) while requiring fewer searches, proving the effectiveness andefficiency of our proposed method.
The Hero game is an outlier here.
But our performance is veryclose to the vanilla MCTS ( N = 30 ) while we use half of the search iterations on average.
Besides,the number of search times decreases more than that on Go.
To sum up, V-MCTS can keep comparable performance under fewer search iterations while simplyreducing the total budget of MCTS will encounter a more significant performance drop.
In addition,the savings of search cost is more substantial in easier environments.
5.4 Ablation Study
The results in the previous section suggest that our method reduces the response time of MCTSwhile keeping comparable performance on challenging tasks.
This section tries to figure out whichcomponent contributes to the performance and how the hyperparameters affect it.
Virtual Expansion
In Section 4.2, we introduce the virtual expansion.
To prove the effectiveness ofvirtual expansion, we compare it with another two baseline expansion methods.
One is the vanillaexpansion, mentioned in Algorithm 1, which returns at iteration k and outputs π k .
Another is greedyexpansion, which spends the left N − k simulations in searching the current best action greedily,indicating that ˆ π k ( s, a ) =
( N k ( s, a ) + ( N − k ) 1 b =
argmax
N
k
( s,b ) ) /N .
Briefly, we stop the searchprocess after k = 30 iterations and do N − k times virtual expansion or greedy expansion or nothing,where k = rN and r = 0 .
2 , N = 150 .
[Table]
We compare the winning rate against the sameengine, and the results are listed as Table 3shows.
The winning rate of virtual expansioncan achieve 32%, which is much better than theothers.
Besides, MCTS with greedy expansiondoes not work because it over-exploits and results in severe exploration issues.
Consequently,virtual expansion can generate a better policydistribution because it can balance explorationand exploitation with UCT.
Termination Rule It is significant to explore a better termination rule to keep the sound performancewhile decreasing the tree size as much as possible.
As mentioned in Section 4.1, VET-Rule has two
[Figure]
k= k= k= k= k= k= k= hyperparameters r, ϵ .
Here r is the factor of the minimum budget rN , and ϵ is the minimum distanceˆ∆ s ( k, k/ 2) .
To explore the VET-Rule with better computation and performance trade-off, we doablations for the different values of r and ϵ , respectively.
The default values of r, ϵ are set to 0 .
Figure 2 compares the winning rate as well as the average tree size across the training stage.
Firstly,
Figure 2(a) gives the results of different minimum search times factor r .
The winning probability isnot sensitive to r when r ≥ 0 .
Nevertheless, the average tree size is sensitive to r because V-MCTSis supposed to search for at least rN times.
In addition, there is a performance drop between r = 0 .
1and r = 0 .
Therefore, it is reasonable to choose r = 0 .
2 to balance the speed and the performance.
Besides, the comparisons of the different minimum distance ϵ are shown in Figure 2(b).
A larger ϵmakes the tree size smaller because ˆ∆ s ( k, k/ 2) <
ϵ is easier to satisfy.
In practice, the performance ishighly correlated with ϵ .
In terms of the winning rate, a smaller ϵ outperforms a larger one.
However,better performances are at the cost of more computations.
We suggest selecting an appropriateminimum distance to balance the computation and performance ( r = 0 .
2 , ϵ = 0 . 1 ).
5.5 Visualization of V-MCTS’s Adaptive Behavior
In this section, we display the adaptive mechanism through visualizations and performance analysis.
We find that (1) ˆ π k ( s ) is close to π N ( s ) ; (2) V-MCTS terminates earlier for simpler states.
Specifically, we choose some states at different time steps on one game of Go against the GnuGowith a trained model.
And Figure 3 is the visualization of the policy distributions heatmap.
Here,our agent is the black one.
The two rows are the heatmap visualization for oracle MCTS ( π N ) and
V-MCTS ( ˆ π k
when ˆ∆ s ( k, k/ 2) <
ϵ ).
The darker the color is on the grid, the more the correspondingaction is visited during the search.
In general, ˆ π k is close to the π N at distinct states, indicating thatthe virtual expanded policy obtained after virtual expansion is close to the oracle one.
Furthermore, the less valuable actions there are, the sooner the V-MCTS will terminate.
For example,on Go games, the start states are usually not complex because there are only a few stones on theboard, but the situations are more complicated in shuban, the closing stage of the game.
Notably, thetermination occurs earlier in the start states (columns 1, 2), but it is the opposite when the situation ismore complicated.
More importantly, the termination step k is not related to the number of Go pieces.
Therefore, we can conclude that V-MCTS makes adaptive terminations according to the situations ofthe current states and generate near-oracle policies.
Specifically, it terminates the search loop earlierwhen dealing with easier states, which has a better computation and performance trade-off.
Discussion
This paper proposes a novel method named V-MCTS to accelerate the MCTS to determine thetermination of search iterations.
It can maintain comparable performances while reducing half of thetime to search adaptively.
We believe that this work can be one step toward applying the MCTS-basedmethods to some real-time domains.
One limitation of our work is that it cannot deal with theenvironments of continuous action space.
In the future, we will plan to extend to the continuousaction space with early termination.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section
• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary
• Did you include the license to the code and datasets?
[N/A]
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]