L EFT - HAND H EAVY -T AILS AND THE E FFECTIVENESS OF THE P OLICY AND V ALUE N ETWORKS IN DNNBASED BEST - FIRST SEARCH FOR S OKOBAN P LANNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite the tremendous success of applying traditional combinatorial search methods in various NP-complete domains such as SAT and CSP as well as using deep reinforcement learning to tackle two-player games such as Go, certain classes of PSPACE-hard planning problems have remained out of reach.
Even carefully designed domain-speciﬁc solvers can fail quickly due to the exponential combinatorial search space on hard instances.
Recent work that combines traditional search methods, such as best-ﬁrst search and Monte Carlo tree search, with Deep Neural Networks’ (DNN) heuristic prediction has shown promising progress.
These methods can solve a signiﬁcant number of hard planning instances beyond specialized solvers.
To better understand why these approaches work, we studied the interplay of the policy and value networks in DNN-based best-ﬁrst search on the Sokoban domain and show the surprising effectiveness of the policy network, further enhanced by the value network, as a guiding heuristic for the search.
To further understand the phenomena, we studied the cost distribution of the search algorithms and found that Sokoban planning instances can have heavy-tailed runtime distributions, with tails both on the left and right-hand sides.
In particular, for the ﬁrst time, we show the existence of left-hand heavy tails and propose an abstract tree model that can empirically explain the appearance of these tails.
We provide extensive experiment data supporting our model.
The experiments show the critical role of the policy network as a powerful heuristic guiding the search, which can lead to left-hand heavy tails with polynomial scaling by avoiding exploring exponentially sized sub-trees.
Our results also demonstrate the importance of random restart strategies, as are widely used in traditional combinatorial solvers, for DNN-based search to avoid left and right-hand heavy tails.
I NTRODUCTION
Combinatorial search is a key domain for artiﬁcial intelligence.
Unfortunately, combinatorial domains usually have intractable theoretical complexity, such as NP-complete, PSPACE-complete, or even undecidable.
In the past few decades, we have observed tremendous progress for practical problem solving in NP-hard domains with wide applicability in, for example, circuit design (Hong et al., 2010), hardware veriﬁcation (Gupta et al., 2006), and mathematical discovery (Konev & Lisitsa, 2014).
SAT solvers based on conﬂict-driven clause learning can solve instances with thousands of variables and clauses in seconds, which demonstrates surprising scaling performance despite SAT being an NP-complete task (Silva & Sakallah, 2003).
In contrast, practical combinatorial search in PSPACE-hard domains has remained a signiﬁcant practical challenge.
PSPACE-hard problems can be generally divided into two main categories: two-player games , such as Go, Chess, and Amazons (Lichtenstein & Sipser, 1980; Fraenkel &
Lichtenstein, 1981; Hearn, 2005), and single-agent planning problems (a.k.a. combinatorial puzzles), such as Sokoban and problems formalized by PDDL (Planning Domain Deﬁnition Language) (Culberson, 1997; Bylander, 1994).
Recent achievements in the deep learning community inspired an approach of augmenting Monte Carlo tree search (MCTS) with Deep Neural Networks’ (DNN) heuristic predictions.
AlphaGo (Silver et al., 2016) became the ﬁrst Go software to beat professional human players in 2016, and its newer and more general version AlphaZero (Silver et al., 2017) can achieve, tabula rasa , superhuman performance in many other challenging game domains.
These advances naturally raise the question of whether we can build a general framework to learn different planning domains with minimum modiﬁcations.
A key challenge is that hard planning instances often require intricate action sequences with hundreds of steps, and any deviation can lead to a dead-end state with no path to goal states.
To address this issue, a more systematic and complete search, such as best-ﬁrst search, is preferred over MCTS (Agostinelli et al., 2019).
Traditional search methods augmented with neural networks’ heuristics have shown promising progress in planning domains (Shen et al., 2020; Rivlin et al., 2020; Ferber et al., 2020; Feng et al., 2020a;b).
These methods can solve a signiﬁcant number of hard planning instances that specialized solvers cannot solve.
To gain a better understanding of these approaches, we studied the interplay of the policy and value networks in DNN-based best-ﬁrst search algorithms.
Our experiments show the surprising effectiveness of the policy network, further enhanced by the value network, as a guiding heuristic.
To further understand the phenomena, we also studied the cost distribution of DNN-based search.
Speciﬁcally, we explored and generalized the cost distribution proﬁles of search methods from NPhard domains to PSPACE-hard planning domains, and show heavy-tailed cost distributions exist ubiquitously among planning instances.
For the ﬁrst time, we found and characterized left-hand heavy tails , which are different from well-studied right-hand heavy tails (also abbreviated as heavy tails in the literature). Left-hand heavy tails occur when instances become extremely hard and most runs cannot ﬁnish in a reasonable time limit. The solver needs to be ”lucky” and occasionally hit a short run.
In contrast, right-hand heavy tails characterize mildly hard instances whose majority randomized runs have short runtime.
Meanwhile, the solver can be ”unlucky” and occasionally hit an extremely long run which makes the expected runtime exponential.
We propose an abstract search tree model that can empirically explain the appearance of left-hand heavy tails and provide extensive experiment data supporting our model. The experiments show the critical role of the policy network as a powerful heuristic guiding the search, which can lead to left-hand heavy tails with polynomial scaling by avoiding exploring exponentially sized sub-trees.
Randomized combinatorial solvers use various techniques, such as randomized tie-breaking and random variable ordering, to carefully inject a controlled amount of randomization into a deterministic search procedure (Crawford & Baker, 1994; Bresina, 1996; Gomes et al., 1998).
The randomization step requires careful engineering and analysis of the solver since excess randomization can hamper the effectiveness of random restarts.
In our approach, we found that uncertainty-aware networks (Huang et al., 2017; Chua et al., 2018; Sedlmeier et al., 2019; Malinin & Gales, 2018) can provide just the right amount of controllable randomization into a deterministic search algorithm.
In this paper, we consider Sokoban as the background planning domain.
The only domain knowledge neural networks and search algorithms receive is 1) input state representation; 2) state-action transition function; 3) deciding goal states. These three components are the minimum requirements to describe any planning domain and we did not use any other Sokoban-speciﬁc techniques, such as dead-end detection.
We evaluated DNN-based search on more than 10 , 000 instances with signiﬁcant variations in underlying combinatorial structure.
Our results also demonstrate the importance of random restart strategies , as are widely used in traditional combinatorial solvers, for deep reinforcement learning (DRL) and deep AI planning systems to avoid left and right-hand heavy tails.
In summary, our overall contributions are as follows:
We studied the interplay of the policy and value networks on DNN-based best-ﬁrst search for the Sokoban domain.
Our experiments show the surprising effectiveness of the policy network, further enhanced by the value network, as a guiding heuristic for the search.
We studied the runtime distribution on more than 10 , 000 instances and propose distributionindependent statistics to quantify the heaviness of tails and effectiveness of random restarts
For the ﬁrst time , we extensively studied left
-hand heavy tails from experiment data , introduce an abstract search tree model with critical nodes and formally show how left-hand heavy tails can arise during the search.
Left-hand heavy tails of runtime distributions are explained by the critical role of the policy network as a guiding heuristic.
Polynomial runtime scaling can occur because the policy network helps avoid exploring exponentially sized sub -trees during the search
We show the importance of using uncertainty-aware networks in the planning domain and how it can add a controllable amount of randomization to a deterministic solver.
We show how a restart strategy can improve DNN-based search’s effectiveness.
In particular, our experiments show for larger budgets, more frequent restarts are more effective.
B ACKGROUND AND R ELATED W ORK
Sokoban as a planning domain. Sokoban is a PSPACE-complete puzzle whose goal is to push a set of boxes into the same number of goal cells in a grid maze with walls (Culberson, 1997).
Sokoban is among the most challenging known AI planning domains. The domain remains challenging even for specialized solvers with signiﬁcant human domain knowledge (Fern et al., 2011; Junghanns & Schaeffer, 2001).
Due to its general search structure and hardness, we use Sokoban as our background domain throughout the paper.
Optimal speedup of Las Vegas algorithms. Let A be a randomized algorithm that always outputs the correct answer when it halts but whose running time is a random variable r A : Z + ∞ → R +0 .
Luby et al.
(1993) proved that when we have full knowledge about the distribution r A , the optimal strategy that achieves the minimum expected time required to obtain an output from A is to repeatedly run
A for the same amount of time t A until it halts.
To calculate t A , let
[Equation]
be the expected halting time of repeatedly running A with time limit t .
Deﬁne l A = inf t< ∞ l ( t ) and l A is ﬁnite for any non-trivial distribution r A , i.e., r A ( ∞ ) < 1 .
Let t A be any ﬁnite value of t such that l ( t )
=
l A , if such a value exists, or t A = ∞ otherwise.
Luby et al.
(1993) also showed that when r A is unknown, the universal strategy that runs A for time limit 1 , 1 , 2 , 1 , 1 , 2 , 4 , 1 , 1 , 2 , 1 , 1 , 2 , 4 , 8 , 1 , 1 , 2 , can achieve estimated halting time O ( l A log( l A )) for any randomized algorithm A .
This bound is optimal among all universal strategies.
hand
) observed ized can particular so instances,
.e runs instance halt extremely runs makes the average [Equation]
Uncertainty-aware network.
There is a line of research on the uncertainty of neural networks to reduce test error, provide conﬁdence estimate, and improve model-based reinforcement learning (Huang et al., 2017; Chua et al., 2018; Sedlmeier et al., 2019; Malinin & Gales, 2018).
Our method augments neural networks with Monte Carlo (MC) dropout to introduce randomization to deterministic search engines (Gal & Ghahramani, 2016). MC dropout enables dropout layers during testing and the dropout rate can control the amount of randomization.
For our experiment, the uncertainty comes from two main sources: (1) the distributional mismatch between the training and test datasets; (2) noises in the training data since the remaining distances (found by specialized solvers) are usually not optimal.
See Section 3 for more details about data preparation.  F ORMAL FRAMEWORK
Policy-guided best-ﬁrst search.
Best-ﬁrst search is an informed search algorithm, which explores a graph by expanding the most promising node chosen according to an evaluation function f ( n ) from the open set (search boundary nodes).
f ( n ) can use both the knowledge acquired so far while exploring the graph, denoted by g ( n ) , and a heuristic function h ( n ) , which estimates the remaining distance to the nearest goal state. Starting from the start state, best-ﬁrst search gradually enlarges the current search graph by consecutively expanding a new node n which minimizes the evaluation function f ( n ) and adds n to the closed set (expanded nodes). The search considers duplicate state detection and merges different nodes with the same state into a single node. Sokoban has unit cost so that g ( n ) represents the depth of the node n .
The heuristic function h ( n ) is estimated by a value network, which is explained in further detail below. Orseau & Lelis (2021) proposed Policy-guided Heuristic Search (PHS) to further learn a policy network, which takes a state s as input and outputs a vector of action probabilities p with components p ( a | s ) for each valid action a of s .
Speciﬁcally, they adapted the evaluation function to [Equation] where ( s 0 , ..., s m ) is the sequence of states from the root node to n .
Orseau & Lelis (2021) also proposed PHS*, a variant of PHS, that uses the evaluation function f ( n ) = [Equation] Both PHS and PHS* require computing the cumulative product of probability predictions among the whole path from the root to n .
In this paper, we use a new evaluation function which only depends on the probability prediction of the incoming nodes of n :
[Equation] Experiment data shows using this simpler evaluation function can consistently solve more instances. Data preparation.
The study of the complexity and practical performance of search methods is greatly hampered by the difﬁculty in collecting real data.
As an alternative, researchers heavily resort to procedurally generated instances or highly structured problem domains (Taylor & Parberry, 2011; Guez et al., 2018).
The randomly generated instances lack sufﬁcient structure and their underlying combinatorial search space is, in some sense, too regular.
[Table]
Our
[Table]
To bridge this gap, we collected all the Sokoban instances from the Sokobano website 2 , resulting in 10871 instances in total.
All these instances were designed by different human authors in the past few decades, have great variation in the underlying structure, serve as the benchmark for specialized solvers, and also exhibit practical interest for humans to solve.
The dataset is orders of magnitude larger than the ones used in previous works on DRL and provides a great challenge for deep heuristic learning.
See Table 1 for Sokoban board statistics compared with previous works.
Notice that the difﬁculty of Sokoban grows exponentially as the number of boxes increases.
To generate supervised training data, we ran Sokolution 3 , a state-of-the-art Sokoban solver, to compute ground truth plans.
Sokolution can solve 8272 out of 10871 total instances given a 10-minute time limit.
We randomly divided the solved 8272 instances into a training set (7435 instances) and a test set (827 instances).
For the remaining unsolved 2609 instances, we randomly sampled 200 instances and reran Sokolution with extended 2-hour time limit to solve.
137 out frominstances remain unsolved and we collected these 137 instances as the hard set to further study the cost distribution of instances that are way harder than the training instances.
For each found plan ( s 0 , a 1 , s 1 , ..., a n , s n ) from the start state s 0 to the goal state s n , we generated training tuples ( s i , l s i , v s i ) with policy label l s i
= a i +1 and remaining distance label
v s i = n − i as training data.
Feng et al.
(2020b) used PUSH as basic actions to achieve the state-of-the-art performance of DRL for the Sokoban domain.
In this work, we use the more elementary action MOVE.
A PUSH action can be divided into two parts: 1) moving to the correct adjacent cell for pushing a box; 2) pushing a box. As a result, PUSH requires more domain knowledge of Sokoban – the framework needs to compute all reachable cells from the current player position and decide which boxes are pushable.
The number of valid PUSH actions can grow linearly on the number of boxes. In contrast, MOVE only consists of four actions (four directions), and way less domain knowledge is required to decide valid moves for any state.
Using MOVE as basic actions will generate plans that are, on average, 3-4 times longer than ones generated with PUSH actions.
The majority of instances considered in this paper have plans containing hundreds or thousands of moves, which provides a great challenge for AI planning. Network architecture and training details.
For each board state of height H and width W , we create an input tensor of shape [4 × H × W ] with four multi-hot feature maps encoding the player position, box positions, goal cells, and player reachable cells (ignoring all boxes), respectively. The policy head outputs a vector of length four representing the probability distribution among four moving directions. The value head outputs a single scalar representing the logarithm of the estimated remaining distance. The network consists of multiple convolutional residual blocks and each block has two extra dropout layers to introduce randomization.
See Appendix B for more details. The parameters θ of the deep neural network are trained with the following loss function: [Equation] where c is the weight decay parameter controlling L 2 weight regularisation.
[Table] Sokolution [Table]
DNN-based A* [Table] [Table] [Table]
p + WA* (ours) ( d + [Table] · h ) / [Table]
Solver statistics. Table 2 above shows solver statistics of Sokolution, a state-of-the-art solver of Sokoban, and DNN-based A*.
We use 8 cores of Xeon 6154 CPUs for proﬁling both solvers (neural networks run on the CPU mode for a fair comparison).
Because of the cost of evaluating the deep net search guidance, DNN-based A* expands signiﬁcantly fewer nodes per second than Sokolution (about a factor of 17). Nevertheless, given 10 minutes, DNN-based A * can solve 69% hard instances that Sokolution cannot solve even given a 2-hour time limit . So, the trained deep net provides much superior search guidance than the hand-crafted guidance in Sokolution. Effectiveness of policy and value networks.
Table 3 shows experiment results for different choices of the evaluation function. As shown in the table, the policy heuristic has a signiﬁcantly larger impact than the value heuristic. Speciﬁcally, the table shows that even the Pure Policy (using only the 1 /p term, i.e., inversely proportional to the policy prediction) signiﬁcantly boosts performance compared to all value heuristics-based search strategies without the policy guidance. (See the rows above “Pure Policy” in Table 3.)
With extra properly added depth and value terms, the performance of Pure Policy can further increase to obtain our best strategies with Policy + A* and Policy + Weighted A* (WA*). [Table] To further study why the policy network is more effective, we studied the performance of both networks in detecting dead-end states since dead-ends are one crucial factor that leads to exponential runtime.
In particular, we randomly sampled 2000 board states from each dataset. For each state, we used Sokolution to detect dead-end successor states. The policy/value network is considered to successfully detect a state if it predicts a higher/lower policy/value to the child on the ground truth plan than any other dead-end child. Table 4 shows that the policy network signiﬁcantly outperforms the value network at detecting dead-ends, and thus can provide better search guidance.
A NALYSIS OF L EFT - HAND H EAVY T AILS
[Figure]
r i g h t t a il s u r v i a l f [Figure] c t i o [Figure]
l [Figure] f t t a il c u m u l a t i v e d i [Figure]
t r i
[Figure]
t
i [Figure] [Figure] Sample statistics about the heaviness of tails.
Luby’s optimal restart strategy requires the full knowledge about the runtime distribution r A of a randomized algorithm A .
Here we introduce sample statistics that can approximate the theoretical optimal values. Speciﬁcally, for each planning instance, we perform multiple randomized searches on it with maximum search budget M and collect a runtime sample S . The runtime of failed searches is recorded as M . We approximate the optimal sample restart time t S and expected sample total runtime with restart l S as
[Equation]
Let T = (1 , 1 , 2 , 1 , 1 , 2 , 4 ... ) be the time limit sequence of the universal strategy. To approximate the expected total runtime l S univ of the universal strategy on S , let a i be the expected total runtime of applying ( T i , T i +1 ,
T i +2 , ... ) on S and we have the constraint [Equation] We only need to calculate a i until the ﬁrst i such that T i ≥ M and set the remaining a i to zeros.
Figure 1 shows runtime statistics for different types of tails. Both left and right-hand heavy tails exhibit orders of magnitude reduction of l S and l S univ over the runtime sample mean, which demonstrates the beneﬁt of using random restarts.
For exponential tails, the expected sample total runtime with restart is very close to the sample mean, and the universal strategy even has a negative effect. To separate the two types of heavy tails, we compared the average runtime of solved instances v.s. the unsolved percentage as shown in Figure 2. An instance is viewed as a heavy-tailed run if random restarts can reduce the expected runtime.
We add further restrictions that a right/left-hand heavy tail requires the unsolved ratio to be less/greater than 10%/90%. For experiment budget concerns, we only plotted 10% randomly sampled instances from the training and test datasets. Figure 2 shows the training dataset has almost the same number of left and right-hand heavy-tailed instances, with the majority of instances not showing the heavy-tailed behavior.
For the hard dataset, the percentage of right-hand heavy tails decreases signiﬁcantly, with more instances shifting to the top side of the ﬁgure and entering the left-hand heavy-tailed area. We hypothesize left-hand heavy tails occur more frequently when the underlying combinatorial structure becomes harder.
Though random restarts can potentially eliminate heavy tails on both sides, left-hand heavy tails provide further intuitions and implications for curriculum learning for DRL.
In particular, they can beneﬁt from a distributed solving procedure in which any solution found by one of the processes can be shared and learned by the curriculum framework (Weng, 2020; Narvekar & Stone, 2018; Narvekar et al., 2020).
Abstract search tree model.
Chen et al.
(2001) proposed an imbalanced tree model to empirically explain right-hand heavy tails.
Here we propose an abstract tree model to empirically characterize left-hand heavy tails.
See Figure 3 for the description for both models.
Search models for planning problems differ from ones for SAT and CSP.
For NP domains, the number of unassigned variables is ﬁxed with O ( n ) where n is the problem size, and the search can assign these variables in any order.
For planning domains, the search needs to assign actions in order from the start state to a goal state with potentially maximum exponential length.
Our proposed model hypothesizes the existence of O ( log ( n )) critical nodes from which a wrong child node selected by the search will result in extra exponential search space.
As shown in the following results, our model does not depend on the actual choice of p as long as p is a constant value in range (0 , 1) .
Theorem 5.1 (The abstract tree model has exponential runtime almost surely). When restricting plans to polynomial length on the input size n , the probability that the abstract tree model has exponential runtime converges to 1 as n goes inﬁnite. The proof is deferred to Appendix A.1.
Theorem 5.2 (Restart achieves polynomial expected halting time). The optimal expected halting time with restart l A and expected halting time using the universal strategy l A univ are both O ( poly ( n )) .
The proof is deferred to Appendix A.2.
Theorem 5.1 states the runtime of the model is exponential on the input size n almost surely, and
Theorem 5.2 shows that the cost distribution has polynomial estimated runtime with restarts.
These two theorems combined show the occurrence of left-hand heavy tails.
Here is an intuitive explanation of the abstract tree model.
For the majority of the nodes on the plan, the deep neural network either provides an accuracy heuristic to choose the right child node or makes a small error of preferring a wrong child node.
As long as the error is small the search can recover from it with extra polynomial steps since the evaluation function penalizes deeper nodes.
For the O (log( n )) critical nodes, the error of estimated heuristics is so large that exponential search is required to jump out from the local search space.
Structure of real search graphs.
To empirically examine our abstract model, we plotted two typical search graphs generated by DNN-based best-ﬁrst search as shown in Figure 4.
We found that the majority of nodes actually do not incur search.
Networks’ heuristics either directly lead the search to the correct child node, or only a very small wrong sub-tree (less than 5 nodes) is explored.
Critical nodes (labeled as blue circles) are extremely rare in the search graph.
However, when encountering such critical nodes, the search explores a large sub-tree with no near goal.
Whether AI planning systems can ﬁnd a macro action routine , i.e., a sequence of algorithmic actions to perform a sub-goal, has a great interest for researchers.
To make a long plan, e.g., prove a hard mathematical theorem, humans usually only make some critical choices of lemmas and schemes, and ﬁll the remaining parts of the proof with little reasoning.
Indeed, the number of required crucial intermediate lemmas to prove a mathematical theorem is quite small, even for challenging open problems.
However, extensive and profound reasoning, search, and enumeration are needed to ﬁnd such lemmas.
The small number of critical lemmas compared with the long proof length reﬂects the prototype of such search graphs.
In our experiment setting, we use MOVE as basic actions.
To perform a real PUSH, the search algorithm needs to compute all reachable cells from the player’s position and calculate the shortest path to the cell adjacent to the box to push.
Such a long sequence of moves before performing an actual push can be viewed as a routine .
As shown in Figure 4, the algorithm can perform a long chain of moves with little local search, which might suggest that macro action routines are implicitly learned as a part of neural networks’ heuristics.
Relation to backdoors.
The proposed model has a close relation to backdoors to typical case complexity.
To explain why solvers scale so well in areas such as planning and ﬁnite model-checking,
Williams et al.
(2003) examined various benchmarks and identiﬁed that for most practically solvable problem instances, after assigning values to logarithmic variables, the remaining problem instance quickly becomes polynomially solvable by propagating constraints.
This result illuminates the prototypical patterns of the structure causing the empirical behavior observed in the International
Planning Competitions benchmarks (Vallati et al., 2015; Cohen & Beck, 2018; Meier et al., 2014).
10101010 3 size of sub-tree 101010101010nu m b e r o f t o t a l o cc u rr e n c e
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
More experiment data about the abstract tree model.
To better illustrate the occurrence probability of different sizes of sub-search space, we randomly sampled 300 solved instances from the test dataset and counted the number of dead sub-trees (no near goal exists) for each size.
As shown in Figure 5a, as the size of sub-trees grows exponentially, the number of sub-trees also decreases exponentially fast.
The experiment data empirically conﬁrms our abstract tree model that critical nodes which incur extra exponential search space have logarithmic occurrence.
This is evidence about how left-hand heavy tails occur in practical planning solving.
Hoffmann et al.
(2006) explained SATPLAN, classical planning as SAT, with backdoor models. In particular, after ﬁnding and assigning logarithmic variables, the remaining problem solving becomes polynomial.
In contrast, ﬁnding these critical nodes is not necessary for expand-style search algorithms.
Left-hand heavy tails are not only caused by the underlying structure of practical instances but also affected by the mysterious generalization ability of neural networks.
It is an interesting future research direction to understand the surprising scaling performance of various heuristics, from conﬂict-driven clause learning for SAT solving to DNN-based search methods.
Solving more instances with random restarts.
The theory of heavy-tailed cost distribution suggests that a sequence of short runs instead of a single long run may make better use of a ﬁxed amount of computational budget.
We explored this idea by considering a ﬁxed number of total expanded nodes allowed for the search.
Figure 5b shows the probability of not solving the instances for the test dataset.
The failure rate of a single run drops the fastest for a small amount of computational budget.
With more total expanded nodes, random restarts gradually achieve better performance.
Speciﬁcally, to solve more instances, the solver needs to increase the total number of compute cycles.
When doing so, the ﬁgure shows there comes a point where more frequent restarts are more effective.
For example, with a budget of around 2,000 nodes, the strategy with 2 restarts becomes more effective than no restarts.
At around 5,000 nodes, 5-restart becomes more effective than 2-restart.
So, to solve a larger fraction of hard instances, more frequent restarts become more effective.
C ONCLUSION
We studied the use of policy (action selection) and value (remaining distance estimate) functions as well as randomization methods for solving hard planning instances using best-ﬁrst search.
Our experiments show the remarkable effectiveness of the policy network and random restarts for the search.
The value network provides additional global search guidance.
We show that uncertainty-aware networks provide an effective way to introduce randomization into the search process leading to increased efﬁciency.
Our runtime distribution results show heavy-tailed distributions with tails on both the left and right-hand sides.
Left-hand heavy tails have not been observed in combinatorial search before.
We also introduce an abstract computational model that explains left-hand heavy tails.
Finally, we show how random restarts can improve the overall search effectiveness
A T HE I MBALANCED T REE M ODEL P ROOFS A. P ROOF OF T HEOREM 5.
Lemma A.1 The abstract tree mode has polynomial runtime if and only if all critical nodes pick the right child.
Proof of Lemma A.1.
Necessity comes from the structure of the model and any picking of left child will immediately result in exponential search space. For sufﬁciency, by the assumption, the depth of the tree model is bounded by O ( poly ( n )) , and the ﬁnal sub-search space with all right child picking is also O ( poly ( n )) . Then we have the total runtime is O ( poly ( n )) + O ( poly ( n ))
=
O ( poly ( n )) .
Q.E.D.
Proof of Theorem 5.1.
By Lemma A.1, we have
[Equation]
for some constant C )
[Equation]
[Equation]
p
[Equation]
assumption
[Equation]
As a conclusion, lim n →∞ Pr( tree mode has exponential runtime )
= 1 .
Q.E.D.
A. P ROOF OF T HEOREM 5.
Proof.
We can assume there exists at least one non-critical node on the plan (if all tree nodes are critical, we can augment an extra non-critical node to the right child of the deepest critical node without affect other properties of the tree model).
Let u be the shallowest non-critical node and d be the depth of node u .
We set the restart time t
A to be the size of the left sub-tree of d .
t A is poly ( n ) by the deﬁnition of the model.
Let q = p d (1 − p ) .
So the expected runtime
[Equation]
( poly ( n ))) · 1 q = O ( poly ( n )) .
By the result of Luby et al.
, l A univ =
O ( l A log( l A ))
=
O ( poly ( n )) .
Q.E.D.
B N ETWORK A RCHITECTURE AND T RAINING DETAILS B. N ETWORK A RCHITECTURE
A single input tensor of board states has shape [4 × H × W ] and a batch of board states can have different heights and widths.
For each batch, we take the maximum H and W of all state tensors as the batch height and width, and zero-pad the empty cells.
The network consists of a single convolution block followed by 16 residual blocks.
The convolutional block applies the following modules:
A convolution of 128 ﬁlters of kernel size 3 × 3 with stride2.
2D batch normalization (Ioffe & Szegedy, 2015) 3.
A ReLU nonlinearity
Each residual block applies the following modules sequentially to its input:
A channel-wise dropout layer with probability 30% of a channel to be zeroed.
A convolution of 128 ﬁlters of kernel size 3 × 3 with stride
2D batch normalization 4.
A Relu nonlinearity 5.
A channel-wise dropout layer with probability 30% of a channel to be zeroed.
A convolution of 128 ﬁlters of kernel size 3 × 3 with stride7.
2D batch normalization 8.
A skip connection that adds the input to the block 9.
A Relu nonlinearity
The output of the residual tower is then fed into two independent heads for computing the policy and value.
Both heads contain an extra residual block followed by a fully connected layer.
The policy head outputs a vector of size 4 and the value head outputs a single scalar.
B. T RAINING DETAILS
We use the AdamW optimizer (Loshchilov & Hutter, 2017) with weight decay 0 . 01 and an initial learning rate 0 . 001 .
We train the network with supervised training data for 200 epochs.
The lastepochs use a learning rate of 0 . 0001 .
We set batch-size to 256.
The whole training procedure took around 70 hours to ﬁnish on 5 Tesla V100 GPU cards.