A LPHA Z ERO - BASED P ROOF C OST N ETWORK TO A ID G AME S OLVING
Anonymous authors
Paper under double-blind review
A BSTRACT
In recent years, the AlphaZero algorithm has achieved super-human playing levels for many games without hand-crafted expert knowledge.
Researchers have taken advantage of AlphaZero’s effectiveness at learning and playing games to help in solving them.
However, a strong player is not necessarily a strong solver.
This paper proposes a novel approach to solving problems by modifying the training target of the AlphaZero algorithm, such that it prioritizes solving the game quickly, rather than winning.
We train a Proof Cost Network (PCN), where proof cost is a heuristic that estimates the amount of work required to solve problems.
This matches the general concept of the so-called proof number from proof number search, which has been shown to be well-suited for game solving.
We propose two speciﬁc training targets.
The ﬁrst ﬁnds the shortest path to a solution, while the second estimates the proof cost.
We conduct experiments on solving 15x15 Gomoku and 9x9 Killall-Go problems with both MCTS-based and focused depth-ﬁrst proof number search (FDFPN) solvers.
Comparisons between using AlphaZero networks and PCN as heuristics show that PCN can solve more problems.
I NTRODUCTION
There are two main goals in the pursuit of strong game-playing agents.
The ﬁrst is to push the boundaries of artiﬁcial intelligence since games can be seen as simpliﬁed models of the real world.
The second involves ﬁnding game-theoretic values , or outcomes given optimal play, for various games (van den Herik et al., 2002).
These two closely related yet separate goals are commonly referred to as playing and solving 1 games, respectively.
For playing, researchers have found success in building super-human level game-playing agents for many games, including Go (Silver et al., 2017; 2016), Chess, Shogi (Silver et al., 2018), and Atari games (Schrittwieser et al., 2020; Mnih et al., 2015).
Achievements for solving include checkers (Schaeffer et al., 2007), Hex up to board sizes of 9x9 (Pawlewicz & Hayward, 2013; Henderson et al., 2009), Go for board sizes up to 5x (van der Werf & Winands, 2009), 15x15 Gomoku (Allis, 1994), among others.
Previously, researchers used hand-crafted heuristics with game-speciﬁc knowledge, combined with search methods such as Monte Carlo tree search (MCTS) (Winands et al., 2008), proof number search (PNS) (Allis et al., 1994), depth-ﬁrst proof number search (DFPN)
(Nagai, 2002; 1999), and threat-space search (Allis et al., 1993), to solve problems by massively pruning away unnecessary branches (Schaeffer et al., 2007; van der Werf et al., 2003; Allis et al., 1996; Allis, 1994).
With the great success of AlphaGo and AlphaZero, there have been attempts to combine neural networks with previous solvers to further improve game solving.
For example, Gao et al.
(2017) incorporated an AlphaGo-like neural network into a state-of-the-art Hex solver to solve 8x8 Hex openings more quickly, which demonstrated that neural networks can have a positive impact when combined with previous techniques and heuristics.
Meanwhile, McAleer et al.
(2018) also trained an AlphaZerolike agent to solve the Rubik’s Cube.
In addition, game solving techniques were also shown to be viable for non-game applications (Kishimoto et al., 2019; Segler et al., 2018).
However, a limitation of using these techniques is that the networks trained with the objective of strong play may not be well-suited for obtaining game-theoretic values.
Since the goal of a strong player is to play moves that maximize win rates (or the highest conﬁdence for winning), as a heuristic it often does not choose moves that result in the fewest nodes in the search tree when solving.
As an example, Agostinelli et al.
(2019) pointed out that AlphaZero-like systems tend to solve problems with longer solutions.
For games with a singular end state, e.g. Rubik’s cube and n-puzzles, they proposed an approach that trains a cost-to-go value function that approximates the cost of ﬁnding the shortest-path solution, which can be viewed as a kind of prioritized-sweeping (Moore & Atkeson, 1993) algorithm, starting from the end state (goal).
However, singular end states are often only exploitable for puzzles.
Indeed, in two-player zero-sum games such as Go, Hex, Gomoku, Connect6, and many others, the game may end in too many conﬁgurations to enumerate practically.
This paper presents a novel approach to solving these problems, while still tending to choose moves that lead to the fastest solutions.
In our approach, we propose a new heuristic, referred to as the proof cost .
We give two concrete examples of how proof cost can be designed.
The ﬁrst acts as a “shortest path” heuristic to a proof, while the second predicts solution tree sizes.
Then, we use AlphaZero training to approximate this heuristic.
The resulting network is referred to as the Proof Cost Network ( PCN ), which can be used as a heuristic with any AND/OR tree search algorithm to improve solving efﬁciency.
Experiments were conducted on 15x15 Gomoku and 9x9 Killall-Go problems.
The results show that
PCN outperforms AlphaZero when used as a heuristic with MCTS-based and Focused DFPN (Gao et al., 2017; Henderson et al., 2009) solvers, which are both commonly used to solve games.
B ACKGROUND 2. T HE G AME - THEORETIC V ALUE AND P ROOF T REES
For a one or two-player game, the game-theoretic value of a game state is the best outcome that can be obtained by all players if they play optimally.
In this paper, we focus on two-player games, and simplify the outcome to only include win/loss; draws are considered losses for both players 2 .
A game state is said to be solved when the game-theoretic value for the state is obtained, and proved/disproved if the player to play for the state has a winning/losing outcome.
In a proof, the winning player needs to ensure a winning response to all possible actions by the losing player.
A proof tree (Pijls & de Bruin, 2001; Stockman, 1979) is an AND-OR search tree representing a strategy of answering actions such that (a) all terminal nodes are wins, (b) the winning player (the
OR-player) contains at least one winning action, (c) all losing player (the AND-player) actions are enumerated, and the OR-player needs to have winning responses to each AND-player action.
A disproof tree is the dual case of the above (with win/loss and AND/OR reversed); a solution tree is a general term used to describe both proofs and disproofs.
2. M ONTE C ARLO T REE S EARCH
Monte Carlo tree search (MCTS)
(Coulom, 2006; Kocsis & Szepesv ´ ari, 2006) is a heuristic best-ﬁrst search algorithm that iteratively repeats the following four phases: selection, expansion, evaluation, and backpropagation.
In the selection phase, starting from the root node, the algorithm chooses a child node according to a speciﬁed selection criterion until a leaf node is reached.
Given a state s during selection, AlphaZero selects an action a ∗ by the PUCT equation (Silver et al., 2018; Rosin, 2011):
[Equation]
where Q ( s, a ) =
W ( s, a ) /N
( s, a ) is the estimated mean win rate, N ( s, a ) is the number of times a is selected at s , W ( s, a ) is the number of wins,
c PUCT is a weight for exploration, and P ( s, a ) is a prior knowledge heuristic.
The above selected leaf then expands all children which are added to the tree.
The leaf node is then evaluated.
In AlphaZero, a two-head network is trained such that it outputs a policy p ( s, a ) , and a value v ( s ) .
The policy is a distribution that estimates how likely each child of s will be visited during selection; p ( s, a ) can then be used as P ( s, a ) in eq.
The scalar v ( s ) is an estimate for the game-theoretic value of s .
During backpropagation we move back up the selection sequence and Q ( s i , a i ) is updated for all ancestors s i .
While originally designed for playing, MCTS can also be used to solve games with AND-OR Boolean backpropagation.
MCTS has been used to solve games and problems for Lines of Action (Winands et al., 2010; 2008), Connect4, Seki in the game of Go (Cazenave & Safﬁdine, 2010), and Connect
(Wei et al., 2015).
2. P ROOF N UMBER S EARCH
Proof number search (PNS) (Allis et al., 1994) is a best-ﬁrst search algorithm that was designed speciﬁcally for solving games.
Each node is associated with two numbers: the proof number ( PN ) and the disproof number ( DN ), which are heuristics that represent the minimum required number of leaf nodes that need to be expanded to prove and disprove the node respectively.
PNS largely follows the same four phases as in MCTS.
What is called the most-proving node ( MPN ) is selected during the selection phase.
Speciﬁcally, the child with the lowest PN at each OR node, or the lowest DN at each AND node is selected; the resulting leaf node is then the MPN that needs to be evaluated.
Following the deﬁnition of the PN/DN, PNS therefore builds the solution tree by expanding and evaluating the least number of nodes.
There are many PNS variants, one of which is the so-called depth-ﬁrst proof number search (DFPN)
(Nagai, 2002; 1999), which addresses the large memory requirements for large searches.
PNS was originally designed such that all newly expanded nodes initialize PN/DN values to 1/1.
To improve the heuristic, domain knowledge can be used to initialize PN/DN values differently, which speeds up the search for a solution (Winands & Schadd, 2010;
Wu et al., 2010; Saito et al., 2006;
Kishimoto & M ¨ uller, 2005; Allis, 1994).
In addition, Henderson (2010) proposed Focused DFPN (FDFPN), which uses heuristics to direct the search toward promising solutions with a technique that is often referred to as progressive widening.
Speciﬁcally, progressive widening involves prioritizing search on only a portion of all possible moves.
The branching factor b at an internal search node is dynamically adjusted by b = b base + (cid:100) µ × | b livechildren |(cid:101) , where b base is set to 1, b livechildren is the number of all yet unsolved children of the node, and µ is a weighting constant.
With an accurate heuristic, FDFPN can solve problems while searching fewer nodes than unmodiﬁed DFPN.
Further, Gao et al.
(2017) proposed FDFPN-CNN by incorporating the policy and value outputs from an AlphaGo-based agent as heuristics to solve 8x8 Hex.
Namely, the policy was used for move ordering and the value was used to control the branching factor by b = b base + (cid:100)
f ( s ) ×| b livechildren |(cid:101) and f ( s ) =
min { µ, 1 + v ( s ) } .
2. T HE A LPHA Z ERO A LGORITHM
AlphaZero (Silver et al., 2018) is a general reinforcement learning algorithm that can achieve superhuman level playing strength for games without requiring any expert domain knowledge.
The training routine consists of two phases: self-play and optimization.
In a nutshell, a two-headed neural network that outputs a policy distribution p and a value scalar v is periodically optimized with data that is collected via self-play.
Self-play games are generated by running a set number of MCTS simulations with the most recently optimized neural network.
In the optimization phase, random positions are sampled from the latest self-play games, where the network is optimized by the following loss function:
[Equation]
where z is the ground truth outcome of the game for that position, π is the MCTS search distribution, c is an L2 regularization weight, and θ represents the network parameters.
O UR A PPROACH 3. E STIMATING P ROOF T REE S IZES
We ﬁrst examine the size of a proof tree, i.e. the number of nodes that need to be searched to obtain a proof for a player of interest from a given state.
Since we do not know the game-theoretic outcome of the game before ﬁnding a solution, either player can be designated the player of interest.
Given limited time and resources, it is almost always preferable to prove a given state by traversing as small a search tree as possible, where the ideal case is called a minimum proof tree .
A minimum proof tree, say rooted at a node representing state s , is deﬁned as a proof tree containing the least possible number of tree nodes, denoted by n ( s ) , for the player of interest at s to win.
m ( s ) is deﬁned likewise, but for the opposite player. With no loss of generality, we focus on n ( s ) for the remainder of this section, unless otherwise speciﬁed.
By deﬁnition, n ( s ) is a perfect measures of resource efﬁciency for proving s .
Unfortunately, in order to derive the true value of n ( s ) , we would need to traverse the game tree starting at s by induction as follows.
Suppose s T are terminal states.
Then, n ( s T ) = 1 if the player to play at s is winning, and n ( s T ) =
∞ otherwise.
Suppose s AND are non-terminal AND nodes.
Then, n ( s AND ) = 1 + (cid:80) s i n ( s i ) , where s i are the children of s AND .
Suppose s OR are non-terminal OR nodes.
Then, n ( s OR ) = 1 + min s i n ( s i ) for all children s i .
To obtain min s i n ( s i ) , we would need to search all children of non-terminal OR nodes; combined with having to search all children of non-terminal AND nodes, the conclusion is that the entire tree rooted at s will have to be searched.
In practice, even if we cannot obtain the true value, a sufﬁciently accurate heuristic estimate of n ( s i ) , which we denote as ¯ n ( s i ) , can be useful in guiding the search.
In the ideal case, ¯ n ( s i ) perfectly preserves the order of n ( s i ) , i.e. ¯ n ( s 1 ) ≤ ¯ n ( s 2 ) if and only if n ( s 1 ) ≤ n ( s 2 ) .
Then, for s OR , we would only have to compute the heuristic ¯ n ( s i ) for all children s i , then choose to search the child with the smallest value.
The difﬁculty, as is always the case in heuristic search, is in designing such a heuristic so that ¯ n ( s i ) preserves the order of n ( s i ) as best as possible.
While prioritization according to ¯ n is straightforward for OR nodes, the AND node case requires some more thought.
Theoretically, in this case ¯ n should have no effect on priority since all children need to be searched for AND nodes.
In practice, with an imperfect heuristic, the order in which children of AND nodes are searched impacts how quickly OR node prioritization mistakes are corrected.
To obtain reasonably accurate heuristics ¯ n and ¯ m , we take advantage of AlphaZero’s ability to converge on accurate positional value estimates through self-play, and modify the algorithm so that it approximates n and m instead. We describe the details on how the training targets are deﬁned in subsection 3.2 and the training loop in subsection 3.3. The heuristics ¯ n and ¯ m are then used in a modiﬁed PNS algorithm, which we describe in subsection 3.4.
3. T HE P ROOF T REE S IZE E STIMATE AS A T RAINING T ARGET
Let us assume that s is the root, is an OR node, has b children, s 1 , ..., s b , and that all ¯ n ( s i ) are available with ¯ n ( s 1 ) ≤ ¯ n ( s 2 ) ≤ ... ≤ ¯ n ( s b ) .
We propose that ¯ n ( s ) :
= ¯ n ( s 1 ) , similar to the way PNs are calculated for OR nodes in PNS.
Since ¯ n ( s ) grows exponentially, the +1 term’s impact is overall negligible.
Moreover, the omission of this term preserves the order of ¯ n among all children, so it can be safely omitted for simplicity.
As described in subsection 3.1, the OR-player should prioritize searching s 1 , or those s i with small i .
The dual case where the root is an AND node is ¯ n ( s ) :
= (cid:80) bi =1 ¯ n ( s i ) , where s i are the children of s , following PNs for AND nodes in PNS.
In order to design the heuristic ¯ n ( s ) such that it preserves the ordering of n ( s ) as accurately as possible, a naive way as describe in the previous subsection would be to perform brute force search to obtain all the ground truths for machine learning.
However, it is apparent that this is not feasible given the exponential tree size growth.
Instead, we use a modiﬁed MCTS in the self-play phase of AlphaZero to collect game episodes, which are then used to learn ¯ n ( s ) .
For the terminal state, the ground truths are set to 1 if the root player wins, and ∞ otherwise.
For all non-terminal OR nodes s t and its child s t +1 in the episode, ¯ n ( s t ) is set to ¯ n ( s t +1 ) .
For all non-terminal AND nodes s t and its child s t +1 in the episode, ¯ n ( s t ) is set to b × ¯ n ( s t +1 ) , where b is the number of children for s t .
We use b × ¯ n ( s t +1 ) for AND nodes as the training target since we do not know the values for all other siblings of s t +1 (recall that self-play generates single episodes), which makes it difﬁcult to obtain (cid:80) bi =1 ¯ n ( s i ) .
Effectively, we are assuming that the costs for all of s t ’s children are equal to s t +1 ’s. There are domain-speciﬁc improvements for many games to improve the accuracy of this estimate.
As an example, the Gomoku position illustrated in Fig.
1 contains what is called a “four [Figure] ¯ n ( s 0 ) =
b 0 × bs¯ n ( s 1 ) =
bs¯ n ( s 2 ) =
bs¯ n ( s 3 ) =s¯ n ( s 4 ) =bbbb threat” (4T) for Black.
We know that Black will achieve a 5-in-a-row and win if White does not block at A, so we can effectively reduce b to 1 .
To differentiate the case where we use heuristics to reduce b like in the 4T example above, we will use b heur to specify the number of valid children when heuristics are used, and b max for the setting where we use the entire action space.
We illustrate three examples for b in 15x15 Gomoku.
First, b max is always 225 since that is the maximum number of moves for the 15x15 board; next, if we use the game independent heuristic where we only consider legal moves, we set b heur to the number of empty grids left on the board; last, if a 4T exists, as in Fig.
1, b heur = 1 .
An example episode is illustrated in Fig.
It starts from s 0 and ends at s 4 with a win (achieving the goal) for the OR-player; b 0 to b 3 are the branching factors of s 0 to s 3 respectively (with no loss of generality to whether b heur or b max is used).
The heuristic function ¯ n ( s ) for the states in the episode are set to the following: ¯ n ( s 3 ) =
¯ n ( s 4 ) = 1 , ¯ n ( s 1 ) =
¯ n ( s 2 ) =
b 2 × ¯ n ( s 3 ) =
b 2 , and ¯ n ( s 0 ) =
b 0 × ¯ n ( s 1 ) =
b 0 × b 2 .
Based on this calculation, we can see that every move counting backwards from the terminal node increases the proof cost by a factor of b equally when using b max .
Thus, the resulting heuristic can be seen as a measure of how many moves are left until the solution, so it can be useful for ﬁnding the shortest path to the solution.
There are some similarities and differences between our heuristic and the concept of PNs in PNS.
First, PN/DN are dynamic quantities meant to change as PNS progresses and new nodes are expanded and evaluated.
In contrast, ¯ n ( s ) is a static estimate of an oracle number that represents the minimum proof tree size.
More speciﬁcally, rather than the PN value, ¯ n ( s ) estimates PN plus the current tree size of s . Second, PNS makes no assumptions as to which player is more likely to win, and so both PN/DN are used.
Since we already have self-play episodes to train from, we add an auxiliary target ¯ m ( s ) , which is conceptually similar to DNs.
In Fig. 2,
[Equation]
Last, in PNS, the search arrives at the MPN by choosing the minimum PN at OR nodes, and the minimum DN at AND nodes.
In our method, while OR nodes behave similarly in that the smallest ¯ n value is chosen, for AND nodes, we prioritize moves which have the maximum ¯ n value.
The OR node estimate is most accurate when the OR-player chooses the move with the lowest ¯ n ( s ) value because n ( s )
=
+ min n ( s i ) .
To leverage the AlphaZero algorithm, where the two players’ goals run counter to each other, the AND-player behavior should be designed such that it tries its best to oppose the OR-player, which in this case means it should pick the largest ¯ n .
Intuitively, the ORplayer will try to prove a win as quickly as possible, while the AND-player will delay as much as it can.
It is important to stress that this difference only affects how the heuristic network is trained, not how search algorithms work.
In other words, the choice of the maximum ¯ n value for AND nodes is used only in the self-play phase described in subsection 2.4 when training the heuristic network.
3. T RAINING THE P ROOF C OST N ETWORK WITH A LPHA Z ERO
We train the Proof Cost Network (PCN)
f θ ( s ) =
( p, v n , v m ) , with parameters θ , such that it predicts the policy distribution p , and the values v n and v m , corresponding to the proof cost ¯ n ( s ) and auxiliary ¯ m ( s ) for the state s .
Intuitively, for p , moves with higher probabilities lead to faster solutions for s , either proof or disproof.
Our method replaces the AlphaZero v ( s ) output, which estimate the win rate (or outcome of the game z ), with v n ( s ) and v m ( s ) , which estimate ¯ n ( s ) and the auxiliary target ¯ m ( s ) .
In practice, since ¯ n ( s ) grows exponentially the closer s is to the root, we replace ¯ n ( s ) with the surrogate l n ( s ) , where l n ( s ) =
log(¯ n ( s )) , likewise with ¯ m ( s ) .
With the single-sided proof cost heuristic ¯ n , our method tends to perform stronger when we designate a speciﬁc player as the OR-player and attempt a proof (as opposed to solving problems without any assumptions as to the winner/loser).
Thus, we need to ﬁrst decide which are the AND/ORplayers, e.g. knowing that Black is proven to be the winner for 15x15 Gomoku, we set Black as the OR-players.
In the self-play phase, ideally, the OR-player should select arg min a i ¯ n ( s i ) among all candidate moves a i which lead to s i .
Intuitively this motivates the self-play agent to solve s as quickly as possible.
Meanwhile, the AND-player should select arg max a i ¯ n ( s i ) to prevent the OR-player from solving the game quickly, as described in subsection 3.2.
To implement this behavior into the MCTS process for move selection during self-play, we make the following changes.
In the MCTS selection phase, we still follow eq.
1, but now Q ( s, a ) is a moving average of v n ( s ) , representing the estimated overall mean cost of solving s with the action a .
Since
Q ( s, a ) is not limited to [ − 1 , 1] , we calculate ¯ Q ( s, a ) by normalizing Q ( s, a ) with the following equation (similar to Schrittwieser et al.
[Equation]
Lastly, to motivate the OR/AND-player to choose small/large values of ¯ n , we simply ﬂip the value to − ¯ Q ( s, a ) for the OR-player.
During AlphaZero optimization, we randomly sample the self-play games and calculate ¯ n and ¯ m as described in subsection 3.2, then convert them to l n and l m accordingly.
The network is optimized by a modiﬁed version of eq. 2, as follows:
[Equation]
where φ is a transformation function that changes scalar values to categorical representations (Schrittwieser et al., 2020).
3. F OCUSED DFPN WITH P ROOF C OST N ETWORK
In this subsection, we present a method that incorporates PCN into the FDFPN solver (Henderson et al., 2009), called FDFPN-PCN.
With the PCN, since the value outputs v n and v m are not scalars between [ − 1 , 1] , we cannot directly apply them as heuristics like FDFPN-CNN (Gao et al., 2017), described in subsection 2.3.
We propose a separate method based on FDFPN and FDFPN-CNN that is able to use the PCN as a heuristic in the FDFPN solver.
The fundamental search behaves just as DFPN would, including PN/DN calculation and the MPN selection mechanism, with two differences.
First, we follow
FDFPN-CNN where the policy output serves as move ordering, and FDFPN where the branching factor is limited by a parameter µ .
Second, since v n and v m take inspiration from PN/DN, we can initialize PN/DN values with the PCN as follows.
Let us assume we are expanding state s , at which point we will call f θ ( s ) .
A naive way would be to set PN
= v n ( s ) and
DN = v m ( s ) directly.
The problem is that we do not have v n ( s i ) and v m ( s i ) for every expanded child.
Additionally, calling f θ ( s i ) for all s i slows the entire search down.
To conserve time, we initialize the PN/DN of s i , then calculate the PN/DN of s with its children’s values.
If s is an OR node, we set PN
= v n ( s ) and
DN = v m ( s ) /b
for all s i , where b is the number of children for s .
On the other hand, if s is an AND node, we set PN = v n ( s ) /b and DN =
v m ( s ) for all s i .
This way, the PN/DN of s will then be v n ( s ) and v m ( s ) , respectively.
E XPERIMENTS 4. T RAINING S ETTINGS
All experiments were performed by solving 15x15 Gomoku and 9x9 Killall-Go problems.
We use two different solvers , one MCTS and the other FDFPN.
The MCTS solver is an AlphaZero player with AND/OR tree logic to propagate win/loss information.
The FDFPN solver follows the mechanism described in 3.4.
Both solvers make use of transposition tables (Nelson, 1985) to conserve evaluation costs when identical positions are encountered during search 3 .
For each experiment, we focus on comparing the number of solved problems by using different networks as heuristics on the same solvers.
There are four network conﬁgurations: (1) no network, (2) α 0 (AlphaZero), (3) PCN- b max , and (4) PCN- b heur .
For fairness, we use the same training settings for each network conﬁguration, described as follows.
We run 400 MCTS simulations for each move during self-play, for a total of 1,500,000 games, and the network is optimized every 5,000 games.
The network contains 5 residual blocks with 64 ﬁlters, is optimized by SGD with 0.9 for momentum, 1e-4 for weight decay, and a ﬁxed learning rate of 0.02.
We use 1080Ti GPUs for training, where the network is implemented with PyTorch (Paszke et al., 2019).
Each 15x
Gomoku training takes about 1,000 GPU-Hours, and for 9x
Killall-Go, about 1,500 GPU-Hours.
For simplicity, we select the last snapshot model for each network in our experiments.
Additional detailed settings are as follows.
In 15x15 Gomoku, the network input has 4 channels, including black and white stones in the current position, and two channels representing the turn color.
The α for Dirichlet noise is set to 0.05.
The maximum value ( ∞ ), used to set training targets, is set to 1600.
There are two variations for 15x15 Gomoku: (1) NDK: no domain knowledge is provided, and (2) 4T: uses the four threat heuristic.
For the NDK variation, all legal moves are considered in the solver; b max = 225 , and b heur is the number of legal moves.
For the 4T variation in Gomoku, whenever a 4T occurs, the solver will only consider blocking 4 ; b max = 225 ; b heur = with a 4T and is the number of legal moves else wise.
Note that b max and b heur are not “branching factors” for the solver, but hyperparameters that deﬁne the training labels for ¯ n and ¯ m .
9x
Killall-Go (Cauwet et al., 2015) is similar to the game of Go except the following additional rules.
Black plays four stones ﬁrst on the board as shown in Fig.
4a, then White and Black play normally, as in standard Go.
White wins if any white stones becomes unconditionally alive, otherwise
Black wins.
Namely, Black wins if and only if there are no white stones on the board at the end of the game.
The network input for 9x9 Killall-Go has 18 channels, including black and white stones for the last 8 positions, and two channels representing the turn color.
The α for Dirichlet noise is set to 0.2.
The maximum value ( ∞ ) is set to 500.
In 9x9 Killall-Go, we implement two heuristics for the solver to help the game end quickly; ﬁrst is Benson’s algorithm
(Benson, 1976) which detects unconditional life, which we use to end the game immediately if White achieves unconditional life anywhere on the board 5 ; second, we prohibit players to ﬁll their own true eyes.
There are also two variations for Go, one assuming White is the OR-player, and the other Black; recall that v n is the proof cost estimate for the OR-player.
For both variations, b max is set to 82 ( 9 × 9 + 1 for passing), and b heur is set to the number of legal moves excluding ﬁlling in one’s own eye.
4. S OLVING 15 X 15 G OMOKU PROBLEMS WITH MCTS- BASED S OLVER
A set of 77 problems are generated automatically for the experiment, detailed in
Appendix B. The results for the number of solved problems within 30 minutes are shown in Table 1.
We list the node counts in Appendix C. From the table, when no domain knowledge is provided for 15x15 Gomoku, both PCN- b max and PCN- b heur outperform α 0 , with PCN- b max being better than PCN- b heur .
No Network α
PCN- b max
PCN- b heur 15x15 Gomoku (NDK) 1 /23 [-/43 /
-]/
[Table]
We use Fig.
3, one of the problems, to illustrate that with the help of 4T, PCN- b heur can take advantage of the VCF (victory by continuous four) (gom, 2021) strategy more effectively.
Fig.
3a is a problem where Black plays ﬁrst and wins.
Fig.
3b shows the solution following b heur calculation, where Black wins using VCF.
Namely, if White does not respond to any of Black’s threats, Black wins directly.
In b heur , l ( s ) ≈ 0 since the branching factor is reduced to 1 by 4T. Fig.
3c shows the solution found by PCN- b max , which has the same length as PCN- b heur , but move 7 (Black) is
[Figure]
[Figure] [Figure]
not a 4T. This means that White has many choices, such as moves 8 and 10, that does not result in an immediate loss.
All possibilities need to be searched to ensure a Black win, so limiting White choices narrows down the search and improves efﬁciency.
With b max , since the branching factor is always set to 225, the solver cannot identify 4Ts.
As a result, PCN- b max requires 7,090 nodes while PCN- b heur only uses 2,283 to solve this problem.
We expect that PCN- b heur can perform even better when incorporating more domain knowledge, such as three-threats.
4. S OLVING 9 X 9 K ILLALL -G O PROBLEMS WITH MCTS- BASED S OLVER
Next, we try to solve 81 automatically generated problems for 9x9 Killall-Go.
We exclude Black wins from the generated problem set because they are trivial for PCN for the following reason.
The goal for Black in this game is to own the entire board.
However, since we generate problems from self-play outcomes, for Black winning states close to the end of the game, e.g. in Fig. 4b, Black has many similar moves that lead to victory.
In situations like these, AlphaZero-based agents are known to play passively, unnecessarily extending the game length, which is a well-known problem for MCTS-based players.
In fact, the AlphaZero-based Leela Chess Zero program implemented a “moves left head” in its network to handle this issue (Forst ´ en & Pascutto, 2019), which functions similarly to ¯ n when we use b max .
Whereas the “moves left head” is only used near the end of the game, PCN is used throughout a problem.
As favorable as this is for PCN, it is not a fair comparison, so we excluded Black win problems from the Killall-Go set.
Now we observe results for White win problems.
Both b max and b heur still outperform α 0 as shown in Table 1.
Note that since the baseline and α 0 do not need to designate which side is the OR-player, they only have one result each.
Interestingly, for the variation where Black is set to be the OR-player (and v m is used instead of v n as the proof cost), it can still solve about half of the problems, despite the problem being White win and v m being an auxiliary task.
[Figure]
Fig.
4 shows two examples to illustrate the difference between α 0 and our method.
In Fig.
4c, a problem for White to play and win, White can become Benson safe by directly playing at A, which forms two eyes (square labels).
Both PCN- b max and PCN- b heur only focus on A, which solves the problem directly.
However, since White has many ways to win in this position, α 0 will attempt other moves, e.g. play at B, C, or D to kill the cross-marked black stones, or play at E, F, or G to kill the triangle-mark stones.
Since the goal for AlphaZero is to win, and all of A to G can reach this goal, α 0 does not focus on A only, spending more nodes to solve this problem.
In the problem in Fig.
4d, there is a capturing race between the three white blocks (marked as cross, triangle, and square) and the black block (marked as circle).
White can win the capturing race by directly playing at A (for Black D, reply E, and for E, reply D).
For another sure win, White can ﬁrst extend liberties for the square-marked white blocks by making an exchange , a common technique in
Go, where White plays at B and Black replies at C. For both winning moves at A and B, we observe that α 0 tends to make the exchange at B ﬁrst to increase the win rate, while PCN prefers the faster
A. Again, exploring all possibilities is costly.
So much so that α 0 takes nearly three times the node count of PCN to solve this problem.
4. S OLVING WITH THE FDFPN S OLVER
In this subsection, we try to use the FDFPN solver as described in subsection 3.4 to solve the same problems in previous subsections.
Table 2 shows the results for baseline FDFPN 6 and the three types of [-networks to solve problems on 15x15 Gomoku and 9x
 Killall-Go.-]networks.
In FDFPN, we set µ = 0 .
1 on both 15x15 Gomoku and 9x9 Killall-Go.
The result shows that both PCN- b max and PCN- b heur still outperform α 0 .
When comparing MCTS and FDFPN solvers directly, we ﬁnd that there is no clear winner.
FDFPN can solve the same number of problems for 9x9 Killall-Go, while MCTS-based solver is slightly better for 15x15 Gomoku.
The fact that neither DFPN nor MCTS dominates the other in solving has been corroborated by previous research (Ewalds, 2012; Wei et al., 2015).
However, in either case, PCN has a positive impact on efﬁciency.
We therefore expect PCN to be extendable to other search algorithms.
No Network α
[Table]
D ISCUSSION
The experiments demonstrate that, in general, AlphaZero-like networks can be used to enhance solving without expert knowledge.
It is somewhat surprising that b heur is not consistently better than b max , since it is theoretically a more accurate estimate. A possible explanation is that the main beneﬁt of PCN is contingent upon the ordering of ¯ n , rather than ¯ n itself, and that the limited scope of heuristics may have led to inaccuracies in general.
However, there is potential in ﬁne-tuning a combination of heuristics so that b heur performs even better.
We leave this as a future direction of research.
We note that 15x15 Gomoku has been solved by a combination of PNS, dependency-based search, and threat-space search (Allis, 1994).
Nonetheless, we choose to perform experiments on Gomoku because it is a straight-forward benchmark that is well-known and often used in games.
By excluding the highly speciﬁc dependency-based and threat-space search in our experiments, and by including experiments for the yet unsolved 9x9 Killall-Go, we wish to demonstrate PCN’s generality and positive impact on solving efﬁciency, regardless of whether expert knowledge is provided.
Combining PCN with powerful, highly game-speciﬁc heuristics is left as a future topic of research.
A strong case that supports the idea that neural networks do not clash with hand-crafted heuristics in solving is the results presented by Gao et al.
(2017), where a highly game-speciﬁc engine was combined with networks to enhance solving on 8x8 Hex.
We expect that PCN can be used to solve more challenging game problems in the future with the introduction of more sophisticated knowledge.
Lastly, PCN may be helpful for domains in which heuristics can be difﬁcult to design, e.g. planning, chemistry, material science, among others.
Ethics Statement We do not foresee any subtleties in regards to ethical concerns for this research at this time.
Reproducibility Statement All experiments can be reproduced by following the instructions in the README ﬁle in the supplementary materials, including source code, problem sets, MCTS and
FDFPN solvers, and the trained models used in this paper
A P LAYING S TRENGTH
Although our method does not optimize for strongest playing strength, it is an interesting question worth investigating, since we expect playing and solving to be correlated tasks.
With PCN the ORplayer tends to solve (win) the game as soon as possible, which coincides with strong play.
On the other hand, the goal for the AND-player is to maximum ¯ n ( s ) ; it receives the maximum reward ( ∞ ) if it wins the game.
This encourages the AND-player to win the game if possible, and prevent the OR-player from winning the game as much as possible otherwise.
Overall, PCN optimization preserves playing strength while prioritizing game solving.
Interestingly, the results in Table 3 show that our method is stronger than α 0 for 9x
Killall-Go (with a win rate of 60%), and has nearly the same playing strength as α 0 in 15x15 Gomoku.
Each experiment contains 250 games with alternating Black and White assignment.
Win Rate (WR)
Black WR
White WR 15x15 Gomoku (NDK)
PCN- b max 50.40% ± 6.21% 100.00% 0.80% 15x15 Gomoku (NDK)
PCN- b heur 50.80% ± 6.21% 100.00% 1.60% 15x15 Gomoku (4T)
PCN- b max 50.40% ± 6.21% 96.80% 4.00% 15x15 Gomoku (4T)
PCN- b heur 50.00% ± 6.21% 100.00% 0.00% 9x
Killall-Go
PCN- b max (OR: White) 50.00% ± 6.21% 34.40% 65.60% 9x
Killall-Go
PCN- b heur (OR: White) 60.80% ± 6.07% 66.40% 55.20% 9x
Killall-Go
PCN- b max (OR: Black) 62.25% ± 6.03% 60.00%
[Table]
B P ROBLEM S ET G ENERATION
We generate self-play games to create the problem sets.
For each self-play game, we start from the last move at the end of game and try to solve the game by using the three networks ( α 0 , PCN- b max ,
PCN- b heur ) with an MCTS solver.
If the problem can be solved within 5 minutes, we rollback two moves and try solving the resulting larger problem.
To generate problems with strong discriminating power, we repeat this process until one of the three programs cannot solve the problem withinminutes, upon which the problem is added into the problem set.
For 15x15 Gomoku, we choose Yixin (Sun, 2018), a strong Gomoku program that won several championships from 2012 to 2018, as the problem generating program.
Self-play games were generated with one second per move.
Since 15x15 Gomoku has been proven as Black win (Allis, 1994), and Yixin plays almost optimally, we only select problems with Black wins.
We collect 77 problems for Black wins from two variations, 43 and 34 problems from the NDK and 4T respectively.
Generally, the 34 problems is harder than 43 problems.
For 9x9 Killall-Go, since there are no open-source 9x9 Killall-Go programs, we simply use α 0 and
PCN- b max to generate self-play games.
The self-play games were generated with 400 simulations per move.
In addition, the Dirichlet noise is maintained in order to increase the diversity of self-play games.
We collect 81 problems for White wins, 39 and 42 problems from the self-play games by α 0 and PCN- b max respectively.
Black wins are trivial for PCN but difﬁcult for α 0 (as we mention in subsection 4.3), so we exclude them from our problem set.
All problem sets are included in the supplementary materials.
C D ETAILS FOR S OLVING P ROBLEM S ETS WITH D IFFERENT S OLVERS
Table 4 to 7 and Table 8 to 11 list the node count and the time for solving 15x15 Gomoku and 9x
Killall-Go with different network settings as heuristics on both MCTS-based and FDFPN solvers.
In the table, the fewest node counts and the shortest time to solve the problems among the different settings are bolded, where dashes represent settings where the problems cannot be solved within 30 minutes.
Note that the “no network” conﬁguration searches more nodes than any of the conﬁgurations where networks are used, within the same amount of time.
Each solver runs with one
CPU and one NVIDIA Tesla V100.
In 15x15 Gomoku, the solver without networks can visit nearly 10,000 and 30,000 nodes on MCTS-based and FDFPN solvers respectively, while the solver with networks can only visit 600 and 1,000 nodes on MCTS-based and FDFPN solvers respectively.
In 9x9 Killall-Go, the solver without networks can visit nearly 5,600 and 15,000 nodes on MCTS-based and FDFPN solvers respectively, while the solver with networks can only visit 600 and 950 nodes on MCTS-based and FDFPN solvers respectively.
Despite this massive difference in node counts, network-assisted solvers can still solve more problems.
To further improve network-assisted solvers, we can use asynchronous evaluation between CPU and
GPU.
By doing so, the solver no longer needs to wait for the GPU to compute during evaluation.
The FDFPN-CNN Hex solver by Gao et al.
(2017) is one such example.
In regards to the slower
CNN evaluations, they state that the runtime overhead of CNN evaluation is not an issue, since the sophisticated hand-crafted Hex heuristics can be up to 10 times slower than the CNN evaluations in early positions.
As a result, FDFPN-CNN can solve problems more quickly than DFPN and FDFPN (both without networks) in the same amount of time.
For 15x15 Gomoku, problems 1 to 43 are generated by NDK and problems 44 to 77 are generated by 4T. The NDK problem set is determined to be easier to solve than the 4T set because all NDK problems are solvable by the three networks trained with 4T, except for problem 6.
All of these experiments can be reproduced by following the instructions in the README ﬁle in the “PCN” directory of our supplementary code.
ID
[Table]
No Network
[Table]
ID
NDK 4T
No Network α 0 PCN- b max PCN- b heur
No Network α
PCN- b max PCN- b heur - - - - - 948, 11, 136,  - - - - - - 32, 28,  - - - - - - 63, 364,  - - - - - 233, 294, 549,  - - - - - 660, 657, 651,  - - - - - 879, 536, 584,  - - - - - - 418, 209,  - - - - - 1,008, 35, 61,  - - - - - - 478, 911,  - - - - - - 61, 317,  - - - - - - 273, 249,  - - - - - 633, 479, 453,  - - - - - - 268, 238,  - - - - - 854, 287, 284,  - - - - - 1,147, 464, 891,  - - - - - 261, 581, 361,  - - - - - 553, 72, 587,  - - - - - 535, 328,709 1,005,  - - - - - 802, 585, - - - - - - - 831, 294,  - - - - - 1,042, 342, 411,  - - - - - 199, 120, - - - - - - - 342, 304,  - - - - - 84, 794, 150,  - - - - - 198, 438, 866,  - - - - - - 376, 489,  - - - - - 527, 505, 719,  - - - - - 310, 267, 654,  - - - - - - 538, 272,  - - - - - - 354, 304,  - - - - - 1,048, 415, 437,  - - - - - 95, 77, - - - - - - 329, 928, 675,  - - - - - - 544, 673,
ID
No Network α
OR: White
OR: Black
PCN- b max
PCN- b heur
PCN- b max
PCN- b heur - - 437, 339, - 785,  - - 4, 33, - - - - 192, 211, - - - - 6, 5, 10, 12,  - 1,133, 554, 233, 877, 1,061,  - - 156, 70, - - - - 349, 421, - 925,  - - 409, 449, - - - - 160, 201, - - - - 437, 429, - - - - 529, 400, - - - - 156, 182, 586, 440,  - - 403, 405, 397, 454,  - - 69, 50, - - - 61, 46, 355, 55, - - - 455, 405, - 497,  - - 227, 274, - 791,  - 666, 576, 605, - 693,  - - 570, 574, - 917,  - - 70, 68, 120, 75,  - - 491, 363, - - - - 453, 245, - 830,  - - 476, 453, - - - 259, 544, - - - - - 257, 193, 1,020, - 4,009, - 1, 2, 137, 3,  - - 102, 51, 20, 45,  - 550, 368, 352, 855, 719,  - - 14, 19, 18, 108,  - - 325, 233, 484, 612,  - - 340, 321, - 536,  - 1,072, 650, 506, 483, 507,  - 699, 670, 658, 790, 647,  - 1,152, 136, 94, 182, 237,  - 682, 210, 330, 549, 335,  - 889, 362, 327, 606, 470,  - 690, 319, 214, 241, 249,  - 660, 782, 656, 604, 836,  - 188, 1,014, 214, 458, 285,
ID
No Network α
OR: White
OR: Black
PCN- b max
PCN- b heur
PCN- b max
PCN- b heur - - 391, 646, 590, 723,  - - 381, 425, - 757,  - - 603, 722, - - - - 360, 385, - - - 835, 134, 372, - - - - 49, 94, - - - - - 1,016, - - - - 766, 618, - 833,  - - 456, 289, 980, 700,  - 598, 313, 321, - 205,  - - 661, 499, - - - - 544, 455, - - - 1,058, 458, 237, - - - 1,079, 71, 127, - 625,  - - 165, 210, - - - 524, 258, 270, 418, 406,  - 681, 294, - 416, - - - 429, 763, - - - - 361, 278, 664, 646,  - 702, 176, 123, 600, 323,  - - 263, 279, 431, 249,  - 656, 10, 187, 207, 121,  - - 306, 241, 1,032, 958,  - - 545, 782, - 779,  - - 353, 438, - 421,  - 952, 478, 463, 615, 566,  - 979, 748, 654, 709, 804,  - - 815, 462, - - - - 896, 547, - - - 773, 85, 97, - - - - 536, 656, - - - - 758, 800, 913, - - - 305, 352, 544, - - 761, 403, 445, 680, 553,  - - 564, - - - - 830, 37, 34, 594, 161,  - - 315, 332, - - - - 541, - - - - 712, 553, 451, 892, 606,  - - 779, 320, -
[Table]
ID
NDK 4T
No Network α
[Table]
No Network α
PCN
[Table]
ID
NDK 4T
No Network α 0 PCN- b max PCN- b heur
No Network α
PCN- b max PCN- b heur - - - 1,792, - - 251, 267,  - - - - - - 56, 44,  - - 541, 183,799 34,104, 439, 37, 4,  - - - - - - 93, 159,  - - - - - - 81,989 1,790,  - - - - - - 480, 386,  - - - - - - - 1,610,  - - - - - - 298,338 1,033,  - - - - - - - 1,798,  - - - - - - 88, 125,  - - - - - - - - - - - - - - 356, 337,  - - - - - - - - - - - - - - - - - - - - - - 948,526 1,584,  - - - - - 1,438, 468, 982,  - - - - - - 92, 114,  - - - - - - 1,068, - - - - - - - 519, 958,  - - - - - - 594, 281,  - - - - - - 460, 597,  - - - - - - 1,081, 578,  - - - - 21,638, - 1,300, - - - - 1,374, - 69, 57, 26,  - - - - - - 611,155 1,490,  - - - - - - 709,441 1,297,  - - 654, - - - 60,078 1,507,  - - - - - - 705, 318,  - - - 1,822, - - 102, 251,  - - - - - - - - - - - - - - 709, - - - - - - - 143, 204,  - - - - - - 1,359, - - - - 1,257, - 130, 88, 34,
ID
No Network α
OR: White
OR: Black
PCN- b max
PCN- b heur
PCN- b max
PCN- b heur105, - 251, - - 440, 1,185, 85, 849, - 1,517,  - 165, 66, 48, 282, 254,  156, 11, 25, 58, 402, 598,  12,225, 134, 49, 82, 899, 384,  94, - 229, - - - - 88, 49, 42, 327, 92,  - 324, 99, 84, 550, 1,204,  - 91, 38, 58, 904, 576,  - - 168, - - - - 76, 707, 439, - - - 109, 47, 65, 166, 158,  - 91, 68, 70, 78, 69,  913, 139, 30, 38, 344, - 714, - 594, - 440, - - 74, 56, 73, 716, 649,  - 104, 67, 52, 269, 326,  - 371, 404, 764, - 994,  - 236, 153, 175, 538, 454,  - 49, 52, 81, 840, 52,  - 229, 414, 112, - - - 68, 352, 37, 349, 85,  - 201, 154, 108, - 854,  - 55, - 92, 1,152, 735,  5,751, 70, 72, 73, 228, 151,  18, 155, 308, 119, 311, 210,  267, 17, 30, 24, 26, 27,  - 94, 77, 73, 1,073, 246,  160, 20, 4, 4, 25, 32,  6,306, 19, 41, 41, 116, 74,  - 123, 176, 61, 184, 575,  - 204, 77, 74, 432, 207,  - 453, 455, 142, 368, 292,  9,370, 76, 169, 40, 173, - 18,455, 92, 125, 90, 959, 207,  - 130, 187, 115, - 555,  - 365, 191, 39, 41, 59,  - 171, 75, 92, 153, 96,  1,325, 40, 309, 243, 451, 61,
ID
No Network α
OR: White
OR: Black
PCN- b max
PCN- b heur
PCN- b max
PCN- b heur 1,568, 11, 68, 62, 121, 119,  5,582, 85, 77, 49, 239, 80,  12,159, 208, 250, 727, 1,515, 297,  - 398, 104, 109, 155, 393,  7,868, 51, 116, 95, 105, 127,  - 42, 26, 59, 260, 1,155,  - 332, 481, 470, 517, 481,  - 153, 178, 324, 105, 649,  3,546, 61, 190, 63, 570, 119,  3,463, - 60, 33, 542, 363,  - 119, 59, 57, 172, 226,  - 202, 86, 75, - 1,194,  - 60, 27, 623, 932, - 2,532, 931, 49, 43, - 141,  7,796, 96, 28, 38, 127, 216,  23,427, 78, 44, 42, 51, 61,  - 62, 58, 229, 84, 230,  - 297, 298, 563, 1,277, 567,  - 45, 67, 60, 131, 174,  - - 35, - - 247,  - 76, 96, 134, 396, 99,  3,228, 180, 1,000, 551, 45, 1,004,  1,732, 95, 67, 53, 84, 67,  - 136, 51, 86, 164, 98,  19,265, 94, 199, 193, 212, 166,  - 226, 81, 83, 81, 308,  - 101, 52, 169, 53, 64,  - 191, 120, 92, 948, - - 453, 130, 247, 128, 143,  3,538, 40, 389, 268, 1,212, 761,  - 184, 141, 226, - 754,  - 345, 338, 135, 323, 952,  16,519, 113, 63, 491, - - - 241, 108, 112, 231, 1,488,  - 329, 478, 108, 338, 707,  4,266, 39, 177, 26, 61, 108,  - 216, 162, 222, 516, 591,  5,212, 74, 372, 283, 57, - - 101, 890, 1,313, 75, 214,  3,951, 913, 538, 1,560, - - 5,556, 51, 189, 137, 688, 431,  - 203, 258, 116, 250, 130,
ID
[Table] [Table]
ID
NDK 4T
No Network α 0 PCN- b max PCN- b heur
No Network α
PCN- b max PCN- b heur - - - - - 1,416.
ID
No Network α
OR: White
OR: Black
PCN- b max
PCN- b heur
PCN- b max
PCN- b heur - - 717.
ID
No Network α
OR: White
OR: Black
PCN- b max
PCN- b heur
PCN- b max
PCN- b heur - - 609.
[Table]
ID
NDK 4T
No Network α
[Table]
Network α
[Table]
ID
NDK 4T
No Network α 0 PCN- b max PCN- b heur
No Network α
PCN- b max PCN- b heur - - - 1,531.
ID
No Network α
OR: White
OR: Black
PCN- b max
PCN- b heur
PCN- b max
PCN- b heur123.
ID
No Network α
OR: White
OR: Black
PCN- b max
PCN- b heur
PCN- b max
PCN- b heur 184.
[Table] [Table]
[Table] [Table]
[Table] [Table]