A DVERSARIAL P OLICY G RADIENT FOR A LTERNATING M ARKOV G AMES
Anonymous authors
Paper under double-blind review
A BSTRACT
Policy gradient reinforcement learning has been applied to two-player alternateturn zero-sum games, e.g., in AlphaGo, self-play REINFORCE was used to improve the neural net model after supervised learning.
In this paper, we emphasize that two-player zero-sum games with alternating turns, which have been previously formulated as Alternating Markov Games (AMGs), are different from standard MDP because of its two-agent nature.
We exploit the difference in associated
Bellman-equations, which leads to different policy iteration algorithms.
In principle, as policy gradient method is a kind of generalized policy iteration, we show how these differences in policy iteration are reﬂected in policy gradient for AMGs.
We reformulate an adversarial policy gradient and discuss potential possibilities for developing better policy gradient methods other than self-play REINFORCE.
Speciﬁcally, the core idea is to estimate the minimum rather than the mean for the “critic”.
Experimental results on the game of Hex show that our newly developed Monte-Carlo policy gradient methods are able to learn better pure neural net policy than the REINFORCE variants.
We also describe a board-size independent neural net architecture.
As a by product, we show that when combined with search, using a single neural net model, the resulting program consistently beats MoHex 2.0, the state-of-the-art multi-boardsize Hex program after 2013, on board sizes from 9 × 9 to 13 × 13 .
Furthermore, we show that integrating minimum playout return to Monte-carlo tree search provides additional improvements.
I NTRODUCTION
Reinforcement Learning (RL) is a trial-and-error paradigm where an agent learns by interacting with an environment.
Two problems are involved in reinforcement learning, the prediction problem seeks to learn the true value function of a following policy; the control problem aims to ﬁnd an optimal policy that the long-term expected cumulative reward is maximized.
The environment is assumed to be a Markov Decision Process (MDP)
(Bellman, 1957), which manifests a Markov property.
Compared to dynamic programming, RL methods have the advantage of being model-free which learn by treating the environment as a black box.
Nevertheless, despite the impracticality in real applications, dynamical programming is theoretically fundamental to reinforcement learning (Bertsekas &
Tsitsiklis, 1996).
Indeed, almost all RL algorithms are a form of generalized policy iteration , which improve the policy while estimating a value function (Sutton & Barto, 2017).
Model-free reinforce learning methods have been successfully applied to many domains, including robotics (Kober et al., 2013), Atari games (Mnih et al., 2015; 2016; Schaul et al., 2016), and twoplayer board games (Tesauro, 1995; Silver et al., 2016).
Most of RL algorithms fall into one of two categories: valued ﬁtting and policy gradient.
The value-ﬁtting algorithms try to optimize a value function through iteratively minimizing a sequence of Bellman inconsistencies of observed states or state-action pairs, e.g., on-policy SARSA
(Rummery & Niranjan, 1994) and off-policy Qlearning (Watkins & Dayan, 1992).
The optimal deterministic policy is implicitly learned when the value function converges to optimal.
The primary disadvantage of value-ﬁtting methods, such as Q-learning (Watkins & Dayan, 1992;
Mnih et al., 2015; Wang et al., 2016; Mnih et al., 2016), is that they might be unstable when interacting with function approximation (Bertsekas & Tsitsiklis, 1996;
Sutton et al., 2000).
To gain stable behavior, often extra heuristic techniques (Lin, 1992; Schaul et al., 2016) and extensive hyper- parameter tunning are required.
By contrast, policy gradient methods explicitly represent a policy as a function of parameters: they learn through iteratively adjusting the parameterized policy by following estimation of gradients, thus converging to at least a local maximum (Peters & Bagnell, 2011).
Policy gradients are applicable to continuous control (Silver et al., 2014; Schulman et al., 2015; 2017) or domains with large action space (Silver et al., 2016), whereas action-value learning methods often become infeasible (Lillicrap et al., 2016;
Wang et al., 2017).
Both value and policy based reinforcement learning are well-studied in MDP, where a single agent learns by exploiting a stationary environment.
In this paper, we focus on policy gradient methods for two-player zero-sum games played in alternating turns, i.e., Alternating Markov Games (AMGs) (Littman, 1996).
AMGs are a specialization of Stochastic Games (Shapley, 1953) but a generalization of MDPs by allowing exactly two players pursing diametrical goals.
Many popular two-player games are of such kind, such as chess, checkers, backgammon and Go.
The restriction of zero-sum in AMGs makes it possible to deﬁne a shared reward function such that one agent tries to maximize while other agent tries to minimize it.
Indeed, due to such a property, with a notion of self-play policy, standard reinforcement learning methods have been successful applied to
AMGs (Tesauro, 1995; Silver et al., 2016) by simply negating the reward signals of the opponent’s turns.
In this paper, we reexamine the justiﬁcations of adapting standard reinforcement learning methods to AMGs.
We begin by reviewing the fundamental differences in corresponding Bellman-equations, from which we show how the resulting policy iterations algorithms are disparate.
We reformulate an adversarial policy gradient objective speciﬁcally for AMGs, based on which we discussion potential opportunities for developing new policy gradient methods for AMGs.
Speciﬁcally, we apply our approach to the game of Hex, we show by modifying REINFORCE to estimate the minimum rather than the mean return of a self play policy, stronger pure neural net players are obtained.
F INITE MDP
The ﬁnite MDP is a tuple ( S , A , R , P , γ ) where: S is ﬁnite set of states, A is a set of actions, R :
S ×
A → R is a reward function, and P deﬁnes the probabilistic transition among states.
An agent lives in an environment characterized by the Markov property, i.e., Pr( s t + | s 1 , a 1 , . . .
, s t , a t ) =
Pr( s t +1 | s t , a t ) .
The agent learns by receiving reward signals from interacting with the environment.
The goal is to maximize the expected cumulative reward: E (cid:80) ∞
k
= t γ t − k R t − k +1 , where γ is a discounting factor 0 < γ ≤ 1 that controls the contribution of long-term and short-term rewards; γ = 1 is only possible in episodic tasks.
The basis for reinforcement learning in MDP is a set of Bellman Equations .
For a given policy π :
[Equation]
where p ( s (cid:48) | s, a ) is the transition function and r ( s, a, s (cid:48) ) is the reward of taking a at s arriving s (cid:48) .
It is also popular to use the notion of action-value function:
[Equation]
The above Bellman equations are the basis for policy evaluation , which is to approximate the true value function for a given π .
This corresponds to the prediction problem in RL.
The Optimal Bellman Equation is a recursive relation for the optimal policy:
[Equation]
A value iteration procedure derived by the above optimal Bellman-equation converges to optimal (Bellman, 1957).
Alternatively, it is also possible to derive a policy iteration (Howard, 1960;
Bertsekas & Tsitsiklis, 1996) by combining Equation (1) and (3).
In reinforcement learning, the precise model is usually assumed unknown, the learning typically interleaves policy evaluation and policy improvement , which is summarized as generalized policy iteration (Sutton & Barto, 2017).
A LTERNATING M ARKOV G AMES
Alternating Markov Games are a specialization of Stochastic Games with only two players: it is a tuple ( S 1 , S 2 , A 1 , A 2 , R , P , γ ) where S 1 and S 2 are respectively this and other agent’s states, A
and
A 2 signiﬁes respectively the actions at each player’s states.
It is clear that MDP can be viewed as a special case of AMG by restricting |S 2 | = 0 .
In general
[Equation]
since there are two players in AMG
[Equation]
the policy evaluation thus involves two policies
[Equation]
i
[Equation]
e., π 1 and π 2 .
Bellman equations for policy evaluation are:
[Equation]
Action-value functions can also be deﬁned:
[Equation]
Further, suppose π 1 is the max player while π 2 is the min player, assuming the π 2 is an optimal “counter policy” with respect to π 1 , then we may rewrite the above equation as:
[Equation]
where s ∈ S 1 , s (cid:48)
∈ S 2 and s (cid:48)(cid:48) ∈ S 1 .
[Equation]
[Equation]
The replacement of π 2 with a min operator is caused by the observation that, since π 1 is ﬁxed, the problem reduces to a single-agent MDP where an agent tries to minimize the recevied rewards.
The optimal Bellman equation (also implies Nash equilibrium) can be expressed as:
[Equation]
assuming that states in S 1 belongs to the max player, the other one is the min player.
A value iteration algorithm according to this minimax recursion converges to optimal (Condon, 1992).
However, because the policy evaluation in AMGs consists of two policies π 1 , π 2 , the policy iteration, which alternates between policy evaluation and policy improvement, could have the following four formats:
Algo.1 Fix π t 1 , compute the optimal counter policy π t 2 , then ﬁx π t 2 compute the optimal counter policy π t +11 , alternating this procedure repeatedly, Algo.2 Policy evaluation with π t 1 , π t 2 , swith both π t 1 and π t 2 to greedy policies with respect to current state-value function, continue this procedure repeatedly, Algo.3 Policy evaluation with π t 1 , π t 2 , swith π t 1 to greedy policy with respect to the current statevalue function and then compute the optimal counter policy for π t 2 , continue this procedure repeatedly, Algo.4 Policy evaluation with π t 1 , π t 2 , swith π t 2 to greedy policy with respect to the current statevalue function and then compute the optimal counter policy for π t 1 , continue this procedure repeatedly.
Intuitively speaking, all of those procedures are somewhat sensible, and perhaps would yield practical success.
However, Condon (1990) showed that oscillation could be a problem that prevents
Algo.1 and Algo.2 from converging, only Algo.3 and Algo.4 are correct, guaranteed to converge in general (Hoffman & Karp, 1966).
Algo.3 and Algo.4 are duals.
Variants of Algo.3 or Algo.4 , such as only switching one node to greedy every iteration then optimize the other player’s strategy, are also correct, although they may have slower convergence rate (Condon, 1990).
A DVERSARIAL POLICY GRADIENT
Before presenting adversarial policy gradient for AMGs, we ﬁrst review policy gradient in single agent MDP.
In MDP, the strength of a policy π can be measured by (note that for brevity, we implicitly state that the policy is parameterized by θ ):
[Equation]
where d π ( s ) is a state distribution under π .
The gradient of J ( π ) is:
[Equation]
The above formula is the Policy Gradient Theorem in MDP (Sutton et al., 2000), which implies that the gradient of the strength of a policy can be estimated by sampling according to π .
The requirements are 1) π is differentiable, and 2) q π
( s, a ) can be estimated.
This theorem can also be interpreted as a kind of generalized policy iteration, where the gradient ascent corresponds to policy improvement (Kakade, 2002), q π ( s, a ) is obtained by some policy evaluation.
Depending on how q π ( s, a ) is estimated, policy gradient algorithms can be categorized into two families: MonteCarlo policy gradients that use Monte-Carlo to estimate q π , e.g., REINFORCE (Williams, 1992) and actor-critic (Sutton et al., 2000; Wang et al., 2017) methods that use another learning parameter to approximate the action-value under π .
Recent approaches (Gu et al., 2017) interpolate both.
Analogously, in AMGs, for a pair of parameterized policies π 1 and π 2 , the joint-strength of π 1 and π 2 may be deﬁned as:
[Equation]
where d π 1 ,π 2 ( s ) is the state-distribution given π 1 and π 2 .
A nature question is what is the gradient of J ( π 1 , π 2 ) with respect to π 1 and π 2 respectively?
One tempting derivation is to calculate the the gradient for both π 1 and π 2 simultaneously by treating the other policy as the “environment”.
Similar to the mutual greedy improvement in Algo.2 , such a method tries to adapt based on the value function under current π 1 , π 2 , ignoring the fact that the opponent is a dynamic adversary who will also adapt its strategy for better counter-payoffs.
Another possible algorithm is to ﬁx π 1 and do a ﬁx number of iterations to optimize π 2 by normal policy gradient as in MDP, and then ﬁx π 2 for optimizing π 1 , repeat such alternatively.
However, this algorithm is an analog to Algo.1 , which has shown to be non-convergent in general.
Following Algo.3 and Algo.4 , given the action-value function under π 1 , π 2 , a more reasonable approach for policy improvement is thus to switch π 2 to “greedy” (not necessary every node of π 2 ) and optimize the π 1 by policy gradient.
Therefore, assuming π 1 is the max player, we advocate the following objectives
[Equation]
Consequently, the gradients can be estimated by
[Equation]
The above formulation implies that, when computing the gradient for one policy, the other policy is simultaneously switched to “greedy”.
This joint-change forces the current player to adjust the action preferences according to the worst-case response of the opponent, which is desirable due to the adversarial nature of the game.
4. A DVERSARIAL M ONTE -C ARLO POLICY GRADIENT
A straightforward implementation of Equation (12) is to do separate Monte-Carlo from each nextaction, and then apply the min or max operator.
However, this may not be practical feasible when the action space is large.
We introduce a parameter k , by which only a subset of actions are considered in our adversarial Monte-Carlo policy gradient design.
We conduct minimal modiﬁcation on self-play REINFORCE.
The algorithms work as follows: after a batch of n games is generated by a self-play policy with game results, for each game, a single state-action pair ( s, a ) is uniformly sampled as a training example.
Instead of directly using the observed return z in batch as the estimated action-value for ( s, a ) , we perform extra Monte-Carlo simulations to estimate the minimum return for ( s, a ) .
In below are two methods we propose:
AMCPG-A: Run k self-play games from the ( s, a ) using self-play policy π , take the minimum the k additional returns and z .
AMCGP-B: Sample a s (cid:48) from ( s, a ) according to state-action transition p ( ·| s, a ) , select top k predicated actions using self-policy π , doing separate self-play afterwards, take the minimum of the k additional returns and z .
Note that only “minimum” is used, because we assume the game result is with respect to the player to play at state s .
It is easy to see that in AMCPG-A , if mean operator is used rather than minimum , a self-play REINFORCE variant is recovered.
When k = |A ( s (cid:48) )
| , AMCPG-B becomes a full number
Monte-Carlo estimate, but in this situation, even though each action-value’s estimation is unbiased, bias could still be incurred when applying the min or max .
This is due to the well-known “winner’s curse” problem (Capen et al., 1971; Smith & Winkler, 2006).
By introducing a parameter k , the above two methods can be seen as some kind of “soft min”, which is usually better than applying a “hard min” on noisy estimates.
4. A DVERSARIAL A CTOR -C RITIC
As in MDPs, another possible direction is to implement Equation (12) in an adversarial actor-critic framework.
Compared to the actor-critic in MDP, the difference is that a min operator will be used when estimating the “critic”.
In practice, for example, when neural net is used as a function approximator, the min operator might be easily achieved by employing an architecture with both policy and action-value outputs.
More desirably, as in Q-learning (Fox et al., 2016), the biased caused by min operator can be addressed by certain soft updates.
R ELATED WORK
Previous studies of games are divided among different disciplines, including game theory, reinforcement learning as well as computational complexity.
Shapley (1953) introduced the notion of
Stochastic Games, which is a multi-player framework that generalizes both Markov Decision Process (only one player) and repeated games (only one state).
Condon (1990; 1992) initiated the study of Simple Stochastic Games , which are two-player games played in a directed graph with min , max and restricted probabilistic transition nodes.
While her concern was largely in a computational complexity perspective, Condon (1990) showed that several variants of Hoffman-Karp’s algorithms are incorrect.
Littman (1996) formulated the notion of Alternating Markov Games, which is a more general Simple Stochastic Games without the restriction in action sets and probabilistic transitions.
Littman (1994) also proposed a minimax-Q learning algorithm that is applicable to Alternating Markov Games as well as two-player zero-sum games played with matrix payoffs.
One of the most cited work that uses reinforcement learning to play AMGs is by Tesauro (1995), who trained multi-layer neural network to play backgammon.
Tesauro’s program takes advantage of the symmetric (due to alternating and zero-sum) property of AMGs and learn the optimal value function by “greedy” self-play, relying the stochastic environment for exploration.
It was conjectured that the smoothness of the value function is one major factor to the particular success of
TD-Gammon (Tesauro, 1995).
The recent advances of reinforcement learning are arguably primarily due to the use of deep neural networks (LeCun et al., 2015) as a much more expressive function approximator.
Deep neural nets have been applied to two-agent board game Go (Maddison et al., 2015; Tian & Zhu, 2016; Clark &
Storkey, 2015), leading to super-human Go-playing system AlphaGo (Silver et al., 2016).
AlphaGo consists of several independent components including supervised learning for move prediction, reinforcement learning and Monte-Carlo Tree Search (MCTS)
(Coulom, 2006).
AlphaGo successfully combined those strengths and achieved remarkable success.
Newly updated AlphaGo Zero uses a “heavy” self-play based on MCTS to improve its parameterized neural net (Silver et al., 2017).
A similar approach is Expert Iteration (Anthony et al., 2017).
However, one drawback of such methods is that huge computation could be required when using MCTS self-play for data generation, i.e., when applied to Shogi and Chess, 5000 TPUs were used (Silver et al., 2017).
Policy gradient reinforcement learning, on the other hand, is a conventional fast RL method that can be used to reﬁned a neural net policy by pure neural net self-play.
Indeed, in AlphaGo (Silver et al., 2016), policy gradient RL is used to improve the neural network model obtained by supervised learning.
The improved model is usually less useful in search, primary because its strong bias towards some local optima.
However, due to its better playing strength and fast speed, a possible usage is using it to generate training data for a value net (Silver et al., 2016), which can be integrated into Monte-Carlo tree search.
The policy gradient employed by AlphaGo is a variant of self-play
REINFORCE that resembles to Algo.2 .
A practical innovation is to select an opponent parameter from a previous iteration, so as to increase the stability of the training.
Policy gradient has also been applied to train a balanced policy (Silver & Tesauro, 2009); however, its requirement of an optimal value function makes it less useful in practice (Huang et al., 2010).
Adversarial methods have also been adopted in MDP (Pinto et al., 2017), with the observation that errors may occur in simulated models, thus maximizing a worst-case return will generally produce more robust results.
The alternating procedure proposed by Pinto et al.
(2017) resembles to Algo.1 .
The idea of adversarial learning is also used in generative models (Goodfellow et al., 2014), which leverages adversarial examples to train a more robust classiﬁer.
E XPERIMENTS
In this section, we present experimental results of the proposed adversarial Monte-Carlo policy gradient algorithms in Section 4.
We ﬁrst introduce the testbed game, and present experimental results subsequently.
6. G AME OF H EX
The game of Hex is played on a N × N rhombus board, where Black and White alternatively place a stone on an unoccupied hexagonal cell.
The goal is to connect two sides of the board.
Since its invention, the game of Hex has been an active research problem for both mathematicians (Nash, 1952) and computer scientists (Shannon, 1953).
Nash (1952) proved by strategy stealing argument that from the empty board, the game theoretic value is a ﬁrst-player win.
However, explicit winning strategy is unclear, i.e., after an arbitrary opening move on the board, the theoretical result become unknown.
Hex board can also be mapped into Go-style, in which stones are played at intersections, as shown in Figure 1 (left).
Hex is a game collected in OpenAI gym; usually 11 × 11 is considered as a regular board size, 13 × 13 Hex is also played at computer Olympiad Hex tournaments.
Similar to Go of same board size, Hex is challenging to play primarily because of its large and near-uniform branching factor and the difﬁculty of constructing reliable evaluation function.
Hex is simpler than Go in the sense that perfect play can often be achieved whenever virtual connections are found by H-Search (Anshelevich, 2002).
We consider using policy gradient reinforcement learning to play the game of Hex.
Similar to the previous work in Go, we use deep convolutional neural networks as a function approximator.
Figure 1(right) shows our architecture design.
Instead of ﬁxing the board size to be particular number, we design an architecture that is able to receive different board size inputs, which can be achieved by having multiple parallel inputs with different shape when building the computation graph using
Tensorﬂow (Abadi et al., 2016).
Speciﬁcally, the architecture allows board size 8 ≤ N ≤ 15 .
After boarding padding, each raw board state is processed into 12 binary feature planes, which are respectively: Black stones, White Stones, empty points, to-play plane, black bridge endpoints, white bridge
[Figure]
endpoints, to-play save bridge points, to-play make-connection points, to-play form bridge points, opponent save bridge points, opponent form bridge points, opponent make-connection points.
6. R ESULTS
We apply policy gradient reinforcement learning to improve the resulting policy net trained from supervised learning.
In addition to our proposed adversarial Monte-Carlo policy gradient algorithms, for comparison purpose, we implement three REINFORCE variants, as shown in below.
• REINFORCE-V: Vanilla REINFORCE using a parameterized self-play policy.
After a batch of n self-played games, each game is then replayed to determine batch policy gradient update αn (cid:80) ni (cid:80) T i t ∇ log π
( s it , a it ; θ ) z it , where z it is either +1 or − 1 .
• REINFORCE-A: An “AlphaGo-like” REINFORCE.
It differs from REINFORCE-V by randomly selecting a set of previously saved parameter weights from former iterations as the opponent for self-play.
• REINFORCE-B: A new implementation that for each self-played game, only one stateaction pair is uniformly selected for policy gradient update.
It differs from AMCPG-A by using the mean of all k + 1 observed returns.
All methods are implemented using Tensorﬂow, sharing the same code base.
In fact, they are only different in a few lines of code.
A self-play policy is employed for all algorithms, which is equivalent as forcing π 1 and π 2 to share the same set of parameters.
The game batch size n is set to 128, with opening move for each game played uniform randomly; learning rate is set to 0 .
001 after a grid search, vanilla stochastic gradient ascent is used as the optimizer.
The reward signal is either +1 or − 1 , and only appears after a complete self-play game; no discounting is used.
For all algorithms, the same neural net architecture is adopted, with initial parameter weights obtained from supervised learning on a dataset generated by MoHex (Henderson, 2010) self-play on board size 9 × 9 .
This supervised learning is only on 9 × 9 Hex, trained parameter weights are reused in other board sizes as the starting policy.
We run policy gradient reinforcement learning for 400 iterations for each method, on different board sizes (smaller board 9 × 9 and regular board 11 × 11 ) with varied k .
We use open source player
Wolve 1 (Henderson, 2010) as a benchmark to measure the relative performance of learned models.
[Figure]
[Table]
t
[Table]
a
[Table]
a i n
[Table]
t
W
[Table]
l ve
[Figure]
W n
[Table]
a t e a a i n s t
W o l ve
[Figure]
W i n r a t e a g a i n s t
W o l ve
[Figure]
The primary strength of Wolve comes from its inferior cell analysis augmented H-Search, making it is able to discover winning strategies for perfect play much earlier than unenhanced H-Search, along with more accurate position evaluation with electric resistance evaluation (Anshelevich, 2002;
Henderson, 2010) if no winning virtual connection can be found.
H-Search (Anshelevich, 2002) computes virtual connections by iteratively applying an AND-OR hierarchical algebra, this also makes it orders of magnitude slower than a pure neural net evaluation.
The tournaments with Wolve are played by iterating all opening moves, each repeated 5 times with Wolve as Black or White.
Figure 2 compares the strength of these ﬁve different algorithms.
It is interesting that REINFORCEB is able to achieve similar performance with REINFORCE-V, even though the number of training samples used in the former algorithm is signiﬁcantly less, this is perhaps because the reward signal in the same game is too much correlated.
Consistent with the ﬁnding in Go (Silver et al., 2016), the method REINFORCE-A obtained small performance improvements on both 9 × 9 and 11 ×
Hex.
However, the newly developed algorithms AMCPG-A and AMCPG-B are able to learn better polices.
They tend to learn faster, also generally achieve better results even when k = 1 .
The better performances are more eminent in regular board size 11 × 11 .
Those results conﬁrm the beneﬁt of estimating the minimum , rather than mean return, when applying policy gradient methods to Alternating Markov Games.
We note that policy gradient training on both 9 × 9 and 10 × 10 Hex is very fast.
On an Intel i7CPU computer with single GTX 1080 GPU, most took only a few hours, the longest training by
AMCPG-A, AMCPG-B and REINFORCE-B ( k = 9 , 11 × 11 Hex) took about 12 hours.
For k =case, AMCPG-A, AMCPG-B and REINFORCE-B are about twice slower than REINFORCE-V and
REINFORCE-A.
6.2. M INIMUM RETURN IN M ONTE -C ARLO T REE S EARCH
In the previous section we have shown policy gradient can beneﬁt from minimum Monte-Carlo return.
State-of-the-art methods for Hex use Monte-Carlo tree search, which can be viewed as using the Monte-Carlo return as a “critic” to guide a tree represented policy.
To further see the effect of worst “critic”, we proceed to apply minimum rollout return to MoHex 2.0 (Huang et al., 2013), the world champion Monte-Carlo tree search Hex player since 2013.
Mohex 2.0 is an improved version of MoHex 2011 (Arneson et al., 2010), with stronger virtual connections (Pawlewicz et al., 2015) and pattern based playout.
In MoHex 2.0, after an in-tree phase, a leaf node is selected, then one or more rollouts are conducted thereafter by a fast playout policy trained on 13 × 13 expert Hex games.
MoHex 2.0 back-propagates the average rollout result to the tree.
Following the same idea in the previous section, we modify
MoHex 2.0 to use the minimum of the rollout results instead.
To see the strength of minimum return, we experimentally compare three programs: MoHex 2.0, MoHex-A and MoHex-Mixed.
MoHexA is directly modiﬁed on MoHex 2.0’s code, for all programs, at the leaf node, 2 playouts are conducted.
The only difference between MoHex-A and MoHex 2.0 is that MoHex-A backs up the minimum playout results (w.r.t the player to play at the leaf) while MoHex 2.0 uses average playout results.
Another program MoHex-Mixed uses the linear combination of the minimum and average return, with factors 0 .
5 and 0 .
Table 1 summarizes the comparative results.
It is easy to see that MoHex-A and MoHex-Mixed can still achieve slightly better results, due to their deliberate consideration of minimum “critics”.
same number of simulations.
All programs use the default parameters the same as MoHex 2.0 except that number playouts per visit is set to 2.
On each board size, every opening was tried 10 times with each player as black or white.
1690 games were played for each match.
[Table]
6.2. M O H EX -CNN FOR MULTIPLE BOARD SIZE H EX
Unlike previous work in Go (Silver et al., 2017) or Hex (Anthony et al., 2017)
[Table]
the neural net we proposed has no board size dependent fully connected layers, this makes it possible to easily reuse the trained neural network on other board sizes other than 9 × 9 .
To see the generalization ability across different board sizes in search, we incorporate the trained neural net to MoHex 2.0, the resulting program is MoHex-CNN.
The difference between MoHex-CNN and MoHex 2.0 is that
MoHex-CNN replaces the prior probability of MoHex 2.0 with the policy neural net trained on 9 ×
Hex games,
i
[Table]
e., the neural net model before policy gradient learning
[Table] [Table]
Table 2 contains the results of MoHex-CNN on board sizes from 9 × 9 to 13 × 13 .
We note that board sizes smaller than 9 are not investigated because they can easily be solved by virtual connection computation or tree based proof number search (Henderson, 2010).
MoHex 2.0 was not designed to play board size > 13 , so we stop at board size 13 .
From Table 2, it is easy to notice that MoHexCNN is able to defeat MoHex 2.0 on every board size, which is remarkable since the neural net model was only trained on a set of 9 × 9 Hex games for only a few hours on a single GPU.
The computation overhead is also very small, because in our implementation, neural net evaluation is computed in parallel with prior pruning 2 when expanding a node.
For 13 × 13 Hex, we have seen that adversarial Monte-Carlo return slightly improved MoHex 2.0, we further show the results in Table 3 by combining the minimum Monte-Carlo return with MoHexCNN.
Same as the notation in previous section, MoHex-CNN-A uses the minimum of two playout returns while MoHex-CNN-Mixed uses the linear combination of minimum and average by factors 0 .
5 and 0 .
[Table]
Results in Table 3 indicate that using the mixed return achieve the best performance.
This is perhaps because, different from policy gradient, the in-tree phase in MCTS is already an implicit form minimax .
However, results in Table 3 suggest that a moderate skewing the critic to the worst-case still helped the search.
6.2. C OMPARISON WITH E X I T
ExIt (Anthony et al., 2017) was competed against MoHex 2011 (Arneson et al., 2010) on 9 × 9 Hex.
To have a grasp of the relative strengths of ExIt, MoHex 2.0 and MoHex-CNN, we also present results of MoHex 2.0 and MoHex-CNN played against MoHex 2011 on 9 × 9 Hex 3 .
Following the experiment in ExIt, we show the results with 10 4 and 10 5 simulations, in Table 4.
[Table]
It is easy to see that when 10 4 simulations is used, both MoHex 2.0 and MoHex-CNN has similar winrates as ExIt.
However, the 10 5 simulations MoHex 2.0 and MoHex-CNN obtain much better winrates than ExIt (Anthony et al., 2017).
We further note that, the state-of-art for 9 × 9 Hex is that all openings have been solved (Pawlewicz & Hayward, 2013) 4 .
This parallel solver has been embedded into MoHex 2.0, but in default setting, it is turned off.
C ONCLUSIONS
We reviewed MDP and AMG, and proposed a new adversarial policy gradient paradigm that uses the worst-case return as the “critic”.
We introduced two practical adversarial Monte-Carlo policy gradient methods.
We evaluated our algorithms on game of Hex, the experimental comparisons show that the new algorithms are notably better than routinely adopted self-play REINFORCE variants.
We also applied minimum return to Monte-carlo tree search, and observed better performance on 13 ×
Hex.
Another contribution we made is a multi-boardsize neural net architecture, we demonstrated that a single neural net model trained on smaller board size can effectively generalize to larger board sizes.
As a result, using it as a prior knowledge, we are able to improve MoHex 2.0 consistently from 9 × 9 to 13 × 13 Hex.
Since in practice, expert data is often scarce, difﬁcult to generate with limited computation resources, in this paper we provided a more feasible approach for high-performance playing for multi-boardsize Hex.
On the other hand, based on the adversarial policy gradient formulation in AMGs the Monte-carlo policy gradients either self-play REINFORCE-V or others, are not unbiased anymore, yet still have the drawback of being sample-inefﬁcient and may exhibit high variance.
In this sense, an adversarial actor-critic framework is perhaps more appealing.
In Monte-carlo tree search framework, value neural network can also be used to replace Monte-carlo rollouts.
We leave this line of research as future work
A PPENDIX A P SEUDOCODE
We present the pseudocode of AMCPG-A and AMCPG-B .
Let z ( s j , a j ) be the outcome with respect to action a j at s j in e i ;
From ( s j , a j ) , z (cid:48) ( s j , a j ) =
  s
Write ( s j , a j , R i ) to the batch ;
[Equation] [Equation]
←
[Equation]
We also present the pseudo-code of Algo.1 , Algo.2 and Algo.3 in Section 3.
B E XPERIMENTS B. S UPERVISED LEARNING FOR INITIALIZATION
Since high quality data is not instantly available, to initialize the neural net weight, we ﬁrst generated a dataset containing 10 6 state-action pairs by computer player MoHex played against MoHex, on 9 × 9 board.
We train the policy neural net to maximize the log-likelihood on this dataset.
The training only took 100 , 000 steps with mini-batch 64, optimized using Adam (Kingma & Ba, 2015) with learning rate 0 . 001 .
The resulting neural net could be used for multiple board sizes, its win-percentages against 1-play
Wolve on 9 × 9 and regular board size 11 × 11 are respectively 13 .
2% and 4 .
6% (iterated all opening moves, 10 trails each opening with Wolve as Black or White).
Input feature has 12 binary planes, utilizing a “bridge pattern” in Hex.
B. M O H EX 2.0’ S IN - TREE PHASE
Starting from the root node, MoHex 2.0 selects a child node according to the score function in below:
[Equation]
where N ( s ) , N ( s, a ) are respectively the visit counts of s and ( s, a ) , q ( s, a ) is the Q-value of ( s, a ) ,
R ( s, a ) is the RAVE value, ρ ( s, a ) is the prior probability from move pattern weights, w, c b ,
c pb are weighting parameters.
MoHex-CNN uses the same formula except that the prior probability is from neural net.
do
Fix π t 1 ;
Compute the optimal counter strategy for player 2 given player 1’s stragy is ﬁxed;
Let π t 2 be the computed optimal counter strategy;
Fix π
Compute the optimal counter strategy for player 1 given player 2’s strateg is ﬁxed;
Let π t + be the computed optimal counter strategy; then
; 17 Procedure Algo
Suppose player 1 is the max player, palyer 2 is the min player;
; stable false ; while not stable do
Fix
Compute the value
V t under π t 1 ,
2 by policy evaluation;
Let π t
be the greedily (max) modiﬁed policy w.r.t V t ;
Let π t
be the greedily (min) modiﬁed policy w.r.t V t ; then stable ←
; 31 Procedure Algo
Suppose player 1 is the max player, palyer 2 is the min player;
π
π
t ← 0 ;
stable ← false ; while not stable do
Fix π t 1 , π t 2 ;
Compute the value function V t under π t 1 , π t 2 by policy evaluation;
Let π t + be the greedily (max) modiﬁed policy w.r.t V t ;
Fix π t +11 ;
Compute the optimal counter policy for player 2 given player 1’s policy is ﬁxed;
Let π t + be this optimal counter policy;
t ← t + 1 ;
if π t 1 = π t − 1and π t 2 = π t − 1then
stable ← true ;
[Table]
W ITH DFPN SOLVER TURNED ON
In computer Hex Olympiad tournament
[Table]
MoHex is usually played with the parallel solver turned on.
To demonstrate the full strength of MoHex-CNN compared to MoHex 2.0, we perform another experiments by limiting both programs with the same thinking time 5s per move, all with the DFPN solver turned on.
[Table]
As expected
[Table]
the results show there on smaller board size like 9 × 9 , due to the DFPN solver in both players, there is a signiﬁcant decrease of MoHex-CNN’s winrate.
However, on
[Table]
[Table] [Table]