Safe Opponent-Exploitation Subgame Refinement
Anonymous Author(s)
AffiliationAddress email
Abstract
In zero-sum games, an NE strategy tends to be overly conservative confrontedwith opponents of limited rationality, because it does not actively exploit theirweaknesses.
From another perspective, best responding to an estimated opponentmodel is vulnerable to estimation errors and lacks safety guarantees.
Inspiredby the recent success of real-time search algorithms in developing superhuman
AI, we investigate the dilemma of safety and opponent exploitation and present anovel real-time search framework, called Safe Exploitation Search (SES), whichsmoothly interpolates between the two extremes of online strategy refinement.
Weprovide SES with a theoretically upper-bounded exploitability and a lower-boundedevaluation performance.
Additionally, SES enables computationally efficient onlineadaptation to a possibly updating opponent model, while previous safe exploitationmethods have to recompute for the whole game.
Empirical results show that SESsignificantly outperforms NE baselines and previous algorithms while keepingexploitability low at the same time
Introduction
Behind the recent breakthroughs of superhuman AIs in Go [Silver et al., 2016, 2017, Schrittwieseret al., 2020], heads-up no-limit Texas hold’em (HUNL)
[Brown et al., 2018, Moravcík et al., 2017,
Brown and Sandholm, 2019, Brown et al., 2020], and Hanabi [Lerer et al., 2020], search plays a vitalrole.
In perfect information games, Monte Carlo tree search (MCTS) is widely applied to improvepolicy’s strength.
In zero-sum imperfect information games such as poker, search algorithms are usedto find a Nash equilibrium (NE) approximation in subgames encountered in real time [Brown and
Sandholm, 2017, Burch et al., 2014].
They are both theoretically sounded and empirically powerful.
In zero-sum games, NE-based search algorithms [Burch et al., 2014, Moravcik et al., 2016, Brownand Sandholm, 2017, Brown et al., 2018] find safe strategies with low exploitability and producestrong baselines against all opponents [Brown and Sandholm, 2019].
However, it may be overlyconservative confronted with opponents with limited rationality, and fail to take advantage of theirweaknesses to obtain higher rewards [McCracken and Bowling, 2004, Johanson et al., 2007, Liand Miikkulainen, 2018].
From another perspective, there have been extensive studies on opponentexploitation to address the problem.
Some typical works [Carmel and Markovitch, 1996, Billingset al., 2003, Gilpin and Sandholm, 2006, Li and Miikkulainen, 2018] model the opponent’s strategybased on previous observations and then search for a new strategy to exploit this model.
However,these methods often neglect the significance of the strategy safety, thus being highly exploitable.
Few exceptions including Johanson et al.
[2007] and Ganzfried and Sandholm [2015a] aim to searchfor safe and robust counter-strategies.
Ganzfried and Sandholm [2015a] provides a characterization ofsafe deviations from NE in repeated games.
Restricted Nash response (RNR) [Johanson et al., 2007]finds a Pareto optimal strategy with respect to safety and exploitation in the full game.
However, it is computationally inefficient because it needs to recompute a strategy for the whole game whenever the opponent model is updated.
This can be even infeasible in an online setting where the opponentmodel is being updated continuously with streamed data.
In this paper, we study the dilemma of safety and opponent exploitation and present a new scalablereal-time search framework Safe Exploitation Search (SES) that smoothly interpolates between thetwo extremes of strategy search, hence unifying safe search and opponent exploitation.
It enablescomputationally efficient online adaptations to a continuously changing opponent model, which ishard to address by previous safe exploitation algorithms.
The safety criterion requires the refinedstrategy to stay close to NE, formally speaking, to expose limited exploitability against any opponents,while the opponent exploitation criterion requires the strategy to adapt to its specific opponent and toexploit its weaknesses.
We propose a novel maximization objective in the subgame search frameworkwhich combines the safety objective and exploitation, controlled by the exploitation level α .
Weconstruct a new gadget game to optimize this objective, which enables our method’s scalability tolarge games such as Texas Hold’em.
Theoretically, we prove that SES is guaranteed to outperform
NE at the cost of some constant increase in its own exploitability confronted with non-NE opponents.
Empirically, we evaluate the effectiveness of our search algorithm in 1 didactic matrix game 2 pokergames: Leduc Hold’em [Southey et al., 2005] and Flop Hold’em Poker (FHP)
[Brown et al., 2019].
The experiment results demonstrate that our algorithm significantly outperforms NE baselines againstnon-NE opponents and keeps low exploitability at the same time.
Additionally, we show that SES isnot only much more computationally efficient than previous safe exploitation methods but also morerobust to estimation errors in opponent models.
Related work
This paper investigates the problem of safe opponent exploitation in two-player zero-sum imperfectinformation games.
We propose a novel search algorithm that balances between NE and exploitingopponents.
Two major relevant research areas are search algorithms in imperfect information gamesand opponent exploitation.
Search in imperfect information games.
In recent literature, search techniques are witnessed to beimportant in developing strong AI strategies in both perfect and imperfect information games [Burchet al., 2014, Moravcik et al., 2016, Brown and Sandholm, 2017].
Texas hold ’em poker is widelyemployed as a benchmark for imperfect information games.
A primary part of the long-term researchon Texas hold’em poker is the evolution of subgame solving algorithms, which aim at achieving amore accurate Nash equilibrium approximation in the subgame encountered given a pre-computedstrategy for the full game which we refer to as the blueprint strategy.
Unsafe search [Billings et al.
,2003, Ganzfried and Sandholm, 2015b, Gilpin and Sandholm, 2006, 2007] estimates the subgamereach probability assuming the opponent follows blueprint, and searches for a refined subgamestrategy.
Subgame resolving [Burch et al., 2014] and maxmargin search [Moravcik et al., 2016]are theoretically sounded safe search algorithms which ensure that the subgame strategy obtainedis no worse than the blueprint.
They search in a gadget game and achieve safety by providing theopponent with the option not entering the current subgame.
DeepStack [Moravcík et al.
, 2017]and Libratus [Brown et al., 2018] build strong poker AIs with the aid of search.
Beyond poker,search algorithms for subgame refinement have also shown promise in improving joint strategies incooperative imperfect information games such as Hanabi [Lerer et al., 2020] and the bidding phase ofcontract bridge [Tian et al., 2020].
The purpose of our search algorithm is different from previousmethods in poker literature.
We seek to exploit opponents while keeping exploitability low, ratherthan simply approximating NE.
Opponent exploitation.
Most previous opponent exploitation researches [Carmel and Markovitch,1996, Billings et al., 2003, Gilpin and Sandholm, 2006, Li and Miikkulainen, 2018] typically modelthe opponent’s strategy based on previous observations and then search for a new strategy to exploitthis model, but put little emphasis on safety.
One similar work is Johanson et al.
[2007] which proposes p -restricted Nash response (RNR) to finda safe exploitation strategy to the estimated opponent’s strategy.
It calculates a Nash equilibriumfor the whole game restricting that the opponent plays the estimated strategy σ fix with probability p ,and any strategy with probability 1 − p .
In that paper, Johanson et al.
[2007] prove that a p -RNRto σ fix is Pareto optimal with respect to exploitation and safety.
However, it does not provide anexplicit bound.
Additionally, whenever the estimated opponent model changes or we want to use a different p to balance between safety and exploitation, the original p -RNR has to recompute thestrategy for the whole game.
It is computationally inefficient in an online setting, where the opponentmodel is updated after every round with new game data.
Our algorithm instead takes modeling errorinto account and provides explicit bounds for both safety and exploitation.
With the aid of real-timesearch, it only searches for strategies in subgames encountered instead of the whole game.
Ganzfried and Sandholm [2015a] study safe exploitation strategies in repeated games, which is adifferent setting from this paper.
Intuitively, it achieves safety by risking in exploitability at mostwhat it has earned over NE in expectation in previous rounds.
Therefore, its expected value in thewhole repeated game is never worse than the NE.
In contrast, this paper focuses on the safety ofstage game strategies.
Furthermore, our algorithm is complementary to Ganzfried and Sandholm[2015a].
Ganzfried and Sandholm [2015a] calculate an ε -safe best response for the whole game ateach iteration with LP.
This procedure is one of the main limitations on the algorithm’s scalability.
Our algorithm can be a possible efficient substitute for the calculation.
Bernasconi-de Luca et al.
[2021] use a UCB-like algorithm to learn how to encourage opponent’sengagement in repeated games setting by guaranteeing that the opponent’s utility lies in the desiredrange.
Moravcík et al.
[2017] use a similar mixing distribution technique as ours to speed-up resolvingprocedure for NE by fixing the distribution of infosets on top of the subgame with a prediction ofsome unknown NE strategy.
This work does not study opponent exploitation.
Besides, it does notprovide a safety or an exploitation bound for the algorithm.
To our knowledge, we are the first paper to investigate the safe opponent exploitation problemin subgame resolving schemes.
Subgame resolving enables online adaptations to a continuouslychanging opponent model, eliminating the need to recompute a whole game strategy.
It offerscomputational benefits in practical opponent exploitation circumstances.
Our experiments in section5 demonstrate its efficiency and robustness.
There is extensive research [Albrecht and Stone, 2018] on agent modeling.
However, this paperonly focuses on the safe exploitation algorithm, but not the agent modeling techniques.
We can useoff-the-shelf agent modeling algorithms to estimate the opponent’s strategies.
Notations and background
An extensive-form imperfect information game G =
( P, H, Z, A, χ, ρ, · , σ c , u, I ) describes sequential interactions among agents, where agents have private information.
A finite set P consists of nplayers and a chance node c which represents the stochastic nature of the environment.
The set ofnon-terminal decision nodes is denoted as H , and Z is a set of terminal nodes or leaves.
The set ofpossible actions is A , and χ : H → 2 | A | is a function which assigns to each decision node h ∈ H aset of legal actions.
A player function ρ : H → P assigns to each decision node a player p ∈ P whoacts at that node.
If action a leads from h to h ′ , we write h · a = h ′ .
If there exists a sequence ofactions leading from h to h ′ , we write h ⊑ h ′ .
At each node h ∈ H , the acting player p = ρ ( h ) picksan action from legal actions a ∈ χ ( h ) , and leads node h into its child h · a .
The chance node alwayssamples an action from its own distribution σ c , which is common knowledge to all players.
Utilityfunctions are u = ( u 1 , u 2 , . . .
, u n ) , where u
i : Z → R defines the utility of player i at terminal nodez ∈ Z .
The nature of imperfect information is characterized by infosets I =
( I 1 , I 2 , . .
, I n ) , where
I i
=
( I i, 1 , . . .
, I i,k i ) is a partition of H for player i .
Two states in the same infoset must have thesame acting player and the same legal action sets.
We use I ( h ) to denote the infoset that h belongs to.
A player p cannot distinguish between states
h 1 and h 2 if I ( h 1 ) =
I ( h 2 ) , and thus should behaveidentically on all states in the same infoset.
The strategy of a player p is σ p : I p ×
A → R , where σ p ( I, a ) is a distribution over valid actions oninfoset I .
For simplicity, we also use σ p ( h, a ) to denote σ p ( I ( h ) , a ) .
We use π σ ( h ) to denote the probability of reaching state h from the root when agents choose a strategy profile σ = ⟨ σ 1 , σ 2 , . . .
, σ n ⟩ .
Formally, π σ ( h ) =
(cid:81) h ′ · a ⊑ h σ ρ ( h ′ )
( h ′ , a ) .
We use π σ − p ( h ) =
(cid:81) h ′ · a ⊑ h
∧ ρ ( h ′ ) ̸
= p σ ρ ( h ′ )
( h ′ , a ) to denote the probability of reaching h when player p always chooses the action that leads to h wheneverpossible.
π σ ( h, h ′ ) is the reaching probability of h ′ from h .
π σ ( h · a, h ′ ) is the the probability ofreaching h ′ from h if action a is taken at h .
These probabilities can be formally defined in a similarmanner.
A blueprint is a pre-computed strategy for the full game.
The expected utility of player p given strategy profile σ is u σp =
[Equation]
z ∈
Z π σ ( z ) u p ( z ) .
The counterfactual value v σp ( I, a ) is the expected utility that player p will obtain after taking action a at infoset I ,given the joint policy profile is σ .
Mathematically, it is the weighted sum of expected values at allstates h ∈
I .
[Equation]
In the rest of the paper, we focus on two-player zero-sum games with perfect recall.
Zero-sum means∀ z ∈ Z, u 1 ( z ) + u 2 ( z ) = 0 .
Perfect recall means that no player will forget the information whichhas been obtained previously in the game.
This is a common assumption in related literature.
A best response strategy BR
[Equation]
) = arg
[Equation]
for player p is the strategy that maximizehis own expected utility against fixed opponent strategy σ − p .
The exploitability of strategy σ p is
[Equation]
where σ ∗ is the optimal strategy, and is an NE in two-playerzero-sum games.
It measures the performance of σ p against its best response comparing with the
NE.
A counterfactual best response CBR p ( σ − p ) is a strategy where σ p ( I, a ) >
if and only ifv σp ( I, a ) ≥ max b v σp ( I, b ) .
Counterfactual best response is a best response, but not vice versa.
Thecounterfactual best response value CBV σ − p p
[Equation]
p is the expected utility of thecounterfactual best response policy.
Since we focus on two-player zero-sum games, we will use
CBV σ p ( I ) as a shorthand notation for CBV σ p − p ( I ) .
We follow the imperfect information subgame definition as in Burch et al.
An augmentedinfoset contains states which cannot be distinguished by the remaining players.
Definition 3.1.
An imperfect information subgame S is a forest of trees, closed under both thedescendant relation and membership within augmented infosets for any player.
Let S top be the set ofnodes which are roots of each tree in S .
Method
In this section, we introduce our novel search algorithm called safe exploitation search (SES), whichexploits the weaknesses of the opponent while ensuring a bounded exploitability efficiently.
Let σbe the pre-computed blueprint strategy.
Without loss of generality, assume we search for player 2’srefined strategy σ S 2 by applying SES to all subgames S ∈ S .
Finally, the refined strategy for P2 aftersearch is σ ′ 2 , which is the same as σ 2 in { I i 2 |∀ S ∈ S , I i 2 / ∈ S } and is replaced with σ S 2 in S ∈ S .
4.1 Safe Exploitation Search
Our algorithm offers a unified approach to balance these two demands with theoretical guarantees.
The objective of our search algorithm is to find a new subgame strategy σ S 2 for S ∈ S whichmaximizes
[Equation]
where α ∈ [0 , 1] is a hyper-parameter controlling the exploitation level, and ˆ p ( I j 1 ) is the estimatedprobability of player 1 entering infoset I j 1 ∈ S top .
Given P2’s strategy (which is the blueprint σ 2 )and P1’s actual strategy (which does not have to be the blueprint σ 1 ), the real probability of player1 entering infoset I j 1 ∈ S top (which we denote as p ( I j 1 ) ) is determined.
ˆ p ( I j 1 ) is an estimation ofp ( I j 1 ) .
For instance, in poker, it is the estimated distribution of private cards player 1 holds.
Boththeoretically and empirically, such estimation does not have to be fully accurate.
It can be done withoff-the-shelf opponent modeling techniques, which lies beyond the focus of this paper.
Intuitively, the maximization objective achieves a balance between opponent exploitation andsafety , controlled by exploitation level α .
The first part of the objective is maximized when
ˆ p ( I
2 1 ( I j 1 ) is minimized.
It aims at finding a strategy σ S 2 which results in thelowest value for P1 under the assumption that the reach probabilities is ˆ p .
It can be interpreted as exploiting the estimated P1’s strategy.
The second part of the objective demands the resolvedstrategy to behave well against any reach probability distribution.
We use the subgame margin
[Equation]
[Moravcik et al., 2016] which can be regarded as the worstcase utility increase for P2.
Our search objective is a convex combination of exploitation and safety, which is closely related toprevious safe exploitation research [McCracken and Bowling, 2004, Johanson et al., 2007].
RNR[Johanson et al., 2007] calculates an exploitation strategy by computing an NE in the full game,restricting the opponent to play its fixed strategy with probability p and any other strategy with 1 − p .
RNR is proved to be Pareto optimal with respect to safety and exploitation.
However, it neitherprovides an explicit bound on exploitability and performance nor takes modeling errors into account.
Furthermore, without search, RNR has to recompute for the whole game whenever the opponent’sstrategy changes, which limits its efficiency.
Experiment section 5.3 demonstrates that SES is muchmore computationally efficient, and section 5.4 shows that, even we augment RNR with searchframework, SES is still much more robust to estimation errors in opponent strategy.
By maximizing the objective 2, we provide sound theoretical results for both safety and opponentexploitation.
Additionally, we provide analyses of how (1) exploitation level α , (2) accuracy ofopponent modeling, and (3) strength of the blueprint strategy impact the theoretical bound.
Bygradually increasing α from 0 to 1, our algorithm tends to exploit rather than keep safe.
Theorem 4.1.
(safety) Let S be a disjoint set of subgames S .
Let σ ∗ = ⟨ σ ∗ 1 , σ ∗ 2 ⟩ be the NE where P2’sstrategy is constrained to be the same with σ 2 outside S .
Define ∆ = max S ∈ S , I
[Equation]
|
[Equation]
( I [Equation] 1 ) −v σ 1 ( I i 1 ) | .
Let ˜ p
( I i 1 ) be the reach probability given by σ ∗ 1 .
Let ˆ p
( I i 1 ) be the estimation of reachprobability p ( I i 1 ) given by the real opponent strategy.
Define τ = max S ∈ S ,I i 1 ∈ S top | ˆ p ( I i 1 ) − ˜ p
( I i 1 ) ˜
p ( I i 1 )
| .
Whenever 1 − (2 τ + 1) α > 0 , we have a bounded exploitability given by:
[Equation]
Recall that σ ′ 2 is the refined strategy after search.
The proof is provided in Appendix B. Thistheorem implies that the exploitability of the new strategy is smaller than that of strategy σ ∗ 2 plus aconstant value, which is the closest strategy to NE if constrained to differ from σ 2 only in S .
Thecorresponding theoretical result of maxmargin search [Moravcik et al., 2016], a safe search algorithmwith no opponent exploitation abilities, is exp( σ ′ 2 ) ≤ exp( σ ∗ 2 )+2∆ .
Comparing these two results, wecan interpret the term 2 / (1 − (2 τ + 1) α ) as the additional risk introduced by exploiting the opponent.
If exploitation level α = 0 , then our bound is as tight as that of maxmargin search [Moravcik et al.
The bound also gets tighter if the τ gets smaller, or the blueprint σ 2 is closer to σ ∗ 2 .
Theorem 4.2.
(opponent exploitation) Let ϵ =
∥
ˆ p − p ∥ 1 be the L1 distance of the distributionp ( I i 1 ) and ˆ p ( I i 1 ) .
Let η = min S ∈ S max I j 1 ∈ S top (cid:16) CBV 1 ( I j 1 , σ S 2 ) − CBV 1 ( I j 1 , σ ∗ 2 ) (cid:17) ≥ 0 .
We use
BR [ S ,σ p ] p ( σ ) to denote the strategy for player p which maximizes its utility in subgame S ∈ S againstσ − p under the constraint that BR [ S ,σ p ] p ( σ ) and σ p differs only inside S .
By maximizing objective 2,for all S ∈ S , the refined strategy σ ′ 2 satisfies
[Equation]
The proof is provided in Appendix B. Observe that the reach probability p is characterized by P1’sstrategy outside S and ˆ p is its estimation.
Because the search algorithm always find a strongerresponse strategy for P1 in S (which is exactly BR [ S ,σ 1 ]( σ ′ 2 ) ) as well, opponent exploitation refersto adapting to P1’s strategy σ 1 outside S .
This theorem implies that the utility of the new strategyσ ′ 2 is lower bounded by the utility of σ ∗ 2 when both confronted with P1’s unknown strategy outside 226 S .
It provides theoretical guarantees for the opponent exploitation ability of our algorithm.
ϵ canbe interpreted as estimation error.
The lower bound increases if the estimation error get smaller orthe blueprint σ 2 is closer to σ ∗ 2 .
We show empirically how exploitation level α and estimation errorimpact both safety and exploitation abilities in section 5.
[Figure]
In order to find σ S 2 which maximize objective 2, a straight-forward method is to reformulate themaximization problem as a Linear Programming problem [Moravcik et al., 2016].
However, LPsolvers [Koller et al., 1994] cannot handle large-scale problems.
Alternatively, inspired by Moravciket al.
[2016], we create a gadget game and then apply iteration-based NE algorithms such as CFR[Zinkevich et al., 2007, Tammelin et al., 2015, Lanctot et al., 2009] in the gadget game.
The gadgetgame is carefully designed such that the NE solution found in it is exactly the solution to the originaloptimization problem.
As shown in Figure 1, the original subgame is copied into two identical parts S 1 , S 2 in the gadgetgame.
Player 2’s infosets stretch over both branches, while player 1 can distinguish between the twoparts.
We use P1 to denote player 1, and P2 for player 2.
The procedure of constructing such gadgetgame can be summarized into 4 steps as described below:
Create a chance node at the top of the gadget game.
It goes to the left part with probability1 − α , and the right part with probability α .
The outcome is visible to P1 but not P2.
Therefore,corresponding nodes in both branches are in the same infosets for P2, and his strategy σ S 2 will be thesame for both parts.
Since σ S 1 is the best-response to σ S 2 and the two parts only differ at how to go toeach infoset of player 1, player 1 will also keep his strategy the same in both parts.2.
We subtract u 1 ( z ) by v σ 1 ( I i ) for all z ⊑ h, h ∈
I i 1 , and add u 2 ( z ) by v σ 1 ( I i 1 ) in order to keepthe subgame zero-sum.
By doing so, the objective of a Nash Equilibrium of p 2 will change frommaximizing − CBV σ S 2 1 ( I i 1 ) to maximizing v σ 1 ( I i 1 ) − CBV σ S 2 1 ( I i 1 ) .3.
As for the left part of the gadget game, the P1 node enables P1 to enter an arbitrary infoset
I i 1 .
The following chance nodes sample a specific state with probability proportional to π σ − 1 ( h ) forall h ∈
I 1 .
Since this is a zero-sum game, in an NE strategy, he will enter the one with lowestv σ 1 ( I i 1 ) − CBV 1 ( I i 1 , σ ∗ 2 ) which is exactly the minimization in the second term of SE ( σ S 2 ) .4.
The chance node in the right part will sample an infoset I i 1 according to reach probability ˆ p
( I i 1 ) .
The following chance nodes again sample a specific state with probability proportional to π σ − 1 ( h ) forall h ∈
I 1 .
So the NE objective of this part is exactly the summation in the first term of SE ( σ S 2 ) .
The pseudocode of SES is shown in Appendix A. We also provide a didactic matrix game in Appendix
D as an example to show the necessity of considering safety and expected payoff simultaneously, andto demonstrate the superiority of SES over a simple mixing strategy.
Experiment
Our experiment is done in Leduc Hold’em
[Southey et al., 2005] and
Flop Hold’em Poker (FHP)[Brown et al., 2019].
Leduc Hold’em is a smaller-scale poker game and FHP is a larger one.
The rulesof these two pokers are provided in Appendix C.
We demonstrate the exploitability and evaluationperformance of SES against opponents of various strengths.
The exploitability measures a searchalgorithm’s safety, while head-to-head evaluation measures the ability of opponent exploitation.
We also illustrate how estimation accuracy of opponent’s strategy and the exploitation level α impact theresults.
Please refer to Appendix E for implementation details.
5.1 Opponents
In our experiments, we test the performance of our algorithm against opponents of various strengths.
We create 3 types of opponents with 3 random seeds each.
The first one is an approximation of NE,and is regarded as a strong opponent.
For the second and third type of opponents, we enumerateevery infoset in the blueprint strategy and shift the action distribution randomly with probability
Pr shuffle = 0 .
3 or 0 .
We multiply the probability of each action by a random variable from
Uniform (0 , 1) , and then re-normalize the probability distribution.
The procedure is motivated by
Brown et al.
[2018], in which such method is applied to create a number of diverse but reasonablystrong agents.
Even when Pr shuffle = 0 .
7 , the strategy keeps close to NE with average L1 distance ofeach infoset 0.132 comparing to 1.036 of a random strategy to NE.
So they are regarded as opponentswho are not fully rational but with competitive strength.
[Figure]
5.2 Safe Opponent Search
In Figure [Figure] we demonstrate the head-to-head evaluation performances and corresponding exploitability of the refined strategies found by SES against opponents of various strengths, under differentexploitation level α and when ϵ = 0 . 0 , 0 . 3 , 0 . 6 , 0 . 9 , 1 . 2 .
Please refer to Appendix E for details of generating opponentestimations.
The blue line is the result of blueprint strategy without conducting any search.
Generally speaking, SES balances between safety and opponent exploitation.
The increase ofexploitation level α helps win more chips from opponents, while resulting in the increase of thestrategy’s own exploitability.
As can be seen in Figure 2, the exploitability increases when theexploitation level α grows from 0 to 1 , which is consistent with Theorem 4.1.
One exception is in
FHP when ϵ is small: the exploitability surprisingly keeps decreasing even if SES puts more emphasison opponent exploitation.
Similar situations have also occurred in previous literature [Brown and
Sandholm, 2017].
The reason is that our opponent is quite close to NE outside the subgame whichwill make ˆ p close to ˜ p when ϵ is small, which means the τ in Theorem 4.1 is small.
As a result,we will have a low-exploitability resolved strategy when using unsafe search and the exploitabilityincreases as ϵ increases.
When the estimation is completely correct ( ϵ = 0 . 0 , the yellow line), the expected payoff in FHP(3rd row in Figure 2) increases as the exploitation level α grows higher.
In Leduc poker, since thegame is very small, the pre-computed blueprint is very close to NE.
Therefore, when confrontedwith relatively strong opponents ( Pr shuffle = 0 . 0 , 0 . 3 ) which are also close to NE, actually few thingscan be done other than sticking with the blueprint.
So the improvement introduced by SES is small.
When facing relatively weak opponent ( Pr shuffle = 0 . 7 ), the improvement margin is slightly larger.
SES relies on an estimation of opponent’s strategy.
In order to test the robustness of our algorithmwhen the prediction of p ( I i 1 ) is not accurate, we evaluate the performance of our algorithm withdifferent values of estimation error ϵ .
As illustrated in Figure 2, the exploitability increases and theexpected payoff drops when ϵ grows larger.
The result is expected since an accurate estimation alwaysprovides benefits.
However, it also demonstrates that SES can still achieve a trade-off between safetyand opponent exploitation even when ϵ is considerably high.
For instance, in FHP, ϵ is between 0 and2, and ϵ = 1 .
2 means that the predicted distribution is almost random.
When ϵ ≤ 0 . 6 , the expectedpayoff still keeps increasing with respect to α .
In case of a bad estimation, we can always choosesmaller α to ensure safety.
5.3 Comparison with Restricted Nash Response
[Figure]
row represents a type of opponent with Pr shuffle = 0 . 0 , 0 .
The
-axis is the parameter α for and p for
We also compare SES with restricted Nash response (RNR)
[Johanson et al., 2007], a previous safeexploitation algorithm, in FHP.
RNR calculates an NE for for the whole game restricting that theopponent plays the estimated strategy σ fix with probability p , and any strategy with probability 1 − p .
In each round, we limit the computation time of RNR(normal) to 10 CPU second 1 , which is the samefor SES.
However, as stated in section 4.1, RNR needs to recompute a strategy for the whole game ineach round.
It cannot converge in 10s.
So we also compare with RNR(big), which has a budget of10M CFR iterations in each round (around 190 CPU second in time).
In contrast, SES only uses 10M error is shuffle
[Table]
CFR iterations to calculate its blueprint once.
As is shown in Figure 3, SES significantly outperforms
RNR(normal) in both exploitability and evaluation.
SES also achieves much lower exploitability than
RNR(big)
and comparable evaluation results with much less computation time.
5.4 Comparison with EXP-STRATEGY
It is possible to augment p − RNR with real-time search.
For a subgame S , we can create a similargadget game to SES in Figure 1.
The difference is that it keeps the opponent strategy fixed to itsestimation in the whole exploitation part, while SES only uses the reach probability ˆ p calculated fromthe estimated opponent strategy and allows the search algorithm to find opponent strategies in theexploitation part as well.
To our best knowledge, this algorithm does not exist in previous literature,and can be regarded as an ablation study for SES.
We call it EXP-STRATEGY.
Table 1 shows the exploitability and evaluation performance of SES and EXP-STRATEGY underdifferent exploitation level α (or p in RNR).
We add errors in the estimation of opponent strategy,and use the same estimation for both algorithms.
SES performs much better than EXP-STRATEGY:maintains lower exploitability and achieves higher evaluation performances
The experiment showsthat EXP-STRATEGY is sensitive to modeling errors, because it relies on the estimated strategy inthe whole subgame.
SES, which exploits a distribution of infoset instead of the full opponent strategy,is more robust.
Please refer to Appendix F for full comparison results.
Similar to SES, we derive the theoretical bounds for both exploitability and exploitation for EXPSTRATEGY.
And the theory also demonstrates that EXP-STRATEGY is more sensitive to theaccuracy of the estimation.
Please refer to Appendix G for details.
Conclusion
We propose a novel safe exploitation search (SES) algorithm which unifies both safe search andopponent exploitation.
With the aid of real-time search, SES can make online adaptations to achanging opponent model.
We also prove safety and opponent exploitation guarantees of SES in
Theorem 4.1 and Theorem 4.2.
The experimental results in our designed matrix game confirm theexistence of the refined strategy which is both safe and actively exploiting the opponent.
In gamesof poker, our method outperforms NE baselines while keeping exploitability low.
SES is also muchmore efficient than previous safe exploitation algorithms without search.
Additionally, SES is morerobust to opponent modeling errors.
One limitation of SES is that it relies on an estimation of opponent.
Although the assumption iscommon in opponent exploitation literature, efficient opponent modeling is still an active researcharea.
Additionally, the exploitation level α is now regarded as a hyperparameter in SES.
We find thatα should be tuned for each specific purpose and automatically learning α is not trivial (see Appendix
H for a brief discussion).
We leave this for future work.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] The code and licence of the code would be released upon the paper acceptance.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[No] To our
knowledge, our work does not pose negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] All proofs are
included in the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See section 5.1 and
Appendix E.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
b) Did you mention the license of the assets?
[N/A]
c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you
are using/curating contains personally identifiableinformation or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation
N/ A]
A
Algorithm
The pseudocode of SES in Poker is shown in Algorithm
Algorithm 1 Procedure of in Poker
Initialize the game environment
Initialize the private cards distribution p of player 2 as uniform distribution
Initialize the prediction of opponent private cards distribution
Compute a blueprint strategy ⟨ σ 1 , σ 2 ⟩ while Game Not Terminate do
P ← current player if P = P 1 then ▷ Opponent Player
Observe opponent action a
Update the prediction of opponent’s private cards based on a else if P = P 2 then ▷ Resolving Player if Resolve Current Node then
Compute v σ 1 ( I i 1 )
Construct Gadget Game σ S 2 ← CFR ( Gadget Game ) else σ S 2 ← σend if
Sample action a based on σ Sp ( I i 2 ) ← p ( I i 2 ) · σ S 2 ( I i 2 , a )
Normalize p else ▷ Chance Node
Sample public cards C
= { c 1 , c 2 , ..., c k } from current chance node ∀
i = 1 , 2 , ..., k, p c i ←
Normalize p end if end while
B Proofs
B.1 Proof of Theorem 4.
Theorem 4.1 (safety) Let S be a disjoint set of subgames S .
Let σ ∗ = ⟨ σ ∗ 1 , σ ∗ 2 ⟩ be the NE where P2’sstrategy is constrained to be the same with σ 2 outside S .
Define ∆ = max S ∈ S ,I i 1 ∈ S top | CBV σ ∗ 2 1 ( I i 1 ) −v σ 1 ( I i 1 ) | .
Let ˜ p
( I i 1 ) be the reach probability given by σ ∗ 1 .
Let ˆ p
( I i 1 ) be the estimation of reachprobability p ( I i 1 ) given by the real opponent strategy.
Define τ = max S ∈ S ,I i 1 ∈ S top | ˆ p ( I i 1 ) − ˜ p
( I i 1 ) ˜
p ( I i 1 )
| .
Whenever 1 − (2 τ + 1) α > 0 , we have a bounded exploitability given by:
[Equation]
Proof:
For simplicity, we will omit the subscript of CBV σ ∗ 2by default.
In order to prove Theorem 1, wewill use mathematical induction on the level of the infoset.
The depth L has the same definition as in
Brown and Sandholm [2017], i.e.
• For all the infosets which are direct parents of the subgames, we define L ( I ) = 0 .
• For the infosets that are not ancestors of the subgames, we define L ( I ) = 0 .
• For any infosets that are ancestors of the subgames, we define
[Equation]
I
[Equation]
That is, it has a higher level than any
of its successors
Base Case of Induction
Firstly, we will prove that for any infoset with level 0, the inequality of theorem 1 holds.
Forconvenience, we consider that theorem 1 in a specific subgame S .
We will prove the infoset at the top of the subgame first.
Since SE ( σ ∗ 2 ) ≥ (1 − α )( − ∆)
+α (cid:80) i ˆ p
( I i 1 )( − ∆)
= − ∆ , we have
[Equation]
since σ S 2 = arg max ˜ σ 2 SE (˜ σ 2 ) .
Furthermore, we have(cid:88)
[Equation]
where the second term (cid:80) i ˜ p
( I i 1 )( CBV
[Equation]
2 ( I i 1 ) − CBV
[Equation]
S 2 ( I i 1 )) is no larger than 0 because 514 (cid:80) i ˜ p
( I 1 ) CBV σ ∗ 2 ( I i 1 ) is exactly what σ ∗ 2 minimized.
Otherwise, σ ∗ 2 can change the strategy inthe subgame so that he will get higher reward against σ ∗ 1 which conflicts the definition of NE.
And we will further decompose I i 1 ∈ S top into two parts, { I i, − 1 } and { I i, + 1 } .
They have the propertythat CBV σ ∗ 2 ( I i, − 1 ) − CBV
[Equation]
S 2 ( I i, − 1 ) ≤ 0 and CBV σ ∗ 2 ( I i, + 1 ) − CBV σ S 2 ( I i, + 1 ) > 0 .
And since(cid:80) i
˜ p
( I i [Equation] ( CBV
[Equation]
2 ( I i 1 ) − CBV
[Equation]
S 2 ( I i 1 )) ≤ 0 as discussed above, we have(cid:88)
[Equation]
[Equation]
8) which implies that(cid:88)
[Equation]
Then we have
[Equation]
The last inequation holds since min I j 1 (cid:16) [Equation] 1 ) − [Equation]
[Equation]
I
[Equation]
1 ) (cid:17) ≤ 0 since σ ∗ 2 is the strategywith lowest exploitability by only changing strategy of σ 2 in the subgames.
Back to Equation 7, we have
[Equation]
And substitute it into Equation 6,
[Equation]
so that
[Equation]
for all I j 1 in the subgame.
And for infoset I out of the subgame with level 0, since the refined strategy σ S 2 and blueprint strategyσ 2 are the same here, the CBV value is exactly the same and the inequality holds.
Inductive Step
The inductive step mostly follows that of Brown and Sandholm [2017].
Since CBV σ S 2 ( I 1 ) ≤ CBV σ ∗ 2 ( I 1 ) + 2 1 − (2 τ +1)
α ∆ holds for every subgame S , σ ′ 2 will also satisfythis inequation since ∆ and τ are defined as maximum over all subgames.
Now, suppose CBV σ ′ 2 ( I 1 ) ≤ CBV σ ∗ 2 ( I 1 ) + 2 1 − (2 τ +1)
α ∆ holds for any infoset with level lower orequal to k , we will prove that it also holds for infoset with level k + 1 .
By definition of CBV ( I 1 ) ,
[Equation]
We can swap the two summations above since the game is perfect recall
[Equation]
then
[Equation]
By substituting the definition of CBV ( I ′ 1 ) into the equation above,
[Equation]
And by the induction hypothesis,
[Equation]
Because I 1 is out of the subgame and σ ∗ 2 , σ 2 is exactly the same outside the subgame, we will get
[Equation]
Finally, by mathematical induction we get
[Equation] [Equation]
B.2 Proof of Theorem
Theorem 4.2 (opponent exploitation) Let ϵ =
∥
ˆ p − p ∥ 1 be the L1 distance of the distribution p
( I i 1 ) andˆ p ( I i 1 ) .
Let η = min S ∈ S max I j 1 ∈ S top (cid:16) CBV 1 ( I j 1 , σ S 2 ) − CBV 1 ( I j 1 , σ
[Equation]
2 ) (cid:17) ≥ 0 .
We use BR [ S ,σ p ] p ( σ )to denote the strategy for player p which maximizes its utility in subgame S ∈ S against σ − p underthe constraint that BR [ S ,σ p ] p ( σ ) and σ p differs only inside S .
By maximizing objective 2, for all S ∈ S ,the refined strategy σ ′ 2 satisfies
[Equation]
Proof: Still, we only consider a specific subgame S first.σ S 2 is maximizing
[Equation]
So, we have
[Equation]
) and
[Equation]
Therefore,
[Equation]
which means
[Equation]
⇔
[Equation]
Since η is defined as minimum over all subgames, we have
[Equation] [Equation]
C Poker RulesRules of Leduc Poker
Leduc Poker is a two players game.
In Leduc Poker, there are 6 cards in total, three ranks( { J, Q, K } )with two suits( { a, b } ) each.
And at the beginning, every player should put 2 chip into the pot andthen will be dealt with one private card.
Then, two players alternatively bet.
They can call, raise andfold.
If any of them fold, the game ends and all chips in the pot belongs to the other player.
Andwhen a player call, he has to put chips in the pot to ensure that he contributes equal chips as the otherplayer in the pot.
When a player raise, he has to ensure that he contributes more chips than the otherplayer in the pot.
A betting round ends when a player calls.
Leduc Poker is divided into two betting rounds.
In the first round, a private card is dealt to eachplayer and then two player start to bet.
After the first betting round ends and nobody folds, a publiccard is dealt on board and the second betting round starts.
When the second round ends, both of theplayer show their private hands and the stronger hands win.
If a player’s private card has the samerank as the public card, then he wins.
Otherwise, we have J < Q < K and the higher one wins.
And in each betting round, there will be at most two raises in our experiment and each raise shouldcontribute 2 more chip in the first round and 4 more chips in the second round.
[Table]
Rules of Flop Hold’em Poker
The rules of Flop Hold’em Poker is similar to that of Leduc Poker.
In FHP, we use the standard52-card deck.
At the beginning, the first player will contribute 1 chip to the pot and the second playerwill contribute 2 chips.
And then they will be dealt with 2 private cards each and the first player startto bet.
There are still two betting rounds and the raise sizes are both 2 chips.
At the end of the firstbetting round, there will be 3 public cards dealt on board.
And the players will show their privatecard at the end of the second betting round and the larger one wins the game.
In FHP, we have thesame rule of card order as a standard Texas hold ’em .
D Matrix Game
In this part, we offer a matrix game as an example to show the necessity of considering safety andexpected payoff simultaneously, and to demonstrate the superiority of SES over a simple mixingstrategy, which follows a best response to the estimated opponent model with probability α andfollows the blueprint with probability 1 − α .
In the matrix game shown in Table 2, let’s consider two specific NEs.
In both NEs, P1 will play L/Mwith 0 .
5 probability each.
P2 will play U/O with 0 .
5 probability in the first NE and O/D with 0 .
5probability in the second NE.
Suppose the blueprint strategy is the first NE.
Consider the case when
P1 plays a rather weak strategy that he will only play R. We apply SES to search for P2’s refinedstrategy.
When the estimation of opponent strategy is accurate such that ˆ p = p , the best response of P2 isalways playing F, which is highly exploitable, while SES finds the second NE under proper α .
Togive more details, the exploitability and expected payoff of the strategy found by SES and the mixingstrategy are shown in Figure 4.
We can see that SES achieves both lower exploitability and betterperformance than the mixing strategy at almost all α values.
Theoretically, for a mixing strategy mix p ( σ 1 , σ ′ 1 ) which plays σ 1 with probability p , and σ ′ 1 with1 − p
[Johanson et al., 2007], we have the following propositions.
Proposition D.1.
Utility of a mixing strategy: u (mix p ( σ 1 , σ ′ 1 ) , σ 2 ) =
pu ( σ 1 , σ 2 )+(1 − p ) u ( σ ′ 1 , σ 2 ) .
Proposition D.2.
Exploitability of a mixing strategy: exp(mix p ( σ 1 , σ ′ 1 ))
≤ p exp( σ 1 )
+ (1 −p )
exp
σ ′ 1 ) .
The example in Figure 4 shows that the bound is tight.
Compared with the corresponding theorems 4.1 and 4.2 of SES, we find that the simple mixingstrategy cannot provide a safety guarantee and is inferior to SES.
In some circumstances, exp( σ unsafe )can be quite large.
Therefore, the exploitability of the mixing strategy can also be large even for arather small p .
In contrast, as shown in Theorem 1, SES’s exploitability does not rely on exp( σ unsafe ) ,which makes our bound for SES tighter than that of mixed strategy in most cases.
E Implementation Details
Leduc Poker.
In Leduc Poker, we solve for a blueprint strategy using a variant of CFR algorithm[Lanctot et al., 2009, Tammelin et al., 2015] with 1M iterations in the full game.
Then we applysearch in subgames when the board card is dealt.
c h i p s / h
Evaluation
SESMixing 0.
1.0     c h i p s / h
Exploitability
SESMixing
Flop Hold’em Poker (FHP).
As for FHP, there are 1,286,792 different infosets for each bettingsequence.
We cluster them into 200 infosets by an abstraction algorithm
[Johanson et al., 2013] inorder to make equilibrium finding feasible.
Then, we compute a blueprint strategy in this abstractionwith 10,000,000 iterations.
We apply search immediately once the flop cards are dealt.
Generate estimated opponent strategy.
Given p and ϵ , we use the following procedure to generatethe corresponding ˆ p such that | p − ˆ p | = ϵ . 1.
In the beginning, we set ˆ p = p .
We randomly choose asubset S + from index set S of ˆ p and we set S − =
S − S
+ .
We ensure that (cid:80) i ∈ S
+ (1 .
− ˆ p ( i ))
> ϵ/ 2and (cid:80) i ∈
S − ˆ p ( i ) > ϵ/ 2 by the following procedure: to get S + , S − , we will randomly permutethe index of ˆ p first to get permutation A .
Then, we will sample k ∼ Uniform (1 , | A | ) .
We willcheck whether S + =
{ A 1 , ..., A k } and S − =
{ A k +1 , ..., A | A | } satisfy the condition above.
We willcontinue the process until S + , S − satisfy the condition above.
We divide ϵ/ 2 into 10,000 pieceswith value ϵ 20 , 000 assigned to each piece.
For S + , we will do reject sampling to sample 10,000pieces to different indexes.
We will pick i ∈ S + randomly each time.
If ˆ p ( i ) + ϵ 20 , 000 < 1 . 0 , we willassign one piece to i and increase ˆ p ( i ) by ϵ 20 , 000 .
The process will continue until we have assignedall 10 , 000 pieces.
We will do the same thing for S − except that only when p ( i ) − ϵ 20 , 000 > 0 wewill assign one piece to i .
F Comparison with EXP-STRATEGY
Table 3, 4, and 5 show the comparison between SES and EXP-STRATEGY.
G Theoretical Bound of EXP-STRATEGY
Similar to SES, EXP-STRATEGY maximizes the following objective,
[Equation]
[Equation]
Theorem G.1.
(safety) Let S be a disjoint set of subgames S .
Let σ ∗ = ⟨ σ ∗ 1 , σ ∗ 2 ⟩ be the NE where P2’sstrategy is constrained to be the same with σ 2 outside S .
Define ∆ = max S ∈ S ,I i 1 ∈ S top | CBV σ ∗ 2 1 ( I i 1 ) −v σ 1 ( I i 1 ) | .
We use BR [ S ,σ p ] p ( σ ) to denote the strategy for player p which maximizes its utility insubgame S ∈ S against σ − p under the constraint that BR [ S ,σ p ] p ( σ ) and σ p differs only inside S .
Let ˆ p
( I i 1 ) be the estimation of reach probability p
( I i 1 ) given by the real opponent strategy. DefineΩ
= max S max i ( CBV σ ∗ 2 ( I i 1 ) + CBV
[Equation]
σ 1 ( I i 1 )) .
We have a bounded exploitability given by:
[Equation]
E XPLOITABILITY E VALUATION
[Table]
E
[Table]
Proof:
[Equation]
i
[Equation]
−
[Equation]
[Equation]
min
[Equation]
[Equation]
[Equation]
E XPLOITABILITY E VALUATION
[Table]
where the fourth line to the fifth line is derived from the fact that the fourth line is exactly the objectiveof σ S 2 subtracted a constant .
That is,
[Equation]
Follows the same induction step,
[Equation]
Theorem G.2.
(opponent exploitation) Let ϵ =
∥
ˆ p − p ∥ 1 be the L1 distance of the distribution p
( I i 1 )and ˆ p ( I i 1 ) .
Let η = min S ∈ S max I j 1 ∈ S top (cid:16) CBV 1 ( I j 1 , σ S 2 ) − CBV 1 ( I j 1 , σ ∗ 2 ) (cid:17) ≥ 0 .
By maximizingobjective 27, for all S ∈ S , the refined strategy σ ′ 2 satisfies
[Equation]
Proof:
Still, we only consider a specific subgame S first [Equation] σ S 2 is maximizing
[Equation]
So, we have
[Equation]
34) and
[Equation]
Therefore,
[Equation]
36) which means
[Equation]
1 ) V 2 ( I i 1 , ˆ σ 2 , σ ∗ 2 ) (37)
Since η is defined as minimum over all subgames and Ω is defined as maximum over all subgames,we have
[Equation]
[Equation]
H Choosing α
We find that automatically choosing α by explicitly incorporating modeling accuracy into searchformulation is non-trivial.
It still requires additional information like the risk you are willing to take.(i) Without further specifications, there is always a trade-off between safety and exploitation regardless of opponent modeling accuracy.
Since the prediction is made based on past observation, highaccuracy (confidence) cannot guarantee the success in the future (since we do not have the assumption that the opponent will not change his behavior in the next round).
Therefore, to incorporateaccuracy(confidence) into objective, we need more information like the risk you are willing to take.
Quantifying the relationship between modeling confidence and α theoretically will require moreassumptions about the opponent’s behavior than this paper does.(ii)
Empirically, exploitation and safety correspond to a fixed α and prediction accuracy differ indifferent games (FHP and Leduc) as our experiments imply.
Practically, we can use learning algorithms (e.g., bandit) to automatically select α in each round withrespect to one’s specific goal.