Original: S AFE
Revised: S AFE

Original: O PPONENT -E XPLOITATION S UBGAME
Revised: O PPONENT -E XPLOITATION S UBGAME

Original: R EFINEMENT
Revised: R EFINE MENT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Search algorithms have been playing a vital role in the success of superhuman AI in both perfect information and imperfect information games.
Revised: Search algorithms have been playing a vital role in the success of superhuman AI in both perfect information and imperfect information games.

Original: Speciﬁcally, search algorithms can generate a reﬁnement of Nash equilibrium (NE) approximation in games such as Texas hold’em with theoretical guarantees.
Revised: Speciﬁcally, search algorithms can generate a reﬁnement of Nash equilibrium (NE) approximation in games such as Texas hold’em with theoretical guarantees.

Original: However, when confronted with opponents of limited rationality, an NE strategy tends to be overly conservative, because it prefers to achieve its low exploitability rather than actively exploiting the weakness of opponents.
Revised: However, when confronted with opponents of limited rationality, an NE strategy tends to be overly conservative, because it prefers to achieve its low exploitability rather than actively exploiting the weakness of opponents.

Original: In this paper, we investigate the dilemma of safety and opponent exploitation.
Revised: In this paper, we investigate the dilemma of safety and opponent exploitation.

Original: We present a new real-time search framework that smoothly interpolates between the two extremes of strategy search, hence unifying safe search and opponent exploitation.
Revised: We present a new real-time search framework that smoothly interpolates between the two extremes of strategy search, hence unifying safe search and opponent exploitation.

Original: We provide our new strategy with a theoretically upper-bounded exploitability and lower-bounded reward against an opponent.
Revised: We provide our new strategy with a theoretically upper-bounded exploitability and lower-bounded reward against an opponent.

Original: Our method can exploit the weakness of its opponent without signiﬁcantly sacriﬁcing its exploitability.
Revised: Our algorithm enables computationally efﬁcient online adaptations to a possibly changing opponent model.

Original: Empirical results show that our method signiﬁcantly outperforms NE baselines when opponents play non-NE strategies and keeps low exploitability at the same time.
Revised: Empirical results show that our method signiﬁcantly outperforms NE baselines when opponents play non-NE strategies and keeps low exploitability at the same time.

Revised: It is also much more efﬁcient than previous safe exploitation baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Behind the recent breakthroughs of superhuman AIs in Go (Silver et al., 2016; 2017; Schrittwieser et al., 2020), heads-up no-limit Texas hold’em (HUNL) (Brown et al., 2018; Moravcík et al., 2017;
Revised: Behind the recent breakthroughs of superhuman AIs in Go (Silver et al., 2016; 2017; Schrittwieser et al., 2020), heads-up no-limit Texas hold’em (HUNL) (Brown et al., 2018; Moravcík et al., 2017;

Original: Brown & Sandholm, 2019; Brown et al., 2020),  Hanabi (Bard et al., 2020),and contract bridge (Tian et al., 2020b), search plays a vital role.
Revised: Brown & Sandholm, 2019; Brown et al., 2020), and Hanabi (Lerer et al., 2020), search plays a vital role.

Original: In perfect information games, Monte Carlo tree search (MCTS) can be viewed as a powerful policy improvement operator.
Revised: In perfect information games, Monte Carlo tree search (MCTS) is widely applied to improve policy’s strength.

Original: In zero-sum imperfect information games such as poker, search algorithms are widely applied to ﬁnd a Nash equilibrium (NE) approximation in subgames encountered in real time (Brown & Sandholm, 2017; Burch et al., 2014a).
Revised: In zero-sum imperfect information games such as poker, search algorithms are used to ﬁnd a Nash equilibrium (NE) approximation in subgames encountered in real time (Brown & Sandholm, 2017; Burch et al., 2014a).

Original: They  both have soundedmonotonic improvement guarantees in theory, and show tremendous performance promotion empirically.
Revised: They are both theoretically sounded and empirically powerful.

Original: In fully-cooperative imperfect information games,  search algorithms proposed in Tian et al.
Revised: In fully-cooperative imperfect information games, the search algorithm proposed in Tian et al.

Original: (2020b) and Lerer et al. (2020) are also proved to never be detrimental to the current policy. 
Revised: (2020b) is proved to never be detrimental to the current policy. Lerer et al. (2020) also ensures original performance asymptotically.

Original: In zero-sum games, NE-based search algorithms (Burch et al., 2014a; Moravcik et al., 2016; Brown & Sandholm, 2017; Brown et al., 2018)
Revised: In zero-sum games, NE-based search algorithms (Burch et al., 2014a; Moravcik et al., 2016; Brown & Sandholm, 2017; Brown et al., 2018)

Original: ﬁnd safe strategies with low exploitability and produce strong baselines against all opponents (Brown & Sandholm, 2019).
Revised: ﬁnd safe strategies with low exploitability and produce strong baselines against all opponents (Brown & Sandholm, 2019).

Original: However, it may be overly conservative confronted with opponents with limited rationality and fail to take advantages of their weaknesses to obtain higher rewards.
Revised: However, it may be overly conservative confronted with opponents with limited rationality and fail to take advantage of their weaknesses to obtain higher rewards (McCracken & Bowling, 2004; Johanson et al., 2007; Li & Miikkulainen, 2018).

Original: From the other perspective, there have been extensive studies on opponent exploitation to address the problem.
Revised: From the other perspective, there have been extensive studies on opponent exploitation to address the problem.

Original: Some typical works (Carmel & Markovitch, 1996; Billings et al., 2003; Gilpin & Sandholm, 2006; Li & Miikkulainen, 2018) model the opponent’s strategy based on previous observations and then search for a new strategy to exploit this model.
Revised: Some typical works (Carmel & Markovitch, 1996; Billings et al., 2003; Gilpin & Sandholm, 2006; Li & Miikkulainen, 2018) model the opponent’s strategy based on previous observations and then search for a new strategy to exploit this model.

Original: However, these methods often neglect the signiﬁcance of the strategy safety, thus being highly exploitable by the opponent.
Revised: However, these methods often neglect the signiﬁcance of the strategy safety, thus being highly exploitable by the opponent.

Original: Few exceptions including Johanson et al.
Revised: Few exceptions including Johanson et al.

Original: and Ganzfried & Sandholm (2015a) aimed to search for safe and robust  strategies, but theyeither require an accurate estimation of opponent’s strategy or are not scalable to large games such as Texas Hold’em.
Revised: and Ganzfried & Sandholm (2015a) aimed to search for safe and robust counter strategies, but they are computationally inefﬁcient in an online setting where the opponent model is being updated continuously with streamed data.

Original: In this paper, we study the dilemma of safety and opponent exploitation and present a new scalable real-time search framework Safe Exploitation Search (SES) that smoothly interpolates between the two extremes of strategy search, hence unifying safe search and opponent exploitation.
Revised: In this paper, we study the dilemma of safety and opponent exploitation and present a new scalable real-time search framework Safe Exploitation Search (SES) that smoothly interpolates between the two extremes of strategy search, hence unifying safe search and opponent exploitation.

Revised: It enables computationally efﬁcient online adaptations to a continuously changing opponent model, which is hard to address by previous safe exploitation algorithms.

Original: The safety criterion requires the reﬁned strategy to stay close to NE, formally speaking, to expose limited exploitability against any opponents, while the opponent exploitation criterion requires the strategy to adapt to its speciﬁc opponent and to exploit its weaknesses.
Revised: The safety criterion requires the reﬁned strategy to stay close to NE, formally speaking, to expose limited exploitability against any opponents, while the opponent exploitation criterion requires the strategy to adapt to its speciﬁc opponent and to exploit its weaknesses.

Original: We propose a novel maximization objective which combines the safety objective and exploitation, controlled by the exploitation level α .
Revised: We propose a novel maximization objective which combines the safety objective and exploitation, controlled by the exploitation level α .

Original: We construct a new gadget game to optimize this objective, which enable our method’s scalability to large games such as Texas Hold’em.
Revised: We construct a new gadget game to optimize this objective, which enables our method’s scalability to large games such as Texas Hold’em.

Original: Theoretically, we prove that SES is guaranteed to outperform NE at the cost of some constant increase in its own exploitability confronted with non-NE opponents.
Revised: Theoretically, we prove that SES is guaranteed to outperform NE at the cost of some constant increase in its own exploitability confronted with non-NE opponents.

Original: Empirically, we evaluate the effectiveness of our search algorithm in 1 didactic matrix game 2 poker games: Leduc Hold’em (Southey et al., 2005) and
Revised: Empirically, we evaluate the effectiveness of our search algorithm in 1 didactic matrix game 2 poker games: Leduc Hold’em (Southey et al., 2005) and

Original: Flop Hold’em Poker (FHP)(Brown et al., 2019).
Revised: Flop Hold’em Poker (FHP)(Brown et al., 2019).

Original: The experiment results demonstrate that our algorithm signiﬁcantly outperforms NE baselines against non-NE opponents and keeps low exploitability at the same time.
Revised: The experiment results demonstrate that our algorithm signiﬁcantly outperforms NE baselines against non-NE opponents and keeps low exploitability at the same time.

Revised: Additionally, SES is much more computationally efﬁcient than previous safe exploitation baselines.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: This paper investigates the problem of safe opponent exploitation in two-player zero-sum imperfect information games.
Revised: This paper investigates the problem of safe opponent exploitation in two-player zero-sum imperfect information games.

Original: We propose a novel search algorithm which balances between NE and exploiting opponents.
Revised: We propose a novel search algorithm which balances between NE and exploiting opponents.

Original: Two major relevant research areas are search algorithms in imperfect information games, and opponent exploitation.
Revised: Two major relevant research areas are search algorithms in imperfect information games, and opponent exploitation.

Original: Search in imperfect information games.
Revised: Search in imperfect information games.

Original: Search techniques are proven to be essential in developing strong AI strategies in both perfect and imperfect information games (Burch et al., 2014a; Moravcik et al., 2016; Brown & Sandholm, 2017).
Revised: In recent literature, search techniques are witnessed to be important in developing strong AI strategies in both perfect and imperfect information games (Burch et al., 2014a; Moravcik et al., 2016; Brown & Sandholm, 2017).

Original: Texas hold ’em poker is widely employed as a benchmark for imperfect information games.
Revised: Texas hold ’em poker is widely employed as a benchmark for imperfect information games.

Original: A primary part of the long-term research on Texas hold’em poker is the evolution of subgame solving algorithms, which aim at achieving a more accurate Nash equilibrium approximation in the subgame encountered given a pre-computed strategy for the full game which we refer to as the blueprint strategy.
Revised: A primary part of the long-term research on Texas hold’em poker is the evolution of subgame solving algorithms, which aim at achieving a more accurate Nash equilibrium approximation in the subgame encountered given a pre-computed strategy for the full game which we refer to as the blueprint strategy.

Original: Unsafe search (Billings et al., 2003; Ganzfried & Sandholm, 2015b; Gilpin & Sandholm, 2006; 2007) estimates the subgame reach probability assuming the opponent follows blueprint, and searches for a reﬁned subgame strategy.
Revised: Unsafe search (Billings et al., 2003; Ganzfried & Sandholm, 2015b; Gilpin & Sandholm, 2006; 2007) estimates the subgame reach probability assuming the opponent follows blueprint, and searches for a reﬁned subgame strategy.

Original: Subgame resolving (Burch et al., 2014a) and maxmargin search (Moravcik et al., 2016) are theoretically sounded safe search algorithms which ensure that the subgame strategy obtained is no worse than the blueprint.
Revised: Subgame resolving (Burch et al., 2014a) and maxmargin search (Moravcik et al., 2016) are theoretically sounded safe search algorithms which ensure that the subgame strategy obtained is no worse than the blueprint.

Original: They search in a gadget game and achieve safety by providing the opponent with the option not entering the current subgame.
Revised: They search in a gadget game and achieve safety by providing the opponent with the option not entering the current subgame.

Original:  Libratus (Brown et al., 2018) is the ﬁrst superhuman poker AI which is considered the milestone in the research of imperfect information games. It incorporates reach search and nested search techniques.
Revised: DeepStack (Moravcík et al., 2017) and Libratus (Brown et al., 2018) build strong poker AIs with the aid of search.

Original: Beyond poker, search algorithms for subgame reﬁnement have also shown promise in improving joint strategies in cooperative imperfect information games such as Hanabi (Bard et al., 2020) and the bidding phase of contract bridge (Tian et al., 2020a).
Revised: Beyond poker, search algorithms for subgame reﬁnement have also shown promise in improving joint strategies in cooperative imperfect information games such as Hanabi (Lerer et al., 2020) and the bidding phase of contract bridge (Tian et al., 2020a).

Original: The purpose of our search algorithm is different from previous methods in poker literature.
Revised: The purpose of our search algorithm is different from previous methods in poker literature.

Original: We seek to exploit opponents while keeping exploitability low, rather than simply approximating NE.
Revised: We seek to exploit opponents while keeping exploitability low, rather than simply approximating NE.

Original: Opponent exploitation.
Revised: Opponent exploitation.

Original: Most previous opponent exploitation researches (Carmel & Markovitch, 1996; Billings et al., 2003; Gilpin & Sandholm, 2006; Li & Miikkulainen, 2018) typically model the opponent’s strategy based on previous observations and then search for a new strategy to exploit this model, but put little emphasis on safety.Some opponent exploitation methods (Johanson et al., 2007; Ganzfried & Sandholm, 2015a) also rely on Linear Programming (Koller et al., 1994).
Revised: Most previous opponent exploitation researches (Carmel & Markovitch, 1996; Billings et al., 2003; Gilpin & Sandholm, 2006; Li & Miikkulainen, 2018) typically model the opponent’s strategy based on previous observations and then search for a new strategy to exploit this model, but put little emphasis on safety.

Original: Therefore, their scalability to large games such as Texas hold’em is limited.

Original: One similar work is Johanson et al.
Revised: One similar work is Johanson et al.

Original: (2007) which also aims to ﬁnd a method tosafely exploit the opponent.
Revised: (2007) which proposes p -restricted Nash response (RNR) to ﬁnd a safe exploitation strategy to the estimated opponent’s strategy.

Revised: It calculates a Nash equilibrium for the whole game restricting that the opponent plays the estimated strategy σ ﬁx with probability p , and any strategy with probability 1 − p .

Original: In that paper, Johanson et al.
Revised: In that paper, Johanson et al.

Original: (2007) assumes thatthe opponent plays according to a known strategy σ ﬁx  with probability p and  an arbitrary unknown strategy with probability 1 − p .
Revised: (2007) prove that a p -RNR to σ ﬁx is Pareto optimal with respect to exploitation and safety. However, it does not provide an explicit bound.

Original: The best response to this hybrid strategy will be used as the ﬁnal result. However, it requires an accurate estimation of thefull opponent strategy which can be rather hard to obtain. While our algorithm only requires the opponent’s private information distribution like distribution of private cards in Texas
Revised: Additionally, whenever the estimated opponent model changes or we want to use a different p to balance between safety and exploitation, p -RNR has to recompute the strategy for the whole game. It is computationally inefﬁcient in an online setting, where the opponent model is updated after every round with new game data.

Original: Hold ’em and allows for estimation errors theoretically. If the estimation is accurate enough, our algorithm can beneﬁt from exploiting the revealed opponent’s weaknesses. Otherwise, due to the safety guarantee, our algorithm’s exploitability is still bounded, which means that its performance is also lower bounded.
Revised: Our algorithm instead takes modelling error into account and provides explicit bounds for both safety and exploitation. With the aid of real-time search, it only searches for strategies in subgames encountered instead of the whole game.

Original: We provide experiments investigating into the compact of different estimation accuracies in section 5. Another similar work is (Ganzfried & Sandholm, 2015a).
Revised: Our experiments show that it is more efﬁcient than Johanson et al. Ganzfried & Sandholm (2015a) study safe exploitation strategies in repeated games, which is a different setting from this paper.

Revised: Intuitively, it achieves safety by risking in exploitability at most what it has earned over NE in expectation in previous rounds.

Original: Their algorithm will guarantee the expectation payoff of each round thesame as playing NE.
Revised: Therefore, its expected value in the whole repeated game is never worse than the NE.

Original: The main idea is to ﬁnd thebest response strategy against estimated opponent strategy with (cid:15) safety guarantee.
Revised: In contrast, this paper focuses on the safety of stage game strategies. Furthermore, our algorithm is complementary to Ganzfried & Sandholm (2015a).

Original: That is, it will lose at most (cid:15) more than a 
Revised: Ganzfried & Sandholm (2015a) calculate an ε -safe best response for the whole game at each iteration with LP. This procedure is one of the main limitations on the algorithm’s scalability. Our algorithm, which only reﬁnes strategies in subgames in real-time, can be a possible substitute for LP. To our knowledge, we are the ﬁrst paper to investigate the safe opponent exploitation problem in subgame resolving schemes.

Original: NE in expectation confronted with a best response opponent. But Ganzfried & Sandholm (2015a) relies on LP solver (Koller et al., 1994) to ﬁnd that strategy which makes it infeasible in large games.
Revised: Subgame resolving enables online adaptations to a continuously changing opponent model, eliminating the need to recompute a whole game strategy. It offers computational beneﬁts in practical opponent exploitation circumstances.

Original: It is worthwhile mentioning that there also have been extensive research (Albrecht & Stone, 2018) on agent modeling.
Revised: It is worthwhile mentioning that there also has been extensive research (Albrecht & Stone, 2018) on agent modeling.

Original: However, this paper onlyonly focuses on the theoretical and empirical results of the search algorithm, but not the agent modeling techniques.
Revised: However, this paper only focuses on the theoretical and empirical results of the search algorithm, but not the agent modeling techniques.

Original: We  use off-the-shelf agent modeling algorithms to estimate opponent’s strategies.
Revised: We can use off-the-shelf agent modeling algorithms to estimate opponent’s strategies.

Original: Agent modeling provides the ability to reason about the others, making predictions of their strategies, types, etc.
Revised: Agent modeling provides the ability to reason about the others, making predictions of their strategies, types, etc.

Original: He et al.
Revised: He et al.

Original: (2016) embeds opponent model learning into deep Q-learning, which stabilizes the training process faced with opponents whose policy is changing over time.
Revised: (2016) embeds opponent model learning into deep Q-learning, which stabilizes the training process faced with opponents whose policy is changing over time.

Original: LOLA (Foerster et al., 2018) considers the evolution of other agents in the training process of multi-agent reinforcement learning.
Revised: LOLA (Foerster et al., 2018) considers the evolution of other agents in the training process of multi-agent reinforcement learning.

Original: Raileanu et al.
Revised: Raileanu et al.

Original: (2018) proposes to model others from one’s own policy.
Revised: (2018) proposes to model others from one’s own policy.

Original: Rabinowitz et al.
Revised: Rabinowitz et al.

Original: (2018) uses meta learning to predict strategies of different kinds of agents.
Revised: (2018) uses meta learning to predict strategies of different kinds of agents.

Original:  N OTATIONS AND BACKGROUND3. E XTENSIVE - FORM IMPERFECT INFORMATION GAME
Revised:  N OTATIONS AND BACKGROUND

Revised: An extensive-form imperfect information game G =

Original: An extensive-form imperfect information game G describes sequential interactions among agents, where agents have private information. G is a tuple ( P, H, Z, A, χ, ρ, (cid:118) , σ c , u, I ) .
Revised: ( P, H, Z, A, χ, ρ, · , σ c , u, I ) describes sequential interactions among agents, where agents have private information.

Original: P is a ﬁnite set  of n players including a chance node c representing the stochastic nature of the environment.
Revised: A ﬁnite set P consists of n players and a chance node c which represents the stochastic nature of the environment.

Original: H is a set of non-terminal decision nodes, or histories. Z is a set of terminal nodes, or leaves.
Revised: The set of non-terminal decision nodes is denoted as H , and Z is a set of terminal nodes, or leaves.

Original: A is a set of possible actions, and χ : H → 2 | A | is a function which assigns to each decision node h ∈ H a set of legal actions.
Revised: The set of possible actions is A , and χ : H → 2 | A | is a function which assigns to each decision node h ∈ H a set of legal actions.

Original:  ρ : H → Pis a player function, which assigns to each decision node a player p ∈ P who acts at that node.
Revised: A player function ρ : H → P assigns to each decision node a player p ∈ P who acts at that node.

Original:  (cid:118) : H × A → H ∪ Z is the successor function.
Revised: If action a leads from h to h (cid:48) , we write h · a = h (cid:48) . If there exists a sequence of actions leading from h to h (cid:48) , we write h (cid:118) h (cid:48) .

Original: At each node h ∈ H , the acting player p = ρ ( h ) picks an action from legal actions a ∈ χ ( h ) , and leads node h into its child h · a(cid:118) h (cid:48) .
Revised: At each node h ∈ H , the acting player p = ρ ( h ) picks an action from legal actions a ∈ χ ( h ) , and leads node h into its child h · a .

Original: The chance node always samples an action from its own distribution σ c , which is common knowledge to all players.
Revised: The chance node always samples an action from its own distribution σ c , which is common knowledge to all players.

Original:  u = ( u 1 , u 2 , . . 
Revised: Utility functions are u = ( u 1 , u 2 , . . .

Original:  u n ) are utility functions, where u i : Z → R deﬁnes the utility of player i at terminal node z ∈ Z .
Revised: , u n ) , where u i : Z → R deﬁnes the utility of player i at terminal node z ∈ Z .

Original: The nature of imperfect information is characterized by infosets I =
Revised: The nature of imperfect information is characterized by infosets I =

Original: ( I 1 , I 2 , . .
Revised: ( I 1 , I 2 , . .

Original: , I n ) , where I i =
Revised: , I n ) , where I i =

Original: ( I i, 1 , . . .
Revised: ( I i, 1 , . . .

Original: , I i,k i ) is a partition of H for player i .
Revised: , I i,k i ) is a partition of H for player i .

Original: Two states in the same infoset must have the same acting player and the same legal action sets.
Revised: Two states in the same infoset must have the same acting player and the same legal action sets.

Original: We use I ( h ) to denote the infoset that h belongs to.
Revised: We use I ( h ) to denote the infoset that h belongs to.

Original: A player p cannot distinguish between states h 1 and hif I ( h 1 ) =
Revised: A player p cannot distinguish between states h 1 and h 2 if I ( h 1 ) =

Original: I ( h 2 ) , and thus should behave identically on all states in the same infoset.
Revised: I ( h 2 ) , and thus should behave identically on all states in the same infoset.

Original: To give a better understanding of these notations, we offer an example.

Original: In a two player Texas

Original: Hold’em, P = { 1 , 2 , c } where c is the card dealer.

Original: H consists of all non-terminal betting sequences when the game does not end.

Original: Z consists of the states when a player folds or the betting ends.

Original: A = { call, raise, fold } .

Original: For player p , states whose public cards, betting sequence, and private cards of himself are the same belong to the same infoset of player p (regardless of the private cards of his opponent).

Original: The strategy of a player p is σ p : I p ×
Revised: The strategy of a player p is σ p : I p ×

Original: A → R , where σ p ( I, a ) is a distribution over valid actions on infoset I .
Revised: A → R , where σ p ( I, a ) is a distribution over valid actions on infoset I .

Original: For simplicity, we also use σ p ( h, a ) to denote σ p ( I ( h ) , a ) .
Revised: For simplicity, we also use σ p ( h, a ) to denote σ p ( I ( h ) , a ) .

Original: We use π σ ( h ) to denote the probability of reaching state h from the root when agents choose a strategy proﬁle σ
Revised: We use π σ ( h ) to denote the probability of reaching state h from the root when agents choose a strategy proﬁle σ

Original: = (cid:104) σ 1 , σ 2 , . .
Revised: = (cid:104) σ 1 , σ 2 , . .

Original: , σ n (cid:105) .
Revised: , σ n (cid:105) .

Original: Formally, π σ ( h ) =
Revised: Formally, π σ ( h ) =

Original: (cid:81) h (cid:48)
Revised: (cid:81) h (cid:48)

Original: · a (cid:118) h σ ρ ( h (cid:48) )
Revised: · a (cid:118) h σ ρ ( h (cid:48) )

Original: ( h (cid:48) , a ) .
Revised: ( h (cid:48) , a ) .

Original: We use π σ − p ( h ) =
Revised: We use π σ − p ( h ) =

Original: (cid:81) h (cid:48)
Revised: (cid:81) h (cid:48)

Original: · a (cid:118) h
Revised: · a (cid:118) h

Original: ∧ ρ ( h (cid:48) )
Revised: ∧ ρ ( h (cid:48) )

Original: (cid:54) =
Revised: (cid:54) =

Original: p σ ρ ( h (cid:48) )
Revised: p σ ρ ( h (cid:48) )

Original: ( h (cid:48) , a ) to denote the probability of reaching h when player p always chooses the action that leads to h whenever possible.
Revised: ( h (cid:48) , a ) to denote the probability of reaching h when player p always chooses the action that leads to h whenever possible.

Original: π σ ( h, h (cid:48) ) is the reaching probability of h (cid:48) from h .
Revised: π σ ( h, h (cid:48) ) is the reaching probability of h (cid:48) from h .

Original: π σ ( h · a, h (cid:48) ) is the the probability of reaching h (cid:48) from h if action a is taken at h .
Revised: π σ ( h · a, h (cid:48) ) is the the probability of reaching h (cid:48) from h if action a is taken at h .

Original: These probabilities can be formally deﬁned in a similar manner.
Revised: These probabilities can be formally deﬁned in a similar manner.

Original: The expected utility of player p given strategy proﬁle σ is u σp = (cid:80)z
Revised: The expected utility of player p given strategy proﬁle σ is u σ p = (cid:80)

Original:  ∈
Revised: z ∈

Original: Z π σ ( z ) u p ( z ) .
Revised: Z π σ ( z ) u p ( z ) .

Original: The counterfactual value v σp ( I, a ) is the expected utility that player p will obtain after taking action a at infoset
Revised: The counterfactual value v σp ( I, a ) is the expected utility that player p will obtain after taking action a at infoset

Original: I , given the joint policy proﬁle is σ .
Revised: I , given the joint policy proﬁle is σ .

Original: Mathematically, it is the weighted sum of expected values at all states h ∈
Revised: Mathematically, it is the weighted sum of expected values at all states h ∈

Original: I .
Revised: I .

Original: [Equation]
Revised: [Equation]

Original: In the rest of the paper, we focus on two-player zero-sum games with perfect recall.
Revised: In the rest of the paper, we focus on two-player zero-sum games with perfect recall.

Original: Zero-sum means ∀ z ∈ Z, u 1 ( z ) + u 2 ( z ) = 0 .
Revised: Zero-sum means ∀ z ∈ Z, u 1 ( z ) + u 2 ( z ) = 0 .

Original: Perfect recall means that no player will forget the information which has been obtained previously in the game.
Revised: Perfect recall means that no player will forget the information which has been obtained previously in the game.

Original: This is a common assumption in related literature.
Revised: This is a common assumption in related literature.

Original: A best response strategy BR p ( σ ) =
Revised: A best response strategy BR p ( [Equation]

Original: arg max σ p u (cid:104) σ p ,σ − p (cid:105) p for player p is the strategy that maximize his own expected utility against ﬁxed opponent strategy σ − p .
Revised: ) = [Equation] for player p is the strategy that maximize his own expected utility against ﬁxed opponent strategy σ − p .

Original: The exploitability of strategy σ p is exp( σ p ) =
Revised: The exploitability of strategy σ p is [Equation] ) =

Original: u σ ∗ p − u (cid:104) σ p ,BR − p ( σ ) (cid:105) p where σ ∗ is the optimal strategy, and is an NE in two-player zero-sum games.
Revised: [Equation] p where σ ∗ is the optimal strategy, and is an NE in two-player zero-sum games.

Original: It measures the performance of σ p against its best response comparing with the
Revised: It measures the performance of σ p against its best response comparing with the

Original: NE.
Revised: NE.

Original: A counterfactual best response CBR p ( σ  ) is a strategy where σ p ( I, a ) > 

Revised: A counterfactual best response CBR p ( σ − p ) is a strategy where σ p ( I, a ) >

Original: if and only if v σp ( I, a ) ≥ max b v σp ( I, b ) .
Revised: if and only if v σp ( I, a ) ≥ max b v σp ( I, b ) .

Original: Counterfactual best response is a best response, but not vice versa.
Revised: Counterfactual best response is a best response, but not vice versa.

Original: The counterfactual best response value CBV σp ( I ) = v (cid:104)
Revised: The counterfactual best response value CBV [Equation] −

Original: CBR p ( σ ) ,σ − p (cid:105) p  is the expected utility of the counterfactual best response policy.
Revised: [Equation] p [Equation] is the expected utility of the counterfactual best response policy.

Original: Since we focus on two-player zero-sum games, we will use
Revised: Since we focus on two-player zero-sum games, we will use

Original: CBV σ p ( I ) as a shorthand notation for CBV σ p − p ( I ) .
Revised: CBV σ p ( I ) as a shorthand notation for CBV σ p − p ( I ) .

Original: 3. S UBGAME

Original: We follow the imperfect information subgame deﬁnition as in Burch et al.
Revised: We follow the imperfect information subgame deﬁnition as in Burch et al.

Original: (2014b).
Revised: (2014b).

Original: An augmented infoset contains states which cannot be distinguished by the remaining players.
Revised: An augmented infoset contains states which cannot be distinguished by the remaining players.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: An imperfect information subgame S is a forest of trees, closed under both the descendant relation and membership within augmented infosets for any player.
Revised: An imperfect information subgame S is a forest of trees, closed under both the descendant relation and membership within augmented infosets for any player.

Original: Let S top be the set of nodes which are roots of each tree in S .
Revised: Let S top be the set of nodes which are roots of each tree in S .

Original: In Texas Hold ’em, all nodes in a subgame must share the same betting sequence and public cards.

Original: That is, they only differs in the private cards of players.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we introduce our novel search algorithm called Safe Exploitation Search, which exploits the weaknesses of opponent while ensuring a bounded exploitability.
Revised: In this section, we introduce our novel search algorithm called safe exploitation search (SES), which exploits the weaknesses of opponent while ensuring a bounded exploitability.

Original: Let σ be the precomputed blueprint strategy.
Revised: Let σ be the precomputed blueprint strategy.

Original: Without loss of generality, assume we search for player 2’s reﬁned strategy σ S 2 by applying Safe Exploitation Search to all subgames S ∈ S .
Revised: Without loss of generality, assume we search for player 2’s reﬁned strategy σ S 2 by applying SES to all subgames S ∈ S .

Original: Finally, the reﬁned strategy for P2 after search is σ (cid:48) 2 , which is the same as σ 2 in { I i 2 |∀ S ∈ S , I i 2 / ∈ S } and is replaced with σ S 2 in S ∈ S .
Revised: Finally, the reﬁned strategy for P2 after search is σ (cid:48) 2 , which is the same as σ 2 in { I i 2 |∀ S ∈ S , I i 2 / ∈ S } and is replaced with σ S 2 in S ∈ S .

Original: 4. S AFE E XPLOITATION S EARCH
Revised: 4. S AFE E XPLOITATION S EARCH

Original: Our algorithm offers a uniﬁed approach to balance between these two demands with theoretical guarantees.
Revised: Our algorithm offers a uniﬁed approach to balance between these two demands with theoretical guarantees.

Original: The objective of our search algorithm is to ﬁnd a new subgame strategy σ S 2 for S ∈ S which maximizes
Revised: The objective of our search algorithm is to ﬁnd a new subgame strategy σ S 2 for S ∈ S which maximizes

Original: [Equation]
Revised: [Equation]

Original: where α ∈ [0 , 1] is a hyper-parameter controlling the exploitation level, and ˆ p ( I i 1 ) is the estimated probability of player 1 entering infoset I j 1 ∈ S top .
Revised: where α ∈ [0 , 1] is a hyper-parameter controlling the exploitation level, and ˆ p ( I i 1 ) is the estimated probability of player 1 entering infoset I j 1 ∈ S top .

Original: Given P2’s strategy (which is the blueprint σ 2 ) and P1’s actual strategy (which does not have to be the blueprint σ 1 ), the real probability of playerentering infoset I j 1 ∈ S top (which we denote as˜ p ( I j 1 ) ) is determined.
Revised: Given P2’s strategy (which is the blueprint σ 2 ) and P1’s actual strategy (which does not have to be the blueprint σ 1 ), the real probability of playerentering infoset I j 1 ∈ S top (which we denote as p ( I j 1 ) ) is determined.

Original: ˆ p ( I i 1 ) is just an estimation of˜ p ( I j 1 ) .
Revised: ˆ p ( I i 1 ) is just an estimation of p ( I j 1 ) .

Original: For instance, in poker, it is the estimated distribution of private cards player 1 holds.
Revised: For instance, in poker, it is the estimated distribution of private cards player 1 holds.

Original: Both theoretically and empirically, such estimation does not have to be fully accurate.
Revised: Both theoretically and empirically, such estimation does not have to be fully accurate.

Original: It can be done with off-the-shelf opponent modeling techniques, which lies beyond the focus of this paper.
Revised: It can be done with off-the-shelf opponent modeling techniques, which lies beyond the focus of this paper.

Original: Though seemingly complicated, intuitively, the maximization objective achieves a balance between opponent exploitation and safety , controlled by exploitation level α .
Revised: Though seemingly complicated, intuitively, the maximization objective achieves a balance between opponent exploitation and safety , controlled by exploitation level α .

Original: The ﬁrst part of the objective is maximized when (cid:80) 
Revised: The ﬁrst part of the objective is maximized when (cid:80) I j 1 ∈ S top

Original: [Equation] CBV σ S 2 1 ( I j 1 ) is minimized.
Revised: ˆ p ( I j 1 ) CBV σ S 2 1 ( I j 1 ) is minimized.

Original: It aims at ﬁnding a strategy σ S 2 which results in the lowest value for P1 under the assumption that the reach probabilities remain the same.
Revised: It aims at ﬁnding a strategy σ S 2 which results in the lowest value for P1 under the assumption that the reach probabilities is ˆ p .

Original: It can be interpreted as exploiting the estimated P1’s strategy.
Revised: It can be interpreted as exploiting the estimated P1’s strategy.

Original: The second part of the objective demands the resolved strategy to behave well against any reach probability distribution.
Revised: The second part of the objective demands the resolved strategy to behave well against any reach probability distribution.

Original: We use the subgame margin min
Revised: We use the subgame margin min

Original: I j 1 ∈ S top

Original: [Equation] − [Equation] ) (cid:17) (Moravcik et al., 2016) which can be regarded as the worst-case utility increase for P2.
Revised: I j 1 ∈ S top (cid:16) v σ 1 ( I j 1 ) − CBV σ S 2 1 ( I j 1 ) (cid:17) (Moravcik et al., 2016) which can be regarded as the worst-case utility increase for P2.

Original: By maximizing the objective 2, we provide sound theoretical results for both safety and opponent exploitation.
Revised: By maximizing the objective 2, we provide sound theoretical results for both safety and opponent exploitation.

Original: Additionally, we provide analyses of how (1) exploitation level α , (2) accuracy of opponent modeling, and (3) strength of the blueprint strategy impact the theoretical bound.
Revised: Additionally, we provide analyses of how (1) exploitation level α , (2) accuracy of opponent modeling, and (3) strength of the blueprint strategy impact the theoretical bound.

Original: By gradually increasing α from 0 to 1, our algorithm tends to exploit rather than keeping safety.
Revised: By gradually increasing α from 0 to 1, our algorithm tends to exploit rather than keeping safety.

Original: Theorem 1.
Revised: Theorem 1.

Original: (safety)  S is a disjoint set of subgames S .
Revised: (safety) Let S be a disjoint set of subgames S .

Original: Let σ ∗  2  be  P2’s strategy with lowest exploitability which differs with σ 2 only in S .
Revised: Let σ ∗ = (cid:104) σ ∗ 1 , σ ∗ 2 (cid:105) be the NE where P2’s strategy is constrained to be the same with σ 2 outside S .

Original: Deﬁne ∆ = max S ∈ S , I [Equation]{+i 1 ) − v σ 
+}
Revised: Deﬁne ∆ = max S ∈ S ,I i 1 ∈ S top | CBV σ ∗ 2 1 ( I {+i 1 ) − v σ 
+}

Original: [Equation]
Revised: ( I i 1 ) | .

Original: S top | CBV σ [Equation]
Revised: Let ˜ p

Original: [Equation] I [Equation] 1 ) − v σ 1 ( I i 1 ) | .
Revised: ( I i 1 ) be the reach probability given by σ ∗ 1 and P2’s blueprint.

Original: Let ˜ p
Revised: Let ˆ p

Original: ( I i 1 ) be the real reach probability, and ˆ p ( I i 1 ) be its estimation.
Revised: ( I i 1 ) be the estimation of reach probability p ( I i 1 ) calculated by the real opponent strategy σ 1 .

Original: Deﬁne 
Revised: Deﬁne τ =

Original: [Equation]
Revised: [Equation]

Original:  Whenever 1 − (2 τ + 1) α > 0 , we have a bounded exploitability given by:
Revised: | . Whenever 1 − (2 τ + 1) α > 0 , we have a bounded exploitability given by:

Original: [Equation]
Revised: [Equation]

Original: Recall that σ (cid:48) 2 is the reﬁned strategy after search.
Revised: Recall that σ (cid:48) 2 is the reﬁned strategy after search.

Original: The proof is provided in Appendix A.
Revised: The proof is provided in Appendix A.

Original: This theorem implies that the exploitability of the new strategy is smaller than that of strategy σ ∗ 2 plus a constant value, which is the closest strategy to NE if constrained to differ from σ 2 only in S .
Revised: This theorem implies that the exploitability of the new strategy is smaller than that of strategy σ ∗ 2 plus a constant value, which is the closest strategy to NE if constrained to differ from σ 2 only in S .

Original: The corresponding theoretical result of maxmargin search (Moravcik et al., 2016), a safe search algorithm with no opponent exploitation abilities, is exp( σ (cid:48) 2 )
Revised: The corresponding theoretical result of maxmargin search (Moravcik et al., 2016), a safe search algorithm with no opponent exploitation abilities, is exp( σ (cid:48) 2 )

Original: ≤ exp( σ ∗ 2 )+2∆ .
Revised: ≤ exp( σ ∗ 2 )+2∆ .

Original: Comparing these two results, we can interpret the term 2 / (1 − (2 τ + 1) α ) as the additional risk introduced by exploiting the opponent.
Revised: Comparing these two results, we can interpret the term 2 / (1 − (2 τ + 1) α ) as the additional risk introduced by exploiting the opponent.

Original: If exploitation level α = 0 , then our bound is as tight as that of maxmargin search (Moravcik et al., 2016).
Revised: If exploitation level α = 0 , then our bound is as tight as that of maxmargin search (Moravcik et al., 2016).

Original: Note that τ is the difference between the estimated distribution and the real distribution, which can be interpreted as opponent modeling error.

Original: The bound also gets tighter if theopponent modeling error τ gets smaller, or the blueprint σ 2 is closer to σ ∗ 2 .
Revised: The bound also gets tighter if the τ gets smaller, or the blueprint σ 2 is closer to σ ∗ 2 .

Original: Theorem 2.
Revised: Theorem 2.

Original: (opponent exploitation) Let (cid:15) =
Revised: (opponent exploitation) Let (cid:15) =

Original: (cid:107) ˆ p − p (cid:107) 1 be the L1 distance of the distribution p
Revised: (cid:107) ˆ p − p (cid:107) 1 be the L1 distance of the distribution p

Original: ( I i 1 ) and ˆ p ( I i 1 ) .
Revised: ( I i 1 ) and ˆ p ( I i 1 ) .

Original: Let η = min S ∈ S max I j 1 ∈ S top (cid:16) CBV 1 ( I j 1 , σ S 2 ) − CBV 1 ( I j 1 , σ ∗ 2 ) (cid:17) ≥ 0 .
Revised: Let η = min S ∈ S max I j 1 ∈ S top (cid:16) CBV 1 ( I j 1 , σ S 2 ) − CBV 1 ( I j 1 , σ ∗ 2 ) (cid:17) ≥ 0 .

Original: We use BR [ S ,σ p ] p ( σ ) to denote the strategy for player p which maximizes its utility in subgame S ∈ S against σ − p under the constraint that BR [ S ,σ p ] p ( σ ) and σ p differs only inside S .
Revised: We use BR [ S ,σ p ] p ( σ ) to denote the strategy for player p which maximizes its utility in subgame S ∈ S against σ − p under the constraint that BR [ S ,σ p ] p ( σ ) and σ p differs only inside S .

Original: By maximizing objective 2, for all S ∈ S , the reﬁned strategy σ (cid:48) 2 satisﬁes
Revised: By maximizing objective 2, for all S ∈ S , the reﬁned strategy σ (cid:48) 2 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: The proof is provided in Appendix A. Observe that the reach probability p is characterized by P1’s strategy outside S and ˆ p is its estimation.
Revised: The proof is provided in Appendix A. Observe that the reach probability p is characterized by P1’s strategy outside S and ˆ p is its estimation.

Original: Because the search algorithm always ﬁnd a stronger response strategy for P1 in S (which is exactly BR [ S ,σ 1 ]( σ (cid:48) 2 ) ) as well, opponent exploitation refers to adapting to P1’s strategy σ 1 outside S .
Revised: Because the search algorithm always ﬁnd a stronger response strategy for P1 in S (which is exactly BR [ S ,σ 1 ]( σ (cid:48) 2 ) ) as well, opponent exploitation refers to adapting to P1’s strategy σ 1 outside S .

Original: This theorem implies that the utility of the new strategy σ (cid:48) 2 is lower bounded by the utility of σ ∗ 2 when both confronted with P1’s unknown strategy outside S .
Revised: This theorem implies that the utility of the new strategy σ (cid:48) 2 is lower bounded by the utility of σ ∗ 2 when both confronted with P1’s unknown strategy outside S .

Original: It provides theoretical guarantees for the opponent exploitation ability of our algorithm.
Revised: It provides theoretical guarantees for the opponent exploitation ability of our algorithm.

Original: (cid:15) can be interpreted as estimation error.
Revised: (cid:15) can be interpreted as estimation error.

Original: The lower bound increases if the estimation error get smaller or the blueprint σ 2 is closer to σ ∗ 2 .
Revised: The lower bound increases if the estimation error get smaller or the blueprint σ 2 is closer to σ ∗ 2 .

Original: We show empirically how exploitation level α and estimation error impact both safety and exploitation abilities in section 5.
Revised: We show empirically how exploitation level α and estimation error impact both safety and exploitation abilities in section 5.

Original: 4. G ADGET G AME
Revised: 4. G ADGET G AME

Original: In order to ﬁnd σ S 2 which maximize objective 2, a straight-forward method is to reformulate the maximization problem as a Linear Programming problem (Moravcik et al., 2016).
Revised: In order to ﬁnd σ S 2 which maximize objective 2, a straight-forward method is to reformulate the maximization problem as a Linear Programming problem (Moravcik et al., 2016).

Original: However, current

Original:  LP solvers (Koller et al., 1994) cannot handlesuch large-scale problems.
Revised: However, LP solvers (Koller et al., 1994) cannot handle large-scale problems.

Original: Alternatively, inspired by Moravcik et al.
Revised: Alternatively, inspired by Moravcik et al.

Original: (2016), we create a gadget game and then apply iteration-based NE algorithms such as CFR (Zinkevich et al., 2007;
Revised: (2016), we create a gadget game and then apply iteration-based NE algorithms such as CFR (Zinkevich et al., 2007;

Original: Tammelin et al., 2015; Lanctot et al., 2009) in the gadget game.
Revised: Tammelin et al., 2015; Lanctot et al., 2009) in the gadget game.

Original: The gadget game is carefully designed such that the NE solution found in it is exactly the solution to the original optimization problem.
Revised: The gadget game is carefully designed such that the NE solution found in it is exactly the solution to the original optimization problem.

Original: As shown in Figure 1, the original subgame is copied into two identical parts S 1 , S 2 in the gadget game.
Revised: As shown in Figure 1, the original subgame is copied into two identical parts S 1 , S 2 in the gadget game.

Revised: Player 2’s infosets stretch over both branches, while player 1 can distinguish between the two parts.

Original: The procedure of constructing such gadget game can be summarized into 4 steps: (i).
Revised: The procedure of constructing such gadget game can be summarized into 4 steps: (i).

Original: Create a chance node at the top of the gadget game.
Revised: Create a chance node at the top of the gadget game.

Original: (ii)
Revised: (ii)

Original: For the left part of the gadget game, we construct a
Revised: For the left part of the gadget game, we construct a

Original: P1 node to let P1 choose an infoset
Revised: P1 node to let P1 choose an infoset

Original: I i 1 to enter.
Revised: I i 1 to enter.

Original: The following chance node samples a speciﬁc state with probability proportional to π σ − 1 ( h ) for all h ∈
Revised: The following chance node samples a speciﬁc state with probability proportional to π σ − 1 ( h ) for all h ∈

Original: I 1 .
Revised: I 1 .

Original: (iii)
Revised: (iii)

Original: For the right part, create a chance node 
Revised: For the right part, create a chance node sampling an infoset I i 1 .

Revised: The following chance node again samples a speciﬁc state with probability

Original: S
Revised: S

Original: C C
Revised: C C

Original: C
Revised: C

Original: P
Revised: P

Original: S
Revised: S

Original: C C 
Revised: C C C

Original: C C
Revised: C C

Original: C 1 −𝛼 𝛼
Revised: [Figure]

Original: P P
Revised: P P

Original: L
Revised: L

Original: M
Revised: M

Original: R
Revised: R

Original: U
Revised: U

Original: O  9.
Revised: O  9.

Original: D  9.
Revised: D  9.

Original: F - -sampling an infoset I i 1 . The following chance node again samples a speciﬁc state with probability  proportional to π σ − 1 ( h ) for all h ∈
Revised: F - -  proportional to π σ − 1 ( h ) for all h ∈

Original: I 1 .
Revised: I 1 .

Original: (iv).
Revised: (iv).

Original: Shift the utility of the gadget game by v σ 1 ( I i 1 ) .
Revised: Shift the utility of the gadget game by v σ 1 ( I i 1 ) .

Original: The details are described below.
Revised: The details are described below.

Original: The chance node at the top goes to the left part with probability 1 − α , and the right part with probability α .
Revised: The chance node at the top goes to the left part with probability 1 − α , and the right part with probability α .

Original: The outcome is visible to P1 but not P2.
Revised: The outcome is visible to P1 but not P2.

Original: Therefore, the left and the right part are  the same  for P2, and his strategy σ S 2 will be the same for both parts.
Revised: Therefore, corresponding nodes in both branches are in the same infosets for P2, and his strategy σ S 2 will be the same for both parts.

Original: And since σ S 1 is the best-response to σ S 2 and the two parts only differ at how to go to each infoset of player 1, player 1 will also keep his strategy the same in both parts.
Revised: Since σ S 1 is the best-response to σ S 2 and the two parts only differ at how to go to each infoset of player 1, player 1 will also keep his strategy the same in both parts.

Original: We subtract u 1 ( z ) by v σ 1 ( I i 1 ) for all z (cid:118) h, h ∈
Revised: We subtract u 1 ( z ) by v σ 1 ( I i 1 ) for all z (cid:118) h, h ∈

Original: I i 1 , and add u 2 ( z ) by v σ 1 ( I i 1 ) in order to keep the subgame zero-sum.
Revised: I i 1 , and add u 2 ( z ) by v σ 1 ( I i 1 ) in order to keep the subgame zero-sum.

Original: By doing so, the objective of a Nash Equilibrium of p 2 will change from maximizing − CBV σ S 2 1 ( I i 1 ) to maximizing v σ 1 ( I i 1 ) − CBV σ S 2 1 ( I i 1 ) .
Revised: By doing so, the objective of a Nash Equilibrium of p 2 will change from maximizing − CBV σ S 2 1 ( I i 1 ) to maximizing v σ 1 ( I i 1 ) − CBV σ S 2 1 ( I i 1 ) .

Original: As for the left part of the gadget game, the P1 node on the second level in Figure 1 enables P to enter an arbitrary infoset I i 1 .
Revised: As for the left part of the gadget game, the P1 node on the second level in Figure 1 enables P to enter an arbitrary infoset I i 1 .

Original: Since this is a zero-sum game, in an NE strategy, he will enter the one with lowest v σ 1 ( I i 1 ) − CBV 1 ( I i 1 , σ ∗ 2 ) which is exactly the minimization in the second term of
Revised: Since this is a zero-sum game, in an NE strategy, he will enter the one with lowest v σ 1 ( I i 1 ) − CBV 1 ( I i 1 , σ ∗ 2 ) which is exactly the minimization in the second term of

Original: SE ( σ S 2 ) .
Revised: SE ( σ S 2 ) .

Original: The chance node on the second level of the right part will sample an infoset I i 1 according to reach probability ˆ p
Revised: The chance node on the second level of the right part will sample an infoset I i 1 according to reach probability ˆ p

Original: ( I i 1 ) .
Revised: ( I i 1 ) .

Original: So that the NE objective of this part is exactly the summation in the ﬁrst term of
Revised: So that the NE objective of this part is exactly the summation in the ﬁrst term of

Original: SE ( σ S 2 ) .
Revised: SE ( σ S 2 ) .

Original: 4. M ATRIX G AME
Revised: 4. M ATRIX G AME

Original: In this part, we offer a matrix game as an example to show the necessity of considering safety and expected payoff simultaneously, and to demonstrate the superiority of SES over a simple mixing strategy, which follows a best response to the estimated opponent model with probability α and follows the blueprint with probability 1 − α .
Revised: In this part, we offer a matrix game as an example to show the necessity of considering safety and expected payoff simultaneously, and to demonstrate the superiority of SES over a simple mixing strategy, which follows a best response to the estimated opponent model with probability α and follows the blueprint with probability 1 − α .

Original: In the matrix game shown in Table 1, there are two  NEs.
Revised: In the matrix game shown in Table 1, let’s consider two speciﬁc NEs.

Original: In both NE, P1 will play L/M with 0 .probability each.
Revised: In both NEs, P1 will play L/M with 0 . 5 probability each.

Original: P2 will play U/O with 0 .
Revised: P2 will play U/O with 0 .

Original: 5 probability in the ﬁrst NE and O/D with 0 . 5 probability in the second NE.
Revised: 5 probability in the ﬁrst NE and O/D with 0 .probability in the second NE.

Original: Suppose the blueprint strategy is the ﬁrst NE.
Revised: Suppose the blueprint strategy is the ﬁrst NE.

Original: Consider the case when P1 plays a rather weak strategy that he will only play action R.
Revised: Consider the case when P1 plays a rather weak strategy that he will only play action R.

Original: We apply SES to search for P2’s reﬁned strategy.
Revised: We apply SES to search for P2’s reﬁned strategy.

Original: c h
Revised: c h

Original: i p s / h
Revised: i p s / h

Original: Evaluation
Revised: Evaluation

Original: SESMixing 0.
Revised: SESMixing 0.

Original: 1.0     c h i p s / h
Revised: 1.0     c h i p s / h

Original: Exploitability
Revised: Exploitability

Original: SESMixing
Revised: SESMixing

Original: When the estimation of opponent strategy is accurate such that ˆ p = p , the best response of P2 is always playing F, which is highly exploitable, while SES ﬁnds the second NE under proper α .
Revised: When the estimation of opponent strategy is accurate such that ˆ p = p , the best response of P2 is always playing F, which is highly exploitable, while SES ﬁnds the second NE under proper α .

Original: To give more details, the exploitability and expected payoff of the strategy found by SES and the mixing strategy are shown in Figure 2.
Revised: To give more details, the exploitability and expected payoff of the strategy found by SES and the mixing strategy are shown in Figure 2.

Original: We can see that SES achieves both lower exploitability and better performance than the mixing strategy at almost all α values.
Revised: We can see that SES achieves both lower exploitability and better performance than the mixing strategy at almost all α values.

Original: The reason behind this success is that P always playing R is a Gift Strategy (Ganzfried & Sandholm, 2015a) in the designed matrix game and
Revised: The reason behind this success is that P always playing R is a Gift Strategy (Ganzfried & Sandholm, 2015a) in the designed matrix game and

Original: SES manages to utilize such gift strategy.
Revised: SES manages to utilize such gift strategy.

Original: Strategy σ − p is a Gift Strategy if it is not a best response to a NE strategy σ ∗ p .
Revised: Strategy σ − p is a Gift Strategy if it is not a best response to a NE strategy σ ∗ p .

Original: Therefore, we can switch to σ ∗ p to get better performance against σ − p while also keeping exploitability low.
Revised: Therefore, we can switch to σ ∗ p to get better performance against σ − p while also keeping exploitability low.

Original: However, the simple mixing strategy cannot ﬁnd such “good" NE strategy so that it will perform much worse in both exploitability and expected payoff.
Revised: However, the simple mixing strategy cannot ﬁnd such “good" NE strategy so that it will perform much worse in both exploitability and expected payoff.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: Our experiment is done in Leduc Hold’em (Southey et al., 2005) and
Revised: Our experiment is done in Leduc Hold’em (Southey et al., 2005) and

Original: Flop Hold’em Poker (FHP)(Brown et al., 2019).
Revised: Flop Hold’em Poker (FHP) (Brown et al., 2019).

Original: Leduc Hold’em is a smaller-scale poker games and FHP is a larger one.
Revised: Leduc Hold’em is a smaller-scale poker games and FHP is a larger one.

Original: The rules of these two pokers are provided in Appendix B.
Revised: The rules of these two pokers are provided in Appendix B.

Original: We demonstrate the exploitability and evaluation performance of SES against opponents of various strengths.
Revised: We demonstrate the exploitability and evaluation performance of SES against opponents of various strengths.

Original: The exploitability measures a search algorithm’s safety, while head-to-head evaluation measures the ability of opponent exploitation.
Revised: The exploitability measures a search algorithm’s safety, while head-to-head evaluation measures the ability of opponent exploitation.

Original: We also illustrate how estimation accuracy of opponent’s strategy and the exploitation level α impact the results.
Revised: We also illustrate how estimation accuracy of opponent’s strategy and the exploitation level α impact the results.

Revised: Please refer to Appendix C for implementation details.

Original: 5. O PPONENTS
Revised: 5. O PPONENTS

Original: In our experiments, we test the performance of our algorithm against opponents of various strengths.
Revised: In our experiments, we test the performance of our algorithm against opponents of various strengths.

Original: For both Leduc Poker and FHP, we create 3 types of opponents with 3 random seeds each.
Revised: For both Leduc Poker and FHP, we create 3 types of opponents with 3 random seeds each.

Original: The ﬁrst type of opponent is an approximation of NE in the full game, and is regarded as a strong opponent.
Revised: The ﬁrst type of opponent is an approximation of NE in the full game, and is regarded as a strong opponent.

Original: It is computed in the same way as the blueprint strategy with different seed.
Revised: It is computed in the same way as the blueprint strategy with different seed.

Original: For the second and third type of opponents, we enumerate every infoset in the blueprint strategy and shift the action distribution randomly with probability Pr shufﬂe = 0 .
Revised: For the second and third type of opponents, we enumerate every infoset in the blueprint strategy and shift the action distribution randomly with probability Pr shufﬂe = 0 .

Original: 3 or 0 .
Revised: 3 or 0 .

Original: We multiply the probability of each action by a random variable from Uniform (0 , 1) , and then re-normalize the probability distribution.
Revised: We multiply the probability of each action by a random variable from Uniform (0 , 1) , and then re-normalize the probability distribution.

Original: The procedure is motivated by Brown et al.
Revised: The procedure is motivated by Brown et al.

Original: (2018), in which such method is applied to create a number of diverse but reasonably strong agents.
Revised: (2018), in which such method is applied to create a number of diverse but reasonably strong agents.

Original: Even when Pr shufﬂe = 0 . 7 , the strategy keeps close to NE with average L1 distance of each infoset 0.132 comparing to 1.036 of a random strategy to NE.
Revised: Even when Pr shufﬂe = 0 . 7 , the strategy keeps close to NE with average L1 distance of each infoset 0.132 comparing to 1.036 of a random strategy to NE.

Original: So they are regarded as opponents who are not fully rational but with competitive strength.
Revised: So they are regarded as opponents who are not fully rational but with competitive strength.

Original: 5. L EDUC P OKER

Original: In Leduc Poker, we solve for a blueprint strategy using a variant of CFR algorithm (Lanctot et al., 2009; Tammelin et al., 2015) with 1,000,000 iterations in the full game.

Original: Then we apply search in subgames when the board card is dealt.

Original: [Figure]

Original: 5. F LOP H OLD ’ EM P OKER (FHP)
Revised: 5. S AFE O PPONENT S EARCH

Original: As for FHP, there are 1,286,792 different infosets for each betting sequence.

Original: We cluster them into 200 infosets by an abstraction algorithm (Johanson et al., 2013) in order to make equilibrium ﬁnding feasible.

Original: Then, we compute a blueprint strategy in this abstraction.

Original: We apply search immediately once the ﬂop cards are dealt.

Original: In Figure 3, we demonstrate the head-to-head evaluation performances and corresponding exploitability of the reﬁned strategies found by SES against opponents of various strengths, under different exploitation level α and estimation errors of opponent’s strategy.
Revised: In Figure 3, we demonstrate the head-to-head evaluation performances and corresponding exploitability of the reﬁned strategies found by SES against opponents of various strengths, under different exploitation level α and estimation errors of opponent’s strategy.

Original: Different lines in each plot refers to corresponding estimation error (cid:15) , which is the L1 distance of ˆ p and p .
Revised: Different lines in each plot refers to corresponding estimation error (cid:15) , which is the L1 distance of ˆ p and p .

Original: We evaluate our reﬁned  strategy when (cid:15)
Revised: We evaluate our reﬁned [Figure] strategy when (cid:15)

Original: = 0 . 0 , 0 . 3 , 0 .
Revised: = 0 . 0 , 0 . 3 , 0 .

Original: The blue line is the result of blueprint strategy without conducting any search.
Revised: The blue line is the result of blueprint strategy without conducting any search.

Original: Generally speaking, SES balances between safety and opponent exploitation.
Revised: Generally speaking, SES balances between safety and opponent exploitation.

Original: The increase of exploitation level α helps win more chips from opponents, while resulting in the increase of the strategy’s own exploitability.
Revised: The increase of exploitation level α helps win more chips from opponents, while resulting in the increase of the strategy’s own exploitability.

Original: As can be seen in Figure 3, the exploitability increases when the exploitation level α grows from 0 to 1 , which is consistent with Theorem 1.
Revised: As can be seen in Figure 3, the exploitability increases when the exploitation level α grows from 0 to 1 , which is consistent with Theorem 1.

Original: One exception is in FHP when (cid:15) is small: the exploitability surprisingly keeps decreasing even if SES puts more emphasis on opponent exploitation.
Revised: One exception is in FHP when (cid:15) is small: the exploitability surprisingly keeps decreasing even if SES puts more emphasis on opponent exploitation.

Original: Similar situations have also occurred in previous literature (Brown &
Revised: Similar situations have also occurred in previous literature (Brown &

Original: Sandholm, 2017).
Revised: Sandholm, 2017).

Original: The reason is that our opponent is quite close to NE outside the subgame which will make ˆ p close to ˜ p when (cid:15) is small, which means the τ in Theorem 1 is small.
Revised: The reason is that our opponent is quite close to NE outside the subgame which will make ˆ p close to ˜ p when (cid:15) is small, which means the τ in Theorem 1 is small.

Original: As a result, we will have a low-exploitability resolved strategy when using unsafe search and the exploitability increases as (cid:15) increases.
Revised: As a result, we will have a low-exploitability resolved strategy when using unsafe search and the exploitability increases as (cid:15) increases.

Original: When the estimation is completely correct ( (cid:15) = 0 . 0 , the yellow line), the expected payoff in FHP increases as the exploitation level α grows higher.
Revised: When the estimation is completely correct ( (cid:15) = 0 . 0 , the yellow line), the expected payoff in FHP increases as the exploitation level α grows higher.

Original: In Leduc poker, since the game is very small, the pre-computed blueprint is very close to NE.
Revised: In Leduc poker, since the game is very small, the pre-computed blueprint is very close to NE.

Original: Therefore, when confronted with relatively strong opponents ( Pr shuﬄe = 0 . 0 , 0 . 3 ) which are also close to NE, actually few things can be done other than sticking with the blueprint.
Revised: Therefore, when confronted with relatively strong opponents ( Pr shuﬄe = 0 . 0 , 0 . 3 ) which are also close to NE, actually few things can be done other than sticking with the blueprint.

Original: So the improvement introduced by SES is small.
Revised: So the improvement introduced by SES is small.

Original: When facing relatively weak opponent ( Pr shuﬄe = 0 . 7 ), the improvement margin is slightly larger.
Revised: When facing relatively weak opponent ( Pr shuﬄe = 0 . 7 ), the improvement margin is slightly larger.

Original: SES relies on an estimation of opponent’s strategy.
Revised: SES relies on an estimation of opponent’s strategy.

Original: In order to test the robustness of our algorithm when the prediction of p ( I i 1 ) is not accurate, we evaluate the performance of our algorithm with different values of estimation error (cid:15) .
Revised: In order to test the robustness of our algorithm when the prediction of p ( I i 1 ) is not accurate, we evaluate the performance of our algorithm with different values of estimation error (cid:15) .

Original: As illustrated in Figure.3, the exploitability increases and the expected payoff drops when (cid:15) grows larger.
Revised: As illustrated in Figure.3, the exploitability increases and the expected payoff drops when (cid:15) grows larger.

Original: The result is expected since an accurate estimation always provides beneﬁts.
Revised: The result is expected since an accurate estimation always provides beneﬁts.

Original: However, it also demonstrates that SES can still achieve a trade-off between safety and opponent exploitation even when (cid:15) is considerably high.
Revised: However, it also demonstrates that SES can still achieve a trade-off between safety and opponent exploitation even when (cid:15) is considerably high.

Original: For instance, in FHP, (cid:15) is between 0 and 2, and (cid:15)
Revised: For instance, in FHP, (cid:15) is between 0 and 2, and (cid:15)

Original: = 1 . 2 means that the predicted distribution is almost random.
Revised: = 1 . 2 means that the predicted distribution is almost random.

Original: When (cid:15) ≤ 0 .
Revised: When (cid:15) ≤ 0 .

Original: 6 , the expected payoff still keeps increasing with respect to α .
Revised: 6 , the expected payoff still keeps increasing with respect to α .

Original: In case of a bad estimation, we can always choose smaller α to ensure safety.
Revised: In case of a bad estimation, we can always choose smaller α to ensure safety.

Revised: 5. C OMPARISON WITH R ESTRICTED N ASH R ESPONSE

Revised: [Figure]

Revised: We also compare SES with restricted Nash response (RNR) (Johanson et al., 2007) in FHP.

Revised: In each round, we limit the computation time of RNR(normal) to 10s, which is the same for SES.

Revised: However, as stated in section 2, RNR needs to recompute a strategy for the whole game in each round.

Original: C ONCLUSION
Revised: It cannot converge in 10s. So we also compare with RNR(big), which has a budget of 10M CFR iterations in each round (around 190s in time). In contrast, SES only uses 10M CFR iterations to calculate its blueprint once. As is shown in Figure 4, SES signiﬁcantly outperforms RNR(normal) in both exploitability and evaluation.

Revised: SES also achieves much lower exploitability than RNR(big) and comparable evaluation results with much less computation time.

Revised:  C ONCLUSION

Original: We propose a novel Safe Exploitation Search (SES) algorithm which uniﬁes both safe search and opponent exploitation.
Revised: We propose a novel safe exploitation search (SES) algorithm which uniﬁes both safe search and opponent exploitation.

Revised: With the aid of real-time search, SES can make online adaptations to a changing opponent model.

Original: We also prove safety and opponent exploitation guarantees of SES in Theorem 1 and Theorem 2.
Revised: We also prove safety and opponent exploitation guarantees of SES in Theorem 1 and Theorem 2.

Original: The experimental results in our designed matrix game conﬁrm the existence of the reﬁned strategy which is both safe and actively exploiting the opponent.
Revised: The experimental results in our designed matrix game conﬁrm the existence of the reﬁned strategy which is both safe and actively exploiting the opponent.

Original: In games of poker, our method outperforms NE baselines while keeping exploitability low.
Revised: In games of poker, our method outperforms NE baselines while keeping exploitability low.

Revised: SES is also much more efﬁcient than previous safe exploitation algorithms without search.

Original: The exploitation level α is now regarded as a hyperparameter in our algorithm.
Revised: The exploitation level α is now regarded as a hyperparameter in our algorithm.

Original: However, ideally, α should be learnt automatically from opponents and should be adaptive to opponent’s strategy change.
Revised: However, ideally, α should be learnt automatically from opponents, and should be adaptive to opponent’s strategy change.

Original: We leave this for future work.
Revised: We leave this for future work.

Original: A A PPENDIX A A. P ROOF OF T HEOREM
Revised: A P ROOFS A. P ROOF OF T HEOREM

Original: Theorem 1.
Revised: Theorem 1.

Original: (safety)  S is a disjoint set of subgames S .
Revised: (safety) Let S be a disjoint set of subgames S .

Original: Let σ ∗  2  be  P2’s strategy with lowest exploitability which differs with σ 2 only in S .
Revised: Let σ ∗ = (cid:104) σ ∗ 1 , σ ∗ 2 (cid:105) be the NE where P2’s strategy is constrained to be the same with σ 2 outside S .

Original: Deﬁne ∆ = max S ∈ S ,I i 1 ∈ S top | CBV σ ∗ 2 1 ( I i 1 ) − v σ 

Revised: Deﬁne ∆ = max S ∈ S ,I i 1 ∈ S top | CBV σ ∗ 2 1 ( I i 1 ) − v σ

Original: ( I i 1 ) | .
Revised: ( I i 1 ) | .

Original: Let ˜ p
Revised: Let ˜ p

Original: ( I i 1 ) be thereal reach probability, and ˆ p ( I i 1 ) be its estimation.
Revised: ( I i 1 ) be the reach probability given by σ ∗ 1 and P2’s blueprint.

Original: Deﬁne τ = max S ∈ S ,I i 1 ∈ S top | ˆ p ( I i 1 ) − ˜ p 
Revised: Let ˆ p ( I i 1 ) be the estimation of reach probability p ( I i 1 ) calculated by the real opponent strategy σ 1 . Deﬁne τ =

Original: ( I i 1 ) ˜
Revised: [Equation]

Original: p ( I i 1 )

Original: | .
Revised: | .

Original: Whenever 1 − (2 τ + 1) α > 0 , we have a bounded exploitability given by:
Revised: Whenever 1 − (2 τ + 1) α > 0 , we have a bounded exploitability given by:

Original: [Equation]
Revised: [Equation]

Original: Proof
Revised: Proof:

Original: For simplicity, we will omit the subscript of CBV σ ∗ 2by default.
Revised: For simplicity, we will omit the subscript of CBV σ ∗ 2by default.

Original: In order to prove Theorem 1, we will use mathematical induction on the level of the infoset.
Revised: In order to prove Theorem 1, we will use mathematical induction on the level of the infoset.

Original: The depth L has the same deﬁnition as in
Revised: The depth L has the same deﬁnition as in

Original: Brown & Sandholm (2017), i.e.
Revised: Brown & Sandholm (2017), i.e.

Original: • For all the infosets which are direct parents of the subgames, we deﬁne L ( I ) = 0 .
Revised: • For all the infosets which are direct parents of the subgames, we deﬁne L ( I ) = 0 .

Original: • For the infosets that are not ancestors of the subgames, we deﬁne L ( I ) = 0 .
Revised: • For the infosets that are not ancestors of the subgames, we deﬁne L ( I ) = 0 .

Original: • For any infosets that are ancestors of the subgames, we deﬁne
Revised: • For any infosets that are ancestors of the subgames, we deﬁne

Original: [Equation]
Revised: [Equation]

Original: I

Original: [Equation]

Original: L

Original: [Equation]

Original: I

Original: [Equation]

Original: ) + 1 .
Revised: ) + 1 .

Original: That is, it has a higher level than any of its successors.
Revised: That is, it has a higher level than any of its successors.

Original: B ASE C ASE OF I NDUCTION
Revised: B ASE C ASE OF I NDUCTION

Original: Firstly, we will prove that for any infoset with level 0, the inequality of theorem 1 holds.
Revised: Firstly, we will prove that for any infoset with level 0, the inequality of theorem 1 holds.

Original: For convenience, we consider that theorem 1 in a speciﬁc subgame S .
Revised: For convenience, we consider that theorem 1 in a speciﬁc subgame S .

Original: We will prove the infoset at the top of the subgame ﬁrst.
Revised: We will prove the infoset at the top of the subgame ﬁrst.

Original: Since SE ( σ ∗ 2 ) ≥ (1 − α )( − ∆)
Revised: Since SE ( σ ∗ 2 ) ≥ (1 − α )( − ∆)

Original: + α (cid:80) i ˆ p
Revised: + α (cid:80) i ˆ p

Original: ( I i 1 )( − ∆)
Revised: ( I i 1 )( − ∆)

Original: = − ∆ , we have
Revised: = − ∆ , we have

Original: [Equation]
Revised: [Equation]

Original: ) since σ S 2 = arg max ˜ σ 2 SE (˜ σ 2 ) .
Revised: since σ S 2 = arg max ˜ σ 2 SE (˜ σ 2 ) .

Original: Furthermore, we have(cid:88)
Revised: Furthermore [Equation] we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) where the second term (cid:80) i ˜ p
Revised: where the second term (cid:80) i ˜ p

Original: ( I i 1 )( CBV σ ∗ 2 ( I i 1 ) − CBV σ S 2 ( I i 1 )) is no larger than 0 because (cid:80)i ˜
Revised: ( I i 1 )( CBV σ ∗ 2 ( I i 1 ) − CBV σ S 2 ( I i 1 )) is no larger than 0 because (cid:80)

Original: p
Revised: i

Original: ( I 1 ) CBV σ ∗ 2 ( I i 1 ) is exactly what σ ∗ 2 minimized.
Revised: [Equation] 2 ( I i 1 ) is exactly what σ ∗ 2 minimized.

Original: Otherwise, σ ∗ 2 can change the strategy in the subgame so that he will get higher reward against σ ∗ 1 which conﬂicts the deﬁnition of NE.
Revised: Otherwise, σ ∗ 2 can change the strategy in the subgame so that he will get higher reward against σ ∗ 1 which conﬂicts the deﬁnition of NE.

Original: And we will further decompose I i 1 ∈ S top into two parts, { I i, − 1 } and { I i, + 1 } .
Revised: And we will further decompose I i 1 ∈ S top into two parts, { I i, − 1 } and { I i, + 1 } .

Original: They have the property that CBV σ ∗ 2 ( I i,
Revised: They have the property that CBV σ ∗ 2 ( I i,

Original: [Equation]
Revised: [Equation]

Original: 1 ) − CBV σ S 2 ( I i, − 1 ) ≤ 0 and CBV σ
Revised: 1 ) − CBV σ S 2 ( I i, − 1 ) ≤ 0 and CBV σ

Original: [Equation]
Revised: [Equation]

Original: 2 ( I i, + 1 ) − CBV σ S 2 ( I i, + 1 ) > 0 .
Revised: 2 ( I i, + 1 ) − CBV σ S 2 ( I i, + 1 ) > 0 .

Original: And since (cid:80) i ˜ p
Revised: And since (cid:80) i ˜ p

Original: [Equation]
Revised: [Equation]

Original: I i 1 )
Revised: I i 1 )

Original: [Equation]
Revised: [Equation]

Original: CBV
Revised: CBV

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: I i
Revised: I i

Original: [Equation]
Revised: [Equation]

Original: − CBV
Revised: − CBV

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: I i 1 )) ≤ 0 as discussed above, we have
Revised: I i 1 )) ≤ 0 as discussed above, we have

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: 8) which implies that (cid:88)
Revised: 8) which implies that (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: The last inequation holds since min I j 1 (cid:16) CBV σ ∗ 2 ( I j 1 ) − CBV σ S 2 ( I j 1 ) (cid:17) ≤ 0 since σ ∗ 2 is the strategy with lowest exploitability by only changing strategy of σ 2 in the subgames.
Revised: The last inequation holds since min I j 1 (cid:16) CBV σ ∗ 2 ( I j 1 ) − CBV σ S 2 ( I j 1 ) (cid:17) ≤ 0 since σ ∗ 2 is the strategy with lowest exploitability by only changing strategy of σ 2 in the subgames.

Original: Back to Equation
Revised: Back to Equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: And substitute it into Equation 6,
Revised: And substitute it into Equation 6,

Original: [Equation]
Revised: [Equation]

Original: ≥ − ∆
Revised: ≥ − ∆

Original: [Equation]
Revised: [Equation]

Original: so that
Revised: so that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for all I j 1 in the subgame.
Revised: for all I j 1 in the subgame.

Original: And for infoset I out of the subgame with level 0, since the reﬁned strategy σ S 2 and blueprint strategy σ 2 are the same here, the CBV value is exactly the same and the inequality holds.
Revised: And for infoset I out of the subgame with level 0, since the reﬁned strategy σ S 2 and blueprint strategy σ 2 are the same here, the CBV value is exactly the same and the inequality holds.

Original: I NDUCTIVE S TEP
Revised: I NDUCTIVE S TEP

Original: The inductive step mostly follows that of Brown & Sandholm (2017).
Revised: The inductive step mostly follows that of Brown & Sandholm (2017).

Original: Since CBV
Revised: Since CBV

Original: [Equation]
Revised: [Equation]

Original: ( I
Revised: ( I

Original: [Equation]
Revised: [Equation]

Original: ≤ CBV
Revised: ≤ CBV

Original: [Equation]
Revised: [Equation]

Original: ( I
Revised: ( I

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1) α ∆ holds for every subgame S , σ (cid:48) 2 will also satisfy this inequation since ∆ and τ are deﬁned as maximum over all subgames.
Revised: 1) α ∆ holds for every subgame S , σ (cid:48) 2 will also satisfy this inequation since ∆ and τ are deﬁned as maximum over all subgames.

Original: Now, suppose CBV σ (cid:48)
Revised: Now, suppose CBV σ (cid:48)

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: 1) α ∆ holds for any infoset with level lower or equal to k , we will prove that it also holds for infoset with level k + 1 .
Revised: 1) α ∆ holds for any infoset with level lower or equal to k , we will prove that it also holds for infoset with level k + 1 .

Original: By deﬁnition of CBV ( I 1 ) ,
Revised: By deﬁnition of CBV ( I 1 ) ,

Original: [Equation]
Revised: [Equation]

Original: We can swap the two summations above since the game is perfect recall, then
Revised: We can swap the two summations above since the game is perfect recall, then

Original: [Equation]
Revised: [Equation]

Original: By substituting the deﬁnition of CBV ( I (cid:48) 1 ) into the equation above,
Revised: By substituting the deﬁnition of CBV ( I (cid:48) 1 ) into the equation above,

Original: [Equation]
Revised: [Equation]

Original: And by the induction hypothesis
Revised: And by the induction hypothesis

Original: [Equation]
Revised: [Equation]

Original: Because I 1 is out of the subgame and σ ∗
Revised: Because I 1 is out of the subgame and σ ∗

Original: [Equation]
Revised: [Equation]

Original: σ 2 is exactly the same outside the subgame
Revised: σ 2 is exactly the same outside the subgame

Original: [Equation]
Revised: [Equation]

Original: we will get
Revised: we will get

Original: [Equation]
Revised: [Equation]

Original: Finally, by mathematical induction we get
Revised: Finally, by mathematical induction we get

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Theorem 2.
Revised: Theorem 2.

Original: (opponent exploitation) Let (cid:15) =
Revised: (opponent exploitation) Let (cid:15) =

Original: (cid:107) ˆ p − p (cid:107) 1 be the L1 distance of the distribution p
Revised: (cid:107) ˆ p − p (cid:107) 1 be the L1 distance of the distribution p

Original: ( I i 1 ) and ˆ p ( I i 1 ) .
Revised: ( I i 1 ) and ˆ p ( I i 1 ) .

Original: Let η = min S ∈ S max I j 1 ∈ S top (cid:16) CBV 1 ( I j 1 , σ S 2 ) − CBV 1 ( I j 1 , σ ∗ 2 ) (cid:17) ≥ 0 .
Revised: Let η = min S ∈ S max I j 1 ∈ S top (cid:16) CBV 1 ( I j 1 , σ S 2 ) − CBV 1 ( I j 1 , σ ∗ 2 ) (cid:17) ≥ 0 .

Original: We use BR [ S ,σ p ] p ( σ ) to denote the strategy for player p which maximizes its utility in subgame S ∈ S against σ − p under the constraint that BR [ S ,σ p ] p ( σ ) and σ p differs only inside S .
Revised: We use BR [ S ,σ p ] p ( σ ) to denote the strategy for player p which maximizes its utility in subgame S ∈ S against σ − p under the constraint that BR [ S ,σ p ] p ( σ ) and σ p differs only inside S .

Original: By maximizing objective 2, for all S ∈ S , the reﬁned strategy σ (cid:48) 2 satisﬁes u
Revised: By maximizing objective 2, for all S ∈ S , the reﬁned strategy σ (cid:48) 2 satisﬁes u

Original: [Equation]
Revised: [Equation]

Original: Proof: Still, we only consider a speciﬁc subgame S ﬁrst.
Revised: Proof: Still, we only consider a speciﬁc subgame S ﬁrst.

Original: σ S 2 is maximizing
Revised: σ S 2 is maximizing

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: So, we have
Revised: So, we have

Original: [Equation]
Revised: [Equation]

Original: 22) and
Revised: 22) and

Original: [Equation]
Revised: [Equation]

Original: Therefore, αf
Revised: Therefore, αf

Original: [Equation]
Revised: [Equation]

Original: 24) which means (cid:88)
Revised: 24) which means (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ⇔
Revised: ⇔

Revised: [Equation]

Revised: ⇔

Original: [Equation]
Revised: [Equation]

Original: Since η is deﬁned as minimum over all subgames
Revised: Since η is deﬁned as minimum over all subgames, we have

Original: [Equation]
Revised: [Equation]

Original: we have

Original: [Equation]
Revised: [Equation]

Original: B A PPENDIX B R ULES OF L EDUC P OKER
Revised: B P OKER R ULES R ULES OF L EDUC P OKER

Original: Leduc Poker is a two players game.
Revised: Poker is a two players game.

Original: In Leduc Poker, there are 6 cards in total, three ranks( { J, Q, K } ) with two suits( { a, b } ) each.
Revised: In Leduc Poker, there are 6 cards in total, three ranks( { J, Q, K } ) with two suits( { a, b } ) each.

Original: And at the beginning, every player should put 1 chip into the pot and then will be dealt with one private card.
Revised: And at the beginning, every player should put 1 chip into the pot and then will be dealt with one private card.

Original: Then, two players alternatively bet.
Revised: Then, two players alternatively bet.

Original: They can call, raise and fold.
Revised: They can call, raise and fold.

Original: If any of them fold, the game ends and all chips in the pot belongs to the other player.
Revised: If any of them fold, the game ends and all chips in the pot belongs to the other player.

Original: And when a player call, he has to put chips in the pot to ensure that he contributes equal chips as the other player in the pot.
Revised: And when a player call, he has to put chips in the pot to ensure that he contributes equal chips as the other player in the pot.

Original: When a player raise, he has to ensure that he contributes more chips than the other player in the pot.
Revised: When a player raise, he has to ensure that he contributes more chips than the other player in the pot.

Original: A betting round ends when a player calls.
Revised: A betting round ends when a player calls.

Original: Leduc Poker is divided into two betting rounds.
Revised: Leduc Poker is divided into two betting rounds.

Original: In the ﬁrst round, a private card is dealt to each player and then two player start to bet.
Revised: In the ﬁrst round, a private card is dealt to each player and then two player start to bet.

Original: After the ﬁrst betting round ends and nobody folds, a public card is dealt on board and the second betting round starts.
Revised: After the ﬁrst betting round ends and nobody folds, a public card is dealt on board and the second betting round starts.

Original: When the second round ends, both of the player show their private hands and the stronger hands win.
Revised: When the second round ends, both of the player show their private hands and the stronger hands win.

Original: If a player’s private card has the same rank as the public card, then he wins.
Revised: If a player’s private card has the same rank as the public card, then he wins.

Original: Otherwise, we have J < Q < K and the higher one wins.
Revised: Otherwise, we have J < Q < K and the higher one wins.

Original: And in each betting round, there will be at most two raises in our experiment and each raise should contribute 1 more chip in the ﬁrst round and 2 more chips in the second round.
Revised: And in each betting round, there will be at most two raises in our experiment and each raise should contribute 1 more chip in the ﬁrst round and 2 more chips in the second round.

Original: R ULES OF F LOP H OLD ’ EM P OKER
Revised: R ULES OF F LOP H OLD ’ EM P OKER

Original: The rules of Flop Hold’em Poker is similar to that of Leduc Poker.
Revised: The rules of Flop Hold’em Poker is similar to that of Leduc Poker.

Original: In FHP, we use the standard 52-card deck.
Revised: In FHP, we use the standard 52-card deck.

Original: At the beginning, the ﬁrst player will contribute 1 chip to the pot and the second player will contribute 2 chips.
Revised: At the beginning, the ﬁrst player will contribute 1 chip to the pot and the second player will contribute 2 chips.

Original: And then they will be dealt with 2 private cards each and the ﬁrst player start to bet.
Revised: And then they will be dealt with 2 private cards each and the ﬁrst player start to bet.

Original: There are still two betting rounds and the raise sizes are both 2 chips.
Revised: There are still two betting rounds and the raise sizes are both 2 chips.

Original: At the end of the ﬁrst betting round, there will be 3 public cards dealt on board.
Revised: At the end of the ﬁrst betting round, there will be 3 public cards dealt on board.

Original: And the players will show their private card at the end of the second betting round and the larger one wins the game.
Revised: And the players will show their private card at the end of the second betting round and the larger one wins the game.

Original: In FHP, we have the same rule of card order as a standard Texas hold ’em .
Revised: In FHP, we have the same rule of card order as a standard Texas hold ’em .

Revised: C I MPLEMENTATION D ETAILS

Revised: Leduc Poker.

Revised: In Leduc Poker, we solve for a blueprint strategy using a variant of CFR algorithm (Lanctot et al., 2009; Tammelin et al., 2015) with 1M iterations in the full game.

Revised: Then we apply search in subgames when the board card is dealt.

Revised: Flop Hold’em Poker (FHP).

Revised: As for FHP, there are 1,286,792 different infosets for each betting sequence.

Revised: We cluster them into 200 infosets by an abstraction algorithm (Johanson et al., 2013) in order to make equilibrium ﬁnding feasible.

Revised: Then, we compute a blueprint strategy in this abstraction with 10,000,000 iterations.

Revised: We apply search immediately once the ﬂop cards are dealt.