L AZY -CFR: FAST AND NEAR - OPTIMAL REGRET MINIMIZATION FOR EXTENSIVE GAMES WITH IMPERFECT INFORMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Counterfactual regret minimization (CFR) methods are effective for solving twoplayer zero-sum extensive games with imperfect information.
However, the vanilla
CFR has to traverse the whole game tree in each round, which is time-consuming in large-scale games.
In this paper, we present Lazy-CFR, a CFR algorithm that adopts a lazy update strategy to avoid traversing the whole game tree in each round.
We prove that the regret of Lazy-CFR is almost the same as the regret of the vanilla
CFR and only needs to visit a small portion of the game tree.
Thus, Lazy-CFR is provably faster than CFR.
Empirical results consistently show that Lazy-CFR is fast in practice.
I NTRODUCTION
Extensive games provide a mathematical framework for modeling the sequential decision-making problems with imperfect information, which is common in economic decisions, negotiations and security.
We focus on solving t wo-player zero-sum e xtensive g ames with i mperfect information (TEGI).
In a TEGI, there is an environment with uncertainty and two players on opposite sides (Koller & Megiddo, 1992).
Counterfactual regret minimization (CFR) (Zinkevich et al., 2008) provides a state-of-the-art approach for solving TEGIs with much progress in practice (Brown & Sandholm, 2017b; Moravˇcík et al., 2017; Brown & Sandholm, 2019a).
Regret minimization techniques are ﬁrst introduced to solve
TEGIs based on the observation that minimizing the regrets of both players makes the time-averaged strategy converge to the Nash Equilibrium (NE) (Nisan et al., 2007).
CFR (Zinkevich et al., 2008) further bounds the original regret with a summation of many immediate counterfactual regrets on each information set (infoset).
These immediate counterfactual regrets are deﬁned by the counterfactual rewards and can be iteratively minimized by existing online learning algorithms, e.g., regret matching (RM) (Blackwell et al., 1956).
A limitation of CFR is that it requires traversing the whole game tree in each round, which is timeconsuming in large-scale games due to the fact that we have to apply RM to every immediate regret in each round.
Existing works on avoiding traversing the whole game tree can be mainly divided into two categories: Pruning-based CFR (Brown & Sandholm, 2015; 2017a) and Monte-Carlo CFR (MC-CFR) (Lanctot et al., 2009).
These algorithms can signiﬁcantly speed up the vanilla CFR in practice.
However, pruning-based algorithms may degenerate in the worst case.
And the performance of MC-CFR depends on the structure of the game and the chosen online learning algorithm.
Contributions:
In this paper, we explore another approach to address the problem.
• We present an algorithm named Lazy-CFR, which exploits a lazy update technique to avoid traversing the whole game tree.
Lazy-CFR divides the time horizon into segments,
i
e., disjoint subsets with consecutive elements, and updates the strategy only at the beginning of each segment and keeps the strategy the same within each segment.
By this way we only need to access each infoset at the beginning of each segment.
• We then extend the analysis on the regret of the vanilla CFR in (Burch, 2018) to LazyCFR.
We show that the regret of Lazy-CFR is almost the same as that of the vanilla CFR.
Combining with the analysis on the number of updates in Lazy-CFR, we
show
that LazyCFR converges much faster than the vanilla CFR in theory, with some extra cost of memory.
• Finally, we bound
the regret from
below.
By comparing with the regret lower bound, we show that the regret upper bounds of Lazy-CFR and the vanilla CFR are near-optimal.
We empirically evaluate Lazy-CFR, the vanilla CFR, MC-CFR (Lanctot et al., 2009), CFR+ (Bowling et al., 2017) and MC-CFR+ on the standard benchmarks, Leduc Hold’em (Southey et al., 2005) and heads-up ﬂop hold’em poker (Brown et al., 2019).
It is noteworthy that the same idea of Lazy-CFR can also be applied to CFR+, and we name the resulted algorithm Lazy-CFR+.
The analysis on
Lazy-CFR can be directly applied to Lazy-CFR+.
Experiments show that Lazy-CFR and Lazy-CFR+ works well in practice as suggested by the theory.
N OTATIONS AND P RELIMINARIES
We ﬁrst introduce the notations and deﬁnitions of extensive games and TEGIs.
Then we introduce an online learning concept of regret minimization, followed by CFR, which is based on the connection between TEGIs and regret minimization.
For clarity, most important notations appeared in this work and their descriptions are listed in the look-up table in appendix A.
2. E XTENSIVE GAMES
Extensive games (see Osborne & Rubinstein, 1994, pg.
200 for a formal deﬁnition) compactly model the decision-making problems with sequential interactions among multiple agents.
An extensive game can be represented by a game tree H of histories, where a history is a sequence of actions in the past.
Suppose that there are N players participating in an extensive game and let c denote the chance player which is usually used to model the uncertainty in the environment.
Let [ N ] := { 1 , · · · , N } .
A player function P is a mapping from H to [ N ] ∪ { c } such that P ( h ) is the player who takes an action after h .
And each player i ∈ [ N ] receives a reward u
i ( h ) ∈ [ − 1 , 1] at a terminal history h .
Let A ( h ) denote the set of valid actions of P ( h ) after h , that is, ∀ a ∈ A ( h ) , ( h, a ) ∈ H .
Let
A = max h |A ( h ) | .
A strategy of player i is a function σ i that assigns h a distribution over A ( h ) if
P ( h ) =
i .
A strategy proﬁle σ consists of the strategy for each player, that is, σ = { σ 1 , · · · , σ N } .
We will use σ − i to refer to all the strategies in σ except σ i .
And we use the pair ( σ i , σ − i ) to denote the full strategy proﬁle.
In games with imperfect information, actions of other players are partially observable to player i ∈
[ N ] .
So for player i , some different histories may not be distinguishable.
Thus, the game tree can be partitioned into disjoint information sets (infoset).
Let I i denote the collection of player i ’s infosets.
We have that two histories h, h (cid:48) ∈
I ∈
I i are not distinguishable to player i .
Thus, σ i must assign the same distribution over actions to all histories in infoset I ∈
I i
if P ( I ) = i .
With a little abuse of notations, we let σ i ( I ) denote the strategy of player i on infoset
I ∈
I i .
It is clear that infosets of a player also form a tree, called infoset tree.
We let π σ ( h ) denote the probability of arriving at history h .
Obviously, we can decompose π σ ( h ) into the product of each player’s contribution, that is, π σ ( h ) =
(cid:81)
[ N ] ∪{ c } π iσ ( h ) .
Similarly, we deﬁne π σ ( I ) =
(cid:80)
h ∈
I π σ ( h ) as the probability of arriving at infoset I and let π iσ ( I ) denote the corresponding contribution of player i .
Let π − i σ ( h ) and π − i σ ( I ) denote the product of the contributions on arriving at h and I , respectively, of all players except player i .
With the above notations, a two-player zero-sum extensive game with imperfect information (TEGI) is an extensive game with N = 2 and u 1 ( h ) + u 2 ( h )
=
for all terminal histories.
In game theory, the solution of a game is often referred to a Nash equilibrium (NE) (Osborne &
Rubinstein, 1994).
In this paper, we concern on computing an approximation of an NE, namely an (cid:15) -NE
(Nisan et al., 2007).
With a little abuse of notations, let u i ( σ ) denote the expected reward of player
i if all players take actions according to σ .
An (cid:15) -NE is a strategy proﬁle
σ such that ∀ i ∈ [ N ] , u i ( σ ) ≥ max σ (cid:48) ,i u i (( σ (cid:48) ,i , σ − i ))
− (cid:15).
And the (cid:15) -NE in a TEGI can be efﬁciently computed by regret minimization; see later in this section.
2. R EGRET MINIMIZATION
Now we introduce regret , a core concept in online learning (Cesa-Bianchi & Lugosi, 2006).
Many powerful online learning algorithms can be framed as minimizing some kinds of regret, therefore known as regret minimization algorithms.
Generally, the regret is deﬁned as follows:
Deﬁnition 1 (Regret) .
Consider the case where a player takes actions repeatedly.
At each round, the player selects an action w t ∈ Σ , where Σ is the set of valid actions.
At the same time, the environment selects a reward function f t .
Then, the overall reward of the player is (cid:80) Tt
=1 f t ( w t ) , and the regret is deﬁned as R T = max w (cid:48) ∈ Σ
(cid:80)
Tt =1 f t ( w (cid:48) ) − (cid:80) Tt =1 f t ( w t ) .
One important example of online learning is online linear optimization (OLO) in which f t is a linear function.
If Σ is the set of distributions over some discrete set, an OLO can be solved by standard regret minimization algorithms, e.g., regret matching (RM) (Blackwell et al., 1956;
Abernethy et al., 2011) or AdaHedge (Freund & Schapire, 1997).
As CFR employs RM or AdaHedge as a sub-procedure, we summarize them as follows:
Deﬁnition 2 (Online linear optimization (OLO), regret matching (RM) and AdaHedge) .
Consider the online learning problem with linear rewards.
In each round t , an agent plays a mixed strategy w t ∈ ∆( A ) , where ∆( A ) is the set of distributions, while an adversary selects a vector c t ∈ R |A| .
The reward of the agent at this round is (cid:104) w t , c t (cid:105) where (cid:104)· , ·(cid:105) denotes the operator of inner product.
The goal of the agent is to minimize the regret: R oloT = max w ∈ ∆( A ) (cid:80)
Tt =1 (cid:104) w, c t (cid:105) − (cid:80)
Tt =1 (cid:104) w t , c t (cid:105) .
Let R oloT, + ( a ) = max
[Equation] [Equation]
=
[Equation]
t
[Equation]
a ) − (cid:80) Tt =1 (cid:104) w t , c t (cid:105) ) , in RM, w t +1 ( a ) =
R olot, + ( a ) / (cid:80) a (cid:48)
R olot
[Equation]
+ ( a (cid:48) ) , if max a (cid:48) R olot, + ( a (cid:48) ) > 0 , and w t +1 ( a ) = 1 |A| otherwise.
According to the result in (Blackwell et al., 1956), RM has the following regret bound:
[Equation]
Let s t ( a ) =
exp( − η t (cid:80) tt (cid:48)
=1 c t (cid:48) ( a )) , AdaHedge picks w t ( a ) =
s t ( a ) / ( (cid:80) a (cid:48)
s t ( a (cid:48) )) , where η t is the learning rate that can be tuned adaptively (De Rooij et al., 2014).
According to (De Rooij et al., 2014), Adahedge has the regret bound R oloT ≤ O (cid:18)(cid:113) log |A| (cid:80)
Tt =1 max a ∈A c 2 t ( a ) (cid:19) .
2. C OUNTERFACTUAL REGRET MINIMIZATION (CFR)
CFR is developed on a connection between (cid:15) -NE and regret minimization.
This connection is naturally established by considering repeatedly playing a TEGI as an online learning problem.
It is worthy to note that there are two online learning problems in a TEGI, one for each player.
Suppose player i takes σ it at time step t and let σ t = ( σ 1 t , σ 2 t ) .
Consider the online learning problem for player i by setting w t :
=
σ it and f it ( σ i ) :
= u i (( σ i , σ − i t )) .
The regret for player i is
R iT := max σ i R iT ( σ i ) , where R iT ( σ i ) :
= (cid:80) Tt =1 u
i (( σ i , σ − i t )) − (cid:80) Tt =1 u
i (( σ it , σ − i t )) .
Furthermore, consider the time-averaged strategy ¯ σ iT ( I ) :
= (cid:80) t π iσt ( I ) σ it ( I ) (cid:80) t π iσt ( I ) .
Then, it is well-known that :
Lemma 1 ((Nisan et al., 2007)) .
If ∀ i, 1 T R iT ≤ (cid:15)/ 2 , then (¯ σ 1 T , ¯ σ 2 T ) is an (cid:15) -NE.
Speciﬁcally, let σ | I → σ (cid:48)
( I ) denote the strategy generated by modifying σ ( I ) to σ (cid:48)
( I ) and u i ( σ, I ) denote the reward of player i conditioned on arriving at the infoset
I if the strategy σ is executed.
(Zinkevich et al., 2008) decomposes the regret R iT into the summation of immediate regrets as 2 :
[Equation]
Further, (Zinkevich et al., 2008) upper bounds Eq.
(2) by the counterfactual regret :
[Equation]
For convenience, we call π − i σ t ( I )
u
i ( σ t | I → a , I ) the counterfactual reward of action a
at round t .
Notice that Eq.
(3) essentially decomposes the regret of a TEGI into O ( |I| ) OLOs.
So that, in each round, we can apply RM directly to each individual OLO to minimize the counterfactual regret.
And the original regret max σ R iT ( σ ) is also minimized since the counterfactual regret is an upper bound.
L AZY -CFR: A L AZY U PDATE A LGORITHM FOR TEGI S
The above CFR procedure that applies RM to solve each OLO has to traverse the whole game tree, which is very time-consuming in large scale games.
In this section, we present Lazy-CFR, an efﬁcient
CFR algorithm with a lazy update strategy based on the insight that updating the strategy on every infoset is not indispensable.
Intuitively, this is because the regret is determined by the norm of the vector of counterfactual reward on each node (see Eq.
(1)); and on most nodes, the corresponding norm is very small, since π − i σ t is a probability (see Eq.
(3)), thereby can be updated in a lazy manner.
3. L AZY UPDATE FOR OLO S
Their lengths of time are 6 and 2 respectively.
We start by presenting a lazy update strategy for an OLO in Defn.
As illustrated in Fig.
1, a lazy update algorithm for OLOs consists of two steps: (1) It divides the set of time steps [ T ] into n intervals, that is, { t i , t i +1 , · · · , t i +1 − 1 } ni =
where 1 = t 1 < t 2 · · · < t n +1 = T
+ 1 .
(2) It updates w t at time steps t
= t
i for some i and keeps w t the same within each segment.
That is, the OLO with
T steps collapses into a new OLO with n steps and accordingly the vector selected by the adversary in the collapsed OLO at time step j is c (cid:48)
j = (cid:80) t j +1 − 1 t =
t j c t , where c t is the vector selected by the adversary in the original OLO at time step t .
According to the known result in Eq.
(1), the regret of the RM with lazy update (Lazy-RM) is bounded by O ( (cid:112) A (cid:80) ni
=1 max a c (cid:48)
i ( a )) .
Thus, if the segmentation rule is under a reasonable design, that is, (cid:80) ni
=1 max a c (cid:48)
i ( a ) 2 ≈ (cid:80)
Tj =1 max a c j ( a ) 2 , then the regrets of Lazy-RM and the vanilla RM are similar in amount.
As we shall show soon, we can expect a reasonable segmentation rule in CFR.
Input: a constant B > 0 .
while t < T do for all i ∈ { 1 , 2 } do
Q = { I r } where I r is the root of the infosets tree.
while Q is not empty.
do
Pop I from Q .
Update the strategy on I via RM.
For I (cid:48) ∈ γ
( I ) , if m t ( I (cid:48) ) ≥ B , push I (cid:48) into Q , i.e., Q = Q ∪ { I (cid:48) } .
end while end for for all h ∈ H such that the strategy on some history after h has been modiﬁed.
do
Update the reward vector on h .
end for end while
Output the time-averaged strategy.
Though Lazy-RM does not need to update the strategy at each round, a straightforward implementation of Lazy-RM still has a running time of O ( AT ) , which is the same as applying RM directly.
This is because we have to compute (cid:80) t i + t
= t
i +1 c t .
Fortunately, this problem can be addressed in TEGIs by exploiting the structure of the game tree (see Sec. 3.2).
3. L AZY -CFR
We now use lazy update to solve TEGIs.
According to Eq.
(3), the regret minimization procedure can be divided into O ( |I| ) OLOs, one for each infoset.
Speciﬁcally, for each infoset
I ∈
I i , P ( I )
=
i , we divide the set of
[ T ] into n ( I ) segments { t j ( I ) , · · · , t j +1 ( I ) − 1 } n ( I ) j =1 , following step (1).
Let r j
( I, a ) = (cid:80) t j +
( I )
[Equation]
1 t =
[Equation]
j ( I ) π − i
[Equation]
t ( I ) u
i ( σ t | I → a , I ) denote the summation of the counterfactual rewards over segment j , and let r j ( I ) =
[ r j ( I, a )] a ∈A ( I ) denote the vector consisting of r j ( I, a ) .
Similar to lazy update for OLOs, we only update the strategy on infoset I at t j ( I ) .
Let σ (cid:48)
j ( I ) denote the strategy after the j -th update on infoset I , that is, σ t ( I ) :
= σ (cid:48)
j ( I ) for t ∈ [ t j ( I ) , t j +1 ( I ) − 1] .
According to Eq.
(1), we can bound the regret of the collapsed OLO on infoset I as R lazyT ( I ) :=
[Equation]
The above procedure is quite straightforward.
However, as discussed above, in order to have an efﬁcient implementation, one critical step is to deﬁne a proper segmentation rule of each OLO.
Below, we present one rule for Lazy-CFR, with which we can: 1) achieve a regret similar in amount to the regret of the vanilla CFR; 2) avoid updating the whole game tree; and 3) compute r j ( I ) efﬁciently.
Speciﬁcally, let τ t ( I ) denote the last time step we update the strategy on infoset I before time t .
We have τ t ( h ) =
τ t ( I ) for h ∈
I .
Let m t ( I ) :
= (cid:80) tτ =
τ t ( I )+1 π − i σ τ
( I ) denote the summation of reach probabilities after τ t ( I ) , which is contributed by all players except i .
Let subt ( I ) denote the subtree rooted at infoset I .
For convenience, for I ∈
I i
if P ( I ) =
i , we call I a decision point.
Let γ ( I ) denote the set of decision points after I such that every I (cid:48) in the path from I to I (cid:48)(cid:48)
∈ γ
( I ) is not a decision point.
Formally, γ ( I ) is a subset of subt ( I ) such that ∀
I (cid:48) ∈
γ
( I ) , P ( I (cid:48) )
=
i and ∀ I (cid:48)(cid:48) ∈ subt ( I ) , if I (cid:48)(cid:48) is an ancestor of I (cid:48)
∈ γ
( I ) , then P ( I (cid:48)(cid:48) )
(cid:54)
=
i or I (cid:48)(cid:48)
=
I .
For convenience, we suppose P ( I r ) =
i where I r is the infoset of the empty history.
Then, our segmentation rule is deﬁned as a procedure that updates the strategies on infosets recursively as follows: 1) We update the strategy on I r in every round; 2) For infoset
I , if we update the strategy on I at time step t , the time steps from τ t ( I ) + 1 to t forms a segment in the Lazy-RM of I .
That is, we compute r j ( I ) for the corresponding j and then apply RM to I ; 3) after updating the strategy on infoset I , we keep on updating the strategies on the infosets from γ ( I ) with m t ≥ B where B > 0 is a constant.
Alg.
1 presents an outline of Lazy-CFR.
We’ll formally analyze its performance in Sec. 4 and now we brieﬂy discuss why Alg.
1 converges faster than CFR.
Let’s tentatively assume that we can compute m t and r j efﬁciently (See Sec.
3.2.1 for details), then the convergence rate of Lazy-CFR depends on the total number of updates on strategy and its regret.
As for the number of updates on strategy, it is obviously upper bounded by 1 B (cid:80)
Tt =1 (cid:80)
I π − i σ t ( I ) which is much smaller than T |I| as π − i σ t ( I ) is probability.
As for the regret, it is easy to see that in Alg. 1, m t ( I ) ≤
d ( I ) B where d ( I ) is the depth of I in the tree.
With (cid:107) r j
( I ) (cid:107) 2 ≤ max t m t ( I ) and Eq.
(1), we can upper bound the regret of the
Lazy-RM on I by O ( B d ( I ) (cid:112)
An ( I ))
≤ O ( B d ( I ) √ AT ) .
Therefore, the overall regret, i.e., Eq.
(2), of Lazy-CFR is O
( |I| D B√ AT ) .
This regret bound is the same to the bound of CFR in (Zinkevich et al., 2008) within a gap of D B which is usually a logarithm of |I| .
In Sec. 4, we further reﬁne this bound and show it is comparable with the best known regret bound of CFR in (Burch, 2018).
3.2. I MPLEMENTATION
As mentioned in Sec.
3.1, if we compute m t ( I ) and r j ( I ) directly, we still have to spend O ( AT ) time on each infoset.
In this section, we show how to efﬁciently implement Alg. 1 by exploiting the structure of the game tree.
More speciﬁcally, we deﬁne data structures (DS) used in the computations of m t , r j and then show when and how to update them.
Updating these data structures can be implemented by depth ﬁrst search (DFS) and we only spend O (1) time to update the DSs on each history visited by DFS.
The detailed pseudo-code is in Appx E.
W.L.O.G., we consider the DSs used in Lazy-CFR for player i .
First, we store u i ( h | σ t ) in DS
U ( h ) and the strategy proﬁle in Γ( I ) , i.e., U ( h ) =
u i ( h | σ t ) , Γ i ( I ) =
σ it ( I ) at time step t .
And how to update U and Σ is standard as other CFR algorithms.
Recall that our target is to compute m t ( I ) =
(cid:80) h ∈
I m t ( h ) =
(cid:80)
h ∈
I (cid:80) tt (cid:48) =
τ t ( I )+1 π − i σ t (cid:48)
( h ) and r j ( I, a ) =
(cid:80) h r j
( h, a ) = (cid:80) h ∈
I (cid:80) tt (cid:48) = τ t ( I )+1 π − i σ t (cid:48)
( h ) u
i (( h, a ) | σ t (cid:48) )
where ( h, a ) denotes the successor of h after action a .
It is noteworthy that we do not need to compute m t and r j for every history in every round.
Instead, we just need to make sure m t ( h ) and r j ( h, a ) can be computed efﬁciently when Alg.
1 visits h .
Before diving into details, we intuitively explain how to efﬁciently compute r j ( h ) and m t ( h ) .
For m t ( h ) , suppose we have stored (cid:80) τ t ( pa ( h ))
t = τ t ( h ) π − i t ( h ) in some data structure α ( h ) , then we can compute m t ( h ) as m t ( h )
=
(cid:80) h (cid:48) :
h ∈ subt ( h (cid:48) )
α ( h (cid:48) )
Γ − i ( h (cid:48) , h ) where Γ( h (cid:48) , h ) denotes the probability of reaching h from h (cid:48) contributed by Γ − i . As for r j ( h ) , suppose the strategy on histories in subt ( h ) is never modiﬁed during [ τ t ( h ) , t ] , then we can compute r j ( h )
= m t ( h ) U
( h ) in time O (1) , as U ( h ) is also ﬁxed in [ τ t ( h ) , t ] . In Alg. 1, it is possible that the opponent’s strategy on some h (cid:48)(cid:48) ∈ subt ( h ) is modiﬁed during [ τ t ( h ) , t ] , thus, we need more data structures to maintain the cumulative reach probability and cumulative counterfactual reward.
More speciﬁcally, we use two additional DSs α ( h ) and β ( h, a ) .
In α ( h ) , we store summations of reach probabilities and we store a summation of counterfactural rewards in β ( h, a ) .
These DSs should satisfy the following properties:
Property 1.
In round t , α, β, ˆ α should satisfy:
[Equation]
= τ t ( h {+)+
+} π − i σ t (cid:48)
( h ) , β ( h, a ) =
(cid:80) t 1 ( h ) t (cid:48)
=
τ t ( h )+1 π − i σ t (cid:48)
( h ) u
i (( h, a ) | σ t (cid:48) ) and ˆ α ( h ) =
(cid:80) t 2 ( h ) t (cid:48)
=
t 1 ( h )
+1 π − i σ t (cid:48)
( h ) for τ t ( h ) ≤ t 1 ( h ) ≤ t 2 ( h ) ≤ t .
We will introduce t 1 , t 2 soon.
If h (cid:48)
=
pa ( h ) , then t 1 ( h (cid:48) )
=
t 2 ( h ) ; 2)
If h is the root, t 2 ( h ) = t .
So that [ τ t ( h ) , t ] = (cid:83) h (cid:48) :
h ∈ subt ( h (cid:48) )
[ t 1 ( h (cid:48) )
+ 1 , t 2 ( h (cid:48) )] .
For all h (cid:48) ∈ subt ( h ) , P ( h (cid:48) )
(cid:54)
=
i , the strategy on h (cid:48) is never modiﬁed on time steps between t 1 ( h ) and t 2 ( h ) .
So that for t (cid:48) ∈ [ t 1 ( h ) , t 2 ( h )] and h (cid:48) ∈ subt ( h ) , π − i σ t (cid:48) ( h, h (cid:48) )
=
π − i Γ ( h, h (cid:48) ) .
Intuitively, t 1 ( h ) is the last time step that U ( h ) is modiﬁed and t 2 ( h ) is the last time step that the strategy on pa ( h ) is modiﬁed.
Before discussing how to update these DSs, we discuss how to use them to compute m t and r j .
It is easy to check that Eq.
(4) is true with Prop. [-
-]1, we have:
[Equation] where π − i Γ ( h (cid:48) , h ) denotes the probability from h (cid:48) to h contributed by − i with strategy proﬁle Γ . We postpone the derivation of Eq. (4) to Appx.
E.
With the fact that m t ( h ) = m t ( pa ( h )) π − i Γ ( pa ( h ) , h )
+ + α ( h ) where pa ( h ) is the parent of h , computing m t only spends O (1) time on each node as when visiting h , m t ( pa ( h )) must have been computed in DFS.
The last challenge is how to update these DSs to satisfy the desired properties in Prop 1.
Speciﬁcally, let S 1 ,t denote the set of histories such that if h ∈ S , the strategy on subt ( h ) is modiﬁed at round t and S 2 ,t = { h : h / ∈ S 1 ,t , pa ( h ) ∈ S 1 ,t } .
As illustrated in Fig.
2, we only update DSs on
S t = S 1 ,t ∪ S 2 ,t as:
Update on the DSs: 1) If Γ on the infoset of h is modiﬁed, set α ( h ) =
β ( h, a ) = 0 ; else 2) for h ∈ S t , and set α ( h )
= m t ( h ) , β ( h, a )
= m t ( h ) U
(( h, a )) .
With above update rule, it is easy to check that if Prop 1 is true at t , then it is still true at t + 1 .
Lazy-CFR+ and Lazy-LCFR
: We can also apply lazy update to CFR+ (Bowling et al., 2017) and
LCFR (Brown & Sandholm, 2019a), which are improvements of CFR.
To get Lazy-CFR+ and
Lazy-LCFR, we only need to replace RM by the corresponding OLO solvers, and use their methods of computing time-averaged strategy as in (Bowling et al., 2017) and (Brown & Sandholm, 2019a) respectively.
T HEORETICAL ANALYSIS
We now present the theoretical results, starting with the regret for members of CFR with lazy update.
4. R EGRET UPPER BOUND
We extend the regret analysis on the vanilla CFR in (Burch, 2018) to the members of CFR with lazy update.
Speciﬁcally, let ξ i ( σ )
= (cid:80) Dd =1 (cid:113)(cid:80)
I : d ( I )=
d π iσ ( I ) , ξ = max i,σ ξ i ( σ ) and η ( σ ) =
ξ i ( σ ) (cid:113)
A max I,j ( (cid:80) t j +1 ( I ) t =
t j ( I )+1 π − i σ t ( I )) which are parameters depending on the structure of the game and segmentation rule.
Thm. 1 provides a regret bound for a CFR algorithm with an arbitrary segmentation rule.
By applying Thm.
1, we can get the regret bound of CFR and Lazy-CFR which are comparable with Corollary 2 in (Burch, 2018).
Theorem 1.
The regret of CFR with lazy update can be bounded as R iT ( σ ) ≤ O ( √ Tη ( σ )) .
Lemma 2.
With RM, the regret of the vanilla CFR is bounded by O ( ξ √ AT ) and the regret of
Lazy-CFR is bounded by O ( ξ √ DAT ) .
Proof.
It is easy to see that for the vanilla CFR, we have η ( σ ) ≤ ξ i ( σ ) √ A and for Lazy-CFR, we have η ( σ ) ≤ ξ i ( σ ) √ AD .
With Thm. 1, we get the regret bounds.
4. T IME AND SPACE COMPLEXITY
With the implementation in Sec.
3.2.1 and Appx E, the running time of Lazy-CFR is bounded by
O ( (cid:80) t | S t | ) .
Obviously, (cid:80) t | S t | =
O (max σ (cid:80) t ) π − i σ ( h ) .
Thus, we can bound Lazy-CFR’s time complexity as:
Lemma 3 (Time complexity) .
The time complexity of Alg.
1 is O
( T max σ (cid:80) h
π − i σ ( h )) .
To show how small max σ (cid:80) I π − i σ ( I ) is, we make the following mild assumption which leads to
Corollary 1.
Assumption 1. 1, If P ( h ) =
i , then P (( h, a )) (cid:54) =
i ; 2, The tree of infosets for each player is a full
A -ary tree; 3, Every infoset in the tree of infosets is corresponding to n nodes in the history tree.
Corollary 1.
If a TEGI satisﬁes Assumption 1, then ∀ σ, (cid:80) h ∈I
i π − i σ ( h ) =
O ( n (cid:112) |I
i | ) .
According to Lemmas 2 3 and 1, the regret of Lazy-CFR is about O ( √ D ) times larger than that of
CFR, whilst the running time is about O ( (cid:112) |I| ) times faster than CFR per round under Assumption 1.
Thus, according to Lemmas 1, 2 and with a bit algebra calculation, we know that Alg.
1 is about O ( (cid:112) |I| /D )
times faster than the vanilla CFR to achieve the same approximation error.
The improvement is signiﬁcant in large scale TEGIs.
Space complexity : A potential limitation of Lazy-CFR is that its space complexity is O ( | H | ) as we have to store the data structures α, β, ˆ α on histories.
In contrast, the space complexity is O ( |I| ) for some popular implementation of CFR.
In heads-up ﬂop hold’em poker (FHP) (Brown et al., 2019), we have | H | ≈ 10 12 and |I| ≈ 10 9 .
Accordingly, Lazy-CFR needs about 10 TB to store the data structures while CFR needs about 10 GB, which is still affordable in common storage systems.
Note that we do not optimize the space complexity in this work.
It is worth of a systematical investigation to derive an algorithm as fast as Lazy-CFR without additional memories, e.g., by designing a better segmentation rule as well as a better implementation.
4. R EGRET LOWER BOUND
In the analysis of lower bound, we consider the standard adversarial setting in online learning, in which an adversary selects σ − i t and a reward function u it : Z → [ − 1 , 1] where Z is the set of terminal nodes in the infoset tree of player i .
Thus, to get a lower bound, we need to explicitly construct σ − i t and u it ( I ) , I ∈
Z .
This setting is consistent with the analysis of the regret upper bound, in which we do not make any assumption on how u it , σ − i t changes over rounds.
Intuitively, by extending the lower bound analysis of OLO (Cesa-Bianchi & Lugosi, 2006) to infoset tree structured problem, we bound
Eq.
2 from below as in Thm.
2 and for the proof, see Appendix D.
Theorem 2.
For an algorithm A generating σ it given the history in the past, let R iT, A denote the regret of A in the ﬁrst T rounds, we have lim
A →∞ lim T →∞ min
A max π − i σt ,u t
R iT, A ξ i √ T/D log A ≥ 1 .
By comparing the regret lower bound in Theorem 2 and the regret upper bounds of CFR and
Lazy-CFR as in Lem 2, we can see that the regret of CFR and Lazy-CFR are both near-optimal. [Figure]
R ELATED WORK
Monte-Carlo and pruning-based methods: There are several variants of CFR which attempt to avoid traversing the whole game tree at each round.
Monte-Carlo based CFR (MC-CFR) (Lanctot et al., 2009; Burch N, 2012) uses Monte-Carlo sampling to avoid updating the strategy on infosets with small probability of arriving at.
Pruning-based variants (Brown & Sandholm, 2017a; 2015) skip the branches of the game tree if they do not affect the regret, but their performance can deteriorate to the vanilla CFR in the worst case.
And dynamic thresholding (Brown et al., 2017) directly prunes the branches with small reach probabilities.
In this work, we do not compare with pruning-based method (Brown & Sandholm, 2019b) since the pruning technique is orthogonal to lazy update.
Existing analyses of regret : Lanctot et al.
(2009); Burch N (2012); Burch (2018) reﬁned the regret upper bound of CFR.
Our analysis is essentially an extension of the regret analysis on the vanilla
CFR in (Burch, 2018) to other variants of CFR with lazy-update.
E XPERIMENT
In this section, we empirically evaluate [-the CFR, CFR+, LCFR variants which prunes the histories with π − i σ t ( h ) = 
-]our algorithm against existing CFR variants.
We measure the exploitability of these algorithms.
The exploitability of a strategy ( σ 1 , σ 2 ) can be interpreted as the approximation error to the Nash equilibrium.
The exploitability is deﬁned as max σ (cid:48) , 1 u 1 (( σ (cid:48) , 1 , σ 2 ))
+ max σ (cid:48) , 2 u 2 (( σ 1 , σ (cid:48) , 2 )) .
Experiments are conducted on variants of two common benchmarks in imperfect-information game solving: (1) the Leduc hold’em (Southey et al., 2005) which is a simplifed version of the heads-up no-limit hold’em poker with 6 cards; and (2) heads-up ﬂop hold’em poker (Brown et al., 2019)
(FHP) which is similar to heads-up no-limit Texas hold’em poker without the last two rounds of betting.
In our experiments of Leduc Hold’em poker, we restrict players not to bid more than 6 or 8 times the big blind.
The number of infosets in the generated game trees are about 40000 and 455000 for each player, respectively.
We run each algorithm on each Leduc Hold’em game for 30000 seconds.
In the experiments of FHP, to control the size of the game, we consider a simpliﬁed game such that in the deck, there are 2 suits and 5 cards in each suit.
And we restrict players not to bid more than 7 times the big blind.
The size of the game is about 10 9 .
We measure algorithms by the number of nodes touched which is independent with implementation and hard-ware.
We empirically compare Lazy-(CFR,CFR+,LCFR) with existing methods, including the vanilla CFR, CFR+, LCFR, MC-(CFR,CFR+,LCFR) and MC-LCFR with negative regret-pruning (MC-LCFR-P) which was used in developing Pluribus (Brown & Sandholm, 2019b). In our experiments, we use RM (RM+) as the OLO solver.
In Lazy-(CFR,CFR+,LCFR), we set B ∈ { 0 . 1 , 1 . 0 } .
We evaluate the
CFR, CFR+, LCFR variants which prunes the histories with π − i σ t ( h ) = 0 in the recursive tree walk as they don’t affect the regret.
For MC-(CFR,CFR+,LCFR), we use the external-sampling scheme.
In the experiments on MC-LCFR-P, we use the following parameters: we run MC-LCFR in the ﬁrstminutes; after that, in each iteration, we run MC-LCFR with probability 0.05 and run MC-LCFR-P with probability 0.95; in MC-LCFR-P, we prune those branches with average regret less than −times the big blind.
Fig.
3 presents the results.
We can see that the performance of Lazy-CFR(+) has a similar performance to CFR(+) on the variants of Leduc Hold’em.
This is because in the experiments of CFR(+) on Leduc
Hold’em, there are a large portion of histories with π − i t ( h ) = 0 on average.
And on the larger game,
Lazy-CFR(+) signiﬁcantly outperforms CFR(+).
The performance of Lazy-LCFR is much worse than LCFR on all games.
This might be because our segmentation rule is designed for OLO with a uniform weight in each iteration.
And LCFR assigns more weights on later iterations.
C ONCLUSIONS
In this work we propose a new framework to develop efﬁcient variants of CFR with an analysis shows that our algorithm is provably faster than the vanilla CFR.
The ﬁnal algorithm runs fast in practice, but with some extra cost on space complexity.
It is worth of a systematical study to reduce the space complexity.
A T HE TABLE OF NOTATIONS
We provide a table of some important notations in our paper in this section.
Notation
Explanation [ n
[Table]
the set { 1 , 2 , · · · , n } [ a, b ] the set { a, a + 1 , · · · , b }
[Table]
The tree of histories
[Table]
Infomation set
[Table]
the depth of the game tree
[Table]
the depths of I and h
[Table]
the
[Table]
tree
[Table]
the
[Table]
a
[Table]
the
[Table]
I
[Table]
action a
[Table]
I i is the infoset tree of player
i
and I
=
I 1 ∪ I
[Table]
The player function, i.e., P ( h ) and P ( I ) are the players who take actions on h and I , respectively.
[Table]
the strategy of player i ;
[Table]
the strategy of all players;
[Table]
the strategy of all players except player
i
[Table]
=
[Table]
P
[Table]
h )
[Table]
h )
and σ
[Table]
I )
= σ
P
( I
) ( I )
[Table]
the reach probability of h
and I contributed by all players
[Table]
the reach probability of h
and I contributed by player
i
[Table]
the reach probability of h
and I contributed by all players except i
[Table]
the reach probability from h (cid:48) to h contributed by all players except i if h ∈ subt ( h (cid:48)
[Table]
the reward received by player i at leaf h
[Table]
the reward received by player i with strategy proﬁle σ
[Table]
the reward received by player i at h with strategy proﬁle σ
[Table]
the strategy proﬁle generated from σ by
replacing σ ( I ) by σ (cid:48)
( I ) pa ( h ) , pa ( I ) the parents of h and I respectively subt ( h ) , subt ( I ) the subtree rooted at h
and I respectively γ ( I ) , I ∈ I
i a subset of subt ( I ) such that for all I (cid:48)
∈
γ
( I ) then P ( I (cid:48) )
=
i ; consider I (cid:48)(cid:48) ∈ subt ( I ) , if I (cid:48)(cid:48) is an ancestor of I (cid:48) , then P ( I (cid:48)(cid:48) )
(cid:54) =
i (cid:104) a, b (cid:105) the inner product of vectors a and b w t the action selected by the player at time step t c t the reward vector at time step t
R oloT the regret of the OLO, i.e., R oloT = max w ∈ ∆( A ) (cid:80) Tt =1 (cid:104) w, c t (cid:105) − (cid:80)
Tt =1 (cid:104) w t , c t (cid:105) ∆
( A ) the set of distributions over set
A
R iT ( σ i )
R iT ( σ i ) =
(cid:80) Tt =1 u i ( σ i , σ − i t ) − (cid:80) Tt =1 u i (( σ it , σ − i t ))
R iT the regret of player i , i.e., R iT = max σ i R iT ( σ i ) ¯ σ iT the time-averaged strategy with ¯ σ iT ( I ) =
(cid:80) Tt =1 π iσt ( I ) σ t
( I ) (cid:80) Tt =1 π iσt ( I )
τ t ( I ) , τ t ( h ) the last time step we update the strategy on infoset I (history h ) before time
t m t ( h ) the sum of reach probabilities, m t ( h ) =
(cid:80) tt (cid:48)
=
τ t ( h )+1 π − i σ
[Equation]
(cid:48)
[Equation]
h ) m t
( I )
[Equation]
r j ( h, a )
the sum of counterfactural rewards over the j -th segment of h
r j ( I, a ) r
[Equation] [Equation]
, a )
[Table] [Table]
data structure with U ( h ) =
u i ( h | σ t ) at time step t
[Table]
data structure with Σ( I ) =
σ t ( I ) at time step t
[Table]
[Table] [Table]
data structures such that, at time step t , m t ( h ) =
α ( h ) + (cid:80) h (cid:48) :
h ∈ subt ( h ) ˆ α ( h ) π − i Γ ( h ) and r j ( h, a ) =
β j ( h, a ) + U (( h, a ))
(cid:80) h (cid:48) :
h ∈ subt ( h (cid:48) )
ˆ α ( h ) π − i Γ ( h )
[Table]
a parameter depends on the structure of the game tree with ξ i ( σ )
=
(cid:80) I ∈I i ,P ( I )=
i π iσ ( I )
[Table]
ξ = max i,σ ξ i ( σ )
[Table]
a parameter depends on the segmentation rule with η ( σ ) =
Aξ i ( σ ) max I,j ( (cid:80) t j +
( I ) t =
t j ( I )+1 π − i σ t ( I ))
B D ERIVATION OF REGRET BOUNDS OF RM AND A DA H EDGE
The regret upper bound of RM and AdaHedge used in this work can be easily derived from previous works.
We include their derivations only for completeness.
Derivation of Eq.
The standard analysis of RM in Cor 2.1 on pg.
13 in (Cesa-Bianchi & Lugosi, 2006) shows the regret of RM is bounded by:
[Equation]
Obviously, we have | (cid:80) |A| a (cid:48)
=1 w t ( a (cid:48) )
c t ( a (cid:48) )
| ≤ max a | c t ( a ) | since w t is a vector of probability.
We have:
[Equation]
We ﬁnished the proof.
Derivation of the regret bound of AdaHedge.
Let v t denote the variance of reward if we take a random action according to the distribution w t deﬁned in AdaHedge.
Theorem 6 in (De Rooij et al., 2014) provides a second-order bound as:
[Equation]
where κ is a constant.
It is known that for a random variable x ∈ [ a, b ] , its variance is no more than ( a − b ) 2 / 4 .
Thus, v t ≤ max a c 2 t ( a ) .
We ﬁnished the proof.
C P ROOF OF T HEOREM
We now extend the proof of the Theorem 1 in (Burch, 2018) to CFR with lazy update.
We ﬁrst introduce some additional terminology.
Consider B ⊆
I i , let ξ ( B ) = max σ − i (cid:80) I ∈ B (cid:80)
h ∈
I π − i σ − i ( h ) .
Now we prove Theorem
[Equation]
Proof.
Let κ denote the constant involved in Eq.
With Eq.
(2) and the regret bound of RM, we have
[Equation]
 
And then apply Jensen’s inequality and with some calculations
[Equation]
we have
[Equation]
( I (cid:48) )
(cid:33) · (cid:88)
I : d ( I )=
d   π iσ ( I ) (cid:118)(cid:117)(cid:117)(cid:116)
A n ( I ) (cid:88)
j = max a r 2 j ( I, a ) /T
 =
κ
D (cid:88) d =   (cid:88)
I (cid:48) ,t,d ( I (cid:48) )
=
d π iσ ( I (cid:48) ) π − i σ t ( I (cid:48) ) 
 (cid:88)
I : d ( I )=
d (cid:32) π iσ
( I ) (cid:80) I (cid:48) ,t,d ( I (cid:48) )
=
d π iσ ( I (cid:48) )
π − i σ t ( I (cid:48) ) (cid:33)   (cid:118)(cid:117)(cid:117)(cid:116)
A n ( I ) (cid:88)
j = max a r 2 j ( I, a ) /T
 =
κ
D (cid:88) d =   (cid:88)
I (cid:48) ,t,d ( I (cid:48) )
=
d π iσ ( I (cid:48) ) π − i σ t ( I (cid:48) ) 
 (cid:88)
I : d ( I )=
d (cid:32) π iσ
( I ) (cid:80) t π − i σ t ( I ) (cid:80) I (cid:48) ,t,d ( I (cid:48) )
=
d π iσ ( I (cid:48) )
π − i σ t ( I (cid:48) ) (cid:33)   (cid:118)(cid:117)(cid:117)(cid:116)
A (cid:80) n (
[Table]
[Equation]
The Jensen’s inequality applies here, in which we move the term of probability outside the square root into it.
[Equation]
The last inequality utilizes the fact that (cid:80) I (cid:48) ,t,d ( I (cid:48) )
=
d π iσ ( I (cid:48) )
π − i σ t ( I (cid:48) ) ≤ T as π − i σ t
( I ) π iσ ( I ) is the probability of arriving at infoset I under strategy ( σ i , σ − i t ) and there is at most 1 infoset been arrived at each level.
It is easy to see that max a r 2 j ( I, a ) ≤ ( (cid:80) t j +1 ( I ) t
= t j ( I )+1 π − i σ t ( I )) 2 .
With straight-forward computations, we ﬁnish the proof:
[Equation]
I
[Equation]
, this is because
[Equation]
D T HE REGRET LOWER BOUND ANALYSIS
Recall that the analysis on the regret upper bound is under the standard adversarial setting in online learning, that is, it does not depend on how the opponent’s strategy and the utility vary over time steps.
So we here make the same adversarial assumption that there is an adversary choose both σ − i t and u it .
It is worthy to note that in our construction of the adversary, the utility function may also vary over time.
For convenience, let − i denote all the players except i
[Equation]
and let ζ
[Equation]
P ( I )=
i π iσ
i ( I ) } .
Let D :=
{ I ∈ I i : ∃ σ i ∈ ζ, π iσ i ( I ) > 0 } .
It can be shown that D forms a subtree of I i .
Intuitively, our construction on σ − i t and u it can be divided into two stages:
For I ∈ Z, I / ∈ D , u it ( I ) = − 1 for all t .
This enforces player i take actions on D , otherwise, it will always receive reward − 1 . 2.
In each round t , for I ∈ D , I / ∈ Z, P ( I ) (cid:54) =
i , we ﬁrst generate a random variable a ( I ) ∼ Multinomial(1 , 1 A (I) ) , 3 and then set σ − i t ( I, a ( I ))
= 1 , and σ − i t
( I, a ) = 0 for a (cid:54) =
a ( I ) .
Intuitively
this step separates R iT into O (( ξ i ) 2 ) isolated OLOs, each of which is of A actions and
would
be repeated for about O ( T/ ( ξ i )
rounds
, since only one of them will
be
triggered
on in each time step according
to our construction
on
σ − i
t
Thus combined with the lower bound proved by (Cesa-Bianchi & Lugosi, 2006),
each
OLO incurs a regret of about Ω
( √ T
log
A/ξ i )
, and
we can
informally
provide a lower bound of
Ω
(( ξ i ) 2 /ξ i √ T log A )
= Ω( ξ i √ T log A )
which is
formally described in Theorem 2.
Before proving Theorem 2, we ﬁrst address some trivial cases and show some intuitions on the way we construct the worst case of σ − i t and u it .
Let ϕ i ( σ )
= (cid:80) I ∈I
i π iσ ( I ) .
It is easy to see that ϕ
i ( σ ) > ( ξ i ( σ )) 2 /D .
And we are going to show that
[Equation]
To start, we make an implicit assumption that |A ( I ) | ≥ 2 , ∀ I ∈
I i , P ( I )
=
i .
Otherwise we can merge these infosets as we have no choice but choose the only action, which contributes nothing to the regret.
In addition, we assume i and − i take actions alternatively.
We now show that it is sufﬁcient to focus on the subtree D
(Note that D is a subtree rooted at I r ).
By setting u t ( I ) = − 1 for I ∈
Z and I / ∈ D , we know that player
i will not take actions to go out of
D .
And if player − i goes out of D , this round will contribute nothing to the regret as player i will always receive reward − 1 .
Moreover, we assume P ( pa ( I ))
=
i, ∀
I ∈ Z .
Otherwise, we can merge the subtree rooted at pa ( I ) , I ∈
Z into one single leaf node (i.e. an infoset I ∈
Z after the mergence), and P ( pa ( I ))
=
i, ∀ I ∈
Z in the new merged infoset tree.
The following property will be useful in our proof.
Lemma 4.
Let ϕ i D ( σ i ) =
(cid:80) I ∈D
π iσ i ( I ) .
Moreover, let ζ (cid:48)
= { σ i : σ i ( I, a ) = 0 , for P ( I ) =
i, I ∈
D , ( I, a ) / ∈ D} .
Then ∀ σ i D , ( σ i D ) (cid:48)
∈ ζ (cid:48) , we have that ϕ
i D ,σ i D
= ϕ i D , ( σ i D ) (cid:48) .
Proof.
To start, let ϕ i D ( σ i , I )
= 1 π iσi ( I ) (cid:80) I (cid:48) ∈ subt ( I ) ,P ( I (cid:48) )
=
i π iσ
i ( I (cid:48) ) , I ∈ D , where subt ( I ) means the subtree rooted at I (Note that it’s possible that I (cid:48) ∈ subt ( I ) , I (cid:48) / ∈ D ).
Let ϕ
i D ( I ) =
max σ i ϕ i D ( σ i , I ) , and σ i, ∗ = arg max σ i ϕ i D ( σ i , I ) .
We ﬁrst show that, ∀ I ∈ D , P ( I ) =
i , ϕ i D ( J i ( I, a j )) =
ϕ i D
( J i ( I, a k )) , j, k ∈ |A
D ( I ) | where A D ( I ) consists of a ∈ A ( I ) such that
J i ( I, a ) ∈ D .
With the deﬁnition of D , ∀ I ∈ D , ∃ σ i 0 ∈ ζ , s.t.
π iσ
i
( I ) > 0 .
If ϕ i D ( J i ( I, a j )) (cid:54) =
ϕ i D
( J i ( I, a k )) ,
W.L.O.G. we assume ϕ i D
( J i ( I, a j )) <
ϕ i D
( J i ( I, a k )) , then ∀ σ i 0 such that π iσ i
( I ) > 0 , we have ϕ i D (
[Equation]
i 0 , I ) =
1 + (cid:88)
[Equation]
With the last inequality, we can construct a ( σ i 0 ) (cid:48) with ϕ i D (( σ i 0 ) (cid:48) , I r ) ≥ ϕ i D ( σ i 0 , I r ) and ( σ i 0 ) (cid:48)
( I (cid:48) )
(cid:54) =
σ i
( I (cid:48) )
only if I (cid:48) ∈ subt ( I ) .
Formally, we deﬁne ( σ i 0 ) (cid:48) as:
• ( σ i 0 ) (cid:48)
( I (cid:48) )
=
σ i
( I (cid:48) ) , I (cid:48) / ∈ subt ( I ) • ( σ i 0 ) (cid:48)
( I (cid:48) )
=
σ i, ∗ ( I (cid:48) ) , I (cid:48) ∈ subt ( I ) , I (cid:48) (cid:54) =
I • ( σ i 0 ) (cid:48)
( I, a ) =
σ i
( I, a ) , a (cid:54) =
a j , a k • ( σ i 0 ) (cid:48)
( I, a j ) =
• ( σ i 0 ) (cid:48)
( I, a k ) =
σ i
( I, a j ) + σ i 0 (
I, a k )
As σ i 0 ∈ ζ , ϕ i D (( σ i 0 ) (cid:48) , I r ) = ϕ
i D ( σ i 0 , I r ) , which means σ i
( I, a j ) = 0 , due to π iσ i
( I ) > 0 .
Thus, ∀ σ i 0 such that π iσ i
( I ) > 0 , we have π iσ i
( J i ( I, a j )) = 0 , which means J i ( I, a j ) / ∈ D (notice that if π iσ i
( I ) = 0 , π iσ i
( J i ( I, a j )) = 0 ), that is contradict to our assumption J i
( I, a j ) ∈ D .
Thus ϕ
i
[Equation] [Equation]
[Equation] [Equation]
D ( I ) .
Now we can show that ϕ i D ( σ i , I ) =
ϕ i D
( I ) , ∀ σ i , ∀ I with mathematical induction.
Let D denotes the depth of D .
As we assume P ( pa ( I ))
=
i, ∀
I ∈ Z , P ( I r ) =
i if D is odd and
P ( I r ) = − i
if D is even.
We separately discuss them as follows:
• D = 1 : Obviously ϕ i D ( σ i , I r ) =
ϕ D ( I r ) = 1 .
• D is even: If ∀
a ∈
A D ( I r ) , J i ( I r , a ) satisﬁes that ϕ
i D ( σ i , J i
( I r , a ))
=
ϕ D ( J i ( I r , a )) ,
then
[Equation]
Thus we get ϕ
i D ( σ i
I r ) =
ϕ i D
( I r )
• D is odd and D (cid:54)
= 1 :
If ∀ a ∈
A D ( I r ) , J i ( I r , a ) satisﬁes that ϕ
i D ( σ i , J i
( I r , a ))
=
ϕ D ( J i ( I r , a )) , then ∀ σ i ∈ ζ (cid:48) , ϕ i D ( σ i , I r ) =
1 + (cid:88) a ∈A D
( I )
σ i ( I r , a ) ϕ
i D ( σ i , J i
( I r , a ))
=1 + (cid:88) a ∈A D
( I )
σ i ( I r , a ) ϕ
i D ( J i
( I r , a ))
=1 + [ (cid:88) a ∈A D ( I ) σ i ( I r , a )] · ϕ i D ( J i ( I r , a j )) , ∀ a j ∈ A D ( I )
(due to ϕ i D ( I r , a j ) =
ϕ
i D ( I r , a k ) , ∀ a j ,
a k ∈ A D ( I ) )
=1 + ϕ
i D
( J i ( I r , a j ))
= ϕ
i
D
( I r
(as ∀
σ
i , ϕ i D (
σ
i , I
[Equation] [Equation]
[Equation]
ϕ
i D (
J
i ( I r , a )) , a ∈ A D ( I ) is
independent of σ
i
We can get ϕ
i D ( σ i , I r ) =
ϕ i D
( I r ) as well.
So ∀ σ i , ∀ A, ϕ i D ( σ i , I r ) =
ϕ i D
( I r ) .
As ϕ i D ,σ i = ϕ
i D ( σ i , I ) =
ϕ i D
( I ) is independent from σ i , we ﬁnally prove this lemma.
As ϕ i D ,σ i is independent of the choice of σ i , we will drop the subscript σ i , that is, we will use ϕ i D instead of ϕ i D ,σ i in the following proof.
Now we prove Theorem 2.
Proof.
As we discussed before, it’s sufﬁcient to focus on the subtreee D , so all of the terms (e.g. the regret) in the following proof are deﬁned on D , not I i .
For D , P ( pa ( I ))
=
i, ∀
I ∈ Z , we use the following procedure to generate σ − i t and u it for each round:
• u t ( I ) ∼ Bernoulli(0 . 5) , ∀ I ∈
Z .
• ∀ I ∈ D , P ( I ) (cid:54)
=
i , we deﬁne p D ( I ) as:
[Equation]
Notice that this term only depends on D , so once we determine D , we can immediately get this p D ( I ) .
Each turn we ﬁrst sample a ( I ) from Multinomial (1 , p D ( I )) , then let σ − i t ( I ) =
a ( I ) .
In the following proof we denote this generating procedure as M and use the notation E M [ · ] as the expectation over this generating procedure.
Let n t ( I ) ,
I ∈ D denotes the cumulative arriving time player i arrives at I in the ﬁrst t rounds.
With a little abuse of notation, we use the term R iT ( I ) , I ∈ D to represent the regret of the subtree rooted at I in the ﬁrst T turns, i.e.
[Equation]
I
[Equation]
i
[Equation]
D
We will prove that lim
A →∞ lim T →∞ E M R iT,alg / (cid:113) ϕ
i D T log A≥ 1 , ∀ alg with mathematical induction, where R iT,alg is deﬁned in Theorem 2.
Notice that under the assumption that P ( pa ( I ))
=
i, ∀
I ∈ Z , P ( I r ) =
i if D is odd while P ( I r ) =
− i if D is even.
In our proof we will discuss them separately.
• D = 1 : As (Auer et al., 1995; Freund & Schapire, 1997; Cesa-Bianchi & Lugosi, 2006; Dani et al., 2008) have shown, for K -arm online linear optimization problem,
lim K →∞ lim T →∞ E M R iT,
[Equation]
T log K ≥ 1 , ∀ alg , which is consistent to our proposition with ϕ i D = 1 4 .
• D is even: From the deﬁnition we can get that ϕ
i D ( I r ) =
(cid:80) a ∈A D ( I r ) ϕ
i D ( J i
( I r , a ))
[Equation]
then
[Equation]
In the ﬁrst equation, we decompose the overall expected regret to the summation of regrets on subtrees.
This decomposition can be derived in a similar way to that of Eq.
(2) in
Appendix C.
In the second equation we transform the limitation of T into limitation of n T ( J i ( I r , a )) , ∀
a ∈
A D ( I r ) .
As the adversary selects action a j with probability p D
( I r ) j > 0 where p D ( I r )
j denotes the j -th element of p D ( I r ) ,
when T → ∞ , the n T ( J i ( I r , a )) → ∞ , ∀ a ∈
A D ( I r ) as well.
The inequality is from induction and
lim
T →∞ n
T
J i
I r , a )) /T
= p D ( I ) j , a j ∈ A D ( I r ) due to
the strong law of large numbers
• D is odd and D
[Equation]
1 : For convenience, let R
[Equation]
) denote the immediate regret on the root node for algorithm alg .
We ﬁrst show that under our construction
E M R iT,alg,imm ( I r ) ≥ 0 , ∀ alg .
Notice that in our construction, E M (cid:80) Tt =1 u t ( σ t | I r → a , I r ) = 0 .
5 T, ∀
a ∈ A D ( I r ) .
Meanwhile, E M (cid:80) Tt =1 u t ( σ t , I r ) = 0 .
5 T as well.
Thus,
[Equation]
This inequality is true for all σ t (i.e. alg ), and we can get
[Equation]
Then, similar to the case when D is even, we can get:
[Equation]
=
[Equation]
Similarly, we decompose the overall expected regret to each subtree in the ﬁrst equation.
Notice that ∀ alg , lim T →∞ (cid:80) a ∈A D
( I r ) ,n T ( J i ( I r ,a )) < ∞ E M R in T ( J i ( I r ,a )) ,alg < ∞ , lim T →∞ (cid:80)
[Equation]
∈A D ( I r
[Equation]
,n T ( J i ( I r ,a )) < ∞ E M R in T ( J i ( I r ,a )) ,alg / √ T = 0 .
Thus the second inequality is true.
The last inequality is true by (cid:80) i √ a i ≥ (cid:112)(cid:80) i a i .
Notice that (cid:80) a ∈A D ( I r ) ϕ
i D ( J i
( I r , a ))
n T ( J i ( I r , a )) =
( ϕ i D ( I r ) − 1) T (We can see this by n T ( J i
( I r , a ))
= (cid:80) Tt =
σ it ( I r , a ) and ∀ t, (cid:80) a ∈A D
( I r ) ϕ
i D ( J i
( I r , a ))
σ it ( I r , a ) =
ϕ
i D ( I r ) − 1 , which can be simply derived by the deﬁnition of ϕ i D and P ( I r ) = i ), and lim T →∞ (cid:80) a ∈A D ( I r ) ,n T ( J i ( I r ,a )) <
∞ ϕ iZ ( J i
( I r ,a ))
n T ( J i ( I r ,a ))
ϕ iZ ( I r ) T = 0 .
Thus we can get the limitation of T → ∞ .
Thus, with mathematical induction, we prove that lim T →∞ ,A →∞ E M
[Equation]
≥ 1 , while with the deﬁnition of D and Lemma 4, we can get ϕ
i D =
ϕ
i , thus we can get the mini-max lower bound in
Theorem 2.
E T HE DETAILS OF IMPLEMENTATION
We now discuss the detailed implementation of Lazy-CFR.
To start, we derive Eq.
According to
Property 1, we have:
[Equation]
The ﬁrst equation is the deﬁnition of m t ; the second line is according to Prop 1.2; the third line is derived from the deﬁnition of α and π − i σ and the last line is due to Property 1.3.
Similarly, we can write r j as:
[Equation]
We now present how to implement of the ideas in 3.2.1.
1: A two-player zero-sum extensive game.
2: Randomly initialize Γ .
3: ∀ h ∈ H, i ∈ { 1 , 2 } , compute the counterfactual reward U i ( h )
=
u i ( h | Γ) , flag i ( h )
=
− 1 .
4: ∀ I ∈
I , s ( I ) =
0 , ∀ a ∈ A ( I ) , r ( I, a )
= 0 .
5: while t < T do 6: for all i ∈ { 1 , 2 } do 7: α i ( h r )+ = 1 . 0 .
UPDATE1 ( I ir , i ) where I ir is the root of infoset tree
I i .
9: end for 10: ∀ i ∈ { 1 , 2 } UPDATE2 ( h r , i, t ) where h r denotes the root of the history tree.
11: end while 12: RETURN ¯ σ .
Algorithm
1: m ( I )
2: for all h ∈ I do 3: θ i ( h ) =
( θ i ( pa ( h ))
+ ˆ α i ( pa ( h )))
× π − i Γ ( pa ( h ) , h ) .
4: m ( I )+ =
α i ( h ) + θ i ( h ) 5: end for 6: if m ( I ) ≥ 1 then 7: if P ( I ) =
i then 8: for all a ∈ A ( I ) do 9:
UPDATE1( ( I, a ) , t ).
10: end for 11: ∀ h ∈
I, updﬂag ( h, t, i ) .
12: ∀ a ∈ A ( I ) , r ( I, a ) = 0 .
13: for all a ∈ A ( I ) , h ∈ I do 14: r ( I, a )+ = β ( h, a )
+ θ i
( h )
U i (( h, a )) .
15: end for 16: Γ i ( I ) =
RM ( r ( I )) .
17: else 18: ∀ a ∈ A ( I ) , UPDATE1((I,a),i, t).
19: end if 20: end if
Algorithm 4 updﬂag ( h, t, i ) 1: if h is not the root of history tree and flag i ( h ) (cid:54)
=
t then 2: flag i ( h ) =
t.
3: updflag ( pa ( h ) , t, i ) .
4: end if
Algorithm 5 UPDATE2 ( h, i, t ) 1: Let i (cid:48) denote the opponent of player i .
2: if flag i ( h ) =
t then 3: ∀ a ∈ A ( h ) UPDATE2 (( h, a ) , i, t ) .
4: update U 1 ( h ) and U 2 ( h ) .
5: α i ( h ) = 0 .
6: ˆ α i (cid:48) ( h ) = 0 .
7: ∀ a ∈ A ( h ) , β i (cid:48)
( h, a )
= 0 .
8: else 9: α i (cid:48)
( h )
+
=
( θ i (cid:48) ( pa ( h ))
+ ˆ α i (cid:48) ( pa ( h )))
× π − i (cid:48) Γ (cid:48)
( pa ( h ) , h ) .
10: ˆ α i (cid:48)
( h )
+
=
( θ i (cid:48) ( pa ( h ))
+ ˆ α i (cid:48) ( pa ( h )))
× π − i (cid:48) Γ (cid:48)
( pa ( h ) , h ) .
11: ∀ a ∈ A ( h ) , β i (cid:48)
( h, a )
+ = U i (cid:48)
( h, a )( θ i (cid:48) ( pa ( h ))
+ ˆ α i (cid:48) ( pa ( h )))
× π − i (cid:48) Γ (cid:48)
( pa ( h ) , h ) .
12: end if