Sample-Efficient Learning of Correlated Equilibria in Extensive-Form Games
Anonymous Author(s)
AffiliationAddress email
Abstract
Imperfect-Information Extensive-Form Games (IIEFGs) is a prevalent model forreal-world games involving imperfect information and sequential plays.
The
Extensive-Form Correlated Equilibrium (EFCE) has been proposed as a naturalsolution concept for multi-player general-sum IIEFGs.
However, existing algorithms for finding an EFCE require full feedback from the game, and it remainsopen how to efficiently learn the EFCE in the more challenging bandit feedbacksetting where the game can only be learned by observations from repeated playing.
This paper presents the first sample-efficient algorithm for learning the EFCEfrom bandit feedback.
We begin by proposing K -EFCE—a more generalizeddefinition that allows players to observe and deviate from the recommended actionsfor K times.
The K -EFCE includes the EFCE as a special case at K = 1 , andis an increasingly stricter notion of equilibrium as K increases.
We then designan uncoupled no-regret algorithm that finds an ε -approximate K -EFCE within(cid:101)
O (max i X i
A Ki /ε 2 ) iterations in the full feedback setting, where X i and A
i are thenumber of information sets and actions for the i -th player.
Our algorithm works byminimizing a wide-range regret at each information set that takes into account allpossible recommendation histories.
Finally, we design a sample-based variant ofour algorithm that learns an ε -approximate K -EFCE within (cid:101)
O (max i X i
A K
+
i /ε 2 )episodes of play in the bandit feedback setting.
When specialized to K = 1 , thisgives the first sample-efficient algorithm for learning EFCE from bandit feedback.
Introduction
This paper is concerned with the problem of learning equilibria in Imperfect-Information ExtensiveForm Games (IIEFGs) [28].
IIEFGs is a general formulation for multi-player games with bothimperfect information (such as private information) and sequential play and has been used formodeling and solving real-world games such as Poker [23, 31, 7, 8], Bridge [38], Scotland Yard [36],and so on.
In a two-player zero-sum IIEFG, the standard solution concept is the celebrated notion of
Nash Equilibrium (NE) [34], that is, a pair of independent policies for both players such that no playercan gain by deviating.
However, in multi-player general-sum IIEFGs, computing an (approximate)
NE is PPAD-hard and unlikely to admit efficient algorithms [12].
A more amenable class of solutionconcepts is the notion of correlated equilibria [4], that is, a correlated policy for all players such thatno player can gain by deviating from the correlated play using certain types of deviations.
The notion of Extensive-Form Correlated Equilibria (EFCE) proposed by Von Stengel and Forges[39] is a natural definition of correlated equilibria in multi-player general-sum IIEFGs.
An EFCE is acorrelated policy that can be thought of as a “mediator” of the game who recommends actions to eachplayer privately and sequentially (at visited information sets), in a way that disincentivizes any playerto deviate from the recommended actions.
Polynomial-time algorithms for computing EFCEs have been established, by formulating as a linear program and using the ellipsoid method [24, 35, 25],min-max optimization [16], or uncoupled no-regret dynamics using variants of counterfactual regretminimization [11, 20, 33, 2].
However, all the above algorithms require that the full game is known (or full feedback ).
In the morechallenging bandit feedback setting where the game can only be learned by observations from repeatedplaying, it remains open how to learn EFCEs sample-efficiently.
This is in contrast to other types ofequilibria such as NE in two-player zero-sum IIEFGs, where sample-efficient learning algorithmsunder bandit feedback are known [29, 19, 27, 5].
A related question is about the structure of the
EFCE definition: An EFCE only allows players to deviate once from the observed recommendations,upon which further recommendations are no longer revealed and the player needs to make decisionson her own instead.
This may be too restrictive to model situations where players can still observe therecommendations after deviating [33].
It is of interest how we can extend the EFCE definition in astructured fashion to disincentivize such stronger deviations while still allowing efficient algorithms.
This paper makes steps towards answering both questions above, by proposing stronger and moregeneralized definition of EFCEs, and designing efficient learning algorithms under both full-feedbackand bandit-feedback settings.
We consider IIEFGs with m players, H steps, where each player i has
X i information sets and A i actions.
Our contributions can be summarized as follows.
• We propose K -EFCE , a natural generalization of EFCE, at which players have no gains whenallowed to observe and deviate from the recommended actions for K times (Section 3).
At K = 1 ,the K -EFCE is equivalent to the existing definition of EFCE based on trigger policies.
ForK ≥ 1 , the K -EFCE are increasingly stricter notions of equilibria as K increases.
• We design an algorithm K Extensive-Form Regret Minimization ( K -EFR) which finds an εapproximate K -EFCE within (cid:101)
O
(max i X
i A K
∧ H
i /ε 2 ) iterations under full feedback (Section 4).At K = 1 , our linear in max
i X i dependence improves over the best known (cid:101)
O
(max i X 2 i ) dependence for computing ε -approximate EFCE.
At K > 1 , this gives a sharp result for efficientlycomputing (the stricter) K -EFCE , improving over the best known (cid:101) O
(max i
X 2 i
A K ∧ H
i /ε 2 ) iteration complexity of Morrill et al.
[33].• We further design Balanced K -EFR—a sample-based variant of K -EFR—for the more challengingbandit-feedback setting (Section 5).
Balanced K -EFR learns an ε -approximate K -EFCE within(cid:101)
O (max i X i A K
∧ H
+
i /ε 2 ) episodes of play.
This is the first line of results for learning EFCE andK -EFCE from bandit feedback, and the linear in X i dependence matches information-theoreticlower bounds.
Technically, our bandit-feedback result builds on a novel stochastic wide rangeregret minimization algorithm SWRH EDGE , as well as sample-based estimators of counterfactual
loss functions using newly designed sampling policies, which may be of independent interest.
1.1 Related work
Computing Correlated Equilibria from full feedback
The notion of Extensive-Form Correlated
Equilibria (EFCE) in IIEFGs is introduced in Von Stengel and Forges [39].
Huang and von Stengel[24] design the first polynomial time algorithm for computing EFCEs in multi-player IIEFGs fromfull feedback, using a variation of the Ellipsoid against hope algorithm [35, 25].
Farina et al.
[16] laterpropose a min-max optimization formulation of EFCEs which can be solved by first-order methods.
Celli et al.
[11] and its extended version [20] design the first uncoupled no-regret algorithm forcomputing EFCEs.
Their algorithms are based on minimizing the trigger regret (first considered in
Dudik and Gordon [13], Gordon et al.
[22]) via counterfactual regret decomposition [44].
Morrillet al.
[33] propose the stronger definition of “Behavioral Correlated Equilibria” (BCE) using general“behavioral deviations”, and design the Extensive-Form Regret Minimization (EFR) algorithm tocompute a BCE by using a generalized version of counterfactual regret decomposition.
They alsopropose intermediate notions such as a “Twice Informed Partial Sequence” (TIPS) (and its K -shotgeneralization) as an interpolation between the EFCE and BCE.
Our definition of K -EFCE offers anew interpolation between the EFCE and BCE that is different from theirs, as the deviating playerin
K -EFCE does not observe and does not follow recommended actions after K deviations hashappened, whereas the deviator in K -shot Informed Partial Sequence resumes to following after
K deviations has happened.
The iteration complexity for computing an ε -approximate correlatedequilibrium in both [20, 33] scales quadratically in max i ∈ [ m ] X i .
Our K -EFR algorithm for the full feedback setting builds upon the EFR algorithm, but specializes to the notion of K -EFCE , andachieve an improved linear in max i ∈ [ m ]
X i iteration complexity.
Apart from the EFCE, there are other notions of (coarse) correlated equilibria in IIEFGs such as
Normal-Form Coarse-Correlated Equilibria (NFCCE) [44, 10, 9, 18, 43], Extensive-Form CoarseCorrelated Equilibria (EFCCE) [17], and Agent-Form (Coarse-)Correlated Equilibria (AF(C)CE) [37,39]; see [32, 33] for a detailed comparison.
All above notions are either weaker than or incomparablewith the EFCE, and thus results there do not imply results for computing EFCE.
Learning Equilibria from bandit feedback A line of work considers learning Nash Equilibria(NE) in two-player zero-sum IIEFGs and NFCCE in multi-player general-sum IIEFGs from banditfeedback [29, 19, 15, 21, 42, 40, 27, 5]; Note that the NFCCE is weaker than (and does not implyresults for learning) EFCE.
Dudík and Gordon [14] consider sample-based learning of EFCE insuccinct extensive-form games; however, their algorithm relies on an approximate Markov-Chain
Monte-Carlo (MCMC) sampling subroutine that does not lead to an end-to-end sample complexityguarantee.
To our best knowledge, our results are the first for learning the EFCE (and K -EFCE )under bandit feedback.
Preliminaries
We formulate IIEFGs as partially-observable Markov games (POMGs) with tree structure and perfectrecall, following [27, 5].
For a positive integer i , we denote by [ i ] the set { 1 , 2 , · · · , i } .
For a finite set
A , we let ∆( A ) denote the probability simplex over A .
Let (cid:0) nm (cid:1) denote the binomial coefficient (i.e.number of combinations) of choosing m elements from n different elements, with the conventionthat (cid:0) nm (cid:1)
=
if m > n .
We use m ∧ n to denote min { m, n } .
Partially observable Markov game We consider an episodic, tabular, m -player, general-sum,partially observable Markov game
[Equation]
where S is the state space of size |S| = S
; X i is the space of information sets (henceforth infosets )for the i th player, which is a partition of S (i.e., x i ⊆ S for all x i ∈
X i , and S =
⊔ x
i ∈X i x i where⊔ stands for disjoint union) with size |X i | =
X i , and we also use x i : S → X i to denote the i thplayer’s emission (observation) function; A i is the action space for the i th player with size |A
i | =
A i ,and we let A
=
A 1 × · · · A m denote the space of joint actions a = ( a 1 , . .
. , a m ) ; H ∈ Z ≥ 1 isthe time horizon; p 0 ∈ ∆( S ) is the distribution of the initial state s 1 ; p h : S ×
A → ∆( S ) aretransition probabilities where p h ( s h
+1 | s h , a h ) is the probability of transiting to the next state s h +1from state-action ( s h , a h ) ∈ S × A ; and r i,h : S
× A → [0 , 1] is the deterministic 1 reward functionfor the i th player at step h .
Tree structure and perfect recall assumption We use a POMG with tree structure and perfect recall to formulate imperfect information games, following [27, 5].
We assume that the game has a treestructure: for any state s ∈ S , there is a unique step h and history ( s 1 , a 1 , . . .
, s h − 1 , a h − 1 , s h = s )to reach s .
Precisely, for any policy of the players, for any realization of the game (i.e., trajectory) ( s ′ k , a ′ k ) k ∈ [ H ] , conditionally to s ′ l
= s , it almost surely holds that l = h and ( s ′ 1 , . . .
, s ′ h ) =(
s 1 , . . .
, s h ) .
We also assume perfect recall, which means that each player remembers its past information sets and actions.
In particular, for each information set (infoset) x i ∈
X i for the i th player, there isa unique history of past infosets and actions ( x i, 1 , a i, 1 , . . .
, x i,h − 1 , a i,h − 1 , x i,h = x i ) leading to x i .
This requires that X i can be partitioned to H subsets ( X
i,h ) h ∈ [ H ] such that x i,h ∈ X i,h is reachableonly at time step h .
We define X i,h :=
|X i,h | .
Similarly, the state set S can be also partitioned into
H subsets ( S h ) h ∈
[ H ] .
As we mostly focus on the i th player, we use x h to also denote x i,h and usethem interchangeably.
For s ∈ S and x i ∈
X i , we write s ∈ x
i if infoset x i contains the state s .
With an abuse ofnotation, for s ∈ S , we let x i ( s ) denote the i th player’s infoset that s belongs to.
For any h < h ′ ,x i,h ∈
X i,h , x i,h ′ ∈
X i,h ′ , we write x i,h ≺ x i,h ′ if the information set x i,h ′ can be reached frominformation set x i,h by i th player’s actions; we write ( x i,h , a i,h ) ≺ x i,h ′ if the infoset x i,h ′ can be reached from infoset x i,h by i th player’s action a i,h .
For any h < h ′
and x i,h ∈
X i,h , welet C h
′ ( x i,h , a i,h ) :
= { x ∈ X i,h ′ :
( x i,h , a i,h ) ≺ x } and C h ′ ( x i,h ) :
= { x ∈ X i,h ′ :
x i,h ≺ 139 x } = ∪ a i,h ∈A i
C h ′
( x i,h , a i,h ) denote the infosets within the h ′ -th step that are reachable from (i.e.children of) x
i,h or ( x i,h , a i,h ) , respectively.
For shorthand, let C ( x i,h , a i,h ) :
=
C h
+1 ( x i,h , a i,h ) and 141 C ( x i,h ) :
=
C h
+1 ( x i,h ) denote the set of immediate children.
Policies We use π i =
π i,h ( ·| x i,h ) } h ∈
[ H ] ,x i,h ∈X i,h to denote a policy of the i th player,
where eachπ i,h ( ·| x i,h ) ∈ ∆( A i ) is the action distribution at infoset x i,h .
We say π i is a pure policy if π i,h ( ·
| x i,h )takes some single action deterministically for any ( h, x i,h ) ; in this case we let π i ( x i,h ) =
π i,h ( x i,h )denote the action taken at infoset x i,h for shorthand.
We use π = { π i } i ∈ [ m ] to denote a product policyfor all players, and let π − i = { π j } j ∈ [ m ] ,j ̸
= i denote policies of all players other than the i th player.
We call π a pure product policy if π i is a pure policy for all i ∈ [ m ] .
Let Π i denote the set of allpossible policies for the i th player and Π = (cid:81) i ∈
[ m ] Π i denote the set of all possible product policies.
Any probability measure π on Π induces a correlated policy , which executes as first sampling aproduct policy π = { π i } i ∈ [ m ] ∈ Π from probability measure π and then playing the product policyπ .
We also use π to denote this policy.
A correlated policy π can be viewed as a mediator of thegame which samples π ∼ π before the game starts, and privately recommends action sampled fromπ i ( ·| x i ) to the i th player when infoset x i ∈
X i is visited during the game.
Reaching probability With the tree structure assumption, for any state s h ∈ S h and actions a ∈ A ,there exists a unique history ( s 1 , a 1 , . . .
, s h
= s, a h = a ) ending with ( s h
= s, a h = a ) .
Given anyproduct policy π , the probability of reaching ( s h , a h ) at step h can be decomposed as
[Equation]
where we have defined the sequence-form transitions p 1: h and sequence-form policies π i, 1: h as
[Equation]
where ( s h ′ , a h ′ )
h ′ ≤ h − 1 is the unique history of states and actions that leads to s h by the treestructure; x i,h
= x i ( s h ) is the i th player’s infoset at the h -th step, and ( x i,h ′ , a i,h ′ )
h ′ ≤ h − 1 isthe unique history of infosets and actions that leads to x i,h by perfect recall.
We also defineπ i,h : h ′ ( x i,h ′ , a i,h ′ ) :
= (cid:81)
h ′ h ′′ =
h π i,h ′′ ( a i,h ′′ | x i,h ′′ ) for any 1 ≤ h ≤ h ′ ≤ H .
Value functions and counterfactual loss functions Let V πi :=
E π [ (cid:80)
Hh =1 r
i,h ] denote the valuefunction (i.e. expected cumulative reward) for the i th player under policy π .
By the product formof the reaching probability in (1), the value function V πi admits a multi-linear structure over thesequence-form policies.
Concretely, fixing any sequence of product policies { π t } Tt =1 where eachπ t = { π ti } i ∈ [ m ] , we have
[Equation]
For any sequence of policies { π t } Tt =1 , we also define the counterfactual loss functions [44]{ L ti,h ( x i,h , a i,h ) } i,h,x i,h ,a i,h as:
[Equation]
i,
[Equation] [Equation]
i,
[Equation] [Equation]
Intuitively, L ti,h ( x i,h , a i,h ) measures the i th player’s expected cumulative loss (one minus reward)conditioned on reaching ( x i,h , a i,h ) , weighted by the (environment) transitions and all other players’policies π t − i at all time steps, and the i th player’s own policy π ti from step h
+ 1 onward.
We willomit the i subscript and use L th to denote the above when clear from the context.
Feedback protocol We consider two standard feedback protocols for our algorithms: full feedback ,and bandit feedback .
In the full feedback case, the algorithm can query a product policy π t ={ π ti } i ∈ [ m ] in each iteration and observe the counterfactual loss functions { L ti,h ( x i,h , a i,h ) } i,h,x i,h ,a i,hexactly 2 .
In the bandit feedback case, the players can only play repeated episodes with some policiesand observe the trajectory of their own infosets and rewards from the environment.
K Extensive-Form Correlated Equilibria
We now introduce the definition of K Extensive-Form Correlated Equilibria ( K -EFCE ) and establishits relationship with existing notions of correlated equilibria in IIEFGs.
3.1 Definition of K -EFCE
Intuitively, a K -EFCE is a correlated policy in which no player can gain if allowed to deviate fromthe observed recommended actions K times, and forced to choose her own actions without observingfurther recommendations afterwards.
To state its definition formally, we categorize all possiblerecommendation histories (henceforth rechistories ) at each infoset x i,h ∈
X i,h (for the i th player) intotwo types, based on whether the player has already deviated K times from past recommendations:
(1) A Type-I rechistory ( ≤ K − 1 deviations happened) at x i,h is any action sequence b 1: h − 1 ∈
A h − 1 isuch that (cid:80) h − 1 k =1 1 { a k ̸ = b k } ≤ K − 1 , where ( a 1 , . . .
, a h − 1 ) is the unique sequence of actionsleading to x i,h .
Let Ω (I) ,K i ( x i,h ) denote the set of all Type-I rechistories at x i,h .
(2) A Type-II rechistory ( K deviations happened) at x i,h is any action sequence b 1: h ′ ∈
A h ′ i
withlength h ′ <
h such that (cid:80) h
′ − 1 k =1 1 { a k ̸ = b k } =
K − 1 and a k ̸ = b k , where ( a 1 , . .
. , a h − 1 ) isthe unique sequence of actions leading to x i,h .
Let Ω (II) ,K i ( x i,h ) denote the set of all Type-IIrechistories at x i,h .
We now define a K -EFCE strategy modification ( 0 ≤ K ≤ ∞ ) for the i th player.
Definition 1 ( K -EFCE strategy modification) .
A K -EFCE strategy modification ϕ (for the
i th
player)is a mapping ϕ of the following form: At any infoset x i,h ∈
X i,h , for any Type-I rechistory b 1: h − 1 ∈Ω (I) ,K i ( x i,h ) , ϕ swaps any recommended action b h into ϕ ( x i,h , b 1: h − 1 , b h ) ∈
A i ; for any Type-IIrechistory b 1: h ′ ∈ Ω (II) ,K i ( x i,h ) , ϕ directly takes action ϕ
( x i,h , b 1: h ′ )
∈
A i .
Let Φ Ki denote the set of all possible K -EFCE strategy modifications for any 0 ≤ K ≤ ∞ .
Formally,for any ϕ ∈ Φ Ki and any pure policy π i ∈
Π i , we define the modified policy ϕ
⋄ π i as in Algorithm 1.
We parse the modified policy ϕ ⋄ π i (Algorithm 1) as follows.
Upon receiving the infoset x i,h ateach step h , the player has the rechistory b containing all past observed recommended actions.
Then,if b is Type-I, i.e. at most K − 1 deviations have happened (Line 4), then the player observes the current recommended action b h
= π
i,h ( x i,h ) , takes a potentially swapped action a h
= ϕ
( x i,h , b , b h )(Line 6), and appends b h to the recommendation history (Line 7).
Otherwise, ( x i,h , b ) is Type-II,i.e. K deviations have already happened.
In this case, the player does not observe the recommendedaction, and instead takes an action a h
= ϕ
( x i,h , b ) , and does not update b (Line 10).
We now define K -EFCE as the equilibrium induced by the K -EFCE strategy modification setΦ Ki .
With slight abuse of notation, we define ϕ ⋄ π for any correlated policy π to be the policy( ϕ ⋄ π i ) × π − i where π ∼ π is the product policy sampled from π .
Definition 2 ( K -EFCE ) .
A correlated policy π is an ε -approximate K Extensive-Form Correlated
Equilibrium ( K -EFCE )
if
[Equation]
We say π is an (exact) K -EFCE
if K -EFCEGap( π ) = 0 .
3.2 Properties of K -EFCE
The K -EFCE is closely related to various existing definitions of correlated equilibria in IIEFGs.
We show that the special case of K = 1 is equivalent to the existing definition of EFCE basedon trigger policies (Proposition C.1); The K -EFCE are indeed stricter equilibria as K increases(Proposition C.2);
The two extreme cases K = 0 and K = ∞ are equivalent to (Normal-Form)
Coarse Correlated Equilibrium and the “Behavioral Correlated Equilibria” of [33], respectively(Proposition C.3).
Due to the space limit, the full statements and proofs are deferred to Appendix C.
Computing K -EFCE from full feedback
Algorithm description We first present our algorithm for computing K -EFCE in the full-feedbacksetting.
Our algorithm K Extensive-Form Regret Minimization ( K -EFR), described in Algorithm 2,is
an uncoupled no-regret algorithm aiming to minimize the following K -EFCE regret
[Equation]
By standard online-to-batch conversion, achieving sublinear K -EFCE regret for every player impliesthat the average joint policy over all players is
an approximate K -EFCE
(Lemma E.1).
At a high level, our Algorithm 2 builds upon the EFR algorithm of Morrill et al.
[33] to minimizethe K -EFCE regret R Ti,K , by maintaining a regret minimizer R x
i,h (using algorithm R EG A LG ) ateach infoset x i,h ∈
X i that is responsible for outputting the policy π ti ( ·
| x i,h ) ∈ ∆
A i (Line 8) whichcombine to give the overall policy π ti for the t -th iteration.
Core to our algorithm is the requirement that R x i,h ∼ R EG A LG should be able to minimize regretswith time-selection functions and strategy modifications (also known as the wide range regret ) [30, 6].
Specifically, R x i,h needs to control the regret
[Equation]
(7) for all possible Type-I rechistories b 1: h −
1 ∈ Ω (I) ,K i ( x i,h ) simultaneously, where (cid:81)
h − 1 k
=
π ti ( b k | x k )
=:
S tb [Equation] is the time-selection function (i.e. a weight function) associated with this b 1: h − 1 (cf.
Line 5),and Ψ s
= { ψ : A i → A i } is the set of all swap modifications from the action set A i onto itself.(An analogous regret for Type-II rechistories is also controlled by R x
i,h .)
Controlling these “local"regrets at each x i,h guarantees that the overall K -EFCE regret is bounded, by the K -EFCE regretdecomposition (cf.
Lemma E.2).
To control this wide range regret, we instantiate R EG A LG as WRH EDGE (Algorithm 4; cf.
Appendix
A.2), which is similar as the wide regret minimization algorithm in [26], with a slight modification ofthe initial weights suitable for our purpose (cf.
The learning rate is set as
[Equation]
for all x i,h ∈
X i .
With this algorithm in place, at each iteration, R x i,h observes all time selectionfunctions (Line 7), computes the policy for the current iteration (Line 8), and then observes the lossvector L ti,h ( x i,h , · ) (Line 9) that is useful for updating the policy in the next iteration.
Theoretical guarantee We are now ready to present the theoretical guarantee for K -EFR.
Theorem 3 (Computing K -EFCE from full feedback) .
For any 0 ≤ K ≤ ∞ , ε ∈ (0 , H ] , let allplayers run Algorithm 2 together in a self-play fashion where R EG A LG is instantiated as Algorithm4 with learning rates specified in (8).
Let π t = { π ti } i ∈
[ m ] denote the joint policy of all players at thet ’th iteration.
Then the average policy π = Unif( { π t } Tt =1 ) satisfies K -EFCEGap( π ) ≤ ε , as longas the number of iterations
[Equation]
where ι = log(max i ∈ [ m ] A i ) is a log factor and O ( · ) hides poly( H ) factors.
In the special case of K = 1 , Theorem 3 shows that K -EFR can compute an ε -approximate 1-EFCEwithin (cid:101)
O (max i ∈
[ m ] X i
A
i /ε 2 ) iterations.
This improves over the existing (cid:101) O
(max i ∈ [ m ] X 2 i
A 2 i /ε 2 ) iteration complexity of Celli et al.
[11], Farina et al.
[20] by a factor of X i A i .
Also, compared with the iteration complexity of the optimistic algorithm of [3] which is at least (cid:101) O (max i ∈ [ m ] X 4 − δ
i
A 4 / 3 i /ε 4 / 3 ) 3 ,we achieve lower X i dependence (though worse ε dependence).
For 1 < K ≤ ∞ , Theorem 3 gives a sharp (cid:101) O
( (cid:0) HK ∧ H (cid:1) (max i ∈ [ m ] X i
A K ∧ H i ) /ε 2 ) iteration complexity for computing K -EFCE .
This improves over the (cid:101) O
( (cid:0) HK ∧ H (cid:1) (max i ∈ [ m ] X 2 i
A K ∧ H i ) /ε 2 ) rateof EFR [33] instantiated to the K -EFCE problem.
Also, note that although the term A K ∧ H iis exponential in K (for K ≤ H ), this is sensible since it is roughly the same scale as thenumber of possible recommendation histories, which is also the “degree of freedom" within a
K -EFCE strategy modification.
Apart from learning equilibria, Algorithm 2 also achieves a low
K -EFCE regret when controlling the i th player only and facing potentially adversarial opponents:
R Ti,K ≤ (cid:101) O
( (cid:113)(cid:0) HK ∧ H (cid:1)
X i
A K ∧ H
i
T ) (Corollary F.1).
In particular, the (cid:101) O
( √ X i T ) scaling is optimal upto log factors, due to the fact that R Ti,K ≥ R Ti, 0 (i.e. the vanilla regret) and the known lower bound
R Ti, 0 ≥ Ω( √ X i T ) in IIEFGs [41].
Proof overview Our Theorem 3 follows from a sharp analysis on the K -EFCE regret of Algorithm 2, by incorporating (i) a decomposition of the K -EFCEGap into local regrets at each infosetwith tight leading coefficients (Lemma E.2), and (ii) loss-dependent upper bounds for the widerange regret of WRH EDGE (Lemma A.2), which when plugged into the aforementioned regretdecomposition yields the improved dependence in ( X i
A K ∧ H i ) over the analysis of Morrill et al.
[33](Lemma F.1 & F.2).
The full proof can be found in Appendix F.
Input: Policy π ti , π t − i .
Balanced exploration policies { π ⋆,hi } h ∈
[ H ] .
1: for K ≤ h ′ <
h ≤ H , W ⊆
[ h ′ ] with | W | =
K and ending in h ′ do 2:
Set policy π t, ( h,h ′ ,W ) i ← ( π ⋆,hi,k ) k ∈ W ∪{ h ′
+1 ,...,h } · ( π ti,k ) k ∈
[ h ′ ] \ W · π ti, ( h +1): H .
Play π t, ( h,h ′ ,W ) i × π t − i for one episode, observe trajectory ( x t, ( h,h ′ ,W ) i,, a t, ( h,h ′ ,W ) i,, r t, ( h,h ′ ,W ) i,, . . .
, x t, ( h,h ′ ,W ) i,H , a t, ( h,h ′ ,W ) i,H , r t, ( h,h ′ ,W ) i,H ) .
4: for all ( x i,h , b 1: h ′ ) ∈ Ω (II) ,K i do 5:
Find ( x i, 1 , a 1 ) ≺ · · · ≺ ( x i,h − 1 , a h − 1 ) ≺ x i,h . 6:
Set W ← { k ∈ [ h ′ ] : b k ̸ = a k } 7:
Construct loss estimator for all a ∈ A i (below a h ∈
A i is arbitrary): (cid:101)
L t ( x i,h ,b 1: h ′ )
( a ) ← 1 (cid:110) ( x t, ( h,h ′ ,W ) i,h , a t, ( h,h ′ ,W ) i,h ) =
( x i,h , a ) (cid:111)
π t, ( h,h ′ ,W ) i, 1: h ( x i,h , a ) ·
H (cid:88) h ′′ =
h (cid:16) 1 − r t, ( h,h ′ ,W ) i,h ′′ (cid:17) .
Output: Loss estimators (cid:110)(cid:101) L t ( x i,h ,b 1: h ′ ) ( · ) (cid:111) ( x i,h ,b 1: h ′ ) ∈ Ω (II) ,K i .
Learning K -EFCE from bandit feedback
We now present Balanced K -EFR, a sample-based variant of K -EFR that achieves a sharp samplecomplexity in the more challenging bandit feedback setting.
Our algorithm relies on the followingbalanced exploration policy [19, 5].
Recall that |C h ( x i,h ′ , a i,h ′ ) | is the number of descendants of( x i,h ′ , a i,h ′ ) within the h -th layer of the i th player’s game tree (cf.
Section 2).
Definition 4 (Balanced exploration policy) .
For any 1 ≤ h ≤ H , we define π ⋆,hi , the ( i th player’s)balanced exploration policy for layer
h asπ ⋆,hi,h ′ ( a h ′
| x h ′ ) :
=
|C h ( x i,h ′ , a i,h ′ )
| / |C h
( x i,h ′ )
| for all ( x i,h ′ , a i,h ′ )
∈
X i,h ′ ×
A i , h ′ ≤ h − 1 , and π ⋆,hi,h ′ ( a i,h ′ | x i,h ′ ) :
= 1 /A
i for h ′ ≥ h .
Note that there are H such policies, one for each layer h .
We remark that the construction of π ⋆,hirequires knowledge about the descendant relationships among the i th player’s infosets, which is amild requirement (e.g. can be efficiently obtained from one traversal of the i th player’s game tree).
Algorithm description (sampling part)
Our Balanced K -EFR (deferred to Algorithm 7) buildsupon the full feedback version of K -EFR (Algorithm 2).
The main new ingredient within Algorithm 7is to use sample-based loss estimators obtained by two balanced sampling algorithms (Algorithm 3& 6), one for each type of rechistories.
Here we present the sampling algorithm for Type-II rechistories in Algorithm 3; The sampling algorithm for Type-I rechistories (Algorithm 6) is designedsimilarly and deferred to Appendix G.1 due to space limit.
Algorithm 3 performs two main steps:
• Line 1-3 (Sampling): Construct policies { π t, ( h,h ′ ,W ) i } that are interlaced concatenations of thecurrent π ti and the balanced policy π ⋆,hi , and play one episode using each policy against π t − i .•
Line 7: Construct loss estimators { (cid:101)
L x i,h ,b 1: h ′ ( a ) } x i,h ,b 1: h ′ ,a by (9), which for each x i,h andb 1: h ′ ∈ Ω (II) ,K i ( x i,h ) is an unbiased estimator of counterfactual losses { L th ( x i,h , a ) } a ∈A i thatwill be used by Algorithm 7 to be fed into the regret minimization algorithm R EG A LG .
We remark that the sampling policies { π t, ( h,h ′ ,W ) i } in Algorithm 3 are interlaced concatenationsof π t i and π ⋆,hi along time steps h , where the policy to take at each h is determined by W .
Thesepolicies are generalizations of the sampling policies in the Balanced CFR algorithm of Bai et al.
[5](which can be thought of as a simple non-interlacing concatenation).
They allow time-selection awaresampling : Each loss estimator (cid:101)
L t ( x i,h ,b 1: h ′ ) ( · ) achieves low variance relative to the correspondingtime selection function S tb 1: h ′ .
Further, there is an efficient sharing of sampling policies, as here roughly (cid:0)
[Equation]
loss estimators (one for each ( x i,h , b 1: h ′ ) ) are constructed using only (amuch lower number of) H (cid:0) HK ∧ H (cid:1) policies.
Stochastic wide-range regret minimization Algorithm 7 requires the wide-range regret minimization algorithm R EG A LG to additionally handle the stochastic setting, i.e. minimize the wide-rangeregret (e.g. (7)) when fed with our sample-based loss estimators.
Here, we instantiate R EG A LG to be
SWRH EDGE (Algorithm 5), a stochastic variant of WRH EDGE , with hyperparameters
[Equation]
SWRH EDGE is a non-trivial extension of WRH EDGE to the stochastic setting, as in each round itadmits multiple sample-based loss estimators, one for each time selection function, with the samemean (cf.
Line 8).
This is needed since Algorithm 3 uses different sampling policies to construct theloss estimator (cid:101)
L tx
i,h ,b 1: h ′ ( · ) for each b 1: h ′ ∈ Ω (II) ,K i ( x i,h ) (cf.
Theoretical guarantee We now present our main result for the bandit feedback setting.
Theorem 5 (Learning K -EFCE from bandit feedback) .
For any 0 ≤ K ≤ ∞ , ε ∈ (0 , H ] andp ∈ [0 , 1) , letting all players run Algorithm 7 together in a self-play fashion for T iterations, with
R EG A LG instantiated as SWRH EDGE (Algorithm 5) with hyperparameters in (10) .
Let π t ={ π ti } i ∈
[ m ] denote the joint policy of all players at the t ’th iteration.
Then, with probability atleast 1 − p , the correlated policy π = Unif( { π t } Tt =1 ) satisfies K -EFCEGap( π ) ≤ ε , as long as
T ≥ O ( H 3 (cid:0) HK ∧ H (cid:1) (max i ∈ [ m ]
X i A K
∧ H
+1 i )
ι/ε 2 ) .
The total number of episodes played is
[Equation]
where ι = log(8 (cid:80) i ∈ [ m ] X i
A i /p ) is a log factor and O ( · ) hides poly( H ) factors.
To our best knowledge, Theorem 5 provides the first result for learning K -EFCE under banditfeedback.
The sample complexity (cid:101) O ( (cid:0) HK ∧ H (cid:1) 2 max i ∈ [ m ] ( X i A K
∧ H
+1 i ) /ε 2 )
(ignoring m , H factors)has only an (cid:0) HK ∧ H (cid:1)
A i additional factor over the iteration complexity in the full feedback setting(Theorem 3), which is natural—The (cid:0) HK ∧ H (cid:1) comes from the number of episodes sampled withineach iteration (Lemma G.1), and the A i arises from estimating loss vectors from bandit feedback.
Inparticular, the special case of K = 1 provides the first result for learning EFCEs from bandit feedback,with sample complexity (cid:101) O
(max i ∈ [ m ] X i
A 2 i /ε 2 ) .
We remark that the linear in X i dependence at all
K ≥ 0 is optimal, as the sample complexity lower bound for the K = 0 case (learning NFCCEs)is already Ω(max i ∈ [ m ]
X i A
i /ε 2 ) [5] 4 .
Also, the policies { π ti } Tt =1 maintained in Algorithm 7 alsoachieves sublinear K -EFCE regret.
However, strictly speaking, this is not a regret bound of ouralgorithm, as the sampling policies π t, ( h,h ′ ,W )
i actually used are not π ti .
Proof overview The proof of Theorem 5 builds on the analysis in the full-feedback case, andfurther relies on several new techniques in order to achieve the sharp linear in max i ∈ [ m ]
X i samplecomplexity: (1) A regret bound for the SWRH EDGE algorithm under the same-mean condition(Lemma A.3), which may be of independent interest; (2) Crucial use of the balancing propertyof π ⋆,hi (Lemma B.4) to control the variance of the loss estimators (cid:101)
L t ( x i,h ,b 1: h ′ ) ( · ) , which in turnproduces sharp bounds on the regret terms and additional concentration terms (Lemma G.4-G.9).
Thefull proof can be found in Appendix G.3.
Conclusion
This paper proposes K -EFCE , a generalized definition of Extensive-Form Correlated Equilibriain Imperfect-Information Games, and designs sharp algorithms for computing K -EFCE under fullfeedback and learning a K -EFCE under bandit feedback.
Our algorithms perform wide-range regretminimization over each infoset to minimize the overall K -EFCE regret, and introduce new efficientsampling policies to handle bandit feedback.
We believe our work opens up many future directions,such as accelerated techniques for computing K -EFCE from full feedback, learning other notions ofequilibria from bandit feedback, as well as empirical investigations of our algorithms.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Yes
(b) Did you describe the limitations of your work
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] As our
work is theoretical, we do not foresee any negative societal impacts
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[N/A]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board ) approvals, if applicable
N A]
[Table]
you the estimated hourly wage paid participants and the total amount
[Table]
spent on participant compensation
N A] [Table]