Efficient Φ -Regret Minimization in Extensive-Form Games via Online Mirror Descent
Anonymous Author(s)
AffiliationAddress email
Abstract
A conceptually appealing approach for learning Extensive-Form Games (EFGs)is to convert them to Normal-Form Games (NFGs).
This approach enables usto directly translate state-of-the-art techniques and analyses in NFGs to learning
EFGs, but typically suffers from computational intractability due to the exponentialblow-up of the game size introduced by the conversion.
In this paper, we address this problem in natural and important setups for the Φ -Hedge algorithm—
Ageneric algorithm capable of learning a large class of equilibria for NFGs.
We showthat Φ -Hedge can be directly used to learn Nash Equilibria (zero-sum settings),
Normal-Form Coarse Correlated Equilibria (NFCCE), and Extensive-Form Correlated Equilibria (EFCE) in EFGs.
We prove that, in those settings, the Φ -Hedgealgorithms are equivalent to standard Online Mirror Descent (OMD) algorithmsfor EFGs with suitable dilated regularizers, and run in polynomial time.
This newconnection further allows us to design and analyze a new class of OMD algorithms based on modifying its log-partition function.
In particular, we design animproved algorithm with balancing techniques that achieves a sharp (cid:101) O ( √ XAT )
EFCE-regret under bandit-feedback in an EFG with X information sets, A actions,and T episodes.
To our best knowledge, this is the first such rate and matches theinformation-theoretic lower bound.
Introduction
Extensive form games (EFGs) is a natural formulation for multi-player games with imperfect information and sequential play, which models real-world games such as Poker [9, 10], Bridge [45], Scotland
Yard [41], Diplomacy [7] and has many other important applications such as cybersecurity [32],auction [39], marketing [29].
In multi-player general-sum EFGs, computing an approximate Nashequilibrium (NE) [40] is PPAD-hard [15] and thus likely intractable.
A reasonable and computationally tractable solution concept in general-sum EFGs is the extensive-form correlated equilibria(EFCE)
It is known that, as long as each player runs an uncoupled dynamicsminimizing a suitable EFCE-regret, their average joint policy will converge to an EFCE [28].
Existing algorithms of minimizing the EFCE-regret are mostly built upon the regret decompositiontechniques [51], which utilize the structure of the game and the set of policy modifications [13, 38, 23,42].
For example, Morrill et al.
[38] decomposes the EFCE-regret to local regrets at each informationset (infoset) with each of them handled by a local regret minimizer; Farina et al.
[23] utilizes thetrigger structure of the policy modification set to decompose the regret to external-like regrets.
There are at least two alternative approaches to designing regret minimization algorithms for EFGs.
The first is to convert a EFG to a normal-form game (NFG) and use NFG-based algorithms such asΦ -Hedge [28].
This approach typically admits simple algorithm designs and sharp regret bounds bydirectly translating existing results in NFGs [44].
However, the conversion introduces an exponential blow-up in the game size, and makes such algorithms computationally intractable in general.
Thecomputational efficiency of these NFG-based algorithms is recently investigated by Farina et al.
[25]in the external regret minimization problem, who provided an efficient implementation of an NFGbased algorithm using “kernel tricks”.
The second is to use Online Mirror Descent (OMD) algorithmsvia suitably designed regularizers over the parameter space.
This approach has been successfullyimplemented in minimizing the external regret [35] but not yet generalized to the EFCE-regret, as itremains unclear how to design suitable regularizers for the policy modification space.
In this paper, we develop the first line of EFCE-regret minimization algorithms along both lines ofapproaches above, and identify an equivalence between them.
We consider EFCE-regret minimizationin EFGs with X infosets, A actions, and maximum L 1 -norm of sequence-form policies bounded by∥ Π ∥ 1 (cf.
Section 2.2 for the formal definition).
Our contributions can be summarized as follows:
• We present an efficient implementation of the Φ -Hedge algorithm for minimizing the extensiveform trigger regret, by recursively evaluating the gradient of a log-partition function (Section 3.1).The implementation
further
reveals that this algorithm (via reparametrization) is equivalent to an
OMD algorithm with dilated regularizers, which we term as EFCE-OMD (Section C.1).
• We show that EFCE-OMD achieves trigger regret bound (cid:101) O
( (cid:112)
∥
Π ∥ 1 T ) under full feedback and(cid:101) O
( (cid:112) X ∥ Π ∥ 1 AT ) under bandit feedback (Section 3.3).
Notably, the proofs are done using the
corresponding NFG analysis straightforwardly
and is independent of the actual implementation.
• We design an improved algorithm Balanced EFCE-OMD
and show that it achieves a sharp
(cid:101) O ( √ XAT ) trigger regret under bandit feedback (Section 4).
This improves over EFCE-OMD by
a factor of ∥ Π ∥ 1 and is the first to match the information-theoretic lower bound.
The algorithmworks by modifying the above log-partition function using a variety of balancing techniques,and is equivalent to another OMD algorithm (but no longer an NFG algorithm).• As another example of our framework, we show that the Φ -Hedge algorithm for vanilla (external)regret minimization in EFGs, along with its efficient implementation via “kernelization” developed recently in [25], is actually equivalent to standard OMD with dilated entropy (Section 5).
1.1 Related work
Φ -regret minimization and correlated equilibrium The Φ -regret minimization framework wasintroduced in Greenwald and Jafari [28] and Stoltz and Lugosi [44].
In particular, Greenwald and
Jafari [28] showed that uncoupled no Φ -regret dynamics leads to Φ -correlated equilibria, a generalizednotion of correlated equilibria introduced by Aumann [5].
Stoltz and Lugosi [44] then developed afamily of Φ -regret minimization algorithms using the fixed-point method (including the Φ -Hedgealgorithm considered in this paper), and derived explicit regret bounds.
Two important special casesof Φ -regret are the internal regret and swap regret in normal-form games [43, 8].
A recent line ofwork developed algorithms with O (polylog T ) swap regret bound in normal-form games [2, 3].
Regret minimization in EFG from full feedback
A line of work considers external regret minimization in EFGs from full feedback [51, 12, 11, 21, 50].
In particular, Zhou et al.
[50] achieves(cid:101) O ( √ XT ) external regret.
The recent work of Farina et al.
[25] develops the first algorithm to achieve(cid:101)
O ( ∥ Π ∥ 1 polylog T ) external regret in EFGs by converting it to an NFG and invoking the fast rate of
Optimistic Hedge [16], along with an efficient implementation via the “kernel trick”.
Our Φ -regretframework covers their algorithm as a special case, and we further show that their algorithm (alongwith its efficient implementation) is equivalent to the standard OMD with dilated entropy.
The notion of Extensive-Form Correlated Equilibria (EFCE) in EFGs was introduced in Von Stengeland Forges [46].
Optimization-based algorithms for computing computing EFCEs in multi-player
EFGs from full feedback have been proposed in Huang and von Stengel [31], Farina et al.
Gordon et al.
[27] first proposed to use uncoupled EFCE-regret minimization dynamics to compute
EFCE; however, they do not explain how to efficiently implement each iteration of the dynamics.
Recent works [13, 23, 38, 42] developed uncoupled EFCE regret minimization learning dynamicswith efficient implementation; All of these algorithms are based on counterfactual regret decomposition [51] and minimizing each trigger regret (first considered by Dudik and Gordon [17], Gordonet al.
[27]) using a different regret minimizer.
Celli et al.
[13] decomposed the regret to each laminarsubtree, but they did not give an explicit regret bound.
Farina et al.
[23] decomposed the regret to each trigger sequence and used CFR type algorithm to minimize the regret on each trigger sequenceand achieved an ˜ O ( √ X 2 T ) EFCE-regret bound.
Morrill et al.
[38], Song et al.
[42] decomposedthe regret to each information set and use regret minimization algorithms with time-selection functions [8, 33] to minimize the regret on each information set, giving (cid:101) O ( √ X 2 T ) and (cid:101) O ( √ XT ) regretbounds respectively.
In this paper, we show that the simple Φ -Hedge algorithm, which has an efficientimplementation and an intuitive interpretation, can also achieve the state-of-art (cid:101) O ( √ XT ) regretbound in the full feedback setting.
Regret minimization in EFG from bandit feedback Minimizing the external regret in EFGs frombandit feedback is considered in a more recent line of work [36, 22, 19, 24, 49, 47, 34, 6].
Dudík and
Gordon [18] consider sample-based learning of EFCE in succinct extensive-form games; however,their algorithm relies on an approximate Markov-Chain Monte-Carlo sampling subroutine that doesnot lead to a sample complexity guarantee.
A concurrent work by Song et al.
[42] also achieves (cid:101) O
( X/ε 2 ) sample complexity for learning EFCEunder bandit feedback (when only highlighting X ) using the Balanced K -EFR algorithm.
Our workachieves the same linear in X sample complexity, but using a very different algorithm (Balanced
EFCE-OMD).
We also remark that the algorithm of [42] cannot minimize the EFCE-regret againstadversarial opponents from bandit feedback like our algorithm, as their algorithm requires playingmultiple episodes against a fixed opponent, which is infeasible when the opponent is adversarial.
Preliminaries
2.1 Φ -regret minimization and Φ -Hedge algorithm
Consider a generic linear regret minimization problem on a policy set Π ⊂ R d ≥ 0 with respect to apolicy modification set Φ ⊂ R d × d .
Here Π is a convex compact subset of R d , and Φ is a convexcompact subset of R d × d , where each ϕ ∈ Φ is a policy modification function which is a lineartransformation from R d to R d that maps Π to itself ( ϕ (Π) ⊆
Π ).
For any algorithm that plays policies{ µ t } Tt =1 within T rounds and receives loss functions { ℓ t } Tt =1 ⊂ R d ≥ 0 , the Φ -regret is defined as
[Equation]
The Φ -regret subsumes the vanilla regret (i.e. external regret) as a special case by taking Φ to bethe set of all constant modifications Φ ext :=
{ ϕ µ ⋆ : µ ⋆ ∈ Π } where ϕ µ ⋆ µ = µ ⋆ for all µ ∈ Π .
Another widely studied example is the swap regret [8] (and the closely related internal regret [26])for normal-form games, where Π = ∆ d is the probability simplex over d actions, and Φ is the set ofall stochastic matrices (i.e. those mapping ∆
d to itself).
A primary motivation for minimizing theΦ -regret is for computing various types of Correlated Equilibria (CEs) in multi-player games usingthe online-to-batch conversion (see e.g. [14]), which has been established in many games and hasbeen a cornerstone in the online learning and games literature.
Φ -Hedge algorithm
A widely used strategy for minimizing the Φ -regret is to use any (black-box)linear regret minimization algorithm on the Φ set to produce a sequence of { ϕ t } Tt =1 ⊂ Φ , combinedwith the fixed point technique (e.g. [43])—Output policy µ t that satisfies the fixed-point equationϕ t µ t = µ t in each round t .
In the common scenario where Φ is the convex hull of a finite number ofvertices , i.e. Φ = conv(Φ 0 ) where Φ 0 is a finite subset of Φ , a standard regret minimization algorithmover Φ is Hedge (a.k.a.
Exponential Weights) [4], leading to the Φ -Hedge algorithm (Algorithm 1).
It is a standard result ([44], see also Lemma A.1) that Algorithm 1 achieves Φ -regret bound
[Equation]
By choosing η > 0 , this result implies a quite desirable bound Reg Φ ( T ) ≤ L (cid:112) 2 log | Φ 0 | · T in thefull-feedback setting (assuming bounded loss ⟨ ϕµ t , ℓ t ⟩ ≤ L ), and can also be used to prove regretbounds in the bandit-feedback setting.
2.2 Extensive-form games (EFGs) and extensive-form trigger regret
In this paper, we consider m -player imperfect-information extensive-form games (EFGs) with perfectrecall (see Appendix B.1 for detailed definitions).
For the purpose of this work, we consider an
Algorithm
[Equation] [Equation]
alternative formulation of EFGs—Tree-Form Adversarial Markov Decision Processes (TFAMDP)
This model is equivalent to studying EFGs from the perspective of a single player, while treating allother players as adversaries who can change both transitions and rewards in each round.
Tree-form adversarial MDP
We consider an episodic, tabular TFAMDP which consists of thefollowings ( H, {X h } h ∈ [ H ] , A , T , { p th } h ∈{ 0 }∪ [ H ] ,t ≥ 1 , {
R th } h ∈ [ H ] ,t ≥ 1 ) .
Here H ∈ N + is the horizonlength;
X h is the space of information sets (henceforth infosets ) at step h with size |X h
| =
X hand (cid:80) Hh =1 X h
=
X ; A is the action space with size |A| =
A .
Next, T = {C ( x, a ) } ( x,a ) ∈X×Adefines the tree structure over the infosets and actions, where C ( x h , a h ) ⊂
X h +1 denotes the set ofimmediate children of ( x h , a h ) .
Furthermore, {C ( x h , a h ) } ( x h ,a h ) ∈X h ×A forms a partition of X h +1 .
143 It directly follows from the tree structure of TFAMDP that the player has perfect recall , i.e., for anyinfoset x h ∈ X h , there is a unique history ( x 1 , a 1 , . . .
, x h − 1 , a h − 1 ) that leads to x h .
Furthermore,p t 0 ( · ) ∈
∆ X 1 is the initial distribution over X 1 at episode t ; p th ( ·| x h , a h ) is the transition probabilityfrom ( x h , a h ) to its immediate children C ( x h , a h ) at episode t ; R th ( ·| x h , a h ) is the distribution of thestochastic reward r ∈ [0 , 1] received at ( x h , a h ) at episode t , with expectation R th ( x h , a h ) .
At the beginning of episode t , an adversary will first choose the initial distribution p t 0 , transition{ p th } h ∈ [ H ] , and reward distribution { R th } h ∈ [ H ] .
Then in the bandit feedback setting, at each step 150 h , the player observes the current infoset x h , takes an action a h , receives a bandit feedback of thereward r th ∼ R th ( ·| x h , a h ) , and the environment transitions to the next state x h +1 ∼ p th ( ·| x h , a h ) .
Policies We use µ = { µ h ( ·| x h ) } h ∈ [ H ] ,x h ∈X h to denote a policy, where each µ h ( ·| x h ) ∈
∆ A is theaction distribution at infoset x h .
We say µ is a deterministic policy if µ h ( ·| x h ) takes some singleaction with probability 1 for any ( h, x h ) .
Let Π denote the set of all possible policies.
We denote thesequence form representation of policy µ ∈ Π by
[Equation]
(3) where ( x 1 , a 1 , . . .
, x h − 1 , a h − 1 ) is the unique history of x h .
We also identify µ as a vector in R XA ≥ 0 ,whose ( x h , a h ) -th entry is equal to its sequence form µ 1: h ( x h , a h ) .
Let ∥ Π ∥ 1 := max µ ∈ Π ∥ µ ∥ 1 ,which admits bound ∥ Π ∥ 1 ≤ X but can in addition be smaller (cf.
Appendix B.3).
Expected loss function Given any policy µ t at round t , the total expected loss received at round t(which equals to H minus the total rewards within round t ) is given by
[Equation]
where the loss function for the t -th round is given by ℓ
[Equation]
=
[Equation]
where ( x 1 , a 1 , . .
, x h − 1 , a h − 1 ) is the unique history that leads to x h .
In the full feedback setting, thelearner is further capable of observing the full loss vector ℓ t ∈ R XA ≥ 0 at the end of each round t .
Subtree and subtree policies For any g ≤ h , x g ∈ X g , x h ∈ X h , and any action a g , a h ∈ A ,we say x h or ( x h , a h ) is in the subtree rooted at x g , written as x h ⪰ x g or ( x h , a h )
⪰ x g , if x g iseither equal to x h or is a part of the unique preceding history ( x 1 , a 1 , . . .
, x h − 1 , a h − 1 ) which leadsto x h .
Similarly, we say x h or ( x h , a h ) is in the subtree of ( x g , a g ) , written as x h ≻ ( x g , a g ) or
( x h , a h ) ⪰ ( x g , a g ) , if ( x g , a g ) is either equal to ( x h , a h ) (only in the latter case), or is a part of theunique preceding history ( x 1 , a 1 , . . .
, x h − 1 , a h − 1 ) which leads to x h .
For any g ∈ [ H ] , and any infoset x g ∈ X g , we use µ x g = { µ x g h ( ·| x h ) ∈
∆
A : x h ⪰ x g } to denotea subtree policy rooted at x g .
We use Π x g and V x g to denote the set of all subtree policies and theset of all deterministic subtree policies rooted at x g .
We denote the sequence form representation ofµ x g ∈ Π x g by:
[Equation]
Similarly, we can also identify any subtree policy µ x g ∈ Π x g as a vector in R XA ≥ 0 , whose ( x h , a h ) -thentry is equal to its sequence form µ x g g : h ( x h , a h ) (which is non-zero only on the subtree rooted at x g ).
Extensive-form trigger regret The notion of trigger regret is introduced in [27, 13, 23].
An(extensive-form) trigger modification ϕ x g a g
→ m xg is a policy modification that modifies any policyµ ∈ Π as follows:
When x g is visited and a g is about to be taken (by µ ), we say x g a g is triggered , inwhich case the subtree policy rooted at x g is then replaced by m x g ∈ Π x g .
One can verify that thetrigger modification ϕ x g a g → m xg can be written as a linear transformation that maps from Π to Π :
[Equation]
a g ∈ R XA × XA .
Here, E ⪰ x g a g is a diagonal matrix with diagonal entry 1 at all ( x h , a h ) satisfying ( x h , a h ) ⪰ ( x g , a g ) ,and zero otherwise, and e x g a g ∈ R XA is an indicator vector whose only non-zero entry is 1 at ( x g , a g ) .
We say ϕ x g a g
→ v xg is a deterministic trigger modification if v x g ∈ V x g is a deterministic subtreepolicy.
We denote the set of all deterministic trigger modifications and its convex hull as Φ Tr 0 and Φ Trrespectively, where
[Equation]
The (extensive-form) trigger regret is then defined as the difference in the total loss when comparingagainst the best extensive-form trigger modification in hindsight.
We note that the trigger regret is aspecial case of Φ -regret (1) with Φ = Φ Tr .
Definition 1 (Extensive-Form Trigger Regret) .
For any algorithm that plays policies µ t ∈ Π at roundt ∈
[ T ] , the extensive-form trigger regret (also the EFCE-regret) is defined as
[Equation]
From trigger regret to Extensive-Form Correlated Equilibrium (EFCE)
The importance ofextensive-form trigger regret is in its connection to computing EFCE:
By standard online-to-batchconversion [13, 23], if all players have low trigger regret (with Reg Tr i ( T ) for the i th player), thenthe average joint policy π is an ε -EFCE, where ε = max i ∈ [ m ] Reg Tr i ( T ) /T
(cf.
Appendix B.2).
We remark in passing by taking Φ =
Φ ext , low Φ -regret implies learning (Normal-Form) Coarse
Correlated Equilibria in EFGs, as well as Nash Equilibria in the two-player zero-sum setting [6].
Efficient Φ -Hedge for Extensive-Form Trigger Regret Minimization
In this section, we study the Φ -Hedge algorithm (Algorithm 1) for minimizing the trigger regret.
Naively, Algorithm 1 requires maintaining and updating p t ∈ ∆ Φ 0 (cf.
Line 6), whose computationalcost is linear in | Φ Tr 0 | which can be exponential in X in the worst case 1 .
We begin by deriving anefficient implementation of the iterate
ϕ t ∈ Φ (of Line 3) directly by exploiting the structure of Φ Tr 0 .
3.1 Efficient implementation of Φ Tr -Hedge algorithm
We first use a standard trick to convert the computation of ϕ t (Line 3 & 6, Algorithm 1) in Φ -Hedgeto evaluating the gradient of a suitable log-partition function.
This is stated in the lemma below (forany generic Φ 0 ), whose proof can be found in Appendix C.2.
Lemma 2 (Conversion to log-partition function) .
Define the log-partition function F Φ 0 : R d × d → R
[Equation]
Then Line 3 of Φ -Hedge
(Algorithm 1) has a closed-form update for all t ≥ 1 :
[Equation]
Eq.
(8) suggests a strategy for evaluating ϕ t = −∇ F Φ 0 ( η (cid:80) t − 1 s =1 M s ) —So long as the vertex set Φ 0has some structure that allows efficient evaluation of the sum of exponentials on the numerators anddenominators (i.e. faster than naive sum), ϕ t may be computed directly in sublinear in | Φ 0 | time, andthere is no need to maintain the underlying distribution p t ∈ ∆ Φ 0 .
The following lemma enables such an efficient computation for the log-partition function F Tr := F Φ Tr(and its gradient) associated with the trigger modification set Φ = Φ Tr .
This lemma (proof deferred to
Appendix C.3) is a consequence of the specific structure of Φ 0 (cf.
(5)), whose elements are indexedby a sequence x g a g and a deterministic subtree policy v x g ∈ V x g .
Lemma 3 (Recursive expression of F Tr and ∇ F Tr ) .
For any loss matrix M ∈ R XA × XA , the EFCElog-partition function can be written as
[Equation]
where for any x h ⪰ x g ,
[Equation]
Furthermore, define λ = ( λ x g a g ) x g a g ∈X×A ∈ ∆ XA and m = ( m x g a g ) x g a g ∈X×A with m x g a g ∈Π x g (and also identified as a vector in R XA ) as
[Equation]
then we have
[Equation]
Above, λ = ( λ x g a g ) x g a g ∈X×A ∈ ∆ XA is a probability distribution over X × A , and m =( m x g a g ) x g a g ∈X×A ∈ M ≡ (cid:81) g,x g a g Π x g a g is a collection of subtree policies m x g a g , where eachm x g a g ∈ Π x g is a subtree policy that specifies an action distribution
m x g a g ,h ( a h | x h ) for everyx
h ⪰ x g , and can be identified with a vector in R XA (c.f. Section 2.2).
The recursive structure in Lemma 3 offers a roadmap for evaluating ( λ, m ) and thus ∇ F Tr ( M ) in
O ( X 2 A 2 ) time (formal statement in Appendix C.4).
Applying Lemma 3 with M = η (cid:80) t − 1 s =1 M sgives an efficient implementation of (8), i.e. the Φ -Hedge algorithm with Φ = Φ Tr .
For clarity, wesummarize this in Algorithm 2.
We remark that the parameters ( λ t , m t ) therein can also be expressedin terms of ( λ t − 1 , m t − 1 ) and M t − 1 , which we present in Algorithm 3 (the equivalent “OMD” form)in Appendix C.1.
We also note that the fixed point equation ϕ t µ = µ in Line 5 can be solved in
O ( X 2 A 2 ) time [23, Corollary 4.15].
3.2 Equivalence to FTRL and OMD
We now show that Algorithm 2 is equivalent to FTRL and OMD with suitable dilated entropies anddivergences (hence the name EFCE-OMD).
We define the trigger dilated entropy function and triggerdilated KL divergence function over ( λ, m ) ∈ ∆ XA × M as
[Equation]
where H ( · ) and D KL ( ·∥· ) are the (negative) Shannon entropy and KL divergence; and for any x g ,
H x g ( · ) is the dilated entropy
D x g ( ·∥· ) is the dilated KL divergence [30] both for the subtreerooted at x g (detailed definitions
Appendix C
Require:
Learning rate η > 0 .
1: for t = 1 ,
[Equation]
T do 2:
For each x g a g ∈ X ×A , from the reverse order of x h , compute m tx g a g ,h ( a h | x h ) and F tx g a g ,x h
[Equation]
Compute λ tx g
[Equation]
g as
[Equation]
Compute ϕ t = ϕ λ
, m
) where is
Compute the policy µ t , which is a solution of the fixed point equation ϕ t µ t = µ t .
Receive loss ℓ t = { ℓ th ( x h , a h ) } ( x h
[Equation]
a h ) ∈X×A ∈ R XA ≥ 0 .
Compute matrix loss M t = ℓ t ( µ t ) ⊤ ∈ R XA × XA ≥.
Lemma 4 (Equivalent formulations of Φ Tr -hedge) .
For any sequence of loss functions { M t } t ≥ 1 , theiterates ( λ t , m t ) in Algorithm 2 (i.e. (14) - (16) ) are equivalent to (i.e. satisfy) the following FTRLupdate on H Tr and OMD update on D Tr :
[Equation]
The proof of Lemma 4 follows directly by the concrete forms of ( λ t , m t ) in (14)-(16), and can befound in Appendix C.6.
3.3 Regret bound under full feedback and bandit feedback
We now present the regret bounds of Algorithm 2.
We emphasize that these regret bounds are simpleconsequence of the generic bound for Φ -Hedge in (2), and their proofs do not depend on the actualimplementation of Algorithm 2 developed in the preceding two subsections.
We first consider the fullfeedback setting, where the full expected loss vector ℓ t ∈ R XA ≥ 0 is received after each episode.
Theorem 5 (Regret bound of EFCE-OMD under full feedback) .
Running Algorithm 2 with η =
O ( (cid:112) ∥
Π ∥ 1 ι/ ( H 2 T )) achieves the following trigger regret bound
[Equation]
where ι := log( XA ) is a log factor.
The proof of Theorem 5 is simply by applying (2) and observing that log(Φ
Tr 0 ) ≤ ∥ Π ∥ 1 log A +log( XA ) (see Appendix D.1).
This theorem shows that the Φ Tr -Hedge algorithm gives (cid:101) O ( √ XT )trigger regret bound, which matches the information-theoretic lower bound Ω( √ XT ) [48, Theorem 2]up to a (cid:101)
O
(poly( H )) factor, and is slightly better than the (cid:101) O ( √ XAT ) upper bound of [42,
Corollary
F.3] though their definition of EFCE-regret is slightly stricter (thus higher) than ours.
In the bandit feedback setting, the learner only observes her own rewards and infosets.
In this casewe replace ℓ t in Algorithm 2 with the following loss estimator (with IX bonus γ ) proposed in [34]:
[Equation]
We show that EFCE-OMD achieves the following guarantee in the bandit feedback setting (proofin Appendix D.2).
The proof follows by plugging the loss estimator (cid:101) ℓ t into (2) and additionallybounding concentrations (which we remark is a better strategy than using a naive bandit-based lossestimator in the corresponding NFG space).
Theorem 6 (Regret bound of EFCE-OMD under bandit feedback) .
Run Algorithm 2 with lossestimator { (cid:101) ℓ t } Tt =1 (19) , η = (cid:112)
∥
Π ∥ 1 log A/ ( HXAT ) , and γ = (cid:112) ∥
Π ∥ 1 ι/ ( XAT ) .
Then we havethe following trigger regret bound with probability at least 1 − δ :
[Equation]
where ι = log(3 XA/δ ) is a log term.
To our best knowledge, Theorem 6 gives the first trigger regret bound against adversarial opponentsand bandit feedback.
This (cid:101) O
( (cid:112) XA ∥
Π ∥ 1 T ) rate is √ XA worse than Theorem 5 (ignoring H andlog factors), and is at most (cid:101) O ( √ X 2 AT ) using ∥ Π ∥ 1 ≤ X .
Balanced EFCE-OMD for bandit feedback
We now build upon the EFCE-OMD algorithm (Algorithm 2) to develop a new algorithm, Balanced
EFCE-OMD , and show that it achieves near-optimal extensive-form trigger regret guarantee underbandit feedback.
Due to space limit, here we discuss the two key modifications in the algorithmdesign, and defer the full algorithm description to Algorithm 4 (Appendix E).
Key modification I: “Rebalancing” the log-partition function Building on the balancing technique of [6], we start from Eq.
(9) and (10) of the log partition function, and rescale the innerfunctions F x g a g ,x h using balanced exploration policies { µ ⋆,hg : h ( x h , a h ) } g,x h ,a h (see Definition E.1for the formal definition), and rescale the outer function F Tr by XA .
Concretely, for any matrix
M ∈ R XA × XA , we define the balanced EFCE log-partition function as
[Equation] [Equation]
where for any x h ⪰ x g (using µ ⋆,hg : h := µ ⋆,hg : h ( x h , a h ) as shorthand, which depends on x h but not a h ),
F ⋆x g a g ,x h
[Equation]
⋆
[Equation]
g
[Equation]
Key modification II: New loss estimator under bandit feedback We use an adaptive family ofbandit-based loss estimators { (cid:101) ℓ t,x g a g } x g a g ⊂ R XA ≥ 0 , one for each ( x g ,
a g ) ∈
X × A , defined as(cid:101) ℓ t,x g a g h ( x h , a h ) :
= 1 { ( x th ,a th )=( x h
[Equation]
a h ) } (
[Equation] [Equation]
where µ tx g a g := µ t 1: g ( x g , a g ) for shorthand.
The main difference of (22) over (19) is in the adaptive
IX bonus term on the denominator that scales with γ but is different for each x g a g .
We then placeeach µ tx g a g (cid:101) ℓ t,x g a g into the x g a g -th column of a matrix loss estimator (cid:102) M t , or in matrix form,
[Equation]
With (20)-(22) at hand, our algorithm Balanced EFCE-OMD is defined as the negative gradient of
F Trbal evaluated at the cumulative loss estimators:
[Equation]
[Equation]
and µ t ∈ Π solves the fixed point equation ϕ t µ t = µ t .
Similar as EFCE-OMD, (23) also admitsefficient implementations in both FTRL and OMD form (cf.
Algorithm 4 & 5).
The corresponding( λ t , m t ) is also equivalent to running a FTRL/OMD algorithm with respect to a balanced dilatedentropy/KL-divergence over ϕ ∈
Φ Tr (cf.
Lemma E.4 and Appendix E.3 for details).
Main result We now present the theoretical guarantee of Algorithm 4 (proof in Appendix F).
Theorem 7.
Balanced EFCE-OMD (Algorithm 4) with η = (cid:112) XAι/
H 4 T and γ = 2 (cid:112) XAι/
H
Tachieves the following extensive-form trigger regret bound with probability at least 1 − δ :
[Equation]
where ι = log(10 XA/δ ) is a log term.
The (cid:101) O ( √ XAT ) trigger regret asserted in Theorem 7 improves over Theorem 6 by a factor of (cid:112)
∥ Π ∥ 1 ,and matches the information-theoretic lower bound up to poly ( H ) and log factors 2 .
By the online-tobatch conversion (Appendix B.2), Theorem 7 also implies an (cid:101) O
( H 4 XA/ε 2 ) sample complexity forlearning EFCE under bandit feedback (assuming same game sizes for all m players).
This improvesover the best known (cid:101) O
( mH 6 XA 2 /ε 2 ) sample complexity in the recent work of Song et al.
Overview of techniques The proof of Theorem 7 is significantly more challenging than that of
Theorem 6, even though the algorithm itself is designed by appearingly simple modifications.
Thehappens since Algorithm 4, unlike Algorithm 2, no longer necessarily corresponds to any normal-formalgorithm.
The technical crux of the proof is to bound the nonlinear part of F Trbal (with respect to thelosses), which we do by carefully controlling a series of second-order terms utilizing the balancedpolicies within F Trbal and the new adaptive IX bonus within { ℓ t,x g a g } x g a g (Lemma F.5-F.8).
Equivalence of OMD and Vertex MWU for external regret minimization
As another illustration of our framework, we now choose Φ =
Φ ext = conv { Φ ext 0 } to be the setof external policy modifications, which modify any policy to some deterministic policy.
In thiscase, the Φ ext -Hedge algorithm minimizes the external regret in EFGs.
In this section, we show thatΦ ext -Hedge, same as the vertex MWU algorithm considered in Farina et al.
[25], is actually equivalentto the OMD with dilated entropy [30].
Let { ℓ t } t ≥ 1 ⊂ R XA ≥ 0 be an arbitrary sequence of loss vectors.
Vertex MWU We use V to denote all the deterministic sequence-form policies, which can also beviewed as the vertex set of the policy set Π .
A simple reformulation (cf.
Appendix G) shows thatΦ ext -Hedge
(Algorithm 1) gives the vertex MWU algorithm considered by Farina et al.
[Equation]
[Equation]
OMD with dilated entropy Another popular algorithm for external regret minimization is the
OMD algorithm on the sequence-form policy space with the dilated entropy [30, 35]:
[Equation]
Theorem 8 (Equivalence of OMD and Vertex MWU) .
For any sequence of loss vectors { ℓ t } t ≥ 1 ,
OMD with dilated entropy is equivalent to Vertex MWU, that is, (25) and (24) give the same { µ t } t ≥ 1 .
The proof of Theorem 8 can be found in Appendix G.1.
Our proof also reveals that the efficientimplementation of Vertex MWU developed by Farina et al.
[25] using the “kernel trick” is actuallyequivalent to the standard linear-time efficient implementation of OMD with dilated entropy.
Conclusion
In this paper, we present an efficient implementation of the Φ -Hedge algorithm for minimizing theextensive form trigger regret.
The algorithm is equivalent to OMD with dilated regularizers, andachieves efficient regret bounds under both full feedback and bandit feedback.
We also design animproved algorithm Balanced EFCE-OMD, which achieves a sharp trigger regret bound under banditfeedback.
We believe our work leads to many open questions, such as efficient implementationsof Φ -Hedge with more general Φ sets (e.g. the behavioral modifications considered in [38, 42]), oraccelerated polylog( T ) Φ -regret bounds under full feedback by optimistic algorithms
Checklist
For all authors...
(a) Do
the
main
claims made in the abstract and introduction accurately
reflect
the paper’scontributions
and scope?
[Yes
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] As this
work is theoretical, we do not foresee any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the
type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]