Mutation-Driven Follow the Regularized Leader for Last-Iterate Convergence in Zero-Sum Games
Abstract
In this study, we consider a variant of the Follow the Regularized Leader (FTRL) dynamics in twoplayer zero-sum games.
FTRL is guaranteed to converge to a Nash equilibrium when time-averaging the strategies, while a lot of variants suffer from the issue of limit cycling behavior, i.e., lack the last-iterate convergence guarantee.
To this end, we propose mutant FTRL (M-FTRL), an algorithm that introduces mutation for the perturbation of action probabilities.
We then investigate the continuous-time dynamics of M-FTRL and provide the strong convergence guarantees toward stationary points that approximate Nash equilibria under full-information feedback.
Furthermore, our simulation demonstrates that M-FTRL can enjoy faster convergence rates than FTRL and optimistic
FTRL under full-information feedback and surprisingly exhibits clear convergence under bandit feedback.
INTRODUCTION
Our study focuses on the problem of learning an equilibrium in two-player zero-sum games.
In order to find an equilibrium in two-player zero-sum games, we need to solve a minimax optimization (or saddle-point optimization) in the form of min x max y f
( x, y ) .
Motivated by advances of multi-agent reinforcement learning [Busoniu et al., 2008] and Generative Adversarial Networks (GANs) [Goodfellow et al., 2014], the development of algorithms that efficiently approximate the solution of the minimax optimization is attracting considerable interest [Blum and Monsour, 2007,
Daskalakis et al., 2018].
There are a lot of studies focusing on developing no-regret learning algorithms where the iterate-average strategy profile converges to a Nash equilibrium of two-player zero-sum games [Banerjee and Peng, 2005, Zinkevich et al., 2007,
Daskalakis et al., 2011].
However, well-known no-regret learning algorithms such as Follow the Regularized Leader (FTRL) are shown to cycle and fail to converge without timeaveraging [Mertikopoulos et al., 2018, Bailey and Piliouras, 2018].
In recent years, several studies have developed and analyzed algorithms whose trajectory of updated strategies directly converges to an equilibrium without forming a cycle, such as optimistic FTRL (O-FTRL)
[Daskalakis et al., 2018, Daskalakis and Panageas, 2019, Mertikopoulos et al., 2019, Wei et al., 2021, Lei et al., 2021].
This convergence property is known as last-iterate convergence .
However, establishing the explicit convergence rates of optimistic multiplicative weights update, which is tantamount to O-FTRL with entropy regularization, requires that the equilibrium in underlying games must be unique [Daskalakis and Panageas, 2019, Wei et al., 2021].
In this study, as an alternative, we propose mutant FTRL(M-FTRL), an algorithm that introduces mutation for the perturbation of action probabilities.
We first identify the discrete-time version of the M-FTRL dynamics and then modify it to the continuous-time version to provide the theoretical analysis.
We prove the followings: 1) M-FTRL dynamics induced by the entropy regularizer is equivalent to replicator-mutator dynamics (RMD)
[Hofbauer et al., 2009,
Zagorsky et al., 2013, Bauer et al., 2019]; 2) for general regularization functions, the strategy trajectory of M-FTRL converges to a stationary point of the RMD; 3) the trajectory of M-FTRL with the entropy regularizer converges to an approximate Nash equilibrium at an exponentially fast rate.
To the best of our knowledge, we are the first to provide the convergence result for RMD in two-player zero-sum games.
Furthermore, our simulation demonstrates that M-FTRL can enjoy faster convergence rates than FTRL and optimistic
FTRL under full-information feedback, i.e., M-FTRL converges to a stationary point, which approximates a Nash [Figure] equilibrium, faster.
It also exhibits clear convergence under partial-information or bandit feedback, where each player takes the feedback about the payoffs from his or her chosen actions.
We empirically observe the last-iterate convergence behavior in the M-FTRL dynamics, as well as under fullinformation feedback, while neither FTRL nor O-FTRL reveals such behavior.
This is surprising because it is an open question if a last-iterate convergence guarantee is provided under bandit feedback.
RELATED LITERATURE
Average-iterate convergence There are a lot of previous studies focusing on developing no-regret learning algorithms that enjoy average-iterate convergence in two-player zero-sum games [Cesa-Bianchi and Lugosi, 2006, Zinkevich et al., 2007, Hofbauer et al., 2009, Syrgkanis et al., 2015].
FTRL is one of the most widely studied no-regret learning algorithm and has been shown to be convergent if the equilibrium is deterministic or strict [Mertikopoulos et al., 2018, Giannou et al., 2021].
If the equilibrium strategy is a mixed strategy with full support, FTRL’s trajectory can be recurrent [Mertikopoulos et al., 2018].
For extensive-form games, counterfactual regret minimization [Zinkevich et al., 2007] and its variants have been developed as a no-regret learning algorithm
[Gibson et al., 2012, Tammelin, 2014, Lanctot et al., 2017, Schmid et al., 2019, Brown and Sandholm, 2019, Davis et al., 2020].
However, most of these algorithms have not been proven that the last-iterate strategy converges.
Last-iterate convergence In recent years, various algorithms using an optimistic online learning framework [Rakhlin and Sridharan, 2013a,b] have been proposed for last-iterate convergence in minimax optimization.
Optimistic gradient descent ascent [Daskalakis et al., 2018, Mertikopoulos et al., 2019, Wei et al., 2021] and optimistic multiplicative weights update [Daskalakis and Panageas, 2019,
Wei et al., 2021, Lei et al., 2021] are the variants of O-FTRL, and they have been shown to enjoy the last-iterate convergence guarantee in constrained and unconstrained saddle optimization problems.
Furthermore, Nguyen et al.
[2021] have proposed the no-regret learning algorithm, which exhibits the last-iterate convergence in asymmetric repeated games.
In contrast to their optimistic modification of FTRL, which boosts updates for expected utitilities, our method is motivated by replicator-mutator dynamics and provides an alternative way to enjoy the last-iterate convergence guarantee.
Replicator-mutator dynamics Evolutionary game theory has been strongly related to learning dynamics.
In fact, it is well-known that cross learning converges to the replicator dynamics (RD) in the continuous-time limit [Börgers and
Sarin, 1997, Bloembergen et al., 2015], similarly to FTRL.
On the other hand, RMD [Hofbauer and Sigmund, 1998] has been overlooked in the context of learning.
Introducing mutation empirically makes numerical errors in computation small
[Zagorsky et al., 2013].
However, it makes difficult to analyze the properties.
Some notable exceptions report that mutation stabilizes the dynamics [Bomze and Burger, 1995,
Bauer et al., 2019].
Let π µ be an interior stationary point of
RMD with mutation rate µ , then π µ is ε -Nash equilibrium of the underlying game for ε = µ [Bauer et al., 2019].
Also, evolutionary game dynamics such as RD typically exhibits continua of stationary points and is unlikely to converge to a unique, stable stationary point.
Mutation dissolves continua of neutrally stable equilibria into isolated, asymptotically stable ones [Bomze and Burger, 1995].
PRELIMINARIES 3.1 TWO-PLAYER ZERO-SUM NORMAL-FORM GAME
A two-player normal-form game is defined by utility functions u i ∈ [ − u max , u max ] A 1 × A 2 , where A i is the finite action space for player i ∈ { 1 , 2 } .
In a two-player zero-sum normal-form game, u i satisfies u 1 ( a 1 , a 2 )+ u 2 ( a 1 , a 2 ) =
for all a 1 ∈ A 1 and a 2 ∈
A 2 .
In this game, each player i selects action a i ∈
A i simultaneously.
Then, player i receives utility u
i ( a 1 , a 2 ) .
Let us denote π
i ∈ ∆( A i ) as a mixed strategy for player i , where ∆( A i ) := { p ∈ [0 , 1] |
A i | | (cid:80) a i ∈ A i p ( a i ) = 1 } represents the probability simplex on A i . We define a strategy profile as π = ( π 1 , π 2 ) .
For a given strategy profile π , the expected utility for player i is given by v πi =
E a ∼ π [ u i ( a 1 , a 2 )] .
We further define the conditional expected utility of taking action a i ∈
A i as q πi ( a i ) =
E a − i ∼ π − i [ u i ( a i , a − i ) | a i ] , where − i represents the opponent of player i .
Finally, we denote the conditional expected utility vector as q πi =
( q πi ( a i ))
a i ∈
A i .
3.2 NASH EQUILIBRIUM AND EXPLOITABILITY
A common solution concept for two-player games is a Nash equilibrium [Nash, 1951], where no player cannot improve his/her expected utility by deviating from his/her specified strategy.
In two-player zero-sum normal-form games, a Nash equilibrium π ∗ = ( π ∗ 1 , π ∗ 2 ) ensures the following condition: ∀ π 1 ∈ ∆( A
[Equation]
) , ∀ [Equation] [Equation] An ϵ -Nash equilibrium ( π 1 , π 2 ) is an approximation of a
Nash equilibrium, which satisfies the following inequality:
[Equation]
≤ ϵ. Furthermore, we call exploit( π ) :
= max ˜ π 1 ∈ ∆( A 1 ) v ˜ π 1 ,π 2+ max ˜ π 2 ∈ ∆( A 2 ) v π 1 ,
˜ π 2as exploitability of a given strategy profile π .
Exploitability is a metric for measuring how close π is to a Nash equilibrium π ∗ in two-player zero-sum games [Johanson et al., 2011, 2012, Lockhart et al., 2019, Timbers et al., 2020, Abe and Kaneko, 2021].
From the definition, a
Nash equilibrium π ∗ has the lowest exploitability of 0 .
3.3 PROBLEM SETTING
In this study, we consider the setting where the game is played repeatedly for T iterations.
At each iteration t ∈ [ T ] , each player i determines the (mixed) strategy π ti ∈ ∆( A i ) based on the past-observed feedback.
Then, each player i observes the new feedback.
In this study, we focus on two feedback cases: full-information feedback and bandit feedback .
At the end of the iteration t under full-information feedback, player i observes the conditional expected utility vector ( q π t i ( a i ))
a i ∈
A i as feedback.
Under bandit feedback, each player i chooses an action a ti according to π ti .
Then, each player observes the realized utility u i ( a t 1 , a t 2 ) .
FTRL is a widely used learning algorithm in the repeated game setting.
For player i , FTRL methods are defined with regularization function ψ i : ∆( A i )
→ R , which is strictly convex and continuously differentiable on ∆( A i ) .
In FTRL, each player i determines her strategy π ti at iteration t as follows:
[Equation]
where η > 0 is the learning rate.
3.4 OTHER NOTATIONS
We denote the interior of the probability simplex ∆( A i ) by ∆ ◦
( A i ) := { p ∈ ∆( A i )
| ∀ a i ∈
A i , p ( a i ) > 0 } .
For a strictly convex and continuously differentiable function ψ , the associated Bregman divergence is defined as
D ψ ( x, x ′ )
=
ψ ( x ) − ψ ( x ′ ) − ⟨∇ ψ ( x ′ ) , x − x ′ ⟩ .
The
Kullback-Leibler divergence , which is the Bregman divergence with the entropy regularizer ψ ( x ) =
(cid:80) i x i
ln x i , is denoted by KL( x, x ′ )
= (cid:80)
i x i
ln x i x ′ i .
Besides, we define the sum of Bregman divergences and sum of KullbackLeibler divergences as D ψ ( π, π ′ ) =
(cid:80) 2 i =
1 D ψ i ( π i , π ′ i ) and
KL( π, π ′ )
= (cid:80)
i =1 KL( π i , π ′ i ) , respectively.
MUTANT FOLLOW THE REGULARIZED LEADER
In this section, we introduce Mutant Follow the Regularized Leader (M-FTRL), which is inspired by the RMD [Hofbauer and Sigmund, 1998, Zagorsky et al., 2013].
Let us see what happens in a biased version of the Rock-PaperScissors game, see Table 1.
Figure 1 compares trajectories of RD and RMD with varying mutation parameters µ (see (RMD for the differential equation of RMD). Note that µ represents the parameter that controls the strength of mutation.
Figure 1a shows that the trajectories form a cycle and never converge to the Nash equilibrium because the game is intransitive.
Note, however, that the time-averaged trajectory of FTRL converges to interior Nash equilibria in two-player zero-sum games [Hofbauer et al., 2009].
In contrast, Figures 1b and 1c exhibit a clear convergence to the unique stationary point, which is almost equivalent to the interior Nash equilibrium (the red dot) without taking the time average.
As the mutation parameter increases to 1 . 0 , although the stationary point becomes far from the Nash equilibrium, it is still asymptotically stable in Figure 1d.
Thus, mutation is expected to ensure that the trajectory of a learning dynamics reaches an approximated equilibrium.
4.1 ALGORITHM
We propose a discrete-time version of the M-FTRL algorithm
under two feedback cases: full-information feedback
Require:
Time horizon T , learning rate η , regularization function ψ i , mutation parameter µ , update frequency
N , initial strategy π 0 i 1: c i ← (cid:16) 1 | A i | (cid:17) a i ∈ A
i 2: τ ←3: Initialize z 0 i so that π 0 i = arg max p ∈ ∆( A i ) (cid:8)(cid:10)
z
i , p (cid:11) − ψ i ( p ) (cid:9) 4: for t
= 1 , 2 , · · · , T do 5:
Compute strategy π ti by π ti = arg max p ∈ ∆( A i ) (cid:8)(cid:10)
z ti , p (cid:11) − ψ i ( p ) (cid:9) 6: for a ∈ A i do 7: z t +1 i ( a ) ← z ti ( a )+ η (cid:16) q π t
i ( a )+ µ π t i ( a ) ( c i ( a ) − π ti ( a )) (cid:17) 8: end for 9: τ ← τ +10: if τ = N then 11: c i ← π ti 12: τ ←13: end if 14: end for and bandit feedback. First, we provide the strategy update rule under full-information feedback: [Equation]
where η > 0 is the learning rate, µ > 0 is the mutation parameter , and c i ∈ ∆ ◦ ( A i ) is the reference strategy . As shown in Figure 1b-1d, strategies π t i updated by (1) would converge to the stationary point, which is different from the Nash equilibrium of the original game. The stationary point is a 2 µ -Nash equilibrium of the original game, and the stationary point is not Nash equilibrium unless ( c 1 , c 2 ) is a Nash equilibrium (see Theorem 5.4).
Therefore, for convergence to a Nash equilibrium of the original game, we introduce a technique to adapt the reference strategy.
That is, we copy probabilities from π ti into c i every N ( ≤ T ) iterations.
This technique is similar to the direct convergence method by [Perolat et al., 2021].
The pseudo-code of our algorithm with adaptive reference strategies is presented in
Algorithm 1.
Under bandit feedback, each player i needs to estimate
[Equation]
µ,ti
[Equation]
from the realized utility u i ( a t 1 , a t 2 ) .
Similarly to [Wei and Luo, 2018, Ito, 2021], we construct the following estimator ˆ q [Equation]
i : ˆ q
[Equation]
It is easy to confirm that ˆ q µ,ti is an unbiased estimator of q µ,ti .
Under bandit feedback, M-FTRL updates the strategy π ti by the following update rule, which uses ˆ q µ,ti instead of q µ,ti in (1): [Equation]
Note that M-FTRL does not require any information about the opponent’s strategy
π t − i under bandit feedback.
THEORETICAL ANALYSIS
In this section, we provide the theoretical relationship between RMD and M-FTRL and the last-iterate convergence guarantee of M-FTRL.
Instead of the discrete-time version of M-FTRL algorithm, we analyze the theoretical properties of the following continuous-time version of M-FTRL dynamics:
[Equation]
First, we show that this dynamics is a generalization of
RMD
[Bauer et al., 2019].
That is, the dynamics of M-FTRL with the entropy regularizer ψ i ( p ) =
(cid:80) a i ∈
A i p ( a i ) ln p ( a i ) induces RMD:
Theorem 5.1.
The dynamics defined by (3) with the entropy regularizer ψ i ( p ) =
(cid:80) a i ∈
A i p ( a i ) ln p ( a i ) is equivalent to replicator-mutator dynamics:
[Equation]
The proof of this theorem is shown in Appendix D.1.
From here, we derive the relationship between the stationary point π µ of (RMD) (i.e., the strategy profile that satisfies d dt π µi ( a i ) = 0 for all i ∈ { 1 , 2 } and a i ∈ A i ) and the updated strategy profile π t .
Note that, from Lemma in [Bauer et al., 2019], for any µ > 0 there exists π µ ∈ [-(cid:81) 
-](cid:81)i =1 ∆ ◦
( A i ) such that π µ is a stationary point of (RMD).
Thus, π µ is well-defined.
We first derive the time derivative of the (sum of) Bregman divergence between π µ and π t :
Theorem 5.2.
Let π µ ∈ (cid:81) 2 i =1 ∆( A i ) be a stationary point of (RMD).
Then, π t updated by M-FTRL satisfies that:
[Equation]
Furthermore, if the regularizer is entropy ψ i ( p ) =
[Equation]
a i ∈ A i p ( a i ) ln p ( a i ) , then π t satisfies that:
[Equation]
where ξ = min i ∈{ 1 , 2 } ,a
i ∈ A i c i ( a i ) π µi ( a i ) .
The first statement implies that ddt D ψ ( π µ , π t ) = 0 holds if and only if π t = π µ , and
∀ π t ̸ = π µ , ddt D ψ ( π µ , π t ) < 0 .
Thus, by Lyapunov arguments [Khalil, 2015], the Bregman divergence between π µ and π t converges to 0 , and then π t converges to π µ .
Note that Theorem 5.2 holds for all stationary points of (RMD).
This means that for a fixed µ and ( c i ) 2 i =1 , the stationary point is unique.
From the second statement, we can show that exponential convergence rates can be achieved when using the entropy regularizer:
Corollary 5.3.
Assume that the regularizer is entropy ψ i ( p ) =
(cid:80) a i ∈
A i p ( a i ) ln p ( a i ) .
Then, M-FTRL’s trajectory converges to a stationary point of (RMD) exponentially fast, i.e., KL [Equation] , π t ) ≤ KL
[Equation]
µ ,
[Equation] 0 ) exp ( [Equation] Finally, combining this corollary and Lemma 3.5 in [Bauer et al., 2019], we can derive the exploitability bound of π t .
Theorem 5.4.
Assume that the regularizer is entropy ψ i ( p ) =
(cid:80) a i ∈
A i p ( a i ) ln p ( a i ) .
Then, the exploitability for
M-FTRL is bounded as:
[Equation]
Theorem 5.4 means that π t converges to a 2 µ -Nash equilibrium exponentially fast.
The proof of the theorem is shown in Section 5.2.
5.1 PROOF SKETCH OF THEOREM 5.
We sketch below the proof of Theorem 5.2.
The complete proof and the associated lemmas are presented in Appendix D.2-D.4.
Proof of the first part of Theorem 5.2.
First, we derive the time derivative of the Bregman divergence between π ∈ (cid:81)i =1 ∆( A i ) and π t :
Lemma 5.5.
For any π ∈ (cid:81) 2 i =1 ∆( A i ) , π t updated by
M-FTRL satisfies that:
[Equation]
The proof of
Lemma 5. stems from the fact that
[Equation]
( π i ) (cid:1) .
Next, we derive the relationship between the expected utilities v π µ and v π ′ i ,π µ − i for any π ′
i ∈ ∆( A i ) :
Lemma 5.6.
Let π µ ∈ {+(cid:81) 
 i =1+} ∆( A i ) be a stationary point of (RMD).
Then, for any i ∈ { 1 , 2 } and π ′ i ∈ ∆( A i ) :
[Equation]
i
[Equation]
− i
[Equation]
This result can be shown by the fact that π µ is the stationary point of (RMD), i.e., π µi ( a i ) (cid:0) q π µ i ( a i ) − v π µ i (cid:1) + µ ( c i ( a i ) − π µi ( a i )) = {+
 for all i ∈ { 1 , 2 }+} .
By combining Lemmas 5.5 and 5.6, we can obtain:
[Equation]
a
[Equation]
A i
[Equation] a
[Equation]
a
[Equation]
[Equation]
a
[Equation]
where the third equality follows from (cid:80) 2 i =1 v π µ i = 0 by the definition of zero-sum games.
This concludes the first statement of the theorem.
Proof of the second part of Theorem 5.2.
Let us define
[Equation]
From the first part of the theorem, we have:
[Equation]
[-ξ i ln (cid:32) 
-]
where the second inequality follows from x ≥ ln(1 + x ) for all x >
[Equation]
and the third inequality follows from the concavity of the ln( · ) function and Jensen’s inequality for concave functions.
On the other hand, if ψ i ( p ) = (cid:80) a i ∈
A i p ( a i ) ln p ( a i ) , then D ψ i ( π µ , π t ) =
KL( π µ , π t ) .
[-(cid:80) 
-]
From this fact and (4), we have:
[Equation]
This concludes the second statement of the theorem.
5.2 PROOF OF THEOREM 5.
From the definition of exploitability, we have: exploit
[Equation]
[Table]
[Equation]
where the second inequality follows from Hölder’s inequality, and the last inequality follows from Lemma 11.6.1 in [Cover and Thomas, 2006].
From Lemma 3.5 of [Bauer et al., 2019], a stationary point π µ of (RMD) satisfies that for all i ∈ { 1 , 2 } and a i ∈
A i , q π µ i
[Equation]
≤ µ .
Therefore, the term of
[Equation]
can be bounded as:
[Equation]
where the second equality follows from (cid:80) 2 i =1 v π µ i = 0 by the definition of zero-sum games.
By combining (5), (6), and Corollary 5.3, we have:
[Equation]
where the second inequality follows from √ a + √ b ≤ (cid:112) 2( a + b ) for a, b > 0 .
This concludes the statement
[Table]
EXPERIMENTS
In this section, we empirically evaluate M-FTRL.
We compare its performance to those of FTRL and O-FTRL
[Table] We conduct experiments on the following games: biased rock-paper-scissors (BRPS), a normal-form game with multiple Nash equilibria (M-Eq) [Table] and random utility games.
BRPS and M-Eq have the following utility matrix, respectively:
[Table] [Table]
[Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
P
[Figure]
[Figure] The set of Nash equilibria in M-Eq is given by:
[Equation]
For random utility games, we generate each component in a utility matrix uniformly at random in [0 , 1] .
We consider random utility games with action sizes | A 1 | = | A 2 | = 10 and | A 1 | = | A 2 | = 50 .
For each game, we average the results for 100 instances.
We generate the initial strategy profile πuniformly at random in (cid:81) 2 i =1 ∆ ◦ ( A i ) for each instance.
We use the entropy regularizer ψ i ( p ) =
(cid:80) a i ∈
A i p ( a i ) ln p ( a i ) in all experiments.
6.1 FULL-INFORMATION FEEDBACK
First, we provide the results under full-information feedback.
In these experiments, we analyze the performance of MFTRL with a fixed reference strategy c[-i = 
 | A-]
i = (cid:16) 1 |
A i | (cid:17)
a i ∈
A i and one with adaptive reference strategies (Algorithm 1).
We set the learning rate to η = 10 − 1 for all algorithms, and set the mutation parameter to µ = 10 − 2 for M-FTRL.
For MFTRL with adaptive reference strategies, we set N = 4 ,in BRPS and M-Eq, and N = 20 , 000 in the random utility games.
Figure 2 shows the average exploitability of π t updated by each algorithm.
We find that the exploitability of MFTRL converges to a constant value faster than FTRL and
O-FTRL.
Furthermore, by adapting the reference strategy, the exploitability of M-FTRL’s strategy profile quickly converges to 0 .
We provide additional experimental results with varying mutation parameters µ ∈ { 10 − 3 , 5 × 10 − 3 , 10 − 2 , 10 − 1 , 1 } in Appendix B.
Next, we compare the trajectories of strategies updated by each algorithm.
Figure 3 shows the trajectories of π t updated by each algorithm from an instance of RBPS. Note [Figure] [Figure]
[Figure]
1.0 R
P S Equilibrium Point Start Point End Point (c) O-FTRL that in this figure, we set the initial strategy to π 0 i =
| A i | for i ∈ { 1 , 2 } .
We can observe that FTRL’s strategies cycle around the Nash equilibrium strategy, and O-FTRL’s
strategies gradually approach the Nash equilibrium strategy.
Unlike these methods, M-FTRL’s strategies quickly approach the stationary point.
Figure 4 shows the initial strategies and final strategies for player 1 in M-Eq.
We find that M-FTRL’s strategy profile converges to a unique stationary point regardless of the setting of the initial point, while O-FTRL’s strategy profile converges to a different Nash equilibrium for each instance.
This result highlights the uniqueness property of the stationary point from Theorem 5.2.
6.2 BANDIT FEEDBACK
Next, we provide the results under bandit feedback.
We set the learning rate to η = 10 − 4 for all algorithms, and set the mutation parameter to µ = 10 − 2 for M-FTRL.
In the bandit feedback experiments, we focus on the performance of MFTRL with a fixed reference strategy c
i = 1 | A i | .
In FTRL and O-FTRL algorithms, we use the unbiased estimator by [Lattimore and Szepesvári, 2020] as the estimator of q π t i so that the estimator takes values in ( −∞ , u max ] for computational stability.
We provide further details on the estimator in Appendix A. Note that M-FTRL does not need this estimator, but it is sufficient to use the importanceweighted estimator in (2).
Figure 5 shows the average exploitability of π t updated by each algorithm, and Figure 6 shows the trajectories of π t updated by each algorithm from an instance of RBPS.
We can see that unlike the experimental results under fullinformation feedback, O-FTRL’s trajectory does not converge to a Nash equilibrium.
On the other hand, M-FTRL’s trajectory converges near a stationary point.
These results suggest that M-FTRL has the last-iterate convergence property even under bandit feedback.
CONCLUSION
In this study, we proposed M-FTRL, a simple FTRL algorithm that incorporates mutation for last-iterate convergence to a stationary point.
We proved that the M-FTRL dynamics induced by the entropy regularizer is equivalent to RMD.
Besides, we showed that the trajectory of M-FTRL with general regularization functions converges to a stationary point of the RMD.
The numerical simulation reveals that MFTRL outperforms the state-of-the-art FTRL and O-FTRL in a variety of two-player zero-sum games.
In future studies, we will extend M-FTRL algorithm and provide a theoretical analysis to more complex games, such as extensive-form games and Markov games.
Acknowledgements
Atsushi Iwasaki was supported by JSPS KAKENHI Grant
Numbers JP21H04890 and JP20K20752.
[-(cid:80) 
-]