A SYNCHRONOUS G RADIENT P LAY IN Z ERO -S UM M ULTI - AGENT G AMES
A BSTRACT
Finding equilibria via gradient play in competitive multi-agent games has been attracting a growing amount of attention in recent years, with emphasis on designing efficient strategies where the agents operate in a decentralized and symmetric manner with guaranteed convergence.
While significant efforts have been made in understanding zero-sum two-player matrix games, the performance in zerosum multi-agent games remains inadequately explored, especially in the presence of delayed feedbacks, leaving the scalability and resiliency of gradient play open to questions.
In this paper, we make progress by studying asynchronous gradient plays in zero-sum polymatrix games under delayed feedbacks.
We first establish that the last iterate of entropy-regularized optimistic multiplicative weight updates (OMWU) method converges linearly to the quantal response equilibrium (QRE), the solution concept under bounded rationality, in the absence of delays.
While the linear convergence continues to hold even when the feedbacks are randomly delayed under mild statistical assumptions, it converges at a noticeably slower rate due to a smaller tolerable range of learning rates.
Moving beyond, we demonstrate entropy-regularized OMWU—by adopting two-timescale learning rates in a delay-aware manner—enjoys faster last-iterate convergence under fixed delays, and continues to converge provably even when the delays are arbitrarily bounded in an average-iterate manner.
Our methods also lead to finite-time guarantees to approximate the Nash equilibrium (NE) by moderating the amount of regularization.
To the best of our knowledge, this work is the first that aims to understand asynchronous gradient play in zero-sum polymatrix games under a wide range of delay assumptions, highlighting the role of learning rates separation.
I NTRODUCTION
Finding equilibria of multi-player games via gradient play lies at the heart of game theory, which permeates a remarkable breadth of modern applications, including but not limited to competitive reinforcement learning (RL) (Littman, 1994), generative adversarial networks (GANs) (Goodfellow et al., 2014) and adversarial training (Mertikopoulos et al., 2018).
While conventional wisdom leans towards the paradigm of centralized learning (Bertsekas & Tsitsiklis, 1989), retrieving and sharing information across multiple agents raise questions in terms of both privacy and efficiency, leading to a significant amount of interest in designing decentralized learning algorithms that utilize only local payoff feedbacks, with the updates at different agents executed in a symmetric manner.
In reality, there is no shortage of scenarios where the feedback can be obtained only in a delayed manner (He et al., 2014), i.e., the agents only receive the payoff information sent from a previous round instead of the current round, due to communication slowdowns and congestions, for example.
Substantial progress has been made towards reliable and efficient online learning with delayed feedbacks in various settings, e.g., stochastic multi-armed bandit (Pike-Burke et al., 2018;
Vernade et al., 2017), adversarial multi-armed bandit (Cesa-Bianchi et al., 2016; Li et al., 2019), online convex optimization (Quanrud & Khashabi, 2015; McMahan & Streeter, 2014) and multi-player game (Meng et al., 2022; H´eliou et al., 2020;
Zhou et al., 2017). Typical approaches to combatting delays include subsampling the payoff history (Weinberger & Ordentlich, 2002; Joulani et al., 2013), or adopting
[Table]
adaptive learning rates suggested by delay-aware analysis (Quanrud & Khashabi, 2015; McMahan & Streeter, 2014; Hsieh et al., 2020; Flaspohler et al., 2021). Most of these efforts, however, have been limited to either the asymptotic convergence to the equilibrium (Zhou et al., 2017; H ´ eliou et al., 2020) or the study of individual regret , which characterizes the performance gap between an agent’s learning trajectory and the best policy in hindsight.
It remains highly inadequate when it comes to guaranteeing finite-time convergence to the equilibrium in a multi-player environment, especially in the presence of delayed feedbacks, thus leaving the scalability and resiliency of gradient play open to questions.
In this work, we initiate the study of asynchronous learning algorithms for an important class of games called zero-sum polymatrix games (also known as network matrix games (Bergman & Fokin, 1998)), which generalizes two-player zero-sum matrix games to the multiple-player setting and serves as an important stepping stone to more general multi-player general-sum games.
Zero-sum polymatrix games are commonly used to describe situations in which agents’ interactions are captured by an interaction graph and the entire system of games are closed so that the total payoffs keep invariant in the system.
They find applications in an increasing number of important domains such as security games (Cai et al., 2016), graph transduction (Bernardi, 2021), and more.
In particular, we focus on finite-time last-iterate convergence to two prevalent solution concepts in game theory, namely Nash Equilibrium (NE) and Quantal Response Equilibrium (QRE) which considers bounded rationality (McKelvey & Palfrey, 1995).
Despite the seemingly simple formulation, few existing works have achieved this goal even in the synchronous setting, i.e., with instantaneous feedback.
Leonardos et al.
(2021) studied a continuous-time learning dynamics that converges to the
QRE at a linear rate.
Anagnostides et al.
(2022) demonstrated Optimistic Mirror Descent (OMD) (Rakhlin & Sridharan, 2013) enjoys finite-time last-iterate convergence to the NE, yet the analysis therein requires continuous gradient of the regularizer, which incurs computation overhead for solving a subproblem every iteration.
In contrast, an appealing alternative is the entropy regularizer, which leads to closed-form multiplicative updates and is computationally more desirable, but remains poorly understood.
In sum, designing efficient learning algorithms that provably converge to the game equilibria has been technically challenging, even in the synchronous setting.
1. O UR CONTRIBUTIONS
In this paper, we develop provably convergent algorithms—broadly dubbed as asynchronous gradient play —to find the QRE and NE of zero-sum polymatrix games in a decentralized and symmetric manner with delayed feedbacks.
We propose an entropy-regularized Optimistic Multiplicative
Weights Update (OMWU) method (Cen et al., 2021), where each player symmetrically updates their strategies without access to the payoff matrices and other players’ strategies and initiate a systematic investigation on the impacts of delays on its convergence under two schemes of learning rates schedule.
Our main contributions are summarized as follows
• Finite-time last-iterate convergence of single-timescale OMWU.
We begin by showing that, in the synchronous setting, the single-timescale OMWU method—when the same learning rate is adopted for extrapolation and update—achieves last-iterate convergence to the QRE at a linear rate, which is independent of the number of agents as well as the size of action spaces (up to logarithmic factors).
In addition, this implies a last-iterate convergence to an ϵ -approximate NE in (cid:101) O
( ϵ − 1 ) iterations by adjusting the regularization parameter, where (cid:101) O ( · ) hides logarithmic dependencies.
While the last-iterate linear convergence to QRE continues to hold in the asynchronous setting, as long as the delay sequence follows certain mild statistical assumptions, it converges at a slower rate due to a smaller tolerable range of learning rates, with the iteration complexity to find an ϵ -NE degenerating to (cid:101) O
( ϵ − 2 ) .
• Finite-time convergence of two-timescale OMWU.
To accelerate the convergence rate in the presence of delayed feedback, we propose a two-timescale OMWU method which separates the learning rates of extrapolation and update in a delay-aware manner for applications with constant and known delays (e.g. from timestamp information).
The learning rate separation is critical in bypassing the convergence slowdown encountered in the single-timescale case, where we show that two-timescale OMWU achieves a faster last-iterate linear convergence to QRE in the presence of constant delays, with an improved (cid:101)
O
( ϵ − 1 ) iteration complexity to ϵ -NE that matches the rate without delay.
We further tackle the more practical yet challenging setting where the feedback sequence is permutated by bounded delays—possibly in an adversarial manner—and demonstrate provable convergence to the equilibria in an average-iterate manner.
We summarize the iteration complexities of the proposed methods for finding ϵ -approximate solutions of QRE and NE in Table 1.
To the best of our knowledge, this work presents the first algorithm design and analysis that focus on equilibrium finding in a multi-player game with delayed feedbacks.
In contrast, most of existing works concerning individual regret in the synchronous/asynchronous settings typically yield average-iterate convergence guarantees (see e.g., Bailey (2021);
Meng et al.
and fall short of characterizing the actual learning trajectory to the equilibrium.
1. N OTATION AND PAPER ORGANIZATION
Denote by [ n ] the set { 1 , · · · , n } and by ∆( S ) the probability simplex over the set S .
Given two probability distributions p, p ′ ∈ ∆( S ) , the KL divergence from p ′ to p is defined by
KL (cid:0) p ∥ p ′ (cid:1) :
= (cid:80) k ∈ S p ( k ) log p ( k ) p ′ ( k ) .
For any vector z = [ z i ] 1 ≤ i ≤ n ∈ R n , we use exp( z ) to represent [exp( z i )] 1 ≤ i ≤ n .
The rest of this paper is organized as follows.
Section 2 provides the preliminary on zero-sum polymatrix games and solution concepts.
Performance guarantees of single-timescale
OMWU and two-timescale OMWU are presented in Section 3 and Section 4, respectively.
Numerical experiments are provided in Section 5 to corroborate the theoretical findings, and finally, we conclude in Section 6.
The proofs are deferred to the appendix.
P RELIMINARIES
In this section, we introduce the formulation of zero-sum polymatrix games as well as the solution concept of NE and QRE.
We start by defining the polymatrix game.
Definition 1 (Polymatrix game) .
Let G :=
{ ( V, E ) , { S i } i ∈ V , { A ij } ( i,j ) ∈ E } be an n -player poly- matrix game, where each element in the tuple is defined as follows.
• An undirected graph ( V, E ) , with V =
[ n ] denoting the set of players and E the set of edges; • For each player i ∈ V , S i represents its action set, which is assumed to be finite; • For each edge ( i, j
) ∈ E , A ij ∈ R | S i |×| S
j | and A ji ∈ R | S j |×| S i | represent the payoff matrices associated with player i and j , i.e
., when player i and player j choose s i ∈ S i and s j ∈ S j , the received payoffs are given by A ij ( s i , s j ) , A ji ( s j , s i ) , respectively.
Utility function.
Given the strategy profile s =
( s 1 , · · · , s n ) ∈ S =
(cid:81)
i ∈ V S
i taken by all players, the utility function u i : S → R of player i is given by
[Equation]
Suppose that player i adopts a mixed/stochastic strategy or policy, π i ∈ ∆( S i ) , where the probability of selecting s i ∈ S i is specified by π i ( s i ) .
With slight abuse of notation, we denote the expected utility of player i with a mixed strategy profile π = ( π
[Equation] , π [Equation] ) ∈ ∆
[Equation]
It turns out to be convenient to treat π i and π as vectors in R | S i | and R (cid:80) i ∈ V | S i | without ambiguity, and concatenate all payoff matrices associated with player i into
[Equation]
where A ij is set to 0 whenever ( i, j ) / ∈ E .
In particular, it follows that A ii = 0 for all i ∈ V .
With these notation in place, we can rewrite the expected utility function (1) as
[Equation]
where A i π ∈ R | S i | can be interpreted as the expected utility of the actions in S i for player i .
In addition, we denote the maximum entrywise absolute value of payoff by ∥ A ∥ ∞ = max i,j ∥
A ij ∥ ∞ =
max i ∥ A i ∥ ∞ , and the maximum degree of the graph by d max = max i ∈
V deg i , where deg i is the degree of player i .
Moreover, we denote S max = max i | S i | as the maximum size of the action space over all players.
Zero-sum polymatrix games.
The game G is a zero-sum polymatrix game if it holds that (cid:80) i ∈
V u i ( s ) = 0 , ∀ s ∈ S .
This immediately implies that for any strategy profile π ∈ ∆( S ) , it follows that (cid:80) i ∈
V u
i ( π ) = 0 .
Nash equilibrium (NE).
A mixed strategy profile π ⋆ =
( π ⋆ 1 , · · · , π ⋆n ) is a Nash equilibrium (NE) when each player i cannot further increase its own utility function u i by unilateral deviation, i.e., u i ( π ′ i , π ⋆ − i ) ≤ u i ( π ⋆i , π ⋆ − i ) , for all i ∈ V, π ′ i ∈ ∆( S i ) , where the existence is guaranteed by the work (Cai et al., 2016).
Here we denote the mixed strategies of all players other than i by π − i and write u i ( π i , π − i ) =
u i ( π ) .
To measure how close a strategy π ∈ ∆( S ) is to an NE, we introduce
[Equation]
which measures the largest possible gain in the expected utility when players deviate from its strategy unilaterally.
A mixed strategy profile π is called an ϵ -approximate Nash equilibrium ( ϵ -NE) when NE-Gap ( π ) ≤ ϵ , which ensures that u i ( π ′ i , π − i ) ≤ u i ( π i , π − i )
+ ϵ , for all i ∈ V, π ′ i ∈ ∆( S i ) .
Quantal response equilibrium (QRE).
The quantal response equilibrium (QRE), proposed by
McKelvey & Palfrey (1995), generalizes the classical notion of NE under uncertain payoffs or bounded rationality, while balancing exploration and exploitation.
A mixed strategy profile π ⋆τ =
( π ⋆ 1 ,τ , · · · , π ⋆n,τ ) is a QRE when each player assigns its probability of action according to the ex- pected utility of every action in a Boltzmann fashion, i.e., for all i ∈ V
[Equation]
where τ > 0 is the regularization parameter or temperature.
Equivalently, this amounts to maximizing an entropy-regularized utility of each player (Mertikopoulos & Sandholm, 2016), i.e., u i,τ ( π ′ i , π ⋆ − i,τ ) ≤ u i,τ ( π ⋆i,τ , π ⋆ − i,τ ) for all i ∈ V , π ′ i ∈ ∆( S i ) .
Here, the entropy-regularized utility function u i : S → R of player i is given by
[Equation]
where H ( π i ) =
− π ⊤
i log π i denotes the Shannon entropy of π i .
In Leonardos et al.
(2021), it is shown that a unique QRE exists in a zero-sum polymatrix game.
Similarly, we can measure the proximity of a strategy π to a QRE by
[Equation]
A mixed strategy profile π is called an ϵ -QRE when QRE-Gap τ ( π ) ≤ ϵ .
According to the straightforward relationship NE-Gap ( π ) ≤ QRE-Gap τ ( π ) + τ log S max , it follows immediately that we can link an ϵ/ 2 -QRE to ϵ -NE by setting τ = ϵ 2log S max .
This facilitates the translation of convergence to the QRE to one regarding the NE by appropriately setting the regularization parameter τ . Initialize
= π
[Equation]
i do
Receive t ) i ) . When t ≥ 1 , update π i according to
[Equation]
Update π i according to the single-timescale rule
[Equation]
or the two-timescale rule
[Equation]
for
P ERFORMANCE GUARANTEES OF SINGLE - TIMESCALE OMWU
In this section, we present and study the entropy-regularized OMWU method (Cen et al., 2021) for finding the QRE of zero-sum polymatrix games.
Whilst the method is originally proposed for finding QRE in a two-player zero-sum game, the update rule naturally generalizes to the multi-player setting as
[Equation]
where η > 0 is the learning rate and π ( t +1) serves as a prediction for π ( t +1) via an extrapolation step
[Equation]
In the asynchronous setting, however, each agent i receives a delayed payoff vector A
i π ( κ ( t ) i ) instead of A
i π ( t ) in the t -th iteration, where κ ( t ) i = max { t − γ ( t ) i , 0 } , with γ ( t ) i ≥ 0 representing the length of delay.
The detailed procedure is outlined in Algorithm 1 using the single-timescale rule (9) for extrapolation.
3. P ERFORMANCE GUARANTEES WITHOUT DELAYS
We first present our theorem concerning the last-iterate convergence of single-timescale OMWU for finding the QRE in the synchronous setting, i.e. γ ( t ) i = {+
+} for all i ∈ V and t ≥ 0 .
For any π, π ′ ∈ V , let KL (cid:0) π ∥ π ′ (cid:1) =
(cid:80) i ∈ V KL (cid:0) π i ∥ π ′ i (cid:1) .
Theorem 1 (Last-iterate convergence without delays) .
Suppose that the learning rate η of singletimescale OMWU in Algorithm 1 obeys 0 < η ≤ min (cid:110) 12 τ ,
[Equation]
(cid:111) , then for any T ≥ 0 , the iterates π ( T ) and π ( Ts ) converge at a linear rate according to
KL
[Equation]
11a)
Furthermore, the QRE-gap also converges linearly according to
[Equation]
Theorem 1 demonstrates that as long as the learning rate η is sufficiently small, the last iterate of single-timescale OMWU converges to the QRE at a linear rate.
Compared with prior works for finding approximate equilibrium for zero-sum polymatrix games, our approach features a closedform multiplicative update and a fast linear last-iterate convergence.
Some remarks are in order.
• Linear convergence to the QRE.
Theorem 1 implies an iteration complexity of (cid:101) O (cid:16) 1 ητ log 1 ϵ (cid:17) for finding an ϵ -QRE in a last-iterate manner, which leads to an iteration complexity of (cid:101) O (cid:16)(cid:16) d max ∥ A ∥ ∞ τ
+
1 (cid:17) log ϵ (cid:17)
by optimizing the learning rate in Theorem 1.The result is especially appealing as it avoids direct dependency on the number of agents n as well as the size of action spaces (up to logarithmic factors), suggesting that learning in competitive multi-agent games can be made quite scalable as long as the interactions among the agents are sparse (so that the maximum degree of the graph d max is much smaller than the number of agents n ).
• Last-iterate convergence to ϵ -NE.
By setting τ appropriately, we end up with an iteration complexity of (cid:101) O (cid:16) d max ∥
A ∥ ∞ ϵ (cid:17) for achieving last-iterate convergence to an ϵ -NE, which outperforms the best existing last-iterate rate of (cid:101) O (cid:0) n ∥ A ∥ ∞ /ϵ 2 (cid:1) from Leonardos et al. (2021) by at least a factor of n/ ( d max ϵ ) .
Remark 1.
Our results trivially extend to the setting of weighted zero-sum polymatrix games (Leonardos et al., 2021), which amounts to adopting different learning rates { η i } i ∈ V at each player.
In this case, the iteration complexity becomes (cid:101) O (cid:16) max i ∈ [Equation] i τ log 1 ϵ (cid:17) .
In addition, our convergence result readily translates to a bound on individual regret as detailed in Appendix C.
P ERFORMANCE GUARANTEES UNDER RANDOM DELAYS
We continue to examine single-timescale OMWU in the more challenging asynchronous setting.
In particularly, we show that the last iterate of single-timescale OMWU continues to converge linearly to the QRE at a slower rate, as long as the delays satisfy some mild statistical assumptions given below.
Assumption 1 (Random delays) .
Assume that for all i ∈ V , t ≥ 0 , the delay γ ( t ) i is independently generated and satisfies
[Equation]
Additionally, there exists some constant ζ > 1 , such that L {+≜ (cid:80) ∞ ℓ =
+}
ζ ℓ E ( ℓ ) < ∞ .
We remark that Assumption 1 is a rather mild condition that applies to typical delay distributions, such as the Poisson distribution (Zhang et al., 2020), as well as distributions with bounded support (Recht et al., 2011; Liu et al., 2014; Assran et al., 2020).
Roughly speaking, Assumption 1 implies that the probability of the delay decays exponentially with its length, where ζ − 1 approximately indicates the decay rate.
We have the following theorem.
Theorem 2 (Last-iterate convergence with random delays) .
Under Assumption 1, suppose that the regulari-zation parameter τ < min { 1 , d max ∥ A ∥ ∞ } and the learning rate η of single-timescale OMWU in Algorithm 1 obeys
[Equation]
then for any T ≥ 1 , the iterates π ( T ) and π ( T ) converges to π ⋆τ at the rate
[Equation]
(14a
[Equation]
Furthermore, the QRE-gap also converges linearly according to
[Equation]
14b
[Equation]
Theorem suggests that the iteration complexity to ϵ -QRE is no more than (cid:101)
[Equation]
(cid:16) max (cid:110)
d
[Equation]
max
[Equation]
∥
[Equation]
∞ ( L + 1) , ζζ
[Equation] (cid:111) [Equation] 1 ϵ (cid:17) after optimizing the learning rate, whose range is more limited compared with the requirement in Theorem 1without delays.
In particular, the range of the learning rate is proportional to the regularization parameter τ , an issue we shall try to address by resorting to two-timescale learning rates in OMWU.
To facilitate further understanding, we showcase the iteration complexity for finding ϵ -QRE/NE under two typical scenarios: bounded delay and Poisson delay.
• Bounded random delay.
When the delays are bounded above by some maximum delay γ , Assumption 1 is met with ζ = 1 + γ − 1 and L =
eγ ( γ + 1) .
Plugging into Theorem 2 yields an iteration complexity of (cid:101) O (cid:16) d 2 max ∥ A ∥ 2 ∞ ( γ +1) 2 τlog 1 ϵ (cid:17) for finding an ϵ -QRE, or (cid:101) O (cid:16) d 2 max ∥ A ∥ 2 ∞ (
[Equation]
) 2 ϵ
[Equation]
for finding an ϵ -NE, which increases quadratically as the maximum delay increases.
Note that these rates are worse than those without delays (cf.
Theorem 1).
• Poisson delay.
When the delays follow the Poisson distribution with parameter 1 /T , it suffices to set ζ = 1 + T − 1 and L = eT (1 + T ) Assumption 1.
This leads to an iteration complexity of (cid:101) O (cid:18)
d 2 max ∥ A ∥ 2 ∞ T 2 τlog 1 ϵ (cid:19) for finding an ϵ -QRE, or (cid:101) O (cid:18)
d 2 max ∥ A ∥ 2 ∞ T 2 ϵ(cid:19) for finding an ϵ -NE, which is similar to the bounded random delay case.
P ERFORMANCE GUARANTEES OF TWO - TIMESCALE OMWU
While Theorem 2 demonstrates provable convergence of single-timescale OMWU with random delays it remains unclear whether the update rule can be better motivated in more general asynchronous settings and whether the convergence can be further ensured under adversarial delays.
Indeed theoretical insights from previous literature (Mokhtari et al., 2020; Cen et al., 2021) suggest the critical role of π ( t ) as a predictive surrogate for π ( t ) in enabling fast convergence which no longer holds when π ( t ) is replaced by a delayed feedback from π ( κ ( t ) i ) .
To this end we propose to replace the extrapolation update (9) with one equipped with a different learning rate:
[Equation]
which adopts a larger learning rate ¯ η > η to counteract the delay
Intuitively, a choice of η ≈ ( γ ( t ) i + 1) η would allow π ( κ ( t ) i ) to approximate π ( t ) by taking the intermediate updates { π ( l ) : κ ( t ) i ≤ l < t } into consideration.
We refer to this update rule as the two-timescale entropy-regularized OMWU, whose detailed procedure is again outlined in Algorithm 1 using (10) for extrapolation.
4. P ERFORMANCE GUARANTEES UNDER CONSTANT AND KNOWN DELAYS
To highlight the potential benefit of learning rate separation, we start by studying the convergence of two-timescale OMWU in the asynchronous setting with constant and known delays, which has been studied in (Weinberger & Ordentlich, 2002;
Flaspohler et al., 2021; Meng et al., 2022).
We have the following theorem, which reveals a a faster linear convergence to the QRE by using a delay-aware two-timescale learning rate design.
Theorem 3 (Last-iterate convergence with fixed delays) .
Suppose that the delays γ ( t ) i = γ are fixed and known.
Suppose that the learning rate η of two-timescale OMWU in Algorithm 1 satisfies
[Equation]
and η is determined by 1 − ητ
=
(1 − ητ ) ( γ +1) , then the last iterate π ( t ) and π ( t ) converge to the
QRE at a linear rate: for T ≥ γ ,
[Equation]
In addition, the QRE-gap converges linearly according to
QRE
[Equation]
By optimizing the learning rate η , Theorem 3 implies that two-timescale OMWU takes at most (cid:101) O (cid:16)
[Equation]
log 1 ϵ (cid:17) iterations to find an ϵ -QRE in a last-iterate manner, which translates to an iteration complexity of (cid:101) O (cid:16) d max ∥
A ∥ ∞ ( γ +
[Equation]
2 ϵ (cid:17) for finding an ϵ -NE.
This significantly improves over the iteration complexity of (cid:101) O (cid:0) d 2 max ∥ A ∥ 2 ∞ ( γ + 1) 2 /ϵ 2 (cid:1) for single-timescale OMWU, verifying the positive role of adopting two-timescale learning rate in enabling faster convergence.
4. P ERFORMANCE GUARANTEES WITH PERMUTED BOUNDED DELAYS
The above result requires the exact information of the delay, which may not always be available.
Motivated by the need to address arbitrary or even adversarial delays, we consider a more realistic scenario, where the payoff sequence arrives in a permuted order (Agarwal & Duchi, 2011) constrained by a maximum bounded delay (McMahan & Streeter, 2014; Wan et al., 2022).
Assumption 2 (Bounded delay) .
For any i ∈ V and t > 0 , it holds that γ ( t ) i
[Equation]
Assumption 3 (Permuted feedback) .
For any t > 0 , the payoff vector at the t -th iteration is received by agent i only once.
The payoff at the 0 -th iteration can be used multiple times.
The following theorem unveils the convergence of two-timescale OMWU to the QRE in an average sense under permutated bounded delays.
Theorem 4 (Average-iterate convergence under permutated delays) .
Under Assumption 2 and 3, suppose that the learning rate η of two-timescale OMWU in Algorithm 1 satisfies η ≤ min (cid:110)
[Equation]
, and η is determined by 1 − ητ
=
(1 − ητ ) ( γ +1) , then for
T > 2 γ , it holds that
[Equation]
Furthermore, the average QRE-gap can be bounded by
[Equation]
Theorem 4 guarantees that the best iterate among { π ( t ) } 2 γ<t ≤ T is an ϵ -QRE as long as
T is on the order of (cid:101) O (cid:16) nd 3 max ∥
A ∥ 3 ∞ ( γ + [Equation]
[Equation]
2 τ 2 ϵ (cid:17) , which translates to an iteration complexity of [Equation] O
[Equation]
(cid:17) for finding an ϵ -NE.
While the rate seems slower than the previous theorems, Theorem 4 holds under arguably the weakest delay assumptions, where it can be even adversarially bounded.
We remark that the result in (16) also guarantees the convergence of the last iterate π ( t ) to the QRE asymptotically, although without a finite-time rate.
This is in sharp contrast to typical average-iterate analysis that only applies to 1 T (cid:80)
Tt =1 π ( t ) without implications on the convergence of the last iterate π ( t ) . Remark 2.
The analysis in this section can be generalized to more commonly-used delay models where the reward information is not assumed to be observed once per round (Quanrud & Khashabi, 2015; Joulani et al., 2013), i.e., in every round an agent may observe multiple reward feedbacks from previous iterations or receive no information.
This can be achieved by storing reward feedbacks in a buffer memory and picking one for policy update every round in a First-In-First-Out manner.
N UMERICAL EXPERIMENTS
In this section, we verify our theoretical findings by investigating the performance of both singletimescale and two-timescale OMWU on randomly generated zero-sum entropy-regularized polymatrix games with n = 10 , | S i | = 10 , i ∈ V and τ = 0 .
For each ( i, j ) ∈ E , we set A ij = − A ⊤ ji with entries of A ij independently sampled from the uniform distribution over [ − 1 , 1] .
All the results are averaged over five independent runs.
In Fig.
1 (a), we compare the performance of single-timescale OMWU in both synchronous and asynchronous settings, with delay uniformly sampled from { 0 , 1 , . . .
We adopt the optimal learning rate η from { 0 . 1 , 0 . 05 , 0 . 02 , 0 . 01 , . . . } that yields the highest accuracy.
The method [Figure] achieves linear convergence in both cases, yet the convergence rate is slowed down by delayed feedbacks in the asynchronous setting.
Fig.
1 (b) and (c) compare the effect of different choices of learning rates η, η on the performance of the proposed methods, where the feedback is permutated with bounded delay γ = 25 (cf.
Assumptions 2 and 3).
In general, two-timescale OMWU outperforms single-timescale OMWU given appropriate choices of learning rate η .
On the other hand, (c) demonstrates that the choice of ¯ η = τ − 1 (1 − (1 − ητ ) γ +1 ) suggested by the theory (marked with star) indeed leads to near-optimal performance of two-timescale OMWU.
Figure 2 shows KL (cid:0) π ⋆τ ∥ π ( t ) (cid:1) with respect to the number of iterations of single-timescale and twotimescale OMWU under different asynchronous scenarios, with optimal choices of η and τ = 0 . 1 .
In particular, two-timescale OMWU adopts the extrapolation learning rate suggested by theory ¯ η = τ − 1 (1 − (1 − ητ ) γ +1 ) .
While both methods yield linear convergence to the QRE, two-timescale method outperforms its single-timescale counterpart in the case with constant and known delay and the case where the feedback is permutated with bounded delay, which verifies our theory.
[Figure]
C ONCLUSION
This paper studies asynchronous gradient play in zero-sum polymatrix games, by investigating the convergence behaviors of entropy-regularized OMWU with delayed feedbacks under two different schedules of the learning rates.
We demonstrate that single-timescale OMWU enjoys a linear lastiterate convergence to the QRE even under mild statistical delays.
However, the presence of the delay noticeably limits the allowable range of learning rates and slows down the convergence.
To mitigate the impact, we further show that the method benefits from adopting a two-timescale learning rate in a delay-aware manner, which achieves a faster last-iterate convergence when the delay is fixed and known, and continues to converge provably even when the delays are arbitrarily bounded in an average-iterate manner.
We believe our work lays the foundation for further understandings of delayed feedback in games under symmetric and independent learning.
A CKNOWLEDGEMENT
S. Cen and Y. Chi are supported in part by the grants ONR N00014-19-1-2404, NSF CCF-1901199,
CCF-2106778 and CNS-2148212. S. Cen is also gratefully supported by Wei Shen and Xuehong Zhang Presidential Fellowship, and Nicholas Minnici Dean’s Graduate Fellowship in Electrical and Computer Engineering at Carnegie Mellon University.
R. Ao is supported by the Elite Undergraduate Training Program of School of Mathematical Sciences at Peking University
Table of Contents
A Further related works
B Proof for single-timescale OMWU (Section 3)
B.1 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.2 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C Regret analysis of single-timescale OMWU
C.1 Proof of Theorem 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.2 Proof of Theorem 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D Proof for two-timescale OMWU (Section 4)
D.1 Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Proof of Theorem 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E
Proof
of auxiliary lemmas
E.1 Proof of Lemma 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.2 Proof of Lemma 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.3 Proof of Lemma 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.4 Proof of Lemma 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.5 Proof of Lemma 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.6 Proof of Lemma 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.7 Proof of Lemma 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.8 Proof of Lemma 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.9 Proof of Lemma 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A F URTHER RELATED WORKS
Learning in two-player zero-sum matrix games.
Freund & Schapire (1999) proved that Multiplicative Weights Update (MWU) method achieve an average-iterate convergence rate of O (1 / √ T ) through the lens of regret analysis.
Daskalakis et al.
(2011) is the first to achieve an optimal convergence rate of O (1 /T ) with the excessive gap technique of Nesterov (Nesterov, 2005a;b).
Rakhlin & Sridharan (2013) achieved the same rate with OMD, which is more commonly referred to as
OMWU when entropy regularization is in use for the mirror descent update rule.
In terms of lastiterate convergence, Daskalakis & Panageas (2018) established asymptotic last-iterate convergence for OMWU assuming the uniqueness of NE.
Wei et al.
(2021) improved upon the analysis under the same assumption by showing a problem-dependent linear rate of convergence, which is extended to a class of extensive-form games (Lee et al., 2021).
Cen et al.
(2021) showed that entropy-regularized
OMWU converges linearly to the QRE of two-player zero-sum matrix game, which translates to an iteration complexity of (cid:101) O
(1 /T ) for finding an ϵ -NE, without assuming its uniqueness; the linear convergence to the QRE continues to hold with smooth value updates (Cen et al., 2022).
Sokota et al.
(2022) showed that linear convergence to QRE can be achieved without resorting to optimistic update rules, e.g., using entropy-regularized MWU, albeit with a more restrictive learning rate.
It is worth pointing out that the idea of learning rate separation has been explored for equilibrium finding in two-player zero-sum games with instant feedback (Fasoulakis et al., 2022) and online learning with delayed feedback (Hsieh et al., 2020), but lacks study in an asynchronous multi-player game setting.
Asynchronous optimization.
Asynchronous and decentralized optimization algorithms have been extensively studied since the proposal of Bertsekas & Tsitsiklis (1989), where a number of agents seek to find an approximate global optimizer of a common loss function, by performing iterative gradient-based methods in a collaborative manner.
Typical approaches including parallelizing the computation of gradient with regard to data (Tong et al., 2020), or parallelizing the model updates by imposing coordinate update rules (Nesterov, 2012; Liu et al., 2014; Liu & Wright, 2015).
Delayed gradient (feedback) is also common in these scenarios due to the existence of other agents updating the model.
Moreover, the zero-sum polymatrix setting considered in this work is inherently noncollaborative by requiring every agent to maximize its own utility function and compete with other agents, and leads to substantially difference analysis techniques.
B P ROOF FOR SINGLE - TIMESCALE OMWU (S ECTION 3)
Before delving into the main proof, we first record a useful lemma pertaining to a basic property of zero-sum polymatrix games; the proof is deferred to Appendix E.1.
For i ∈ V , we denote by
N i = { j : ( i, j ) ∈ E } the neighbors of agent i in the graph ( V, E ) .
For notational simplicity, we denote by x 1 = y
the equivalence between two vectors x and y up to a global shift, i.e.,
[Equation]
(17) for some constant c ∈ R , where 1 is the all-one vector.
Lemma 1.
For any zero-sum polymatrix game G , it holds that for π, π ′ ∈ ∆( S ) that
[Equation]
Or equivalently, (cid:80) i ∈ V (cid:2) π ⊤ i A i π ′
+ ( π ′ i )
⊤ A i π (cid:3) = 0 .
It follows that (cid:88) i ∈ V (cid:10)
[Equation]
B. P ROOF OF T HEOREM
We start with the following lemma that characterizes the iterates of OMWU [Equation] which generalizes Cen et al.
(2021, Lemma 1) for zero-sum two-player games to zero-sum polymatrix games.
The proof can be found in Appendix E.
[Equation]
Lemma 2.
The iterates of OMWU based on the update rule (9) satisfy
[Equation]
To continue, by the definition of KL divergence
[Equation]
we have
[Equation]
In addition,
[Equation]
Summing up the above two relations
[Equation]
in view of Lemma
[Equation]
it holds that
[Equation]
We now proceed to bound the terms of interest one by one
[Equation] Bounding KL (cid:0) π ⋆τ ∥ π ( t ) (cid:1) .
We aim to control the right-hand-side (RHS) of (19).
Based on the update rule of π ( t +1)
i in Algorithm 1, we have
[Equation]
It follows that
[Equation]
where the last line follows from Pinsker’s inequality.
Summing the inequality over i ∈ V
[Equation]
we get
[Equation]
Plugging the above inequality back into (19) yields
[Equation]
With the choice of the learning rate
[Equation]
it holds that 1 − ητ − ηd max ∥ A ∥ ∞ > 0 and [Equation]
[Equation]
This allows us to further relax (22) by
[Equation]
Let us now introduce the potential function of iterates
[Equation]
which allows us to simply the previous inequality as
[Equation]
where the last equality follows from the definition π (0)
=
π (0) .
Hence, we have
[Equation]
Bounding KL (cid:0) π ⋆τ ∥ π ( t +1) (cid:1) .
Following similar approaches to (21), we can bound
[Equation]
Summing the inequality over i ∈ V leads to
[Equation]
On the other hand [Equation] by the definition of KL divergence
[Equation]
we have
[Equation]
[Equation]
Combining the above two inequalities
[Equation]
we get
[Equation]
Plugging the above inequality back into (22)
[Equation]
we have
[Equation]
where the second and third inequalities follow from the choice of the learning rate
[Equation]
and the last line follows from the definition of the potential function L ( t ) .
Then the result follows from (24) as
[Equation]
Bounding the QRE-Gap.
Finally, we bound the QRE-gap, which can be linked to the KL divergence using the following lemma.
The proof can be found in Appendix E.
[Equation]
Lemma 3.
For any π ∈ ∆( S ) and QRE π ⋆τ ∈ ∆( S ) , it holds that
[Equation]
Lemma 3 tells us
[Equation]
With KL (cid:0) π ⋆τ ∥ π ( t ) (cid:1) controlled in the above
[Equation]
we still need to control KL (cid:0) π ( t ) ∥
π ⋆τ (cid:1) .
From (22), it follows that
[Equation]
Plugging them back to (27), we arrive at
[Equation]
B. P ROOF OF T HEOREM
We begin with bounding the KL divergence KL (cid:0) π ⋆τ ∥
π ( t ) (cid:1) and then move to bound the QRE-gap by linking it to the KL divergence.
Bounding the term KL (cid:0) π ⋆τ ∥ π ( t ) (cid:1) .
We start with the following equation
[Equation]
where its proof follows a similar deduction as (19).
Our first target is to bound the last two terms on the RHS of (28) with
[Equation]
Let us introduce the potential function of iterates
[Equation]
which will be used repetitively in the rest of this proof.
For notational simplicity, let Ψ ( l ) i =
when l < 0 .
Step 1: bounding
[Equation]
Following a similar argument as (21), we get
[Equation]
To control the term (cid:13)(cid:13) π ( κ ( t +1) i ) j − π ( κ
[Equation]
t ) i
[Equation]
(cid:13)(cid:13) 2 1 ,
when t = 0 , we have
[Equation]
by Pinsker’s inequality.
For t ≥ 1 , consider the decomposition
[Equation]
it then follows that
[Equation]
where the last line applies Pinsker’s inequality.
Depending on whether γ ( t +1) i > 0 , we proceed to bound the terms (cid:13)(cid:13) π ( κ ( t +1) i )
[Equation]
(cid:13)(cid:13) 2 1 in (29) considering the following two cases based on (31).
• γ ( t +1) i = 0 .
Then
[-j − π ( t ) j (cid:13)(cid:13) 2 1 + 
 (cid:13)(cid:13) π ( t ) j − π ( κ ( t ) i )-][Equation]
[-j (cid:13)(cid:13) 
-]
[Equation]
where the last step uses (31) and
[Equation]
via again Pinsker’s inequality.
• γ ( t +1) i >
Then it follows similarly that
[Equation]
Combining the above two bounds together
[Equation]
we get
[Equation]
when t > 0 .
In view of (30) when t =
[Equation]
the above bound (32) holds for all t ≥ 0 .
Plugging the above inequality into (29) yields
[Equation]
Step 2: bounding ( π ( t +1)
i
[Equation]
−
[Equation]
⋆τ ) .
Let us begin with the following decomposition
[Equation]
where the second term in the RHS of (34) can be bounded by
[Equation]
Following similar deduction of (32) for the second term
[Equation]
we attain
[Equation] [Equation]
Plugging the above inequality back to (34) results in
[Equation]
Step 3: combining the bounds.
For simplicity, we introduce the short-hand notation
[Equation]
and
[Equation]
Combining (33) and (35) into (28)
[Equation]
and summing over i ∈ V gives
[Equation]
γ
[Equation]
i
[Equation]
l
[Equation]
t
[Equation]
i
[Equation]
l
[Equation]
≥
[Equation]
t
[Equation]
l [Equation] t − γ ( t ) i γ ( t ) i
[Equation]
( l ) j (cid:33) [Equation] (37) where we make use of the fact (cid:88)
[Equation]
from Lemma 1 in the first inequality, and the second inequality uses the relation (cid:88)
[Equation]
Step 4: finishing up via averaging the delay.
We now evaluate the expectation of
KL (cid:0) π ⋆τ ∥ π ( t +1) (cid:1) .
Recall that we use subscript E γ ( t ) [ · ] to represent the conditional expectation given γ ( t ) =
{ γ ( t ) i } i ∈ V .
We shall first control the conditional expectation of the last term in (37).
Observing that π ( l +1) j , π ( l ) j are independent of γ ( t ) i for j ∈ N i and l ≤ t − 1 .
Using the definition of
E ( t − l ) , we have (cid:88) i ∈ V [Equation] j
[Equation]
i [Equation] [Equation] where the second line follows from the definition of E ( t − l ) in Assumption 1.
Applying a similar argument to bound (cid:80)
i
[Equation]
V
[Equation]
(cid:80)
[Equation]
Ψ ( l
[Equation]
(cid:105) , and taking expectation of γ ( t ) , γ ( t +1) on both sides of (37), we get
[Equation]
Taking expectation on both sides over all the delays yields
[Equation]
Telescoping over t = 0 , 1 , . .
, T , we get
[Equation]
which leads to the desired bound if
[Equation]
Proof of (41) .
To begin, notice that with the choice of the learning rate
[Equation]
it follows that
[Equation]
and
[Equation]
as τ ≤ d max ∥ A ∥ ∞ .
Both of these relations will be useful in our follow-up analysis.
Now, taking the definition of U ( t ) (cf. (39)), we have
T (cid:88)
[Equation]
where the second half of the RHS can be further controlled via
[Equation] [Equation]
where the first line follows by changing the order of summation, the second line follows from the change of variable l ′ =
l − t , and the last line follows from (42a) and the definition of L in Assumption 1.
Plugging the above relation back leads to
[Equation]
where the second line results from (42b).
Bounding the term KL (cid:0) π ⋆τ ∥ π ( t +1) (cid:1) .
With a similar deduction of (19), we get
[Equation]
Following the similar argument of (35)
[Equation]
we have (
[Equation]
Summing over i ∈ V and plugging into (44) yields
[Equation]
Taking expectation on both sides over all delays and using (38) leads to
[Equation]
Notice that with the choice of the learning rate
[Equation]
we have
[Equation]
and
[Equation]
by combining (43) and (40).
It follows that
[Equation]
and
[Equation]
where (i) is by the bound (1 − ητ ) − 1 ≤ ζ and (ii) uses the definition of L in Assumption 1.
Plugging the above inequalities into (45) leads to
[Equation]
Then from (40) we have
[Equation]
Bounding the QRE-Gap.
Combining (27) and (46), we have
[Equation]
where the second line uses the learning rate bound
[Equation]
C R EGRET ANALYSIS OF SINGLE - TIMESCALE OMWU
For completeness, we also provide the regret analysis of single-timescale OMWU in both synchronous and asynchronous settings, which might be of independent interest.
To begin, for τ ≥ 0 , the regret for each player i ∈ V is defined as
[Equation]
which measures the performance gap compared to the optimal fixed strategy in hindsight for player
i , when the rest of the players follow the strategies derived from Algorithm 1.
Synchronous setting.
We begin with the following no-regret guarantee of single-timescale
OMWU in the synchronous setting.
Theorem 5 (No-regret without delays) .
Suppose all players i ∈ V follow single-timescale OMWU in Algorithm 1 with the initialization π (0)
i = 1 | S i
| 1 and the learning rate obeys 0 < η ≤ 1 4 d max ∥ A ∥ ∞ +4 τ .
Then, for T ≥ 1 , it holds that
[Equation]
By optimizing the learning rate η , Theorem 5 suggests that the regret is bounded by
[Equation]
up to logarithmic factors.
Compared with the OMD method for multi-agent games in Anagnostides et al.
(2022), which only provided the regret bound for τ = 0 , our bound is more general by allowing entropy regularization.
Moreover, our bound is tighter by a factor of n/d max by exploiting the graph connectivity pattern, which is significant for large sparse graphs.
Asynchronous setting.
We next move to the asynchronous case, and show that single-timescale
OMWU continues to enjoy no-regret learning as long as the delays have finite second-order moments.
Assumption 4 (Random delays) .
Recall the definition of E ( ℓ ) in (12) .
There exists some constant σ > 0 , such that E (cid:104) ( γ ( t ) i ) 2 (cid:105) ≤ (cid:80) ∞
ℓ =0 E ( ℓ ) ≤ σ 2 , for all t ≥ 0 and i ∈ V .
Clearly Assumption 4 is weaker than Assumption 1, since it only requires the second-order moments to be finite, instead of an exponential decay of γ ( t ) i .
We have the following theorem.
Theorem 6 (No-regret with random delays) .
Under Assumption 4, suppose all players i ∈
V follow single-timescale OMWU in Algorithm 1 with the initialization π (0) i = 1 | S i | 1 , the regularization parameter τ < min { 1 , ∥ A ∥ ∞ } , and the learning rate obeys 0 < η ≤ τ 24 d 2 max ∥
A ∥ 2 ∞ ( σ 2 +1) .
Then, for
T ≥ 1 , it holds that
[Equation]
Theorem 6 guarantees that the iterate among { π ( t ) } t ≥ 1 enjoys a regret bound on the order of
[Equation]
by optimizing the learning rate η .
C. P ROOF OF T HEOREM
Recall the expression of the regret
Regret i
[Equation]
where
Regret i,τ (cid:0) π i , T (cid:1) := T (cid:88) t = u i,τ ( π i , π ( t ) − i ) −
T (cid:88) t = u i,τ ( π ( t ) ) = T (cid:88) t = (cid:16)(cid:10)
π
i − π ( t +1)
i , A
i π ( t +1) (cid:11) + τ H ( π i ) − τ H ( π ( t +1) i ) (cid:17) .
Therefore, it is sufficient to bound Regret i,τ (cid:0)
π i , T (cid:1) for any π i ∈ ∆( S i ) .
To begin, we record the following useful lemma whose proof can be found in Appendix E.4.
Lemma 4.
For any τ ≥ 0 and T ≥ 0 , we have (cid:88)
i
[Equation]
Let us now proceed with the following regret decomposition
[Equation]
We proceed to bound each term on the RHS of (50).
• To begin, note that log π ( t +1)
i
[Equation]
The first term in (50) can then be written as
[Equation]
where the second step is derived from the definition of KL divergence.
• Similarly, the second term in (50) has the form
[Equation]
• Moving to the third term on the RHS of (50), we first make the following claim, which shall be proven at the end of this proof:
[Equation]
With (53) in place, we have −
[Equation]
The latter term can be further bounded by (cid:16) (cid:88)
[Equation]
i
[Equation]
π ( t ) j − π ( t ) j
(cid:13)(cid:13) 2 1 (cid:17) ≤ 4 deg i (cid:88)
j ∈N i (cid:16) KL (cid:0) π ( t +1)
j ∥ π ( t ) j (cid:1) + KL (cid:0) π
( t ) j ∥ π ( t ) j (cid:1)(cid:17) , where it follows respectively from Cauchy-Schwarz inequality, ∥ a + b ∥ 21 ≤ 2 (cid:0) ∥ a ∥ 21 + ∥ b ∥ 21 (cid:1) [Equation] and Pinsker’s inequality.
Plugging this into the previous inequality leads to −
[Equation]
Plugging (51), (52) [Equation] and (54) into (50) yields
[Equation]
Telescoping the sum over t = 0 , 1 , . .
, T leads to
Regret i,
[Equation]
where the last line follows from the fact that KL (cid:0) π i ∥ π (0) i (cid:1) ≤ log | S i | and 1 /η >
τ .
The proof is thus complete if we can establish
[Equation]
Therefore, it remains to establish (53) and (57), which shall be completed as follows.
Proof of (53) .
By the update rules of π ( t +1) i and π ( t +1)
i , from (20) we can deduce that
[Equation]
By Pinsker’s inequality, we have
[Equation]
In addition,
[Equation]
Plugging the above two relations into (58) then leads to (53). Proof of (57) .
Summing (55) over i ∈ V gives (cid:88) i ∈ V
Regret i,τ (cid:0) π i , T
[Equation]
1 (cid:1) ≤
[Equation] η KL (cid:0) π ∥
[Equation]
(0) (cid:1)
[Equation]
η
T (cid:88)
[Equation]
(cid:0) π ( t +1)
∥
π ( t + [Equation] [Equation] (59) where the last line follows from π (0) =
π (0) , KL (cid:0) π ∥ π (0) (cid:1) ≤ (cid:80) k ∈ V log | S k | for any π since π (0) is a uniform distribution, as well as the choice of the learning rate such that
[Equation]
and
[Equation]
Taking supremum over π on both sides of (59) and applying Lemma 4 gives (57) as advertised.
C. P ROOF OF T HEOREM
Similar to the proof of Theorem 5 in Appendix C.1, it suffices to bound Regret i,τ (cid:0)
π i , T (cid:1) for any π i ∈ ∆( S i ) , where
Regret
[Equation]
[Equation]
Consider the following decomposition:
[Equation]
We now bound each term on the RHS of (61).
For simplicity, we reuse the short-hand notation in (36).
• To begin with, note that π ( t +1)
i
[Equation]
for some normalization constant c i .
Thus we have
[Equation]
where the second step is derived from the definition of KL-divergence
• Similarly [Equation] it holds that
[Equation]
• For the term
[Equation]
in (61)
[Equation]
following the deduction of (33)
[Equation]
we get
[Equation]
• For the last term in (61), it similarly follows that
[Equation]
Plugging (62) (63) (64) (65) into (61) yields
[Equation]
Note that
[Equation]
Then we have
[Equation]
Since the learning rate satisfies
[Equation]
taking expectation on both sides of (66) leads to
[Equation]
≤
[Equation] [Equation]
where we use the fact (cid:80) j ∈N i Ψ ( l ) j ≤ Ψ ( l ) and the definition of E ( t − l ) .
Since (cid:80) ∞ l =0 E ( l ) ≤ σ 2 by definition in Assumption 4, summing (67) over t = 0 , 1 , . . .
, T yields
[Equation]
It remains to establish
[Equation]
Proof of (69) .
Summing (66) over i ∈ V gives
[Equation]
Taking expectation of on both sides and using (38) leads to
E (cid:34)(cid:88)
[Equation]
Summing over t = 0 , 1 , . .
. , T yields
E (cid:34)(cid:88) i ∈ V
Regret i,
[Equation]
where the second line follows from
[Equation]
due to τ ≤ d max ∥ A ∥ ∞ and η ≤ τ 24 d 2 max ∥
A ∥ 2 ∞ ( σ 2 +1) .
Taking supremum with respect to π on both sides, in view of Lemma 4, we arrive at the advertised bound (69).
D P ROOF FOR TWO - TIMESCALE OMWU (S ECTION 4) D. P ROOF OF T HEOREM
Bounding KL (cid:0) π ⋆τ ∥ π ( t ) (cid:1) .
For notational convenience, we set π ( t ) =
π ( t ) =
π (0) for t < 0 .
The following lemma parallels Lemma 2 by focusing on delayed feedbacks.
The proof is postponed to
Appendix E.5.
Lemma 5.
Assuming constant delays γ ( t ) i
= γ , the iterates of OMWU based on the update rule (10) satisfy
[Equation]
By following a similar argument in (19), we conclude that
[Equation]
It boils down to control the term − (cid:10) log π ( t − γ +1)
− log π ( t +1) , π ( t − γ +1) − π ( t +1) (cid:11) .
When t ≥ γ , by taking logarithm on the both sides of the update rules (7) and (10), we have
[Equation]
and
[Equation]
Subtracting the above equalities and taking inner product with π ( t − γ +1)
i − π ( t +1) i gives
[Equation]
where the log π ( t − γ ) i terms cancel out due to the choice 1 − ητ
=
(1 − ητ )
γ +1 .
Summing over i ∈ V ,
[Equation]
Using the triangle inequality, we can bound (cid:13)(cid:13) π ( t − 2 γ ) − π
( t −
[Equation]
− l
[Equation]
1) (cid:13)(cid:13) [-
 +-]1 as (cid:13)(cid:13)
[Equation]
Substitution of the bound into (72) yields
[Equation]
[Table]
[Equation]
Plugging the above inequality into (71) and recursively applying the inequality gives
[Equation]
[Table]
[Equation]
where (i) results from basic calculation
[Equation]
[Table]
and (ii) is due to η ≤ min (cid:110)
[Table]
(cid:111) . To proceed, we introduce the following lemma concerning the error KL (cid:0) π ⋆τ ∥ π ( γ ) (cid:1) , with the proof postponed to Appendix E.6.
Lemma 6.
With constant delays γ ( t ) i
= γ , the iterates of OMWU based on the update rule (10) satisfy
[Equation]
With the lemma above in mind, we can continue to bound (74) by
[Equation]
≤
[Equation]
π [Equation] [Equation] +
[Equation]
1 −
[Equation]
[Equation]
Bounding KL (cid:0) π ⋆τ ∥
π ( t − γ +1) (cid:1) .
By definition of KL divergence, we have
KL
[Equation]
=
[Equation]
=
[Equation]
⋆τ [Equation] ( t +1) , log π ( t +1)
[Equation]
log π ( t − γ +1) (cid:11) . It remains to control the term (cid:10) π ⋆τ − π ( t +1) , log π ( t +1) − log π ( t − γ +1) (cid:11) .
By following a similar argument in (73), we have
[Equation]
t − γ (cid:13)(cid:13)
{+π ⋆i,τ − π ( t +1) i (cid:13)(cid:13) 1 (cid:16)(cid:13)(cid:13) π ( l 1 − γ ) j − π ( l 1 ) j (cid:13)(cid:13) 
 + (cid:13)(cid:13) π+}
( l 1 − γ +1)
j − π ( l 1 ) j (cid:13)(cid:13) 1 (cid:17)
[Equation]
[Equation]
Substitution of the above inequality into (75) yields
[Equation]
[Table]
[Equation]
where (i) results from t (cid:88)
[Equation]
and (ii) is due to the bound established in (74).
Finally, applying Lemma 6 yields
[Equation]
[Equation]
Bounding the QRE gap.
With Lemma 3, we have
[Equation]
, where the last step results from (76).
D. P ROOF OF T HEOREM
Bounding the term KL (cid:0) π ⋆τ ∥ π ( t ) (cid:1) .
Recall that the update rule of π ( t ) i ( k ) is given by
[Equation]
We introduce an auxiliary variable (cid:101) π ( t ) i :
[Equation]
78) which can be viewed as a conceptual alternative update of π ( t ) i with a different step size (cid:101) η ( t ) i >satisfying
[Equation]
or equivalently
[Equation]
It directly follows that (cid:101) η ( t ) i ≥ η .
Since κ ( t ) i ≤ t , we have 1 − (cid:101) η ( t ) i τ ≥ 1 − ( γ + 1 − t + κ ( t ) i ) ητ ≥ 1 − ( γ + 1) ητ , which implies (cid:101) η ( t ) i ≤ ( γ + 1) η.
For notational convenience, we set (cid:101) π ( t ) i =
π (0) , (cid:101) η ( t ) i
= η and κ ( t ) i
=
when t ≤ 0 .
The following lemma establishes a one-step analysis, with the proof postponed to Appendix E.7.
Lemma 7.
When t ≥ 1 , it holds that
[Equation]
79) where
[Equation]
We proceed to control the term (cid:10) log π ( κ ( t ) i )
i
[Equation]
π
[Equation]
t )
[Equation]
(cid:11) .
By definition, we have
[Equation]
[Equation]
and
[Equation]
when κ ( t ) i ≥ 1 .
Subtracting the two equations yields
[Equation]
(80) where the log π ( κ ( t ) i − 1) terms cancel out due to (1 − (cid:101) η ( t ) i τ )(1 − ητ ) t − κ ( t ) i = 1 − ητ .
It follows that
[Equation]
The next lemma establishes an upper bound on the term (cid:80) t [Equation]
κ
[Equation]
i
[Equation]
1 , with the proof postponed to Appendix E.8.
Lemma 8.
Let ν j ( t ) denote the time index when agent j receives the payoff from the t -th iteration, i.e., κ ( ν j ( t ))
j = t .
For t = 0 , we set ν j (0) to an arbitrary index that satisfies κ ( ν j (0))
j = 0 .
When t ≥ 2 γ + 1 , it holds that
[Equation]
Plugging Lemma 8 into (81) gives
[Equation]
[Equation]
where (i) results from Young’s inequality
[Equation]
and (ii) follows from (cid:13)(cid:13)(cid:13)(cid:13) π
[Equation]
κ [Equation] i − [Equation]
π
[Equation]
π
[Equation]
κ
[Equation]
γ + 1)
[Equation] t [Equation] Plugging (82) into (79) and summing over i ∈ V yields
[Equation]
where we denote (cid:80) i ∈ V ψ ( l ) i by ψ ( l ) for notation simplicity.
We then seek to sum the above equation over t = 2 γ + 1 , · · · , T .
Before proceeding, we note that
[Equation]
and that
T (cid:88)
[Equation]
where the first step is due to the mapping t (cid:55)→ ν
j
( κ ( κ ( t − 1) i ) i ) being injective when t ≥ 2 γ +
(cf.
Assumptions 2, 3).
Note that ψ ( t ) j = 0 when t ≤ 0 and hence can be safely discarded.
Taken together, we arrive at ητ
[Equation] π ⋆τ ∥ π (2 γ ) (cid:1) − η T (cid:88) t =2 γ + (cid:88) i ∈ V ( π ( κ ( t ) i )
i − π ⋆i,τ )
⊤ A i ( π ( κ ( t ) i ) − π ⋆τ ) − (cid:16)− 14 ηd max ∥
A ∥ ∞ ( γ + 1) 5 / 2 (cid:17)
T (cid:88) t =2 γ + ψ ( t ) + 12 ηd max ∥
A ∥ ∞ ( γ + 1) 5 / T + γ (cid:88) l = ψ ( l ) + 2 ηd max ∥ A ∥ ∞ ( γ + 1) 5 / T + γ − 1 (cid:88) t = ψ ( t ) ≤ (1 − ητ ) KL (cid:0) π ⋆τ ∥
π (2 γ ) (cid:1) − η T (cid:88) t =2 γ + (cid:88) i ∈ V ( π ( κ ( t ) i ) i − π ⋆i,τ )
⊤ A i ( π ( κ ( t ) i ) − π ⋆τ ) − (cid:16)− 28 ηd max ∥
A ∥ ∞ ( γ + 1) 5 / 2 (cid:17)
T (cid:88) t =2 γ + ψ ( t ) + 14 ηd max ∥
A ∥ ∞ ( γ + 1) 5 / 2 (cid:88) l ∈ Γ ψ ( l )
[Equation]
where Γ = { 1 , · · · , 2 γ } ∪ { T + 1 , · · · , T + γ } .
The last step results from the choice of learning rate η ≤ 1 28 d max ∥
A ∥ ∞ ( γ +1) 5 / 2 .
It now remains to bound the terms (cid:80)
Tt =
2 γ +1 (cid:80) i ∈
[Equation]
( π ( κ
[Equation]
i − π ⋆i,τ )
⊤ A i ( π ( κ ( t ) i ) − π ⋆τ ) , KL (cid:0) π ⋆τ ∥ π (2 γ ) (cid:1) and (cid:80) l ∈ Γ ψ ( l ) .
In view of Lemma 1, we have −
[Equation]
We remark that each ( π ( κ ( t ) i )
i
[Equation]
π
[Equation]
− π ⋆τ ) term will cancel out due to the mapping [Equation] κ ( t ) i being injective when t ≥ γ .
In addition, we have a crude bound
[Equation]
for every i ∈ V [Equation] t ≥ 0 .
Applying the bound to the remaining nγ terms gives
[Equation]
The remaining terms KL (cid:0) π ⋆τ ∥ π (2 γ ) (cid:1) and ψ ( l ) can be bounded with the following lemma, with the proof postponed to Appendix E.9.
Lemma 9.
It holds for all i ∈ V and t ≥ 0 that
[Equation]
In addition
[Equation]
we have
[Equation]
[Equation]
Putting all pieces together
[Equation]
we continue from (84) and show that
[Equation]
l ∈ Γ ψ ( l ) ≤ KL (cid:0) π
⋆i,τ ∥ π (0) i (cid:1) + 8 ηnγd max ∥
A ∥ ∞ + ηγ (cid:16) nd max ∥
A ∥ ∞ (2 γ + 11) + 3 τ (cid:88) i ∈
V log | S i | (cid:17) ≤ KL (cid:0)
π
⋆i,τ ∥ π (0) i (cid:1) + 8 ηn (cid:104) γd max ∥
A ∥ ∞ + γ (cid:16) d max ∥
A ∥ ∞ (2 γ + 11) + 3 τ log S max (cid:17)(cid:105) ≤ KL (cid:0) π ⋆i,τ ∥ π (0) i (cid:1) + n + 24 ητnγ log S max . Bounding the term KL (cid:0) π
⋆ τ ∥ π ( t − γ +1) (cid:1) .
By definition of KL divergence, we have
[Equation]
=
[Equation]
It follows directly from the update rules that
[Equation]
which enables us to control the term (cid:10) π ⋆i,τ
[Equation]
i
[Equation]
In the same vein as Lemma 8, we can bound the term (cid:80)
[Equation]
π
[Equation]
(cid:13)
[Equation]
1 with { ψ ( l )
[Equation]
} , as detailed in the following lemma.
The proof is omitted due to its similarity with that of Lemma
[Equation]
Lemma 10.
When t ≥ 2 γ [Equation] it holds that
[Equation]
Plugging the above lemma into (89), we have
[Equation]
where (i) results from similar arguments in (82) and (ii) invokes Pinsker’s inequality.
Substitution of the above inequality into (88) and summing over i ∈ V leads to
[Equation]
Summing the above inequality over t
[Equation]
[Equation]
and adding
[Equation]
[Equation]
∥ π ⋆i,τ (cid:1) to the both sides,
[Equation]
[Table] [Equation]
Here, (i) invokes the bound established in (84).
We remark that our choice of learning rate
[Equation]
guarantees[-1 − 28(
 + [Equation]-]
[Equation]
This taken together with (85) and Lemma 9 gives
[Equation]
Bounding the QRE gap.
With Lemma 3, we have
[Equation]
Since the mapping t (cid:55)→ ν i ( t ) is injective
[Equation]
we have
[Equation]
i ∈
[Equation]
Combining the above two equalities gives
[Equation]
[Table]
where the last step results from (90).
E P ROOF OF AUXILIARY LEMMAS E. P ROOF OF L EMMA
To prove this lemma, we recall a key observation in Cai et al.
(2016) that allows one to transform a zero-sum polymatrix game G =
{ ( V, E ) , { S i } i ∈ V , { A ij } ( i,j ) ∈ E } into a pairwise constant-sum polymatrix game (cid:101)
G = { ( V, E ) , { S i } i ∈ V , { (cid:101)
A ij } ( i,j ) ∈ E } such that
(1) For every player i ∈ V , it has the same payoff in G and (cid:101) G :
[Equation]
(2) For each pair ( i, j
) ∈ E , i ̸
= j , the two-player game (cid:101) G is constant-sum, i.e., there exist constants α ij =
α ji , such that
[Equation]
holds for all s i ∈ S i , s j ∈ S j
We are now in a place to prove Lemma 1.
Let (cid:101) G be the pairwise constant-sum polymatrix game associated with G after the above payoff preserving transformation.
We have
[Equation]
, where the penultimate line uses (91), and the last line uses the fact that (cid:101) G is also a zero-sum polymatrix game, which satisfies
[Equation]
for any arbitrary s ∈ S .
E. P ROOF OF L EMMA
In view of the update rule (7), we have
[Equation]
for some constant c i .
On the other hand, it follows from the expression of QRE in (4) that
[Equation]
for some constant c ⋆i .
By combining the above two equalities and taking the inner product with π ( t +1)
i − π ⋆i,τ , we have
[Equation]
Summing the above equality over i ∈ V gives
[Equation]
where the last line follows from (cid:80)
i ∈
V (cid:104) ( π ( t +
[Equation]
i
[Equation]
[-= 
-]
due to Lemma 1, as well as that the game is zero-sum.
E. P ROOF OF L EMMA
Recalling the definition
[Equation]
where the inequality holds since max π ′ i ∈ ∆( S i ) u i,τ [Equation] π ′
[Equation]
, π − i
[Equation]
−
[Equation]
π
[Equation]
≥ u
[Equation]
τ
[Equation]
π
[Equation] , π − [Equation] ) − u i,τ ( π ) = 0 for all i ∈ V .
We now proceed to decompose
[Equation]
where the first line follows from (cid:80)
i
∈
{+V ( u i τ ( π ) − τ H ( π i )) = (cid:80) i ∈ V (cid:0) u i,τ ( π ⋆ τ ) − τ H ( π ⋆ i,τ ) (cid:1) = 
+}
by the definition of zero-sum games.
It boils down to control the terms on the RHS of (94).
• To control the first term, by the definition of u i,τ in (5) (see also (3))
it follows that
[Equation]
[Equation]
which each summand can be further bounded by Young’s inequality and Pinsker’s inequality as
[Equation]
Summing the inequality over i, j gives
[Equation]
• Regarding the second term, we have
[Equation]
where the penultimate step follows from (92) and the last step invokes Lemma
[Equation]
Moving to the last term, we have
[Equation]
[Equation]
where the second line follows again from (92).
Plugging (95), (96) and (97) into (94) gives (cid:88)
[Equation]
Taking maximum over π ′ finishes the proof.
E. P ROOF OF L EMMA
Let (cid:101) π
( T ) =
1 T + [Equation] Tt =
π ( t +1) , then (cid:101) π ( T ) ∈ ∆( S ) .
The proof is completed if we can show
[Equation]
where the first inequality holds trivially since Regret i,τ (cid:0) T + 1 (cid:1) ≥ Regret i,τ (cid:0)(cid:101) π ( T ) i , T (cid:1) .
It then boils down to show the second inequality of the above relation.
From the definition of zero-sum polymatrix games, it holds that
[Equation]
[Equation]
In addition, applying Jensen’s inequality gives
[Equation]
Combining the above two relations yields (cid:88) i ∈ V Regret i,
[Equation]
which concludes the proof.
E. P ROOF OF L EMMA
Taking logarithm on the both sides of (7), we have
[Equation]
On the other hand, the definition of QRE in (4) gives
[Equation]
Subtracting the two equalities and taking inner product with π ( t − γ +1)
i − π ⋆i,τ , we get
[Equation]
Summing the above equality over i ∈ V leads to
[Equation]
where the final step results from Lemma 1.
E. P ROOF OF L EMMA
Recall from (71) that
[Equation]
− γ
[Equation]
When t < γ , we have π ( t − γ +1)
i =
π (0) .
It follows that
[Equation]
and that
[Equation]
[Equation]
Therefore
[Equation]
we can bound the term
[Equation]
Plugging the above inequality into (100) leads to
[Equation]
Applying the above inequality recursively to the iterates [Equation] [Equation]
γ −
[Equation]
we arrive at
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
E. P ROOF OF L EMMA
Taking logarithm on the both sides of (77) and (78)
[Equation]
we get
[Equation]
or equivalently
[Equation]
Taking inner product with π ⋆i,τ − π (
[Equation]
) i ,
[Equation]
[Equation] [Equation] By definition of KL divergence, we have
[Equation]
and
[Equation]
[Equation]
Taken together, we get
[Equation]
On the other hand
[Equation]
taking logarithm of (78) and making inner product with π ( κ ( t
[Equation]
i ) i
[Equation]
Following a similar discussion in (19) gives
[Equation]
Plugging the above equation into (102),
[Equation]
Rearranging the terms finishes the proof.
E. P ROOF OF L EMMA
For notational convenience, we set
[Equation]
−
[Equation]
for all i ∈ V, t ≥ 0 .
By triangular inequality, we have ϕ ( t ) i ≥ (cid:13)(cid:13)(cid:13) π ( t )
[Equation]
− π ( t − 1) i (cid:13)(cid:13)(cid:13) 1 .
In addition, we denote by t 1 ∧ t 2 :=
min { t 1 , t 2 } and t 1 ∨ t 2 := max { t 1 , t 2 } .
For 0 < t 1 < t 2 , it holds that (cid:13) (cid:13) π ( κ ( t
[Equation]
i ) j
[Equation]
j
[Equation]
≤ [Equation]
[Equation]
Therefore
[Equation]
we have
[Equation]
Since 0 ∨ ( t − γ ) ≤ κ ( t ) i ≤ t ≤ ν i ( t ) ≤ t + γ for all i ∈ V , t ≥ 0 , the first term can be bounded by
[Equation]
κ
[Equation]
In addition, the mapping k (cid:55)→ ν j ( κ ( k ) i ) is injective when k ≥ γ (cf.
Assumption 2 and 3).
It follows that
[Equation]
Plugging the above inequalities into (105) yields t
[Equation]
ϕ ( l ) j + ( t + 1 − κ ( t ) i )( γ + 1) ϕ ( ν j ( κ ( κ ( t − 1) i ) i )) j + ( γ + 1) t + γ (cid:88) l = t − 2 γ ϕ ( l ) j ≤ 2( γ + 1) t + γ (cid:88) l = t − 2 γ ϕ ( l ) j + ( γ + 1) 2 ϕ ( ν j ( κ ( κ ( t − 1) i ) i ))
j .
Finally, we control the term ϕ ( t ) i with ψ ( t ) i as:
[Equation]
i
[Equation]
[Equation]
[Equation]
− [Equation] (cid:13)(cid:13) [Equation]
η
[Equation]
i
[Equation]
[Equation]
η
[Equation]
[Equation]
where (i) applies Cauchy-Schwarz inequality [Equation] (ii) invokes Pinsker’s inequality and (iii) is due to [Equation]
η [Equation] τ ≤ ( γ + 1) ητ ≤ 1 / 2 .
Combining the above two inequalities finishes the proof.
E. P ROOF OF L EMMA
We start with verifying the claim (86).
Recall that
[Equation]
We introduce the following standard Lemma (see e.g., (Cen et al., 2020, Appendix A.2)), which allows us to bound control KL (cid:0) π i ∥ π ′ i (cid:1) properly:
Lemma 11.
Given π i , π ′ i ∈ ∆( S i ) and w ∈ R | S i | with log π i 1 = log π
′ i + w , we have
[Equation]
Therefore, it suffices to figure out the terms log π ( t ) i −
[Equation]
i ,
[Equation]
π (
[Equation]
π ( t ) [Equation]
− log π ( κ ( t ) i ) i and log π ( κ ( t ) i )
i − log π ( t − 1) i .
• Bounding KL (cid:0) π ( t ) i ∥ π ( t − 1) i (cid:1) and KL (cid:0) π ( t ) i ∥ (cid:101) π ( t ) i (cid:1) .
The following equations follow directly from (77) and (78):
[Equation]
In addition, we have the following bound w.r.t.
the order of (cid:13)(cid:13) log π ( t − 1) i (cid:13)(cid:13) ∞ , which we shall establish momentarily.
[Equation]
This taken together with Lemma 11 yields
[Equation]
• Bounding KL (cid:0)(cid:101) π ( t ) i ∥ π ( κ [Equation] (cid:1) .
When κ ( t ) i ≥ 1 , we recall from (80) that:
[Equation]
κ
[Equation]
which leads to a crude bound
[Equation]
When κ ( t ) i = 0 , we have
[Equation]
which yields
[Equation]
• Bounding
[Equation]
π
[Equation]
Note that log
[Equation]
This yields, by equations (107), (110), (111) and associated bounds,
[Equation]
Putting all pieces together
[Equation]
we conclude that
[Equation]
It remains to prove the claim (87):
[Equation]
where the third step results from log π (2 γ )
i
[Equation]
and Lemma
[Equation]
Proof of the claim (108) .
First, we prove by induction that for any k, l ∈ S i ,
[Equation]
Note that the claim trivially holds for t = 0 with the uniform initialization π (0)
i
[Equation]
| [Equation] Assume that (112) holds for all t ′ ≤ t − 1 .
Note that log π ( t )
i
[Equation]
we have
[Equation]
[Equation]
where the second line follows from the induction hypothesis (112).
This completes the induction at the t -th iteration.
It follows that for all i ∈ V and t ≥ 0 ,
[Equation]