Uncoupled Learning Dynamics with O (log T ) Swap Regret in Multiplayer Games
Abstract
In this paper we establish efficient and uncoupled learning dynamics so that, when employed by all players in a general-sum multiplayer game, the swap regret of each player after T repetitions of the game is bounded by O (log T ) , improving over the prior best bounds of O (log 4 ( T )) .
At the same time, we guarantee optimal
O ( √ T ) swap regret in the adversarial regime as well.
To obtain these results, our primary contribution is to show that when all players follow our dynamics with a time-invariant learning rate, the second-order path lengths of the dynamics up to time T are bounded by O (log T ) , a fundamental property which could have further implications beyond near-optimally bounding the (swap) regret.
Our proposed learning dynamics combine in a novel way optimistic regularized learning with the use of self-concordant barriers .
Further, our analysis is remarkably simple, bypassing the cumbersome framework of higher-order smoothness recently developed by Daskalakis, Fishelson, and Golowich (NeurIPS’21).
Introduction
Online learning and game theory share an intricately connected history tracing back to the inception of the modern no-regret framework with Robinson’s analysis of fictitious play [Robinson, 1951] and Blackwell’s approachability theorem [Blackwell, 1956].
Indeed, the no-regret framework addresses the fundamental question of how independent and decentralized agents can “learn” with only limited feedback from their environment, and has led to celebrated connections with gametheoretic equilibrium concepts [Hart and Mas-Colell, 2000, Foster and Vohra, 1997].
One of the remarkable features of these results is that the learning dynamics are fully uncoupled [Hart and
Mas-Colell, 2000]: each player is completely agnostic to the utilities of the other players.
Thus, there is no communication between the players or any centralized authority dictating behavior throughout the game.
Instead, the only “coordination device” is the common history of play.
An additional desideratum, which is fundamentally tied to the no-regret framework, is what Daskalakis et al.
[2011] refer to as strong uncoupledness : 1 players have no information whatsoever about the game (even their own utilities), and they only make decisions based on the utilities received as feedback throughout the repeated game.
In this context, it is well-known that there are broad families of no-regret learning algorithms that, after T repetitions, guarantee regret bounded by O ( √ T ) , and this bound is known to be insuperable in adversarial environments [Cesa-Bianchi and Lugosi, 2006].
However, this begs the question:
What if the player is not facing adversarial utilities, but instead is competing with other learning agents in a repeated game?
This question was first formulated and addressed by Daskalakis et al.
[2011], who devised strongly uncoupled dynamics converging with a near-optimal rate of O ( log TT ) in zero-sum games, a substantial improvement over the O (1 / √ T ) rate obtained via traditional approaches within the no-regret framework.
Thereafter, there has been a considerable amount of effort in strengthening their result, leading to extensions along several important lines [Rakhlin and
Sridharan, 2013, Syrgkanis et al., 2015, Chen and Peng, 2020, Farina et al., 2019, Daskalakis et al., 2021, Anagnostides et al., 2022a, Wei and Luo, 2018, Foster et al., 2016, Anagnostides et al., 2022b].
In particular, in a recent breakthrough result, Daskalakis et al.
[2021] showed that when all players in a general game employ an optimistic variant of multiplicative weights update (MWU) (henceforth
OMWU ), the external regret of each player grows as O (log 4 ( T )) .
That result was also subsequently extended to the substantially more challenging performance measure of swap regret [Anagnostides et al., 2022a].
Perhaps the main drawback of the latter results is the complexity of the analysis, relying on establishing a refined property for the dynamics they refer to as higher-order smoothness .
Our primary contribution in this paper is to develop a novel and much simpler framework, which furthermore improves the prior state of the art O (log 4 ( T ))
regret bounds to O (log T ) in general multiplayer games.
1.1 Overview of Our Contributions
Before we state our main result, let us first introduce some basic notation.
We assume that each player i ∈ [[ n ]] selects at every iteration t of the repeated game a probability distribution (mixed strategy) over the set of available actions x ( t ) i [-
  (-]∈ ∆( A i ) (see Section 2 for further details).
The following theorem is the primary contribution of our work.
Theorem 1.1 (Precise Statement in Theorem 4.4) .
There exist strongly uncoupled no-swap-regret learning dynamics so that when employed by all players with learning rate η = Θ(1) , the secondorder path lengths of the dynamics up to any time T ∈ N are bounded by O (log T ) ; that is,
[Equation]
We are not aware of even an o ( T ) bound for the second-order path lengths—under a time-invariant learning rate—prior to our work, except for very restricted classes of games such as zero-sum games.
The dynamics of Theorem 1.1 combine: (i) the celebrated no-swap-regret template of Blum and
Mansour [2007]; (ii) the optimistic follow the regularizer leader (OFTRL) algorithm of Syrgkanis et al.
[2015]; and (iii) using a self-concordant barrier as a regularizer.
The latter was introduced in online learning in the seminal work of Abernethy et al.
[2008], where the authors obtained the first near-optimal and efficient online learning algorithm for linear bandit optimization; the way we leverage the log-barrier in the setting of no-regret learning in games is novel, and crucially leverages the local norm induced by the regularizer.
The dynamics of Theorem 1.1 are also efficiently implementable (see Remark 4.7).
The implication of Theorem 1.1 is perhaps surprising in view of the inherent cycling aspect of no-regret learning in general games.
Indeed, it is by now well-understood that any no-regret dynamics
Reference
Algorithm
Swap Regret in Games
Adversarial Swap Regret
Blum and Mansour [2007]
E.g. , BM-MWU —–
O ( (cid:112) m log( m ) T )
Chen and Peng [2020]
BM-OMWU
O ( √ n ( m log( m )) 3 / 4 ) T 1 / 4 ) (cid:101) O ( √ mT )
Anagnostides et al.
[2022a]
BM-OMWU
O ( nm 4 log( m )log 4 ( T ))
This paper
BM-OFTRL-LogBar
O ( nm 5 / 2 log( T ))
O ( (cid:112) m log( m ) T )
The second column indicates the algorithm internally employed by the “master” BM algorithm; our construction uses OFTRL with log-barrier regularization (Section 3).
Further, m is the maximum number of actions available to each player.
We point out that in the adversarial swap regret bound we have suppressed lower order factors in terms of T .
We further remark that the near-optimal internal regret guarantee of Anagnostides et al.
[2022a] in turn implies O ( nm log( m ) log 4 ( T ))
swap regret for each individual player, but is obtained via the algorithm of Stoltz and Lugosi [2005].
will fail to converge—at least for certain games ( e.g. , see [Milionis et al., 2022]).
Nevertheless,
Theorem 1.1 implies that players will change their strategies arbitrarily slowly as the game progresses.
As such, players will observe utilities that exhibit very small variation over time, immediately implying near-optimal swap regret.
Corollary 1.2 (Precise Statement in Corollaries 4.5 and 4.6) .
There exist strongly uncoupled noswap-regret learning dynamics so that when employed by all players, the individual swap regret of each player is bounded by O (log T ) .
At the same time, when faced against adversarial utilities each player guarantees O ( √ T ) swap regret.
Corollary 1.2 improves over the prior best bounds of O (log 4 ( T ))
[Daskalakis et al., 2021, Anagnostides et al., 2022a]; a comparison with prior works regarding the algorithm of Blum and Mansour [2007] is given in Table 1.
In fact, Corollary 1.2 yields, to our knowledge, the first no-regret guarantee in general games for uncoupled methods when players use a time-invariant learning rate, a feature that has been extensively motivated in prior works (see, e.g. , the discussion in [Bailey and
Piliouras, 2019]).
Corollary 1.2 also establishes near-optimality in the adversarial regime as well, a crucial desideratum in this line of work.
Finally, swap regret is a powerful notion of hindsight rationality, trivially subsuming external regret.
In particular, in light of well-established connections (see Theorem 2.3), we obtain the best known rate of convergence of O ( log TT ) to correlated equilibria in general games.
Corollary 1.3.
There exist strongly uncoupled learning dynamics so that, when employed by all players, the average correlated distribution of play after T repetitions of the game is an O ( log TT ) - approximate correlated equilibrium.
From a technical standpoint, our approach is conceptually remarkably simple and direct.
Specifically,
Theorem 1.1 is shown by first establishing the RVU bound—a fundamental property first identified in [Syrgkanis et al., 2015, Definition 3]—for swap regret in Theorem 4.3; the key ingredient is
Lemma 4.2, which crucially leverages the local norm induced by the log-barrier regularizer over the simplex.
Next, Theorem 1.1 follows directly by making a seemingly trivial observation: swap regret is always nonnegative .
A related approach was recently employed in [Anagnostides et al., 2022c] for external regret, but only works for very restricted classes of games such as zero-sum.
As such, we bypasses the cumbersome framework of higher-order smoothness recently introduced by Daskalakis et al.
1.2 Further Related Work
The first accelerated dynamics in general games were established by Syrgkanis et al.
In particular, they identified a broad class of no-regret learning dynamics—satisfying the so-called RVU property—for which the sum of the players’ regrets is O (1) .
On the other hand, they only obtained an O ( T 1 / 4 ) bound for the individual external regret of each player.
This is crucial given that the rate of convergence to coarse correlated equilibria is driven by the maximum of the external regrets.
It is important to note that a bound for the sum of the external regrets does not necessarily translate to a bound for the maximum since external regrets can be negative.
This is in stark contrast to swap regret
(Observation 2.1), a property crucially leveraged in our work.
Furthermore, the O ( T 1 / 4 ) bounds for the individual external regret in [Syrgkanis et al., 2015] were only recently extended to swap regret by Chen and Peng [2020].
The main challenge with swap regret—which is also the main focus of our paper—is that the underlying dynamics are much more complex, maintaining and aggregating over multiple independent external regret minimizers.
In addition, the dynamics involve a fixed point operation—namely, the stationary distribution of a Markov chain—posing new challenges compared to the analysis of no-external-regret algorithms [Chen and Peng, 2020].
Finally, a very intriguing approach for obtaining near-optimal no-external-regret dynamics was recently introduced by Piliouras et al.
The main caveat of that result is that the dynamics they propose are not uncoupled, which has been a central desideratum in the line of work on no-regret learning in games.
For this reason, the result in [Piliouras et al., 2021] is not directly comparable with the previous approaches.
Preliminaries
In this section we introduce the basic background on online optimization and learning in games.
For a comprehensive treatment on the subject we refer the interested reader to the excellent book of Cesa-Bianchi and Lugosi [2006].
Conventions We denote by N = { 1 , 2 , . . .
} the set of natural numbers.
We use the shorthand notation [[ n ]] := { 1 , 2 , . . . , n } .
Subscripts are typically used to indicate the player, or a parameter uniquely associated with a player (such as an action available to the player).
On the other hand, superscripts are reserved almost exclusively for the (discrete) time index, which is represented via the variable t .
Also, the r -th coordinate of a d -dimensional vector x ∈ R d is denoted by x [ r ] .
Finally, we let log( · ) be the natural logarithm.
2.1 Online Learning and Phi-Regret
Let X ⊆ R d be a nonempty convex and compact set of strategies, for some d ∈ N .
In the online learning framework the learner has to select at every iteration t ∈ N a strategy x ( t ) ∈ X .
Then, the environment—be it the “nature” or some “adversary”—returns a (linear) utility function u ( t ) :
X ∋ x (cid:55)→ ⟨ x , u ( t ) ⟩ , for some utility vector u ( t ) ∈ R d , so that the learner receives a utility of ⟨ x ( t ) , u ( t ) ⟩ at time t .
In the full information model the learner receives as feedback the entire utility function, represented by u ( t ) .
The canonical measure of performance in online learning is based on the notion of regret , or more generally, on Phi-regret [Greenwald and Jafari, 2003, Stoltz and Lugosi, 2007, Gordon et al., 2008].
Formally, for a set of transformations Φ :
X → X , the Φ -regret of a regret minimization algorithm R up to a time horizon T ∈ N is defined as
[Equation]
Naturally, a broader collection of transformations leads to a stronger notion of hindsight rationality; canonical instantiations of Phi-regret include:
(i) External regret (denoted by Reg ): Φ includes only constant transformations ; (ii) Swap regret (denoted by SwapReg ): Φ includes all possible linear transformations.
As such, swap regret induces the more powerful notion of hindsight rationality.
We point out that our main focus in this paper (Section 4) will be for the special case where X is the probability simplex.
A crucial property of swap regret is that SwapReg ≥ 0 , as formalized below.
Observation 2.1.
Fix any time horizon T ∈ N .
For any sequence of utilities u (1) , . . .
, u ( T ) and any sequence of strategies x (1) , . .
, x ( T ) it holds that SwapReg T ≥ 0 .
In proof, just consider the identity transformation Φ ∋ ϕ :
x (cid:55)→ x in (1).
In contrast, this property does not necessarily hold for external regret.
Moreover, it will be convenient to model a regret minimization algorithm R as a black box which interacts with its environment via the following two subroutines.
(i) R .
N EXT S TRATEGY () :
R returns the next strategy of the learner; (ii) R .
O BSERVE
U TILITY ( u ) : R receives as feedback from the environment a utility vector u , and may adapt its internal state accordingly.
2.2 No-Regret Learning and Correlated Equilibria
A fundamental connection ensures that as long as all players employ no-swap-regret learning dynamics (in the sense that SwapReg T
=
o
( T ) ), the average correlated distribution of play converges to the set of correlated equilibria [Hart and Mas-Colell, 2000, Foster and Vohra, 1997, Blum and Mansour, 2007].
Before we formalize this connection, let us first introduce some basic background on games.
Finite Games Let [[ n ]] := { 1 , 2 , . . . , n } be the set of players, with n ≥ 2 .
In a (finite) game, represented in normal form , each player i ∈ [[ n ]] has a finite set of actions
A i ; for notational simplicity, we will let m i := |A
i | ≥ 2 .
For a given joint action profile a = ( a 1 , . . . , a n ) ∈ ×
n i =
1 A i , the (normalized) utility received by player i is given by some arbitrary function u i : × n i =1 A
i →
[ − 1 , 1] .
Players are allowed to randomize by selecting a (mixed) strategy x i ∈ ∆( A i ) :
= (cid:110) x ∈ R |A
i | ≥ 0 : (cid:80) a i ∈A i x [ a i ] = 1 (cid:111) ; that is, a probability distribution over the available actions.
For a joint strategy profile x
= ( x 1 , . . .
, x n ) , player i receives an expected utility of E a ∼ x
[ u i ( a )] = (cid:80) a ∈A u
i ( a ) (cid:81) j ∈ [[ n ]] x j [ a j ] .
In the problem of no-regret learning in games, every player receives as feedback at time t ∈ N a utility vector u ( t ) i ∈ R |A
i | , so that u
[Equation]
for
[Equation]
here, we used the notation a − i to represent the joint action profile excluding i ’s component, and analogously for the notation x − i .
No other information is available to the player .
We are now ready to introduce the concept of a correlated equilibrium due to Aumann [1974].
Definition 2.2 (Correlated Equilibrium [Aumann, 1974]) .
A probability distribution µ over × n i =1 A i is an ϵ -approximate correlated equilibrium , for ϵ ≥ 0 , if for any player i [-
-]∈
[[ n ]] and any swap function ϕ
i : A i
→ A i ,
[Equation]
Theorem 2.3 (Folklore) .
Suppose that each player i ∈ [[ n ]] employs a no-swap-regret algorithm such that the cumulative swap regret up to time T ∈ N is upper bounded by SwapReg Ti .
Further, let µ ( t ) :
= x ( t ) 1 ⊗ x ( t ) 2 ⊗ · · · ⊗ x ( t ) n be the product distribution at time t ∈ [[ T ]] , and ¯ µ := 1 T (cid:80) Tt =1 µ ( t ) be the average correlated distribution of play up to time T .
Then, ¯ µ is a max ni
=1 { SwapReg Ti /T } - approximate correlated equilibrium.
Consequently, a central challenge for correlated equilibria is that the rate of convergence is driven by the maximum of the swap regrets; this is in contrast to, for example, the rate of convergence of the (utilitarian) social welfare in smooth games , which is driven by the sum of the players’ external regrets [Syrgkanis et al., 2015, Roughgarden, 2015].
Optimistic Learning with Self-Concordant Barriers
Optimistic follow the regularizer leader (OFTRL)
[Syrgkanis et al., 2015] is a predictive variant of the standard FTRL paradigm.
Specifically, OFTRL maintains an internal prediction vector m ( t ) ∈ R d , and can be expressed with the following update rule for t ∈ N .
[Equation]
here, η > 0 serves as the learning rate , and R is the regularizer .
For convenience, we also define x (0) :
= arg min x ∈X R ( x ) .
Unless specified otherwise, (OFTRL) will be instantiated with m ( t ) :
= u ( t − 1) , for t ∈ N .
(For convenience in the analysis, and without any loss, we assume that players initially obtain the utilities corresponding to the other players’ strategies at time t = 0 .)
In [Syrgkanis et al., 2015] the regularizer R was assumed to be 1 -strongly convex with respect to some (static) norm ∥ · ∥ on R d .
On the other hand, we are introducing an important twist: R will be a self-concordant barrier function over X .
In this context, we first extend (in Appendix B) the so-called RVU bound established in [Syrgkanis et al., 2015] under self-concordant regularization.
More precisely, we assume that X has nonempty interior int( X ) .
Further, for u ∈ R d the primal local norm with respect to x ∈ int( X ) is defined as ∥ u ∥
x = (cid:112) u ⊤ ∇ 2 R ( x ) u , while the dual norm is defined as ∥ u ∥ ∗ , x := (cid:112) u ⊤ ( ∇ 2 R ( x )) − 1 u , assuming that R nondegenerate—in the sense
that its
Hessian is positive definite.
Finally, for the purpose of the analysis, we let g ( t ) denote the be the leader sequence (see (BTL) in Appendix B); no attempt was made to optimize universal constants.
Theorem 3.1 ( RVU for Self-Concordant Regularizers) .
Suppose that R is a nondegenerate selfconcordant function for int( X ) .
Moreover, let η > 0 be such that η ∥ u ( t ) − m ( t ) ∥ ∗ , x ( t ) ≤ 12 and η ∥ m ( t ) ∥ ∗ , g ( t − 1) ≤ 12 for all t ∈
[[ T ]] .
Then, the regret Reg T ( x ∗ ) of (OFTRL) with respect to any comparator x ∗ ∈ int( X ) under any sequence of utilities u (1) , . . .
, u ( T ) can be bounded by
R ( x ∗ )
[Equation]
Here, we also used the standard notation Reg T ( x ∗ ) :=
[Equation]
Next, we instantiate
Theorem 3.1 using the log-barrier on the (probability) simplex: R ( x ) = − (cid:80) dr =1 log( x [ r ]) .
While the probability simplex has empty interior, there is a simple transformation on the relative interior relint(∆ d ) that addresses that issue (see Appendix B).
Corollary 3.2 ( RVU for Log-Barrier on the Simplex) .
Suppose that R is the log-barrier on the simplex and η ≤ 116 .
Then, the regret of (OFTRL) under any sequence of utilities u (1) , . .
, u ( T ) can be bounded as
[Equation]
for any x ∗ ∈ relint(∆ d ) , where ∥ x ( t ) − x
[Equation]
We remark that a similar regret bound for optimistic mirror descent [Rakhlin and Sridharan, 2013] under log-barrier regularization was shown by [Wei and Luo, 2018, Theorem 7].
Main Result
In this section we sketch the proof of our main result, namely Theorem 1.1, leading to Corollaries 1. and 1.3; detailed proofs are deferred to Appendix C. In this context, we first employ the general template of Blum and Mansour [2007] for constructing a no-swap-regret minimizer R swap over the simplex.
We proceed with a brief overview of their construction (summarized in Algorithm 1).
In the sequel, we first perform the analysis from the perspective of a single player, without explicitly indicating so in our notation.
The Algorithm of Blum and Mansour Blum and Mansour [2007] construct a “master” regret minimization algorithm R swap by maintaining a separate and independent external regret minimizer
R a for every action a ∈ A .
To compute the next strategy, R swap first obtains the strategy x ( t ) a ∈ ∆( A ) of R a , for every a ∈ A .
Then, a (row) stochastic matrix Q ( t ) ∈ S |A| is constructed, so that the row associated with action a ∈ A is equal to the distribution x ( t ) a , while R swap outputs as the next strategy x
( t ) ∈ ∆( A ) any stationary distribution of Q ( t ) ; that is, ( Q ( t ) ) ⊤ x t ) =
x ( t ) .
Next, upon observing a utility u ( t ) ∈ R |A| , R swap forwards to each individual regret minimizer R a the utility u ( t ) a := u ( t ) x
( t ) [ a ] ∈ R |A| .
This construction is summarized in Algorithm 1.
t ) ← 0 ∈
A|
for do
Q ( t ) [ a, · ] ← R a . N EXT S TRATEGY () x ( t ) ← S TATIONARY D ISTRIBUTION ( Q ( t ) return x ( t ) 7 function O BSERVE U TILITY ( u ( t ) ) for a ∈ A do
R a .
O BSERVE U TILITY ( x ( t ) [ a ] u ( t
Blum and Mansour [2007] showed that this algorithm guarantees no-swap-regret as long as each individual regret minimizer has sublinear external regret ; this is formalized in the theorem below.
Theorem 4.1 (From External to Swap Regret [Blum and Mansour, 2007]) .
Let SwapReg T be the swap regret of R swap and Reg Ta be the external regret of R a , for each a ∈ A , up to time T ∈ N .
Then,
[Equation]
In this context, we will instantiate each individual regret minimizer R a with (OFTRL) under logbarrier regularization—and the same learning rate η > 0 .
We will refer to the resulting algorithm as BM - OFTRL - LogBar .
A central ingredient in our proof of Theorem 1.1 is to establish that the resulting no-swap-regret algorithm R swap will enjoy an RVU bound, as stated in Theorem 4.3.
To this end, we first apply Corollary 3.2 for each individual regret minimizer R a , implying that
SwapReg T = (cid:80) a ∈A Reg Ta (by Theorem 4.1) is upper bounded as
[Equation]
The log T factor derives from the diameter of the log-barrier regularizer (see Theorem A.9), and appears to be unavoidable using our approach.
Now the crux in establishing an RVU bound for R swap is to upper bound the last term in (2) in terms of the “movement” of the stationary distribution.
This is exactly where the local norm induced by the log-barrier turns out to be crucial, leading to the following key technical ingredient.
Lemma 4.2.
Suppose that each regret minimizer R a employs (OFTRL) with log-barrier regularization and η ≤ 116 .
Then, for any t ∈ N ,
[Equation]
Intuitively, this lemma ensures that the “movement” of the stationary distribution is smooth in terms of the “movement” of each row of the transition matrix Q ( t ) .
To show this, we use the Markov chain tree theorem (Theorem C.3), which provides a closed-form combinatorial formula for the stationary distribution of an ergodic Markov chain, along with the fact that the log-barrier regularizer guarantees “multiplicative stability” of the iterates (Corollary C.1).
While similar in spirit results have been documented in the literature for dynamics akin to MWU [Candogan et al., 2013, Chen and
Peng, 2020], our proof of Lemma 4.2 crucially hinges on the local norm induced by the log-barrier regularizer.
Thus, we are now ready to derive an RVU bound for swap regret.
Theorem 4.3 ( RVU Bound for Swap Regret) .
Suppose that each R a employs (OFTRL) with logbarrier regularization and η ≤ 1 128 √ m .
Then, for T ≥ 2 , the swap regret of R swap is bounded as
[Equation]
This theorem follows directly from (2) and Lemma 4.2.
So far we have focused on bounding the swap regret of each player when faced against arbitrary utilities.
Next, we use Theorem 4.3 to establish a new fundamental property when all players employ the dynamics.
Our proof crucially relies on the seemingly insignificant fact that SwapReg Ti ≥ 0 (recall Observation 2.1).
Theorem 4.4 (Log-Bounded Second-Order Path Lengths) .
Suppose that each player i ∈ [[ n ]] employs
BM - OFTRL - LogBar with η =
[Equation]
Then, for T ≥ 2 ,
[Equation]
Proof.
Consider any player i ∈ [[ n ]] .
Given that | u i ( a ) | ≤ 1 , for any a ∈ A (by the normalization assumption), we have that for any t ∈ [[ T ]] , [Equation] i
[Equation]
where we used that the total variation distance between two product distributions is bounded by the sum of the total variations of each individual marginal distribution [Hoeffding and Wolfowitz, 1958].
Thus,
[Equation]
As a result, using Theorem 4.3 we conclude that (cid:80) ni
=1 SwapReg Ti can be upper bounded by 2 log T [Equation] [Equation] ≤
[Equation]
[Equation]
m
[Equation]
η [Equation] (cid:88) t = [Equation]
∥ 21 , since η =
[Equation]
[ n ]] {√ m j } .
But, given that 0 ≤ (cid:80) ni =1 SwapReg Ti , we conclude that
[Equation]
We are not aware of even o ( T ) bounds for the second-order path lengths in prior works (using a time-invariant learning rate), except in very restricted classes of games such as zero-sum and potential games [Anagnostides et al., 2022c].
An example of the implication of Theorem 4.4 in a variant of
Shapley’s game [Shapley, 1964, Daskalakis et al., 2010] is illustrated in Figure 1.
Although the dynamics appear to cycle, and the Nash gap—the maximum of the best response gaps—is always large, the players are changing their (mixed) strategies with gradually diminishing speed; further discussion and experiments are included in Appendix D.
As an immediate consequence, combining Theorem 4.4 with Theorem 4.3 implies near-optimal individual swap regret.
[Figure]
Corollary 4.5 (Near-Optimal Individual Swap Regret) .
Suppose that all players use
BM - OFTRL - LogBar with η =
[Figure]
[Equation]
Then, the individual swap regret SwapReg Ti up to time T ≥ 2 of each player i [-
-]∈
[[ n ]] can be bounded as
[Equation]
We point out that our distributed protocol makes the very mild assumption that each player knows an upper bound on the total number of players and the maximum number of actions in order to appropriately tune the learning rate.
Further, as is the case with the result in [Daskalakis et al., 2021], the individual regret of each player predicted by Corollary 4.5 grows linearly with the number of players.
This can be unsatisfactory in games with a large number of players— i.e. , n ≫ 1 .
For this reason, in Theorem C.4 we refine and improve the guarantee of Corollary 4.5 in games where the utility of each player depends only on a small number of other players, and each player’s actions only affect a small number of others players; no other constraint is imposed on the game.
Understanding whether the linear dependence on n is necessary to obtain near-optimal (swap) regret is left as an interesting question for future work.
Finally, we adapt the learning dynamics so that each player enjoys at the same time near-optimal swap regret in the adversarial regime as well.
Corollary 4.6 (Adversarial Robustness) .
There exist dynamics such that when all players follow them the individual swap regret of each player grows as in Corollary 4.5.
Moreover, when faced against adversarial utilities, such that ∥ u ( t ) i ∥ ∞ ≤ 1 for all t ∈ [[ T ]] , the algorithm guarantees that
[Equation]
Our adaptation is particularly natural: If all players follow the prescribed protocol, Theorem 4. implies that the observed utilities of each player i will be such that (cid:80) tτ =1 ∥ u ( [Equation] i
[Equation]
O (log t ) .
So, if at any time the player identifies that the previous condition was violated, it suffices to switch to a no-swap-regret minimizer (such as BM-MWU) tuned to face advarsarial losses—in which case it is crucial to use a vanishing learning rate η = O (1 / √ T ) .
Remark 4.7 (Numerical Precision) .
As is standard, we assumed that the iterates of (OFTRL) were computed exactly, without taking into account issues relating to numerical precision.
To justify this, one can use Damped Newton’s method in order to determine an ϵ -nearby point to the optimal in
O (log log(1 /ϵ )) iterations [Nemirovski and Todd, 2008].
This would extend all the regret bounds with up to an O ( ϵT ) error.
So, with only O (log log T ) repetitions of Damped Newton’s method (per iteration) the error in the regret bounds becomes O (1) , and all of our guarantees immediately extend; see [Farina et al., 2022, Appendix A.5] for an analogous extension under approximate iterates.
Discussion
Our main contribution in this paper was to establish a fundamental new property characterizing the trajectories of certain uncoupled no-regret learning dynamics, summarized in Theorem 1.1.
This property directly guarantees the best known and near-optimal bound of O (log T ) for the swap regret incurred by each player in a general multiplayer game.
Investigating further consequences of Theorem 1.1 is an interesting direction for the future.
We also believe that our framework could have new implications for learning in games with partial information; e.g. , see [Wei and Luo, 2018].
Another interesting avenue is to extend our scope to more general and combinatorial sets beyond the probability simplex, in order to (efficiently) encompass, for example, games in extensive form .
Further, our no-swap-regret learning dynamics have external regret trivially bounded according to
Corollary 4.5.
Consequently, our construction yields no-external-regret learning dynamics with a more favorable dependence on T compared to [Daskalakis et al., 2021] ( log T compared to the log 4 ( T ) of the latter), but with a worse dependence on the number of actions (polynomial rather than logarithmic).
Our method also has higher per-iteration complexity.
For these reasons, extending the scope of our framework beyond self-concordant regularization is an important direction for future research.
Indeed, we conjecture that OMWU has bounded second-order path lengths, a property that would imply the first uncoupled learning dynamics with bounded regret, but establishing that likely requires new insights.
Acknowledgments
We are thankful to the anonymous NeurIPS reviewers for many helpful comments.
Ioannis Anagnostides is grateful to Dimitris Achlioptas for helpful discussions.
Christian Kroer is supported by the Office of Naval Research Young Investigator Program under grant N00014-22-1-2530.
Haipeng
Luo is supported by the National Science Foundation under grant IIS-1943607 (part of this work was done while he was visiting the Simons Institute for the Theory of Computing). Tuomas Sandholm is supported by the National Science Foundation under grants IIS-1901403 and CCF-1733556.