Clairvoyant Regret Minimization: Equivalence with Nemirovski’s Conceptual Prox Method and Extension to General Convex Games
Abstract
A recent paper by Piliouras et al.
[23, 24] introduces an uncoupled learning algorithm for normal-form games—called Clairvoyant MWU (CMWU).
In this paper we show that CMWU is equivalent to the conceptual prox method described by Nemirovski [22].
This connection immediately shows that it is possible to extend the CMWU algorithm to any convex game, a question left open by Piliouras et al.
We call the resulting algorithm—again equivalent to the conceptual prox method— Clairvoyant OMD .
At the same time, we show that our analysis yields an improved regret bound compared to the original bound by Piliouras et al.
, in that the regret of CMWU scales only with the square root of the number of players, rather than the number of players themselves.
1. Introduction
A recent line of work has focused on identifying learning algorithms such that, when used by all players in a game, each player’s regret grows polylogarithmically in the number of repetitions T , improving over the traditional (and unimprovable) O ( √ T ) bounds 1 of no-regret algorithms for the more adversarial setting in which no assumption about the algorithm used by other agents is made (see, e g. , [1, 7–10, 13, 14, 16, 25, 27]).
By leveraging well-known connections between regret and equilibria in games ( e g. , [15, 17, 18, 26]), such learning algorithms can then be used as computational approaches to equilibrium finding, leading to ˜ O
(1 /T ) convergence to coarse correlated equilibria.
This reduction from no-regret learning to equilibrium computation is largely the preferred approach in practice [2, 4, 5, 12].
In a recent work, Piliouras et al.
[23] depart from this no-regret learning perspective, by introducing a new algorithm that they call Clairvoyant MWU (CMWU) for normal-form games.
CMWU is a variant of the popular multiplicative weights updates (MWU) algorithm, where action probabilities are scaled exponentially according to their payoff at each time step.
In CMWU this scaling is assumed to be done at time t with respect to a tight approximation of the payoff that the player will see at that same time t (hence the adjective clairvoyant ).
In CMWU, the iterates generated by the algorithm are not known to yield regret bounds, and as a decentralized protocol the CMWU dynamics require the players to coordinate on repeatedly computing a fixed point over a sequence of iterations.
However, the CMWU dynamics, whether centralized or decentralized, are attractive as a method for computing a coarse correlated equilibrium (CCE).
[Table]
In particular, they yield a very competitive O (log T/T ) rate of convergence, while requiring only a single gradient computation as well as a linear-time closed-form strategy update at every iteration.
In this paper, we show that the CMWU algorithm can be viewed as an instantiation of the conceptual prox method (CPM), which has been studied extensively in the first-order methods literature [6, 20, 22].
Nemirovski [22] discusses the CPM as a conceptual algorithm that achieves an O (1 /T ) rate of convergence to a solution to a monotone variational inequality (VI).
Monotone VI generalizes for example the problem of computing a two-player zero-sum Nash equilibrium.
He labels it a conceptual method because each step of the algorithm requires solving a non-trivial variational inequality, which means that it is not clear that the algorithm is implementable.
He then shows, however, that an approximate solution to this VI can be computed in logarithmic time in the required precision, because the VI corresponds to a solution to a fixed point of a mapping which is a contraction.
Finally, he goes on to show that in fact one can get the same rate by performing only two steps of the contraction.
This results in the famous mirror prox algorithm.
What we show is that the CMWU algorithm can be viewed as a specialization of CPM to the setting where the feasible set is the Cartesian product of the player’s strategy spaces, and the operator used in the variational inequality is the gradient operator for each player.
However, here one must depart from the CPM perspective of Nemirovski, because his paper focuses on monotone operators.
This is because his method ultimately requires computing the average of the CPM iterates, in order to achieve convergence.
For that reason, CMWU, and our generalization, is not strictly covered by Nemirovski’s results.
The key insight is to realize that if we are only interested in the regret of the players, then no averaging is needed, in which case one can show that the CPM method achieves constant regret, even in the case of a non-monotone operator, as is the case for the operator associated to general-sum games.
Using our perspective on CPM as a method for computing a sequence of strategies with low regret, we show that it is possible to generalize the CMWU method, which only applies to normal-form games, to games with arbitrary convex and compact decision sets, and utility functions that are concave with bounded gradients that are also Lipschitz continuous.
This answers an open problem of Piliouras et al.
[23], where they ask whether CMWU could be generalized to exactly such a setting.
We call the resulting algorithm
Clairvoyant Online Mirror Descent (COMD).
We stress that while this particular result for the algorithm is new, the algorithm is really an instantiation of the CPM of Nemirovski [22].
By appealing to the contraction argument of [22], we also show that it is possible to improve the regret bound of CMWU in the case of normal-form games: we improve the dependence on the number of players n from being linear to only √ n .
Finally, we go on to develop concrete bounds for the COMD method in the case where only a finite number of steps of the contraction are performed.
We show that in this case one can perform O (log t ) steps of the contraction at iteration t , while retaining the guarantee of constant regret.
An immediate consequence of our result is that by instantiating this approximate variant of COMD with a dilated entropy regularizer [12, 19, 21], we get the first algorithm that has an O (log T/T ) rate of convergence to a normal-form coarse correlated equilibrium for extensive-form games, while requiring a linear-time update at every iteration.
2. Setting 2.1. Convex Games and Variational Inequality
We let [[ n ]] := { 1 , 2 , . . . , n } be a set of players, with n ∈ N := { 1 , 2 , . . . } .
In this paper, we operate on convex games , whereby each player i ∈ [[ n ]] has a nonempty convex and compact set of strategies
X i ⊆
R d i .
For a joint strategy profile x = ( x 1 , . . .
, x n ) ∈ × nj =1 X j , the reward of player i is given by a differentiable concave utility function u i : × n j =1 X j → R , subject to the following standard assumptions:
(Concavity) u
i ( x i , x − i ) is concave in x i for any x − i =
( x 1 , . .
, x i − 1 , x i +1 , . . .
, x n ) ∈ × j ̸
= i X j ; 2.
(Bounded gradients) for any ( x 1 , . . .
, x n ) ∈ × nj =1 X j , ∇ x i u i ( x 1 , . . .
, x n ) is bounded, 3.
( L -smoothness)
The gradient ∇ x i u i is Lipschitz-continuous.
In the rest of the paper, we will often find it beneficial to view strategy updates in the game as global , that is, operating on all players at the same time rather than each player individually.
For that reason, we now introduce notation to operate on the Cartesian product of all strategy spaces.
First, we denote the sum of dimensions of the strategy spaces of the players with the letter d := d 1 + · · · + d n .
The joint strategy space of the game is Z :=
X 1 ×· · ·×X n , and we will consistently denote elements in Z using the letter z or variants thereof.
Given a vector z ∈ R d , we will denote as z
i ∈ R d i the portion of the vector belonging to player i , that is, we let ( z 1 , . . .
, z n ) ∈
R d 1 × · · · × R d n be the (unique) vectors such that z = ( z 1 , . . .
, z n ) .
In this global notation over the game, a key quantity associated with the game is the vector-valued function mapping strategies to payoff gradients for all players, that is,
[Equation]
When viewed through the global lenses of the function F , the properties listed above imply the following:
Assumption 1 For an appropriate primal-dual norm pair ( ∥ · ∥ , ∥ · ∥ ∗ ) over R d , the game operator F :
Z → R d satisfies:
• (Boundedness) ∥
F ( z ) ∥ ∗ ≤ B for all z ∈ Z .
• (Lipschitz continuity)
There exists L > 0 such that ∥ F ( z ) − F ( z ′ )
∥ ∗ ≤ L ∥ z − z ′ ∥ .
The variational inequality associated with the operator F , that is, the problem of finding z ∈ Z such that
[Equation]
is exactly equivalent to the problem of computing a Nash equilibrium of the game (see, e.g. , [11, Proposition 1.4.2]).
We remark that generally F is not a monotone operator, that is, there might exist z ,
z ′ ∈
Z such that [Equation] ) , z − z ′ ⟩ ̸≥ 0 .
Nonetheless, in this paper we will be concerned with applying the conceptual prox method, which was designed for monotone operators, to (1).
2.2. Proximal Setup
For each player i ∈ [[ n ]] , we assume that a strongly convex regularizer φ i : Z → R has been chosen.
Each regularizer φ i induces a generalized notion of distance—called Bregman divergence —over X i , defined as
[Equation]
We combine the regularizer φ i for each player’s strategy space into a global , composite regularizer
[Equation]
→ R , [Equation] Correspondingly, the Bregman divergence induced by φ is the function D ( · ∥ · ) : Z × Z → R ≥ 0 ,
[Equation]
As long as each φ i is strongly convex, then so is φ .
Specifically, in the rest of the paper we operate under the following assumption.
Assumption 2 The per-player regularizers φ
i are chosen so that the global regularizer φ : Z → R ≥ 0 is 1 -strongly convex with respect to the norm ∥ · ∥ : Z → R for which Assumption 1 holds.
With that, we are able to define the prox operator , which we define for the global space Z and global regularizer φ .
Given a center z ∈ Z and a gradient g ∈ R d , the prox operator Π z ( g ) generalizes the notion of gradient step away from z in the direction of − g , and is defined as follows.
Definition 1 (Prox operator)
The prox operator associated with φ is defined as
[Equation]
for any center z ∈ Z and a vector g ∈ R d .
We recall standard properties of the prox operator in Appendix B.
3. Conceptual Prox Method
The key observation underpinning the conceptual prox method is the following straightforward lemma (all proofs are in the Appendix).
Lemma 2 Let t ∈ N and z ( t − 1) ∈
Z be arbitrary.
If the point z ( t ) satisfies the fixed point equation z ( t ) =
Π z ( t − 1) (cid:0) ηF ( z ( t ) ) (cid:1) , then
[Equation]
and in particular, for all players i ,
[Equation]
By noting that the right-hand side of (4) is telescopic, Theorem 2 immediately implies the following.
Corollary 3 (Constant per-player regret) Let z (
) ∈
Z be arbitrary, and suppose that recursively z ( t ) ∈
Z at all times t ∈ N satisfies the following fixed-point equation
[Equation]
Then, at all times T ∈ N , the per-player regret is upper bounded as
[Equation]
The algorithm defined in Theorem 3 is called the conceptual prox method (CPM) (see also Nemirovski [22]).
Theorem 3 shows that the per-player regret cumulated up to any time T by the fixed-point iterates z ( t ) =
Π z ( t − 1) (cid:0) F ( z ( t ) ) (cid:1) produced by the CPM is bounded by the range of the divergence
D
i ( · ∥ z (0) i ) , a quantity independent of time.
For example, when X i = ∆ m is the m -simplex, z
(0) i is the uniform strategy, and φ i is negative entropy, then max x ∈X
i D i ( x ∥ z (0) i )
= log m .
Existence and Computation of Fixed-Point Solutions
At this stage, it is perhaps unclear why the fixed points ( ⋆ ) exist and how one can compute them.
The key lies in the following observation, which dates back to at least the work of Nemirovski [22]:
Observation 1 At all times t the map w (cid:55)→ Π z ( t − 1) ( ηF ( w )) is ηL -Lipschitz continuous.
Hence, as long as η < 1 /L , the above function is a contraction, and the fixed point is therefore unique.
Consequently, convergence to an ϵ -fixed point can be achieved via a number of iterations that scales proportially to log(1 /ϵ ) .
This is straightforward: the proximal operator itself is 1 -Lipschitz continuous (Theorem 7), and ηF is ηL -Lipschitz continuous, so their composition is ηL -Lipschitz continuous.
So, at least approximate fixed-point solutions are easy to compute.
In the next section we quantify the error introduced by the approximation in the fixed-point solution.
4. Conceptual Prox Method with Approximate Fixed Points
A more refined analysis of the argument employed in Theorem 3 takes into account error in the computation of the fixed-point solutions ( ⋆ ).
We start by relaxing Theorem 2.
Lemma 4 Let t ∈ N and z ( t − 1) ∈
Z be arbitrary.
Let w ( t ) ∈
Z be an approximate fixed point, in the sense that for some ϵ ( t ) ≥ 0 ,
[Equation]
Then, the point z ( t ) :
=
Π z ( t − 1) (cid:0) ηF ( w ( t ) ) (cid:1) satisfies
[Equation]
[Equation]
and in particular, for all players i , η ⟨∇ x i u i ( w ( t ) ) , ˆ z i − w ( t ) i ⟩ ≤ − D i ( ˆ z i ∥ z ( t ) i )
+ D i
( ˆ z i ∥ z ( t − 1) i ) − D i ( z ( t ) i ∥ z ( t − 1) i ) +
ηBϵ ( t ) ∀ ˆ z i ∈
X i .
Repeating the analysis we already carried out for Theorem 3, this time using Theorem 4, we obtain the following.
Corollary 5 Let z (
) ∈
Z be arbitrary, and recursively let z ( t ) , w ( t ) ∈
Z at all times t ∈ N be such that
[Equation]
Then, at all times T ∈ N , the per-player regret associated with iterates w ( t ) is upper bounded as
[Equation]
Observation 2 When the choice ϵ ( t ) = 1 /t 2 is used, then the sum of errors (cid:80) Tt =1 ϵ ( t ) is an additive constant bounded by 2 , and therefore does not affect the constant per-player regret guarantees, while at the same time requiring O (log t ) fixed-point iterations per iteration of the learning algorithm.
4.1. Centralized Implementation
By combining Theorem 5 together with the concrete choice of errors ϵ ( t ) given in Observation 2 and fixedpoint iterations on the map Π z ( t − 1) ◦ ηF , which is a contraction for η ≤ 1 / (2 L ) (Observation 1), we obtain
Algorithm 1, whose properties summarized in Theorem 6 follow directly from the preceding discussion.[-Theorem 
-]
Data: z (
) ∈
Z initial point, 0 < η ≤ 1 / (2 L ) learning rate, ϵ ( t ) desired fixed-point approximation error 1 for t = 1 , 2 , . . .
do w ( t ) ← z
( t − 1) while (cid:13)(cid:13)(cid:13) w ( t ) − Π z ( t − 1) (cid:16) ηF ( w ( t ) )
(cid:17)(cid:13)(cid:13)(cid:13) >
ϵ ( t ) do w ( t ) ← Π z ( t − 1) (cid:0) ηF ( w ( t ) )
(cid:1)
[ ▷ Fixed-point iteration]
z ( t ) ← Π z ( t − 1) (cid:0) ηF ( w ( t ) ) (cid:1)
{+Theorem 
+}
At all times t = 1 , 2 , . . .
in Algorithm 1, 1.
The internal while loop runs for at most log 2 (max z , z ′ ∈Z ∥
z − z ′ ∥ )
+ log 2 1 ϵ ( t ) iterations. The iterates z ( t ) produced by the algorithm achieve regret upper bounded by
[Equation]
for each player i .
Correspondingly, the average product distribution of play 1 t (cid:80) tτ
=1 z ( τ ) 1 ⊗· · ·⊗ z ( τ ) n is a κ -coarse correlated equilibrium of the game, with
[Equation]
Once again, we remark that the choice ϵ ( t )
= 1 /t 2 for all t is natural, and leads to constant per-player regret, as well as convergence to a coarse correlated equilibrium of the convex game at the rate 1 /t .
Alternatively, if the total number of iterations T was known in advance, the choice ϵ ( t )
= 1 /T would lead to a similar result.
We refer to Algorithm 1 as a centralized implementation because it operates directly on the product space Z .
We believe that this is the natural setting in which to analyze the algorithm.
However, as we lay out in the next section, the individual steps that make Algorithm 1 can be expanded to have a learning dynamic flavor for each player, with some important caveats.
4.2. Decentralized Implementation: Clairvoyant OMD
In this section we show that Algorithm 1 can be implemented in the form of a decentralized learning algorithm where each player independently updates their strategy upon observing the gradient of their utility.
The key is in the observation that, given the definition of φ
= φ 1 + · · · + φ n , the proximal operator on
Line 4 decomposes as
[Equation]
which corresponds to an OMD update (with linearized losses) for each player.
Then, by fixing the number of fixed-point iterations (that is, repetitions of the while loop) to the quantity
[Equation]
it becomes guaranteed that the approximation error of the approximate fixed-point generated after N ( t ) −iterations is less than ϵ ( t ) , and thus we can rewrite Algorithm 1 as in Algorithm 2.[-: z (0) i desired fixed-point approximation error 1 t ′ ←
 w ( t ′ ) ← z (0) 3 for t = 1 , 2 , . . .-]
As Algorithm 2 is amounts to an alternative implementation of Theorem 6, the guarantees of Theoremapply to Algorithm 2 as well.
We also remark that in the latter implementation, each player i ∈ [[ n ]] crucially updates their approximate fixed-point strategy w ( t ′ )
i using as gradient vector the gradient of their utility evaluated in the strategy profile ( w ( t ′ − 1), . .
, w ( t ′ − 1) n ) .
This is perfectly consistent with the framework of learning in games, where each player updates their strategy based on their gradient at the current strategy profile.
One caveat with this implementation is that only the iterates z ( t ) (and not the approximate fixed-point iterates w ( t ′ ) ) are guaranteed to cumulate low regret.
In other words, in the learning-in-games interpretation, only the subsequence of iterates { w ( N (1) ) , w ( N (1) + N (2) ) , . . . } is guaranteed to have low regret.
Therefore, when interpreted as a learning algorithm, Algorithm 2 provides uncoupled learning dynamics, but is not a noregret algorithm in the classic sense.
Nonetheless, a predictable subsequence of iterates guarantees constant regret, and therefore the algorithm can be used to extract an approximate coarse correlated equilibrium.
The name Clairvoyant OMD was chosen to explicitly point out that the algorithm is a generalization, to general convex games, of the Clairvoyant MWU algorithm recently introduced by Piliouras et al.
[23] in the special case of normal-form games and for certain specific choices of regularizers—see also Appendix C.
We remark also that while Clairvoyant MWU was originally introduced as a centralized algorithm, the latter decentralized interpretation was preferred in the later revision of the paper by the same authors [24]. Data: z (0) i ∈ X i
{+0 < η desired fixed-point approximation error 1 t ′ ←
+}
w ( t ′ ) ← z (0) 3 for t = 1 , 2 , . . . do [ ▷ Begin while Algorithm 1] for k = 1 , . .
. , N ( t ) do for each player i ∈
[[ n ]] , in parallel do t ′ ← t ′ + w ( t ′ )
i ← arg min x
i ∈X i (cid:110) − η ⟨∇ x i u i ( w ( t ′ − 1), . .
, w ( t ′ − 1) n ) , x i ⟩ + D i ( x i ∥
z ( t − 1) i ) (cid:111) [
End
] z ( t ) ← w ( t ′ )
In Appendix C we discuss how the previous discussion informs an improved analysis of the Clairvoyant
MWU algorithm for normal-form games.
Appendix A. Example: Normal-Form Games
A notable class of convex games are normal-form games.
In normal-form games, each player i ∈ [[ n ]] has a finite set of actions of cardinality d i .
For any possible combination of actions of the players, each player receives a utility, which we assume to be in the range [ − V, V ] .
Players are free to select any distribution over their actions as their strategy, that is, a point in the probability simplex
[Equation]
Each player’s utility as a function of the choice of distributions is the expected utility corresponding to actions sampled for those distributions.
We use the following primal-dual norm pair ( ∥ · ∥ ∆ , ∥ · ∥ ∗ ∆ ) on the Cartesian product space R d
[Equation]
For this choice of norms, it is immediate to check that the operator F satisfies Assumption 1 for the choice
B := √ n V and L := √ n V .
A standard choice of regularization for normal-form games is negative entropy , that is, the regularizer
[Equation]
for all player i ∈ [[ n ]] .
Negative entropy is 1 -strongly convex with respect to the ℓ 1 norm, and therefore the composite regularizer φ = φ 1 + · · · + φ n is 1 -strongly convex with respect to the norm ∥ · ∥ ∆ introduced above, in compliance with Assumption 2.
Furthermore, we remark that for any player i , the uniform strategy c
i : =
(1 /d i , . . .
, 1 /d i ) satisfies
[Equation]
Appendix B. Known Properties of Prox Operators
The following properties of proximal operators are standard in the literature (see, e.g. , Nemirovski [22]).
Lemma 7 (The prox operator is Lipschitz continuous)
Given any center z ∈ Z , the prox operator is
Lipschitz continuous with constant 1 :
[Equation]
We remark that the above inequality uses the primal-dual norm pair for which φ is 1 -strongly convex, which is the same primal-dual norm pair for which Assumption 1 holds, as per Assumption 2.
Lemma 8 Π z ( g ) =
z ∗ if and only if ⟨ g − ∇ φ ( z ) + ∇ φ ( z ∗ ) , ˆ z − z ∗ ⟩ ≥
for all ˆ z ∈ Z .
Theorem 8 simply states the (necessary and sufficient) first-order optimality condition for the prox operator problem defined in (2).
It immediately implies the following.
Corollary 9 For all z ∈ Z and g , one has
[Equation]
In Appendix A we discuss the proximal setup in normal-form games, an important special case of convex games.
Appendix C. The Special Case of Clairvoyant MWU
In the special case of normal-form games where each player’s simplex strategy space ∆ d i has been equipped with the negative entropy regularizer (Appendix A), the OMD-like update step on Line 7 of Algorithm 2 has the closed-form solution
[Equation]
By plugging the above closed formula into Algorithm 1 and Algorithm 2, we recover the centralized and decentralized versions of the Clairvoyant MWU algorithm introduced by Piliouras et al.
As already mentioned in Appendix A, in that setting the game operator F is upper bounded (with respect to the dual norm ∥ · ∥ ∗ ∆ ) by B and is L -Lipschitz continuous with respect to the ( ∥ · ∥ ∆ , ∥ · ∥ ∗ ∆ ) norm pair, where B = L = √ n V and V is the maximum absolute utility for any player in the game.
Furthermore, negative entropy is 1 -strongly convex with respect to ∥ · ∥ ∆ .
Hence, by choosing
[Equation]
from Theorem 6 we recover a per-player regret of
[Equation]
for the iterates z (1) , . .
, z ( T ) produced by Algorithms 1 and 2.
The number of intermediate iterates w ( t ′ ) produced by Algorithm 2 in this case is
[Equation]
where the last step uses the fact that
[Equation]
This shows that from O ( T log T + T log n ) iterates of w ( t ′ )
it is possible to extract a subsequence of T iterates (those corresponding to w ( N (1) ) , w ( N (1) + N (2) ) , and so on), that cumulate O ( √ nV log( d i )) regret.
This regret bound refines that of Piliouras et al.
[24] in the dependence on the number of players ( √ n rather than n ).
Appendix D. Proofs
Proof of Theorem 9 Expanding the definition of the Bregman divergence, the statement is equivalent to
[Equation]
which in turn is equivalent to
[Equation]
Applying Theorem 8 yields the statement.
Proof of Theorem
The statement follows immediately from summing (4) for t = 1 , . . .
, T , and noticing that the terms
− D i ( ˆ z i ∥
[Equation]
( t )
i
[Equation]
D i
[Equation]
ˆ
[Equation]
i ∥
[Equation]
) telescope:
[Equation]
for all ˆ z i ∈
X i .
Dividing by η and taking a maximum over ˆ z i ∈
X i yields first inequality in the statement.
The second inequality follows immediately by using the fact that divergences are always nonnegative.
Proof of Theorem 4 The proof of the second part of the statement is identical to that of Theorem 2.
Hence, we focus on proving (6).
Again, we start from Theorem 9, this time applied with g = ηF ( w ( t ) ) and
z = z ( t − 1) :
[Equation]
where the second inequality follows from the definition of the dual norm, and the last inequality from using the definition of B , introduced in Assumption 1, and (5).
Rearranging yields (6).
Proof of Theorem 2 Take any ˆ z ∈ Z and use Theorem 9 with g = ηF ( z ( t ) ) and z =
z ( t − 1) :
[Equation]
which is exactly (3).
Since the inequality holds for any ˆ z ∈ Z , it holds in particular for any vector of the form ˆ z =
( z ( t ) 1 , . . .
, z ( t ) i − 1 ,
ˆ z i , z ( t ) i +1 , . . .
, z ( t ) n ) ∈ Z .
Substituting this particular choice into (3) and expanding the definitions yields (4).