Beyond Time-Average Convergence: Near-Optimal Uncoupled Online Learning via Clairvoyant Multiplicative Weights Update
Abstract
In this paper we provide a novel and simple algorithm, Clairvoyant Multiplicative
Weights Updates (CMWU), for convergence to Coarse Correlated Equilibria (CCE) in general games.
CMWU effectively corresponds to the standard MWU algorithm
but where all agents, when updating their mixed strategies, use the payoff profiles based on tomorrow’s behavior, i.e. the agents are clairvoyant.
CMWU achieves constant regret of ln( m ) /η
in all normal-form games with m actions and fixed step-sizes η .
Although CMWU encodes in its definition a fixed point computation, which in principle could result in dynamics that are neither computationally efficient nor uncoupled, we show that both of these issues can be largely circumvented.
Specifically, as long as the step-size η is upper bounded by 1 ( n − 1) V , where n is the number of agents and [0 , V ] is the payoff range, then the CMWU updates can be computed linearly fast via a contraction map.
This implementation results in an uncoupled online learning dynamic that admits a O (log T ) -sparse sub-sequence where each agent experiences at most O ( nV log m ) regret.
This implies that the CMWU dynamics converge with rate O ( nV log m log T/T ) to a CCE and improves on the current state-of-the-art convergence rate of uncoupled online learning dynamics [13, 1].
Introduction
The connection between online learning and game theory has been extensively studied for decades.
The emergence of online learning algorithms guaranteeing comparable payoffs with the highestrewarding strategy has provided a landmark method of understanding how selfish agents can act in an adversarial environment, while the notion of Coarse Correlated Equilibrium (CCE) has provided a game-theoretic characterization of the limiting behavior of such online learning dynamics [4, 20, 35, 21].
An important aspect of online learning dynamics is that agents can collectively learn a CCE without having explicit knowledge of the full game description.
By definition, online learning algorithms take as input the sequence of vectors corresponding to the payoff of each possible action, making no assumption about how these payoff vectors are derived [22].
This idea of information exchange can be concisely described with the following distributed protocol, known in the literature as uncoupled online learning dynamics [11]:
No agent is aware of their payoff matrix or the payoff matrix of any other agent.
At each round t , each agent i announces their mixed strategy x ti . 3.
Based on the announced strategy vector x t :=
( x t 1 , . . .
, x tn ) , each agent i learns only their payoff vector
u i ( x t − i ) .
Each agent i uses ( u i ( x 0 − i ) , . .
, u i ( x t − i )) to update their mixed strategy at round t + 1 .
If agents use one of the classical no-regret online learning algorithms (e.g. Hedge, Regret Matching, Multiplicative Weights Update (MWU) etc), then the time-average behavior of the resulting uncoupled online learning dynamics will converge to CCE with rate Θ(1 / √ T )
Despite the fact that Θ(1 / √ T ) has proven to be the optimal time-average regret that an agent can achieve in the adversarial case, the question of determining which update rule gives the fastest convergence to CCE has remained open.
Over the years, a series of works has proposed update rules with better and better rates [9, 12, 24, 33, 37, 10] until the recent seminal work of Daskalakis et.
al [13], which established that Optimistic Multiplicative Weights Update (OMWU) admits O (log 4 T/T ) rate of convergence.
This rate matches (up to logarithmic factors) the lower bound Ω(1 /T ) on the convergence rate of any uncoupled online learning dynamic [11].
Self-Play and Algorithmic Applications
Apart from theoretical interest, uncoupled online learning dynamics admit important applications which make the above line of research even more exciting and significant [35, 15, 25, 28, 39, 32, 27].
Some of their interesting algorithmic properties are: i ) their decentralized nature permits efficient distributed implementation, ii ) the implicit access to the game via payoff vectors permits gameabstractions with vastly reduced size, and iii ) their iterative guarantees can be preferable over the all-or-nothing guarantees of linear programs.
A notable example of their algorithmic success is the design of state-of-the art AI poker programs based on iterative self-play that outperform previous
LP-based approaches, and were able to compete with human professionals [6, 38, 40].
We remark that this result has motivated a parallel line of research studying the convergence rates of uncoupled online learning dynamics for extensive form games [17, 8, 16, 18, 25].
Our Contribution and Results
All previous works in this area couple the goals of minimizing adversarial regret and fast time-average convergence to CCE [9, 12, 24, 33, 15, 37, 10].
Since many of the aforementioned applications are in the realm of self-play where all agents are programmed to follow the update rule, guaranteeing adversarial no-regret is not a necessity.
Indeed, it is not clear why time-average behavior should be the only way to deduce a CCE.
In fact, any simple and efficient deduction rule would serve the algorithmic benefits of uncoupled online learning dynamics [18, 25].
Motivated by the above, our contributions can be summarized as follows:
• We introduce a novel update rule, called Clairvoyant Multiplicative Weights Update (CMWU) that produces sequences of strategy profiles with constant regret in general games.
• In its generic form, CMWU is a centralized update rule and does not fit in the online learning framework.
However, based on CMWU we design an uncoupled online learning dynamic called CMWU dynamics which gives fast convergence to CCE beyond the time-average sense .
• More precisely, we establish that given any trajectory of length T of CMWU dynamics, the log T -sparse sub-trajectory always admits constant regret for all agents.
As a result, the time-average behavior of the O (log T ) -sparse sub-trajectory converges to CCE with Θ (log T/T ) rate, improving on the previous state-of-the-art Θ (cid:0) log 4 T/T (cid:1) achieved by Daskalakis et al.
• The update rule of CMWU dynamics (presented in Algorithm 2) admits a simple form and an efficient implementation (requiring only a single step of the MWU algorithm at each round).
The proof of convergence for CMWU dynamics is also far simpler than previous proofs of convergence of uncoupled online learning dynamics [13].
In Table 1 we summarize the most important results concerning the convergence to CCE of uncoupled online learning dynamics.
Remark 1.1 .
All the results mentioned in Table 1 additionally admit ˜ O ( √ T ) adversarial guarantees.
As mentioned above, this means that once a subset of the agents act adversarially by not following the update rule of the online learning dynamic, the agents following the update rule are guaranteed to experience at most ˜ O ( √ T ) regret.
The reason why CMWU is able to achieve better guarantees with simpler analysis comes from the fact that it neglects the adversarial no-regret guarantees that are Update Rule Deduction Rule [Table] irrelevant in self-play/training settings and focuses on minimizing the regret in specific parts of the sequence.
Remark 1.2 .
In another related work [Table] [1] established an online learning dynamic that converges with rate Θ( n log( m ) log 4 T/T ) to Correlated Equilibria (CE), a subset of Coarse Correlated Equilibria .
Finally, following our work [Table] [2] produced time-average convergence to CE via online learning at a rate Θ( nm 5 / 2 log T/T ) .
In comparison, our dependency on the number of actions m is exponentially smaller at Θ(log( m )) .
Furthermore, their update rule is based on self-concordant regularization and thus admits high per-iteration complexity, a point which is mentioned by the authors. Remark 1.3 .
Similarly to other standard no-regret algorithms, CMWU is designed to compute CCE via an uncoupled online learning dynamic and thus cannot be viewed as a rational behavioral assumption for selfish agents who seek to minimize their individual cost. One concrete example where the assumption fails is the case of auctions with a no-regret buyer [5], where the auctioneer can provably manipulate the behavior of any buyer learning according to a wide class of no-regret algorithms to guarantee themselves optimal revenue. This comes at the expense of higher prices for the buyer, since the auctioneer can take advantage of the commitment of the buyer to a noregret learning strategy.
Hence, it is clear that the learning in games community should discuss and differentiate in greater detail the uncoupled online learning dynamics that come as natural game-play and uncoupled dynamics that come as a means of decentralized, efficient computation .
The Philosophy and Design of CMWU
We introduce a radically different philosophy in the design of online learning algorithms.
We shift away from the prevailing paradigm by defining a novel algorithm that we call Clairvoyant Multiplicative Weights Update (CMWU).
CMWU is MWU equipped with a mental/simulated/synthetic model (jointly shared across all agents) about the state of the system in its next period.
Each agent records its mixed strategy, i.e., its belief about what it expects to play in the next period in this shared mental model, which is internally updated using MWU without any changes to the real-world behavior up until it equilibrates, thus marking its consistency with the next day’s real-world outcome.
It is then and only then that agents take action in the real-world, effectively doing so with “full knowledge” of the state of the system on the next day, i.e., they are clairvoyant.
CMWU acts as MWU with one day look-ahead, achieving bounded regret.
CMWU update rule is closely related with the Proximal Point
Method (PPM) [29, 26, 34, 31] which is an implicit (and therefore unimplementable) method that admits arbitrarily fast convergence in the context of convex minimization.
Similarly to PPM, in order to implement the update rule of CMWU one would require access to the explicit description of the game and to additionally solve a fixed-point problem .
Our main technical contribution consists of establishing that for sufficiently small step-sizes, the update rule of CMWU can be computed via the iterations of a contraction map.
This not only provides a way to compute the CMWU update rule using a simple and efficient process, but also opens a pathway for decentralized computation in the context of uncoupled online learning dynamics .
The basic idea behind CMWU dynamics (Algorithm 2) is to punctuate the history of play with anchor points which are equally spaced in distance Θ(log T ) , such that any anchor point is the CMWU update of the previous anchor point.
In this way, the regret in the anchor sequence is constant for any agent.
Moreover, the intermediate points between two anchor points correspond to the iterations of the contraction map.
Surprisingly enough, all of the above can be implemented with just an if condition and an MWU-esque exponentiation (see Algorithm 2).
Preliminaries & Model 2.1 Normal Form Games
We begin with basic definitions from game theory.
A finite normal-form game Γ ≡ Γ( N , S , u ) consists of a set of players N = { 1 , ..., n } where player i may select from a finite set of actions or pure strategies S i .
Each player has a payoff function u
i : S ≡ (cid:81) i S i
→ R assigning reward u i ( s ) to player i .
It is common to describe u i with a payoff tensor A ( i ) where u i ( s )
=
A ( i ) s .
Let m = max i |S i
| denote the maximum number of pure strategies in Γ , and let V = max i,s | A ( i ) s | denote the maximum payoff value of any strategy in Γ .
Players are also allowed to use mixed strategies x i =
( x is i )
s i ∈S
i ∈ ∆( S i )
≡ X i .
The set of mixed strategy profiles is X
= (cid:81)
i X i .
A strategy is fully mixed if x is i > 0 for all s i ∈ S
i and i ∈ N .
Individuals compute the payoff of a mixed strategy linearly using expectation.
Formally, u i ( x ) = (cid:88) s
∈S u i ( s ) (cid:89) i ∈N x is i .
We also introduce additional notation to express player payouts for brevity in our analysis later.
Let v is i ( x ) =
u i ( s i ; x − i ) 1 denote the reward i receives if i opts to play pure strategy s i when everyone else commits to their strategies described by x .
This results in u i ( x ) = ⟨ v i ( x − i ) , x i ⟩ .
Next we will introduce the notion of Coarse Correlated Equilibria (CCE), which is the key equilibrium notion we explore in our work.
Definition 2.1.
A probability distribution µ over pure strategy profiles s = ( s 1 , . . .
, s n ) ∈ S is called an ϵ -approximate Coarse Correlated Equilibrium if for each agent i ∈ [ n ] ,
[Equation]
Definition 2.2.
Given a strategy profile x :=
( x 1 , . .
, x n ) ∈ X , µ x denotes the product probability distribution over strategy profiles s =
( s 1 , . . .
, s n ) ∈ S induced by x , µ x ( s ) :
= Π s i ∈ s x is i .
2.2 Uncoupled Online Learning Dynamics in Games
We study games from a learning perspective where agents iteratively update their mixed strategies over time based on the performance of pure strategies in prior iterations via an uncoupled, online adaptive algorithm.
We will start by describing one of the most classical online learning algorithms,
Multiplicative Weights Update (MWU).
The update rule for MWU can be written as
[Equation]
The remarkable guarantee of MWU and many other online learning algorithms is that the actual payoff for agents is close to the highest-rewarding action of the game.
This property is formally captured via the notion of regret .
[Equation]
Definition 2.3 (Regret) .
Given a sequence of mixed strategies x 0 , . . .
, x T − 1 the regret of agent i ,
R i ( T ) , is defined as
[Equation]
Once agent i adopts MWU, then they are ensured to experience sub-linear regret, R i ( T ) ≤ O (cid:16) √ T (cid:17) no matter how the other agents update their strategies [9].
This implies that the time-averaged difference between the payoff of the best fixed strategy and the actual produced reward goes to zero with rate O (1 / √ T ) .
Any online learning algorithm that is able to guarantee R i ( T )
=
o ( T ) is called no-regret .
There exists a folklore connection between any no-regret online learning algorithms and Coarse
Correlated Equilibrium.
Theorem 2.4 (Folkore 2 ) .
Given a sequence of mixed strategies ( x 0 , . . .
, x T − 1 ) , then the probability distribution ˆ µ := (cid:80) T − 1 t =0 µ x t /T is an ( R ( T ) /T )
-approximate
Coarse Correlated Equilibrium where
R ( T ) :
= max i ∈
[ n ] R i ( T ) .
Theorem 2.4 implies that if all agents update their strategies with a no-regret online learning algorithm , then the time-average strategy vector converges to CCE with rate o ( T ) /T .
As a result, the timeaverage strategy vector converges to CCE as T → ∞ .
In Algorithm 1, we present a generic description of uncoupled online learning dynamics.
1: for all round t = 0 , · · · , T − 1 do 2:
Each player i ∈ [ n ] , broadcasts its mixed strategy x ti ∈ X i 3:
Each i n ] , learns only its reward vector u
i ( x t − i ) .
Each i n ] , + end for If for example the update rule of MWU is implemented at Step 4 of Algorithm 1, then the distribution ˆ µ := (cid:80) T − 1 t =0 µ x t /T is an (1 / √ T ) -approximate
CCE.
Clairvoyant MWU
In this section, we introduce a novel learning algorithm for games that we call Clairvoyant Multiplicative Weights Updates (CMWU).
Critically, CMWU, unlike MWU, forms self-confirming predictions/beliefs about what all opponents will play in the next time instance.
Namely, all agents will form the same belief about what agent i will play in the next period t + 1 (cid:0) x
t +1 i (cid:1) .
These beliefs/estimates are such that when agents simulate an extra period of play in their mind and update their current strategies using MWU, the resulting strategy for each agent i is x t +1 i .
All agents accurately predict the behavior of all other agents tomorrow , in other words they are clairvoyant !
The update rule for (CMWU) is as follows:
[Equation]
CMWU is an implicit method: the new strategy x t +1 appears on both sides of the equation, and thus the method needs to solve an algebraic/fixed point equation for the unknown x t +1 .
Theorem 3.1.
The algebraic system of equations in (CMWU) defined by an arbitrary game Γ , an arbitrary tuple of learning rates η i , and any state x t , always admits a solution.
Theorem 3.1 follows directly by the Brouwer fixed-point theorem.
Having established the fact that the agents can always collectively compute a next step of the CMWU update rule, we present the remarkable property of CMWU stated in Theorem 3.2.
Theorem 3.2.
Let x 0 , . . .
, x T − 1 be a sequence of mixed strategies such that all pairs of consecutive mixed strategies ( x t , x t +1 ) satisfy Equation (CMWU) .
Then, each agent i has bounded regret ≤ log | S i | /η
i .
The proof of Theorem 3.2 follows by standard arguments in online learning literature (e.g. Lemma 5. in [22]).
Moreover, we directly obtain the following corollary:
Corollary 3.3.
Let x 0 , . . .
, x T − 1 be a sequence of mixed strategies such that all pairs of consecutive mixed strategies ( x t , x t +1 ) satisfy Equation (CMWU) .
Then, the probability distribution ˆ µ := (cid:80) T −t =0 µ x t /T is a (log m/ηT ) -approximate CCE where η := min i ∈ n η i .
From a computational complexity perspective, solving the algebraic/fixed point equation (CMWU) is in general a hard problem.
For instance, the computation of a Nash Equilibrium [30] which has proven to be PPAD -complete
[14], reduces to the computation of a solution for equation (CMWU) once η → ∞ .
Moreover, even if we assume that the agents possess unlimited computational power, it is not clear how they can compute a solution to Equation (CMWU) in the context of uncoupled online learning dynamics .
In Section 3.1, we show that if each η i is upper bounded by some game-dependent parameters, (CMWU) is a contraction map (Theorem 3.6, Corollary 3.7).
This not only implies that there exists a unique fixed-point solution that can be computed very efficiently, but also that the
Clairvoyant MWU update rule can be simulated via an uncoupled online learning dynamic.
3.1 Uniqueness of Fixed Point via Map Contraction
We will establish uniqueness of the fixed point in CMWU for a specific range of step-sizes.
The proof will be based on an application of the Banach fixed-point theorem (mapping theorem or contraction mapping theorem) [3].
Thus, we simultaneously provide a constructive method to compute these fixed points with linear convergence rate.
In what follows, it will be useful to consider (MWU) as a map from a vector of payoffs v i =
( v i 1 , . . .
, v i | S i | ) to mixed strategies parameterized by the current initial position x ti :
[Equation]
We first establish the fact that f x ti ( v i ) is an 2 η i -continuous mapping.
Lemma 3.4.
For any choice of x ti ∈ ∆( S i ) , the ( MWU f ) map f x ti : R | S i | → ∆( S i ) satisfies that for any utility vectors v i , v ′ i ∈ R | S i | ,
[Equation]
In the rest of the section we establish that once all η i are selected sufficiently small then Equation MWU f admits a unique fixed point and is in fact a contraction map.
Definition 3.5.
The distance between the strategy profiles x = ( x 1 , . . .
, x n ) ∈ X and x ′ =
( x ′ 1 , . . . , x ′ n ) ∈ X is defined as D ( x, x ′ ) :
= max 1 ≤ i ≤ n ∥ x
i − x ′ i ∥ 1 .
In Theorem 3.6 we show that the computation of CMWU is a contraction map once all η i do not exceed a game-dependent constant.
Theorem 3.6.
Consider the mixed strategy profile ( x t 1 , x t 2 , . . .
, x tn ) and the map G : X (cid:55)→ X defined as follows:
[Equation]
Then for any x, x ′ ∈ X ,
[Equation]
where η is the maximum step-size over all players, [0 , V ] is the payoff range of the game and n is the number of players.
Proof.
Let us denote by d TV ( x, x ′ )
the total variation distance between product distributions x, x ′ .
D (
[Equation]
[Equation] [Equation]
[Equation]
for some player i ∈ [ n ]
[Equation]
by Lemma (3.4)
[Equation]
properties of [Equation] variation
[Equation]
Corollary 3.7.
The ( MWU f ) map with maximum step-size η <
[Equation] n [Equation] 1) V is a contraction and thus converges to its unique fixed point at a linear rate.
Uncoupled Clairvoyant MWU Online Learning Dynamics
In this section we present an uncoupled online learning dynamic based on the Clairvoyant MWU update rule, which we call CMWU dynamics.
More precisely, the agents follow the distributed protocol described in Algorithm 1 while each agent i runs Algorithm 2 internally to update their strategy x ti at each round.
To simplify notation in Algorithm 2, we set the number of actions of any agent i to be m :=
|S i | .
Algorithm 1: Input: η > 0 , k ∈ N 2: x − 1 i ← (1 /m, . . .
, 1 /m ) and z − 1 i ← (1 /m, . .
, 1 /m ) 3: for each round t = 0 , · · · , T − 1 do 4: if t mod k ==
then 5: x ti ← x t − 1 i 6:
Agent i broadcasts the mixed strategy x ti and then receives the payoff vector
v i ( x t − i ) .
Updates z ti such that for all s i ∈ S i ,
[Equation]
8: else 9: z ti ← z t [Equation] [Equation]
Updates x ti such that for all s i ∈ S i ,
[Equation]
Agent i broadcasts the mixed strategy x ti and then receives the payoff vector
v i ( x t − i ) .
12: end if 13: end for
Theorem 4.1.
Let x 0 , . . .
, x T − 1 be the strategy vector once each agent internally adopts Algorithmwith η = 1 / 2 nV and k = ⌈ log T ⌉ .
Then for each agent i ,
T ′ (cid:88)
[Equation]
where T ′ =
(cid:4) T − 1 k (cid:5) .
Thus, the distribution ˆ µ := (cid:80) T ′ τ =0 µ x kτ /T
′ is a Θ ( nV log m log T/T ) - approximate CCE.
Proof.
Notice that z ti =
z
⌊ t/k ⌋ i when ( t mod k ) ̸
= 0 and that
[Equation]
As a result, the sequence z
i , z ki , . . .
, z kτi , . . .
is the sequence produced by MWU with look-ahead (BeThe-Regularized-Leader) applied on the sequence reward vectors v i ( x 0 − i ) , v i ( x k − i ) , . .
. , v i ( x kτ − i ) , . . .
and it is known to admit the following regret guarantee 3 ,
[Equation]
= − [Equation] nV
[Equation]
m Up next, we establish that ∥ z k · τ i − x k · τ i ∥ 1 ≤ 8 / 2 k for all τ ≥ 1 .
Let τ m :=
( τ − 1) · k + m .
By the definition of Algorithm 2, x k · τ i = x
[Equation] − 1 i and z τ m i =
z τ 0 i for all m = 0 , . . .
, k − 1 .
Thus,
[Equation]
←
[Equation]
) for all s [Equation] S
[Equation] Using Equation MWU f of Section 3.1, the above system of equations can be concisely written as,
[Equation]
and since all agents follow Algorithm 2, x τ m =
G (cid:0) x
τ m − 1 (cid:1) where G ( x ) :=
( f z τ 01 ( x ) , . . . ,
f z τ 0 n ( x )) .
Since η := 1 / 2 nV by Theorem 3.6, G ( x ) is a contraction map with constant 1 / 2 .
Thus,
[Equation]
The second inequality follows by D ( x τ 1 , x τ 0 ) ≤ 2 (see Definition 3.5).
Recall that x k · τ = x τ k −and that z k · τ = G (cid:0) x τ k − 1 (cid:1) (Equation 3).
As a result, for each agent
i
[Equation]
We are now ready to complete the proof of Theorem 4.1,
T ′ (cid:88) τ = ⟨ v i ( x k · τ − i ) , x k · τ i ⟩ ≥ T ′ (cid:88) τ [Equation] [Equation]
[Equation]
i )
[Equation]
k
[Equation]
x
[Equation]
⟩| ≥
[Equation]
(cid:88)
τ
[Equation] [Equation]
⟩ − [Equation] (cid:88)
[Equation]
[Equation]
[Equation]
−
[Equation]
≥ T ′ (cid:88) τ = ⟨ v i ( x k · τ − i ) , z k · τ i ⟩ − 8 V T k 2 k − ∥ v i ( x 0 − i )
{+∥ ∞ · ∥ x 0 i − z 
+}
i ∥≥ max x i ∈X i T ′ (cid:88) τ = ⟨ v i ( x k · τ − i ) , x i ⟩ − 2 nV log m − 10 V ≥ max x i ∈X i
T ′ (cid:88) τ = ⟨ v i ( x k · τ − i ) , x i ⟩ − 12 nV log m
Conclusion
In this paper we analyze a novel algorithm achieving the strongest to-date results for time-average convergence to coarse correlated equilibria (CCE) in general games.
We call this Clairvoyant
Multiplicative Weights Updates (CMWU).
Although CMWU includes in its definition a fixed-point computation and effectively enables the agents to access the future behavior of their opponents, we show that it can be implemented efficiently in an online and totally uncoupled manner.
A critical conceptual innovation of the uncoupled implementation of CMWU is that the CCE computation is done not via the standard techniques (uniform time averaging or last-iterate implementation) but an averaging of a pre-specified and game independent O (log T ) -sparse sub-sequence of the whole history of play.
This type of implementation, as far as we know, is novel not only from a theoretical but even from a practical perspective.
Indeed, learning in games has become a fundamental as well as ubiquitous tool for numerous machine learning applications.
These include the most well known AI headline success stories such as
Generative Adversarial Networks [19], achieving superhuman performance in diverse settings such as Go [36], heads-up Poker [28], many-player Poker [7] a.o.
Despite this wide range of settings, there is a roughly common pattern in achieving these results - design an optimization-driven learning dynamic and have it compete against itself in self-play.
In poker style applications, the typical technique is to approximately solve the zero-sum game via low-regret time-averaging (typically uniform, sometimes with recency bias) over the whole history of play.
In other applications such as Go, where the policy/strategy encoding is achieved by DeepNets with many parameters and any notion of averaging is totally impractical, the hope is that self-play leads to ever improving agents with the solution being the last-iterate of the self-play process.
Our analysis shows that exploring different averaging techniques enables efficient, simple-to-implement, uncoupled and state-of-the-art algorithms to solving general normal-form games.
This points to a rather under-explored hyperparameter of online algorithm design, the averaging policy , and raises tantalizing questions about extending our results both in theory as well as experimentally to more complex settings. Acknowledgments
This research/project is supported by the National Research Foundation, Singapore and DSO National
Laboratories under the AI Singapore Programme (AISG Award No: AISG2-RP-2020-016), NRF Fellowship NRF-NRFF2018-07, NRF2019-NRF-ANR095 ALIAS grant, grant PIE-SGP-AI-2020-01, AME Programmatic Fund (Grant No. A20H6b0151) from the Agency for Science, Technology and Research (A*STAR),
Provost’s Chair Professorship grant RGEPPV2101 and by EPSRC grant EP/R018472/1. Ryann Sim gratefully acknowledges support from the SUTD President’s Graduate Fellowship (SUTD-PGF)
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We argue for the results described in the abstract and intro in the remaining sections of the paper.
(b) Did you describe the limitations of your work?
[Yes] We dedicate a large portion of the paper to explaining how to circumvent the limitations of our proposed algorithm (see Section 4).
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our work is primarily theoretical, with no potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] We discuss the assumptions needed for each result explicitly.
(b) Did you include complete proofs of all theoretical results?
[Yes] All proofs are either given directly after the result, or deferred to the Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]