Paper: Slingshot Perturbation to Learning in Monotone Games (ICLR-2024)

Reviewer comment (Theory):
The authors didn't give any impossibility results (lower bounds) on the convergence rate. I would suggest the authors to present some lower bound results to contrast with the upper bound results they give, even if the lower bounds can be from previous work.

Author response:
Thank you for your valuable comment regarding the tightness of our derived convergence rates. We agree that establishing lower bounds for our algorithm in both full and noisy feedback settings would be a promising future direction. In fact, to the best of our knowledge, no existing research has derived a lower bound for the noisy feedback setting. We will strive to provide a lower bound for our algorithm in full and noisy feedback settings in future work.

Reviewer comment (Theory):
The $O(T^{-1/10})$ bound for finding exact Nash equilibrium in the noisy feedback case (Theorem 5.2) does not seem to be tight. It would be great if the authors can provide some discussion on the tightness of this result.

Author response:
Our convergence rate in the noisy feedback could be improved because we have roughly derived the upper bound on $\langle \nabla_{\pi_i}v_i(\pi^{\mu, \sigma^k}), \pi^{\mu, \sigma^{k-1}} - \sigma^k\rangle$ by using Cauchy–Schwarz inequality in the proof of Lemma F.2. We will mention the potential for improvement in our convergence rate in the subsequent revision.

Reviewer comment (Theory):
While the convergence rates to approximate equilibria (Section 4) are presented for any general $G$ functions, the convergence rates for exact Nash equilibria (Section 5) are only for $G$ being the squared $\ell^2$-distance. The authors say that Theorem G.1 and G.2 are for other $G$ functions, but they are only asymptotic results, with no quantitative convergence rates. Given that the main contribution of this paper is a unified framework, I feel that a quantitative convergence rate for general $G$ function is needed.

Author response:
We would like to emphasize that existing studies have only provided asymptotic last-iterate convergence results for special cases of $G$, such as KL divergence [Perolat et al., 2021] or reverse KL divergence [Abe et al., 2023]. Our work significantly contributes to this field by (i) generalizing these results to a wider class of $G$ functions, and (ii) deriving a non-asymptotic last-iterate convergence rate for a specific $G$ function. We acknowledge the importance and challenge of extending Theorems 5.1 and 5.2 to general $G$ functions, which we view as an intriguing open problem for future work.

Reviewer comment (Theory):
The cited related work [Cai-Luo-Wei-Zheng, 2023, Uncoupled and Convergent Learning in Two-Player Zero-Sum Markov Games] shows that the Nash equilibria of 2-player zero-sum games can be found by an uncoupled learning algorithm with bandit feedback with $O(T^{-1/8})$ convegence rate. This is better than the $O(T^{-1/10})$ bound in this paper. Of course I'm not saying [Cai-Luo-Wei-Zheng, 2023] outperforms this paper, because this paper's result holds for more general monotone games. But I wonder whether [Cai-Luo-Wei-Zheng, 2023]'s ideas can be used to improve this paper's result? In particular, improving the convergence rate or proving results for the bandit feedback case?

Author response:
As you pointed out, Cai et al. [2023] have analyzed settings that are subtly different from ours, both in terms of the feedback type (bandit feedback vs noisy feedback), and the type of game (two-player zero-sum matrix games or Markov games vs $N$-player monotone games). This distinction makes it challenging to draw a straightforward comparison between the convergence rate of theirs and ours in Theorem 5.2.

Moreover, the algorithm proposed by Cai et al. [2023] involves a decreasing perturbation strength $\mu_t$ over iteration $t$. However, in our method, as indicated in Theorem 4.2, a smaller $\mu$ necessitates a correspondingly smaller learning rate $\eta$. This could potentially slow down the convergence rate and make it difficult to find an appropriate learning rate. Therefore, for the sake of practicality, we have opted to maintain $\mu$ as a constant independent of $t$ in our algorithm.

Reviewer comment (Theory):
At first glance, the claims about "exact Nash equilibrium" computation can seem misleading.

Author response:
Thanks for pointing this out to us. We have replaced the term ‘’exact Nash equilibrium’’ with ‘’Nash equilibrium of the underlying game’’ in our latest version (highlighted in red) to clarify what the term means.

Reviewer comment (Theory):
The algorithm is not fully decentralized. It could be possible to discuss whether the process would converge when the frequency of update of the slingshot strategy and $\mu$ varied across players.

Author response:
When each player may have a distinct perturbation strength $\mu_i>0$, convergence results very similar to the theorems in Section 4 are expected to be obtained. This is because strong convexity for every perturbed payoff function is still maintained. Consequently, we expect the behavior to be consistent with these theorems, substituting $\mu$ with the smallest perturbation strength $\min_{i\in [N]}\mu_i$ across all players.

For the scenarios where the frequency of updates of the slingshot strategy, $T_{\sigma}$, varies among players, that is, each player updates the slingshot strategy at his or her own timing, we assume that each player independently adheres to the lower bound on $T_{\sigma}$ as defined in Theorems 5.1 or 5.2. Under this condition, we are expected to obtain similar convergence rates as in these theorems.

Therefore, we believe that we can fully decentralize our algorithm with ease. We also agree that further analysis of this asymmetric setup would present an intriguing and promising avenue for future research.

Reviewer comment (Theory):
Do you think that the slingshot perturbation framework achieves no adversarial regret?

Author response:
Although a no-regret guarantee is not crucial for ensuring last-iterate convergence, we guess the FTRL-SP would achieve the no-regret property. This is because FTRL-SP updates the slingshot strategy so that it converges to the best response against the current strategy of the opponent. However, this does not always admit an optimal rate. We will mention this issue in the subsequent revision upon your request.

Reviewer comment (Experiments):
What would happen if a fixed number of agents did not update their slingshot strategy every time or every player had their own frequency of updating the slingshot strategy?

Author response:
We anticipate that if some agents do not update their slingshot strategy, or if $T_{\sigma}=\infty$ for some agents, it would disrupt the last-iterate convergence property. Specifically, the strategy of the player with $T_{\sigma}=\infty$  at least would not reach a Nash equilibrium strategy.

With regards to the scenario where each player has their own $T_{\sigma}$, we have addressed this in our response to Weakness 2.

Reviewer comment (Theory):
Would it be possible to tune $\mu$ as well to achieve better convergence rates?

Author response:
We appreciate your insightful suggestion regarding the tuning of $\mu$ for better convergence rates. Indeed, setting $\mu$ to a small value that depends on $T$ could potentially lead to improved convergence rates. However, as we have implied in Theorem 4.2, a smaller $\mu$ would necessitate a smaller learning rate $\eta$, which could make the balancing between $\mu$ and $\eta$ quite challenging. For the sake of practicality, we have opted to maintain $\mu$ as a constant independent of $T$.

Reviewer comment (Theory):
It is not clear if noise ruins the guarantees of the algorithms already present in the literature. Usually no-regret algorithm still converge in full feedback, to some sort of equilibria even in the presence of noisy feedback as long as the loss used are unbiased. The authors should discuss better why existing techniques fail, why now they only fail in terms of experimental evaluation.

Author response:
Standard no-regret algorithms like MWU converge to a Nash equilibrium only when one averages strategies over iterations to obtain the exact equilibrium strategy, since the dynamics make an orbit around it. While this property is well-known as average-iterate convergence, we focus on last-iterate convergence, where an algorithm converges to an equilibrium without taking the average of the strategies. We hope this clarifies the difference between the concepts of average-iterate and last-iterate.

Previous studies [Daskalakis and Panageas, 2019, Mertikopoulos et al., 2019, Wei et al., 2021] have shown that optimistic no-regret algorithms, such as OWMU and OGD, enjoy last-iterate convergence only with full feedback, but fail with noisy feedback [Abe et al. 2023]. Roughly speaking, the success with full feedback is due to the proper bound of the variation of the gradient feedback vectors $\sum_{i=1}^N\\|\widehat{\nabla}\_{\pi_i}v_i(\pi^t) -\widehat{\nabla}\_{\pi_i}v_i(\pi^{t-1})\\|^2$ over iterations. More formally, we define the path length of the gradient feedbacks as $\sum_{s=1}^t\sum_{i=1}^N\\|\widehat{\nabla}\_{\pi_i}v_i(\pi^s) -\widehat{\nabla}\_{\pi_i}v_i(\pi^{s-1})\\|^2$. Noisy feedback makes the path length large, leading to the failure of optimistic no-regret algorithms to converge to an equilibrium.

Thus, achieving last-iterate convergence with noisy feedback is still a significantly challenging task. We will explain the intuition in our paper's Introduction or related literature section.

Reviewer comment (Novelty):
My second, and far more pressing, concern is on the technical innovation of the work. The "slingshot" perturbation was studied extensively in recent papers, as also noted by the authors (Sokota et al. 2023, Bernasconi et al. 2022, Perlolat at al. 2021, Liu et al. 2023). The main technical contribution of the paper is the introduction of the noise into the feedback, but it not clearly well explained why this is challenging or important and what technical novelty does the problem require.

Author response:
The techniques devised in this paper are manifold. 

- We have developed a subtly different perturbation technique from the existing ones you mentioned so that it enables the FTRL or MD dynamics to converge to an approximate equilibrium in underlying games (an equilibrium in perturbed games).
- The existing studies only show the sort of convergence only with full feedback, and we have been successful with noisy feedback. So, we are the first to establish last-iterate convergence with noisy feedback.
- Our perturbation technique also provides a comprehensive view of payoff-regularized algorithms such as Perorat et al. [2021] and Abe et al. [2022].
- Furthermore, the idea of the slingshot strategy update leads us to find a Nash equilibrium in underlying games (not an approximate one). In other words, it is innovative that we mix the perturbation technique with the slingshot strategy update and exhibit last-iterate convergence with both full and noisy feedback, which has not been achieved so far.

Reviewer comment (Theory):
I’m puzzled about some results in section 5. In particular, how is it possible to have some meaningful guarantees as the gradient of \(G(\pi,\sigma)\) can be arbitrary large in general. For example, if \(\sigma\) is close to the boundary and \(G\) is the KL divergence, probably the results of section 5 only require very nice properties of \(G\) and thus it limits its applicability.

Author response:
No, we do **not** require the gradient of $G$ to be bounded for Theorems 5.1, 5.2, G.1, and G.2. Therefore, we believe that these theorems hold for a wide range of divergence functions.

Reviewer comment (Theory):
It is also not clear to me why you do the slingshot update. You give guarantees on the exploitability of the last iterate strategy, but not on the convergence rate to the strategy itself, as you do in theorem 4.2 and 4.10 for approximate Nash.

Author response:
First, we would like to clarify that the convergence of exploitability means that the last-iterate strategy itself converges to an equilibrium. Therefore, our results (Theorems 5.1 and 5.2) inherently indicate that we have provided the convergence rate of the strategy itself. Exploitability is a standard measure of proximity to a Nash equilibrium, as used in several studies [Cai et al., 2022a,b, Abe et al., 2023, Cai and Zheng, 2023].