Paper: Regularization is Enough for Last-Iterate Convergence in Zero-Sum Games (ICLR-2024)

Reviewer comment (Novelty):
The main issue pertains to the novelty of the results. There are many existing papers with closely related results, such as 1) "Last-iterate convergence with full- and noisy-information feedback in two-player zero-sum games;" 2) "A unified approach to reinforcement learning, quantal response equilibria, and two-player zero-sum games;" 3) "Modeling strong and human-like gameplay with KL-regularized search;" and 4) an unpublished paper "No-Regret Learning in Strongly Monotone Games Converges to a Nash Equilibrium." Some of those papers are cited, but the discussion is inadequate, such as the comparison with magnetic mirror descent.

Author response:
**1. About novelty, and some of those papers are cited, but the discussion is inadequate.**

We acknowledge that some of the discussion about related work was inadequate in the original submission, which is also argued by other reviewers. We have carefully addressed this issue and made some necessary modifications in the new version paper (see Section 1.1 on page 2~3). Please also refer to the responses to related comments raised by other reviewers.

Reviewer comment (Experiments):
Regarding the experimental evaluation (Section 5.1), experiments on very small games such as Kuhn or Leduc can be misleading, and it's hard to draw any definite conclusions. The reviewer recommends using larger games.

Author response:
**2. Regarding the experimental evaluation (Section 5.1), experiments on very small games such as Kuhn or Leduc can be misleading, and it's hard to draw any definite conclusions. I would recommend using larger games.**

In Section 5.1, our primary objective was to further validate our theoretical results, with a specific focus on convergence rather than scalability. To ensure a fair comparison with related works and for the sake of consistency with common experimental environments in the literature, we opted for the widely used Kuhn or Leduc game settings. It is true that these games have relatively small sizes, but they still capture the complexity of sequential decision-making, making them suitable for our convergence-focused analysis.

Additionally, we acknowledge the importance of demonstrating the scalability potential of our proposed approach. In Section 5.2, we have included experiments with a larger game, Phantom Tic-Tac-Toe, to showcase the scalability of our approach.

Reviewer comment (Theory):
What is more concretely the problem of using optimism or extra-gradient in the context of Figure 1? The paper states that these "...can impede convergence, particularly when the real-time policy exhibits chaotic behavior," but the reviewer does not follow this reasoning.

Author response:
**3. What is more concretely the problem of using optimism or extra-gradient in the context of Figure 1? You write that "...can impede convergence, particularly when the real-time policy exhibits chaotic behavior," but I don't think I am following this.**

In Figure 1, the primary focus is on illustrating the guidance of second-order information in dynamic settings for different types of games. Specifically:

1. In a Hamiltonian game, second-order information points toward the NE, facilitating convergence. The introduction of second-order information aids in convergence in this context.
2. Conversely, in a potential game, second-order information diverges from the NE. In such cases, introducing second-order information can decelerate convergence or even lead to divergence.

The key reason for this divergence in potential games lies in the fact that the computation of second-order information depends on the gradient flow. In contrast, regularization, which is not dependent on the gradient flow, tends to provide more robust convergence performance.

Furthermore, optimism or extra-gradient can be considered as first-order approximations of second-order information. Therefore, they may inherit similar issues in scenarios where second-order information introduces challenges.

Reviewer comment (Presentation):
The reviewer is not sure of the purpose of Section 2.3 and asks the authors to explain.

Author response:
**4. I am not sure I see the purpose of Section 2.3. Can the authors explain?**

The intention behind Section 2.3 is to illustrate that the introduction of regularization, as discussed in the paper, can be interpreted from the perspective of enhancing the potential component of a game. Specifically, it highlights that the regularization term in the objective function can be viewed as augmenting the potential component, which, in turn, facilitates the achievement of last-iterate convergence. 

In the revised version, we have moved Section 2.3 to Appendix C to ensure compliance with page limitations while maintaining the completeness of the content.

Reviewer comment (Novelty):
By introducing regularization to the underlying two-player zero-sum game, the game becomes strongly monotone (strongly-convex–strongly-concave). Since RegFTRL is equivalent to running FTRL on a regularized strongly monotone game, the linear last-iterate convergence of RegFTRL follows from the fact that Mirror Descent (MD) or Follow the Regularized Leader (FTRL, the lazy projection version of MD) has linear last-iterate convergence. This approach is also studied in many recent works [1,2] and the paper should discuss the difference and their unique contribution more clearly.

Author response:
We thank the reviewer for pointing out work [1], and acknowledge our work shares some similarities with it. However, we would like to highlight some distinction between these two approaches.

1. Our paper focuses a complementary scenario. While [1] considers a broader class of monotone games beyond NFGs, their analysis could not encompass the behavior-form EFGs considered in our paper.
2. When concentrating on NFGs under full-information feedback, our analyses differ from [1], leading to distinct conclusions. Specifically:
   - Our proposed step size range is larger, especially when $g=D_{\psi}$, where our range is $(0, \frac{1}{\tau}]$ compared to their $(0, \frac{2\tau}{3\tau^2+8})$.
   - The derived convergence rates differ slightly. Our rates are of the form $(1+\eta\tau\lambda)^{-t}$, whereas [1] is $(1-\eta\tau\lambda/2)^{t}$.
3. Additionally, we have strengthened our results by adding sublinear rates to the exact Nash equilibrium, i.e, Theorem 4 in the revised version (see Theorem 4 on page 7). Notably, our results hold for any general-case regularization, whereas [1] specifically considers the l2-norm. 

To enhance clarity, we have added a comparison with the relevant literature, especially [1], in the related work (see Section 1.1 on page 3) and the remark following our theorems (see Remark 2 on page 6).

Reviewer comment (Theory):
The statement that “In practical terms, the implementation of the optimistic update approach often necessitates the computation of multiple gradients at each iteration, making it intricate and resource-intensive” is not true for OMWU or OGDA (refers to Optimistic Gradient Descent-Ascent), which only require computation of one gradient in each iteration.

Author response:
Our original wording was indeed misleading. We have modified "computation of multiple gradients" to "computation of multiple strategies" to accurately convey the idea. In the context of RegFTRL, the computation involves a single strategy sequence $\pi_t$ at each iteration. On the other hand, in the case of OMWU or OGDA, maintenance of two strategy sequences, $\pi_t$ and $\hat\pi_t$, is required. The computation of $\pi_{t+1}$ in each iteration involves both $\pi_t$ and $\hat\pi_{t+1}$. Therefore, the distinction lies in the number of strategy sequences involved in the computation. Furthermore, OGDA has high per-iteration complexity due to the costly projection operations at each iteration, which adds to the computational burden.

Reviewer comment (Theory):
It is unclear if the current approach gives more general results, specifically: (a) extension to monotone games; (b) last-iterate convergence *rates* to Nash equilibrium; (c) convergence (*rates*) under noisy feedback / bandit feedback.

Author response:
We would like to express our sincere apologies for the oversight on the revised version of Theorem 4. We have reverted to the original statement of Theorem 4, guaranteeing the asymptotic convergence to the Nash Equilibrium. However, we would like to emphasize that previous work typically provides asymptotic convergence only, whereas Abe's work [1] was conducted concurrently with ours. While we share some results with Abe's work, it is crucial to note that the analytical proof they employ differs from ours, and their results do not extend to the behavior-form EFGs in our paper. Moving forward,  we acknowledge the complexity associated with the extension of Theorem 4 to general-case regularization with guaranteed rate, which is also an interesting question in future research.

Reviewer comment (Writing):
The paper requires significant proofreading. There are many typos and missing articles (e.g., "continue-time" and "continues-time" should be "continuous-time" on page 5) and quantities aren't necessarily always clearly defined (e.g., $r^h$ should be either explicitly given a name or otherwise introduced at the bottom of page 3 since the way it is currently written it is assume that the reader should know what $r^h$ is/that it already has been introduced).

Author response:
We have proofread the paper several times again in order to eliminate any possible typos.

Reviewer comment (Presentation):
The preliminaries could be made more substantial. A discussion of FTRL (at least explicitly mentioning the FTRL update) would be appropriate in the preliminaries (or earlier in the introduction).

Author response:
Following your advice, in the main text, we have added an explicit but brief mention of the FTRL update in the preliminaries (see Section 2.2 on page 4), and provide a detailed explanation in the Appendix C to adhere to page limitations.

Reviewer comment (Presentation):
Perhaps you can note once after the preliminaries all proofs are included in the appendix instead of explicitly creating a proof environment for each theorem statement to state that the proof can be found in the appendix.

Author response:
Thanks for the advice, and we have adopted your recommendation in the revised version.

Reviewer comment (Presentation):
Why do the plots in Figures 3 and 4 start at $10^2$ and $10^3$? It seems important to note the performance early on as well.

Author response:
In the early iterations, the exploitability of various algorithms experiences rapid declines, and the distinction among them is not apparent. By focusing on later iterations, where the differences among these algorithms become more evident, we aimed to provide a clearer and more meaningful comparison. However, we acknowledge the importance of presenting a complete picture. In response to your feedback, we have added figures in the revised paper to include all iterations (see Figure 3 and 4 on page 8), ensuring a more comprehensive view of the algorithms' performance over the entire range.

Reviewer comment (Experiments):
Is there a reason you are not comparing to SOTA CFR variants (e.g. CFR$^+$ [a], DCFR [b], PCFR$^+$ [c]) in your EFG experiments in Figure 4?

Author response:
We have added CFR+ as a baseline in the revised version (see Figure 4 on page 8).

Reviewer comment (Theory):
Do you request uniqueness of NE in zero-sum game?

Author response:
No, we do not request the uniqueness of NE. The uniqueness of the equilibrium is ensured by Theorem 1 in our paper, and therefore, we do not rely on the uniqueness assumption.

Reviewer comment (Theory):
In NFGs (0-sum), your algorithm converge always to an \eps-NE?

Author response:
Yes, our algorithm converges to an $\epsilon$-NE in NFGs without the use of annealing or adaption approaches. However, as indicated by experiments and theorems in our paper, the proposed algorithm can also converge to an NE through the adaption approach.

Reviewer comment (Reproducibility):
What is the reason that you pass to FollowMu in the experimental section?

Author response:
In the experimental section, we employed FollowMu as part of a two-fold evaluation. The first evaluation assesses RegFTRL to substantiate our theoretical findings, while the second one evaluates the neural-based algorithm, FollowMu, which is derived from RegFTRL, in order to validate the effectiveness and practical performance of RegFTRL with neural network function approximation.