Paper: Last-Iterate Convergence Properties of Regret-Matching Algorithms in Games (ICLR-2024)

Reviewer comment (Writing):
I’m confused about the notation \delta_{\ge}: aren’t all the strategies in \delta already probability distribution that whose sum is exactly 1?

Author response:
It is correct that all the strategies in $\Delta$ are probability distributions whose sum is exactly 1. The ExRM+ and SPRM+ algorithm (proposed in [1]) use the regret operator (Equation 2) which is defined over the lifted space $\Delta_{\ge}:= \\{  u \in R+, 1^T u \ge 1 \\}$. We can map a point $z$ in $\Delta_{\ge}$ to a point in $\Delta$ by $\ell_1$ normalization: the normalized point $z’ = g(z) = z/||z||_1$ is a probability distribution.

Reviewer comment (Evaluation):
For the numerical run on the 3X3 algorithm, it would be very insightful to plot the actual trajectory of the algorithms (it should be relatively easy to plot in 2D since the strategies are probability distributions over 3 actions).

Author response:
Thanks for this suggestion. We have run additional experiments to obtain plots showing the trajectories of RM+, alt. RM+, PRM+, alt. PRM+,  ExRM+, and SPRM+ on the $3\times 3$ matrix. We plot the last 2000 iterations (out of $10^5$ iterations) of both players in 2D. The results show that the trajectories of RM+, alt. RM+, PRM+ cycle while alt. PRM+,  ExRM+, and SPRM+ converge. The empirical results can be found at this anonymized link: https://docdro.id/KUGvPvg
We thank you for this interesting point, we will add this in the revised paper.

Reviewer comment (Experiments):
Related, I’m curious if best-iterate convergence of RM+ etc is any better?

Author response:
No. As shown in Figure 1 in the paper, the duality gap of the best-iterate of RM+ is also at least $10^{-1}$ after $10^5$ iterates. Thus the non-convergence holds for both the best-iterate and last-iterate. Moreover, in additional experiments above, we observe the cycling behavior of RM+, which further shows that the best-iterate does not converge.

Reviewer comment (Theory):
The bulk of the work in Sections 4 and 5 is, AFAICT, to deal with solutions of the VI that are not NE. I’m a bit confused about it – when you introduced the VI notation I expected that NE would be the only solutions. Maybe you could give a simple example of a non-NE solution? That would really help build intuition.

Author response:
This is not correct. Every solution of the VI corresponds to a NE after normalization. In section 4 and 5, the main technical challenge is that some solutions do not satisfy the Minty condition. In the following, we first explain with an example that every solution of the VI corresponds to a NE after normalization. 

**Every solution of the VI corresponds to a NE after normalization**: Specifically, the VI is defined over the lifted space $Z_{\ge} = \Delta_{\ge}^{d_1} \times \Delta_{\ge}^{d_2}$  as defined above (also in page 4 at the paragraph introducing ExRM+ and SPRM+). Note that for  $x \in \Delta^{d_1}_{\ge}$, $\sum_i x_i$ could be greater than 1. Thus $x$ may not be a probability distribution and may not be in an NE. However, we have shown in Lemma 2 that for any solution $z = (x, y)$ of the VI in the lifted space (they may not be probability distributions), they corresponds to a Nash equilibrium in the sense that after $\ell_1$ normalization, the point $g(z) = (x/||x||_1, y /||y||_1)$ must be a Nash equilibrium. 

**Example**: The Rock-Paper-Scissors game has a unique Nash equilibrium where both players use $ x^* = y^*=  (\frac{1}{3}, \frac{1}{3}, \frac{1}{3})$. In the lifted space, any point of the form $(Qx^*, R y^*)$ with $ Q, R \ge 1$ is a solution to the variational inequality $VI(F, Z_{\ge})$. When $Q > 1$ or $R > 1$,  $z = (Qx^*, R y^*)$ is not a Nash equilibrium but corresponds to the Nash equilibrium $g(z) = (x^*, y^*)$ after $\ell_1$ normalization.

**Some solutions do not satisfy the Minty condition**: Let $(x, y)$ be any Nash equilibrium. In Fact 1, we show that $(Q x, Qy)$ for any $Q \ge 1$ satisfies the Minty condition for the $VI(F, Z_\ge)$. However, there exists solutions of the form $(Qx, Ry)$ where $Q \ne R$ and such solutions may not satisfy the Minty condition. The existence of non-Minty solutions of the VI adds difficulty to proving last-iterate convergence. In section 4 and 5, we overcome this challenge by establishing structural characterization of the limit points of the learning dynamics (Lemma 3).

Reviewer comment (Theory):
I probably missed something, but I didn’t understand why you need to talk about best iterate vs last iterate in Section 4.2. Does the guarantee of Lemma 1 not imply something about the convergence of last z^t?

Author response:
Lemma 1 does not immediately imply last-iterate convergence, unless one first establishes stronger properties of the operator, which are not known to hold in general in our setting. The key is that Lemma 1 establishes geometric decrease in distance on the lifted joint strategy set $\mathcal{Z}_{\ge}$. However, this does not imply a geometric improvement of the distance for the normalized strategies.

Reviewer comment (Experiments):
The min-max game itself is known to be an LP and thus solved in polynomial time. Iterative algorithms are one of the approaches to solving the LPs. The paper should consider comparing the state-of-the-art LP solvers with iterative algorithms. Although iterative algorithms are easy to implement, practical LP solvers have been improved for a long time and could be faster than naive iterative algorithms.

Author response:
**Q**: *The min-max game itself is known to be an LP and thus solved in polynomial time. Iterative algorithms are one of the approaches to solving the LPs. The paper should consider comparing the state-of-the-art LP solvers with iterative algorithms. Although iterative algorithms are easy to implement, practical LP solvers have been improved for a long time and could be faster than naive iterative algorithms.*

**A**: Thanks for your comment! The focus on iterative algorithms like Regret Matching (RM), RM+ and their variants over Linear Programming (LP) methods is driven by two reasons. 

One of our motivations to study RM and its variants is that they are simple online learning algorithms, and understanding the convergence properties of uncoupled learning dynamics in games is an important question at the intersection of machine learning and game theory. In a multi-agent learning scenario, agents interact with each other and update their strategies in their own interest. Existence of simple learning dynamics that converge to Nash equilibrium in a day-to-day sense (i.e. in last iterate) provide stability guarantees for multi-agent systems, and provide some justification for using Nash equilibrium to predict the outcome of real-life multi-agent interactions. Prior to our work, the literature had focused on the behavior of optimistic variants of FTRL and OMD. Yet RM-type algorithms are very simple, and have a history of performing well in practice (more on that in the next paragraph), and thus their last-iterate convergence behavior is of interest in its own right.

Our study is also motivated by the strong practical performance of RM-type iterative algorithms, especially in the context of large-scale extensive-form games. While LP methods might offer advantages in smaller games, their application becomes less feasible in larger, more complex scenarios. This is evident from the historical context: the last notable application of LP for offline two-player zero-sum games was in 2005, which successfully solved Rhode Island Hold’em [1, 2]. However, the practicality of LP diminishes as the game size increases. The size of LP formulations is linear in the game tree size, making the use of simplex or Interior Point Method (IPM) iterations extremely slow for larger games. Since 2007, there has been a significant shift in the field towards using iterative algorithms like Counterfactual Regret Minimization (CFR) [3], dilated Entropy-regularized algorithms such as excessive gap technique (EGT) and CFR+ [4]. We remark that CFR and CFR+ build upon RM and RM+, respectively. Every superhuman poker AI developed since then has utilized CFR+ [5, 6, 7]. Despite their strong practical performance in large-scale games, RM-type algorithms are not well-understood theoretically, which motivates our study.

[1] Gilpin, Andrew, and Tuomas Sandholm. “Optimal Rhode Island Hold’em Poker.” AAAI, 2005 

[2] Gilpin, Andrew, and Tuomas Sandholm. "Lossless abstraction of imperfect information games." Journal of the ACM, 2007

[3] Zinkevich, M., Johanson, M., Bowling, M., Piccione, C. Regret minimization in games with incomplete information. NeurIPS, 2007

[4] Tammelin, Oskari. "Solving large imperfect information games using CFR+." 2014

[5] Bowling, M., Burch, N., Johanson, M., & Tammelin, O. Heads-up limit hold’em poker is solved. Science, 2015

[6] Brown, Noam, and Tuomas Sandholm. "Superhuman AI for heads-up no-limit poker: Libratus beats top professionals." Science, 2018

[7] Brown, Noam, and Tuomas Sandholm. "Superhuman AI for multiplayer poker." Science, 2019

Reviewer comment (Theory):
I do not understand why the last-iterate property is important. The convergence results can still be obtained by averaging the outputs of iterative OLO-based algorithms (such as Hedge). Maybe a more important issue is the speed itself, not whether the property holds or not.

Author response:
**Q**: *I do not understand why the last-iterate property is important. The convergence results can still be obtained by averaging the outputs of iterative OLO-based algorithms (such as Hedge). Maybe a more important issue is the speed itself, not whether the property holds or not.*

**A**: We can average the iterates and get convergence results. However, last-iterate convergence is still important from both theoretical and practical point of view, as we have motivated in the introduction of the paper. 
If we view learning as a model of agents’ strategic behavior, then only the last-iterate convergence of learning dynamics guarantees that agents’ strategies (day-to-day behavior) stabilize at Nash equilibrium. Learning dynamics with only average-iterate convergence is unstable and exhibits cycling and divergence behavior. For example, the Hedge algorithm does not converge in last-iterate even for simple zero-sum games [8]. The problem of understanding last-iterate convergence for learning in games has received extensive attention in recent years, see e.g [9, 10, 11, 12, 13] and more references therein. 
Last-iterate is widely used in practice for simplicity. In some cases, iterate averaging can be cumbersome and even impractical when neural networks are involved. Thus it is crucial to develop theoretical tools to understand last-iterate convergence of learning algorithms.

Reviewer comment (Experiments):
In summary, the paper focuses only on iterative algorithms, but as a solver of a certain LP, there are more alternatives to compare.

Author response:
**Q**: *The min-max game itself is known to be an LP and thus solved in polynomial time. Iterative algorithms are one of the approaches to solving the LPs. The paper should consider comparing the state-of-the-art LP solvers with iterative algorithms. Although iterative algorithms are easy to implement, practical LP solvers have been improved for a long time and could be faster than naive iterative algorithms.*

**A**: Thanks for your comment! The focus on iterative algorithms like Regret Matching (RM), RM+ and their variants over Linear Programming (LP) methods is driven by two reasons. 

One of our motivations to study RM and its variants is that they are simple online learning algorithms, and understanding the convergence properties of uncoupled learning dynamics in games is an important question at the intersection of machine learning and game theory. In a multi-agent learning scenario, agents interact with each other and update their strategies in their own interest. Existence of simple learning dynamics that converge to Nash equilibrium in a day-to-day sense (i.e. in last iterate) provide stability guarantees for multi-agent systems, and provide some justification for using Nash equilibrium to predict the outcome of real-life multi-agent interactions. Prior to our work, the literature had focused on the behavior of optimistic variants of FTRL and OMD. Yet RM-type algorithms are very simple, and have a history of performing well in practice (more on that in the next paragraph), and thus their last-iterate convergence behavior is of interest in its own right.

Our study is also motivated by the strong practical performance of RM-type iterative algorithms, especially in the context of large-scale extensive-form games. While LP methods might offer advantages in smaller games, their application becomes less feasible in larger, more complex scenarios. This is evident from the historical context: the last notable application of LP for offline two-player zero-sum games was in 2005, which successfully solved Rhode Island Hold’em [1, 2]. However, the practicality of LP diminishes as the game size increases. The size of LP formulations is linear in the game tree size, making the use of simplex or Interior Point Method (IPM) iterations extremely slow for larger games. Since 2007, there has been a significant shift in the field towards using iterative algorithms like Counterfactual Regret Minimization (CFR) [3], dilated Entropy-regularized algorithms such as excessive gap technique (EGT) and CFR+ [4]. We remark that CFR and CFR+ build upon RM and RM+, respectively. Every superhuman poker AI developed since then has utilized CFR+ [5, 6, 7]. Despite their strong practical performance in large-scale games, RM-type algorithms are not well-understood theoretically, which motivates our study.

[1] Gilpin, Andrew, and Tuomas Sandholm. “Optimal Rhode Island Hold’em Poker.” AAAI, 2005 

[2] Gilpin, Andrew, and Tuomas Sandholm. "Lossless abstraction of imperfect information games." Journal of the ACM, 2007

[3] Zinkevich, M., Johanson, M., Bowling, M., Piccione, C. Regret minimization in games with incomplete information. NeurIPS, 2007

[4] Tammelin, Oskari. "Solving large imperfect information games using CFR+." 2014

[5] Bowling, M., Burch, N., Johanson, M., & Tammelin, O. Heads-up limit hold’em poker is solved. Science, 2015

[6] Brown, Noam, and Tuomas Sandholm. "Superhuman AI for heads-up no-limit poker: Libratus beats top professionals." Science, 2018

[7] Brown, Noam, and Tuomas Sandholm. "Superhuman AI for multiplayer poker." Science, 2019

Reviewer comment (Theory):
Are the analyses for the last iterate property useful for constructing a new online-to-batch conversion technique (e.g., averaging all outputs of OCO algorithms)?

Author response:
**Q**: *Are the analyses for the last iterate property useful for constructing a new online-to-batch conversion technique (e.g., averaging all outputs of OCO algorithms)?*

**A**: We think the online-to-batch technique is not related to last-iterate convergence. This is because online convex optimization is a single-agent learning problem, where the goal is to minimize regret against an adversary; while learning in games is a multi-agent learning problem and the goal is to prove last-iterate convergence to Nash equilibrium. The following examples further illustrate the differences between regret minimization and last-iterate convergence in games.

1. The Extragradient (EG) algorithm suffers linear regret [11] in OCO but when both players employ EG in a zero-sum game, their joint strategy converges to a Nash equilibrium [13].
2. The online gradient descent (OGD) algorithm is a no-regret online algorithm but when both players employ OGD in a zero-sum game, their joint strategy diverges from Nash equilibrium [8]. 

[8] Mertikopoulos, Panayotis, Christos Papadimitriou, and Georgios Piliouras. "Cycles in adversarial regularized learning." SODA, 2018

[9] Daskalakis, Constantinos, and Ioannis Panageas. "Last-Iterate Convergence: Zero-Sum Games and Constrained Min-Max Optimization." ITCS, 2019

[10] Lin, T., Zhou, Z., Mertikopoulos, P., & Jordan, M. I.. Finite-time last-iterate convergence for multi-agent learning in games. ICML 2020.

[11] Golowich, N., Pattathil, S., & Daskalakis, C.  Tight last-iterate convergence rates for no-regret learning in multi-player games. NeurIPS, 2020

[12] Wei, C. Y., Lee, C. W., Zhang, M., & Luo, H. Linear Last-iterate Convergence in Constrained Saddle-point Optimization. ICLR, 2021

[13] Cai, Y., , Oikonomou, A., Zheng W. Finite-Time Last-Iterate Convergence for Learning in Multiplayer Games. NeurIPS 2022