Paper: Observer Uncertainty of Learning in Games from a Covariance Perspective (ICLR-2024)

Reviewer comment (Theory):
Although the authors claim that the 'input uncertainty' is well-motivated, I am still skeptical about this. For a convergent trajectory, measuring such an uncertainty could tell us (i) how fast we approach the solution; (ii) the initialization would not affect the convergence. However, for a non-convergent trajectory, measuring this uncertainty seems only to tell us whether the trajectory remains bounded or how fast it diverges. Although the results in this paper seem solid, I do not know what are the further implications for these cycling or divergent behaviors.

Author response:
Please refer to the second point of the global respond for more discussion on the concept of input uncertainty. As it was shown in previous works, the dynamical behaviors of FTRL is complex, it can exhibit recurrence [1] and chaotic behaviors [2]. However, these properties only provide qualitative descriptions about the hardness of predicting the dynamics of these algorithms. The results in the current paper focus on the quantitative viewpoint, and demonstrates new characteristics of these dynamics, such as the tradeoff between cumulative strategies and payoff. For more discussion on significance, applications if needed see also our related response to Reviewer kXrY.

Reviewer comment (Evaluation):
Some results related to cumulative strategies lack interpretation. The adaptation of cumulative payoffs is easy to understand because we could map it into the strategy space through $\nabla h^*$. However, the adaptation of cumulative strategies seems just to facilitate the definition of the Hamiltonian system. Dividing it by $t$ yields a more intuitive meaning, i.e., the averaged strategies. Consequently, I think that the results in Theorem 5.1 could be transformed into those related to averaged strategies. Then the difference between the covariances of averaged strategies and cumulative payoffs can be better analyzed, because they represent the averaged-iterate and last-iterate behavior, respectively. In particular, I think it is also worth discussing the relationship between the results in Theorem 5.1 and the no-regret property of FTRL as well as the convergence of the averaged strategies. Finally, I do not understand the meaning of the covariance between cumulative strategies and payoffs.

Author response:
Cumulative strategies have a nature game-theoretical explanation: they denote the historical frequencies of strategies used by a player, and they also provide a new coordinate system to describe the dynamics of the two players system. Cumulative strategies and payoffs are related by the identity $y_i(t) = y_i(0) + A^{(ij)} X_j(t)$, for $i,j = 1,2$, thus one player's  cumulative strategies can determine another player's cumulative payoff (although the payoff matrix is unknown), this also explain why we can trace the behaviors of a two players system from the viewpoint of a single player.

We appreciate your suggestion to consider average strategies. Our results on the covariance of cumulative strategies can be used to derive corollaries about the average strategies using the relation $\text{Var}(X(t)/t) = \text{Var}(X(t))/t^2$. For example, in the alternating setting, Theorem 5.1 implies that when $AA^{\top}$ is non-singular, the variance of average strategies tends to $0$ at a rate of $O(1/t^2)$. In case $AA^{\top}$ is singular, the variance of average strategies tends towards a non-zero value, and this number can be determined through computational methods. This suggests that in such cases, the initial distribution will not concentrate as in non-singular cases.

Reviewer comment (Theory):
The form of symplectic discretization may be not rigorous. If we adopt the symplectic Euler method in [1, Theorem VI.3.3], the update rule of $Y^t$ in symplectic discretization (Type I method) should be $Y^{t+1} = Y^{t} - \eta \nabla_X H( X^t, Y^{t+1} )$, while the authors use $Y^{t+1} = Y^{t} - \eta \nabla_X H( X^t, Y^{t} )$. Although for the Hamiltonian function defined in Proposition 3.1, the two versions are equivalent, I think adopting the latter one may be misleading and less rigorous.

Author response:
Thank you for pointing this out, we have fixed this in the update manuscripts. The parts that were modified are  marked by red color.

Reviewer comment (Theory):
The lower bound in Theorem 5.2 involves the covariance, while the amplification rates in Theorem 5.1 only hold for the Euclidean regularizer. For general regularizers, without the amplification rates of the covariance, the result in Theorem 5.2 itself is less meaningful.

Author response:
In addition to providing a numerical lower bound, Theorem 5.2 also offers insight into the tradeoff phenomenon between the standard deviations of cumulative payoffs and strategies for more general regularizers. In the case of GDA, where the standard deviations can be calculated precisely, an example of such a tradeoff phenomenon is presented in figure (1) in the main paper. Note that the maximum points of the curve representing standard deviations of cumulative payoffs coincide with  minimum points of the curve representing standard deviations of cumulative strategies, and vice versa. This insight cannot be obtained by analyzing the growth rate of standard deviations of cumulative payoffs and strategies independently since Theorem 5.1 only indicates that both curves are bounded.

Reviewer comment (Writing):
The authors could give more intuition on the definition of the Hamiltonian function.

Author response:
The Hamiltonian function used in the current paper comes from [3], and a similar non-canonical Hamiltonian system formulation (also known as a Possion system) for continuous time  mirror descent algorithms is also presented in [4]. Theorem 5.1 of [3] demonstrates that the Hamiltonian function defined here is inherently connected to the Bregman divergence, which is a commonly used concepts in optimization, plus a an additional term determined by the regularizers and equilibrium of the game.

Reviewer comment (Theory):
The dependence of $\mu$ in Theorem 5.1 on the step size should be clarified.

Author response:
As shown in the proof (Appendix C.2.2 ), the constant  $\mu$ in Theorem 5.1 has norm $\lvert \mu \lvert = 1+\gamma \eta^2$, where $\eta$ is the step size and $\gamma$ is the maximal eigenvalue of $AA^{\top}$.

Reviewer comment (Presentation):
To capture the exponential growth, one could change the y-axis to a log scale. Similarly, to capture quadratic (or more generally, polynomial) growth rate in figures, one could change both axes to log scales.

Author response:
Thank you for the suggestion, we have changed the y-axis to a log scale in the update manuscripts. Please refer to the new manuscripts in the system.

Reviewer comment (Theory):
The results about differential entropy in Section 4 are in terms of MWU and AltMWU, while covariance evolution results in Theorem 5.1 only hold for the Euclidean regularizer, i.e., GDA and AltGDA. They are not directly comparable. However, in the appendix, the authors say that the results in Section 4 hold for more general regularizers. Why not adopt the same regularizer in the main text or state a more general result in Section 4?

Author response:
We present the results in Section 4 using MWU and AltMWU to enable a direct comparison with the results in [2], where differential entropy is used as a measure of uncertainty and the differential entropy of MWU is calculated. However, the proof of our results in Section 4 doesn't rely on any special choice of regularizer.

Reviewer comment (Novelty):
The main issue is with regards to the motivation and the significance of the results. Although the authors already attempt to discuss the motivation at length in the introduction, there is still no concrete motivation or applications for those results.

Author response:
>Motivation, significance of the results, and actual applications.

Please refer to the global respond for more explanation of the motivation.

For the significance : Game dynamics particularly based on zero-sum like game are seminal components in standard AI architectures such as GANS [1] (7000+ refs), PSRO [2] (600+ refs) and are actually useful even in n-player games (Diplomacy [3]) The typical way to control such dynamics so far has been to aim/hope for global convergence to equilibrium, however, there are many hurdles to this tasks both from a computational perspective [4,5] as well as from a dynamical systems perspective [6]. Here we aim at a different type of "stability" altogether. Stability/predictability of orbits *without* needing to converge to Nash as a necessary intermediary step from proving stability. By showing that this is possible we open the door for new multi-agent AI architectures that although maybe design around the pipe-dream of global  convergence they can still be practically useful. 

Actual applications: (Computer Poker). Poker is of course a standard benchmark for AI and a real world application.  This direction of work could provide some insights behind the empirical success of CFR+ [7,8] in poker. CFR+ is a variant of Counter Factual Regret (CFR) minimization [9] with improved empirical performance. A key difference between CFR+ and vanilla CFR is switching from simultaneous updates to alternating updates. Our results provide some hints on why this could lead to increased performance. Increased orbit stability, reduced payoff variances due to symplectic-like discretization would lead to faster time average convergence to equilibrium and thus improved performance. Of course our current work is not on CFR and extensive form games but normal form games, nevertheless, we believe these are promising steps that provide a new vocabulary to study such real-world systems.

Reviewer comment (Evaluation):
Overall, the paper provides throughout several facts about the game dynamics but without explaining the significance of the characterization.

Author response:
>Motivation, significance of the results, and actual applications.

Please refer to the global respond for more explanation of the motivation.

For the significance : Game dynamics particularly based on zero-sum like game are seminal components in standard AI architectures such as GANS [1] (7000+ refs), PSRO [2] (600+ refs) and are actually useful even in n-player games (Diplomacy [3]) The typical way to control such dynamics so far has been to aim/hope for global convergence to equilibrium, however, there are many hurdles to this tasks both from a computational perspective [4,5] as well as from a dynamical systems perspective [6]. Here we aim at a different type of "stability" altogether. Stability/predictability of orbits *without* needing to converge to Nash as a necessary intermediary step from proving stability. By showing that this is possible we open the door for new multi-agent AI architectures that although maybe design around the pipe-dream of global  convergence they can still be practically useful. 

Actual applications: (Computer Poker). Poker is of course a standard benchmark for AI and a real world application.  This direction of work could provide some insights behind the empirical success of CFR+ [7,8] in poker. CFR+ is a variant of Counter Factual Regret (CFR) minimization [9] with improved empirical performance. A key difference between CFR+ and vanilla CFR is switching from simultaneous updates to alternating updates. Our results provide some hints on why this could lead to increased performance. Increased orbit stability, reduced payoff variances due to symplectic-like discretization would lead to faster time average convergence to equilibrium and thus improved performance. Of course our current work is not on CFR and extensive form games but normal form games, nevertheless, we believe these are promising steps that provide a new vocabulary to study such real-world systems.

Reviewer comment (Presentation):
Footnotes should come after punctuation marks.

Author response:
>Minor stylistic issues :

Thank you for pointing out these issues. We have fixed these issues and updated the manuscript in the system. Footnotes are moved after punctuation marks, and the references have been unified in the (authors,year) format. Thank you again!

Reviewer comment (Writing):
The references are not used appropriately; for example, Go Silver et al. (2016) should instead be Go (Silver et al.).

Author response:
>Minor stylistic issues :

Thank you for pointing out these issues. We have fixed these issues and updated the manuscript in the system. Footnotes are moved after punctuation marks, and the references have been unified in the (authors,year) format. Thank you again!