Paper: Generative Adversarial Equilibrium Solvers (ICLR-2024)

Reviewer comment (Presentation):
The title in the PDF is still the template title.

Author response:
A clash in macro definitions seems to have caused the title to be missing from the paper, this will be fixed for the camera-ready version.

Reviewer comment (Theory):
The paper concentrates on finding a single equilibrium, yet many games have multiple equilibria. Incorporating a discussion on equilibrium selection would enhance the work.

Author response:
We will make sure to include a discussion on equilibrium selection, but we note that a strength of our method is that it does not require practitioners to have to select an equilibrium selection criteria which might be hard to decide.

Reviewer comment (Presentation):
The overall presentation of this paper would benefit from further refinement. The figures within the paper are small and appear blurry due to the absence of vector graphics formats such as .pdf or .svg. Upgrading the figures to vector graphics would improve their clarity and overall visual impact.

Author response:
We will make sure to include graphics which are clearer.

Reviewer comment (Evaluation):
How does the performance of GAES degrade with approximate discriminators in practice? Is there a way to quantify the required discriminator accuracy?

Author response:
**On the effectiveness in approximating the best response.** As we show in additional experiments in Appendix G.1., the accuracy of the discriminator directly impacts the accuracy of the generator, however these two architectures can be tuned independently. We find it interesting to see that 3 of the 5 games can be solved accurately, even when the discriminator is not optimized. This seems to suggest that the accuracy needed for the discriminator depends on the equilibrium structure induced by the payoff structure of the pseudo-game, a direction which future work can explore.

Reviewer comment (Theory):
Could you apply GAES to find other solution concepts like correlated equilibria? How would the formulation need to change?

Author response:
**On generalizing GAES for correlated equilibria.** Yes, GAES can be directly extended to compute correlated equilibria, and in fact to any equilibrium notion which can be expressed as a $\Phi$-equilibrium [1] where $\Phi$ is a set of action deviations. To do so, one can replace the cumulative regret with the $\Phi$-cumulative regret, i.e., the sum of unilateral improvements by deviations within a set $\Phi$-action deviations. In this more general setting, the discriminator would then simply output action deviations within $\Phi$.

Reviewer comment (Theory):
It is often claimed that the method maps pseudo-games to GNE, and it is not clear whether that claim is theoretically sound. Although computing GNE is intractable, the paper alludes (for example in the abstract) that under a distribution over problem instances the problem could be easier, but this is not clearly established.

Author response:
We want to emphasize that the main goal of our method is indeed to compute GNE, and the reason our convergence results are only to a stationary point of the expected exploitability (rather than its minimum) is simply due to our desire to propose broad computational results that apply to nearly all pseudo-games, including those which are PPAD-complete to solve (e.g., the pseudo-games studied in our experiments). If we make stronger assumptions, for instance if in addition to Assumption 1, we assume that the pseudo-game is zero-sum, and the generator is affine in its parameters, then our convergence result implies convergence to a GNE since the expected exploitability then becomes convex. *All this said, we understand your concern and will make the necessary adjustments to our paper to clarify these subtleties.*

Reviewer comment (Theory):
In Footnote 4, it is claimed that the method obtains the state-of-the-art \(O(1/\epsilon^3)\) complexity, an improvement over \(O(1/\epsilon^6)\), stated as the previous state of the art. The authors need to explain more precisely the specific class of problems this applies to, since there are many variants of the PL condition in the literature.

Author response:
**On convergence complexity.** Thank you for pointing out these papers to us, especially "Doubly smoothed GDA for constrained nonconvex-nonconcave minimax optimization." which seems to going to appear at NeurIPS seems very relevant and we were not aware of them. At a high-level, in all three papers, ignoring acceleration-type methods–-which are rarely used in the training of neural networks in practice due to their complexity—the best convergence complexity achieved is $O(\varepsilon^{-4})$ (see for instance [2]), while our algorithm—which makes use of no acceleration techniques—achieves a complexity of $O(\varepsilon^{-3})$. As a result, in the light of these more recent (and some unpublished) results, our algorithm still achieves a faster convergence rate than single-loop algorithms with no-acceleration.

Reviewer comment (Novelty):
Some existing papers seem to obtain much better dependency than claimed in Footnote 4, including: "Faster single-loop algorithms for minimax optimization without strong concavity," "Faster stochastic algorithms for minimax optimization under Polyak-Lojasiewicz Conditions," and "Doubly smoothed GDA for constrained nonconvex-nonconcave minimax optimization." These should be addressed in the discussion.

Author response:
**On convergence complexity.** Thank you for pointing out these papers to us, especially "Doubly smoothed GDA for constrained nonconvex-nonconcave minimax optimization." which seems to going to appear at NeurIPS seems very relevant and we were not aware of them. At a high-level, in all three papers, ignoring acceleration-type methods–-which are rarely used in the training of neural networks in practice due to their complexity—the best convergence complexity achieved is $O(\varepsilon^{-4})$ (see for instance [2]), while our algorithm—which makes use of no acceleration techniques—achieves a complexity of $O(\varepsilon^{-3})$. As a result, in the light of these more recent (and some unpublished) results, our algorithm still achieves a faster convergence rate than single-loop algorithms with no-acceleration.

Reviewer comment (Novelty):
The algorithmic approach is very close to that of the paper "Exploitability minimization in games and beyond," which limits the extent of the algorithmic contribution of the present paper. The authors should highlight the comparisons in more detail.

Author response:
**On the differences between our paper and Goktas and Greenwald’s “Exploitability Minimization in Games and Beyond”.**  
While Goktas and Greenwald observe that the exploitability minimization problem for *only one* pseudo-game can be expressed as a min-max optimization, it is not obviously true that one can minimize the expected exploitability via a min-max optimization problem. The novelty in our method lies in noticing that we can compute the expected exploitability after computing the expected cumulative regret, by optimizing over the space of best-response functions from pseudo-games to actions, rather than the space of actions individually for every pseudo-game.

Reviewer comment (Presentation):
The title of the submission document is the default one.

Author response:
1) A clash in macro definitions seems to have caused the title to be missing from the paper, this will be fixed for the camera-ready version.

Reviewer comment (Presentation):
The references need to be polished — many papers that were published years ago are cited only with the arXiv version, and there is inconsistency in whether URLs are included. These issues should be fixed.

Author response:
2) We chose to cite the Arxiv versions of the paper to allude to the most recent version of the papers. That said, to address your concern we will cite the published versions of the papers when the Arxiv versions we have consulted match the published versions.

Reviewer comment (Presentation):
There are underfull equations in Observation 1 and immediately below it.

Author response:
3) We will fix these equation formatting issues for the camera-ready.

Reviewer comment (Writing):
The relevance of the paper by Daskalakis et al. (2009) in the context of Footnote 4 about min-max optimization is unclear.

Author response:
4) Thank you for pointing this citation out. It seems that we accidentally cited the wrong paper from Daskalakis et al., while we sought to cite [2].

Reviewer comment (Presentation):
The appendix contains many overfull equations that need to be formatted properly.

Author response:
5) Once again, we will fix these equation formatting issues for the camera-ready.

Reviewer comment (Theory):
Assume strong concavity in assumption 1, however, the utility function is not strong convex in Arrow-Debreu competitive economy.

Author response:
Our theorems directly generalize to the concave game settings and as such apply to Arrow-Debreu markets as well; see part 3) of the below answer to your questions and the discussion in lines 235-240 for more information.

Reviewer comment (Theory):
Do not provide guarantee for the performance on the training set.

Author response:
Our results provide computational complexity guarantees on the training dataset (Theorem 4.1), whose average exploitability achieved on the training set generalizes to the test set with a sample complexity result (Theorem 4.2). Even though Theorem 4.1 does not provide a bound on the value of the average exploitability of the training set, it tells us that average exploitability on the training set, cannot be minimized via first-order deviations, hence suggesting the correctness of the algorithm we use. We note that lack of a full characterization of the average exploitability on the training set is a common occurrence in the literature for learning equilibrium mappings (see for instance [4] and [5]).

Reviewer comment (Experiments):
Use different network architecture in two experiments which means GAES is not a general solver for GNE.

Author response:
Just like any neural network, GAES also faces an architecture search problem, and for it to solve the task at hand a search over optimal architectures has to be done. However, this does not affect the generality of GAES. To give a theoretical explanation, according to our sample complexity bounds, the accuracy of GAES depends on the choice of the generator and discriminator hypothesis classes which appear as a "bias" term corresponding to the difference in performance between the optimal generator and discriminator in the class of all possible functions and the optimal ones in the hypothesis class. That is, the choice of a hypothesis class will affect the bias term in performance. As the selection of a neural network architecture effectively determines the hypothesis space we chose, if the architecture of the network is not well-chosen, then the bias of GAES will go up, hence degrading accuracy. This, however, does not take away the fact that with an appropriate neural network architecture GAES is guaranteed to be ``general’’ as the bias term will go down to 0.

Reviewer comment (Evaluation):
Do you measure the difference between the results and the optimal action in the feasible set? Notice that the results “is on average better than at least 99% of the action profiles” in the experiments.

Author response:
The distance between the candidate equilibrium actions outputted by GAES and any GNE is measured in payoff space in terms of the exploitability.  As such the phrasing “is on average better than at least 99% of the action profiles” means that the GAES achieves an exploitability lower than 99% of all other action profiles in the feasible action space.

Reviewer comment (Theory):
Is there any guarantee when the utility function only satisfy convexity?

Author response:
Yes, our theoretical guarantees apply more broadly to any concave game—that is we can drop the strong concavity condition. As mentioned in lines 235-240, our theorems generalize to the concave game settings by replacing the exploitability with the regularized exploitability, however for consistency and simplicity we chose to present our results under these stronger assumptions, since in experiments non-regularized exploitability performed better.

Reviewer comment (Experiments):
Would it be possible to scale GAES to modern games that consists of multiple neural networks (e.g., GANs, multi-agent RL problems, etc.)? If not, what would be the main technical challenges to do so?

Author response:
Thank you so much for your encouraging and thorough review! It means a lot for us!

Regarding your question. GAES extends directly to games $\mathcal{G} \in \Gamma$ in which players are $n \in \mathbb{N}$ “neural players” with each player $i \in \mathbb{N}$ “choosing” an action (i.e., network weights) from an action space (i.e., a subset of $\mathbb{R}^d$ where $d$ is the number of parameters) which maximizes some desired payoff function $u_i^{\mathcal{G}}$. In such settings, GAES can be seen a meta-learning method, where any game $\mathcal{G} \in \Gamma$ is described as parameters of the problem.

For instance, for generative adversarial neural networks for image classification, we have two neural players consisting of a generator and a discriminator for whom the payoff functions are respectively given as the expected negated cross-entropy and expected cross-entropy over a data set of images. As such, each game is characterized by the data set of images, and GAES can be trained over *distributions* of image datasets, so as to output optimal network weights for the generator and discriminator for a given image dataset.