Paper: Approximating Nash Equilibria in Normal-Form Games via Stochastic Optimization (ICLR-2024)

Reviewer comment (Presentation):
I do not have any questions, as the presentation of this work is quite clear and intuitive, along with some illustrations to help readers understand it well.

Author response:
Thank you for your encouraging feedback! One of the goals of this paper is to open up the challenge of approximating Nash equilibria to researchers without a predominant game theory background (and in particular, to those with more of a machine learning / optimization background). We are glad to hear that you valued the step-by-step explanation and accompanying illustrations!

Reviewer comment (Theory):
The loss function only captures fully mixed equilibria. The authors address this by considering quantal-response equilibrium. As a result, a zero loss can only serve as an approximation to a Nash equilibrium.

Author response:
Lemma 14 is composed of two terms. The first term is a positive constant that remains even when our proposed loss is zero. Therefore, yes, as you say, this clearly indicates our loss can only be used to approximate Nash equilibria (hence the title of the paper).

Reviewer comment (Experiments):
There is limited empirical evaluation on real-world games. Most experiments involve small synthetic games from previous research. It's possible that more complex games may expose certain limitations.

Author response:
The scope of this work is normal-form games, which is the natural starting point for studying more complex game settings and arguably the bedrock of most traditional economic theory. If by real-world games, you mean games like poker (extensive-form) or some video games (Markov games), then yes, we do not cover that in this work. We mention extensive-form games as one of our future goals in the conclusion.

Reviewer comment (Experiments):
How well does the approach scale to larger real-world games with structures like symmetry? Are techniques such as grouping actions necessary?

Author response:
The structure of symmetric games can be exploited in our approach. Recall that every symmetric NFG has a symmetric NE. Therefore, we can reduce our search space over joint profiles of size $nm$ to a search over a single player’s mixed strategy of size $m$. Moreover, if you inspect the gradient of our loss in equation (9) which contains a sum over all players $k$, we can simplify this as well to a sum over player $\ell$ and any single player $k \ne \ell$ (scaled by $n-1$). Due to symmetry and the fact that all players play the same background strategy in a symmetric NE, each of the summands is the same, so we only need to retain one. See Figure 4 where we explicitly take advantage of the symmetry of the game to both solve it and visualize it; we present all player strategies as a single 1-d variable ($p$) on the x-axis. If instead of player / payoff-symmetry, you meant action-symmetry, then yes, grouping actions would help to scale our approach when two actions are equivalent.

Reviewer comment (Experiments):
Have the authors attempted more advanced optimization methods like Momentum SGD or Adam to address saddle points?

Author response:
We believe this is a very interesting direction for future work. We have tried Adam actually on Blotto (10c, 3f, 4p) by reparameterizing the mixed strategy as a softmax over real-valued logits (vanilla Adam assumes unconstrained optimization). We found that Adam improves on SGD slightly ($\epsilon$ drops from $0.029$ to $0.020$), but we still see a saddle-shape in the landscape when we inspect the loss at Adam’s final step. This suggests Adam helped cope with the saddle point problem, but is not a complete solution. We explicitly decided against including these preliminary results so as to encourage follow-up work by non-convex / deep-learning optimization experts with respect to a variety of non-vanila SGD algorithms such as Adam, Adagrad, saddle-point  avoidant solvers, learning-to-learn approaches, a.o. We believe that the ICLR audience would be the perfect set of experts to take this on.

Reviewer comment (Theory):
What heuristics could be employed to guide equilibrium selection when multiple equilibria exist?

Author response:
A simple heuristic would be to regularize our proposed loss with a second loss term encoding an additional criterion, e.g., maximizing welfare. You could anneal this term over learning to try to find the equilibrium with the highest welfare.

Reviewer comment (Experiments):
The empirical section (6.2) could use some additional explanation/discussion. The baselines that are compared against should be explicitly stated in the body text (and cited appropriately) instead of just stating that they are the baselines used in Gemp et al. 2022's simulations.

Author response:
Thank you for your positive review and for your constructive feedback!

We agree, the empirical section 6.2 is quite short and there are several details (in particular, descriptions of the baselines) in Appendix K that we will pull into the main body.

Reviewer comment (Presentation):
The flow of Section 6 generally could be improved (one suggestion is provided below).

Author response:
Thank you for your positive review and for your constructive feedback!

We also appreciate your suggestion regarding flow. Would you agree with a structure such as this?
- SGD
- - Discussion
- - Experiments
- Bandits
- - Discusision
- - Theory
- - Experiments

Reviewer comment (Writing):
There is a typo in the definition of the projected-gradient (it should be $\Pi_{T\delta^{d-1}}(z) = z - \frac{1}{d} \mathbf{1}^\top z \mathbf{1}$)

Author response:
Thank you for your positive review and for your constructive feedback!

And finally, great catch on the typo. Yes, we are missing the “ones vector”.

Reviewer comment (Presentation):
In Section 6, it is confusing to introduce SGD first and then the bandit algorithm and then immediately have a subsection discussing the bandit algorithm, especially since it is stated that "in the next section, we find it performs well empirically in games previously examined by the literature." Perhaps it makes sense to include the SGD experiments before any discussion of the bandit approach.

Author response:
Thank you for your positive review and for your constructive feedback!

We also appreciate your suggestion regarding flow. Would you agree with a structure such as this?
- SGD
- - Discussion
- - Experiments
- Bandits
- - Discusision
- - Theory
- - Experiments