Paper: Stateless Mean-Field Games: A Framework for Independent Learning with Large Populations (ICLR-2024)

Reviewer comment (Theory):
The results in Theorems 1 and 2 discount the effect of mean-field approximation and need more explanations. In my personal understanding, the mean-field approximation is implemented to avoid the dependency of the number of agents $N$ in sample complexity (I understand that in the finite-player setting, the complexity can scale as $\log N$ due to for example the union bound in concentrations). However, Theorems 1 and 2 suggest that $T$ should at least grow as $poly(N)$ for a meaningful result. This implies that the results in this paper cannot be extended to the setting $N=\infty$ considered in the previous papers, e.g., [1]. This may suggest that the algorithms proposed in this paper is not suitable for the large population problem. In addition, the justification for this polynomial dependency by the comparison with [2] below Corollary 2 may be unreasonable, since mean-field approximation is not considered in [2].

Author response:
## Question 1: (regarding dependence on $N$)

We emphasize that (Guo et al. 2019) pointed out by the reviewer and many others in mean-field game literature rely on a very strong oracle/assumption: their algorithms are **centralized** and can **prescribe any behavior to an infinite population** and observe the resulting dynamics while learning (as in a “population generative model”). In a more realistic setting, however, the population distribution will be induced by the behavior of learners interacting with the game over many rounds leading to complicated interactions. We consider the much more challenging and realistic setting where $N$ decentralized learners repeatedly play the game while locally observing interactions. Namely, unlike past work, our setting is:
- **Finite-agent**: learning occurs by playing the finite-agent game, not by centrally simulating an infinite player approximation,
- **Independent/decentralized:** agents can not communicate and their observations are local,
- **Limited feedback:** agents can only partially observe rewards leading to a more realistic learning model.

All in all, the setting of SMFG is much more challenging compared to many works in MFG literature. Our analysis incurs a dependence on $poly(N)$, however, we solve a more challenging and realistic learning problem.

We agree with the reviewer that the setting of multi-player MAB as in Lugosi et al., 2022 can not be directly compared to our setting: in fact, the collision model employed by Lugosi et al., 2022 and others in MMAB literature has a **much more restrictive structure than our setting**. The SMFG incorporates a wide range of complicated payoff functions (including the much simpler binary collision rewards) hence the SMFG problem is much more complicated than the MMAB setting. The reference to Lugosi et al., 2022 served as an analogy for how independent/decentralized learning can introduce a dependence on $N$ even for the much more restrictive setting of collision models.

Finally, while the scaling of the number of iterations to $\mathcal{O}(N^{3/2})$ is indeed a polynomial dependence in the decentralized learning with bandit feedback, it might not be prohibitively large for the large class of payoff functions considered under decentralized learning. The MFG approximation is introduced to tractably solve a difficult problem, but the decentralized learning model nevertheless can introduce a dependence on $N$. Once again, while not a directly comparable result, we point out for example the polynomial $\mathcal{O}(N^{12})$ dependence in the decentralized case for congestion games [5].

## Comparison with Lugosi et al.: 

Regarding the work of Lugosi et al., the reason **we consider their model restrictive is due to the collision model**. In their case, if multiple agents choose a particular arm $k$, the reward obtained is a constant zero, and if an agent alone plays the arm $k$, it obtains a constant reward $r_k$ associated with the arm. In fact, it’s easy to formulate an NE in the setting of Lugosi et al. in the centralized case: we can simply place the $N$ agents to the arms with the highest rewards. There is no need for a mean-field approximation to make the problem tractable, their game model is simple enough.

In our case, the reward map $\mathbf{F}$ is an **unknown nonlinear function of action occupancies** (see beginning of Section 2.1). For instance, the payoff of arm $k$ can arbitrarily depend on how many agents played another arm $k’$ as well as how many agents played $k$. Therefore our game model is much more complicated to solve, even in the centralized learning case it is a computational challenge to formulate an NE. The mean-field approximation makes our problem tractable.

## Regarding MFG and the work of Lasry and Lions

Regarding the seminal work of Lasry and Lions and subsequent research in mean-field RL, we point out that past MFG works solve the continuum limit (the MFG) directly without consideration of the real-world $N$ player game. This is considerably easier.

In other words, there is no dependence on $N$ in past works because there is **no concept of agents**, rather they are simplified to a distribution $\mu$. In this regard past work is **theoretically convenient** but **not realistic**: in the real world, regardless of the mean-field approximation the game played is always an $N < \infty$ player game. This is precisely the contribution of our paper: we could solve MF-VI (page 6) directly easily to obtain a bound similar to past work without a notion of independent agents, but instead for the first time we prove that efficient independent learning with $N < \infty$ agents is possible.

We refer the reviewer to the table provided above to reviewer qRij for a comparison with the past line of work that has followed the seminal paper by Lasry and Lions.

## Sublinear dependence on $N$

We would also like to point out that $N$ appears **sublinearly** in our bounds in Theorem 1 and 2, which is much more tractable compared to an arbitrary polynomial dependence. **Even in games such as potential games or congestion games**, a $poly(N)$ cost is incurred when solving the game, which we reference in our original response. The $\log N$ dependence of Lugosi et al. is possible in the much simple collision model for the rewards, but not in the case when the game has more complicated reward dynamics such as ours.

Reviewer comment (Theory):
The bias term $O(1/\sqrt{N})$ needs more discussion. Although the authors explain that this potentially comes from the independent learning setting, such a term does not appear in other independent learning settings, e.g., the potential games and two-player zero-sum games. I do not fully understand why there is a bias term in the problem setting considered in this paper.

Author response:
## Question 2: (regarding non-vanishing bias term)

The bias term is standard for mean-field games [1,2,3] and can be considered as trading off a small amount of bias/exploitability (when $N$ is large) for a **tractable approximate solution of an otherwise intractable problem**. Our work can be seen as proof that this approximation can be performed without explicitly constructing the $N\rightarrow\infty$ approximate problem by a centralized algorithm but rather independently by participants of the real-world $N$-player game.

We also note that SMFG solves a different class of problems compared to congestion games or potential games as we show in Section B.2 that SMFG in general does not admit a potential structure. While in the highly structured and restrictive settings of potential games and two-player zero-sum games, it is possible to solve the problem exactly, even for symmetric games the task of finding exact NE can be computationally intractable [4].

Reviewer comment (Theory):
Based on the above points, I am not sure whether these concerns arise from the loose analysis of the upper bounds or from the problem formulation itself. It will be helpful to derive the lower bound for these two concerns.

Author response:
## Question 3: (regarding lower bounds)

Developing such lower bounds for multi-agent systems is an active topic of research; such results are sparse even for structured games such as potential games. One result we are aware of is https://arxiv.org/pdf/2303.12287.pdf which provides hardness results of independent learning for Markov games. More relevantly, the works https://arxiv.org/pdf/1511.00785.pdf and https://arxiv.org/pdf/1606.04550.pdf provably show a lower bound on the query complexity of computing $\varepsilon$-NE growing exponentially in the number of players $N$, demonstrating that the mean-field approximation of our SMFG framework for large is relevant. Such lower bound results for SMFG, while being of fundamental interest, are out of the scope of this paper.

Reviewer comment (Theory):
It seems they find an approximate nash equilibrium rather than an exact one.

Author response:
As the reviewer points out, our algorithms produce an approximate NE, however, this bias term is standard for mean-field games [1,2,3] and can be considered as trading off a small amount of bias/exploitability (when $N$ is large) for a **tractable approximate solution of an otherwise intractable problem**. In general, as $N$ gets large (as in many real-world problems) the bias will vanish as our bounds explicitly show.

Reviewer comment (Theory):
Is finding a exact nash equilibrium possible with this methodology?

Author response:
Regarding the question of whether an exact NE can be computed, intractability results exist even in the case the reward function is exactly known and learning is not decentralized [4].

Reviewer comment (Novelty):
There might be significant overlap between the theoretical claims of the paper and what is already known about congestion games, variational inequalities, and game dynamics. Approaching congestion games with variational inequalities instead of explicitly exploiting game potentials (e.g., via best response dynamics) is somewhat standard; the fact that such approaches still work when one relaxes cost functions from being monotone increasing to just monotone is a fairly direct implication—assuming the understanding that SMFG = Congestion Game but with both monotone increasing/decreasing costs is correct.

Author response:
We thank reviewer qRij for their comments.

## SMFG vs. congestion games:

As Appendix B.2 demonstrates, in general, SMFGs are not congestion (or more generally potential) games. While combinations of VI and no-regret dynamics are thoroughly analyzed in structured games such as potential games or for the subclass congestion games, for our class of symmetric games independent learning guarantees have been largely absent. Our work is thus more directly comparable to mean-field games literature.

One parallel can be drawn with non-atomic congestion games which also approximate the infinite-agent limit (Roughgarden et al., 2004), however, we are not aware of any algorithms in this setting where learning is performed by $N$ agents in a decentralized manner. One of the main contributions of the work is utilizing the infinite-player approximation to understand independent learning in the more realistic finite-player setting with explicit finite-sample and finite-agent bounds.

In case the reviewer has particular works for comparison with our setting, we would be happy to provide further differences.

Reviewer comment (Novelty):
One of the main claimed technical contributions is replacing (what is effectively) gradient descent with (what is effectively) L2-regularized gradient descent. The paper claims three benefits of this change: (1) it makes the problem strongly monotone (avoiding the need for the extragradient method), (2) it helps with stochasticity (though convergence despite noise seems to follow from normal stochastic approximation and nothing specific to the regularization), and (3) it allows decentralized implementation across players, which seems to be the primary emphasis. However, this does not appear to be a new observation, as a standard way of learning equilibria in online learnable (such as monotone) setups is through no-regret dynamics.

Author response:
## Independent learning and no-regret dynamics:

While no-regret dynamics are analyzed to provide learning guarantees for various classes of games, they can in general cycle or fail to converge. Despite many centralized learning results with strong oracles (e.g. Perrin et al., 2020 and others mentioned in the paper), independent learning guarantees in MFG literature have so far been understudied. We fill the gap in MFG literature by showing that certain algorithms (combined with particular exploration strategies) can be used to approximately solve large-scale games without communication.

Finally, we note that due to the particular regularization induced by $\tau$, our algorithms are not direct adaptations of no-regret methods. Instead, a particular regularization term depending on population size is required for optimal rates. A similar case can be made for the bandit feedback case: to obtain optimal rates, a particular exploration probability $\varepsilon$ depending on $N$ must be used.

Reviewer comment (Theory):
Theorem 1 should probably state the dependence on \( K \) instead of hiding it as a constant—it seems that the correct dependence should be \(\log(K)\), at least in the full information setting (e.g., using exponentiated gradient descent).

Author response:
## Dependence on $K$:

Finally, we thank the reviewer for pointing out the dependence on $K$ in our bounds. Since the primary focus of our complexity results is to establish the dependencies on the number of agents $N$ and the number of iterations $T$, we move the explicit dependency on $K$ to the appendix due to space considerations. We will make the dependence explicit in the main paper as the reviewer suggested.