Paper: Sample-Efficient Multi-Agent RL: An Optimization Perspective (ICLR-2024)

Reviewer comment (Writing):
I think that the paper should make it clearer that MAMEX is not computationally efficient.

Author response:
(i). The equilibrium computation over the set of pure policies.

The policy evaluation step refers to Lines 3 and 4 in MAMEX (Algorithm 1). If we regard these two lines as an oracle, MAMEX is oracle-efficient. The computational complexity of this oracle depends on the type of equilibrium. For CCE, the oracle can be approximately implemented by mirror descent [1], where the time of calculating $V(\pi)$ depends on $L$ and $L$ is the number of iterations. Hence, the computational complexity of this oracle depends only on the number of iterations rather than the joint pure policy space $|\Pi|$, and MAMEX can be calculated efficiently. However, for Nash Equilibrium, calculating the equilibrium is PPAD-complete. So MAMEX is computationally inefficient for learning NE.

(ii). The sup over the function class in the policy evaluation routine

We believe that it can be solved by using similar techniques in [2], where they alternately update the policy and the hypothesis function (the model parameter in the model-based version or the Q-function in the model-free setting). Particularly, we apply gradient ascent when updating the hypothesis function. In MAMEX, if we use policy iteration or soft policy iteration to solve a CCE, their alternate updates are also available here.

Reviewer comment (Theory):
The lack of computational efficiency is due to (i) the equilibrium computation over the set of pure policies in step 4 of Algorithm 1, which is often over combinatorial sets like for example the set of deterministic policies in the tabular setting, and (ii) the sup over the function class in the policy evaluation routine.

Author response:
(i). The equilibrium computation over the set of pure policies.

The policy evaluation step refers to Lines 3 and 4 in MAMEX (Algorithm 1). If we regard these two lines as an oracle, MAMEX is oracle-efficient. The computational complexity of this oracle depends on the type of equilibrium. For CCE, the oracle can be approximately implemented by mirror descent [1], where the time of calculating $V(\pi)$ depends on $L$ and $L$ is the number of iterations. Hence, the computational complexity of this oracle depends only on the number of iterations rather than the joint pure policy space $|\Pi|$, and MAMEX can be calculated efficiently. However, for Nash Equilibrium, calculating the equilibrium is PPAD-complete. So MAMEX is computationally inefficient for learning NE.

(ii). The sup over the function class in the policy evaluation routine

We believe that it can be solved by using similar techniques in [2], where they alternately update the policy and the hypothesis function (the model parameter in the model-based version or the Q-function in the model-free setting). Particularly, we apply gradient ascent when updating the hypothesis function. In MAMEX, if we use policy iteration or soft policy iteration to solve a CCE, their alternate updates are also available here.

Reviewer comment (Theory):
In the tabular case or in the Linear MDP case would there exist an efficient way to implement MAMEX?

Author response:
By replacing the optimistic value function $\overline{V}_i^{(t),\pi}$ in Algorithm 4 (DOPMD) of [1] by the regularized payoff function in (3.1), one can get an approximated CCE by mirror descent using FTRL analysis in Section 6.6 in [4]. To be more specific, using the FTRL analysis, the bound of (a) in Section I.3 of [1] can be transformed to the following inequality

$$\max_{\pi_i \in \Pi_i}\sum_{t=1}^T [\overline{V}\_i(\pi_i \times \pi_{-i}^t) - \overline{V}\_i(\Lambda_i^t \times \pi_{-i}^t)] \le O(H\sqrt{\log |\Pi_i|T} ).$$
Then by Azuma-Hoeffding's inequality, we can have
$$ \begin{aligned}\max_{\pi_i \in \Pi_i}\sum_{t=1}^T [\overline{V}\_i(\pi_i \times \Lambda_{-i}^t) - \overline{V}\_i(\Lambda_i^t \times \Lambda_{-i}^t)] &\le \max_{\pi_i \in \Pi_i}\sum_{t=1}^T [\overline{V}\_i(\pi_i \times \Lambda_{-i}^t) - \overline{V}\_i(\Lambda_i^t \times \Lambda_{-i}^t)] + O\left(H\sqrt{T\log (\sum_{i \in [n]}|\Pi_i|/\delta)}\right)\le O\left(H\sqrt{T\log (\sum_{i \in [n]}|\Pi_i|/\delta)}\right).\end{aligned}$$

From the inequality above, the mixture policy $\Lambda^{out}=\frac{1}{T}\sum_{t=1}^T \Lambda^t_1\times \cdots \times \Lambda_n^t$ is the approximated CCE by 
$$\max_{\pi_i \in \Pi_i}\overline{V}\_i(\pi_i \times \Lambda_{-i}^{out})-\overline{V}\_i(\Lambda_i^{out} \times \Lambda_{-i}^{out}) \le \max_{\pi_i \in \Pi_i}\sum_{t=1}^T [\overline{V}\_i(\pi_i \times \Lambda_{-i}^t) - \overline{V}\_i(\Lambda_i^t \times \Lambda_{-i}^t)]\le O\left(H\sqrt{\log (\sum_{i \in [n]}|\Pi_i|/\delta)/T}\right)$$
if $T$ is large.

In general, the DOPMD is not computationally efficient since the calculation of the payoff function $\overline{V}$ is as hard as solving nonconvex optimization. However, there are some simple cases in which the algorithm can be computationally efficient.

For example, if we assume the class of value function as a linear function class $V_{f}^{(i),\pi}(s) = \langle Q_f^{(i)}(s,a),\pi\rangle$ and $Q_f^{(i)}(s,a) = \langle \phi^{(i)}(s,a), \theta_f\rangle$, and the empirical loss function $L^{(i),k}(f,\pi,\tau)$ is a quadratic function of $\theta_f$. Then, the calculation of the payoff function in (3.1) can be implemented efficiently because it is given by maximizing a concave quadratic function of $\theta_f$. Moreover, the sampling procedure in Line 3 of DOPMD can be implemented efficiently by Langevin dynamics ([2]) as long as we can obtain the gradient of $\overline{V}\_i(\pi)$ in (3.1) with respect to $\pi$, i.e., the policy gradient. By using Langevine dynamics, we can sample $\Lambda_i^{t+1} \propto_{\pi_i} \Lambda_i^t \cdot \exp(\eta_i \cdot \overline{V}\_i(\pi_i\times \pi_{-i}^t))$ in Line 6 of DOPMD by the following iteration:

$$\pi^{(k+1)} = \pi^{(k)} - \nabla_{\pi^{(k)}} \log (\Lambda_i^0)-\alpha \sum_{t'=1}^t\nabla_{\pi^{(k)}} (\eta_i \cdot \overline{V}\_i(\pi^{(k)}\times \pi_{-i}^t)) + \sqrt{2\alpha}\epsilon^{(k)},$$

where $\alpha$ is the step-size and $\varepsilon^{(k)}$ is a  sequence of standard normal vectors.

Hence, it avoids calculating  $\overline{V}\_i(\pi)$ for all policies $\pi \in \Pi^{\text{pur}}$. In fact, we can parameterize the policy $\pi_i$ as $\pi_\theta$, and thus the loss function $L^{(i),k}(f,\pi,\tau)$ is quadratic in $\theta$ and $V_{f}^{(i),k,\pi}(\rho)$ is linear in $\theta$. Then, the regularized payoff function $\overline{V}\_i(\pi_i \times \pi_{-i}^t)$ is always a differentiable function with respect to $\pi_i$, and the policy distribution $\Lambda_i^{t}$ is always log-concave, which guarantees that the Langevin dynamics converges to the true distribution [3].

Reviewer comment (Theory):
Do you think that the MADC are necessary quantities that should appear in lower bounds too?

Author response:
We think the MADC is a necessary quantity that should appear in lower bounds. However, even for the single-agent DC ([3]), the lower bound is still lacking. We leave it as a future work.

Reviewer comment (Novelty):
While I like the results presented in this paper, it is not very clear to me what the relationship between MADC and the existing measures (such as MADMSO, the single agent decoupling coefficient). I believe the paper would benefit from more discussion and comparison on that.

Author response:
In general, we do not see a clear relation between MADC and MADMSO, in the sense that our complexity measures cannot be reduced to theirs and vice versa. However, when applying to some specific examples, the inducing results are comparable. For example, when applying to the linear MG with dimension $d$,  the paper [3] achieves a $\widetilde{O}(n^2d^2H^4\max_{i=1}^n \log |\Pi_i|/\varepsilon^2)$ sample complexity, while our paper can achieve a $\widetilde{O}(d^2H^2 + n^2H^2(\max_{i=1}^n\log |\Pi_i|)^2/\varepsilon^2)$ sample complexity.
 Compared to [3],  our result has a higher degree in term $\max_{i=1}^n \log|\Pi_i|$. When $\max_{i=1}^n \log |\Pi_i|$ is relatively small and can be ignored, our $\widetilde{O}((n^2+d^2)H^2/\varepsilon^2)$ sample complexity is smaller than the sample complexity result $\widetilde{O}(n^2d^2H^2/\varepsilon^2)$ in [3]. In addition,  our complexity measure also captures model-free settings, while the algorithm and the complexity measure MA-DEC in [3] heavily depend on the model and cannot be directly applied to the model-free setting.

Reviewer comment (Theory):
When $n = 1$, the problems are reduced to a single agent RL. In this case, is MADC reduced to DC, or is it more general than DC? (since it is not restricted to greedy policy)

Author response:
It is more general than DC since the policy $\pi^k$ in (2.3) is not restricted to the greedy policy. In other words, the MADC in $n=1$ is not less than DC. A function class of MDP with small DC can also have large MADC for $n=1$.

Reviewer comment (Theory):
If MADC reduces to DC when $n=1$, does an RL instance with finite DC imply the MG version of the RL instance also has finite MADC (and thus can be solved efficiently?)

Author response:
There is no nontrivial example in which MADC is infinite since MADC in tabular MG is also finite ($O(|S||A|)$) when states and actions are finite.  However, an RL instance with a small DC cannot imply the MG version of the RL instance also has a small MADC, since the policy $\pi^k$ is not restricted to the greedy policy. Hence, it needs to further check whether the multi-agent version of the RL instance satisfies the Equation (2.3) for every policy sequences $\{\pi^k\}_{k \in [K]}$.

Reviewer comment (Theory):
How should one compare MADC and MADMSO? We know that tabular MG and linear mixture MG have finite MADMSO and MADC. Are there any classes of games that have finite MADC but not finite MADMSO or vice versa?

Author response:
In general, we do not see a clear relation between MADC and MADMSO, in the sense that our complexity measures cannot be reduced to theirs and vice versa. However, when applying to some specific examples, the inducing results are comparable. For example, when applying to the linear MG with dimension $d$,  the paper [3] achieves a $\widetilde{O}(n^2d^2H^4\max_{i=1}^n \log |\Pi_i|/\varepsilon^2)$ sample complexity, while our paper can achieve a $\widetilde{O}(d^2H^2 + n^2H^2(\max_{i=1}^n\log |\Pi_i|)^2/\varepsilon^2)$ sample complexity.
 Compared to [3],  our result has a higher degree in term $\max_{i=1}^n \log|\Pi_i|$. When $\max_{i=1}^n \log |\Pi_i|$ is relatively small and can be ignored, our $\widetilde{O}((n^2+d^2)H^2/\varepsilon^2)$ sample complexity is smaller than the sample complexity result $\widetilde{O}(n^2d^2H^2/\varepsilon^2)$ in [3]. In addition,  our complexity measure also captures model-free settings, while the algorithm and the complexity measure MA-DEC in [3] heavily depend on the model and cannot be directly applied to the model-free setting.

Reviewer comment (Theory):
It seems that the current framework does not distinguish NE/CCE/CE. It seems that as long as the NE/CCE/CE equilibrium oracle is efficient, it takes the same amount of samples to find NE/CCE/CE. As NE is a much stricter notion of equilibrium than CCE, for example, one might expect more samples to be needed (or in other words, harder) to learn NE. So are the results tight for each of NE/CCE/CE or is it an upper bound for the worst-case regret needed for finding any equilibrium?

Author response:
The results is not tight for each of NE/CCE/CE. For example, one can use the decentralized function approximation to learn CCE with polynomial sample complexity for tabular general-sum MG ([1]), while any general framework that learns NE/CCE/CE ([2,3,4,5] and our paper) suffers an exponential term $\prod_{i=1}^n |A_i|$ for the tabular general-sum MG due to the intrinsic difficulty of learning NE. 


Instead of getting the optimal result in one particular equilibrium, our goal is to provide an unified algorithmic framework that learns NE/CCE/CE for both model-based and model-free RL under the general function approximation.

Reviewer comment (Novelty):
I would like to see the authors compare / contrast their approach with PSRO [1, 2]. PSRO also consists of two components (single agent optimization to compute a best-response and computing the equilibrium of a normal-form game), applies to Markov games, leverages function approximation, and can learn CCE, CE, and NE.

Author response:
> 1. Comparison between MAMEX and PSRO
 
The PSRO learns the equilibrium in the following way: At first, every player chooses a uniform policy as their strategy. The algorithm then calculates the equilibrium by a meta-solver, and trains an oracle that outputs the best response $\pi_i$ of the equilibrium for player $i$. After that, the algorithm adds $\pi_i$ into the strategy space of the player $i$. Last, the algorithm simulates all the new joint policy and construct a  new normal-form game for the next iteration. The algorithm can finally converge to an equilibrium.   However, in each iteration, it should simulate all the new joint policies and estimate the return. Consequently, the sample complexity increases exponentially as the iteration rounds increase. 


Different from PSRO, MAMEX utilizes the function approximation technique to the value function. The precise characterization of the structure of the value function can help us to evaluate the policy without actually simulating the environment with more samples. To be more specific, at each round, instead of simulating the environment and getting a Monte-Carlo return of each joint policy, MAMEX only needs to solve a regularized optimization problem over the function space of the value function. The solution of the optimization problem is used to be a payoff for the normal-form game. Since solving this optimization subproblem does not need additional samples, MAMEX bypasses the requirement for exponential samples to simulate the environment and estimate the value for each joint policy $\pi$. This characteristic enhances its sample efficiency in comparison to PSRO.

Reviewer comment (Novelty):
Note I am increasing my score conditioned on some discussion of the limitations of scaling this approach to practical sized games (see my last follow-up comment) as well as including a discussion of a comparison with PSRO in related work.

Author response:
>2. Comparison with PSRO and EGTA

 The comparisons of EGTA are shown below. We have added the comparisons of PSRO and EGTA in the related work section in Appendix A.

The EGTA uses a graph to represent the deviation for all profiles, and then identify the Nash equilibrium and whether a  profile is relatively stable. Note if we want to evaluate all profiles and construct the whole graph,  we need many samples to estimate the payoff functions for each profile, and then identify all the deviations and construct the graph.

The authors also mention that if one cannot evaluate all the profiles because of the lack of samples, one possible approach is to apply machine learning or regression techniques to fit a payoff function over the entire profile space given the available data. In fact, we use the function approximation technique to derive an estimate of all the profiles without estimating all the profiles. Thus, our approach can be considered as a learning approach to evaluate all the profiles without simulating environments using many samples.

Reviewer comment (Writing):
Pure Policy: I was confused by the definition of pure policies. You state that the set of pure policies is a subset of each agent's local policies, but this is still too vague. I assume a pure policy is a deterministic mapping from states to actions, but can you clarify / confirm this?

Author response:
> 2. Pure Policy set: 

The pure policy $\Pi_i$ for each player $i$ is a concrete policy set, so an agent can follow any mixture of the pure policy $\Delta(\Pi_i)$. The equilibrium is defined over the mixture of these pure policies. That is, each pure policy can be viewed as a pure strategy of the normal form game, and each joint policy can be viewed as a mixed strategy. 

For example, if the pure policy set is the deterministic policy set, the agent can execute any mixture of the deterministic policy. The equilibrium is then defined on the mixture of the deterministic policy, which reduces to the standard equilibrium defined in a normal-form game. 

If the pure policy set for each player only consists of the log linear policy $\Pi_i = \\{\pi_\theta: \pi_\theta(\cdot \mid s) = \text{Softmax}(\theta^T\phi(s,\cdot)), \theta \in \mathbb{R}^d, \|\theta\|_2 \le 1\\}$, then one agent can only execute the mixture of the log linear policy. Now the equilibrium is defined in a $|\Pi_1| \times |\Pi_2| \times \cdot \times |\Pi_n|$ multi-player normal-form game, where each log linear policy in $\Pi_i$ is a strategy for player $i$, and each joint log linear policy has a payoff.

Reviewer comment (Writing):
Equation 3.1: Can you state somewhere that $\eta > 0$ is a hyperparameter that you are introducing?

Author response:
> 3. Can you state somewhere that $\eta > 0$ is a hyperparameter that you are introducing?

In Theorem 3.1, we choose $\eta = 4/\sqrt{K}$.

Reviewer comment (Presentation):
Can you please bring Algorithm 1 into the main body?

Author response:
> 4. Can you please bring Algorithm 1 into the main body?

Thanks for your reminder. We will put Algorithm 1 in the main body in our revised version.

Reviewer comment (Theory):
You say "We prove that MAMEX achieves a sublinear regret for learning NE/CCE/CE in classes with small MADCs", but haven't you proven sublinear regret regardless of how large the MADC is (as long as it is finite, Assumption 2.7)? Do you have an interesting example in which MADC is infinite?

Author response:
> 5. You say "We prove that MAMEX achieves a sublinear regret for learning NE/CCE/CE in classes with small MADCs", but haven't you proven sublinear regret regardless of how large the MADC is (as long as it is finite, Assumption 2.7)? Do you have an interesting example in which MADC is infinite?


Thanks for your reminder. It is better to say "We prove that MAMEX achieves a sublinear regret for learning NE/CCE/CE, and the regret is smaller when the MADC of the function class is smaller."  In other words, the bound becomes vacuous when MADC is infinity. 

There is no nontrivial example in which MADC is infinite since MADC in tabular MG is also finite ($O(|S||A|)$).  Of course, an MG with infinite states and no structure has infinite MADC.

Reviewer comment (Theory):
To me, there lacks motivation to study equilibrium learning in a centralized manner, particularly when it does not consider any global value optimization. Equilibrium seems to be a concept under which selfish players cannot make unilateral move, and is usually used to characterize the steady state when every player plays independently and selfishly. However, if the players are in coordination, perhaps they can aim higher, such as higher social welfare. Can you give more motivations on centralized learning NE/CE/CCE without considering social welfare? A related question is: why is the Reg_{NE/CE/CCE} a meaningful measure for the quality of a centralized MARL algorithm?

Author response:
> 1. Motivation to study equilibrium learning in a centralized manner

Centralized or decentralized learning is a separate question from whether the game is cooperative or competitive. For example, in a noncooperative game, each agent might first run a centralized RL algorithm on a game environment to learn the equilibrium policy $\hat \pi$, and then deploy her component $\hat \pi^{(i)}$. Here this environment takes the joint policy as the input and outputs the trajectory. Therefore, it is crucial to investigate the equilibrium for a multi-agent RL problem, which has been also studied in recent years ([1,2,3,4,6,7]).

To be more specific, there are three motivations for learning the equilibrium in a centralized manner:

* AlphaZero ([5]): It learns a Nash equilibrium in a two-player zero-sum Markov Game to learn how to compete with the opponent. The **centralized** RL algorithm is deployed to the two-player zero-sum Markov game and a policy $\pi^{(1)}$ and $\pi^{(2)}$ are learned. After the algorithm converges, the policy $\pi^{(1)}$ is deployed to play against human. 

* Financial Market: When $n$ market participants choose the best strategies based on the equilibrium, the market may reach a relatively stable state. So they have enough motivation to learn an equilibrium $(\pi^{(1)},\cdots,\pi^{(n)})$ by following a centralized algorithm, and the $i$-th market participants take the policy $\pi^{(i)}$. 

Furthermore, if the goal of the agents is to achieve social welfare, our algorithm can be modified to learn the policy with optimal social welfare. In particular, we only need to change the equilibrium-solving oracle to an oracle that maximizes the social welfare

Reviewer comment (Novelty):
There are some previous works also on quantifying sample complexity of multi-agent RL, including Wang et al. 2023, Foster et al., 2023, Chen et al. 2022b. In this work, the comparisons to these previous works are all about the "simplicity" of the objective. The comparisons on "sample complexity", however, are missing. How to compare the new notion MADC with the complexity measure proposed in these works? Do they have any relations?

Author response:
> 2. comparisons in the "sample complexity" side and relations between several complexity measures

* Compared to the previous works that study the general function approximation of the general-sum MGs ([1,2,3]).

[2] and [3] both study the model-based RL problem with centralized function approximation. Since [2] only studies the PAC learning setting with sample complexity results instead of the regret result,  to compare with them, we use our sample complexity result in Corollary D.4 to give a comparison. To learn a $\varepsilon$-NE/CCE/CE, both our results and their results have the form $O(\text{poly}(d_{\text{parameter}},\log |\Pi|, \log |M|)/\varepsilon^2)$, where $M$ is the model and $d_{\text{parameter}}$ is the complexity measure (MADC in our paper, MADEC in [2,3]). When applying to the tabular MG, both our work and their works suffer an exponential term $\prod_{i=1}^n A_i$, due to the centralized function approximation. When applying to the linear MG with dimension $d$,  the paper [2] achieves a $\widetilde{O}(dH^2\log |M|/\varepsilon^2)$ sample complexity, the paper [3] achieves a $\widetilde{O}(n^2d^2H^4\max_{i=1}^n \log |\Pi_i|/\varepsilon^2)$ sample complexity, while our paper can achieve a $\widetilde{O}(d^2H^2 + n^2H^2(\max_{i=1}^n\log |\Pi_i|)^2/\varepsilon^2)$ sample complexity. Compared to [2], the term $\log|M| \ge\Omega( A)$ (using an appropriate discretization ([8])) usually suffers the curse of multi-agency, while our result avoids this term and does not suffer the curse of multi-agency. Compared to [3],  our result has a higher degree in term $\max_{i=1}^n \log|\Pi_i|$. When $\max_{i=1}^n \log |\Pi_i|$ is small and can be ignored, our $\widetilde{O}((n^2+d^2)H^2/\varepsilon^2)$ sample complexity is smaller than the  result $\widetilde{O}(n^2d^2H^2/\varepsilon^2)$ in [3].

When applying to the linear mixture MG, the work [2] derives a $\widetilde{O}(d^2H^3/\varepsilon^2)$ sample complexity, our paper achieves a smaller $\widetilde{O}((n^2+d^2)H^2/\varepsilon^2)$ sample complexity, and [3] does not derive an accurate result.

Moreover, both [2] and [3] cannot be easily extended to the model-free settings since (a) their algorithms involve an optimization subproblem (Line 4 of Algorithm 9 in [2], Line 5 of Algorithm 1 in [3]) or a sampling procedure (Line 6 of Algorithm 9 in [2]) over the model space, and (b) their essential complexity measures (MDEC in [2], MA-DEC in [3]) heavily depends on the model. In contrast, our complexity measure MADC is flexible to be applied to both model-free RL and model-based RL.

The recent work [1] uses a decentralized function approximation, so they provide a polynomial sample complexity result for learning CCE for the tabular general-sum MG. However, they do not provide a regret result, and the function approximation results are not directly comparable due to the different decentralized function approximation.

* Relations 

We do not see a clear relation in general, in the sense that our complexity measures cannot be reduced to theirs and vice versa.   However, when applying to some specific examples, the regret of our work and previous works is comparable.   In addition,  our complexity measure also captures model-free settings.

Reviewer comment (Writing):
There are some typos in the paper (e.g. page 8 solveing).

Author response:
Thanks for your reminder. We will fix them in our next version. If we have additional space, we will move the definition of $\ell$ and the algorithm code into the main body.

Reviewer comment (Writing):
It is not easy to understand the paper without looking at the appendix. For example, it would be better to include the definition of $\{l^{i,s}\}_{i \in [n]}$ in the main paper.

Author response:
Thanks for your reminder. We will fix them in our next version. If we have additional space, we will move the definition of $\ell$ and the algorithm code into the main body.

Reviewer comment (Evaluation):
How do the authors compare the regret results with previous SotA algorithms?

Author response:
* Compared to the previous papers that study the general function approximation of the general-sum MGs ([1,2,3]).

Two papers [2] and [3] both study the model-based RL problem with centralized function approximation. Since [2] only studies the PAC learning setting with sample complexity results instead of the regret result,  to compare with them, we use our sample complexity result in Corollary D.4 to give a comparison. To learn a $\varepsilon$-NE/CCE/CE, both our results and their results have the form $O(\text{poly}(d_{\text{parameter}},\log |\Pi|, \log |M|)/\varepsilon^2)$, where $M$ is the model and $d_{\text{parameter}}$ is the complexity measure (MADC in our paper, MADEC in [2,3]). When applying to the tabular MG, both our work and their works suffer an exponential term $\prod_{i=1}^n A_i$, due to the centralized function approximation. When applying to the linear MG with dimension $d$,  the paper [2] achieves a $\widetilde{O}(dH^2\log |M|/\varepsilon^2)$ sample complexity, the paper [3] achieves a $\widetilde{O}(n^2d^2H^4\max_{i=1}^n \log |\Pi_i|/\varepsilon^2)$ sample complexity, while our paper can achieve a $\widetilde{O}(d^2H^2 + n^2H^2(\max_{i=1}^n\log |\Pi_i|)^2/\varepsilon^2)$ sample complexity. Compared to [2], the term $\log|M| \ge\Omega( A)$ (using an appropriate discretization ([8])) usually suffers the curse of multi-agency, while our result avoids this term and does not suffer the curse of multi-agency. Compared to [3],  our result has a higher degree in term $\max_{i=1}^n \log|\Pi_i|$. When $\max_{i=1}^n \log |\Pi_i|$ is relatively small and can be ignored, our $\widetilde{O}((n^2+d^2)H^2/\varepsilon^2)$ sample complexity is smaller than the  result $\widetilde{O}(n^2d^2H^2/\varepsilon^2)$ in [3].

When applying to the linear mixture MG, the work [2] derives a $\widetilde{O}(d^2H^3/\varepsilon^2)$ sample complexity, our paper achieves a smaller $\widetilde{O}((n^2+d^2)H^2/\varepsilon^2)$ sample complexity, and [3] does not derive an accurate result.

Moreover, both [2] and [3] cannot be easily extended to the model-free settings since (a) their algorithms involve an optimization subproblem (Line 4 of Algorithm 9 in [2], Line 5 of Algorithm 1 in [3]) or a sampling procedure (Line 6 of Algorithm 9 in [2]) over the model space, and (b) their essential complexity measures (MDEC in [2], MA-DEC in [3]) heavily depends on the model. In contrast, our complexity measure MADC is flexible to be applied to both model-free RL and model-based RL.



* Compared to the previous SoTA algorithm for specific examples.

For zero-sum linear MG, the previous work [4] provides a $\widetilde{O}(d^{3/2}H^2\sqrt{K})$ regret when $\sum_{h=1}^H r_h \in [0,H]$, while MAMEX achieves a $\widetilde{O}(dH^3\sqrt{K}+H^3\sqrt{K}\log |\Pi|)$ regret. When the pure policy class for each player is the log linear policy class with dimension $d_\pi$, MAMEX yields a $\widetilde{O}((d+nd_\pi)H^3\sqrt{K})$ regret.

For zero-sum linear mixture MG, the previous work [5] provides a minimax-optimal $\widetilde{O}(dH\sqrt{K})$ regret, while MAMEX yields a $\widetilde{O}(dH^5\sqrt{K})$ regret. Thus our regret matches their results in terms of $d$ and $K$, with an additional factor $H^4$.