Paper: On the Hardness of Constrained Cooperative Multi-Agent Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
The bound $\Delta$ seems to be trivial? It just comes from the geometric sum and assuming that the risk at each step is less than 1?

Author response:
**A:** Yes, the simple bound $\Delta\le \frac{1}{1-\gamma}$ directly comes from the geometric sum and the assumption that the risk at each step is less than 1. In the revision, we have moved this result to the remark after Theorem 2. The main result of Theorem 2 is to prove the non-zero duality gap.

Reviewer comment (Theory):
I'm not sure I see the only if part of Theorem 7.

Author response:
**A:** Thanks for pointing this out. In the revised paper, we have added multiple bold subtitles to highlight the parts of "if" and "only if" in the proof of Theorem 7.

Reviewer comment (Evaluation):
It would be good to offer some advice on when to use which type of algorithm.

Author response:
**A:** Thanks for the suggestion. When the Q function can be approximated by a factorized form, the primal algorithm is preferred since the advantage gaps $\zeta_k$ in the convergence rate are small. Otherwise, there is no significant difference between these two algorithms based on our current understanding.

Reviewer comment (Experiments):
Some simulation would be helpful to illustrate the results.

Author response:
**A:** Thanks for the suggestion. We are currently conducting MARL experiments in a real world environment and will keep the reviewer posted once the results are available.

Reviewer comment (Theory):
The paper seems to want to say that the problem is NP-hard but stops short. It seems that the quadratic equality is more of an analogy rather than equivalence to standard optimization problems. It would be good to make this more precise.

Author response:
**A:** Thanks for the great suggestion. We agree that it is challenging to figure out the specific complexity class of such a problem, and we think this is an interesting and fundamental problem deserving future research. We also realize that our statement of hardness does not reflect the actual contribution of the paper, as also pointed out by Reviewer WbB4. Therefore, in the revised paper, we have changed the emphasis
of our claim. Specifically, we changed "hardness" to "duality gap" in the title, and changed it to "challenging" or "challenges" throughout the paper. We also changed the emphasis of our main contribution, focusing on the nonzero duality gap and development of algorithms and their convergence results.

Regarding the quadratic equality, our Theorem 1 strictly proved that the original RL problem (1) on the policy $\pi$ is equivalent to the optimization problem (4) on the occupation measure $\nu$. So this is not an analogy.

Reviewer comment (Theory):
For the duality, the authors didn't discuss the connection to the duality of constrained Markov potential games: Provably Learning Nash Policies in Constrained Markov Potential Games. Since constrained cooperative Markov games are a particular case, a non-zero duality gap directly follows.

Author response:
**A:** Thank you very much for pointing out this related work [1], which studies constrained Markov potential game with competing agents.
In the case where all the agents share the same reward function $r_0$, the potential function $\Phi$ of the constrained Markov potential game reduces to the value function $V_0$ associated with $r_0$. Thus, we agree that the non-zero duality gap result in their Proposition 3 also applies to our case. In the revision, we have cited this work and added a remark after Theorem 2 to mention the related results.  

We also want to highlight the differences between our work and this related work [1]. To elaborate, their work focuses on Nash equilibrium of constrained Markov potential game, which is very different from the optimal product policy of constrained cooperative MARL studied in our work. More specifically, in the special case where all the agents share the same reward (i.e., cooperative case), their Nash equilibrium 
corresponds to a feasible product policy $\pi$ where each agent's policy $\pi^{(m)}$ is optimal fixing all the other agents' policies $\pi^{(\backslash m)}$. Such a notion of agent-wise optimal product policy is weaker than the global optimal product policy studied in our work. 

[1] Alatur, P., Ramponi, G., He, N., \& Krause, A. (2023). Provably Learning Nash Policies in Constrained Markov Potential Games. ArXiv:2306.07749.

Reviewer comment (Theory):
Due to the non-zero duality gap, the proposed two algorithms suffer some gaps caused by the multi-agent and constraint coupling. It is more expected that the primal-dual algorithm suffers a duality gap. The primal algorithm has a dependence on an advantage gap that is less expected. It is not clear if these gaps are necessary and which one is better.

Author response:
**A:** Great questions. Our current analysis cannot avoid the duality gap for the primal-dual algorithm and the advantage gap for the primal algorithm. We think the dependence of the primal-dual algorithm on the duality gap is unavoidable due to the nature of primal-dual updates. In general, we believe this is a challenging problem that deserves future research. 

On the other hand, we find that it is possible to remove the advantage gap for the primal algorithm by carefully selecting the initialization point. Specifically, in Example 1, all the advantage gaps $\zeta_0, \zeta_1, \zeta_2>0$, but the primal algorithm can still converge to the optimal solution under certain choices of initialization. Hence, the primal algorithm can be very sensitive to the initialization, indicating that the dependence of the convergence on the advantage gap $\zeta_k$ may not necessarily be tight. This indicates the possibility to address this issue by carefully selecting the initialization, which we leave for future study.

It is unclear whether the duality gap or the advantage gap is better, as neither of the primal-dual and primal algorithms strictly outperform the other one.

Reviewer comment (Experiments):
Experiments are done with artificial examples. It is favorable to check the performance of the two actor-critic algorithms for solving real constrained tasks, and compare it with existing methods as mentioned.

Author response:
**A:** Thanks for the suggestion. We are currently conducting MARL experiments in a real constrained task  
and will keep the reviewer posted once the results are available.

Reviewer comment (Theory):
Is the non-product form of optimal policy the only reason for the duality gap? Does there exist a more fundamental metric that can characterize the cause of the duality gap?

Author response:
**A:** Yes. In fact, even partially product form of optimal policy (e.g., only one agent is independent from all the other agents) can lead to nonzero duality gap. For example, the policy $\pi(a|s)=\pi^{(1,2)}(a^{(1,2)}|s)\pi^{(3)}(a^{(3)}|s)$ is a partially product policy for three agents, where agents 1 and 2 take joint action $a^{(1,2)}$ that is independent from the action $a^{(3)}$ taken by agent 3. Such a (partial) product form of the policy is the only reason that causes the nonzero duality gap.

Reviewer comment (Theory):
As shown in convergence analysis, the effect of the duality gap in different algorithms can be different. Does this suggest a better way to design algorithms? Is it possible to remove such gap dependence?

Author response:
**A:** Great questions. Our current analysis cannot avoid the duality gap for the primal-dual algorithm and the advantage gap for the primal algorithm. We think the dependence of the primal-dual algorithm on the duality gap is unavoidable due to the nature of primal-dual updates. In general, we believe this is a challenging problem that deserves future research. 

On the other hand, we find that it is possible to remove the advantage gap for the primal algorithm by carefully selecting the initialization point. Specifically, in Example 1, all the advantage gaps $\zeta_0, \zeta_1, \zeta_2>0$, but the primal algorithm can still converge to the optimal solution under certain choices of initialization. Hence, the primal algorithm can be very sensitive to the initialization, indicating that the dependence of the convergence on the advantage gap $\zeta_k$ may not necessarily be tight. This indicates the possibility to address this issue by carefully selecting the initialization, which we leave for future study.

It is unclear whether the duality gap or the advantage gap is better, as neither of the primal-dual and primal algorithms strictly outperform the other one.

Reviewer comment (Theory):
The advantages of the two algorithms are discussed in terms of policy iterates, which are stronger than the output policy in algorithms. Is it possible to show them in convergence theory?

Author response:
**A:** We think the reviewer was referring to the last-iterate convergence of the policy $\pi_T$. If so, we think it is possible to prove this result by leveraging some existing techniques. Specifically, for the primal-dual algorithm, the existing work [2] has established last-iterate convergence in single-agent constrained RL by introducing a **regularized** Lagrangian function. This idea can be applied to our multi-agent setting to prove the last-iterate convergence. For the primal algorithm, we may consider introducing entropy-regularization to the natural policy gradient (NPG) updates, 
since entropy-regularized NPG has  been shown to have last-iterate convergence in single-agent RL [3].

[2] Ding, D., Wei, C. Y., Zhang, K., \& Ribeiro, A. (2023). Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs. ArXiv:2306.11700.

[3] Cen, S., Cheng, C., Chen, Y., Wei, Y., \& Chi, Y. (2022). Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research, 70(4), 2563-2578.

Reviewer comment (Novelty):
As mentioned, prior works also studied constrained cooperative Markov games and the authors have improved the analysis. Can the authors illustrate the analysis differences due to the lack of zero duality gap? It is useful if the authors could compare assumptions and results with them in a table.

Author response:
**A:** Thanks for the great suggestion. To the best of our knowledge, no existing approaches for constrained cooperative MARL has convergence guarantee on the optimality gap. Specifically, as we discussed after Q1 in the introduction, the existing works only obtained  either convergence of gradient norm (weaker than the optimality gap), or convergence of only part of the policy instead of the full algorithm. In the revision, we have added Table 1 to summarize this comparison.

Reviewer comment (Theory):
A weakness of the authors' result is the lack of a definitive answer as to the hardness of the problem of Constrained Cooperative MARL. Indeed, the fact that the optimization program corresponds to a mathematical program with bilinear constraint functions is an indication of its potential hardness, yet there is a multitude of refined computational complexity classes that it could belong to. Is the problem *total*, i.e., is the problem guaranteed to have a solution? If so, it will belong to the TFNP complexity class. Then, does it belong to some known classes such as PPAD, PPA, CLS, PLS?

Author response:
Good question. The optimization problem in Theorem 1 has a solution since the feasible set is compact and the objective function is continuous. However, we do agree that it is challenging to figure out the complexity class of such a problem, and we think this is an interesting and fundamental problem deserving future research.

Reviewer comment (Presentation):
The narrative and even the title of the introductory text could benefit by stressing the contribution of extending previous results that considered correlated policies to quantifying the effect of being restricted to product policies, rather than putting the focus on the hardness, since there is no definitive answer of the computational complexity.

Author response:
We really appreciate the reviewer's great suggestion on changing the emphasis of our claim. We do agree that our statement of hardness does not reflect the actual contribution of the paper. In the revised paper, we have 
changed "hardness" to "duality gap" in the title, and changed it to "challenging" or "challenges" throughout the paper. We also changed the emphasis of our main contribution, focusing on the nonzero duality gap and development of algorithms and their convergence results.

Reviewer comment (Theory):
The assumption of the decomposition schema of the Q-function, that the advantage gaps depend on, seems limiting. Is the assumption necessary to ensure the computational tractability of the problem? Moreover, in practice, similar assumptions (on the linear decomposition of the Q-function) can harm performance, even in the unconstrained setting (e.g. see the comparison between VDN and QMIX (Rashid et al. 2020)).

Author response:
**A:** Great question. Note that the convergence result in Theorem 4 does not rely on this assumption. Moreover, we showed in Appendix H that this assumption is necessary and sufficient to ensure a vanishing advantage gap $\zeta_k$. We agree with the reviewer that this assumption is restricted and may fail to hold in practical scenarios, and this echos our key point that constrained cooperative MARL problems can be challenging to solve.

Reviewer comment (Experiments):
It is not clear by the authors if the paper can be compared with other state-of-the-art algorithms (if exist) in terms of the optimality gap (in the constrained cooperative MARL setting).

Author response:
**A:** Good question. To the best of our knowledge, no existing approaches for constrained cooperative MARL has convergence guarantee on the optimality gap. Specifically, as we discussed after Q1 in the introduction, the existing works only obtained  either convergence of gradient norm (weaker than the optimality gap), or convergence of only part of the policy instead of the full algorithm. In the revision, we have added Table 1 to summarize this comparison.

Reviewer comment (Theory):
In the proof of Theorem 3, the authors have assigned $\lambda_k$ with a value larger than $\lambda_{k,{max}}$ (page 19). Can the authors be more explicit about why they are able to assign $\lambda_k$ with such a value?

Author response:
**A:** Thank you very much for pointing out this and we have fixed it in the revised paper. Specifically, we changed the value of $\widetilde{\lambda} _ k$ from $2\lambda _ {k,\max}I\{V_k(\pi_t)\le \xi_k\}$ to $\lambda_{k,\max}I\{V_k(\pi_t)\le \xi_k\}$. Accordingly, we have doubled the value of $\lambda _ {k,\max}$ in Lemma 1 and showed that the optimal dual variable $\lambda_k^*\le \frac{1}{2}\lambda_{k,\max}$ (not $\lambda _ {k,\max}$).

Reviewer comment (Theory):
In the proof of Theorem 7, can the authors be more explicit about the proof steps and explain what $\pi_{\omega}$ and $\omega$ are?

Author response:
**A:** Thanks for checking our proof so carefully. In our revised paper, we have added bold subtitles to highlight the proofs of "if" and "only if" parts. We also added more intermediate steps and explanations to the proof of ``if'' part (i.e. the Q-function decomposition implies $\zeta_k=0$). We have also changed $\pi_{\omega}$ and $\omega$ to $\pi$. Thanks for pointing out this inconsistent notation.