From: ICLR.cc/2023/Conference/Paper1610/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper1610/Authors
title: Resonpse to Reviewer UQcj
comment: Dear Reviewer UQcj,

Thanks for your time spent on it. Our response will be presented in the form of fact list for your quick review.
1. Most of our paper revisions (in blue) haven't introduced any extra information or changed the theoretical ideas, just for more clear guidance.
2. $\pi=\prod_i \pi^i$ is a widely used premise in CTDE-based methods, even in HAPPO (please see Sec. 2.1 in HAPPO) and CoPPO, since the policies are independent during the fully decentralized execution.
3. The key factor that makes IPPO/MAPPO fail in the differentiable game (Fig 1 in the HAPPO) is conflicting update caused by the lack of interaction between two agents, not $\pi=\prod_i \pi^i$. 
4. Regarding point 3, our assumption doesn't have a direct relationship with Fig.1 in the HAPPO paper. 
5. In FP3O, every single pipeline $p$ does keep other policies unchanged. Thus, the assumption is reasonable.
6. The condition of point 5 is enough for our subsequent proofs. There is no need to generalize the assumption to the joint policy case.

We sincerely hope that our replies have helped you solve your concerns on Missing Citation, Flaws, Experiments. If you have unsolved concerns or further concerns on these points, you are welcomed to point them out and we are happy to explain more.

(Details for point 3: specifically, the two agents in MAPPO/IPPO mind only themselves and follow their own gradients due to the lack of interaction. Therefore, two agents will be updated in their own optimal direction (see two red arrows in Fig 1 in the HAPPO). The sequential update scheme of HAPPO enables an agent to be updated first, and then the latter agent is updated according to the first results thus avoiding update conflict and winning the game in Fig 1 in the HAPPO. MAPPO and HAPPO are both under the setting of $\pi=\prod_i \pi^i$).

From: ICLR.cc/2023/Conference/Paper1610/Reviewer_UQcj
title: Thanks for the response
comment: Thanks for the long response. I have to first acknowledge that I don't have sufficient time to fully review so many changes --- I think most of the theoretical analyses are now in blue, which looks a bit skeptical. There are way too many additional reviewing efforts to get this paper to this conference from my perspective.

Anyway, I have tried my best in my limited time to go through the proof. I think the fundamental issue is that in the proof, the authors simply take the decomposed form $\prod_i \pi_i$ to replace the joint policy $\pi$ without any compromise. I would remark that the decomposed assumption that $\pi\approx\prod_i \pi_i$ is the key factor that makes IPPO/MAPPO problematic (you may want to check Fig 1 in the HAPPO). Specifically, assumption 1 is problematic. For every single policy $\pi_i(s,a_i)$, if you keep other policies unchanged, the assumption is correct. However, if you consider the joint policy $\pi$, the assumption can be violated (again see Fig.1 in the HAPPO paper). I think this is the key issue why you would have the strange conclusion that after the first phase, the objective improves, which shouldn't be the case. 

Anyway, I apologize again that I didn't check every single word in the revised draft and I have turned down my confidence accordingly since I really don't have time. But based on the current format of the paper (and with so many critical revisions), I would keep my score and hope the authors could substantially improve their work for a much more rigorous and cleaner version.

From: ICLR.cc/2023/Conference/Paper1610/Authors
title: Paper Revisions for Reviewer BvRf
comment: Dear Reviewer BvRf,

Thank you for your constructive feedback and suggestions. We have revised our paper in response to your comments. Specifically:
* We have provided detailed discussion and proper justification for assumption 1 in Appendix B.6.
* We have stated the assumption in Theorem 3. 

All updates have been highlighted in blue in revised paper.

From: ICLR.cc/2023/Conference/Paper1610/Authors
title: Response to Reviewer BvRf
comment: Dear Reviewer BvRf,

Once again! Thank you very much for your constructive comments and the time spent on it. We hope our following response can solve your concerns.

> Decomposition

The decomposition of the joint advantage function is the one-step deduction result of Eq. (1), (2) therefore having similarity with HAPPO. However, this similarity only occurs in the first step of our FP3O. Unlike the serial optimization scheme in HAPPO, we establish $n$ equivalent parallel decomposition schemes (please see Eq. (4)). Please allow us to make it clearer. 

* The first part of our decomposition at all pipelines (i.e. $n$ parallel components of $M_{\pi}^{i_1}(\tilde{\pi}^\varnothing,{\tilde{\pi}}^{i_1}),…, M_{\pi}^{i_n}(\tilde{\pi}^\varnothing,{\tilde{\pi}}^{i_n})$) plays a key role in achieving parallel update scheme in FP3O because they enable every agent to start an optimization pipeline simultaneously without considering the influence of other agents (please see independence step in Algorithm 2). This is the crucial basis to achieve a general-purpose parallel update scheme. 
* Then, the optimization of the second part $M_{\pi}^{-i_1}(\tilde{\pi}^{i_1}, \tilde{\pi}^{-i_1}),…, M_{\pi}^{-i_n}(\tilde{\pi}^{i_n}, \tilde{\pi}^{-i_n})$ in these parallel pipelines is achieved by our proposed non-overlapping selection and pipeline interaction. Dependence step helps to correct the deviation from the joint optimization objective and can help establish an important interaction between different agents (please see pipeline interaction on Page 6 or Eq. (10)).

* Regarding the entire structure, FP3O has $n$ first parts, also has $n$ second parts. Meanwhile, they are assigned to different pipelines and are optimized in parallel simultaneously. Therefore, FP3O forms a highly parallel structure. This enforces our model to consider the complex influence relationship between all parallel pipelines (see Section 4.2). The derivation of FP3O (see Section 4.3), together with its proofs, is entirely based on the perspective of the influence between parallel pipelines and parallel policy updates (see Section 4.4).

Based on the abovementioned three points, FP3O and HAPPO have significant differences. Our motivation for decoupling the joint advantage function into two parts is to implement full-pipeline parallel update scheme. This structure enables our FP3O to be applied in different network types (i.e., full param sharing, partial param sharing, non-param sharing). In contrast, HAPPO is not applicable to these scenarios.

>Assumption

Thanks for your comment. We apologize for missing the proper justification for assumption 1. We value your suggestions and have provided more detailed discussions on assumption 1 in Appendix B.6, and stated this assumption in Theorem 3. 

In cooperative MARL, the joint policy is the most direct factor that affects whether the joint optimization objective can be improved. For example, CoPPO[1] directly optimize the joint policy $\prod_i\pi^i$to improve the joint optimization objective. Intuitively, the increase/decrease of joint policy probability $\prod_i\pi^i$ does not mean the increase/decrease of single policy $\pi^i$ probability. 
However, in our FP3O, we have shown in Appendix B.6 that the increase/decrease of single policy probability $\pi^{i_p}$ or $\pi^{j_p}$ is equivalent to that of joint policy probability at both independence step and dependence step. Thus, the change of a single policy becomes an effective way to guide the change of joint policy in our FP3O (please see Eq. (20)-(23) for more details). And, the inequality of a single policy (completely equivalent to the joint policy) in our assumption becomes a reasonable way to get the joint objective improved. Our motivation for Assumption 1 also originates from this point.

### Reference
[1] Wu Z, Yu C, Ye D, et al. Coordinated proximal policy optimization. NeurIPS. 2021.

From: ICLR.cc/2023/Conference/Paper1610/Authors
title: Paper Revisions for Reviewer UQcj
comment: Dear Reviewer UQcj,

Thank you for your thoughtful feedback and suggestions. We have revised our paper in response to your comments. Specifically:
* We have added CoPPO in Section 2 and perform additional experiments with CoPPO baseline on more SMAC scenarios.
* We have added PG-AR to the discussion on the future work in Section 6.
* We have added the insights of Eq. (7) below Algorithm 2 on Page 6.
* We have updated Eq. (7) to a meaningful version with $\tilde{\pi}^\ast$ denoting the candidate update policy.
* We have added, in the Appendix B.5, more detailed proofs of why the second term of Eq. (7) can be omitted in Eq. (8).
* We have updated the Theorem 3 for more clearly showing that the change of all policies when jointly updating all pipelines has been considered.
* We have updated some contexts (e.g., Eq. (11), (12), (24), (28), (31)) in Subsection 4.4/Appendix B.6 to give more clear guidance to Theorem 3. 

All updates have been highlighted in blue in the revised paper.

From: ICLR.cc/2023/Conference/Paper1610/Authors
title: Response to Reviewer UQcj (1/5)
comment: ## Response to Missing Citation

Dear Reviewer UQcj,

Once again! We would like to thank you for the in depth review of the paper and the time spent on it. And, we appreciate the thoughtful feedback and suggestions. Below we detail our answer to your concerns.

>For Reference [1]

Thanks for your constructive comments on CoPPO [1]. In response to your suggestion, we have added CoPPO in Section 2 and additional experiments in Appendix E.2. However, CoPPO does not perform the same update rule as the dependence step of our FP3O algorithm. There are three significant differences:
1) Different clip objective: **FP3O**: for dependence step of every single pipeline $p$, the clip objective is only the policy of agent $j_p$, i.e., ${\rm clip}\Big(\frac{\pi_{\theta^{j_p}}^{j_p}(a^{j_p}|s)}{\pi_{\theta_{\rm old}^{j_p}}^{j_p}(a^{j_p}|s)},1\pm\epsilon\Big)$，**CoPPO**: the clip objective is the joint policy of all agents ${\rm clip}\Big((\prod_{j\neq i} r^j)r^i,1\pm\epsilon\Big)$ (see Eq. (7), (8) in [1]).
2) Different update scheme：**CoPPO**: CoPPO directly updates the joint policy during optimization, which means that term $\prod_{j\neq i}r^j$ in Eq. (8) in [1] will constantly change after every mini-batch update. Thus, the optimization objective for each agent is non-stationary. **FP3O**: for dependence step of every single pipeline $p$, we only update one policy $\pi_{\theta^{j_p}}^{j_p}$ and take $\frac{\pi_{\theta_{\rm mid}^{-j_p}}^{-j_p}(a^{-j_p}|s)}{\pi_{\theta_{\rm old}^{-j_p}}^{-j_p}(a^{-j_p}|s)}$ in Eq. (10) as constant during the whole process of dependence step. 
3) Different framework: The overall framework of our FP3O is to assign different agents to different pipelines for parallel updating. In this case, a single pipeline is not independent, and will also be affected by the update results of other pipelines. Thus, we establish the interaction between these pipelines, such as $\pi_{\theta^{j_p}}^{j_p}$ and take $\frac{\pi_{\theta_{\rm mid}^{-j_p}}^{-j_p}(a^{-j_p}|s)}{\pi_{\theta_{\rm old}^{-j_p}}^{-j_p}(a^{-j_p}|s)}$ in Eq. (10) which reflects the mutual utilization of different pipelines. However, CoPPO doesn’t have the above property.

>For Reference [2]

1. (Fixed) Order: In fact, HAPPO adopts random order instead of fixed order, which has been stated in the second paragraph on Page 8 in [3], i.e., “the order in HATRPO/HAPPO can be randomised at each iteration, which also offers desirable convergence property”. Thus, the order is not responsible for the unfeasible parameter sharing or partial parameter sharing in HAPPO and is not the issue we want to solve in our paper. We have clarified the true limitation in HAPPO: *the sequential update scheme will cause uncontrolled KL divergence and poor data efficiency*, which is exactly the issue we want to address, not the (fixed) order. We have highlighted the relevant contents in red in Section 1 and Appendix A for convenience. If we have any misleading statements about the (fixed) order in our paper, we will appreciate you pointing them out. Hope this response can help you with this misunderstanding.  
2. Attention Architecture: We appreciate the reviewer for providing new insight to make param sharing available in HAPPO. The attention network design of PG-AR[2] is elaborate and non-general. Maybe, reference [4] is a more direct solution to HAPPO. It also designs an attention architecture to enable HAPPO to be used with parameter sharing. They both try to design a specific network architecture as the solution, which is significantly different from the major objective of our work, i.e., designing  a general-purpose algorithm that is applicable to varieties of network types including MLP, RNN, LSTM, param sharing, partial param sharing, non-param sharing, etc. Nevertheless, we note that PG-AR provides a good insight for our future work to explore the application of FP3O on the elaborately designed network. Therefore, in revised paper, we have added PG-AR[2] and [4] to the discussion on future work in Section 6.


### Reference
[1] Coordinated proximal policy optimization, Wu Z, Yu C, Ye D, et al., NeurIPS. 2021.

[2] Revisiting some common practices in cooperative multi-agent reinforcement learning, Fu W, Yu C, Xu Z, et al., ICML. 2022.

[3] Trust region policy optimisation in multi-agent reinforcement learning, Kuba J G, Chen R, et al., ICLR, 2022.

[4] Multi-Agent Reinforcement Learning is a Sequence Modeling Problem, Wen M, Kuba J G, Lin R, et al., NeurIPS. 2022.

From: ICLR.cc/2023/Conference/Paper1610/Authors
title: Response to Reviewer UQcj (2/5)
comment: ## Response to Flaws (1/2)

>Eq. (6), (7)

* Eq. (6) and (7) are the importance sampling forms of $A_{\symbfit{\pi}}^{i_p}(s,a^{\varnothing },a^{i_p})$ and $A_{\symbfit{\pi}}^{-i_{p}}(s,a^{i_p},\symbfit{a}^{-i_{p}})$ respectively. Adding up Eq. (6) and Eq. (7) is the reverse of advantage decomposition. Naturally, the result will be the joint importance sampling objective of $A_{\symbfit{\pi}}(s,\symbfit{a})$. The reason why we decouple the joint advantage function into Eq(6) and Eq(7) is to quantify the contribution of different agents to the joint return, which holds the same insight as the advantage decomposition problems in the MARL community [3][5]. In these works, if we put all decomposed parts together, we will get the same result, i.e., the joint advantage function or its importance sampling objective. In the revised paper, we have added more insights of Eq. (6), Eq. (7) below Algorithm 2 on Page 6.

* Thanks for your suggestion on subscript. However, changing the subscript of the expectation operator from $\symbfit{a}\sim \pi$ to $\symbfit{a}^{-i_p}\sim \pi$ will make Eq. (17) no longer hold. In addition, this changing violates the definition of $M_{\pi}^{-i_p}(\tilde{\pi}^{i_p}, \tilde{\pi}^{-i_p})$ (note that the subscript $\symbfit{a}^{i_{1:m}}\sim \tilde{\pi}^{i_{1:m}}$ in Definition 1 involves $a^{i_p}, \symbfit{a}^{i_{1:m}-i_{p}}$) thus breaking the inequality in Theorem 2. It is a valuable suggestion to rewrite Eq. (7) in a more meaningful version. Thus, we have updated the candidate update policy as $\tilde{\pi}^\ast$ in Eq. (6), (7), which can reflect the candidate optimization agent of independence step and dependence step. Moreover, in the original paper, we also gave a further version of Eq. (7) in Pipeline Interaction on Page 6 (i.e., $\mathbb{E}_{a^{j_p} \sim \tilde{\pi}^{j_p}, \symbfit {a}^{-j_p}\sim \hat{\pi}^{-j_p}} [{A}_\pi^{-i_p}(s, a^{i_p}, \symbfit {a}^{-i_p})]$).

>The second ratio term in Eq. (7)

Although, the subscript of Eq. (7), as we explained above, cannot be changed to $a^{-i_p}\sim\pi$, the second ratio term in Eq. (7) can be omitted in Eq. (8) thanks to the parameterized policies $\theta$. To yield our FP3O, we first parameterize the policies of all agents with neural networks. At dependence step, we aim to optimize the policy network of agent $j_p$, i.e., $\theta^{j_p}$ with the policies of other agents as constants (please see non-overlapping selection). Based on this implementation, when we execute $\mathop{\rm maximize}\limits_{\theta^{j_p}}M_{\pi_{\theta_{\rm old}}}^{-i_p}$ and gradient backpropagation for $\theta^{j_p}$, the second ratio term, as a constant, has no contribution to the gradients of $\theta^{j_p}$. This guarantees that our omission is reasonable. Nevertheless, we value your comments and have added more detailed proofs in Appendix B.5.

### Reference

[3] Trust region policy optimisation in multi-agent reinforcement learning, Kuba J G, Chen R, et al., ICLR, 2022.

[5] Settling the variance of multi-agent policy gradients, Kuba J G, Wen M, Meng L, et al. NeurIPS. 2021.

From: ICLR.cc/2023/Conference/Paper1610/Authors
title: Response to Reviewer UQcj (3/5)
comment: ## Response to Flaws (2/2)

>Theorem 3
* Let's consider the following case. If in Theorem 3, we just consider a single policy $\pi^{i_p}$ is changed at step 1 and a single policy $\pi^{j_p}$ is changed at step 2, the inequality we should prove will be $L_{\pi}(\hat{\pi}^{i_p},\tilde{\pi}^{j_p}\cdot\pi^{-i_p-j_p})\geq L_{\pi}(\hat{\pi}^{i_p},\pi^{-i_p})\geq L_\pi(\pi^{i_p},\pi^{-i_p})$ rather than$L_{\pi}(\tilde{\pi}^{i_p},\tilde{\pi}^{-i_p})\geq L_{\pi}(\hat{\pi}^{i_p},\hat{\pi}^{-i_p})\geq L_\pi(\pi^{i_p},\pi^{-i_p})$. Thus, it is by taking the factor that all policies are changed at different pipelines into account that we have proved Theorem 3 in Appendix B.6, which can be clearly checked by the superscript (i.e.,$\pi$, $\hat{\pi}$, $\tilde{\pi}$) in Eq (24), (25), (28), (29). As a result, the lower bounds of all pipelines are equal (i.e., $L_{\pi}(\tilde{\pi}^{i_1},\tilde{\pi}^{-i_1})=,…,=L_{\pi}(\tilde{\pi}^{i_n},\tilde{\pi}^{-i_n}$) (see Eq. (31)), so Theorem 3 can cover the full-pipeline case.

* Beyond the changes of all policies, FP3O also considers the inter-relationships between pipelines (i.e., pipeline interaction on Page 6). At the dependence step, all pipelines will mutually utilize the results of independence step (middle policies $\hat{\pi}$) to make policy approximation. We have also considered this policy approximation between all pipelines in our proof for Theorem 3 (see Eq. (27)).

* Our work is not a trivial extension of HAPPO. FP3O is a highly parallel structure (non-overlapping selection, pipeline interaction and the mutual influence between different pipelines on the joint policy change). Its derivation and proofs must consider the influence of parallel policy updates, which is fundamentally different from HAPPO. We have updated Theorem 3 and some contexts (e.g., Eq. (11), (12), (24), (28), (31)) in Subsection 4.4/Appendix B.6 to give more clear guidance to Theorem 3.

>The presentation of the algorithm

As we explained above, there are three significant differences between dependence step and CoPPO (please see Response to Missing Citation). Therefore, whether CoPPO can guarantee monotonicity has no direct relationship with the dependence step. The dependence step only optimizes the second part $A_{\symbfit {\pi}}^{-i_{p}}(s,a^{i_p},\symbfit{a}^{-i_{p}})$ of the joint advantage function, and its monotonicity depends on the optimization of the first part in independence step (see Eq. (29), (30)). As a result, describing FP3O as an additional CoPPO update following a MAPPO update is unsuitable. 

Please allow us to make the two stages in FP3O clearer. 
* At independence step, the optimization objectives of all pipelines (i.e. $n$ parallel components of $M_{\pi}^{i_1}(\tilde{\pi}^\varnothing,{\tilde{\pi}}^{i_1}),…, M_{\pi}^{i_n}(\tilde{\pi}^\varnothing,{\tilde{\pi}}^{i_n})$) play a key role in achieving parallel update scheme in FP3O because they enable every agent to start an optimization pipeline simultaneously without considering the influence of other agents (please see independence step in Algorithm 2). This is the crucial basis to achieve a general-purpose parallel update scheme. However, independence step could cause deviation from the joint optimization objective (i.e., $A_\pi(s,\symbfit{a})$) since the objective is just the first part of it (i.e., $A_\pi^{i_p}(s,a^{\varnothing },\symbfit{a}^{i_p})$).

* Dependence step with $M_{\pi}^{-i_1}(\tilde{\pi}^{i_1}, \tilde{\pi}^{-i_1}),…, M_{\pi}^{-i_n}(\tilde{\pi}^{i_n}, \tilde{\pi}^{-i_n})$ helps to correct the deviation from the joint optimization objective. Its monotonicity depends on the optimization of independence step. Additionally, dependence step can help establish an important interaction (see pipeline interaction on Page 6 and Eq. (10)) between different agents so that their updates can be coordinated.

* Regarding the entire structure, FP3O has $n$ optimization objectives at independence step, also has $n$ optimization objectives at dependence step. Meanwhile, they are assigned to different pipelines and are optimized in parallel simultaneously. Therefore, FP3O forms a highly parallel structure. This enforces our model to consider the complex influence relationship between all parallel pipelines (see Section 4.2). The derivation of FP3O (see Section 4.3), together with its proofs, is entirely based on the perspective of the influence between parallel pipelines and parallel policy updates (see Section 4.4).

From: ICLR.cc/2023/Conference/Paper1610/Authors
title: Response to Reviewer UQcj (4/5)
comment: ## Response to Experiments

> CoPPO and PG-AR
 
Regarding “Respond to Missing Citation”, again, we appreciate the reviewer for the suggestion on CoPPO. We have performed additional experiments with CoPPO baseline on more SMAC scenarios in Appendix E.2. As for PG-AR, we have clarified there is no issue of (fixed) order in HAPPO and our paper. Moreover, PG-AR designs a specific network architecture as the solution, which is significantly different from the major objective of our work, i.e., designing one algorithm that is applicable to varieties of network types. In our paper, the settings of all algorithms are identical including all hyperparameters and network inputs for a fair comparison, while PG-AR cannot adopt the same networks (e.g., mlp, rnn, NoPS, FuPS, PaPS in Table 5, 7) as other algorithms. Therefore, PG-AR may not be suitable as a baseline, but as the future work, in our paper.

> Reliability

In the MAPPO paper [6], the main results are obtained by elaborate hyperparameter settings (see Table 14 in [6]). However, in our paper, the hyperparameters of all algorithms (FP3O, HAPPO, MAPPO, IPPO, CoPPO) are not as complicated as [6], so the results have differences. 

To prove that our results of MAPPO are reliable (a more effective way is to check our codes of MAPPO/IPPO in the supplementary materials), we list some reproduction results of MAPPO in some existing papers for comparison:

| Scenarios | Ours | Reference [1] | Reference [4] |
|  ----  | ----  |  ----  | ----  |  
| bane vs. bane | $100.0\pm3.5$ |*|*|
| 2c vs. 64zg | $100.0\pm2.5$ |*|*|
| corridor |$100.0\pm2.9$ |*|*|
| 3s5z | $90.6\pm18.5$ | $\approx91.0\pm10.0$ | $72.5\pm26.5$ |
| 5m vs. 6m | $81.3\pm19.0$ | $\approx77.0\pm25.0$ |$\times$|
| 6h vs. 8z | $87.5\pm12.9$ |$\times$| $88.4\pm5.7$ |

*: not compared as our reproduction result for MAPPO has reached 100% on these maps.

$\times$: not available in their original papers.

As we can see, on easy map (i.e., bane vs. bane, 2c vs. 64zg), the reproduction results of MAPPO in our paper are 100%. On hard and super hard maps (i.e., 3s5z, 5m vs. 6m, corridor, 6h vs. 8z), our reproduction results are also reliable in the MARL community. Among them, our result on super hard corridor also reaches 100%.

We have provided all the source code (including FP3O, HAPPO, MAPPO, IPPO and CoPPO) in the supplementary material and specify all training details (e.g. hyperparameter settings), repetitive experiments (e.g., the number of random seeds), and the type of computing infrastructure (e.g. the type of GPUs) in Appendix D. All reproducible materials are welcome to be checked for our supportable results on MAPPO/IPPO.

### Reference
[1] Coordinated proximal policy optimization, Wu Z, Yu C, Ye D, et al., NeurIPS. 2021.

[4] Multi-Agent Reinforcement Learning is a Sequence Modeling Problem, Wen M, Kuba J G, Lin R, et al., NeurIPS, 2022.

[6] The surprising effectiveness of ppo in cooperative, multi-agent games, Yu C, Velu A, Vinitsky E, et al., arXiv preprint arXiv:2103.01955, 2021.

From: ICLR.cc/2023/Conference/Paper1610/Authors
title: Response to Reviewer UQcj (5/5)
comment: ## Response to Summary

Dear Reviewer UQcj,

We hope our replies above have helped solve your all concerns and helped make the main theorem of FP3O clearer: it is derived based on the complex parallel relationship between our different pipelines (i.e., non-overlapping selection, pipeline interaction and the mutual influence between different pipelines on the joint policy change). Thus, our work is significantly different from HAPPO.

From: ICLR.cc/2023/Conference/Paper1610/Authors
title: Paper Revisions for Reviewer 7HFV
comment: Dear Reviewer 7HFV,

Thank you for your feedback. We have revised our paper in response to your comments. Specifically:
* We have provided proper justification for assumption 1 in Appendix B.6.
* We have stated the assumption in Theorem 3. 
* We have added the discussion on why the dependence step is favorable below Algorithm 2 on Page 6.
* We have updated Theorem 3 to more clearly show that the change of all policies when jointly updating all pipelines has been considered.
* We have updated some contexts (e.g., Eq. (11), (12), (24), (28), (31)) in Subsection 4.4/Appendix B.6 to give more clear guidance to Theorem 3. 

All updates have been highlighted in blue in the revised paper.

From: ICLR.cc/2023/Conference/Paper1610/Authors
title: Response to Reviewer 7HFV
comment: Dear Reviewer 7HFV,

Once again, thank you for your precise review and constructive feedback. We have read your comments carefully. Hope our following responses can answer your concerns.
>Assumption

Thanks for your suggestions, we have provided proper justification for Assumption 1 in Appendix B.6, and stated this assumption in Theorem 3. 

In cooperative MARL, the joint policy is the most direct factor that affects whether the joint optimization objective can be improved. For example, CoPPO[1] directly optimize the joint policy to get the joint optimization objective improved. Intuitively, the increase/decrease of joint policy probability does not mean the increase/decrease of single policy probability. However, in our FP3O, we have shown in Appendix B.6 that the increase/decrease of single policy probability is equivalent to that of joint policy probability at both independence step and dependence step of each pipeline. Thus, the change of a single policy at independence step and dependence step is an effective way to guide the change of the joint policy. And, the inequality of a single policy (completely equivalent to the joint policy) in our assumption becomes a reasonable way to get the joint objective improved (please see Eq. (20)-(23) for details).

>Dependence step

The joint advantage function is decoupled into two parts (i.e., $A_{\symbfit {\pi}}^{i_p}(s,a^{\varnothing},a^{i_p})$ and $A_{\symbfit {\pi}}^{-i_{p}}(s,a^{i_p},\symbfit{a}^{-i_{p}})$ as shown in Lemma 1. If we drop the dependence step and perform the independence step alone, there will be two issues. 1) For a single pipeline, the optimization objective is no longer the true objective (i.e., $A_{\symbfit{\pi}}(s,\symbfit{a})$), but the first part of it. In this case, the optimization direction will deviate from the true objective at each policy iteration. 2) At independence step, each agent’s optimization is completely independent without considering the interaction with other agents. Lack of interaction between agents may lead to update conflict (see Fig. 1 in [2]), which is a crucial problem in the multi-agent domain [2][3][4]. 

Therefore, the dependence step is to 1) correct the deviation from the true objective induced by the independence step, 2) help establish an interaction between different agents so that their updates are coordinated. In response to your comment, we have added the discussion on why dependence step is favorable on Page 6.

>Theorem 3

Theorem 3 is based on simultaneous changes of all policies at different pipelines thus covering the full-pipeline case. Please allow us to make it more clearly.
* Let's consider the following case. If in Theorem 3, we just consider a single pipeline (i.e., a single policy $\pi^{i_p}$ is changed at step 1 and a single policy $\pi^{j_p}$ is changed at step 2), the inequality we should prove will be $L_\pi(\hat{\pi}^{i_p},\tilde{\pi}^{j_p}\cdot \pi^{-i_p-j_p})\geq L_\pi(\hat{\pi}^{i_p},\pi^{-i_p})\geq L_\pi(\pi^{i_p},\pi^{-i_p})$ rather than $L_\pi(\tilde{\pi}^{i_p},\tilde{\pi}^{-i_p})\geq L_\pi(\hat{\pi}^{i_p},\hat{\pi}^{-i_p})\geq L_\pi(\pi^{i_p},\pi^{-i_p})$ in Theorem 3. Thus, it is by taking the factor that all policies are changed at different pipelines into account that we have proved Theorem 3 in Appendix B.6, which can be clearly checked by the superscript (i.e.,$\pi$, $\hat{\pi}$, $\tilde{\pi}$) in Eq (24), (25), (28), (29). As a result, the lower bounds of all pipelines are equal (i.e., $L_{\pi}(\tilde{\pi}^{i_1},\tilde{\pi}^{-i_1})=,…,=L_{\pi}(\tilde{\pi}^{i_n},\tilde{\pi}^{-i_n}$) (see Eq. (31)), so Theorem 3 can cover the full-pipeline case.
* Beyond the changes of all policies, FP3O also considers the inter-relationships between pipelines (i.e., pipeline interaction on Page 6). At the dependence step, all pipelines will mutually utilize the results of independence step (middle policies $\hat{\pi}$) to make policy approximation. We have also considered this policy approximation between all pipelines in our proof for Theorem 3 (see Eq. (27)).

For clearer guidance to Theorem 3, we updated Theorem 3 and some contexts (e.g., Eq. (11), (12), (24), (28), (31)) in Subsection 4.4/Appendix B.6. 

## Reference 
[1] Wu Z, Yu C, Ye D, et al. Coordinated proximal policy optimization. NeurIPS. 2021.

[2] Kuba J G, Chen R, et al. Trust region policy optimisation in multi-agent reinforcement learning. ICLR, 2022.

[3] Wen Y, Yang Y, Luo R, et al. Probabilistic recursive reasoning for multi-agent reinforcement learning. ICLR, 2019.

[4] Foerster J, Farquhar G, Afouras T, et al. Counterfactual multi-agent policy gradients. AAAI. 2018, 32(1).

From: ICLR.cc/2023/Conference/Paper1610/Authors
title: Response to Reviwer z4CL
comment: Dear Reviewer z4CL,

Thank you for your positive feedback! We feel encouraged to see that you think our contributions are well-written and motivated! We hope our response can help with your remaining concerns.

> Non-cooperative MARL setting

Similar to all existing PPO-based algorithms [1][2][3][4], our method is also modeled on the cooperative multi-agent discounted Markov decision process (please see problem formulation in Section 3), where all agents have a shared reward and all agents aim to maximize a shared objective: $\mathcal{J}(\symbfit{\pi})$. The shared reward makes the advantage decomposition without any additional restrictions possible (see Eq. (1), (2)), and our equations, lemmas and theorems are derived under this basic setting. Thus, directly applying this framework to a non-cooperative MARL setting is not suitable, even though non-cooperative task is an important setting. The difficulty in generalizing our framework to non-cooperative games is how to establish a private reward objective function for each agent. MADDPG[5] is working in both cooperative and competitive scenarios, and the latter is known to be a much harder problem. As the same CTDE-based algorithm, MADDPG can provide us insight into this problem and we will leave it for future work.

## Reference 
[1] de Witt C S, Gupta T, Makoviichuk D, et al. Is independent learning all you need in the starcraft multi-agent challenge?. arXiv preprint arXiv:2011.09533, 2020.

[2] Yu C, Velu A, Vinitsky E, et al. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.

[3] Wu Z, Yu C, Ye D, et al. Coordinated proximal policy optimization. Advances in Neural Information Processing Systems, 2021, 34: 26437-26448.

[4] Kuba J G, Chen R, et al. Trust region policy optimisation in multi-agent reinforcement learning. ICLR, 2022.

[5] Lowe R, Wu Y I, Tamar A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in Neural Information Processing Systems, 2017, 30.

From: ICLR.cc/2023/Conference/Paper1610/Reviewer_7HFV
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This work focuses on the problem of policy optimization in cooperative multi-agent RL, where different agents share the same reward. The proposed FP3O method provably achieves monotone policy improvement and demonstrates good empirical performance in numerical experiments.
strength_and_weaknesses: Strength:

- The proposed method achieves good empirical performance compared with other methods.

- Theoretical analysis shows that the proposed method achieves monotonic policy improvement under suitable assumption.

Weaknesses:

- The analysis imposes Assumption 1 to prove monotonic improvement of FP3O. However, the assumption lacks proper justification, especially for the inequality concerning the independence step and dependence step. Please also clarify that additional assumption is needed in the statement of Theorem 3.

- The main motivation of the proposed method, as the manuscript claims, is the guaranteed monotonic policy improvement. A natural observation is that: monotonic policy improvement can be achieved by the independence step (11) alone (i.e., simply dropping all "dependence step" (12)), which is simply the independent TRPO method. While Theorem 3 shows that an additional dependence step further improves the policy, this can be done with another round of independence step as well. Therefore, it would be helpful to provide a discussion on why a dependence step is more favorable in the proposed method.

===============================

- I have read reviewer UQcj's comment on the statement of Theorem 3 and agree that the result only shows the monotonic improvement of every single optimization pipeline. This conclusion does not naturally apply to the full-pipeline optimization due to non-stationarity, hence the discussion on the monotonic improvement guarantee of FP3O algorithm is flawed and misleading. 
clarity,_quality,_novelty_and_reproducibility: See "Strength and Weakness".
summary_of_the_review: This work proposes a new policy optimization method for cooperative multi-agent RL, which achieves provable monotonic policy improvement over iterations and yields good empirical results. Overall the paper is well-written. The reviewer is not familiar with empirical RL research and is therefore unable to comment on the implementation details and numerical experiments. However, theoretical motivation of the algorithm is not convincing as 
(1) it relies on an additional assumption
(2) monotonic policy improvement is only demonstrated on an individual optimization pipeline and does not apply to the full-pipeline update
(3) a lack of discussion on why dependence step is more favorable.
Therefore the reviewer vote for "3: reject, not good enough".
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1610/Reviewer_z4CL
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper studies the cooperative MARL problems and proposes the Full-Pipeline PPO (FP3O) algorithm, where agents are dynamically allocated to different pipelines and then updated with the proposed one-separation trust region optimization. They show that FP3O avoids restrictive factors and enjoys provable monotonic improvement. Extensive experiments are conducted to show the performance of FP3O.
strength_and_weaknesses: ## Strength
- This paper is well-written and nicely organized. The proofs seem to be theoretically sound to me.
- The topic of this paper is the core of the relevant literature and is of great significance.
- The proposed framework is novel to me and is nicely motivated. The theoretical guarantee is strong.
- The experiments are extensive and nicely presented, showing great performance over the baseline methods. 

## Weaknesses
- I wonder is it possible to generate this framework for the non-cooperative MARL setting? It would be helpful if the author could provide some discussions on the key difficulties there.

==================================================
I read reviewer UQcj's comments and the authors' response. The authors' explanation about why the second ratio term in Eq(7) could be omitted makes sense to me. But I still can't follow the response to flaws in Thm 3. Due to the concern about the potential flaws of the theory, I believe the paper should be further polished. 

Also, I read the authors' response to concerns about Assump. 1, I don't see why "the increase/decrease of joint policy probability is completely equivalent to the increase/decrease of single policy probability" serves as a valid justification of Assump. 1. Rather, this assumption seems directly implies the improvement results given the equivalence above, making the assumption ad-hoc. 

In conclusion, I think this paper needs to be further polished, and hence I need to decrease my score.


clarity,_quality,_novelty_and_reproducibility: This paper is of nice quality and is doing good in clarity and novelty.
summary_of_the_review: In summary, I think this paper is well-written, with great significance and novelty. The proposed method is general, theoretically justified, and competitive with the baseline methods, validated by extensive experiments. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1610/Reviewer_UQcj
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a new multi-agent RL algorithm called FP3PO. The core idea builds upon the HAPPO algorithm with the aim to extending the multi-agent trust region idea to the setting of parameter sharing and parallel updating (instead of sequential updating). The empirical results show that FP3PO outperforms HAPPO and MAPPO.

However, the algorithm derivations have significant flaws.
strength_and_weaknesses: ## Strength
The paper points out two significant limitations of the HAPPO: (1) the fixed updating order from agent 1 to n, which may not be the most appropriate choice; (2) HAPPO requires a sequential update, which can make parameter sharing particularly non-trivial.

Therefore, the paper proposes a simple idea to address these two issues: for (2), we can only update two agents; for (1) we can average over all possible orders ---- note that since we are only updating two agents, a non-overlapping scheme can be applied as a clever order design. 

The idea is intuitive, which should be appreciated. However, the derivations have flaws.

## Weakness
### Missing Citation
Before moving into the details of derivation issues, I would like to point out that a few important references are missing in this paper.

Regarding issue (2), i.e., HAPPO requires a sequential update scheme, the authors may want to refer to the [CoPPO algorithm](https://openreview.net/forum?id=iCJFwoy1T-q). The CoPPO performs exactly the same update rule as the dependent stage of the FP3PO algorithm (without the first independent stage).

Regarding issue (1), i.e., HAPPO cannot be applied to the parameter sharing setting, the authors may want to refer to the [PG-AR algorithm](https://arxiv.org/abs/2206.07505), which attention architecture and a randomized ordering in training to ensure the learned policy can have shared parameter and generalize to an arbitrary updating order. 

[1] Coordinated Proximal Policy Optimization, Zifan Wu, Chao Yu, Deheng Ye, Junge Zhang, haiyin piao, Hankz Hankui Zhuo, NeurIPS 2021

[2] Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning, Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, Yi Wu, ICML 2022

### Flaws

1. **Equation (6) and (7)**: Equation (6) and (7) are technically correct. However, I would like to point out that these two equations do not make too much sense. If you simply add up eq(6) and eq(7), you will get the joint importance sampling objective for the joint return. That is, Eq(7) does not bring any additional insight at all, since it can be directly obtained by deducing eq(6) (which is straightforward to get as well) from the joint return. _I would suggest the authors consider a more meaningful _conditioned_ version of eq(7)_ by changing the subscript of the expectation operator from $\mathbf{a}\sim\pi$ to $\mathbf{a}^{-i_p}\sim\pi$. The unconditioned subscripts possibly cause future flaws, I guess.

2. **The second ratio term in Eq(7) cannot be omitted**. It is stated in the first line on page 7 that the ratio term of $\pi^{i_p}$ can be omitted. This statement can be true if the subscript in Eq(7) is the conditioned version, i.e., $a^{-i_p}\sim\pi$, rather than the joint action, i.e., $a\sim\pi$. Unfortunately, since a joint action is considered in the expectation, the ratio term can be no longer omitted.

3. **Theorem 3 cannot guarantee the monotonicity of FP3PO**. Theorem 3 is technically correct. However, the statement under theorem 3 is wrong. Theorem 3 shows that for each individual $i_p$, the pipeline ensures monotonicity, which naturally follows the derivation in HAPPO ---- this is just a special case of HAPPO by only updating two agents.  _However, the conclusion may not hold when _jointly_ updating all pipelines._ This is because, in the proof of theorem 3, the authors assume that a single policy with index $i_p$ is changed. However, in the proposed algorithms (Algo.2), all the policies are changed (assuming the setting of parameter sharing) after the independent stage, which breaks the proof. 

4. **The presentation of the algorithm**. Now, let's re-examine the two stages in the proposed algorithm (Algo.2). Stage one is exactly MAPPO (or IPPO) with individual policy ratio and PPO. Stage two is exactly the CoPPO [1] algorithm with the reference policy as the middle policy. Following the derivation in CoPPO, stage two clearly ensures monotonicity. However, stage one, which is exactly MAPPO, would not ensure monotonicity. From this perspective, we can again notice that the second ratio term in Eq(7) cannot be omitted at all. That term is to correct the violated monotonicity induced by the stage one update rule. Interestingly, the experiment results suggest that FP3PO outperforms MAPPO by a clear margin. Considering the actual structure of FP3PO, the experiment suggests that the improvements are actually from the second stage, i.e., an additional CoPPO update following an MAPPO update. Such a conclusion can be particularly counter-intuitive in my opinion ---- if MAPPO is bad, why adding another CoPPO update rule, which makes policy updates even more aggressive, can make the result better? I would suggest the authors completely re-examine the proposed algorithm and provide a better presentation with more insights. 

### Experiments
1. As I mentioned, it would be great to include more baselines including the CoPPO algorithm [1] and the PG-AR algorithm [2]
2. The variance of IPPO and MAPPO in table 2 is particularly large, which contradicts the reported numbers in the MAPPO paper. This makes the results less convincing.  
clarity,_quality,_novelty_and_reproducibility: In general, I think the writing of the paper is a bit messy. The motivation and ideas are simple but the presentation involves so many technical steps, which are unnecessarily complex. Lots of the theorems can be directly adapted from the HAPPO paper without the need to re-introducing a lot of new notations. I would suggest the authors re-organize the paper to make it more concise. 

Regarding quality, novelty and reproducibility, please refer to my comments on weaknesses. 
summary_of_the_review: My major complaint about this paper is the problematic algorithm derivation, which has been stated in the previous context. 
correctness: 1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1610/Reviewer_BvRf
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposed a CTDE PPO-based MARL algorithm F3PO which could be optimized simultaneously. This paper showed the surrogate of the joint policy improvement could be decoupled into two parts: the independent part which is about one agent $i_p$ and the dependent part which is about the other agents $-i_p$. F3PO is the algorithm to optimize these two parts of the objective.  The practical algorithm of F3PO could be divided into two steps: in the first step, each pipeline $p$ optimizes the independent part of agent $i_p$ to obtain the middle policy $\hat{\pi}_{i_p}$; in the second step, each pipeline $p$ optimize the policy of one another agent $j_p$ in the dependent objective of agent $i_p$ to obtain the new policy $\tilde{\pi}\_{j\_p}$. This paper claims that the new joint policy guarantees monotonic improvement.

strength_and_weaknesses: ### Strength

1. The main strength of this paper lies in the empirical results, which show the proposed method outperforms baselines in a variety of tasks under different parameter-sharing schemes.   

2. Full-pipeline optimization is new and empirically effective.  

### Weaknesses

My main concern about this paper is the theoretical results.

1.  Novelty:  The idea of the decomposition of the joint advantage function over different subsets of agents is similar to HAPPO. And the decomposition in Theorem 2 could actually be seen as part of the decomposition in HAPPO.

2. Assumption: The main theoretical result in this paper is the monotonic improvement guarantee of the joint policy (Theorem 3). However, the proof of this conclusion depends on Assumption 1 which supposes that if $A\_{\boldsymbol{\pi}}(s,\boldsymbol{a}) > 0$, then $\tilde{\pi}\_i(a\_i|s) \ge  \hat{\pi}\_i(a_i|s) \ge \pi_i(a\_i|s)$ and  if $A\_{\boldsymbol{\pi}}(s,\boldsymbol{a}) < 0$, then $\tilde{\pi}\_i(a\_i|s) \le  \hat{\pi}\_i(a\_i|s) \le \pi\_i(a\_i|s)$. I think this assumption is too strong and idealistic which reduces the value of this theoretical result significantly. Moreover, such an important assumption is even not mentioned in the main text and is only included in the Appendix. I think this arrangement is not appropriate and I doubt the motivation for such an arrangement. 
clarity,_quality,_novelty_and_reproducibility: As mentioned above, the assumption should be made clear in the main paper, not in the appendix. 

The claims, such as the monotonic improvement guarantee, are not well-supported. 
summary_of_the_review: The proposed method empirically outperforms baselines. However, the proposed decomposition of the advantage function is somewhat similar to HAPPO. The main theoretical results are based on a strong assumption which however is idealistic and not even mentioned in the main text. Therefore, the claim of theoretical results, i.e., the monotonic improvement, is not well supported. 

Based on these concerns, I currently lean toward rejection. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough