From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: I thank the authors for their submission and active engagement during the discussion period. This is a borderline paper. On the positive side, reviewers acknowledged the ablation study, and stated that the idea is interesting. On the negative side, reviewers had concerns around novelty [DA2S] and theoretical justification [kSBb] as well as empirical evaluation [kSBb,Z9UZ]. During internal discussions, reviewer kSBb and Z9UZ agree that the empirical evaluation needs to be improved. Therefore, I recommend rejection at this point but highly encourage the authors to take the feedback into account for a revised resubmission to the next venue.
justification_for_why_not_higher_score: Concerns around empirical evaluation shared by two reviewers.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper2324/Authors
title: Important response to naming and novelty
comment: Thank you very much for your reply.

**The novelty concern** raised by Reviewer DA2S has been addressed ("the authors have addressed all of my concerns"), and Reviewer DA2S believes that "the update made to the submission merits an increase in the review rating." In our revised paper (Lines 45-56), we also explained our improvement to VAST (raised by Reviewer kSBb). We would deeply appreciate it if you could re-evaluate our novelties based on the following response.

GoMARL is an absolutely innovative approach to automatic grouping, parameter sharing, and novel use of hypernetworks in terms of dynamic structure and fine-grained learning. The misconception that our paper lacks novelty may be due to how we write. For clarity and reading convenience, each section of our paper starts with a module description. *The informative analysis of the reasons and advantages of our approach is discussed next, highlighting our key idea and novelty insights*. Here we list the inspiring and key novelties our method contributes to our community:

1. Dynamic structure for fine-grained reinforcement learning. GoMARL adopts hypernetwork in a very novel way to realize the number/size-flexible groups guided by fine-grained information. All baselines in our paper use hypernetworks, but they do not achieve GoMARL’s functionality. Our novelty is not reflected in the use of hypernetworks but in *how and why we use hypernetworks* to achieve flexible grouping.  In Lines 167-170, 208-216, 231-233, and 236-244, we analyze the challenges we address by using hypernetworks and their advantages, namely:
   * Bind each agent (or group) to specific weights, thus allowing dynamic group size/number changes under traditional neural networks.
   * Unlike existing value factorizations, we propose group-wise information to drive intra-group cooperation. We utilize hypernetworks to integrate agent info $e^{a_i}$ and group-wise info $s^{g_j}$ into the gradients to guide policy learning in a fine-grained manner, which provides sufﬁciently informative signals to break the performance bottleneck [1].

2. Automatic grouping guidelines. Reviewer DA2S questioned our use of LASSO at first and is now convinced. Our novelty lies in the automatic end-to-end group learning without any domain knowledge, which LASSO cannot achieve. In fact, the fundamental idea of the dynamic grouping is the novel sparsity-driven scheme detailed in Lines 150-161, but not LASSO, a very common regularization in DL. We appreciate its simplicity and non-parametric trait and implement our novel sparsity-driven scheme with it only as an attempt. Alternative sparsity regularization terms can also realize this functionality under the proposed scheme. Thus, LASSO is not our main focus. We have modified this part in Lines 171-176 and emphasized the novelty of the sparsity-driven grouping mechanism.

3. Specialized agent networks with full parameter sharing. All the reviewers recognize this novelty, and thus we no longer emphasize it here.

GoMARL's novel designs of automatic grouping, specialized agent networks with agent info, and especially the novel use of hypernetworks to achieve dynamic structure and fine-grained learning will significantly inspire the MARL community. We attached great importance to this concern and revised our submission a lot.

**As for the naming**, we will follow your suggestion and rename GoMARL as "Dynamic Group-wise Value Factorization with automatic grouping" (DGVF) in the final version.

We sincerely invite you to revisit our novelty and re-evaluate GoMARL with our deep gratitude. *If you have further concerns about our novelties, please tell us in detail; your re-evaluation is very vital to us*.

From: ICLR.cc/2023/Conference/Paper2324/Reviewer_Z9UZ
title: Thank you for the response
comment: I understood the meaning of GoMARL, but I feel there may be more distinct naming.
I also understood other points, but at this stage, I did not increase the rating because I agreed with other reviewers’ opinions about the novelty.

From: ICLR.cc/2023/Conference/Paper2324/Authors
title: To all reviewers
comment: We sincerely thank all reviewers for your detailed and constructive comments. We highly value all feedback and have responded to address all concerns. In addition, we have revised our paper in accordance with all the suggestions, and *the main revisions include but are not limited to the following*:

* A more in-depth introduction of grouping and GoMARL, and a more logical and informative Introduction Section that illustrates our novelty and GoMARL's strengths are contained.
* The revised Method Section highlights our method's motivation, insights, reasons, and advantages to emphasize the novelty and scientific value of GoMARL.
* A new baseline method recommended by Reviewer kSBb is added in both SMAC and GRF environments with extra experiments.
* A new proposition for analyzing GoMARL's satisfaction of the IGM principle is shown in Appendix B.
* A more detailed experimental setup for fair comparisons is in Appendix C.1.
* Extra ablation/component studies of the specialized agent network are added in Appendix D to further demonstrate our superiority.
* Figures are modified to eliminate potential ambiguity and misunderstanding.
* Update missing related work and references.
* The language is emphasized, rephrased, and rearranged in various places throughout the paper to better highlight the focus of GoMARL, the strengths, and the innovations of our method.

Here we only list some crucial revisions, and detailed enhancements are also implemented. *All modifications and critical ideas are highlighted in a comfortable color for your reading convenience*. We sincerely invite all reviewers to read our revision, and it would be highly appreciated with our deep deep gratitude if you would re-evaluate GoMARL based on the new version and our responses.

From: ICLR.cc/2023/Conference/Paper2324/Authors
title: Response to Reviewer kSBb (PART 1)
comment: Thank you very much for your interest in our paper and implementation! We have revised our paper in line with your detailed review and helpful comments. Extra experiments are also conducted. Following are our responses, and we hope they will address all your concerns. 

**Response 1: Timestep difference caused by parallel training**

Our experiments are all trained with *8 parallel runners* for 10M steps in both testbeds, as mentioned in Lines 276-278. Therefore the timestep is not comparable to some papers which conduct experiments with only one "episode" runner. When resources are available, parallel training can facilitate faster convergence of methods and is very common in RL and MARL communities. For example, renowned published MARL papers [1-7] utilize 4, 8, or 32 parallel actors for training, respectively (mostly 8). There are still many other papers taking 8 parallel runners that are not listed here. Our experiments strive for a fair comparison without deliberately tuning any algorithm worse, as detailed in the following response.

**Response 2: Fair Comparison**

*This is a very, very important question. We will address many of your concerns in this response, and we sincerely hope you can patiently read the entire response.*

Firstly, GoMARL is "built on Ft-QMIX" to some extent. However, since GoMARL proposes a new agent network and group mixing network, the finetuned hyperparameters of Ft-QMIX can *only* improve our total mixing network, while the remaining central GoMARL structure was not tuned. Besides, all baseline algorithms utilize the same mixing network of QMIX and all heritage Ft-QMIX's hyperparameter for this commonly-used mixing network (i.e., our total mixing network) during our comparison. Therefore, if GoMARL is defined as "built on Ft-QMIX", all baseline algorithms (except for vanilla QMIX) are also "built on Ft-QMIX" since *they all heritage and only heritage the tuned hyperparameters of Ft-QMIX's mixing network*. The comparison is still fair among these methods.

Secondly, adding vanilla-QMIX in our experiments *is not intended to prove that GoMARL is better than it*. After all, GoMARL is already better than Ft-QMIX, and there is no need to compare it with vanilla-QMIX to show our superiority. We added vanilla-QMIX only to avoid potential misunderstanding caused by the performance difference between Ft-QMIX and that of QMIX in other papers. Therefore, vanilla-QMIX does not affect the fair comparison.

Besides, code-level optimizations are studied in depth in [3,8,9] and are proven to impact methods' performances significantly. Therefore, *we strictly limited tricky code-level implementations (e.g., reward clipping/scaling/normalization, gradient clipping, observation normalization, learning rate annealing, death agent masking, etc.) of all methods*, ensuring the comparisons are as fair as possible. Specific parameter tuning in diverse scenarios is also unfair, so our experiments use fixed parameters for all methods in all scenarios. Most baselines in this paper use the open-source, best-performing implementation[3], which is proposed to compare methods fairly with no special code-level tricks. RODE and ROMA are not included in this framework, but we also utilized the same setting for them to ensure fairness. Many papers produce results that differ significantly from the original ROMA and RODE papers, even those from the same author or research team. For example, the performance of ROMA in the RODE paper is significantly worse than in the original paper. The original RODE paper shows that RODE outperforms QPLEX. However, another paper [10] by the same author shows that QPLEX performs much better than RODE. In addition, papers by other researchers, such as [5, 11-13], also have different results from the original papers in multiple environments. This may indicate that RODE and ROMA are not very stable. However, in our submission, they have one of the best performances among published papers, far better than their performances in the above references and even better than the original paper in some environments, absolutely with no unfair tuning or tricks.

Lastly, we do not share your view that GoMARL and Ft-QMIX perform similarly. GoMARL significantly outperforms Ft-QMIX in 6 of the 8 scenarios, while all methods perform very similarly in 8m_vs_9m and corridor, indicating that these two scenarios are approximate in difficulty for all methods, with only Ft-QMIX and GoMARL have apparent advantages. In summary, only Ft-QMIX maintains some training advantages (i.e., finetuning) in our comparison that have no unfair code-level implementation tricks. However, GoMARL performs much better than Ft-QMIX in both SMAC and GRF (especially on GRF), fully demonstrating the superiority of our methods.

We detailed the experimental setup for fair comparison in Appendix C.1. We would sincerely appreciate it if you would re-evaluate our experiments in light of our response and the revised paper.

From: ICLR.cc/2023/Conference/Paper2324/Authors
title: Response to Reviewer kSBb (PART 2)
comment: **Response 3: Extra ablation experiments**

The ablation of $\lambda_{SD}$ was not studied at first since we ablated it with the agent network as a whole. Your request for additional ablations of $\lambda_{SD}$ is reasonable, and the results are shown in Appendix D of our revised paper. $\lambda_{SD}$ is set to a small value of 0.03, but it plays an essential role in policy specialization and is validated in the ablations. Although this scaling factor is small, the ratio of SD loss to TD loss is stable between 1/4 and 1 during training, which is proper for policy learning. Please pay attention to more than just the scaling factor value. What really matters is the scale of the loss value.

Your request for additional ablations of $w_1$ and $w_2$ is unattainable. Ablations of $w_1$ and $w_2$ together directly degenerate GoMARL to traditional flat value factorization like VDN and QMIX. Ablations of $w_2$ will invalidate the generation of $Q_{group}$ since $w_1$ only produces hidden values, and the absence of $w_2$ will influence the adaption to the dynamic group number. These two ablations will crack GoMARL's key idea or destroy the basic structure for policy learning. $w_1$ is the only meaningful ablation. It decides the dynamic grouping and is already studied before in Lines 305-326.

Your concern about performing specialized agent networks on other baselines is very interesting and crucial. The results are shown in Appendix D. Baselines are improved when equipped with our agent network (much smaller variance and a 5~20% win rate increase). Most importantly, even equipped with our networks, baselines fail to surpass GoMARL regarding efﬁciency and performance. GoMARL with vanilla parameter sharing looks much inferior to GoMARL with specialized agents, so your doubt about whether the performance gain is caused only by specialized agent networks is insightful. This experiment fully illustrates that dynamic group learning may reduce the efﬁciency in the early stage of training. However, in the middle and late stages, the learned grouping has a signiﬁcant effect when equipped with specialized agents. Thus, the grouping module and specialized agents are both crucial. Lastly, the baseline without parameter sharing is not included. It maintains a network for each agent, which requires exponentially increasing time and resources. Therefore, it has no practical and research significance, which is why parameter-sharing has been proposed and widely studied. In our experiments, a baseline without parameter sharing is not meaningful (it requires much more parameters, training time, and resources). Even methods like ROMA and CDS proposed specifically for parameter-sharing do not implement this baseline.

**Response 4: Theoretical analysis and novelty**

In the old version, the introduction of grouping through natural systems is "intuitive" and "not enough". Therefore, we revised the Introduction section and added the missing related work VAST. The comparison with VAST is also available in our revised version.

Like QMIX, ROMA, RODE, VAST you mentioned, and many other excellent works, GoMARL is an empirical method that effectively improves efficiency and cooperation. We believe that even "intuitive" and "expected" ideas need to be validated scientifically, perhaps more so, as they are often easily taken for granted without careful analyses. Science is a process of furthering our understanding by both validating (strengthening) and invalidating (updating) our pre-existing priors. Our work happens to fall into the first category, but both categories should be taken as equally important.

The starting point of GoMARL is the dynamic grouping of agents to promote cooperation, not to improve the expressiveness of the mixer. We are also concerned with theoretical studies such as [16,17] that focus on this problem, but the strong assumptions make their performance unsatisfactory. We added Proposition 1 in our new submission, which proves that GoMARL maintains the IGM principle.

Your misconception that our paper "just proposes a complicated mixer and provides an intuitive explanation" may be due to how we write. For clarity and reading convenience, each section of our paper starts with a module description. *The informative analysis of the reasons and advantages of our approach is placed next, highlighting our key idea and novelty*. Due to this special structure, we think you may unconsciously overlook some insightful discussions. For example, we analyze the challenges we address and our advantages in Lines 167-170, 208-216, 231-233, and 236-244. GoMARL is an innovative approach to automatic grouping and a novel use of hypernetworks in terms of dynamic structure and fine-grained learning. Reviewer Z9U9 also recognizes our novelty. We attach great importance to your concern about our novelty and have rearranged our linguistic emphasis. We sincerely invite you to read the above lines and hope you will revisit the novelty of GoMARL.

From: ICLR.cc/2023/Conference/Paper2324/Authors
title: Response to Reviewer kSBb (PART 3)
comment: We hope our response will give you a deeper understanding and a better impression of GoMARL. *Please re-evaluate our method in light of our responses and the revised paper. It would be highly appreciated with our deep deep gratitude if all your concerns have been addressed and you are willing to raise your score.* If you have further questions or concerns, we are very happy to discuss them with you through new comments in the following discussion period.

References  
[1] Yihan Wang, et al. Dop: Off-policy multi-agent decomposed policy gradients. In International Conference on Learning Representations, 2020. (4 actors)   
[2] Tonghan Wang, et al. Roma: Multi-agent reinforcement learning with emergent roles. In International Conference on Machine Learning, 2020. (8 actors)   
[3] Jian Hu, et al. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning. arXiv e-prints arXiv:2102.03479, 2021. (8 actors)   
[4] Yaodong Yang, et al. Q-value path decomposition for deep multiagent reinforcement learning. In International Conference on Machine Learning, 2020. (8 actors)   
[5] Shariq Iqbal, et al. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference on Machine Learning, 2021. (8 actors)   
[6] Yali Du, et al. Liir: Learning individual intrinsic reward in multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 2019. (32 actors)   
[7] Meng Zhou, et al. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 2020. (32 actors)   
[8] Christian Schroeder de Witt, et al. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533, 2020.   
[9] Chao Yu, et al. The surprising effectiveness of ppo in cooperative multi-agent games. arXiv preprint arXiv:2103.01955, 2021.   
[10] Siyang Wu, et al. Containerized distributed value-based multi-agent reinforcement learning. arXiv preprint arXiv:2110.08169. 2021.   
[11] Jiechuan Jiang, Zongqing Lu. The emergence of individuality. In International Conference on Machine Learning, 2021.   
[12] Lulu Zheng, et al. Episodic multi-agent reinforcement learning with curiosity-driven exploration. Advances in Neural Information Processing Systems, 2021.   
[13] Jeewon Jeon, et al. MASER: Multi-agent reinforcement learning with subgoals generated from experience replay buffer. In International Conference on Machine Learning, 2022.   
[14] Kyunghwan Son, et al. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, 2019.   
[15] Lipeng Wan, et al. Greedy-based value representation for optimal coordination in multi-agent reinforcement learning. In International Conference on Machine Learning, 2022.

From: ICLR.cc/2023/Conference/Paper2324/Authors
title: Response to Reviewer Z9UZ
comment: We thank the reviewer for the insightful review and comments! We revised our paper in line with your suggestions and addressed your questions and concerns below. 

**Response 1: The meaning of “Group-oriented”**

Our Group-oriented MARL method realizes an automatic group learning mechanism, based on which we further propose a group-wise value factorization and introduce specialization into agents' policies. The core of GoMARL is grouping, and the policy learning is also group-driven; hence we named our method "group-oriented" MARL. 

We agree that "group-oriented may be obvious in MARL", but the truth is that a solid grouping mechanism is still an open question in the MARL community. Most existing methods require domain knowledge or a priori settings, as described in Lines 45-56. Although the idea of group-oriented learning is obvious, there is a lack of end-to-end learnable methods. Thus, in addition to training agents individually, most methods train the team as a whole and adopt flat value factorization in the CTDE paradigm, which often falls into the performance bottleneck [1]. This is why GoMARL should be proposed as a new baseline to facilitate the further development of the MARL community.

[1] Thomy Phan, Fabian Ritz, Lenz Belzner, et al. VAST: Value function factorization with variable agent sub-teams. Advances in Neural Information Processing, 2021.

**Response 2: CDS citations**

Thank you very much for your suggestion. The core of GoMARL is dynamic grouping, and the specialization is only a subsidiary enhancement, so in our old version, we only highlighted works related to grouping. During revision, we followed your suggestion and cited important ideas of CDS in the methods section and related work section. In addition, as you pointed out, "CDS solves the diversity problem in a similar but *different manner* with GoMARL"; it maintains parts shared among agents and agent-speciﬁc modules to achieve diversity. GoMARL, on the other hand, innovatively implements specialized policies while achieving full sharing among agents, so the third contribution in our Introduction is still one of our novelties.

**Response 3: Computation of $\textbf{w}_1^{g_1},\textbf{w}_1^{g_2},\dots,\textbf{w}_1^{g_m}$**

The first half of your understanding is correct, i.e., $Q^a$ is sent to $\textbf{w}_1^{g_1},\textbf{w}_1^{g_2},\dots,\textbf{w}_1^{g_m}$ to compute the mixed group-wised value function. In Fig.2(right), the illustration of $Q^a$ also has the same meaning. Due to space constraints, we omitted arrows indicating inputs, which may confuse you. We have modified Fig.2 in our revised version to help you understand our approach more clearly. 

Concretely, $\textbf{w}_1^{g_j}$ is generated by concatenating the $w_1^i$ of agents in the group $g_j$, $\forall a_i\in g_j$, where $w_1^i$ is produced by the w1 generator in Fig.2. As shown by the purple dashed line in Fig.2(right), the generated weights $w_1^i$ will only be multiplied with $Q^i$ of the corresponding agent $a_i$, without affecting other agents' $Q$. We take advantage of this network feature to achieve agents' dynamic changes in the various groups. It is also because of this feature, $\textbf{w}_1^{g_j}$ can be generated by a simple concatenation on $w_1^i$, $\forall a_i \in g_j$, to compute $Q^{g_j}$. We also refined the explanation of the concat-operation in Fig.2, and we sincerely invite you to read our revised version.

We hope our response will give you a deeper understanding and a better impression of GoMARL. Please re-evaluate our method in light of our responses and the revised submission. *It would be highly appreciated if all your concerns have been addressed and you are willing to raise your score. If you have further questions or concerns, we are very happy to discuss them with you through new comments in the following discussion period*.

From: ICLR.cc/2023/Conference/Paper2324/Authors
title: Response to Reviewer DA2S (PART 1)
comment: Thank you very much for your detailed review and helpful comments! We have revised our paper in line with your concerns and suggestions. Following are our responses to your concerns.

**Response 1: Novelty of GoMARL**

1. About hypernetworks. Hypernetwork itself cannot realize the number/size-flexible groups of GoMARL. The baseline methods in this paper all use hypernetworks, but they do not achieve GoMARL’s functionality. Our novelty is not reflected in the use of hypernetworks but in *how and why we use hypernetworks* to achieve flexible grouping. For clarity and reading convenience, each section of our paper starts with a module description. *The informative analysis of the reasons and advantages of our approach is placed next, highlighting our key idea and novelty*. Due to this special structure, you may unconsciously overlook some discussions. In Lines 167-170, 208-216, 231-233, and 236-244, we analyze the challenges we address by using hypernetworks and their advantages, namely:
   * Bind each agent (or group) to specific weights, thus allowing dynamic group size/number changes.
   * Unlike other value factorizations, we propose group-wise information to drive intra-group cooperation and utilize hypernetworks to integrate agent info $e^{a_i}$ and group-wise info $s^{g_j}$ into the gradients to guide policy learning in a fine-grained manner.

2. Regarding LASSO, our novelty lies in the automatic end-to-end group learning without any domain knowledge, which LASSO cannot achieve. In fact, the fundamental idea of the dynamic grouping is the novel sparsity-driven scheme detailed in Lines 150-161, but not LASSO, a very common regularization in DL. We appreciate its simplicity and non-parametric trait and implement our novel sparsity-driven scheme with it only as an attempt. Alternative sparsity regularization terms can also realize this functionality under the proposed scheme. Thus, LASSO is not our main focus. We have modified this part in Lines 171-176 and emphasized the novelty of the grouping mechanism instead of LASSO.

GoMARL is an absolutely innovative approach to automatic grouping and a novel use of hypernetworks in terms of dynamic structure and fine-grained learning. It is not an "effective combination of previous works." Reviewer Z9U9 also recognizes our novelty. We attach great importance to your concern about our novelty and have rearranged our submission's linguistic emphasis. We sincerely invite you to re-read the above lines and hope you will revisit the novelty of GoMARL.



**Response 2: Fragmented figures and colors with no apparent purpose**

We have tried to show all the details of GoMARL in one figure before, but we found that the figure would be too large and confusing, which is not conducive to reading and understanding. Therefore, we disassembled GoMARL and placed component figures in the textual part to facilitate a clear grasp of our design. In addition, we provided an overview and an overall architecture omitted details in Figures 1 and 4, respectively, to maintain the integrity and promote complete understanding.

The colors used in this paper are consistent. Specifically, blue denotes the agent network. Purple is used for parts related to grouping. Orange represents the mixing network used to generate group-wise $Q_{group}$, and yellow denotes the inter-group mixing network. Green shows all the hypernetworks and modules with special functions designed using hypernetworks. The only part that tends to cause ambiguity is the upper MLP of the agent network in Figures 3 and 4, consisting of a blue border and green padding, by which we wish to express that this MLP is part of the agent (blue) but is generated by a hypernetwork (green). We have modified this MLP to pure blue to remove the ambiguity. We hope our response can help you better understand our diagrams.

From: ICLR.cc/2023/Conference/Paper2324/Authors
title: Response to Reviewer DA2S (PART 2)
comment: **Response 3: Justification on experiments**

Thank you very much for your suggestion. We have detailed the experimental setup for fair comparison in Appendix C.1 of our revised version. It is worth emphasizing that the baseline algorithms in this paper use the open-source, best-performing implementation[1], which is proposed to compare methods fairly with no special code-level tricks. Although we only marked Ft for Ft-QMIX (to avoid conflict with original-QMIX), all methods heritage Ft-QMIX's finetuned hyperparameters of the shared components (e.g., the mixing network). About ROMA and RODE, many published papers produce results that significantly differ from the original papers. Even some papers by the same authors cannot reproduce the original performances. For example, the performance of ROMA in the RODE paper is significantly worse than in the original paper. The original RODE paper shows that RODE outperforms QPLEX. However, another paper[2] by the same author shows that QPLEX performs much better than RODE. In addition, papers by other researchers, such as [3-6], also have different results from the original papers in multiple environments. These conflicts may indicate that the performances of RODE and ROMA are not very stable. However, ROMA and RODE in our paper have one of the best performances among published papers, far better than their performances in the above references and even better than the original papers in some environments, *absolutely with no unfair tuning*. GoMARL's group learning is fundamentally different from role learning, and the failure of role-designating does not indicate that our group-wise value factorization is also meaningless. RODE and ROMA are both excellent and inspiring works, but their performances are unstable and not satisfactory enough. This is why GoMARL should be proposed as a new baseline to facilitate the further development of our community.

We hope our response will give you a more comprehensive understanding of GoMARL. *It would be highly appreciated if all your concerns have been addressed and you are willing to raise your score. If you have new concerns or questions after reading our response, we sincerely hope to hear from you and will be very happy to further discuss with you during the precious time before the end of the rolling discussion period*. Thank you very much again for your suggestion and contribution to our work!


[1] Jian Hu, Siyang Jiang, Seth Austin Harding, et al. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2102.03479, 2021.   
[2] Siyang Wu, Tonghan Wang, Chenghao Li, et al. Containerized distributed value-based multi-agent reinforcement learning. arXiv preprint arXiv:2110.08169. 2021.  
[3] Shariq Iqbal, Christian A Schroeder De Witt, Bei Peng, et al. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference on Machine Learning, 2021.   
[4] Jiechuan Jiang, Zongqing Lu. The emergence of individuality. In International Conference on Machine Learning, 2021.  
[5] Lulu Zheng, Jiarui Chen, Jianhao Wang, et al. Episodic multi-agent reinforcement learning with curiosity-driven exploration. Advances in Neural Information Processing Systems, 2021.  
[6] Jeewon Jeon, Woojun Kim, Whiyoung Jung, et al. MASER: Multi-agent reinforcement learning with subgoals generated from experience replay buffer. In International Conference on Machine Learning, 2022.

From: ICLR.cc/2023/Conference/Paper2324/Reviewer_DA2S
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: Authors present a grouping mechanism based on the theoretically sound variable selection algorithm LASSO and on the shape/number-flexible network scheme inspired by hypernetworks. A trainable grouping module concatenates those subsets of agents, which otherwise would have had to figure out each of their 'role' in the MARL task as if every agent were distinct individual coordinators. GoMARL is tested in various evaluation environments, and the grouping intuition is explained in empirical results with references to how each agent has indeed learned to belong in different groups.
strength_and_weaknesses: GoMARL works well, and the grouping idea seems to manifest itself as expected in empirical settings. Components have been ablated away for a more comprehensive evaluation, and, where applicable, fine tuning is done for better reproducibility and comparison.

Novelty is perhaps the biggest question facing GoMARL. Borrowing from a seminal work like LASSO and then adopting hypernetworks without much apparent modification makes the reader wonder what important and challenging problems have been solved but somehow left unexplained in the writing. Brief mentions to bio-inspired origins of grouping are left unconnected and totally isolate. Presentation also lacks a well-interpreted understanding of how GoMARL does better than other role-inducing/grouping algorithms such as RODE and ROMA. In fact, QPLEX and Ft-QMIX doing better than RODE/ROMA raises doubts as to whether the latter have been appropriately tuned.
clarity,_quality,_novelty_and_reproducibility: Clarity
The writing is easy to follow, but the figures feel fragmented and inconsistent. Colors have been assigned with no apparent purpose and only obfuscates the understanding of the figures.

Quality
Evaluation environments are varied, but the baseline tuning needs some justification (i.e., if plain value factorization schemes do better than role-designating schemes, is there any point in introducing roles at all?).

Novelty
A detailed description of what challenges were faced, circumvented/solved in order to adopt LASSO and hypernetworks would be a welcome addition to the paper.
summary_of_the_review: Paper presents an interesting and effective combination of well-known previous works. The combination is shown to demonstrate an intuitive idea of grouping. Arguments of novelty can be made much stronger, and some evaluation details need clearer explanation.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2324/Reviewer_Z9UZ
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors introduced the concept of grouping into multi-agent reinforcement learning (MARL) and provided a formulation of Group-oriented MARL (GoMARL). The contribution of this paper is as follows: 

1. They introduce an adaptive group learning solution for cooperative MARL inspired by the variable selection, partitioning agents into dynamic groups to promote efﬁcient cooperation. 

2. They present an architecture to estimate global and group-wise joint action-values with varying group numbers and agent numbers per group, realizing a dynamic end-to-end training. 

strength_and_weaknesses: Strength: 
* The idea of dynamic grouping is interesting and may have a novelty.
* Overall, the paper was well-written and I can understand most of the ideas.
* The experimental results clearly show the superiority of the proposed approach.

Weakness:
* The authors referred to CDS (Li et al. 2021) in the Introduction, but for a diversity problem in MARL, they ignored it in Sections 3,4,5. CDS also solved the diversity problem in a similar but different manner. At this stage, the third contribution in the Introduction was not contributed. 
* Although the presentation was overall clear, some points were unclear to me (described as the specific comments).

Specific comments
* Title: What does “Group-oriented” specifically mean? I did not find the explanation and I consider “group-oriented” may be obvious in MARL problems except for learning the agent individually.
* Figs 2 and 4: The computation of w_1^{g1}, …, w_1^{gm} using Q^a. In Fig. 2 left and Fig. 4, Q^a is just sent to w_1^{g1}, …, w_1^{gm} (if so, how to compute it?). However, in the group adjustment operator in Fig. 2 right, Q^a (but expressed as Q^1,...,Q^n) is used. Which is correct? And moreover, I did not find a related explanation in the main text. I hope it will become clearer. 

clarity,_quality,_novelty_and_reproducibility: The presentation was clear in total except for the above comments. The quality was also good in total because of the above contributions and strengths, but I found the above weaknesses. The experiments may be reproduced because they provided the code and may give necessary information. 

summary_of_the_review: Based on the comment above, I consider that the strength of this paper outperformed the weakness, but I had some concerns at this stage. Thus I cannot give a higher rating.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2324/Reviewer_kSBb
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: Inspired by grouping in natural systems, the authors propose a new mixer for value decomposition, which decomposes the joint action-value into group-wise values. The automatic grouping module progressively divides the team into dynamic groups. To deal with similar behaviors caused by parameter sharing,  the authors propose specialized agent networks to make the agents in the same group learn similar policies and possess diverse strategies across groups. 
strength_and_weaknesses: ### Strength 
* The paper is well-written.
* The experiments are performed on both SMAC and GRF and the proposed method seems to perform better. 

### Weaknesses

* The paper does not provide a theoretical analysis of why and how grouping helps cooperative multi-agent reinforcement learning, e.g., how group-wise decomposition influences the representation ability of the mixer. Just analogizing natural systems is not enough. Therefore, from my point of view, the paper just proposes a complicated mixer and provides an intuitive explanation.

* Due to the lack of theoretical analysis, my main concern is the experiments.
  1. The performance of QMIX is very poor. And I find the learning curves of QMIX within 10M timesteps are similar to the curves within 2M in other published papers. What is the difference? 
  2. I find Ft-QMIX shows the most similar performance compared with GoMARL. Is GoMARL built on Ft-QMIX? If so, it is unfair to compare with vanilla QMIX and other non-finetuned baselines, since code-level optimizations are different.
  3. Ablation studies are limited.  (1) Ablation studies on the automatic grouping module are missing. Automatic grouping module is an important part of GoMARL. I am curious about the performance of removing automatic grouping module and w2. (2) Ablation studies of $L_{SD}$ are also missing. (3) In the source code, I find $\lambda_{SD}$ is very small (0.03). Does $L_{SD}$ really produce an effect?
  4. I find vanilla parameter sharing shows poor performance, which means that the proposed group-wise decomposition does not work and even damages the performance (3s5z_vs_3s6z). Is the performance gain just caused by specialized agent networks？If performing specialized agent network on other baselines, will other baselines show similar performance to GoMARL? Moreover, in ablation studies of parameter sharing, the baseline without parameter sharing should also be included.
clarity,_quality,_novelty_and_reproducibility: For novelty, VAST[1] also decomposes the joint action-value into group-wise values, and the framework of VAST is very similar to Figure 1. Thus, VAST should be cited and carefully compared. 

Overall, this paper is well-organized, but Group Adjustment Operator in Figure 2 may be simplified. 

[1] Phan T, Ritz F, Belzner L, et al. VAST: Value Function Factorization with Variable Agent Sub-Teams. NeurIPS 2021. 

summary_of_the_review: Overall, the paper is not theoretically grounded, the experimental results are limited and problematic in the current form, and important related work is missing. So I think this paper needs to be further improved.

correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough