Paper: Solving Homogeneous and Heterogeneous Cooperative Tasks with Greedy Sequential Execution (ICLR-2024)

Reviewer comment (Experiments):
The algorithm is tested exclusively on specially designed tasks. There are no additional experiments conducted on common benchmarks.

Author response:
1. The algorithm is tested exclusively on specially designed tasks. 

    a. Our work aims to address the challenge of learning both homogeneous and heterogeneous policies simultaneously in value decomposition methods. **Therefore, it is crucial to validate our approach in tasks that require both types of policies.** Unfortunately, existing environments such as SMAC, GRF, and overcooked do not meet this requirement, which necessitated the design of new scenarios. 

    b. Furthermore, the tasks encompass a variety of behaviors necessary for completion, and the effectiveness of a task is influenced not just by the environment, but also by the nature of the policy required for its completion. In the environments we have utilized, agents are required to concurrently learn both homogeneous and heterogeneous policies. This dual learning requirement substantially elevates the complexity of these tasks, making them challenging and rich in diversity.

Reviewer comment (Experiments):
The ablation study section fails to provide an in-depth discussion of the functions of the key components in the algorithm. I would appreciate a more detailed description of the ablated algorithms and a clearer explanation of the results.

Author response:
2. About a more detailed description of the ablated algorithms and a clearer explanation of the results. 
    
    We provide the following clearer explanation of the results in our ablation study and modify the revision accordingly: 
    
    a. **Ablation of Greedy Actions:** Our first ablation test involved evaluating our method without using greedy actions, meaning we relied on marginal contributions instead of greedy marginal contributions. The results indicated a significant performance degradation when greedy actions were not utilized. This finding aligns with our analysis in Section 4.2, where we discussed how using greedy actions helps overcome the relative over-generalization problem. 

    b. **Ablation of Marginal Contributions:** The second ablation focused on evaluating our method without marginal contributions. In this scenario, we did not use the critic as proposed in Section 4.1 and directly fitted the policy value function Q_s^i to Q_{tot}. The results demonstrated the challenges of using an implicit credit assignment method to learn the policy when the sequential execution policy does not satisfy the IGM principle, underscoring the importance of our proposed critic. 

    c. **Effect of Number of Sampling in MC Method:** We also investigated the impact of varying the number of sampling in the Monte Carlo (MC) method. The findings showed that a small number of samples could lead to suboptimal action selection, failing to identify the greedy value of actions. Conversely, a sufficiently large number of sampling (beyond 5) did not yield additional performance improvements, indicating an optimal range for the number of sampling. 

    d. **Performance with Increased Numbers of Agents:** Lastly, we tested our method in the Lift task with double and triple the number of agents. These results affirmed our method's robustness, demonstrating effective performance management even with a larger number of agents. 
    
    We believe these clarifications provide a comprehensive understanding of the key components and functions in our algorithm, as well as the implications of each ablation test conducted.

Reviewer comment (Evaluation):
To generate a joint action $u$, each agent must calculate its action $a$ sequentially, which prevents parallel processing. Therefore, with each agent requiring $t$ time to produce its action, a single environment step requires $n*t$ units of time for $n$ agents. Compared with baseline algorithms like QMIX, does GSE require more wall-clock time for training and evaluation? If so, could you provide a rough estimate of the additional time required?

Author response:
3. About the additional time required for training and evaluation. 
    
    It is true that the sequential execution in GSE necessitates more time per step, as each agent calculates its action sequentially. This additional execution time can be estimated as (n-1)*t for n agents, where t is the time taken by each agent to produce its action. However, it's important to note that **this sequential execution primarily impacts the execution phase and not the training phase**. During training, as the transitions are collected, the sequential nature of execution does not impede the parallel processing capabilities. Therefore, the additional time required during training is mitigated. We appreciate the suggestion to focus on minimizing additional execution time. We have included a discussion about this limitation in our work, underscoring its significance. Thank you for this valuable input.

Reviewer comment (Reproducibility):
During training, is the execution sequence of all agents fixed, or do you shuffle the sequence?

Author response:
4. During training, is the execution sequence of all agents fixed, or do you shuffle the sequence?
    
    We employ a fixed order for the execution sequence of agents, which is determined based on the agent IDs.

Reviewer comment (Writing):
What does 'training with a larger scale of agents' mean in Section 6.3 & Figure 6?

Author response:
5. What does 'training with a larger scale of agents' mean in Section 6.3 & Figure 6? 
    
    We tested our method in the Lift task with double and triple the number of agents. These results affirmed our method's robustness, demonstrating effective performance management even with a larger number of agents.

Reviewer comment (Presentation):
Figure 2 with the architecture is hard to read and not well explained.

Author response:
We have added a more detailed description to accompany Figure 2 in our manuscript. This includes:  
    
- Upper (Blue): A depiction of the critic architecture that we elaborated on in Section 4.1.  
- Lower (Green): An illustration of the framework used for calculating the greedy marginal contribution. This is based on $Q_c^i$ and $V_c^i$, as well as the sequential execution policy $Q_s^i$.

Reviewer comment (Reproducibility):
The use of Monte Carlo method to estimate the optimal cooperative actions with previous agents to maximize the greedy marginal contribution is mentioned with no other details. How is the number of actions selected?

Author response:
The number of actions selected for this estimation process is treated as a hyperparameter in our method. In our ablation studies, we have evaluated the effect of different numbers of sampling on the final results. Specifically, we examined how the sample number M affects performance. The results demonstrate that a small value of M can be problematic as it may not select the greedy value of actions. However, a reasonably large value of M is sufficient as increasing M beyond 5 does not further improve performance.

Reviewer comment (Experiments):
In the Overcooked scenario, I understand why the size of the map affects the complexity, but there is no indication of how large the maps are. I believe the number of agents is fixed to two. Have you tried a larger number of agents?

Author response:
To provide a clearer understanding of the environment's complexity, we have included more detailed information about the Overcooked scenario and schematic representations of the maps in the appendix. The sizes of the maps used in our experiments are 5x5 for easy and medium and 7x7 for hard.  
    
    Regarding the exploration with a larger number of agents, we have conducted evaluations in the Lift task using double or triple agents. The results of these evaluations are presented in the fourth figure of Figure 6. These results demonstrate that our method effectively handles a larger number of agents, maintaining performance without significant degradation.

Reviewer comment (Evaluation):
A trend that should be discussed more in detail is in Overcooked, where there is a large standard deviation for both Easy and Medium, with Easy having a decreasing return past 900 episodes. It seems that it didn't converge.

Author response:
The performance in the Overcooked environment is impacted by a variety of factors. **Our settings of difficulty are calibrated based on the ease of discovering rewards.** In simpler scenarios, characterized by more interactive objects and smaller maps, agents have increased opportunities for interaction. **This design, while lowering the barrier for exploration, potentially escalates the difficulty of policy learning as agents progress.** For instance, in scenarios with multiple onions, agents may quickly learn the value of picking up an onion. However, as they develop, they might attempt to pick up different onions, leading to mis-cooperation. This issue is less likely in harder scenarios with fewer onions. These are the reasons that lead to some degree of fluctuation. **However, since the primary challenge of learning is relative overgeneralization and the difficulty in discovering successful instances, our setting is appropriate.** We have added a discussion about this phenomenon in our work. Thank you once again for this suggestion.

Reviewer comment (Presentation):
Section V already introduces comparison methods that are presented in Section VI. It is better instead to introduce the comparison methods in Section V so that the reader doesn't have to guess what methods are they.

Author response:
We have reorganized the content to introduce the comparison methods directly in Section V. We believe that this revised order will enhance the readability and coherence of our manuscript, making it easier for readers to follow the logic and rationale behind our comparative analysis.

Reviewer comment (Presentation):
It is worth including graphically the map for the overcooked environment.

Author response:
We have included images of the map in the appendix of our manuscript.

Reviewer comment (Presentation):
Please ensure to include the correct venue for papers, instead of just including the arxiv version, e.g., the MAVEN paper was published in NeurIPS 2019.

Author response:
We have revised the citation format throughout our manuscript. We have also updated the manuscript to include the correct publication venues for referenced papers. We appreciate your guidance on these matters and believe these revisions enhance the accuracy and professionalism of our manuscript.