Paper: Constructing Informative Subtask Representations for Multi-Agent Coordination (ICLR-2024)

Reviewer comment (Writing):
Multiple related works on MARL are included for comparison in Section 1, but the intuition behind the proposed algorithm is still not very clear.

Author response:
Thanks for your comments. LSVQ is a generic method that can be integrated with various MARL algorithms. As shown in Figure 2, the backbone MARL methods can be QMIX as well as Weighted QMIX, QPLEX, MAPPO, LICA, and so on. LSVQ introduces a subtask learner, which selects a subtask for the agent based on its observation. Then, LSVQ modifies the structure of the policy network, which additionally takes the agent's subtask as input to facilitate coordination. Under the setting of Dec-POMDP, LSVQ has to do subtask assignment with only individual observations available. Therefore, LSVQ maps the observation to latent space, and uses the latent vectors to reconstruct the true global state during training, which can be viewed as maximizing the mutual information $\mathcal{I}(s;\boldsymbol{z})$. In this way, the global information is injected into the latent vectors, and agents can infer the global state by selecting subtask $z_i$ based on its observation.

Reviewer comment (Writing):
Could you give more explanation on the statement from Section 1: "Nevertheless, these approaches aim to build differentiated subtask representations but neglect the information that ought to be incorporated within the subtask."?

Author response:
A1: We assert that a good subtask representation should be informative. This objective can be formalized as maximizing the mutual information $\mathcal{I}(s;\boldsymbol{z})$, which implies that the subtask representation can help to infer the global state. Only with the guide of global information can agents divide up the task and cooperate effectively. In previous works like LDSA, the subtask encoder is updated via complex loss function to make the representations of different subtasks distinguishable, but there is no global information included in to facilitate agents' coordination. This is where LSVQ makes improvements.

Reviewer comment (Novelty):
It would be better to include discussion and comparisons with related works on multi-agent skill (a.k.a., option) discovery.

Author response:
A2: We briefly introduce some related subtask-based studies in Section 1. A more detailed related work section will be included in the future edition.

Reviewer comment (Theory):
Could you explain why $q(z|o_a)$ is designed to be deterministic rather than a distribution on all subtasks?

Author response:
A3: We calculate $q(z|o_a)$ deterministically because we adopt vector quantization to learn the subtask representations. The representations of $K$ subtasks are saved in a codebook with shape $K\times D$ where $D$ is the dimension of representation. The observation $o_a$ is quantized to a $D$-dimensional vector, and the index of its nearest neighbor vector in the codebook corresponds to the selected subtask. This step of calculating the nearest neighbor makes $q(z|o_a)$ deterministic. Vector quantization offers many benefits. It guarantees distinguishable subtask representations, without the need for an additional loss function to control the discrepancy between them like other subtask-based methods do. It enables agents to select various subtasks in different stages and situations, as well as prevents the agent from frequently changing the selected subtasks in an episode, which could lead to unstable training.

Reviewer comment (Evaluation):
The results reported in the original paper of LDSA are clearly better than the ones shown in this paper, e.g. results on "5m_6m" and "3s5z_3s6z".

Author response:
A4: We tested LDSA with the official code released on Openreview. The difference is that in the original paper, the version of the StarCraftII game engine is 4.10. In our experiments, we use version 4.6.2 instead. Experiments in previous studies show that version 4.6.2 is more challenging, with a drop in algorithm performance observed in most scenarios. This is the reason for the discrepancy in the results of LDSA.

Reviewer comment (Novelty):
While the approach seems somewhat novel to me, I am missing a related work section that discusses conceptual differences to other works that encode observations with alternative techniques, e.g., attention, or dynamically group agents for value decomposition such as (in addition to all the works briefly listed in the introduction): [1] S. Iqbal et al., “Actor-Attention-Critic for Multi-Agent Reinforcement Learning”, ICML 2019; [2] T. Phan et al., “VAST: Value Function Factorization with Variable Agent Sub-Teams”, NeurIPS 2021; [3] M. Wen et al., "Multi-Agent Reinforcement Learning is A Sequence Modeling Problem", NeurIPS 2022.

Author response:
LSVQ introduces the concept of subtasks to improve the diversity of agents' policies. This technique is independent of the structures of the observation encoder or the value mixing network. LSVQ can be easily integrated with other methods such as QPLEX [1] and MAAC [2] and make further improvements based on their advanced modules. Therefore, we mainly compare LSVQ with other subtask-based methods in Section 1. A detailed related work section will be included in the future edition.

Reviewer comment (Theory):
According to [4], the global state cannot be reconstructed from observations alone. However, in Dec-MDPs (a special case of Dec-POMDP), the reconstruction is possible. Thus, I am not sure about the general validity of the proposed approach.

Author response:
Although the observation function $\mathcal{O}$ can be stochastic, each agent would receive a sampled observation that could reflect the original distribution to a certain extent, and LSVQ can still work in such situations. Theoretically, the global state cannot be reconstructed perfectly from observations, but fortunately the global state is available during training. According to Figure 1, we assume the input of the subtask learner is the global state and consider the individual observations as vectors that have been processed by different channels. The global state is mapped into the latent space, then LSVQ attempts to reconstruct the global state based on the latents. The state reconstruction using latents is even harder than using observations alone, but as the quality of reconstruction improves with training, the latents become more informative as the mutual information $\mathcal{I}(s;\boldsymbol{z})$ increases. So the latents can serve as good subtask representations.

Reviewer comment (Experiments):
The x-axes vary for different scenarios in Figures 3, 4, and 6 (either 2 million steps or 5 million), which is somewhat confusing. I wonder how the plots would look like, if the plots with only 2 million steps were run until 5 million steps as there could be a chance that the baselines outperform LSVQ in the long run.

Author response:
For experiments on SMAC, most studies follow this setting: 5 million steps for the most difficult scenarios, and 2 million steps for others. For example, in *27m_vs_30m*, curves begin to rise at 500k steps, and 2 million steps are enough to predict the trends of the learning curves in practice. In *corridor*, curves rise from about 1.5 million steps, so 5 million steps is a proper setting. The learning curves would always stay the trend if the training continues.

Reviewer comment (Experiments):
In the maps `5m_vs_6m`, `MMM2`, and `27m_vs_30m`, QMIX performs worse than reported in the original introduction of the benchmark [7]. Furthermore, since the original SMAC authors consider SMAC to be outdated themselves [8], I suggest to evaluate on the newer SMACv2, which exhibits more stochasticity and better aligns with the general Dec-POMDP setting.

Author response:
Results on SMACv2 will be available in the appendix in the future edition.

Reviewer comment (Novelty):
The reviewer is concerned about the novelty of this paper. It seems that HSL (Heterogeneous Skill Learning) [1] shares the similar idea of this paper. A discussion on the difference between these two works should be added.  

[1] Liu, Yuntao, et al. "Heterogeneous Skill Learning for Multi-agent Tasks." *Advances in Neural Information Processing Systems* 35 (2022): 37011-37023.

Author response:
We have read the original paper of HSL and found that HSL is quite different from LSVQ but has a lot in common with other methods. HSL encodes the one-hot vector of skill-id $z_j$ to the latent representation space and introduces a learning objective to make the representations of different skill-ids distinguishable. This approach shares the same idea with LDSA [1], **with little global information in the representation**. The skill selecting process and the loss used to update the skill selector in HSL are also very similar to those in RODE [2]. The overall loss function includes several regularization terms, and **additional hyperparameters are needed for balancing the loss terms**. Instead, LSVQ integrates both the subtask encoding and subtask assignment into the VQ-VAE framework, which has a more solid theoretical foundation and naturally avoids generating identical representations for distinct subtasks. As described in Section 3.2 of our paper, the overall loss function is concise with few additional hyperparameters.

Reviewer comment (Experiments):
The proposed method should be integrated into more advanced algorithms in MARL, such as MAPPO, HAPPO and MAT. More experiments should be conducted to strengthen the quality of this paper.

Author response:
We integrated LSVQ with QMIX and MADDPG in our experiments to show the compatibility of LSVQ with both value decomposition methods and multi-agent actor-critic methods. There are two reasons for not using more advanced algorithms. First, as the baselines [1,2,3,4] use QMIX for value mixing, LSVQ keeps consistent with them for fairness. Second, the effectiveness of LSVQ is reflected in two aspects: whether it is superior to other subtask-based methods, and how much it improves the original algorithm (e.g., LSVQ-QMIX improves QMIX). Therefore, we think it is not necessary to integrate LSVQ with more advanced methods. But we agree that it would be better if we display the results of LSVQ-MAPPO instead of LSVQ-MADDPG in the ablation study.

Reviewer comment (Experiments):
To compare LSVQ-QMIX with QMIX, the scale of the Q network in QMIX is enlarged to achieve a comparable parameter count with that of LSVQ-QMIX. In the reviewer's opinion, the scale of the LSVQ-QMIX ought to be shrunk rather than enlarging the scale of QMIX.

Author response:
We follow previous studies [1,3] which also enlarge the scale of QMIX for ablation studies. As shown in Figure 2(a) in our paper, the MLP, GRU, and the mixing network are exactly the same as those in QMIX. The design of the subtask learner follows the principle of simplicity, and further reduction of parameters can lead to insufficient network fitting capability. Therefore, we consider increasing the QMIX network for ablation to be reasonable.

Reviewer comment (Evaluation):
In Figure 10, the results of LSVQ-QMIX against other baselines on GRF scenarios are not satisfactory compared with LDSA. Reasons for performance degradation should be analyzed and provided.

Author response:
Thank you for your careful reading of our paper. We apologize for not providing an analysis of this phenomenon. The most common reason is that different tasks require different capabilities of algorithms. For example, QMIX shows stable performances in many scenarios, but it fails in *corridor* which requires sufficient exploration. To remedy this, the exploration rate in QMIX needs to be increased [5]. The design of LDSA may be better suited to GRF scenarios, but it couldn't outperform LSVQ in overall performance. Further study on the GRF results would certainly help us to understand and improve LSVQ in greater depth.

Reviewer comment (Experiments):
The method depends on finding the right number of sub-tasks K.

Author response:
In our experiments, we found that LSVQ is very stable and insensitive to its hyperparameters. Theoretically, the number of subtasks $K$ corresponds to the number of distinct policies. A small $K$ would restrict the variety of the agents' behaviors, while a large $K$ leads to over-specialization of the strategy and unstable performances of agents. As shown in Appendix B.1, we set $K=8$ in all the scenarios across different environments and get promising results. Users can tune $K$ if necessary, but $K=8$ is a moderate choice and works well.

Reviewer comment (Experiments):
Is it possible to study how entangled the latent representations and sub-tasks are? Do authors see a way in which disentanglement could help here?

Author response:
This is an interesting perspective. We are not familiar with the area of disentanglement, so we took a brief look at it. The subtasks and representations are a one-to-one match, and it is more practical to study how entangled the latent representations and the global state are. In previous subtask-based methods [1,2], an additional regularization term is needed to make the representations of distinct subtasks distinguishable. Meanwhile, disentanglement helps to learn mutually independent subtask representations, which can replace the regularization term mentioned above and implicitly increase the discrepancy between representations with a more solid theoretical foundation. Existing disentanglement methods mainly focus on VAEs, and further study is required to combine disentanglement with VQ-VAE in our paper.

Reviewer comment (Experiments):
How much tuning is needed and how difficult is it to find the right sub-task representation size for each method?

Author response:
In our experiments, we found that LSVQ is very stable and insensitive to its hyperparameters. Theoretically, the number of subtasks $K$ corresponds to the number of distinct policies. A small $K$ would restrict the variety of the agents' behaviors, while a large $K$ leads to over-specialization of the strategy and unstable performances of agents. As shown in Appendix B.1, we set $K=8$ in all the scenarios across different environments and get promising results. Users can tune $K$ if necessary, but $K=8$ is a moderate choice and works well.