Paper: Generalising Multi-Agent Cooperation through Task-Agnostic Communication (ICLR-2024)

Reviewer comment (Experiments):
The weakness of the approach lies in the method of generating offline datasets, which involves applying a random policy in different environments. In such a setup, the offline dataset often contains episodes with low returns, and with these episodes, it is hard to learn an informative representation due to this inefficient exploration strategy. It is essential to acknowledge that the quality of the dataset is crucial, given that the encoder aims to distill intrinsic information from the environment.

Author response:
Great point. The offline dataset may contain episodes with low returns with respect to one of the tasks within the environment that we learn online. Fortunately, our autoencoder is trained without the reward. This means that even in very sparse reward scenarios, we should be able to learn a good state representation. If this remains an issue, future work could mitigate this problem (without biasing the offline dataset) by using curiosity-driven exploration (Pathak et al., 2017) instead to sample sparse states more frequently (see our limitations section).

Reviewer comment (Experiments):
Another potential issue is dataset bias. The authors have not explicitly stated that the dataset was generated across different tasks. In a task-agnostic setting, the learned latent variable should ideally capture data from an environment with a distribution of tasks, facilitating the learning of environment-specific representations.

Author response:
Our offline dataset is collected through random exploration in the environment. As a result, there is no reward signal (and hence, no task bias) in our offline data. The task-agnostic strategy we learn can therefore be applied to any task in the environment via defining any reward function within it. For example, when we pre-train the set autoencoder for the Discovery task, we sample data randomly from the environment. This environment can have the Discovery task or the Flocking task (or any other suitable task) defined within it. Since we sample randomly, there is no task bias in the offline dataset.

Reviewer comment (Experiments):
The same for the Discovery, where the differences between tasks appear to be primarily related to the differently generated locations of targets. If I am correct, this might not provide a strong basis for presenting distinct tasks.

Author response:
A task $\tau$ within an environment $\mathcal{E}$ is defined by a reward function $R_{\tau, \mathcal{E}}(s, a)$ (please see our explanation in the introduction). Thus, two tasks within the same environment are distinguished by their environment and task-relative reward functions. For example, in collaborative cooking, the reward for *cramped* and *circuit* differ due to alternative arrangements of the cooking stations, pots, and ingredients. The rewards yielded for the same agent state are different as a result. Similarly, going from Flocking to Discovery, the reward is now for agents to "cover" targets rather than flocking amongst them.

Reviewer comment (Evaluation):
The evaluation presented is incomplete. The author only demonstrates results for a single task rather than a distribution of tasks. A robust task-agnostic method should ideally outperform other baselines across a range of tasks within a specific environment.

Author response:
A limitation of existing benchmarks is that it is very difficult to find a range of challenging alternative tasks from the same environment. We have done our best to accommodate this gap in the field by demonstrating our task-agnostic method is effective in multiple different types of environments, action and observation spaces, and simulators for at least two different tasks in the same environment. While we developed this paper, we deliberated extensively around which environments are suitable to test this work. We have included a discussion around this in Appendix F, including reasons why 3D environments like SMAC and GRF are unsuitable. In addition, we would like to highlight that Melting Pot is becoming a recognised MARL benchmark, featuring as a NeurIPS 2023 challenge [B] and that VMAS has been used in recent state-of-the-art work [C, D], including being directly supported by the new TorchRL library [E]. Both are known and growing benchmarks in the community.

Reviewer comment (Experiments):
OOD setting is interesting, but I was wondering if it possible to unfreeze the parameters of the autoencoder when training policy under the OOD setting. This would shed light on how adaptive the communication strategy truly is.

Author response:
This is an interesting point. However, we believe fine-tuning task-agnostic communication is out-of-scope as it does not fit the objective of our work, which is to achieve sample-efficient communication by training communication only once and deploying it to solve multiple tasks. Thank you for raising this point. It is still interesting to consider ways to improve adaptability without sacrificing task-agnostic communication by, for example, improving unbiased offline dataset collection instead.

Reviewer comment (Experiments):
Environments are limited to 2-D environments. Some more complicated 3D environments are truly expected.

Author response:
A limitation of existing benchmarks is that it is very difficult to find a range of challenging alternative tasks from the same environment. We have done our best to accommodate this gap in the field by demonstrating our task-agnostic method is effective in multiple different types of environments, action and observation spaces, and simulators for at least two different tasks in the same environment. While we developed this paper, we deliberated extensively around which environments are suitable to test this work. We have included a discussion around this in Appendix F, including reasons why 3D environments like SMAC and GRF are unsuitable. In addition, we would like to highlight that Melting Pot is becoming a recognised MARL benchmark, featuring as a NeurIPS 2023 challenge [B] and that VMAS has been used in recent state-of-the-art work [C, D], including being directly supported by the new TorchRL library [E]. Both are known and growing benchmarks in the community.

Reviewer comment (Experiments):
The task-specific method used as a baseline is not explained in the main text (how and what does it learn?), making it hard to understand how it can be compared with the task-agnostic method.

Author response:
Thanks for highlighting this. We have now clarified this point in Section 4.2.

Reviewer comment (Experiments):
The no-comms baseline is not expected to be a strong competitor since it has less information than the other two methods.

Author response:
Indeed. We included this baseline to verify that communication was being utilised.

Reviewer comment (Experiments):
Pre-training the method purely offline with random policies may be problematic because the other agents’ observations depend on their policies. If the agents act randomly, their observations differ from when they act according to a specific policy, which can lead to serious problems in some environments.

Author response:
Great point. The offline dataset may contain episodes with low returns with respect to one of the tasks within the environment that we learn online. Fortunately, our autoencoder is trained without the reward. This means that even in very sparse reward scenarios, we should be able to learn a good state representation. If this remains an issue, future work could mitigate this problem (without biasing the offline dataset) by using curiosity-driven exploration (Pathak et al., 2017) instead to sample sparse states more frequently (see our limitations section). Fine-tuning task-agnostic communication is not a suitable approach. It does not fit the objective of our work, which is to achieve sample-efficient communication by training communication only once and deploying it to solve multiple tasks. We suggest future work should focus on improving unbiased offline dataset collection instead. Thanks for making this important point. We will add more discussion on this topic to the paper.

Reviewer comment (Theory):
The convergence and regret results can be considered out-of-scope or distracting, as the method does not contribute to reinforcement learning itself but rather to encoding observations. In partially observable environments, encoders are common, so analysing convergence and regret here is not particularly interesting.

Author response:
We agree that 3.1 and 3.2 may break the flow of the paper and this could warrant moving them to an appendix. Thank you for affirming this. Nevertheless, these theorems are novel in the context, showing that our approach of reconstructing the Markov state leads to interesting and important properties that are not general to all encoder approaches. For example, Theorem 3.2 warrants great practicality. When there is an information bottleneck (such as the latent state) or when deploying our approach in realistic scenarios, like where communication connectivity between the entire multi-agent team is not possible, we still know that the regret scales linearly. As a result, the paper would be incomplete without stating Theorem 3.1 and 3.2.

Reviewer comment (Theory):
The limitation of acting on a Markov state is not discussed. Specifically, even with a fixed latent vector dimension, as the number of agents grows, the information from all agents scales exponentially. Methods that prune communication or transmit only task-relevant information do not face this limitation.

Author response:
Great observation. Whilst we have demonstrated in 4.3 that we can scale far beyond the training distribution, the limitations of the fixed-size latent state can be seen when we reach 10 agents. As you insightfully suggest, to go beyond this, it is possible to incorporate methods which prune communication on top of our work. This would only affect which elements appear in the set that the set autoencoder encodes. As a result, they can be applied to the communication topology without any modifications to our method.

Reviewer comment (Experiments):
Why is the focus restricted to dec.MDPs instead of dec-POMDPs? Even though in dec-POMDPs the joint observation is not necessarily the state, it is a more common setting in multi-agent reinforcement learning, and the proposed method could be used to estimate the joint observation instead. This generalization would also make it possible to use more common benchmark environments (SMAC, MPE, LBF, etc.).

Author response:
This is a great question. We restrict our theoretical focus to Dec-MDPs because the field in general has not solved POMDPs or Dec-POMDPs. Focusing on Dec-MDPs allows us to motivate our method by focusing on reconstructing the Markov state which brings several nice results (particularly Theorem 3.2, which is useful in practice). Nevertheless, we agree that Dec-POMDPs are a more common setting in the literature and have included an additional comment in the paper describing how Theorem 3.2 applies to Dec-POMDPs. We emphasise also that the benchmark tasks we use in both VMAS and Melting Pot _are also Dec-POMDPs_. We have included a discussion in Appendix F regarding our choice of MARL benchmarks. In addition, we would like to highlight that Melting Pot is becoming a recognised MARL benchmark, featuring as a NeurIPS 2023 challenge [B] and that VMAS has been used in recent state-of-the-art work [C, D], including being directly supported by the new TorchRL library [E]. Both are known and growing benchmarks in the community.

Reviewer comment (Experiments):
In the experiments, do the policies share parameters or do they learn independently?

Author response:
For our experiments on Melting Pot, the policies do not share parameters. For experiments on VMAS, they do. Please see Appendix C for further details. Thank you for highlighting this.

Reviewer comment (Presentation):
In figure 2, does $N$ represent the number of agents (which should be $n$) or the number of close-by neighbors?

Author response:
In Section 4.2, $N$ is the number of agents. In real world deployment however, this should be the number of agents within the communication range of agent $i$.

Reviewer comment (Writing):
In the conclusions, what does it mean to use “full connectivity between agents”?

Author response:
We assume agents are within the communication range of all other agents in our experiments. In graph neural network-based communication, this is not required as information can be propagated to out-of-range agents via agents that are in range. In the conclusion, we describe how something similar may be implemented using our method to overcome this full connectivity assumption. We will clarify this in the paper. Thank you.

Reviewer comment (Experiments):
The experimental environment is overly simplistic; I recommend employing more convincing and commonly used benchmarks such as SMAC and GRF.

Author response:
Thank you for this observation. We agree that challenging environments are essential to demonstrate the efficacy of research in MARL. While we were developing this work, we deliberated thoroughly on the experimental benchmark, considering both SMAC and Google Research Football (GRF) in the process. We believe we have good reasons for choosing not to use these (perhaps more well-known) benchmarks, as follows:

While SMAC is commonly used in prior literature, many of its environments can be solved with open-loop policies (i.e. with observations of just the agent ID and time step) [A]. As a result, communication is not necessary to solve it. While SMACv2 resolves some of these issues, the objective remains simply to kill all the enemy agents. Consequently, this environment does not have enough task variation to test task-agnostic communication, the main contribution of this paper.

GRF is indeed challenging. However, it is not a suitable benchmark to evaluate communication as it provides full pixel-frame observations, thus giving the entire visual state to each agent and eliminating any need for communication. Moreover, GRF allows control over only a single "active" player at a time. Since only one agent is active at any time step, communication is not applicable in the usual sense, and the environment is neither a Dec-MDP nor Dec-POMDP. Like SMAC, GRF also does not feature task variation as the only objective is to win at the game of football.

We believe Melting Pot represents similarly (if not more) challenging tasks. Like SMAC and GRF it features high-dimensional pixel-based observations and complex objectives. It is a new, but state-of-the-art benchmark, even featuring as a NeurIPS 2023 challenge [B]. Hence, it is quickly becoming a recognised MARL benchmark. VMAS is also an existing and known benchmark, having been used in recent state-of-the-art work [C, D], including being directly supported by the new TorchRL library [E]. The visualisations of VMAS appear simple, but the dynamics are complex, going beyond kinematics by simulating elastic collisions, rotations, and joints. Thus, while the environments are conceptually basic, VMAS still represents a significant (and _realistic_) challenge to agents.

Finally, we would like to emphasise that although we use IPPO in our experiments, our method is algorithm-agnostic. As new state-of-the-art algorithms and benchmarks appear, our approach can be applied to these new algorithms to solve even more challenging problems as the field evolves.

Thank you for bringing this nuanced issue to light and allowing us to discuss this problem in detail. We have added wording to further elaborate our choice of simulators in the paper to clarify our rationale (in particular, please see Appendix F). We will continue to consider what benchmarks are available and update the paper if we can find suitable environments to conduct further tests on. We believe challenging MARL benchmarks are essential for the progression of the field.

Reviewer comment (Theory):
In essence, the paper models the global environment in advance through the use of an autoencoder. If the global state can be reconstructed from the partial state, it implies that the invisible state can be deduced from the visible state. In this case, there would be no need for communication.

Author response:
Interesting point! We are glad to discuss this. Consider a problem, like Discovery, where agents must "cover" targets (i.e. have multiple agents come within some range of a target) to gain reward. Now, if the agents are invisible, the only way to solve this task is for the agents to communicate their positions and whether they have discovered a target in order for other agents to go to them and cover the target. Thus, communication is required.

This example elucidates the more general point that in a Dec-MDP (and in Dec-POMDPs in general), the global state cannot be reconstructed from the partial state seen by an agent. In our work, we do not imply that the global state can be reconstructed from the partial state. We attempt to approximate the global state using the information that we have (whichever observations we have received from other agents). Our set autoencoder has learned the latent information/underlying patterns during pre-training which allow it to attempt to do so. In a Dec-MDP, when we have information from all agents, we have enough information to theoretically reconstruct the global state—but communication is required in order to get information from all other agents.

Thank you for highlighting this important point. We are working to make this clearer in the paper.

Reviewer comment (Writing):
Some typos: page3 As the the global observation

Author response:
Great catch! We have fixed this in the paper. Thank you.

Reviewer comment (Theory):
Theorem 3.1 and 3.2 seem to be trivial.

Author response:
3.1 and 3.2 are simple and we agree that this may warrant moving them to an appendix. Thank you for affirming this. Nevertheless, these theorems are novel in the context, showing that our approach of reconstructing the Markov state leads to interesting and important properties. For example, Theorem 3.2 warrants great practicality. When there is an information bottleneck (such as the latent state) or when deploying our approach in realistic scenarios, like where communication connectivity between the entire multi-agent team is not possible, we still know that the regret scales linearly. As a result, the paper would be incomplete without stating Theorem 3.1 and 3.2. We agree that it should perhaps consume less of the main body and we appreciate you highlighting this.

Reviewer comment (Reproducibility):
What are the detailed structures of the autoencoder?

Author response:
Thank you for highlighting that this is missing. We are working towards ensuring that the paper is completely self-contained. We have now added a full description of the set autoencoder to the paper (see Appendix B).

A description of the architecture of the autoencoder can also be found in (Kortvelesy et al., 2023, Figure 1). In this architecture, an encoder assigns input elements (known as "keys") to values and encodes them with a neural network. Simultaneously, it encodes the keys with another network. The sum of the element-wise products of the encoded values and keys forms the latent state to which a cardinality embedding is added. A similar decoding process decodes the cardinality of the set and produces the corresponding number of keys and "queries". The element-wise product of the queries by the keys produces a "disentangled" encoding which is decoded by a final network that produces the reconstructed set. For details of the hyperparameters we use for this set autoencoder, please see Appendix E (particularly paragraph 2). Note that we use the default implementation provided by the author, which can also be found at [A].

Thank you again for pointing out this gap in the paper. We have added a more complete description of the architecture.

Reviewer comment (Reproducibility):
What is the exact version of the task-specific algorithm used in experiments?

Author response:
Great question! We use IPPO, just as we do for the other baselines (we have detailed our training setup in Section 4.1 and Appendix E). The only difference between the task-specific and the task-agnostic baselines is that instead of pre-training the set autoencoder with random samples in the environment, we pre-train the set autoencoder by learning a policy (again with IPPO) to execute a _similar_ (but clearly distinct) task in the same environment. Then, we take this set autoencoder, freeze its weights, and deploy it to learn the main task (just as we do with the task-agnostic baseline). We describe which tasks we do pre-training on for each corresponding main task in Section 4.2. Thanks again for pointing this out. We will ensure to make this clearer in the main text.