Paper: Learning Multi-Agent Communication with Contrastive Learning (ICLR-2024)

Reviewer comment (Evaluation):
The improvement over the baselines is not obvious, especially in Traffic Junction in Figure 2. The standard error is also too large with a lot of overlaps, so it may need more seeds of experiments.

Author response:
**More seeds**

For fairness, we chose the number of seeds following our closest baseline AEComm. Note that CACL outperforms AEComm on Traffic Junction more than AEComm outperformed their own baseline in their paper. The task demonstrates that even if all methods fully solve a task, CACL can reach perfect reward faster.  

**Standard errors**

We will run more seeds but may not have results in time for the end of discussion. We expect close results on Traffic Junction because we finely tuned our baselines, they are notably better than previous work [(Singh et al, 2018)](http://arxiv.org/abs/1812.09755). Our results on PP and FindGoal are still better despite strong baselines as well.

Reviewer comment (Experiments):
The proposed algorithm CACL is only tested on three tasks. The paper could benefit from additional experiments on some more challenging tasks with partial observability where communications are intuitively beneficial.

Author response:
**More challenging environments**

We use the same environments as baselines but made harder. E.g. in Predator-Prey, predators must fully surround a moving prey without being able to see each other, so communication is essential. In Find-Goal, we also reduce field-of-view to make communication key to good performance.  

**Starcraft**

Very few works in MARL communication have evaluated on SMAC, and none use decentralized training. Just recently [SMACv2](https://openreview.net/pdf?id=5OjLGiJW3u) argues that SMAC is solvable without communication since its "partial observability is not particularly meaningful". We evaluate CACL in line with previous work. Compared to decentralized training baselines, we evaluate on harder, larger tasks ([AEComm](https://arxiv.org/abs/2110.15349)) and more tasks ([PL](https://arxiv.org/abs/1912.05676)).

Reviewer comment (Theory):
The policy $\pi$ defined in section 3 seems to be only conditional on the local observation $\tau^i$. Should it also be conditional on the communicated messages?

Author response:
**Policy is conditioned on messages**

We include messages as part of the observation and have clarified this in section 3. Thank you for pointing it out.

Reviewer comment (Theory):
If some local observations miss important information, the approximated global states reconstructed by the message encoders may be very different from the true ones. Will this make the contrastive learning not meaningful?

Author response:
**CACL with different local information**

CACL does not explicitly reconstruct the whole state with each message, but brings messages closer together. So a local observation without the goal cannot communicate goal location. But its message can communicate that the goal is *not* in that location, which should be meaningfully cohesive with another agents’ message containing goal location.

Reviewer comment (Theory):
Does CACL require all-to-all communications, i.e., each agent communicate to all the other agents? If so, CACL is not scalable with large number of agents.

Author response:
**All-to-all communication is not necessary**

Other agents’ messages act as positive / negative samples, so reducing all-to-all communication is possible, it just reduces the number of positive / negative samples. Scaling to more agents increases the number of samples so limiting communication directions should be completely fine.  

**Scaling**

We believe the novelty of our method and connection to contrastive learning is our main contribution. Scaling is interesting and our method could be augmented with targeted or limited communication, as you suggested, to counter this issue. We believe our contribution is sufficient and hope to tackle scaling in future work.

Reviewer comment (Reproducibility):
During a time step, each agent receives multiple approximations of the global state from the communicated messages. This seems to include redundant information if the messages are not selectively received. How does CACL handle redundant information?

Author response:
**Redundant information**

You are correct that messages may contain redundant information, especially if the observations overlap. In our particular setup, this doesn’t seem to be a problem as the message encoder learns to extract useful information.

Reviewer comment (Experiments):
Conduct further experiments to fill in Figure 4 in more detail (instead of just 3 or 4 checkpoints along the curve)

Author response:
Thank you very much for your review. We are glad that you enjoyed our work and find it “clear”, “well-scoped [...] with good results”, and a “strong paper” that will be an “important baseline for future emergent communication work”. We are currently running more experiments to fill in Figure 4 as well as more trials for Traffic Junction, that will be included in the final paper. We would also be happy to rework Figure 3 to better present the information if the reviewer has suggestions.

Reviewer comment (Experiments):
Run more trials, especially in the traffic junction where variance is high and not all methods seem to have converged.

Author response:
Thank you very much for your review. We are glad that you enjoyed our work and find it “clear”, “well-scoped [...] with good results”, and a “strong paper” that will be an “important baseline for future emergent communication work”. We are currently running more experiments to fill in Figure 4 as well as more trials for Traffic Junction, that will be included in the final paper. We would also be happy to rework Figure 3 to better present the information if the reviewer has suggestions.

Reviewer comment (Presentation):
The example of predator prey in figure 1 (right) is a bit confusing. I would not agree that the given examples correspond to similar views; for example, the first view (counting from the top) seems more similar to the third view rather than to the second view.

Author response:
**Figure 1 reworked**

We have reworded the caption. Messages are positive samples if they are derived from [observations of] the same state. We’ve also changed the images to clarify our connection to image multi-view learning

Reviewer comment (Writing):
In the loss function, the RL loss is not defined. It would be good to have it for clarity purposes.

Author response:
**RL Loss Added**

Thank you for pointing that out. We have added a RL loss in Appendix A.1 for clarity, with text in red.

Reviewer comment (Writing):
In section 3, do the authors mean $m_{t-1}^{-i}$ instead of $m_{t-1}^{-1}$?

Author response:
**Section 3 notation**

Thank you for finding these minor errors. We have corrected it to m_{t-1}^{-i} and a^i  in the updated paper in red.

Reviewer comment (Writing):
In section 3 "At each time step, each agent $i \in N$ chooses an action $a ∈ A^i$": shoud be $a^i$ since ahead $a$ is defined as the joint action.

Author response:
**Section 3 notation**

Thank you for finding these minor errors. We have corrected it to m_{t-1}^{-i} and a^i  in the updated paper in red.

Reviewer comment (Evaluation):
I have questions about whether it is reasonable to evaluate the similarity of messages of different agents by simply looking at a window of a few timesteps in the trajectory. The observations corresponding to the generated messages can be different in important aspects from one timestep to the other, and thus would require distinct messages that could be biased by the contrastive loss. I am unsure whether this would scale to more complex cases, since it could not capture these differences in the observations.

Author response:
**Sliding Window Reasonability**

Contrastive learning doesn’t force messages within a timestep to be exactly the same, just more similar, allowing for variation within the window. We expect it to work with more complex observations e.g. [TACO (Zheng et al, 2023)](https://arxiv.org/abs/2306.13229) use a similar time window for contrastive learning in single agent RL on DM control suite and Meta-World.

Reviewer comment (Reproducibility):
The authors mention that the setting followed is a fully decentralised setting where the agents do not share parameters or gradients (section 1); does this apply to the message encoder? I.e., do the agents share the same message encoder or does each one of them use a separate encoder to generate messages?

Author response:
**Message Encoder are not shared**

Agents are fully decentralized and have completely separate message encoders, highlighting the difficulty of the task.

Reviewer comment (Experiments):
I believe another potential direction for further work would be to investigate how to make methods such as the proposed one work together with the reward. I.e., in section 5.5 it was shown that currently it is detrimental. Yet, have the authors thought whether the method can be improved in any way in order to take advantage of the reward as well?

Author response:
**Learning Representation with Reward**

Our negative results reinforce a similar finding in AEComm, and likely stem from the two training signals (SSL and RL) conflicting, causing instability. A possible future approach could alternate training one signal then the other or add an attention layer that conditions on the state to decide on a weighted mixture of the two losses.

Reviewer comment (Theory):
In section 4, the negative samples are defined as from outside the current time window or other trajectories. This is not technically sound since it would be possible for agents to encounter similar states at different trajectories (which would be considered as negative by the proposal). It is suggested that the authors should discuss such cases in detail and figure out more solid principle to decide positive/negative samples for contrastive learning.

Author response:
**CACL’s negative sample window is sufficient**

Although edge-cases can exist, contrastive learning does not require perfect negative mining, just a good general rule. E.g. [TACO (Zheng et al, 2023)](https://arxiv.org/abs/2306.13229) uses a similar time window for negative samples in contrastive learning for single agent RL.

Reviewer comment (Experiments):
The selected benchmark for comparison is kind of limited. Public MARL evaluation platforms like SMAC[1] or MATE[2] which involves higher complexity should be considered for more pursuasive comparison. In addition, in the compared task of Traffic Junction, the improvement seems to be marginal.

Author response:
**Our benchmarks are thorough and robust**

We extend and harden benchmarks used in baselines, e.g. AEComm, see section 5.1 for details, and they are nearly identical or harder versions of benchmarks in the most recent papers you reference (I2C and ToM2C). SMAC does not generally require communication. MATE is interesting but all of its communication baselines use centralized training and we focus on benchmarks with existing decentralized training baselines.

Reviewer comment (Experiments):
The compared baseline methods are sort of obselete. Newer published works in recent 3 years should get included. (especially new work in 2022-2023).

Author response:
**Baselines are SOTA**

Our baselines are current state-of-the-art in decentralized communication. Can the reviewer point to other decentralized RL + communication methods? All noted papers seem to be centralized training methods.

Reviewer comment (Experiments):
It is encouraged for the authors to demonstrate the scalability of the proposed approach, like in a continuous state/action space environments which may involve a large number of agents with quite dense communication load. In such cases, would the proposed scheme be better than the most recent communication-based work like ATOC[1], MF-MARL[2], TarMAC[3], I2C[4], ToM2C[5]?

Author response:
**Scalability of CACL**

All noted works (ATOC, TarMAC,...) are centralized training methods whereas CACL is decentralized training which should be better at scaling to large number of agents [(Lyu et al, 2021 Section 6.3)](https://arxiv.org/pdf/2102.04402.pdf). We do not investigate communication overhead, but targeting methods (e.g. TarMAC) could be applied to CACL as well.

Reviewer comment (Evaluation):
Besides showing the similarity of messages among multiple states, the exact improvement from such a contrastive learning method should be analyzed. For instance, it is better to compare the adjacency/disparity of positive/negative sample pairs before/after the proposed training.

Author response:
**Evaluating Contrastive Learning**

As requested, we plot the distance between positive and negative samples (i.e. our contrastive loss) over the course of training in Figure 9 in Appendix A.4. As training evolves, contrastive loss decreases so positive and negative samples become more separated.