From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: Summary:
This paper studies emergent communication in multiagent populations. First it shows a surprising negative result: when all agents communicate with each other, the equilibrium communication strategy is independent of population size. Moreover, this strategy need not have the properties of a structured language. However, the paper goes on the demonstrate, if senders and receivers are partitioned in a particular way, then structured language does emerge, and all agents converge on the same mutually intelligible structure.

Strengths:
* Well motivated with interesting background psychology
* Surprising findings that shed light on prior failures of scaling up multiagent communication
* Good theoretical analysis
* Interesting empirical results

Weaknesses:
* The experiments are limited to just a few referential games


justification_for_why_not_higher_score: All reviewers advocate accepting this paper, finding it to be informative and sound. No major weaknesses were identified, but the reviewers do provide many suggestions that could improve the paper. I concur with the consensus and find the paper very interesting. I think poster is the appropriate level given that, although the paper will potentially be impactful, the results are not quite there yet to show if the method is broadly applicable and useful in practice.
justification_for_why_not_lower_score: See above.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper4630/Reviewer_uuc8
title: Thanks for your reply!
comment: Thanks for your detailed response. The response addressed most of my concerns about the experiments. I recommend further discussing the limitation and future work in the main paper. It is helpful for follow-up work.

From: ICLR.cc/2023/Conference/Paper4630/Reviewer_ULLE
title: Reviewer response
comment: Thank you for taking the time to respond to my comments.  I believe the latest changes address my concerns with the work.

From: ICLR.cc/2023/Conference/Paper4630/Authors
title: General Response
comment: We thank all reviewers for their helpful feedback and suggestions. We responded to their individual comments in dedicated threads. We have also uploaded a new revision of the paper addressing some of their concerns. In the revised draft, new passages are indicated in blue

The main changes in this version are:
- revised the beginning of the introduction and Section 6.1 to better justify why emergent languages that are more structured are desirable
- added forward references to the main sections in the introduction, to make the flow of the paper more transparent
- included additional results for the “New partners” experiment with different graph types (ring lattices with higher connectivity)

From: ICLR.cc/2023/Conference/Paper4630/Authors
title: Response to Reviewer 2JSn
comment: We thank the reviewer for their time and in particular for their in-depth suggestions regarding the presentation of the paper. We have attempted to address the most salient points while contending with the limited space. Specifically, we made the following changes to the paper (indicated in blue in our revision)

> In the introduction, adding what you do in each section

In the last two paragraphs of the introduction, we have added forward links to the relevant sections where appropriate.

> You have explained that "standard procedure" do not showcase the features of the reality. You could consider elaborating on that and explain more on how partitioning allows those features (of course you have explained them- you could consider elaborating and being more precise ), how scaling up to populations can be beneficial and how the way it is done matters. What are the implications and broader connections to natural language?

Reviewer ULLE also voiced a similar concern. We made several modifications to the introduction and to Section 6.1 to emphasize the fact that emergent languages that showcase more “natural language”-like features may be more relevant for downstream applications. In particular, we made the following changes:

- Changed the end of the first paragraph in the introduction to: “This line of work has applications to semi-supervised learning. **For example, agents that develop languages exhibiting universal properties of natural languages may be used as useful initialization for downstream tasks such as image captioning (Lazaridou et al. 2020) or representation learning (Dessi et al. 2021)**.”
- Added a sentence in the second paragraph of the introduction: “for example agents are able to solve the task successfully while using languages that are not compositional (Kottur et al., 2017; Chaabouni et al., 2020). **These idiosyncrasies of the emergent languages can preclude their use in practical applications (Lazaridou et al., 2020).**"
- Modified the end of the first paragraph in 6.1 to: “**It has been argued that compositionality makes languages easier to learn (Davidson, 1965). Consequently, emergent communication protocols that are compositional may ultimately be easier to understand by humans**.”

From: ICLR.cc/2023/Conference/Paper4630/Authors
title: Response to Reviewer uuc8
comment: We thank the reviewer for their feedback. Below, we address their specific concerns individually:

> Two communication graphs are adopted in the experiments, as shown in figure 2. However, the fully-connected graph and the circular graph are the two extreme cases. Maybe it is necessary to add a randomly sampled graph as a reference.

We agree with the reviewer that the paper would be stronger with experiments on graphs situated between the two extremes of circular and fully connected graphs. Accordingly, we have updated the submission to include experiments on a wider variety of graphs.

We do not report results on completely random graphs as this may introduce a large number of factors of variation, for instance: is the graph connected? What is the diameter? What is the average path length between any two agents? What is the distribution of the number of neighbors of each agent? Etc… (See [Kim & Oh, 2021](https://proceedings.neurips.cc/paper/2021/file/92dfa194391a59dc65b88b704599dbd6-Paper.pdf))

Instead, we report results on ring graphs of increased connectivity. A ring graph with connectivity $k$ is essentially a circular graph where each vertex is connected to its $k$-nearest neighbors on each side. This allows us to interpolate between circular populations ($k=1$) and fully connected populations ($k=n/2$) while jointly (1) increasing the number of communication partner of a given agent and (2) decreasing the path length between any two agents, two properties of the graph that are arguably most impactful on the mutual intelligibility between agents.

We perform the same experiment as Section 5 with graphs of connectivities $k=2$ and $k=3$ (resp. 4 and 6 training partners for each agent) on the Attribute/values dataset. We report results in Appendix D in the revision. We observe that across the board, populations with greater connectivity exhibit higher accuracies in both conditions, but that  "New partners" accuracy is still higher in partitioned populations.

>The experiments are all conducted on the referential games, where the receiver only needs to pick a candidate from a set as the prediction of the original object. What if we extend the game into a sequential decision problem? For example, the sender may send messages as the instructions, and the receiver needs to achieve some goals (such as navigation to the goals) by following such instructions. I wonder whether the proposed method can be applied to these games.

We have kept the focus on the referential game due to its simplicity (success at the task is trivially conditioned on successful communication) and ubiquity in the literature. That being said, experimenting with partitioning (and populations in general) in broader varieties of communication games would certainly be a promising avenue for future work. In general, we expect partitioning to help in cases where populations of agents perform different roles (e.g. senders and receivers in our work, instruction givers and followers in the reviewer’s example) and where co-adaptation may lead to idiosyncratic communication protocols.

From: ICLR.cc/2023/Conference/Paper4630/Authors
title: Response to Reviewer ULLE
comment: We thank the reviewer for their very encouraging feedback. Below, we address their specific concerns:

> the value of learning more "structured" languages in this setting is not fully articulated

We thank the reviewer for pointing out this concern, which was also raised by reviewer 2JSn. We have rephrased the paper in several places, notably in the introduction and in Section 6.1 (“Measuring Compositionality”) to emphasize that languages that exhibit more structure may be more useful in downstream tasks. The modified passages are indicated in blue in the revised submission. In particular, we made the following changes:

- Changed the end of the first paragraph in the introduction to: “This line of work has applications to semi-supervised learning. **For example, agents that develop languages exhibiting universal properties of natural languages may be used as useful initialization for downstream tasks such as image captioning (Lazaridou et al. 2020) or representation learning (Dessi et al. 2021)**.”
- Added a sentence in the second paragraph of the introduction: “for example agents are able to solve the task successfully while using languages that are not compositional (Kottur et al., 2017; Chaabouni et al., 2020). **These idiosyncrasies of the emergent languages can preclude their use in practical applications (Lazaridou et al., 2020).**"
- Modified the end of the first paragraph in 6.1 to: “**It has been argued that compositionality makes languages easier to learn (Davidson, 1965). Consequently, emergent communication protocols that are compositional may ultimately be easier to understand by humans**.”

> the training objectives form general-sum game between the senders and receivers, and so some discussion of whether behaviors reliably converge, or whether cyclic behavior was observed, would be useful.

The reviewer is correct that there is no theoretical guarantee that simultaneous gradient converges. In practice, we find that with appropriate choice of hyper-parameters (typically a small enough learning rate, entropy regularization and the ADAM optimizer), behaviours converge consistently. This is in line with what has been observed in related work. In the revised submission, we have added a sentence to acknowledge this fact in Section 4.3 (“Population Training”)

> Examples of the languages actually learned by the agents would help the reader understand the practical significance of the topological similarity measure.

This is an interesting suggestion. If time permits, we will try to look through agent outputs and see if there is some discernible pattern which we might report as qualitative examples in the appendix.

> a more detailed analysis of whether there is a trade-off between the topological similarity and the accuracy of the learned languages in the ImageNet task would be helpful in understanding whether the partitioned approach to training might be detrimental in some cases.

We did not report topographic similarities on the ImageNet dataset since the metric does not enjoy the same properties as the attribute/values objects on images. Specifically, distance in pixel spaces do not really correspond to what we would expect to be meaningful differences in natural language descriptions for example. This is a known downside of the metric on image data (see e.g. Chaabouni et al., 2022).

That being said, on the attribute/values dataset, we did not find that partitioning significantly impacted accuracy. However, we observed that partitioning generally made convergence slower (as seen in Figure 4 for example).

> A minor point, but using pi to denote both a sender's strategy, and the posterior over objects induced by that strategy is somewhat confusing when reading Section 3, so perhaps a different notation should be considered.

We agree with this criticism. We originally used pi to underline the fact that the optimal posterior is tied to the speaker’s policies. We understand that this can be confusing, but we could not come up with an alternative notation that made the distinction between strategy and posterior clearer while still highlighting their relationship. Any suggestion from the reviewer would be greatly appreciated.

[Edited to fix formatting]

From: ICLR.cc/2023/Conference/Paper4630/Reviewer_ULLE
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper considers the problem of using population-based training to learn more "natural" languages for communicating between agents.  When two agents (a "sender" and a "receiver") are trained to solve a communication task together, the communication protocol, or language, that they learn often lacks structure, and simply maps arbitrary strings of symbols to different objects.  Surprisingly, training populations agents, and randomly pairing senders and receivers, does not lead to the agents converging to a more significantly structured language than training a single pair of agents.  This paper argues that this is due to the fact that no matter how large the population, the population-level training objective can be maximized by a single receiver policy that simply memorizes the (arbitrary) languages of each sender.

To address this issue, this work modifies the population-based training procedure so that while senders are rewarded for being able to communicate with multiple receivers, each receiver is only required to understand communication from a single sender.  This prevents receivers from directly optimizing their behavior with respect to the average behavior of the senders, while requiring that the senders be interpretable by many distinct receivers.  The experimental results demonstrate that the under this "partitioned" training paradigm, receiver agents are able to correctly interpret communication from senders that they were never paired with during training (which is not observed without partitioning).  Perhaps more importantly, the languages learned by the senders demonstrate significantly more structure (as measured by the topological similarity between the language and the underlying object space), than those learned without this partitioning mechanism.
strength_and_weaknesses: The main contribution of this work is that it provides a convincing explanation for the somewhat counterintuitive phenomenon that population-based training does not more structured and less arbitrary synthetic languages in cooperative communication tasks.  They also present a straightforward solution to this issue through a simple modification of the training process, and demonstrate that this recovers the behavior that was originally expected from population-based training.  There are potential applications of these ideas to other topics such as zero-shot coordination in RL or human-AI interaction more generally.

The main weakness of the work is that the value of learning more "structured" languages in this setting is not fully articulated.  Practical use cases for this training setup, even if only hypothetical, would be helpful in understanding how we should evaluate emergent languages relative to one another.  One example might be to use an emergent language to evaluate to train an agent designed to adapt to existing languages (eg. a particular human language) in an online fashion.  Another might be to design visual communication protocols for user interfaces that are structure in a way that makes it easy for humans to interpret them.  The lack of such motivation means that we cannot determine how useful metrics such as topological similarity really are as proxies for performance in realistic applications.
clarity,_quality,_novelty_and_reproducibility: Overall the work is technically correct, and well presented.  They indicate that source code for their experiments will be available upon publication.  While the work builds heavily on existing methods and results, it presents novel and useful insights about the problem.

A few suggestions for potential improvements:
- the training objectives form general-sum game between the senders and receivers, and so some discussion of whether behaviors reliably converge, or whether cyclic behavior was observed, would be useful.
- Examples of the languages actually learned by the agents would help the reader understand the practical significance of the topological similarity measure.
- a more detailed analysis of whether there is a trade-off between the topological similarity and the accuracy of the learned languages in the ImageNet task would be helpful in understanding whether the partitioned approach to training might be detrimental in some cases.
- A minor point, but using pi to denote both a sender's strategy, and the posterior over objects induced by that strategy is somewhat confusing when reading Section 3, so perhaps a different notation should be considered.
summary_of_the_review: Overall the work provides useful, and somewhat surprising, insights that have not been previously published.  It also demonstrates that a simple modification to population-based training can substantially change the nature of the solutions learned in cooperative communication tasks, which raises the question of whether similar changes might be useful for population based training in other multi-agent settings.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper4630/Reviewer_uuc8
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper revisits the population-based training method in emergent multi-agent communication. The authors analyze the standard population-based training protocol and point out that the overall objective is the same as that of a single-pair training protocol. The co-adaptation of receivers to senders in the standard method is believed to limit the effect of population. To better make use of population, the paper proposes to partition agents into pairs and one receiver is only trained with its associated sender. The authors claim that such partitioning enhances zero-shot communication with new partners as well as the compositionality of the emergent language. Empirical results in two referential games verify the statements and the effectiveness of the proposed method.
strength_and_weaknesses: Strength：
- This paper is well-written and the figures are demonstrative, making it easy to follow. 
- The proposed method is quite simple yet effective. I appreciate the intuition of reducing co-adaptation to enhance the compositionality of the emergent language. 
- The main results and the ablation study clearly show the effectiveness of the method and the influence of all the components and factors.

Weakness:
- Two communication graphs are adopted in the experiments, as shown in figure 2. However, the fully-connected graph and the circular graph are the two extreme cases. Maybe it is necessary to add a randomly sampled graph as a reference.
- The experiments are all conducted on the referential games, where the receiver only needs to pick a candidate from a set as the prediction of the original object. What if we extend the game into a sequential decision problem? For example, the sender may send messages as the instructions, and the receiver needs to achieve some goals (such as navigation to the goals) by following such instructions. I wonder whether the proposed method can be applied to these games.
clarity,_quality,_novelty_and_reproducibility: Clarity and Quality: The overall paper is well-written and easy to follow. The figures are clear.
Novelty: The main idea is original but the potential impact maybe is limited.
Reproducibility: The implementation details are available.
summary_of_the_review: The paper is interesting and solid. But the experiments are not convincing enough, requiring comparing with different patterns of graphs, e.g., random graphs. Besides, demonstrating the feasibility of extending the method to a sequential decision-making setting will help the method be more general and impactful.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4630/Reviewer_2JSn
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The contribution of the paper is twofold: The paper explores if the “standard training protocol” in multi agent communication exhibits features observed in natural language, namely, if agents are able to communicate with new partners within the same population, and if lager populations tend to develop more structured languages. Authors show that, it is not the case. Moreover, they show that objective function of the standard training approach for multi agents is the same as that of a single pair of agents due to receivers co-adapting to their training partners. Second, authors propose an alternative training approach which partitions sender-receiver pairs which limits co-adaptation of receiver agents. They show that this alternative approach maximizes an objective function that involves two parts:
one function to maximize the “internal communication” between pair of agents and another function to maximize the “mutual intelligibility” across agents. They empirically show that this partitioning approach increases the effect of population size on compositionality and allows emergence of a more structured language, thus allowing to exhibit those features in natural language.



strength_and_weaknesses: 
The authors study an interesting and motivating problem. Analysis is done to explore the objective functions in both cases: standard and  partitioned. They have performed number of experiments to analyze different hypothesis and to showcase the claims and the results are compelling. Provides how the work is related to other work.

I like the organization of the paper (sections containing individual task.) However, the structure can be improved. Here are some suggestions.

In the introduction, adding what you do in each section. This will help the readers to better understand the flow of the paper and your results.
Adding a discussion explaining your results and what the broader connections are. You have explained that "standard procedure" do not showcase the features of the reality. You could consider elaborating on that and explain more on how partitioning allows those features (of course you have explained them- you could consider elaborating and being more precise ), how scaling up to populations can be beneficial and how the way it is done matters. What are the implications and broader connections to natural language? Consider explaining how "simple design choices can have a great impact on the training dynamics and ultimately the effect of population on the emergent language"

Conclusion looks a bit vague. " exhibit some of the properties","Motivated by an analysis of populations at the equilibrium"-be a bit explicit.

Add future work.

All the points made above are suggestions, and I would defer to the authors who best know their work and are in the best position to decide how to make their work more accessible.




clarity,_quality,_novelty_and_reproducibility: The paper could be more clear. 
summary_of_the_review: I believe the authors have obtained some interesting results to explore and answer an interesting problem. However the flow and the explanations could be improved. In my opinion, how the results can be connected at large could be discussed more.
I have made some suggestions to improve the understandability of the paper. After revision paper can be accepted for publication.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold