Paper:
Recent work has studied the emergence of language among deep reinforcement learning agents that must collaborate to solve a task. Of particular interest are the factors that cause language to be compositional-i.e. express meaning by combining words which themselves have meaning. Evolutionary linguists have found that in addition to structural priors like those already studied in deep learning, the dynamics of transmitting language from generation to generation contribute significantly to the emergence of compositionality. In this paper, we introduce these cultural evolutionary dynamics into language emergence by periodically replacing agents in a population to create a knowledge gap, implicitly inducing cultural transmission of language. We show that this implicit cultural transmission encourages the resulting languages to exhibit better compositional generalization.

Review:
This paper studies whether language composition may emerge by partially re-sampling new agents inside a pool of language agents. They set up a consistent experimental setting for assessing compositionality,  assess different agent architectures, e.g., memory vs. memoryless agents,  and explore how the language remains close to each other by re-sampling new agents.
The paper is well-motivated with substantial background literature on the cognitive science and emergent communication side. The claim is clear, the hypotheses are well-stated, and the experiments look solid (I particularly appreciated the paragraph on shortcoming evaluation). In the end, I enjoy reading the paper despite its density, and I could see that the authors made quite some effort in that direction.
Improvement direction, questions:
 - The authors made were careful not to take ownership of Kottur et al. 's works. Yet, the writing sometimes gives the feeling that their work is solely an extension of Kottur's work, which is not the case. It also gives the feeling that Kottur et al. is the only valid experimental setting, which is not the case (at the authors pointed out in the related work section). Thus, I would recommend to summarise at some point the similarity/difference between the two papers, or at least stop referring the paper every two lines! 
 - Replacement strategy: the authors use simple replacement strategies, and conclude that it has little impact. Althought It sounds reasonable in the current setting, the conclusion may be a bit premature. I would recommend to discuss further this result with complementary experiments could be the following: see the impact of epsilon, trying tournament strategies, why 8 populations (this sounds a bit arbitrary too). I would also like to put those observations in perspective with the evolutionary literature [1], and even provide a full paragraph in the related work section.
 - Population dynamics: I am missing a key element in the paper: an analysis of the population dynamics. Although the paper deals with generational transmissions, there are no experiments that analyze the evolution of language generations after generations. Most of the experiments deal with the final convergence state. Again, I would recommend having a look at the evolutionary literature to see which protocol they use to analyze such behavior.
 - Literature side: The authors did an excellent job on the emergent communication and cognitive science side. I think that it is worth extending the comparison further. For instance: 
    * generational transmission can be studied in the light of game theory [2] where compositionality can be seen as a Nash Equilibrium between agent. 
    * generational transmission is a form of dynamic distillation [3]
    * and evolutionary algorithms!
 - I had some difficulties in understanding Fig4, and the final-take away correctly. Would it be possible to give me one or two examples to correctly parse the table? More generally, I would recommend to add a few lines with some concrete and cherry-picked examples from the experiments to help the reader to have more intuition). 
 - In a similar spirit, it is hard to interpret the distance in Figure 3. What would correspond to an increase of 1pt of distance? Having said that, the experiment is sound, and it is insightful.
 - reproducible: having a final table in the array in the appendix could be very helpful 
 - crazy experiment: even if I am also a DRL addict, I would be curious to train one of the models with evolutionary algorithms (CMA-ES over parameters, for instance) to assess whether RL has an impact on compositionality (or it is solely the experimental protocol that matters). 
 - I may have missed this point, but how many seeds did you use to run your experiments? 
 - I may have also missed this point, what is the average length of the dialogue. Can you upload (non-understandable) dialogue example? 
Last point... but it does not undermine the soundness of the experimental protocol! 
 - In the end, Is 25 accuracy points really compositionality? What would be the score of simple strategies with overcomplete tokens? What is the score of the minimal vocab if we are only correct with one modality, two modalities?
Remarks:
 - in the introduction, you mention that previous old agents have grounded language, I am not sure whether we can speak of grounded language here, they have a predefined language, but it is not grounded. 
 - Please remove the bold sentence in the introduction :) The claim is clear!
 - P11: Alg undefined
 - P12: the legend cannot be read
 
Conclusion
I am familiar with this type of experimental protocols, and I am well aware that they are never-ending works. There are always more experiments to do, more parameters to analyze. The final question is the following: is this paper have enough of these never-ending experiments? I think that this paper is just above this threshold by a short margin, and I vouch for weak accept.
However, I am missing at least one dynamic figure (to see the impact of the population along time, which is one of the core concepts of the paper), and there are several links with other ML communities that still have to be highlighted (especially evolutionary algorithms). 
Besides, I somehow feel that the authors pursue two different goals in this paper: they both analyze memory/memoryless complete/overcomplete agents, which is somehow orthogonal to the general transmission hypothesis. Maybe, It would have made more sense to focus on one (or two) of the models and change the experimental setting on them (population size, training time, etc.) 
In the end, I would favor a weak accept. 
I am open to discussion regarding this scoring.
[1] Bäck, Thomas, and Frank Hoffmeister. "Extended selection mechanisms in genetic algorithms." (1991).
[2] Lanctot, Marc, et al. "A unified game-theoretic approach to multiagent reinforcement learning." Advances in Neural Information Processing Systems. 2017.
[3] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 (2015).