Paper:
Published as a conference paper at ICLR 2018 EMERGENT COMMUNICATION THROUGH NEGOTIATION
Multi-agent reinforcement learning offers a way to study how communication could emerge in communities of agents needing to solve specific problems. In this paper, we study the emergence of communication in the negotiation environment, a semi-cooperative model of agent interaction. We introduce two communication protocols -one grounded in the semantics of the game, and one which is a priori ungrounded and is a form of cheap talk. We show that self-interested agents can use the pre-grounded communication channel to negotiate fairly, but are unable to effectively use the ungrounded channel. However, prosocial agents do learn to use cheap talk to find an optimal negotiating strategy, suggesting that cooperation is necessary for language to emerge. We also study communication behaviour in a setting where one agent interacts with agents in a community with different levels of prosociality and show how agent identifiability can aid negotiation.

Review:
The authors describe a variant of the negotiation game in which agents of different type, selfish or prosocial, and with different preferences. The central feature is the consideration of a secondary communication (linguistic) channel for the purpose of cheap talk, i.e. talk whose semantics are not laid out a priori. 
The essential findings include that prosociality is a prerequisite for effective communication (i.e. formation of meaningful communication on the linguistic channel), and furthermore, that the secondary channel helps improve the negotiation outcomes.
The paper is well-structured and incrementally introduces the added features and includes staged evaluations for the individual additions, starting with the differentiation of agent characteristics, explored with combination of linguistic and proposal channel. Finally, agent societies are represented by injecting individuals' ID into the input representation.
The positive:
- The authors attack the challenging task of given agents a means to develop communication patterns without apriori knowledge.
- The paper presents the problem in a well-structured manner and sufficient clarity to retrace the essential contribution (minor points for improvement).
- The quality of the text is very high and error-free.
- The background and results are well-contextualised with relevant related work. 
The problematic:
- By the very nature of the employed learning mechanisms, the provided solution provides little insight into what the emerging communication is really about. In my view, the lack of interpretable semantics hardly warrants a reference to 'cheap talk'. As such the expectations set by the well-developed introduction and background sections are moderated over the course of the paper.
- The goal of providing agents with richer communicative ability without providing prior grounding is challenging, since agents need to learn about communication partners at runtime. But it appears as of the main contribution of the paper can be reduced to the decomposition of the learnable feature space into two communication channels. The implicit relationship of linguistic channel on proposal channel input based on the time information (Page 4, top) provides agents with extended inputs, thus enabling a more nuanced learning based on the relationship of proposal and linguistic channel. As such the well-defined semantics of the proposal channel effectively act as the grounding for the linguistic channel. This, then, could have been equally achieved by providing agents with a richer input structure mediated by a single channel. From this perspective, the solution offers limited surprises. The improvement of accuracy in the context of agent societies based on provided ID follows the same pattern of extending the input features.
- One of the motivating factors of using cheap talk is the exploitation of lying on the part of the agents. However, apart from this initial statement, this feature is not explicitly picked up. In combination with the previous point, the necessity/value of the additional communication channel is unclear.
Concrete suggestions for improvement:
- Providing exemplified communication traces would help the reader appreciate the complexity of the problem addressed by the paper.
- Figure 3 is really hard to read/interpret. The same applies to Figure 4 (although less critical in this case).
- Input parameters could have been made explicit in order to facilitate a more comprehensive understanding of technicalities (e.g. in appendix).
- Emergent communication is effectively unidirectional, with one agent as listener. Have you observed other outcomes in your evaluation?
In summary, the paper presents an interesting approach to combine unsupervised learning with multiple communication channels to improve learning of preferences in a well-established negotiation game. The problem is addressed systematically and well-presented, but can leave the reader with the impression that the secondary channel, apart from decomposing the model, does not provide conceptual benefit over introducing a richer feature space that can be exploited by the learning mechanisms. Combined with the lack of specific cheap talk features, the use of actual cheap talk is rather abstract. Those aspects warrant justification.