Paper:
Multi-agent collaboration is required by numerous real-world problems. Although distributed setting is usually adopted by practical systems, local range communication and information aggregation still matter in fulfilling complex tasks. For multi-agent reinforcement learning, many previous studies have been dedicated to design an effective communication architecture. However, existing models usually suffer from an ossified communication structure, e.g., most of them predefine a particular communication mode by specifying a fixed time frequency and spatial scope for agents to communicate regardless of necessity. Such design is incapable of dealing with multi-agent scenarios that are capricious and complicated, especially when only partial information is available. Motivated by this, we argue that the solution is to build a spontaneous and self-organizing communication (SSoC) learning scheme. By treating the communication behaviour as an explicit action, SSoC learns to organize communication in an effective and efficient way. Particularly, it enables each agent to spontaneously decide when and who to send messages based on its observed states. In this way, a dynamic inter-agent communication channel is established in an online and self-organizing manner. The agents also learn how to adaptively aggregate the received messages and its own hidden states to execute actions. Various experiments have been conducted to demonstrate that SSoC really learns intelligent message passing among agents located far apart. With such agile communications, we observe that effective collaboration tactics emerge which have not been mastered by the compared baselines.

Review:
Summary: 
This paper expands on the work on 'emergent communication' with 2 innovations: 
- The architecture has a separate 'message channel' that processes the incoming and outgoing messages mostly independently of the hidden state of the agent. There are also dedicated architecture elements for the interaction between the hidden state and the message stream. 
- The outgoing message is gated with a 'speak' action: only when the agent takes the speak-action at time step t is a message sent out at timestep t+1. 
Comments for improvement: 
-The paper proposes a rather complicated architecture, with many moving part. In the paper's current form it is extremely hard to see which part of this architecture contribute to the success of the method. A set of ablation studies on the different components would indeed be very helpful. 
-Using the word 'thought' to describe the hidden state of the agent is rather distracting.
-Equation (1): This just seems to be the policy gradient term for a factorised action space across 'environment action' and 'communication action'. The only obvious difference is that the policy here is shown to condition on the state representation s_t, rather than on the input. Is that intended?
-The paper suffers from a lot of undefined notation, e.g. the s_t above. Please clarify.
-In Figure 2b) the MCU is shown to produce the action a_t as an output. That seems like a mistake. 
-Figure 4): The results seem to be extremely unstable, which is a well known issue for independent learning. Recent work (MADDPG, COMA) has shown that centralised critics can drastically avoid these instabilities and improve final performance. Did you compare against using a centralised critic, V(central state), rather the V(observation)? Also, using a single seed on this kind of unstable learning process renders the results highly non-conclusive. 
-In Figure (5), what are the red-arrows? Do these correspond to the actual actions taken by the agents or are they simply annotations? It would be good to see how far the communication range is by comparison. Also, why is there a blob of 'communicating' agents far from the enemy? 
-Are different methods in the large scale battle task trained in self-play and then pitched against other methods in a round-robin tournament after training has finished or are they trained against each other? 
-In Figure 6 (a), why are average rewards changing over the course of training? I would expect this to be a zero-sum setting in self-play. 
-I couldn't find any supplementary material referenced in the text for the details. Instead the paper seems to have another copy of the paper itself attached in the pdf. This makes it hard to evaluate the paper given that few details around training are provided in the main text. 
Overall I am concerned that the learning method used in the paper (independent baseline) is known to be unstable and to produce poor results in the multi-agent setting (see COMA and MADDPG). This raises the concern that the communication channel is mostly useful for overcoming the issues introduced from having a decentralised critic.