Paper:
Multi-agent reinforcement learning systems aim to provide interacting agents with the ability to collaboratively learn and adapt to the behaviour of other agents. In many real-world applications, the agents can only acquire a partial view of the world. Here we consider a setting whereby most agents' observations are also extremely noisy, hence only weakly correlated to the true state of the environment. Under these circumstances, learning an optimal policy becomes particularly challenging, even in the unrealistic case that an agent's policy can be made conditional upon all other agents' observations. To overcome these difficulties, we propose a multi-agent deep deterministic policy gradient algorithm enhanced by a communication medium (MADDPG-M), which implements a two-level, concurrent learning mechanism. An agent's policy depends on its own private observations as well as those explicitly shared by others through a communication medium. At any given point in time, an agent must decide whether its private observations are sufficiently informative to be shared with others. However, our environments provide no explicit feedback informing an agent whether a communication action is beneficial, rather the communication policies must also be learned through experience concurrently to the main policies. Our experimental results demonstrate that the algorithm performs well in six highly non-stationary environments of progressively higher complexity, and offers substantial performance gains compared to the baselines.

Review:
This paper is clearly written and explains everything in a good detail. I have a few questions about the design of the algorithm and experiments that I will explain next. Most importantly, I am confused why the communication actions are modeled with continuous actions. Also, the communicating agent idea is incorporated in MADDPG paper, and the contribution of the proposed network is unclear to me.  Right now, I am leaning toward weak reject now but I might update my evaluations after seeing the authors' feedback.
1) First, your construction of communication medium simply seems to be learning a method for graph sparsification and this deserves some explanation.  Also, I think that using the graph terminology for describing the communication medium structure will significantly improve the clarity of the paper. For example, I assume that by saying that m^t = ...=m^{t+C-1} you mean that you simply fix the communication graph structure for C steps, not the communicating observations. Based on your notations, it is a little bit confusing -- in your notations $m^t$ is the set of observation that flows through the graph which should be different than $m^{t+1}$.
2) Even MADDPG is very challenging to train! Now, this paper utilizes two MADDPG, and that is something that concerns me a lot. I don't think that replicating the results of this paper is possible by other people. How much was the cost of the hyper-parameters search? 
3) Why the decision of where to send the observations is modeled with a continuous control action? It can be simply modeled with discrete action in a more efficient way, right? What I mean is that $c_i$ can be a binary which tells whether send an observation or not. Am I missing anything?
4) In section 2, you argue that in the original MADDPG paper, there is no inter-agent communication. As far as I remember, they have some experiments for cooperative communication or covert communication in which the communication is allowed between the agents. I would like to know more about this statement; maybe I am missing something. Why you are not designing the communication network which is a differentiable medium such as Foerster 2015? Isn't that efficient?
5) In alternating case, I don't see (intuitively) why the communication should help to improve upon MADDPG. My intuition is that each agent will be the gifted one 1/3 on average. This means that the agents cannot perceive who gives the correct information and the policy should converge to a point where the communication does not give any new information.
6) I would like to see what will happen with C=1? I think this hyper-parameter deserves some analysis to see how it affects the performance of the proposed method.
7) In section 5, you say that in original DDPG, there is no real need for inter-agent communication". This is a little bit strict statement, I guess. For example in the case with full observability, the agents can send messages which conveys their intention and help each other.
Minor:
* I would suggest using partially observability terminology instead of saying noisy observation because I think it includes a more general class of the problems to solve.
* "that a coupled through a communication medium" -> "that are coupled through a communication medium"
* In section 4.2, it is unclear to me what is the exploration strategy. Please explain more.
* section 5.2: Using the term lower bound is not accurate. Try changing it to something else or use with quotations: "lower bound"
* What will happen you choose the top-k rule for sending the information? For example, does top-2 (two-to-one) rule improve the results? The experiments might be added in future.
-----------------------------------
post rebuttal: the authors have responded to my main questions, and I would like to increase my score, but I cannot agree with them on possile future extensions of this work, e.g. in learning from pixels.