Paper:
A variety of cooperative multi-agent control problems require agents to achieve individual goals while contributing to collective success. This multi-goal multiagent setting poses difficulties for recent algorithms, which primarily target settings with a single global reward, due to two new challenges: efficient exploration for learning both individual goal attainment and cooperation for others' success, and credit-assignment for interactions between actions and goals of different agents. To address both challenges, we restructure the problem into a novel two-stage curriculum, in which single-agent goal attainment is learned prior to learning multi-agent cooperation, and we derive a new multi-goal multi-agent policy gradient with a credit function for localized credit assignment. We use a function augmentation scheme to bridge value and policy functions across the curriculum. The complete architecture, called CM3, learns significantly faster than direct adaptations of existing algorithms on three challenging multi-goal multi-agent problems: cooperative navigation in difficult formations, negotiating multi-vehicle lane changes in the SUMO traffic simulator, and strategic cooperation in a Checkers environment.

Review:
The paper presents a new method CM3 for multi-goal multi-agent settings where agent must care about the rewards of others as well as theirs. The method restructures the problem into two-stage curriculum where individual agents solve the problem in the individual setting of the environment in the first stage and then more agents are introduced in the multi-agent setting in second stage. The method also uses function augmentation to only learn parameters which are necessary for single agent in first stage and then more parameters are introduced in the second stage. The results are shown on three environments which show that CM3 outperforms the baselines.
The paper is well-written, motivated and clear to read. A lot of important stuff has been pushed into appendix and experiments/setup require more details but overall I believe paper has significant contributions and results. Therefore, I assign a rating of weak accept which I am happy to raise if clarity of paper can be improved.
The paper doesn’t comment on parameter counts of CM3 compared with baselines which is also an important factor in choosing one method over the other. I would like to see more quantitative analysis as presented in IAC to understand what is happening behind the scenes. It is very surprising to see that IAC is outperforming COMA in most of the tasks which is opposite of what COMA paper suggested.
On the note of curriculum [1] and [2] uses curriculum in traffic junction settings to improve overall performance as agent increases. This can be compared to moving from stage 1 to stage 2 but instead we move from less number of agents to more. [2] also suggests that using individualized rewards help in better credit assignment. There is also an assumption in the setup that private observations from all other agents are readily available. It would be interesting to see what would happen if agents have to communicate their private state. So, the paper is also missing discussion on communication protocols (discrete, continuous, through critic) [1][2]. 
It is hard to directly compare through the charts. So, tables in Appendix E and F should be moved to the main text. The experiment section needs to be extended and the main model section needs to be decreased in the content amount.
It seems like you missed 1/n in advantage function’s equation in Section 3, Multi-agent credit assignment section. Figure 6 has been mentioned in 6 second paragraph but doesn’t appear until the last page.
[1] Sukhbaatar, Sainbayar, and Rob Fergus. "Learning multiagent communication with backpropagation." In Advances in Neural Information Processing Systems, pp. 2244-2252. 2016.
[2] Singh, Amanpreet, Tushar Jain, and Sainbayar Sukhbaatar. "Learning when to communicate at scale in multiagent cooperative and competitive tasks." arXiv preprint arXiv:1812.09755 (2018).