Paper:
We derive a new intrinsic social motivation for multi-agent reinforcement learning (MARL), in which agents are rewarded for having causal influence over another agent's actions. Causal influence is assessed using counterfactual reasoning. The reward does not depend on observing another agent's reward function, and is thus a more realistic approach to MARL than taken in previous work. We show that the causal influence reward is related to maximizing the mutual information between agents' actions. We test the approach in challenging social dilemma environments, where it consistently leads to enhanced cooperation between agents and higher collective reward. Moreover, we find that rewarding influence can lead agents to develop emergent communication protocols. We therefore employ influence to train agents to use an explicit communication channel, and find that it leads to more effective communication and higher collective reward. Finally, we show that influence can be computed by equipping each agent with an internal model that predicts the actions of other agents. This allows the social influence reward to be computed without the use of a centralised controller, and as such represents a significantly more general and scalable inductive bias for MARL with independent agents.

Review:
INTRINSIC SOCIAL MOTIVATION VIA CAUSAL INFLUENCE IN MULTI-AGENT RL
Main Idea: The authors consider adding a reward term to standard MARL which is the mutual information between its actions and the actions of others. They show that adding this intrinsic social motivation can lead to increased cooperation in several social dilemmas. 
Strong Points:
-	This paper is a novel extension of ideas from single agent RL to multi agent RL, there are clear benefits from doing reward shaping in the right way to make deep RL work better.
-	The paper focuses on cooperative environments which is an underfocused area in RL right now
 
Weak Points:
-	There is missing discussion of a lot of literature. The causal influence term can be thought of as a form of reward shaping. There is little discussion on the (large) literature on reward shaping to get MARL to exhibit good behavior. 
-	The results feel quite thin. Related to the point above: the theory of different types of reward shaping (e.g. optimistic Q-learning, prosociality, etc…) are well understood. It is not clear to me under what conditions the authors’ proposed augmentation to the reward function will lead to better or worse outcomes. The experiments in this paper are quite simple and only span a small set of environments so it would be good to have at least some formal theory.
-	Social dilemmas don’t seem like the best application. The authors define the social dilemma as: “For each individual agent, ‘defecting’ i.e. non-cooperative behavior has the highest payoff.” With the intrinsic motivation, agents learn to cooperate. This is good, however, if we’re thinking about situations where agents aren’t trained together and have their own rewards (the authors’ example: “autonomous vehicles are likely to be produced by a wide variety of organizations and institutions with mixed motivations”) then won’t these agents be exploited by rational agents? Other solutions to this problem (e.g. recent papers on tit-for-tat by Lerer & Peysakhovich or LOLA by Foerster et al. construct agents where defectors get explicitly punished and so don’t want to try exploiting). Is there something I am missing here? Do the agents learn to punish non-cooperators (if no, isn’t it rational at that point to just not cooperate and won’t self-driving cars trained via this method get exploited by others)? 
-	Relate to the point(s) above: a better environment for application here seems to be coordination games/”Stag Hunt” games where it is known that MARL converges to poor equilibria and many other methods e.g. optimistic Q-learning or prosociality have been invented to make things work better. Perhaps the method proposed here will work better than these (and it has the appealing property that it does not require the ability to observe the other agents' rewards as e.g. prosociality does)
-	This paper contains some quite grandiose language connecting the proposed reward shaping to “how humans learn” (example: It may also have correlates in human cognition; experiments show that newborn infants are sensitive to correspondences between their own actions and the actions of other people, and use this to coordinate their behavior with others) it’s unclear to me that humans experience extra reward for their actions having high mutual information (and/or causal information with others). While it’s fine to argue some of these points at a high level I would suggest scrubbing the text of the gratuitous references to this.
Nits:
“Crawford & Sobel (1982) find that once agents’ interests diverge by a finite amount, no communication is to be expected.” – this is an awkward phrasing of the Crawford and Sobel result (it can be read as “if interests diverge by any epsilon there can be no communication”). The CS result is that information revealed in communication (in equilibrium) is proportional to amount of common interest.