We derive a new intrinsic social motivation for multi-agent reinforcement learning (MARL), in which agents are rewarded for having causal influence over another agent's actions. Causal influence is assessed using counterfactual reasoning. The reward does not depend on observing another agent's reward function, and is thus a more realistic approach to MARL than taken in previous work. We show that the causal influence reward is related to maximizing the mutual information between agents' actions. We test the approach in challenging social dilemma environments, where it consistently leads to enhanced cooperation between agents and higher collective reward. Moreover, we find that rewarding influence can lead agents to develop emergent communication protocols. We therefore employ influence to train agents to use an explicit communication channel, and find that it leads to more effective communication and higher collective reward. Finally, we show that influence can be computed by equipping each agent with an internal model that predicts the actions of other agents. This allows the social influence reward to be computed without the use of a centralised controller, and as such represents a significantly more general and scalable inductive bias for MARL with independent agents. arXiv:1810.08647v2 [cs. LG] 14 Dec 2018 reward structure, similar to Prisoner's Dilemma. For each individual agent, 'defecting' (non-cooperative behavior) has the highest payoff. However, the collective reward will be better if all agents choose to cooperate. The paradoxical payoff structure of these tasks make achieving cooperative social dynamics extremely challenging for typical RL agents. We show that social influence allows agents to learn to cooperate in these environments, and make the following contributions: • We demonstrate that deep RL agents trained with the proposed social influence reward cooperate to attain higher collective reward than baseline deep RL agents (Mnih et al., 2016). In some cases, this cooperation is attained because influencer agents learn to use their actions as an emergent communication protocol, analogous to behavior seen in animals (von Frisch, 1969). • Motivated by the previous point, we apply the influence reward to training deep RL agents to use an explicit communication channel, as in Foerster et al. (2016). We demonstrate that the communication protocols trained with the influence reward meaningfully relate to agents' actions, and that once again, agents trained with the influence reward achieve better collective outcomes. • We demonstrate that there is a significant correlation between being influenced through communication messages and obtaining higher individual return, suggesting that influential communication is beneficial to the agents that receive it. • Finally, rather than computing social influence using a centralised training framework as in prior work (e.g. Foerster et al. (2017; 2016) ), we extend the approach by attaching an internal Model of Other Agents (MOA) network to each agent and training it to predict the actions of every other agent. The agent can then simulate counterfactual actions and use its own internal MOA to predict how these will affect other agents, thus computing its own intrinsic influence reward. Castañeda, Iain Dunning, Tina Zhu, Kevin R McKee, Raphael Koster, et al. Inequity aversion improves cooperation in intertemporal social dilemmas. In Advances in neural information processing systems (NIPS), Montreal, Canada, 2018. Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. Empowerment: A universal agent-centric measure of control. In Evolutionary Computation, 2005. The 2005 IEEE Congress on, volume 1, pp. 128-135. IEEE, 2005. Kevin N. Laland. Darwin's unfinished symphony : how culture made the human mind / Kevin N. Laland. of linguistic communication from referential games with symbolic and pixel input. arXiv preprint arXiv:1804.03984, 2018. Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning in sequential social dilemmas. In
Jacques et al. REF use the mutual information between the agent's policy and a goal-independent policy to shape the agent's behaviour towards hiding or spelling out its current goal.
Intrinsic Social Motivation via Causal Influence in Multi-Agent RL
ArXiv
ArXiv
Computer Science Mathematics