Many tasks in AI require the collaboration of multiple agents. Typically, the communication protocol between agents is manually specified and not altered during training. In this paper we explore a simple neural model, called CommNN, that uses continuous communication for fully cooperative tasks. The model consists of multiple agents and the communication between them is learned alongside their policy. We apply this model to a diverse set of tasks, demonstrating the ability of the agents to learn to communicate amongst themselves, yielding improved performance over non-communicative agents and baselines. In some cases, it is possible to interpret the language devised by the agents, revealing simple but effective strategies for solving the task at hand. We consider the setting where we have J agents, all cooperating to maximize reward R in some environment. We make the simplifying assumption of full cooperation that each agent receives R, independent of their contribution. In this setting, there is no difference between each agent having its own controller, or viewing them as pieces of a larger model controlling all agents. Taking the latter perspective, our controller is a large feed-forward neural network that maps inputs for all agents to their actions, each agent occupying a subset of units. A specific connectivity structure between layers arXiv:1605.07736v1 [cs.LG] 25 May 2016 (a) instantiates the broadcast communication channel between agents and (b) propagates the agent state. Because the agents will receive reward, but not necessarily supervision for each action, reinforcement learning is used to maximize expected future reward. We explore two forms of communication within the controller: (i) discrete and (ii) continuous. In the former case, communication is an action, and can naturally be handled by reinforcement learning. In the continuous case, the signals passed between agents are no different than hidden states in a neural network; thus credit assignment for the communication can be performed using standard backpropagation (within the outer RL loop). We use policy gradient [36] with a state specific baseline for delivering a gradient to the model. Denote the states in an episode by s(1), ..., s(T ), and the actions taken at each of those states as a(1), ..., a(T ), where T is the length of the episode. The baseline is a scalar function of the states b(s, θ), computed via an extra head on the model producing the action probabilities. Beside maximizing the expected reward with policy gradient, the models are also trained to minimize the distance between the baseline value and actual reward. Thus after finishing an episode, we update the model parameters θ by (1) Here r(t) is reward given at time t, and the hyperparameter α is for balancing the reward and the baseline objectives, which set to 0. Abstract abstract troduction work we make two contributions. First, we simplify and extend the graph neural network ture of ??. Second, we show how this architecture can be used to control groups of cooperating odel plest form of the model consists of multilayer neural networks f i that take as input vectors c i and output a vector h i+1 . The model takes as input a set of vectors {h 0 1 , h 0 2 , ..., h 0 Abstract ct ction e make two contributions. First, we simplify and extend the graph neural network ??. Second, we show how this architecture can be used to control groups of cooperating 42 learning is used to maximize expected future reward. We explore two forms of communication within 43 the controller: (i) discrete and (ii) continuous. In the former case, communication is an action, and 44 will be treated as such by the reinforcement learning. In the continuous case, the signals passed 45 between agents are no different than hidden states in a neural network; thus credit assignment for the 46 communication can be performed using standard backpropagation (within the outer RL loop). We use policy gradient [33] with a state specific baseline for delivering a gradient to the model. Denote the states in an episode by s(1), ..., s(T ), and the actions taken at each of those states 49 as a(1), ..., a(T ), where T is the length of the episode. The baseline is a scalar function of the 50 states b(s, ✓), computed via an extra head on the model producing the action probabilities. Beside 51 maximizing the expected reward with policy gradient, the models are also trained to minimize the 52 distance between the baseline value and actual reward. Thus, after finishing an episode, we update 53 the model parameters ✓ by Here r(t) is reward given at time t, and the hyperparameter ↵ is for balancing the reward and the 55 baseline objectives, set to 0.03 in all experiments. 3 Model We now describe the model used to compute p(a(t)|s(t), ✓) at a given time t (ommiting the time 58 index for brevity). Let s j be the jth agent's view of the state of the environment. The input to the 59 controller is the concatenation of all state-views s = {s 1 , ..., s J }, and the controller is a mapping 60 a = (s), where the output a is a concatenation of discrete actions a = {a 1 , ..., a J } for each agent. Note that this single controller encompasses the individual controllers for each agents, as well as 62 the communication between agents. One obvious choice for is a fully-connected multi-layer neural network, which could extract 64 features h from s and use them to predict good actions with our RL framework. This model would 65
CommNet REF learns continuous multiagent communication for cooperative tasks in an end-to-end supervised and reinforcement learning setting.
Learning Multiagent Communication with Backpropagation
ArXiv
ArXiv
Computer Science Mathematics