C ORRECTING EXPERIENCE REPLAY FOR MULTI - AGENT COMMUNICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider the problem of learning to communicate using multi-agent reinforcement learning (MARL).
A common approach is to learn off-policy, using data sampled from a replay buffer.
However, messages received in the past may not accurately reﬂect the current communication policy of each agent, and this complicates learning.
We therefore introduce a ‘communication correction’ which accounts for the non-stationarity of observed communication induced by multi-agent learning.
It works by relabelling the received message to make it likely under the communicator’s current policy, and thus be a better reﬂection of the receiver’s current environment.
To account for cases in which agents are both senders and receivers, we introduce an ordered relabelling scheme.
Our correction is computationally efﬁcient and can be integrated with a range of off-policy algorithms.
We ﬁnd in our experiments that it substantially improves the ability of communicating
MARL systems to learn across a variety of cooperative and competitive tasks.
I NTRODUCTION
Since the introduction of deep Q-learning (Mnih et al., 2013), it has become very common to use previous online experience, for instance stored in a replay buffer, to train agents in an ofﬂine manner.
An obvious difﬁculty with doing this is that the information concerned may be out of date, leading the agent woefully astray in cases where the environment of an agent changes over time.
One obvious strategy is to discard old experiences.
However, this is wasteful – it requires many more samples from the environment before adequate policies can be learned, and may prevent agents from leveraging past experience sufﬁciently to act in complex environments.
Here, we consider an alternative, Orwellian possibility, of using present information to correct the past, showing that it can greatly improve an agent’s ability to learn.
We explore a paradigm case involving multiple agents that must learn to communicate to optimise their own or task-related objectives.
As with deep Q-learning, modern model-free approaches often seek to learn this communication off-policy, using experience stored in a replay buffer (Foerster et al., 2016; 2017; Lowe et al., 2017; Peng et al., 2017).
However, multi-agent reinforcement learning (MARL) can be particularly challenging as the underlying game-theoretic structure is well known to lead to non-stationarity, with past experience becoming obsolete as agents come progressively to use different communication codes.
It is this that our correction addresses.
Altering previously communicated messages is particularly convenient for our purposes as it has no direct effect on the actual state of the environment (Lowe et al., 2019), but a quantiﬁable effect on the observed message, which constitutes the receiver’s ‘social environment’.
We can therefore determine what the received message would be under the communicator’s current policy, rather than what it was when the experience was ﬁrst generated.
Once this is determined, we can simply relabel the past experience to better reﬂect the agent’s current social environment, a form of off-environment correction (Ciosek & Whiteson, 2017).
We apply our ‘communication correction’ using the framework of centralised training with decentralised control (Lowe et al., 2017; Foerster et al., 2018), in which extra information – in this case the policies and observations of other agents – is used during training to learn decentralised multiagent policies.
We show how it can be combined with existing off-policy algorithms, with little computational cost, to achieve strong performance in both the cooperative and competitive cases.
B ACKGROUND
Markov Games
A partially observable Markov game (POMG) (Littman, 1994; Hu et al., 1998) for
N agents is deﬁned by a set of states S , sets of actions
A 1 , ..., A N and observations O 1 , ..., O N for each agent.
In general, the stochastic policy of agent i may depend on the set of action-observation histories H
i ≡ ( O i × A i ) ∗ such that π
i : H i × A i → [0 , 1] .
In this work we restrict ourselves to history-independent stochastic policies
π i : O i ×
A i
→
The next state is generated according to the state transition function P : S×A 1 × . . .
×A n ×S → [0 , 1] .
Each agent i obtains deterministic rewards deﬁned as r i : S ×
A 1 × ...
× A n → R and receives a deterministic private observation o i : S → O i .
There is an initial state distribution ρ 0 : S → [0 , 1] and each agent i aims to maximise its own discounted sum of future rewards E s ∼ ρ π , a ∼ π [ (cid:80) ∞ t =
γ t r
i
s, a )] where π = { π 1 , . . .
, π n } is the set of policies for all agents, a = ( a 1 , . . .
, a N ) is the joint action and ρ π is the discounted state distribution induced by these policies starting from ρ 0 .
Experience Replay As an agent continually interacts with its environment it receives experiences ( o t , a t , r t +1 , o t +1 ) at each time step.
However, rather than using those experiences immediately for learning, it is possible to store such experience in a replay buffer, D , and sample them at a later point in time for learning (Mnih et al., 2013).
This breaks the correlation between samples, reducing the variance of updates and the potential to overﬁt to recent experience.
In the single-agent case, prioritising samples from the replay buffer according to the temporal-difference error has been shown to be effective (Schaul et al., 2015).
In the multi-agent case, Foerster et al.
(2017) showed that issues of non-stationarity could be partially alleviated for independent Q-learners by importance sampling and use of a low-dimensional ‘ﬁngerprint’ such as the training iteration number.
MADDPG Our method can be combined with a variety of algorithms, but we commonly employ it with multi-agent deep deterministic policy gradients (MADDPG) (Lowe et al., 2017), which we describe here.
MADDPG is an algorithm for centralised training and decentralised control of multi-agent systems (Lowe et al., 2017; Foerster et al., 2018), in which extra information is used to train each agent’s critic in simulation, whilst keeping policies decentralised such that they can be deployed outside of simulation.
It uses deterministic policies, as in DDPG (Lillicrap et al., 2015), which condition only on each agent’s local observations and actions.
MADDPG handles the nonstationarity associated with the simultaneous adaptation of all the agents by introducing a separate centralised critic Q µ i ( o , a ) for each agent where µ corresponds to the set of deterministic policies µ i : O
→ A of all agents.
Here we have denoted the vector of joint observations for all agents as o .
The multi-agent policy gradient for policy parameters θ of agent i is:
[Equation]
where D is the experience replay buffer which contains the tuples ( o , a , r , o (cid:48) ) .
Like DDPG, each
Q µ i is approximated by a critic Q wi which is updated to minimise the error with the target.
[Equation]
where y
= r
i + γQ wi ( o (cid:48) , a (cid:48) ) is evaluated for the next state and action, as stored in the replay buffer.
We use this algorithm with some additional changes (see Appendix A.3 for details).
Communication One way to classify communication is whether it is explicit or implicit.
Implicit communication involves transmitting information by changing the shared environment (e.g. scattering breadcrumbs).
By contrast, explicit communication can be modelled as being separate from the environment, only affecting the observations of other agents.
In this work, we focus on explicit communication with the expectation that dedicated communication channels will be frequently integrated into artiﬁcial multi-agent systems such as driverless cars.
Although explicit communication does not formally alter the environmental state, it does change the observations of the receiving agents, a change to what we call its ‘social environment’ 1 .
For agents which act in the environment and communicate simultaneously, the set of actions for each agent
A
i
=
A ei × A mi is the Cartesian product of the sets of regular environment actions A ei and explicit communication actions A mi .
Similarly, the set of observations for each receiving agent
O i
= O ei × O mi is the Cartesian product of the sets of regular environmental observations
O ei and explicit communication observations O mi .
Communication may be targeted to speciﬁc agents or broadcast to all agents and may be costly or free.
The zero cost formulation is commonly used and is known as ‘cheap talk’ in the game theory community (Farrell & Rabin, 1996).
In many multi-agent simulators the explicit communication action is related to the observed communication in a simple way, for example being transmitted to the targeted agent with or without noise on the next time step.
Similarly, real world systems may transmit communication in a well understood way, such that the observed message can be accurately predicted given the sent message (particularly if error-correction is used).
By contrast, the effect of environment actions is generally difﬁcult to predict, as the shared environment state will typically exhibit more complex dependencies.
M ETHODS
Our general starting point is to consider how explicit communication actions and observed messages might be relabelled using an explicit communication model.
This model often takes a simple form, such as depending only on what was communicated on the previous timestep.
The observed messages o mt +1 given communication actions a mt are therefore sampled (denoted by ∼ ) from:
[Equation]
Examples of such a communication model could be an agent i receiving a noiseless message from a single agent j such that o mi,t +1 = a mj,t , or receiving the message corrupted by Gaussian noise o mi,t +1 ∼ N ( a mj,t , σ ) where σ is a variance parameter.
We consider the noise-free case in the multiagent simulator for all bar one of our experiments, although the general idea can be applied to more complex, noisy communication models.
A communication model such as this allows us to correct past actions and observations in a consistent way.
To understand how this is possible, we consider a sample from a multi-agent replay buffer which is used for off-policy learning.
In general, the multi-agent system at current time t (cid:48) receives observations o t (cid:48) , collectively takes actions a t (cid:48) using the decentralised policies π , receives rewards r t (cid:48) +1 (split into environmental rewards r et (cid:48)
+1 and messaging costs r mt (cid:48) +1 ) and the next observations o t (cid:48)
+1 .
These experiences are stored as a tuple in the replay buffer for later use to update the multiagent critic(s) and policies.
For communicating agents, we can describe a sample from the replay buffer D as the tuple:
[Equation]
where we separately denote environmental ( e ) and communication ( m ) terms, and t indexes a time in the past (rather than the current time t (cid:48) ).
For convenience we can ignore the environmental tuple of observations, actions and reward as we do not alter these, and consider only the communication tuple ( o mt , a mt , r mt +1 , o mt +1 ) .
Using the communication model at time t (cid:48) , we can relate a change in a mt to a change in o mt +1 .
If we also keep track of a mt − 1 we can similarly change o mt .
In our experiments we assume for simplicity that communication is costless (the ‘cheap talk’ setting), which means that r mt +1 = 0 , however in general we could also relabel rewards using a model of communication cost p ( r mt +1 | a mt ) .
Equipped with an ability to rewrite history, we next consider how to use it, to improve multi-agent learning.
3. O FF - ENVIRONMENT RELABELLING
A useful perspective for determining how to relabel samples is to consider each multi-agent experience tuple separately, from the perspective of each agent, rather than as a single tuple received by all agents as is commonly assumed.
For a given agent’s tuple, we can examine all the observed messages which constitutes its social environment (including even messages sent to other agents, which will be seen by a centralised critic).
These were generated by past policies of other agents, and since then these policies may have changed due to learning or changes in exploration.
Our ﬁrst approach is therefore to relabel the communication tuple ( o m t , a m t , r mt
+1 , o mt +1 )
i for agent i by instead querying the current policies of other agents, replacing the communication actions accordingly and using the transition model to compute the new observed messages.
For agent i this procedure is:
[Equation]
where ¬ i indicates agents other than i and we use ˆ z to indicate that z has been relabelled from its original value.
Once the message has been relabelled for all agents, we can construct the overall relabelled joint observation by concatenation:
[Equation]
We illustrate our Communication Correction (CC) idea in Figure 1 for the case of two agents, one sending out communication (the Speaker) and the other receiving communication (the Listener).
We experiment with feedforward policies which condition actions on the immediate observation, but this general idea could also be applied with recurrent networks using a history of observations ˆ a m ¬ i,t ∼ π ¬ i ( a m ¬ i,t | h ¬ i,t ) ; we discuss this in Future Work (Section 6).
In our feedforward case, we sample an extra o t − 1 in order to determine (using the other agents’ policies) the new ˆ o mt , which allows us to relabel at the point of sampling from the replay buffer.
Our relabelling approach could also straightforwardly be incorporated with attention-based models which also learn to whom to communicate (Das et al., 2019), but for our experiments we assume this is determined by the environment rather than the model.
3. O RDERED RELABELLING
One additional complexity to our approach is that the policies of the other agents may themselves be conditioned on received communication in addition to environmental observations.
Our initial description ignores this effect, applying only a single correction.
However, we can better account for this by sampling from the replay buffer an extra k samples into the past.
Starting from the k ’th sample into the past, we can set ˆ o t − k
=
o t − k .
Using Equations 5 and 6, we can then relabel according to:
[Equation]
We iteratively apply this correction until ˆ o t +1 is generated.
In general, this is an approximation if the starting joint observation o t − k depends on communication, but the corrected communication would likely be less off-environment than before.
Furthermore, for episodic environments an exact correction could be found by correcting from the ﬁrst time step of each episode.
In our experiments we consider a Directed Acyclic Graph (DAG) communication structure which also allows for exact corrections.
In general a DAG structure may be expressed in terms of an adjacency matrix D which is nilpotent; there exists some positive integer n such that D m = 0 , ∀ m ≥ n .
If s is the smallest such n , we can set k
= s − 1 which allows information to propagate from the root nodes of the DAG to the leaves.
Agents which are not root nodes will not need k updates for the inﬂuence of their messages to be propagated and so, for efﬁciency, for messages c steps into the past we only generate messages which will have a downstream effect c steps later (where 0 < c ≤ k ).
In general, we call our approach an Ordered Communication Correction (OCC), as opposed to our previous First-step Communication Correction (FCC) which only does one update.
3. I MPLEMENTATION
We include the full algorithm in Appendix A.9.
We ﬁnd in our experiments that relabelling can be done rapidly with little computational cost.
Although different agents require different relabelling, the majority of the relabelling is shared (more so proportionally for increasing N ).
For simplicity, we can therefore relabel only a single multi-agent experience for the N agents, and then correct for each agent by setting its own communication action back to its original value, as well as the downstream observations on the next time step.
Once the sampled minibatch has been altered for each agent, we then use it for training an off-policy multi-agent algorithm.
In our experiments we use MADDPG and Multi-Agent Actor-Critic (MAAC)
(see Appendix A.4)
(Iqbal & Sha, 2019)
but our method could also be applied to other algorithms, such as value-decomposition networks (Sunehag et al., 2017) and QMIX (Rashid et al., 2018).
R ESULTS
We conduct experiments in the multi-agent particle environment 2 (MPE), a world with a continuous observation and discrete action space, along with some basic simulated physics.
For common problems in the MPE, immediate observations summarise relevant history, such as velocity, such that optimal policies can be learned using feedforward networks, which we use for both policy and critic. We provide precise details on implementation and hyperparameters in Appendix A.1.
4. C OOPERATIVE C OMMUNICATION WITH 5 LANDMARKS
Our ﬁrst experiment, introduced by Lowe et al.
(2017), is known as Cooperative Communication (Figure 2).
It involves two cooperative agents, a Speaker and a Listener, placed in an environment with landmarks of differing colours.
On each episode, the Listener must navigate to a randomly selected landmark; and both agents obtain reward proportional to its negative distance from this target.
However, whilst the Listener observes its relative position from each of the differently coloured landmarks, it does not know which landmark is the target.
Instead, the colour of the target landmark can be seen by the Speaker, which is unable to move.
The Speaker can however communicate to the Listener at every time step, and so successful performance on the task corresponds to it helping the Listener to reach the target.
Whilst Lowe et al.
(2017) considered a problem involving only 3 landmarks (and showed that decentralised DDPG fails on this task), we increase this to 5.
This is illustrated in Figure 2, which shows a particular episode where the dark blue Listener has correctly reached the dark blue square target, due to the helpful communication of the Speaker.
We show performance on this task in Figure 3.
Perhaps surprisingly both MADDPG and MAAC struggle to perfectly solve the problem in this case, with reward values approximately corresponding to only reliably reaching 4 of the 5 possible targets.
We also implement a multi-agent ﬁngerprint for MADDPG (MADDPG+FP) similar to the one introduced by Foerster et al.
(2017), by including the training iteration index as input to the critic (see Appendix A.5),
but we do not ﬁnd it to improve performance.
By contrast, introducing our communication correction substantially improves both methods, enabling all 5 targets to be reached more often.
By looking at smoothed individual runs for MADDPG+CC we can see that it often induces distinctive rapid transitions from the 4 target plateau to the 5 target solution, whereas
MADDPG does not.
We hypothesise that these rapid transitions are due to our relabelling enabling the Listener to adapt quickly to changes in the Speaker’s policy to exploit cases where it has learned to select a better communication action (before it unlearns this).
Only the Speaker knows the target colour and must guide the
Listener to the target landmark.
We also consider a version of Cooperative Communication in which randomly, 25% of the time, a sent message is not received.
By relabelling with the same communication model, which also drops 25% of messages, we ﬁnd again that MADDPG+CC does much better than MADDPG (Appendix A.6).
4. H IERARCHICAL C OMMUNICATION
We next consider a problem with a hierarchical communication structure, which we use to elucidate the differences between ﬁrststep and ordered communication corrections (MADDPG+FCC vs MADDPG+OCC).
Our Hierarchical Communication problem (Figure 4) involves four agents.
One agent is a Listener and must navigate to one of four coloured landmarks, but it cannot see what the target colour is.
The remaining three Speaker agents can each see different colours which are certain not to be the target colour (indicated by their own colour in the diagram).
However, only one
Speaker can communicate with the Listener, with the rest forming a communication chain.
To solve this task, the ﬁrst Speaker must learn to communicate what colour it knows not to be correct, the middle Speaker must integrate its own knowledge to communicate the two colours which are not correct (for which there are 6 possibilities), and the ﬁnal Speaker must use this to communicate the identity of the target landmark to the Listener, which must navigate to the target.
We analyse performance of MADDPG, MADDPG+FCC and MADDPG+OCC on this task in Figure 5.
Whilst MADDPG+FCC applies the communication correction for each agent, it only does this over one time step, which prevents newly updated observations being used to compute the next correction.
By contrast, the ordered MADDPG+OCC, with k = 3 , starts from the root node, updates downstream observations and then uses the newly updated observations for the next update and so on (exploiting the DAG structure for more efﬁcient updates).
Our results show that MADDPG learns very slowly on this task and performs poorly, and MADDPG+FCC also performs poorly, with no a signiﬁcant improvement over MADDPG.
By contrast, MADDPG performs markedly better, learning at a much more rapid pace, a higher mean performance
We would like to ﬁnd out if the improved performance may be related to the ability of our method to recover correlations in communication.
We therefore also examine the correlation matrices for the vector of joint communication actions.
After having trained our MADDPG+OCC agents for 30 , 000 episodes, we can compare samples of communication from the starting point of learning and after learning has taken place.
We see that the correlation matrices are substantially different, with an intricate structure after learning reﬂecting the improved performance on the task.
Without a communication correction, a ‘before learning’ sample would be unchanged, and the sample would therefore be a poor reﬂection of the current social environment.
Using MADDPG+FCC we recover some of this structure, but there are still large differences, whereas MADDPG+OCC recovers this perfectly.
This indicates that an ordered relabelling scheme is beneﬁcial, and suggests that it may be increasingly important as the depth of the communication graph increases.
4. C OVERT C OMMUNICATION
In our next experiment, we consider a competitive task ﬁrst introduced by Lowe et al.
(2017) called Covert Communication.
In this task there are three agents; two Allies, one being a Speaker and another a Listener, and an Adversary.
The Speaker sends messages which are received by both the Listener and the Adversary.
However, whilst the Speaker would like the Listener to decode the message, it does not want the Adversary to decode the message.
Both Speaker and Listener observe a cryptographic ‘key’, which varies per episode and which the Adversary does not have access to.
Reward for the Allies is the difference between how well the Listener decodes the message and how well the Adversary decodes the message (with 0 reward corresponding to both agents decoding equally well).
One of the reasons this problem is interesting is because, unlike the previous Speaker-Listener problems which were ultimately cooperative, here there are competing agents.
This is known to be able to induce large amounts of non-stationarity in communication policies and the environment.
We therefore expect our experience relabelling to be effective in such situations, whether it be used for the Allies or the Adversary.
Our results in Figure 6 demonstrate this; using the communication correction for the Allies but not the Adversary improves the Allies performance, whereas using it for the Adversary but not the Allies degrades the Allies performance.
We ﬁnd that this is because the communication correction allows agents to rapidly adapt their policies when their opponents change their policy to reduce their reward.
For MADDPG+CC Allies, this involves ignoring the key and instead learning to exploit structure in the Adversary’s behaviour to do better than if the opponent were random (see Appendix A.8 for an analysis).
4. C OOPERATIVE C OMMUNICATION WITH M ULTIPLE T ARGETS
In our ﬁnal experiment, we apply our method to a situation in which the Speaker also takes environment actions and communicates cooperatively with two Listeners (Figure 7).
There are 5 landmarks and each agent is assigned a randomly chosen target landmark.
Each agent must navigate to cover its own target but only the Speaker sees the three target colours and so must communicate with the Listeners, sending different messages to each of them, to guide them to their targets.
To facilitate direct comparison of our plot with Figure 3, we divide the summed rewards by the number of agents.
As with Cooperative Communication, we ﬁnd that MADDPG+CC substantially outperforms MADDPG on this task.
R ELATED W ORK
Multi-agent RL has a rich history (Busoniu et al., 2008).
Communication is a key concept; however, much prior work on communication relied on predeﬁned communication protocols.
Learning communication was however explored by Kasai et al.
(2008) in the tabular case, and has been shown to resolve difﬁculties of coordination which can be difﬁcult for independent learners (Mataric, 1998; Panait & Luke, 2005).
Recent work in the deep RL era has also investigated learning to communicate, including how it can be learned by backpropagating through the communication channel (Foerster et al., 2016; Sukhbaatar et al., 2016; Havrylov & Titov, 2017; Peng et al., 2017; Mordatch & Abbeel, 2018).
Although we do not assume such convenient differentiability in our experiments, our method is in general applicable to this case, for algorithms which use a replay buffer.
Other approaches which have been used to improve multiagent communication include attention-based methods (Jiang & Lu, 2018; Iqbal & Sha, 2019; Das et al., 2019), intrinsic objectives (Jaques et al., 2019; Eccles et al., 2019) and structured graph-based communication (Agarwal et al., 2019).
Improvements to multi-agent experience replay were also considered by Foerster et al.
(2017) who used decentralised training.
Importance sampling as an off-environment correction was only found to provide slight improvements, perhaps due to the classical problem that importance ratios can have large or even unbounded variance (Robert & Casella, 2013), or with bias due to truncation.
Here we focus speciﬁcally on communicated messages; this allows us to relabel rather than reweight samples and avoid issues of importance sampling.
Of course, our method does not alter environment actions and so importance sampling for these may still be beneﬁcial.
In addition, it may in some cases be beneﬁcial to condition our relabelled messages on these environment actions, perhaps using autoregressive policies (Vinyals et al., 2017).
Another approach that has seen some success is using the training iteration number as a simple ‘ﬁngerprint’ for the critic (Foerster et al., 2017).
Although this can be more effective than importance sampling, we did not ﬁnd that it improved performance, perhaps because we use centralised rather than decentralised training, which can better handle issues of non-stationarity.
Our approach also bears a resemblance to Hindsight Experience Replay (HER) (Andrychowicz et al., 2017), which can be used for environments which have many possible goals.
It works by replacing the goal previously set for the agent with one which better matches the observed episode trajectory, which is particularly valuable in sparse reward problems where any given episode is unlikely to be rewarded.
This idea has been applied to hierarchical reinforcement learning (Levy et al., 2018), a ﬁeld which can address single-agent problems by invoking a hierarchy of communicating agents (Dayan & Hinton, 1993;
Vezhnevets et al., 2017).
In such systems, goals are set by a learning agent, and one can also relabel its experience by replacing the goal it previously set with one which better reﬂects the (temporally-extended) observed transition (Nachum et al., 2018).
Such ideas could naturally be combined with multi-agent HRL methods (Ahilan & Dayan, 2019; Vezhnevets et al., 2019; Ma & Wu, 2020), however they rely on communication corresponding to goals or reward functions.
In contrast, our method can be applied more generally, to any communicated message.
F UTURE W ORK
Similar to prior work using the MPE we experimented with problems which could be solved using feedforward policies (Lowe et al., 2017; Iqbal & Sha, 2019; Agarwal et al., 2019).
In the future it would be interesting to extend our idea to richer problems solvable only by recurrent policies, which depend on the history of observations and actions.
One strategy would be to build on existing methods which use experience replay with recurrent networks (Hausknecht & Stone, 2015; Kapturowski et al., 2018) whilst additionally relabelling when traversing the replay buffer.
However, an additional challenge in this setting is that a communicating agent may signal its intended environmental actions prior to taking them, and a naive relabelling would not account for this.
When relabelling in the recurrent setting it may therefore be beneﬁcial to condition on these ﬁxed future environment actions, or perhaps to consider the communication and environment as part of an hierarchical whole, and relabel them in a coordinated manner.
These merit further investigation.
We also did not conduct experiments with loopy communication graphs.
For problems of this nature, an exact correction requires relabelling from the start of each episode (although a k steps back approximation may be sufﬁcient in some cases).
Whilst the general principal of the OCC updates are unchanged in this case, it may be more computationally efﬁcient to sample and sequentially relabel full episodes of experience rather than individual experiences, as was done in this work.
C ONCLUSIONS
We have shown how off-policy learning for communicating agents can be substantially improved by relabelling experiences.
Our communication correction exploited the simple communication model which relates a sent message to a received message, and allowed us to relabel the received message with one more likely under the current policy.
To address problems with agents who were both senders and receivers, we introduced an ordered relabelling scheme, and found overall that our method improved performance on both cooperative and competitive tasks.
Our method provides a window into a general strategy for learning in non-stationary environments, in which acquired information in the present is used to alter past experience in order to improve future behaviour.
A A PPENDIX A. H YPERPARAMETERS
For all algorithms and experiments, we used the Adam optimizer with a learning rate of 0 .
001 and τ = 0 . 01 for updating the target networks.
The size of the replay buffer was 10 7 and we updated the network parameters after every 100 samples added to the replay buffer.
We used a batch size of 1024 episodes before making an update.
We trained with 20 random seeds for all experiments and show using a shaded region the standard error in the mean.
For MADDPG we use the implementation of Iqbal & Sha (2019) 3 .
For MADDPG and all MADDPG variants, hyperparameters were optimised using a line search centred on the experimental parameters used in Lowe et al.
(2017) but with 64 neurons per layer in each feedforward network (each with two hidden layers).
We found a value of γ = 0 .
75 worked best on Cooperative Communication with 6 landmarks evaluated after 50 , 000 episodes.
We use the Straight-Through Gumbel Softmax estimator with an inverse temperature parameter of 1 to generate discrete actions (see A.2).
For MAAC and MAAC+CC, we use the original implementation with unchanged parameters of
Iqbal & Sha (2019) 4 .
For our feedforward networks this corresponded to two hidden layers with 128 neurons per layer, 4 attend heads and γ = 0 .
A. D ISCRETE OUTPUT WITH S TRAIGHT -T HROUGH G UMBEL S OFTMAX
We work with a discrete action space and so for MADDPG, following prior work (Lowe et al., 2017; Iqbal & Sha, 2019), we use the Gumbel Softmax Estimator (Jang et al., 2016; Maddison et al., 2016).
In particular we use the Straight-Through Gumbel Estimator, which uses the GumbelMax trick (Gumbel, 1954) on the forward pass to generate a discrete sample from the categorical distribution, and the continuous approximation to it, the Gumbel Softmax, on the backward pass, to compute a biased estimate of the gradients.
The Gumbel Softmax enables us to compute gradients of a sample from the categorical distribution.
Given a categorical distribution with class probabilites π , we can generate k-dimensional sample vectors y ∈ ∆ n − 1 on the simplex, where:
[Equation]
for i = 1 , . . .
, k , where g i are samples drawn from Gumbel(0,1) 5 and β is an inverse temperature parameter.
Lower values of β will incur greater bias, but lower variance of the estimated gradient.
A. MADDPG
Although the original MADDPG proposed the multi-agent policy gradient of Equation 1, this can cause over-generalisation when a ∼ D is far from the current policies of other agents (Wei et al., 2018).
We use a correction introduced by Iqbal & Sha (2019) (for all variants of MADDPG), by sampling actions from the feedforward policy µ rather than the replay buffer, and providing this as input to the centralised critic.
[Equation] [Equation]
One minor difference from Iqbal & Sha (2019) is that they generate the discrete action of other agents using the greedy action whereas we instead take a Gumbel-Softmax sample with β = 1 .
using the procedure u ∼ Uniform(0 , 1) and computing g = − log( − log( u )) )
A. M ULTI - AGENT A CTOR A TTENTION C RITIC
Introduced by Iqbal & Sha (2019), MAAC learns the centralised critic for each agent by selectively paying attention to information from other agents.
This allows each agent to query other agents for information about their observations and actions and incorporate that information into the estimate of its value function.
In the case of MAAC+CC there is just a single shared critic and so we provide all altered samples to the same critic and do N critic updates (where N is the number of agents), and compare to MAAC which also does N critic updates but with the unaltered samples.
A. M ULTI - AGENT F INGERPRINT
We incorporate a multi-agent ﬁngerprint similar to the one introduced by Foerster et al.
We do this by additionally storing the training iteration number.
As training iteration numbers can get very large, and we ﬁnd this hurts performance (we do not do use batch normalisation), we divide this number by 100 , 000 before storing it (we typically train over 40 , 000 episodes, which corresponds to 1 , 000 , 000 iterations).
We then train MADDPG in the conventional way, but additionally provide this number to the centralised critic.
Foerster et al.
(2017) also varied exploration and so provide the exploration parameter to the decentralised critics.
We do not do this as we keep exploration ﬁxed throughout training (with Gumbel β = 1 ).
A. C OOPERATIVE C OMMUNICATION WITH D ROPPED M ESSAGES
We adapt the Cooperative Communication experiment such that 25% of sent messages are not received (the receiver sees all zeros).
When correcting communication we compute what the new message would be and then randomly drop 25% of messages (MADDPG+CC).
We drop messages by sampling from the communication model, irrespective of whether the original message that was sent was dropped or not (although an alternative approach of dropping the same messages could be explored in future).
We ﬁnd that applying this correction (MADDPG+CC) leads to better performance than MADDPG (Figure 8; left).
We also test a relabelling approach which ignores the dropped message probability and simply relays them without error (MADDPG+CC No Error).
As expected, this does much worse than MADDPG+CC, with performance approximately equal to MADDPG (although initial learning is faster).
Perhaps surprisingly, dropping messages 25% of the time does not provide an impediment to MADDPG+CC learning an effective policy.
As the Listener can observe its own instantaneous velocity, it is possible that when no message is received it learns to continue moving towards the target it is already heading to, which works most of the time.
Interestingly, if we compare a model trained with dropped messages to one trained without by evaluating it on the same task but with different probabilities of dropped messages, we ﬁnd that the model trained with dropped messages is substantially more robust (Figure 8; right).
A. E XPERIMENTAL E NVIRONMENTS
We provide a description of our environments in the main text but provide further information here.
The episode length for all experiments was 25 time steps.
For Cooperative Communication we alter the original problem by including two extra landmarks and by providing reward according to the negative distance from the target rather than the negative squared difference (as this elicits better performance in general).
Our reward plots show the reward per episode.
For Hierarchical Communication, the information as to the target landmark is distributed amongst
Speakers who cannot move and communicate in a chain to the Listener.
As with Cooperative Communication we display rewards per episode and use the negative distance rather than the negative square distance.
For Covert Communication we plot the average reward per time step and use the original task, with one minor change of dividing the reward by 2.
This ensures that the magnitude of the Allies rewards is less than or equal to 1 (for example, when the Allies are always right and the Adversary is always wrong the Allies reward is 1).
For Cooperative Communication with Multiple Targets, each agent must navigate to its target landmark but only the Speaker sees all three target colours.
The Speaker sends separate discrete messages to each Listener.
A shared reward equal to the summed negative distance of each agent from its target is provided to all agents.
Our reward plots show the reward per episode divided by 3 (the number of agents).
A. A NALYSIS OF C OVERT C OMMUNICATION
When used for the Allies, our communication correction substantially improved their performance.
Interestingly, we found that rather than use the key, MADDPG+CC is able to learn communication actions which mislead its adversary into making the wrong choice, whilst simultaneously helping the Listener to make the right choice.
This enables MADDPG+CC to achieve more reward than could be achieved against a random opponent (on average this would be 0.5).
To emphasise this, we also trained agents on the same problem but without providing the Allies with the key.
We ﬁnd that when both Allies and the Adversary are trained using MADDPG the reward for the Allies is zero, because the Allies cannot adapt any faster than the Adversary.
By contrast, performance for the MADDPG+CC Allies against a
MADDPG opponent remains very good, with the MADDPG+CC Allies continuing to out-adapt the
Adversary, causing it to perform even worse than random.
A. A LGORITHM
We outline our Ordered Communication Correction algorithm here (First-step CC can be recovered by setting K = 1 ).
We write it for the general case, although fewer updates can be used to achieve equivalent results if the communication graph is a DAG.
For notational simplicity we use a ∼ π ( a | o ) to indicate that each agent’s policy
π i , receiving input
o i was used to generate a i and thus collectively the joint action a .
Note that our description of the set of actions
A i =
A ei × A mi allows agents to both act and communicate simultaneously; in cases where agents only do one of these we allow for a ‘no op’ action.
An alternative way to deﬁne actions for each agent
A i is to divide them into disjoint environment actions A ei and explicit communication actions
A mi such that A ei ∪ A mi =
A i and A ei ∩ A mi = ∅ (Lowe et al., 2019).
To deal with this case one would require only minor alterations to the algorithm.
Given:
An off-policy multi-agent RL algorithm A using centralised training with decentralised policies
communication under
o mt +1 |
mt
sent
foreach episode do
Receive initial state x and joint observations o ; foreach episode do o t − K o t a t r t + o t +
Compute messages
Compute
ˆ
; end for do
Set agent
’s received receiving a m i t
= ˆ a m t , ¬ i ⊕ a m t
; ˆ o m i t
=
ˆ o m t , ¬ l ⊕ o m t , l ; ˆ o m t t
+
=
ˆ o m t +
¬ l ⊕ o m t + l ; ˆ B i t =
( ˆ o i t ˆ a i t r t 1 , ˆ o i t +