T HE
E MERGENCE OF I NDIVIDUALITY IN M ULTIA GENT R EINFORCEMENT L
EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Individuality is essential in human society. It induces the division of labor and thus improves the efﬁciency and productivity.
Similarly, it should also be a key to multi-agent cooperation.
Inspired by that individuality is of being an individual separate from others, we propose a simple yet efﬁcient method for the emergence of individuality (EOI) in multi-agent reinforcement learning (MARL).
EOI learns a probabilistic classiﬁer that predicts a probability distribution over agents given their observation and gives each agent an intrinsic reward of being correctly predicted by the classiﬁer.
The intrinsic reward encourages the agents to visit their own familiar observations, and learning the classiﬁer by such observations makes the intrinsic reward signals stronger and in turn makes the agents more identiﬁable.
To further enhance the intrinsic reward and promote the emergence of individuality, two regularizers are proposed to increase the discriminability of the classiﬁer.
We implement EOI on top of popular MARL algorithms.
Empirically, we show that
EOI outperforms existing methods in a variety of multi-agent cooperative scenarios.
I NTRODUCTION
Humans develop into distinct individuals due to both genes and environments (Freund et al., 2013).
Individuality induces the division of labor (Gordon, 1996), which improves the productivity and efﬁciency of human society.
Analogically, the emergence of individuality should also be essential for multi-agent cooperation.
Although multi-agent reinforcement learning (MARL) has been applied to multi-agent cooperation, it is widely observed that agents usually learn similar behaviors, especially when the agents are homogeneous with shared global reward and co-trained (McKee et al., 2020).
For example, in multi-camera multi-object tracking (Liu et al., 2017), where camera agents learn to cooperatively track multiple objects, the camera agents all tend to track the easy object.
However, such similar behaviors can easily make the learned policies fall into local optimum.
If the agents can respectively track different objects, they are more likely to solve the task optimally.
Many studies formulate such a problem as task allocation or role assignment (Sander et al., 2002; Dastani et al., 2003; Sims et al., 2008).
However, they require that the agent roles are rule-based and the tasks are pre-deﬁned, and thus are not general methods.
Some studies intentionally pursue difference in agent policies by diversity (Lee et al., 2020; Yang et al., 2020) or by emergent roles (Wang et al., 2020a), however, the induced difference is not appropriately linked to the success of task.
On the contrary, the emergence of individuality along with learning cooperation can automatically drive agents to behave differently and take a variety of roles, if needed, to successfully complete tasks.
Biologically, the emergence of individuality is attributed to innate characteristics and experiences.
However, as in practice RL agents are mostly homogeneous, we mainly focus on enabling agents to develop individuality through interactions with the environment during policy learning.
Intuitively, in multi-agent environments where agents respectively explore and interact with the environment, individuality should emerge from what they experience.
In this paper, we propose a novel method for the emergence of individuality (EOI) in MARL.
EOI learns a probabilistic classiﬁer that predicts a probability distribution over agents given their observation and gives each agent an intrinsic reward of being correctly predicted probability by the classiﬁer.
Encouraged by the intrinsic reward, agents tend to visit their own familiar observations.
Learning the probabilistic classiﬁer by such observations makes the intrinsic reward signals stronger and in turn makes the agents more identiﬁable.
In this closed loop with positive feedback, agent individuality emerges gradually.
However, at early learning stage, the observations visited by different agents cannot be easily distinguished by the classiﬁer, meaning the intrinsic reward signals are not strong enough to induce agent characteristics.
Therefore, we propose two regularizers for learning the classiﬁer to increase the discriminability, enhance the feedback, and thus promote the emergence of individuality.
EOI is compatible with centralized training and decentralized execution (CTDE).
We realize EOI on top of two popular MARL methods, MAAC (Iqbal & Sha, 2019) and QMIX (Rashid et al., 2018).
For MAAC, as each agent has its own critic, it is convenient to shape the reward for each agent.
For QMIX, we introduce an auxiliary gradient and update the individual value function by both minimizing the TD error of the joint action-value function and maximizing its cumulative intrinsic rewards.
We evaluate EOI in three scenarios where agents are preferred to take different roles, i.e. ,
Pac-Men, Windy Maze, and Fireﬁghters, and we empirically demonstrate that EOI signiﬁcantly outperforms existing methods.
Additionally, in a micro-task of StarCraft II (Samvelyan et al., 2019) where the need for the division of labor is unknown, EOI also learns faster than existing methods.
By ablation studies, we conﬁrm that the proposed regularizers indeed improve the emergence of individuality even if agents have the same innate characteristics.
R ELATED W ORK
MARL.
We consider the formulation of Decentralized Partially Observable Markov Decision Process (Dec-POMDP), where at each timestep t each agent i receives a local observation o ti , takes an action a ti , and gets a shared global reward r t .
Agents together aim to maximize the expected return
E (cid:80) Tt =
γ t r t , where γ is a discount factor and T is the time horizon.
Many methods have been proposed for Dec-POMDP, most of which adopt CTDE.
Some methods (Lowe et al., 2017; Foerster et al., 2018; Iqbal & Sha, 2019) extend policy gradient into multi-agent cases.
Value function factorization methods (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019) decompose the joint value function into individual value functions.
Communication methods (Das et al., 2019; Jiang et al., 2020) share information between agents for better cooperation.
Behavior Diversiﬁcation.
Many cooperative multi-agent applications require agents to take different behaviors to complete the task successfully.
Behavior diversiﬁcation can be handcrafted or emerge through agents’ learning.
Handcrafted diversiﬁcation is widely studied as task allocation or role assignment.
Heuristics (Sander et al., 2002; Dastani et al., 2003; Sims et al., 2008; Macarthur et al., 2011) assign speciﬁc tasks or pre-deﬁned roles to each agent based on goal, capability, visibility, or by search.
M 3 RL (Shu & Tian, 2019) learns a manager to assign suitable sub-tasks to rule-based workers with different preferences and skills.
These methods require that the sub-tasks and roles are pre-deﬁned, and the worker agents are rule-based.
However, in general, the task cannot be easily decomposed even with domain knowledge and workers are learning agents.
The emergent diversiﬁcation for single agent has been studied in DIAYN (Eysenbach et al., 2019), which learns reusable diverse skills in complex and transferable tasks without any reward signal by maximizing the mutual information between states and skill embeddings as well as entropy.
In multi-agent learning, SVO (McKee et al., 2020) introduces diversity into heterogeneous agents for more generalized and high-performing policies in social dilemmas.
Some methods are proposed for behavior diversiﬁcation in multi-agent cooperation.
ROMA (Wang et al., 2020a) learns a role encoder to generate the role embedding, and learns a role decoder to generate the neural network parameters.
However, there is no mechanism that guarantees the role decoder can generate different parameters, taking as input different role embeddings.
Learning low-level skills for each agent using DIAYN is considered in Lee et al.
; Yang et al.
(2020), where agents’ diverse low-level skills are coordinated by the high-level policy.
However, the independently trained skills limit the cooperation, and diversity is not considered in the high-level policy.
M ETHOD
Individuality is of being an individual separate from others.
Motivated by this, we propose EOI, where agents are intrinsically rewarded in terms of being correctly predicted by a probabilistic classiﬁer that is learned based on agents’ observations.
If the classiﬁer learns to accurately distinguish agents, agents should behave differently and thus individuality emerges.
Two regularizers are introduced for learning the classiﬁer to enhance intrinsic reward signal and promote individuality.
Unlike existing work on behavior diversiﬁcation, EOI directly correlates individuality with the task by intrinsic reward, and thus individuality emerges naturally during agents’ learning.
EOI can be applied to
Dec-POMDP tasks and trained along with CTDE algorithms.
We design practical techniques to implement EOI on top of two popular MARL methods, MAAC and QMIX.
We also propose an unbiased gradient estimation for learning from other agents’ experiences to accelerate learning, which appears in Appendix A.1.
3. I NTRINSIC R EWARD
As illustrated in Figure 1, two agents learn to collect two items, where item b is the easier one.
Collecting each item, they will get a global reward of +1 .
Sequentially collecting the two items by one agent or together is sub-optimal, sometimes impossible when the task has a limited horizon.
The optimal solution is that the two agents go collecting different items simultaneously.
It is easy for both agents to learn to collect the easier item b .
However, after that, item a becomes even harder to be explored, and the learned policies easily fall at local optimum.
Nevertheless, the emergence of individuality can address the problem, e.g. , agents prefer to collect different items.
[Figure]
To enable agents to develop individuality, EOI learns a probabilistic classiﬁer P ( I | O ) to predict a probability distribution over agents given on their observation, and each agent takes the correctly predicted probability as the intrinsic reward at each timestep.
Thus, the reward function for agent i is modiﬁed as
[Equation]
where r is the global environmental reward, p ( i | o i ) is the predicted probability of agent i given its observation o i , and α is a tuning parameter to weight the intrinsic reward.
With the reward shaping, EOI works as follows.
If there is initial difference between agent policies in terms of visited observations, the difference is captured by P ( I | O ) as it is ﬁtted using agents’ experiences.
The difference is then fed back to each agent as an intrinsic reward.
As agents maximize the expected return, the difference in agents’ policies is exacerbated together with optimizing the environmental return.
Therefore, the learning process is a closed loop with positive feedback.
As agents progressively behave more identiﬁably, the classiﬁer can distinguish agents more accurately, and thus individuality emerges gradually.
The classiﬁer P φ ( I | O ) is parameterized by a neural network φ and learned in a supervised way.
At each timestep, we take each agent i ’s observation
o i as input and the agent index i as the label and store the pair <
o i , i > into a buffer B .
φ is updated by minimizing the cross-entropy loss ( CE ), which is computed based on the uniformly sampled batches from B .
3. R EGULARIZERS OF P φ ( I | O )
In the previous section, we assume that there is some difference between agents’ policies.
However, in general, the difference between initial policies is small (even no differences if agents’ policies are initially by the same network weights), and the policies will quickly learn similar behaviors as in the example in Figure 1.
Therefore, the intrinsic rewards are nearly the same for each agent, which means no feedback in the closed loop.
To generate the feedback in the closed loop, the observation needs to be identiﬁable and thus the agent can be distinguished in terms of observations by P φ ( I | O ) .
To address this, we propose two regularizers: positive distance (PD) and mutual information (MI) for learning P φ ( I | O ) .
Positive Distance.
The positive distance is inspired from the triplet loss (Schroff et al., 2015) in face recognition, which is proposed to learn identiﬁable embeddings.
Since o ti and its previous observations { o t − ∆ t i , o t − ∆ t +1 i , · · · , o t − 1 i } are distributed on the trajectory generated by agent i , the previous observations in the ∆ t -length window could be seen as the positives of o ti .
To make the probability distribution on the anchor o t i close to that on the positives, we sample an observation o t (cid:48)
i from { o t − ∆ t i , o t − ∆ t +1 i , · · · , o t − 1 i } and minimize the cross-entropy loss
[Equation]
The positive distance minimizes the intra-distance between the observations with the same “identity”, which hence enlarges the margin between different “identities”.
As a result, the observations become more identiﬁable.
Since the positives are naturally deﬁned on the trajectory, the identiﬁability generated by the positive distance is actually induced by the agent policy.
Deﬁning the negatives is hard but we ﬁnd that just using the positive distance works well in practice.
Mutual Information.
If the observations are more identiﬁable, it is easier to infer the agent that visits the given observation most, which indicates the higher mutual information between the agent index and observation.
Therefore, to further increase the discriminability of the classiﬁer, we maximize their mutual information,
[Equation]
Since we store <
o i , i > of every agent in B , the number of samples for each agent is equal.
Fitting
P φ ( I | O ) using batches from B ensures H ( I ) is a constant.
To maximize MI( I ; O ) is to minimize
H ( I | O ) .
Therefore, equivalently, we sample batches from B and minimize
[Equation]
Therefore, the optimization objective of P φ ( I | O ) is to minimize
[Equation]
[Equation]
where β 1 and β 2 are hyperparameters.
The regularizers increase the discriminability of P φ ( I | O ) , make the intrinsic reward signals stronger to stimulate the agents to be more distinguishable, and eventually promote the emergence of individuality.
In this sense, P φ ( I | O ) not only is the posterior statistics, but also serves as the inductive bias of agents’ learning.
The learning process of EOI is illustrated in Figure 1.
3. I MPLEMENTATION WITH MAAC AND QMIX
Existing methods for reward shaping in MARL focus on independent learning agents e.g. , McKee et al.
; Wang et al.
(2020b);
Du et al.
How to shape the reward in Dec-POMDP for centralized training has not been deeply studied.
Since the intrinsic reward can be exactly assigned to the speciﬁc agent, individually maximizing the intrinsic reward is more efﬁcient than jointly maximizing the sum of all agents’ intrinsic rewards (Hughes et al., 2018).
Adopting this idea, we respectively present the implementation with MAAC and QMIX for realizing EOI.
MAAC is an off-policy actor-critic algorithm, where each agent learns its own critic, thus it is convenient to directly give the shaped reward r + αp φ ( i | o i ) to the critic of each agent i , without modifying other components, as illustrated in Figure 2(a).
The TD error of the critic and the policy gradient are the same as in MAAC (Iqbal & Sha, 2019).
[Figure] [Figure]
In QMIX, each agent i has an individual action-value function Q ai .
All the individual action-value functions are monotonically mixed into a joint action-value Q tot by a mixing network.
Each agent selects the action with the highest individual value, but the individual value has neither actual meaning nor constraints (Rashid et al., 2018).
Therefore, we can safely introduce an auxiliary gradient of the intrinsic reward to the individual action-value function Q ai .
Each agent i learns an intrinsic value function (IVF) Q pi , which takes as input the observation o
i and the individual action-value vector
Q ai ( o i ) and approximates E (cid:80) Tt =
γ t p ( i | o ti ) by minimizing the TD error,
[Equation]
, Q
[Equation]
where y = p φ ( i
[Equation]
o
[Equation]
+ γ ¯ Q pi [Equation]
[Equation]
(6) ¯ Q a i and ¯ Q p i are the target value functions and D is the replay buffer.
In order to improve both the global reward and intrinsic reward, we update Q ai , parameterized by θ i , towards maximizing E [ Q pi ( o i , Q ai ( o i ; θ i ))] along with minimizing the TD error of Q tot (denoted as δ tot ), as illustrated in Figure 2(b).
Since the intrinsic value function is differentiable with respect to the individual action-value vector Q ai ( o i ; θ i ) , we can establish the connection between Q pi and Q ai by the chain rule, and the gradient of θ i is,
[Equation]
MAAC and QMIX are off-policy algorithms, and the environmental rewards are stored in the replay buffer.
However, the intrinsic rewards are recomputed in the sampled batches before each update, since P φ ( I | O ) is co-evolving with the learning agents, the previous intrinsic reward is outdated.
The joint learning process of the classiﬁer and agent policies can be mathematically formulated as a bi-level optimization, which is detailed in Appendix A.2.
E XPERIMENTS
To clearly interpret the mechanism of EOI, we design three scenarios: Pac-Men, Windy Maze, and
Fireﬁghters, which are illustrated in Figure 3, and adopt a micro-task of StarCraft II (Samvelyan et al., 2019).
In the evaluation, we ﬁrst verify the effectiveness of EOI and the two regularizers on both
MAAC and QMIX by ablation studies.
Then, we compare EOI against EDTI (Wang et al., 2020b) to verify the advantages of individuality with sparse reward, and against ROMA (Wang et al., 2020a) and Lee et al.
; Yang et al.
(2020) (denoted as HC and HC-simul respectively) to investigate the advantages of individuality over emergent roles and diversity.
The details about the experimental settings and the hyperparameters are available in Appendix A.3.
4. P AC -M EN
There are four pac-men (agents) initialized at the maze center, and some randomly initialized dots, illustrated in Figure 3 ( left ).
Each agent has a local observation that contains a square view with 5 ×
[Figure]
Agent 1 Agent 2 Agent
Agent
Agent 1 Agent 2 Agent
Agent− 0 .
3 − 0 .
2 − 0 .
4 − 0 .− 0 .
3 − 0 .− 0 .
grids centered at the agent itself.
At each timestep, each agent can move to one of four neighboring grids or eat a dot.
The agents only get a global reward, i.e. , the total eaten dots, at the ﬁnal timestep.
As depicted Figure 4, MAAC and QMIX get the lowest mean environmental reward respectively, since some agents learn to go to the same room and compete for the dots.
This can be veriﬁed by the position distribution of QMIX agents in Figure 5 ( left ), where three agents move to the same room.
MAAC agents behave similarly.
At the early stage of training, it is easy for the agents to explore the bottom room and eat dots there to improve the reward.
Once the agents learn such policies, it is hard to explore other rooms, so the agents learn similar behaviors and fall at the local optimum.
Driven by the intrinsic reward without both regularizers, EOI obtains better performance than MAAC and QMIX.
But the improvement is not signiﬁcant since the observations of different agents cannot be easily distinguished when there is little difference between initial policies.
The regularizers of PD and MI can increase the discriminability of P φ ( I | O ) , providing stronger intrinsic signals.
Guided by P φ ( I | O ) with PD or MI, the agents go to different rooms and eat more dots.
MI theoretically increases the discriminability even the initial policies have no differences, while PD makes the observations distinguishable according to policies.
Combining the advantages of the two regularizers leads to higher and steadier performance, as shown in Figure 4.
With both two regularizers, the agents respectively go to the four rooms and achieve the highest reward, which is indicated in Figure 5 ( mid ).
We also visualize the observations of different agents by kernel PCA, as illustrated in Figure 5 ( right ), where darker color means higher correctly predicted probability of P φ
( I | O ) .
We can see P φ ( I | O ) can easily distinguish agents given their observations.
To further investigate the effect of the regularizers, we show the learning curves of intrinsic reward and environmental reward of EOI with or without regularizers in Figure 6.
EOI with the regularizers converges to higher intrinsic reward than that without regularizers, meaning agents behave more distinctly.
With regularizers, the rapid increase of intrinsic reward occurs before that of environmental reward, which indicates the regularizers make P φ ( I | O ) also serve as the inductive bias for the emergence of individuality.
We also investigate the inﬂuence of the difference between initial policies.
The action distributions (over visited states) of the four initial action-value functions in QMIX are illustrated in Figure 7, where we ﬁnd there is a large difference between them.
The inherent observation makes agents distinguishable initially, which is the main reason EOI without the regularizers works well.
We then initiate the four action-value functions with the same network weights and re-run the experiments.
The performance of EOI without the regularizers drops considerably, while EOI with the regularizers has almost no difference, as illustrated in Figure 8.
Therefore, we can conclude that PD and MI also make the learning more robust to the initial policies.
In other words, even with the same innate characteristics, agent individuality still emerges.
[Equation]
a n i n t r i n s i c r e w a r d p
( i | o )
[Equation]
M e a n e n v i r o n m e n t a l r e w a r d r p
( i | o ) ofEOI+MAAC p ( i |
[Figure]
e a n i n t r i n s i c r e w a r d p
( i |
[Figure] [Figure]
e a n e n v i r
[Figure]
n
[Figure]
e n t a l r e w a r d r
[Figure]
[Figure] [Figure]
[Figure]
In the scenario, there are two pirates (agents) initialized at the bottom of the T-shaped maze, and two treasure boxes initialized at the two ends, as illustrated in Figure 3 ( mid ).
Each agent has a local observation that contains a square view with 5 × 5 grids and could move to one of four neighboring grids or open the box.
There is a wind running right from the dotted line.
Shifted by the wind, forty percent of the time, the agent will move to the right grid whichever action it takes.
The agents could only get a global reward, i.e. , the number of the opened boxes, at the ﬁnal timestep.
Figure 9 shows the learning curves of EOI+MAAC and EOI+QMIX.
Under the effect of the wind, it is easy to go to the right end and open the box, even if the agent acts randomly.
Limited by the time horizon, it is impossible that the agent ﬁrst opens the right box and then goes to the left end for the other box.
MAAC and QMIX only achieve the mean reward of 1 .
Due to the wind and the small state space, the trajectories of the two agents are similar, thus EOI without regularizers provides little help, where the intrinsic reward is not strong enough to induce individuality.
With regularizers, the observations on the right path and the left path can be discriminated gradually.
In the learning process, we ﬁnd that the agents will ﬁrst both learn to open the right box.
Then one of them will change its policy and go left for higher intrinsic reward, and eventually the agents develop the distinct policies and get a higher mean reward.
The left box is hard to explore, thus we also compare EOI with EDTI (Wang et al., 2020b), a multiagent exploration method.
EDTI gives the agent an intrinsic motivation, considering both curiosity and inﬂuence.
It rewards the agent to explore the rarely visited states and to maximize the inﬂuence on the expected returns of other agents.
The optimization of the intrinsic motivation is integrated with PPO (Schulman et al., 2017).
However, EDTI does not perform well in this scenario.
When the agent policies are similar, the two agents will be curious about similar states.
When they both explore novel states, the environmental reward decreases, which draws the agents to go back to the right box again.
So the curiosity is not stable, and we can see its ﬂuctuation during training in Figure 9.
4. F IREFIGHTERS
There are some burning grids in two areas and two rivers in other areas.
Four ﬁreﬁghters (agents) are initiated at the same burning area, illustrated in Figure 3 ( right ).
The agent has a local observation that contains a square view with 5 × 5 grids and can move to one of four neighboring grids, spray water, or pump water.
They share a water tank, which is initiated with four units water.
Once the agent sprays water on the burning grid, it puts out the ﬁre and consumes one unit water.
Once the agent pumps water in the river, the water in the tank increases by one unit.
The agents only get a global reward, i.e. , the number of the extinguished burning grids, at the ﬁnal timestep.
[Figure]
n v i r o n m e n t a l r e w a r d
[Figure] [Figure]
As illustrated by Figure 10 ( left ), MAAC and QMIX fall into local optimum.
The agents learn to put out the ﬁre around the initial position until the water is exhausted, and get the mean reward 4 .
They do not take different roles of pumping water or putting out the ﬁre farther.
Beneﬁted from the emergent individuality, EOI+MAAC achieves the best performance with a clear division of labor, where two agents go to the river for pumping water, and two agents go to different areas for ﬁghting with ﬁre.
EDTI could escape from the local optimum, but it converges to a lower mean reward than EOI.
This is because encouraging curiosity and inﬂuence cannot help division of labor, and the intrinsic motivation of EDTI is too complex to suit the cases with more than two agents.
ROMA (Wang et al., 2020a), based on QMIX, uses a role encoder taking the local observation to generate role embedding, and decodes the role to generate the parameters of individual action-value function in QMIX.
However, ROMA converges to a lower mean reward than EOI+QMIX and learns much slower.
Due to the large parameter space, generating various parameters for the emergent roles is less efﬁcient than encouraging diverse behaviors by reward shaping.
Moveover, since the agents are initiated with the same observations, the role encoder will generate the similar role embeddings, which means similar agent behaviors at the beginning, bringing difﬁculty to the emergent roles.
HC (Lee et al., 2020)
ﬁrst provides each agent a set of diverse skills, which are learned independently by DIAYN (before 6 × 10 3 episodes), then learns a high-level policy to select the skill for each agent.
Since the agents are trained independently without coordinating with others’ skills, the skills could be diverse but might not be useful, e.g. , ignoring the pumping action.
So it converges to a lower mean reward.
We also train the primitive skills and high-level policy simultaneously, i.e. , HC-simul
(Yang et al., 2020), and ﬁnd it falls into local optimum.
Before the diversity emerges in the skills, the high-level policy learns to choose the skill of putting out the nearby ﬁre for the cheap reward, however other skills would not be chosen to learn diversity.
That is to say, HC-simul only considers diversity in the low-level policies but ignores that in the high-level policy.
EOI encourages individuality with the coordination provided by centralized training, considering both of the global reward and other agents’ policies.
Therefore, EOI is more effective than HC and HC-simul.
Moreover, EOI also outperforms
ROMA and HC in Pac-Men, available in Appendix A.4.
4. S TAR C RAFT II
We additionally evaluate EOI on a micromanagement scenario of StarCraft II (Samvelyan et al., 2019) 8m , where 8 Marines face 8 enemy Marines and the need for the division of labor is unknown.
As illustrated in Figure 10 ( right ), EOI learns faster than ROMA and QMIX, showing the classiﬁer
P φ ( I | O ) could be effectively trained and provide useful intrinsic signals in the highly dynamic environment with many agents.
Note that we represent the individual action-value function as MLP, not GRU, which means that the action value is generated only based on o ti , and the agents do not share the neural network parameters so that the result of QMIX is different from that in the original paper (Rashid et al., 2018).
C ONCLUSION
We have proposed EOI, a novel method for the emergence of individuality in MARL.
EOI learns a probabilistic classiﬁer that predicts a probability distribution over agents given their observation and gives each agent an intrinsic reward of being correctly predicted by the classiﬁer.
Two regularizers are introduced to increase the discriminability of the classiﬁer.
We realized EOI on top of two popular
MARL methods and empirically demonstrated that EOI outperforms existing methods in a variety of multi-agent cooperative tasks.
A A PPENDIX A. L EARNING FROM E XPERIENCES OF O THER A GENTS
When agents are exploring the environment, agents have various exploration patterns.
The experiences of one agent can be helpful for the learning of other agents, i.e. , improving data efﬁciency and accelerating learning.
Therefore, we propose an unbiased gradient estimate for learning from other agents’ experiences.
In centralized training, all agents’ experience tuples < O t , A t , r t , O t +1 , s t , s t +1 > are stored in the replay buffer, and sampled as batch Ω to compute the gradients jointly.
For each agent i , the probability of sampled observation is p ( o | i ) .
To use others’ experiences, we randomly rearrange the agent index in a subset of Ω , denoted as ω .
For example, in a two-agent case, two agents exchange o t , a t , and o t +1 to build a new training experience.
In the experiences with shufﬂed index ω , the training experience of each agent i is sampled based on p ( o ) rather than p ( o | i ) .
By Bayes rule,
[Equation]
Since the number of observations for each agent is equal in the training set, p ( i ) is a constant.
Then we have
[Equation]
We use p φ ( i | o ) as importance weight (IW) for the gradient of individual action-value function/policy to make sure that the estimate of the expected gradients based on p ( o ) is unbiased from that based on p ( o | i ) .
[Equation]
[Figure]
Figure 11(a) and 11(b) show the learning curves of EOI (with both PD and MI) with unbiased learning from others’ experiences (LFO) and without importance weight respectively in Pac-Men.
LFO greatly accelerates the learning of EOI since the agents have access to more experiences, and it converges to the similar performance with EOI since the gradient estimation is unbiased.
Applying
LFO to MAAC and QMIX also accelerates the convergence.
Without important weight, LFO leads to lower mean reward, because the gradient estimate using others’ experience is biased, which would harm the learning.
To evaluate how much knowledge is learned from others, in Figure 11(c)
we plot the curves of mean important weight on other agents’ experiences E
i,o [ 1 n − 1 (cid:80) j (cid:54) =
i p φ ( i | o j )] , where i and j are the agent indexes before rearranging.
At the beginning, as the agents are exploring the environment, the mean importance weight is high, which indicates that the agent learns a lot from others’ experiences.
In later phase, the mean importance weight becomes very low, because individuality has emerged and the experiences of agents are very different, the agent cannnot beneﬁt from others’ experiences.
In the experiment, | ω | / | Ω | is set as a constant 0 .
However, enlightened by the trend of the mean importance weight, we could also adjust | ω | / | Ω | according to a pre-deﬁned schedule, e.g. , time-decaying or adaptive methods related to the mean importance weight.
A. M ATHEMATICAL F ORMULATION
Let θ and φ denote the parameters of the joint policies and the probabilistic classiﬁer, respectively.
Then, the whole learning process corresponds to the following bi-level optimization:
[Equation]
where J is the RL objective with intrinsic reward, L is the loss function of the probabilistic classiﬁer, and φ is an implicit function of θ .
Therefore, to solve this optimization, we can iteratively update θ by
[Equation]
where
[Equation]
which is obtained by the implicit function theorem.
In practice, the second-order term is neglected due to high computational complexity, without incurring signiﬁcant performance drop, such as in meta-learning and GANs.
Therefore, we can solve the bi-level optimization by the ﬁrst-order approximation with iterative updates:
[Equation]
A. H YPERPARAMETERS
The hyperparameters of EOI and the baselines in each scenario are summarized in Table 1.
Since
QMIX and MAAC are off-policy algorithms with replay buffer, we do not need to maintain the buffer
[Table]
B but build the training data from the replay buffer D .
For EDTI, ROMA, HC
[Table]
and HC-simul, we use their default settings.
In the experiments, the agents do not share the weights of neural network since parameter sharing causes similar agent behaviors.
All the curves are plotted using mean and standard deviation.
A. A DDITIONAL R ESULTS IN P AC -M EN
In Figure
[Table]
we compare EOI with HC and ROMA in Pac-Men.
EOI converges to a higher reward than these two methods for behavior diversiﬁcation, showing that EOI is more conducive to the performance improvement.
To investigate the performance of EOI in the scenarios where individuality is not necessary, we perform the experiments on an easy version of Pac-Men, where the dots are only distributed in the bottom room, illustrated in Figure 13 ( left ).
To complete this task, the agents do not need to go to different rooms.
As depicted in Figure 13 ( right ), EOI+QMIX converges to the same mean reward with the QMIX, which veriﬁes that EOI does not conﬂict with the success of task where agent individuality is not necessary. In Figure 14(a) [Table]
we test EOI+MAAC with different α to investigate the effect of α on the emergent individuality. When α is too large, the agents will pay much attention to learn the individualized behaviors, which harms the optimization of the cumulative reward. However, the results of EOI+MAAC with different α are all higher than that of the vanilla MAAC without EOI ( α = 0 ), which shows that the proposed intrinsic reward could encourage individualized policies for better cooperation. In
Figure 14(b), we apply EOI to PPO, where each agent is controlled by an independent PPO.
EOI+PPO outperforms the version without EOI, which veriﬁes the effectiveness of EOI on independent learning algorithms.
Intuitively, maximizing the KL-divergence between agents (cid:80) j (cid:54)
= i KL ( p i ( a | o ) ||
p j ( a | o )) for agent
i as a regularization could also encourages the individuality.
In Figure 14(c), we compare
EOI against the KL-divergence and ﬁnd that KL-divergence does not outperform the vanilla MAAC,
[Figure]
[Figure]
[Figure]
and the learning of KL-divergence is not stable.
We think that the KL-divergence term is hard to optimize with more than two agents.
A. S IMILARITY AND D IFFERENCE B ETWEEN EOI AND DIAYN
DIAYN is proposed to learn diverse skills with single-agent RL in the absence of any rewards.
It trains the agent by maximizing the mutual information between skills ( Z ) and states ( S ), maximizing the policy entropy, and minimizing the mutual information between skills and actions ( A ) given the state.
The optimization objective is
[Equation]
To maximize this objective, in practice DIAYN gives the learning agent an intrinsic reward log q ( z | s ) − log p ( z ) , where q ( z | s ) approximates p ( z | s ) and p ( z ) is a prior distribution.
EOI gives each agent an intrinsic reward of p ( i | o ) .
Let agents correspond to skills and observations correspond to states, then EOI has a similar intrinsic reward with DIAYN.
However, simply adopting the intrinsic reward in MARL, which can be seen as EOI without the regularizers, cannot provide strong intrinsic reward signals for the emergence of individuality.
The two regularizers, which capture the unique characteristics of MARL, strength the reward signals and promote the emergence of individuality in
MARL.