R OLE D IVERSITY M ATTERS : A S TUDY OF C OOPERATIVE T RAINING S TRATEGIES FOR M ULTI -A GENT RL
Anonymous authors
Paper under double-blind review
A BSTRACT
Cooperative multi-agent reinforcement learning (MARL) is making rapid progress for solving tasks in a grid world and real-world scenarios, in which agents are given different attributes and goals.
For example, in Starcraft II battle tasks, agents are initialized with the various move, defense, and attack abilities according to their unit types.
Current researchers tend to treat different agents equally and expect them to form a joint policy automatically.
However, ignoring the differences between agents in these scenarios may bring policy degradation.
Accordingly, in this study, we quantify the agent’s difference and study the relationship between the agent’s role and the model performance via Role Diversity , a metric that can describe MARL tasks.
We deﬁne role diversity from three perspectives: policybased, trajectory-based, and contribution-based to fully describe the agents’ differences.
Through theoretical analysis, we ﬁnd that the error bound in MARL can be decomposed into three parts that have a strong relation to the role diversity.
The decomposed factors can signiﬁcantly impact policy optimization on parameter sharing, communication mechanism, and credit assignment strategy.
Role diversity can therefore serve as a ﬂag for selecting a suitable training strategy and helping to avoid possible bottlenecks on current tasks.
The main experimental platforms are based on Multiagent Particle Environment (MPE) and
The
StarCraft Multi-Agent Challenge (SMAC) , with extensions to ensure the requirement of this study are met.
Our experimental results clearly show that role diversity can serve as a robust description for the characteristics of a multi-agent cooperation task and help explain the question of why the performance of different MARL training strategies is unstable according to this description.
In addition, role diversity can help to ﬁnd a better training strategy and increase performance in cooperative MARL.
I NTRODUCTION
Recently, multi-agent reinforcement learning (MARL) has captured people’s attention due to its impressive achievements in the ﬁeld of super human-level intelligence in video games [3, 6, 52, 57], card games [7, 25, 44, 59], and real-world applications [62–64].
These achievements have beneﬁted substantially from the success of single-agent reinforcement learning (RL) [14, 15, 30, 42, 43] and rapid progress of MARL from both the competitive and the cooperative side.
On the competitive side, existing works focus on game theory among adversary agents with guaranteed policy convergence via theoretical analysis [5, 17, 27, 29, 56].
Whereas the achievements on the cooperative MARL are more based on empirical results in cooperative multi-agent system (MAS) [4, 21, 28, 39, 46, 51, 61].
One key problem of cooperative MARL is that whether one algorithm is better than another depends on the MARL tasks as showed in Fig.
1a.
Current researches focus on developing algorithms on the tasks they are good at but lack the study of why the performance declines on other tasks [16, 35, 55, 55, 58].
Even adopting the state-of-the-art algorithms does not guarantee a strong performance [13, 35, 48, 55, 58].
This may due to the varying characteristic (e.g agent’s attributes and goals) of MARL tasks and scenarios, one single algorithm is not able to cover them all, which means we have to change the training strategy according to the current scenario. From this perspective, we need to ﬁnd a metric to describe different MARL tasks and use this description to help determine the best strategy combination as showed in Fig. 1b. Considering that
MARL Algorithm
Communication
Parameter Sharing
Credit Assignment
Task Task
Task n
MARL Algorithm
Performance Performance
Performance …
Role Diversity
Communication
Parameter Sharing
Credit Assignment
Task Task
Task n …
Role Diversity Role Diversity
Algorithm for Task Algorithm for Task
Algorithm for Task n … (a)
MARL Algorithm
Communication
Parameter Sharing
Credit Assignment
Task Task
Task n
MARL Algorithm
Performance Performance
Performance …
Role Diversity
Communication
Parameter Sharing
Credit Assignment
Task Task
Task n …
Role Diversity Role Diversity
Algorithm for Task Algorithm for Task
Algorithm for Task n … (
[Figure]
the main component of MAS is the agents, we propose a new deﬁnition Role in MAS to quantify the agents’ difference and use Role Diversity to describe MAS.
We then analyze how the role diversity impacts the MARL both theoretical and experimental.
For theoretical analysis, we use the decomposed estimation error of the joint action-value function Q tot in MARL to discuss how role diversity impacts the policy optimization process.
The experiment further veriﬁes the theoretical analysis that the role diversity is strongly related to the model performance and can serve as a good description of a MAS.
As shown in Fig.
1, with the role diversity description of each task, we can now avoid possible bottleneck of a MARL algorithm with the combination of different parameter sharing, communication, and credit assignment strategies.
With the deﬁnition of role diversity and the analysis of its impact on MARL, we can also explain the question of why the model performance varies across different tasks.
Role diversity are deﬁned from three aspects: policy-based, trajectory-based, and contribution-based in Sec. 3 which are measured by action distribution, observation overlap, and Q/state value diversity.
Through theoretical analysis, we ﬁnd each type of role diversity has different impact to different terms of decomposed estimation error: algorithmic error, approximation error, and statistical error (Sec. 4).
We conduct comprehensive experiments covering three main topics in MARL: parameter sharing , communication , and credit assignment in Sec. 5 and provide a set of guideline on choosing
MARL training strategy in Sec.
The main experiments are conducted on MPE [28] and SMAC [39] benchmarks, covering a variety of scenarios with different role diversity.
The impact of role diversity is evaluated on representative MARL algorithms including IQL [49], IA2C [31], VDN [48],
QMIX [35], MADDPG [28], and MAPPO [58], covering independent learning methods, centralised policy gradient methods, and value decomposition methods.
The experiment results prove that the model performance of different algorithms and training strategies is largely dependent on the role diversity.
Scenarios with large policy-based role diversity prefer no parameter sharing strategy.
Communication is not needed in scenarios with large trajectory-based role diversity.
Learnable credit assignment modules should be avoided when training on scenarios with large contributionbased role diversity.
The key contributions of this study are as follows:
First, the concepts of the role and role diversity are deﬁned to describe MARL tasks.
Second, a theoretical analysis of how role diversity impacts
MARL policy optimization with estimation error decomposition is built.
Third, role diversity is proven to be strongly related to performance variance when choosing different training strategies, including parameter sharing, communication, and credit assignment on the MARL benchmarks.
Finally, a set of guidelines for selecting a training strategy based on role diversity is provided.
R ELATED W ORK
Researches on the development of cooperative MARL algorithms are mainly in three aspects: parameter sharing, communication and credit assignment.
For parameter sharing, the common approach in cooperative MARL is to fully share the model parameters among the agents [16, 35, 48, 55].
In this way, the policy optimization can beneﬁt from the shared experience buffer with samples from all different agents, providing a higher data efﬁciency.
However, it has also been noted in recent works that parameter sharing is not always a good choice [8, 34, 50].
In some scenarios, a selective parameter sharing strategy, or even no parameter sharing, can signiﬁcantly beneﬁt agent performance and surpass the full parameter sharing.
However, the question of why different parameter sharing strategies have different impacts on different scenarios remains open.
In this study, we ﬁnd that the role diversity can serve as a strong signal for selecting the parameter sharing strategy.
The communication mechanism is an intrinsic part of the multi-agent system (MAS) framework [18, 19, 23, 28, 47].
It provides the current agent with essential information of other agents to form a better joint policy, which substantially impacts the ﬁnal performance.
In some cases, communication restrictions exist, which hinder us from freely choosing communication methods [28, 39]; in most cases, however, the communication is available and it is optional on when to communicate and how to ingest the shared information [19, 45].
We present a comprehensive study on the relationship between role diversity and information sharing via communication mechanisms and demonstrate that role diversity determines the necessity of communication.
For the credit assignment, most cooperative MARL algorithms adopt Q-learning or policy gradient as the basic policy optimization method, which is combined with an extra value decomposition module [16, 35, 48, 55] or shared critic function [13, 28, 58] to optimize the individual policy.
Some other works ﬁnd that leveraging the reward signal is unnecessary; however, optimizing the individual policy independently (independent learning, IL) can still get a strong joint policy [34, 49].
It then becomes slightly difﬁcult to decide which credit assignment method (including IL) is better as there is no single method in cooperative MARL that is robust and always outperforms others (compared to PPO [43] or SAC [14] in single-agent RL) on different tasks.
In this study, we contend that role diversity is the key factor that impacts the performance of different credit assignment strategies.
In the next section, we present the role deﬁnition from three aspects including policy-based, trajectory-based, and contribution-based, and propose the measurement of different role diversity types to describe a MARL task.
R OLE D IVERSITY
Using role to describe the characteristic of the agents in the MARL context has been proven to be effective in many recent works [8, 24, 54, 55].
However, the deﬁnition of the role concept remains largely unclear.
In work [55], the role is deﬁned as the higher-level option in the hierarchical RL framework [20].
In work [8], the role is deﬁned as the environmental impact similarity of a random policy.
These deﬁnitions are intuitive and cannot accurately describe the role difference.
In this study, we attempt to deﬁne the role in a more comprehensively way from three different aspects: policy-based, trajectory-based, and contribution-based.
More speciﬁc scenario-based illustration can be found in Fig.
7 and Fig.
With our reﬁned role, a strong relationship between role diversity and the MARL optimization process can be built and the performance variance can be further explained.
3. P OLICY -B ASED R OLE
In MARL, different agents output different actions based on its current status.
As common sense would indicate, actions taken at the same timestep can indicate different roles [55].
However, there are many exceptions.
For instance, if we have two soccer players passing a ball to each other repeatedly [21], although the action is different at each time step, the roles of these two soccer players can be very similar from the perspective of the whole soccer game.
Therefore, it is not sufﬁcient to distinguish the role difference based on a single timestep.
Instead, we contend that this difference should be deﬁned based on a period.
As this role is purely based on policy distribution π , we refer to it as a policy-based role.
Speciﬁcally, we deﬁne the policy-based role as the statistics of the actions’ frequency over a period, which is n steps backward and forward from the current timestep.
Here, n is the time interval that is half the length of the total time.
More details can be found in Fig.
8a, where we provide a real scenario from SMAC.
Policy-based role difference can be represented as follows:
[Equation]
where T represents the current timestep, n is the time interval, a is the agent index, π is the policy distribution.
We adopt symmetrical KL divergence to measure the distance of different policy-based [Figure] roles.
The total role distance of A agents can be computed as follows:
[Equation]
where d pT represents the policy-based role distance at timestep T , A represents the total agent number, and KL represents Kullback–Leibler divergence.
We also provide a case study of how policy-based role diversity varies in Fig.
2a.
From a real battle scenario (4m vs 3z) taken from SMAC, we ﬁnd three stages including Find a good position , Focus on enemy and Find each one’s best strategy .
In stage 1, agents try to ﬁnd their own best location; the role diversity is large.
In stage 2, agents focus on the same enemy target; the policies become similar and the role diversity is decreased.
In stage 3, the formation of the agents is broken up by the enemies.
Policy-based role diversity again increases as each agent is required to ﬁnd its own best strategy to deal with its current situation.
3. T RAJECTORY -B ASED R OLE
A policy-based role only considers the role diversity from the action distribution perspective.
A slight action difference may not diversify policy-based role; however, this difference can be enlarged by time, which eventually results in a de facto behavior difference.
The most signiﬁcant phenomenon caused by this is the agent’s trajectory.
Again, consider a soccer game where players repeatedly pass a ball to each other; these players may have a similar policy-based role, but their trajectories are different.
This difference is enhanced by the partial observation setting that exists in many popular cooperative MARL environments [39, 46, 61] as the partial observed input from different agents shares a less common pattern when the vision scope is smaller.
Therefore, trajectory-based role diversity is an important supplement to policy-based role diversity.
Generally speaking, we can deﬁne the trajectory-based role as the record of the agent’s movement.
However, the determination of the extent to which two trajectories differ is not a straightforward matter.
To measure this difference, we use an indirect metric called observation overlap percentage.
Using observation overlap to measure the trajectory-based role difference is: I . easy to compute,
II .
able to utilize a constant scale from 0 to 1 and III .
strongly related to observation scope, which means that two trajectories can have varying role distances.
For instance, in an A -agent multi-agent system, the trajectory difference can be computed as follows:
[Equation]
Here, d a 0 ,a 1 T is the observation overlap percentage of the total area between agent a 0 and agent aat timestep T .
An example of calculating the observation overlap percentage in SMAC [39] can be found in Appx.
C. We provide the observation overlap percentage curves in Fig.
2b to demonstrate how trajectory-based role diversity varies in different scenarios.
3. C ONTRIBUTION -B ASED R OLE
To help ensure the authenticity of the modern MAS, agents are initialized with different attributes in recently proposed MARL environments [21, 28, 39, 46].
For instance, in [21], the roles of forward and goalkeeper are quite different and characterized by different observation scopes, action spaces, and reward functions.
The type differences are easy to notice but hard to deﬁne, as is the role distance between them.
Here, we use an indirect variable called contribution to measuring the type difference.
In cooperative
MARL, the ﬁnal target is to obtain an optimal joint policy consisting of A individual policies.
To do this, the credit assignment strategy is proposed in [13, 35, 48] to leverage the reward signals to each agent to achieve individual policy optimization.
A good credit assignment strategy should be able to leverage the reward signal in a manner equal to each agent’s contribution to the global reward.
In this way, the Q value (Q function in off-policy RL) or state value (critic function in on-policy RL) of each agent can be estimated based on the leveraged reward signal.
From this perspective, the Q value or the state value can be regarded as the agent’s contribution to the team.
Generally speaking, we use the Q value or state value to measure the contribution of a single agent.
Contribution-based role diversity can be computed as follows:
[Equation]
Here, v is the Q value or state value of the policy output and d v a 0 ,v a 1 T is the absolute value difference between the agents’ output.
In addition, we use max function to keep the range of contribution-based role diversity from 0 to 1.
We provide the Q value (mean & max) curves in Fig.
2c to demonstrate that the contribution-based role diversity can vary a lot in different scenarios.
More detailed role deﬁnition, connection, and application discussion can be found in Appx.
C, D and E. In this paper, all the role diversity values and curves including d pT , d trajT , and d contT come from VDN[28] and separated training with no communication for robust performance and trainingefﬁcient.
T HEORETICAL A NALYSIS
In this section, we use a simple scenario as an example to illustrate the role diversity.
Suppose each agent makes individual observations and the learning procedure of all agents is independent.
We provide ﬁnite-sample analysis for the estimation error of the joint action-value function and identify the terms corresponding to the role diversity.
We denote Q ∗ tot and Q ∗ i as the optimal joint and individual Q-function respectively and write (cid:107) · (cid:107) p,µ as the L p norm with respect to a probability measure µ. Motivated by [48] and [35], we consider a simple case:
[Equation]
where n is the number of agents and ∆ n is the ( n − 1)
-dimensional probability simplex.
Here, the credit assignment function F is a weighted sum of Q ∗ i , i ∈ [ n ] with non-negative weights.
We then study the excess risk that is the error gap between the estimated and optimal solution:
[Equation]
where w ∗ and ˆ w are the optimal and estimated weights and Q t is the output of FQI algorithm at the iteration t. We further denote Q as the space of individual Q-functions and write ω ( Q ) =
sup Q ∈Q inf Q (cid:48) ∈Q (cid:107) Q (cid:48)
− TQ (cid:107) 22 ,ν .
In Sec.
B.3, we prove that
[Equation]
where N is the sample size and T is the number of iterations.
In addition, φ µ,ν is the concentration coefﬁcient and N 0 represents the 1 /N -covering number of Q .
Please refer to Sec.
B.1 for the detailed deﬁnitions.
The ﬁrst term on the RHS of (5) reﬂects the beneﬁt of credit assignment that is strong related to the Contribution-Based Role (Sec. 3.3).
When Var n ( Q ∗ ) is non-negligible, minimizing (cid:107) w ∗ − ˆ w (cid:107) can signiﬁcantly decrease the excess risk.
The second term that involves ω ( Q ) stands for the approximation error caused by functional approximation in Q .
It depends on the concentration of the sample and the scale of the hypothetical space.
The remaining two tems are statistical error and algorithmic error.
If the sample size is sufﬁciently large and the learning time is long enough, they can be arbitrarily small.
In Sec.
B.5, we assume Q is a sparse ReLU network and TQ is a composition of H ¨ older smooth functions, and analyze the convergence rate as N, T → ∞ .
Next, we demonstrate that the variance term Var n ( Q ∗ ) is related to both the Policy-Based Role (Sec. 3.1) and the Trajectory-Based Role (Sec. 3.2).
We consider the case in which all agents share one estimated Q-function and denote the optimal share Q-function as ¯ Q ∗ .
Sec.
B.4 proves that
Err share
[Equation]
where N (cid:48) 0 is the 1 / ( nN ) -covering number of Q and
[Equation]
The ﬁrst term on the RHS of (6) stands for the bias caused by parameter sharing.
If all Q ∗ i are the same, the bias will disappear.
Therefore, the Policy-Based Role is related to this bias.
Second, the variance Var n ( ¯ Q ∗ ) here is caused by the trajectory diversity.
To reduce this term, we should should ensure that all agents have similar observations.
In addition, the Trajectory-Based Role measures the concentration of all agents’ support set.
It is therefore natural to group the highly overlapped agents into one sub-joint agent via communication mechanism.
This can be compared to the separate case in (5), where approximation error and the learning error are the same.
In Sec.
B.5, we show that the parameter sharing improves the convergence rate of the statistical error via sample pooling, while the communication decreases the convergence rate by activating more input variables.
E XPERIMENT
In this section, we mainly demonstrate how model performance varies with role diversity and how to adjust the training strategy in the context of cooperative MARL.
The experimental results show the following: 1. that the performance of different parameter sharing strategies is strongly related to the Policy-Based Role (Sec. 5.1).
that the beneﬁt brought by different communication mechanisms can be easily affected by the Trajectory-Based Role (Sec. 5.2).
that the performance of the credit assignment method, or the centralized training strategy, is largely dependent on the
Contribution-Based Role (Sec. 5.3).
that the choosing of training strategies should be determined by the scale of role-diversity for different scenarios.
The main experimental platforms are
MPE [28] and SMAC [39].
Extensions are made to fulﬁll the requirements of parameter sharing and the communication mechanism, these include separated training of policy in Sec.
5.1 and information exchange among agents in Sec.
All results come from eight random seeds.
More details regarding the experimental settings can be found in the appendix.
[Table]
P ARAMETER S HARING
Policy-based role inﬂuences the convergence speed and ﬁnal performance of different parameter sharing strategies in cooperative MARL.
The scenarios we choose from the MPE and SMAC benchmarks are simple but diverse, covering policy-based role diversity that ranges from small to large.
In table.
1 and Fig.
3, we provide policy-based role diversity and the model performance curve on these chosen scenarios.
For the SMAC benchmark, we adopt two metrics to count r uT in Eq.
The ﬁrst metric is real policy diversity , which treats each action as an independent.
The second way is semantic policy diversity , which distributes the actions to different groups according to their semantic type (e.g. move & attack).
There is no semantic policy-based role in scenarios chosen from the MPE benchmark as all the actions are of the same semantic type (move).
The policy-based role diversity is then calculated according to Eq.
The base MARL credit assignment strategy we choose in table.
1 is VDN[48] , combined with a fully/partly/no parameter sharing strategy.
For details of how the partial parameter sharing strategy works, please refer to the Appx.
F. We also evaluate other popular credit assignment strategies including IQL[49], IA2C, MADDPG[28] MAPPO[58], MAA2C and QMIX[35] combined with a fully/no parameter sharing strategy in Fig.
IA2C and MAA2C are the extension of A2C[31] on multi-agent scenarios.
Table.
1 outlines the performance of three parameter training strategies including No shared , Partly shared and Shared with the base credit assignment method VDN[48].
Detailed framework and settings can be found in Appx.
F. As the policy-based role diversity increases, the performance of the Shared strategy is degraded in terms of both convergence speed (half training steps) and the ﬁnal reward (full training steps).
One interesting phenomenon also emerges: the same agent type (e.g.
3s vs 5z, 4m vs 4z) does not always indicate small policy-based role diversity, and vice versa (e.g.
1s1m1h1M vs 3z), which means it is hard to deﬁne the role before identifying an adequate policy.
Fig.
3 shows the model performances of two parameter sharing strategies including No shared and
Shared with different credit assignment methods.
For policy gradient-based methods, we extend the training steps from the standard 2M to 20M (10 times) as the convergence speed of policy gradientbased methods (e.g. MAPPO, MAA2C) is slower than Q value-based methods.
From Fig.
3, we ﬁnd that different credit assignment methods have a slight impact on parameter sharing strategies but the trend in which no parameter sharing strategy achieves performance improvement continues to be present as the policy-based role diversity increases.
Here we conclude that, scenarios with large policy-based role diversity prefer no parameter sharing strategy, and vice versa.
Suitable parameter sharing strategy helps obtain faster convergence speed and higher ﬁnal performance.
5. C OMMUNICATION
More information is better.
This principle is common sense in many areas including computer vision or natural language processing.
With a bigger dataset and more detailed and accurate annotations, the model can be better optimized.
However, in reinforcement learning, the data is sampled using the current policy; in most cases, this policy learning starts with a randomly initialized neural network.
In this way, the provision of more information may introduce a burden for the policy optimization and can moreover further degrade the sampled data quality, which gives rise to a vicious
T e s t O n e E p
i s od
e R e w a r d
Step (a) 3s vs 5z
T e s t O n e E p
i s od
e R e w a r d
Step (b) 4m vs 4z
T e s t O n e E p
i s od
e R e w a r d
Step (c) 1c1s1z vs 1c1s3z
T e s t O n e E p
i s od
e R e w a r d
Step (d) 4m vs 5m
T e s t O n e E p
i s od
e R e w a r d
[Figure]
T e s t O n e E p
i s od
e R e w a r d
[Figure]
(f)
[Figure]
T e s t O n e E p
i s od
e R e w a r d
Step (
[Figure]
) 1c1s1z vs 1c1s3z
T e s t O n e E p
i s od
e R e w a r d
Step (h) 4m vs 5m
[Figure]
[Figure] [Figure]
Detailed analysis in circle.
Therefore, it is critical to determine when and how to accept the extra information provided via communication mechanism in the context of cooperative MARL.
As discussed in Sec. 4, the pattern of different agents’ support sets for policy optimization can determine whether or not the extra information is needed.
Notably, the similarity of these support sets is largely dependent on the trajectories of different agents, corresponding to the trajectory-based role diversity deﬁned in
Sec. 3.2.
Small trajectory-based role diversity corresponds to a similar support set pattern, which means that forming a concentrated input is preferred for policy optimization.
Experiment results in table.
7 and Fig.
4 further prove that scenarios with larger observation overlap are more suitable for communication.
Detailed setting of communication mechanism on SMAC can be found in Appx.
G
To prove that small trajectory-based role diversity prefers obtaining extra information via communication and vice versa, we conduct extra experiments to study the relationship between the pattern of the input observation (support set) and the model performance by shrinking the vision scope ( r in eq. 14).
The results can be found in table 2.
The results show that the model performance is strongly related to the vision scope and the trajectory-based role diversity determines whether the small or large vision is preferred.
Small trajectory-based role diversity prefers large vision scope, indicating the similar pattern of support set is better.
Large trajectory-based role diversity prefers small vision scope, which means enlarging the pattern difference beneﬁts the policy optimization.
This further prove that extra information provided by communication forms a similar pattern of support set which is preferred in scenarios with small trajectory-based role diversity.
5. C REDIT A SSIGNMENT
The performance of different credit assignment methods is strongly related to the contributionbased role.
We mainly focus on three representative Q value-based MARL algorithm: VDN[48],
QMIX[35] and IQL[49], and compare their performance on different scenarios that have different contribution-based role diversity measured by Q value according to Eq. 4.
The result can be found
[Table] [Table]
/17.9/19.6 1s1m1h1M vs 4z 8.4/16.0/19.8 (cid:88) 4.9/5.1/6.
8.1/13.5/18.2 (cid:88) 5.5/5.0/5.
7.1/8.5/8.5 1s1m1h1M vs 5z > 0. 6.3/12.6/15.3 (cid:88) 4.2/4.2/3.
6.2/9.9/12.3 (cid:88) 4.0/2.5/4.
in table.
In small Q diversity ( (cid:39) 0 . 5 ) scenarios, QMIX signiﬁcantly outperforms VDN with both shared and no shared strategies.
With the increase of Q diversity, the performance of QMIX starts to degrade.
In scenarios where agents have signiﬁcantly different Q value distribution (Fig.1s1m1h1M vs 3/4/5z), VDN signiﬁcantly outperforms QMIX.
As for IQL, the performance is not as good as VDN and QMIX in most scenarios.
However, IQL is not sensitive to Q diversity and can perform well in easy scenarios like 1s1m1h1M vs 3z.
Combined with theoretical analysis in
Sec. 4, we can conclude that QMIX is not suitable for a large contribution-based role diversity scenario because of the additional value decomposition module, which is a sum function in VDN and a learnable neural network in QMIX.
The neural network fails to minimize the approximation error for Q tot , and is an extra burden when the reward function (or the contribution to the global reward) is diverse.
IQL has no such problem as it treats Q tot as the individual Q value.
From this part, we conclude that using credit assignment methods with learnable value decomposition module should be avoided in scenarios with large contribution-based role diversity.
F INDING A BETTER TRAINING STRATEGY
Based on our theoretical and experimental analysis, we contend that role diversity is a strong candidate metric to aid in selecting the proper training strategy for cooperative MARL.
Speciﬁcally, if the policy-based role diversity is large, we should choose a no parameter sharing strategy, and vice versa.
If the trajectory-based role diversity is large, we should avoid communication or other unnecessary information sharing, and vice versa.
If the contribution-based role diversity is large, the ﬁxed credit assignment method or independent learning method is preferred, and vice versa.
In this way, we can avoid the possible performance bottlenecks in the context of cooperative MARL.
Notably, however, the question remains as to how we might obtain an accurate measurement of the role diversity before the training.
In this study, all the role diversity comes from VDN and separated training with no communication because we require the baseline method in our study to be robust and training-efﬁcient.
The fact however remains that, in some cases, roles are different in ways that are easy to notice and require no trained policy.
C ONCLUSION
In this paper, we deﬁne the role and role diversity to describe a cooperative MARL task and explain the question of why the model performance varies in different scenarios.
We claim that a strong relationship between the role diversity and model performance exists and we prove it through both theoretical analyses on MARL error bound decomposition and experiments conducted on MARL benchmarks.
The experiment results clearly show that the role diversity signiﬁcantly impacts the model performance of different training strategies and this effect is ubiquitous in various environments and algorithms.
Finally, we provide a guideline on choosing the proper training strategies for cooperative MARL based on the role diversity description.
A P ROBLEM F ORMULATION
Multi-agent Reinforcement Learning A cooperative multi-agent task is a decentralized partially observable Markov decision process [33] with a tuple G =
(cid:104)S , A , U , P, r, Z , O, n, γ (cid:105) .
Let S denote the global state of the environment, while A represents the set of n agents and U is the action space.
At each time step t , agent a ∈
A ≡ { 1 , ..., n } selects an action u ∈ U , forming a joint action u ∈ U n , which in turn causes a transition in the environment represented by the state transition function
P ( s (cid:48) | s , u ) :
S × U n × S → [0 , 1] .
All agents share the same reward function r ( s , u ) : S × U n → R , while γ ∈ [0 , 1) is a discount factor.
For any state-action pair, the reward r is bounded by M , i.e. | r | ≤ M.
We consider a partially observable scenario in which each agent makes individual observations z ∈ Z according to the observation function O ( s , a ) : S ×
A → Z .
Each agent has an action-observation history that conditions a stochastic policy π t , creating the following joint action value: Q π ( z t , u t ) =
E s t +1: ∞ , u t +1: ∞
[ R t | z t , u t ] , where R t = (cid:80) ∞
i =
γ i r t
+ i is the discounted return.
Centralized training with decentralized execution Centralized training with decentralized execution (CTDE) is a commonly used architecture in the MARL context.
Each agent is conditioned only on its own action-observation history to make a decision using the learned policy.
The centralized value function provides a centralized gradient to update the individual function based on its output.
Therefore, a stronger individual value function can beneﬁt the centralized training.
B P ROOFS
In this section, we present more detailed results and the proofs for the theoretical analysis in Sec. 4.
In Sec.
B.1, we denote more notations and state the concentration property of Markov decision process.
Sec.
B.2 presents two useful lemmas about the error propagation and one-step approximation respectively.
In Sec.
B.3, we consider a simple example of the decentralized and cooperative MARL and provide the ﬁnite-sample analysis for the estimation error of the joint action-value function.
We use the value decomposition [35, 48] and the ﬁnite-sample results for single-agent RL [10].
For more related results about MARL, please refer to [60] and [53].
Sec.
B.4 studies the parameter sharing case that all agents share one deep Q network.
In Sec.
B.5, we assume Q is a sparse ReLU network and TQ is a composition of H ¨ older smooth functions.
Then we discuss the convergence rate of the statistical error as the sample size tends to inﬁnity.
According to Sec.
B.3, B.4 and B.5, one can ﬁnd that each type of role diversity have different impact to the decomposed estimation error.
Furthermore, we explain the beneﬁts of the training options, e.g. parameter sharing (Sec. 5.1), communication (Sec. 5.2) and credit assignment (Sec. 5.3), and discuss how these options impact the convergence rate of approximation error and statistical error.
We summarize our results as follows:
• The parameter sharing strategy introduces a bias term by constraining the diversity of individual action-value functions, which corresponds to the policy-based role diversity.
At the same time, it speeds up the convergence rate of statistical error by pooling training data.
• The communication mechanism reduces the variance caused by the trajectory-based role diversity but slows down the convergence rate of approximation error by introducing more active input variables.
• When the contribution-based role diversity is nonnegligible, the credit assignment can signiﬁcantly reduce the estimation error of the action-value function.
B. M ORE N OTATIONS AND A SSUMPTIONS
We denote the joint optimal action-value function by
[Equation]
where z is the global state of the environment, u = { u 1 , . .
. , u n } is the action set that collects the action of each agent u i and z i is the observation of the agent i generated from the emission distribution
z i ∼ Λ( z | i, s ) .
We further denote the individual optimal action-value function by Q ∗ i and write
[Equation]
as the vector of all agents’ action-value functions.
According to the value decomposition assumption, the joint optimal Q-function Q ∗ tot can be approximated with
[Equation]
where F ∈ F is a credit assignment function.
The VDN method [48] approximates the joint as a sum of individual action-value functions that condition only on individual observations and actions.
Then a decentralised policy arises simply from each agent selecting actions greedily with respect to its Q i .
Since n is a ﬁxed integer, we write the hypothetical space of credit assign functions that only contains one function as:
[Equation]
The QMIX method [35] generalizes the value decomposition scheme and prove that if
[Equation]
then the global arg max performed on joint Q-function yields the same result as a set of individual arg max operations performed on each agent Q-function, that is
[Equation]
Motivated by VDN and QMIX, we consider a simple case throughout this section:
[Equation] [Equation]
, where ∆ n is the n − 1 dimensional probability simplex.
Suppose the individual action-value function is estimated by the ﬁtted-Q iteration (FQI) algorithm [9, 38].
At the iteration 0 ≤ t ≤ T , we write ˜ Q i,t and π i,t as the output of FQI algorithm and the corresponding greedy policy respectively.
Let Q π i,t be the Q-function corresponding to π i,t .
Then the joint action-value function is estimated by ˆ F ( Q t ) , where
[Equation]
To proceed further, we give the following assumption that controls the similarity between two probability distributions under the Markov decision process.
Assumption 1 .
Let µ, ν ∈ P ( Z × U ) be two probability measures that are absolutely continuous with respect to the Lebesgue measure on Z × U .
Let { π t } be a sequence of joint policies for all the agents, with π t : Z → P ( U ) for all time t. Suppose the initial state-action pair ( z 0 , u 0 ) has distribution µ , and the action u t is sampled from the joint policy π t .
For any integer m , we denote by P π m P π m − 1 · · · P π 1 µ the distribution of { ( z t , u t ) } mt =1 under the policy sequence { π t } t =1 ,...,m .
Then, the m -th concentration coefﬁcient is deﬁned as
[Equation]
where d ( P π m P π m − 1 · · · P π 1 µ ) /dν is the Radon-Nikodym derivative of P
π m P π m − 1 · · · P π 1 µ with respect to ν and the supremum is taken over all possible policies.
Furthermore, let ν be the stationary distribution of the samples { ( z t , u t ) } from the Markov decision process and let µ be a ﬁxed distribution on S × U .
We assume that there exists a constant φ µ,ν such that
[Equation]
To proceed further, we denote Q as the space of individual Q-functions and let
[Equation]
where (cid:107) · (cid:107) 2 ,ν as the L 2 norm with respect to a probability measure ν.
In the following, we take ν as the independent data sampling distribution in the FQI algorithm, e.g. experience replay [26].
We say a collection { Q 1 , . . .
, Q K } ⊆ Q is an δ -cover of Q if for each Q ∈ Q , there exists Q k such that (cid:107) Q − Q k (cid:107) ≤ δ .
The δ -covering number of Q with respect to (cid:107) · (cid:107) is
N ( Q , δ, (cid:107) · (cid:107) ) :
= inf { K ∈ N : there is an δ -cover of Q with respect to (cid:107) · (cid:107)} .
In the following, we take the L ∞ ( Z × U ) norm on Q by
[Equation]
For the sake of simplicity, we rewrite N ( Q , δ, (cid:107)·(cid:107) ) as N δ .
In Sec.
B.3, we study the estimation error of the joint action-value function and prove that the statistical error depends on ln N 0 /N , where Nis the 1 /N -cover of Q and N is the sample size.
For the parameter sharing settings, the sample size increases while the cover number also increases due to the smaller δ.
Thus, we still do not know whether the parameter sharing improves the convergence rate of the statistical error.
So we present a ﬁne-grain analysis to discuss the convergence rate in Sec B.5.
B. U SEFUL L EMMAS
Lemma 1.
(Theorem 6.1 in [10]).
For each agent i ∈ [ n ] , we denote { ˜ Q
i,t } 0 ≤ t ≤ T as the iterates of FQI Algorithm.
Let π i,t be the one-step greedy policy with respect to ˜ Q i,t , and let Q π i,t be the action-value function corresponding to π i,t .
Under Assumption 1 , we have
[Equation]
Proof: Please see Appx C.1 of [10] for a complete proof.
[Equation]
This lemma quantiﬁes the error propagation procedure of each agent action-value functions through each iteration of FQI Algorithm.
The ﬁrst term on the RHS is the one-step statistical error and will not vanish even when the iteration goes to inﬁnity ( T → ∞ ).
For more related error propagation results, please refer to [11, 12, 22, 32, 40].
Lemma 2.
(Theorem 6.2 in [10]).
Let { ( z ij , u ij ) } j ∈ [ N ] be N i.i.d.
random variables.
For each j ∈ [ N ] , let r ij and z (cid:48) ij be the reward and the next state corresponding to ( z ij , u ij ) .
In addition, for any ﬁxed ˜ Q i,t − 1 ∈ Q , we deﬁne y ij = r ij + γ · max u ˜ Q i,t − 1 ( z (cid:48) ij , u ) .
Based on { ( z ij , u ij , y ij ) } j ∈ [ N ] , we deﬁne ˜ Q
i,t as
[Equation]
Then for any δ > 0 , we have
[Equation]
where ω ( Q ) =
sup Q ∈Q inf Q (cid:48) ∈Q (cid:107) Q (cid:48)
− TQ (cid:107) 2 ν
and N δ is the δ -covering number of Q with respect to the norm (cid:107) · (cid:107) ∞ .
Proof: Please see Appx C.2 of [10] for a complete proof.
B. I NDIVIDUAL Q-F UNCTION
Theorem 1 .
We consider the separated strategy that each agent has its own action-value function and reward.
All agents’ learning process is independent.
Suppose { ˜ Q
i,t } 0 ≤ t ≤ T are the output of
FQI Algorithm for the agent i. Let π i,t be the one-step greedy policy with respect to ˜ Q i,t , and let
Q π i,t be the action-value function corresponding to π i,t .
We rewrite
[Equation]
and
[Equation]
Recall that 0 ≤ γ < 1 is the discount factor, the reward function is bounded, i.e., | r ( s , u ) | ≤ M , Q is the space of individual Q-functions and ω ( Q ) =
sup Q ∈Q inf Q (cid:48) ∈Q (cid:107) Q (cid:48)
− TQ (cid:107) 22 ,ν .
Then, under
Assumption 1 , we have
[Equation]
where N 0 is the 1 /N
-covering number of Q with respect to the norm
[Equation] [Equation]
∞ and
[Equation]
Proof : It is easy to see that
[Equation]
Here the ﬁrst term at the RHS of (9):
[Equation]
represents the best achievable estimation error under the value decomposition assumption.
Next we consider the second term in the inequality (9):
[Equation]
For any given z = ( z 1 , . . . , z n ) and u = ( u 1 , . . . , u n ) ,
[Equation]
= n
[Equation]
=
[Equation]
Q ∗
[Equation]
( z
[Equation]
u
[Equation]
) (cid:33) .
The second equality holds since (cid:80) ni =1 ( w ∗ i − ˆ w i )
× c = 0 for any constant c. By the Cauchy–Schwarz inequality, we have
[Equation]
where
[Equation]
is the variance of the output vector of Q ∗ given z and u .
Plugging the positive upper boundary of ( w
[Equation]
− ˆ w ) (cid:62) Q ∗ into the expression of (cid:107) F
[Equation]
1 ,µ , we obtain that
[Equation]
Here the term (cid:107) (cid:113)
Var n (cid:0) Q ∗ ( z , u ) (cid:1) (cid:107) 1 ,µ stands for the diversity of the agents.
Finally, we deal with the third term on the RHS of (9).
Notice that
[Equation]
Therefore, it sufﬁces to consider the deep Q-learning procedure of each agent separately and to give upper bound of (cid:107) Q ∗ i − Q π t
i (cid:107) 1 ,µ for each i ∈ [ n ] .
According to (7) and (8),
[Equation]
We take δ = 1 /N and write N 0 as the 1 /N -covering number of Q
[Equation]
Then
[Equation]
we have
[Equation]
Furthermore,
[Equation]
Combining the results of (10), (11) and (12), we know
[Equation]
Remark.
The term
[Equation]
shows the beneﬁts of learning credit assignment, where w ∗ stands for the best credit assignment scheme.
Here we assume ˆ w is given and do not take the modelling and learning of ˆ w into consideration.
In practice, ˆ w is the output of a credit distribution network and its learning procedure also inﬂuence the convergence properties of individual Q-functions.
On the other hand, Var n ( Q ) corresponds to the contribution-based role diversity in Sec. 3.3.
Therefore, when the variance is nonzero, a good credit assignment ˆ w can the estimation error.
For the parameter sharing case in the next section, we decompose the variance term into the sum of a bias and a variance caused by policy-based role diversity and the trajectory-based role diversity respectively.
This decomposition does not always hold.
Here, we assume that all agents’ learning processes are independent and that each agent has its own reward function.
In practice, these assumptions are idealistic and limited.
B. S HARED Q-F UNCTION
Theorem 2 .
We consider the parameter sharing strategy that all individual agents shares one actionvalue function.
Suppose { ˜ Q t } 0 ≤ t ≤ T are the output of FQI Algorithm.
Let π t be the one-step greedy policy with respect to ˜ Q t , and let Q π t be the action-value function corresponding to π t .
We further denote ¯ Q ∗ as the optimal shared action-value function and write
[Equation]
(cid:1)
[Equation]
Recall that 0 ≤ γ < 1 is the discount factor, the reward function is bounded, i.e., | r ( s , u ) | ≤ M , Q is the space of individual Q-functions and ω ( Q ) =
sup Q ∈Q inf Q
[Equation]
∈Q
[Equation]
Q
[Equation]
− TQ (cid:107) 22 ,ν .
Then, under
Assumption 1 , we have
[Equation]
where N (cid:48) 0 is the 1 / ( nN ) -covering number of Q with respect to the norm (cid:107) · (cid:107) ∞ and
[Equation]
Proof : Similar to the arguments in (9), we have
[Equation]
The term (cid:107) Q ∗ tot − F ∗ ( Q ∗ ) (cid:107) 1 ,µ caused by the value decomposition is the same to that in (9).
So (10) still holds.
For the second term on the RHS of (13),
[Equation]
F
[Equation]
,µ
[Equation]
µ
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
, which is the bias term caused by the parameter sharing.
Next, we turn to a turn that is related to the trajectory-based role diversity.
Similar to (11), we know
[Equation]
On the other hand, by (7) and (8), (cid:107)
[Equation]
We take δ = 1 / ( nN ) and write N (cid:48)
as the 1 / ( nN ) -covering number of Q .
Then, we have
[Equation]
Therefore,
[Equation]
Summarizing the above results
[Equation]
we have
[Equation]
B. C ONVERGENCE R ATE
Similar to the Theorem 4.4 in [10]
[Equation]
we assume that Q belongs to a family of sparse ReLU networks and TQ can be written as compositions of H ¨ older smooth functions.
Here T is the optimal Bellman operator.
We start with the deﬁnition of a ( L + 1) layers and { d j } L +1 j =1 width ReLU networks:
[Equation]
, where σ is the ReLU activation function
[Equation]
and W l and v l are the weight matrix and the bias in the l -th layer, respectively.
The family of sparse ReLU networks is deﬁned as
[Equation]
where ˜ W l is the parameter matrix that contains W l and v l and f j is the j -th output of f.
On the other hand, the set of H ¨ older smooth functions is
[Equation]
β −(cid:98)
β
[Equation]
∞ ≤ H    , where D is a a compact subset of R r , (cid:98)·(cid:99) stands for the ﬂoor function and ∂ α = ∂ α 1 ∂ α 2 · · · ∂ α r with α
= ( α 1 , α 2 , . . .
, α n ) (cid:62) .
Furthermore, we write G ( { p j , t j , β j , H j } j ∈ [ q ] ) as the family of functions that can be decomposed into the composition of a sequence of H ¨ older smooth functions { g j } j ∈ [ q ] .
That is, for any function f ∈ G ( { p j , t j , β j , H j } j ∈ [ q ] ) ,
[Equation]
where for any k ∈ [ p j +1 ] and j ∈ [ q ] , the k -th component of the function g j is a H ¨ older smooth function, i.e., g jk ∈ C t j ([ a j , b j ] t j , β j , H j ) .
For simplicity, we take p j +1 = 1 .
Here we assume that the input of g jk is t j -dimensional, where t j can be much smaller than p j .
More speciﬁc, the deep Q network we used is a sparse ReLU network for any given action u .
Therefore, we rewrite the space of individual Q-functions Q as
[Equation]
any
[Equation]
Furthermore, for any Q ∈ F 0 , we assume TQ is a composition of H ¨ older smooth functions and belongs to the following family:
[Equation]
To proceed further, we denote
[Equation]
and β ∗
[Equation]
= 1 .
Now we are ready to state the following result.
Theorem 3 .
Suppose the assumptions of Theorem 1 hold and for any Q ∈ F 0 , TQ ∈ G 0 , where
T is the optimal Bellman operator.
The sample size N is sufﬁciently large such that there exists a constant ξ > 0 satisﬁes
[Equation]
The network architecture of the Q-function is well designed such that
[Equation]
and s (cid:16) N α
[Equation]
(ln N ) ξ ∗ for some constant ξ ∗ > 1 + 2 ξ.
The number of iterations T is sufﬁciently large, such that
[Equation]
where C (cid:48) is a constant.
Then, under Assumption 1 , we have
[Equation]
Proof: This is a direct conclusion reached by Theorem 4.4 in [10].
That is, for any agent i ∈ [ n ] ,
[Equation]
The approximation error in Theorem 1 that involves ω ( Q ) is bounded above via Theorem 5 in [41].
The upper bound for the cover number N 0 is derived from Theorem 14.5 in [2].
Please refer to
Section 6 in [10] and Sec.
B.3 for a complete proof.
[Equation]
Remark: Note that
[Equation]
which is the algorithmic error that converges to zero at a linear rate of T. In Theorem 3, we assume
T is sufﬁciently large such that this error is negligible comparing to the statistical error.
If we ignore the logarithmic term, the convergence rate of the statistical error is about
[Equation]
Here β ∗ j and t j are parameters of the functional space of TQ .
Therefore, the parameter sharing (Sec. 5.1) keeps β ∗ j and t j unchanged and increases the sample size N to nN by pooling training data.
In addition, t j is the number of active input variables of g j .
Thus, the communication mechanism (Sec. 5.2) slows down the convergence rate by enlarging t j .
C O BSERVATION O VERLAP P ERCENTAGE C ALCULATION C. O VERLAP P ERCENTAGE C ALCULATION IN G AMES
In this part, we demonstrate how to calculate the observation overlap percentage in SMAC [39].
As the partial observable area is circular and the coordinate system is a 2D map with axis X and Y, the observation overlap on one battle scenario can be computed as:
[Equation]
Here r is the vision scope.
Notice that if l < 2 r , d T equals zero as no overlap exists.
We provide the observation overlap curve in Fig.
2b to show how trajectory-based role distance varies in one game.
C. O VERLAP P ERCENTAGE C ALCULATION IN R EAL W ORLD S CENARIO (S EMANTIC )
[Table] [Figure] In this part, we demonstrate how to apply the observation overlap concept and trajectory-based role diversity calculation (Eq. 3) to real-world scenarios.
Different from game scenarios like SMAC and
MPE, the observation of real-world tasks is usually an image.
For example, in the vision language navigation task (VLN[1]), agents take real indoor scene pictures as the input, combine them with language description to locate the target as shown in Fig. 5.
Considering the learning efﬁciency, object detection techniques like YOLO[36] and FasterRCNN[37] are used in VLN to help extract the objects from the scene pictures as semantic information.
The semantic information can be recorded as part of the agents’ trajectory, enabling agents to use the past information for future decisions.
Under the multi-agent setting, agents are required to cooperate and ﬁnd the target together.
Therefore, trajectory overlap should be avoided, which means that large trajectory-based role diversity is preferred in this task, and policies that cause trajectory overlap should be punished.
Directly using scene pictures as input or its feature pattern can bring large noise in observation overlap calculation.
Instead, using semantic information from the detected object can signiﬁcantly reduce the noise and serve as a good observation history representation.
As shown in Fig.
5, the red dotted frames indicate that the blue agent and green agent share some similar observation semantic in their trajectories.
In this way, the trajectory-based role diversity of multi-agent VLN task can be calculated the same as
Eq. 3.
Only the d a 0 ,a 1 T is replaced by the observation semantic overlap, which is the shared detected object percentage in total detected objects.
In this way, without knowing the exact trajectory, we still manage to calculate the trajectory distance.
And using overlap to represent the trajectory-based role diversity, we can keep this metric in a ﬁxed range from 0 to 1.
C.
O VERLAP P
IN
W
Pretrained Backbone Observation Agent
Feature vector .
x
Vector wise
Channel wise
{+Overlap calculation (2 ways) … 
+}
Threshold
Overlap
N 0 and M=
Overlap
M/(N+M)
=
cosine similarity
It is also possible that we can get observation overlap directly based on real image MARL tasks.
As showed in Fig.
Passing the input image to pre-trained CNN/Transformer backbone and getting its feature, we can use cosine similarity or channel-wise similarity to compute the overlap between different observation features as d a 0 ,a 1 T in Eq. 3.
However, these methods can bring large noise to this metric.
Moreover, how to stabilize the reinforcement learning with real pictures as input is still under investigation.
In addition, it is rare in MARL tasks that the only information provided in the training stage is one single image.
Location and communication are necessary auxiliary information to help learn the coordination of agents in most MARL tasks.
Therefore, simply using the raw image to calculate the observation overlap can be a choice, but not the best choice.
We present two illustration ﬁgures for different types of role based on MPE [28] and SMAC [39].
Fig. 7 is based on MPE.
Grey circles and black circles represent agents and goals respectively.
Dashed arrows in different colors represent different actions.
Larger circles receive more rewards when they reach the goal.
[Figure]
Fig. 8 is based on SMAC.
A detailed explanation can be found in the caption.
E C ONNECTIONS OF D IFFERENT R OLES
Is there any redundancy in the deﬁnition of different kinds of role diversity in Sec. 3?
Here we discuss the connections of different role diversity.
From the theoretical perspective, the contribution-based role diversity is a compound description of role diversity.
It corresponds to the variance term in (5).
For the parameter sharing case, we decompose this variance into a sum of two terms: a bias term corresponds to the policy-based role
[Figure]
[Figure]
[Figure]
Types of role difference
[Figure]
[Figure]
[Table]
diversity and a variance term corresponds to the trajectory-based role diversity.
Therefore, under the simple scenario in Sec. 4, we can ﬁnd a clear relationship between different role diversity.
From the experiment perspective, the decomposition in (6) may not hold because of the more complicated settings.
We have discussed this issue in the remark on Page 18.
Here we collect all different role diversity data in Table.
We ﬁnd scenarios like 3s vs 5z have relatively small diversity in the policy-based role while the observation overlap of trajectory diversity is small.
Scenarios like 4m vs 5m have small policy-based role diversity while the observation overlap is large.
Contributebased role diversity can not be inferred from the policy diversity and trajectory diversity, and is more depending on the agents’ behavior difference.
In conclusion, the relationship between different roles exists in MARL training theoretically, while this relationship is not so signiﬁcant in the experimental perspective due to more complicated settings of the real MARL tasks.
Yet the strong relation between different role diversity and the MARL training process still exists with no conﬂict with the conclusion of this paper.
F P ARAMETER S HARING
Four different parameter sharing strategies are tested in our experiment including shared, no shared, partly shared, and selectively shared[8].
For partly shared, we only shared the GRU cell across different agents while keeping the embedding layer of the policy function model separated for each agent.
For selectively shared strategy, we reproduce the grouping results following[8].
An illustration ﬁgure can be found in Fig. [Figure]
F. S ELECTIVELY S HARING THE P ARAMETER
Here we provide the selective parameter sharing strategy result in Table.
5 as a supplement for Table.
The main purpose for doing so is to verify whether this method can serve as a general solution for parameter sharing strategy choosing issue.
Selective parameter sharing strategy partitions the agents into the different groups automatically with an encoder-decoder model.
However, the partition process is before the MARL training stage, which can not fully catch the policy difference.
And according to the grouping result column in Table.
5, the selective parameter sharing strategy tends to divide agents by their initial attributes.
This works well in scenarios like 1s1m1h1M vs 4z and 1c1s1z vs 1c1s3z but ignores the fact that the same type of agents may evolve to different functions during MARL training, which is the weakness of the selective parameter sharing strategy.
[Figure]
[Table]
G C OMMUNICATION F RAMEWORK
The communication mechanism is important for MARL.
The information shared can be location, action, and partial observation as showed in Fig. 10a.
In many cases, communication is optional where the agent should learn when to communicate and how to ingest the information (dash line in
Fig. 10b).
In our experiment, we only consider the observation sharing method where the support set of policy functions contains both self partial observation and the aggregated observation information from other agents.
The aggregated information is obtained by getting the mean value of other agents’ observation and concatenate with the self partial observation.
This means the support set of policy functions is now much similar to the global state.
[Figure]
H C REDIT A SSIGNMENT
Credit assignment is the key part module for cooperative MARL, especially for the value decomposition-based method as it leverages the reward signal to each agent by approximate the
Q tot .
Then the learned individual policies combine to form a joint policy interacting with the MAS [Figure]
I E XPERIMENT T ABLE & C URVE
[Figure]
[Table]
SMAC 4m vs 4z shared 2.6/3.2/3.
[Table]
od w
Step (a) 3s vs 5z
T e s t O n e
E p i s od
e R e w a r d
Step (b) 4m vs 3z
T e s t O n e E p
i s od
e R e w a r d
Step (c) 4m vs 4z
T e s t O n e
[Table]
p i s od
e R e w a r d
Step (d
[Table]
e
E p s od e R e w a r d
Step (e) 1c1s1z vs 1c1s3z
T e s t O n e
E p i s od
e R e w a r d
Step (f) 1s1m1h1M vs 3z
T e s t O n e E p
i s od
e R e w a r d
Step (g) 1s1m1h1M vs 4z T e s t O n e E p i s od e R e w a r d
Step (
[Figure]
) 1s1m1h1M vs 5z
T e s t O n e E p
i s od
e R e w a r d
Step
[Figure]
T e s t O n e E p
i s od
e R e w a r d
Step (
[Figure]
) 4m vs 3z
T e s t O n e E p
i s od
e R e w a r d
Step (c) 4m vs 4z
T e s t O n e
E p i s od
e R e w a r d
Step (
[Figure]
) 4m vs 5m
T e s t O n e
E p i s od
e R e w a r d
Step (e) 1c1s1z vs 1c1s3z
T e s t O n e
E p i s od
e R e w a r d
Step (
[Figure]
) 1s1m1h1M vs 3z
T e s t O n e E p
i s od
e R e w a r d
Step (g) 1s1m1h1M vs 4z
T e s t O n e
E p i s od
e R e w a r d
Step (
[Figure]
T e s t O n e
E p i s od e
R e w a r d
Step
[Figure]
5z
T e s t O n e
E p i s od e
R e w a r d
Step (
[Figure]
4m
[Figure]
5m
T e s t O n e
E p i s od e
R e w a r d
Step
[Figure]
1c1s1z
[Figure]
T e s t O n e
E p i s od e
R e w a r d
[Figure]
T e s t O n e
E p i s
[Figure]
e
R e w a r d
[Figure]
T e s t O n e
E p i s od e
R e w a r d
[Figure]
[Figure] [Figure]
[Figure] [Figure]