B OOSTING M ULTIAGENT R EINFORCEMENT L EARNING VIA P ERMUTATION I NVARIANT AND P ERMUTATION E QUIVARIANT N ETWORKS
A BSTRACT
The state space in Multiagent Reinforcement Learning (MARL) grows exponentially with the agent number.
Such a curse of dimensionality results in poor scalability and low sample efficiency, inhibiting MARL for decades.
To break this curse, we propose a unified agent permutation framework that exploits the permutation invariance (PI) and permutation equivariance (PE) inductive biases to reduce the multiagent state space.
Our insight is that permuting the order of entities in the factored multiagent state space does not change the information.
Specifically, we propose two novel implementations: a Dynamic Permutation Network (DPN) and a Hyper Policy Network (HPN).
The core idea is to build separate entity-wise PI input and PE output network modules to connect the entity-factored state space and action space in an end-to-end way.
DPN achieves such connections by two separate module selection networks, which consistently assign the same input module to the same input entity (guarantee PI) and assign the same output module to the same entity-related output (guarantee PE).
To enhance the representation capability, HPN replaces the module selection networks of DPN with hypernetworks to directly generate the corresponding module weights.
Extensive experiments in SMAC, SMACv2, Google Research Football, and MPE validate that the proposed methods significantly boost the performance and the learning efficiency of existing MARL algorithms.
Remarkably, in SMAC, we achieve 100% win rates in almost all hard and super-hard scenarios (never achieved before
I NTRODUCTION
Multiagent Reinforcement Learning (MARL) has successfully addressed many real-world problems (Vinyals et al., 2019; Berner et al., 2019;
H ¨ uttenrauch et al., 2017).
However, MARL algorithms still suffer from poor sample-efficiency and poor scalability due to the curse of dimensionality, i.e., the joint state-action space grows exponentially as the agent number increases (Li et al., 2022).
A way to solve this problem is to properly reduce the size of the state-action space (van der Pol et al., 2021;
Li et al., 2021).
In this paper, we study how to utilize the permutation invariance (PI) and permutation equivariance (PE) 1 inductive biases to reduce the state space in MARL.
Let G be the set of all permutation matrices 2 of size m × m
and g be a specific permutation matrix of G .
A function f : X → Y where X
=
[ x 1 , . . .
x m ] T , is PI if permutation of the input components does not change the function output, i.e., f ( g [ x 1 , . . .
x m ] T ) =
f
([ x 1 , . . .
x m ] T ) , ∀ g ∈ G .
In contrast, a function f : X → Y where X
=
[ x 1 , . . .
x m ] T and Y =
[ y 1 , . . .
y m ] T , is PE if permutation of the input components also permutes the outputs with the same permutation g , i.e., f ( g [ x 1 , . . .
x m ] T ) =
g [ y 1 , . . .
y m ] T , ∀ g ∈ G .
For functions that are not PI or PE, we uniformly denote them as permutation-sensitive functions.
[Table] [Table] A multiagent environment typically consists of m individual entities, including n learning agents and m − n non-player objects.
The observation o i of each agent i is usually composed of the features of the m entities, i.e., [ x 1 , . . .
x m ] , where x i ∈ X represents each entity’s features and X is the feature space.
If simply representing o i as a concatenation of [ x 1 , . . .
x m ] in a fixed order, the observation space will be |X| m .
A prior knowledge is that although there are m !
different orders of these entities, they inherently have the same information.
Thus building functions that are insensitive to the entities’ orders can significantly reduce the observation space by a factor of 1 m ! .
To this end, in this paper, we exploit both PI and PE functions to design more sample efficient MARL algorithms.
[Figure]
Take the individual Q-network of the StarCraft Multiagent Challenge (SMAC) benchmark (Samvelyan et al., 2019)
Q i ( a i | o i ) as an example.
As shown in Fig.1, the input o i consists of two groups of entities: an ally group o ally i and an enemy group
o enemy i .
The outputs consist of two groups as well, i.e., Q-values for move actions A move i and Q-values for attack actions
A attack i .
Given the same o i arranged in different orders, the Qvalues of A move i should be kept the same, thus we can use PI architectures to make Q i ( A move i | o i ) learn more efficiently.
For Q i ( A attack i | o i ) , since there exists a oneto-one correspondence between each enemy’s features in o enemy i and each attack action in A attack i , permutations of o enemy i should result in the same permutations of A attack i , so we can use PE architectures to make Q i ( A attack i | o i ) learn more efficiently.
To achieve PI, there are two types of previous methods.
The first employs the idea of data augmentation, e.g., Ye et al.
(2020) propose data augmented MADDPG, which generates more training data by shuffling the order of the input components and forcedly maps these generated data to the same output through training.
But it is inefficient to train a permutation-sensitive function to output the same value when taking features in different orders as inputs.
The second type applies naturally PI architectures, such as Deep Sets (Li et al., 2021) and GNNs (Wang et al., 2020b; Liu et al., 2020), to
MARL.
These models use shared input embedding layers and entity-wise pooling layers to achieve
PI.
However, using shared embedding layers limits the model’s representational capacity and may result in poor performance (Wagstaff et al., 2019).
For PE, to the best of our knowledge, it has drawn relatively less attention in MARL community and few works exploit this property.
In general, the architecture of an agent’s policy network can be considered as three parts: ❶ an input layer, ❷ a backbone network (main architecture) and ❸ an output layer.
To achieve PI and PE, we follow the minimal modification principle and propose a light-yet-efficient agent permutation framework, where we only modify the input and output layers while keeping backbone unchanged.
Thus our method can be more easily plugged into existing MARL methods.
The core idea is that, instead of using shared embedding layers, we build non-shared entity-wise PI input and PE output network modules to connect the entity-factored state space and action space in an end-to-end way.
Specifically, we propose two novel implementations: a Dynamic Permutation Network (DPN) and a Hyper Policy Network (HPN).
To achieve PI, DPN builds a separate module selection network, which consistently selects the same input module for the same input entity no matter where the entity is arranged and then merges all input modules’ outputs by sum pooling.
Similarly, to achieve
PE, it builds a second module selection network, which always assigns the same output module to the same entity-related output.
However, one restriction of DPN is that the number of network modules is limited.
As a result, the module assigned to each entity may not be the best fit.
To relax the restriction and enhance the representational capability, we further propose HPN which replaces the module selection networks of DPN with hypernetworks and directly generates the network parameters of the corresponding modules (by taking each entity’s own features as input).
Entities with different features are processed by modules with entity-specific parameters.
Therefore, the model’s representational capability is improved while ensuring the PI and PE properties.
Extensive evaluations in SMAC, SMACv2, Google Research Football and MPE validate that DPN and HPN can be easily integrated into many existing MARL algorithms (both value-based and policy-based) and significantly boost their learning efficiency and converged performance.
Remarkably, we achieve 100% win-rates in almost all hard and super-hard scenarios of SMAC, which has never been achieved before to the best of our knowledge.
The code is available at https://github.com/tjuHaoXiaotian/API-Network.
R ELATED W ORK
To highlight our method, we briefly summarize the related works that consider the PI or PE property.
Concatenation .
Typical MARL algorithms, e.g., QMIX (Rashid et al., 2018) and MADDPG (Lowe et al., 2017) simply represent the set input as a concatenation of the m entities’ features in a fixed order and feed the concatenated features into permutation-sensitive functions, e.g., multilayer perceptron (MLP).
As each entity’s feature space size is |X| , the size of the joint feature space after concatenating will grow exponentially to |X| m , thus these methods suffer sample inefficiency.
Data Augmentation .
To reduce the number of environmental interactions, Ye et al.
(2020) propose data augmented MADDPG which generates more training data by shuffling the order of [ x 1 , . . .
x m ] and additionally updates the model based on the generated data.
However, the method requires more computational resources and is more time-consuming.
Besides, as the generated data contains the same information as the original one, they should have the same Q-value.
But it is inefficient to train a permutation-sensitive function to output the same value when taking differently-ordered inputs.
Deep Set & Graph Neural Network .
Instead of doing data augmentation, Deep Set (Zaheer et al., 2017) constructs a family of PI neural architectures for learning set representations.
Each component x i is mapped separately to some latent space using a shared embedding layer ϕ
( x i ) .
These latent representations are then merged by a PI pooling layer (e.g. sum, mean) to ensure the PI of the whole function, e.g., f ( X ) =
ρ (Σ mi =1 ϕ
x i )) , where ρ can be any function.
Graph Neural Networks (GNNs)
(Veli ˇ ckovi ´ c et al., 2018; Battaglia et al., 2018) also adopt shared embedding and pooling layers to learn functions on graphs.
Wang et al.
(2020b)
; Li et al.
(2021) and Jiang et al.
Liu et al.
(2020) have applied Deep Set and GNN to MARL.
However, due to the use of the shared embedding ϕ ( x i ) , the representation capacity is usually limited (Wagstaff et al., 2019).
Multi-head Self-Attention & Transformer.
To improve the representational capacity, Set Transformer (Lee et al., 2019) employs the multi-head self-attention mechanism (Vaswani et al., 2017) to process every x i of the input set, which allows the method to encode higher-order interactions between elements in the set.
Most recently, Hu et al.
(2021b) adopt Transformer (Vaswani et al., 2017) to MARL and proposes UPDeT, which could handle various input sizes.
But UPDeT is originally designed for transfer learning scenarios and does not explicitly consider the PI and PE properties.
PE Functions in MARL.
In the deep learning literature, some works have studied the effectiveness of PE functions when dealing with problems defined over graphs (Maron et al., 2018; Keriven &
Peyr ´ e, 2019).
However, in MARL, few works exploit the PE property to the best of our knowledge.
One related work is Action Semantics Network (ASN) (Wang et al., 2019), which studies the different effects of different types of actions but does not directly consider the PE property.
P ROBLEM S TATEMENT 3. E NTITY -F ACTORED M ODELING IN D EC -POMDP
A cooperative multiagent environment typically consists of m entities, including n learning agents and m − n non-player objects.
We follow the definition of Dec-POMDP (Oliehoek & Amato, 2016).
At each step, each agent i receives an observation o
i ∈
O i which contains partial information of the state s ∈ S , and executes an action a
i ∈ A i according to a policy π i ( a i
| o i ) .
The environment transits to the next state s ′ and all agents receive a shared global reward.
The target is to find optimal policies for all agents which can maximize the expected cumulative global reward.
Each agent’s individual action-value function is denoted as Q i ( o i , a i ) .
Detailed definition can be found in Appendix B.1.
Modeling MARL in factored spaces is a common practice.
Many recent works (Qin et al., 2022;
Wang et al., 2020b; Hu et al., 2021b; Long et al., 2019; Wang et al., 2019) model the observation and the state of typical MARL benchmarks e.g., SMAC (Qin et al., 2022;
Hu et al., 2021b), MPE (Long et al., 2019) and Neural MMO (Wang et al., 2019), into factored parts relating to the environment, agent itself and other entities.
We follow such a common entity-factored setting that both the state space and the observation space are factorizable and can be represented as entity-related features, i.e., s ∈ S ⊆
R m × d s and o i ∈
O ⊆ R m × d o , where d s and d o denote the feature dimension of each entity in the state and the observation.
For the action space, we consider a general setting that each agent’s actions are composed of two types: a set of m entity-correlated actions A equiv3 and a set of entity-uncorrelated actions A inv , i.e., A i ≜ ( A equiv , A inv ) .
The entity-correlated actions mean that there exists a one-to-one correspondence between each entity and each action, e.g., ‘attacking which enemy’ in SMAC or ‘passing the ball to which teammate’ in football games.
Therefore, a equiv ∈
A equiv should be equivariant with the permutation of o i and a inv ∈
A inv should be invariant.
Tasks only considering one of the two types of actions can be considered special cases.
3. O UR T ARGET : D ESIGNING PI AND PE P OLICY N ETWORKS
Let g ∈ G be an arbitrary permutation matrix.
We define that g operates on o i by permuting the orders of the m entities’ features, i.e., go i , and that g operates on a i
= ( a equiv , a inv ) by permuting the orders of a equiv but leaving a inv unchanged, i.e., ga i ≜ ( ga equiv , a inv ) .
Our target is to inject the
PI and PE inductive biases into the policy network design such that:
[Equation]
where g operates on π i by permuting the orders of π i ( a equiv | o i ) while leaving π i ( a inv | o i ) unchanged.
M ETHODOLOGY 4. T HE H IGH - LEVEL I DEA
[Equation]
[Figure]
Output layer 𝑪𝑪 𝑚𝑚 𝑑𝑑 𝑜𝑜 ② ③ ① 𝑜𝑜 𝑖𝑖
Input 𝑧𝑧 𝑖𝑖 ℎ 𝑖𝑖 𝑫𝑫 𝒂𝒂 𝐞𝐞𝐞𝐞𝐞𝐞𝐞𝐞𝐞𝐞 𝒂𝒂 𝐞𝐞𝐢𝐢𝐞𝐞 𝜋𝜋 𝑖𝑖 ( 𝑎𝑎 inv | 𝑜𝑜 𝑖𝑖 )
𝜋𝜋 𝑖𝑖 ( 𝑎𝑎 equiv | 𝑜𝑜 𝑖𝑖 )
Our target policy network architecture is shown in Fig.2, which consists of four modules: ❶ input module A , ❷ backbone module B which could be any architecture, ❸ output module C for actions a inv and ❹ output module D for actions a equiv .
For brevity, we denote the inputs and outputs of these modules as: z i =
A ( o i ) , h i =
B ( z i ) , π i ( a inv
| o i )
=
C ( h i ) , and π i ( a equiv
| o i )
=
D ( h i ) respectively.
To achieve equation 1, we have to modify the architectures of { A, B, C, D } such that the outputs of
C are PI and the outputs of D are PE.
In this paper, we propose to directly modify A to be PI and modify D to be PE with respect to the input o i , and keep the backbone module B and output module
C unchanged.
The following Propositions show that our proposal is a feasible and simple solution.
Proposition 1 If we make module A become PI, the output of module C will immediately become
PI without modifying module B and C .
Proof.
Given two different inputs go i and o i , ∀ g ∈ G , since module A is PI, then A ( go i ) =
A ( o i ) .
Accordingly, for any functions B and C , we have ( C ◦ B ◦ A ) ( go i ) =
( C ◦ B ◦ A )
( o i ) , which is exactly the definition of PI.
Corollary 1 .
With module A being PI, if not modifying module D , the output of D will also immediately become PI, i.e., ( D ◦ B ◦ A ) ( go i ) =
( D ◦ B ◦ A )
( o i ) , ∀ g ∈ G .
Proposition 2 With module A being PI, to make the output of module D become PE, we must introduce o i as an additional input.
Proof.
According to corollary 1, with module A being PI, module D also becomes PI.
To convert module D into PE, we must have module D know the entities’ order in o i .
Thus we have to add o i as an additional input to D , i.e., π i ( a equiv | o i )
=
D ( h i , o i ) .
Then, we only have to modify the architecture of D such that D
( h i , go i )
=
gD ( h i , o i )
=
gπ i ( a equiv | o i ) , ∀ g ∈ G .
[Figure]
sum 𝑚𝑚
B ② ③ ①
[Figure]
[Figure]
permutation equivariantoutput
[Figure]
matmul matmul matmul Input 𝑑𝑑 𝑜𝑜 𝑜𝑜
𝑖𝑖 𝑧𝑧 𝑖𝑖 ℎ 𝑖𝑖 ② ③ ① 𝑫𝑫 𝒂𝒂 𝐞𝐞𝐞𝐞𝐞𝐞𝐞𝐞𝐞𝐞 𝜋𝜋 𝑖𝑖 ( 𝑎𝑎 equiv | 𝑜𝑜 𝑖𝑖 ) FC ① 𝑾𝑾 𝟐𝟐𝐞𝐞𝐢𝐢 ② 𝑾𝑾 𝟑𝟑𝐞𝐞𝐢𝐢 ③ 𝑾𝑾 𝟏𝟏𝐞𝐞𝐢𝐢 ①
MLP
Gumbel-max ① ② ③ 𝑜𝑜 𝑖𝑖 input
FC Layer
An FC-layer under submoduleview ② ③ ① 𝑾𝑾𝑾𝑾𝑾𝑾
MLP
FC ②
FC ③
Gumbel-max 𝒑𝒑 𝐞𝐞𝐢𝐢
𝒑𝒑 𝐨𝐨𝐞𝐞𝐨𝐨
C 𝒂𝒂 𝐞𝐞𝐢𝐢𝐞𝐞 𝜋𝜋 𝑖𝑖 ( 𝑎𝑎 inv | 𝑜𝑜 𝑖𝑖 ) 𝑨𝑨
Minimal Modification Principle.
Although modifying different parts of { A, B, C, D } may also achieve equation 1, the main advantage of our proposal is that we can keep the backbone module
B (and output module C ) unchanged.
Since existing algorithms have invested a lot in handling
MARL-specific problems, e.g., they usually incorporate Recurrent Neural Networks (RNNs) into the backbone module B to handle the partially observable inputs
, we believe that achieving PI and
PE without modifying the backbone architectures of the underlying MARL algorithms is beneficial.
Following this minimal modification principle, our proposed method can be more easily plugged into many types of existing MARL algorithms, which will be shown in section 5.
In the following, we propose two designs to implement the PI module A and the PE module D .
4. D YNAMIC P ERMUTATION N ETWORK
As a common practice adopted by many MARL algorithms, e.g., QMIX (Rashid et al., 2018), MADDPG (Lowe et al., 2017) and MAPPO (Yu et al., 2021), module A , C and D are Fully Connected (FC) Layers and module B is a Deep Neural Network (DNN) which usually incorporates RNNs to handle the partially observable inputs.
Given an input o i containing m entities’ features, an FC-layer under a submodule view is shown in Fig.3 (left), which has m independent weight matrices.
If we denote W in = (cid:2) W in 1 , . . .
W in m (cid:3) T as the weights of the FC-layer A , the output is computed as 4 :
[Equation]
i
[Equation]
where [ j ] indicates the j -th element.
Similarly, we denote W out =
[ W out 1 , . . .
W out m ] T as the weight matrices of the output layer D .
The output π i ( a equiv | o i ) is computed as:
[Equation]
PI Input Layer A .
According to equation 2, permuting o i , i.e., o ′ i
= go i , will result in a different output z
i , as the j-th input is changed to o ′ i
[ j ] while the j-th weight matrix remains unchanged.
To make an FC-layer become PI, we design a PI weight matrix selection strategy for each o
i
[ j ] such that no matter where o i [ j ] is arranged, the same o i
[ j ] will always be multiplied by the same weight matrix.
Specifically, we build a weight selection network of which the output dimension is m :
[Equation]
where the k -th output p in (cid:0) W in k (cid:12)(cid:12) o i [ j ] (cid:1) indicates the probability that o i
[ j ] selects the k -th weight matrix of W in .
Then, for each o i
[ j ] , we choose the weight matrix with the maximum probability.
However, directly selecting the argmax index is not differentiable.
To make the selection process trainable, we apply a Straight Through Estimator (Van Den Oord et al., 2017) to get the one-hot encoding of the argmax index.
We denote it as ˆ p in ( o i [ j ]) .
The weight matrix with the maximum probability can be acquired by ˆ p in ( o i [ j ]) W in , i.e. selecting the corresponding row from W in .
Overall, no matter which orders the input o i is arranged, the output of layer A is computed as:
[Equation]
[Figure]
Since the selection network only takes each entity’s features o i
[ j ] as input, the same o i
[ j ] will always generate the same probability distribution and thus the same weight matrix will be selected no matter where o i
[ j ] is ranked.
Therefore, the resulting z i remains the same regardless of the arranged orders of o i , i.e., layer A becomes PI.
An illustration architecture is shown in Fig.3 (right).
PE Output Layer D .
To make the output layer D achieve PE, we also build a weight selection network ˆ p out ( o i [ j ]) for each entity-related output.
The j -th output of D is computed as:
[Equation]
For ∀ g ∈ G , the j -th element of o ′ i will always correspond to the same matrix of W out and thus the j -th output of π i ( a equiv | o ′ i ) will always have the same value.
The input order change will result in the same output order change, thus achieving PE.
The architecture is shown in Fig.3 (right).
4. H YPER P OLICY N ETWORK
The core idea of DPN is to always assign the same weight matrix to each o
i [ j ] .
Compared with Deep
Set style methods which use a single shared weight matrix to embed the input o i , i.e., |W in | = 1 ,
DPN’s representational capacity has been improved, i.e., |W in | = m .
However, the sizes of W in and
W out are still limited.
One restriction is that for each o i [ j ] , we can only select weight matrices from these limited parameter sets.
Thus, the weight matrix assigned to each o i
[ j ] may not be the best fit.
One question is whether we can provide an infinite number of candidate weight matrices such that the solution space of the assignment is no longer constrained.
To achieve this, we propose a Hyper
Policy Network (HPN), which incorporates hypernetworks (Ha et al., 2016) to generate customized embedding weights for each o i [ j ] , by taking o i [ j ] as input.
Hypernetworks (Ha et al., 2016) are a family of neural architectures which use one network to generate the weights for another network.
Since we consider the outputs of hypernetworks as the weights in W in and W out , the sizes of these parameter sets are no longer limited to m .
To better understand our motivations, we provide a simple policy evaluation experiment in Appendix A.1, where we can directly construct an infinite number of candidate weight matrices, to show the influence of the model’s representational capacity.
PI Input Layer A .
For the input layer A , we build a hypernetwork hp in ( o i [ j ]) =
MLP ( o i [ j ]) to generate a corresponding W in [ j ] for each o i [ j ] .
[Equation]
As shown in Fig.4, we first feed all o i
[ j ] s (colored by different blues), into the shared hypernetwork hp in ( o i [ j ]) (colored in yellow), whose input size is d o and output size is d o d h 5 .
Then, we reshape the output for each o i
[ j ] to d o × d h and regard it as the weight matrix W in [ j ] .
Different o i
[ j ] s will generate different weight matrices and the same o i
[ j ] will always correspond to the same one no matter where it is arranged.
The output of layer A is computed according to equation 7.
Since each o i [ j ] is embedded separately by its corresponding W in [ j ] and then merged by a PI ‘sum’ function, the PI property is ensured.
[Figure]
tuned VDN.
PE Output Layer D .
Similarly, we build a hypernetwork hp out ( o i [ j ]) =
MLP ( o i [ j ]) to generate the weight matrices W out for the output layer D .
The j -th output of layer D is computed as: π
[Equation]
( a equiv | o ′ i )
[ j ] = h i hp out ( o
[Equation]
i [ j ])
[Equation]
By utilizing hypernetworks, the input o i
[ j ] and the weight matrix W out [ j ] directly correspond oneto-one.
The input order change will result in the same output order change, thus achieving PE.
See
Fig.4 for the whole architecture of HPN.
In this section, when describing our methods, we represent module A and D as networks with only one layer, but the idea of DPN and HPN can be easily extended to multiple layers.
Besides, both
DPN and HPN are general designs and can be easily integrated into existing MARL algorithms to boost their performance.
All parameters of DPN and HPN are trained end-to-end with backpropagation according to the underlying RL loss function.
Implementation details can be found in B.2.
E XPERIMENT 5. S TAR C RAFT M ULTIAGENT C HALLENGE (SMAC AND SMAC V 2)
Setup and codebase .
We first evaluate our methods in SMAC, which is an important testbed for
MARL algorithms.
SMAC consists of a set of StarCraft II micro battle scenarios, where units are divided into two teams: allies and enemies.
The ally units are controlled by the agents while the enemy units are controlled by the built-in rule-based bots.
The agents can observe the distance, relative location, health, shield and type of the units within the sight range.
The goal is to train the agents to defeat the enemy units.
We evaluate our methods in all Hard and Super Hard scenarios.
Following Samvelyan et al.
; Hu et al.
(2021a), the evaluation metric is a function that maps the environment steps to test winning rates.
Each experiment is repeated using 5 independent training runs and the resulting plots show the median performance as well as the 25%-75% percentiles.
Recently, Hu et al.
(2021a) demonstrate that the optimized QMIX (Rashid et al., 2018) achieves the
SOTA performance in SMAC.
Thus, all codes and hyperparameters used in this paper are based on their released project PyMARL2.
Detailed parameter settings are given in Appendix C. Except for
SMAC, we also evaluate our HPN on a new challenging benchmark SMACv2.
Due to the space limit, detailed settings and evaluation results are presented in Appendix A.10.
5.1. A PPLYING HPN TO SOTA FINE - TUNED QMIX
We apply HPN to each Q i ( a i | o i ) of the fine-tuned QMIX and VDN (Hu et al., 2021a).
The learning curves over 8 hard and super hard scenarios are shown in Fig.5.
We conclude that: (1) HPN-QMIX surpasses the fine-tuned QMIX by a large margin and achieves 100% test win rates in almost all scenarios, especially in 5m vs 6m , 3s5z vs 3s6z and 6h vs 8z , which has never been achieved before.
Our HPN-QMIX achieves the new SOTA in SMAC; (2) HPN-VDN significantly improves the performance of fine-tuned VDN, and it even surpasses the fine-tuned QMIX in most scenarios, which minimizes the gaps between the VDN-based and QMIX-based algorithms; (3) HPN significantly improves the sample efficiency of both VDN and QMIX, especially in 5m vs 6m , MMM2 , 3s5z vs 3s6z and 6h vs 8z .
In 5m vs 6m , to achieve the same 80% win rate, HPN-VDN and HPNQMIX reduce the environmental interaction steps by a factor of 1 4 compared with the counterparts.
[Figure]
5.1. C OMPARISON WITH PI AND PE BASELINES
The baselines we compared include: (1) DA-QMIX :
According to (Ye et al., 2020), we apply data augmentation to QMIX by generating more training data through shuffling the input order and using the generated data to additionally update the parameters.
(2) SET-QMIX : we apply Deep Set (Li et al., 2021) to each Q i ( a i | o i ) of QMIX, i.e., all x i s use a shared embedding layer and then aggregated by sum pooling, which can be considered as a special case of DPN, i.e., DPN(1).
GNN-QMIX :
We apply GNN (Liu et al., 2020; Wang et al., 2020b) to each Q i ( a i | o i ) of QMIX to achieve PI.
(4) ASN-QMIX (Wang et al., 2019): ASN-QMIX models the influences of different actions on other agents based on their semantics, e.g., move or attack, which is similar to the
PE property considered in this paper.
(5) UPDeT-QMIX (Hu et al., 2021b): a recently MARL algorithm based on Transformer which implicitly considers the PI and PE properties.
The results are shown in Fig.6.
We conclude that:
[Equation]
≈ SET ≈ DA 6 .
Specifically, (1) HPN-QMIX achieves the best win rates, which validates the effectiveness of our PI and PE design; (2) HPN-QMIX performs better than DPN-QMIX.
The reason is that HPN-QMIX utilizes more flexible hypernetworks to achieve PI and PE with enhanced representational capacity.
(3) UPDeT-QMIX and ASN-QMIX achieve comparable win rates with DPNQMIX in most scenarios, which indicates that utilizing the PI and PE properties is important.
GNN-QMIX and SET-QMIX achieve similar performance.
Although PI is achieved, GNN-QMIX and SET-QMIX still perform worse than vanilla QMIX, especially in 3s5z vs 3s6z and 6h vs 8z .
This confirms that using a shared embedding layer ϕ
( x i ) limits the representational capacity and restricts the final performance.
(5) DA-QMIX improves the performance of QMIX in 3s5z vs 3s6z through much more times of parameter updating.
However, its learning process is unstable and it collapses in all other scenarios due to the perturbation of the input features, which validates that it is hard to make a permutation-sensitive function (e.g., MLP) achieve PI through training solely.
The learning curves of these PI/PE baselines equipped with VDN are shown in Appendix A.2, which have similar results.
All implementation details are shown in Appendix B.2.
Besides, in Appendix
A.1, we also test and analyze these methods via a simple policy evaluation experiment.
5.1. A BLATION : ENLARGING THE NETWORK SIZE OF THE BASELINE
[Figure]
For HPN, incorporating hypernetworks leads to a ‘bigger’ model.
To make a more fair comparison, we enlarge the network size of Q i ( a i | o i ) in fine-tuned QMIX (denoted as BIG-QMIX) such that it has more parameters than HPNQMIX.
Besides, the UPDeT baselines already have comparable or more parameters than HPN.
The detailed sizes of these models are shown in
Table 3 of Appendix A.3.
From Fig.7, we see that simply increasing the parameter number cannot improve the performance.
In contrast, it even leads to worse results.
VDN-based results are presented in Appendix A.3.
5.1. A BLATION : IMPORTANCE OF THE PE OUTPUT LAYER AND INPUT NETWORK CAPACITY (
1) To validate the importance of the PE output layer, we add the hypernetwork-based output layer of HPN to SET-QMIX (denoted as HPN-SET-QMIX).
From Fig.7, we see that incorporating a PE output layer could significantly boost the performance of SET-QMIX.
The converged performance of HPN-SET-QMIX is superior to that of vanilla QMIX.
However, due to the limited represen- tational capacity of the shared embedding layer of Deep Set, the performance of HPN-SET-QMIX is still worse than HPN-QMIX.
The only difference between HPN-SET-QMIX and HPN-QMIX is the number of embedding weight matrices in the input layer.
The results validate that improving the representational capacity of the PI input layer is beneficial.
5.1. A PPLYING HPN TO MAPPO AND QPLEX
[Figure]
The results of applying HPN to MAPPO (Yu et al., 2021) and QPLEX (Wang et al., 2020a) in 5m vs 6m and 3s5z vs 3s6z are shown in
Fig.8.
We see that HPN-MAPPO and HPNQPLEX consistently improve the performance of MAPPO and QPLEX, which validates that
HPN can be easily integrated into many types of
MARL algorithms and boost their performance.
Full results are shown in Appendix A.4.
5. M ULTIAGENT P ARTICLE E NVIRONMENT (MPE)
[Figure]
We also evaluate the proposed DPN and HPN in the cooperative navigation and the predator-prey scenarios of MPE (Lowe et al., 2017), where the actions only consist of movements.
Therefore, only the PI property is needed.
We follow the experimental settings of PIC (Liu et al., 2020) (which utilizes GNN to achieve PI, i.e., GNNMADDPG) and apply our DPN and HPN to the joint Q-function of MADDPG.
We implement the code based on the official PIC codebase.
The learning curves are shown in Fig.9.
We see that our HPN-MADDPG outperforms the GNN-based
PIC, DA-MADDPG and MADDPG baselines by a large margin, which validates the superiority of our PI designs.
The detailed experimental settings and full results are presented in Appendix A.6.
5. G OOGLE R ESEARCH F OOTBALL (GRF)
[Figure]
Finally, we evaluate HPN in two Google Research Football (GRF)
(Kurach et al., 2020) academic scenarios: 3 vs 1 with keeper and counterattack hard.
We control the left team players, which need to coordinate their positions to organize attacks, and only scoring leads to rewards.
The observations consist of 5 parts: ball information, left team, right team, controlled player and match state.
Each agent has 19 discrete actions, including moving, sliding, shooting and passing, where the targets of the passing actions correspond to its teammates.
Detailed settings can be found in Appendix A.7.
We apply HPN to
QMIX and compare it with the SOTA CDS-QMIX (Chenghao et al., 2021).
We show the average win rate across 5 seeds in Fig.10.
HPN can significantly boost the performance of QMIX and
HPN-QMIX outperforms the SOTA method CDS-QMIX by a large margin in these two scenarios.
C ONCLUSION
In this paper, we propose two PI and PE designs, both of which can be easily integrated into existing
MARL algorithms to boost their performance.
Although we only test the proposed methods in model-free MARL algorithms, they are general designs and have great potential to improve the performance of model-based (Yuan et al., 2022), multitask and transfer learning (Wu et al., 2023) algorithms.
Besides, we currently follow the settings of (Qin et al., 2022; Wang et al., 2020b; Hu et al., 2021b; Long et al., 2019), where the configuration of the input-output relationships and the observation/state structure are set manually.
For future works, it is interesting to automatically detect such structural information.
A CKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (Grant No.62106172), the “New Generation of Artificial Intelligence” Major Project of Science & Technology 2030 (Grant
No.2022ZD0116402) and the Science and Technology on Information Systems Engineering Laboratory (Grant No.
WDZC20235250409, No.WDZC20205250407)
We thank Zipeng Dai for the insightful discussions and helpful writing suggestions.
We are very grateful for the valuable and constructive comments of all reviewers
A A DDITIONAL E XPERIMENTAL S ETTINGS AND R ESULTS A. A S IMPLE P OLICY E VALUATION P ROBLEM .
We present a simple policy evaluation experiment to show the influence of the model’s representational capacity on the converged performance.
We consider the following permutation-invariant (PI) models: ❶ Deep Set, ❷ DA-MLP (apply data augmentation to a permutation-sensitive MLP model), ❸
DPN (apply DPN to the same MLP model), ❹ HPN (apply HPN to the same MLP model) and ❺
Attention (use self-attention layers and a pooling function to achieve PI).
The experimental settings are as follows.
There are 2 agents in total.
Each agent i only has one dimension of feature x i .
For the convenience of analyzing, we set that each x i is an integer and x i ∈ { 1 , 2 , ..., 30 } , i.e., each agent i only has 30 different features.
Thus the size of the joint state space ( [ x 1 , x 2 ] ) after concatenating is 30 ∗ 30 = 900 .
To make the policy evaluation task permutation-invariant, we simply set the target value Y of each state [ x 1 , x 2 ] as x 1 ∗ x 2 .
In section 4.3, when introducing HPN, we asked a question that whether we can provide an infinite number of candidate weight matrices of DPN.
In this simple task, we can directly construct a separate weight matrix for each feature x i .
Since x i is discrete (which is enumerable), we can explicitly maintain a parameter table and exactly record a different embedding weight for each different feature x i .
We denote this direct method as ❻ DPN( ∞ ), which means ‘DPN with infinite weights’.
Our HPN uses a hypernetwok to approximately achieve ‘infinite weight’ by generating a different weight matrix for each different input x i .
We use the Mean Square Error (MSE) as the loss function to train these different models.
The code for this simple experiment is also available at https://github.com/tjuHaoXiaotian/API-Network (see ‘ Synthetic Policy Evaluation.py ’).
The comparison of the learning curves and the converged MSE losses of these different PI models are shown in Fig.11 and Table 1 respectively.
We conclude that DPN( ∞ ) > HPN > Attention > DPN > DeepSet > DA-MLP, where ‘ > ’ means ‘performs better than’, which indicates that increasing the representational capacity of the model can help to achieve much less MSE loss.
[Table]
[Figure]
If we keep each x i ∈ { 1 , 2 , ..., 30 } and simply increase the agent number, the changes of the original state space by simple concatenation and the reduced state space by using PI representations are shown in Table 2 below.
We see that by using PI representations, the state space can be significantly reduced.
[Table]
[Figure]
A. C OMPARISON WITH VDN- BASED PI AND PE BASELINES
The learning curves of the PI/PE baselines equipped with VDN are shown in Fig.12, which demonstrate that: HPN ≥ DPN ≥ UPDeT > VDN > GNN ≈ SET ≈ DA 7 .
Specifically, (1) HPN-VDN and DPN-VDN achieve the best win rates; (2) Since UPDeT uses a shared token embedding layer followed by multi-head self-attention layers to process all components of the input sets, the PI and
PE properties are implicitly taken into consideration.
The results of UPDeT-VDN also validate that incorporating PI and PE into the model design could reduce the observation space and improve the converged performance in most scenarios.
(3) GNN-VDN achieves slightly better performance than SET-VDN.
Although permutation-invariant is maintained, GNN-VDN and SET-VDN perform worse than vanilla QMIX, (especially in 3s5z vs 3s6z and 6h vs 8z , the win rates are approximate 0%).
This confirms that the use of a shared embedding layer ϕ
( x i ) for each component x i limits the representational capacities and restricts the final performance.
(4) DA-VDN significantly improves the learning speed and performance of vanilla VDN in 3s5z vs 3s6z by data augmentation and much more times of parameter updating.
However, the learning process is unstable, which collapses in all other scenarios due to the perturbation of the input features, which validates that it is hard to train a permutation-sensitive function (e.g., MLP) to output the same value when taking different orders of features as inputs.
A. A BLATION S TUDIES .
[Figure]
M ed i an T e s t W i n %
[Figure] [Figure]
E NLARGING
THE
N ETWORK S IZE
We also enlarge the agent network of vanilla VDN (denoted as BIG-VDN) such that the number of parameters is larger than our HPN-VDN.
The detailed numbers of parameters are shown in Table 3.
The results are shown in Fig.13.
We see that simply increasing the parameter number cannot always guarantee better performance.
For example, in 5m vs 6m, the win rate of BIG-VDN is worse than the vanilla VDN.
In 3s5z vs 3s6z and 6h vs 8z, BIG-VDN does achieve better performance, but the performance of BIG-VDN is still worse than our HPN-VDN in all scenarios.
A.3. I MPORTANCE OF THE PE OUTPUT LAYER AND THE CAPACITY OF THE PI INPUT LAYER
To validate the importance of the permutation-equivariant output layer, we also add the hypernetwork-based output layer of HPN to SET-VDN (denoted as HPN-SET-VDN).
The results are shown in Fig.13.
We see that incorporating an APE output layer could significantly boost the
[Table]
performance of SET-VDN, and that the converged performance of HPN-SET-VDN is superior to the vanilla VDN in 5m vs 6m and 3s5z vs 3s6z
[Table]
However, due to the limited representational capacity of the shared embedding layer of Deep Set
[Table]
the performance of HPN-SET-VDN is still worse than our HPN-VDN, especially in 6h vs 8z.
Note that the only difference between HPN-VDN and HPN-SET-VDN is the input layer, e
[Table]
g., using hypernetwork-based customized embeddings or a simply shared one.
The results validate the importance of improving the representational capacity of the permutation-invariant input layer.
A. A PPLYING HPN TO QPLEX AND MAPPO
[Figure]
M ed i an T e s t
W i n %
[Figure] [Figure]
M ed i an T e s t
W i n %
[Figure]
To demonstrate that our methods can be easily integrated into many types of MARL algorithms and boost their performance, we also apply HPN to a typical credit-assignment method QPLEX (Wang et al., 2020a)
(denoted as HPN-QPLEX) and a policy-based MARL algorithm MAPPO (Yu et al., 2021) (denoted as HPN-MAPPO).
The results are shown in Fig.14 and Fig.15.
We see that HPN significantly improves the performance of QPLEX and MAPPO, which validates that our method can be easily combined with existing MARL algorithms and improves their performance (especially for super hard scenarios).
A. A PPLYING HPN TO D EEP C OORDINATION G RAPH .
Recently, Deep Coordination Graph (DCG) (B¨ohmer et al., 2020) scales traditional coordination graph based MARL methods to large state-action spaces, shows its ability to solve the relative overgeneralization problem, and obtains competitive results on StarCraft II micromanagement tasks.
Further, based on DCG, (Wang et al., 2021) proposes an improved version, named Context-Aware
SparsE Coordination graphs (CASEC).
CASEC learns a sparse and adaptive coordination graph (Wang et al., 2021), which can largely reduce the communication overhead and improve the performance.
Besides, CASEC incorporates action representations into the utility and payoff functions to reduce the estimation errors and alleviate the learning instability issue.
Both DCG and CASEC inject the permutation invariance inductive bias into the design of the pairwise payoff function q ij ( a i , a j |
o i , o j ) .
They achieve permutation invariance by permuting the input order of [ o i , o j ] and taking the average of both.
To show the generality of our method, we also apply
HPN to the utility function and payoff function of CASEC and show the performance in Fig.16.
The codes of CASEC and HPN-CASEC are also available at https://github.com/tjuHaoXiaotian/APINetwork (see code/src/config/algs/casec.yaml and code/src/config/algs/hpn casec.yaml ).
[Figure]
In Fig.16, we compare HPN-CASEC with the vanilla CASEC in 5m vs 6m. Results show that
HPN can significantly improve the performance of CASEC, which validate that HPN is very easy to implement and can be easily integrated into many existing MARL approaches.
A. M ULTIAGENT P ARTICLE E NVIRONMENT
We evaluate the proposed DPN and HPN on the classical Multiagent Particle Environment (MPE) (Lowe et al., 2017) tasks, where the actions only consist of movement actions.
Therefore, only the permutation invariance property is needed.
We follow the experimental settings of PIC (permutationinvariant Critic for MADDPG, which utilizes GNN to achieve PI, i.e., GNNMADDPG) (Liu et al., 2020) and apply our DPN and HPN to the centralized critic Q-function of MADDPG (Lowe et al., 2017).
Each component x i represents the concatenation of agent i’s observation and action.
The input set X j contains all agents’ observation-actions.
We implement the code based on the official
PIC.
The baselines we considered are PIC (Liu et al., 2020), DA-MADDPG (Ye et al., 2020) and
MADDPG (Lowe et al., 2017).
The tasks we consider are as follows:
• Cooperative navigation : n agents move cooperatively to cover L landmarks in the environment.
The reward encourages the agents to get close to landmarks.
An agent observes its location and velocity, and the relative location of the landmarks and other agents.
• Cooperative predator-prey : n slower
predators work together to chase M fast-moving prey.
The predators get a positive reward when colliding with prey.
Preys are environment controlled.
A predator observes its location and velocity, the relative location of the L landmarks and other predators and the relative location and velocity of the prey.
The learning curves of different methods in the cooperative navigation task (the agent number n = 6 ) and the cooperative predator-prey task (the agent number n = 3 ) are given in Fig.9.
Besides, We further test HPN on two more cooperative navigation tasks with 100 and 200 agents respectively.
The learning curves are shown in Figure 17.
The results show that HPN-MADDPG can significantly improve the performance of vanilla MADDPG and achieves superior sample efficiency and converged performance than PIC.
All experiments are repeated for five runs with different random seeds.
We see that our HPN-MADDPG outperforms the PIC, DA-MADDPG and MADDPG baselines in these two tasks, which validates the superiority of our permutation-invariant designs.
[Figure]
. A. G OOGLE R ESEARCH F OOTBALL
We evaluate HPN in two Google Research Football (GRF) academic scenarios: 3 vs 1 with keeper and counterattack hard.
In these tasks, we control the left team players except for the goalkeeper.
The right team players are controlled by the built-in rule-based bots.
The agents need to coordinate their positions to organize attacks and only scoring leads to rewards.
The observations are factorizable and are composed of five parts: ball information, left team, right team, controlled player information and match state.
Detailed feature lists are shown in Table 4.
Each agent has 19 discrete actions, including moving, sliding, shooting and passing.
Following the settings of CDS (Chenghao et al., 2021), we also make a reasonable change to the two half-court offensive scenarios: we will lose if our players or the ball returns to our half-court.
All methods are tested with this modification.
The final reward is +100 when our team wins, -1 when our player or the ball returns to our half-court, and 0 otherwise.
We apply HPN to QMIX and compare it with the SOTA CDS-QMIX (Chenghao et al., 2021).
In detail, when applying HPN to QMIX, both the PI actions, e.g., moving, sliding and shooting, and the PE actions, e.g., long pass, high pass and short pass are considered.
For each player, since the targets of these passing actions directly correspond to its teammates, we apply the PE output layer to generate the Q-values of these passing actions, where the hypernetwork takes each ally player’s features as input and generates the weight matrices for the passing actions.
Besides, in the official
GRF environment, as we cannot directly control which teammates the current player passes the ball to, we take a max pooling over all ally-related Q-values to get the final Q-values for the three passing actions.
We show the average win rate across 5 seeds in Fig.10.
HPN can significantly boost the performance of QMIX and our HPN-QMIX outperforms the SOTA method CDS by a large margin in these two scenarios.
A. G ENERALIZATION : C AN HPN GENERALIZE TO A NEW TASK WITH A DIFFERENT NUMBER OF AGENTS ?
Apart from achieving PI and PE, another benefit of HPN is that it can naturally handle variable numbers of inputs and outputs.
Therefore, as also stated in the conclusion section, HPN can be potentially used to design more efficient multitask learning and transfer learning algorithms.
For example, we can directly transfer the learned HPN policy in one task to new tasks with different numbers of agents and improve the learning efficiency in the new tasks.
Transfer learning results of
[Table]
5m → 12m, 5m vs 6m → 8m vs 10m, 3s vs 3z → 3s vs 5z are shown in Fig.18.
We see that the previously trained HPN policies can serve as better initialization policies for new tasks.
A. T O ACHIEVE PI AND PE, WHAT IF WE JUST SORT THE ENTITIES ACCORDING TO DISTANCE FROM THE FOCAL AGENT ?
(1) When we first started working on this project, we have also considered a similar baseline: we sort the entities according to (type, distance), i.e., according to their types first and then the relative distances if two entities’ types are same.
But we found that this solution do not always work well.
Here, we provide the learning curves of HPN, QMIX, VDN, SORT-QMIX, and SORT-VDN inhard and super hard scenarios on SMAC in Figure 19.
The results show that the sorting baseline can slightly improve the performance of vanilla QMIX/VDN in 5m vs 6m and 3s5z vs 3s6z.
However, in 8m vs 9m and 6h vs 8z, it harms the performance.
The reason is that each entity has many types of features, e.g., relative x, relative y, relative distance, entity type, health point, shield, etc.
Relative distance is just one of them.
Simply sorting the entities by their relative distances while ignoring the influences of the other features may not be appropriate.
Besides, as different x and y can have the same distance and different distances can have the same order, the same o i
[ j ] may be arranged at different positions and be multiplied by different ‘weight matrices‘ (according to z i = (cid:80) mj
=
o i
[ j ] W in [ j ] ).
Therefore, learning may become unstable if we frequently reorder the inputs by distance only.
(3) Thus, our target is not only matching the observation and action belonging to the same entity but stabilizing the learning process by always assigning the same weight matrix W in [ j ] , i.e., a stable weight, to the same entity features o i
[ j ] no matter where o i
[ j ] is arranged.
In this paper
[Figure]
we propose DPN and HPN to achieve this
[Figure] [Figure]
[Figure]
A. E VALUATE HPN ON SMAC- V
SMAC-v2 makes three major changes to SMAC: randomising start positions, randomising unit types, and restricting the agent field-of-view and shooting range to a cone.
These first two changes increase more randomness to challenge contemporary MARL algorithms.
The third change makes features harder to infer and adds the challenge that agents must actively gather information (require more efficient exploration).
Since our target is not to design more efficient exploration algorithms, we keep the field-of-view and attack of the agents a full circle as in SMAC.
• Random Start Positions : Random start positions come in two different types.
First, there is the surrounded type, where the allied units are spawned in the middle of the map, and
surrounded by enemy units.
This challenges the allied units to overcome the enemies approach from multiple angles at once.
Secondly, there are the reflect position scenarios.
These randomly select positions for the allied units, and then reflect their positions in the midpoint of the map to get the enemy spawn positions.
Example figures are shown in Figure 20 below.
• Random Unit Types : Battles in SMACv2 do not always feature units of the same type each time, as they did in SMAC.
Instead, units are spawned randomly according to certain pre-fixed probabilities.
Units in StarCraft II are split up into different races.
Units from different races cannot be on the same team.
For each of the three races (Protoss, Terran, and Zerg), SMACv2 uses three unit types.
Detailed generation probabilities are shown in Figure 21.
Our HPN can naturally handle the two types of new challenges.
Thanks to the PI and PE properties, our HPN is more robust to the randomly changed start positions of the entities.
Thanks to the entity-wise modeling and using hypernetwork to generate a customized weight matrix for each type of unit, HPN can handle the randomly generated unit types as well.
The comparisons of HPNVDN with VDN on three difficult scenarios across the three races (Protoss, Terran, and Zerg) are shown in Figure 22.
Results show that our HPN significantly improves the sample efficiency and the converged test win rates of the baseline VDN.
1.01e7 0.
protoss_5_units 0.
1.01e7 0.
terran_5_units 0.
1.0 Environment Step 1e7 0.
M ed i an
T e s t W i n %
[Figure]
B T ECHNICAL D ETAILS B. D ECENTRALIZED P ARTIALLY O BSERVABLE MDP
We model a fully cooperative multiagent task as a Dec-POMDP (Oliehoek & Amato, 2016), which is defined as a tuple ⟨N , S , O , A , P, R, Z, γ ⟩ .
N is a set of n agents.
S is the set of global states.
O =
{O 1 , . . .
O n } denotes the observation space for n agents.
A =
A 1 × . . .
×
A n is the joint action space, where A i is the set of actions that agent i can take.
At each step, each agent i receives a private observation o i ∈
O i according to the observation function Z ( s, a ) : S ×
A → O , and produces an action a i ∈
A i by a policy π i ( a i |
o i ) .
All agents’ individual actions constitute a joint action a = ⟨ a 1 , ..., a n ⟩ ∈ A .
Then the joint action a is executed and the environment transits to the next state s ′ according to the transition probability P ( s ′ | s, a ) .
All agents receive a shared global reward according to the reward function R ( s, a ) .
All individual policies constitute the joint policy π = π 1 × . . .
× π n .
The target is to find an optimal joint policy π which could maximize the expected return R t = (cid:80)
Tt =
γ t r ( s t , a t ) , where γ is a discount factor and T is the time horizon.
The joint action-value function is defined as Q π ( s t , a t ) =
E π ,P
[ R t | s t , a t ] .
Each agent’s individual action-value function is denoted as Q i ( o i , a i ) .
B. I MPLEMENTATION D ETAILS OF OUR A PPROACH AND B ASELINES
The key points of implementing the baselines and our methods are summarized here: (1) DPN and HPN :
The proposed two methods inherently support heterogeneous scenarios since the entity’s ‘type’ information has been taken into each entity’s features.
And the sample efficiency can be further improved within homogeneous agents compared to fixedly-ordered representation.
For MMM and MMM2, we implemented a permutation-equivariant ‘rescue-action’ module for the only Medivac agent, which uses similar prior knowledge to ASN and UPDeT, i.e., action semantics.
To focus on the core idea of our methods, we omitted these details in the method section.
The objective to train the weight selection network of PDN.
As stated in the last paragraph of
Section 4, all parameters of DPN are trained end-to-end with backpropagation according to the RL loss function.
The weight selection network and the other networks work cooperatively to minimize the overall RL loss function.
How PI is achieved of DPN.
As described in Section 4.2, for each o i [ j ] , the weight selection network outputs the probability of selecting each weight matrix.
During the forward pass at training step t , given the parameter snapshot of the weight selection network, the output probability of selecting each weight matrix is fixed.
For each o i [ j ] , we select the weight matrix with the maximum probability.
However, directly selecting the argmax index is not differentiable.
To make the selection process trainable, we apply a Straight Through Estimator [7] to get the one-hot encoding of the argmax index.
We denote it as ˆ p in ( o i [ j ]) .
The weight matrix with the maximum probability can be acquired by ˆ p in ( o i [ j ]) · W in .
As selecting the weight matrix with the maximum probability is a deterministic process, according to Equation (5), PI is guaranteed.
Besides, to encourage more exploration at the beginning of training, we also add small gumbel noises (Jang et al., 2016) to the ‘logits‘ within the epsilon anneal time.
Within this interval, PI cannot be strictly guaranteed.
When the epsilon anneal schedule is over, PI will be strictly guaranteed.
The
Straight Through Estimator written in PyTorch is shown below:
② no op stop ↑ ↓ → ← 1 0 0 0 00 1 0 × 0 1 0 0 01 0× randomly generated permutation matrix 𝒈𝒈shuffle the order of the original observation 𝒐𝒐 𝒊𝒊 to generate more new data ③ ① ② ③ ① ② ③ ① ① ② ③ attack ③ attack ② attack ① ② ③ ① 𝒐𝒐 𝒊𝒊𝐚𝐚𝐚𝐚𝐚𝐚𝐚𝐚 𝒐𝒐 𝒊𝒊𝐞𝐞𝐞𝐞𝐞𝐞𝐞𝐞𝐚𝐚 randomly generated permutation matrix 𝒈𝒈use the same permutation matrix 𝒈𝒈to shuffle the ‘attack action’ and the ‘available action mask of attack action’
1 def straight_through(y_soft, dim): # Straight Through Estimator. index = y_soft.max(dim, keepdim= True )[1] y_hard = th.zeros_like(y_soft).scatter_(dim, index, 1.0)
= y_hard
.detach(
+
_soft return ret (2) Data Augmentation (DA) (Ye et al., 2020): we apply the core idea of Data Augmentation (Ye et al., 2020) to SMAC by randomly generating a number of permutation matrices to shuffle the ‘observation’, ‘state’, ‘action’ and ‘available action mask’ for each sample simultaneously to generate more training data.
An illustration of the Data Augmentation process is shown in Fig.23.
A noteworthy detail is that since the attack actions are permutation-equivariant to the enemies in the observation, the same permutation matrix M 2 that is utilized to permute o enemy i should also be applied to permute the ‘attack action’ and ‘available action mask of attack action’ as well.
The code is implemented based on PyMARL2 8 for fair comparison.
(3) Deep Set (Zaheer et al., 2017; Li et al., 2021): the only difference between SET-QMIX and the vanilla QMIX is that the vanilla QMIX uses a fully connected layer to process the fixedlyordered concatenation of the m components in o i while SET-QMIX uses a shared embedding layer
h i = ϕ
( x i ) to separately process each component x i in o i first, and then aggregates all h i s by sum pooling.
The
code
is also implemented based
on PyMARL2 for fair comparison
(4) GNN : Following PIC (Liu et al., 2020) and DyAN (Wang et al., 2020b), we apply GNN to the individual Q-network of QMIX (denoted as GNN-QMIX) to achieve permutation-invariant.
The code is also implemented based on PyMARL2.
(5) ASN (Wang et al., 2019): we use the official code and adapt the code to PyMARL2 for fair comparison.
(6) UPDeT (Hu et al., 2021b): we use the official code 9 and adapt the code to PyMARL2 for fair comparison.
(7) VDN and QMIX
: As mentioned in Section 3, vanilla VDN/QMIX uses fixedly-ordered entityinput and fixedly-ordered action-output (
both
are sorted by agent/enemy indices).
Although VDN and QMIX do not explicitly consider the permutation invariance and permutation equivariance properties, they train a permutation-sensitive function to figure out the input-output relationships according to their fixed positions, which is implicit and inefficient.
The codes for the baselines are also published at https://github.com/tjuHaoXiaotian/API-Network.
C H YPERPARAMETER S ETTINGS
For all MARL algorithms we use in SMAC (Samvelyan et al., 2019) (under the MIT License), we keep the hyperparameters the same as in PyMARL2 (Hu et al., 2021a)
(under the Apache License v2.0).
We list the detailed hyperparameter settings used in the paper below in Table 5 to help peers replicate our experiments more easily.
[Table]
D C OMPUTING E NVIRONMENT
We conducted our experiments on an Intel(R) Xeon(R)
Platinum 8171M CPU @ 2.60GHz processor based system.
The system consists of 2 processors, each with 26 cores running at 2.60GHz (52 cores in total) with 32KB of L1, 1024 KB of L2, 40MB of unified L3 cache, and 250 GB of memory.
Besides, we use 2 GeForce GTX 1080 Ti GPUs to facilitate the training procedure.
The operating system is Ubuntu 16.04.
E L IMITATIONS
We currently follow the settings of (Qin et al., 2022;
Wang et al., 2020b; Hu et al., 2021b;
Long et al., 2019;
Wang et al., 2019), where the configuration of the input-output relationships and the observation/state structures are set manually.
What if the observations are images or the structural information is not available?
The high-level idea of this paper is to leverage some formats of symmetries to reduce the size of the search space.
Since typical MARL benchmarks represent observations as factorizable vectors (which can provide more direct and compact information than images), we currently focus on the permutation symmetries, i.e., PI and PE.
For image inputs, the rotational or reflectional symmetries are more prominent characteristics.
Thus, we could leverage rotation invariance or rotation equivariance to design better MARL algorithms, which is also a novel research direction.
For vector inputs, when the structural information is unknown, a potential solution is:
• (1) Learning action representations using a forward model.
We want to learn action representations that can reflect the effects of actions on the environment and other agents.
The effect of an action can be measured by the induced reward and the change in the states.
• (2) Using all actions’ representations as queries and using all entities’ embedded features (potentially generated by HPN) as keys and values, we leverage the self-attention mechanism to generate the Q-values of each action.
Since the self-attention computation is invariant to the input entities’ order, PI and PE are achieved.
And the input-output relationships may be learned implicitly by the self-attention mechanism.
Automatically detecting such structural information is interesting and we leave this as future works.