SPD: Synergy Pattern Diversifying Oriented Unsupervised Multi-agent Reinforcement Learning
Abstract
Reinforcement learning typically relies heavily on a well-designed reward signal, which gets more challenging in cooperative multi-agent reinforcement learning.
Alternatively, unsupervised reinforcement learning (URL) has delivered on its promise in the recent past to learn useful skills and explore the environment without external supervised signals.
These approaches mainly aimed for the single agent to reach distinguishable states, insufficient for multi-agent systems due to that each agent interacts with not only the environment, but also the other agents.
We propose Synergy Pattern Diversifying Oriented Unsupervised Multi-agent
Reinforcement Learning (SPD) to learn generic coordination policies for agents with no extrinsic reward.
Specifically, we devise the Synergy Pattern Graph (SPG), a graph depicting the relationships of agents at each time step.
Furthermore, we propose an episode-wise divergence measurement to approximate the discrepancy of synergy patterns .
To overcome the challenge of sparse return, we decompose the discrepancy of synergy patterns to per-time-step pseudo-reward.
Empirically, we show the capacity of SPD to acquire meaningful coordination policies, such as maintaining specific formations in Multi-Agent Particle Environment and passand-shoot in Google Research Football.
Furthermore, we demonstrate that the same instructive pretrained policy’s parameters can serve as a good initialization for a series of downstream tasks’ policies, achieving higher data efficiency and outperforming state-of-the-art approaches in Google Research Football.
Introduction
Deep reinforcement learning (RL) has exhibited excellent capabilities in tackling complex tasks, including Go [42, 38], poker [28] and simulated robotic tasks [39, 13].
However, one of the key challenges in deploying RL algorithms is to devise an elaborate reward signal for eliciting desired behaviour of agents [14].
Due to the sharing of the team reward, the situation gets more complicated for cooperative multi-agent reinforcement learning (MARL), especially in real-world applications such as coordination of robot swarms [17] and autonomous vehicle teams [54], where verifying the effectiveness of the designed reward can be prohibitively costly.
As for the single agent, unsupervised learning has been incorporated into RL to acquire diverse skills for the agent without extrinsic reward from the environment, and this scenario is known as unsupervised reinforcement learning (URL).
Recent advances in URL [10, 15, 21, 32, 40] have demonstrated the potential of the agent to explore the environment and learn useful skills with no supervision.
Moreover, when endowed with the task-related extrinsic reward in downstream tasks, these skills exhibit significant generalizability, leading to efficient adaptation.
An important hypothesis explaining the phenomenon is made by DIAYN [10] that the higher the coverage of learned skills over the set of possible behaviour, the greater the probability of gaining effective skills.
Above mentioned URL approaches mostly depict the diversity of skills by the diversity of visited states, however, insufficient for MARL due to the neglect of interactions with other agents.
Intuitively, this coordination “skill” in MARL corresponding to the skill in single-agent URL should be taken into consideration as well.
We term the coordination “skill” as the Synergy Pattern , which is the perennial coordinated behaviour of agents induced by interactions of joint policies and the environment.
In retrospect, the coordination of agents are usually modeled by graphs , such as coordination graphs [12, 4, 52] and graph neural networks [18, 22].
Nevertheless, these graph-based approaches are mainly tasked for communication and deploying the graphs directly to unsupervised
MARL can be challenging.
We diversify the synergy patterns of agents by bridging the graphs and unsupervised MARL.
Specifically, we devise the Synergy Pattern Graph (SPG), in which there is a vertex for each agent and the weight of the edge is derived from the function depicting the relations of agents, as the per-time-step instance of the synergy pattern.
Our method Synergy Pattern Diversifying Oriented Unsupervised
Multi-agent Reinforcement Learning (SPD) focuses on obtaining diverse synergy patterns while following the same hypothesis in URL.
To this end, we approximate the Discrepancy of Synergy
Patterns by quantifying the divergence between two SPG batches, which are sampled from the distributions of different synergy patterns respectively.
The first step is to measure the discrepancy between two SPG elements.
This problem is essentially to measure the similarity of graphs, which has been well-studied in the field of graph matching[41, 27, 33, 9], and one of the promising approaches is the Gromov-Wasserstein discrepancy [6, 53].
With the adoption of the Gromov-Wasserstein discrepancy as the pair-wise measurement, we then formulate the discrepancy of synergy patterns as an episode-wise measurement and deploy it as the pseudo episode return for unsupervised MARL.
Our contributions for SPD, a general framework for unsupervised MARL, are summarized as follows:
i) we devise SPG with a general formulation as the instance to depict the synergy patterns of agents; ii) we propose an episode-wise divergence measurement to estimate the discrepancy of synergy patterns; iii) we derive the formulation for the decomposition of the pseudo episode return to alleviate the problem of the sparse return; iv) The empirical results demonstrate that SPD achieves better performance in MARL compared to conventional URL approaches and shows great potentials to learn synergy patterns with generalizability for downstream tasks.
Related works
Multi-agent reinforcement learning (MARL) has shown great potential in solving multi-agent cooperation tasks.
A popular paradigm in MARL is the centralized training with decentralized execution (CTDE) that combines the advantage of independent Q-learning [46] and joint action learning [7].
Centralized training makes agents cooperate better while decentralized execution benefits the flexible deployment capability.
We mainly focus on the value-based CTDE methods.
Following the CTDE paradigm, a series of works concentrate on distributing the team reward to all agents by value function factorization [45, 35], and deriving the Individual-global-max principle for policy optimality analysis [44, 48, 36].
Another bunch of works introduce roles or skills to agents for better team cooperation [25, 50, 5].
However, these methods usually suffer from poor generalization ability and have considerable performance drop when transferred to unseen scenarios.
Wang et al. [51] reports agents’ generalization by splitting the action space, but the method relies heavily on the handcrafted hyper-parameter.
The concept of agents’ forming a graph is also widely used to represent agents’ cooperation in MARL, including the coordination graph [4, 52] and graph neural networks [18, 22].
But these methods are usually designed for communication, which brings extra costs for decentralized execution.
In contrast to these methods, our proposed method uses the synergy graph to train team skills without rewards from the environment, which brings the agents strong generalization ability in downstream tasks, and takes no extra cost in execution.
Unsupervised reinforcement learning (URL) has been particularly intriguing in the context of
RL, consider the high cost of real-world rollouts with delicately designed reward signal.
This paper is concerned about unsupervised behaviour learning for deep RL that requires learning with only the task-agnostic intrinsic reward.
Early works [11, 1] mainly focuses on controlling the visited states via regulating a latent variable Z , where the mutual information is maximized with visited states.
DIAYN [10] demonstrates the exceptional performance of URL by maximizing I ( Z ; S ) while simultaneously minimizing I ( Z ; A | S ) .
The recent studies have also shown interests on adapting pre-trained skills on MDPs from aspects of the initial state [3] and the dynamics model [40].
Besides,
WURL [15] adopts the idea of optimal transport and increases the discrepancy of learned skills by enlarging the distance of visited state distributions estimated by Wasserstein distance.
These approaches aiming for achieving easily distinguishable states, however, they are deficient to solve the growing size of joint observation space in multi-agent systems.
By contrast, our method devise a graph to depict the relative relations between agents which alleviate the growing size challenge.
There are also works encouraging diversity for effective exploration [20, 24, 31, 55], while they mostly enhance single policy’s diversity or the inter-agent diversity and do not get rid of the dependence of the task-related reward. We follow a similar hypothesis to that in Eysenbach et al.
[10] to learn diverse synergy patterns for gaining useful coordination policies.
Preliminaries 3.1 Notation
We consider a Dec-POMDP [30] for the fully cooperative multi-agent task with n agents, defined by a tuple ⟨S , A , P , R , O , I , Ω , n, γ ⟩ , where s ∈ S and
a i ∈
A denote the true state of the environment and the discrete action chosen by agent i ∈
I ≡ { 1 , . . . , n } , respectively.
γ ∈ [0 , 1) is the discount factor.
The dynamics of the environment can be formulated as the state transition function
P ( s ′ | s, a ) : S ×
A × S → [0 , 1] and the reward function R ( s, a ) : S ×
A → R shared by all agents, where a ∈ A ≡ A n denotes the joint action.
Each agent i only has access to its own observation o i ∈ O derived from the observation function Ω( s, i ) : S × I → O , due to the partial observability .
At each time step t , agent i resolves to execute the action a i according to a stochastic policy π i ( a i | τ i ) :
T ×A → [0 , 1] , where τ i ∈ T ≡ ( O × A ) ∗ denotes the action-observation history of the agent i .
The goal of MARL is to find an optimal joint policy πππ ∗ ≡ (cid:81) ni
=1 π i ∗ satisfying the condition: Q πππ ∗ ( s, a ) ≥
Q πππ ( s, a ) , ∀ πππ and ( s, a ) ∈ S × A , where Q πππ ( s t , a t ) =
E s t +1: ∞ , a t +1: ∞ (cid:2)(cid:80) ∞ i =
γ i r t
+ i | s t , a t (cid:3) is the joint action-value function induced by joint policy πππ .
We adopt the paradigm of centralized training with decentralized execution (CTDE).
During training the true state s and the joint action-observation history τττ ≡ { τ i } ni =1 are accessible to the learning algorithm, while each agent i selects an action based on the policy conditioning only on its own action-observation history
τ i during testing (execution).
3.2 Optimal transport and Gromov-Wasserstein discrepancy
Optimal transport (OT) is a general problem of moving one distribution of mass to another as efficiently as possible.
Likewise, the distinction of two given graphs can be regarded as an optimal transport problem to find the correspondence between their vertices and edges.
Previous work [6] proposed Gromov-Wasserstein Discrepancy , which is extended from Gromov-Wasserstein distance [26] as a measure for the divergence of the graphs.
Definition 3.1.
[6, 53] (Gromov-Wasserstein discrepancy)
Denote a measure graph as G ( V , WWW ,µµµ ) , where V ≡ { v i } |V| i =1 is the set of vertices, ω ij ∈ WWW is the weight of the undirected edge { i, j } between vertices v i , v j ∈ V , and µµµ =
[ µ i ] ∈ Σ V is a Borel probability measure defined on V .
∀ p ∈
[1 , ∞ ] and each G ν , G ς ∈ G , where G denotes the collection of measure graphs, the Gromov-Wasserstein discrepancy between G ν and G ς can be derived as
[Equation]
where Π ( µ ν ,
[Equation]
In general, the Gromov-Wasserstein discrepancy measures the divergence between two graphs by comparing the differences of the edges’ relationship in one graph to that in the other.
We refer the reader to Mémoli [26], Chowdhury & Mémoli [6], Peyré et al.
[34] for mathematical foundations.
Methodology
This section is organized as the following: In Sec. 4.1, we delineate the concept of Synergy Pattern and propose a novel structure of graph as the instance of it, namely Synergy Pattern Graph .
We then carefully derive the formulation for Discrepancy of Synergy Patterns and make an approximation of it for convenience in practice in Sec. 4.2.
In Sec. 4.3, our unsupervised framework SPD is depicted in detail and the mechanism for the decomposition of the pseudo reward is explained.
Sec.
4.4 provides the implementation details for solving the optimization problem.
4.1 Synergy pattern
With the awareness of interactions between agents’ actions, we describe the perennial coordinated behaviour of agents by the Synergy Pattern .
Previous methods [12, 4, 18, 22] utilize the structure of graph to depict it, whereas most of them focus on the local relationship between agent pairs towards a specific task.
To get rid of the extrinsic reward signal, we propose Synergy Pattern Graph (SPG), denoted as G sp ( V , WWW ,µµµ, ζ ) , where v i ∈ V is the vertex for agent
i ∈
I and the weight ω ij ∈ WWW of edge { i, j } is derived from a synergy pattern function which could depict agents’ relative relations:
[Equation]
The synergy pattern function follows a general formulation as any function with range R +0 , for instance, the information-theoretic influence [49]: ζ ( τ i , τ j ) :
= MI πππj | i ( o ′ j ; o i , a i | o j , a j ) or the communication messages [4].
It can depend on more than merely the observations and joint actions as well, whereas we omit this for simplicity.
At each time step, SPG varies depending on the history of the multi-agent system rather than being predefined before training.
We will use G t to denote
G sp ( V , WWW ,µµµ, ζ ) at time step t in the following part of this paper for ease of notation.
Accordingly, the synergy pattern can be described by the distribution of SPG obtained by deploying a joint policy πππ in the environment: P ( G πππ | πππ ) with domain Λ , where Λ is the collection of all SPG.
4.2 Discrepancy of synergy patterns
We use the discrepancy between two distributions of SPG: P ( G πππ 1 | π 1 π 1 π 1 ) and P ( G πππ 2 | π 2 π 2 π 2 ) to denote the discrepancy between two synergy patterns.
This process can be decomposed into two phases: i) measuring the distance between two samples from each distribution respectively; ii) extending the metric for a pair of samples to that for distributions.
The former one is essentially to estimate the discrepancy between graphs.
We adopt the Gromov-Wasserstein discrepancy of graphs, as mentioned in section 3.2, to measure this distance.
Besides, we solve the latter one by applying optimal transport on SPG distributions.
This whole process leads to the following definition:
Definition 4.1.
(Discrepancy of Synergy Patterns) Let p 1 = P
( G πππ 1 | π 1 π 1 π 1 ) and p 2 = P
( G πππ 2 | π 2 π 2 π 2 ) be two distributions of SPG induced by joint polices πππ 1 and πππ 2 , respectively.
Denote an joint distribution ϱ ∈ Γ[ p 1 , p 2 ] as a bijective plan transporting probability mass from p 1 to p 2 , where Γ[ p 1 , p 2 ] is the set of all distributions on product space Λ × Λ with p 1 and p 2 as their marginal probabilities.
Then the discrepancy of synergy patterns is defined as
[Equation]
where d gw is the Gromov-Wasserstein discrepancy according to Eq.
Besides, we show that d sp is a legitimate pseudometric on Λ and the proof is deferred to Appendix A.
However, solving Eq.
(2) for the discrepancy of synergy patterns is almost intractable.
In practice, we collect two SPG batches to approximate the discrepancy of synergy patterns.
Here, we give the definition for this approximation:
Definition 4.2.
(Proximal Discrepancy of Synergy Patterns)
Denote ¯ λ πππ 1 = { G πππ 1 t } B 1 t =0 ∼ P ( G πππ 1 | π 1 π 1 π 1 ) and ¯ λ πππ
=
{ G πππ 2 t } B 2 t =0 ∼ P ( G πππ 2 | π 2 π 2 π 2 ) sampled from the distributions of SPG, where B 1 , B 2 ∈ N + .
Assume B 1 ≤ B 2 by symmetry.
Let D B 1 × B 2 = (cid:2) d gw ( G πππ 1 t 1 , G πππ 2 t 2 ) (cid:3) , ∀ 0 ≤ t 1 ≤ B 1 , 0 ≤ t 2 ≤ B be the cost matrix.
Suppose that an injective function ρ : { 1 , . . .
, B 1 } → { 1 , . . .
, B 2 } denotes an matching function.
By minimizing the total cost, an approximation for the discrepancy of synergy patterns is:
[Equation]
It is vital to point out that the SPG from ¯ λ πππ 2 corresponding to G πππ 1 t may not be G πππ 2 t since the relations of agents may be analogous at different time steps.
Hence, we search the whole ¯ λ πππ 2 for G πππ 2 ρ ( t ) that is relatively adjacent to G πππ 1 t based on the Gromov-Wasserstein discrepancy.
4.3 Synergy pattern diversifying oriented unsupervised multi-agent reinforcement learning
Based on the discrepancy of synergy patterns, we can now delineate the framework for SPD.
Suppose that Z ∈ N + denotes the number of different synergy patterns.
Our goal is to learn a set of joint policies { πππ z } Zz =1 with diverse synergy patterns to explore the environment sufficiently when there is no extrinsic reward.
To increase the divergence among synergy patterns, we formulate the pseudo episode return for the episode data τττ
z produced by the joint policy πππ z :
[Equation]
(4) where 1 ≤ z ≤ Z , and ¯ λ πππ z is the SPG set derived from τττ z .
Besides, we obtain ¯ λ πππ k by sampling a batch of data from the replay buffer of πππ k and ensure ¯ λ πππ k have more elements: | ¯ λ πππ k | ≥ | ¯ λ πππ z | .
Our method is to maximize the expected return: max πππ z E [ ˜ R z ( τττ z , { τττ k } )] , resulting in the maximization of each discrepancy of one synergy pattern to its nearest counterpart.
However, the pseudo episode return in Eq.
(4) depending on the whole episode data τττ z is sparse since it only derives a single final reward for one episode.
This imposes the trouble for MARL algorithms to learn an accurate action-value function Q πππ z ( s, a ) and tends to impair the performance of value-based MARL algorithms.
To bypass this predicament, intuitively, the pseudo episode return ˜
R z ( τττ z , { τττ k } ) can be decomposed into pseudo rewards for every time step according to the optimal matching function ρ ∗ .
Concretely, the pseudo reward at time step t can be formulated as:
[Equation]
where ρ ∗ is the optimal solution to ¯ d sp (¯ λ πππ z , ¯ λ πππ k ) by Eq.
(3), and k ∗ denotes the id of the synergy pattern with minimal discrepancy to ¯ λ πππ z according to Eq.
In other word, the pseudo reward is the
Gromov-Wasserstein discrepancy of the SPG at current time step and the SPG from the counterpart of another synergy pattern.
We discriminate the concepts we proposed in Appendix D for a clearer understanding.
The visualization for the pseudo reward derivation is shown in Fig.
4.4 Implementation
We implement SPD with QMIX [35], learning a set of joint policies that maximize the discrepancy of synergy patterns to each other.
QMIX is designed to have a mixing network structure, learning a global action-value function Q :=
{ Q tot | Q tot ( τττ, a ) =
f s (cid:0) Q 1 ( τ 1 , a 1 ) , . . .
, Q n ( τ n , a n ) (cid:1) , ∂f s ∂Q i ≥ 0 , Q i ( · , · ) ∈ R } which satisfies the IGM principle [44], to accomplish the cooperative task for agents.
During unsupervised learning, we select 1 ≤ z ≤ Z for the synergy pattern at the start of each episode randomly or in turn, and interact with the environment according to the joint policy πππ z throughout the episode.
In practice, the Euclidean distance of o i and o j is adopted as the weight ω
ij of edge { i, j } to build the SPG, i.e., the formulation of synergy pattern function is ζ ( τ i , τ j ) =
∥
o i − o j ∥ 2 .
We will show in Sec.
5 that, even with such a synergy pattern function to grab the simple information of relationships of agents, SPD can still learn useful synergy patterns such as pass-and-shoot in Google football Research and accelerate the learning procedure when endowed with the task-related reward.
Since solving for the Gromov-Wasserstein discrepancy between two measure graphs in Eq.
(1) is a computationally costly process, we adopt [53]’s
Regularized Proximal Gradient method to estimate it.
We set p = 2 in Eq.
(1) in our experiments.
As for the probability measure, we set µµµ = 1 n 1 n since the agents are assumed to have equal contributions, where n is the number of agents or vertices in synergy patterns and 1 n is a size n vector of ones.
In the m -th iteration, the current optimal transport
T ( m ) can be updated by
[Equation]
where L ( WWW ν , WWW ς , T ( n ) )
=
WWW ν µµµ ν 1 ⊤|V ς | + 1 |V ν |
µµµ ⊤ ς WWW ⊤ ς − 2 WWW ν
T WWW ⊤ ς , and Kullback-Leibler (KL) divergence is added as the proximal term.
⟨· , ·⟩ denotes the inner product of two matrices and α is a hyper-parameter to balance two items.
The Eq.
(6) can then be solved with nearly-linear convergence [2] by the Sinkhorn-Knopp algorithm [43, 8].
As soon as T ( m ) converges to a stable optimal transport ˆ T , the estimation of the Gromov-Wasserstein discrepancy can be calculated by plugging ˆ T into Eq.
To solve Eq.
(3), we utilize Kuhn–Munkres algorithm [29], which could solve such an assignment problem in polynomial time, to obtain the optimal matching funciton ρ ∗ .
Therefore, the complete formula for pseudo reward at time step t is as below:
[Equation]
2 ˆ T ii ′ ˆ T jj ′   , (7) where G πππ z t ∈ ¯ λ πππ z , G πππ k ∗ ρ ∗ ( t ) ∈
¯ λ πππ k ∗ , and k ∗ denotes the id of the synergy pattern with minimal discrepancy to ¯ λ πππ z according to Eq.
The whole framework is summarized in Algorithm 1.
Experiments
In this section, we first compare properties of SPD with that of other URL baselines on Multi-agent Particle Environment (MPE) [23] in Sec. 5.1.
And in Sec. 5.2, we evaluate the generalization ability of our method on downstream tasks on Google Research Football (GRF) [19]. Our code is available at https://github.com/thu-rllab/SPD .
5.1 Synergy pattern diversifying
We demonstrate that our method SPD is capable of capturing more inherent properties of multi-agent systems such as the impacts from other agents, comparing to conventional URL algorithms.
In order pattern function ζ described in section Maximal time for episode t max Obtain d
Obtain
Calculate
24: end to verify this claim, we first evaluate the diversity of coordination policies learned by SPD and URL baselines in Multi-agent Particle Environment 2 [23, 47].
We customize the scenario SimpleTag , which is a predator-prey environment, for URL experiments by discarding the extrinsic reward from the environment.
Instead, we are concerned about the diversity of agents’ formations driven by learned coordination policies.
The detailed description of the scenario please refer to Appendix B.1.
Baselines and evaluation We compare our method SPD with several conventional single-agent
URL approaches. Here is the details about the implementation of URL algorithms, which we carried out to learn policies without external reward: • SPD: we follow the narration of the implementation details in Sec. 4.4. The pseudo-reward is derived via Eq.
(1) ∼ (7).
• Diversity Is All You Need (DIAYN)
[10]: The joint policy πππ ( a | τττ z , z ) is conditioned on a discrete latent variable z sampled from a pre-defined categorical distribution p ( z ) : z ∼ p ( z ) . The pseudo-reward with formulation as r z := log q ϕ ( z |{ o i } ni =1 ) − log p ( z ) is used for inspiring agents to visit states that can be distinguished easily, where q ϕ ( z |{ o i } ni =1 ) is a learned discriminator. In our experiment, we replace the actor network πππ ( a | τττ z , z ) shared by all coordination policies with Z individual networks { πππ z } Zz =1 , considering the large observation space and action space.
• Wasserstein Unsupervised Reinforcement Learning (WURL)
[15]: The pseudo-reward is designed to enhance the distance between different policies, and it is obtained by estimating the Wasserstein distance of the induced state distributions: r z = W ( τττ z , ¯ τττ k ) , where τττ z is the trajectory driven by current joint policy πππ z ( a | τττ z ) and ¯ τττ k is a sampled batch from another joint policy πππ k .
/mpe The URL algorithms are implemented with QMIX to learn diverse joint policies { πππ z } Zz =1 , and we set Z = 10 in practice.
The hyper-parameters are kept to be the same, and please refer to Appendix B.2 for details.
For evaluation, each algorithm is carried out with 5 different random seeds and trained with 4 × 10 6 steps interacting with the environment.
All the intrinsic pseudo-rewards are derived based on the positions of agents, which is a part of the original observation space.
[Table]
Since DIAYN and WURL are designed for the single agent case, we deploy them by regarding all agents as a single agent.
Concretely, we use observations from all agents to create the features that DIAYN and WURL need.
We evaluate these algorithms following the same protocol: (i) collect sample transitions except extrinsic reward using each joint policy πππ z ( a | τττ z ) , (ii) calculate the pseudo-rewards { r z } for each time step and then assign them to the trajectory τττ z , (iii) employ learned joint policies once the training procedure is completed to collect { τττ final z } Zz = for evaluation.
We use two metrics to evaluate how diverse are the learned joint policies from different aspects: the diversity of states and the discrepancy among relative relationships of agents induced by { πππ z } Zz =1 .
As for the former one, we adopt the structure of the discriminator q ϕ
( z |{ o i } ni =1 ) from DIAYN, which is a learned neural network to classify each policy πππ z from state samples in { τττ [Equation] } Zz =1 , and use the success rate of the discriminator to measure how diverse are the states, namely Discriminator Success Rate (DSR).
As discussed in Sec.
4.2, the discrepancy of synergy patterns d sp according to Eq.
(2) is an effective measure for the latter one.
Table 1 shows that the performance of all algorithms on DSR is close, whereby SPD significantly outperforms the others on the discrepancy of synergy patterns.
Such results demonstrate that SPD can inspire agents to visit comparably identifiable states as the conventional URL approaches, while further motivating agents to explore with other types of coordination policies in the meanwhile. means using Eq. (4) as final the reward directly. “On-policy ρ ( t ) =
t ” means we apply no optimal transport and set ρ ( t ) = t in Eq. (3) to suit the on-policy manner computing pseudo-reward on-the-fly.
And we reduce the precision of the solution for Eq.
(6) in “Sub-optimal ¯ ρ ”.
Ablation study We examine the following components of our method in detail to show their effects: (i) Reward scale β r : we multiply the pseudoreward in Eq.
(5) with a factor vary in { 0 . 1 , 1 . 0 , 10 . 0 } to change the scale of the final returns received by agents, and 10 . 0 is the default value in the aforementioned experiments.
(ii) Sparse return ˜ R
z : To evaluate the effect of the decomposition of the pseudoreturn in Eq. (4), we directly use it as the pseudoreward of the last step in the episode. (iii) On-policy ρ ( t ) = t : As discussed in Sec. 4.2 that the SPG from ¯ λ πππ 2 corresponding to G πππ 1 t may not be G πππ
t .
To verify this claim, we set ρ ( t ) = t directly and the pseudo-reward becomes to ˜ r (no match) t ( τττ z , { τττ k } ) = d
gw
( G πππ z t , G πππ k ∗ t ) , where G πππ z t ∈ ¯ λ πππ z , G πππ k
∗
t
∈ ¯ λ πππ k ∗ .
(iv) Sub-optimal ¯ ρ : The matching function ρ ∗ is obtained by Kuhn–Munkres algorithm [29].
Therefore, the precision is influenced by the number of iterations, which defaults to N iterKM = 100 in the experiments.
A lower number of iterations N iterKM = 25 is used to show the influence from the precision of the matching function.
We use ¯ d sp as the evaluating indicator and the results is shown in Fig. 2.
The curves start from the time step that the exploration rate epsilon has stopped annealing.
We can summarize that the reward scale plays an important role in learning good synergy patterns.
Canceling the optimal transport for
5.2 Generalizability of learned synergy patterns
In this part we demonstrate the generalization ability of the models trained by SPD.
Specifically, we first train SPD on the complicated MARL environment: Google Research Football [19] without environment reward.
Then we use the trained model as the initialization for different tasks with specific rewards, and use common MARL algorithms to train it.
Task description We choose the map “academy_3_vs_1_with_keeper” with the setting from CDS [5], where three players need to cooperate to shoot and get a score under the defense of an opposing player and a goalkeeper.
To simplify the training, the original environment ends an episode if the ball or the player return to the left half-court.
Players always start at the same place.
Losing the control of the ball also cause the failure.
The team is rewarded 100 at the end of an episode when there is a goal, otherwise − 1 .
On the basis of this scenario which we call “Half-court; Fixed Init”, we design a series of tasks with more difficulty.
(i)
We initialize the agents randomly in a circle with 5% court width’s radius, which increases the difficulty by introducing more randomness.
(ii) We remove the restrictions on the left-half court, which greatly increases the exploration space. (iii)
The possession change will not end the episode, which may increase the percent of the bad samples in the replay buffer, since at the beginning of training the ball is easily robbed by the preset AI opponent.
Utilizing SPD model We first train the models by SPD with Z = 20 on the “Half-court; Possession can change” scenario without the environment reward for 10 6 time steps, as well as WURL [15] and DIAYN [10].
Then we use the same trained parameters as the initialization for QMIX [35] on four different downstream scenarios.
In contrast to the manual selection of the synergy patterns, we test them on the downstream task before MARL training, choosing the parameters of the model with the highest average return as the model initialization.
We show the results in Fig.
3, together with the SOTA on GRF including QMIX [35], QPLEX [48] and CDS [5].
The results show that using the model learned by conventional single-agent URL methods (WURL and DIAYN) as the initialization mostly performs similarly to the baseline QMIX, while WURL slightly outperforms QMIX on the “Full-court” maps.
In contrast, SPD learns faster and finally gets higher winning rate on all scenarios, demonstrating the effectiveness of d sp as an evaluation metric for the difference of agents’ relationship and showing that SPD embeds the model with stronger generalization ability.
Actually, one difference between SPD and these URL methods is that SPD encourages the diversity of the relationship among agents while these diversity methods encourages the diversity of visited states.
Therefore, the models trained by WURL may reach diverse states on the ’Full-court’ maps which need more exploration and we believe this accounts for the slightly better performance.
Visualization of the learned synergy patterns To comprehensively evaluate SPD, we visualizethe trajectories of all the skills and show three of them in Fig. 4.
The trajectories of other skills can be found in Appendix C. We find that agents can perform some meaningful behavior from a human perspective, even without the supervision of the environment reward.
Agents can dribble for a long [Figure] distance, pass between teammates, shoot, and run back together in different skills.
To our concern that the different performance of different joint policies learned by SPD are only due to the randomness of the GRF environment itself but not the inter-policy diversity we want, we deploy each learned joint policy for 10 episodes and visualize them.
The results show that each joint policy exhibits the similar coordination behavior across episodes.
The visualization shows that making the team coordinate more diverse drives agents to learn meaningful behaviour, which is favorable for downstream tasks.  Discussion 6.1 Limitations
There are still some limitations or difficulties with further applications.
Though the discriminator success rate (DSR) used in Sec.
5.1 is not a good measure for relationships of agents, it does evaluate the diversity of visited states in some ways.
The results on DSR show that all of the URL approaches, including our method SPD, are far away from satisfying, implying that there is still space for improvement. Furthermore, a synergy pattern function ζ that embeds more information than the one used in this paper is tend to require higher computational cost. Moreover, incorporating diversity-based exploration technique with SPD may improve the intra-agent diversity as well as inter-policy diversity. These limitations may suggest further research directions.
6.2 Negative societal impact
Due to the domain gap between virtual environments and real-world scenarios, applying our method directly to some real-world applications, such as traffic management, is risky and may result in damage to the system.
To avoid harmful operations, human supervision is required to restrict the learned models.
In this work, we propose a general unsupervised framework Synergy Pattern Diversifying Oriented
Unsupervised Multi-agent Reinforcement Learning (SPD) to learn useful synergy patterns for agents.
Synergy pattern graph (SPG) is devised to depict the relationship among all agents.
Furthermore, we approximate the discrepancy of synergy patterns by applying optimal transport on the batches sampled from the distributions of SPG.
In our experiments, we show that SPD surpasses conventional
URL approaches on capturing the relative relations of agents and learns useful synergy patterns with no task-related reward.
We also empirically demonstrate that the learned policies can accelerate the downstream training procedure and outperform state-of-the-art algorithms in GRF
Acknowledgement
This work was supported by the National Key R&D Program of China under Grant 2018AAA0102801, National Natural Science Foundation of China under Grant 61620106005. A Properties of the discrepancy of synergy patterns as a legitimate pseudometric
Proposition A.1. By the definition 4.1, d sp is a legitimate pseudometric on Λ which can be proved by examining following properties:
i) d sp ( p 1 , p 2 ) ≥ 0 ,
Proof.
As Chowdhury & Mémoli [6] show, the Gromov-Wasserstein discrepancy is a pseudometric on G , where G denotes the collection of measure graphs, leading to d gw ( G πππ 1 , G πππ 2 ) ≥ 0 .
Following the definition in Eq.
(2), proving d sp ( p 1 , p 2 ) ≥ 0 is trivial since all elements are non-negative. ii) d sp ( p, p ) = 0 , Proof. Similarly, based on d gw ( G πππ , G πππ ) = 0 , we can choose ϱ carefully as ϱ 0 with P ( G πππ 1 , G πππ 2 = G )
=
δ ( G ) and P ( G πππ 1 = G, G πππ 2 ) =
δ ( G ) everywhere, where δ is the Dirac distribution.
Hence we have:
[Equation] iii) d sp ( p 1 , p 2 ) = d sp ( p 2 , p 1 ) , Proof.
To avoid unnecessary confusion, we notate the joint distribution of the L.H.S as ϱ l and that of the R.H.S. as ϱ r , respectively.
Denote ϱ l = ϱ 12 as one of the joint distributions when the infimum of the L.H.S. is reached:
[Equation]
We show that the infimum of the R.H.S. is reached when ϱ r = ϱ 21 by contradictions, where ϱcan be obtained by exchanging the probabilities of G πππ 1 and G πππ 2 in ϱ 12 .
Suppose there exists ϱ r = ϱ ′ 21 such that:
[Equation]
Then we can update the joint distribution for the L
[Equation]
H
[Equation]
S. with ϱ l = ϱ ′ 12 by exchanging the probabilities of G πππ 1 and G πππ 2 in ϱ ′ 21 similarly.
Based on the property of the Gromov-Wasserstein discrepancy: d gw
[Equation]
G
[Equation]
, G
[Equation]
[Equation]
d gw ( G πππ 2 , G πππ 1 ) , we have: [Equation] This contradicts the claim that ϱ l is one of the joint distributions when the infimum of the L.H.S. is reached. Hence, we can derive:
[Equation] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table] [Figure] [Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
which finishes the proof.
iv) d sp ( p 1 , p χ ) + d sp ( p 2 , p χ ) ≥ d sp ( p 1 , p 2 ) .
Proof.
We prove the triangle inequality by contradictions similar to iii). Suppose that ∃ p χ , s.t.
d sp ( p 1 , p χ ) + d sp ( p 2 , p χ ) < d sp ( p 1 , p 2 ) .
Denote the joint distributions for the infimum of the discrepancy of synergy patterns as ϱ 1 χ , ϱ 2 χ and ϱ 12 respectively. Let ϱ χ 2 be the joint distribution by exchanging the probability of G πππ 2 and G πππ χ .
Then we can use ϱ 1 χ and ϱ χto find the probabilities of G πππ 1 and G πππ 2 corresponding to the same G πππ χ , obtaining the new joint distribution ϱ ′ 12 . By the triangle inequality of the Gromov-Wasserstein discrepancy, we can write: (cid:20)(cid:90)
[Equation]
1 , G πππ 2 )
d ϱ (cid:21) ϱ = ϱ ′≤ (cid:20)(cid:90) d gw ( G πππ 1 , G πππ χ ) d ϱ (cid:21) ϱ = ϱ 1 χ + (cid:20)(cid:90) d gw ( G πππ χ , G πππ 2 )
d ϱ (cid:21) ϱ = ϱ χ= (cid:20)(cid:90) d gw ( G πππ 1 , G πππ χ ) d ϱ (cid:21) ϱ = ϱ 1 χ + (cid:20)(cid:90) d gw ( G πππ 2 , G πππ χ ) d ϱ (cid:21) ϱ = ϱ 2 χ = d sp ( p 1 , p χ ) + d sp ( p 2 , p χ ) <
d sp ( p 1 , p 2 ) which contradicts the claim that d sp ( p 1 , p 2 ) is the infimum.
Hence we have ∀ p χ , d sp ( p 1 , p χ ) + d sp ( p 2 , p χ ) ≥ d sp ( p 1 , p 2 ) .
Multi-agent Particle Environment (MPE) [23, 47] consists of n agents and l landmarks in a 2D world.
Each agent has to resolve to select the action from its discrete action space to move around.
In practice, the experiments in Sec. 5.1 are carried out in the customed scenario SimpleTag , which is a predator-prey environment with n = 4 , l = 1 . In the original scenario, there are 1 good agent that can move faster to keep itself away from the others and 3 adversaries trying to hit the good agent. Besides, there is a obstacle blocking the way denoted by 1 landmark.
To customize the environment for URL experiments, we discard the extrinsic reward from the environment. Instead, we are concerned about the diversity of agents’ formations driven by learned coordination policies. B.2 Details about hardware and reproducibility
The hardware An AMD Ryzen 3975WX CPU with 32-Cores and three RTX-3090-11G GPUs are employed to run all the experiments with five random seeds.
As for MPE, it takes around 10 hours for
10 6 time steps to train 10 joint policies with SPD.
Besides, SPD takes around 36 hours for 4 × 10time steps in GRF to train 20 joint policies.
Code and reproducibility Our entire code 4 is opened for reproducing all the experiments.
To reproduce the results, please refer to the instruction file “README.md”. Training details We adopt QMIX [35] as our MARL algorithm to learn synergy patterns. Recurrent Neural Network (RNN) is used in the policy to alleviate the partial observability.
The mixing network has one hyper-layer as described in QMIX with 64 units.
The optimizer to optimize the neural networks is “Adam”.
Each URL algorithm is deployed to learn different joint policies ( Z =for MPE and Z = 20 for GRF) and mixing networks every time. We summarize most of the hyper-parameters for the two experiments in Sec. 5 in Table. C Visualization for learned synergy patterns in GRF To comprehensively illustrate the diversity of synergy patterns learned by SPD, we directly visualize the Z = 20 learned joint policies in GRF.
During this procedure, The agents 1 , 2 , 3 are controlled by the learned policies and the opponents are built-in AI. Fig. 5 shows the position trajectories of the agents and the opponents by different learned joint policies. The results demonstrate that the agents can learn useful synergy patterns, such as dribbling ( z = 1 , 2 , 3 , 7 , . . . ), collaborating to maintain the formations ( z = 4 , 8 , 18 ), and passing-and-shooting ( z = 6 ), with only pseudo-reward from SPD.
[Table]
D Discrimination of proposed concepts
We summarize the concepts mentioned in this work in Table 3 for a clearer understanding.
E Evaluation on SMAC
We also assess SPD on SMAC [37] to demonstrate the efficacy of the learnt joint policies and offer a more thorough perspective of the effect of the URL training.
As for our method SPD, we follow the same URL training procedure in Sec.
5.2 and use QMIX as our training method as well.
The
Z = 20 pre-trained policies are learned by QMIX on the same maps as the downstream tasks without external reward but only pseudo-reward provided by SPD for 10 6 time steps, and are tested before
MARL training on the downstream tasks to choose parameters as the initialization.
During the MARL training, each algorithm is carried out with 5 random seeds.
We evaluate each algorithm on three maps: 3m , 2m _ vs _ 1z and 3s5z _ vs _ 3s6z (super-hard).
The results are shown in Fig.
On the map 2m _ vs _ 1z with normal difficulty, SPD performs similarly to the baseline QMIX, while it learns more efficiently compared to QMIX on map 3m .
We believe the fact that these two maps can be explored by ϵ -greedy strategy sufficiently and do not need well-performed team behaviour may account for this.
Since we use QMIX as our training algorithm, the performance is limited by it and is surpassed by CDS and QPLEX.
However, SPD significantly outperforms the baseline QMIX and reach similar performance as CDS on the super-hard map 3s5z _ vs _ 3s6z , which demonstrates that SPD do learn the relationship of agents and encourage team behaviour.
Such results suggest that the role of SPD is more pronounced in tasks where there is a greater need for cooperation.
In addition, combining exploration-based algorithms, such as CDS, with SPD may achieve better performance and foreshadow future research directions.