SPD: Synergy Pattern Diversifying Oriented Unsupervised Multi-agent Reinforcement Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Reinforcement learning typically relies heavily on a well-designed reward signal,which gets more challenging in cooperative multi-agent reinforcement learning.
Alternatively, unsupervised reinforcement learning (URL) has delivered on itspromise in the recent past to learn useful skills and explore the environmentwithout external supervised signals.
These approaches mainly aimed for the singleagent to reach distinguishable states, insufficient for multi-agent systems due tothat each agent interacts with not only the environment, but also the other agents.
We propose Synergy Pattern Diversifying Oriented Unsupervised Multi-agent
Reinforcement Learning (SPD) to learn generic coordination policies for agentswith no extrinsic reward.
Specifically, we devise the Synergy Pattern Graph (SPG),a graph depicting the relationships of agents at each time step.
Furthermore, wepropose an episode-wise divergence measurement to approximate the discrepancyof synergy patterns .
To overcome the challenge of sparse return, we decomposethe discrepancy of synergy patterns to per-time-step pseudo-reward.
Empirically,we show the capacity of SPD to acquire meaningful coordination policies, suchas maintaining specific formations in Multi-Agent Particle Environment and passand-shoot in Google Research Football.
Furthermore, we demonstrate that thesame instructive pretrained policy’s parameters can serve as a good initializationfor a series of downstream tasks’ policies, achieving higher data efficiency andoutperforming state-of-the-art approaches in Google Research Football.
Introduction
Deep reinforcement learning (RL) has exhibited excellent capabilities in tackling complex tasks,including Go [40, 36], poker [27] and simulated robotic tasks [37, 13].
However, one of the keychallenges in deploying RL algorithms is to devise an elaborate reward signal for eliciting desiredbehaviour of agents [14].
Due to the sharing of the team reward, the situation gets more complicatedfor cooperative multi-agent reinforcement learning (MARL), especially in real-world applicationssuch as coordination of robot swarms [16] and autonomous vehicle teams [52], where verifying theeffectiveness of the designed reward can be prohibitively costly.
As for the single agent, unsupervised learning has been incorporated into RL to acquire diverseskills for the agent without extrinsic reward from the environment, and this scenario is known asunsupervised reinforcement learning (URL).
Recent advances in URL [10, 15, 20, 31, 38] havedemonstrated the potential of the agent to explore the environment and learn useful skills withno supervision.
Moreover, when endowed with the task-related extrinsic reward in downstreamtasks, these skills exhibit significant generalizability, leading to efficient adaptation.
An importanthypothesis explaining the phenomenon is made by DIAYN [10] that the higher the coverage oflearned skills over the set of possible behaviour, the greater the probability of gaining effective skills
Above mentioned URL approaches mostly depict the diversity of skills by the diversity of visitedstates, however, insufficient for MARL due to the neglect of interactions with other agents.
Intuitively,this coordination “skill” in MARL corresponding to the skill in single-agent URL should be takeninto consideration as well.
We term the coordination “skill” as the Synergy Pattern , which isthe perennial coordinated behaviour of agents induced by interactions of joint policies and theenvironment.
In retrospect, the coordination of agents are usually modeled by graphs , such ascoordination graphs [12, 4, 50] and graph neural networks [17, 21].
Nevertheless, these graph-basedapproaches are mainly tasked for communication and deploying the graphs directly to unsupervised
MARL can be challenging.
We diversify the synergy patterns of agents by bridging the graphs and unsupervised MARL.
Specifically, we devise the Synergy Pattern Graph (SPG), in which there is a vertex for each agent and theweight of the edge is derived from the function depicting the relations of agents, as the per-time-stepinstance of the synergy pattern.
Our method Synergy Pattern Diversifying Oriented Unsupervised
Multi-agent Reinforcement Learning (SPD) focuses on obtaining diverse synergy patterns whilefollowing the same hypothesis in URL.
To that end, we approximate the Discrepancy of Synergy
Patterns by quantifying the divergence between two SPG batches, which are sampled from thedistributions of different synergy patterns respectively.
The first step is to measure the discrepancybetween two SPG elements.
This problem is essentially to measure the similarity of graphs, which hasbeen well-studied in the field of graph matching[39, 26, 32, 9], and one of the promising approachesis the Gromov-Wasserstein discrepancy [6, 51].
With the adoption of the Gromov-Wassersteindiscrepancy as the pair-wise measurement, we then formulate the discrepancy of synergy patterns asan episode-wise measurement and deploy it as the pseudo episode return for unsupervised MARL.
Our contributions for SPD, a general framework for unsupervised MARL, are summarized as follows:i)
we
devise
SPG
with
a general formulation as the instance to depict the synergy patterns of agents;ii) we propose an episode-wise divergence measurement to estimate the discrepancy of synergypatterns; iii) we derive the formulation for the decomposition of the pseudo episode return to alleviatethe problem of the sparse return; iv) The empirical results demonstrate that SPD achieves betterperformance in MARL compared to conventional URL approaches and shows great potentials tolearn synergy patterns with generalizability for downstream tasks.
Related Works
Multi-agent Reinforcement Learning (MARL) has shown great potential in solving multi-agentcooperation tasks.
A popular paradigm in MARL is the centralized training with decentralizedexecution (CTDE) that combines the advantage of independent Q-learning [44] and joint actionlearning [7].
Centralized training makes agents cooperate better while decentralized executionbenefits the flexible deployment capability.
We mainly focus on the value-based CTDE methods.
Following the CTDE paradigm, a series of works concentrate on distributing the team reward to allagents by value function factorization [43, 34], and deriving the Individual-global-max principle forpolicy optimality analysis [42, 46, 35].
Another bunch of works introduce roles or skills to agents forbetter team cooperation [24, 48, 5].
However, these methods usually suffer from poor generalizationability and have considerable performance drop when transferred to unseen scenarios.
Wang et al.[49] reports agents’ generalization by splitting the action space, but the method relies heavily onthe handcrafted hyper-parameter.
The concept of agents’ forming a graph is also widely used torepresent agents’ cooperation in MARL, including the coordination graph [4, 50] and graph neuralnetworks [17, 21].
But these methods are usually designed for communication, which brings extracosts for decentralized execution.
In contrast to these methods, our proposed method uses the synergygraph to train team skills without rewards from the environment, which brings the agents stronggeneralization ability in downstream tasks, and takes no extra cost in execution.
Unsupervised Reinforcement Learning (URL) has been particularly intriguing in the context of
RL, consider the high cost of real-world rollouts with delicately designed reward signal.
This paperis concerned about unsupervised behaviour learning for deep RL that requires learning with onlythe task-agnostic intrinsic reward.
Early works [11, 1] mainly focuses on controlling the visitedstates via regulating a latent variable Z , where the mutual information is maximized with visitedstates.
DIAYN [10] demonstrates the exceptional performance of URL by maximizing I ( Z ; S ) whilesimultaneously minimizing I ( Z ; A | S ) .
The recent studies have also shown interests on adaptingpre-trained skills on MDPs from aspects of the initial state [3] and the dynamics model [38].
Besides,
WURL [15] adopts the idea of optimal transport and increases the discrepancy of learned skillsby enlarging the distance of visited state distributions estimated by Wasserstein distance.
Theseapproaches aiming for achieving easily distinguishable states, however, are deficient to solve thegrowing size of joint observation space in multi-agent systems.
By contrast, our method devise agraph to depict the relative relations between agents which alleviate the growing size challenge.
Thereare also works encouraging diversity for effective exploration [19, 23, 30, 53], while they mostlyenhance single policy’s diversity or the inter-agent diversity and do not get rid of the dependence ofthe task-related reward We follow a similar hypothesis to that in Eysenbach et al.
[10] to learn diversesynergy patterns for gaining useful coordination policies.
Preliminaries
[Figure]
3.1 Notation
We consider a Dec-POMDP [29] for the fully cooperative multi-agent task with n agents, defined bya tuple ⟨S , A , P , R , O , I , Ω , n, γ ⟩ , where s ∈ S and
a i ∈
A denote the true state of the environmentand the discrete action chosen by agent i ∈
I ≡ { 1 , . . . , n } , respectively.
γ ∈ [0 , 1) is thediscount factor.
The dynamics of the environment can be formulated as the state transition function
P ( s ′ | s, a ) : S ×
A × S → [0 , 1] and the reward function R ( s, a ) : S ×
A → R shared by all agents,where a ∈ A ≡ A n denotes the joint action.
Each agent i only has access to its own observationo i ∈ O derived from the observation function Ω( s, i ) :
S × I → O , due to the partial observability .
At each time step t , agent i resolves to execute the action a i according to a stochastic policy π i ( a i | τ i ) :
T ×A → [0 , 1] , where τ i ∈ T ≡ ( O × A ) ∗ denotes the action-observation history of the agent i .
Thegoal of MARL is to find an optimal joint policy πππ ∗ ≡ (cid:81) ni
=1 π i ∗ satisfying the condition: Q πππ ∗ ( s, a ) ≥
Q πππ ( s, a ) , ∀ πππ and ( s, a ) ∈ S × A , where Q πππ ( s t , a t ) =
E s t +1: ∞ , a t +1: ∞ (cid:2)(cid:80) ∞ i =
γ i r t
+ i | s t , a t (cid:3) is thejoint action-value function induced by joint policy πππ .
We adopt the paradigm of centralized training with decentralized execution (CTDE).
During trainingthe true state s and the joint action-observation history τττ ≡ { τ i } ni =1 are accessible to the learningalgorithm, while each agent i selects an action based on the policy conditioning only on its ownaction-observation history
τ i during testing (execution).
3.2 Optimal Transport and Gromov-Wasserstein Discrepancy
Optimal transport (OT) is a general problem of moving one distribution of mass to another as efficiently as possible.
Likewise, the distinction of two given graphs can be regarded as an optimaltransport problem to find the correspondence between their vertices and edges.
Previous work [6] proposed Gromov-Wasserstein Discrepancy , which is extended from Gromov-Wasserstein distance [25]as a measure for the divergence of the graphs.
Definition 3.1.
[6, 51] (Gromov-Wasserstein discrepancy)
Denote a measure graph as G ( V , WWW ,µµµ ) ,where V ≡ { v i } |V| i =1 is the set of vertices, ω ij ∈ WWW is the weight of the undirected edge { i, j } betweenvertices v i , v j ∈ V , and µµµ =
[ µ i ] ∈ Σ V is a Borel probability measure defined on V .
∀ p ∈
[1 , ∞ ]and each G ν , G ς ∈ G , where G denotes the collection of measure graphs, the Gromov-Wassersteindiscrepancy between G ν and G ς can be derived asd gw ( G ν , G ς ) :
= min T ∈ Π( µ ν , µ ς )   (cid:88) i,j ∈V ν (cid:88) i ′ ,j ′ ∈V
ς (cid:12)(cid:12)(cid:12) ω νij − ω ςi ′ j ′ (cid:12)(cid:12)(cid:12) p T ii
′ T jj ′ 
 1 p , (1) where Π ( µ ν , µ ς ) =
(cid:8) T ≥ 0 | T 1 |V ς | = µ ν , T ⊤ 1 |V ν | = µ ς (cid:9) .
In general, the Gromov-Wasserstein discrepancy measures the divergence between two graphs bycomparing the differences of the edges’ relationship in one graph to that in the other.
We refer thereader to Mémoli [25], Chowdhury & Mémoli [6], Peyré et al.
[33] for mathematical foundations.
Methodology
This section is organized as the following: In Sec. 4.1, we delineate the concept of Synergy Patternand propose a novel structure of graph as the instance of it, namely Synergy Pattern Graph .
We thencarefully derive the formulation for Discrepancy of Synergy Patterns and make an approximation ofit for convenience in practice in Sec.
In Sec. 4.3, our unsupervised framework SPD is depicted indetail and the mechanism for the decomposition of the pseudo reward is explained.
Sec.
4.4 providesthe implementation details for solving the optimization problem.
4.1 Synergy Pattern
With the awareness of interactions between agents’ actions, we describe the perennial coordinatedbehaviour of agents by the Synergy Pattern .
Previous methods [12, 4, 17, 21] utilize the structure ofgraph to depict it, whereas most of them focus on the local relationship between agent pairs towardsa specific task.
To get rid of the extrinsic reward signal, we propose Synergy Pattern Graph (SPG),denoted as G sp ( V , WWW ,µµµ, ζ ) , where v i ∈ V is the vertex for agent
i ∈
I and the weight ω ij ∈ WWW ofedge { i, j } is derived from a synergy pattern function which could depict agents’ relative relations:
[Equation]
The synergy pattern function follows a general formulation as any function with range R +0 , forinstance, the information-theoretic influence [47]: ζ ( τ i , τ j ) :
= MI πππj | i ( o ′ j ; o i , a i | o j , a j ) or the communication messages [4].
It can depend on more than merely the observations and joint actions aswell, whereas we omit this for simplicity.
At each time step, SPG varies depending on the historyof the multi-agent system rather than being predefined before training.
We will use G t to denote
G sp ( V , WWW ,µµµ, ζ ) at time step t in the following part of this paper for ease of notation.
Accordingly, the synergy pattern can be described by the distribution of SPG obtained by deploying ajoint policy πππ in the environment: P ( G πππ | πππ ) with domain Λ , where Λ is the collection of all SPG.
4.2 Discrepancy of Synergy Patterns
We use the discrepancy between two distributions of SPG: P ( G πππ 1 | π 1 π 1 π 1 ) and P ( G πππ 2 | π 2 π 2 π 2 ) to denote thediscrepancy between two synergy patterns.
This process can be decomposed into two phases: i)measuring the distance between two samples from each distribution respectively; ii) extending themetric for a pair of samples to that for distributions.
The former one is essentially to estimate thediscrepancy between graphs.
We adopt the Gromov-Wasserstein discrepancy of graphs, as mentionedin section 3.2, to measure this distance.
Besides, we solve the latter one by applying optimal transporton SPG distributions.
This whole process leads to the following definition:
Definition 4.1.
(Discrepancy of Synergy Patterns) Let p 1 = P
( G πππ 1 | π 1 π 1 π 1 ) and p 2 = P
( G πππ 2 | π 2 π 2 π 2 ) be twodistributions of SPG induced by joint polices πππ 1 and πππ 2 , respectively.
Denote an joint distributionϱ ∈ Γ[ p 1 , p 2 ] as a bijective plan transporting probability mass from p 1 to p 2 , where Γ[ p 1 , p 2 ] is theset of all distributions on product space Λ × Λ with p 1 and p 2 as their marginal probabilities.
Thenthe discrepancy of synergy patterns is defined as
[Equation]
where d gw is the Gromov-Wasserstein discrepancy according to Eq.
Besides, we show that d spis a legitimate pseudometric on Λ and the proof is deferred to Appendix A.
However, solving Eq.
(2) for the discrepancy of synergy patterns is almost intractable.
In practice,we collect two SPG batches to approximate the discrepancy of synergy patterns.
Here, we give thedefinition for this approximation:
Definition 4.2.
(Proximal Discrepancy of Synergy Patterns)
Denote ¯ λ πππ 1 = { G πππ 1 t } B 1 t =0 ∼ P ( G πππ 1 | π 1 π 1 π 1 )and ¯ λ πππ
=
{ G πππ 2 t } B 2 t =0 ∼ P ( G πππ 2 | π 2 π 2 π 2 ) sampled from the distributions of SPG, where B 1 , B 2 ∈ N + .
Assume B 1 ≤ B 2 by symmetry.
Let D B 1 × B 2 = (cid:2) d gw ( G πππ 1 t 1 , G πππ 2 t 2 ) (cid:3) , ∀ 0 ≤ t 1 ≤ B 1 , 0 ≤ t 2 ≤ B 2 be the cost matrix.
Suppose that an injective function ρ : { 1 , . . .
, B 1 } → { 1 , . . .
, B 2 } denotes anmatching function.
By minimizing the total cost, an approximation for the discrepancy of synergypatterns is:
[Equation]
It is vital to point out that the SPG from ¯ λ πππ 2 corresponding to G πππ 1 t may not be G πππ 2 t since the relationsof agents may be analogous at different time steps.
Hence, we search the whole ¯ λ πππ 2 for G πππ 2 ρ ( t ) that isrelatively adjacent to G πππ 1 t based on the Gromov-Wasserstein discrepancy.
4.3 Synergy Pattern Diversifying Oriented Unsupervised Multi-agent ReinforcementLearning
Based on the discrepancy of synergy patterns, we can now delineate the framework for SPD.
Supposethat Z ∈ N + denotes the number of different synergy patterns.
Our goal is to learn a set of jointpolicies { πππ z } Zz =1 with diverse synergy patterns to explore the environment sufficiently when there isno extrinsic reward.
To increase the divergence among synergy patterns, we formulate the pseudoepisode return for the episode data τττ z produced by the joint policy πππ z :
[Equation]
(4) where 1 ≤ z ≤ Z , and ¯ λ πππ z is the SPG set derived from τττ z .
Besides, we obtain ¯ λ πππ k by sampling abatch of data from the replay buffer of πππ k and ensure ¯ λ πππ k have more elements: | ¯ λ πππ k | ≥ | ¯ λ πππ z | .
Our method is to maximize the expected return: max πππ z E [ ˜ R z ( τττ z , { τττ k } )] , resulting in the maximization of each discrepancy of one synergy pattern to its nearest counterpart.
However, the pseudoepisode return in Eq.
(4) depending on the whole episode data τττ z is sparse since it only derivesa single final reward for one episode.
This imposes the trouble for MARL algorithms to learn anaccurate action-value function Q πππ z ( s, a ) and tends to impair the performance of value-based MARLalgorithms.
To alleviate this predicament, intuitively, the pseudo episode return ˜
R z ( τττ z , { τττ k } ) can bedecomposed into pseudo rewards for every time step according to the optimal matching function ρ ∗ .
Concretely, the pseudo reward at time step t can be formulated as:
[Equation]
(5) where ρ ∗ is the optimal solution to ¯ d sp (¯ λ πππ z , ¯ λ πππ k ) by Eq.
(3), and k ∗ denotes the id of the synergypattern with minimal discrepancy to ¯ λ πππ z according to Eq.
In other word, the pseudo reward is the
Gromov-Wasserstein discrepancy of the SPG at current time step and the SPG from the counterpartof another synergy pattern.
We discriminate the concepts we proposed in Appendix F for a clearer understanding.
The visualization for the pseudo reward derivation is shown in Fig.
4.4 Implementation
We implement SPD with QMIX [34], learning a set of joint policies that maximize the discrepancyof synergy patterns to each other.
QMIX is designed to have a mixing network structure, learninga global action-value function Q :=
{ Q tot | Q tot ( τττ, a ) =
f s (cid:0) Q 1 ( τ 1 , a 1 ) , . . .
, Q n ( τ n , a n ) (cid:1) , ∂f s ∂Q i
≥0 , Q i ( · , · ) ∈ R } which satisfies the IGM principle [42], to accomplish the cooperative task for agents.
During unsupervised learning, we select 1 ≤ z ≤ Z for the synergy pattern at the start of each episoderandomly or in turn, and interact with the environment according to the joint policy πππ z throughout theepisode.
In practice, the Euclidean distance of o i and o j is adopted as the weight ω
ij of edge { i, j }to build the SPG, i.e., the formulation of synergy pattern function is ζ ( τ i , τ j ) =
∥
o i − o j ∥ 2 .
Wewill show in Sec.
5 that, even with such a synergy pattern function to grab the simple information ofrelationships of agents, SPD can still learn useful synergy patterns such as pass-and-shoot in Googlefootball Research and accelerate the learning procedure when endowed with the task-related reward.
Since solving for the Gromov-Wasserstein discrepancy between two measure graphs in Eq.
(1) is acomputationally costly process, we adopt [51]’s
Regularized Proximal Gradient method to estimate it.
We set p = 2 in Eq.
(1) in our experiments.
As for the probability measure, we set µµµ = 1 n 1 n sincethe agents are assumed to have equal contributions, where n is the number of agents or vertices insynergy patterns and 1 n is a size n vector of ones.
In the m -th iteration, the current optimal transport
T ( m ) can be updated by
[Equation]
) where L ( WWW ν , WWW ς , T ( n ) )
=
WWW ν µµµ ν 1 ⊤|V ς | + 1 |V ν |
µµµ ⊤ ς WWW ⊤ ς − 2 WWW ν
T WWW ⊤ ς , and Kullback-Leibler (KL)divergence is added as the proximal term.
⟨· , ·⟩ denotes the inner product of two matrices andα is a hyper-parameter to balance two items.
The Eq.
(6) can then be solved with nearly-linearconvergence [2] by the Sinkhorn-Knopp algorithm [41, 8].
As soon as T ( m ) converges to a stableoptimal transport ˆ T , the estimation of the Gromov-Wasserstein discrepancy can be calculated byplugging ˆ T into Eq.
To solve Eq.
(3), we utils Kuhn–Munkres algorithm [28], which could solve such an assignmentproblem in polynomial time, to obtain the optimal matching funciton ρ ∗ .
Therefore, the completeformula for pseudo reward at time step t is as below:
[Equation]
   , (7) where G πππ z t ∈ ¯ λ πππ z , G πππ k ∗ ρ ∗ ( t ) ∈
¯ λ πππ k ∗ , and k ∗ denotes the id of the synergy pattern with minimaldiscrepancy to ¯ λ πππ z according to Eq.
The whole framework is summarized in Algorithm 1.
Experiments
We first compare SPD’s properties with other URL baselines on Multi-agent Particle Environment (MPE) [22] in Sec. 5.1, and in Sec.
5.2 we evaluate SPD’s generalization ability on downstreamtasks on Google Research Football (GRF) [18].
5.1 Synergy pattern diversifying
We demonstrate that our method SPD is capable of capturing more inherent properties of multi-agentsystems such as the impacts from other agents, comparing to conventional URL algorithms.
In order described in
Obtain
Calculate
24: end to verify this claim, we first evaluate the diversity of coordination policies learned by SPD and URLbaselines in Multi-agent Particle Environment 1 [22, 45].
We customize the scenario SimpleTag , whichis a predator-prey environment, for URL experiments by discarding the extrinsic reward from theenvironment.
Instead, we are concerned about the diversity of agents’ formations driven by learnedcoordination policies.
The details of the scenario please refer to Appendix B.1.
Baselines and Evaluation We compare our method SPD with several conventional single-agent
URL approaches, including DIAYN [10] and WURL [15].
We defer the detailed description of thesemethods to Appendix B.2.
The URL algorithms are implemented with QMIX to learn diverse jointpolicies { πππ z } Zz =1 , and we set Z = 10 in practice.
The hyper-parameters are kept to be the same, andplease refer to Appendix B.1 for details.
For evaluation, each algorithm is carried out with 5 differentrandom seeds and trained with 4 × 10 6 steps interacting with the environment.
All the intrinsic pseudorewards are derived based on the positions of agents which is a part of the original observation space.
[Table]
Since DIAYN and WURL are designed for thesingle agent case, we deploy them by regardingall agents as a single agent.
Concretely, we useobservations from all agents to create the featuresthat DIAYN and WURL need.
We evaluate these algorithms following the sameprotocol: (i) collect sample transitions except extrinsic reward using joint policy πππ z ( a | τττ z ) , (ii)calculate the pseudo-reward r z for each time stepand assign them to the trajectory τ τ τ z , (iii) employlearned joint policies once the training procedureis completed to collect { τττ z } Zz =1 for evaluation.
We use two metrics to evaluate how diverse are the learned joint policies from different aspects: the diversity of states and the discrepancy among relative relationships of agents induced by { πππ z } Zz =1 .
As for the former one, we adopt the structure of the discriminator q ϕ
( z |{ o i } ni =1 ) from DIAYN, whichis a learned neural network to classify each policy πππ z from state samples in { τττ final z } Zz =1 , and use thesuccess rate of it to measure how diverse are the states, namely Discriminator Success Rate (DSR).
As discussed in Sec.
4.2, the discrepancy of synergy patterns d sp according to Eq.
(2) is a effectivemeasure for the latter one.
Table 1 shows that the performance of all algorithms on DSR is close,whereby SPD significantly outperforms the others on the discrepancy of synergy patterns.
Such resultsdemonstrate that SPD can inspire agents to visit comparably identifiable states as the conventional
URL approaches while further encouraging agents to explore with other types of coordination policiesin the meanwhile.
Ablation We examine the effect of each component of SPD in Fig.
We use ¯ d sp as the evaluating indicator and the detailed description is deferedto Appendix B.1.
The curves start from the timestep that the exploration rate epsilon has stopped annealing.
We can summarize that the reward scaleplays an important role in learning good synergy patterns.
Canceling the optimal transport for matchingor decreasing its accuracy both harm the performance.
And the “sparse return” curve shows that amortizingthe episode return to each time step simplifies thetraining.
5.2 Generalizability of learned synergy patterns
In this part we demonstrate the generalization abilityof the models trained by SPD.
Specifically, we firsttrain SPD on the complicated MARL environment:
Google Research Football [18] without environmentreward.
Then we use the trained model as the initialization for different tasks with specific rewards, anduse common MARL algorithms to train it.
Task
Description We choose the map“academy_3_vs_1_with_keeper” with the setting from CDS [5], where three players need tocooperate to shoot and get a score under the defenseof an opposing player and a goalkeeper.
To simplify the training, the original environment ends anepisode if the ball or the player return to the left half-court.
Players always start at the same place.
Losing the control of the ball also cause the failure.
The team is rewarded 100 at the end of anepisode when there is a goal, otherwise − 1 .
On the basis of this scenario which we call “Half-court;
Fixed Init”
, we design a series of tasks with more difficulty.
(i)
We initialize the agents randomly in acircle with 5% court width’s radius, which increases the difficulty by introducing more randomness.(ii)
We remove the restrictions on the left-half court, which greatly increases the exploration space.(iii)
The possession change will not end the episode, which may increase the percent of the badsamples in the replay buffer, since at the beginning of training the ball is easily robbed by the preset
AI opponent.
Utilizing SPD model We first train the models by SPD with Z = 20 on the “Half-court; Possessioncan change” scenario without the environment reward for 10 6 time steps, as well as WURL [15] and
DIAYN [10].
Then we use the same trained parameters as the initialization for QMIX [34] on fourdifferent downstream scenarios.
In contrast to the manual selection of the synergy patterns, we testthem on the downstream task before MARL training, choosing the parameters of the model with thehighest average return as the model initialization.
We show the results in Fig.
3, together with the
SOTA on GRF including QMIX [34], QPLEX [46] and CDS [5].
The results show that using themodel learned by conventional single-agent URL methods (WURL and DIAYN) as the initializationmostly performs similarly to the baseline QMIX, while WURL slightly outperforms QMIX on the“Full-court” maps.
In contrast, SPD learns faster and finally gets higher winning rate on all scenarios, demonstrating the effectiveness of d sp as an evaluation metric for the difference of relationship ofagents and showing that SPD embeds the model with stronger generalization ability.
We believe thegap between the improved performances of SPD and the other URL approaches is caused by thefact that SPD does capture more information than the single-agent URL approaches in multi-agentsettings, which is the relationship of agents since it is designed for this.
[Figure]
CurrentPosition
[Figure]
Position
AgentMovement
[Figure]
Visualization of the learned synergy patterns To comprehensively evaluate SPD, we visualize 2the trajectories of all the skills and show three of them in Fig.
The trajectories of other skills canbe found in Appendix A.3.
We find that agents can perform some meaningful behavior from a humanperspective, even without the supervision of the environment reward.
Agents can conduct the ball,pass between teammates, shoot, and run back together in different skills.
To our concern that thedifferent performance of different joint policies learned by SPD are only due to the randomness ofthe GRF environment itself but not the inter-policy diversity we want, we deploy each learned jointpolicy for 10 episodes and visualize them.
The results show that each joint policy exhibits the similarcoordination behavior across 10 episodes.
The visualization shows that making the team coordinatemore diverse drives agents to learn meaningful behaviour, which is favorable for downstream tasks.
Conclusion
In this work, we propose a general unsupervised framework Synergy Pattern Diversifying Oriented
Unsupervised Multi-agent Reinforcement Learning (SPD) to learn useful synergy patterns for agents.
Synergy pattern graph (SPG) is devised to depict the relationship among all agents.
Furthermore,we approximate the discrepancy of synergy patterns by applying optimal transport on the batchessampled from the distributions of SPG
In our experiments we show that SPD surpasses conventional
URL approaches on capturing the relative relations of agents and learns useful synergy patterns withno task-related reward.
We also empirically demonstrate that the learned policies can accelerate the downstream training procedure and outperform state-of-the-art algorithms in GRF
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] .
(b) Did you describe the limitations of your work?
[Yes] See Appendix A.5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix A.6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] .
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] .
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.1.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We submit the whole code.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] .
(b) Did you mention the license of the assets?
[N/A] .
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] .
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] .
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] .
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable
[Table] [Table]
N
[Table]