Large Scale Coordination Transfer for Cooperative Multi-Agent Reinforcement Learning
Abstract
Multi-agent environments with large numbers of agents are difﬁcult to solve due to the complexity associated with drawing sufﬁcient samples for learning.
While recent work has addressed the possibility of using transfer learning to improve sample complexities of reinforcement learning algorithms, methods for transferring knowledge in multi-agent domains across differing numbers of agents have rarely been considered.
To address the bottleneck with sampling from large scale environments, we propose a joint critic structure motivated from graph convolutional networks and coordination graphs that allows for the direct transfer of parameters into environments with varying amounts of agents.
We further consider ﬁne-tuning the transferred policy and critic networks on the target domain and provide the motivation for doing so in cooperative environments where agent behavior is determined by a subset of the total population.
Finally, we provide empirical results validating our claims on such environments, including popular multi-agent benchmark environments.
Introduction
Data collection for a wide variety of multi-agent environments relies on computationally expensive simulations, or in scenarios such as autonomous driving, health informatics, and robotics coordination, potentially hazardous experiments.
As such, the sample complexity of reinforcement learning algorithms remains a prominent bottleneck against their applications in these systems.
Since the state-action space must grow exponentially with the number of agents, popular algorithms can suffer from a curse of dimensionality termed the curse of many agents [12] reﬂected by diminishing sample efﬁciency as the number of agents increases.
Mean-ﬁeld approximation methods [27, 46, 50] as well as factorized policy [14] and value representations [39] have been proposed to counteract the exponential growth of the state-action space; however, the high computational cost of sampling from an environment with many agents and the possible consequences of iterating on real-world scenarios are still signiﬁcant barriers against the use of many algorithms.
We seek to utilize transfer learning , a well-studied ﬁeld in the domain of supervised learning [32], to reduce the need for samples in cooperative large scale environments using related environments of smaller scale.
In the realm of reinforcement learning, transfer learning can take the form of reward shaping [2, 9, 45], where external knowledge of the reward function for the target environment is used to guide policy learning, or learning from demonstrations [17, 22], where expert demonstrations are used to facilitate exploration.
Here, we propose a method that is most akin to few shot learning , where a model must learn to adapt to a target task T while training on a dataset with only a limited number of examples for T [47].
In particular, the external knowledge being transferred in this work takes the form of pre-trained policy networks and value function approximators.
Contributions: The contributions in this paper are threefold.
First, we propose a joint critic structure that allows for the direct transfer of pre-trained parameters (Subsection 3.1).
Commonly, the input of these critics is either the joint state or joint state-action pair, and this scales with the number of agents, preventing model parameters from being transferred between environments with varying number of agents.
Secondly, we consider the possibility of ﬁne-tuning (Subsection 3.2), such as in the few shot learning paradigm, across domains with differing numbers of agents using decentralized execution, a method that has not been addressed for multi-agent reinforcement learning to the best of our knowledge.
Finally, we demonstrate empirically (Section 4) that this type of parameter transfer can produce the best performance on environments where individual agent behaviour is dependent on a subset of the global population.
While general environments may not follow this structure, we validate our results on popular multi-agent benchmark environments.
1.1 Related Work
Single-Agent Methods: For single-agent environments, policy transfer refers to the use of pretrained teacher policies from a set of source domains E s that provide knowledge about some target domain E τ .
Policy distillation then utilizes these teacher policies, in the case of [36, 51] pre-trained on multi-task domains such as Atari 2600 games, to learn a student policy by minimizing the divergence of its action distribution with respect to the teachers’.
The transfer of feature representations, termed representation transfer , is used in works such as [37], where progressive neural networks use representations from previous tasks to assist in learning a new task.
Multi-Agent Methods: Intra-agent transfer techniques reuse knowledge generated by an agent for new domains without requiring an explicit communication channel [6].
The authors of [5] propose a method to calculate a task mapping in an object-oriented manner, transferring value functions between tasks using this description.
For environments with sparse interactions , where other agents effect a local agent only in certain portions of the environment, works such as [18, 58] treat the majority of the state space as a single-agent problem, reusing value functions trained on similar tasks.
Similarly, the authors of [56] naturally reuse policy networks on similar road networks by using zero-shot transfer performance of learned policies as a metric to evaluate robustness for the trafﬁc signal optimization problem.
The majority of literature focuses on intra-agent work; however, inter-agent methods, where agents rely on explicit communication to share knowledge, are appearing more frequently [6].
One popular pattern, action advising , gives the agents the roles of advisors, who share experience by providing action suggestions for others, and advisees, who receive suggestions and decide what action to take [7, 10, 43, 55].
Unlike this paper, none of the previously mentioned works consider the possibility of ﬁne-tuning pre-trained model parameters after transferring to environments with differing amounts of agents .
As the experiments performed in Section 4 reﬂect, oftentimes environment dynamics do not differ greatly with respect to the number of agents and thus can be treated analogously to the multi-task scenario, with each environment being a different “task”.
A similar work, [21], uses policies with graph neural network structures comparable to our proposed critic for zero-shot transfer in robotics control.
However, their method assumes that an underlying communication channel is present during execution in order to aggregate features within the swarm.
We focus instead on the centralized learning with decentralized execution paradigm, where agent policies must run independently at execution time.
Background
We formalize the problem as a Dec-POMDP (decentralized partially-observable Markov decision process) [31], deﬁned by the tuple E =
( S, { U i } n i =1 , P, r, Z, O, n, γ ) where U i is the action space for the i th agent, n is the number of agents, and r ( s t , u t ) is the shared reward function.
Here, S is the global state space, P ( s t +1 | s t , u t ) : S × (cid:81)
i U i
× S → [0 , 1] is the transition function, and each agent receives a local observation z at ∈ Z drawn from its observation function O : S × (cid:81)
i U i
→ Z .
Agents follow stochastic policies π i conditioned on observations or an observation-action trajectory τ ∈ ( Z × U i ) inducing a discounted return of G t = (cid:80) ∞ l =
γ l r t + l .
Our goal is to ﬁnd an optimal joint policy π ∗ such that the joint action value function Q π ∗ ( s t , u t ) =
E [ G t | s t , u t ] ≥ Q π ( s t , u t ) for all π .
We follow transfer learning literature and deﬁne the source domains, where we pre-train policies and critics, as E s =
( S s , { U is } ni =1 , P s , r s , Z s , O s , n s , γ s ) .
Similarly, target domains, which will be used to evaluate zero or few-shot performance, will be deﬁned as E τ =
( S τ , { U iτ } ni =1 , P τ , r τ , Z τ , O τ , n τ , γ τ ) .
2.1 Multi-Agent Reinforcement Learning
Multi-agent environments introduce challenges that are not addressed by single-agent reinforcement learning.
Two of the most prominent are: non-stationarity , where a single agent must deal with the moving target problem as the transition function is conditioned on the actions of other agents, and credit assignment , where an individual agent is unable to determine the impact of their action since agents receive the joint reward.
Thus, as applying single-agent methods generally does not yield good results, algorithms are developed to specialize in the multi-agent case.
For example, although it is surprisingly effective on some multi-agent domains [53], the popular proximal policy optimization algorithm (PPO) [38] is known to be sample inefﬁcient [16], a feature that is especially damaging in the large scale environments focused on by this paper.
In Section 4, we will base the proposed method off of the well-known counterfactual multi-agent policy gradients (COMA) [11] algorithm.
We also use the individual actor-critic (IAC) [24] algorithm as a sanity check to demonstrate that the multi-agent methods are learning as expected.
We describe the algorithms below:
IAC: Although it is a single-agent method, independent actor-critic has demonstrated suprising effectiveness in multi-agent systems [
For a single agent a , IAC applies the policy gradient [40] to train an independent actor-critic, resulting in a (cid:0) π a ( u a | s a ) , Q π a
( s a , u a ) (cid:1) pair.
While able to maximize individual performance, the lack of a coordination mechanism means that IAC will most likely struggle to ﬁnd optimal solutions in complex multi-agent scenarios.
COMA: COMA uses a joint critic that marginalises out a single agent’s action with a counterfactual baseline to address the problem of multi-agent credit assignment.
The advantage function used evaluates the contribution of agent a by computing the average of all counterfactuals ˆ u a while keeping the actions of other agents ﬁxed.
2.2 Graph Convolutional Networks
A graph convolution network (GCN) [23] takes a feature matrix summarizing the attributes of nodes v i ∈ V of a graph G =
( E, V ) as input.
In a manner similar to the standard convolutional neural network (CNN), the GCN then extracts locally connected features and outputs a encoded node-level feature matrix by aggregating a node’s features with its neighbors’.
[23] deﬁnes a graph convolution layer as:
[Equation]
where H ( l ) is the feature matrix of the previous layer, l ,
˜
A =
A
+ I is the adjacency matrix with added self-connections,
˜ D ii = (cid:80)
j ˜
A ij , and W ( l ) is the trainable weight matrix.
˜ D − 12 ˜
A ˜ D composes the normalized graph Laplacian, which is theorized to result in a special form of Laplacian smoothing [25] and is possibly a key reason why GCNs work.
We use graph convolutional networks as a natural choice for a transferable critic structure across source and target environments.
More details are available in Subsection 3.1.
2.3 Self-Attention
The self-attention mechanism models the dependencies between different parts of a sequence.
Originally created for natural language processing, it has been used successfully in machine reading, abstractive summarization, and machine translation [52, 4, 33, 34, 28, 44].
More recently, it has been adopted for multi-agent reinforcement learning and used for communication [20] and modeling relations between groups of agents [19, 48].
Following the terminology in literature [44], we let Q , K , V be the query, key, and value matrices respectively.
Then, the attention weights are generated with respect to the input features F in as follows:
[Equation]
where √ d k is the scaling factor and W q , W k , W v are learnable weight matrices.
Large Scale Coordination Transfer
A coordination graph (CG) [13] is an undirected graph that models inter-agent behaviour by representing the locality of interactions within an environment.
Agents are represented as nodes in the graph, and an edge between two agents represents some dependence between their behaviours.
Joint actions are then encoded by the graph structure and thus can capture complex interactions between agents.
In this paper, we focus on the types of environments coordination graphs have typically been used for, where an agent’s action depends only on a subset of other agents.
We acknowledge that this may be a limiting condition of our method; however, recent work has shown that CGs can be effective in more general environments [1].
Deﬁnition 1 (Decomposable graph).
Let S ⊂ V be some cut of graph G =
( E, V ) , where | S | > 2 .
Call graph G decomposable if there exists some S such that all v ∈ V can exist in cuts isomorphic to
S .
S outlined in black.
Notice that in each case, the graph is only decomposable if cut
Above, we deﬁne decomposable graphs, a structure we exploit to underline the motivation behind our method.
Decomposable coordination graphs can represent scenarios such as robotics swarm control, trafﬁc grid optimization, and city block planning.
We consider the coordination graphs, G s =
( V s , E s ) and
G τ =
( V τ , E τ ) , for a source and target environment respectively, where | V τ | > | V s | .
If the two environments model similar dynamics, e.g. are both trafﬁc grid networks, but differ in the number of agents, we expect some portion of the larger coordination graph to be nearly isomorphic to a portion of the smaller one.
If the larger graph is then decomposable , we can ﬁnd its corresponding cut S and choose the source environment with a coordination graph most similar to S .
Given a decomposable coordination graph for the target environment, the motivation for transferring parameters between the source and target environments is clear; the optimal policy π ∗ s that maximizes the value function in the source environment Q s also maximizes the contribution of its corresponding group of agents to the global reward.
Then, under the condition that agent behaviour in the target environment only depends on neighboring agents (with structure isomorphic to S ), the mapping of π ∗ s from S to cuts of G τ will also maximize reward in the target environment.
For target environments in which a static coordination graph is not known, we instead rely on dynamic coordination graphs [26], where a learned graph structure is conditioned on current state.
Under the assumption that an agent’s actions are only inﬂuenced by its neighborhood, we expect that a source environment that envelopes this neighborhood size is sufﬁcient to capture knowledge about these localities of interaction.
Then, the arguments for transfer learning in the multi-task domain apply as the method falls under the paradigms of policy distillation and representation transfer .
Parameter sharing between agent-level policies is vital to the intuition behind either case.
In areas of a decomposable CG where the cuts isomorphic to S overlap, an agent may otherwise be expected to have two different behaviours.
For dynamic coordination graphs, no static mapping to facilitate transfer of policy networks is deﬁned, and thus, it would be unclear which policy network should transfer onto a speciﬁc agent in the target environment.
3.1 Critic Structure
We propose a joint critic structure that facilitates coordination between agents and captures locality of agent interactions within an environment.
Naturally, due to their ability to aggregate information over a deﬁned graph structure, a graph convolutional network is the clear choice to model coordination graphs.
The default implementation considers the adjacency matrix for the GCN, A , to be a coordination graph with binary edge weights.
This is generally a heuristic choice, but given that we only use static adjacency matrix representations for known decomposable coordination graphs, there will be an obvious graph structure to follow.
Figure 2 depicts the general structure for this joint critic representation; however, instead of using the self-attention block in the Learned Adjacency Matrix module, we pass an adjacency matrix in directly.
Soft Edge Weights: For cases in which the coordination graph is dynamic , we must use a learned adjacency matrix representation, such as the Learned Adjacency Matrix module in Figure 2.
Following the procedure outlined in [26], agent observations are ﬁrst passed through a self-attention block, and the resultant attention weights are then used as a soft adjacency matrix.
As maintaining differentiability would be difﬁcult if edge weights were strictly binary, this allows the module to be trained end-to-end with the rest of the critic.
The self-attention block also outputs its input features F in , used to calculate the query Q and key K matrices, which corresponds to the agent action-observations encoded by a linear layer.
F in is then used as the input to the ﬁrst graph convolutional layer.
We note that general joint critic structures receive the joint state-action pair as input and thus have variable input shape across differing numbers of agents.
The proposed critic structure instead computes the features for all agents in a single parallel batch, reducing computational cost.
This same characteristic is what allows for a direct transfer of critic parameters from the source to the target environment.
Since the input shape is constant and all agents are computed in parallel, critics for environments with varying number of agents have the same number of parameters.
The summation that appears towards the end of Figure 2 aggregates observation-action encoding across agents to produce a joint value Q ( s , a ) that is then used in training.
This type of design has been shown to be effective for multi-agent domains in the form of value decompostion networks (VDN) [39] and on set-based data in the form of deep sets [54].
Multiple Joint Critics: While this section has focused largely on the implementation of a single joint critic, the structure can easily be extended to algorithms with numerous joint critics, such as
MADDPG [30].
Since agent-level policies must share parameters, it follows that these joint critics must also share parameters.
3.2 Fine-tuning
In the supervised learning domain, ﬁne-tuning refers to the initialization of a new model on pre-trained parameters and the subsequent training on a new domain.
If the initialized model is used on the new domain without additional training, it is called zero-shot learning; however, if the model learns on a limited number of examples from the new domain, it becomes few-shot learning.
Our intuition is that there may be some differences in the larger environment which will require agents to adjust their behaviour, as speciﬁed in Section 4.
However, due to the knowledge distilled by training on the source environment, we are able to show empirically that much fewer training samples are needed to achieve similar results.
As a result, we claim that the ﬁne-tuning method can be viewed as few-shot learning.
Algorithm 1 Multiagent Large Scale Coordination Transfer with Fine-Tuning
Require: T s iterations on E s , T τ iterations on E τ , a policy gradient multi-agent reinforcement learning method RL that uses our proposed critic structure
Initialize: Policy parameters θ s and critic parameters φ s .
while t < T s do :
Train agents on the source environment: θ s , φ s = RL ( θ s , φ s , E s ) end while
Transfer: Initalize policy parameters θ τ = θ s and critic parameters φ τ = φ s while t < T τ do :
Train agents on the target environment: θ τ , φ τ = RL ( θ τ , φ τ , E τ ) end while
Evaluate: Measure performance of π θ τ on E τ .
For the base transfer procedure without ﬁne-tuning, we follow Algorithm 1 while skipping training on the target environment E τ .
This may be preferable in cases where environment dynamics do not change when more agents are introduced or when the target environment is extremely expensive to sample from.
Results
We present experiments on three environments: park lights, trafﬁc junction [49], and cooperative navigation [30].
All environments require inter-agent coordination to receive high rewards as evidenced by the performance of the baseline methods.
For each case, we choose the source environment to be the smallest non-trivial subset of agents in the target environment where key environment dynamics are captured.
For example, cooperative navigation requires n = 3 due to the fact that we observe 2 other agents in the observation space, and trafﬁc junction must have 4 agents ( 2 × 2 ) so that each possible conﬁguration for vehicle turns exist.
Environment hyper parameters and minor implementation details can be found in Appendix A.
Table 1 depicts the simulation time of a hundred steps in the corresponding environment.
While these type of explicit computational costs are not often shown in literature due to their dependence on factors such as hardware, cluster usage, and implementation, we include this table to illustrate the importance of maximizing knowledge transferred from the source environment.
All times refer to the same implementation on the same hardware (Intel Xeon Silver 4116 CPU and NVIDIA GeForce
RTX 2080 Ti), and overall cluster usage is kept constant to the best of our ability.
In particular, we observe that cooperative navigation has superlinear simulation times with respect to the number of agents.
This is because it follows a particular paradigm possible for multi-agent settings: the nearest neighbor observation space, where agents receive the observations of their closest neighbors as well as their own.
The related problem of constructing a k-NN graph has been shown to be naively quadratic in the number of agents [3] and even with state-of-the-art optimizations, subquadratic in the worst case [15].
Thus, we claim that zero-shot performance is especially important in this environment; limiting the number of samples needed from the target environment will drastically reduce computational costs. [Table]
For each environment, we compare the same implementation of COMA on two different critics: our proposed architecture, labeled with (GCN) , and a standard multi-layer perceptron.
The ﬁne-tuning procedure outlined in Algorithm 1 is labed with (Fine Tune) .
We establish IAC and IAC (Fine
Tune) as the baseline methods for comparison, given that there is no central coordination scheme for these methods.
IAC (Fine Tune) initializes its parameters with a model pre-trained on the source environment and is ﬁne-tuned on the target environment, like COMA (Fine Tune) ; this is possible since its independent critics keep a constant input shape with respect to the number of agents.
To ensure fair comparison, we keep the number of learnable parameters across all algorithms the same.
Further information can be found in Appendix B.
4.1 Park Lights
We create the park lights environment as a model of outdoor public lighting and as a way to isolate the beneﬁts of our transfer learning approach.
Here, agents are situated on a N × M grid and model individual park lights.
At each timestep, an agent receives information about its 8 adjacent neighbors, containing only knowledge about whether that agent’s light is on or off, the state of its own light, and the number of timesteps its light has been in that state.
Agents must then choose to either toggle their light or do nothing, with each illuminated light also giving light to its 8 adjacent cells.
The global reward is then the function
T − 0 .
1 (cid:80) N
× M i =
t i , where T is the total number of cells lit, either by its corresponding agent or by the 8 agents adjacent to it, and t i is the number of timesteps agent i ’s light has been on.
Figure 3 shows the average performance of the algorithms on episodes of length 100 during training.
Notice that the optimal policy requires that two adjacent agents not have their lights on at the same time and that agents alternate illuminating adjacent squares.
As a result of this essential coordination,
IAC is unable to converge; each individual agent can only observe whether its neighbors has their lights on or off at a given timestep, so it is unable to predict whether it or its adjacent agents should toggle their light.
We see that the zero-shot performance (Figure 4) of the policy trained on the proposed GCN architecture is the best.
This follows logically from the fact that park lights has a decomposable coordination graph since increasing the size of the grid does not change environment dynamics.
Furthermore, even though the return of COMA is comparable to that of COMA (GCN) in the source environment, the performance gap widens in each of the target environments, performing about as well as IAC .
This indicates that a simple MLP critic is not able to capture and distill knowledge [Figure] about agent-level interactions and that the GCN model of the coordination graph is what facilitates knowledge transfer.
4.2 Trafﬁc Junction
The trafﬁc junction environment is a popular benchmark in multi-agent reinforcement learning where cars are randomly added to a trafﬁc grid network and must avoid collisions as they travel to their pre-determined destinations.
At every intersection of the N × M road network, an agent models the trafﬁc light and is able to observe information within a limited range of one grid from itself.
To prevent ﬂickering of lights, agents are able to toggle the color of the trafﬁc light every 5 timesteps.
Even though the trafﬁc junction environment is commonly used in literature, there is no universal reward function that all works agree upon.
[57] suggests that a reward based on the queue length at trafﬁc lights is optimal for learning, so in this work, we follow the implementation given by [56], where the reward function is a combination of queue length , wait time , and vehicle delay .
However, to better indicate transfer performance, we switch to episodic rollouts of length 100 rather than follow an inﬁnite horizon approach.
Furthermore, we introduce the Static baseline, which switches the color of the trafﬁc lights every 5 timesteps (as soon as they can be switched).
This represents a common real-world implementation of trafﬁc lights, and thus is included to better compare results.
[Figure]
In contrast to park lights , the actions of a trafﬁc light agent will eventually inﬂuence the actions of agents beyond those immediately adjacent to it given that vehicles take many timesteps to travel down the road.
While the coordination graph is still structured the same as the road network itself (as an agent does not interact with agents other than its neighbors in the same timestep ), we expect that increasing the grid size has some non-trivial impact on optimal behaviors.
Figure 5 demonstrates that our approach is reasonable in this environment.
We observe that while the zero-shot performance of the proposed critic is high, ﬁne-tuning it with a few samples of the target environment is able to improve its performance.
As a result, our method converges much quicker than the others and generally has the best performance. [Figure]
4.3 Cooperative Navigation
Cooperative navigation is a commonly used scenario for the popular multi-agent particle environment (MPE) [30].
Here, N agents are tasked with cooperatively covering N landmarks across the map and are able to observe information about the k closest landmarks and agents.
We follow the implementation given by the ofﬁcial codebase for [29].
As demonstrated by Table 1, the k -nearest neighbors observation space causes superlinear growth in runtime with respect to the number of agents in the environment.
Thus, we provide ﬁne-tuned methods only for environments up to n = 30 ; we emphasize that zero-shot performance becomes much more critical as the number of agents grows beyond that amount.
Since there is no clear coordination graph encoded into the structure of the environment, we use the Learned Adjacency Matrix module illustrated in Figure 2 to learn a dynamic coordination graph.
To maintain fair comparison, we also include the parameters introduced here into the total count.
We observe in Figure 6 that while the GCN critic with the Learned
Adjacency Matrix module does converge, it is less consistently monotonic when compared to the MLP critic, particularly when ﬁnetuning is used.
Most importantly though, the zero-shot performance of the proposed critic is still high (and the highest in Figure 7), indicating that transfer is effective even in environments where a coordination graph is completely unknown.
This can be explained by the fact that the dynamic coordination graph model may not adjust well to environments with a different amount of agents.
The architecture learns a representation of dynamics in the smaller source environment, which corresponds to strong performance on the target environment; however, we hypothesize that suddenly introducing many more agents destabilizes the self-attention module, as the nearest agents observed by an individual varies more in the larger environments (the n = 3 source environment always observes the same agents).
The module then needs to adjust its learned queries to account for this shift in input domain.
Conclusion and Future Work
In this work, we propose a joint critic architecture that is able to be transferred directly between environments with differing numbers of agents.
While we demonstrate in Section 4 that it performs better than baseline critics and is able to effectively facilitate knowledge distillation, we acknowledge that its reliance on parameter sharing and a simple GCN architecture can be limiting [8].
In future works, we would like to investigate the possibilities of using more of the various critic structures found in literature, such as hypernetworks [35].
We further propose a transfer learning procedure to share knowledge between a single source environment and multiple target environments.
This is effective for the environments that we experiment in; however, there may be cases where utilizing knowledge from multiple source environments can improve performance, such as in the single agent setting [36, 51].
We leave the choice of these source environments – selecting a cut S from the coordination graph of the target environment – to future work.