A Meta-Gradient Approach to Learning Cooperative Multi-Agent Communication Topology
Abstract
In cooperative multi-agent reinforcement learning (MARL), agents often can only partially observe the environment state, and thus communication is crucial to achieving coordination.
Communicating agents must simultaneously learn to whom to communicate (i.e., communication topology) and how to interpret the received message for decision-making.
Although agents can efﬁciently learn communication interpretation by end-to-end backpropagation, learning communication topology is much trickier since the binary decisions of whether to communicate impede end-to-end differentiation.
As evidenced in our experiments, existing solutions, such as reparameterization tricks and reformulating topology learning as reinforcement learning, often fall short.
This paper introduces a meta-learning framework that aims to discover and continually adapt the update rules for communication topology learning.
Empirical results show that our meta-learning approach outperforms existing alternatives in a range of cooperative MARL tasks and demonstrates a reasonably strong ability to generalize to tasks different from meta-training.
Preliminary analyses suggest that, interestingly, the discovered update rules occasionally resemble the human-designed rules such as policy gradients, yet remaining qualitatively different in most cases.
Introduction
There have been signiﬁcant successes of reinforcement learning (RL) recently.
Many RL applications involve multiple agents learning and adapting simultaneously to achieve shared goals, which naturally fall into the framework of cooperative multi-agent RL (MARL).
In addition to challenges in singleagent RL, there are (at least) two challenges unique to cooperative MARL: (i) Since all agents are updating their policies, the environment becomes nonstationary from the perspective of any individual agent, violating assumptions in traditional single-agent RL and perhaps resulting in learning instability; and (ii) Compared to single-agent RL, the issue of partial observability is often more severe in MARL, since each individual agent might only observe a fraction of the information in the global state, leading to potential failure to coordinate.
To cope with (i), the paradigm of centralized training and decentralized execution (CTDE) [1, 2] have been proposed, where agents are trained with access to global information to mitigate the issue of nonstationarity and choose actions during execution independently in a decentralized manner.
To cope with (ii), it is crucial to enable communication among agents during execution [3, 4], where agents share their local information to others, such that each agent can form better knowledge about the global state, leading to better cooperation.
This paper aims to better address multi-agent communication during execution under the CTDE paradigm.
For effective multi-agent communication, individual agents must simultaneously solve the problem of communication interpretation, i.e., how to utilize the received messages from other agents to make better decisions, and the preceding problem of communication topology, i.e., whom to communicate to.
Recently, many research works have shown that it is effective to learn communication interpretation by end-to-end backpropagation (e.g., [5, 3]).
However, the end-to-end learning of communication topology is much trickier, since the binary decisions of to whom to communicate impede end-to-end differentiation.
Prior work either uses reparameterization tricks such as Gumbel-Softmax [6] to force differentiability (e.g., [7]) or reformulates the learning of communication decisions as an RL problem to apply policy gradient methods (e.g., [8]).
However, these existing solutions are not completely satisfactory: Gumbel-Softmax is known to be unstable, especially for RL tasks; the formulation can dramatically slow down training due to the trial-and-error nature of RL, especially considering that the decision space for communication topology is often large.
Motivated by the aforementioned challenges, in this paper we propose to employ a meta-learning framework that formulates the problem of discovering an update rule for learning communication topology.
We then develop an architecture and an algorithm for discovering and adaptively adjusting the update rule in an online fashion, with several key design choices aimed to stabilize and facilitate the meta-learning process.
Experimental results show that our algorithm outperforms alternative baselines such as Gumbel-Softmax and policy gradient for learning communication topology.
Our ablation study conﬁrms the importance of our key design choices for performance.
We also conduct analyses showing that the meta-learned update rule can transfer to different tasks with reasonable effectiveness.
Surprisingly, while it outperforms the policy gradient baseline in most cases, we found that the update rule discovered by the meta-learning framework occasionally resembles patterns of the policy gradient update rule, yet remains qualitatively different.
Related work
Learning to learn.
Meta-learning, aka learning to learn, refers to the objective of improving the learning process itself (usually an optimization process) to be effective for a variety of learning tasks.
This idea has been around since the late 80s with various formulations such as improving the genetic programming process [9], learning update rules for neural networks [10], improving domain-invariant transfer of learned knowledge [11], and, more recently, identifying initial neural network parameters for fast adaptation [12] using meta-gradients, i.e., gradients obtained from backpropagation of the meta-learning objective.
Recently, Xu et al.
[13] introduced the framework of using meta-gradients to improve the learning of an RL agent, which has been applied to various components of an RL algorithm, including intrinsic rewards [14], auxiliary tasks [15], hyperparameters of an actor-critic loss function [16], update target [17, 18].
Meta-learning for multi-agent RL is relatively underexplored.
Du et al.
[19] have applied meta-gradients to the discovery of multi-agent intrinsic rewards.
In contrast, our work has an orthogonal contribution of using meta-gradients to learn multi-agent communication topology.
Communication in cooperative MARL.
Communications between agents play an important role in solving the nonstationarity and partial observability in MARL.
A popular training framework to solve the issue of nonstationarity is CDTE [2], where global information is obtained by all-to-all communication during training (usually achieved by a centralized module) to mitigate nonstationarity, and no communication during execution to enable fast deployment.
However, when agents only partially observe the environment state, the augmentation of communication during execution becomes necessary.
A commonly used way is to broadcast the messages from one agent to all the others during execution if the communication is determined necessary for the current timestep [3][20][21].
However, such broadcasting of information is not only unscalable when the number of agents is large[22], but also can potentially include redundant or even harmful information [23][24].
Thus, ﬁnding topology graphs to selectively communicate is crucial.
The topology can be obtained by trainable networks that make the communication decisions [25][22], or obtained naturally by the physical constraints e.g. only agents within certain distances are viewable [4].
We give a more detailed description of existing methods on ﬁnding communication topology in Section 4.1, in distinction to our approach.
Policy Net 𝜃 ! !
𝑜 "
Comm Net 𝜃 # !
𝒩 " 𝐼 "←% 𝐼 "←% " 𝑜 % 𝑜 %&
Centralizer
Env 𝑜,𝑟,𝛾 𝜋 " agent 𝑖 agent −𝑖 𝜋 ’" agent 𝑖 𝑖 agent 𝑗 !
𝑗 ! agent 𝑘 𝑘 agent 𝑗 " 𝑗 " agent 𝑗 # 𝑗 # 𝐼 $←& =
𝐼 $←& =𝒩 $ = 𝑗 " ,𝑗 ! ,𝑗 # timestep 𝑡  Preliminaries
We consider fully cooperative multi-agent tasks that can be modeled as Markov games [26] augmented with networked communication [4, 25], in which N agents, indexed by i ∈ N := { 1 , .., N } , choose sequential actions.
At each timestep t , the task has a global state s t ∈ S that is Markovian, and observation function f i : S
→ O i yields observation o it ∈
O i , for each agent i where O i is its observation space.
At timestep t , each agent i chooses an action a it ∈
A i a, forming a joint action a t := ( a 1 t , ..., a Nt ) ∈ A := × i
A i that induces a transition of the global state according to the state transition function P ( s (cid:48) | s, a ) : S ×
A × S → [0 , 1] .
The N agents choose actions after selectively communicating through a time-variant network, deﬁned by a undirected graph
G t :=
( N , E t ) with vertex set N and edge set E t ⊆ { ( i, j ) : i, j ∈ N , i (cid:54) = j } .
Speciﬁcally, let
N it :=
{ j : ( i, j ) ∈ E t } be the neighbors of agent i at timestep t .
Based on its observation o it , agent i selects a subset of its neighbors, C it ⊆
N it , and request the observations from the selected subset, o t ( C it ) :
= { o jt : j ∈ C it } .
Then, agent i chooses action according to its policy π i , a it ∼ π i ( o it , o t ( C it )) , conditioning on both its own observation and observations of the selected neighbors.
We consider the fully cooperative setting in which agents optimize their policies with respect to global reward function r ( s, a ) : S ×
A → R and discount factor γ ∈ [0 , 1] .
The discounted return from timestep t is G t = (cid:80) ∞
l =
γ l r t + l , where r t := r ( s t , a t ) is the reward at timestep t .
The agents’ joint policy π = ( π 1 , ..., π N ) induce a value function, which is deﬁned as V π ( s t ) =
E s t +1: ∞ ,a t : ∞ [ G t | s t ] , and action-value function Q π ( s t , a t ) =
E s t +1: ∞ ,a t +1: ∞ [ G t | s t , a t ] .
Methods
Figure 1 depicts our neural network architecture for solving the problem formulated in Sectionwith a learned communication topology.
The architecture is compatible with any instantiation of the CTDE framework.
Speciﬁcally, it consists of a communication network for each agent, a policy network for each agent, and a centralizer module shared by all agents, which are described next in detail.
For the rest of the paper, we omit the subscript of timestep t when the timestep is clear or irrelevant to the context.
Comm Net.
At each timestep, the communication network selects a subset of each agent’s neighbors to communicate with based on its current observation, and thus determining the communication topology.
Speciﬁcally, we assume agent i ’s
observation
o i contains the IDs of its neighbors N i .
Each agent i ’s communication network c i ( · ) takes o i as input and outputs c
i ← j ( o i ) ∈ [0 , 1] , ∀ j ∈ N i , which is interpreted as the probability that agent i communicates with neighboring agent j for its observation o j .
Letting I i ← j be the corresponding Bernoulli random variable with mean c
i ← j ( o i ) , the selected subset of communicating neighbors is thus C i = { j ∈ N i : I i ← j = 1 } .
The output size of the communication network is |N i | .
When |N i | is a constant (e.g., an agent has a constant number of n ( < N ) neighbors), and we can use a simple multi-layer perceptron (MLP) as the communication network.
In general, |N i | is a time-variant variable, and architecture choices such as LSTMs [27], transformers [28], and Graph Convolutional Nets (GCNs) [29] to deal with the variable output size.
We use θ ic to denote the parameters for the communication network of agent i .
Policy Net.
In this paper, we consider deterministic policies which are suitable for continuous action spaces.
Our method can be straightforwardly adapted to the stochastic policy case.
Speciﬁcally, the agent i ’s policy network π i ( · ) takes as input its local observation o i as well as the received observations o ( C i ) and deterministically outputs an action
π i ( o i , o ( C i ))
∈ A i .
In order to scale to an arbitrary number of communicating neighbors, in practice we treat o ( C i ) as a sequence of length
|C i | use an LSTM network that processes the sequence.
The ﬁnal hidden state of the LSTM network is concatenated with local observation o i and then fed into a multi-layer perceptron (MLP) to produce the action.
We use θ iπ to denote the parameters for the policy network of agent i .
Centralizer.
As an instantiation of the CTDE framework, our architecture involves a centralizer module that utilizes global information to guide the training of the modules for decentralized execution.
As the communication and policy networks are the decentralized execution modules, the role of our centralizer is to provide learning signals for these two types of networks.
We follow the prior work and consider the typical centralized critic that seeks to approximate the action-value function of the joint policy, Q ( o, a ) ≈ Q π ( s, a ) , where o :=
( o 1 , ..., o N ) is the joint observation and a :
= ( a 1 , ..., a N ) is the joint action; if the global state s is conveniently available, joint observation o can be replaced by s .
Given joint observation o ∈ O , the communication networks and the policy networks { c i , π i } i ∈N together deﬁne a (stochastic) joint policy π : O → ∆( A ) that maps the joint observation to a distribution over the joint action, where the stochasticity comes from the selection of communicating neighbors by the communication networks (assuming the policy network is deterministic).
Letting the critic be parameterized by θ Q , the critic can be trained by minimizing the temporal difference (TD) error:
[Equation]
where D is the replay buffer recording joint observations and actions; TD target y is computed by the target critic network Q (cid:48) with delayed parameters θ (cid:48) Q and target joint policy π (cid:48) with delayed parameters of the communication and policy networks.
Following MADDPG [2] [Equation] the centralized critic can be used to guide the optimization of each agent
i ’s network parameters by the optimization of max [Equation]
) with J
[Equation]
Using the chain rule on the deterministic policy network a i =
π i ( o i
[Equation] o ( C i ); θ iπ ) [Equation] the gradient w.r.t.
θ iπ is derived as
[Equation]
4.1 Problem of learning communication topology and existing solutions
In contrast with the policy network, the gradient of objective (2) w.r.t.
the communication network’s parameters θ ic cannot be obtained via the chain rule, since the sampling procedure for the communicating neighbors, C i ∼ c i ( o i ; θ ic ) , is non-differentiable.
Before presenting our solution using the meta-gradient framework in Section 4.2, we here review existing solutions in prior work:
No communication, i.e., C it = ∅ ∀ i, t .
At the extreme of the no communication topology, no agent communicates with any neighbor at any time, and thus the action choice is only based on the local observation, i.e., a it ∼ π i ( o it ) ∀ i, t .
Early work on CTDE, e.g., MADDPG [2], uses no communication as the default topology.
No communication suffers from the issue of partial observability of individual agents.
Full communication, i.e., C it =
N it ∀ i, t .
This solution bypasses the problem of communication topology learning by simply letting all agents always communicate with all neighbors.
Prior work such as DIAL [5] and CommNet [3] employs such full communication topology.
Such topology suffers from large communication overhead, especially when the number of agents is large.
Moreover, later work (e.g., TarMAC [20]), as well as our empirical results in Section 5, shows that full communication can even hinder the learning performance, presumably because the agents are forced to interpret usually excessive messages received from all neighbors.
RL for communication topology.
The communication decisions { I i ← j t } j ∈N it made by agent i at timestep t can be viewed as a |N it | -bit action in the RL sense, with reward r t simply being the original task reward.
Therefore, any (multi-agent) RL algorithm can in principle be applied to obtain the update for the communication network
c i that makes the decisions, e.g., REINFORCE [30]:
[Equation]
where the expectation is w.r.t.
trajectories generated by on-policy parameters { θ ic , θ iπ } i ∈N .
This idea of reformulating topology learning as an RL problem has been explored in prior work.
For example,
I3CNet [8] uses REINFORCE to train a single gating mechanism for each agent to decide whether to broadcast its observation to all of its neighbors.
However, no prior work has attempted to use
RL for training agent-to-agent communication decision-making like our communication network.
We include REINFORCE as a baseline in our experiments, yet we do not claim it as our main contribution.
Reparameterization tricks.
Reparameterization tricks are widely used to address the nondifferentiability caused by sampling.
Speciﬁcally, Straight Through Gumbel-Softmax [6] can be used for categorical samplings like I i ← j t ∼ c i ← j ( o it ) .
While convenient to use, Gumbel-Softmax can be very unstable.
Without carefully chosen regularizers, Gumbel-Softmax can easily degenerate to full communication, as evidenced in prior work [7, 31, 32] and our experiments.
4.2 Learning communication networks with meta-gradients
Inspired by the success of single-agent meta-RL [13, 14, 17, 18], we here develop a meta-gradient framework to train the communication networks.
Our main hypothesis of this work is that the communication topology learning rule obtained from our meta-gradient based optimization outperforms the traditional methods based on the policy gradient and Gumbel-Softmax .
The meta-gradient approach will yield a learning rule with independent advantage estimates for each ( i, j ) pair of agents (cf. ˆ A i ← j t in (5)), which potentially reduces variance and speeds up learning compared with the policy gradient method (4) that shares the same advantage estimate G t for all pairs of agents.
Moreover, the meta-learned update rule does not involve any reparameterization tricks such as Gumbel-Softmax that can result in unstable learning.
As for the architecture, the centralizer module is augmented with a backward LSTM network that takes as input the reverse trajectory of all the agents and produces the update direction for each agent’s communication network.
Speciﬁcally, the backward LSTM takes as input x t := [ r t , d t , γ, { c i ← j ( o it ) }
( i,j ) ∈N×N it ] at each timestep t of a trajectory, where r t is the shared reward, d t is the binary value indicating episode termination, γ is the discount factor, and c i ← j ( o it ) is the probability of agent i communicating with neighbor j at timestep t that is computed by communication network c i ( · ) .
Since |N it | is in general a time-variant variable, the size of input x t is time-variant.
We assume |N
it | to be bounded by d (i.e., an agent has at most d neighbors), which is always possible since d can be set to N − 1 .
Thus, the size of input x t is time-invariant, and x t can be ﬂattened for the
LSTM input.
The LSTM outputs, for each agent-neighbor pair of ( i, j ) ∈ N × N it at each timestep t , a scalar ˆ A i ← j t ∈ R that deﬁnes the update of the communication network: [Equation]
[Equation]
Compared with the REINFORCE update in Equation (4), the term of return G t is replaced by ˆ A i ← j t that is computed by the backward LSTM.
We use η to denote the meta-parameters for the backward LSTM network that deﬁnes the update of
Equation (5).
The objective of our meta-learning framework is to ﬁnd the meta-parameters η that leads to high expected return after a number of K updates for the communication network parameters θ c := { θ ic } i ∈N , given some ﬁxed policy network parameters θ π := { [Equation] iπ } i ∈N
[Equation] [Equation]
where the expectation is w.r.t.
random initializations of θ c (0) ; the initial θ c (0) is updated K times to θ c ( K ) using the ﬁxed η , ( θ c (0) → ...
→ θ c ( K ) | η ) ; G ( θ c ( K )) := E [ (cid:80) ∞ t [-=
 -]=0 γ t r t | θ c ( K ) , θ π ] is the expected return for the communication network after the K updates θ c ( K ) and the ﬁxed policy network parameters θ π .
Since the updates of θ c are differentiable with respect to η , we can consider optimizing objective (6) with the meta-gradient derived using the chain rule ∇ η J K ( η ) = ∇ θ c ( K ) J K ( η ) ∇ η θ c ( K ) , with ∇ θ c ( K ) J K ( η ) obtained from the policy gradient, e.g., REINFORCE:
[Equation]
where G t ( θ c ( K )) :
= E [ (cid:80) ∞ t (cid:48) =
t γ t (cid:48) r t (cid:48) | θ c ( K ) , θ π ] is the expected return starting from timestep t .
The true meta-objective (6) and its meta-gradient (7) require backpropagation through the entire K updates of θ c , which is infeasible due to memory constraint if K is large.
In practice, we perform meta-gradient optimization every a small number of K updates while continually updating θ c to the end of learning.
Concurrent parameter updates.
The derivation of meta-gradient (7) assumes ﬁxed policy network parameters, which makes the environment stationary from the perspective of the communication networks and thus the policy gradient can be used to calculate ∇ θ c ( K ) J K ( η ) .
In practice, to speed up learning we consider concurrent updates for all parameters, including the communication networks θ c , the policy networks θ π , and the centralized critic θ Q .
Learned baseline.
To reduce variance, we separately use a baseline V ( o t ; φ )
≈ G t parameterized by φ to approximate the expected return under the current communication and policy networks, which is learned using n -step TD:
[Equation]
The G t ( θ c ( K )) term in meta-gradient (7) is replaced by G φ,nt − V ( o t ; φ ) , where the trajectory for computing G φ,nt is sampled from the on-policy parameters ( θ c ( K ) , θ π ( K )) that have been concurrently updated.
In our empirical work, we also use a learned baseline for the original REINFORCE (4) for fair comparison.
Entropy regularization.
We also propose to add entropy regularization on the communication network outputs c i ← j to encourage exploration and facilitate learning.
The regularized meta-gradient becomes
[Equation]
where H ( · ) is the entropy of the Bernoulli distribution c i ← j that is determined by the post-update θ ic ( K ) , and β ∈ R + is the regularization coefﬁcient.
Blending intermediate meta-gradients.
The regularized meta-gradient ∇ η ( K ) is derived through the parameters after the K -th update, i.e., θ c ( K ) .
In practice, we ﬁnd it effective to blend it with meta-gradients derived through the intermediate parameters, resulting in our ﬁnal update for metaparameters η :
[Equation]
(9) where ∇ η ( k ) is the meta-gradient derived through θ c ( k ) in the same way as Equation (8).
We hypothesize that such blending in Equation (9) can reduce the variance of the gradient estimation and therefore beneﬁt the optimization of the meta-parameters.
The resulting algorithm is speciﬁed in Algorithm 1.
Experiments
We aim to answer the following questions in Sections 5.1-5.3, respectively:
• How effective is our algorithm for learning cooperative communication topology?
• How effective is the meta-parameters after being optimized? • How important are the design choices described in Section 4.2?
• What kind of update rule is learned from the meta-learning process?
Environment.
We consider the following scenarios in multiple particle environment (MPE) [33]: (1) cooperative navigation : N agents move as a team to cover N landmarks.
Each agent observes its Input: A cooperative multi-agent task with agents N , meta-optimization hyperparameter K 1 Initialize parameters for communication and policy networks θ c ≡ { θ ic } i ∈N , θ π ≡ { θ iπ } i ∈N , centralized critic θ Q , baseline φ , meta-parameters η , and replay buffer D 2 repeat θ c (0) , θ π (0) ← θ c , θ π for k = 1 , 2 , ..., K do Generated a trajectory τ ( k − 1) using ( θ c ( k − 1) , θ π ( k − 1))
Add the trajectory to replay buffer D
Update the centralized critic θ Q by minimizing L ( θ Q ) in Equation (1) on samples from D
Update the policy networks as θ π ( k − 1)
→ θ π ( k ) using Equation (3) on samples from D
Update the communication networks as θ c ( k − 1) → θ c ( k ) using Equation (5) with η end
Generated a trajectory τ ( K ) using ( θ c ( K ) , θ π ( K ))
Perform updates for baseline φ on trajectories { τ ( k ) } Kk =
Update η using meta-gradient Equation (9) computed from trajectories { τ ( k ) } Kk =  θ c , θ π ← θ c ( K ) , θ π
( K ) 15 until convergence location and velocity, the relative location of the nearest l landmarks, and the relative location of other agents.
Our experiments include the tasks of ( N = 3 , l = 3) as navigation_N3_l3 , ( N = 6 , l = 1) as navigation_N6_l1 , and ( N = 6 , l = 6) as navigation_N6_l6 .
(2) predator and prey : N cooperating predators (agents) are tasked to capture M preys.
The preys are environment-controlled by ﬁxed policies that are pretrained.
The movement of both predators and preys is impeded by L landmarks.
Each predator observes its location and velocity, the relative location of the nearest l landmarks and the nearest l preys, and the relative location of other predators.
Our experiments include the tasks of ( N = 3 , M = 1 , l = 3) as predator_prey_N3_l3 , ( N = 6 , M = 2 , l = 1) as predator_prey_N6_l1 , and ( N = 6 , M = 2 , l = 6) as predator_prey_N6_l6 .
(3) cooperative push : N cooperating agents are tasked to push a large ball to a target position.
Each agent observes its location and velocity, the relative position of the large ball and the target position, and the relative location of other agents.
We include the tasks of N = 3 as push_N3 and N = 6 as push_N6 .
Implementation details.
Consistent with prior implementations [2, 34] on MPE, the trajectory length is set to be equal to the episode length, which is 25 timesteps.
The meta-parameters are updated every K = 10 updates of the parameters , after a grid search over
K ∈ { 1 ,
All algorithms are implemented using JAX [35].
Baselines.
We compare our meta-gradient approach against the four baselines discussed in Section 4.1: no-communication, full-communication, learning communication topology with the policy gradient algorithm of REINFORCE, and learning communication topology with Straight Through
Gumbel-Softmax (STGS).
For the ablation study in Section 5.3, we additionally compare against variants of our algorithm with changes to our key design choices.
5.1 Effectiveness of the meta-learning framework
Figure 2 shows the training curves comparing our method of meta-learning communication topology against the four baselines.
Please view all of the ﬁgures in this section in color.
Our key observation is that: no method is uniformly the best among all tasks, yet our method is consistently comparable to the best for most tasks while each baseline can perform signiﬁcantly worse in some tasks.
Full communication is, perhaps surprisingly, among the worst in all tasks except for push_N3 .
No communication is among the worst in push_N3 and worse than the best in push_N6 .
REINFORCE is among the worst in navigation_N3_l3 , predator_prey_N6_l1 , and push_N [Figure]
[Figure]
5.2 Effectiveness of the meta-learned update rule
Our algorithm continually updates meta-parameters η online.
An interesting question is: how effective is the communication topology learning rule deﬁned by some ﬁxed η that is obtained after metagradient optimization?
Figure 3 compares training curves for the two tasks of navigation_N6_l and predator_prey_N6_l1 using ﬁxed η that is randomly initialized, or obtained after 10 k or 50 k episodes, with the online updated η as a reference point.
We consider both the settings in which the ﬁxed η is obtained from the same task and from a different task.
As a sanity check, meta-learned η (either ﬁxed or online updated) outperforms random initialized η .
In navigation_N6_l6 , the meta-learned, ﬁxed η can outperform the online updated η , even for the η obtained from the other task of predator_prey_N6_l1 .
However, in predator_prey_N6_l1 , the online updated η tends to perform best.
This indicates that the answer to our question is task-dependent: some tasks may require continually updating meta-parameters for best performance.
5.3 Ablation study and analyses of learned communication topology
Our ablation study that examines the effect of our design choices of entropy regularization and intermediate meta-gradients introduced in Section 4.2, as well as the effect of the number of parameter updates K per meta-update Figure 4 (left) summarizes the ablation results .
The results show that both design choices are crucial for performance.
As for K , an intermediate value such as 5 andtends to work better than the extreme values of 1 and 20 .
To distinguish REINFORCE-learned and meta-learned typologies, we compare the REINFORCE advantage (i.e., G t − V ( o t ; φ ) for Equation (4) with learned baseline φ ) and the meta-learned advantage (i.e., ˆ A i ← j t computed by η for Equation (5)) that respectively deﬁne the two update rules for the communication network.
Figure 4 (right) visualizes the normalized advantages computed from a same episode of predator_prey_N6_l1 .
We present the meta-learned advantage ˆ A i ← j t computed by both the randomly initialized η and the updated η after 50 k episodes, for the agent indexed by i = 1 against its neighbors j ∈ { 2 , ..., 6 } .
It is clear that the meta-learned η yields advantages that differ from those at initialization and are contrastingly different among the neighbors j .
Interestingly, [Figure] although most updates are signiﬁcantly different, there is evidence that the meta-learned update rule occasionally resembles the REINFORCE update rule (see agent 4 ) for some trajectories.
Conclusion and discussion
In this paper, we have proposed a novel meta-learning framework to discover adaptive update rules for learning communication topology in an online manner.
Empirical results showed that our method outperforms existing alternatives in a range of cooperative MARL tasks.
Preliminary analyses suggest that the discovered update rules resemble the human-designed rules such as policy gradients, yet remain qualitatively different.
We next acknowledge several limitations of this work, which point out promising future directions.
Although we provided in Section 5 preliminary analyses on the topology learning update rules discovered by our meta-learning algorithm, its nature is not yet fully understood.
It might be interesting and enlightening to better understand the discovered update rules for future work.
Moreover, our meta-learning algorithm continually adapts the update rules online, and the meta-learned update rules exhibit limited generalization ability as shown in Figure 3.
It remains an open question how we can discover topology learning update rules (deﬁned by ﬁxed meta-parameters) that work well for a large variety of MARL tasks. A Implementation details A.1 Computing resources Our implementation is based on JAX [35] using GPUs.
A single run took approximately 17 hours using an NVIDIA Tesla V100 GPU and 7 CPU cores
[Table]
A.2 Hyperparameters
[Table]
B Additional results
The main body presents the visualization of normalized advantages for a single episode of predator_prey_N6_l1 .
Figure 5 presents visualization for multiple episodes
[Table]
[Table]
[Figure]