Learning to Share in Multi-Agent Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress
Abstract
In this paper, we study the problem of networked multi-agent reinforcement learning (MARL), where a number of agents are deployed as a partially connectednetwork and each interacts only with nearby agents.
Networked MARL requiresall agents make decisions in a decentralized manner to optimize a global objectivewith restricted communication between neighbors over the network.
Inspired by thefact that sharing plays a key role in human’s learning of cooperation, we propose
LToS, a hierarchically decentralized MARL framework that enables agents to learnto dynamically share reward with neighbors so as to encourage agents to cooperateon the global objective through collectives.
For each agent, the high-level policylearns how to share reward with neighbors to decompose the global objective, whilethe low-level policy learns to optimize the local objective induced by the high-levelpolicies in the neighborhood.
The two policies form a bi-level optimization andlearn alternately.
We empirically demonstrate that LToS outperforms existingmethods in both social dilemma and networked MARL scenario across scales.
Introduction
In fully cooperative multi-agent reinforcement learning (MARL) there are multiple agents interactingwith the environment via their joint action to cooperatively optimize an objective.
Many methods ofcentralized training and decentralized execution (CTDE) have been proposed for cooperative MARL,such as COMA (Foerster et al., 2018), QMIX (Rashid et al., 2018), QPLEX (Wang et al., 2021),and FOP (Zhang et al., 2021).
However, these methods suffer from the overgeneralization issue:employed value functions cannot estimate well because agents sometimes choose uncoordinatedactions and thus the optimal policy cannot be learned (Castellini et al., 2019).
Moreover, they maynot easily scale up with the number of agents due to centralized learning (Qu et al., 2020a).
In many MARL applications there are a large number of agents that are deployed as a partiallyconnected network and collaboratively make decisions to optimize the globally averaged return, suchas communication networks (Kim et al., 2019) and trafﬁc signal control (Wei et al., 2019).
To dealwith such scenarios, networked MARL is formulated to decompose the dependency among all agentsinto dependencies between only neighbors.
To avoid decision-making with insufﬁcient information,agents are permitted to exchange messages with neighbors over the network.
In such settings, it isfeasible for agents to learn to make decisions in a decentralized way (Zhang et al., 2018;
Qu et al.
,2020b).
However, there are still difﬁculties of dependency if anyone attempts to make decisionsindependently, e.g., prisoner’s dilemma and tragedy of the commons (Pérolat et al., 2017).
Existingmethods tackle these problems by consensus update of value function (Zhang et al., 2018), creditassignment (Wang et al., 2020), or reward shaping (Chu et al., 2020).
However, these methods relyon either access to global state and joint action (Zhang et al., 2018) or hand-crafted reward functions(Wang et al., 2020; Chu et al., 2020).
Inspired by the fact that sharing plays a key role in human’s learning of cooperation (Eisenbergand Mussen, 1989), we propose learning to share ( LToS ), a hierarchically decentralized learningframework for networked MARL.
LToS enables agents to learn to dynamically share reward withneighbors so as to collaboratively optimize the global objective.
The high-level policies decomposethe global objective into local ones by determining how to share their rewards, while the low-levelpolicies optimize local objectives induced by the high-level policies.
LToS learns in a decentralizedmanner, and we prove that the high-level policies are a mean-ﬁeld approximation of the jointhigh-level policy.
Moreover, the high-level and low-level policies form a bi-level optimization andalternately learn to optimize the global objective.
LToS is a general hierarchical framework for networked MARL and can be easily realized by diversecombinations of RL algorithms.
We currently implement LToS by DDPG (Lillicrap et al., 2016) as thehigh-level policy and DGN (Jiang et al., 2020) as the low-level policy.
We empirically demonstratethat LToS outperforms existing methods for networked MARL in both social dilemma and networked
MARL scenario.
Related Work
There are many recent studies for collaborative MARL.
Most adopt CTDE
(Foerster et al., 2018;
Rashid et al., 2018; Wang et al., 2021; Zhang et al., 2021).
Many of them are constructed on the basisof factorizing the joint Q-function (Rashid et al., 2018; Wang et al., 2021).
However, these factorizedmethods suffer from the overgeneralization issue (Castellini et al., 2019).
Other studies focus moreon decentralized training , to which our work is more closely related, as summarized as follows.
Networked MARL.
Zhang et al.
(2018) and Qu et al.
(2019) proposed consensus update of localvalue functions, where each agent keeps a local copy of the global value function but is assumed tohave global information.
Qu et al.
(2020a) proposed intention propagation between agents, where eachagent updates its policy based on intentions shared by other agents, but the policy may converge slowlydue to propagated intentions over the network.
Qu et al.
(2020b)
and Lin et al.
(2020) investigatedthe exponential decay property, i.e. , the impact of agents on each other decays exponentially in theirgraph distance, while Chu et al.
(2020) introduced a spatial discount factor to capture the inﬂuencebetween agents, which remains hand-tuned.
However, none of these studies provide an explicitmechanism to solve social dilemma in networked MARL.
Reward Design.
Hostallero et al.
(2020) aimed at maximizing social welfare,
but they simplyused temporal difference error for reward shaping.
As temporal difference error in deep RL hardlyconverges to zero, it still biases the optimization objective.
Mguni et al.
(2019) added an extra part tothe original reward as non-potential based reward shaping and used Bayesian optimization to inducethe convergence to a desirable equilibrium between agents.
However, the extra part remains ﬁxedduring an episode, which makes it less capable of dealing with dynamic environments.
Moreover,the reward shaping alters the original optimization problem.
Hughes et al.
(2018) proposed theinequity aversion model to balance agents’ selﬁsh desire and social fairness.
Wang et al.
(2020)considered learning the Shapley value as the credit assignment.
However, these methods still relyon hand-crafted reward designs.
Lupu and Precup (2020) added gifting as an extra action into theoriginal MDP to modify the MDP and objective.
Yang et al.
(2020) proposed that each agent learnsan incentive function and optimizes the policy in terms of both reward and incentives given by otheragents.
Obviously, both methods alter the original objective of optimization.
Unlike existing work, LToS enables agents to learn to dynamically share reward with other agentswithout the bias of the optimization objective such that they can collaboratively optimize the globalobjective in networked MARL.
Background3.1 Networked Multi-Agent Reinforcement Learning
Assume N agents interact with an environment.
Let V = { 1 , 2 , · · · , N } be the set of agents.
Themulti-agent system is modeled as an undirected graph G ( V , E ) , where each agent i serves as vertexi and E⊆ V × V is the set of all edges.
Two agents i, j ∈ V can communicate with each other ifand only if
= ( i j ) ∈ E .
We denote agent i and its all neighbors in the graph together as a set
N i .
The state of the environment s ∈ S transitions upon joint action a ∈ A according to transitionprobability P a : S ×
A × S → [0 , 1] , where joint action set A = × i ∈V
A i .
Each agent i has apolicy π i ∈
Π i : S ×
A i → [0 , 1] , and we denote the joint policy of all agents as π ∈ Π = ×
i ∈V Π i(Zhang et al., 2018).
For networked MARL, a common and realistic assumption is that the rewardof each agent i just depends on its action and the actions of its neighbors (Qu et al., 2020a), i.e. ,r i ( s, a ) =
r i ( s, a N i ) .
Moreover, each agent i may only obtain partial observation o i ∈
O i , but canapproximate the state by the observations of N i (Jiang et al., 2020) or the observation history (Chuet al., 2020), which are all denoted by o i for simplicity .
The global objective is to maximize the sumof cumulative rewards of all agents , i.e. , (cid:80) ∞ t =0 (cid:80) Ni
=1 γ t r ti .
3.2 Markov Game
In such a setting, each agent could individually maximizes its own expected return, which is knownas Markov game.
This may lead to stable outcome or Nash equilibrium, which however is usuallysub-optimal in terms of the global objective.
Given π , the value function of agent i is given by
[Equation]
(1) where p a ∈ P a describes the state transitions.
A Nash equilibrium is deﬁned as (Mguni et al., 2019)
[Equation]
, ∀ i ∈ V , where π − i =
× j ∈V\{ i } π j .
Method
LToS is a decentralized hierarchy.
At each agent, the high-level policy determines the weights ofreward sharing based on low-level policies while the low-level policy directly interacts with theenvironment to optimize the local objective induced by the high-level policies.
Therefore, they forma bi-level optimization and alternately learn towards the global objective.
4.1 Reward Sharing
The intuition of reward sharing is that if agents share their rewards with others, each agent has toconsider the consequence of its actions on others, and thus it promotes cooperation.
In networked
MARL, as the reward of an agent is assumed to depend on the actions of neighbors, we allow rewardsharing only between neighboring agents.
This is because the change of actions of neighbors directlyaffects the reward while the agents outside the neighborhood can only affect the return of the agentindirectly by the change of state distribution.
Moreover, this also ﬁts the setting of networked MARLwith restricted communication between neighbors.
For the graph of V , we additionally deﬁne a set of directed edges, D , constructed from E .
Speciﬁcally,we add a loop d ii ∈ D for each agent i and split each undirected edge e ij ∈ E into two directededges: d ij = ( i, j ) and d ji = ( j, i ) ∈ D .
Each agent i determines a weight w ij ∈ [0 , 1] for eachdirected edge d ij , ∀ j ∈ N i , subject to the constraint (cid:80) j ∈N i w ij = 1 , so that w ij proportion of agenti ’s environment reward r i will be shared to agent j .
Let w ∈ W = ×
d ij ∈D w ij be the weights of thegraph.
Therefore, the shaped reward after sharing for each agent i is deﬁned as
[Equation]
4.2 Hierarchy
Assume there is a joint high-level policy φ ∈
Φ : S × W → [0 , 1] to determine w .
Given φ and w ,we can deﬁne the value function of π at each agent i based on (1) as
[Equation]
It is noteworthy that w is a multidimensional action for an allocation scheme rather than a probabilitydistribution.
In our derivation, we express w as a discrete action for simplicity.
It also holdsfor continuous action as long as we change all the summations to integrals.
Let V φ V ( s ; π ) .
=(cid:80) i ∈V v π i ( s ; φ ) and Q
φ V ( s, w ; π ) .
= (cid:80) i ∈V v π i ( s ; w , φ ) .
Proposition 4.1.
Given π , V φ V ( s ; π ) and Q φ V ( s, w ; π ) are respectively the value function and actionvalue function of φ .
Proof.
The proof is deferred to Appendix A.
Proposition 4.1 implies that φ directly optimizes the global objective by generating w , given π .
Unlike existing hierarchical RL methods, we can directly construct the value function and action-valuefunction of φ based on the value function of π at each agent.
As φ optimizes the global objective given π while π i optimizes the shaped reward individually ateach agent given φ (assuming π convergent to Nash equilibrium or stable outcome, denoted aslim ), they form a bi-level optimization.
Let J φ ( π ) and J π ( φ ) denote the objectives of φ and πrespectively.
The bi-level optimization can be formulated as follows,
[Equation]
4.3 Decentralized Learning
We start from collective learning to achieve global optimization of average reward.
So far, the jointhigh-level policy is still in a centralized form.
Note that the scenario needs a decentralized method andeach agent has its own reward.
Now we turn to learning the joint high-level policy in a decentralizedway.
Let w out i .
= { w ij | j ∈ N i } and w in i .
= { w ji | j ∈ N i } .
The following proposition proves eachagent’s independence of each other on the high level.
Proposition 4.2.
The joint high level policy φ can be learned in a decentralized manner, and thedecentralized high-level policies of all agents form a mean-ﬁeld approximation of φ .
Proof.
The proof is deferred to Appendix A.
Proposition 4.1 and 4.2 indicate that for each agent i , the low-level policy simply learns a localπ i ( a i | s, w in i ) to optimize the cumulative reward of r w i , since r w i is fully determined by w in i accordingto (2) and denoted as r wi from now on.
And the high-level policy φ i just needs to locally determinew out i to optimize the cumulative reward of r φ V .
Therefore, for decentralized learning, (4) can be decomposed locally for each agent i as
[Equation]
Now we use max instead of lim because local policies can be compared and improved in a decentralized manner in a Markov game.
We abuse the notation and let φ and π also denote theirparameterizations respectively.
To solve the optimization, we have
[Equation]
where α is the learning rate for the low-level policy.
Let π (cid:48)
i denote π
i + α ∇ π i J π i ( φ ) , we have
[Equation]
j
[Equation] [Equation]
The second-order derivative is neglected due to high computational complexity, without incurringsigniﬁcant performance drop such as in meta-learning (Finn et al., 2017) and neural architecturesearch (Liu et al., 2019).
Differently, our low-level policy requires more than one gradient step untilconvergence.
Similarly, we have
[Equation]
where β is the learning rate of the high-level policy.
Therefore, we can solve the bi-level optimization(4) by the ﬁrst-order approximations in a decentralized way.
For each agent i , φ i and π i are alternatelyupdated.
[Figure]
In distributed learning, as each agent i usually does nothave access to state, we further approximate φ i ( w out i | s )and π i ( a i | s, w in i ) by φ i ( w out i |
o i ) and π i ( a i
|
o
i , w in i ) , respectively.
Moreover, in network MARL as each agent iis closely related to neighboring agents, (5) can be furtherseen as π i maximizes the cumulative discounted rewardof r wi given φ N i , where φ N i =
×
j ∈N i φ j , and φ i equiv- 170 alently optimizes the global objective given π
N i , where 171 π
N i = ×
j ∈N i π j .
During training, π N i and φ N i are implicitly considered by interactions of w out i and w in i respectively.
The architecture of LToS is illustrated in Figure 1.
At each timestep, the high-level policy of each agent imakes a decision of action w out i as the weights of rewardsharing based on the observation.
Then, the low-level policy takes the observation and w in i as an input and outputsthe action.
Agent i obtains the shaped reward according to w in i for both the high-level and low-levelpolicies.
The gradients are backpropagated along purple dotted lines.
Further, from Proposition 4.1, we have: q φ i i ( s, w out i ; π N i ) =
v π i i ( s ; w in i , φ N i ) , where q φ i
i is theaction-value function of φ i given π N i , v π
i i is the value function of π i given φ N i and conditioned onw in i .
As aforementioned, we approximately have q φ i
i ( o i , w out i ) =
v π i i
( o i ; w in i ) .
We can see that theaction-value function of φ i is equivalent to the value function of π i .
That said, we can use a singlenetwork to approximate these two functions simultaneously.
For a deterministic low-level policy, thehigh-level and low-level policies can share the same action-value function.
In the current instantiationof LToS, we use DGN (Jiang et al., 2020)
(Q-learning) for the low-level policy and DDPG (Lillicrapet al., 2016) for the high-level policy.
Thus, the Q-network of DGN also serves as the critic of DDPG,and the gradient of w in i is calculated based on the maximum Q-value of a i .
For completeness, Algorithm 1 (see Appendix B) gives the training procedure of LToS based on
DDPG and DGN.
More discussions about training LToS are also available in Appendix C.
Experiments
For the experiments, we adopt three scenarios prisoner , jungle , and trafﬁc depicted in Figure 2, whereprisoner and jungle (Jiang et al., 2020) are grid games about social dilemma that easily measuresagents’ cooperation, while trafﬁc is a realistic scenario of networked MARL.
We obey the principleof networked MARL that only allows communication in neighborhood as Zhang et al.
and
Chu et al.
To illustrate the reward sharing scheme each agent learned, we use a simple indicator: selﬁshness , thereward proportion that an agent chooses to keep for itself.
For ablation, we keep the sharing weightsﬁxed for each agent, named ﬁxed LToS.
Throughout the experiments, we additionally compare withthe baselines including DQN and DGN, where DGN also serves the ablation of LToS without rewardsharing as DGN is the low-level policy of LToS.
To maximize the global return directly by centralizedlearning, we use QMIX (Rashid et al., 2018) as a baseline throughout the three scenarios and twoother ones in prisoner .
Moreover, as LToS aims to bring a harmonious cooperation by reward sharingin networked MARL, we compared LToS to three methods for networked MARL, i.e. , ConseNet(Zhang et al., 2018), NeurComm (Chu et al., 2020) and Intention Propagation (abbreviated as IP) (Quet al., 2020a), and LIO (Yang et al., 2020) for incentivized learning, all of which use recurrent neural
[Figure]
network (RNN) or graph neural network (GNN) for the partially observable environment.
Moredetails of hyperparameters are available in Appendix D.
5.1 Prisoner
We use prisoner , a grid game version of the well-known matrix game prisoner’s dilemma from
Sodomka et al.
to empirically demonstrate that LToS is able to learn cooperative policies toachieve the global optimum ( i.e. , maximize globally averaged return).
As illustrated in Figure 2a,there are two agents A and B that respectively start on two sides of the middle of a grid corridorwith full observation.
At each timestep, each agent chooses an action left or right and moves to thecorresponding adjacent grid, and every action incurs a cost − 0 .
There are three goals, two goalsat both ends and one in the middle.
The agent gets a reward +1 for reaching the goal.
The gameends once some agent reaches a goal or two agents reach different goals simultaneously.
This gameresembles prisoner’s dilemma: going for the middle goal (“defect") will bring more rewards than thefarther one on its side (“cooperate"), but if two agents both adopt that, a collision occurs and only oneof the agents wins the goal with equal probability.
On the contrary, both agents obtain a higher returnif they both “cooperate", though it takes more steps.
The highest possible return is 1 .
Figure 3a illustrates the learning curves of all the methods in terms of average return.
Note that for allthree scenarios, we present the average of 5 training runs with different random seeds by solid linesand the min/max value by shadowed areas.
As a result of self-interest optimization, DQN convergesto the “defect/defect" Nash equilibrium where each agent receives an expected reward about 0 .
Sodoes DGN since it only aims to take advantage of its neighbors’ observations while prisoner is afully observable environment already.
Given a hand-tuned reward shaping factor to direct agents tomaximize average return, NeurComm and ﬁxed LToS agents are able to cooperate eventually.
Soare ConseNet and QMIX.
However, they converge slowly.
In contrast, IP agents learn at a slowerpace and its performance is only a little higher than 0 . 5 .
LIO agents cooperate soon enough at thebeginning, but they cannot form steady cooperation and perhaps need longer training to get rid ofsuch instability.
JointDQN, Coco-Q (Sodomka et al., 2013), and LToS perform similarly and outperform othermethods.
JointDQN is one centralized DQN that takes control of joint actions of both agent A and B ,and thus should be able to achieve the best performance but still takes time to converge even in sucha simple two-agent scenario.
As a modiﬁed tabular Q-learning method, Coco-Q introduces the cocovalue (Kalai and Kalai, 2010) as a substitute for the expected return in the Bellman equation andregards the difference as transferred reward.
However, it is speciﬁcally designed for some games, andit is hard to be extended beyond two-player games.
LToS can learn the reward sharing scheme whereone agent at ﬁrst gives all the reward to the other so that both of them are prevented from “defect",and thus achieve the best average return quickly, as observed in the experiment.
By prisoner , weverify that LToS can escape from local optimum by learning to share reward.
5.2 Jungle
Jungle is a scenario about moral dilemma proposed by Jiang et al.
(2020) based on MAgent (Zhenget al., 2017).
As illustrated in Figure 2b, there are N agents and L stationary foods.
At each timestep,
[Figure]
e r
[Figure]
r
[Figure]
each agent can attack or move to one adjacent grid.
Eating (attacking food) brings a positive reward+1 , but attacking other agents obtains a higher reward +2 .
The victim, however, suffers a negativereward − 4 , which makes each attack between agents a negative-sum action.
Moreover, attacking ablank grid gets a small negative reward − 0 .
01 (inhibiting excessive attacks).
We follow the originalsetting of Jiang et al.
(2020): map size = 30 × 30 grids, N = 20 , L = 12 , and the observationconsists of one’s coordinates and a ﬁeld of 11 × 11 grids nearby.
Each agent has 3 closest agents asits neighbors.
Compared to prisoner , jungle has much more agents, and thus JointDQN and Coco-Qare disregarded for this scenario.
Another challenge of jungle is that the network topology is dynamicsince each agent can always move.
Fortunately, the topology change slowly and predictably, andalgorithms may get the time-varying neighbor set N i as part of the input.
Therefore, it is still likelyto estimate the shaped rewards and value functions well.
[Table]
Figure 3b illustrates the learning curves of all the methods, and their performance after convergenceis also summarized in Table 1.
NeurComm, ConseNet and QMIX do not perform well in this task.
In
NeurComm, each agent gets a “delayed global information".
However, a stable pattern of delayedglobal information cannot be formed when the communication is conducted via a dynamic topology.
ConseNet is constructed on the basis of a premise of full observation , and it can hardly learn wellwhen the input is not only partial but also fairly varying in sequence and content.
LIO agents alsoperform badly, since they cannot be distinguished from one another in the dynamic topology andthus fail to learn a proper incentive function.
Moreover, LIO requires opponent modeling, but it ishard to simultaneously model all other agents in a dynamic environment.
QMIX is free from theseproblems.
While aiming at global optimization, like LIO, it realizes that attacking usually means anegative-sum action, but as a result, it avoids attacking as well as eating most of the time and thus onlyachieves a reward slightly higher than 0.
Another possible reason to explain the performance QMIXis its scalability.
As there are 20 agents in the scenario, it can be hard to learn the joint action-valuefunction to directly optimize the average return (Qu et al., 2020a).
Also, we can see that a ﬁxedreward sharing scheme does not bring any gain over DGN.
This is because ﬁxed reward sharing doesnot adapt to the dynamic topology.
By learning proper reward sharing and adjusting to changingcircumstances, LToS outperforms all other baselines.
Note that there is an upper bound for averagereward per step for jungle .
By estimating the average distance between each agent and the food thatis the closest to it at the beginning of one episode, we can give a loose upper bound around 0 .
95 toreﬂect our improvement.
Figure 4 illustrates the representative behaviors of agents learned by difference methods.
For
NeurComm, ConseNet (and the same with LIO), most agents are not even close to the foods, so as toavoid being attacked.
Even though, there are still conﬂict and sneak attack between agents sometimes.
IP and DGN agents learn much better, but agents may still be cautious about each other, which leadsto hesitation when they are near the same food.
LToS agents learn to properly share the food evenif the foods are close ( i.e. , agents are easy to be attacked) as depicted in Figure 4e and demonstrate
[Figure] [Figure]
Besides
[Figure]
we compared LToS and ablation baselines ( i.e. , DGN and ﬁxed LToS) with different numberof agents ( i
[Figure]
e. , from 10 to 50 ) to verify the scalability of LToS.
All the setting remains the same exceptthat the number of agents and food grows proportionally ( i
[Figure]
e. , # agents / # foods = 5 / 3 ).
As depicted in
Figure 5b, LToS can always achieve the best performance as the agent population size increases.
5.3 Trafﬁc
[Table]
In trafﬁc , as illustrated in Figure 2c , we aim to investigate thecapability of LToS in dealing with highly dynamic environmentthrough reward sharing.
We adopt the same problem settingas Wei et al.
In a road network, each agent serves astrafﬁc signal control at an intersection.
The observation of anagent consists of a one-hot representation of its current phase(directions for red/green lights) and the number of vehicles oneach incoming lane of the intersection.
At each timestep, anagent chooses a phase from the pre-deﬁned phase set for thenext time interval, i.e. , 10 seconds.
The reward is set to be thenegative of the sum of the queue lengths of all approaching lanes at current timestep.
The globalobjective is to minimize average wait time of all vehicles in the road network, which is equivalent tominimizing the sum of queue lengths of all intersections over an episode (Zheng et al., 2019).
Theexperiment was conducted on a trafﬁc simulator, CityFlow (Zhang et al., 2019).
We use a 6 × 6 gridnetwork with 36 intersections.
The trafﬁc ﬂows were generated to simulate dynamic trafﬁc ﬂowsincluding both peak and off-peak period, and the statistics are summarized in Table 2.
[Table]
For better demonstration, we choose to show the normalized metric of wait time: the average numberof red lights one vehicle waits for at per intersection, and we can also give a loose lower bound0 .
50 to reﬂect our improvement.
Figure 3c shows the learning curves of all the methods in termsof that in logarithmic form.
The performance after convergence is summarized in Table 3, where
[Figure]
LToS outperforms all other methods.
LToS outperforms DGN, which demonstrates the rewardsharing scheme learned by the high-level policy indeed helps to improve the cooperation of agents.
Without the high-level policy, i.e. , given ﬁxed sharing weights, ﬁxed LToS does not perform well indynamic environment.
This indicates the necessity of the high-level policy.
The performance of IPagents increases so slowly that they cannot converge efﬁciently.
Although NeurComm and ConseNetboth take advantage of RNN for partially observable environments, LToS still outperforms thesemethods, which veriﬁes the great improvement of LToS in networked MARL.
QMIX is conﬁned tosuboptimality (Mahajan et al., 2019).
As observed in the experiment, QMIX tries to release trafﬁcﬂows from one direction while stopping ﬂows from the other direction all the time, because this willonly make two rows of intersections on the border blocked but keep most of the intersections fromany trafﬁc jam all the time.
However, the global optimality actually does not need to be constructedon the sacriﬁce of anyone.
Some similar thing happens to LIO.
It is likely because LIO contains somesensitive parameters (Yang et al., 2020) and agents are hard to learn and coordinate their incentivefunctions since the original reward functions change acutely once an improper operation causes trafﬁccongestion.
An introduction of some explicit coordination mechanism may also alleviate the problem,like that of NeurComm and ConseNet.
We visualize the variation of selﬁshness of all agents during an episode in trafﬁc in Figure 6a and 6b.
Figure 6a depicts the temporal variance of selﬁshness for each agent.
For most agents, there are twovalleys occurred exactly during two peak periods ( i.e. , 0 − 600 s and 1 , 800 − 2 , 400 s).
This is becausefor heavy trafﬁc agents need to cooperate more closely, which can be induced by being less selﬁsh.
We can see this from the fact that selﬁshness is even lower in the second valley where the trafﬁc iseven heavier ( i.e. , 2 vs. 1 vehicles/s).
Therefore, this demonstrates that the agents learn to adjusttheir extent of cooperation to deal with dynamic environment by controlling the sharing weights.
Figure 6b shows the spatial pattern of selﬁshness at different timesteps, where the distribution ofagents is the same as the road network in Figure 2c.
The edge and inner agents tend to have verydifferent selﬁshness.
In addition, inner agents keep their selﬁshness more uniform during off-peakperiods, while they diverge and present cross-like patterns during peak periods.
This shows that handling heavier trafﬁc requires more diverse reward sharing schemes among agents to promote more sophisticatedcooperation.
In addition, we use another network that is part of Shenzhen, China with 33 intersections, and real one-hour trafﬁc ﬂows(Xu et al., 2021).
The performance is also summarized in Table 3.
The experimental results in trafﬁc verify that LToS can also handlehighly dynamic environment in networked MARL.
Conclusion
In this paper, we proposed LToS, a hierarchically decentralized framework for networked MARL.
LToS enables agents to share reward with neighbors so as to encourage agents to cooperate on theglobal objective through collectives.
For each agent, the high-level policy learns how to share rewardwith neighbors to decompose the global objective, while the low-level policy learns to optimizethe local objective induced by the high-level policies in the neighborhood.
Experimentally, wedemonstrate that LToS outperforms existing methods in both social dilemma and networked MARLscenario across scales.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] In Appendix.(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
As wejust propose a new reinforcement learning method, we do not see any potential negativesocietal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[N/A] All our experiments do not claimfor much computation.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]