Can Mean Field Control (MFC) Approximate Cooperative Multi Agent Reinforcement Learning (MARL) with Non-Uniform Interaction?
Abstract
Mean-Field Control (MFC) is a powerful tool to solve Multi-Agent Reinforcement Learning (MARL) problems.
Recent studies have shown that
MFC can well-approximate MARL when the population size is large and the agents are exchangeable.
Unfortunately, the presumption of exchangeability implies that all agents uniformly interact with one another which is not true in many practical scenarios.
In this article, we relax the assumption of exchangeability and model the interaction between agents via an arbitrary doubly stochastic matrix.
As a result, in our framework, the mean-ﬁeld ‘seen’ by different agents are different.
We prove that, if the reward of each agent is an afﬁne function of the mean-ﬁeld seen by that agent, then one can approximate such a non-uniform MARL problem via its associated MFC problem within an error of e
= O ( 1 √ N [ (cid:112) |X| + (cid:112) |U| ]) where N is the population size and |X| , |U| are the sizes of state and action spaces respectively.
Finally, we develop a
Natural Policy Gradient (NPG) algorithm that can provide a solution to the non-uniform MARL with an error O (max { e, (cid:15) } ) and a sample complexity of O ( (cid:15) − 3 ) for any (cid:15) > 0 .
INTRODUCTION
Multi-Agent Systems (MAS) are ubiquitous in the modern world.
Many engineered systems such as transportation networks, power distribution and wireless communication systems can be modeled as MAS.
Modeling, analysis and control of such systems to improve the overall performance is a central goal of research across multiple disciplines.
Multi-Agent Reinforcement Learning (MARL) is a popular approach to achieve that target.
In this article, we primarily focus on cooperative MARL where the goal is to determine policies for each individual agent such that the aggregate cumulative reward of the entire population is maximized.
However, the sizes of joint state, and action spaces of the population grows exponentially with the number of agents.
This makes the computation of the solution prohibitively hard for large MAS.
Two major computationally efﬁcient approaches have been developed to tackle this problem.
The ﬁrst approach restricts its attention to local policies.
In other words, it is assumed that each individual agent makes its decision solely based on its local state/observation.
Algorithms that fall into this category are independent Q-learning (IQL)
[Tan, 1993], centralised training and decentralised execution (CTDE) based algorithms such as VDN [Sunehag et al., 2017], QMIX [Rashid et al., 2018], WQMIX [Rashid et al., 2020], etc.
Unfortunately, none of these algorithms can provide theoretical convergence guarantees.
The other approach is called meanﬁeld control (MFC)
[Angiuli et al., 2022].
It is grounded on the idea that in an inﬁnite population of homogeneous agents, it is sufﬁcient to study the behaviour of only one representative agent in order to draw accurate conclusions about the whole population.
Recent studies have shown that, if the agents are exchangeable, then MFC can be proven to be a good approximation of MARL
[Gu et al., 2021].
Unfortunately, the idea of exchangeability essentially states that all agents in a population uniformly interact with each other (uniform means that all pairwise interactions are the same).
This is not true in many practical scenarios.
For example, in a trafﬁc control network, the congestion at an intersection is highly inﬂuenced by the control policies adopted at its immediate neighbouring intersections.
Moreover, the inﬂuence of an intersection on another intersection rapidly diminishes with increase of their separation distance.
Nonuniform interaction is a hallmark characteristic of many other MASs such as social networks, wireless networks etc.
In the absence of uniformity of the interaction between the agents, the framework of MFC no longer applies, and the problem becomes challenging.
In this paper, we come up a new result which assures that even with non-uniform interac- tions, MFC is a good choice for approximating MARL if the reward of each agent is an afﬁne function of the mean-ﬁeld distributions ‘seen’ by that agent.
We note that the behaviour of agents in multitude of social and economic networks can be modeled via afﬁne rewards (refer the examples given in [Chen et al., 2021]), and thus for many cases of practical interest, MFC can approximate MARL with non-uniform interactions.
1.1 CONTRIBUTIONS
We consider a non-uniform MARL setup where the pairwise interaction between the agents is described by an arbitrary doubly stochastic matrix (DSM).
As a result of non-uniform interaction, the so-called mean-ﬁeld effect of the population on an agent is determined by the identity of the agent.
This is in stark contrast with other existing works [Gu et al., 2021,
Mondal et al., 2022] where the presumption of exchangeability washes away the dependence on identity.
We demonstrate that, if the reward of each agent is an afﬁne function of the mean-ﬁeld distribution ‘seen’ by that agent, then the standard MFC approach can approximate the non-uniform
MARL with an error bound of e (cid:44) O ( 1 √ N [ (cid:112) |X| + (cid:112) |U| ]) , where N is the number of agents and |X| , |U| indicate the sizes of state and action spaces of individual agent.
We would like to emphasize the importance of this result.
MFC is traditionally seen as an approximation method of
MARL when the agents are exchangeable and hence their interactions are uniform.
Uniformity allows us to solve MFC problems by tracking only one representative agent.
In this paper, we show that, under certain conditions, a non-uniform
MARL can also be approximated by the MFC approach.
Thus, although the non-uniform interaction is a major part of the original MARL problem, the assumed afﬁne structure of the reward function allows us to evade non-uniformity while obtaining an approximate solution.
The key result is established in Lemma B.7 (Appendix B.2) where, using the afﬁne structure of the reward function, we show that the instantaneous reward generated from non-uniform MARL can be closely approximated by MFC-generated instantaneous reward.
Finally, using the results of [Liu et al., 2020], in section 5, we design a natural policy-gradient based algorithm that can solve MFC within an error of O ( (cid:15) ) for any (cid:15) > 0 , with a sample complexity of O ( (cid:15) − 3 ) .
Invoking our approximation result, we prove that the devised algorithm can yield a solution that is O (max { e, (cid:15) } ) error away from the optimal
MARL solution, with a sample complexity of O ( (cid:15) − 3 ) for any (cid:15) > 0 .
1.2 RELATED WORKS
Single Agent RL: The classical algorithms in single agent learning include tabular Q-learning Watkins and Dayan [1992], SARSA Rummery and Niranjan [1994], etc.
Although they provide theoretical guarantees, these algorithms can only be applied to small state-action space based systems due to their large memory requirements.
Recently Neural Network (NN) based Q -iteration Mnih et al.
[2015], and policy gradient Mnih et al.
[2016] algorithms have becomes popular due to the large expressive power of NN.
However, they cannot be applied to large MAS due to the exponential blow-up of joint state-space.
MFC as an Approximation to Uniform MARL:
Recently,
MFC is gaining traction as a scalable approximate solution to uniform MARL.
On the theory side, recently it has been proven that MFC can approximate uniform MARL within an error of O (1 / √ N )
[Gu et al., 2021].
However, the result relies on the assumption that all agents are homogeneous.
Later, this approximation result was extended to heterogeneous agents [Mondal et al., 2022].
We would like to clarify that the idea of heterogeneity is different from the idea of non-uniformity.
In the ﬁrst case, the agents are divided into multiple classes.
However, the identities of different agents within a given class are irrelevant.
In contrast, non-uniform interaction takes the identity of each agent into account.
Graphon Approximation: One possible approach to consider non-uniform agent interaction is the notion of Graphon mean-ﬁeld, which is recently gaining popularity in the noncooperative MARL setup [Caines and Huang, 2019, Cui and Koeppl, 2021].
The main idea is to approximate the ﬁnite indices of the agents as a continuum of real numbers and the discrete interaction graph between agents as a continuous, symmetric, measurable function, called graphon, in the asymptotic limit of inﬁnite population.
The unfortunate consequence of this approximation is that one is left to deal with an inﬁnite dimensional mean-ﬁeld distribution.
In order to obtain practical solution from graphon-approximation, one must therefore discretise the continuum of agent indices [Cui and Koeppl, 2021], which limits the use of this approximation.
Our paper establishes that for afﬁne reward functions, we do not need to go to the complexity of Graphon approximation.
Applications of MFC: Alongside the theory, MFC has also become popular as an application tool.
It has been used in ride-sharing [Al-Abbasi et al., 2019], epidemic management [Watkins et al., 2016], congestion control in road network [Wang et al., 2020] etc.
Learning Algorithms for MFC: Both model-free [Angiuli et al., 2022, Gu et al., 2021] and model-based [Pasztor et al., 2021] Q-learning algorithms have been proposed in the literature to solve uniform MARL via MFC with homogeneous agents.
Recently, [Mondal et al., 2022] proposed a policygradient algorithm for heterogeneous-MFC.
NON-UNIFORM INTERACTION
We consider a system comprising of N interacting agents.
The (ﬁnite) state and action spaces of each agent are denoted as X , and U respectively.
Time is assumed to belong to the discrete set, T (cid:44) { 0 , 1 , 2 , · · · } .
The state and action of i -th agent at time t are symbolized as x it and u it .
The empirical state and action distributions of the population of agents at time t are denoted by µ Nt , and ν Nt respectively, and deﬁned as follows.
[Equation]
where δ ( · ) is the indicator function.
Each agent, i ∈ [ N ] (cid:44) { 1 , · · · , N } is endowed with a reward function r and a state transition function P that are of the following forms: r : X ×U×P ( X ) ×P ( U ) → R and P :
[Equation]
( X ) ×P ( U ) → P ( X ) where P ( · ) is the set of all
Borel probability measures over its argument.
In particular, r, P take the followings as arguments: (a) the state, x it and action, a it of the corresponding agent and (b) the weighted state distribution µ i,Nt and the weighted action distribution ν i,Nt of the population as seen from the perspective of the agent.
The terms µ i,Nt and ν i,Nt are deﬁned as follows.
[Equation]
The function W : [ N ] × [ N ] → [0 , 1] dictates the inﬂuence of one agent on another.
In particular, W ( i, j ) speciﬁes how j -th agent inﬂuences i -th agent’s reward and transition functions.
Observe that, for µ i,Nt , and ν i,Nt to be probability distributions, W must be right
[Equation]
In summary, the reward received by the i -th agent at time t can be expressed as r ( x it , u it , µ i,Nt , ν i,Nt ) .
Moreover, the state of the agent at time t + 1 is decided by the following probability law: x it +1 ∼ P ( x it , u i t , µ i,N t , ν i,N t ) .
We would like to point out that, in contrast to our framework, existing works assume reward and state transition to be functions of µ Nt , ν Nt , thereby making the inﬂuence of population to be identical for every agent [Mondal et al., 2022, Gu et al., 2021].
If we take the inﬂuence function W to be uniform i.e.,
W ( i, j ) = 1 /N , ∀ i, j ∈ [ N ] , then ∀ i ∈ [ N ] , µ i,Nt = µ Nt , and ν i,Nt =
ν Nt , which forces our framework to collapse onto that described in the above mentioned papers.
At time t ∈ T , each agent is also presumed to have a policy function π t : X × P ( X ) → P ( U ) that maps ( x it , µ i,Nt ) to a distribution over the action space, U .
In simple words, a policy function π t is a rule that (probabilistically) dictates what action must be chosen by an agent given its current state and the mean-distribution of the population as observed by the agent.
Note that the policy function is presumed to be the same for all the agents as the reward function, r and the transition function, P is taken to homogeneous across the population.
Homogeneity of r, P is a common assumption in the mean-ﬁeld literature [Gu et al., 2021, Vasal et al., 2021].
For a given set of initial states x 0 (cid:44) { x i 0 } i ∈ N , the value of the sequence of policies, π (cid:44) { π t } t ∈ T , for the i -th agent is deﬁned as follows.
[Equation]
where µ i,Nt , ν i,Nt are deﬁned by (3) , (4) , respectively, and the expectation is computed over all the state-action trajectories generated by the transition function P and the sequence of policy functions, π .
The term, γ ∈ [0 , 1] is called the time discount factor.
We would like to emphasize that the value function v
i is dependent on the interaction matrix W (because so are µ i,Nt , and
ν i,Nt ).
However, such dependence is not explicitly shown to keep the notation uncluttered.
The average value function of the entire population is expressed as below.
[Equation]
The goal of MARL is to maximize v MARL ( x 0 , . )
over all policy sequences π .
Such an optimization is hard to solve in general, especially for large N .
Before concluding this section, we would like to point out two important observations that will be extensively used in many of our forthcoming results.
Remark 1.
∀ t ∈ T , the random variables { u it } i ∈
[ N ] are conditionally independent given { x it } i ∈
[ N ] .
In other words, given current states, each agent chooses its action independent of each other.
Remark 2.
∀ t ∈ T , the random variables { x it +1 } i ∈
[ N ] are conditionally independent given { x it } i ∈
[ N ] , and { u it } i ∈
[ N ] .
In other words, given current states and actions, the next state of each agent evolves independent of each other.
MEAN-FIELD CONTROL
MFC is an approximation method of N − agent MARL that takes away many of the complexities of the later.
The main idea of MFC is to consider an inﬁnite population of homogeneous agents, instead of a ﬁnite population as considered in
MARL.
The advantage of such presumption is that it allows us to draw accurate inferences about the whole population by tracking only a single representative agent.
Unfortunately, as stated before, such approximation method is known to work [Gu et al., 2021] when the interactions between different agents are uniform, i.e., W ( i, j ) = 1 /N , ∀ i, j ∈ [ N ] .
In this article, we shall show that, under certain conditions, we can show MFC as an approximation of MARL, even with non-uniform W .
Below we describe the MFC method.
As explained above, in MFC, we only need to track a single representative agent.
Let the state and action of the agent at time t be denoted as x t , and u t respectively.
Also, let µ t , ν t be the state, and action distributions of the inﬁnite population at time t .
The reward and state transition laws of the representative at time t are denoted as r ( x t , u t , µ t , ν t ) and
P ( x t , u t , µ t , ν t ) , respectively.
For a given policy sequence π (cid:44) { π t } t ∈ T , the action distribution ν t can be expressed as a deterministic function of the state distribution, µ t as follows.
[Equation]
In a similar fashion, the state distribution at time t + 1 can be written as a deterministic function of µ t as follows.
[Equation]
For an initial state distribution µ 0 , the value of a sequence of policies π (cid:44) { π t } t ∈ T , is deﬁned as written below.
[Equation]
where r MF ( µ t , π t ) (cid:44) (cid:88)
x ∈X
[Equation]
u ∈U
[Equation]
x
[Equation]
The term r MF ( µ t , π t ) indicates the average reward of the population.
Alternatively, it can also be expressed as the ensemble average of the reward of the representative agent i.e., r MF ( µ t , π t ) =
E [ r ( x t , u t , µ t , ν t )] where the expectation is computed over all possible states x t ∼ µ t , and actions u t ∼ π t ( x t , µ t ) at time t .
The mean distributions µ t , ν t are sequentially determined by (8) , (9) from a given initial state distribution, µ 0 .
The goal of MFC is to maximize v MF ( µ 0 , · ) over all policy sequences.
In the next section, we shall demonstrate that, under certain conditions, v MARL is well-approximated by v MF .
Therefore, in order to solve MARL, it is sufﬁcient to solve its associated MFC.
It is worthwhile to point out that µ t , ν t can be thought of as limiting values of the empirical distributions µ Nt , ν Nt in the asymptotic limit of inﬁnite population.
Note that, µ Nt , ν Nt and thereby, µ t , ν t are NOT dependent on W .
This makes the MFC problem agnostic of W .
In contrary, agents in the
N − agent MARL problem are inﬂuenced by the weighted mean-ﬁeld distribution { µ i,Nt , ν i,Nt } i ∈ [ N ] which do depend on W via (3) , (4) .
Therefore, unlike in the existing works, the mean-ﬁeld representative in our case cannot be described as a randomly chosen typical agent in the limit N → ∞ .
The concept of mean-ﬁeld representative, in our work, is a useful construct that, under certain conditions, can provide well-approximated solution to MARL.
In the next section, we describe how these seemingly incompatible frameworks, namely non-uniform MARL where the behaviour of agents are dependent on W , and the framework of W -agnostic MFC, can be merged together.
MFC AS AN APPROXIMATION TO NON-UNIFORM MARL
Before formally stating our main result, we would like to describe the assumptions that the result is grounded upon.
Our ﬁrst assumption is on the structure of state-transition function.
Assumption 1.
The state-transition function P is Lipschitz continuous with parameter L P with respect to the meandistribution arguments.
Mathematically, the inequality, | P ( x, u, µ 1 , ν 1 ) − P ( x, u, µ 2 , ν 2 ) |≤ L P [ | µ 1 − µ 2 | 1 + | ν 1 − ν 2 | 1 ] holds ∀ x ∈ X , ∀ u ∈ U , ∀ µ 1 , µ 2 ∈ P ( X ) and ∀
ν 1 , ν 2 ∈
P ( U ) .
The symbol | · | 1 denotes L 1 norm.
Assumption 1 states that the transition function, P , is Lipschitz continuous with respect to its mean-ﬁeld arguments.
Essentially, this implies that if the state-distribution changes from µ to µ + ∆ µ , then the corresponding change in the transition-function can be bounded by a term proportional to | ∆ µ | 1 .
Similar property holds for the change in the actiondistribution.
This useful assumption commonly appears in the mean-ﬁeld literature [Gu et al., 2021, Mondal et al., 2022, Carmona et al., 2018].
The second assumption is on the structure of r , the reward function.
Assumption 2.
The reward function, r is afﬁne with respect to mean-distribution arguments.
Mathematically, for some a ∈ R |X| , b ∈ R |U| , and f : X × U
→ R , the equality, r ( x, u, µ , ν ) =
a T µ + b T ν + f ( x, u ) holds ∀ x ∈ X , ∀ u ∈ U , ∀ µ ∈ P ( X ) , and ∀ ν ∈ P ( U ) .
Assumption 2 dictates that the reward is an afﬁne function of the mean-ﬁeld distributions.
Although this assumption does not allow us to encapsulate a large variety of reward functions, we would like to point out that the behaviour of agents in multitude of social and economic networks can be modeled via afﬁne rewards (refer the examples given in [Chen et al., 2021]).
We shall provide one explicit example at the end of this section.
We would also like to reiterate that the beneﬁt of this seemingly restrictive assumption of afﬁne reward is it allows us to apply the principles of MFC to an arbitrarily interacting N -agent system which is notoriously complex to solve in general.
The immediate corollary of Assumption 2 is that the reward function is bounded and Lipschitz continuous.
The formal proposition is given below.
Corollary 1.
If the reward function, r satisﬁes Assumption 2 , then for some M R , L R > 0 , the following holds
[Equation]
and
[Equation]
The third assumption concerns the set of allowable policy functions.
Assumption 3.
The set of allowable policy functions, Π , is such that each of its element is Lipschitz continuous with respect to its mean-state distribution argument.
Mathematically, ∀ π ∈ Π , the following inequality holds
[Equation]
≤ L Q | µ 1 − µ 2 |for some L Q > 0 and ∀ x ∈ X , ∀ µ 1 , µ 2 ∈ P ( X ) .
Assumption 3 states that the allowable policy functions must be Lipschitz continuous with respect to its state-distribution argument.
Such requirement typically holds for neural network based policies and are commonly presumed to be true in the literature
[Gu et al., 2021, Cui and Koeppl, 2021,
Pasztor et al., 2021].
The ﬁnal assumption imposes some constraints on the interaction function, W .
Assumption 4.
The interaction function, W is such that,
[Equation]
In conjunction with (5) , this assumption implies that W is doubly-stochastic.
Assumption 4 requires W to be an N × N doubly stochastic matrix (DSM).
Such presumption is commonly applied in many multi-agent tasks, e.g., distributed consensus [Alaviani and Elia, 2019a], distributed optimization [Alaviani and Elia, 2019b], and multi-agent learning [Wai et al., 2018].
We now state our main result.
Theorem 1.
Let, x 0 (cid:44) { x i 0 } i ∈ [ N ] be the initial states in an
N -agent non-uniform MARL problem and µ 0 be its associated empirical distribution deﬁned by (1) .
Assume Π to be a set of policies that obeys Assumption 3, and π (cid:44) { π t } t ∈ T is a sequence of policies such that π t ∈ Π , ∀ t ∈ T .
If
Assumption 1, 2 and 4 hold, then
[Equation]
1 − γS P − 1 − γ (cid:21) (12) whenever γS P < 1 where S P (cid:44) (1 + L Q ) + L P (2 + L Q ) ,
S R (cid:44) M R (1+ L Q )+ L R (2+ L Q ) , C P (cid:44) 2+ L P , and C R (cid:44) | b | 1 + M F .
The parameters L P , b , L Q , L R , M R have been deﬁned in Assumption 1, 2, 3, and Corollary 1, respectively.
The term M F is such that | f ( x, u ) | ≤ M F , ∀ x ∈ X , ∀ u ∈
U where f is stated in Assumption 2.
The functions v MARL , and v MF are deﬁned in (7) , (10) respectively.
Theorem 1 has an important implication.
Speciﬁcally, it states that, if reward and transition functions respectively are afﬁne and Lipschitz continuous functions of the meandistributions, and the interaction between the agents is described by a DSM, then the solution of MFC is at most
O (1 / √ N ) error away from the solution of the non-uniform
MARL problem.
Therefore, the larger the number of agents, the better is the MFC-based approximation.
It also describes how the approximation error changes with the sizes of the state, and action spaces.
Speciﬁcally, if all other parameters are kept ﬁxed, then the error increases as O ( (cid:112) |X| + (cid:112) |U| ) .
In other words, if individual state and action spaces are large, then MFC may not be a good approximation to non-uniform
MARL.
Now we shall discuss one example where the reward, transition function and the interaction function satisfy Assumption 1, 2, and 4 respectively.
Example 1.
A version of this model has been adapted in [Subramanian and Mahajan, 2019] and [Chen et al., 2021].
Consider a network of N ﬁrms operated by a single operator.
All of the ﬁrms produce the same product but with varying quality.
A discrete set X (cid:44) { 1 , 2 , · · · , Q } (state-space) describes the possible levels of quality of the product.
At each time instant, each ﬁrm decides whether to invest to improve the quality of its product which leads to the following action set: U = { 0 , 1 } .
If at time t , the i -th ﬁrm decides to invest, i.e., u it
= 1 , its current quality, x it , improves according to the following transition law.
[Equation]
x it otherwise where χ is a uniform random variable between [0 , 1] , and ¯ µ i,Nt is average product quality of its K < N neighbouring ﬁrms.
The intuition is that improving product quality might be difﬁcult if the quality maintained in the local economy is high.
Formally, we assume that each ﬁrm equally inﬂuences and is inﬂuenced by K other ﬁrms.
Hence, W ( i, j ) = 1 /K for all i, j ∈ [ N ] that inﬂuence each other and W ( i, j )
=otherwise.
The local average product quality is computed as, ¯ µ i,Nt (cid:44) (cid:80) x ∈X x µ i,Nt ( x ) where µ i,Nt is given in (3) .
At time t , the i -th ﬁrm earns a positive reward, α R x it due to its revenue, a negative reward, β R ¯ µ i,Nt due to the average local quality, and a cost λ R u it due to investment.
Hence, the total reward can be expressed as follows.
[Equation]
Clearly, in this example, Assumption 1, 2, and 4 are satisﬁed.
4.1 PROOF OUTLINE
In this subsection, we shall provide a brief sketch of the proof of Theorem 1.
Step 0 :
The difference between v MARL and v MF is essentially the time-discounted sum of differences between the average N -agent reward and average mean-ﬁeld (MF) reward at time t .
Our ﬁrst goal, therefore, is to estimate the difference between these rewards.
Step 1 : Average N -agent reward at t depends on weighted empirical distributions { µ i,Nt } i ∈ [ N ] , { ν i,Nt } i ∈ [ N ] whereas average MF reward depends on the distributions µ t , ν t .
To estimate their difference, we ﬁrst compute the difference between average N -agent reward at t and average MF reward at the same instant generated from the distribution µ Nt .
This estimate is provided by Lemma B.7 in the Appendix.
Assumption 2 is invoked to establish this result.
Step 2 : Next we estimate the difference between the average MF reward generated by µ Nt and that generated by µ t .
Lemma B.3 in the Appendix bounds this difference by a term proportional to | µ Nt − µ t | .
Step 3 : Using Lemma B.2 and B.6, we now establish a recursive relation on | µ Nt − µ t | .
Via induction, we can now write this difference as a function of t .
Step 4 : Finally, by computing a time-discounted sum of all the upper bounds described above, we arrive at the desired result.
SOLUTION OF MFC VIA NATURAL POLICY GRADIENT ALGORITHM
In this section, we develop a Natural Policy Gradient (NPG) algorithm to solve the MFC problem.
By virtue of Theorem 1, it provides an approximate solution to the non-uniform
MARL problem.
Recall from section 3 that, in MFC, it is sufﬁcient to track only one representative agent.
At time t , that agent takes its decision u t based on its own state x t , and the mean-ﬁeld state distribution µ t .
Thus, MFC essentially reduces to a single-agent Markov Decision Problem (MDP) with extended state space X × P ( X ) and action space U .
To solve MFC, it is therefore sufﬁcient to consider only stationary policies [Puterman, 2014].
Let the set of stationary policies be denoted by Π and its elements be parameterized by Φ ∈ R d .
For a given policy π Φ ∈ Π , we shall deﬁne its sequence as π Φ (cid:44) { π Φ , π Φ , · · · } .
Let,
Q Φ be the Q-function associated with policy π Φ .
We deﬁne
Q Φ ( x, µ , u ) for arbitrary x ∈ X , µ ∈ P ( X ) , and u ∈ U , as follows.
[Equation]
where the expectation is over u t +1 ∼ π Φ ( x t +1 , µ t +1 ) , and x t +1 ∼ P ( x t , u t , µ t , ν t ) , ∀ t ∈ T .
The mean-ﬁeld distributions { µ t +1 , ν t } t ∈ T are updated via deterministic update equations (8) , and (9) .
We now deﬁne the advantage function as follows.
[Equation]
where the expectation is over u ∼ π Φ ( x, µ ) .
Let, v ∗ MF ( µ 0 ) =
sup Φ ∈ R d v MF ( µ 0 , π Φ ) where v MF is the value function of MFC problem and is deﬁned in (10) .
Let, { Φ j } Jj =1 be a sequence of parameters that are generated by the NPG algorithm
[Liu et al., 2020, Agarwal et al., 2021] as follows.
[Equation]
The term η is deﬁned as the learning parameter.
The function
L ζ Φ j µ 0 and the distribution ζ Φ j µ 0 are deﬁned below.
[Equation]
NPG update (15) indicates that, at each iteration, one must solve another minimization problem to obtain the gradient direction.
It can be solved by applying a stochastic gradient descent (SGD) approach.
In particular, the update equation, in this case, turns out to be the following: w j,l +1 = w j,l − α h j,l [Liu et al., 2020].
The term α is the learning rate for this sub-problem.
The update direction h j,l can be deﬁned as follows.
[Equation]
where ( x, µ , u ) ∼ ζ Φ j µ 0 , and ˆ A Φ j is a unbiased estimator of
A Φ j .
The process to obtain the samples and the estimator has been detailed in Algorithm 1 in the Appendix I. We would like to point out that Algorithm 1 is based on Algorithmof [Agarwal et al., 2021].
We summarize the whole NPG process in Algorithm 1.
Algorithm
Input: η, α : Learning rates, J, L : Number of execution steps w 0 , Φ 0 : Initial parameters, µ 0 : Initial state distribution
Initialization: Φ ← Φ1: for j ∈ { 0 , 1 , · · · , J − 1 } do 2: w j, 0 ← w3: for l ∈ { 0 , 1 , · · · , L − 1 } do 4:
Sample ( x, µ , u ) ∼ ζ Φ j µ
and ˆ A Φ j ( x, µ , u ) using
Algorithm5:
Compute h j,l using (18) w j,l +1 ← w j,l − α h j,l 6: end for 7: w j ← 1 L (cid:80) Ll =1 w j,l 8: Φ j +1 ← Φ j + η w j 9: end for
Output: { Φ 1 , · · · , Φ J } : Policy parameters
The global converge of NPG is stated in Lemma 1 which is a direct consequence of Theorem 4.9 of [Liu et al., 2020].
However, the following assumptions are needed to establish the Lemma.
These are similar to Assumptions 2.1, 4.2, 4.
respectively in [Liu et al., 2020].
Assumption 5.
∀ Φ ∈ R d , ∀ µ 0 ∈ P ( X ) , for some χ > 0 ,
F µ 0 (Φ) − χI d is positive semi-deﬁnite where F µ 0 (Φ) can be expressed as follows.
[Equation]
Assumption 6.
∀ Φ ∈ R d , ∀ µ ∈ P ( X ) , ∀ x ∈ X , ∀ u ∈ U , |∇ Φ log π Φ ( x, µ )( u ) | 1 ≤ G for some positive constant G .
Assumption 7.
[Equation]
, Φ
[Equation]
P
[Equation]
X
[Equation]
for some positive constant M .
Assumption
[Equation]
where Φ ∗ is the parameter of the optimal policy.
Lemma 1.
Let { Φ j } Jj =1 be the sequence of policy parameters obtained from Algorithm 1.
If Assumptions 5 − 8 hold, then the following inequality holds for some η, α, J, L ,
[Equation]
for arbitrary initial parameter Φ 0 and initial state distribution µ 0 ∈ P ( X ) .
The parameter (cid:15) bias is a constant.
The sample complexity of Algorithm 1 is O ( (cid:15) − 3 ) .
The bias (cid:15) bias turns out to be small for rich neural network based policies [Liu et al., 2020].
Intuitively, it indicates the expressive power of the policy class, Π .
Lemma 1 establishes that Algorithm 1 can approximate the optimal mean-ﬁeld value function with an error bound of (cid:15) , and a sample complexity of O ( (cid:15) − 3 ) .
Using Theorem 1, we can now state the following result.
Theorem 2.
Let x 0 (cid:44) { x i 0 } i ∈ [ N ] be the initial states in an N -agent system and µ 0 their associated empirical distribution.
Assume that { Φ j } Jj =1 are the policy parameters generated from Algorithm 1, and the set of policies, Π satisﬁes Assumption 3.
If Assumptions 1, 2, 4, 5 - 8 are satisﬁed, then, for any (cid:15) > 0 , the following inequality holds for certain choices of η, α, J, L
[Equation]
where e (cid:44)
[Equation]
whenever γS P < 1 where S P is given in Theorem 1.
The term, C is a constant and the parameter (cid:15) bias is deﬁned in
Lemma 1.
The sample complexity of the process is O ( (cid:15) − 3 ) .
Proof.
Note that following inequality,
[Equation]
Using Theorem 1, the ﬁrst term can be bounded by C (cid:48)
e for some constant C (cid:48) .
The second term can be bounded by √ (cid:15) bias / (1 − γ )
+ (cid:15) with a sample complexity of O ( (cid:15) − 3 )
(Lemma 1).
Assigning C = 2 max { C (cid:48) , 1 } , we conclude the result.
Theorem 2 guarantees that Algorithm 1 can yield a policy such that its associated value is O (max { e, (cid:15) } ) error away from the optimal value of the non-uniform MARL problem.
Moreover, it also dictates such a policy can be obtained with a sample complexity of O ( (cid:15) − 3 ) .
EXPERIMENTS
Let the policy sequence that maximizes the mean-ﬁeld value function v MF ( µ 0 , · ) be denoted as π ∗ MF where µ 0 indicates the empirical distribution of the initial joint state, x N 0 .
We deﬁne the percentage error as follows.
[Equation]
We can approximately obtain π ∗ MF using Algorithm 1.
Fig.plots the value of error (deﬁned in (20) ) as a function of N for the reward, transition function, and interaction model described in Example 1.
The values of various parameters used in this numerical experiment are provided in the description of Fig.
Evidently, the error decreases with N .
Notice that the reward function stated in Example 1 (thereby that is used for generating Fig. 1) is linear in its mean-ﬁeld distribution argument.
In Fig.
2, we exhibit the error as a function of N with the following non-linear reward function.
[Equation]
The term σ is a measure of non-linearity.
All other parameters are same as stated in Example 1.
Observe that if σ = 1 , the reward function stated above turns out to be identical to the reward function given in Example 1.
In Fig.
2a, and 2b we plot error for σ = 1 . 1 , 1 . 2 respectively.
In both of these scenarios, we see the error to be a decreasing function of N .
The hyperparameter values used in
Algorithm 1 are as follows:
α = η = 10 − 3 , J = L = 10 2 .
We use a feed forward neural network with a single hidden layer as the policy approximator.
This indicates that although our MFC-based approximation results are theoretically proven for afﬁne rewards only, they empirically hold for non-afﬁne rewards as well.
The codes for generating these results are publicly available at: https://github.com/washim-uddin-mondal/UAI
CONCLUSION
In this article, we consider a multi-agent reinforcement learning (MARL) problem where the interaction between agents is described by a doubly stochastic matrix.
We prove that, if the reward function is afﬁne, one can well-approximate this non-uniform MARL problem via an associated MeanField Control (MFC) problem.
We obtain an upper bound of the approximation error as a function of the number of agents, and also propose a natural policy gradient (NPG) algorithm to solve the MFC problem with polynomial sample complexity.
The obvious drawback of our approach is the restriction on the structure of the reward function.
Therefore, extension of our techniques to non-afﬁne reward functions is an important future goal.
Acknowledgements
W. U. M., and S. V. U. were partially funded by NSF Grant
No. 1638311 CRISP Type 2/Collaborative Research: Criti-
[Figure]
cal
Transitions in the Resilience and Recovery of Interdependent Social and Physical Networks.