S AMPLE AND C OMMUNICATION -E FFICIENT D ECENTRALIZED A CTOR -C RITIC A LGORITHMS WITH F INITET IME A NALYSIS
Anonymous authors
Paper under double-blind review
A BSTRACT
Actor-critic (AC) algorithms have been widely adopted in decentralized multi-agent systems to learn the optimal joint control policy.
However, existing decentralized
AC algorithms either need to share agents’ sensitive information, e.g., local actions and policies , or are not sample and communication-efﬁcient.
In this work, we develop two decentralized AC and natural AC (NAC) algorithms that are sample and communication-efﬁcient and avoid sharing agents’ local actions and policies .
In both algorithms, agents share only noisy rewards and adopt mini-batch local policy gradient updates to improve sample and communication efﬁciency.
Particularly for decentralized NAC, we develop a decentralized Markovian SGD algorithm with an adaptive mini-batch size to efﬁciently compute the natural policy gradient.
Under Markovian sampling and linear function approximation, we prove that the proposed decentralized AC and NAC algorithms achieve the state-of-the-art sample complexities O ( (cid:15) − 2 ln (cid:15) − 1 ) and O ( (cid:15) − 3 ln (cid:15) − 1 ) , respectively, and achieve an improved communication complexity O ( (cid:15) − 1 ln (cid:15) − 1 ) .
Numerical experiments demonstrate that the proposed algorithms achieve lower sample and communication complexities than the existing decentralized AC algorithm.
I NTRODUCTION
Multi-agent reinforcement learning (MARL) has achieved great success in various application domains, including control (64; 10; 49), robotics (62), wireless sensor networks (24; 65), intelligent systems (69), etc.
In MARL, a set of fully decentralized agents interact with a dynamic environment following their own policies and collect local rewards, and their goal is to collaboratively learn the optimal joint policy that achieves the maximum expected accumulated reward.
Classical policy optimization algorithms have been well developed and studied, e.g., policy gradient (PG) (47), actor-critic (AC) (23) and natural actor-critic (NAC) (37; 7).
In particular, AC-type algorithms are more computationally tractable and efﬁcient as they take advantages of both policy gradient and value-based updates.
However, in the multi-agent setting, decentralized AC is more challenging to design compared with the centralized AC, as the algorithm updates involve sensitive agent information, e.g., local actions, rewards and policies, which must be kept locally in the decentralized learning process.
In the existing designs of decentralized AC, the agents need to share either their local actions (68; 67; 8; 36; 70; 27; 19; 26; 11) or local rewards (15; 33; 32) with their neighbors, and hence are not desired.
This issue is addressed by Algorithm 2 of (68) at the cost of learning a parameterized model to estimate the averaged reward, yet this approach requires extra learning effort and the reward estimation can be inaccurate.
Moreover, existing decentralized AC algorithms are not sample and communication-efﬁcient, and do not have ﬁnite-time convergence guarantee, especially under the practical Markovian sampling setting.
Therefore, we aim to address the following important question.
• Q1:
Can we develop a decentralized AC algorithm that is convergent, sample and communicationefﬁcient, and does not require sharing agents’ local actions and policies ?
On the other hand, as an important variant of the decentralized AC, decentralized NAC algorithm has not been formally developed and rigorously analyzed in the existing literature.
In particular, a major challenge is that we need to develop a fully decentralized and computationally tractable scheme to compute the inverse of the high dimensional Fisher information matrix, and this scheme must be both sample and communication efﬁcient.
Hence, we want to ask:
• Q2:
Can we develop a computationally tractable and communication-efﬁcient decentralized NAC algorithm that has a low ﬁnite-time sample and communication complexity?
In this study
[Table]
we provide afﬁrmative answers to the above two questions by developing fully decentralized AC and NAC algorithms that are sample and communication-efﬁcient
[Table]
and do not reveal agents’ local actions and policies.
We also develop rigorous ﬁnite-time analysis of these algorithms under Markovian sampling.
Our contributions are summarized as follows.
[Table]
1. O UR C ONTRIBUTIONS
We develop fully decentralized AC and NAC algorithms and analyze their ﬁnite-time sample and communication complexities under Markovian sampling.
Our results and comparisons to existing works are summarized in Table 1 1 .
Our decentralized AC and NAC algorithms adopt the following novel designs to accurately estimate the policy gradient in an efﬁcient way
• Noisy Rewards: In a decentralized setting, local policy gradients (estimated locally by the agents) involve the average of all agents’ local rewards.
To help agents estimate this averaged reward without revealing the raw local rewards , we let them share Gaussian-corrupted local rewards with their neighbor, and the variance of the Gaussian noise can be adjusted by each agent to reach its desired level.
• Mini-batch Updates: We apply mini-batch Markovian sampling to both the decentralized actor and critic updates.
This approach 1) helps the agents obtain accurate estimations of the corrupted averaged reward; 2) signiﬁcantly reduces the variance of policy gradient caused by Markovian sampling; and 3) signiﬁcantly reduces the communication frequency and complexity.
Moreover, for our decentralized NAC algorithm, we additionally adopt the following design to compute the inverse of the Fisher information matrix in an efﬁcient and decentralized way.
• Decentralized Natural Policy Gradient:
By reformulating the natural policy gradient as the minimizer of a quadratic program, we
develop a decentralized SGD with Markovian sampling that allows the agents to estimate the corresponding local natural gradients by communicating only scalar variables with their neighbors.
In particular, in order to minimize the sample complexity of the decentralized SGD, we set the batch size to be exponentially increasing.
Theoretically, for the ﬁrst time, we provide ﬁnite-time convergence analysis of decentralized AC and NAC algorithms under Markovian sampling.
Speciﬁcally, we prove that our decentralized AC and NAC algorithms achieve the overall sample complexities O ( (cid:15) − 2 ln (cid:15) − 1 ) and O ( (cid:15) − 3 ln (cid:15) − 1 ) , respectively, and both match the state-of-the-art complexities of their centralized versions (58).
Moreover, both decentralized algorithms achieve a signiﬁcantly reduced overall communication complexity O ( (cid:15) − 1 ln (cid:15) − 1 ) .
In particular, our analysis involves new technical developments.
First, we need to characterize the bias and variance of (natural) policy gradient and stochastic gradient caused by the noisy rewards and the inexact local averaging steps, and control them with proper choices of batch sizes and number of local averaging steps.
Second, when using decentralized Markovian SGD to compute the inverse Fisher information matrix, we need to use an exponentially increasing batch size to achieve an optimized sample complexity bound.
Such a Markovian SGD with adaptive batch size has not been studied before and can be of independent interest.
1. R ELATED W ORK
Convergence analysis of AC and NAC.
In the centralized setting, the AC algorithm was ﬁrstly proposed by (23) and later developed into the natural actor-critic (NAC) algorithm (37; 7).
Speciﬁcally, (37) does not provide any convergence result, while (22; 5) and (20; 6; 7) establish the asymptotic convergence rate of centralized AC and NAC, respectively, which is weaker than our ﬁnite-time convergence results.
Furthermore, (52; 25; 38; 59; 54) and (52) establish the ﬁnite-time convergence rate of centralized AC and NAC, respectively.
Please refer to Table 1 for their sample complexities.
Moreover, (58) improve the ﬁnite-time sample complexities of the above works to the state-of-theart result for both centralized AC and NAC by leveraging mini batch sampling, and our sample complexities match these state-of-the-art results .
In the decentralized setting, a few works have established the almost sure convergence result of
AC (15; 27; 46; 33), but they do not characterize the ﬁnite-time convergence rate and the sample complexity.
To the best of our knowledge, there is no formally developed decentralized NAC algorithm.
Decentralized TD-type algorithms.
The ﬁnite-time convergence of decentralized TD(0) has been obtained using i.i.d samples (50; 14; 51; 28) and Markovian samples (45; 51), respectively, without revealing the agents’ local actions, policies and rewards .
Decentralized off-policy TD-type algorithms have been studied in (34; 44; 9; 12).
Decentralized AC in other MARL settings.
Some works apply decentralized AC to other MARL settings that are very different from ours.
For example, (43; 36; 17; 11; 55) studied adversarial game.
(30) studied a mixed cooperative-competitive environment where each agent maximizes its own Q function (30).
(11) proposed Delay-Aware Markov Game which considers delay in Markov game.
(66; 31) studied linear control system and linear quadratic regulators instead of an MDP.
(53) studied sequential prisoner’s dilemmas.
Policy gradient algorithms.
Policy gradient (PG) and natural policy gradient (NPG) are popular policy optimization algorithms.
(1) characterizes the iteration complexity (i.e., number of episodes) of centralized PG and NPG algorithms by assuming access to exact policy gradient.
They also established a sample complexity result O ( (cid:15) − 6 ) in the i.i.d.
setting for NPG, which is worse than the state-of-the-art result O ( (cid:15) − 3 ln (cid:15) − 1 ) of both centralized NAC (58) and our decentralized NAC with
Markovian samples.
(3) proposes decentralized PG in a simple cooperative MARL setting, where all the agents share one action and the same policy, and they establish a iteration complexity in the order of O ( (cid:15) − 4 ) .
(13; 71) apply decentralized PG to Markov games.
(2) applies decentralized NPG to a different cooperative MARL setting where each agent observes its own state, takes its own action and has access to these information of its neighbors.
Value-based algorithms.
Value-based algorithms have also been develop for MARL.
Speciﬁcally, (21; 18) develop distributed Q-learning in a simpliﬁed cooperative MARL setting, where the agents share a joint action.
In particular, (18) characterizes the convergence rate of a value function-based convergence error, which is a different optimality measure from that of AC-type algorithms.
(35) applies distributed Q-learning to another cooperative MARL setting, where each agent observes its own state and takes its own action.
It establishes an asymptotic convergence guarantee, and no convergence rate is given.
(39) develops a value propagation algorithm that uses primal-dual method to minimize a soft Bellman error in the MARL setting.
Under an assumption that the variance of the stochastic gradient is uniformly bounded, it establishes a non-asymptotic convergence rate to an approximate stationary point.
R EVIEW OF M ULTI -A GENT R EINFORCEMENT L EARNING
In this section, we ﬁrst introduce some standard settings of RL.
Consider an agent that starts from an initial state s 0 ∼ ξ and collects a trajectory of Markovian samples { s t , a t , R t } t ⊂ S ×
A × R by interacting with an underlying environment (with transition kernel P ) following a parameterized policy π ω with induced stationary state distribution µ ω .
The agent aims to learn an optimal policy that maximizes the expected accumulated reward J ( ω ) =
(1 − γ ) E (cid:2) (cid:80) ∞
t =
γ t R t (cid:3) , where γ ∈ (0 , 1) is a discount factor.
The marginal state distribution is denoted as P ω ( s t ) and the visitation measure is deﬁned as ν ω ( s ) :
= (1 − γ ) (cid:80) ∞
t =
γ t P ω ( s t = s ) , both of which depend on the policy parameter ω ∈ Ω and the transition kernel P .
We also deﬁne the mixed transition kernel P ξ ( ·| s, a ) :
= γ P ( ·| s, a ) + (1 − γ ) ξ ( · ) , whose stationary state distribution is known to be ν ω .
In the multi-agent RL (MARL) setting, M agents are connected via a fully decentralized network and interact with a shared environment.
The network topology is speciﬁed by a doubly stochastic communication matrix W ∈ R M × M .
At any time t , all the agents share a common state s t .
Then, every agent m takes an action a ( m ) t following its own current policy π ( m ) t ( ·| s t ) parameterized by ω ( m ) t .
After all the actions a t := { a ( m ) t } Mm =1 are taken, the global state s t transfers to a new state s t +1 and every agent m receives a local reward R ( m ) t .
In this MARL setting, each agent m can only access the global state { s t } t , its own actions { a ( m ) t } t and rewards { R ( m ) t } t and policy π ( m ) t .
Next, deﬁne the joint policy π t ( a t
| s t ) :
= (cid:81)
Mm =1 π ( m ) t ( a ( m ) t | s t ) parameterized by ω t =
[ ω (1) t ; . .
; ω ( M ) t ] , and deﬁne the average reward R t := 1 M (cid:80)
Mm =1 R ( m ) t .
The goal of the agents is to collaboratively learn the optimal joint policy that maximizes the expected accumulated average reward J ( ω ) :
= (1 − γ ) E (cid:2) (cid:80) ∞ t =
γ t R t (cid:12)(cid:12)(cid:12) s 0 ∼ ξ (cid:3) .
Throughout, we consider the setting that the agents keep interacting with the environment and observing a trajectory of MDP transition samples, which are further used to learn the optimal joint policy.
S AMPLE AND C OMMUNICATION -E FFICIENT D ECENTRALIZED AC
In this section, we propose a decentralized actor-critic (AC) algorithm that is sample and communication-efﬁcient and avoids revealing agents’ local actions, policies and raw rewards .
We ﬁrst consider a direct extension of the centralized AC to the decentralized case.
As each agent m has its own policy π ( m ) , it aims to update the policy parameter ω ( m ) using the local policy gradient ∇ ω ( m ) J ( ω ) .
Under linear approximation of the value function V θ ( s ) ≈ φ ( s ) (cid:62) θ where φ ( s ) is the feature vector, the local policy gradient has the following stochastic approximation.
[Equation]
1) where a ( m ) t ∼ π ( m ) t (
[Equation]
s
[Equation] [Equation]
s t
[Equation]
1 ∼ P
[Equation] [Equation]
s t
[Equation]
a t ) , s (cid:48)
t
[Equation]
1 ∼
[Equation]
a
[Equation] [Equation]
Here, θ ( m ) t is agent m ’s critic parameter and ψ ( m ) t
[Equation]
| s t ) is the local score function.
It is clear that both θ ( m ) t and ψ ( m ) t ( a ( m ) t | s t ) can be obtained/computed by agent m using the local information.
However, the average reward R t requires agent m aggregating the local rewards from all the other agents, which raises concerns.
In the existing literature on decentralized AC, this issue is avoided by either 1) sharing the agents’ actions with each other instead (68; 67; 8; 36; 70; 27; 19; 26; 11), yet the action information is also highly sensitive; or 2) learning a parameterized model to estimate the average reward (68), which requires extra learning effort and does not provide an accurate estimation.
Hence, we are motivated to develop a simpler approach that provides accurate estimation of the average reward while avoids sharing raw local rewards .
Efﬁcient Policy Gradient Estimation.
We propose a decentralized policy gradient estimation scheme that improves the sample and communication efﬁciency and avoids revealing the agents’ local actions, policies and raw rewards .
First, in order for each agent to estimate the average reward R t in eq.
(1), we let each agent m generate a noisy local reward (cid:101) R ( m ) t
[Equation]
) and share with other agents, where e ( m ) t ∼ N (0 , σ 2 m ) 2 .
The noise variance is determined by the agent based on its desired level.
Speciﬁcally, every agent m ﬁrst initializes its local estimation of the averaged reward
R ( m ) t using its own noisy reward, i.e., R ( m ) t, 0 = (cid:101) R ( m ) t .
Then, each agent m performs decentralized local averaging with its neighbors N m for T (cid:48) iterations, i.e.,
[Equation]
After that, agent m obtains the ﬁnal estimate R ( m ) t := R
[Equation]
It can be shown that R ( m ) t converges to the averaged noisy reward 1 M (cid:80)
Mm =1 (cid:101) R ( m ) t exponentially fast.
Ideally, by averaging these noisy local rewards over the M agents, the variance of the noise in the ﬁnal estimation will be scaled by a factor of 1 M .
Therefore, to obtain an accurate estimation, the network needs to have a sufﬁciently large number of agents, which does not always hold in practice.
To address this issue, we let each agent m collect a mini-batch of N Markovian samples in each iteration t to estimate the local policy gradient, which then takes the following form.
[Equation]
where R ( m )
i is an estimation of R i obtained by agent m following the process described in eq.
Intuitively, each R ( m ) i is corrupted by a zero-mean noise with variance O ( 1 M ) due to averaging over the agents.
Then, the mini-batch samples further help scale the noise variance by a factor of 1 N .
Consequently, with a sufﬁciently large batch size N , we can obtain an accurate estimation of the averaged reward and hence the policy gradient.
To summarize, our decentralized policy gradient estimation scheme has the following advantages.
• Avoid sharing raw rewards: The agents share only noisy rewards (cid:101) R ( m ) t with their neighbors, and the noise variance can be adjusted based on the desired level such that R ( m ) t is unknown to the other agents.
This is in contrast to other decentralized AC algorithms where the agents need to either share local actions, rewards or collaboratively learn an additional parameterized reward model.
• Sample-efﬁcient: The mini-batch updates help greatly suppress the noise variance of the local policy gradient in (4) and improve its estimation accuracy.
On the other hand, mini-batch policy gradient also helps reduce the optimization variance caused by Markovian sampling and leads to a good ﬁnite-time sample complexity as we prove later.
We note that there is no trade-off between noise variance and sample efﬁciency here, because for highly noisy local rewards we can choose a large batch size to suppress the overall estimation error to the desired level.
• Communication-efﬁcient: The mini-batch updates also signiﬁcantly reduce the communication frequency as well as the complexity as we prove later.
In comparison, the
existing decentralized
AC requires to perform one communication round per Markovian sample.
Remark.
We note that the local mini-batch policy gradient update in eq.
(4) can be computed in an accumulative way by the agent when observing the mini-batch of transition samples on the ﬂy.
There is no need to store all these samples and perform a large batch computation.
Fully Decentralized Critic Update.
The critic parameters of the agents are updated following the standard decentralized TD-type algorithm.
Speciﬁcally, consider the t -th local critic update of each agent m .
It ﬁrst collects a mini-batch of N c Markovian samples.
Then, starting from a ﬁxed initialization θ ( m ) t, 0 = θ − 1 , agent m performs T c iterations of decentralized TD updates as follows, where { s t } t ∈ N follows the transition kernel P and a ( m ) t
[Equation]
Then, the updated critic parameter is set to be θ ( m ) t := θ ( m ) t,T c .
To further reduce the consensus error, we perform additional T (cid:48)
c steps of local model averaging, as also adopted in (12).
The pseudo code of the entire decentralized AC algorithm is summarized in Algorithms 1 and 2 below.
Algorithm
Initialize: Actor-critic parameters ω 0 , θ − 1 .
for actor iterations t = 0 , 1 , . . .
, T − 1 do (cid:73) Critic update on θ t : by Algorithm 2.
(cid:73) Collect N Markovian samples by eq.
for agents m = 1 , ..., M in parallel do (cid:73) Send noisy local rewards and perform
T (cid:48) local average steps following eq.
(cid:73)
Compute the estimated local policy gradient (cid:98) ∇ ω ( m ) J ( ω t ) following eq.
(cid:73) Actor update on ω t : ω ( m ) t +1 = ω
( m ) t + α (cid:98) ∇ ω
( m ) J ( ω t ) .
end end
Output: ω (cid:101) T with (cid:101) T uniform ∼ { 1 , 2 , . .
, T } .
Initialize:
Critic parameter θ t,
= θ − 1 .
for critic iterations t (cid:48)
= 0 , 1 , . .
, T c − 1 do (cid:73) Collect N c Markovian samples following policy π t and transition kernel P .
for agents m = 1 , ..., M in parallel do (cid:73) Send local critic parameters.
(cid:73) Decentralized TD update in eq.
end endfor iterations t (cid:48)
=
T c , ..., T c + T (cid:48)
c − 1 do for agents m = 1 , ..., M in parallel do (cid:73) θ ( m ) t,t (cid:48)
+
=
(cid:80) m (cid:48) ∈N m W m,m (cid:48)
θ ( m (cid:48) )
t,t (cid:48) .
end endOutput: θ t
= θ t,T c + T (cid:48) c .
F INITE -T IME C ONVERGENCE A NALYSIS OF D ECENTRALIZED AC
In this section, we analyze the ﬁnite-time convergence of Algorithm 1 and characterize the sample and communication complexities.
All the notations and universal constants are summarized in
Appendices A & F respectively.
We ﬁrst introduce the following standard assumptions that have been widely adopted in the existing literature.
Assumption 1.
Regarding the transition kernels P , P ξ , denote µ ω , ν ω respectively as their stationary state distributions under policy π ω and denote P , P ξ respectively as their marginal state distributions.
Then, there exist constants κ > 0 and ρ ∈ (0 , 1) such that for all t ≥ 0 ,
[Equation]
[Equation]
where d TV ( P, Q ) denotes the total-variation distance between probability measures P and Q .
Assumption 2.
There exist constants C ψ , L ψ , L π > 0 such that for all ω, (cid:101)
ω ∈ Ω , s ∈ S and a ∈ A , (cid:107) ψ ω ( a | s ) (cid:107) ≤ C ψ , (cid:107) ψ (cid:101) ω ( a | s ) − ψ ω ( a | s ) (cid:107) ≤ L ψ (cid:107) (cid:101) ω − ω (cid:107) and d TV (cid:0) π (cid:101) ω ( ·| s ) , π ω ( ·| s ) (cid:1) ≤ L π (cid:107) (cid:101) ω − ω (cid:107) .
Assumption 3.
There exists R max > 0 such that for any agent m and any Markovian sample ( s, a, s (cid:48) ) , we have 0 ≤ R ( m ) ( s, a, s (cid:48) ) ≤ R max .
Assumption 4.
The feature vectors satisfy (cid:107) φ ( s ) (cid:107) ≤ 1 for all s ∈ S .
There exists a constant λ φ >such that λ min (cid:0) E s ∼ µ ω [ φ ( s ) φ ( s ) (cid:62) ] (cid:1) ≥ λ φ for all ω .
Assumption 5.
The communication matrix W ∈ R M × M of the decentralized network is doubly stochastic, and its second largest singular value satisﬁes σ W ∈ [0 , 1) .
Assumption 1 has been widely considered in the existing literature (4; 38; 61; 56; 41; 58; 12) and it holds for any time-homogeneous Markov chains with ﬁnite-state space and any uniformly ergodic
Markov chains.
Assumption 2 introduces boundedness and Lipschitzness to the policy and its associated score function (63; 58), and holds for many parameterized policies such as Gaussian policy (25) and Boltzman policy (16).
Assumption 4 can always hold by normalizing the feature vector φ ( s )
Assumption 5 is widely used in decentralized optimization (42; 40) and multi-agent reinforcement learning (45; 51; 12), which ensures that all the decentralized agents can reach a global consensus.
With the above assumptions, we obtain the following ﬁnite-time convergence result of the decentralized AC algorithm.
Throughout, we follow (58; 54) and deﬁne the critic approximation error as ζ criticapprox :=
sup ω E s
∼ ν ω ( V ω ( s ) − φ ( s ) (cid:62) θ ∗ ω ) 2 where θ ∗ ω is the optimal critic parameter (see its deﬁnition right before Lemma D.3 in Appendix D).
We also deﬁne sample complexity as the total number of
Markovian samples required for achieving E [ (cid:107)∇ J ( ω ) (cid:107) 2 ] ≤ (cid:15) .
All the universal constants are listed in
Appendix F.
Theorem 1.
Let Assumptions 1–5 hold and adopt the hyperparameters of the decentralized TD in Algorithm 2 following Lemma D.4.
Choose α ≤ 14 L J , T (cid:48) ≥ ln M 2ln σ − 1 W .
Then, the output of the decentralized AC in Algorithm 1 has the following convergence rate.
[Equation]
Moreover, to achieve E (cid:2)(cid:13)(cid:13) ∇ J ( ω (cid:101) T ) (cid:13)(cid:13) 2 (cid:3) ≤ (cid:15) for any (cid:15) ≥ 128 C 2 ψ ζ criticapprox , we can choose T, N, N c =
O ( (cid:15) − 1 ) and T c , T (cid:48) c , T (cid:48)
=
O (ln (cid:15) − 1 ) .
Consequently, the overall sample complexity is T ( T c N c + N ) =
O ( (cid:15) − 2 ln (cid:15) − 1 ) , and the communication complexities for synchronizing linear model parameters and rewards are T ( T c + T (cid:48) c )
=
O ( (cid:15) − 1 ln (cid:15) − 1 ) and TT (cid:48)
=
O ( (cid:15) − 1 ln (cid:15) − 1 ) , respectively.
Remark.
We note that the constraint on (cid:15) is naturally induced by the critic approximation error.
In particular, if this approximation errors vanish, for example, when the dimension of features equals the number of states and the parameterized policy space is sufﬁciently expressive, then we can achieve arbitrarily small target accuracy.
To the best of our knowledge, Theorem 1 provides the ﬁrst ﬁnite-time analysis of decentralized AC under Markovian sampling.
To elaborate, under any pre-speciﬁed variance σ 2 m of the reward noise, our result shows that the gradient norm asymptotically converges to the order O ( N − 1 + N − 1 c + ζ criticapprox ) , which can be made arbitrarily close to the linear model approximation error ζ criticapprox by choosing sufﬁciently large batch sizes N, N c .
In particular, exact gradient convergence can be achieved when there is no model approximation error.
The overall sample complexity of our decentralized AC is
O ( (cid:15) − 2 ln (cid:15) − 1 ) , matching the state-of-the-art complexity result for centralized AC (58).
Moreover, with proper choices of the batch sizes N, N c
= O ( (cid:15) − 1 ) , the overall communication complexity is signiﬁcantly reduced to O ( (cid:15) − 1 ln (cid:15) − 1 ) .
The proof of Theorem 1 relies on developing several new algorithmic and technical developments to reduce the communication complexity of both the decentralized actor and critic updates while establishing tight convergence error bounds for both components.
We further elaborate on these novel technical developments below.
• To achieve an overall reduced communication complexity, we adopt mini-batch updates in both the actor and critic steps to reduce the communication frequency, as opposed to the single sample-based update adopted in the existing work on decentralized TD learning (45).
Speciﬁcally, in the analysis of the decentralized TD described in Algorithm 2 (see Lemma D.4), the mini-batch updates with batch size O ( (cid:15) − 1 ) substantially improve the communication complexity from O ( (cid:15) − 1 ln (cid:15) − 1 ) to O (ln (cid:15) − 1 ) while help achieve the state-of-the-art sample complexity.
Eventually, this together with the mini-batch updates in the decentralized actor steps help achieve the desired overall low communication complexity.
• To achieve the state-of-the-art overall sample complexity, it is critical that the policy gradient vanishes fast, which further requires a fast convergence of the decentralized TD learning.
However, although the standard T c decentralized mini-batch TD updates can yield a small convergence error for the global critic model (i
e., the average of all local critic models), it still suffers from a relatively large consensus error.
To resolve this issue, we introduce an additional T (cid:48)
c global consensus steps in Algorithm 2 to reduce the consensus error.
It is proved that a small number O (ln (cid:15) − 1 ) of such steps sufﬁces to yield a desired TD error.
• We inject random noises into the local raw rewards R ( m ) t to protect the information.
These noises introduce additional Markovian bias and variance to the local policy gradients in (4).
Fortunately, as proved in Lemma D.6, by applying mini-batch policy gradient updates, we are able to control the bias and variance induced by the noisy rewards to an acceptable level that does not affect the overall sample and communication complexities.
D ECENTRALIZED N ATURAL AC
Natural actor-critic (NAC) is a popular variant of the AC algorithm.
It utilizes a Fisher information matrix to perform a natural policy gradient update, which helps attain the globally optimal solution in terms of the function value convergence.
In this section, we develop a fully decentralized version of the NAC algorithm that is sample and communication-efﬁcient.
T }
A major challenge of developing fully decentralized NAC algorithm is computing the inverse Fisher information matrix-vector product involved in the natural policy gradient update.
To explain, ﬁrst recall the exact natural policy gradient update of the centralized NAC algorithm, i.e., ω t +1 = ω
t + αF ( ω t ) − 1 ∇ J ( ω t ) , where F ( ω t ) :=
E s t ∼ ν ωt ,a t ∼ π t ( ·| s t ) (cid:2) ψ t ( a t | s t ) ψ t ( a t | s t ) (cid:62) (cid:3) is the Fisher information matrix.
However, in the multi-agent case, it is challenging to perform the natural policy gradient update in a decentralized manner.
This is because the Fisher information matrix F ( ω t ) is based on the concatenated multi-agent score vector
[Equation]
and the inverse matrix-vector product F ( ω t ) − 1 ∇ J ( ω t ) is not separable with regard to each agent’s policy parameter dimensions.
Next, we develop a fully decentralized scheme to implement the natural policy gradient update in the multi-agent setting.
First, note that the natural policy gradient update h ( ω t ) :
= F ( ω t ) − 1 ∇ J ( ω t ) is equivalent to the solution of a quadratic program, i.e.,
[Equation]
Therefore, we can apply K steps of SGD with Markovian sampling to solve this problem and obtain an estimated natural policy gradient update.
Speciﬁcally, starting from the initialization h t, 0 = h t −(obtained in the previous iteration), in the k -th SGD step, we sample a mini-batch B t,k 3 of N k
Markovian samples to estimate ∇ f ω t ( h ) as
[Equation]
B t
[Equation]
) , where (cid:98) ∇ J ( ω t ; B t,k ) is estimated in the same decentralized way as eq.
(4) using the mini-batch of samples B t,k .
In particular, each agent m needs to compute the corresponding local gradient 1 N
[Equation]
which ψ ( m ) t ( a ( m ) i | s i ) and (cid:98) ∇ ω ( m ) J ( ω t ; B t,k ) can be computed/estimated by the agent m .
Then, it sufﬁces to obtain an estimate of the scalar ψ t ( a i | s i ) (cid:62) h t,k , which can be rewritten as (cid:80) Mm =1 ψ ( m ) t ( a ( m ) i | s i ) (cid:62) h ( m ) t,k .
This summation can be easily estimated by the decentralized agents through local averaging.
Speciﬁcally, each agent m locally computes z ( m ) i, 0 = ψ ( m ) t (
[Equation]
h ( m ) t,k and performs T z steps of local averaging, i.e., z ( m ) i,(cid:96) +
[Equation]
= (cid:80)
[Equation]
m,
[Equation]
(cid:48)
z ( m (cid:48)
[Equation]
i
[Equation]
, (cid:96) = 0 , 1 , . .
, T z − 1 .
After that, the quantity Mz ( m ) i,T z can be proven to converge to the desired summation (cid:80)
Mm =1 ψ ( m ) t ( a ( m ) i | s i ) (cid:62) h ( m ) t,k exponentially fast.
Finally, the local gradient for agent m is approximated as
[Equation]
Then, the agent m performs the following SGD updates to obtain h ( m ) t :
[Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
We emphasize that the above mini-batch SGD updates use Markovian samples .
In particular, as shown in Section 6, we need to develop an adaptive batch size scheduling scheme for this SGD in order to reduce its sample complexity.
We summarize the decentralized NAC in Algorithm 3.
F INITE - TIME C ONVERGENCE A NALYSIS OF D ECENTRALIZED NAC
To analyze the decentralized NAC, we introduce the following additional standard assumptions.
Assumption 6.
There exists a constant λ F > 0 such that λ min (cid:0) F ( ω ) (cid:1) ≥ λ F > 0 , ∀ ω ∈ Ω .
Assumption 7.
There exists C ∗ > 0 such that for ω ∗ =
arg max ω ∈ Ω J ( ω ) and any ω ∈ Ω ,
[Equation]
Assumption 6 ensures that the Fisher information matrix F ( ω ) is uniformly positive deﬁnite, and is also considered in (63; 29; 60).
Assumption 7 regularizes the discrepancy between the stationary state-action distributions ν ω ∗ ( s ) π ω ∗ ( a | s ) and ν ω ( s ) π ω ( a | s ) (52; 57).
We obtain the following ﬁnite-time convergence result of the decentralized NAC algorithm.
Throughout, we follow (52; 58; 60) and deﬁne the actor approximation error as ζ actorapprox :
= sup ω min h E
s ∼ ν ω ,a ∼ π ω ( ·| s ) (cid:2)(cid:0) ψ ω ( a | s ) (cid:62) h − A ω ( s, a ) (cid:1) 2 (cid:3) .
All universal constants are listed in Appendix F.
Theorem 2.
Let Assumptions 1–7 hold and adopt the hyperparameters of the decentralized TD in
Algorithm 2 following Lemma D.4.
Choose hyperparameters α ≤ min (cid:0) 1 ,
[Equation]
T
[Equation]
and N k ∝ (1 − ηλ F / 2) − k/ 2 .
Then, the output of Algorithm 3 satisﬁes
[Equation]
Moreover, to achieve J ( ω ∗ ) − E (cid:2) J ( ω (cid:98) T ) (cid:3) ≤ (cid:15) for any (cid:15) ≥ 2 C ψ (cid:113) c
16 ζ criticapprox + 2 c 24 ζ criticapprox + 2 C ∗ (cid:112) ζ actorapprox , we can choose T
= O ( (cid:15) − 1 ) , N, N c
= O ( (cid:15) − 2 ) , T c , T (cid:48) c , T (cid:48) , T z ,
K = O (ln (cid:15) − 1 ) .
Consequently, the overall sample complexity is T ( T c N c + N ) =
O ( (cid:15) − 3 ln (cid:15) − 1 ) , and the communication complexities for synchronizing linear model parameters and rewards are T ( T c + T (cid:48) c )
=
O ( (cid:15) − 1 ln (cid:15) − 1 ) and TT (cid:48)
=
O ( (cid:15) − 1 ln (cid:15) − 1 ) , respectively.
Theorem 2 provides the ﬁrst ﬁnite-time analysis of fully decentralized natural AC algorithm.
Our result proves that the function value optimality gap converges to the order O (cid:0) N − 1 / 2 c + (cid:113) ζ criticapprox + (cid:112) ζ actorapprox (cid:1) , which can be made arbitrarily close to the actor and critic approximation error by choosing a sufﬁciently large batch size N c .
In particular, exact global optimum can be achieved when there is no model approximation error.
We note that the overall sample complexity of our decentralized
NAC is O ( (cid:15) − 3 ln (cid:15) − 1 ) , matching the state-of-the-art complexity result for centralized NAC (58).
Moreover, with the mini-batch updates, the overall communication complexity is signiﬁcantly reduced to O ( (cid:15) − 1 ln (cid:15) − 1 ) .
Similar to that of Theorem 1, our analysis of Theorem 2 also leverages the mini-batch decentralized
TD updates to reduce the communication complexity and deal with the bias and variance of the local policy gradient introduced by noisy rewards.
In addition, decentralized NAC uses mini-batch
SGD with Markovian sampling to solve the quadratic problem in eq.
Here, we use a special geometrically increasing batch size scheduling scheme, i.e., N k ∝ (1 − ηλ F / 2) − k/ 2 , to achieve the best possible convergence rate under the total sample budget that (cid:80)
Kk
=1 N k
=
N and obtain the desired overall sample complexity result.
Such an analysis of SGD with Markovian sampling under adaptive batch size scheduling has not been studied in the literature and can be of independent interests.
E XPERIMENTS
We simulate a fully decentralized ring network with 6 agents.
Please refer to Appendix E for detailed environment setup.
We implement four decentralized AC-type algorithms and compare their performance, namely, our Algorithms 1 and 3, an existing decentralized AC algorithm (Algorithm 2 of (68)) that uses a linear model to parameterize the agents’ averaged reward (we name it DAC-RP1 for decentralized AC with reward parameterization), and our proposed modiﬁed version of DAC-RP1 to incorporate minibatch, which we refer to as DAC-RP100 with batch size N = 100 .
For our Algorithm 1, we choose T = 500 , T c = 50 , T (cid:48) c
= 10 , N c = 10 , T (cid:48)
=
T z = 5 , β = 0 .
5 , { σ m } 6 m =1 = 0 .
1 , and consider batch size choices N = 100 , 500 , 2000 .
Algorithm 3 uses the same hyperparameters as those of Algorithm 1 except that T = 2000 in Algorithm 3.
For DAC-RP1, we set learning rates β θ = 2( t + 1) − 0 .
9 , β v = 5( t + 1) − 0 .
8 and batch size N = 1 as mentioned in (68).
The modiﬁed
DAC-RP100 adopts the same learning rates as Algorithm 1 with N = 100 .
Figure 1 plots the accumulated reward J ( ω t ) v.s. communication and sample complexity.
Each curve includes 10 repeated experiments, and its upper and lower envelopes denote the 95% and 5% percentiles of the 10 repetitions, respectively.
For our decentralized AC algorithm (left two ﬁgures), its communication and sample complexities for achieving a high accumulated reward are signiﬁcantly reduced under a larger batch size N .
This matches our theoretical understanding in Theorem 1 that a large N helps reduce the communication frequency and policy gradient variance.
In comparison,
DAC-RP1 (with N = 1 ) has almost no improvement on the accumulated reward.
Moreover, although the modiﬁed DAC-RP100 (with N = 100 ) outperforms DAC-RP1, its performance is much worse than our Algorithm 1 with N = 100 .
This performance gap is due to two reasons: (i) Both DAC-RP algorithms suffer from an inaccurate parameterized estimation of the averaged reward, and their mean relative reward errors are over 100%.
In contrast, our noisy averaged reward estimation achieves a mean relative error in the range of 10 − 5 ∼ 10 − 4 ;(ii) Both DAC-RP algorithms apply only a single
TD update per-round, and hence suffers from a large mean relative TD error (about 2% and 1% for DAC-RP1 and DAC-RP100, respectively)whereas our algorithms perform multiple TD learning updates per-round and achieve a smaller mean relative TD error (about 0 .
For our decentralized
NAC algorithm (right two ﬁgures), one can make similar observations and conclusions.
C ONCLUSION
We developed fully-decentralized AC and NAC algorithms that are efﬁcient and do not reveal agents’ local actions and policies .
The agents share noisy reward information and adopt mini-batch updates to improve sample and communication efﬁciency.
Under Markovian sampling and linear function approximation, we proved that our decentralized AC and NAC algorithms achieve the state-of-the-art sample complexities O ( (cid:15) − 2 ln (cid:15) − 1 ) and O ( (cid:15) − 3 ln (cid:15) − 1 ) , respectively, and they both achieve a small communication complexity O ( (cid:15) − 1 ln (cid:15) − 1 ) .
Numerical experiments demonstrate that our algorithms achieve better sample and communication complexity than the existing decentralized AC algorithm that adopts reward parameterization.
Appendix Table of Contents
A Notations
B Proof of Theorem
C Proof of Theorem
D
Supporting Lemmas
E Experiment Setup and Additional Results
E.1 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.2 Gradient Norm Convergence Results in Ring Network . . . . . . . . . . . . . .
E.3 Additional Experiments in Fully Connected Network . . . . . . . . . . . . . . .
F Constant scalars
A N OTATIONS
Norms:
For any vector x , we denote (cid:107) x (cid:107) as its (cid:96) 2 norm.
For any matrix X , we denote (cid:107) X (cid:107) , (cid:107) X (cid:107) F as its spectral norm and Frobenius norm, respectively.
Difference matrix: ∆ := I − 1 M 11 (cid:62) , where 1 denotes a column vector that consists of 1s.
Moments of random vectors: For a random vector X , we deﬁne its variance and covariance matrix as Var ( X ) :
= E (cid:107) X − E X (cid:107) 2 and Cov ( X ) :
=
E (cid:0) [ X − E X ][ X − E X ] (cid:62) (cid:1) , respectively.
It is well known that E (cid:107) X (cid:107) 2 =
Var ( X ) + (cid:107) E X
(cid:107) 2 and that Var ( X ) =
tr [ Cov ( X )] .
Score function: At any time t , The joint score function ψ t ( a t | s t ) :
= ∇ ω ln π t ( a t | s t ) can be decomposed into individual score functions ψ ( m ) t
[Equation]
( a
[Equation]
| s t ) as ψ
[Equation]
( a t | s t )
[Equation]
[ ψ (1) t ( a (1) t | s t ) , . . .
, ψ ( M ) t ( a ( M ) t | s t )] .
Reward functions: At any time t , we denote R ( m ) t := R ( m ) ( s t , a t , s t +1 ) and R t :=
R ( s t , a t , s t +1 ) , where R ( s, a, s (cid:48) )
= 1 M (cid:80)
Mm =1 R ( m ) ( s, a, s (cid:48) ) .
Policy gradient: The policy gradient theorem (48) shows that
[Equation]
where A ω ( s, a ) :
= Q ω ( s, a ) − V ω ( s ) denotes the advantage function.
In the decentralized case, we have the approximations V ω ( s t )
≈ φ ( s t ) (cid:62) θ ,
Q ω ( s t , a t )
≈ R t + γφ
( s (cid:48)
t +1 ) (cid:62) θ where s (cid:48)
t +1 ∼
P ( ·| s t , a t ) .
Therefore, we can stochastically approximate the partial policy gradient as eq.
(1), i.e., for m = 1 , ..., M ,
[Equation]
We also deﬁne the following mini-batch stochastic (partial) policy gradient.
[Equation]
Filtrations: We deﬁne the following ﬁltrations for Algorithms 1 & 3.
[Equation]
[Equation]
B P ROOF OF T HEOREM
Theorem 1.
Let Assumptions 1–5 hold and adopt the hyperparameters of the decentralized TD in Algorithm 2 following Lemma D.4.
Choose α ≤ 14 L J , T (cid:48) ≥
[Equation]
σ
[Equation]
W .
Then, the output of the decentralized AC in Algorithm 1 has the following convergence rate.
[Equation]
Moreover, to achieve E (cid:2)(cid:13)(cid:13) ∇ J ( ω (cid:101) T ) (cid:13)(cid:13) 2 (cid:3) ≤ (cid:15) for any (cid:15) ≥ 128 C 2 ψ ζ criticapprox , we can choose T, N, N c =
O ( (cid:15) − 1 ) and T c , T (cid:48) c , T (cid:48)
=
O (ln (cid:15) − 1 ) .
Consequently, the overall sample complexity is T ( T c N c + N ) =
O ( (cid:15) − 2 ln (cid:15) − 1 ) , and the communication complexities for synchronizing linear model parameters and rewards are T ( T c + T (cid:48) c )
=
O ( (cid:15) − 1 ln (cid:15) − 1 ) and TT (cid:48)
=
O ( (cid:15) − 1 ln (cid:15) − 1 ) , respectively.
Proof.
Concatenating all the agents’ actor updates in Algorithm 1, we obtain the joint actor update ω t +1 = ω
t + α (cid:98) ∇ J ( ω t ) .
Then, the item 7 of Lemma D.5 implies that
[Equation]
where (i) and (ii) use the inequalities (cid:107) x (cid:107) 2 ≤ 2 (cid:107) x
− y (cid:107) 2 + 2 (cid:107) y (cid:107) 2 and x (cid:62) y ≥ − 12 (cid:107) x (cid:107) 2 − 12 (cid:107) y
(cid:107)for any x, y ∈ R d , respectively
[Equation]
and (iii) uses the condition that α ≤ 14 L J .
Then, summing up the inequality above over t = 0 , 1 , . .
, T − 1 yields that
[Equation]
Rearranging the equation above and taking expectation on both sides yields that
[Equation] [Equation]
where (i) uses the item 4 of Lemma D.5 and eq.
(39) of Lemma D.
(The condition of Lemma D. that T (cid:48) ≥ ln M 2ln( σ − 1 ) holds).
This proves the error bound of Theorem 1.
Finally, for any (cid:15) ≥ 128 C 2 ψ ζ criticapprox , it can be easily veriﬁed that the following hyperparameter choices make the error bound in (11) smaller than (cid:15) and also satisfy the conditions of this Theorem and those in Lemma D.4 that β ≤ min (cid:0) λ B 8 C 2 B , 4 λ B
[Equation]
1 −
[Equation]
B
[Equation]
N c
[Equation]
(cid:0)
[Equation]
B [
[Equation]
( κ
[Equation] [Equation]
[Equation] [Equation]
C P ROOF OF T HEOREM
Theorem 2.
Let Assumptions 1–7 hold and adopt the hyperparameters of the decentralized TD in
Algorithm 2 following Lemma D
[Equation]
Choose hyperparameters α ≤ min (cid:0) 1 ,
[Equation]
and N k ∝ (1 − ηλ F / 2) − k/ 2 .
Then
[Equation]
the output of Algorithm 3 satisﬁes
[Equation]
c
[Equation]
Moreover, to achieve J ( ω ∗
[Equation]
for any (cid:15) ≥ 2 C ψ (cid:113) c
16 ζ criticapprox + 2 c 24 ζ critic approx + 2 C ∗ (cid:112) ζ actorapprox , we can choose T
= O ( (cid:15) − 1 ) , N, N c
= O ( (cid:15) − 2 ) , T c , T (cid:48) c , T (cid:48) , T z ,
K = O (ln (cid:15) − 1 ) .
Consequently, the overall sample complexity is T ( T c N c + N ) =
O ( (cid:15) − 3 ln (cid:15) − 1 ) , and the communication complexities for synchronizing linear model parameters and rewards are T ( T c + T (cid:48) c )
=
O ( (cid:15) − 1 ln (cid:15) − 1 ) and TT (cid:48)
=
O ( (cid:15) − 1 ln (cid:15) − 1 ) , respectively.
Proof.
Concatenating all the agents’ actor updates in Algorithm 3, we obtain the joint actor update ω t +1 = ω
t + αh t .
Then, the item 7 of Lemma D.5 implies that
[Equation]
[Table]
[Equation]
where (i) uses the notation that h ( ω t ) (cid:52) = F ( ω t ) − 1 ∇ J ( ω t ) and the inequality that (cid:107) x (cid:107) 2 ≤ 2 (cid:107) x − y (cid:107)
+ 2 (cid:107) y (cid:107) 2 for any x,
y ∈ R d , (ii) uses the item 3 of Lemma D.7 and the inequality that x (cid:62) y ≥ −
[Equation]
2 (cid:107) y (cid:107) 2 for any x, y ∈ R d , and (iii) uses the condition that α ≤ min (cid:16)
[Equation]
Taking expectation on both sides of the above inequality, summing over t = 0 , 1 , . .
, T − 1 and rearranging, we obtain that
[Equation]
where (i) uses the item 4 of Lemma D.5 and the item 8 of Lemma D.
[Equation]
By Assumption
[Equation]
ln π ω ( s, a ) is an L ψ -smooth function of ω .
Denote ω ∗ := arg min ω ∈ Ω J ( ω ) and denote E ω ∗ as the unconditional expectation over s ∼ ν ω ∗ , a ∼ π ω ∗ ( ·| s ) .
We obtain that
E
[Equation]
where (i) uses the inequality that (cid:107) x (cid:107) 2 ≤ 2 (cid:107) x − y (cid:107) 2 + 2 (cid:107) y (cid:107) 2 for any x, y ∈ R d and the notation that h ( ω t ) (cid:52) = F ( ω t ) − 1 ∇ J ( ω t ) , (ii) uses Cauchy-Schwarz inequality, the items 3 & 6 of Lemma
D.7, the inequality that E (cid:107) X (cid:107) ≤ (cid:113) E (cid:2) (cid:107) X (cid:107) 2 (cid:3) for any random vector X and the equality that
E ω
[Equation]
(cid:3) (See its proof in Lemma 3.2 of (1).).
Averaging the inequality above over t = 0 , 1 , . .
, T − 1 and rearranging it yields that
[Equation]
[Table]
[Equation]
14) where (i) uses the deﬁnition of
KL divergence that
[Equation]
E
[Equation]
s
[Equation]
and eqs.
(13) & (54), (ii) uses the condition that α ≤ 1 and the inequality that (cid:112)(cid:80) ni
=1 x i ≤ (cid:80) ni
=1 √ x i for any n ∈ N + and x 1 , . . .
, x n ≥ 0 , (iii) uses the notations that c 17 :=
[Equation]
ψ R max λ 2 F , c 18 :=
C ψ √ c
+ c 10 L ψ (cid:16) 1+ 4 C 4 ψ λ 2 F (cid:17) , c 19 :=
C ψ √ c 11 + c 11 L ψ (cid:16) 1 + 4 C 4 ψ λ 2 F (cid:17) , c 20 :=
C ψ √ c
+ c 12 L ψ (cid:16)
+ 4 C 4 ψ λ 2 F (cid:17) , c 21 :=
C ψ √ c 13 + c 13 L ψ (cid:16) 1 + 4 C 4 ψ λ 2 F (cid:17) , c 22 :=
C ψ √ c 14 + c 14 L ψ (cid:16) 1 + 4 C 4 ψ λ 2 F (cid:17) , c 23 :=
C ψ √ c 15 + c 15 L ψ (cid:16) 1 + 4 C 4 ψ λ 2 F (cid:17) , c 24 := c
16 L ψ (cid:16) 1 + 4 C 4 ψ λ 2 F (cid:17) .
This proves the error bound of Theorem 2.
Finally, for any (cid:15) ≥ 2 C ψ (cid:113) c
16 ζ criticapprox + 2 c 24 ζ criticapprox + 2 C ∗ (cid:112) ζ actorapprox , it can be veriﬁed that the following hyperparameter choices make the error bound in (14) smaller than (cid:15) and satisfy all the conditions of this Theorem and those in Lemma D.4 that β ≤ min (cid:0) λ B 8 C 2 B , 4 λ B , 1 − σ 2 C B (cid:1) , N c ≥ (cid:0) 2 λ B + 2 β (cid:1) 192 C 2 B [1+( κ − 1) ρ ] (1 − ρ ) λ B .
[Equation]
[Table]
[Equation]
D S UPPORTING L EMMAS
First, we extend the Lemma F.3 of (12) to the Lemma D.1 below.
The item 1 of Lemma D. generalizes the case n = 1 to any n ∈ N + , the items 2 & 3 remain unchanged, and the item 4 is added for convenience of our convergence analysis.
Lemma D.1.
The doubly stochastic matrix W and the difference matrix ∆ =
I − 1 M 11 (cid:62) have the following properties:
∆ W n = W n ∆ =
W n − 1 M 11 (cid:62) for any n ∈ N + . 2.
The spectral norm of W satisﬁes (cid:107) W (cid:107) = 1 . 3.
For any x ∈ R M and n ∈ N + , (cid:107) W n ∆ x (cid:107) ≤ σ nW (cid:107) ∆ x (cid:107) ( σ W is the second largest singular value of W ).
Hence, for any H ∈ R M × M , (cid:107) W n ∆ H (cid:107) F ≤ σ nW (cid:107) ∆ H (cid:107) F . 4.
(cid:13)(cid:13) W n − 1 M 11 (cid:62) (cid:13)(cid:13) ≤ σ nW , (cid:13)(cid:13) W n − 1 M 11 (cid:62) (cid:13)(cid:13) F ≤ σ nW √ M for any n ∈ N + .
Proof.
The proof of items 2 & 3 can be found in (12).
We prove the item 1 and item 4.
We prove item 1 by induction.
The case n = 1 of the item 1 can be proved by the following two equalities, as shown in (12).
[Equation]
Suppose the case of n = k holds for a certain k ∈ N + , then the following two equalities proves the case of n = k + 1 and thus proves the item 1.
[Equation]
The item 4 can be proved by the following two inequalities.
[Equation]
[Equation]
where (i) uses the item 1, (ii) and (iii) use the item 3 ( H =
I in (iii)), and (iv) uses the fact that ∆ has M diagnoal entries 1 − 1 M and M ( M − 1) off-diagnoal entries − 1 M , which implies that
[Equation]
Next, we extend the Lemma F.2.
of (12) to the Lemma D.2 below.
Lemma D.2.
Suppose the Markovian samples { s i , a i } i ≥ 0 are generated following the policy π ω and transition kernel P (cid:48) (can be P or P ξ ), and s (cid:48)
i +1 ∼ P ( ·| s i , a i ) .
Then, for any deterministic mapping
X : S ×
A × S × S → R p × q ( p, q ∈
N
+ are arbitrary.)
such that (cid:107) X
( s, a, s (cid:48) , (cid:101) s ) (cid:107) F ≤ C x and for any s, s (cid:48) , (cid:101) s ∈ S , a ∈ A , we have
[Equation]
where X = E (cid:2) X ( s i , a i , s i +1 , s (cid:48)
i +1 )
(cid:12)(cid:12)
s i (cid:3) with s i ∼ µ ω (or ν ω ) when P (cid:48)
=
P (or P ξ ).
Proof.
Denote Y ( s, a, s (cid:48) ) :
= E (cid:101) s ∼P (cid:48) ( ·| s,a ) (cid:2)
X ( s, a, s (cid:48) , (cid:101) s ) (cid:12)(cid:12) s, a, s (cid:48) (cid:3) which satisﬁes (cid:107) Y ( s, a, s (cid:48) )
(cid:107) ≤ C x and E s i ∼ ν ω (cid:2)
Y ( s i , a i , s i +1 ) (cid:3) =
X .
Hence, Lemma F.2 of (12) can be applied to Y ( s, a, s (cid:48) ) and obtain the following inequality
[Equation]
Therefore, we obtain that
[Equation]
n n + n (cid:48) − 1 (cid:88) i = n (cid:48)
X ( s i , a i , s i +1 , s (cid:48)
i +1 )
− X (cid:13)(cid:13)(cid:13) 2 F (cid:12)(cid:12)(cid:12) { s i , a i , s i +1 } n + n (cid:48) − 1 i = n (cid:48) (cid:105) =
(cid:13)(cid:13)(cid:13) E (cid:104)n n + n (cid:48) (cid:88) i
= n (cid:48)
X ( s i , a i , s i +1 , s (cid:48)
i +1 )
− X (cid:12)(cid:12)(cid:12) { s i , a i , s i +1 } n + n (cid:48) − 1 i = n (cid:48) (cid:105)(cid:13)(cid:13)(cid:13)
F + Var (cid:104)n n + n (cid:48) − 1 (cid:88) i = n (cid:48)
X ( s i , a i , s i +1 , s (cid:48)
i +1 )
(cid:12)(cid:12)(cid:12) { s i , a i , s i +1 } n + n (cid:48) − 1 i = n (cid:48) (cid:105) ( i ) =
(cid:13)(cid:13)(cid:13)n n + n (cid:48) − 1 (cid:88) i = n (cid:48)
Y ( s i , a i , s i +1 )
− X (cid:13)(cid:13)(cid:13) 2 F +nn + n (cid:48) − 1 (cid:88) i = n (cid:48)
Var (cid:2) X
( s i , a i , s i +1 ,
[Equation]
) where (i) uses the conditional independency among { s (cid:48) i +1 } ( t +1)
N − 1 i = tN on { s i , a i , s i +1 } n + n (cid:48)
− 1 i = n (cid:48) and (ii) uses the fact that (cid:107) X
( s i , a i , s i +1 , s (cid:48)
i +1 )
(cid:107) F ≤ C x .
Finally, eq.
(18) can be proved via the following inequality.
[Equation] [Equation]
where (i) takes the conditional expectation of eq.
(20) on s (cid:48) n and (ii) uses eq.
Next, we prove the following Lemmas D.3 & D.4 on the decentralized TD in Algorithm 2.
We ﬁrst deﬁne the following useful notations.
[Equation] [Equation]
] (cid:1) > 0 , see Assumption 4.
[Equation]
=1 b ( m ) ( s, a, s (cid:48) ) .
b ( m ) t := 1 N c (cid:80) ( t +1) N c − 1 i = tN c b ( m ) ( s i , a i , s i +1 ) .
b t := 1 M (cid:80) Mm =1 b ( m ) t .
b ω :=
E s ∼ µ ω ,a ∼ π ω ( ·| s ) ,s (cid:48) ∼P ( ·| s,a ) (cid:2)
b ( s, a, s (cid:48) )
(cid:3) .
θ ∗ ω :
= B − 1 ω b ω , which is the optimal critic parameter under policy π ω .
Lemma D.3.
The following bounds hold for Algorithm 2.
(cid:107) B ( s, s (cid:48) )
(cid:107) F , (cid:107) B t (cid:107) F , (cid:107) B ω (cid:107) F ≤ C B := 1 + γ , (cid:107) b ( m ) ( s, a, s (cid:48) )
(cid:107) ,
(cid:107) b ( s, a, s (cid:48) )
(cid:107) , (cid:107) b ( m ) t
[Equation]
, (cid:107) b t (cid:107) , (cid:107) b ω (cid:107) ≤ C b := R max .
θ (cid:62) B ω θ ≤ − λ B 2 (cid:107) θ (cid:107) 2 uniformly for all ω , where λ B :=
2(1 − γ ) λ φ > 0 .
(cid:107) θ ∗ ω (cid:107) ≤ R θ := 2 C b λ B uniformly for all ω .
Proof.
We ﬁrst prove the item 1.
Notice that for any vectors x, y ∈ R d , (cid:107) xy (cid:62) (cid:107) F = (cid:118) (cid:117) (cid:117)(cid:116) d (cid:88) i =
d (cid:88)
[Equation]
Hence
[Equation]
we obtain that
[Equation] [Equation]
The other terms listed in the item 1 can be proved by applying the Jensen’s inequality to the convex function (cid:107) · (cid:107)
[Equation]
Next
[Equation]
we prove the item
[Equation]
where we use the underlying distribution that s ∼ µ ω , a ∼ π ω ( ·| s ) , s (cid:48) ∼ P ( ·| s, a ) .
We obtain that
[Equation]
[Table]
[Equation]
where (i) uses the fact that s, s (cid:48) ∼ µ ω which is the stationary state distribution with the transition kernel P and the policy π ω
[Equation]
and (ii) uses Assumption
and we denote λ B :=
2(1 − γ ) λ φ > 0 .
Finally
[Equation]
the item 3 can be proved via the following inequality.
[Equation] [Equation]
where (i) uses the item 2.
Lemma D.4.
Under Assumptions 1–5 and choosing β ≤ min (cid:0)
[Equation]
B
[Equation]
2 β
[Equation]
, Algorithm 2 has the following convergence rate.
[Equation] [Equation]
Moreover, to achieve (cid:80) Mm =
[Equation]
(cid:13)(cid:13) 2 (cid:12)(cid:12) ω
t (cid:3) ≤ (cid:15) , we can choose T c , T (cid:48) c
= O (cid:2) ln( (cid:15) − 1 ) (cid:3) and N c
= O ( (cid:15) − 1 ) .
Consequently, the sample complexity is T c N c
= O (cid:2) (cid:15) − 1 ln( (cid:15) − 1 ) (cid:3) and the communication complexity is T c + T (cid:48)
c
= O (cid:2) ln( (cid:15) − 1 ) (cid:3) .
Proof.
In Algorithm 2, by averaging the TD update rule (26) over the agents m ∈ M , we obtain that the averaged critic parameter θ t,t (cid:48) :
= 1 M (cid:80) Mm =1 θ ( m ) t,t (cid:48) follows the following update rule
[Equation]
which can be viewed as a centralized TD update using the Markovian samples { s i , a i } i from the transition kernel P and the joint policy π t .
Therefore, Theorem 4 in (58) can be directly applied to analyze this centralized TD update and obtain the following convergence rate of θ t,t (cid:48) , since all the conditions of that theorem are met 4 .
[Equation] [Equation]
where (i) uses the condition that β ≤ 4 /λ B , the item 3 of Lemma D.3 and the constant that c 1 := 1920 ( C 2 B R 2 θ + C 2 b ) [1+( κ − 1) ρ ] (1 − ρ
[Equation]
λ
[Equation]
B , (ii) uses the constant that c 3 := 2 (cid:0)(cid:13)(cid:13)
[Equation]
1 (cid:13)(cid:13) 2 + R 2 θ (cid:1) .
Next
[Equation]
we consider the consensus error (cid:107)
[Equation]
2 where we deﬁne Θ
[Equation] [Equation]
[Equation]
Note that the critic-step (26) can be rewritten into the following matrix form
[Equation]
which further implies that for any t (cid:48)
= 0 , 1 , . .
, T c − 1 ,
[Equation]
where (i) uses the item 1 of Lemma D.1, (ii) uses the item 3 of Lemma D.1 and the item 1 of Lemma
D.3, (iii) uses the condition that β ≤ 1 − σ W 2 C B and the item 1 of Lemma D.3.
Telescoping the inequality above yields that
[Equation]
where (i) uses the equality that ∆Θ
=
O due to the initial condition that Θ t, 0 =
[ θ − 1 ; . .
; θ − 1 ] (cid:62) .
On the other hand, the ﬁnal T (cid:48)
c local average steps in Algorithm 2 can be rewritten into the following matrix form
[Equation]
Hence, the average critic parameter θ t,t (cid:48) does not change in these local average steps, i.e.,
[Equation]
Therefore
[Equation]
we obtain that
[Equation]
where (i) and (ii) use the items 1 and 3 of Lemma D.1 respectively
[Equation]
(iii) uses eq.
[Equation]
(iv) denotes that c 2 := 2 (cid:0) 2 MC b 1 − σ W (cid:1) 2 .
Combining eqs.
(27) & (31) yields that
[Equation]
In the inequality above, replacing θ ( m ) t,T c + T (cid:48) c from Algorithm 2 by its corresponding variable θ ( m ) t from
Algorithm 1 proves eq.
Finally, it can be easily veriﬁed that the following hyperparameter choices make the error bound in (25) smaller than (cid:15) and also satisfy the conditions of Lemma D.4.
[Equation] [Equation]
Lemma D.5.
For any ω, (cid:101) ω ∈ Ω , s ∈ S and a ( m ) ∈
A m ( A m denotes the action space for the agent m ), the following properties hold.
(cid:107) ψ ( m ) ω ( a ( m ) | s ) (cid:107) ≤ C ψ , where ψ ( m ) ω ( a ( m ) | s ) :
= ∇ ω ( m ) ln π ( m ) ω ( a ( m ) | s ) .
(cid:107) ψ ( m ) (cid:101) ω ( a ( m ) | s ) − ψ ( m ) ω
( a ( m ) | s ) (cid:107) ≤ L ψ (cid:107) (cid:101) ω ( m ) − ω ( m ) (cid:107) .
d TV (cid:2) π ( m ) (cid:101) ω
( m ) ( ·| s ) , π ( m ) ω ( m ) ( ·| s ) (cid:3) ≤ L π (cid:107) (cid:101) ω ( m ) − ω ( m ) (cid:107) .
4. 0 ≤ V ω
( s ) , Q ω ( s, a ) ≤ (1 − γ )
R max , 0 ≤ J ( ω ) ≤ R max .
d TV (cid:2) ν ω ( ·| s ) , ν (cid:101) ω ( ·| s ) (cid:3) ≤ L ν (cid:107) ω (cid:48) − ω
(cid:107) where L ν :=
L π [1 + log ρ ( κ − 1 )
+ (1 − ρ ) − 1 ] .
d TV (cid:2) Q (cid:101) ω ( s, a ) , Q ω ( s, a ) (cid:3) ≤ L Q (cid:107) (cid:101)
ω − ω (cid:107) where L Q := 2 R max L ν 1 − γ .
J ( ω ) is L J -smooth where L J :=
R max (4 L ν + L ψ ) / (1 − γ ) .
(cid:107)∇ J ( ω ) (cid:107) ≤ D J :=
C ψ R max 1 − γ .
F ( ω ) is L F -Lipschitz where L F := 2 C ψ ( L π C ψ + L ν C ψ + L ψ ) .
h ( ω ) is L h -Lipschitz where L h :
= 2 λ − 1 F
( D J λ − 1 F L F + L J ) .
Proof.
For any ω ( m ) , (cid:101) ω ( m ) ∈
Ω m , s ∈ S and a ( m ) ∈
A m , arbitrarily select ω ( m (cid:48) )
= (cid:101) ω
( m (cid:48) ) ∈
Ω m (cid:48) , a ( m (cid:48) )
∈
A m (cid:48) for every m (cid:48) ∈ { 1 , ..., M } / { m } .
Denote ω =
[ ω (1) ; . . .
; ω ( M ) ] , (cid:101) ω
=
[ (cid:101) ω (1) ; . .
; (cid:101) ω ( M ) ] , a =
[ a (1) , . .
, a ( M ) ] .
Notice that the joint score vector has the following decomposition
[Equation]
[Equation]
Hence, the items 1 & 2 can be proved via the following two inequalities, respectively.
[Equation]
where (i) uses Assumption
[Equation]
Next
[Equation]
we prove the item 3.
Notice that
[Equation]
[Table]
[Equation]
where (i) denotes that π ω
( A | s )
= (cid:82) A π ω ( a | s ) da , (ii) uses the relation that × m ∈M A m ⊂ A , (iii) uses our construction
that ω
( m (cid:48) ) = (cid:101) ω
( m (cid:48) ) ∈ Ω m
[Equation]
, ∀ m (cid:48) ∈ { 1 , ..., M } / { m } , and (iv) uses A m (cid:48)
=
A m (cid:48) to achieve the supremum.
Therefore, the item 2 can be proved via the following inequality.
[Equation]
where (i) uses Assumption 2.
The item 4 can be proved by the following three inequalities that use Assumption 3.
[Equation]
The proof of the items 5 – 7 can be found in the proof of Lemma 3, Lemma 4 and Proposition 1 of (58), respectively.
Next, the item 8 is proved by the following inequality.
[Equation]
where (i) applies Jensen’s inequality, (ii) uses Assumption 2 and the item
[Equation]
Next, the item 9 is proved by the following inequality.
[Equation]
[Table]
where (i) applies triangle inequality and then Jensen’s inequality to the norm (cid:107)·(cid:107)
[Table]
(ii) uses Assumption 2, (iii) uses the equality that (cid:82) S ν ω ( s ) ds
= (cid:82) A π ω ( a | s ) da = 1 as well as the inequlities that (cid:82)
A
[Equation]
π
[Equation]
(based on Assumption 2) and
Finally, the item 10 is proved by the following inequality
[Equation]
where (i) uses the items 7 &
[Equation]
and (ii) uses the inequality that (cid:107) F ( ω ) − 1 (cid:107)
=
λ max ( F ( ω ) − 1 ) =
λ min [ F ( ω )] − 1 ≤ λ − 1 F for all ω (since F ( ω ) and F ( ω ) − 1 are positive deﬁnite) and the item 9.
Next, we bound the approximation error of the following stochastic (partial) policy gradients.
[Equation]
Lemma D.6.
Let Assumptions 1-5 hold and adopt the hyperparameters of the decentralized TD in
Algorithm 2 following Lemma D.4.
Choose T (cid:48) ≥
[Equation]
Then, the following properties hold.
The estimated average reward R ( m ) i has the following bias and variance bound.
[Equation]
where R i :=
[ R (1) i ; . . .
; R ( M ) i ] denotes the joint reward.
The stochastic policy gradients have the following error bound.
[Equation]
) where F t,k := σ (cid:2) F t ∪ σ (cid:0) { s i , a i , s i +
[Equation]
s (cid:48)
i +
[Equation]
{ e (
[Equation]
Proof.
We will ﬁrst prove the item 1.
When R i :=
[ R (1) i ; . . .
; R ( M ) i ] is given and ﬁxed, the randomness of (cid:101) R ( m ) i :=
R ( m ) i (1 + e ( m ) i ) and (cid:98) ∇ ω ( m ) J ( ω t ) deﬁned in eq.
(4) only comes from the noises { e ( m ) i } Mm =1 .
Since { e ( m ) i } Mm = are independent noises with zero mean and variances σ 21 , . . .
, σ 2 M , (cid:101) R i :=
[ (cid:101) R (1) i ; . .
; (cid:101) R ( M ) i ] has the following moments
[Equation]
cov (cid:2) (cid:101) R i | R i (cid:3) = diag (cid:2)
( R (1) i ) 2 σ 21 , . . .
, ( R ( M ) i ) 2 σ 2 M (cid:3) := Σ i .
Hence, (cid:98) R i :=
[ R (1) i , . . .
, R ( m ) i ] (cid:62) =
W T (cid:48) (cid:101) R i (the second “ = ” comes from eq.
(3) and the notations that (cid:101) R ( m ) i := (cid:98) R ( m ) i, 0 and that (cid:98) R ( m ) i := (cid:98) R ( m ) i,T (cid:48) ) has the moment that E (cid:2) (cid:98) R i | R i (cid:3)
= W T (cid:48) R i and
Cov (cid:2) (cid:98) R i
| R i (cid:3) =
W T (cid:48)
Σ i ( W T (cid:48) )
(cid:62) .
Therefore, eq. (37) can be proved as follows
[Equation]
=
[Equation]
11 (cid:62)
[Equation]
(cid:13)(cid:13)(cid:13)≤ (cid:13)(cid:13)(cid:13)
[Equation]
(cid:48) − 1 M 11 (cid:62) (cid:13)(cid:13)(cid:13)
[Equation]
where 1 is a M -dim vector of 1’s
[Equation]
(i) uses the inequality that (cid:107) R i (cid:107) 2 = (cid:80)
Mm =
( R ( m ) i ) 2 ≤ MR 2max (based on Assumption 3) and the item 4 of Lemma D.1.
Then, eq.
(38) can be proved as follows
M (cid:88)
[Equation]
where (i) uses the equality
that tr ( Y (cid:62) ) =
tr ( Y ) and the inequality (41) below in which X =
W T (cid:48) − 1 M 11 (cid:62) and the m -th entry of v m ∈ R M is 1 while its other entries are 0, (ii) uses the itemof Lemma D
[Equation]
1 and the equality that tr ( xy (cid:62) ) =
y (cid:62) x for any x, y ∈ R M , (iii) uses the condition that
T (cid:48) ≥ [ln M ] / [2 ln( σ − 1 W )] and the item 1 of Lemma D.1, (iv) uses Assumption 3.
[Equation]
Next
[Equation]
we will prove eq.
(39) in the item 2, where the error term can be decomposed as follows
[Equation]
where we use the following notations that
[Equation] [Equation]
Conditioned on the following ﬁltration
[Equation]
the error term (I) can be bounded as follows.
[Equation]
( a ( m ) i | s i )
(cid:12)(cid:12)(cid:12) F (cid:48) t (cid:105) ( iii
[Table]
≤
M (cid:88) m = (cid:13)(cid:13)(cid:13) E (cid:104)
N ( t +1) N − 1 (cid:88) i = tN (cid:0) R ( m )
i − R i (cid:1)(cid:12)(cid:12)(cid:12) F (cid:48) t (cid:105) ψ ( m ) t ( a ( m ) i | s i ) (cid:13)(cid:13)(cid:13)+
N
M (cid:88) m = ( t +1) N − 1 (cid:88) i = tN
Var (cid:2)(cid:0) R ( m )
i − R i (cid:1) ψ ( m ) t ( a ( m ) i | s i ) (cid:12)(cid:12) F (cid:48) t (cid:3) ( iv ) ≤ M (cid:88) m
= (cid:104)
N ( t +1) N − 1 (cid:88) i = tN
E (cid:0) R ( m )
i − R i (cid:12)(cid:12) F (cid:48) t (cid:1)(cid:105) 2 (cid:13)(cid:13) ψ
( m ) t ( a ( m ) i | s i ) (cid:13)(cid:13)+
N
M (cid:88) m = ( t +1) N − 1 (cid:88) i = tN (cid:13) (cid:13) ψ ( m ) t ( a ( m )
i | s i ) (cid:13)(cid:13) 2 var (cid:2) R ( m )
i − R i (cid:12) (cid:12) F (cid:48) t (cid:3)
[Equation]
[Equation]
where (i) uses the deﬁnitions of (cid:98) ∇ ω ( m ) J ( ω t ) and g ( m ) t deﬁned in eqs.
(33) & (44) respectively, (ii) uses the relation that E (cid:107) X (cid:107) 2 =
Var ( X ) + (cid:107) E X
(cid:107) 2 for any random vector X , (iii) uses the facts that ψ ( m ) t ( a ( m ) i | s i ) ,
R i ∈ F (cid:48) t are ﬁxed while { R ( m ) i } ( t +1) N − 1 i = tN are random and independent given F (cid:
) t , (iv) uses the equality that Var ( xY )
= (cid:80) dj
=1 var ( xy j )
= (cid:80) dj
=
y 2 j var ( x ) = (cid:107) y (cid:107) 2 var ( x ) for any random scalar x and ﬁxed vector Y =
[ y 1 , . . .
, y d ] ∈ R d (Here we denote y = ψ ( m ) t ( a ( m )
[Equation]
applies Jensen’s inequality to the convex function ( · ) 2 and uses the item 1 of Lemma D.5 as well as the fact that R i ∈ F (cid:48) t is ﬁxed, (vi) uses eqs.
(37) & (38) and the fact that the conditional distribution of R ( m ) i on R i ∈ F (cid:48) t is the same as that on F (cid:48)
t since the noise e ( m )
i is independent from any other variables.
Then we bound the error term (II) of eq.
(42) as follows.
[Equation] [Equation]
( m
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
N
[Equation] [Equation]
(cid:13)(cid:13)
[Equation]
where (i) applies Jensen’s inequality to the convex function (cid:107) · (cid:107) 2 , (ii) uses Assumption 4 and the item 1 of Lemma D.5.
To bound the error term (III) of eq.
(42), denote that
[Equation]
which satisﬁes (cid:107) X
( s, a, s (cid:48) , (cid:101) s ) (cid:107) ≤ (cid:2)
| R ( s, a, (cid:101) s ) | + (cid:13)(cid:13) γφ ( (cid:101) s )
+ φ ( s ) (cid:13)(cid:13)(cid:13)(cid:13) θ ∗ ω t (cid:13)(cid:13)(cid:3)(cid:13)(cid:13)
[Equation]
( a | s ) (cid:13)(cid:13) ≤ C ψ ( R max + 2 R
θ ) (the second ≤ uses the item 3 of Lemma D.3) and X =
E s i ∼ ν t (cid:2) X
( s i , a i , s i +1 , s (cid:48)
i +1 )
(cid:12)(cid:12) F t (cid:3) =
g ∗ t where s N , ω t ∈ F t := σ (cid:0) { θ ( m ) t (cid:48) } m ∈M , 0 ≤ t (cid:48) ≤ t ∪ { s i , a i , s (cid:48)
i +1 , { e ( m ) i } m ∈M }
[Equation]
1 i =0 ∪ { s tN } (cid:1) are ﬁxed.
Hence, Lemma D.2 yields that
[Equation]
Next, we bound the error term (IV) of eq.
Notice that g
[Equation]
[Table] [Table]
[Equation]
where (i) applies Jensen’s inequality to the convex function (cid:107) · (cid:107) 2 , (ii) uses the inequality that (cid:107) x
+ y (cid:107) 2 ≤ 2 (cid:107) x (cid:107) 2 + 2 (cid:107) y (cid:107) 2 for any x,
y ∈ R d , (iii) uses the inequality that P ( s (cid:48) | s, a ) ≤ γ − 1 P ξ ( s (cid:48) | s, a ); ∀ s, s (cid:48) ∈ S , a ∈ A , (iv) uses the equality that (cid:82) S×A
ν t ( s ) π t ( a | s ) P ξ ( (cid:101) s | s, a ) dsda =
ν t ( (cid:101) s ) , and (v) uses the notation that ζ criticapprox := sup ω
E s ∼ ν ω (cid:2)(cid:12)(cid:12) V ω ( s )
− φ ( s ) (cid:62) θ ∗ ω (cid:12)(cid:12) 2 (cid:3) .
Substituting eqs.
(47),(48),(50)&(52) into eq.
(42) yields that
[Equation]
where θ ( m ) t , ω t ∈ F t are ﬁxed
[Equation]
and we take the conditional expectation of eq.
(47) on F t ⊂ F (cid:48) t and denote
that c 4 := 4 MC
[Equation]
[Equation] [Equation]
[Equation]
Substituting eq.
(25) into the unconditional expectation of eq.
(53) yields that
[Equation]
where we denote that c 5 :
= 16 c 2 C 2 ψ , c 6 := 32 Mc 3 C 2 ψ , c 8 := 32 Mc 1 C 2 ψ .
This proves eq.
Equation (40) can be proved in the same way as that of proving eq.
There are two differences.
First, (cid:98) ∇ J ( ω t ; B t,k ) uses the minibatch B t,k of size N k while (cid:98) ∇ J ( ω t ) uses batchsize N .
Second, eq.
(40) is conditioned on the ﬁltration F t,k := σ (cid:2) F t ∪ σ (cid:0)(cid:8) s i , a i , s i +1 , s (cid:48)
i +1 , { e ( m
[Equation]
i
[Equation]
which includes not only the ﬁltration F t use by eq.
(53) but also the minibatches ∪ k − 1 k (cid:48) =
0 B t,k (cid:48) used by the previous ( k − 1 ) SGD steps.
Lemma D.7.
Implementing Algorithm 3 with η ≤
[Equation]
K ≥
[Equation]
F / 2) ( K − 1) / 2 and N k ∝ (1 − ηλ F / 2)
− k/ 2 , the involved quantities have the following properties, where E ω denotes the expectation under the underlying distributions that s ∼ ν ω , a ∼ π ω ( ·| s ) .
λ F ≤ λ max [ F ( ω )]
= (cid:107) F ( ω ) (cid:107) ≤ C 2 ψ , ∀ ω .
12 ≤ 1 − ηC 2 ψ ≤ (cid:13)(cid:13)
I − ηF ( ω ) (cid:13)(cid:13) ≤ 1 − ηλ F , so η ≤ 12 λ F . 3.
C − 2 ψ ≤ (cid:107) F ( ω ) − 1 (cid:107) ≤ λ − 1 F .
For any ω, x ∈ R d ω , x (cid:62) F ( ω ) − 1 x ≥ C − 2 ψ (cid:107) x (cid:107) 2 . 4.
(cid:13)(cid:13) h ( ω ) (cid:13)(cid:13) ≤ 1 λ F (cid:13)(cid:13)
∇ J ( ω ) (cid:13)(cid:13) ≤ D J λ F . 5.
h ( ω ) =
arg min h E ω (cid:2)(cid:0) ψ ω ( a | s ) (cid:62) h − A ω ( s, a ) (cid:1) 2 (cid:3) , so E ω (cid:2)(cid:0) ψ ω ( a | s ) (cid:62) h ( ω ) − A ω ( s, a ) (cid:1) 2 (cid:3) ≤ ζ actorapprox where s ∼ ν ω , a ∼ π ω ( ·| s ) .
[Equation]
h t approximates the natural gradient h ( ω t ) with the following error bound.
[Equation]
Proof.
The item 1 is proved by the following inequality
[Equation]
where (i) uses Assumption 6, (ii) uses the fact that F ( ω ) is positive deﬁnite implied by Assumption 6, (iii) applies Jensen’s inequality to the convex function (cid:107) · (cid:107) and (iv) uses Assumption 2.
Next we will prove the item 2.
On one hand,
[Equation]
where (i) uses the item 1, (ii) uses the condition that η ≤ 1 2 C 2 ψ .
On the other hand, λ
[Equation]
where (i) uses the fact that I − ηF ( ω ) is positive deﬁnite based on eq.
Hence, eqs.
(55) & (56) prove the item 2.
The item 3 can be proved by the fact that F ( ω ) − 1 is positive deﬁnite with minimum eigenvalue λ max [ F ( ω )] − 1 ≥ C − 2 ψ and maximum eigenvalue λ min [ F ( ω )] − 1 ≤ λ − 1 F implied by the item 1.
The item 4 can be proved by the following inequality.
[Equation]
1 F D J , where (i) uses the item 3 and (ii) uses the item 8 of Lemma D.5.
Next we will prove item 5.
Consider the following function of x ∈ R d ω .
[Equation]
Since ∇ 2 f ( ω ) = F ( ω ) is positive deﬁnite, f is strongly convex quardratic and thus it has unique minimizer h ( ω )
= F ( ω ) − 1 ∇ J ( ω ) obtained by solving h from the equation ∇ f ω ( h ) = F ( ω ) h − ∇ J ( ω )
= 0 .
Hence,
[Equation]
which proves the item 5.
The item 6 can be proved by the following inequality.
[Equation]
where (i) uses Assumption 7 and the item 5.
Multiplying − 1 to the above inequality proves the item
[Equation]
Next, the item 7 can be proved as follows.
[Equation]
where (i) uses the conditions that N k ∝ (1 − ηλ F / 2) − k/ 2 and (cid:80) K − 1 k =
N k = N and (ii) uses the condition that N ≥
[Equation]
Finally, we will prove the item 8.
Until the end of this proof, we use the underlying distribution that a i ∼ π t ( ·| s i ) , s i +1 ∼ P ξ ( ·| s i , a i ) for tN ≤ i ≤ ( t + 1)
N − 1 in the t -th iteration of the multi-agent
NAC algorithm (Algorithm 1).
The local averaging steps of z i,(cid:96) :=
[ z (1) i,(cid:96) , . .
, z ( M ) i,(cid:96) ] (cid:62) yield the following consensus error bound.
[Equation]
where z T z := 1 M (cid:80) Mm =1 z ( m ) i,T z , (i) and (ii) use the items 1 and 3 of Lemma D.1 respectively, (iii) uses the equality that (cid:107) ∆ (cid:107)
= 1 , and (iv) uses the item 1 of Lemma D.5.
Then, we deﬁne the following stochastic gradients of function f ω .
[Equation]
[Equation]
where (cid:98) ∇ ω ( m ) J ( ω t ; B t,k ) and (cid:98) ∇ J ( ω t ; B t,k ) are deﬁned in eqs.
(35) & (36) respectively.
Hence,
[Equation]
where (i) uses the equality that ψ t
[Equation]
a
[Equation]
∈M z
[Equation]
i
[Equation]
= Mz T z , (ii) uses the item 1 of
Lemma D.5.
Since, ω t , h t,k ∈ F t,k while { s i , a i } i ∈B t,k are random.
Hence,
E
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where (i) uses the inequalities that (cid:107) x + y (cid:107) 2 ≤ 2 (cid:107) x (cid:107) 2 + 2 (cid:107) y (cid:107) 2 for any x, y ∈ R d , (ii) uses the fact that h t,k ∈ F t,k , and (iii) uses eq.
and applies Lemma D.2 to the quantity that
X ( s, a, s (cid:48) , (cid:101) s ) =
ψ t ( a | s ) ψ t ( a | s ) (cid:62) in which ω t ∈ F t,k is ﬁxed and (cid:107)
X ( s, a, s (cid:48) , (cid:101) s ) (cid:107) F ≤ C 2 ψ .
Combining eqs.
(59) & (60) yields that
[Equation]
Therefore,
E
[Equation]
[Equation] [Equation]
=
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
( i )
[Equation] [Equation]
[Equation]
f
[Equation]
h
[Equation]
,k )
[Equation]
t )
[Equation]
F t,k
[Equation] [Equation]
[Equation]
( ii )
[Equation] [Equation]
t )
[Equation]
t
[Equation]
t
[Equation]
[Equation] [Equation]
=
[Equation]
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where (i) uses the inequality that (cid:107) x + y (cid:107) 2 ≤ (1+ ηλ F ) (cid:107) x (cid:107) 2 +[1+( ηλ F ) − 1 ] (cid:107) y (cid:107) 2 for any x, y ∈ R d , (ii) uses the notation that ∇ f ω t ( h )
= F ( ω t ) h − ∇ J ( ω t ) =
F ( ω t )
[ h − h ( ω t )] and the fact that ω t , h t,k ∈ F t,k , (iii) uses eq.
(61) and the item 2 of this Lemma, (iv) uses the conditions that T z ≥ ln(3 D J C 2 ψ ) ln( σ − 1 W ) and the item 7 of this Lemma, and (v) uses the notation that c 9 :
= 18 C 4 ψ D 2 J ( κ +1 − ρ ) λ 2 F (1 − ρ )
+ c 7 .
Then, taking unconditional expectation of the above inequality and iterating it over k = 0 , 1 , . .
, K −yield that
[Equation]
[Table] [Table]
[Equation]
where (i) uses the notation that h t, 0 = h t , the item 7 of this Lemma and the inequality that (cid:80) K
[Equation]
1 k
[Equation]
2 ηλ F , (ii) uses Lemma D.4, (iii) uses the inequality
that
[Equation] [Equation]
[Equation]
) 2 implied by the item 2 of this Lemma, (iv) uses the inequality that (cid:107) x
+ y + z (cid:107) 2 ≤ 3 (cid:107) x (cid:107) 2 + 3 (cid:107) y (cid:107)
+ 3 (cid:107) z (cid:107) 2 , ∀ x, y, z ∈ R d , and (v) uses the itemsof this Lemma.
Taking unconditional expectation of the above inequality and iterating it over t yield that
[Equation]
C 4 ψ M 2 σ 2 T z W + c 4 σ 2 T (cid:48) W + 16 C 2 ψ ζ criticapprox (cid:1) + 512 C 2 ψ λ 2 F (cid:16) σ 2 T (cid:48)
c W β 2 c 2 + 2 M (cid:104) c 3 (cid:16)
1 − λ B 8 β (cid:17) T c + c 1 N c (cid:105)(cid:17) + 256 c
Nηλ 3 F ( ii ) ≤ (cid:104) 3 (cid:16) 1 − ηλ F (cid:17) K (cid:105) t (cid:104)(cid:16) 1 − ηλ F (cid:17) K E
(cid:2)(cid:13)(cid:13)
h − 1 − h ( ω 0 )
(cid:13)(cid:13) 2 (cid:3) +λ 2 F (cid:0) C 4 ψ M 2 σ 2 T
z
W
+ c 4 σ 2 T (cid:48)
W + 16 C 2 ψ ζ criticapprox (cid:1) + 256 C 2 ψ λ 2 F (cid:16) σ 2 T (cid:48)
c W β 2 c 2 + 2 M (cid:104) c 3 (cid:16)
1 − λ B 8 β (cid:17) T c + c 1 N c (cid:105)(cid:17) + 128 c
Nηλ 3 F (cid:105)
+ 12 D 2 J λ 2 F (cid:16) 1 − ηλ F (cid:17) K +λ 2 F (cid:0) C 4 ψ M 2 σ 2 T z
W
+ c 4 σ 2 T (cid:48)
W + 16 C 2 ψ ζ critic approx (cid:1)
[Table]
[Equation]
[Table]
[Equation]
[Table]
where (i) uses the inequality that 3(1 − ηλ F / 2) K ≤ 1 implied by the condition that
K ≥
[Equation]
(1 − ηλ F / 2) − 1 ] , (ii) uses eq.
(62) with t = 0 , (iii) uses the condition that N ≥ 2304 C 4 ψ ( κ +1 − ρ ) ηλ
F (1 − ρ )(1 − ηλ F / 2) ( K − 1) / 2 as well as the inequalities that (cid:13)(cid:13) h − 1 − h ( ω 0 ) (cid:13)(cid:13) 2 ≤ 2 (cid:13)(cid:13)
h − 1 (cid:13)(cid:13) 2 + 2 (cid:13)(cid:13) h
( ω 0 )
(cid:13)(cid:13) 2 ∗ ≤ 2 (cid:13)(cid:13)
h − 1 (cid:13)(cid:13)
+ 2 D 2 J λ − 2 F
(* uses the item 4 of this Lemma) and that 3(1 − ηλ F / 2) K ≤ 1 , (iv) denotes that c 10 := 2 (cid:107) h − 1 (cid:107)
+ 14 D 2 J λ 2 F + c 9 λ 2 F C 4 ψ , c 11 := 48 C 4 ψ M 2 λ 2 F , c 12 := 48 c 4 λ 2 F , c 13 := 768 c 2 C 2 ψ λ 2 F , c 14 :=
Mc 3 C 2 ψ λ 2 F , c 15 :=
Mc 1 C 2 ψ λ 2 F , c 16 :
= 768 C 2 ψ λ 2 F .
This proves the item 8 of this Lemma.
E E XPERIMENT S ETUP AND A DDITIONAL R ESULTS
In this section, we present all the details of experiment setup and additional experiment results.
E. E XPERIMENT S ETUP
We simulate a fully decentralized ring network with 6 fully decentralized agents, using communication matrix with diagonal entries 0 .
4 and off-diagnonal entries 0 .
The shared state space containsstates and each agent can take 2 actions.
We adopt the softmax policy π ω ( a | s ) ∝ e ω s,a .
The entries of the transition kernel and the reward functions are independently generated from the standard
Gaussian distribution (with proper normalization of the absolute value for the transition kernel).
We use the rows of a 5-dimensional identity matrix as state features.
We set the discount factor γ = 0 .
We implement and compare four decentralized AC-type algorithms in this multi-agent MDP: our decentralized AC in Algorithm 1, our decentralized NAC in Algorithm 3, an existing decentralized
AC algorithm (Algorithm 2 of (68)) that uses a linear model to parameterize the agents’ averaged reward R ( s, a, s (cid:48) )
= (cid:80) i λ
i f i ( s, a, s (cid:48) )
(we name it DAC-RP1 for decentralized AC with reward parameterization) 5 , and our proposed modiﬁed version of DAC-RP1 to incorporate minibatch, which we refer to as DAC-RP100 with batch size N = 100 .
For our Algorithm 1, we choose
T = 500 , T c = 50 , T (cid:48) c
= 10 , N c = 10 , T (cid:48)
=
T z = 5 , β = 0 .
5 , { σ m } 6 m =1 = 0 .
1 , and consider batch size choices N = 100 , 500 , 2000 .
Algorithm 3 uses the same hyperparameters as those of
Algorithm 1 except that T = 2000 in Algorithm 3.
We select α = 10 , 50 , 200 for Algorithm 1 with
N = 100 , 500 , 2000 respectively, and T z = 5 , α = 0 .
5 , 2 , η = 0 .
8 , K = 50 , 100 , 200 ,
N k ≡ 2 , 5 , 10 for Algorithm 3 with N = 100 , 500 , 2000 , respectively.
For DAC-RP1 that was originally designed for discount factor γ = 1 , we slightly adjust it to ﬁt our setting where 0 < γ < 1 6 .
(68) deﬁned the Q-function Q θ ( s, a )
=
E (cid:2) r t +1 − J ( θ ) (cid:3) for policy parameter θ and used the temporal differences δ it =
r it +1 − µ it + V t
+
( v it )
− V t ( v it ) and (cid:101) δ it =
R t ( λ it ) − µ it + V t +1 ( v it ) − V t ( v it ) for critic
For this adjusted DAC-RP1, we select diminishing stepsizes β θ = 2( t + 1) − 0 .
9 , β v = 5( t + 1) − 0 .as recommended in (68) and use the rows of a 1600-dimensional identity matrix as the reward features { f i ( s, a, s (cid:48) ) :
s, s (cid:48) ∈ S , a ∈ A} ( i = 1 , 2 , . .
. , 1600) to fully express R ( s, a, s (cid:48) ) over all the 5 × 2 6 × 5 = 1600 triplets ( s, a, s (cid:48) ) .
DAC-RP100 has batchsizes 100 and 10 for actor and critic updates respectively, and selects constant stepsizes β v = 0 .
5 , β θ = 10 .
This setting is similar to
Algorithm 1 with N = 100 to inspect the reason of performance difference between Algorithm 1 and
DAC-RP1.
All the algorithms are repeated 10 times using initial state
and the same initial actor parameter ω 0 generated from standard Gaussian distribution.
E. G RADIENT N ORM C ONVERGENCE R ESULTS IN R ING N ETWORK
Figure 2 plots (cid:107)∇ J ( ω t ) (cid:107) 2 v.s. communication complexity ( t ( T c + T c + T (cid:48) )
=
65 t , t ( T c + T c +
T (cid:48)
+ T z ) =
70 t and 2 t for Algorithms 1 & 3, and both DAC-RP algorithms, respectively)
7 and sample complexity ( t ( T c N c + N ) , 2 t and 110 t for both of our AC-type algorithms, DAC-RP1 and
DAC-RP100, respectively).
For each curve, its upper and lower envelopes denote the 95% and 5% percentiles of the 10 repetitions, respectively.
Similar to the result of accumulative reward J ( ω t ) shown in Figure 1, it can be seen from Figure 2 that the communication and sample efﬁciency of both our decentralized AC and NAC algorithms improve with larger batchsize due to reduced gradient variance, which matches our understanding in Theorems 1 & 2.
Our decentralized AC and NAC algorithms signiﬁcantly outperform DAC-RP1 which has batchsize 1.
Using mini-batch, DAC-RP100 outperforms a lot than DAC-RP1, and converges to critical points earlier than Algorithm 1.
However, it can be seen from Figure 1 that such early convergence turns out to have much lower J ( ω t ) than Algorithm 1 with N = 100 and N c = 10 .
Such a performance gap is caused by two reasons: (i) Both DAC-RP1 and DAC-RP100 suffer from an inaccurate parameterized estimation of the averaged reward, and the mean relative estimation errors of both DAC-RP1 and DAC-RP100 are over 100% 9 .
In contrast, our noisy averaged reward estimation achieves a mean relative error in the range of
Both DAC-RP
[Equation] [Equation]
[Equation] [Equation]
− r t ) and DAC-RP100 apply only a single TD update per-round, and hence suffers from a larger mean
TD learning error (about 2% and 1% for DAC-RP1 and DAC-RP100, respectively), whereas our algorithms perform multiple TD learning updates per-round and achieve a smaller mean relative error (about 0 .
3% and 0 .
07% for our decentralized AC and NAC respectively)
All these relative errors are averaged over iterations.
E. A DDITIONAL E XPERIMENTS IN F ULLY C ONNECTED N ETWORK
To investigate the effect of network topology on the performance of our algorithms, we also conduct the above experiments on a fully connected network with 6 fully decentralized agents, using communication matrix with diagonal entries 0.4 and all the other entries 0.12.
The MDP environment and all the hyperparameters are the same as the above experiments for ring network.
Figures 3 & 4 plot the learning curves of the optimality gap J ∗ − J ( ω t ) and (cid:107)∇ J ( ω t ) (cid:107) 2 respectively for fully connected network.
To make comparison, we plot J ∗ − J ( ω t ) and (cid:107)∇ J ( ω t ) (cid:107) in Figures 5 & 2 respectively for the above experiments with ring network.
It can be seen by comparing these ﬁgures that network topology does not much affect the performance of these algorithms, so the conclusions for ring network that we summarized right before this subsection also holds for fully connected network.
[Figure]
[Equation]
F C ONSTANT SCALARS
The following global constants are frequently used.
M : The number of agents.
γ : Discount rate.
R max : The reward bound such that 0 ≤ R ( m ) ( s, a, s (cid:48) ) ≤ R max for any s, s (cid:48) ∈ S and a ∈ A (Assumption 3).
Hence, 0 ≤ R ( m ) ( s, a, s (cid:48) ) , R ( m ) i , R i ≤ R max .
σ W ∈ [0 , 1) :
The
second
largest singular
value
of W
ω ∗ := max ω J ( ω ) denotes the optimal policy parameter.
The following constants are deﬁned in Lemma D.3.
C B := 1 + γ .
C b := R max .
λ φ := λ min (cid:0) E s ∼ µ ω
[ φ ( s ) φ ( s ) (cid:62) ] (cid:1) > 0 satisﬁes Assumption 4.
λ B := 2(1 − γ ) λ φ > 0 .
(Assumption 4 implies that λ φ > 0 .)
R θ := 2 C b λ B .
The policy-related norm bounds and Lipschitz parameters are deﬁned as follows.
C ψ , L ψ , L π > 0 deﬁned in Assumption 2: For all s ∈ S , a ∈ A and ω, (cid:101) ω , (cid:107) ψ ω ( a | s ) (cid:107) ≤ C ψ , (cid:107) ψ (cid:101) ω (
a | s ) − ψ ω ( a | s ) (cid:107) ≤ L ψ (cid:107) (cid:101) ω − ω (cid:107) and d TV (cid:0) π (cid:101) ω ( ·| s ) , π ω ( ·| s ) (cid:1) ≤ L π (cid:107) (cid:101) ω − ω (cid:107) .
L ν :=
L π [1 + log ρ ( κ − 1 )
+ (1 − ρ ) − 1 ] .
L Q := 2 R max L ν 1 − γ .
L J := R max (4 L ν + L ψ ) / (1 − γ ) .
D J :=
C ψ R max 1 − γ .
L F := 2 C ψ ( L π C ψ + L ν C ψ + L ψ ) .
L h := 2 λ − 1 F
( D J λ − 1 F L F + L J ) where λ F :=
inf ω ∈ Ω λ min [ F ( ω )] > 0 ( λ min denotes the minimum eigenvalue) which satisﬁes Assumption 6.
The following constants are deﬁned to simplify the notations in the proof.
[Equation]
c
[Equation] [Equation]
R 2 θ (cid:1)
where θ − 1 is the initial parameter of decentralized TD (Algorithm 2).
[Equation]
c 5 :
[Equation]
c 6 :=
[Equation]
c 8 :=
[Equation]
2 ψ
[Equation] [Table]