Finite-Time Analysis of Fully Decentralized Single-Timescale Actor-Critic
Anonymous Author(s)
AffiliationAddress email
Abstract
Decentralized Actor-Critic (AC) algorithms have been widely utilized for multiagent reinforcement learning (MARL) and have achieved remarkable success.
Apart from its empirical success, the theoretical convergence property of decentralized AC algorithms is largely unexplored.
The existing finite-time convergenceresults are derived based on either double-loop update or two-timescale step sizesrule, which is not often adopted in real implementation.
In this work, we introducea fully decentralized AC algorithm, where actor, critic, and global reward estimatorare updated in an alternating manner with step sizes being of the same order, namely,we adopt the single-timescale update.
Theoretically, using linear approximation forvalue and reward estimation, we show that our algorithm has sample complexity of˜ O ( ϵ − 2 ) under Markovian sampling, which matches the optimal complexity withdouble-loop implementation (here, ˜ O hides a log term).
The sample complexitycan be improved to O ( ϵ − 2 ) under the i.i.d.
sampling scheme.
The central toestablishing our complexity results is the hidden smoothness of the optimal criticvariable we revealed.
We also provide a local action privacy-preserving versionof our algorithm and its analysis.
Finally, we conduct experiments to show thesuperiority of our algorithm over the existing decentralized AC algorithms.
Introduction
Multi-agent reinforcement learning (MARL) [16, 30] has been very successful in various models ofmulti-agent systems, such as robotics [14], autonomous driving [37], Go [25] etc.
MARL has beenextensively explored in the past decades; see, e.g., [18, 20, 41, 26, 8, 22].
These works either focuson the setting where an central controller is available, or assuming a common reward function for allagents.
Among the many cooperative MARL settings, the work [42] proposes the fully decentralized
MARL with networked agents.
In this setting, each agent maintains a private heterogeneous rewardfunction and agents can only access local/neighboring information through communicating with itsneighboring agents on the network.
Then, the objective of all agents is to jointly maximize the averagelong-term reward through interacting with environment modeled by multi-agent Markov decisionprocess (MDP).
They proposed the decentralized Actor-Critic (AC) algorithm to solve this MARLproblem, and showed its impressive performance.
However, the theoretical convergence propertiesof such class of decentralized AC algorithms are largely unexplored; see [41] for a comprehensivesurvey.
In this work, our goal is to establish the strong finite-time convergence results under this fullydecentralized MARL setting.
We first review some recent progresses on this line of research below.
Related works and motivations.
The first fully decentralized AC algorithm with provable convergence guarantee was proposed by [42], and they achieved asymptotic convergence results undertwo-time scale step sizes, which requires actor’s step sizes to diminish in a faster scale than the critic’sstep sizes.
The sample complexities of decentralized AC were established recently.
In particular, [6] and [11] independently propose two communication efficient decentralized AC algorithms with optimal sample complexity of O ( ε − 2 log( ε − 1 )) under Markovian sampling scheme.
Their analysis arebased on double-loop implementation, where each policy optimization step follows a nearly accuratecritic optimization step (a.k.a.
policy evaluation), i.e., solving the critic optimization subproblem toε -accuracy.
Such a double-loop scheme requires careful tuning of two additional hyper-parameters,which are the batch size and inner loop size.
In particular, the batch size and inner loop size need to beof order O ( ε − 1 ) and O (log( ε − 1 )) in order to achieve their sample complexity results, respectively.
In practice, single-loop algorithmic framework is often utilized, where one updates the actor andcritic in an alternating manner by performing only one algorithmic iteration for both of the twosubproblems; see, e.g., [23, 18, 15, 39].
The work [38] proposes a new decentralized AC algorithmbased on such a single-loop alternative update.
Nevertheless, they have to adopt two-timescale stepsizes rule to ensure convergence, which requires actor’s step sizes to diminish in a faster scale thanthe critic’s step sizes.
Due to the separation of the step sizes, the critic optimization sub-problemis solved exactly when the number of iterations tends to ∞ .
Such a restriction on the step size willslow down the convergence speed of the algorithm.
As a consequence, they only obtain sub-optimalsample complexity of O ( ε − 52 ) .
In practice, most algorithms are implemented with single-timescalestep size rule, where the step sizes for actor and critic updates are of the same order.
Though thereare some theoretical achievements for single-timescale update in other areas such as TDC [31] andbi-level optimization [4], similar theoretical understanding under AC setting is largely unexplored.
Indeed, even when reducing to single-agent setting, the convergence property of single-timescale
AC algorithm is not well established.
The works [9, 10] establish the finite-time convergence resultunder a special single-timescale implementation, where they attain the sample complexity of O ( ε − 2 ) .
However, their analysis is based on an algorithm where the critic optimization step is formulated as aleast-square temporal difference (LSTD) at each iteration, where they need to sample the transitiontuples for ˜ O ( ε − 1 ) times to form the data matrix in the LSTD problem.
Then, they solve the LSTDproblem in a closed-form fashion, which requires to invert a matrix of large size.
Later, [4] obtains thesame sample complexity using TD(0) update for critic variables under i.i.d.
sampling.
Nonetheless,their analysis highly relies on the assumption that the Jacobian of the stationary distribution is
Lipschitz continuous, which is not justified in their work.
The above observations motivate us to ask the following question:
Can we establish finite-time convergence result for decentralized AC algorithm with single-timescale step sizes rule?
Main contributions.
By answering this question positively, we have the following contributions:
• We design a fully decentralized AC algorithm, which employs a single-timescale step sizesrule and adopts Markovian sampling scheme.
The proposed algorithm allows communicationbetween agents for every K c iterations with K c being any integer lies in [1 , O ( ε − 12 )] , ratherthan communicating at each iteration as adopted by previous single-loop decentralized ACalgorithms [38, 42].• Using linear approximation for value and reward estimation, we establish the finite-timeconvergence result for such an algorithm under the standard assumptions.
In particular, weshow that the algorithm has the sample complexity of e O ( ε − 2 ) , which matches the optimal complexity up to a logarithmic term.
In addition, we show that the logarithmic term can be removed under the i.i.d.
sampling scheme.
Note that these convergence results are valid for
all the above mentioned choices for K c .
• To preserve the privacy of local actions, we propose a variant of our algorithm which utilizesnoisy local rewards for estimating global rewards.
We show that such an algorithm willmaintain the optimal sample complexity at the expense of communicating at each iteration.
The underlying principle for obtaining the above convergence results is that we reveal the hiddensmoothness of the optimal critic variable , so that we can derive an approximate descent on theaveraged critic’s optimal gap at each iteration.
Consequently, we can resort to the classic convergenceanalysis for alternating optimization algorithms to establish the approximate ascent property of theoverall optimization process, which leads to the final sample complexity results.
Another technical highlight is the Lyapunov function we construct for measuring the progress of ouralgorithm.
Such a construction is motivated by [4], which analyzes bi-level optimization algorithm.
However, our Lyapunov function is different from theirs as it involves the additional optimal gap ofaveraged critic and reward estimator, which is necessary for dealing with the decentralized setting.
We finish this section by remarking that our convergence results are even new for single agent ACalgorithms under the setting of single-timescale step sizes rule.
Preliminary
In this section, we introduce the problem formulation and the policy gradient theorem, which servesas the preliminary for the analyzed decentralzed AC algorithm.
Suppose there are multiple agents aiming to independently optimize a common global objective, andeach agent can communicate with its neighbors through a network.
To model the topology, we definethe graph as G =
( N , E ) , where N is the set of nodes with |N| =
N and E is the set of edges with|E| = E .
In the graph, each node represents an agent, and each edge represents a communicationlink.
The interaction between agents follows the networked multi-agent MDP.
2.1 Markov decision process
A networked multi-agent MDP is defined by a tuple ( G , S , {A i } i ∈N , P , { r i } i ∈ [ N ] , γ ) .
G denotes thecommunication topology (the graph), S is the finite state space observed by all agents, A i representsthe finite action space of agent i .
Let A :
= A 1 × · · · × A N denote the joint action space and
P ( s ′ | s, a ) : S ×
A × S → [0 , 1] denote the transition probability from any state s ∈ S to any state 107 s ′ ∈ S for any joint action a ∈ A .
r i : S ×
A → R is the local reward function that determines the 108 reward received by agent i given transition ( s, a ) ; γ ∈ [0 , 1] is the discount factor.
For simplicity, we will use a :=
[ a 1 , · · · , a N ] to denote the joint action, and θ :=
[ θ 1 , · · · , θ N ] ∈
R d θ × N to denote joint parameters of all actors, with θ i ∈
R d θ .
Note that different actors may havedifferent number of parameters, which is assumed to be the same for our paper without loss ofgenerality.
The MDP goes as follows:
For a given state s , each agent make its decision a i basedon its policy a i ∼ π
θ i ( ·| s ) .
The state transits to the next state s ′ based on the joint action of all theagents: s ′ ∼ P ( ·| s, a ) .
Then, each agent will receive its own reward r i ( s, a ) .
For the notation brevity,we assume that the reward function mapping is deterministic and does not depend on the next statewithout loss of generality.
The stationary distribution induced by the policy π θ and the transitionkernel is denoted by µ π θ ( s ) .
Our objective is to find a set of policies that maximize the accumulated discounted mean rewardreceived by agents
[Equation]
Here, k represents the time step.
¯ r ( s k , a k ) :
= 1 N
P
Ni =1 r
i
( s k , a k ) is the mean reward among agentsat time step k .
The randomness of the expectation comes from the initial state distribution µ 0 ( s ) , thetransition kernel P , and the stochastic policy π θ i ( ·| s ) .
2.2 Policy gradient Theorem
Under the discounted reward setting, the global state-value function, action-value function, andadvantage function for policy set θ , state s , and action a , are defined as
[Equation]
To maximize the objective function defined in (1), the policy gradient [28] can be computed as follow
[Equation]
where d π θ ( s ) := (1 − γ )
P ∞ k =0 γ k P ( s k = s ) is the discounted state visitation distribution underpolicy π θ , and ψ π θ ( s, a ) := ∇ log π θ ( s, a ) is the score function.
Following the derivation of [42], the policy gradient for each agent under discounted reward settingcan be expressed as
[Equation]
Decentralized single-timescale actor-critic
[Table]
[Table]
14: end
15: end for
We introduce the decentralized single-timescale AC algorithm; see Algorithm 1.
In the remainingparts of this section, we will explain the updates in the algorithm in details.
In fully-decentralized MARL, each agent can only observe its local reward and action, while tryingto maximize the global reward (mean reward) defined in (1).
The decentralized AC algorithm solvesthe problem by performing online updates in an alternative fashion.
Specifically, we have N pairs ofactor and critic.
In order to maximize J ( θ ) , each critic tries to estimate the global state-value function
V π θ ( s ) defined in (2), and each actor then updates its policy parameter based on approximated policygradient.
We now provide more details about the algorithm.
Critics’ update.
We will use ω i ∈ R d ω to denote the i th critic’s parameter and ¯ ω := 1 N
P
Ni =1 ω i torepresent the averaged parameter of critic.
The i th critic approximates the global value function as
V π θ ( s )
≈ ˆ V ω
i ( s ) .
As we will see, the critic’s approximation error can be categorized into two parts, namely, theconsensus error 1 N
P
[Equation]
, which measures how close the critics’ parameters are; and the 145 approximation error
∥ ¯ ω − ω ∗ ( θ ) ∥ , which measures the approximation quality of averaged critic.
In order for critics to reach consensus, we perform the following update for all critics
[Equation]
[Equation] where W ∈ R n × n is a weight matrix for communication among agents, whose property will bespecified in Assumption 5; K c denotes the consensus frequency.
To reduce the approximation error, we will perform the local TD(0) update [29] as
[Equation]
where ξ := ( s, a, s ′ ) represents a transition tuple, g ic ( ξ, ω ) :
= δ i ( ξ, ω ) ∇ ˆ V ω ( s ) is the update direction,δ i ( ξ, ω ) :
= r i ( s, a ) + γ ˆ V ω ( s ′ ) − ˆ V ω ( s ) is the local temporal difference error (TD-error).
β k is thestep size for critic at iteration k .
Q
R ω projects the parameter into a ball of radius of R ω containingthe optimal solution, which will be explained when discussing Assumption 1 and 2.
Actors’ update.
We will use stochastic gradient ascent to update the policy’s parameter, and thestochastic gradient is calculated based on policy gradient theorem in (3).
The advantage function
A π θ ( s, a ) can be estimated by
[Equation]
with a sampled from π θ ( ·| s ) .
However, to preserve the privacy of each agents, the local rewardcannot be shared to other agents under the fully decentralized setting.
Thus, the averaged reward¯ r ( s k , a k ) is not directly attainable.
Consequently, we need a strategy to approximate the averagedreward.
In this paper, we will adopt the strategy proposed in [42].
In particular, each agent i will havea local reward estimator with parameter λ i ∈ R d λ , which estimates the global averaged reward as¯ r ( s k , a k )
≈
ˆ r
λ i ( s k , a k ) .
Thus, the update of the i th actor is given by
[Equation]
where ˆ δ ( ξ, ω, λ ) :
= ˆ r λ ( s, a ) + γ ˆ V ω ( s ′ )
− ˆ V ω ( s ) is the approximated advantage function.
α k is thestep size for actor’s update at iteration k .
Reward estimators’ update.
Similar to critic, each reward estimator’s approximation error can bedecomposed into consensus error and the approximation error.
For each local reward estimator, we perform the consensus step to minimize the consensus error as
[Equation]
To reduce the approximation error, we perform a local update of stochastic gradient descent.
[Equation]
where g ir ( ξ, λ ) :
= ( r i ( s, a ) −
ˆ r λ
( s, a ))
∇ ˆ r λ ( s, a ) is the update direction.
η k is the step size forreward estimator at iteration k .
Note the calculation of g ir ( ξ, λ ) does not require the knowledge of s ′ ;we use ξ in (8) just for notation brevity.
Similar to critic’s update,
Q
R λ projects the parameter into a 173 ball of radius of R λ containing the optimal solution.
In our Algorithm 1, we will use the same order for α k , β k , and η k and hence, our algorithm is insingle-timescale .
Linear approximation for analysis.
In our analysis, we will use linear approximation for both criticand reward estimator variables, i.e. ˆ V ω ( s ) :
= ϕ ( s ) T ω ; ˆ r λ ( s, a ) :
= φ ( s, a ) T λ , where ϕ ( s ) : S →
R d ω and φ ( s, a ) : S ×
A → R d λ are two feature mappings, whose property will be specified in thediscussion of Assumption 1.
Algorithm for preserving the local action.
Note that in Algorithm 1, the reward estimators needthe knowledge of joint actions in order to estimate the global rewards.
To preserve the privacy oflocal actions, we further propose a variant of Algorithm 1, which estimates the global rewards bycommunicating noisy local rewards; see [6] for the original idea.
However, to maintain the optimalsample complexity, such an approach requires O (log( ε − 1 )) communication rounds for each iteration.
We postpone the detailed design and analysis of such an algorithm scheme into Appendix B.
Remarks on sampling scheme.
The unbiased update for critic and actor variables requires samplingfrom µ π θ and d π θ , respectively.
However, in practical implementations, states are usually collectedfrom an online trajectory (Markovian sampling), whose distribution is generally different for µ π θand d π θ .
Such a distribution mismatch will inevitably cause biases during the update of critic andactor variables.
One has to bound the corresponding error terms when analyzing the algorithm.
Inthis work, we will provide the analysis for both sampling schemes.
Main Results
In this section, we first introduce the technical assumptions used for our analysis, which are standardin the literature.
Then, we present the convergence results for both actor and critic variables underi.i.d.
sampling and Markovian sampling.
4.1 Assumptions
Assumption 1 (bounded rewards and feature vectors) .
All the local rewards are uniformly bounded,i.e., there exists a positive constants r max such that | r i ( s, a ) | ≤ r max , for all feasible ( s, a ) andi ∈ [ N ] .
The norm of feature vectors are bounded such that for all s ∈ S , a ∈ A ,
∥ ϕ ( s ) ∥ ≤ 200 1 , ∥ φ ( s, a ) ∥ ≤ 1 .
Assumption 1 is standard and commonly adopted; see, e.g., [3, 35, 38, 24, 21].
This assumption canbe achieved via normalizing the feature vectors.
Assumption 2 (negative definiteness of A θ,ϕ and A θ,φ ) .
There exists two positive constants λ ϕ , λ φsuch that for all policy θ , the following two matrices are negative definite
[Equation]
with λ max ( A θ,ϕ ) ≤ λ ϕ , λ max ( A θ,φ ) ≤ λ φ , where λ max ( · ) represents the largest eigenvalue.
Assumption 2 can be achieved when the matrices Φ ϕ :=
[ ϕ ( s 1 ) , · · · , ϕ ( s |S| )] and Φ φ :=[ φ ( s 1 , a 1 ) , · · · , φ ( s |S| , a |A| )] have full row rank, which ensures that the optimal critic and reward 208 estimator are unique; see also [24, 34].
Together with Assumption 1, we can show that the norm ofω ∗ ( θ ) and λ ∗ ( θ ) are bounded by some positive constant, which justifies the projection steps.
Assumption 3 (Lipschitz properties of policy) .
There exists constants C ψ , L ψ , L π such that forall θ, θ ′ , s ∈ S and a ∈ A , we have (1) .
| π θ ( a | s ) − π θ
a | s ) | ≤ L π ∥ θ − θ ′ ∥ ; (2) .
∥ ψ θ ( s, a ) − 212 ψ θ ′
( s, a ) ∥
≤ L ψ
∥ θ − θ ′ ∥ ; (3) .
∥ ψ θ ( s, a ) ∥ ≤ C ψ .
Assumption 3 is common for analyzing policy-based algorithms; see, e.g., [33, 32, 11].
The assumption ensures the smoothness of objective function J ( θ ) .
It holds for a large range of policy classessuch as tabular softmax policy [1], Gaussian policy [7], and Boltzman policy [13].
Assumption 4 (irreducible and aperiodic Markov chain) .
The Markov chain under π θ and transitionkernel P ( ·| s, a ) is irreducible and aperiodic for any θ .
Assumption 4 is a standard assumption, which holds for any uniformly ergodic Markov chains andany time-homogeneous Markov chains with finite-state space.
It ensures that there exists constantsκ > 0 and ρ ∈ (0 , 1) such that
[Equation]
Assumption 5 (doubly stochastic weight matrix) .
The communication matrix W is doubly stochastic,i.e. each column/row sum up to 1.
Moreover, the second largest singular value ν is smaller than 1.
Assumption 5 is a common assumption in decentralized optimization and multi-agent reinforcementlearning; see, e.g., [27, 5, 6].
It ensures the convergence of consensus error for critic and rewardestimator variables.
4.2 Sample complexity under i.i.d. sampling
Theorem 1 (sample complexity under i.i.d.
sampling) .
Suppose Assumptions 1-5 hold.
Considerthe update of Algorithm 1 under i.i.d.
sampling.
Let α k
= ¯ α √ K for some positive constant ¯ α ,β k = C 9 2 λ ϕ α k , and η k = C 10 2 λ φ α k , K c ≤ O ( α − 12 k ) , where K denotes the total number of iterations.
Then, we have
[Equation]
where C 9 , C 10 are positive constants defined in the proof.
The proof of Theorem 1 can found in Appendix E.1.
It establishes the iteration complexity of
O (1 / √ K ) , or equivalently, sample complexity of O ( ε − 2 ) for Algorithm 1.
Note that actors, critics,and reward estimators use the step sizes of the same order.
The sample complexity matches theoptimal rate of SGD for general non-convex optimization problem.
To explain the errors in (9), let usdefine the approximation error as the following:
[Equation]
The error ε app captures the approximation power of critic and reward estimator.
Similar termsalso appear in the literature (see e.g., [35, 1, 21]).
Such an approximation error becomes zero intabular case.
The error ε sp is inevitably caused by the mismatch between discounted state visitationdistribution d π θ and stationary distribution µ π θ ; see, e.g., [38, 24].
It is defined as
[Equation]
When γ is close to 1, the error becomes small.
This is because d π θ approaches to µ π θ when γ goes to1.
In the literature, some works assume that sampling from d π θ is permitted, thus eliminate this error;see, e.g., [4].
4.3 Sample complexity under markovian sampling
Theorem 2 (sample complexity under Markovian sampling) .
Suppose Assumptions 1-5 hold.
Consider the update of Algorithm 1 under Markovian sampling.
Let α k
= ¯ α √ K for some positive constant¯
α , β k = C 9 2 λ ϕ α k , and η k = C 10 2 λ φ α k , K c ≤ O ( α − 12 k ) , where K is the total number of iterations.
Then,we have
[Equation]
where C 9 , C 10 are positive constants defined in proof.
We put the proof of Theorem 2 in Appendix E.2.
In Markovian sampling, the updates are biased forcritics, actors, and reward estimators.
The error will decrease as the Markov chain mixes, and thelogarithmic term is due to the cost for mixing.
Theorem 2 establishes the iteration complexity of O (log 2 K/ √ K ) , or equivalently, sample complexity of e O ( ε − 2 ) for Algorithm 1.
It matches the state-of-the-art sample complexity of decentralized ACalgorithms, which are implemented in double-loop fashion [11, 6].
4.4 Proof sketch
We present the main elements for the proof of Theorem 2, which helps in understanding the differencebetween classical two-timescale/double-loop analysis and our single-timescale analysis.
The proof of
Theorem 1 follows the same framework with simpler sampling scheme.
Under Markovian sampling, it is possible to show the following inequality, which characterizes theascent of the objective.
[Equation]
To analyze the errors of critic ∥ ω ∗ ( θ k ) − ω ik +1 ∥ 2 and reward estimator ∥ λ ∗ ( θ k ) − λ ik +1 ∥ 2 , the twotimescale analysis requires O ( α k ) <
min {O ( β k ) , O ( η k ) } in order for these two errors to converge.
The double-loop approach runs lower-level update for
O (log( ε − 1 )) times with batch size O ( ε − 1 )to drive these errors below ε and hence, they cannot allow inner loop size and bath size to be O (1)simultaneously.
To obtain the convergence result for single-timescale update, the idea is to furtherupper bound these two lower-level errors by the quantity
O ( α k E ∥
g ia ( ξ k , ω ik +1 , λ ik +1 ) ∥ 2 ) (through aseries of derivations), and then eliminate these errors by the ascent term α k 2 E ∥ g ia ( ξ k , ω ik +1 , λ ik +1 ) ∥ 2 .
We mainly focus on the analysis of critic’s error through the proof sketch.
The analysis for rewardestimator’s error follows similar procedure.
We start by decomposing the error of critic as
N
[Equation]
[Equation]
The first term represents the consensus error, which can be bounded by the next lemma.
Lemma 1.
Suppose Assumptions 1 and 5 hold.
Consider the sequence { ω ik } generated by Algorithm 1,then the following holds
[Equation]
where ω 0 :
=
[ ω 1 , · · · , ω N ] T , Q := I − 1 N 11 T , k ′ :=
⌊ kK c ⌋ ∗ K c .
The constant ν ∈ (0 , 1) is thesecond largest singular value of W .
Based on Lemma 1 and follow the step size rule of Theorem 2, it is possible to show ∥ Q ω k +1 ∥ 2 F =
P
Ni =1 ∥ ω ik +
[Equation]
¯ ω k + [Equation] ∥ 2 =
O ( K 2 c β 2 k ) .
Let K c
= O ( β − 12 k ) , we have ∥ Q ω k +1 ∥ 2 F
=
O
( β k ) , which 278 maintains the optimal rate.
To analyze the second term in (12), we first construct the following Lyapunov function
[Equation]
Then, it remains to derive an approximate descent property of the term ∥ ¯ ω k − ω ∗ ( θ k ) ∥ 2 in (13).
Towards that end, our key step lies in establishing the smoothness of the optimal critic variablesshown in the next lemma.
Lemma 2 (smoothness of optimal critic) .
Suppose Assumptions 1-3 hold, under the update of
Algorithm 1, there exists a positive constant L µ, 1 such that for all θ, θ ′ , it holds that
[Equation]
where ∇ ω ∗ ( θ ) denotes the Jacobian of ω ∗ ( θ ) with respect to θ .
This smoothness property is essential for achieving our ˜ O (1 / √ K ) convergence rate.
To the best of our knowledge, the smoothness of ω ∗ ( θ ) has not been justified in the literature.
Equipped with Lemma 2, we are able to establish the following lemma.
Lemma 3 (Error of critic) .
Under Assumptions 1-5, consider the update of Algorithm 1.
Then, itholds that
[Equation]
Here, Z K :=
min { z ∈ N + | κρ z − 1 ≤ min { α k , β k , η k }} , C 9 , λ ϕ are constants specified in appendix,and C K 1 and C K 2 are of order
O (log( K )) and O (log 2 ( K )) d respectively.
Plug (15) into (14), we can establish the approximate descent property of ∥ ¯ ω k − ω ∗ ( θ k ) ∥ 2 in (13):
[Equation]
Finally, plugging (11), (14), and (16) into (13) gives the ascent of the Lyapunov function, which leadsto our convergence result through steps of standard arguments.
Numerical results
In this section, our objective is to illustrate the empirical sample complexity and communicationcomplexity of the proposed algorithms.
We also implement the algorithm in [6] to serve as a baseline,which employs double-loop algorithmic framework.
Our simulation is based on the groundedcommunication environment proposed in [19]; see Appendix A for detailed set up.
Through thediscussion, we refer the algorithm in [6] as "DLDAC", the Algorithm 1 as "SDAC-re", the Algorithm 2as "SDAC-noisy" (see Appendix B).
We also provide the result which assumes full reward is availableto serve as baseline, which we refer as "SDAC-full".
We set K r = 5 for "SDAC-noisy"; K c = 1for "SDAC-re", "SDAC-noisy", and "SDAC-full".
We choose T c = 5 (loop size), T ′ c = 1 (criticconsensus number every iteration), T ′ =
5 (reward consensus number every iteration) for "DLDAC".
The sample complexity and communication complexity are shown in Figure 1.
The results areaveraged over 10 Monte Carlo runs.
As we can see, the proposed two algorithms achieve significantlyhigher reward than "DLDAC" in terms of both sample complexity and communication complexity.
Moreover, their performances approach the baseline “SDAC-full", where the global reward is assumedto be available, indicating that the reward approximation is nearly accurate.
Due to space limit, wewill put additional experiments on the comparison with existing decentralized AC algorithms and theablation study of hyper-parameters to Appendix A.
Conclusion and future direction
In this paper, we studied the convergence of fully decentralized AC algorithm under practical singletimescale update for the first time.
We designed such an algorithm which maintains the optimalsample complexity of e O ( ε − 2 ) under less communications.
We also proposed a variant to preserve theprivacy of local actions by communicating noisy rewards.
Extensive simulation results demonstratethe superiority of our algorithms’ empirical performance over existing decentralized AC algorithms.
One limitation of our work is that we only study the convergence to stationary point.
Thus, we leavethe research on the avoidance of saddle points and convergence to global optimum as promisingfuture directions.