F INITE -T IME E RROR B OUNDS FOR D ISTRIBUTED L INEAR S TOCHASTIC A PPROXIMATION
A BSTRACT
This paper considers a novel multi-agent linear stochastic approximation algorithm driven by Markovian noise and general consensus-type interaction, in which each agent evolves according to its local stochastic approximation process which depends on the information from its neighbors.
The interconnection structure among the agents is described by a time-varying directed graph.
While the convergence of consensus-based stochastic approximation algorithms when the interconnection among the agents is described by doubly stochastic matrices (at least in expectation) has been studied, less is known about the case when the interconnection matrix is simply stochastic.
For any uniformly strongly connected graph sequences whose associated interaction matrices are stochastic, the paper derives ﬁnite-time bounds on the mean-square error, deﬁned as the deviation of the output of the algorithm from the unique equilibrium point of the associated ordinary differential equation.
For the case of interconnection matrices being stochastic, the equilibrium point can be any unspeciﬁed convex combination of the local equilibria of all the agents in the absence of communication.
Both the cases with constant and time-varying step-sizes are considered.
In the case when the convex combination is required to be a straight average and interaction between any pair of neighboring agents may be uni-directional, so that doubly stochastic matrices cannot be implemented in a distributed manner, the paper proposes a push-type distributed stochastic approximation algorithm and provides its ﬁnite-time bounds for the performance by leveraging the analysis for the consensus-type algorithm with stochastic matrices.
I NTRODUCTION
The use of reinforcement learning (RL) to obtain policies that describe solutions to a Markov decision process (MDP) in which an autonomous agent interacting with an unknown environment aims to optimize its long term reward is now standard Sutton & Barto (1998).
Multi-agent or distributed reinforcement learning is useful when a team of agents interacts with an unknown environment or system and aims to collaboratively accomplish tasks involving distributed decision-making.
Distributed here implies that agents exchange information only with their neighbors according to a certain communication graph.
Recently, many distributed algorithms for multi-agent RL have been proposed and analyzed (Zhang et al., 2019).
The basic result in such works is of the type that if the graph describing the communication among the agents is bi-directional (and hence can be represented by a doubly stochastic matrix), then an algorithm that builds on traditional consensus algorithms converges to a solution in terms of policies to be followed by the agents that optimize the sum of the utility functions of all the agents; further, both ﬁnite and inﬁnite time performance of such algorithms can be characterized (Doan et al., 2019; Zhang et al., 2018b).
This paper aims to relax the assumption of requiring bi-directional communication among agents in a distributed RL algorithm.
This assumption is arguably restrictive and will be violated due to reasons such as packet drops or delays, differing privacy constraints among the agents, heterogeneous capabilities among the agents in which some agents may be able to communicate more often or with more power than others, adversarial attacks, or even sophisticated resilient consensus algorithms being used to construct the distributed RL algorithm.
A uni-directional communication graph can be represented through a (possibly time-varying) stochastic – which may not be doubly stochastic – matrix being used in the algorithm.
As we discuss in more detail below, relaxing the assumption of a doubly stochastic matrix to simply a stochastic matrix in the multi-agent and distributed RL algorithms that have been proposed in the literature, however, complicates the proofs of their convergence and ﬁnite time performance characterizations.
The main result in this paper is to provide a ﬁnite time bound on the mean-square error for a multi-agent linear stochastic approximation algorithm in which the agents interact over a time-varying directed graph characterized by a stochastic matrix.
This paper, thus, extends the applicability of distributed and multi-agent RL algorithms presented in the literature to situations such as those mentioned above where bidirectional communication at every time step cannot be guaranteed.
As we shall see, this extension is technically challenging and requires new proof techniques that may be of independent interest.
Related Work (See Appendix B for more related work)
Many distributed reinforcement learning algorithms have now been proposed in the literature.
In this setting, each agent can receive information only from its neighbors, and no single agent can solve the problem alone or by ‘taking the lead’.
A backbone of almost all distributed RL algorithms proposed in the literature is the consensus-type interaction among the agents, dating back at least to Tsitsiklis (1984).
Many works have analyzed asymptotic convergence of such RL algorithms using ODE methods (Zhang & Zavlanos, 2019;
Zhang et al., 2018b; Suttle et al., 2020; Zhang et al., 2018a).
This can be viewed as an application of ideas from distributed stochastic approximation (Kushner & Yin, 1987; Stankovi´c et al., 2010;
Huang, 2012; Stankovi´c & Stankovi´c, 2016; Bianchi et al., 2013; Stankovi´c et al., 2016).
Finite-time performance guarantees for distributed RL have also been provided in works, most notably in Doan et al.
; Wang et al.
; Zhang et al.
; Sun et al.
Zeng et al.
The assumption that is the central concern of this paper and is made in all the existing ﬁnite-time analyses for distributed RL algorithms is that the consensus interaction is characterized by doubly stochastic matrices (Doan et al., 2019; 2021;
Wang et al., 2020; Zhang et al., 2021; Sun et al., 2020;
Zeng et al., 2020)
at every time step, or at least in expectation, i.e., W 1 = 1 and 1 (cid:62) E ( W ) = 1 (cid:62) (Bianchi et al., 2013).
Intuitively, doubly stochastic matrices imply symmetry in the communication graph, which almost always requires bidirectional communication graphs.
More formally, the assumption of doubly stochastic matrices is restrictive since distributed construction of a doubly stochastic matrix needs to either invoke algorithms such as the Metropolis algorithm
(Xiao et al., 2005) which requires bi-directional communication of each agent’s degree information; or to utilize an additional distributed algorithm (Gharesifard & Cort´es, 2012) which signiﬁcantly increases the complexity of the whole algorithm design.
Doubly stochastic matrices in expectation can be guaranteed via so-called broadcast gossip algorithms which still requires bi-directional communication for convergence (Bianchi et al., 2013).
In a realistic network, especially with mobile agents such as autonomous vehicles, drones, or robots, uni-directional communication is inevitable due to various reasons such as asymmetric communication and privacy constraints, non-zero communication failure probability between any two agents at any given time, and application of resilient consensus in the presence of adversary attacks (Vaidya et al., 2012; LeBlanc et al., 2013), all leading to an interaction among the agents characterized by a stochastic matrix, which may further be timevarying.
The problem of design of distributed RL algorithms with time-varying stochastic matrices and characterizing either their asymptotic convergence or ﬁnite time analysis remains open.
As a step towards solving this problem, we propose a novel distributed stochastic approximation algorithm and provide its convergence analyses when a time-dependent stochastic matrix is being used due to uni-directional communication in a dynamic network.
One of the ﬁrst guarantees to be lost as the assumption of doubly stochastic matrices is removed is that the algorithm converges to a “policy” that maximizes the sum of reward functions of all the agents.
Instead, the convergence is to a set of policies that optimize a convex combination of the network-wise accumulative reward, with the exact combination depending on the limit product of the inﬁnite sequence of stochastic matrices.
Nonetheless, by deﬁning the error as the deviation of the output of the algorithm from the eventual equilibrium point, we derive ﬁnite-time bounds on the mean-square error.
We consider both the cases with constant and time-varying step sizes.
In the important special case where the goal is to optimize the average of the individual accumulative rewards of all the agents, we provide a distributed stochastic approximation algorithm, which builds on the push-sum idea (Kempe et al., 2003) that has been used to solve distributed averaging problem over strongly connected graphs, and characterize its ﬁnite-time performance.
Thus, this paper provides the ﬁrst distributed algorithm that can be applied (e.g., in Temporal difference (TD) learning, see Appendix D) to converge to the policy maximizing the team objective of the sum of the individual utility functions over time-varying, unidirectional, communication graphs, and characterizes the ﬁnite-time bounds on the mean-square error of the algorithm output from the equilibrium point under appropriate assumptions.
Technical Innovation and Contributions There are two main technical challenges in removing the assumption of doubly stochastic matrices being used in the analysis of distributed stochastic approximation algorithms.
The ﬁrst is in the direction of ﬁnite-time analysis.
For distributed RL algorithms, ﬁnite-time performance analysis essentially boils down to two parts, namely bounding the consensus error and bounding the “single-agent” mean-square error.
For the case when consensus interaction matrices are all doubly stochastic, the consensus error bound can be derived by analyzing the square of the 2-norm of the deviation of the current state of each agent from the average of the states of the agents.
With consensus in the presence of doubly stochastic matrices, the average of the states of the agents remains invariant.
Thus, it is possible to treat the average value as the state of a ﬁctitious agent to derive the mean-square consensus error bound with respect to the limiting point.
More formally, this process relies on two properties of a doubly stochastic matrix W , namely that (1) 1 (cid:62) W = 1 (cid:62) , and (2) if x t +1 =
Wx t , then (cid:107) x
t +1 − ( 1 (cid:62) x t +1 ) 1 (cid:107) 2 ≤ σ 2 ( W ) (cid:107) x t − ( 1 (cid:62) x t ) 1 (cid:107)where σ 2 ( W ) denotes the second largest singular value of W (which is strictly less than one if W is irreducible).
Even if the doubly stochastic matrix is time-varying (denoted by W t ), property (1) still holds and property (2) can be generalized as in Nedi´c et al.
Thus, the square of the 2-norm (cid:107) x t − ( 1 (cid:62) x t ) 1 (cid:107) 22 is a quadratic Lyapunov function for the average consensus processes.
Doubly stochastic matrices in expectation can be treated in the same way by looking at the expectation.
This is the core on which all the existing ﬁnite-time analyses of distributed RL algorithms are based.
However, if each consensus interaction matrix is stochastic, and not necessarily doubly stochastic, the above two properties may not hold.
In fact, it is well known that quadratic Lyapunov functions for general consensus processes x t +1 =
S t x t , with S t being stochastic, do not exist (Olshevsky & Tsitsiklis, 2008).
This breaks down all the existing analyses and provides the ﬁrst technical challenge that we tackle in this paper.
Speciﬁcally, we appeal to the idea of quadratic comparison functions for general consensus processes.
This was ﬁrst proposed in Touri (2012) and makes use of the concept of “absolute probability sequences”.
We provide a general analysis methodology and results that subsume the existing ﬁnite-time analyses for single-timescale distributed linear stochastic approximation and TD learning as special cases (see Appendix D).
The second technical challenge arises from the fact that with stochastic matrices, the distributed RL algorithms may not converge to the policies that maximize the average of the utility functions of the agents.
To regain this property, we propose a new algorithm that utilizes a push-sum protocol for consensus.
However, ﬁnite-time analysis for such a push-based distributed algorithm is challenging.
Almost all, if not all, the existing push-based distributed optimization works build on the analysis in Nedi ´ c & Olshevsky (2014); however, that analysis assumes that a convex combination of the entire history of the states of each agent (and not merely the current state of the agent) is being calculated.
This assumption no longer holds in our case.
To obtain a direct ﬁnite-time error bound without this assumption, we propose a new approach to analyze our push-based distributed algorithm by leveraging our consensus-based analyses to establish direct ﬁnite-time error bounds for stochastic approximation.
Speciﬁcally, we tailor an “absolute probability sequence” for the pushbased stochastic approximation algorithm and exploit its properties.
Such properties have never been found in the existing literature and may be of independent interest for analyzing any push-sum based distributed algorithm.
We now list the main contributions of our work.
We propose a novel consensus-based distributed linear stochastic approximation algorithm driven by Markovian noise in which each agent evolves according to its local stochastic approximation process and the information from its neighbors.
We assume only a (possibly time-varying) stochastic matrix being used during the consensus phase, which is a more practical assumption when only unidirectional communication is possible among agents.
We establish both convergence guarantees and ﬁnite-time bounds on the mean-square error, deﬁned as the deviation of the output of the algorithm from the unique equilibrium point of the associated ordinary differential equation.
The equilibrium point can be an “uncontrollable” convex combination of the local equilibria of all the agents in the absence of communication.
We consider both the cases of constant and time-varying step-sizes.
Our results subsume the existing results on convergence and ﬁnite-time analysis of distributed RL algorithms that assume doubly stochastic matrices and bi-directional communication as special cases.
In the case when the convex combination is required to be a straight average and interaction between any pair of neighboring agents may be uni-directional, we propose a push-type distributed stochastic approximation algorithm and establish its ﬁnite-time performance bound.
It is worth emphasizing that it is straightforward to extend our algorithm from the straight average point to any pre-speciﬁed convex combination.
Since it is well known that TD algorithms can be viewed as a special case of linear stochastic approximation (Tsitsiklis & Roy, 1997), our distributed linear stochastic approximation algorithms and their ﬁnitetime bounds can be applied to TD algorithms in a straight-forward manner; (see distributed TD( λ ) in Appendix D).
Notation We use X t to represent that a variable X is time-dependent and t ∈ { 0 , 1 , 2 , . . .
} is the discrete time index.
The i th entry of a vector x will be denoted by x i and, also, by ( x ) i when convenient.
The ij th entry of a matrix A will be denoted by a ij and, also, by ( A ) ij when convenient.
We use 1 n to denote the vectors in R n whose entries all equal to 1 ’s, and I to denote the identity matrix, whose dimension is to be understood from the context.
Given a set S with ﬁnitely many elements, we use |S| to denote the cardinality of S .
We use (cid:100)·(cid:101) to denote the ceiling function.
A vector is called a stochastic vector if its entries are nonnegative and sum to one.
A square nonnegative matrix is called a row stochastic matrix, or simply stochastic matrix, if its row sums all equal one.
Similarly, a square nonnegative matrix is called a column stochastic matrix if its column sums all equal one.
A square nonnegative matrix is called a doubly stochastic matrix if its row sums and column sums all equal one.
The graph of an n × n matrix is a direct graph with n vertices and a directed edge from vertex i to vertex j whenever the ji -th entry of the matrix is nonzero.
A directed graph is strongly connected if it has a directed path from any vertex to any other vertex.
For a strongly connected graph G , the distance from vertex i to another vertex j is the length of the shortest directed path from i to j ; the longest distance among all ordered pairs of distinct vertices i and j in G is called the diameter of G .
The union of two directed graphs, G p and G q , with the same vertex set, written G p ∪ G q , is meant the directed graph with the same vertex set and edge set being the union of the edge set of G p and G q .
Since this union is a commutative and associative binary operation, the deﬁnition extends unambiguously to any ﬁnite sequence of directed graphs.
D ISTRIBUTED L INEAR S TOCHASTIC A PPROXIMATION
Consider a network consisting of N agents.
For the purpose of presentation, we label the agents from 1 through N .
The agents are not aware of such a global labeling, but can differentiate between their neighbors.
The neighbor relations among the N agents are characterized by a time-dependent directed graph G t =
( V , E t ) whose vertices correspond to agents and whose directed edges (or arcs) depict neighbor relations, where
V = { 1 , . .
, N } is the vertex set and E t =
V × V is the edge set at time t .
Speciﬁcally, agent j is an in-neighbor of agent i at time t if ( j, i )
∈
E t , and similarly, agent k is an out-neighbor of agent
i at time t if ( i,
k ) ∈
E t .
Each agent can send information to its out-neighbors and receive information from its in-neighbors.
Thus, the directions of edges represent the directions of information ﬂow.
For convenience, we assume that each agent is always an in- and out-neighbor of itself, which implies that G t has self-arcs at all vertices for all time t .
We use N it and N i − t to denote the in- and out-neighbor set of agent i at time t , respectively, i.e.,
[Equation]
It is clear
that N it and N
i − t are nonempty as they both contain index i .
We propose the following distributed linear stochastic approximation over a time-varying neighbor graph sequence { G t } .
Each agent i has control over a random vector θ it which is updated by
[Equation]
where w ijt are consensus weights, α t is the step-size at time t , A ( X t ) is a random matrix and
b i ( X t ) is a random vector, both generated based on the Markov chain { X t } with state spaces X .
It is worth noting that the update of each agent only uses its in-neighbors’ information and thus is distributed.
Remark
The work of Kushner & Yin (1987) considers a different consensus-based networked linear stochastic approximation as follows:
[Equation]
whose state form is Θ t +1 = W t Θ [Equation] +
[Equation]
Θ
[Equation] A [Equation] X t ) (cid:62) + α t B ( X t ) , and mainly focuses on asymptotically weakly convergence for the ﬁxed step-size case (i.e., α t = α for all t ).
Under the similar set of conditions, with its condition (C3.4’) being a stochastic analogy for Assumption 6, Theorem 3.1 in
Kushner & Yin (1987) shows that equation 2 has a limit which can be veriﬁed to be the same as θ ∗ , the limit of equation 1.
How to apply the ﬁnite-time analysis tools in this paper to equation 2 has so far eluded us.
The two updates equation 1 and equation 2 are analogous to the “combine-thenadapt” and “adapt-then-combine” diffusion strategies in distributed optimization (Chen & Sayed, 2012).
[Equation]
We impose the following assumption on the weights w ijt which has been widely adopted in consensus literature (Jadbabaie et al., 2003; Olfati-Saber et al., 2007; Nedi´c & Liu, 2017).
Assumption 1 There exists a constant β > 0 such that for all i, j ∈ V and t , w ijt ≥ β whenever j ∈ N it .
For all i ∈ V and t , (cid:80) j ∈N it w ijt = 1 .
Let W t be the N × N matrix whose ij th entry equals w ijt if j ∈ N it and zero otherwise.
From
Assumption 1, each W t is a stochastic matrix that is compliant with the neighbor graph G t .
Since each agent i is always assumed to be an in-neighbor of itself, all diagonal entries of W t are positive.
Thus, if G t is strongly connected, W t is irreducible and aperiodic.
To proceed, deﬁne
[Equation]
Then, the N linear stochastic recursions in equation 1 can be combined and written as
[Equation]
The goal of this section is to characterize the ﬁnite-time performance of equation 1, or equivalently equation 3, with the following standard assumptions, which were adopted e.g. in Srikant &
Ying (2019); Doan et al.
Assumption 2 There exists a matrix A and vectors b i , i ∈ V , such that
[Equation]
i , i ∈ V .
Deﬁne b max = max i ∈V sup x ∈X (cid:107)
b i ( x ) (cid:107) 2 < ∞ and A max = sup x ∈X (cid:107)
A ( x ) (cid:107) 2 < ∞ .
Then, (cid:107) A (cid:107) 2 ≤ A max and (cid:107) b
i (cid:107) 2 ≤ b max , i ∈ V .
Assumption 3 Given a positive constant α , we use τ ( α ) to denote the mixing time of the Markov chain { X t } for which
[Equation]
The Markov chain { X t } mixes at a geometric rate, i.e., there exists a constant C such that τ ( α ) ≤ − C log α .
Assumption 4 All eigenvalues of A have strictly negative real parts, i.e., A is a Hurwitz matrix.
Then, there exists a symmetric positive deﬁnite matrix P , such that A (cid:62) P + PA = − I .
Let γ max and γ min be the maximum and minimum eigenvalues of P , respectively.
Assumption 5 The step-size sequence { α t } is positive, non-increasing, and satisﬁes (cid:80) ∞
t =
α t = ∞ and (cid:80) ∞ t =
α 2 t < ∞ .
To state our ﬁrst main result, we need the following concepts.
Deﬁnition 1 A graph sequence { G t } is uniformly strongly connected if there exists a positive integer
L such that for any t ≥ 0 , the union graph ∪ t + L − 1 k = t
G k is strongly connected.
If such an integer exists, we sometimes say that { G t } is uniformly strongly connected by sub-sequences of length L .
Remark 2 Two popular joint connectivity deﬁnitions in consensus literature are “ B -connected” (Nedi´c et al., 2009) and “repeatedly jointly strongly connected” (Cao et al., 2008).
A graph sequence { G t } is B -connected if there exists a positive integer B such that the union graph ∪ ( k +1)
B − 1 t = kB
G t is strongly connected for each integer k ≥ 0 .
Although the uniformly strongly connectedness looks more restrictive compared with B -connectedness at ﬁrst glance, they are in fact equivalent.
To see this, ﬁrst it is easy to see that if { G t } is uniformly strongly connected, { G t } must be B -connected; now supposing { G t } is B -connected, for any ﬁx t , the union graph ∪ t +2 B − 1 k = t
[Equation]
must be strongly connected, and thus { G t } is uniformly strongly connected by sub-sequences of length 2 B .
Thus, the two deﬁnitions are equivalent.
It is also not hard to show that the uniformly strongly connectedness is equivalent to “repeatedly jointly strongly connectedness” provided the graphs under consideration all have self-arcs at all vertices, as “repeatedly jointly strongly connectedness” is deﬁned upon “graph composition”.
[Equation]
Deﬁnition 2 Let { W t } be a sequence of stochastic matrices.
A sequence of stochastic vectors { π t } is an absolute probability sequence for { W t } if π (cid:62) t
= π (cid:62) t +1 W t for all t ≥ 0 .
This deﬁnition was ﬁrst introduced by Kolmogorov (Kolmogoroff, 1936).
It was shown by Blackwell (Blackwell, 1945) that every sequence of stochastic matrices has an absolute probability sequence.
In general, a sequence of stochastic matrices may have more than one absolute probability sequence; when the sequence of stochastic matrices is “ergodic”, it has a unique absolute probability sequence (Nedi´c & Liu, 2017).
It is easy to see that when W t is a ﬁxed irreducible stochastic matrix
W , π t is simply the normalized left eigenvector of W for eigenvalue one.
More can be said.
Lemma 1 Suppose that Assumption 1 holds.
If { G t } is uniformly strongly connected, then there exists a unique absolute probability sequence { π t } for the matrix sequence { W t } and a constant π min ∈ (0 , 1) such that π it ≥ π min for all i and t .
Let (cid:104) θ (cid:105) t
= (cid:80) Ni =
1 π it θ it , which is a column vector and convex combination of all θ it .
It is easy to see that (cid:104) θ (cid:105) t =
( π (cid:62) t Θ t ) (cid:62) =
Θ (cid:62) t π t .
From Deﬁnition 2 and equation 3, we have π (cid:62) t +1 Θ t +1 =
π (cid:62) t +1 W t Θ t + α t π (cid:62) t +1 W t Θ t A ( X t ) (cid:62) + α t π (cid:62) t +1 B ( X t ) =
π (cid:62) t Θ t + α t π (cid:62) t Θ t A ( X t ) (cid:62) + α t π (cid:62) t +1 B ( X t ) , which implies that
[Equation]
Asymptotic performance of equation 1 with any uniformly strongly connected neighbor graph sequence is characterized by the following two theorems.
Theorem 1 Suppose that Assumptions 1, 2 and 5 hold.
Let { θ it } , i ∈ V , be generated by equation 1.
If { G t } is uniformly strongly connected, then lim t →∞ (cid:107) θ it − (cid:104) θ (cid:105) t (cid:107) 2 = 0 for all i ∈ V .
Theorem 1 only shows that all the sequences { θ it } , i ∈ V , generated by equation 1 will ﬁnally reach a consensus, but not necessarily convergent or bounded.
To guarantee the convergence of the sequences, we further need the following assumption, whose validity is discussed in Remark 3.
Assumption 6 The absolute probability sequence { π t } for the stochastic matrix sequence { W t } has a limit, i.e., there exists a stochastic vector π ∞ such that lim t →∞ π t = π ∞ .
Theorem 2 Suppose that Assumptions 1–6 hold.
Let { θ it } , i ∈ V , be generated by equation 1 and θ ∗ be the unique equilibrium point of the ODE
[Equation]
where A and b i are deﬁned in Assumption 2 and π ∞ is deﬁned in Assumption 6.
If { G t } is uniformly strongly connected, then all θ it will converge to θ ∗ both with probability 1 and in mean square.
Remark
Though Assumption 6 may look restrictive at ﬁrst glance, simple simulations show that the sequences { θ it } , i ∈ V , do not converge if the assumption does not hold.
It is worth emphasizing that the existence of π ∞ does not imply the existence of lim t →∞ W t , though the converse is true.
Indeed, the assumption subsumes various cases including (a) all W t are doubly stochastic matrices, and (b) all W t share the same left eigenvector for eigenvalue 1, which may arise from the scenario when the number of in-neighbors of each agent does not change over time (Olshevsky & Tsitsiklis, 2013).
An important implication of Assumption 6 is when the consensus interaction among the agents, characterized by { W t } , is replaced by resilient consensus algorithms such as
Vaidya et al.
; LeBlanc et al.
(2013) in order to attenuate the effect of unknown malicious agents, the resulting dynamics of non-malicious agents, in general, will not converge, because the resulting interaction stochastic matrices among the non-malicious agents depend on the state values transmitted by the malicious agents, which can be arbitrary, and thus the resulting stochastic matrix sequence, in general, does not have a convergent absolute probability sequence; of course, in this case, the trajectories of all the non-malicious agents will still reach a consensus as long as the step-size is diminishing, as implied by Theorem 1.
Further discussion on Assumption 6 can be found in Appendix C.
[Equation]
We now study the ﬁnite-time performance of the proposed distributed linear stochastic approximation equation 1 for both ﬁxed and time-varying step-size cases.
Its ﬁnite-time performance is characterized by the following theorem.
Let η t = (cid:107) π t − π ∞ (cid:107) 2 for all t ≥ 0 .
From Assumption 6, η t converges to zero as t → ∞ .
Theorem 3 Let the sequences { θ it } , i ∈ V , be generated by equation 1.
Suppose that Assumptions 1–4, 6 hold and { G t } is uniformly strongly connected by sub-sequences of length L .
Let q t and m t be the unique integer quotient and remainder of t divided by L , respectively.
Let δ t be the diameter of ∪ t + L − 1 k = t
G k , δ max = max t ≥ 0 δ t , and
[Equation]
where 0 < α < min { K 1 ,
[Equation]
1) Fixed step-size: Let α t
= α for all t ≥ 0 .
For all t ≥ T 1 ,
N
[Equation]
i =
[Equation]
step
[Equation]
size
[Equation]
α t = α 0 t +1 with α 0 ≥ γ max 0 .
[Equation]
For all t ≥ LT
[Equation]
Here T 1 , T 2 , K 1 , K 2 , C 1 − C 6 are ﬁnite constants whose deﬁnitions are given in Appendix A.1.
Since π it is uniformly bounded below by π min ∈ (0 , 1) from Lemma 1, it is easy to see that the above bound holds for each individual E [ (cid:107) θ it − θ ∗ (cid:107) 22 ] .
To better understand the theorem, we provide the following remark.
Remark
In Appendix E.2.1, we show that both (cid:15) and (1 − 0 .
9 α γ max ) lie in the interval (0 , 1) .
It is easy to show that (cid:15) is monotonically increasing for δ max and L , monotonically decreasing for β and π min .
Also, lim t →∞ (cid:80)
[Equation]
Therefore, the summands in the ﬁnite-time bound equation 7 for the ﬁxed step-size case are exponentially decaying except for the constant C 2 , which implies that lim sup t →∞ (cid:80) Ni {+=
+}
π it E [ (cid:107) θ it − θ ∗ (cid:107) 2 2 ] ≤ C 2 , providing a constant limiting bound.
From Appendix A, C 2 is monotonically increasing for γ max , δ max , b max and L , and monotonically decreasing for γ min , π min and β .
In Appendix E.2.2, we show that lim t →∞ 1 t (cid:80) tk =1 η k = 0 , which implies that the ﬁnite-time bound equation 8 for the time-varying step-size case converges to zero as t → ∞ .
We next comment on 0 .
1 in the inequality deﬁning α .
Actually, we can replace 0 .
1 with any constant c ∈ (0 , 1) , which will affect the value of (cid:15) and the feasible set of α , with the latter becoming 0 < α < min { K 1 ,
[Equation]
Thus, the smaller the value of c is, the smaller is the feasible set of α , though the feasible set is always nonempty.
For convenience, we simply pick c = 0 .
1 in this paper; that is why we also have 0 .
9 in equation 7.
Lastly, we comment on α 0 in the time-varying step-size case.
We set α 0 ≥ γ max 0 .
9 for the purpose of getting a cleaner expression of the ﬁnite-time bound.
For α 0 < γ max 0 .
9 , our approach still works, but will yield a more complicated expression.
The same is true for Theorem 5.
(cid:3)
Technical Challenge and Proof Sketch As described in the introduction, the key challenge of analyzing the ﬁnite-time performance of the distributed stochastic approximation equation 1 lies in the condition that the consensus-based interaction matrix is time-varying and stochastic (not necessarily doubly stochastic).
To tackle this, we appeal to the absolute probability sequence π t of the time-varying interaction matrix sequence and introduce the quadratic Lyapunov comparison function (cid:80) Ni =
π it E [ (cid:107) θ it − θ ∗ (cid:107) 22 ] .
Then, using the inequality (cid:80) Ni =1 π it E [ (cid:107) θ it − θ ∗ (cid:107) 22 ] ≤ 2 (cid:80) Ni
=1 π it E
[ (cid:107) θ it − (cid:104) θ (cid:105) t (cid:107) 22 ] + 2 E
[ (cid:107)(cid:104) θ (cid:105) t − θ ∗ (cid:107) 22 ] , the next step is to ﬁnd the ﬁnite-time bounds of (cid:80) Ni =
π it E [ (cid:107) θ it −(cid:104)
θ (cid:105) t (cid:107) 22 ] and E [ (cid:107)(cid:104)
θ (cid:105) t − θ ∗ (cid:107) 22 ] , respectively.
The latter term is essentially the “singleagent” mean-square error.
Our main analysis contribution here is to bound the former term for both ﬁxed and time-varying step-size cases.
P USH -SA
The preceding section shows that the limiting state of consensus-based distributed stochastic approximation depends on π ∞ , which leads to a convex combination of the local equilibria of all the agents in the absence of communication, but the convex combination is in general “uncontrollable”.
Note that this convex combination will correspond to a convex combination of the network-wise accumulative rewards in applications such as distributed TD learning.
In an important case when the convex combination is desired to be the straight average, the existing literature e.g. Doan et al.
(2019; 2021) relies on doubly stochastic matrices whose corresponding π ∞ =
(1 /N ) 1 N .
As mentioned in the introduction, doubly stochastic matrices implicitly require bi-directional communication between any pair of neighboring agents; see e.g. gossiping (Boyd et al., 2006) and the Metropolis algorithm (Xiao et al., 2005).
A popular method to achieve the straight average target while allowing uni-directional communication between neighboring agents is to appeal to the idea so-called “pushsum” (Kempe et al., 2003), which was tailored for solving the distributed averaging problem over directed graphs and has been applied to distributed optimization (Nedi´c & Olshevsky, 2014).
In this section, we will propose a push-based distributed stochastic approximation algorithm tailored for uni-directional communication and establish its ﬁnite-time error bound.
Each agent i has control over three variables, namely y it , ˜ θ it and θ it , in which y it is scalar-valued with initial value 1, ˜ θ it can be arbitrarily initialized, and θ i
= ˜
θ i 0 .
At each time t ≥ 0 , each agent i sends its weighted current values ˆ w jit y it and ˆ w jit (˜ θ it + α t A ( X t )
θ t + α t b i
( X t )) to each of its current out-neighbors j ∈ N i − t , and updates its variables as follows:
[Equation]
where ˆ w ijt = 1 / |N
j − t | .
It is worth noting that the algorithm is distributed yet requires that each agent be aware of the number of its out-neighbors.
Asymptotic performance of equation 9 with any uniformly strongly connected neighbor graph sequence is characterized by the following theorem.
Theorem 4 Suppose that Assumptions 2–5 hold.
Let { θ it } , i ∈ V , be generated by equation 9 and θ ∗ be the unique equilibrium point of the ODE
[Equation]
where A and b i are deﬁned in Assumption 2.
If { G t } is uniformly strongly connected, then θ it will converge to θ ∗ in mean square for all i ∈ V .
In this section, we deﬁne (cid:104) ˜
θ (cid:105) t = 1 N (cid:80) Ni =
˜
θ
it
and (cid:104)
θ (cid:105) t = 1 N (cid:80) Ni
=
θ it .
To help understand these deﬁnitions, let ˆ W t be the N × N matrix whose ij -th entry equals ˆ w ijt if j ∈ N it , otherwise equals zero.
It is easy to see that each ˆ W t is a column stochastic matrix whose diagonal entries are all positive.
Then, π t = 1 N 1 N for all t ≥ 0 can be regarded as an absolute probability sequence of { ˆ W t } .
Thus, the above two deﬁnitions are intuitively consistent with (cid:104)
θ (cid:105) t in the previous section.
Finite-time performance of equation 9 with any uniformly strongly connected neighbor graph sequence is characterized by the following theorem.
Let µ t
=
(cid:107) A ( X t )( (cid:104) θ (cid:105) t − (cid:104)
˜
θ (cid:105) t ) (cid:107) 2 .
In Appendix E.3, we show that (cid:107)(cid:104) θ (cid:105) t − (cid:104)
˜
θ (cid:105) t (cid:107) 2 converges to zero as t → ∞ , so does µ t .
Theorem 5 Suppose that Assumptions 2–4 hold and { G t } is uniformly strongly connected by subsequences of length L .
Let { θ it } , i ∈ V , be generated by equation 9 with α t
= α 0 t +1 and α 0 ≥ γ max 0 .
Then, there exists a nonnegative ¯ (cid:15) ≤ (1 − 1 N NL ) 1 L such that for all t ≥ ¯ T ,
N (cid:88) i =
[Equation]
where ¯ T and C 7 − C 12 are ﬁnite constants whose deﬁnitions are given in Appendix A.2.
In Appendix E.3, we show that lim t →∞ 1 t (cid:80) tk =1 µ k = 0 , which implies that the ﬁnite-time bound equation 11 converges to zero as t → ∞ .
It is worth mentioning that the theorem does not consider the ﬁxed step-size case, as our current analysis approach cannot be directly applied for this case.
Proof Sketch and Technical Challenge
Using the inequality (cid:80) Ni =1 E [ (cid:107) θ it +1 − θ ∗ (cid:107) 22 ] ≤ 2 (cid:80) Ni =1 E [ (cid:107) θ it +1 − (cid:104)
˜
θ (cid:105) t (cid:107) 22 ] + 2 N E
[ (cid:107)(cid:104) ˜ θ (cid:105) t − θ ∗ (cid:107) 22 ] , our goal is to derive the ﬁnite-time bounds of (cid:80) Ni =1 E [ (cid:107) θ it +1 − (cid:104)
˜
θ (cid:105) t (cid:107) 22 ] and E [ (cid:107)(cid:104)
˜ θ (cid:105) t − θ ∗ (cid:107) 22 ] , respectively.
Although this looks similar to the proof of Theorem 3, the derivation is quite different.
First, the iteration of (cid:104) ˜
θ (cid:105) t is a single-agent SA plus a disturbance term (cid:104) θ (cid:105) t − (cid:104)
˜
θ (cid:105) t , so we cannot directly apply the existing single-agent SA ﬁnitetime analyses to bound E
[ (cid:107)(cid:104) ˜ θ (cid:105) t − θ ∗ (cid:107) 22 ] ; instead, we have to show that (cid:104) θ (cid:105) t − (cid:104)
˜
θ (cid:105) t will diminish and quantify the diminishing “speed”.
Second, both the proof of showing diminishing (cid:104) θ (cid:105) t − (cid:104)
˜
θ (cid:105) t and derivation of bounding (cid:80)
Ni =1 E [ (cid:107) θ
it +1 − (cid:104)
˜
θ (cid:105) t (cid:107) 22 ] involve a key challenge: to prove the sequence { θ it } generated from the Push-SA equation 9 is bounded almost surely.
To tackle this, we introduce a novel way to constructing an absolute probability sequence for the Push-SA as follows.
From equation 9, θ it +
[Equation]
=
[Equation]
) θ
[Equation]
y jt + α t
b
j
( X t )
y jt ] , where ˜ w ijt =
( ˆ w ijt y jt ) /
( (cid:80) Nk
=1 ˆ w ikt y kt ) .
We show that each matrix ˜ W t =
[ ˜ w ijt ] is stochastic, and there exists a unique absolute probability sequence { ˜ π t } for the matrix sequence { ˜ W t }
such that ˜ π it ≥ ˜ π min for all i ∈ V and t ≥ 0 , with the constant ˜ π min ∈ (0 , 1) .
Most importantly, we show two critical properties of { ˜ W t } and { ˜ π t } , namely lim t →∞ (Π ts =
˜ W s )
= 1 N 1 N 1 (cid:62) N and ˜
π
it y it
= 1 N for all i, j ∈ V and t ≥ 0 , which have never been reported in the literature though push-sum based algorithms have been extensively studied.
C ONCLUDING R EMARKS
In this paper, we have established both asymptotic and non-asymptotic analyses for a consensusbased distributed linear stochastic approximation algorithm over uniformly strongly connected graphs, and proposed a push-based variant for coping with uni-directional communication.
Both algorithms and their analyses can be directly applied to TD learning.
One limitation of our ﬁnitetime bounds is that they involve quite a few constants which are well deﬁned and characterized but whose values are not easy to compute.
Future directions include leveraging the analyses for resilience in the presence of malicious agents and extending the tools to more complicated RL.