R EINFORCEMENT L EARNING UNDER A M ULTIAGENT P REDICTIVE S TATE R EPRESENTATION M ODEL : M ETHOD AND T HEORY
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper proposes an optimal policy learning algorithm under a novel multi-agent predictive state representation reinforcement learning model.
Compared to the state-of-the-art methods, the most striking feature of our approach is to introduce a dynamic interaction graph to the model, which allows us to represent each agent’s predictive state based on its “neighborhood” agents.
Methodologically, we develop an online algorithm that simultaneously learns the predictive state representation and agent policies.
Theoretically, we propose an upper bound of the L 2 -norm of the learnt predictive state representation.
Empirically, we provide positive numerical results on both a MAMuJoCo robotic learning experiment and a multi-agent particle learning environment to demonstrate the efﬁcacy of the method.
I NTRODUCTION
Real-world multi-agent systems are considered partially observable since agents do not have a complete perception of the environment states.
The predictive state representation (PSR) (Littman et al., 2001) is a representation of the dynamic system state by a vector of predictions of tests conditioned on a history.
The tests are sequences of actions and observations, which are true if and only if all the observations occur, given that all the actions taken and the histories are sequences of actions and observations recorded before.
It records a sequence of actions and observations and focuses on observable quantities rather than the system’s dynamics.
Therefore, compared with learning the hidden-state-based partially observable Markov decision process (POMDP) Kaelbling et al.
(1998) models from observation data, learning PSR should be easier and less prone to local minimal problems (Singh et al., 2012). In addition, in the cases where the moment-method is used to learn a model for controlled systems that incorporates actions, PSR is considered as a more general setting to reconstruct the structure of the dynamic system than POMDP (Azizzadenesheli et al., 2016; Hamilton, 2014). Furthermore, (Littman et al., 2001; James & Singh, 2004; Singh et al., 2012) showed that PSR can offer an alternative more compact state representation for POMDP models. Overall, PSR is considered a generalization of POMDP in terms of representation power and the development of learning algorithms.
Due to the above beneﬁts of PSR, it has been used as the basis of some model-based single-agent reinforcement learning (James & Singh, 2004; Boots et al., 2011; Hamilton et al., 2014; Hefny et al., 2018a).
Moreover, the predictive states that act as sufﬁcient statistics for the states of multiagent dynamic systems could potentially help multi-agent decision-making in the complicated, partially observable domains.
However, there is no method for developing multi-agent predictive state representation (MAPSR) for multi-agent reinforcement learning (MARL).
Due to this reason, we focused on building an algorithm for learning the optimal policies by representing the multi-agent dynamic systems with the MAPSR.
An existing algorithm by (Chen et al., 2020) provides a spectral framework for learning MAPSR using a tensor, with its dimensions representing the agents and joint history.
A tensor decomposition is implemented to learn the model parameters.
This type of algorithm avoids explicit estimation of the latent state and can get an unbiased estimation if an inﬁnite amount of data is available.
However, it is not easy to incorporate prior information such as sparsity or structure because each new source of information leads to new moments and a different and more complex set of equations to solve.
(Hefny et al., 2015) represent the distributions in PSR via observable statistics and learn PSR by supervised learning, alleviating the above problem.
Later, (Hefny et al., 2018a) extend this method to develop a new one called predictive state controlled model that can represent the system model where the agents can affect the environment.
Due to the trend of deep RL, (Hefny et al., 2018b) developed an end-to-end policy learning method by using the states represented by a predictive state controlled model as the belief state to ﬁt the policy function to train RL.
Both the PSR and RL parameters are updated by using a combined loss objective.
The PSR part maintains a sufﬁcient representation of distributions under the partially observable environment, the deep RL algorithm brings efﬁciency for learning.
The environment of MARL can be partially observable and always non-stationary (one agent learning while the other agents’ actions can inﬂuence the environment)
(Bowling & Veloso, 2002).
Predictive state controlled model can model the prediction of systems states affected by actions; if we design a state representation model for MARL environment that can predict the future observations of agents upon exerting future actions of other agents and conditioned on past histories, then the non-stationary environment caused by other agents learning can be modeled.
Due to the above reason, we seek to develop a MARL-based PSR model and learning algorithm, which we call MAPSRL.
It could potentially ameliorate both the problem of partially observable observations and non-stationarity by other agents’ interaction for multi-agent decision making.
The paper’s ﬁrst goal is to develop a PSR model for MARL environment.
In MARL settings, the distribution of observed observations is dependent on its actions and relevant to other agents and histories.
So far, many works (Ryu et al., 2020;
Liu et al., 2020b; Niu et al., 2021) that take other agents’ actions into consideration have been shown to achieve competitive performance.
In order to consider the interactive relationship under the MAPSR, we introduce a dynamic interaction graph to characterize the “neighborhoods” of agents.
Such a graph topology, which is often easy to obtain nowadays, enables us to encode the adjacency relationship of agents into our MAPSR model.
We build our model under three common graphs, namely static complete graphs, static non-complete graphs, and dynamic graphs, covering as many real-world scenarios as possible.
Under the existence of such a dynamic interaction graph, we build predictions of its observations by considering its neighbor’s actions for every agent.
We call this primitive interactive predictions.
Then an agent’s
PSR will be formulated as a linear combination of those primitive interactive predictions.
Due to the decentralized formulation, our model allows implementing the learning of the MAPSR on a single agent level while maintaining other agents’ interactions through the encoding of the interactive graph topology.
As set as our second goal, we analytically upper bound the L 2 -norm difference between the underlying true and learned predictive state representations, providing theoretical guarantees for the performance of our learning algorithm under the MAPSR model in all three graph settings.
Our paper’s third goal is to develop an algorithm that integrates the MAPSR model to the MARL.
We develop an online algorithm that simultaneously learns the MAPSR, and the agents’ policies end-toend with a fully differentiable neural network structure.
Our algorithm provides a new model-based
MARL, and it has the ﬂexibility to be extended to most of the ofﬂine MARL algorithms.
We focus on the continuous action space, which is more practical and common to real-world applications such as robotic control.
Finally, we test our algorithm through systematic numerical experiments on MAMujoCo robotic learning experiments (de Witt et al., 2020) and multi-agent particle learning environments (Ryu et al., 2020), and compare our proposed method against two baselines as detailed in Section 6.
The results demonstrate the efﬁciency of our method.
R ELATED W ORKS
Partially Observable Environment.
Real-world agents often experience situations that the observed signals are aliased and do not fully determine their state in the system.
This is particularly true for multiple agents environments where agents have partial observability due to limited communication (Oliehoek & Amato, 2016).
In accommodation to the partially observable environment, POMDP has been adopted by (Kaelbling et al., 1998), and the algorithms (Kaelbling et al., 1998; Cassandra, 1998;
Thrun, 1999; Pineau et al., 2003; Poupart & Vlassis, 2008; Platt Jr et al., 2010) for determining an optimal policy have shifted to using the probability distributions (belief state) over the state space instead of exact state space.
In general, they have high complexity or suffer from local optima.
Moreover, the most common POMDP policy learning assumes the agent has access to a priori knowledge of the system.
The access to such prior knowledge has a premise that the agent has considerable domain knowledge (Kaelbling et al., 1998).
However, it is expected that the real-world agents learn the system model and thus a planning policy further without knowledge of the domain.
Overview of PSR.
(Littman et al., 2001; James & Singh, 2004) introduced the PSR over an expressive and robust framework for modeling dynamical systems and deﬁned PSR as a representation of state by using a vector of predictions of fully observable quantities (tests) conditioned on past events (histories).
A predictive model is constructed directly from execution traces in the PSR framework, utilizing minimal prior information about the domain.
The PSR paradigm subsumes POMDP as a special case (Littman et al., 2001).
PSR is considered much more compact than POMDP (Aberdeen et al., 2007).
The spectral learning method has been proved to show success for learning the PSR (Boots et al., 2011; Jiang et al., 2018).
There are other classes of dynamical system learning algorithms that are based on likelihood-based optimization or sampling approaches (Frigola et al., 2013), but they are prone to poor local optima.
The spectral learning represents the estimated state by sufﬁcient statistics of future observations and estimates the model parameters by method of moments.
However, this line of algorithms is hard to incorporate prior information (Hefny et al., 2015).
Thus, (Hefny et al., 2015; 2018a) introduce the supervised learning method to learn PSR and proves its convergence.
Although many works study PSR in discrete action space (Hsu et al., 2012; Siddiqi et al., 2010; Boots et al., 2011), (Boots et al., 2013) proposes Hilbert space embedding (HSE)-PSR to deal with continuous actions.
(Hefny et al., 2018a) uses an approximation of HSE-PSR by Random Fourier transform (RFF) and built a more principled generalization of PSR to deal with high dimensions.
However, all of these studies aim for the single agent scenario.
PSR and RL.
The predictive states estimated by the PSR are considered as states in a fully observable
Markov Decision Process so that the value function is learned on these states.
This line of work has been done in the single-agent environment (Boots & Gordon, 2010; Boots et al., 2011; Hamilton et al., 2014; Venkatraman et al., 2017; Hefny et al., 2018b).
Especially, (Hefny et al., 2018b) proposes the recurrent predictive state in the RNN network.
Moreover, the learning PSR and policy functions are connected with the end-to-end training.
MAPSR and MARL.
MAPSR model is formulated by (Chen et al., 2020) and the model parameters are learnt by tensor decomposition.
This formulation is aligned with the spectral algorithm, so it also has the same issue as the single-agent case.
MARL algorithms have been developed to learn the Markov Game environment (Lyu & Amato, 2020; Son et al., 2019;
Zhang et al., 2019; Rashid et al., 2018;
Foerster et al., 2018; Lowe et al., 2017).
The partially observable states are pervasive in this environment, bringing non-stationarity to the learning algorithm (Bowling & Veloso, 2002) so that the learning needs to consider the non-stationarity.
Recent works propose more sophisticated deep MARL algorithms for multi-agent problems under the paradigm of centralized training with decentralized execution
(Zhou et al., 2020; Sunehag et al., 2017; Lowe et al., 2017; Foerster et al., 2018; Rashid et al., 2018).
Other than their methods, our approach also accounts for the importance of macroscopic measures of underline systems.
Also, providing a predictive state as a belief state rather than the partially observable observations can further help the agent learn.
Like the single-agent case, we hope our method can further improve these algorithms.
M ODEL D ESCRIPTION OF MAPSR 3. R EVIEW OF S INGLE A GENT PSR
Predictive State Representation (PSR).
A prediction of a state is deﬁned as the conditional probability of seeing a test’s observations in sequence given that actions of the test are taken in sequence from a history (Littman et al., 2001).
Givn a ﬁnite observation space O ( o ∈ O ) and action space A ( a ∈ A ).
A test of length k at time t , is deﬁned as a sequence of action-observation pairs that starts at time t and ends at time t + k − 1 , { ( o l , a l ) } t + k − 1 l = t = { o t , a t , o t +1 , a t +1 , · · · , o l , a l , · · · , o t + k − 1 , a t + k − 1 } .
A history , at time t is a sequence of action-observation pairs that start from the beginning of time and ends at time t − 1 , { ( o l , a l ) } t − 1 l =1 = { o 1 , a 1 , · · · , o l , a l , · · · , o t − 1 , a t − 1 } .
Hilbert Space Embedding Predictive State Controlled Model.
In this work, we are interested in extending PSR to decision makings of controlled systems with continuous actions.
So we use the model introduced by (Hefny et al., 2018a).
In this model, predictive state Q t satisﬁes
Q t ψ at = E
[ ψ ot | ψ at ; ψ ht ] and extended predictive state P t satisﬁes P t ξ at = E
[ ξ ot | ξ at ; ψ ht ] (i.e., Q t and P t are conditional linear expectation operators which maps to the conditional expectation of future observations), where ψ ot := φ O ( { o l } t + k − 1 l = t ) and ψ at :=
φ A ( { a l } t + k − 1 l = t ) are feature maps by kernels k O ,
k A over future observation and action features.
The extended predictive state compared to predictive state adds one more pair of { ( a t + k , o t + k ) } to the prediction.
The ξ at and ξ ot are the corresponding extended feature maps, which satisfy ξ ot = ψ
ot ⊗ φ ot and ξ at = ψ at ⊗ φ at .
Here φ ot :=
φ o ( o t ) and φ at :=
φ a ( a t ) are the shorthands for one time feature map by k o and k a .
We use ⊗ to denote the transposed Khatri–Rao product for two matrices with the same number of rows, and each row of the resultant matrix is the vectorzied outer product of the corresponding row vectors in the two matrices.
Also, we use the ψ ht :=
ψ h ( { ( o l , a l ) } t − 1 l =1 ) to deﬁne a set of features extracted from previous observations and actions (typically from a ﬁxed length window ending at t − 1 ).
3. MAPSR F ORMULATION
For n -agent systems, we denote joint observations as { o t,i } ni =1 and joint actions as { a t,i } ni =1 at any time t .
The joint test is then a collection of tests for all agents { ( o l, 1 , a l, 1 ) , · · · , ( o l,n , a l,n ) } t + k − 1 l = t and the joint history is a collection of histories for all agents { ( o l, 1 , a l, 1 ) , · · · , ( o l,n , a l,n ) } t − 1 l =1 .
A prediction of multi-agent state is thus the conditional probability of a joint test given a joint history.
Our Formulation of MAPSR.
We bring the HSE-PSR model to the multi-agent settings.
For any agent i in an n -agent system, given kernels { k O i } ni =1 , { k A i } ni =1 , { k o i } ni =1 , { k a i } ni =1 , we use ψ ot,i , ψ at,i , ξ ot,i , ξ at,i , φ ot,i , φ at,i and ψ ht,i to denote its corresponding feature maps, same as the previous deﬁnitions for single agent.
The superscripts o , a , h are local observations, actions, and histories for agent i .
Moreover, we use the Q t,i and P t,i to denote its predictive state and extended predictive state.
3. T HE E STIMATION FOR MAPSR M ODEL C OMPONENTS
As stated by (Littman et al., 2001), a complete PSR model can build a recursive rule to update itself.
In other words, if we are given Q t , we can get Q t +1 via the Bayes’ rules.
We use a supervised learning method proposed by (Hefny et al., 2018a) with three types of models to update the predictive states: For any agent i , P t,i = W i
( Q t,i ) , Q t +1 ,i =
F i ( P t,i , o t,i , a t,i ) , and o t,i =
Z i ( Q t,i , a t,i ) .
Typically, W i , Z i are learnable linear maps, and F i is non-linear but known in advance.
Since the model components, Q t,i and P t,i are operators, so they have to be learned as well.
The W i , Z i can be learned from regression if we know the estimation of Q t,i and P t,i , see Appendix A.2.2 for details of estimation of W i , Z i .
We focus on the details of estimation of Q t,i , P t,i to section 4.1.1.
D YNAMIC I NTERACTION G RAPH F OR MAPSR M ODEL
Many works have considered graph representation of the multi-agent network (Liu et al., 2020b; Ryu et al., 2020).
In general, the relationship between agents is characterized as an undirected graph.
We introduce a dynamic interaction graph to represent the MAPSR by considering the interaction between agents.
Deﬁnition 1.
Let G =
( V, E ) , including the set V of nodes and set E of the edges.
Each node represents the agent entry, and the edge represents the relationship between the two adjacent agents.
Here we suppose the graph structure is given, that means the number of nodes ( n ), the number of edges ( m ), the edge weights, and the maximum number of degrees ( k ) are available to us.
We think that this kind of presupposition is very reasonable because in the real world, for example, there are multiple robots; we can quickly get the geographic position of the robots through sensors, then calculate the structure of the graph formed by them.
4. S TATIC C OMPLETE G RAPH
We starts with a static complete graph G c , where the relationship between nodes are invariant to time change.
A complete graph has m = n ( n − 1) / 2 edges, where m = | E | .
For each agent, we then represent its PSR by considering other agents’ interactions by
[Equation]
same for P t,i . To consider the interactive behavior between agents, we introduce two additional notations Q t,i,j and P t,i,j .
Let { ( i, j ) } ni,j =1 represents a pair of agents on a n multi-agent system.
The Q t,i,j is a primitive predictive state of i ’s observation ( ψ ot,i ) by intervening agent j ’s action ( ψ at,j ) and observing agent i ’s observation history and agent j ’s action history ( ψ ht,i,j ).
And P t,i,j is the extended counterpart.
If i = =
j , then it becomes an exact single agent scenario.
Similarly, we use the same approach to represent P t,i .
Each agent’s PSR Q t,i and extended PSR P t,i are modeled by fully considering all other available agents.
Based on equation (1), in practice, estimating Q t,i and P t,i denoted as (cid:98) Q t,i and (cid:98) P t,i requires us to get (cid:98) Q t,i,j and (cid:98) P t,i,j at ﬁrst.
4.1. E STIMATION OF Q t,i,j , P t,i,j
To estimate the P t and Q t for single agent, (Hefny et al., 2018a) use the supervised learning method.
They show that (cid:98) Q t =
M ψ ot | ψ at ; ψ ht = C
ψ ot ψ at | ψ ht ( C ψ at ψ at | φ ht + λI ) − 1 , where M A | B ; c is a linear operator that satisﬁes E
[ A | B = b ; C = c ] =
M A | B ; c b , and C XY :=
E [ φ ( X ) ⊗ φ ( Y )] is the uncentered covariance operator, and C XY | z is covariance of X and Y given Z
= z .
They estimate
C ψ ot ψ at | ψ ht , C ψ at ψ at | φ ht by sampling data, then get the estimation (cid:98) Q t .
The same procedure is used to estimate (cid:98) P t by replacing the features ψ ot and ψ at with their extend counterparts ξ ot and ξ at .
Similar to the single-agent case above, the representation of Q t,i,j can be achieved as
[Equation]
To estimate C ψ ot,i ψ at,j | ψ ht,i,j and C ψ at,j ψ at,j | ψ ht,i,j , we learn two linear maps T i,j and U i,j such that
T i,j (cid:0) ψ ht,i,j (cid:1)
≈ C ψ ot,i ψ at,j | ψ ht,i,j and U i,j (cid:0) ψ ht,i,j (cid:1)
≈ C ψ at,j ψ at,j | ψ ht,i,j .
The training examples for T i,j and
U i,j consist of pairs ( ψ ht,i,j , ψ ot,i ⊗ ψ at,j ) and ( ψ ht,i,j , ψ at,j ⊗ ψ at,j ).
The learning of (cid:98) P t,i,j can be done in a similar way.
More details can be found in Appendix A.2.
After calculation of (cid:98) Q t,i,j and (cid:98) P t,i,j , under the static complete graph setting where the interaction is considered, deﬁned in equation (1), we can get the estimate of (cid:98) Q t,i and (cid:98) P t,i .
4.1. T HEORETICAL G UARANTEE OF E STIMATION OF Q i , P i U NDER T HE S TATIC C OMPLETE G RAPH
Theoretically, we show that the difference between Q i and its estimator (cid:98) Q i is bounded with high probability.
Theorem 1.
Let π Θ be a data collection policy and H is the range of π Θ on joint histories.
If
Equation 1 and 2 used, then for all h ∈ H and any (cid:15) ∈ (0 , 1) , such that N > t 2 Aj log(2 d Aj /(cid:15) ) v (cid:16) C ψaj (cid:17) where N is the number of time points we collect sample, then (cid:13)(cid:13)(cid:13) (cid:98) Q i − Q i (cid:13)(cid:13)(cid:13) is bounded as below with probability at least 1 − 3 (cid:15) , (cid:13)(cid:13)(cid:13) (cid:98) Q
[Equation]
− Q
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
where
[Equation]
Here ∆ 1 , ∆ 2 are two other relevant bounds, we provide them in Appendix E.1.
u ( · ) , v ( · ) denote the largest, smallest eigenvalue of a matrix.
And γ = t 2 Aj log(2 d Aj /(cid:15) )
[Equation]
C ψaj
[Equation]
N < 1 is a constant that depends on the magnitude of the norm of ψ aj (we assume (cid:13)(cid:13) ψ aj (cid:13)(cid:13) ≤ t A j ), the dimension of ψ aj ( d A j ), the (uncentered) covariances ( C ψ aj := E (cid:2)
ψ aj ψ aTj (cid:3) ) and the sample size
( N )
Theorem 1 says we need at least N samples for the bound in equation (3) to be valid.
We give the proofs in Appendix E.
It is not hard to obtain a bound for P i using the same approach.
We omit that.
4. E XTENSION TO S TATIC N ON -C OMPLETE G RAPH
In large-scale multi-agent systems, the number of agents is large, and not all agents need to interact with each other.
A static non-completed graph can perfectly represent such a situation.
For example, in a given static non-complete graph G s , we know its maximum number of degree k , and we use the binary n × n matrix with each entry as I i,j to indicate the interaction between two agents.
Then the MAPSR for each agent will be
[Equation]
Lemma 1.
Under the same environment depicted in Theorem 1 and given a G s with k maximum number of the degree to represent agents, then the bound in Theorem 1 can be rewritten as
[Equation]
The conclusion of Lemma 1 is evident since we replace the n with the k neighbors, the total error bound is also decreased approximately as kn .
4. D YNAMIC G RAPH
Real-world multi-agents can also formulate a time-dependent dynamic graph G d , rather than a static graph.
A dynamic graph has its structure dynamically changing with time.
In other words, the edges can be inserted or deleted across time.
The dynamic graph brings more challenges to the representation as the interaction relationship among agents changes constantly.
Braha & Bar-Yam (2009), Ma et al.
(2017) and Zhao et al.
(2010) consider a dynamic graph as a set of ordered static graphs.
For each time point, we are given a static graph such that G d = { G d 1 , G d 2 , · · · , G dt } , and a time-dependent given binary matrix with I t,i,j indicating the interaction between two agents at each time.
Then we have the agent-wise PSR as
[Equation]
Compared to (4), the coefﬁcient I t,i,j is time-dependent, which brings a challenge to our theoretical bound.
We consider a dynamic graph experiences a trajectory path, assuming every node has a chance at least p to interact by connection with another node at any time point.
For example, for a node i , if we take the union set of the nodes interacted with i over the path, then the union set could form a static complete graph; in other words, if i connects j , then we can obtain a valid sample to estimate Q i,j as the complete static graph does, if not, then we skip to the next time point.
For the complete static graph, we need the trajectory to run at least N time points to collect enough data to estimate our conditional operator accurately Q i,j .
Furthermore, The total number of time points needed by i until the N th interaction with j follows a negative binomial distribution NB ( N, p ) .
On average, we need Np time points before we see i, j completely connecting N times.
Now we consider node i could interact with every node in a set of nodes ( J : | J | = n − 1 ) for N number of time points.
We assume the interaction between two nodes ( i, j ∈ J ) does not affect their interaction with other nodes.
Thus we have a set of independent negative binomial random variables { J l } n − 1 l =1 ∼ NB ( N, p ) to characterize the interaction of i with j ∈ J .
So we are interested in the expectation of the maximum of J 1 , . .
, J n − 1 , a statistics that tells us the expected maximum number of time points of collection of measurements needed for the node i to be able to connect with every node j ∈ J for at least N number of time points.
We denote it as J { 1 ,...,n − 1 } and we have
[Equation]
Lemma 2.
Under the same environment depicted in 1 and given a dynamic graph G d with every node has a chance at least p to interact with another node in a one-time point, let N be the number of time points we collect data of measurements in order to get the bound in Theorem 1, if Equationand 2 used, then we need at least
[Equation]
F [ K ( q, n, N )] + O (1) , (7) total number of time points, where q = 1 − p , K ( q, n, N ) :
= log 1 /q ( n − 1)
+ ( N − 1) log 1 /q (cid:104) log 1 /q ( n − 1) (cid:105)
+ ( N − 1) log 1 /q
p − log 1 /q ( N − 1)!
, F is a periodic C ∞ -function of period 1 and mean value 0 whose Fourier-coefﬁcients are given by ˆ F ( k )
= − 1 log ( 1 q )Γ( − 2 kπi log ( 1 q )) for k ∈ Z \ { 0 } .
Then (cid:13)(cid:13)(cid:13) (cid:98) Q i − Q
i (cid:13)(cid:13)(cid:13) achieves the same bound as in Theorem 1 with probability at least 1 − 3 (cid:15) . In other words, (cid:13)(cid:13) (cid:13) (cid:98) Q i − Q
i (cid:13)(cid:13) (cid:13) ≤ n ∆ .
(7) is an asymptotic expansion of the right-hand of (6) .
We give the proof in Appendix E.3.
Lemma 2 gives the worst-case bound for our estimation under the dynamic graph.
The result tells us that if we need 1 more sample of measurement for our algorithm to converge on the complete static graph, we need roughly log 1 /q log 1 /q ( n − 1) more samples on the dynamic graph.
So as long as we allow enough learning time, the algorithm can converge with high probability.
Complexity with Increasing Agents : For the complete static graph, we need to evaluate (cid:98) Q i,j and (cid:98) P i,j every time point, which requires O ( n 2 ) operations and space.
Overall, with an increased number of agents, our MAPSR has a polynomial O ( n 2 ) scaled complexity, which is feasible for learning in a large number of agents environment and is more efﬁcient compared to the centralized MAPSR (Chen et al., 2020) theoretically, which is combinatorially sample complex, the analysis is shown in Appendix A.4.
For a non-complete static graph with k maximum number of degrees for k (cid:28) n , which is more common in the real world, because a very far-away robot will not likely affect the targeted robot, the operation will be signiﬁcantly decreased to O ( k 2 ) .
After we get the (cid:98) Q t,i and (cid:98) P t,i , we build the update rule by using W i , F i , Z i , as described in section 3.3.
D ECISION -M AKING F RAMEWORK W ITH MAPSR M ODEL
Previous work on single-agent proposed an end-to-end training algorithm (Hefny et al., 2018a) for
PSR model and policy learning.
Here we design an algorithm for multi-agent settings and incorporate our MARPSR model containing the interactive graph component.
We propose an online learning algorithm to learn the MAPSR and agent policies simultaneously.
Our algorithm is shown in Appendix B.2.
We call the algorithm MAPSRL .
The algorithm can be divided into two parts.
Firstly, we have a state tracking component, which lets the agents explore the environment and learn an initial predictive state and its parameters W i and Z i .
The agents ﬁrst enter the initialization phase (Line 1-4), where we initialize MAPSR by executing an exploration policy to collect data.
The agents explore the environment and learn the MAPSR parameters under the given interactive graph.
Secondly, we also have a policy component, in which, by starting from the initialized MAPSR and a random policy, the agents experience a T length trajectory under the current policy function that maps predictive states { Q t,i } ni =1 to actions { a t,i } ni =1 (line 5-18).
The agents then update to the next predictive state as introduced in subsection 3.3.
Agents also save trajectories (actions, observations, predictive states, next predictive states, and rewards) over the path.
Finally, a
MARL algorithm conducts the policy learning component.
We use a diagram in Appendix B.3 to illustrate this process.
We update the MARL and MAPSR model parameter Θ = { Θ MAPSR , Θ MARL } (line 19) by minimizing the following objective function.
[Equation]
Here the l 1 (Θ MARL ) is the loss for MARL, and l 2 (Θ MAPSR , MARL ) is the MSE between prediction and actual observation.
And p ( τ i | Θ) is the distribution over trajectories induced by the policy and
MAPSR.
Θ MAPSR =
{ W i , Z i } ni =1 denotes MAPSR’s parameters, and Θ MARL denotes the MARL part.
Usually, in a continuous environment, each agent’s policy will be parameterized by its actor network that outputs the mean and diagonal covariance of a Gaussian distribution over the continuous action space.
We could also give a parameter sharing actor network that maps individual PSR to parameters of a Gaussian distribution over the individual action space if agents are homogeneous.
Now we devise our MAPSR based MARL algorithm in detail.
Our algorithm follows the actorcritic framework.
We ﬁrst develop an algorithm under partially observable environments, where each agent has its own critic and actor independently, the gradient of the policy is written as ∇ θ i J ( θ i ) =
E [ ∇ θ i log π θ i ( a i | Q i )
( r i + γV π i ( Q t +1 ,i )
− V π i ( Q i ))] , and the independent critic is updated by minimizing the loss L = E [( V ( Q i ) − y i ) 2 ] .
Here the predictive states Q i estimated by the MAPSR are considered as states to ﬁt the value and policy functions.
We call it MAPSRL-1 .
As we know, independent actor-critic (IAC) (Foerster et al., 2018) would not work well since the environment is not stationary under multi-agent setting, the MADDPG by (Lowe et al., 2017) solves a non-stationary environment by considering other agents actions; however, every agent needs a separate critic that has the global information.
Instead, we use a centralized critic like as in COMA (Foerster et al., 2018) and LICA (Zhou et al., 2020) to save space.
Our centralized critic minimizes the loss L = E
[( V π ( Q 1 , · · · , Q n ) − y ) 2 ] .
Unlike MADDPG using the deterministic policy, the gradient is taken on the policy parameter space, which is reﬂected in LICA (Zhou et al., 2020).
So our policy gradient is written as ∇ θ
i J
( θ i ) =
E [ ∇ θ i log π θ i ( a i | Q i ) ∇ θ
i
V π ( Q π θ 1 1 , · · · , Q π θn n )] , Also, to solve the non-stationary environments when each agent is learning, it uses the joint predictive states as input.
The beneﬁts are 1.
PSR ( Q ) is a function of action ( a ), bringing agents’ action information into the policy gradient, 2.
Also, our interactive MAPSR brings the effect of other agents’ actions to the policy gradient, so it does not learn a deterministic policy like MADDPG rather a policy that affects the predictions of future observations considering other agents’ actions 3.
Without the need to input other agents’ actions, which saves the input space since the predictive state is a conditional operator that considers other agents’ actions.
We call it MAPSRL-2 .
We put our integration details in Appendix B.
[Figure]
E XPERIMENTS
[Figure]
of MAPSRL under some OpenAI Gym
- ments ( Witt et al., 2020), such as
-agent swimmer, hopper, ant robotic agent is represented as a body graph, where vertices (joints) are connected by adjacent edges (body segments) as shown in Appendix Figure 5.
Each agent controls its joints based on the local information observed.
All tasks are learned under the partially observable environments by manually hiding some observations for each agent.
The goals of the multi-agent systems are aligned with their corresponding single-agent ones.
However, different from the single-agent system, the agents in the multi-agent system need to collaborate to reach their goals.
For the interactive graph, for simplicity, we considered a complete static graph, where we assume every agent is connected with all other agents in the graph.
We defer the details of the robotic agents to Appendix C.1 and the details of the experimental setup to Appendix C.2.
Baselines and Evaluation.
We run 50 iterations for each experiment and collect M =trajectories in every iteration with a maximum of 1000 steps in every trajectory.
After each iteration, we compute the average return R =
M (cid:80) ni =1 (cid:80) Mb =1 (cid:80) T b t =1 r ti,b on a batch of M trajectories, where, T b is the length of the b th trajectory.
We repeat this process using ten different random seeds and report the average and a standard deviation.
To verify the effectiveness of interactive graph, we introduce a baseline called Independent PSR learning (IPSR); in this model, we do not consider the graph, so we formulate n independent single PSR without considering their interactions, which means there are no Q
i,j any more but only Q i .
The architecture of MAPSRLand MAPSRL-2 remains the same.
To verify the advantage of PSR, we introduce another baseline (MARL) where we take out the MAPSR entirely, so it matches with the MARL run on a partially observable environment.
For MAPSRL-1, its MARL baseline is IAC (Foerster et al., 2018), and for
MAPSRL-2, is MADDPG (Lowe et al., 2017).
Results And Discussion Figure 1 illustrates the empirical average return vs. the number of interactions with the environment measured in time steps.
Our MAPSRL methods consistently outperform IPSR and get the highest rewards under partially observable environments, which justiﬁes the representation power of the interactive graph to assist agents in learning in the non-stationary environment with limited observation when the MARL algorithm has a defect (IAC).
Moreover, it can also boost the performance of the existing good MARL algorithm (MADDPG).
To further verify the effect of learning the PSR part, we also plotted the predicted trajectory to verify the MAPSR’s performance for predicting the observations in Figure 2.
We plotted the predicted observations vs. actual observations in iterations 1 , and 40 , respectively, for MAPSRL-2.
We plotted a row × columns ﬁgure, with each row representing the observation feature and each column representing each agent.
As we can see from the ﬁgures, the ﬁrst iteration of the learning does not predict the actual observation very well; it has some mismatches.
However, as learning progresses, the predictions
[Figure] [Figure]
To enrich our algorithm environment, we also test our algorithm (MAPSRL-2) in multi-agent particle environments, using the benchmark by (Lowe et al., 2017), please check Appendix D.2 for experiment details.
We test our algorithm for large n cases.
In this environment, the agents can have cooperative goals such that all agents must maximize a shared return and conﬂicting competitive goals.
We set up the environments where agents can only perform physical actions but not communication; however, to achieve the goals, agents need explicit communication about others’ locations to achieve the best reward.
These partially observable environments give us the motivation to test our method.
We report the rewards in
Figure 3.
We see that MAPSRL outperforms IPSR and baseline, in terms of the convergence speed and ﬁnal attained rewards, with a different number of agents.
The predictive states convey the information that can help communication between agents in limited communication and observation environments.
C ONCLUSION
We propose a MAPSR model, extending ideas from single-agent predictive state representations to a multi-agent scenario, during the process, we introduce the dynamic interactive graph to model agents’ interactions.
Furthermore, we provide the theoretical guarantees of the MAPSR model.
Finally, a learning algorithm that supports gradientbased deep MARL methods is developed.
Our method provides a model-based MARL framework under a partially observable environment.
The experiments proved that our model assumption is valid by observing the highest return while reducing the observations’ prediction error over trajectories.
E THICS S TATEMENT
The authors of this paper claimed that we have read the Code of Ethics, adhered to it, and explicitly acknowledged this during the submission process.
We claimed that we did not perform any of the following studies that involve human subjects, practices to data set releases, potentially harmful insights, methodologies and applications, potential conﬂicts of interest and sponsorship, discrimination/bias/fairness concerns, privacy and security issues, legal compliance and research integrity issues.
R EPRODUCIBILITY S TATEMENT
We have an appendix section after the main paper to include the model supplements, algorithm details, method diagram, experiment settings and proofs for our main theorems.
We provide a link which directs to our codebase in the supplementary material.
A PPENDICES T O R EINFORCEMENT L EARNING UNDER A M ULTI - AGENT P REDICTIVE S TATE R EPRESENTATION M ODEL : M ETHOD AND T HEORY A M ETHOD D ETAILS A. M ATHEMATICAL M ODEL OF PSR
Here, we give the mathematical model for PSR. As in section 3.1, we let τ t := { ( o l , a l ) } t + k − 1 l = t as a test (a sequence of action-observation pairs) at time t , then we have a subset τ ot := { o l } t + k − 1 l = t for observations, and τ at := { a l } t + k − 1 l = t for actions.
And we let h t := { ( o l , a l ) } t − 1 l =1 as the history at time t .
A test is executed at time t if we intervene to select the sequence of actions speciﬁed by the test.
It is said to succeed at time t if it is executed and the sequence of observations in the test matches the observations by the system (Boots et al., 2013).
The prediction of a length- k test τ t is deﬁned as the probability distribution that the test succeed at time t , given history h t : Pr (
[Equation]
Pr (
[Equation]
Let T = { τ z } , z ∈ N is a ﬁnite set of core tests.
And we use T o = { τ o 1 , · · · , τ o | T | } and T a = { τ a 1 , · · · , τ a | T | } to indicate its observation and action parts.
A.1. L INEAR PSR
A linear PSR is the prediction vector
[Equation]
that contains the probabilities of success of the tests in T , if and only if for any test
[Equation]
Where f τ : [0 , 1] | T | (cid:55)→ [0 , 1] is a linear projection function.
For simplicity of notation, we use the same notations for Q t P t , W, F , which only follows the meaning deﬁned in this section.
It means that knowing the probabilities for the tests in T is sufﬁcient for computing the probabilities for all other tests in the system.
The prediction vector is a sufﬁcient statistic for the system at time t , so we call it as state for the PSR at time t .
Therefore, PSR can represent state by using a vector of predictions of fully observable quantities (tests) conditioned on past events (histories).
The prediction function f τ is linear and has one-to-one relationship to a test τ such that f τ ( Q t ) =
f (cid:62) τ Q t ∀ t .
The linear PSR can still represent systems with nonlinear dynamics.
To maintain predictions in T , we need to update the state Q t .
To do that, we predict the success of any core test τ z prepended by a new action a and observation o at time t + k − 1 , which we call oaτ z .
Based on Bayes rule, we have
[Equation]
f aoτ z , f ao ∈ R | T | are linear operators such that ( ∀ τ z ∈ T , ∀ a ∈ A , ∀ o ∈ O ) .
Then let F aoτ be the matrix with its columns as f aoτ z for all τ z ∈ T .
Then the updated state in PSR is obtained by
[Equation]
[Equation]
Given the initial prediction vector Q 1 , the PSR can update with equation 12.
This recursive application of Bayes rule to a belief state is called the Bayes ﬁlter.
Now we have seen the extended prediction vector, so we deﬁne extended predictive state as:
[Equation]
Clearly, we can see there exists a linear extension operator W such that P t = W ( Q t ) .
And given the new observation, we can have a Bayes ﬁlter F such that Q t +1 =
F ( P t , o, a ) .
We also introduced these operators in section 3.3.
A.1. R EPRESENTATION OF S TATE AS C ONDITIONAL E XPECTATION OF S UFFICIENT S TATISTICS
Instead of learning the distribution, here we recover the idea of representation of states as a conditional expectation of sufﬁcient statistics and using the supervised learning method to learn (Hefny et al., 2015).
Let history h
ot is a sequence of observation that starts from the beginning of time and ends at time t − 1 .
We deﬁne the belief state b t =
Pr ( s t | h ot ) , where s t is the current state of the world.
b t +1 =
Pr ( s t +1 | h ot +1 ) is the next time belief state.
We call b t as "belief state", which represents the probability distribution over state space, also represents the knowledge and uncertainty about the true state of the system.
In dynamic system, the task of getting the updated b t +1 with given b t and new observation o t is called ﬁltering.
The task of estimating the Pr ( s t +1 | h ot ) with given current b t without incorporating any new observation is called one-step prediction.
Instead maintaining a belief b t over states, spectral algorithms try to recover observable operators that can be used to perform ﬁltering and prediction directly, by maintaining the expected value of a sufﬁcient statistic of future observations.
Let a test τ ot is a length k sequence of observation deﬁned before.
As the recursive Bayes rule holds, we can get
the next time prediction vector Pr ( τ ot +1 |
h ot +1 ) using the new observation o t + k and the current prediction Pr ( τ ot | h ot ) , we also deﬁne an extended prediction vector Pr ( τ ot +1 | h ot ) .
These prediction vectors characterize the state of the system and they can be estimated by observable quantities.
Given Pr ( τ ot +1 | h ot ) , ﬁltering becomes the task of getting the updated prediction vector Pr ( τ ot +1 |
h ot +1 ) , "conditioning" on o t .
One-step prediction becomes getting the Pr ( τ ot +1 | h ot ) , "marginalizing" over o t .
Therefore, the spectral algorithms avoid explicitly estimating the latent state or the initial, transition, or observation distributions.
We let Q t =
E [ ν t | h ot ] , where ν t a vector of features that determines the distribution of future observations Pr ( τ ot | h ot ) .
For simplicity of notation, we use the same notations for Q t , P t , W, F .
Let P t =
{+E [ ϕ t | h ot ] , where ϕ t is a vector of features that determines the distribution of observations Pr ( τ ot +
+}
| h ot ) .
We call Q t is the transformed predictive state.
So
Q t +1 = E (cid:2)
ν t +1 |
h
ot +1 (cid:3) , is the updated predictive state, and P t is the extended predictive state.
Let h ot be the feature vector of history
h
ot .
In Hidden Markov Models and Kalman ﬁlters, the extended state P t is linearly related to the predictive state Q t .
Which is P t = WQ t .
Estimation of the W can be done using linear regression with samples ν t and ϕ t , however, due to the overlap between observation windows, the noise terms on ν t and ϕ t are correlated, which will cause biased estimate.
The instrumental regression (Pearl et al., 2000;
Stock et al., 2012) is employed.
h ot is a instrumental variable that do not overlap with sequence { l } l = t + k − 1 l = t and { l } l = t + k l = t .
The correlation corr (h ot , e ( ϕ t ))
{+= 
+}
and corr (h ot , e ( ν t ))
= 0 , where e is a measure of error.
By taking the conditional expectation of P t = WQ t given h ot , we have
[Equation]
[Equation]
Based on the above relationship, we ﬁrst estimate the E [ ϕ t | h ot ] and E [ Q t | h ot ] by sample h
ot , ν t , and ϕ t , we then use the estimates to compute W .
So if we start with Q 1 , we can compute P 1 = WQ 1 , and get the Q t +1 =
F ( P 1 , o 1 ) , where F is the Bayes ﬁlter to update the state.
A. L EARNING FOR MAPSR IN D ETAILS
Here we introduce the details about learning MAPSR, which supplements the section 3.3 and 4.1.1.
A.2. L EARNING F OR (cid:98) Q t,i,j AND (cid:98) P t,i,j
To calculate (cid:98) Q t,i,j , we introduce two sets of linear operators T i,j and U i,j , such that T i,j ( ψ ht,i,j ) ≈
C ψ ot,i ψ at,j | ψ ht,i,j and U i,j ( ψ ht,ij )
≈ C ψ at,j ψ at,j | ψ ht,i,j .
We estimate them by using two ridge regressions:
[Equation]
[Equation]
L represents the ridge regression loss, and R represents the regularizer.
After learning T i,j and U i,j , we can get the estimate of C ψ ot,i ψ at,j | ψ ht,i,j and C ψ at,j ψ at,j | ψ ht,i,j .
Then we get the (cid:98) Q t,i,j by equation 2.
Similarly, we use extended features to obtain (cid:98) P t,i,j .
A.2. E XTENSION , F ILTERING AND P REDICTION F UNCTIONS
Filtering.
To obtain (cid:98) Q t +1 ,i from (cid:98) Q t,i , (cid:98) P t,i , we use ﬁltering.
We denote F i as the ﬁltering function.
We describe the ﬁltering process as below.
From { o i , a i } ni =1 , we obtain the embedding { φ ot,i , φ at,i } ni =1 .
We then compute the observation covariance
[Equation]
We then multiply the extended state by inverse observation covariance to change predicting φ ot,i into conditioning on φ ot,i .
[Equation]
18) × here is to denote n − mode (matrix or vector) product, × φ
ot,i means multiplying the tensor by a matrix (or vector) in mode φ
ot,i .
We condition on φ ot,i and φ at,i to obtain shifted state.
[Equation]
Based on the updating rule, Q t +1 ,i =
F i ( P t,i , o t,i , a t,i ) , and P t,i =
W i ( Q t,i ) , we write the ﬁltering equation.
[Equation]
where P t,i :=
M ψ ot +1 ,i ⊗ φ ot,i | ψ at +1 ,i ,φ at,i ; ψ ht
[Equation]
i .
F i usually is known because it is obtained through the above calculation using known quantities, however, W i , Z i must have to be learned by using regressions.
Extension.
The W i can be learned by kernel regression if we know the (cid:98) Q t,i and (cid:98) P t,i .
Previous work (Hefny et al., 2018a) demonstrated the kernel regression model for learning single-agent PSR, here we extend to the MAPSR.
We set the model parameter W i .
We optimized a ridge regression problem for W i .
[Equation] Prediction. We can also get the prediction about the next one time observation o t by the regression function such as: [Equation]
We solve the prediction regression function Z i by another ridge regression:
[Equation] A. T ENSOR D ECOMPOSITION OF MAPSR Existed Formulation. (Chen et al., 2020) use a n + 1 multi-dimensional tensor called system dynamics tensor D ∈ R |T 1 |×···×|T n |×|H| to represent the system dynamics of the MAPSR, with |T 1 | representing the cardinality of the tests for agent 1 , n denoting the number of agents, and the |H| being the cardinality of the joint history.
Each element of the tensor is a probability of a joint test given joint histories.
Given the system dynamics tensor, ﬁnding the latent predictive state can be transferred into ﬁnding a minimal linearly independent set from the system dynamics tensor, and it can be solved by spectral method such as tensor decomposition.
As mentioned earlier, this formulation can not satisfy our needs.
In Proposition 1 of Appendix A.4, we also give the sample complexity analysis such that the sample size needed to formulate D scales exponentially with the length of tests and number of agents.
Here we give a summary of tensor decomposition of their method.
Given a system dynamic tensor D such that:
[Equation]
Here ◦ is the outer product.
D 1 , · · · , D n , F are matrices.
The factor matrices D 1 , D n , F consist of the vectors, i.e., D 1 =
[ D 1:1 D 1:2 · · · D 1: R ] ∈ R |T 1 |× R .
A colon is used to indicate all elements of a mode, thus, the R th column of D 1 is denoted by D 1: R .
For any i 1 ∈ { 1 , . . .
, |T 1 |} , i n ∈ { 1 , . . . , |T n |} , and k ∈ { 1 , . .
. , |H|} , after the decomposition, we get
[Equation]
where { i 1 , · · · , i n , k } are index corresponding to the speciﬁc dimension of the D .
The last dimension of the tensor D is compressed in a matrix F , and its row vector x k =
[ x k (1) . . .
x k ( R )]
∈ R 1 × R , k ∈ { 1 , . .
. , |H|} is a summary of joint history and can be considered as a compressed version of the system
predictive state vector p
( Q | h k ) , the joint history h k ∈ H ( k ∈ [1 , |H| ]) at time step s = | h k | , where Q is the core joint test set.
The whole ﬁbers listed in the set Q form a basis of the space spanned by the mode-(n+1) ﬁbers of tensor D .
Thus, by constructing the vector m = ( λ ∗ D 1 i 1 : ∗ · · · D ni n : ) T , where ∗ is Hadamard product.
D 1 i 1 : denotes the i 1 -th row vector of D 1 .
Then we could rewrite the previous equation as
[Equation]
m ( r ) is a scalar that m ( r ) =
λ r D 1 i 1 r · · · D ni n r .
And x k is the system state vector and m is the prediction parameter, both of them are obtained by the tensor decomposition.
A. A NALYSIS OF S AMPLE C OMPLEXITY FOR F ORMULATING THE S YSTEM D YNAMIC T ENSOR
The paper (Chen et al., 2020) does not analyze the sample complexity to construct D , which is a n + 1 multi-dimensional tensor D ∈ R |T 1 |×···×|T n |×|H| .
We give this analysis.
The D i 1 , ··· ,i n ,k :=
Pr ( t 1 i 1 , · · · , t ni n | h k ) is an element of that tensor D such that t 1 i 1 is the i 1 -th test of agent 1, similarly, the t ni n is the i n -th test of agent n , and h k is the joint history.
Proposition 1.
In a n -agents system, assume every agent has the same observation and action space |O| , |A| , for a length- k test, to formulate a complete system dynamics tensor D deﬁned in equation 21.
Assume each entry of the tensor needs S samples to give a sufﬁcient estimation using Monte-Carlo roll-out method, then the total sample size is at least ( |O||A| ) kn S , if the agents are homogeneous, in other words, they are permutation invariant such that identity does not matter, then the total sample size is (cid:0) |O||A| k + n − 1 n (cid:1) S .
Proof.
At one time step, for any agent, it has |O||A| different combinations for the joint test, then for a length of k tests, it follows that ( |O||A| ) k number of different choices.
Then the tensor D would need ( |O||A| ) kn elements to cover all the possible length k tests.
So the total sample size is ( |O||A| ) kn S .
If the agents are homogeneous, then the ordering does not matter, for each agent, we have ( |O||A| )
k number of different choices for length k test, so the total choices for n agents are (cid:0) |O||A| k + n − 1 n (cid:1) , then we need (cid:0) |O||A| k + n − 1 n (cid:1) S samples.
The sample complexity is exponentially scaled with the number of agents and length of tests.
B A LGORITHM AND I NTEGRATING WITH MARL METHOD
We ﬁrst introduce some backgrounds on two multi-agent frameworks: Multi-agent Markov Decision
Process (MMDP) and Multi-agent Partially observable Markov Decision Process (MPOMDP) since the algorithms in our paper are developed based on the framework of MPOMDP.
B. M ULTI - AGENT MDP AND M ULTI - AGENT POMDP MODEL B.1. MMDP
MMDP model is a tuple (cid:0) S, N, { O i } i ∈ [ n ] { A i } i ∈ [ n ] , T, R (cid:1) , where S and N are ﬁnite sets of states and agents.
A i is a ﬁnite set of actions available to agent i ; T : S ×
A 1 × · · · × A n × S (cid:55)→ [0 , 1] is a transition function; and R : S (cid:55)→ R is the reward function.
Each agent i obtains reward as function of the state and agent’s action
r
i : S ×
A i (cid:55)→ R , and receives a private observation from the state by the observation channel o i : S (cid:55)→ O i .
The state has distribution d : S (cid:55)→ [0 , 1] .
Each agent aims to maximize its own total expected return r i = (cid:80) Tt
{+=
+}
{+γ t r ti where γ is a discounted factor and T is the time horizon. In a shared reward situation, there is a team reward function r : S × A i · · · × A n (cid:55)→ R , agents aim to maximize one shared total expected return r = (cid:80) Tt =
 γ t r t . B.1. MPOMDP+}
A MPOMDP model is a tuple (cid:0) S, N, { O i } i ∈ [ n ] { A i } i ∈ [ n ] , { Ω i } i ∈ [ n ] , T, R (cid:1) , where ( S, A i , T i , O i , Ω i , R ) describe a single-agent POMDP.
O i is the set of observations the agent i can make.
Ω i : S × A i ×
O i (cid:55)→ [0 , 1] is the agent’s observation channel function, which speciﬁes probabilities of observations given agent’s actions and resulting states. ( S, A i , T i , R i ) describes a single agent MDP; and each agent i obtaines reward as function of the state and agetn’s action r i : S ×
A i (cid:55)→ R . Each agent aims to maximize its own total expected return r i = (cid:80) Tt
{+=
 γ t r ti where γ is a discounted factor and T is the time horizon. In POMDP, an agent’s belief about the sate is represented as probability distribution over S .+}
The agent has prior belief b 0 ,i The agent’s current belief, b t,i over S , is continuously revised based on new observations and expected results of performed actions. The belief update takes into account changes in initial belief, b t − 1 ,i , due to action a t,i , executed at time t − 1 , and the new observation, o t,i . The new time belief state can be obtained from basic probability theory as follows:
[Equation]
, where β is the normalizing factor. B. A LGORITHM : MAPSRL
We give the details of MAPSRL in Algorithm 1.
B. MAPSRL IN D IAGRAM
We also use a diagram (Fig 4) to depict the algorithm.
The algorithm runs k iterations; each iteration ﬁrst uses the policy to roll out data and uses the regressions to obtain the PSR parameters.
Then it executes the policy phase by using the PSR as the input of policy to generate action and using the current PSR parameters to update the predictive state.
At the end of the policy phase, it updates both the PSR and policy parameters.
The policy parameters and the linear relationship captured by
W i and Z i are parameterized by the neural network, section C.3 has the network architectures.
The loss is a composite loss that includes loss from actor-critic and the loss from the predictive state representation.
The next iteration will re-learn the PSR parameters using the newly generated data based on the current policy obtained from the previous iteration; then, it does a soft update to update the PSR parameters with ones obtained at the previous iteration. B. I NTEGRATING T WO C OMMON MARL A LGORITHMS I NTO MAPSR M ODEL
Here we provide a brief intuitive introduction about how we connect existed MARL algorithms with MAPSR. Please look at Fig 4 for demonstration. ψ ot,i , ψ at,j , and ψ ht,i,j are embedding of ( o t : t + k − 1 ,i , a t : t + k − 1 ,j ) , and ( o 1: t − 1 ,i , a 1: t − 1 ,j ) .
We also have embedding for extended part, labeled as ξ . Q t,i,j and P t,i,j are estimated by tensor regression using the embedding vectors.
G represents the given 1: Input: Learning rate η , a graph G , a static complete graph G c or static non-compete graph G s or dynamic graph G t 2: Initialize MARL Policy Θ MARL randomly 3: for k = 1 , 2 , 3 , · · · iterations do 4:
State Tracking Phase 5:
Sample b = 1 , 2 , 3 , · · · M batch of initial trajectories: { ( o bt , a bt ) } Mb =1 from existed policy obtained from previous iteration k − 1 : { π k − 1 i } ni = 6:
Let π θ MARL =
{ π k − 1 i } ni =1 if available or the initial policy; 7:
Given G , calculate (cid:98) Q i and (cid:98) P i , and obtain the initial W i , Z i , F i : 8: (1).Regression Q t,i,j = T i,j ◦ U i,j ( h t,i,j ) to get the (cid:98) Q t,i,j and (cid:98) P t,i,j 9: (2).Given the graph G , using equation 1, 4, or 5 to obtain (cid:98) Q t,i , (cid:98) P t,i 10: (3).Given the (cid:98) Q t,i , (cid:98) P t,i , to obtain W i , Z i , F i 11: 12:
Policy Phase 13:
Initialize MAPSR parameters Θ MAPSR =
{ Q 1 ,i , W i , Z i } ni =1 from state tracking phase and previous iteration by a soft-update:
W [Equation] =
[Equation]
i ,
[Equation] [Equation]
F i 14: for b = 1 , 2 , 3 , · · · M batch of trajectories from { π k − 1 i } ni =1 do 15:
Reset episode: a b 0 ,i , o b 0 ,i 16: for t = 0 , 1 , 2 , · · · T roll-in in each trajectory do 17: for Each agent i do 18:
Get observation o bt,i and reward r bt,i 19:
Extension P bt,i =
W i ( Q bt,i ) 20:
Filtering Q bt +1 ,i =
F i ( Q bt,i , a bt,i , o bt,i , W i ) 21:
Execute a bt +1 ,i ∼ π k −
i
( Q bt +1 ,i ) 22:
Predict (cid:98) o
bt,i =
Z i ( Q bt,i , a bt,i ) 23:
Collect o bt,i , (cid:98) o bt,i , a bt,i , r bt,i , Q bt,i , Q bt +1 ,i 24: end for 25: end for 26: end for 27:
Update Θ using D = {{{ o bt,i , (cid:98) o bt,i , a bt,i , r bt,i , Q bt,i } ni =1 } Tt =1 } Mb =1 : Θ ← Update (Θ k − 1 , D, η ) as in Equation (8), get W ki , Z ki , and π ki , the linear operator W i , Z i , and nonlinear operator π i are all parameterized by neural network.
28: end for 29: Output: Return Θ = (Θ MAPSR , Θ MARL ) graph, the estimation of Q t,i and P t,i are based on graph G and equation 1, equation 4 and equationfor reference.
Please also go to subsection 3.2 and section 4.1.1 for a verbal description.
The policy network uses the predictive state as input to return the action.
The agent takes the action to get the observation.
The ﬁlter F i takes predictive state, action, and observation as inputs to get the next predictive state.
We shows a centralized critic and gives the description in paragraph B.4.2.
The loss is composed into two parts, L θ MAPSR =
(cid:107)
Z i { F i [ W i ( Q t − 1 ,i ) , a t − 1 ,i , o t − 1 ,i ] , a t,i } − o t,i (cid:107) 22 , and L θ critic , which we gives detailed explanation in equation 29.
Please also go to section 5 and
Algorithm B.2 for more details about the framework.
B.4. MAPSRL-
It is based on the IAC (Foerster et al., 2018) which directly applies the single-agent policy gradient to have each agent learn independently, with the idea behind independent Q-learning (Tan, 1993), with actor-critic in place of Q-learning. IAC trains an actor-critic pair for each agent, resulting in actors π i ( a i | o i ) and critics V i ( o i , a i ) .
[Equation]
[Figure] Section B.4 and section C.3 give a detailed verbal description about this architecture. While IQL and IAC agents display a strong ability to optimize individual rewards (Tan, 1993), the lack of global information and a mechanism for cooperation means they are likely to settle for sub-optimal solutions.
Here we use the predictive state Q i to ﬁt the value and policy functions. And we train an actor-critic pair for each agent, resulting in actors π i,θ i ( a i | Q i ) and critics V i ( Q i ) .
[Equation]
If the agents are homogeneous, we can share the critic and actor network.
We have the critic loss as
[Equation]
[Equation] Here (cid:98) V is the target value function.
B.4. MAPSRL-
The algorithm borrows the idea from MADDPG (Lowe et al., 2017) and LICA (Zhou et al., 2020) using the gradient of the PSR-value function with respect to policy to direct the policy gradient update.
MADDPG (Lowe et al., 2017) is an extension of deep deterministic actor-critic policy gradient (DDPG) (Lillicrap et al., 2015) to multi-agent setting such that let each agent’s own critic is augmented with extra information about the actions of other agents, while their individual actor maintains a local state or observation.
The gradient of each agent is: [Equation] The action-value function V i is updated as
[Equation]
Where a (cid:48) is the set of target policies with delayed parameters.
Unlike MADDPG using the deterministic policy, the gradient is taken on the policy parameter space, which is reﬂected in LICA Zhou et al. [Equation]
Our method also uses the centralized critic to save space.
Also, in order to solve the non-stationary environments when each agent is learning, it uses the joint predictive states as input without considering other agents’ actions, which saves the input space since the predictive state is a conditional operator that considers other agents’ actions.
The critic loss is deﬁned as below:
[Equation]
Even though the original MADDPG has the current actions as input to the value function so that the environment is stationary, we take out this operation because the predictive state already considers the effect actions can bring.
C E NVIRONMENT AND E XPERIMENT C.
[Figure]
Here, we give the details of setting up our multi-agent environment - MAMuJoCo.
As introduced in section 6, the many agents are constructed by separating a existed single agent into parts, and each agent will only control a part of the whole agent (Figure 5).
Partially Observable space : MAMuJoCo is a simulated robotic environment, the partially observable property is achieved by only allowing partial information to the agents.
For all environments, only the angles of the agent’s joints are visible to the network; the velocities are hidden.
Action : Each agent’s action space in MAMuJoCo is given by the joint action space overall motors controllable by that agent.
Observation : For each agent i , observations are constructed by inferring which body segments and joints are observable by an agent i .
Each agent can always observe all joints within its sub-graph.
A conﬁgurable parameter k ≥ 0 determines the maximum graph distance to the agent’s subgraph at which joints are observable.
For example, k = 0 means agents can only observe their own joints and body parts, while k = 1 means it can observe its adjacent joints, which has 1 graph distance to the agent.
The agent observation is then given by a ﬁxed order concatenation of each observable graph element’s representation vector.
Depending on the environment and conﬁguration, representation vectors may include attributes such as position, velocity, and external body forces.
In addition to joint and body segment-speciﬁc observation categories, agents can also be conﬁgured to observe the robot’s central torso’s position and velocity attributes.
C. E XPERIMENT SETUP
We select three experiments from MAMuJoCo and give a detailed description of the experiments’ setup.
In all environments, the agent has the goal to maximize the velocity of the ﬁrst coordinate for the team.
We use k to denote the maximum observation distances to the subgraph.
We use ∆ d to denote the ﬁrst coordinate position difference between a time difference ∆ t .
Finally, we use R to denote the reward function.
Table 1 has the conﬁguration details for these parameters.
C. N EURAL N ETWORK A RCHITECTURE
We implement all algorithms using deep neural networks as function approximators.
We ensure that all policy, value, and action-value functions have the same neural network architecture among all algorithms to the extent each algorithm allows for a fair comparison. [Table]
For our experiments with continuous action spaces, a Gaussian distribution with a diagonal covariance matrix is used.
The policy network maps from the input feature to a Gaussian distribution vector µ .
Moreover, µ = [ mean, std ] , where mean is a vector speciﬁes the action means, and std vector speciﬁes the standard deviation.
In the implementation of all actor-critic method, all the actor-network is parameterized by a multilayer perceptron (MLP) with two hidden layers of size 400 and 300 respectively and ReLU activation, which takes in the individual agent’s predictive state and outputs the mean and covariance of a
Gaussian policy.
The critic network is also an MLP with two hidden layers with 400 and 300 units, respectively.
For MAPSRL-1, the critic network is used to approximate per-agent utilities, which receives each agent’s predictive state as input.
For IAC, same as MAPSRL-1, it receives agent local observation and individual action as input.
In MAPSRL-2, there is a shared critic network that approximates all agents utilities, which receives all agents’ predictive states as input.
In MADDPG, the critic receives the global state and the joint action of all agents as input.
The global state consists of the complete state information from the original OpenAI Gym environment.
Each decentralized actor (i.e., policy) network takes in each agent’s observation and outputs the agent’s action vector.
C. M ODEL P ARAMETERS
The hyper parameters for the the MAMuJoCo environments are in Table 2.
D S UPPLEMENT E XPERIMENTS D. MAM UJO C O S UPPLEMENT E XPERIMENTAL R ESULTS
We plotted the predictive observations compared to actual observations in Figure 2 at the beginning of the learning process (iteration 1 ) and end of the learning (iteration 40 ).
We also show the results of iteration ten and iteration 20 in Figure 6.
By comparing with iterations 1 and iterations 40 in
Figure 2, we see that the iteration 40 has the smallest difference between predictive observation and true observation, and the difference gets increased as the iteration goes to the earlier stage of the learning process.
So the predictive accuracy is improved incrementally with the learning progresses.
D. M ULTI -A GENT P ARTICLE E NVIRONMENT
We also test MAPSR into another environment, multi-agent particle environment (Lowe et al., 2017).
The agents are displaced into a 2-dimensional coordinate.
This environment does not assume that all agents have identical action and observation spaces.
We run experiments using a different number of agents on two environments, the predator-prey, and cooperative-push.
We use the same conﬁguration [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] In Table [Figure] we report the hyperparameters for multi-agent particle environments.
E D ETAILED P ROOFS
We give the proof of Theorem 1 in Appendix E.1, and we ﬁrst introduce the following Lemmas to prepare the proof.
Deﬁnition 2.
Let X 1 . . .
X k be independent random variables of dimensionality d X 1 . . .
d X k such that (cid:107) X k (cid:107) <
t x k .
Let { ( x 1 j . . .
, x k j ) } N j =1 be the N i.i.d samples from distribution of X 1 , . . . , X k ,
[Table] Length of test window [Table] Length of history window
[Table] the C X 1 := E [ X 1 X T 1 ] and (cid:98) C X 1 = 1 N (cid:80) Nj =1 x 1 j x T 1 j , and C X 1 X 2 :=
E [ X 1 X T 2 ] and (cid:98) C X 1 ,X 2 = 1 N (cid:80) Nj =1 x 1 j x T 2 j .
Also, we use the · ) , v ( · ) to denote the largest, smallest eigenvalue of a matrix.
Lemma 3 ((Tropp, 2015)) .
Let a j be a ﬁnite sequence of independent random, Hermitian matrices with dimension d .
Assume that 0 ≤ v ( a j ) and u ( a j ) ≤ L for each j .
Let S
=
(cid:80)
j a j , then for any η ∈ [0 , 1] , it follows that
[Equation]
Corollary 1.
Let X be a random variable, for any (cid:15) ∈ (0 , 1) such that N > t 2 x log(2 d X /(cid:15) ) v ( C X ) the following holds with probability at least 1 − (cid:15)
[Equation]
In other words, if N large enough, then (cid:98) C X and C X will be close enough.
Proof.
Deﬁne S j = 1 /Nx j x Tj .
Then it follows that u ( S j ) ≤ L = t 2 x /N and deﬁne (cid:15) :
= 2 d X e − σNv ( C X ) /t 2 x , which implies that σ = t 2 x log(2 d X /(cid:15) ) v ( C X ) N .
Then it follows from Matrix Chernoff
Inequality in Lemma 3 that P r ( v ( (cid:98) C X ) ≤ (1 − σ ) v ( C X )) ≤ (cid:15)
Lemma 4 ((Tropp, 2015)) .
A ﬁnite sequence { a j } of independent, random matrices with common dimensions a × b , and assume that E [ a j ] = 0 and (cid:107) a j (cid:107) ≤ L for each j , let S =
(cid:80)
j a j as a random matrix.
Let V ar ( S ) be the variance statistics such that V ar ( S )
= max { (cid:13)(cid:13) E [ SS T ] (cid:13)(cid:13) , (cid:13)(cid:13) E [ S T S ] (cid:13)(cid:13) } , then
[Equation]
Corollary 2.
With at least probability 1 − (cid:15) that
[Equation]
where L = t y t x + (cid:107) C Y X (cid:107) ≤ 2 t y t x and V ar = max { t 2 y (cid:107) C X (cid:107) , t 2 x (cid:107) C Y (cid:107)} + (cid:107) C Y X (cid:107) 2 ≤ 2 t 2 y t 2 x
Proof.
Let X, Y be two random variables, and let a ﬁnite sequence { a j } of independent random matrices to satisfy a j =
y j x
Tj − C Y X .
So the a j will have dimensions d X × d Y .
Let random matrix
S =
(cid:80)
j a j .
It follows that E [ a j ] = 0 and (cid:107) a j (cid:107) =
(cid:13)(cid:13)
y j x T j − C Y X (cid:13)(cid:13) ≤ (cid:107)
y k
(cid:107) (cid:107) x k (cid:107) + (cid:107) C Y X (cid:107) ≤
[Equation]
Similarly, (cid:13)(cid:13) E [ SS T ] (cid:13)(cid:13) ≤ N [Equation] y (cid:107) C X
(cid:107)
+ (cid:107) C Y X (cid:107) 2 ) .
By applying lemma [Equation] we have (cid:15) =
Pr ( (cid:107) S (cid:107) ≥
Nc ) ≤ ( d X + d Y ) e [Equation] Lc [Equation]
[Equation]
and therefore, it implies that
[Equation]
N
Corollary 3.
For random variable X with dimensionality d X and (cid:107) X (cid:107) ≤ t x , with probability 1 − (cid:15) , it follows that
[Equation]
where L = t 2 x √ v ( C X ) + t x
Proof.
The proof is similarly to the the proof of corollary 2, deﬁne a j = (cid:80) − 1 / 2 X x j x Tj − C 1 / 2 X ,
S =
(cid:80) a j then it follows that E [ a j ] = 0 and (cid:107) a j (cid:107) ≤
[Equation]
x √ v ( C X ) + t x
[Equation]
E [ S T
[Equation]
] (cid:13)(cid:13) =
[Equation]
E [ SS T ] [Equation] [Equation] 2 ) ≤
[Equation]
[Equation]
Applying lemma 4 to get
[Equation]
it follows that
[Equation]
Lemma 5.
For two random variables X, Y , let (
cid:98) C Y X
=
C Y X + ∆ Y X , and (cid:98) C X
= C X + ∆ X where
E [∆ Y X ] and E [∆ X ] are not necessarily zero and (cid:98) C X is symmetric positive semideﬁnite.
Deﬁne
A = C Y X C − 1 X and (cid:98)
A = (cid:98)
C Y X ( (cid:98) C X + λ ) − 1 .
Then it follows that:
[Equation]
Proof.
[Equation]
It follows that
[Equation]
For M 1 , by using facts U − 1 − V − 1 =
U − 1 ( V − U )
V − 1 and C Y X
=
C 1 / 2 Y PC 1 / 2 X , where P is a correlation matrix with (cid:107) P (cid:107) ≤ 1 ,
[Equation]
Corollary 4.
Let { ( x k , y k ) }
Nk =1 be i.i.d samples from two random variables
X, Y with dimensions d X and d Y and (uncentered) covariances C X and C Y .
Assume (cid:107) X (cid:107) ≤ t x and (cid:107) Y (cid:107) ≤ t y .
Deﬁne
A = C Y X C − 1 X and (cid:98)
A = (cid:98)
C Y X ( (cid:98) C X + λ ) − 1 .
For any (cid:15) ∈ (0 , 1) such that N > t 2 x log(2 d X /(cid:15) ) v ( C X ) the following holds with probability at least 1 − 3 (cid:15)
[Equation]
where
[Equation]
Proof.
It follows by applying Corollaries 1,2,3 to Lemma 5.
By union bound, each condition has probability 1 − (cid:15) , so the total events are bounded
[Equation]
Lemma 6.
For two random variables X, Y , let (
cid:98) C Y X
=
C Y X + ∆ Y X , and (cid:98) C X
= C X + ∆ X where
E [∆ Y X ] and E [∆ X ] are not necessarily zero and (cid:98) C X is symmetric but not positive semideﬁnite.
Deﬁne A = C Y X C − 1 X and (cid:98)
A = (cid:98) C Y X (cid:98) C X ( (cid:98) C
2 X + λ ) − 1 .
Then it follows that:
[Equation]
Proof.
[Equation]
For M 1 , by using facts U − 1 − V − 1 =
U − 1 ( V − U )
V − 1 and C Y X
=
C 1 / 2 Y PC 1 / 2 X , where P is a correlation matrix with (cid:107) P (cid:107) ≤ 1 , it follows that
[Equation]
Therefore,
[Equation]
Corollary 5.
Let { ( x k , y k ) }
Nk =1 be i.i.d samples from two random variables
X, Y with dimensions d X and d Y and (uncentered) covariances C X and C Y .
The E [∆ Y X ] and E [∆ X ] is not necessarily zero and C x is symmetric but not necessarily positive semideﬁnite.
Assume (cid:107) X (cid:107) ≤ t x and (cid:107) Y (cid:107) ≤ t y .
Deﬁne A = C Y X C − 1 X and (cid:98)
A = (cid:98) C Y X (cid:98) C X ( (cid:98) C
2 X + λ ) − 1 .
For any (cid:15) ∈ (0 , 1) such that N > t 2 x log(2 d X /(cid:15) ) v ( C X ) the following holds with probability at least 1 − 3 (cid:15)
[Equation]
≤
[Equation]
(cid:107) + λ v ( C X )(1 − γ ) + λ + (cid:107) C Y X
(cid:107) (cid:107) ∆ X (cid:107) + (cid:107) ∆ Y X (cid:107) (cid:107)
C X
(cid:107) + (cid:107) ∆ Y X
(cid:107) (cid:107) ∆ X (cid:107) v ( C X ) 2 (1 − γ )
+ λ where γ = t 2 x log(2 d X /(cid:15) ) v ( C X ) N
Proof.
It follows by applying Corollaries 1,2,3 to Lemma 6.
Also by union bound, so the total events are bounded 1 − 3 (cid:15)
Theorem 2.
Grabner 1997 (3.3) (Grabner & Prodinger, 1997) Consider there are n independent copies X 1 , . . .
X n i.i.d negative binomial random variables, with parameters deﬁned as NB ( b, p ) , and our goal is to calculate the expectation of the maximum of these N random variables E n = E { max( X 1 , . . . , X n ) } then we have following asymptotic solution
[Equation]
(where F is a periodic C ∞ − function of period 1 and mean value 0 whose Fourier-coefﬁcients are given by ˆ F ( k ) = − [Equation]
[Equation]
and q
[Equation]
1 − p )
We omit the proof, interested readers could go to (Grabner & Prodinger, 1997) for details.
E . P ROOF OF T HEOREM
We ﬁrst prove the bound for (cid:13) (cid:13)(cid:13) (cid:98) Q i,j − Q i,j (cid:13) (cid:13)(cid:13) .
Proposition 2.
Let π Θ be a data collection policy and H is the range of π Θ on joint histories.
If
Equation 2 used, then for all h ∈ H and any (cid:15) ∈ (0 , 1) , (cid:13)(cid:13)(cid:13) (cid:98) Q i,j ( ψ h ) − Q
i,j ( ψ h ) (cid:13)(cid:13)(cid:13) is bounded as below with probability at least 1 − 3 (cid:15) .
[Equation]
where ∆ 1 follows the bound 38 and ∆ 2 follows the bound 36, and γ = t 2 Aj log(2 d Aj /(cid:15) ) v
[Equation]
C
[Equation]
Proof.
Let T i,j is the tensor such that C ψ oi ψ aj | ψ hi,j = T i,j × h ψ hi,j , and U i,j is the tensor such that C ψ aj | ψ hi,j = U i,j × h ψ
hi,j and for simplicity without loss meaning, we use C ψ aj | ψ hi,j to denote
C ψ aj ψ aj | ψ hi,j .
Then we have
[Equation]
We ﬁnish the above proof by prooﬁng the (cid:107) T i,j − T i,j (cid:107) and (cid:107)
U
i,j − U i,j (cid:107) are bounded by using
Corollary 4.
[Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Equation]
where
[Equation]
and
[Equation]
where
[Equation]
Then using the equation 2 and corollary 5 to obtain the bound for Q i,j
[Table] where ∆ 1 follows the bound 38 and ∆ 2 follows the bound 36, and γ = t 2 Aj log
[Table]
Aj /
[Table]
C [Table]
[Equation]
Now we start to prove our theorem 1.
Proof.
The equation 1 says:
[Equation]
here we assume the agents are homogeneous, in other words, each pair of { ( o i , a i ) } ni =1 coming from the same spaces O , A .
They are permutation invariant and their identities do not matter.
Thus, the bound of Q i,j is invariant to agents.
Under the assumption of static fully complete graph, for any agent Q i = (cid:80) nj =1 Q i,j , thus
[Equation]
E. P ROOF OF L EMMA
Here we prove Lemma 1.
Proof.
Equation 4 says
[Equation]
where I i,j is an indicator function to denote if two agents are connected.
Under the assumption that the static non-complete graph has maximum of number of degrees k and the agents are homogeneous, we have (cid:80)
j I i,j < = k .
Thus, (cid:13)
[Equation]
E. I NFORMAL P ROOF OF L EMMA
Here we make some intuitions for the proof of Lemma 2.
As we already give the proof sketch in our main paper.
Lemma 2 is a direct application of Theorem 2.
Here our random variable
J 1 , · · · , J n ∼ NB ( r, p ) , where { ( J i ) } ni =1 represents the number of time points node i needs before it meets node j number of r times.
For the complete static graph, we need at least N sample for the bound in equation 3 to be valid; in other words, we need the trajectory to run at least N time points to collect enough data to estimate our conditional operator accurately Q i,j .
For the dynamic graph, each time t , the two nodes are randomly connected with probability p , if it connects, then we can obtain a valid sample to estimate Q i,j ; if not, then we skip to the next time step.
For the dynamic graph node i , if we take the union set of the nodes i connected over the trajectory path, then the union set could form a static complete graph.
We say the two graphs are equivalent.
The number of time points needed by i until the N th connection with j for each of the pairs ( i, j ) follows the same distribution J ∼ NB ( N, p ) .
Then we are interested in the expectation of the maximum of J 1 , . .
, J n − 1 , we denote it as J { 1 ,...,n − 1 } .
E { J { 1 ,...,n − 1 } } means on average, how many time points ( N (cid:48) )
we need for all nodes other than i at least meets N times with i .
Obviously, this N (cid:48) >
= N since p ∈ [0 , 1] .
The calculation of this expectation is solved by (Grabner & Prodinger, 1997).
And we also put their result in Theorem 2.