G RAPH -A SSISTED P REDICTIVE S TATE R EPRESENTATIONS FOR M ULTI -A GENT P ARTIALLY O BSERVABLE S YSTEMS
A BSTRACT
We study reinforcement learning for partially observable multi-agent systems where each agent only has access to its own observation and reward and aims to maximize its cumulative rewards.
To handle partial observations, we propose graph-assisted predictive state representations (GAPSR), a scalable multi-agent representation learning framework that leverages the agent connectivity graphs to aggregate local representations computed by each agent.
In addition, our representations are readily able to incorporate dynamic interaction graphs and kernel space embeddings of the predictive states, and thus have strong flexibility and representation power.
Based on GAPSR, we propose an end-to-end MARL algorithm that simultaneously infers the predictive representations and uses the representations as the input of a policy optimization algorithm.
Empirically, we demonstrate the efficacy of the proposed algorithm provided on both a MAMuJoCo robotic learning experiment and a multi-agent particle learning environment.
I NTRODUCTION
In partially observable decision-making systems, it is pivotal for the agents to infer the latent state of the system from past observations.
The predictive state representation (PSR) (Littman et al., 2001) is a representation of the history by a vector of predictions of set of tests conditioning on the history.
In particular, a history is a sequence of part observations and actions.
A test is sequences of future actions and future observations, which is true if and only if all the observations occur, given that all the actions taken.
Each entry of the predictive state of representation of is given by the conditional probability that the corresponding test holds true given that history.
A fundamental assumption of PSR model is that these representations of the history are a sufficient statistics for predicting future observations and thus are able to represent the underlying state which is unobserved (Littman et al., 2001;
Hefny et al., 2015; Sun et al., 2016).
A key feature of PSR is that it only involves observable actions and observations and also encodes the latent state.
Thus, the system dynamics can be recovered from data without resorting to the estimation of the unobserved latent states.
Furthermore, using kernel embeddings of conditional distributions, PSR can be embedded into reproducing kernel Hilbert spaces (RKHS) (Boots et al., 2013), which further enhances the representation powerful of PSR.
Furthermore, Littman et al.
; James & Singh (2004); Singh et al.
(2012) shows that PSR offers an alternatively more compact state representation than POMDP models.
Therefore, PSR is considered as a powerful representation learning tool and has been used as an indispensable component of various model-based single-agent reinforcement learning methods (James & Singh, 2004; Boots et al., 2011; 2013; Hamilton et al., 2014; Sun et al., 2016; Hefny et al., 2015; 2018a).
Furthermore, when it comes to multi-agent systems with partial observations, is desirable to construct predictive states for multi-agent reinforcement learning (MARL).
However, directly extending singleagent PSR would require building tests based on joint observations and joint actions.
As a result, learning the PSR involves tensor decomposition which is computationally prohibitive (Chen et al., 2020).
Moreover, another drawback of such an approach is that oftentimes the agents only have local observations.
Furthermore, in multi-agent systems, the observations of each agent are influenced by the actions of other agents.
As a result, the representations should also take the other agents into account.
Specifically, we aim to address the following question:
Can we construct multi-agent predictive representations based on local information for reinforcement learning?
In this work, we provide a positive answer to this question.
Specifically, we propose a new framework of multi-agent PSR named as Graph-Assisted Predictive State Representations ( GAPSR ), which utilizes local information possessed by each agent to construct scalable representations.
Specifically, we assume each agent has local observations but is able to observe joint actions.
Let n be the number of agents.
For any agent i , we first construct n primitive interaction predictions { q i,j } j ∈ [ n ] via single-agent PSR, where q i,j is constructed using agent i ’s local observations and agent j ’s local actions.
Then agent i ’s final predictive representations
q
i is obtained by aggregating { q i,j } j ∈ [ n ] according to the underlying agent connectivity graph which characterize the proximity among agents.
Finally, these graph assisted representations are used as the proxy of state variable and fed into a standard MARL algorithm, e.g., multi-agent actor-critic for decision making.
Our GAPSR framework enjoys a few desired properties.
First, GAPSR aggregates the information about other agents via the agent connectivity graph, which encodes the interaction among agents.
Such an aggregation mechanism enables us to implement GAPSR on a single-agent level while maintaining other agents’ interactions through the encoding of the interactive graph topology.
Second, the GAPSR of each agent involve the actions of all the other agents connected to that agent.
In other words, GAPSR correctly captures the non-stationarity caused by the other agents’ actions.
Besides, the agent connectivity graphs can take various forms.
In particular, we build our model under three common graphs, namely static complete graphs, static non-complete graphs, and dynamic graphs, covering as many real-world scenarios as possible.
Third, utilizing kernel embedding of conditional distributions, the predictive representations lie in RKHS with strong representation power.
Finally, GAPSR are readily to be incorporated into any state-of-the-art MARL algorithm as a proxy of the state variable in an end-to-end fashion.
Thus, we believe that GAPSR could be a promising representation learning framework for multi-agent partially observable systems.
Finally, we implement an instantiation of the proposed algorithm that combines GAPSR with multiagent actor-critic, where we simultaneously learn GAPSR and the agents’ policies in an end-to-end manner with fully differentiable neural network structures.
We test our algorithm through systematic numerical experiments on MAMuJoCo robotic learning experiments (de Witt et al., 2020) and multiagent particle learning environments (Ryu et al., 2020), and compare our proposed method against two baselines as detailed in Section 6.
The results demonstrate the efficiency of our method.
R ELATED W ORKS
Partially Observable Environment.
Real-world agents often experience situations that the observed signals are aliased and do not fully determine their state in the system.
This is particularly true for multiple agents environments where agents have partial observability due to limited communication (Oliehoek & Amato, 2016).
In accommodation to the partially observable environment, POMDP has been adopted by Kaelbling et al.
(1998), and the algorithms (Kaelbling et al., 1998; Cassandra, 1998;
Thrun, 1999; Pineau et al., 2003; Poupart & Vlassis, 2008; Platt Jr et al., 2010) for determining an optimal policy have shifted to using the probability distributions (belief state) over the state space instead of exact state space.
In general, they have high complexity or suffer from local optima.
Moreover, the most common POMDP policy learning assumes the agent has access to a priori knowledge of the system.
The access to such prior knowledge has a premise that the agent has considerable domain knowledge (Kaelbling et al., 1998).
However, it is expected that the real-world agents learn the system model and thus a planning policy further without knowledge of the domain.
Overview of PSR.
Littman et al.
; James & Singh (2004) introduced the PSR over an expressive and robust framework for modeling dynamical systems and defined PSR as a representation of state by using a vector of predictions of fully observable quantities (tests) conditioned on past events (histories).
A predictive model is constructed directly from execution traces in the PSR framework, utilizing minimal prior information about the domain.
The PSR paradigm subsumes POMDP as a special case (Littman et al., 2001).
PSR is considered much more compact than POMDP (Aberdeen et al., 2007).
The spectral learning method has been proved to show success for learning the PSR (Boots et al., 2011; Jiang et al., 2018).
There are other classes of dynamical system learning algorithms that are based on likelihood-based optimization or sampling approaches (Frigola et al., 2013), but they are prone to poor local optima.
The spectral learning represents the estimated state by sufficient statistics of future observations and estimates the model parameters by method of moments.
However, this line of algorithms is hard to incorporate prior information (Hefny et al., 2015).
Thus, Hefny et al.
(2015; 2018a) introduce the supervised learning method to learn PSR and proves its convergence.
Although many works study PSR in discrete action space (Hsu et al., 2012; Siddiqi et al., 2010;
Boots et al., 2011), Boots et al.
(2013) propose Hilbert space embedding (HSE)-PSR to deal with continuous actions.
Hefny et al.
(2018a) use an approximation of HSE-PSR by Random Fourier transform (RFF) and built a more principled generalization of PSR to deal with high dimensions.
However, all of these studies aim for the single agent scenario.
PSR and RL.
The predictive states estimated by the PSR are considered as states in a fully observable
Markov Decision Process so that the value function is learned on these states.
This line of work has been done in the single-agent environment (Boots & Gordon, 2010; Boots et al., 2011; Hamilton et al., 2014; Venkatraman et al., 2017; Hefny et al., 2018b).
Especially, Hefny et al.
(2018b) propose the recurrent predictive state in the RNN network.
Moreover, the learning PSR and policy functions are connected with the end-to-end training.
MAPSR and MARL.
A close related work Chen et al.
(2020), which proposes a multi-agent PSR model that is formulated as a n -way tensor, where n is the number of agents.
As a result, learning the parameters is achieved by decomposing the n -way tensor, which becomes computationally prohibitive when the number of agents is large.
Moreover, it seems challenging to incorporate benign structures such as sparsity into the model, as doing so leads to a more complicated set of moment equations.
Furthermore, our work belongs to the vast literature on MARL for (partially observable)
Markov
Games.
See, e.g., Lyu & Amato (2020); Son et al.
; Zhang et al.
; Rashid et al.
Foerster et al.
; Lowe et al.
; Baker et al.
Yu et al.
(2021) and the references therein.
Also see Zhang et al.
(2021); Gronauer & Diepold (2022); Canese et al.
(2021) for recent surveys on MARL.
Recent works propose more sophisticated deep MARL algorithms for multi-agent problems under the paradigm of centralized training with decentralized execution (Zhou et al., 2020;
Sunehag et al., 2017; Lowe et al., 2017; Foerster et al., 2018; Rashid et al., 2018).
Our work seems not directly comparable to these works as our goal is to learn good representations in the partially observable multi-agent systems via model-based RL, whereas most of these works adopt model-free methods or focus on Markov games without partial observations.
B ACKGROUND : S INGLE -A GENT PSR
In this section, we introduce the background on single-agent PSR.
Also see §A.1 for a more detailed introduction.
Predictive State Representation (PSR).
A prediction of a state is defined as the conditional probability of seeing a test’s observations in sequence given that actions of the test are taken in sequence from a history (Littman et al., 2001).
Givn a finite observation space O
( o ∈ O )
and action space A ( a ∈ A ).
A test of length k at time t , is defined as a sequence of action-observation pairs that starts at time t and ends at time t + k − 1 , { ( o l , a l ) } t + k − 1 l = t = { o t , a t , o t +1 , a t +1 , · · · , o l , a l , · · · , o t + k − 1 , a t + k − 1 } .
A history , at time t is a sequence of action-observation pairs that start from the beginning of time and ends at time t − 1 , { ( o l , a l ) } t − 1 l =1 = { o 1 , a 1 , · · · , o l , a l , · · · , o t − 1 , a t − 1 } .
Hilbert Space Embeddings of Predictive State Representations.
In this work, we are interested in extending PSR to decision makings of controlled systems with continuous actions.
So we use the model introduced by Hefny et al.
(2018a).
In this model, predictive state q t and extended predictive state p t satisfies
[Equation]
(i.e., q t and p t are conditional linear expectation operators which maps to the conditional expectation of future observations), where
[Equation]
are feature maps by kernels k O ,
k A over future observation and action features.
The extended predictive state compared to predictive state adds one more pair of { ( a t + k , o t + k ) } to the prediction.
The ξ at and ξ ot are the corresponding extended feature maps, which satisfy
[Equation]
Here ϕ
ot :=
ϕ o ( o t ) and ϕ at :=
ϕ a ( a t ) are the shorthands for one time feature map by k o and k a .
We use ⊗ to denote the transposed Khatri–Rao product for two matrices with the same number of rows, and each row of the resultant matrix is the vectorzied outer product of the corresponding row vectors in the two matrices.
Also, we use
[Equation] [Equation]
h
[Equation]
4) to define a set of features extracted from previous observations and actions (typically from a fixed length window ending at t − 1 ).
3. N OTATION OF PSR F OR M ULTI -A GENT SETTING
In the subsequent section, we will bring the HSE-PSR model to the multi-agent settings.
In the following, we introduce the notation for the multi-agent setup.
For any agent i in an n -agent system, given a set of kernels { k O i } ni =1 , { k A i } ni =1 , { k o i } ni =1 , { k a i } ni =1 for every agent, we define agent i ’s feature maps using the single-agent PSR presented in the previous section.
In particular, for all i ∈ [ n ] , we let ψ ot,i and ψ at,i be defined as in (2) using agent i ’s local observations and actions, respectively.
Similarly, we let ξ ot,i and ξ at,i be defined in (3), and let ϕ ot,i and ϕ at,i be defined using ϕ o i and ϕ a i , respectively.
Furthermore, for all i, j ∈ [ n ] , we let ψ ht,i,j be defined as in (4) using the observations of agent i and the actions of agent j .
Moreover, we use the q t,i and p t,i to denote its predictive state and extended predictive state, whose definitions will be presented in the next section.
D YNAMIC I NTERACTION G RAPH F OR GAPSR M ODEL
Many works have considered graph representation of the multi-agent network (Liu et al., 2020b; Ryu et al., 2020).
In general, the relationship between agents is characterized by an undirected graph.
We introduce a dynamic interaction graph to represent the GAPSR by considering the interaction between agents.
Definition 1.
Let G =
( V, E ) , including the set V of nodes and set E of the edges.
Each node represents the agent entry, and the edge represents the relationship between the two adjacent agents.
Here we suppose the graph structure is given, that means the number of nodes ( n ), the number of edges ( m ), the edge weights, and the maximum number of degrees ( k ) are available to us.
We think that this kind of presupposition is very reasonable because in the real world, for example, there are multiple robots; we can quickly get the geographic position of the robots through sensors, then calculate the structure of the graph formed by them.
4. S TATIC C OMPLETE G RAPH
We starts with a static complete graph G c , where the relationship between nodes are invariant to time change.
A complete graph has m = n ( n − 1) / 2 edges, where m = | E | .
For each agent, we can represent its PSR by considering other agents’ interactions by
[Equation]
same for p t,i .
To consider the interactive behavior between agents, we introduce two additional notations q t,i,j and p t,i,j .
Let { ( i, j ) } ni,j =1 represents a pair of agents on a n multi-agent system.
The q t,i,j is a primitive predictive state of i ’s observation ( ψ o t,i ) by intervening agent j ’s action ( ψ a t,j ) and observing agent i ’s observation history and agent j ’s action history ( ψ ht,i,j ), and p t,i,j is the extended counterpart, where they satisfy the relationship for single agent case in (1)
[Equation]
If i = j , then it becomes an exact single agent scenario.
Similarly, we use the same approach to represent p t,i as shown in (5).
Each agent’s PSR q t,i and extended PSR p t,i are modeled by fully considering all other available agents.
Based on equation (5), in practice, estimating q t,i and p t,i denoted as (cid:98) q t,i and (cid:98) p t,i requires us to get (cid:98) q t,i,j and (cid:98) p t,i,j at first.
4.1. E STIMATION OF q t,i,j AND p t,i,j
To estimate the p t and q t for just one agent, Hefny et al.
(2018a) use the supervised learning method.
They show that (cid:98) q t =
M ψ ot | ψ at ; ψ ht = C
ψ ot ψ at | ψ ht ( C ψ at ψ at | ϕ ht + λI ) − 1 , where M A | B ; c is a linear operator that satisfies E
[ A | B = b ; C = c ] =
M A | B ; c b , and C XY :=
E [ ϕ ( X ) ⊗ ϕ ( Y )] is the uncentered covariance operator, and C XY | z is covariance of X and Y given Z
= z .
They estimate C ψ ot ψ at | ψ ht ,
C ψ at ψ at | ϕ ht by sampling data, then get the estimation (cid:98) q t .
The same procedure is used to estimate (cid:98) p t by replacing the features ψ ot and ψ at with their extend counterparts ξ ot and ξ at .
Similar to the single-agent case above, the representation of q t,i,j can be achieved as
[Equation]
To estimate C ψ ot,i ψ at,j | ψ ht,i,j and C ψ at,j ψ at,j | ψ ht,i,j , we learn two linear maps T i,j and U i,j such that
C ψ ot,i ψ at,j | ψ ht,i,j ≈ T i,j (cid:0) ψ ht,i,j (cid:1) and C ψ at,j ψ at,j | ψ ht,i,j ≈ U i,j (cid:0) ψ ht
i,j (cid:1) .
The training examples for T i,j and
U i,j consist of pairs ( ψ ht,i,j , ψ ot,i ⊗ ψ at,j ) and ( ψ ht,i,j , ψ at,j ⊗ ψ at,j ).
The learning of (cid:98) p t,i,j can be done in a similar way.
More details can be found in §A.2.
After calculation of (cid:98) q t,i,j and (cid:98) p t,i,j , under the static complete graph setting where the interaction is considered, defined in equation (5), we can get the estimate of (cid:98) q t,i and (cid:98) p t,i .
4.1. T HEORETICAL G UARANTEE OF E STIMATION OF q i AND p i U NDER THE S TATIC C OMPLETE G RAPH
Theoretically, we show that the difference between q i and its estimator (cid:98) q i is bounded with high probability.
Theorem 1.
Let π Θ be a data collection policy and H is the range of π Θ on joint histories.
If
Equation (5) and (7) used, then for all h ∈ H and any ϵ ∈ (0 , 1) , such that N > t
[Equation]
where N is the number of time points we collect sample, then ∥ (cid:98) q i − q i ∥ is bounded as below with probability at least 1 − 3 ϵ ,
[Equation]
where
[Equation]
Here ∆ 1 , ∆ 2 are two other relevant bounds, we provide them in §E.1.
u ( · ) , v ( · ) denote the largest, smallest eigenvalue of a matrix.
And γ = t 2 Aj log(2 d Aj /ϵ ) v (cid:16) C ψaj (cid:17) N < 1 is a constant that depends on the magnitude of the norm of ψ aj (we assume (cid:13)(cid:13) ψ aj (cid:13)(cid:13) ≤ t A j ), the dimension of ψ aj ( d A j ), the (uncentered) covariances ( C ψ aj := E (cid:2)
ψ aj ψ aTj (cid:3) ) and the sample size ( N ).
Theorem 1 says we need at least N samples for the bound in equation (8) to be valid.
We give the proofs in §E. It is not hard to obtain a bound for p i using the same approach.
We omit that.
4. E XTENSION TO S TATIC N ON -C OMPLETE G RAPH
In large-scale multi-agent systems, the number of agents is large, and not all agents need to interact with each other.
A static non-completed graph can perfectly represent such a situation.
For example, in a given static non-complete graph G s , we know its maximum number of degree k , and we use the binary n × n matrix with each entry as I i,j to indicate the interaction between two agents.
Then the
GAPSR for each agent will be
[Equation]
Lemma 1.
Under the same environment depicted in Theorem 1 and given a G s with k maximum number of the degree to represent agents, then the bound in Theorem 1 can be rewritten as ∥ (cid:98) q i − q i ∥ ≤ k ∆ .
The conclusion of Lemma 1 is evident since we replace the n with the k neighbors, the total error bound is also decreased approximately as kn .
4. D YNAMIC G RAPH
Real-world multi-agents can also formulate a time-dependent dynamic graph G d , rather than a static graph.
A dynamic graph has its structure dynamically changing with time.
In other words, the edges can be inserted or deleted across time.
The dynamic graph brings more challenges to the representation as the interaction relationship among agents changes constantly.
Braha & Bar-Yam (2009), Ma et al.
(2017) and Zhao et al.
(2010) consider a dynamic graph as a set of ordered static graphs.
For each time point, we are given a static graph such that G d = { G d 1 , G d 2 , · · · , G dt } , and a time-dependent given binary matrix with I t,i,j indicating the interaction between two agents at each time.
Then we have the agent-wise PSR as
[Equation]
Compared to (9), the coefficient I t,i,j is time-dependent, which brings a challenge to our theoretical bound.
We consider a dynamic graph experiences a trajectory path, assuming every node has a chance at least p to interact by connection with another node at any time point.
For example, for a node i , if we take the union set of the nodes interacted with i over the path, then the union set could form a static complete graph; in other words, if i connects j , then we can obtain a valid sample to estimate q i,j as the complete static graph does, if not, then we skip to the next time point.
For the complete static graph, we need the trajectory to run at least N time points to collect enough data to estimate our conditional operator q i,j accurately.
Furthermore, The total number of time points needed by i until the N th interaction with j follows a negative binomial distribution NB ( N, p ) .
On average, we need Np time points before we see i, j completely connecting N times.
Now we consider node i could interact with every node in a set of nodes ( J : | J | = n − 1 ) for N number of time points.
We assume the chances being interacted between two nodes ( i, j ∈ J ) does not affect their interaction with other nodes.
Thus we have a set of independent negative binomial random variables { J l } n − 1 l =1 ∼ NB ( N, p ) to characterize the interaction of i with j ∈ J .
So we are interested in the expectation of the maximum of J 1 , . .
, J n − 1 , a statistics that tells us the expected maximum number of time points of collection of measurements needed for the node i to be able to connect with every node j ∈ J for at least N number of time points.
We denote it as J { 1 ,...,n − 1 } and we have
E {
[Equation] [Equation]
Lemma 2.
Under the same environment depicted in 1 and given a dynamic graph G d with every node has a chance at least p to interact with another node in a one-time point, let N be the number of time points we collect data of measurements in order to get the bound in Theorem 1, if Equationand 7 used, then we need at least
[Equation] [Equation]
total number of time points, where q = 1 − p , K ( q, n, N ) :
= log 1 /q ( n − 1)
+ ( N − 1) log 1 /q (cid:104) log 1 /q ( n − 1) (cid:105)
+ ( N − 1) log 1 /q
p − log 1 /q ( N − 1)!
, F is a periodic C ∞ -function of period 1 and mean value 0 whose Fourier-coefficients are given by ˆ F ( k )
= −
[Equation]
for k ∈ Z \ { 0 } .
Then ∥ (cid:98) q i − q i ∥ achieves the same bound as in Theorem 1 with probability at least 1 − 3 ϵ .
In other words, ∥ (cid:98) q i − q i ∥ ≤ n ∆ .
Moreover, (12) is an asymptotic expansion of the right-hand of (11) .
We give the proof in §E.3.
Lemma 2 gives the worst-case bound for our estimation under the dynamic graph.
The result tells us that if we need 1 more sample of measurement for our algorithm to converge on the complete static graph, we need roughly log 1 /q [log 1 /q ( n − 1)] more samples on the dynamic graph.
So as long as we allow enough learning time, the algorithm can converge with high probability.
Complexity with Increasing Agents .
For the complete static graph, we need to evaluate (cid:98) q i,j and (cid:98) p i,j every time point, which requires O ( n 2 ) operations and space.
Overall, with an increased number of agents, our GAPSR has a polynomial O ( n 2 ) scaled complexity, which is feasible for learning in a large number of agents environment and is more efficient compared to the centralized MAPSR (Chen et al., 2020) theoretically, which is combinatorially sample complex, the analysis is shown in §A.4.
For a non-complete static graph with k maximum number of degrees for k ≪ n , which is more common in the real world, because a very far-away robot will not likely affect the targeted robot, the operation will be significantly decreased to O ( k 2 ) .
4. T HE E STIMATION FOR GAPSR M ODEL C OMPONENTS
As stated by Littman et al.
(2001), a complete PSR model can build a recursive rule to update itself.
In other words, given the q t,i,j , the GAPSR model can calculate q t +1 ,i,j by incorporating a new observation, a so called filtering process in the dynamical system.
We achieve this with three models proposed by Hefny et al.
(2018a):
For any agent i , p t,i,j = W i,j ( q t,i,j ) , q t +1 ,i,j = F i,j ( p t,i,j , o t,i , a t,j ) , and o t,i =
Z i,j ( q t,i,j , a t,j ) .
The last model is used to predict the observation o t,i with action a t,j and PSR q t,i,j .
Typically, W i,j , Z i,j are learnable linear maps, and F i,j is non-linear and differentiable but known in advance.
The W i,j , Z i,j are learned from regression after we estimate q t,i,j and p t,i,j , see §A.2.2 for details of estimation of W i,j , Z i,j .
D ECISION -M AKING F RAMEWORK W ITH GAPSR M ODEL
Previous work on single-agent proposed an end-to-end training algorithm
(Hefny et al., 2018a) for
PSR model and policy learning.
Here we design an algorithm for multi-agent settings and incorporate our GAPSR model containing the interactive graph component.
We propose an online learning algorithm to learn the GAPSR and agent policies simultaneously.
Our algorithm is shown in §B.2.
We use a diagram in §B.3 to illustrate this process.
Our algorithm has three components.
Firstly, we estimate the GAPSR the model parameters W i,j and Z i,j (line 4-11).
The agents use an existed T length trajectory generated based on the iteration k − 1 ’s policy (a random policy is used if it is the first time), and learn the GAPSR parameters under the given interactive graph by regressions with details introduced in 4.4.
Secondly, we use the just learned GAPSR to generate the predictive state representations q and p (line 13-28).
In particular, by starting from the initialized GAPSR and executing the iteration k − 1 ’s policy, the agents experience a T length trajectory.
The agents perform a series of extension, filtering, and prediction steps to generate the q t and p t .
Every agent also obtains action by executing its policy function that maps predictive states q t,i to action a t,i ∼ π k −
i ( q t,i ) .
Agents then save trajectories (actions, observations, predictive states, and rewards) over the path.
Lastly, learning and planning via multi-agent actor-critic (line 30-34): agents use learned representations as the state and feed them into a MARL algorithm, e.g., MADDPG (Lowe et al., 2017).
The MARL algorithm conducts the policy learning.
Here we give two examples of
GAPSR based MARL algorithms, which are implemented for our experiments.
They follow the actor-critic framework.
We first develop an algorithm under partially observable environments, where each agent has its own critic and actor independently, it is analogous to the independent actor-critic (IAC)
(Foerster et al., 2018).
The gradient of the policy is written as ∇ θ
i J
( θ i ) =
E τ i ∼ p ( τ i | Θ i ) (cid:2)(cid:80)
Tt =1 ∇ θ
i log π θ i ( a t,i | q t,i )
( r i + γV π i w i ( q t +1 ,i ) − V π i w i ( q t,i ))
(cid:3) , and the independent critic is updated by minimizing the loss L c =
E τ i ∼ p ( τ i | Θ i )
[( V π i w i ( q i ) − y i ) 2 ] .
Here the predictive states q i generated by the GAPSR are considered as states to fit the value function.
π θ : Q × A → [0 , 1] is the stochastic policy that maps to the probability density of actions with parameter θ .
We call it GAPSR-1 .
As we know, IAC would not work well since the environment is not stationary under a multi-agent setting, the MADDPG by Lowe et al.
(2017) solves a non-stationary environment by considering other agents actions; however, every agent needs a separate critic that has the global information.
We used a centralized critic to minimize the loss L c = E q ,r,o (cid:2) ( Q π w ( q 1 , · · · , q n , a 1 , · · · , a n ) − y ) 2 (cid:3) .
So our policy gradient is written as ∇ θ
i J
( θ i ) =
E q ,r,o (cid:2) ∇ θ i π
θ i ( q i ) ∇ a i Q πw ( q 1 , · · · , q n , a 1 , · · · , a n ) | a i = π θi ( q i ) (cid:3) .
To fully consider other agents’ information, it uses the joint predictive states as input.
Here by abuse of notation, we use π θ : Q → A to indicate the deterministic policy with parameter θ .
We call it GAPSR-2 .
We put our integration details in §B.4.
Our algorithm has the following characteristics.
First, due to its decoupled structure, it is a general algorithm in the sense that planning can utilize any MARL algorithm for multi-agent MDP.
Second, our algorithm is an end-to-end framework, in the implementation, we build an additive loss function and fully differentiate it with respect to model parameters.
In particular, we update the
MARL and GAPSR model parameter Θ = { Θ GAPSR , Θ MARL } (line 32)
by minimizing the following additive objective function:
[Equation] [Equation]
[Equation]
Here the L 1 (Θ MARL ) is the objective function for MARL, for example, if the actor-critic used, it will be the negative action value function with policy parameter θ plus the critic loss with parameter w , and Θ MARL =
( θ, w ) .
L 2 (Θ GAPSR , Θ MARL ) is the MSE between prediction and actual observation.
And p ( τ | Θ) is the distribution over trajectories induced by the policy and GAPSR.
Θ MAPSR =
{ W i,j , Z i,j } denotes GAPSR’s parameters.
The α 1 and α 2 are hyper-parameters to penalize differently on two losses.
Third, PSR, as a function of action, can bring agents’ action information into the policy gradient; our interactive GAPSR even brings the effect of other agents’ actions to the policy gradient as well
[Figure]
E XPERIMENTS
[Figure]
A v e r a g e r e w a r d
Environments.
We evaluate the performance of our GAPSR on a collection of MARL tasks under some OpenAI Gym MAMuJoco environments (de Witt et al., 2020), such as multi-agent swimmer, hopper, and ant.
Each robotic agent is represented as a body graph, where vertices (joints) are connected by adjacent edges (body segments) as shown in Appendix Figure 5.
Each agent controls its joints based on the local information observed.
All tasks are learned under the partially observable environments by manually hiding some observations for each agent.
The goals of the multi-agent systems are aligned with their corresponding single-agent ones.
However, different from the single-agent system, the agents in the multi-agent system need to collaborate to reach their goals.
For the interactive graph, for simplicity, we considered a complete static graph, where we assume every agent is connected with all other agents in the graph.
We defer the details of the robotic agents to §C.1 and the details of the experimental setup to §C.2.
Baselines and Evaluation.
We run 50 iterations for each experiment and collect M = 100 trajectories in every iteration with a maximum of 1000 steps in every trajectory.
After each iteration, we compute the average return
R = 1 M (cid:80) ni
=1 (cid:80)
Mb =1 (cid:80) T b t =1 r ti,b on a batch of
M trajectories, where, T b is the length of the b th trajectory.
We repeat this process using ten different random seeds and report the average and a standard deviation.
To verify the effectiveness of interactive graph, we introduce a baseline called Independent PSR learning (IPSR); in this model, we do not consider the graph, so we formulate n independent single PSR without considering their interactions, which means there are no q i,j any more but only q i .
The architecture of GAPSR-1 and
GAPSR-2 remains the same.
To verify the advantage of PSR, we introduce another baseline (MARL) where we take out the GAPSR entirely, so it matches with the MARL run on a partially observable environment.
For GAPSR-1, its MARL baseline is IAC (Foerster et al., 2018), and for GAPSR-2, is
MADDPG (Lowe et al., 2017).
Results And Discussion Figure 1 illustrates the empirical average return vs. the number of interactions with the environment measured in time steps.
Our GAPSR methods consistently outperform IPSR and get the highest rewards under partially observable environments, which justifies the representation power of the interactive graph to assist agents in learning in the non-stationary environment with limited observation when the MARL algorithm has a defect (IAC).
Moreover, it can also boost the performance of the existing good MARL algorithm (MADDPG).
To further verify the effect of learning the PSR part, we also plotted the predicted trajectory to verify the GAPSR’s performance for predicting the observations in Figure 2.
We plotted the predicted observations vs. actual observations in iterations 1 , and 40 , respectively, for GAPSR-2.
We plotted a row ×
[Figure] [Figure]
To enrich our algorithm environment, we also test our algorithm (GAPSR-2) in multi-agent particle environments, using the benchmark by (Lowe et al., 2017), please check §D.2 for experiment details.
We test our algorithm for large n cases.
In this environment, the agents can have cooperative goals such that all agents must maximize a shared return and conflicting competitive goals.
We set up the environments where agents can only perform physical actions but not communication; however, to achieve the goals, agents need explicit communication about others’ locations to achieve the best reward.
These partially observable environments give us the motivation to test our method.
We report the rewards in Figure 3.
We see that GAPSR outperforms IPSR and baseline, in terms of the convergence speed and final attained rewards, with a different number of agents.
The predictive states convey the information that can help communication between agents in limited communication and observation environments.
C ONCLUSION
We propose a GAPSR model, extending ideas from single-agent predictive state representations to a multi-agent scenario, during the process, we introduce the dynamic interactive graph to model agents’ interactions.
Furthermore, we provide the theoretical guarantees of the GAPSR model.
Finally, a learning algorithm that supports gradientbased deep MARL methods is developed.
Our method provides a model-based MARL framework under a partially observable environment.
The experiments proved that our model assumption is valid by observing the highest return while reducing the observations’ prediction error over trajectories.
A M ETHOD D ETAILS A. M ATHEMATICAL M ODEL OF PSR
Here, we give the mathematical model for PSR.
As in Section 3, we let τ t := { ( o l , a l ) } t + k − 1 l = t as a test (a sequence of action-observation pairs) at time t , then we have a subset τ
ot :=
{ o l } t + k − 1 l = t for observations, and τ at := { a l } t + k − 1 l = t for actions.
And we let h t := { ( o l , a l ) } t − 1 l =1 as the history at time t .
A test is executed at time t if we intervene to select the sequence of actions specified by the test.
It is said to succeed at time t if it is executed and the sequence of observations in the test matches the observations by the system (Boots et al., 2013).
The prediction of a length- k test τ t is defined as the probability distribution that the test succeed at time t , given history h t : Pr (
[Equation]
ot
[Equation]
Let T = { τ z } , z ∈ N is a finite set of core tests.
And we use T o = { τ o 1 , · · · , τ o | T | } and T a = { τ a 1 , · · · , τ a | T | } to indicate its observation and action parts.
A.1. L INEAR PSR
A linear PSR is the prediction vector
[Equation]
(16) that contains the probabilities of success of the tests in T , if and only if for any test τ
[Equation]
(17) where f τ : [0 , 1] | T | (cid:55)→ [0 , 1] is a linear projection function.
For simplicity of notation, we use the same notations for q t p t , W, F , which only follows the meaning defined in this section.
It means that knowing the probabilities for the tests in T is sufficient for computing the probabilities for all other tests in the system.
The prediction vector is a sufficient statistic for the system at time t , so we call it as state for the PSR at time t .
Therefore, PSR can represent state by using a vector of predictions of fully observable quantities (tests) conditioned on past events (histories).
The prediction function f τ is linear and has one-to-one relationship to a test τ such that f τ ( q t ) =
f ⊤ τ q t ∀ t .
The linear PSR can still represent systems with nonlinear dynamics.
To maintain predictions in T , we need to update the state q t .
To do that, we predict the success of any core test τ z prepended by a new action a and observation o at time t + k − 1 , which we call oaτ z .
Based on Bayes rule, we have
[Equation]
and f aoτ z , f ao ∈ R | T | are linear operators such that ( ∀ τ z ∈ T , ∀ a ∈ A , ∀ o ∈ O ) .
Then let F aoτ be the matrix with its columns as f aoτ z for all τ z ∈ T .
Then the updated state in PSR is obtained by
[Equation]
Given the initial prediction vector q 1 , the PSR can update with equation 19.
This recursive application of Bayes rule to a belief state is called the Bayes filter.
Now we have seen the extended prediction vector, so we define extended predictive state as:
[Equation]
Clearly, we can see there exists a linear extension operator W such that p t = W ( q t ) .
And given the new observation, we can have a Bayes filter F such that q t +1 =
F ( p t , o, a ) .
We also introduced these operators in section 4.4.
A.1. R EPRESENTATION OF S TATE AS C ONDITIONAL E XPECTATION OF S UFFICIENT S TATISTICS
Instead of learning the distribution, here we recover the idea of representation of states as a conditional expectation of sufficient statistics and using the supervised learning method to learn (Hefny et al.,
Let history h
ot is a sequence of observation that starts from the beginning of time and ends at time t − 1 .
We define the belief state b t =
Pr ( s t | h ot ) , where s t is the current state of the world.
b t +1 =
Pr ( s t +1 | h ot +1 ) is the next time belief state.
We call b t as "belief state", which represents the probability distribution over state space, also represents the knowledge and uncertainty about the true state of the system.
In dynamic system, the task of getting the updated b t +1 with given b t and new observation o t is called filtering.
The task of estimating the Pr ( s t +1 | h ot ) with given current b t without incorporating any new observation is called one-step prediction.
Instead maintaining a belief b t over states, spectral algorithms try to recover observable operators that can be used to perform filtering and prediction directly, by maintaining the expected value of a sufficient statistic of future observations.
Let a test τ ot is a length k sequence of observation defined before.
As the recursive Bayes rule holds, we can get
the next time prediction vector Pr ( τ ot +1 |
h ot +1 ) using the new observation o t + k and the current prediction Pr ( τ ot | h ot )
, we also define an extended prediction vector Pr ( τ ot +
| h ot ) .
These prediction vectors characterize the state of the system and they can be estimated by observable quantities.
Given Pr ( τ ot +1 | h ot ) , filtering becomes the task of getting the updated prediction vector Pr ( τ ot +1 |
h ot +1 ) , "conditioning" on o t .
One-step prediction becomes getting the Pr ( τ ot +1 | h ot ) , "marginalizing" over o t .
Therefore, the spectral algorithms avoid explicitly estimating the latent state or the initial, transition, or observation distributions.
We let q t =
E [ ν t | h ot ] , where ν t a vector of features that determines the distribution of future observations Pr ( τ ot | h ot ) .
For simplicity of notation, we use the same notations for q t , p t , W, F .
Let p t =
E [ φ t | h ot ] , where φ t is a vector of features that determines the distribution of observations Pr ( τ ot +1 | h ot ) .
We call q t is the transformed predictive state.
So q t +1 = E (cid:2)
ν t +1 |
h
ot +1 (cid:3) , is the updated predictive state, and p t is the extended predictive state.
Let h ot be the feature vector of history
h
ot .
In Hidden Markov Models and Kalman filters, the extended state p t is linearly related to the predictive state q t .
Which is p t =
W q t .
Estimation of the W can be done using linear regression with samples ν t and φ t , however, due to the overlap between observation windows, the noise terms on ν t and φ t are correlated, which will cause biased estimate.
The instrumental regression (Pearl et al., 2000;
Stock et al., 2012) is employed.
h ot is a instrumental variable that do not overlap with sequence { l } l = t + k − 1 l = t and { l } l = t + k l = t .
The correlation corr (h ot , e ( φ t ))
=
and corr (h ot , e ( ν t ))
= 0 , where e is a measure of error.
By taking the conditional expectation of p t = W q t given h ot , we have E
[Equation]
[Equation]
Based on the above relationship, we first estimate the E [ φ t | h ot ] and E [ q t | h ot ] by sample h
ot , ν t , and φ t , we then use the estimates to compute W .
So if we start with q 1 , we can compute p 1 = W q 1 , and get the q t +1 = F ( p 1 , o 1 ) , where F is the Bayes filter to update the state.
A. L EARNING FOR GAPSR IN D ETAILS
Here we introduce the details about learning GAPSR, which supplements the section 4.4 and 4.1.1.
A.2. L EARNING F OR (cid:98) q t,i,j AND (cid:98) p t,i,j
To calculate (cid:98) q t,i,j , we introduce two sets of linear operators T i,j and U i,j , such that T i,j ( ψ ht,i,j ) ≈
C ψ o t,i ψ a t,j | ψ h t,i,j and U i,j ( ψ h t,ij )
≈ C ψ at,j ψ at,j | ψ ht,i,j .
We estimate them by using two ridge regressions:
[Equation]
L represents the ridge regression loss, and R represents the regularizer.
After learning T i,j and U i,j , we can get the estimate of C ψ ot,i ψ at,j | ψ ht,i,j and C ψ at,j ψ at,j | ψ ht,i,j .
Then we get the (cid:98) q t,i,j by equation 7.
Similarly, we use extended features to obtain (cid:98) p t,i,j .
A.2. E XTENSION , F ILTERING AND P REDICTION F UNCTIONS
Filtering.
To obtain (cid:98) q t +1 ,i,j from (cid:98) q t,i,j , (cid:98) p t,i,j , we use filtering.
We denote F i,j as the filtering function.
We describe the filtering process as below.
From { o i } ni =1 and { a j } nj =1 , we obtain the embedding { ϕ ot,i } ni =1 and { ϕ at,j } nj =1 .
We then compute the observation covariance
[Equation]
We then multiply the extended state by inverse observation covariance to change predicting ϕ
ot,i into conditioning on ϕ ot,i .
[Equation]
) × here is to denote n − mode (matrix or vector) product, × ϕ
ot,i means multiplying the tensor by a matrix (or vector) in mode ϕ
ot,i .
We condition on ϕ
ot,i and ϕ at,j to obtain shifted state.
[Equation]
[Equation]
Based on the updating rule, q t +1 ,i,j = F i,j ( p t,i , o t,i , a t,j ) , and p t,i =
W i,j ( q t,i,j ) , we write the filtering equation.
[Equation]
where p t,i :=
M ψ ot +1 ,i ⊗ ϕ
ot,i | ψ at +1 ,j ,ϕ at,j ; ψ ht,i,j .
F i,j usually is known because it is obtained through the above calculation using known quantities, however, W i,j , Z i,j must have to be learned by using regressions.
Extension.
The W i,j can be learned by kernel regression if we know the (cid:98) q t,i,j and (cid:98) p t,i,j .
Previous work (Hefny et al., 2018a) demonstrated the kernel regression model for learning single-agent PSR, here we extend to the GAPSR.
We set the model parameter W i,j .
We optimized a ridge regression problem for W i,j ,
[Equation]
Prediction.
We can also get the prediction about the next one time observation o t by the regression function such as:
[Equation]
We solve the prediction regression function Z i,j by another ridge regression:
[Equation]
A. T ENSOR D ECOMPOSITION OF MAPSR
Existed Formulation.
(Chen et al., 2020) use a n + 1 multi-dimensional tensor called system dynamics tensor D ∈ R |T 1 |×···×|T n |×|H| to represent the system dynamics of the MAPSR, with |T 1 | representing the cardinality of the tests for agent 1 , n denoting the number of agents, and the |H| being the cardinality of the joint history.
Each element of the tensor is a probability of a joint test given joint histories.
Given the system dynamics tensor, finding the latent predictive state can be transferred into finding a minimal linearly independent set from the system dynamics tensor, and it can be solved by spectral method such as tensor decomposition.
As mentioned earlier, this formulation can not satisfy our needs.
In Proposition 1 of Appendix A.4, we also give the sample complexity analysis such that the sample size needed to formulate D scales exponentially with the length of tests and number of agents.
Here we give a summary of tensor decomposition of their method.
Given a system dynamic tensor D such that:
[Equation]
Here ◦ is the outer product.
D 1 , · · · , D n , F are matrices.
The factor matrices D 1 , D n , F consist of the vectors, i.e., D 1 =
[ D 1:1 D 1:2 · · · D 1: R ] ∈ R |T 1 |× R .
A colon is used to indicate all elements of a mode, thus, the R th column of D 1 is denoted by D 1: R .
For any i 1 ∈ { 1 , . . .
, |T 1 |} , i n ∈ { 1 , . . . , |T n |} , and k ∈ { 1 , . .
. , |H|} , after the decomposition, we get
[Equation]
where { i 1 , · · · , i n , k } are index corresponding to the specific dimension of the D .
The last dimension of the tensor D is compressed in a matrix F , and its row vector x k =
[ x k (1) . . .
x k ( R )]
∈ R 1 × R , k ∈ { 1 , . .
. , |H|} is a summary of joint history and can be considered as a compressed version of the system
predictive state vector p
( Q | h k ) , the joint history h k ∈ H ( k ∈ [1 , |H| ]) at time step s = | h k | , where Q is the core joint test set.
The whole fibers listed in the set Q form a basis of the space spanned by the mode-(n+1) fibers of tensor D .
Thus, by constructing the vector m = ( λ ∗ D 1 i 1 : ∗ · · · D ni n : ) T , where ∗ is Hadamard product.
D 1 i 1 : denotes the i 1 -th row vector of D 1 .
Then we could rewrite the previous equation as
[Equation]
) m ( r ) is a scalar that m ( r ) =
λ r D 1 i 1 r · · · D ni n r .
And x k is the system state vector and m is the prediction parameter, both of them are obtained by the tensor decomposition.
A. A NALYSIS OF S AMPLE C OMPLEXITY FOR F ORMULATING THE S YSTEM D YNAMIC T ENSOR
The paper (Chen et al., 2020) does not analyze the sample complexity to construct D , which is a n + 1 multi-dimensional tensor D ∈ R |T 1 |×···×|T n |×|H| .
We give this analysis.
The D i 1 , ··· ,i n ,k :=
Pr ( t 1 i 1 , · · · , t ni n | h k ) is an element of that tensor D such that t 1 i 1 is the i 1 -th test of agent 1, similarly, the t ni n is the i n -th test of agent n , and h k is the joint history.
Proposition 1.
In a n -agents system, assume every agent has the same observation and action space |O| , |A| , for a length- k test, to formulate a complete system dynamics tensor D defined in equation 32.
Assume each entry of the tensor needs S samples to give a sufficient estimation using Monte-Carlo roll-out method, then the total sample size is at least ( |O||A| ) kn S , if the agents are homogeneous, in other words, they are permutation invariant such that identity does not matter, then the total sample size is (cid:0) |O||A| k + n − 1 n (cid:1) S .
Proof.
At one time step, for any agent, it has |O||A| different combinations for the joint test, then for a length of k tests, it follows that ( |O||A| ) k number of different choices.
Then the tensor D would need ( |O||A| ) kn elements to cover all the possible length k tests.
So the total sample size is ( |O||A| ) kn S .
If the agents are homogeneous, then the ordering does not matter, for each agent, we have ( |O||A| )
k number of different choices for length k test, so the total choices for n agents are (cid:0) |O||A| k + n − 1 n (cid:1) , then we need (cid:0) |O||A| k + n − 1 n (cid:1) S samples.
The sample complexity is exponentially scaled with the number of agents and length of tests.
B A LGORITHM AND I NTEGRATING WITH MARL METHOD
We first introduce some backgrounds on two multi-agent frameworks: Multi-agent Markov Decision
Process (MMDP) and Multi-agent Partially observable Markov Decision Process (MPOMDP) since the algorithms in our paper are developed based on the framework of MPOMDP.
B. M ULTI - AGENT MDP AND M ULTI - AGENT POMDP MODEL B.1. MMDP
MMDP model is a tuple (cid:0) S , N, {O i } i ∈ [ n ] {A i } i ∈ [ n ] , T, R (cid:1) , where S and N are finite sets of states and agents.
A i is a finite set of actions available to agent i ; T : S ×
A 1 × · · · × A n × S → [0 , 1] is a transition function; and R : S → R is the reward function.
Each agent i obtains reward as function of the state and agent’s action
r
i : S ×
A i → R , and receives a private observation from the state by the observation channel
o i : S
→ O i .
The state has distribution d : S → [0 , 1] .
Each agent aims to maximize its own total expected return r i = (cid:80) Tt
=
γ t r ti where γ is a discounted factor and T is the time horizon.
In a shared reward situation, there is a team reward function r : S × A i · · · × A n → R , agents aim to maximize one shared total expected return r = (cid:80) Tt =
γ t r t .
[Table] [Table]
A
MPOMDP model is a tuple (cid:0) S , N, {O i } i ∈ [ n ] {A i } i ∈ [ n ] , { Ω i } i ∈ [ n ] , T, R (cid:1) , where ( S ,
A
i , T i , O i , Ω i , R ) describe a single-agent POMDP.
O i is the set of observations the agent i can make.
Ω i : S
×
A
i × O i
→
[0 , 1] is the agent’s observation channel function, which specifies probabilities of observations given agent’s actions and resulting states.
( S , A
i , T i , R i ) describes a single agent MDP; and each agent i obtaines reward as function of the state and agetn’s action r i : S
× A i → R .
Each agent aims to maximize its own total expected return r i = (cid:80) Tt
=
γ t r ti where γ is a discounted factor and T is the time horizon.
In POMDP, an agent’s belief about the sate is represented as probability distribution over S .
The agent has prior belief b 0 ,i
The agent’s current belief, b t,i over S , is continuously revised based on new observations and expected results of performed actions.
The belief update takes into account changes in initial belief, b t − 1 ,i , due to action a t,i , executed at time t − 1 , and the new observation, o t,i .
The new time belief state can be obtained from basic probability theory as follows:
b i ( s t ) =
β Ω i ( o t,i , s t , a t − 1 ,i ) (cid:80) s t
[Equation]
1 ∈ S
[Equation]
t
[Equation] [Equation]
( s t
[Equation] [Equation]
( s t , a t,i , s t − 1 ) , where β is the normalizing factor.
B. A LGORITHM : GAPSR
We give the details of GAPSR in Algorithm 1.
B. GAPSR IN D IAGRAM
We also use a diagram (Fig 4) to depict the algorithm.
The algorithm runs k iterations; each iteration first uses the policy to roll out data and uses the regressions to obtain the PSR parameters.
Then it uses the PSR as the input of policy to generate action and using the current PSR parameters to update the predictive state.
At the end of this phase, it updates both the PSR and policy parameters.
The policy parameters and PSR parameters W i and Z i are parameterized by the neural network, section C. has the network architectures.
The loss is a composite loss that includes loss from actor-critic and the loss from the predictive state representation.
The next iteration will re-learn the PSR parameters using the newly generated data based on the current policy obtained from the previous iteration; then, it does a soft update to update the PSR parameters with ones obtained at the previous iteration.
B.
[Equation]
Here we provide a brief intuitive introduction about how we connect existed MARL algorithms with
GAPSR.
Please look at Fig 4 for demonstration.
ψ ot,i , ψ at,j , and ψ ht,i,j are embedding of ( o t : t + k − 1 ,i , a t : t + k − 1 ,j ) , and ( o 1: t − 1 ,i , a 1: t − 1 ,j ) .
We also have embedding for extended part, labeled as ξ .
q t,i,j and p t,i,j are estimated by regression using the embedding vectors.
G represents the given graph, the estimation of q t,i and p t,i are based on graph G and equation 5, equation 9 and equation 10 for reference.
Please also go to section 4 for detailed description.
The policy network uses the predictive state as input to return the action or its distribution.
The agent takes the action to get the observation.
The filter F i,j takes predictive state, action, and observation as inputs to get the next predictive state.
We present a centralized critic and give the description in paragraph B.4.2.
The loss is composed into two parts, we give a detailed explanation in section 5.
Please also go to section 5 and Algorithm B. for more details about the framework.
1: Input
Sample trajectories from previous iteration k
} ni
Let π ki =
π k −
i if available or the initial policy 7:
Given G , calculate p
,i, i,j i j , tions.
Generation of Predictive Representations
Initialize GAPSR parameters Θ GAPSR
= { W i,j , Z i,j } from GAPSR Model Estimation Phase and previous iteration by a soft-update:
[Equation]
15: for b = 1 , 2 , 3 , · · · M batch of trajectories from { π k − 1 i } ni =1 do 16:
Reset episode: a b 1 , o b 1 for all agents 17: for t = 1 , 2 , · · · T roll-in in each trajectory do 18: for Each agent i do 19:
Get observation o bt,i and reward r bt,i and its neighbor’s action a bt,j 20:
Extension p bt,i,j = W
i,j ( q bt,i,j ) 21:
Filtering q bt +1 ,i,j = F i,j ( q bt,i,j , a bt,j , o bt,i , W i,j ) 22:
Predict (cid:98) o
bt,i =
Z i,j ( q bt,i,j , a bt,j ) 23:
Given the graph G , obtain q bt +1 ,i 24:
Execute a bt +1 ,i ∼ π k − 1 i ( q bt +1 ,i ) 25:
Collect o bt,i , (cid:98) o bt,i , a bt,i , r bt,i , q bt,i 26: end for 27: end for 28: end for 29: 30:
Learning Multi-agent Actor-Critic 31:
Update Θ using D = {{{ o bt,i , (cid:98) o bt,i , a bt,i , r bt,i , q bt,i , q bt,i,j } ni =1 } Tt =1 } Mb =1 : Θ ← Update (Θ k − 1 , D, η ) as in Equation (13): 32:
L (Θ) =
α 1 L 1 (Θ MARL ) + α 2 L 2 (Θ GAPSR , Θ MARL )
L 1 (Θ MARL ) =
− J ( θ ) + L c ( w )
L c = E q ,r,o (cid:2) ( Q πw ( q 1 , · · · , q n , a 1 , · · · , a n ) − y ) 2 (cid:3) ∇ θ
i J ( θ i ) =
E q ,r,o (cid:2) ∇ θ i π
θ i ( q i ) ∇ a i Q πw ( q 1 , · · · , q n , a 1 , · · · , a n ) | a i = π θi ( q i ) (cid:3)
L 2 (Θ GAPSR , Θ MARL ) =
(cid:80) ni
=1 E τ ∼ p ( τ | Θ) (cid:104)(cid:80)
j
(cid:13)(cid:13)
Z i,j (cid:0) F i,j ( W i,j ( q t − 1 ,i,j )) ⊗ ϕ at,j (cid:1) − ϕ
ot,i (cid:13)(cid:13) 2 2 (cid:105) 33:
Get W ki,j , Z ki,j , and π ki 34: end for
: Output: Return Θ =
(Θ GAPSR , Θ MARL )
B.4. GAPSR-
It is based on the IAC (Foerster et al., 2018) which directly applies the single-agent policy gradient to have each agent learn independently, with the idea behind independent Q-learning (Tan, 1993), with actor-critic in place of Q-learning.
[Figure]
IAC trains an actor-critic pair for each agent, resulting in actors π i ( a i | o i ) and critics V i
( o i , a i ) .
[Equation]
While IAC agents display a strong ability to optimize individual rewards (Tan, 1993), the lack of global information and a mechanism for cooperation means they are likely to settle for sub-optimal solutions.
Here we use the predictive state q i to fit the value and policy functions.
And we train an actor-critic pair for each agent, resulting in actors π θ i
( a i | q i ) and critics V i ( q i ) .
[Equation]
If the agents are homogeneous, we can share the critic and actor network.
We have the critic loss as
[Equation]
Here (cid:98) V is the target value function.
Thus, the update of parameters is given by:
[Equation]
B.4. GAPSR-
The algorithm extends from MADDPG (Lowe et al., 2017) using the deterministic policy gradient of the PSR-value function.
MADDPG is an extension of deep deterministic actor-critic policy gradient (DDPG) (Lillicrap et al., 2015) to multi-agent setting such that let each agent’s own critic is augmented with extra information about the actions of other agents, while their individual actor maintains a local state or observation.
The gradient of each agent is:
[Equation]
The critic loss is
[Equation]
where { (cid:98) Q πi } is the set of target value functions with delayed parameters.
Our method considers the centralized critic.
Also, in order to solve the non-stationary environments when each agent is learning, it uses the joint predictive states with joint actions as input
[Equation]
The critic loss is defined as below:
[Equation]
Thus, the update of parameters is given by:
[Equation]
1 , · · · , q t,n , a t, 1 , · · · , a t,n ) , (44) w ′ =
w + 1 T
T (cid:88) t =
η w δ t ∇ w Q π ( q t, 1 , · · · , q t,n , a t, 1 , · · · , a t,n ) , (45) θ ′
i
=
θ i + 1 T
T (cid:88) t
= η θ ∇ θ i π
θ i ( a i | q i ) ∇ a i Q π ( q 1 , · · · , q n , a 1 , · · · , a n ) | a i =
π ( q i ) .
C E NVIRONMENT AND E XPERIMENT C.
[Figure]
Here, we give the details of setting up our multi-agent environment - MAMuJoCo.
As introduced in section 6, the many agents are constructed by separating a existed single agent into parts, and each agent will only control a part of the whole agent (Figure 5).
Partially Observable space : MAMuJoCo is a simulated robotic environment, the partially observable property is achieved by only allowing partial information to the agents.
For all environments, only the angles of the agent’s joints are visible to the network; the velocities are hidden.
Action : Each agent’s action space in MAMuJoCo is given by the joint action space overall motors controllable by that agent.
Observation : For each agent i , observations are constructed by inferring which body segments and joints are observable by an agent i .
Each agent can always observe all joints within its sub-graph.
A configurable parameter k ≥ 0 determines the maximum graph distance to the agent’s subgraph at which joints are observable.
For example, k = 0 means agents can only observe their own joints and body parts, while k = 1 means it can observe its adjacent joints, which has 1 graph distance to the
[Table]
agent.
The agent observation is then given by a fixed order concatenation of each observable graph element’s representation vector.
Depending on the environment and configuration, representation vectors may include attributes such as position, velocity
[Table]
and external body forces.
In addition to joint and body segment-specific observation categories, agents can also be configured to observe the robot’s central torso’s position and velocity attributes
[Table]
C. E XPERIMENT SETUP
We select three experiments from MAMuJoCo and give a detailed description of the experiments’ setup.
In all environments, the agent has the goal to maximize the velocity of the first coordinate for the team.
We use k to denote the maximum observation distances to the subgraph.
We use ∆ d to denote the first coordinate position difference between a time difference ∆ t .
Finally, we use R to denote the reward function.
Table 1 has the configuration details for these parameters.
C. N EURAL N ETWORK A RCHITECTURE
We implement all algorithms using deep neural networks as function approximators.
We ensure that all policy and and action-value functions have the same neural network architecture among all algorithms to the extent each algorithm allows for a fair comparison.
Usually, in a continuous environment, each agent’s policy will be parameterized by its actor network that outputs the mean and diagonal covariance of a Gaussian distribution over the continuous action space.
For our experiments with continuous action spaces, a Gaussian distribution with a diagonal covariance matrix is used.
The policy network maps from the input feature to a Gaussian distribution vector µ .
Moreover, µ = [ mean, std ] , where mean is a vector specifies the action means, and std vector specifies the standard deviation.
For deterministic policy, it maps to the action vector.
In the implementation of actor-critic method, all the actor-network is parameterized by a multi-layer perceptron (MLP) with two hidden layers of size 400 and 300 respectively and ReLU activation, which takes in the individual agent’s predictive state and outputs the mean and covariance of a
Gaussian policy for stochastic policy, or the action vector for deterministic policy.
The critic network is also an MLP with two hidden layers with 400 and 300 units, respectively.
For GAPSR-1, the critic network is used to approximate per-agent utilities, which receives each agent’s predictive state as input.
For IAC, same as GAPSR-1, it receives agent local observation and individual action as input.
In GAPSR-2, there is a shared critic network that approximates all agents utilities, which receives all agents’ predictive states and the joint action of all agents as input.
The global state consists of the complete state information from the environment.
In MADDPG, the critic receives the global state and the joint action of all agents as input.
Each decentralized actor (i.e., policy) network takes in each agent’s observation and outputs the agent’s action vector.
[Table]
C. M ODEL P ARAMETERS
The hyper parameters for the the MAMuJoCo environments are in Table
[Table]
D S UPPLEMENT E XPERIMENTS D. MAM UJO C O S UPPLEMENT E XPERIMENTAL R ESULTS
We plotted the predictive observations compared to actual observations in Figure 2 at the beginning of the learning process (iteration 1 ) and end of the learning (iteration 40 ).
We also show the results of iteration ten and iteration 20 in Figure 6.
By comparing with iterations 1 and iterations 40 in
Figure 2, we see that the iteration 40 has the smallest difference between predictive observation and true observation
[Table]
and the difference gets increased as the iteration goes to the earlier stage of the learning process.
So the predictive accuracy is improved incrementally with the learning progresses.
D. M ULTI -A GENT P ARTICLE E NVIRONMENT
We also test GAPSR into another environment, multi-agent particle environment (Lowe et al., 2017).
The agents are displaced into a 2-dimensional coordinate.
This environment does not assume that all agents have identical action and observation spaces.
We run experiments using a different number of agents on two environments, the predator-prey, and cooperative-push.
We use the same configuration in (Liu et al., 2020a).
The network has the same design as MAMuJoCo in section C.3, except we use
MLP with two hidden layers with the same 128 units respectively, so we do not repeat the description.
In Table 3, we report the hyperparameters for multi-agent particle environments.
In the supplementary materials we provide the code and instructions.
[Figure]
Environments
[Table]
E D ETAILED P ROOFS
We give the proof of Theorem 1 in Appendix E.
[Table]
and we first introduce the following Lemmas to prepare the proof.
Definition 2.
Let X 1 . . .
X k be independent random variables of dimensionality d X 1 . . .
d
X k such that ∥ X k ∥ < t x k .
Let { ( x 1 j . . .
, x k j ) } Nj =1 be the N i.i.d samples from distribution of X 1 , . . .
, X k , the C X 1 :=
E [ X 1 X T 1 ] and (cid:98) C X 1 = 1 N (cid:80) Nj =1 x 1 j x T 1 j , and C X 1 X 2 := E [ X 1 X T 2 ] and (cid:98) C X 1 ,X 2 = 1 N (cid:80) N j =1 x 1 j x T 2 j .
Also, we use the · ) , v ( · ) to denote the largest, smallest eigenvalue of a matrix.
Lemma 3 ((Tropp, 2015)) .
Let a j be a finite sequence of independent random, Hermitian matrices with dimension d .
Assume that 0 ≤ v ( a j ) and u ( a j ) ≤ L for each j .
Let S
=
(cid:80)
j a j , then for any η ∈ [0 , 1] , it follows that
[Equation]
Corollary 1.
Let X be a random variable, for any ϵ ∈ (0 , 1) such that N > t 2 x
[Equation]
the following holds with probability at least 1 − ϵ
[Equation]
In other words, if N large enough, then (cid:98) C X and C X will be close enough.
Proof.
Define S
j
= 1 /Nx j x Tj .
Then it follows that u ( S j ) ≤ L = t 2 x /N and define ϵ := 2 d X e − σNv ( C X ) /t
2 x , which implies that σ = t 2 x log(2 d X /ϵ ) v ( C X ) N .
Then it follows from Matrix Chernoff
Inequality in Lemma 3 that P r ( v ( (cid:98) C X ) ≤ (1 − σ ) v ( C X ))
≤ ϵ .
Lemma 4 ((Tropp, 2015)) .
A finite sequence { a j } of independent, random matrices with common dimensions a × b , and assume that E [ a j ] = 0 and ∥ a j ∥ ≤ L for each j , let S =
(cid:80)
j a j as a random matrix.
Let V ar ( S ) be the variance statistics such that V ar ( S )
= max { (cid:13)(cid:13) E [ SS T ] (cid:13)(cid:13) , (cid:13)(cid:13) E [
[Equation]
S ] (cid:13)(cid:13) } , then
[Equation]
Corollary 2.
With at least probability 1 − ϵ that
[Equation]
(( d Y + d X ) /ϵ )
L 3 N .
where L = t y t x + ∥ C Y X ∥ ≤ 2 t y t x and V ar = max { t 2 y ∥
C X ∥ , t 2 x ∥ C Y ∥} + ∥ C Y X
∥ 2 ≤ 2 t 2 y t 2 x .
Proof.
Let X, Y be two random variables, and let a finite sequence { a j } of independent random matrices to satisfy a j =
y j x
Tj − C Y X .
So the a j will have dimensions d X × d Y .
Let random matrix
S =
(cid:80)
j a j .
It follows that E [ a j ] = 0 and ∥ a j ∥ =
(cid:13)(cid:13)
y j x Tj − C Y X (cid:13)(cid:13) ≤ ∥
y k ∥
∥ x k ∥ + ∥ C Y X ∥ ≤ t y t x + ∥ C Y X ∥ ,
[Equation]
[Equation] [Equation]
Similarly, (cid:13)(cid:13) E [ SS T ] (cid:13)(cid:13) ≤ N ( t 2 y ∥
C X ∥ + ∥ C Y X ∥ 2 ) .
By applying lemma 4, we have ϵ =
Pr ( ∥ S ∥ ≥
Nc ) ≤ ( d X + d Y ) e (cid:18) − Nc 2 / 2 Var + Lc/ 3 (cid:19) and therefore, it implies that
[Equation]
Corollary 3.
For random variable X with dimensionality d X and ∥ X ∥ ≤ t x , with probability 1 − ϵ , it follows that
[Equation]
where L = t 2 x √ v ( C X ) + t x
Proof.
The proof is similarly to the the proof of corollary 2, define a j = (cid:80) − 1 / 2 X
[Equation]
S =
(cid:80) a j then it follows that E [ a j ] = 0 and ∥ a j ∥ ≤
[Equation]
Applying lemma 4 to get
[Equation]
it follows that
[Equation]
Lemma 5.
For two random variables X, Y , let (
cid:98) C Y X
=
C Y X + ∆ Y X , and (cid:98) C X
= C X + ∆ X where
E [∆ Y X ] and E [∆ X ] are not necessarily zero and (cid:98) C X is symmetric positive semidefinite.
Define
A = C Y X C − 1 X and (cid:98)
A = (cid:98)
C Y X ( (cid:98) C X + λ ) − 1 .
Then it follows that:
[Equation]
Proof.
[Equation]
It follows that
[Equation]
For M 1 , by using facts U − 1 − V − 1 =
U − 1 ( V − U )
V − 1 and C Y X
=
C 1 / 2 Y PC 1 / 2 X
[Equation]
where P is a correlation matrix with ∥ P ∥ ≤ 1 ,
[Equation]
Corollary 4.
Let { ( x k , y k ) }
Nk =1 be i.i.d samples from two random variables
X, Y with dimensions d X and d Y and (uncentered) covariances C X and C Y .
Assume ∥
X ∥ ≤ t x and ∥ Y ∥ ≤ t y .
Define
A = C Y X C − 1 X and (cid:98)
A = (cid:98)
C Y X ( (cid:98) C X + λ ) − 1 .
For any ϵ ∈ (0 , 1) such that N > t 2 x log(2 d X /ϵ ) v ( C X ) the following holds with probability at least 1 − 3 ϵ :
[Equation]
where α = 2 t x
[Equation]
Proof.
It follows by applying Corollaries 1,2,3 to Lemma 5.
By union bound, each condition has probability 1 − ϵ , so the total events are bounded by
[Equation]
Lemma 6.
For two random variables X, Y , let (
cid:98) C Y X
=
C Y X + ∆ Y X , and (cid:98) C X
= C X + ∆ X where
E [∆ Y X ] and E [∆ X ] are not necessarily zero and (cid:98) C X is symmetric but not positive semidefinite.
Define A =
C
Y X C − 1 X and (cid:98)
A = (cid:98) C Y X (cid:98) C X ( (cid:98) C 2 X + λI ) − 1 .
Then it follows that:
[Equation]
Proof.
[Equation]
For M 1 , by using facts U − 1 − V − 1 =
U − 1 ( V − U )
V − 1 and C Y X
=
C 1 / 2 Y PC 1 / 2 X , where P is a correlation matrix with ∥ P ∥ ≤ 1 , it follows that
[Equation]
Therefore,
[Equation]
Corollary 5.
Let { ( x k , y k ) }
Nk =1 be i.i.d samples from two random variables
X, Y with dimensions d X and d Y and (uncentered) covariances C X and C Y .
The E [∆ Y X ] and E [∆ X ] is not necessarily zero and C x is symmetric but not necessarily positive semidefinite.
Assume ∥ X ∥ ≤ t x and ∥ Y ∥ ≤ t y .
Define A =
C
Y X C − 1 X and (cid:98)
A = (cid:98) C Y X (cid:98) C X ( (cid:98) C
2 X + λ ) − 1 .
For any ϵ ∈ (0 , 1) such that N > t 2 x log(2 d X /ϵ ) v ( C X ) the following holds with probability at least 1 − 3 ϵ :
[Equation]
where
[Equation]
Proof.
It follows by applying Corollaries 1,2,3 to Lemma 6.
Also by union bound, so the total events are bounded 1 − 3 ϵ .
Theorem 2.
Grabner 1997 (3.3) (Grabner & Prodinger, 1997) Consider there are n independent copies X 1 , . . .
X n i.i.d negative binomial random variables, with parameters defined as NB ( b, p ) , and our goal is to calculate the expectation of the maximum of these N random variables E n = E { max( X 1 , . . . , X n ) } then we have following asymptotic solution:
[Equation]
(where F is a periodic C ∞ − function of period 1 and mean value 0 whose Fourier-coefficients are given by ˆ F ( k ) = −
[Equation]
for
[Equation]
Z
[Equation]
0 } , and q = 1 − p ).
We omit the proof, interested readers could go to (Grabner & Prodinger, 1997) for details.
E. P ROOF OF T HEOREM
We first prove the bound for ∥ (cid:98) q i,j − q i,j ∥ .
Proposition 2.
Let π Θ be a data collection policy and H is the range of π Θ on joint histories.
If
Equation 7 used, then for all h ∈ H and any ϵ ∈ (0 , 1) , (cid:13)(cid:13)(cid:98) q i,j ( ψ h ) − q i,j ( ψ h ) (cid:13)(cid:13) is bounded as below with probability at least 1 − 3 ϵ .
[Equation]
where ∆ 1 follows the bound (55) and ∆ 2 follows the bound (53) , and γ = t 2 Aj log(2 d Aj /ϵ )
v
[Equation]
C
[Equation]
Proof.
Let T i,j is the tensor such that C ψ oi ψ aj | ψ hi,j = T i,j × h ψ hi,j , and U i,j is the tensor such that C ψ aj | ψ hi,j = U i,j × h ψ
hi,j and for simplicity without loss meaning, we use C ψ aj | ψ hi,j to denote
C ψ aj ψ aj | ψ hi,j .
Then we have
[Equation]
We finish the above proof by proofing the ∥ T i,j − T i,j ∥ and ∥ U i,j − U i,j ∥ are bounded by using
Corollary 4.
[Equation]
hi
[Equation]
53) where
[Equation]
and
[Equation]
Then using the equation (7) and corollary 5 to obtain the bound for q
i
[Equation]
j
[Equation]
where ∆ 1 follows the bound (55) and ∆ 2 follows the bound (53)
[Equation]
and γ
[Equation]
t
[Equation]
Now we start to prove our Theorem
[Equation]
Proof.
The Equation (5) says:
[Equation]
here we assume the agents are homogeneous, in other words, each pair of { ( o i , a i ) } ni =1 coming from the same spaces O , A .
They are permutation invariant and their identities do not matter.
Thus, the bound of q i,j is invariant to agents.
Under the assumption of static fully complete graph, for any agent q i = (cid:80) nj =1 q i,j , thus
[Equation]
ROOF OF L EMMA
Here we prove Lemma 1.
Proof.
Equation (9) says
[Equation]
where I i,j is an indicator function to denote if two agents are connected.
Under the assumption that the static non-complete graph has maximum of number of degrees k and the agents are homogeneous, we have (cid:80)
j I i,j < = k .
Thus,
[Equation]
E. I NFORMAL P ROOF OF L EMMA
Here we make some intuitions for the proof of Lemma 2.
As we already give the proof sketch in our main paper.
Lemma 2 is a direct application of Theorem 2.
Here our random variable
J 1 , · · · , J n ∼ NB ( r, p ) , where { ( J i ) } ni =1 represents the number of time points node i needs before it meets node j number of r times.
For the complete static graph, we need at least N sample for the bound in equation 8 to be valid; in other words, we need the trajectory to run at least N time points to collect enough data to estimate our conditional operator accurately q i,j .
For the dynamic graph, each time t , the two nodes are randomly connected with probability p , if it connects, then we can obtain a valid sample to estimate q i,j ; if not, then we skip to the next time step.
For the dynamic graph node i , if we take the union set of the nodes i connected over the trajectory path, then the union set could form a static complete graph.
We say the two graphs are equivalent.
The number of time points needed by i until the N th connection with j for each of the pairs ( i, j ) follows the same distribution J ∼ NB ( N, p ) .
Then we are interested in the expectation of the maximum of
J 1 , . .
, J n − 1 , we denote it as J { 1 ,...,n − 1 } .
E { J { 1 ,...,n − 1 } } means on average, how many time points ( N ′ )
we need for all nodes other than i at least meets N times with i .
Obviously, this N ′ >
= N since p ∈ [0 , 1] .
The calculation of this expectation is solved by (Grabner & Prodinger, 1997).
And we also put their result in Theorem 2.