R EINFORCEMENT L EARNING UNDER A M ULTIAGENT P REDICTIVE S TATE R EPRESENTATION M ODEL : M ETHOD AND T HEORY
A BSTRACT
This paper proposes an optimal policy learning algorithm under a novel multi-agent predictive state representation reinforcement learning model.
Compared to the state-of-the-art methods, the most striking feature of our approach is to introduce a dynamic interaction graph to the model, which allows us to represent each agent’s predictive state based on its “neighborhood” agents.
Methodologically, we develop an online algorithm that simultaneously learns the predictive state representation and agent policies.
Theoretically, we propose an upper bound of the L 2 -norm of the learnt predictive state representation.
Empirically, we provide positive numerical results on both a MAMuJoCo robotic learning experiment and a multi-agent particle learning environment to demonstrate the efficacy of the method.
I NTRODUCTION
Real-world multi-agent systems are considered partially observable since agents do not have a complete perception of the environment states.
The predictive state representation (PSR) (Littman et al., 2001) is a representation of the dynamic system state by a vector of predictions of tests conditioned on a history.
The tests are sequences of actions and observations, which are true if and only if all the observations occur, given that all the actions taken and the histories are sequences of actions and observations recorded before.
It records a sequence of actions and observations and focuses on observable quantities rather than the system’s dynamics.
Therefore, compared with learning the hidden-state-based partially observable Markov decision process (POMDP) Kaelbling et al.
(1998) models from observation data, learning PSR should be easier and less prone to local minimal problems (Singh et al., 2012).
In addition, in the cases where the moment-method is used to learn a model for controlled systems that incorporates actions, PSR is considered as a more general setting to reconstruct the structure of the dynamic system than POMDP (Azizzadenesheli et al., 2016; Hamilton, 2014).
Furthermore, (Littman et al., 2001; James & Singh, 2004; Singh et al., 2012) showed that PSR can offer an alternative more compact state representation for POMDP models.
Overall, PSR is considered a generalization of POMDP in terms of representation power and the development of learning algorithms.
Due to the above benefits of PSR, it has been used as the basis of some model-based single-agent reinforcement learning (James & Singh, 2004; Boots et al., 2011; Hamilton et al., 2014; Hefny et al., 2018a).
Moreover, the predictive states that act as sufficient statistics for the states of multiagent dynamic systems could potentially help multi-agent decision-making in the complicated, partially observable domains.
However, there is no method for developing multi-agent predictive state representation (MAPSR) for multi-agent reinforcement learning (MARL).
Due to this reason, we focused on building an algorithm for learning the optimal policies by representing the multi-agent dynamic systems with the MAPSR.
An existing algorithm by (Chen et al., 2020) provides a spectral framework for learning MAPSR using a tensor, with its dimensions representing the agents and joint history.
A tensor decomposition is implemented to learn the model parameters.
This type of algorithm avoids explicit estimation of the latent state and can get an unbiased estimation if an infinite amount of data is available.
However, it is not easy to incorporate prior information such as sparsity or structure because each new source of information leads to new moments and a different and more complex set of equations to solve.
(Hefny et al., 2015) represent the distributions in PSR via observable statistics and learn PSR by supervised learning, alleviating the above problem.
Later, (Hefny et al., 2018a) extend this method to develop a new one called predictive state controlled model that can represent the system model where the agents can affect the environment.
Due to the trend of deep RL, (Hefny et al., 2018b) developed an end-to-end policy learning method by using the states represented by a predictive state controlled model as the belief state to fit the policy function to train RL.
Both the PSR and RL parameters are updated by using a combined loss objective.
The PSR part maintains a sufficient representation of distributions under the partially observable environment, the deep RL algorithm brings efficiency for learning.
The environment of MARL can be partially observable and always non-stationary (one agent learning while the other agents’ actions can influence the environment)
(Bowling & Veloso, 2002).
Predictive state controlled model can model the prediction of systems states affected by actions; if we design a state representation model for MARL environment that can predict the future observations of agents upon exerting future actions of other agents and conditioned on past histories, then the non-stationary environment caused by other agents learning can be modeled.
Due to the above reason, we seek to develop a MARL-based PSR model and learning algorithm, which we call MAPSRL.
It could potentially ameliorate both the problem of partially observable observations and non-stationarity by other agents’ interaction for multi-agent decision making.
The paper’s first goal is to develop a PSR model for MARL environment.
In MARL settings, the distribution of observed observations is dependent on its actions and relevant to other agents and histories.
So far, many works (Ryu et al., 2020;
Liu et al., 2020b; Niu et al., 2021) that take other agents’ actions into consideration have been shown to achieve competitive performance.
In order to consider the interactive relationship under the MAPSR, we introduce a dynamic interaction graph to characterize the “neighborhoods” of agents.
Such a graph topology, which is often easy to obtain nowadays, enables us to encode the adjacency relationship of agents into our MAPSR model.
We build our model under three common graphs, namely static complete graphs, static non-complete graphs, and dynamic graphs, covering as many real-world scenarios as possible.
Under the existence of such a dynamic interaction graph, we build predictions of its observations by considering its neighbor’s actions for every agent.
We call this primitive interactive predictions.
Then an agent’s
PSR will be formulated as a linear combination of those primitive interactive predictions.
Due to the decentralized formulation, our model allows implementing the learning of the MAPSR on a single agent level while maintaining other agents’ interactions through the encoding of the interactive graph topology.
As set as our second goal, we analytically upper bound the L 2 -norm difference between the underlying true and learned predictive state representations, providing theoretical guarantees for the performance of our learning algorithm under the MAPSR model in all three graph settings.
Our paper’s third goal is to develop an algorithm that integrates the MAPSR model to the MARL.
We develop an online algorithm that simultaneously learns the MAPSR, and the agents’ policies end-toend with a fully differentiable neural network structure.
Our algorithm provides a new model-based
MARL, and it has the flexibility to be extended to most of the offline MARL algorithms.
We focus on the continuous action space, which is more practical and common to real-world applications such as robotic control.
Finally, we test our algorithm through systematic numerical experiments on MAMujoCo robotic learning experiments (de Witt et al., 2020) and multi-agent particle learning environments (Ryu et al., 2020), and compare our proposed method against two baselines as detailed in Section 6.
The results demonstrate the efficiency of our method.
R ELATED W ORKS
Partially Observable Environment.
Real-world agents often experience situations that the observed signals are aliased and do not fully determine their state in the system.
This is particularly true for multiple agents environments where agents have partial observability due to limited communication (Oliehoek & Amato, 2016).
In accommodation to the partially observable environment, POMDP has been adopted by (Kaelbling et al., 1998), and the algorithms (Kaelbling et al., 1998; Cassandra, 1998;
Thrun, 1999; Pineau et al., 2003; Poupart & Vlassis, 2008; Platt Jr et al., 2010) for determining an optimal policy have shifted to using the probability distributions (belief state) over the state space instead of exact state space.
In general, they have high complexity or suffer from local optima.
Moreover, the most common POMDP policy learning assumes the agent has access to a priori knowledge of the system.
The access to such prior knowledge has a premise that the agent has considerable domain knowledge (Kaelbling et al., 1998).
However, it is expected that the real-world agents learn the system model and thus a planning policy further without knowledge of the domain.
Overview of PSR.
(Littman et al., 2001; James & Singh, 2004) introduced the PSR over an expressive and robust framework for modeling dynamical systems and defined PSR as a representation of state by using a vector of predictions of fully observable quantities (tests) conditioned on past events (histories).
A predictive model is constructed directly from execution traces in the PSR framework, utilizing minimal prior information about the domain.
The PSR paradigm subsumes POMDP as a special case (Littman et al., 2001).
PSR is considered much more compact than POMDP (Aberdeen et al., 2007).
The spectral learning method has been proved to show success for learning the PSR (Boots et al., 2011; Jiang et al., 2018).
There are other classes of dynamical system learning algorithms that are based on likelihood-based optimization or sampling approaches (Frigola et al., 2013), but they are prone to poor local optima.
The spectral learning represents the estimated state by sufficient statistics of future observations and estimates the model parameters by method of moments.
However, this line of algorithms is hard to incorporate prior information (Hefny et al., 2015).
Thus, (Hefny et al., 2015; 2018a) introduce the supervised learning method to learn PSR and proves its convergence.
Although many works study PSR in discrete action space (Hsu et al., 2012; Siddiqi et al., 2010; Boots et al., 2011), (Boots et al., 2013) proposes Hilbert space embedding (HSE)-PSR to deal with continuous actions.
(Hefny et al., 2018a) uses an approximation of HSE-PSR by Random Fourier transform (RFF) and built a more principled generalization of PSR to deal with high dimensions.
However, all of these studies aim for the single agent scenario.
PSR and RL.
The predictive states estimated by the PSR are considered as states in a fully observable
Markov Decision Process so that the value function is learned on these states.
This line of work has been done in the single-agent environment (Boots & Gordon, 2010; Boots et al., 2011; Hamilton et al., 2014; Venkatraman et al., 2017; Hefny et al., 2018b).
Especially, (Hefny et al., 2018b) proposes the recurrent predictive state in the RNN network.
Moreover, the learning PSR and policy functions are connected with the end-to-end training.
MAPSR and MARL.
MAPSR model is formulated by (Chen et al., 2020) and the model parameters are learnt by tensor decomposition.
This formulation is aligned with the spectral algorithm, so it also has the same issue as the single-agent case.
MARL algorithms have been developed to learn the Markov Game environment (Lyu & Amato, 2020; Son et al., 2019;
Zhang et al., 2019; Rashid et al., 2018;
Foerster et al., 2018; Lowe et al., 2017).
The partially observable states are pervasive in this environment, bringing non-stationarity to the learning algorithm (Bowling & Veloso, 2002) so that the learning needs to consider the non-stationarity.
Recent works propose more sophisticated deep MARL algorithms for multi-agent problems under the paradigm of centralized training with decentralized execution
(Zhou et al., 2020; Sunehag et al., 2017; Lowe et al., 2017;
Foerster et al., 2018; Rashid et al., 2018).
Other than their methods, our approach also accounts for the importance of macroscopic measures of underline systems.
Also, providing a predictive state as a belief state rather than the partially observable observations can further help the agent learn.
Like the single-agent case, we hope our method can further improve these algorithms.
M ODEL D ESCRIPTION OF MAPSR 3. R EVIEW OF S INGLE A GENT PSR
Predictive State Representation (PSR).
A prediction of a state is defined as the conditional probability of seeing a test’s observations in sequence given that actions of the test are taken in sequence from a history (Littman et al., 2001).
Givn a finite observation space O ( o ∈ O ) and action space A ( a ∈ A ).
A test of length k at time t , is defined as a sequence of action-observation pairs that starts at time t and ends at time t + k − 1 , { ( o l , a l ) } t + k − 1 l = t = { o t , a t , o t +1 , a t +1 , · · · , o l , a l , · · · , o t + k − 1 , a t + k − 1 } .
A history , at time t is a sequence of action-observation pairs that start from the beginning of time and ends at time t − 1 , { ( o l , a l ) } t − 1 l =1 = { o 1 , a 1 , · · · , o l , a l , · · · , o t − 1 , a t − 1 } .
Hilbert Space Embedding Predictive State Controlled Model.
In this work, we are interested in extending PSR to decision makings of controlled systems with continuous actions.
So we use the model introduced by (Hefny et al., 2018a).
In this model, predictive state q t satisfies q t ψ at = E
[ ψ ot | ψ at ; ψ ht ] and extended predictive state p t satisfies p t ξ at = E [ ξ ot | ξ at ; ψ ht ] (i.e., q t and p t are conditional linear expectation operators which maps to the conditional expectation of future observations), where ψ ot := ϕ O ( { o l } t + k − 1 l = t ) and ψ at :=
ϕ A ( { a l } t + k − 1 l = t ) are feature maps by kernels k O ,
k A over future observation and action features.
The extended predictive state compared to predictive state adds one more pair of { ( a t + k , o t + k ) } to the prediction.
The ξ at and ξ ot are the corresponding extended feature maps, which satisfy ξ ot = ψ ot ⊗ ϕ ot and ξ at = ψ at ⊗ ϕ at .
Here ϕ ot :=
ϕ o ( o t ) and ϕ at :=
ϕ a ( a t ) are the shorthands for one time feature map by k o and k a .
We use ⊗ to denote the transposed Khatri–Rao product for two matrices with the same number of rows, and each row of the resultant matrix is the vectorzied outer product of the corresponding row vectors in the two matrices.
Also, we use the ψ ht :=
ψ h ( { ( o l , a l ) } t − 1 l =1 ) to define a set of features extracted from previous observations and actions (typically from a fixed length window ending at t − 1 ).
3. MAPSR F ORMULATION
For n -agent systems, we denote joint observations as { o t,i } ni =1 and joint actions as { a t,i } ni =1 at any time t .
The joint test is then a collection of tests for all agents { ( o l, 1 , a l, 1 ) , · · · , ( o l,n , a l,n ) } t + k − 1 l = t and the joint history is a collection of histories for all agents { ( o l, 1 , a l, 1 ) , · · · , ( o l,n , a l,n ) } t − 1 l =1 .
A prediction of multi-agent state is thus the conditional probability of a joint test given a joint history.
Our Formulation of MAPSR.
We bring the HSE-PSR model to the multi-agent settings.
For any agent i in an n -agent system, given kernels { k O i } ni =1 , { k A i } ni =1 , { k o i } ni =1 , { k a i } ni =1 , we use ψ ot,i , ψ at,i , ξ ot,i , ξ at,i , ϕ ot,i , ϕ at,i and ψ ht,i to denote its corresponding feature maps, same as the previous definitions for single agent.
The superscripts o , a , h are local observations, actions, and histories for agent i .
Moreover, we use the q t,i and p t,i to denote its predictive state and extended predictive state.
D YNAMIC I NTERACTION G RAPH F OR MAPSR M ODEL
Many works have considered graph representation of the multi-agent network (Liu et al., 2020b; Ryu et al., 2020).
In general, the relationship between agents is characterized as an undirected graph.
We introduce a dynamic interaction graph to represent the MAPSR by considering the interaction between agents.
Definition 1.
Let G =
( V, E ) , including the set V of nodes and set E of the edges.
Each node represents the agent entry, and the edge represents the relationship between the two adjacent agents.
Here we suppose the graph structure is given, that means the number of nodes ( n ), the number of edges ( m ), the edge weights, and the maximum number of degrees ( k ) are available to us.
We think that this kind of presupposition is very reasonable because in the real world, for example, there are multiple robots; we can quickly get the geographic position of the robots through sensors, then calculate the structure of the graph formed by them.
4. S TATIC C OMPLETE G RAPH
We starts with a static complete graph G c , where the relationship between nodes are invariant to time change.
A complete graph has m = n ( n − 1) / 2 edges, where m = | E | .
For each agent, we can represent its PSR by considering other agents’ interactions by
[Equation]
same for p t,i .
To consider the interactive behavior between agents, we introduce two additional notations q t,i,j and p t,i,j .
Let { ( i, j ) } ni,j =1 represents a pair of agents on a n multi-agent system.
The q t,i,j is a primitive predictive state of i ’s observation ( ψ ot,i ) by intervening agent j ’s action ( ψ at,j ) and observing agent i ’s observation history and agent j ’s action history ( ψ ht,i,j ).
And p t,i,j is the extended counterpart.
If i =
=
j , then it becomes an exact single agent scenario.
Similarly, we use the same approach to represent p t,i .
Each agent’s PSR q t,i and extended PSR p t,i are modeled by fully considering all other available agents.
Based on equation (1), in practice, estimating q t,i and p t,i denoted as (cid:98) q t,i and (cid:98) p t,i requires us to get (cid:98) q t,i,j and (cid:98) p t,i,j at first.
4.1. E STIMATION OF q t,i,j , p t,i,j
To estimate the p t and q t for single agent, (Hefny et al., 2018a) use the supervised learning method.
They show that (cid:98) q t =
M ψ ot | ψ at ; ψ ht = C
ψ ot ψ at | ψ ht ( C ψ at ψ at | ϕ ht + λI ) − 1 , where M A | B ; c is a linear operator that satisfies E
[ A | B = b ; C = c ] =
M A | B ; c b , and C XY :=
E [ ϕ ( X ) ⊗ ϕ ( Y )] is the uncentered covariance operator, and C XY | z is covariance of X and Y given Z
= z .
They estimate C ψ ot ψ at | ψ ht , C ψ at ψ at | ϕ ht by sampling data, then get the estimation (cid:98) q t .
The same procedure is used to estimate (cid:98) p t by replacing the features ψ ot and ψ at with their extend counterparts ξ ot and ξ at .
Similar to the single-agent case above, the representation of q t,i,j can be achieved as
[Equation]
To estimate C ψ ot,i ψ at,j | ψ ht,i,j and C ψ at,j ψ at,j | ψ ht,i,j , we learn two linear maps T i,j and U i,j such that
C ψ ot,i ψ at,j | ψ ht,i,j ≈ T i,j (cid:0) ψ ht,i,j (cid:1) and C ψ at,j ψ at,j | ψ ht,i,j ≈ U i,j (cid:0) ψ ht i,j (cid:1) .
The training examples for T i,j and
U i,j consist of pairs ( ψ ht,i,j , ψ ot,i ⊗ ψ at,j ) and ( ψ ht,i,j , ψ at,j ⊗ ψ at,j ).
The learning of (cid:98) p t,i,j can be done in a similar way.
More details can be found in Appendix A.2.
After calculation of (cid:98) q t,i,j and (cid:98) p t,i,j , under the static complete graph setting where the interaction is considered, defined in equation (1), we can get the estimate of (cid:98) q t,i and (cid:98) p t,i .
4.1. T HEORETICAL G UARANTEE OF E STIMATION OF q i , p i U NDER T HE S TATIC C OMPLETE G RAPH
Theoretically, we show that the difference between q i and its estimator (cid:98) q i is bounded with high probability.
Theorem 1.
Let π Θ be a data collection policy and H is the range of π Θ on joint histories.
If
Equation 1 and 2 used, then for all h ∈ H and any ϵ ∈ (0 , 1) , such that N >
[Equation]
[Equation]
where N is the number of time points we collect sample, then ∥ (cid:98) q i − q i ∥ is bounded as below with probability at least 1 − 3 ϵ ,
[Equation]
where
[Equation]
[Table] [Table] Here ∆ 1 , ∆ 2 are two other relevant bounds, we provide them in Appendix E.1.
u ( · ) , v ( · ) denote the largest, smallest eigenvalue of a matrix.
And γ = t 2 Aj log(2 d Aj /ϵ ) v (cid:16) C ψaj (cid:17) N < 1 is a constant that depends on the magnitude of the norm of ψ aj (we assume (cid:13)(cid:13) ψ aj (cid:13)(cid:13) ≤ t A j ), the dimension of ψ aj ( d A j ), the (uncentered) covariances ( C ψ aj := E (cid:2)
ψ aj ψ aTj (cid:3) ) and the sample size ( N ).
Theorem 1 says we need at least N samples for the bound in equation (3) to be valid.
We give the proofs in Appendix E.
It is not hard to obtain a bound for p i using the same approach.
We omit that.
4. E XTENSION TO S TATIC N ON -C OMPLETE G RAPH
In large-scale multi-agent systems, the number of agents is large, and not all agents need to interact with each other.
A static non-completed graph can perfectly represent such a situation.
For example, in a given static non-complete graph G s , we know its maximum number of degree k , and we use the binary n × n matrix with each entry as I i,j to indicate the interaction between two agents.
Then the
MAPSR for each agent will be
[Equation]
Lemma 1.
Under the same environment depicted in Theorem 1 and given a G s with k maximum number of the degree to represent agents, then the bound in Theorem 1 can be rewritten as ∥ (cid:98) q i − q i ∥ ≤ k ∆ .
The conclusion of Lemma 1 is evident since we replace the n with the k neighbors, the total error bound is also decreased approximately as k n .
4. D YNAMIC G RAPH
Real-world multi-agents can also formulate a time-dependent dynamic graph G d , rather than a static graph.
A dynamic graph has its structure dynamically changing with time.
In other words, the edges can be inserted or deleted across time.
The dynamic graph brings more challenges to the representation as the interaction relationship among agents changes constantly.
Braha & Bar-Yam (2009), Ma et al.
(2017) and Zhao et al.
(2010) consider a dynamic graph as a set of ordered static graphs.
For each time point, we are given a static graph such that G d = { G d 1 , G d 2 , · · · , G dt } , and a time-dependent given binary matrix with I t,i,j indicating the interaction between two agents at each time.
Then we have the agent-wise PSR as
[Equation]
Compared to (4), the coefficient I t,i,j is time-dependent, which brings a challenge to our theoretical bound.
We consider a dynamic graph experiences a trajectory path, assuming every node has a chance at least p to interact by connection with another node at any time point.
For example, for a node i , if we take the union set of the nodes interacted with i over the path, then the union set could form a static complete graph; in other words, if i connects j , then we can obtain a valid sample to estimate q i,j as the complete static graph does, if not, then we skip to the next time point.
For the complete static graph, we need the trajectory to run at least N time points to collect enough data to estimate our conditional operator q i,j accurately.
Furthermore, The total number of time points needed by i until the N th interaction with j follows a negative binomial distribution NB ( N, p ) .
On average, we need Np time points before we see i, j completely connecting N times.
Now we consider node i could interact with every node in a set of nodes ( J : | J | = n − 1 ) for N number of time points.
We assume the chances being interacted between two nodes ( i, j ∈ J ) does not affect their interaction with other nodes.
Thus we have a set of independent negative binomial random variables { J l } n − 1 l =1 ∼ NB ( N, p ) to characterize the interaction of i with j ∈ J .
So we are interested in the expectation of the maximum of J 1 , . .
, J n − 1 , a statistics that tells us the expected maximum number of time points of collection of measurements needed for the node i to be able to connect with every node j ∈ J for at least N number of time points.
We denote it as J { 1 ,...,n − 1 } and we have E
[Equation]
Lemma 2.
Under the same environment depicted in 1 and given a dynamic graph G d with every node has a chance at least p to interact with another node in a one-time point, let N be the number of time points we collect data of measurements in order to get the bound in Theorem 1, if Equationand 2 used, then we need at least
[Equation]
total number of time points, where q = 1 − p , K ( q, n, N ) :
= log 1 /q ( n − 1)
+ ( N − 1) log 1 /q (cid:104) log 1 /q ( n − 1) (cid:105)
+ ( N − 1) log 1 /q
p − log 1 /q ( N − 1)!
, F is a periodic C ∞ -function of period 1 and mean value 0 whose Fourier-coefficients are given by ˆ F ( k )
= − 1 log ( 1 q )Γ( − 2 kπi log ( 1 q )) for k ∈ Z \ { 0 } .
Then ∥ (cid:98) q i − q i ∥ achieves the same bound as in Theorem 1 with probability at least 1 − 3 ϵ .
In other words, ∥ (cid:98) q i − q i ∥ ≤ n ∆ .
(7) is an asymptotic expansion of the right-hand of (6) .
We give the proof in Appendix E.3.
Lemma 2 gives the worst-case bound for our estimation under the dynamic graph.
The result tells us that if we need 1 more sample of measurement for our algorithm to converge on the complete static graph, we need roughly log 1 /q log 1 /q ( n − 1) more samples on the dynamic graph.
So as long as we allow enough learning time, the algorithm can converge with high probability.
Complexity with Increasing Agents : For the complete static graph, we need to evaluate (cid:98) q i,j and (cid:98) p i,j every time point, which requires O ( n 2 ) operations and space.
Overall, with an increased number of agents, our MAPSR has a polynomial O ( n 2 ) scaled complexity, which is feasible for learning in a large number of agents environment and is more efficient compared to the centralized MAPSR (Chen et al., 2020) theoretically, which is combinatorially sample complex, the analysis is shown in Appendix A.4.
For a non-complete static graph with k maximum number of degrees for k ≪ n , which is more common in the real world, because a very far-away robot will not likely affect the targeted robot, the operation will be significantly decreased to O ( k 2 ) .
4. T HE E STIMATION FOR MAPSR M ODEL C OMPONENTS
As stated by (Littman et al., 2001), a complete PSR model can build a recursive rule to update itself.
In other words, given the q t,i,j , the MAPSR model can calculate q t +1 ,i,j by incorporating a new observation, a so called filtering process in the dynamical system.
We achieve this with three models proposed by (Hefny et al., 2018a): For any agent i , p t,i,j = W i,j ( q t,i,j ) , q t +1 ,i,j = F i,j ( p t,i,j , o t,i , a t,j ) , and o t,i = Z i,j ( q t,i,j , a t,j ) . The last model is used to predict the observation.
Typically, W i,j , Z i,j are learnable linear maps, and F i,j is non-linear and differentiable but known in advance.
The W i,j , Z i,j are learned from regression after we estimate q t,i,j and p t,i,j , see Appendix A.2.2 for details of estimation of W i,j , Z i,j .
D ECISION -M AKING F RAMEWORK W ITH MAPSR M ODEL
Previous work on single-agent proposed an end-to-end training algorithm
(Hefny et al., 2018a) for
PSR model and policy learning.
Here we design an algorithm for multi-agent settings and incorporate our MARPSR model containing the interactive graph component.
We propose an online learning algorithm to learn the MAPSR and agent policies simultaneously.
Our algorithm is shown in Appendix B.2.
We call the algorithm MAPSRL .
The algorithm can be divided into two parts.
Firstly, we have a state tracking component, which lets the agents explore the environment and learn an initial predictive state and its parameters W i and Z i .
The agents first enter the initialization phase (Line 1-4), where we initialize MAPSR by executing an exploration policy to collect data.
The agents explore the environment and learn the MAPSR parameters under the given interactive graph.
Secondly, we also have a policy component, in which, by starting from the initialized MAPSR and a random policy, the agents experience a T length trajectory under the current policy function that maps predictive states { q t,i } ni =1 to actions { a t,i } ni =1 (line 5-18).
The agents then update to the next predictive state as introduced in subsection 4.4.
Agents also save trajectories (actions, observations, predictive states, next predictive states, and rewards) over the path.
Finally, a
MARL algorithm conducts the policy learning component.
We use a diagram in Appendix B.3 to illustrate this process.
We update the MARL and MAPSR model parameter Θ = { Θ MAPSR , Θ MARL } (line 19) by minimizing the following objective function.
[Equation]
Here the L 1 (Θ MARL ) is the objective function for MARL, for example, if the Actor-Critic used, it will be the − J ( θ ) + L c ( w ) , the negative action value function with policy parameter θ plus the critic loss with parameter w , and Θ MARL = ( θ, w ) .
Usually, in a continuous environment, each agent’s policy will be parameterized by its actor network that outputs the mean and diagonal covariance of a Gaussian distribution over the continuous action space. We could also give a parameter sharing actor network that maps individual PSR to parameters of a Gaussian distribution over the individual action space if agents are homogeneous. L 2 (Θ MAPSR , Θ MARL ) = (cid:80) n i =1 E τ ∼ p ( τ | Θ) (cid:104)(cid:80) j
(cid:13) (cid:13) Z i,j (cid:0) F i,j ( W i,j ( q t − 1 ,i,j )) ⊗ ϕ a t,j (cid:1) − ϕ o t,i (cid:13) (cid:13) 2 2 (cid:105) is the MSE between prediction and actual observation. And p ( τ | Θ) is the distribution over trajectories induced by the policy and MAPSR.
Θ MAPSR =
{ W i,j , Z i,j } denotes MAPSR’s parameters. The gradient is equal to ∇ Θ MAPSR L 2 (Θ MAPSR , Θ MARL ) = 1 T (cid:80) ni =1 (cid:80) Tt =1 (cid:80)
j (cid:104)(cid:13)(cid:13)
Z i,j (cid:0) F i [Equation] ( W i, [Equation]
( q t
[Equation]
1 ,i,
[Equation]
)) ⊗
[Equation]
(cid:1)
[Equation]
(cid:13)(cid:13) [Equation] (cid:105) . The α 1 and α 2 are hyper-parameters to penalize differently on two losses.
Now we devise our MAPSR based MARL algorithm in detail.
Our algorithm follows the actorcritic framework.
We first develop an algorithm under partially observable environments, where each agent has its own critic and actor independently, the gradient of the policy is written as ∇ θ i J ( θ i ) =
E τ i ∼ p ( τ i | Θ i ) (cid:2)(cid:80) Tt =1 ∇ θ i log π θ i ( a t,i | q t,i )
( r i + γV π i w i ( q t +1 ,i ) − V π i w i ( q t,i ))
(cid:3) , and the independent critic is updated by minimizing the loss L c = E τ i ∼ p ( τ i | Θ i ) [( V π i w i ( q i ) − y i ) 2 ] .
Here the predictive states q i estimated by the MAPSR are considered as states to fit the value function.
π θ : Q × A → [0 , 1] is the stochastic policy that maps to the probability density of actions with parameter θ .
We call it MAPSRL-1 .
As we know, independent actor-critic (IAC) (Foerster et al., 2018) would not work well since the environment is not stationary under multi-agent setting, the MADDPG by (Lowe et al., 2017) solves a non-stationary environment by considering other agents actions; however, every agent needs a separate critic that has the global information.
We used a centralized critic minimizes the loss L c = E q ,r,o (cid:2) ( Q πw ( q 1 , · · · , q n , a 1 , · · · , a n ) − y ) 2 (cid:3) .
So our policy gradient is written as ∇ θ
i J
( θ i ) =
E q ,r,o (cid:2) ∇ θ i π
θ i ( q i ) ∇ a i Q πw ( q 1 , · · · , q n , a 1 , · · · , a n ) | a i = π θi ( q i ) (cid:3) . Here by abuse of notation, we use π θ : Q → A to indicate the deterministic policy with parameter θ .
Also, to solve the non-stationary environments when each agent is learning, it uses the joint predictive states as input.
The benefits are 1.
PSR ( q ) is a function of action ( a ), bringing agents’ action information into the policy gradient, 2.
Also, our interactive MAPSR brings the effect of other agents’ actions to the policy gradient.
We call it MAPSRL-2 .
We put our integration details in Appendix B.4.
E XPERIMENTS
[Figure]
A v e r a g e r e
[Figure] r d
We evaluate the performance of our MAPSRL on a tasks under some OpenAI Gym MAMujoco environments (de Witt et al., 2020), such multi-agent swimmer, hopper, and robotic agent is represented as graph, where vertices (joints) are connected by segments) shown in hiding observations of the their corresponding single-agent ones
- ever, from the system in the system to orate to reach their goals setup
Baselines and Evaluation.
We run 50 iterations for each experiment and collect M =trajectories in every iteration with a maximum of 1000 steps in every trajectory.
After each iteration, we compute the average return R =
M (cid:80) ni =1 (cid:80) Mb =1 (cid:80) T b t =1 r ti,b on a batch of M trajectories, where, T b is the length of the b th trajectory.
We repeat this process using ten different random seeds and report the average and a standard deviation.
To verify the effectiveness of interactive graph, we introduce a baseline called Independent PSR learning (IPSR); in this model, we do not consider the graph, so we formulate n independent single PSR without considering their interactions, which means there are no q i,j any more but only q i .
The architecture of MAPSRL-1 and MAPSRL-2 remains the same.
To verify the advantage of PSR, we introduce another baseline (MARL) where we take out the MAPSR entirely, so it matches with the MARL run on a partially observable environment.
For MAPSRL-1, its MARL baseline is IAC (Foerster et al., 2018), and for MAPSRL-2, is MADDPG (Lowe et al., 2017).
Results And Discussion Figure 1 illustrates the empirical average return vs. the number of interactions with the environment measured in time steps.
Our MAPSRL methods consistently outperform IPSR and get the highest rewards under partially observable environments, which justifies the representation power of the interactive graph to assist agents in learning in the non-stationary environment with limited observation when the MARL algorithm has a defect (IAC).
Moreover, it can also boost the performance of the existing good MARL algorithm (MADDPG).
To further verify the effect of learning the PSR part, we also plotted the predicted trajectory to verify the MAPSR’s performance for predicting the observations in Figure 2.
We plotted the predicted observations vs. actual observations in iterations 1 , and 40 , respectively, for MAPSRL-2.
We plotted a row × columns figure, with each row representing the observation feature and each column representing [Figure] each agent.
As we can see from the figures, the first iteration of the learning does not predict the actual observation very well; it has some mismatches.
However, as learning progresses, the predictions get increasingly more accurate.
Note that the actual trajectory is changing according to the current policy and the current policy is optimized based on the further accurate learning of MAPSR
[Figure]
To enrich our algorithm environment
[Figure]
we also test our algorithm (MAPSRL-2) in multi-agent particle environments, using the benchmark by (Lowe et al., 2017) [Figure] please check Appendix D.2 for experiment details.
We test our algorithm for large n cases.
In this environment, the agents can have cooperative goals such that all agents must maximize a shared return and conflicting competitive goals.
We set up the environments where agents can only perform physical actions but not communication; however, to achieve the goals, agents need explicit communication about others’ locations to achieve the best reward.
These partially observable environments give us the motivation to test our method.
We report the rewards in
Figure 3.
We see that MAPSRL outperforms IPSR and baseline, in terms of the convergence speed and final attained rewards, with a different number of agents.
The predictive states convey the information that can help communication between agents in limited communication and observation environments
C ONCLUSION
We propose a MAPSR model, extending ideas from single-agent predictive state representations to a multi-agent scenario, during the process, we introduce the dynamic interactive graph to model agents’ interactions.
Furthermore, we provide the theoretical guarantees of the MAPSR model.
Finally, a learning algorithm that supports gradientbased deep MARL methods is developed.
Our method provides a model-based MARL framework under a partially observable environment.
The experiments proved that our model assumption is valid by observing the highest return while reducing the observations’ prediction error over trajectories.
A PPENDICES T O R EINFORCEMENT L EARNING UNDER A M ULTI - AGENT P REDICTIVE S TATE R
EPRESENTATION M ODEL
M
ETHOD AND T HEORY A M ETHOD D ETAILS A. M ATHEMATICAL M ODEL OF PSR
Here, we give the mathematical model for PSR.
As in section 3.1, we let τ t := { ( o l , a l ) } t + k − 1 l = t as a test (a sequence of action-observation pairs) at time t , then we have a subset τ
ot :=
{ o l } t + k − 1 l = t for observations, and τ at := { a l } t + k − 1 l = t for actions.
And we let h t := { ( o l , a l ) } t − 1 l =1 as the history at time t .
A test is executed at time t if we intervene to select the sequence of actions specified by the test.
It is said to succeed at time t if it is executed and the sequence of observations in the test matches the observations by the system (Boots et al., 2013).
The prediction of a length- k test τ t is defined as the probability distribution that the test succeed at time t , given history h t : Pr ( τ ot | τ at , h t ) =
Pr ( τ ot ,τ at | h t )
Pr ( τ at | h t )
Let T = { τ z } , z ∈ N is a finite set of core tests.
And we use T o = { τ o 1 , · · · , τ o | T | } and T a = { τ a 1 , · · · , τ a | T | } to indicate its observation and action parts.
A.1. L INEAR PSR
A linear PSR is the prediction vector
[Equation]
(9) that contains the probabilities of success of the tests in T , if and only if for any test τ
[Equation]
Where f τ : [0 , 1] | T | (cid:55)→ [0 , 1] is a linear projection function.
For simplicity of notation, we use the same notations for q t p t , W, F , which only follows the meaning defined in this section.
It means that knowing the probabilities for the tests in T is sufficient for computing the probabilities for all other tests in the system.
The prediction vector is a sufficient statistic for the system at time t , so we call it as state for the PSR at time t .
Therefore, PSR can represent state by using a vector of predictions of fully observable quantities (tests) conditioned on past events (histories).
The prediction function f τ is linear and has one-to-one relationship to a test τ such that f τ ( q t ) =
f ⊤ τ q t ∀ t .
The linear PSR can still represent systems with nonlinear dynamics.
To maintain predictions in T , we need to update the state q t .
To do that, we predict the success of any core test τ z prepended by a new action a and observation o at time t + k − 1 , which we call oaτ z .
Based on Bayes rule, we have
[Equation]
f aoτ z , f ao ∈ R | T | are linear operators such that ( ∀ τ z ∈ T , ∀ a ∈ A , ∀ o ∈ O ) .
Then let F aoτ be the matrix with its columns as f aoτ z for all τ z ∈ T .
Then the updated state in PSR is obtained by
[Equation]
Given the initial prediction vector q 1 , the PSR can update with equation 12.
This recursive application of Bayes rule to a belief state is called the Bayes filter.
Now we have seen the extended prediction vector, so we define extended predictive state as:
[Equation]
[Equation]
Clearly, we can see there exists a linear extension operator W such that p t = W ( q t ) .
And given the new observation, we can have a Bayes filter F such that q t +1 =
F ( p t , o, a ) .
We also introduced these operators in section 4.4.
A.1. R EPRESENTATION OF S TATE AS C ONDITIONAL E XPECTATION OF S UFFICIENT S TATISTICS
Instead of learning the distribution, here we recover the idea of representation of states as a conditional expectation of sufficient statistics and using the supervised learning method to learn (Hefny et al., 2015).
Let history h
ot is a sequence of observation that starts from the beginning of time and ends at time t − 1 .
We define the belief state b t =
Pr ( s t | h ot ) , where s t is the current state of the world.
b t +1 =
Pr ( s t +1 | h ot +1 ) is the next time belief state.
We call b t as "belief state", which represents the probability distribution over state space, also represents the knowledge and uncertainty about the true state of the system.
In dynamic system, the task of getting the updated b t +1 with given b t and new observation o t is called filtering.
The task of estimating the Pr ( s t +1 | h ot ) with given current b t without incorporating any new observation is called one-step prediction.
Instead maintaining a belief b t over states, spectral algorithms try to recover observable operators that can be used to perform filtering and prediction directly, by maintaining the expected value of a sufficient statistic of future observations.
Let a test τ o t is a length k sequence of observation defined before.
As the recursive Bayes rule holds, we can get
the next time prediction vector Pr ( τ ot +1 |
h ot +1 ) using the new observation o t + k and the current prediction Pr ( τ ot | h ot ) , we also define an extended prediction vector Pr ( τ ot {++
+} | h ot ) .
These prediction vectors characterize the state of the system and they can be estimated by observable quantities.
Given Pr ( τ ot +1 | h ot ) , filtering becomes the task of getting the updated prediction vector Pr ( τ ot +1 |
h ot +1 ) , "conditioning" on o t .
One-step prediction becomes getting the Pr ( τ ot +1 | h ot ) , "marginalizing" over o t .
Therefore, the spectral algorithms avoid explicitly estimating the latent state or the initial, transition, or observation distributions.
We let q t =
E [ ν t | h ot ] , where ν t a vector of features that determines the distribution of future observations Pr ( τ ot | h ot ) .
For simplicity of notation, we use the same notations for q t , p t , W, F .
Let p t =
E [ φ t | h ot ] , where φ t is a vector of features that determines the distribution of observations Pr ( τ ot [-+
-]+1 | h ot ) .
We call q t is the transformed predictive state.
So q t +1 = E (cid:2)
ν t +1 |
h
ot +1 (cid:3) , is the updated predictive state, and p t is the extended predictive state.
Let h ot be the feature vector of history
h
ot .
In Hidden Markov Models and Kalman filters, the extended state p t is linearly related to the predictive state q t .
Which is p t = W q t .
Estimation of the W can be done using linear regression with samples ν t and φ t , however, due to the overlap between observation windows, the noise terms on ν t and φ t are correlated, which will cause biased estimate.
The instrumental regression (Pearl et al., 2000;
Stock et al., 2012) is employed.
h ot is a instrumental variable that do not overlap with sequence { l } l = t + k − 1 l = t and { l } l = t + k l = t .
The correlation corr (h ot , e ( φ t ))
=
and corr (h ot , e ( ν t ))
= 0 , where e is a measure of error.
By taking the conditional expectation of p t = W q t given h ot , we have
[Equation]
[Equation]
Based on the above relationship, we first estimate the E [ φ t | h ot ] and E [ q t | h ot ] by sample h
ot , ν t , and φ t , we then use the estimates to compute W .
So if we start with q 1 , we can compute p 1 = W q 1 , and get the q t +1 = F ( p 1 , o 1 ) , where F is the Bayes filter to update the state.
A. L EARNING FOR MAPSR IN D ETAILS
Here we introduce the details about learning MAPSR, which supplements the section 4.4 and 4.1.1.
A.2. L EARNING F OR (cid:98) q t,i,j AND (cid:98) p t,i,j
To calculate (cid:98) q t,i,j , we introduce two sets of linear operators T i,j and U i,j , such that T i,j ( ψ ht,i,j ) ≈
C ψ ot,i ψ at,j | ψ ht,i,j and U i,j ( ψ ht,ij )
≈ C ψ at,j ψ at,j | ψ ht,i,j .
We estimate them by using two ridge regressions:
[Equation] [Equation]
L represents the ridge regression loss, and R represents the regularizer.
After learning T i,j and U i,j , we can get the estimate of C ψ ot,i ψ at,j | ψ ht,i,j and C ψ at,j ψ at,j | ψ ht,i,j .
Then we get the (cid:98) q t,i,j by equation 2.
Similarly, we use extended features to obtain (cid:98) p t,i,j .
A.2. E XTENSION , F ILTERING AND P REDICTION F UNCTIONS
Filtering.
To obtain (cid:98) q t +1 ,i,j from (cid:98) q t,i,j , (cid:98) p t,i,j , we use filtering.
We denote F i,j as the filtering function.
We describe the filtering process as below.
From { o i } ni =1 and { a j } nj =1 , we obtain the embedding { ϕ ot,i } ni =1 and { ϕ at,j } nj =1 .
We then compute the observation covariance
[Equation]
We then multiply the extended state by inverse observation covariance to change predicting ϕ ot,i into conditioning on ϕ ot,i .
[Equation]
here is to denote n − mode (matrix or vector) product, × ϕ
ot,i means multiplying the tensor by a matrix (or vector) in mode ϕ
ot,i .
We condition on ϕ ot,i and ϕ at,j to obtain shifted state.
[Equation]
Based on the updating rule, q t +1 ,i,j = F i,j ( p t,i , o t,i , a t,j ) , and p t,i =
W i,j ( q t,i,j ) , we write the filtering equation. [Equation] +
[Equation]
× ϕ [Equation] i ϕ
[Equation] i × ϕ [Equation] j ϕ
[Equation]
where p t,i := M ψ ot +1 ,i ⊗ ϕ ot,i | ψ at +1 ,j ,ϕ at,j ; ψ ht,i,j .
F i,j usually is known because it is obtained through the above calculation using known quantities, however, W i,j , Z i,j must have to be learned by using regressions.
Extension.
The W i,j can be learned by kernel regression if we know the (cid:98) q t,i,j and (cid:98) p t,i,j .
Previous work (Hefny et al., 2018a) demonstrated the kernel regression model for learning single-agent PSR, here we extend to the MAPSR.
We set the model parameter W i,j .
We optimized a ridge regression problem for W i,j .
[Equation]
Prediction.
We can also get the prediction about the next one time observation o t by the regression function such as:
[Equation]
We solve the prediction regression function Z i,j by another ridge regression:
[Equation]
A. T ENSOR D ECOMPOSITION OF MAPSR
Existed Formulation.
(Chen et al., 2020) use a n + 1 multi-dimensional tensor called system dynamics tensor D ∈ R |T 1 |×···×|T n |×|H| to represent the system dynamics of the MAPSR, with |T 1 | representing the cardinality of the tests for agent 1 , n denoting the number of agents, and the |H| being the cardinality of the joint history.
Each element of the tensor is a probability of a joint test given joint histories.
Given the system dynamics tensor, finding the latent predictive state can be transferred into finding a minimal linearly independent set from the system dynamics tensor, and it can be solved by spectral method such as tensor decomposition.
As mentioned earlier, this formulation can not satisfy our needs.
In Proposition 1 of Appendix A.4, we also give the sample complexity analysis such that the sample size needed to formulate D scales exponentially with the length of tests and number of agents.
Here we give a summary of tensor decomposition of their method.
Given a system dynamic tensor D such that:
[Equation]
Here ◦ is the outer product.
D 1 , · · · , D n , F are matrices.
The factor matrices D 1 , D n , F consist of the vectors, i.e., D 1 =
[ D 1:1 D 1:2 · · · D 1: R ] ∈ R |T 1 |× R .
A colon is used to indicate all elements of a mode, thus, the R th column of D 1 is denoted by D 1: R .
For any i 1 ∈ { 1 , . . .
, |T 1 |} , i n ∈ { 1 , . . . , |T n |} , and k ∈ { 1 , . .
. , |H|} , after the decomposition, we get
[Equation]
where { i 1 , · · · , i n , k } are index corresponding to the specific dimension of the D .
The last dimension of the tensor D is compressed in a matrix F , and its row vector x k =
[ x k (1) . . .
x k ( R )]
∈ R 1 × R , k ∈ { 1 , . .
. , |H|} is a summary of joint history and can be considered as a compressed version of the system
predictive state vector p
( Q | h k ) , the joint history h k ∈ H ( k ∈ [1 , |H| ]) at time step s = | h k | , where Q is the core joint test set.
The whole fibers listed in the set Q form a basis of the space spanned by the mode-(n+1) fibers of tensor D .
Thus, by constructing the vector m = ( λ ∗ D 1 i 1 : ∗ · · · D ni n : ) T , where ∗ is Hadamard product.
D 1 i 1 : denotes the i 1 -th row vector of D 1 .
Then we could rewrite the previous equation as
[Equation]
m ( r ) is a scalar that m ( r )
=
λ r D 1 i 1 r · · · D ni n r .
And x k is the system state vector and m is the prediction parameter, both of them are obtained by the tensor decomposition.
A. A NALYSIS OF S AMPLE C OMPLEXITY FOR F ORMULATING THE S YSTEM D YNAMIC T ENSOR
The paper (Chen et al., 2020) does not analyze the sample complexity to construct D , which is a n + 1 multi-dimensional tensor D ∈ R |T 1 |×···×|T n |×|H| .
We give this analysis.
The D i 1 , ··· ,i n ,k :=
Pr ( t 1 i 1 , · · · , t ni n | h k ) is an element of that tensor D such that t 1 i 1 is the i 1 -th test of agent 1, similarly, the t ni n is the i n -th test of agent n , and h k is the joint history.
Proposition 1.
In a n -agents system, assume every agent has the same observation and action space |O| , |A| , for a length- k test, to formulate a complete system dynamics tensor D defined in equation 21.
Assume each entry of the tensor needs S samples to give a sufficient estimation using Monte-Carlo roll-out method, then the total sample size is at least ( |O||A| ) kn S , if the agents are homogeneous, in other words, they are permutation invariant such that identity does not matter, then the total sample size is (cid:0) |O||A| k + n − 1 n (cid:1) S .
Proof.
At one time step, for any agent, it has |O||A| different combinations for the joint test, then for a length of k tests, it follows that ( |O||A| ) k number of different choices.
Then the tensor D would need ( |O||A| ) kn elements to cover all the possible length k tests.
So the total sample size is ( |O||A| ) kn S .
If the agents are homogeneous, then the ordering does not matter, for each agent, we have ( |O||A| )
k number of different choices for length k test, so the total choices for n agents are (cid:0) |O||A| k + n − 1 n (cid:1) , then we need (cid:0) |O||A| k + n − 1 n (cid:1) S samples.
The sample complexity is exponentially scaled with the number of agents and length of tests.
B A LGORITHM AND I NTEGRATING WITH MARL METHOD
We first introduce some backgrounds on two multi-agent frameworks: Multi-agent Markov Decision
Process (MMDP) and Multi-agent Partially observable Markov Decision Process (MPOMDP) since the algorithms in our paper are developed based on the framework of MPOMDP.
B. M ULTI - AGENT MDP AND M ULTI - AGENT POMDP MODEL B.1. MMDP
MMDP model is a tuple (cid:0) S , N, {O i } i ∈ [ n ] {A i } i ∈ [ n ] , T, R (cid:1) , where S and N are finite sets of states and agents.
A i is a finite set of actions available to agent i ; T : S ×
A 1 × · · · × A n × S → [0 , 1] is a transition function; and R : S → R is the reward function.
Each agent i obtains reward as function of the state and agent’s action
r
i : S ×
A i → R , and receives a private observation from the state by the observation channel o i : S → O i .
The state has distribution d : S → [0 , 1] .
Each agent aims to maximize its own total expected return r i = (cid:80) Tt
=
γ t r ti where γ is a discounted factor and T is the time horizon.
In a shared reward situation, there is a team reward function r : S × A i · · · × A n → R , agents aim to maximize one shared total expected return r = (cid:80) Tt =
γ t r t .
B.1. MPOMDP
A
MPOMDP model is a tuple (cid:0) S , N, {O i } i ∈ [ n ] {A i } i ∈ [ n ] , { Ω i } i ∈ [ n ] , T, R (cid:1) , where ( S ,
A
i , T i , O i , Ω i , R ) describe a single-agent POMDP.
O i is the set of observations the agent i can make.
Ω i : S
×
A
i × O i
→ [0 , 1] is the agent’s observation channel function, which specifies probabilities of observations given agent’s actions and resulting states.
( S , A
i , T i , R i ) describes a single agent MDP; and each agent i obtaines reward as function of the state and agetn’s action r i : S
× A i → R .
Each agent aims to maximize its own total expected return r i = (cid:80) Tt
=
γ t r ti where γ is a discounted factor and T is the time horizon.
In POMDP, an agent’s belief about the sate is represented as probability distribution over S .
The agent has prior belief b 0 ,i
The agent’s current belief, b t,i over S , is continuously revised based on new observations and expected results of performed actions.
The belief update takes into account changes in initial belief, b t − 1 ,i , due to action a t,i , executed at time t − 1 , and the new observation, o t,i .
The new time belief state can be obtained from basic probability theory as follows:
[Equation]
where β is the normalizing factor.
B. A LGORITHM : MAPSRL
We give the details of MAPSRL in Algorithm 1.
B. MAPSRL IN D IAGRAM
We also use a diagram (Fig 4) to depict the algorithm.
The algorithm runs k iterations; each iteration first uses the policy to roll out data and uses the regressions to obtain the PSR parameters.
Then it executes the policy phase by using the PSR as the input of policy to generate action and using the current PSR parameters to update the predictive state.
At the end of the policy phase, it updates both the PSR and policy parameters.
The policy parameters and the linear relationship captured by
W i and Z i are parameterized by the neural network, section C.3 has the network architectures.
The loss is a composite loss that includes loss from actor-critic and the loss from the predictive state representation.
The next iteration will re-learn the PSR parameters using the newly generated data based on the current policy obtained from the previous iteration; then, it does a soft update to update the PSR parameters with ones obtained at the previous iteration.
B. I NTEGRATING T WO C OMMON MARL A LGORITHMS I NTO MAPSR M ODEL
Here we provide a brief intuitive introduction about how we connect existed MARL algorithms with
MAPSR.
Please look at Fig 4 for demonstration.
ψ ot,i , ψ at,j , and ψ ht,i,j are embedding of ( o t : t + k − 1 ,i , a t : t + k − 1 ,j ) , and ( o 1: t − 1 ,i , a 1: t − 1 ,j ) .
We also have embedding for extended part, labeled as ξ .
q t,i,j and p t,i,j are estimated by regression using the embedding vectors.
G represents the given graph,
1: Input: Learning rate t 2: Initialize MARL Policy randomly 3: for k = 1 , 2 , 3 , · · · iterations do 4:
State Tracking Phase 5:
Sample b = 1 , 2 , 3 , · · · M batch of initial trajectories: { ( o bt , a bt ) } Mb =1 from existed policy obtained from previous iteration k − 1 : { π k − 1 i } ni = 6:
Let π ki =
π k − {+
+} i if available or the initial policy 7:
Given G , calculate (cid:98) q t,i,j and (cid:98) p t,i,j , and obtain the initial W i,j , Z i,j , F i,j : 8: (1).Regression q t,i,j = T i,j ◦ U i,j ( h t,i,j ) to get the (cid:98) q t,i,j and (cid:98) p t,i,j 9: (2).Given the (cid:98) q t,i,j , (cid:98) p t,i,j , compute W i,j , Z i,j , F i,j 10: (3).Get initial q 1 ,i,j and p 1 ,i,j by using the T i,j and U i,j with the early window of observations. 11: (4).Given the graph G , using equation 1, 4, or 5 to obtain q 1 ,i , p 1 ,i 12: 13:
Policy Phase 14:
Initialize MAPSR parameters Θ MAPSR =
{ W i,j , Z i,j } from state tracking phase and previous iteration by a soft-update:
[Equation]
15: for b = 1 , 2 , 3 , · · · M batch of trajectories from { π k − 1 i } ni =1 do 16:
Reset episode: a b 1 , o b 1 for all agents 17: for t = 1 , 2 , · · · T roll-in in each trajectory do 18: for Each agent i do 19:
Get observation o bt,i and reward r bt,i and its neighbor’s action a bt,j 20:
Extension p bt,i,j = W
i,j ( q bt,i,j ) 21:
Filtering q bt +1 ,i,j = F i,j ( q bt,i,j , a bt,j , o bt,i , W [-i ) 21: Execute a bt +1 ,i ∼ π k − 
 i ( Q bt +1 ,i-]i,j ) 22:
Predict (cid:98) o
bt,i =
Z i,j ( q bt,i,j , a bt,j ) 23:
Given the graph G , obtain q bt +1 ,i 24:
Execute a bt +1 ,i ∼ π k − 1 i ( q bt +1 ,i ) 25: Collect o bt,i , (cid:98) o bt,i , a bt,i , r bt,i , q bt,i 26: end for 27: end for 28: end for 29:
Update Θ using D = {{{ o bt,i , (cid:98) o bt,i , a bt,i , r bt,i , q bt,i , q bt,i,j } ni =1 } Tt =1 } Mb =1 : Θ ← Update (Θ k − 1 , D, η ) as in Equation (8), get W ki,j , Z ki,j , and π ki .
30: end for 31: Output: Return Θ = (Θ MAPSR , Θ MARL ) the estimation of q t,i and p t,i are based on graph G and equation 1, equation 4 and equation 5 for reference.
Please also go to section 4 for detailed description.
The policy network uses the predictive state as input to return the action or its distribution.
The agent takes the action to get the observation.
The filter F i,j takes predictive state, action, and observation as inputs to get the next predictive state.
We present a centralized critic and give the description in paragraph B.4.2.
The loss is composed into two parts, we give a detailed explanation in section 5.
Please also go to section 5 and Algorithm B. for more details about the framework.
B.4. MAPSRL-
It is based on the IAC (Foerster et al., 2018) which directly applies the single-agent policy gradient to have each agent learn independently, with the idea behind independent Q-learning (Tan, 1993), with actor-critic in place of Q-learning. [Figure]
Section B.4 and section C.3 give a detailed verbal description about this architecture.
IAC trains an actor-critic pair for each agent, resulting in actors π i ( a i | o i ) and critics V i
( o i , a i ) .
[Equation]
While IAC agents display a strong ability to optimize individual rewards (Tan, 1993), the lack of global information and a mechanism for cooperation means they are likely to settle for sub-optimal solutions.
Here we use the predictive state q i to fit the value and policy functions.
And we train an actor-critic pair for each agent, resulting in actors π θ i ( a i | q i ) and critics V i ( q i ) .
[Equation]
If the agents are homogeneous, we can share the critic and actor network.
We have the critic loss as
[Equation]
Here (cid:98) V is the target value function.
Thus, the update of parameters is given by:
[Equation]
B.4. MAPSRL-
The algorithm extends from MADDPG (Lowe et al., 2017) using the deterministic policy gradient of the PSR-value function.
MADDPG (Lowe et al., 2017) is an extension of deep deterministic actor-critic policy gradient (DDPG)
(Lillicrap et al., 2015) to multi-agent setting such that let each agent’s own critic is augmented with extra information about the actions of other agents, while their individual actor maintains a local state or observation.
The gradient of each agent is:
[Equation]
[Equation]
The critic loss is
[Equation]
Where { (cid:98) Q πi } is the set of target value functions with delayed parameters.
Our method considers the centralized critic. Also, in order to solve the non-stationary environments when each agent is learning
[Equation]
it uses the joint predictive states with joint actions as input.
[Equation]
The critic loss is defined as below:
[Equation]
Thus, the update of parameters is given by:
[Equation]
, · · · , a t
[Equation]
n ) (34) θ ′ i =
[Equation]
T
T (cid:88) t = η θ ∇ [Equation] ∇
[Equation]
[Equation]
n [Equation] [Equation] C E NVIRONMENT AND E XPERIMENT C. MAM UJO C O E NVIRONMENT S ETUP
[Figure]
[Figure]
[Figure]
Here, we give the details of setting up our multi-agent environment - MAMuJoCo.
As introduced in section 6, the many agents are constructed by separating a existed single agent into parts, and each agent will only control a part of the whole agent (Figure 5).
Partially Observable space : MAMuJoCo is a simulated robotic environment, the partially observable property is achieved by only allowing partial information to the agents.
For all environments, only the angles of the agent’s joints are visible to the network; the velocities are hidden.
Action : Each agent’s action space in MAMuJoCo is given by the joint action space overall motors controllable by that agent.
Observation : For each agent i , observations are constructed by inferring which body segments and joints are observable by an agent i .
Each agent can always observe all joints within its sub-graph.
A configurable parameter k ≥ 0 determines the maximum graph distance to the agent’s subgraph at which joints are observable.
For example, k = 0 means agents can only observe their own joints and body parts, while k = 1 means it can observe its adjacent joints, which has 1 graph distance to the agent.
The agent observation is then given by a fixed order concatenation of each observable graph element’s representation vector.
Depending on the environment and configuration, representation [Table] vectors may include attributes such as position, velocity [Table]
and external body forces. In addition to joint and body segment-specific observation categories, agents can also be configured to observe the robot’s central torso’s position and velocity attributes [Table]
C. E XPERIMENT SETUP
We select three experiments from MAMuJoCo and give a detailed description of the experiments’ setup.
In all environments, the agent has the goal to maximize the velocity of the first coordinate for the team.
We use k to denote the maximum observation distances to the subgraph.
We use ∆ d to denote the first coordinate position difference between a time difference ∆ t .
Finally, we use R to denote the reward function.
Table 1 has the configuration details for these parameters.
C. N EURAL N ETWORK A RCHITECTURE
We implement all algorithms using deep neural networks as function approximators.
We ensure that all policy and and action-value functions have the same neural network architecture among all algorithms to the extent each algorithm allows for a fair comparison.
For our experiments with continuous action spaces, a Gaussian distribution with a diagonal covariance matrix is used.
The policy network maps from the input feature to a Gaussian distribution vector µ .
Moreover, µ = [ mean, std ] , where mean is a vector specifies the action means, and std vector specifies the standard deviation.
For deterministic policy, it maps to the action vector.
In the implementation of actor-critic method, all the actor-network is parameterized by a multi-layer perceptron (MLP) with two hidden layers of size 400 and 300 respectively and ReLU activation, which takes in the individual agent’s predictive state and outputs the mean and covariance of a
Gaussian policy for stochastic policy, or the action vector for deterministic policy.
The critic network is also an MLP with two hidden layers with 400 and 300 units, respectively.
For MAPSRL-1, the critic network is used to approximate per-agent utilities, which receives each agent’s predictive state as input.
For IAC, same as MAPSRL-1, it receives agent local observation and individual action as input.
In MAPSRL-2, there is a shared critic network that approximates all agents utilities, which receives all agents’ predictive states and the joint action of all agents as input.
The global state consists of the complete state information from the environment. In MADDPG, the critic receives the global state and the joint action of all agents as input.
Each decentralized actor (i.e., policy) network takes in each agent’s observation and outputs the agent’s action vector.
C. M ODEL P ARAMETERS
The hyper parameters for the the MAMuJoCo environments are in Table 2. [Table]
D S UPPLEMENT E XPERIMENTS D. MAM UJO C O S UPPLEMENT E XPERIMENTAL R ESULTS
We plotted the predictive observations compared to actual observations in Figure 2 at the beginning of the learning process (iteration 1 ) and end of the learning (iteration 40 ).
We also show the results of iteration ten and iteration 20 in Figure 6.
By comparing with iterations 1 and iterations 40 in
Figure
[Table] we see that the iteration 40 has the smallest difference between predictive observation and true observation [Table] and the difference gets increased as the iteration goes to the earlier stage of the learning process.
So the predictive accuracy is improved incrementally with the learning progresses.
[Figure]
[Table]
Length of test window
[Table]
Length of history window
[Table] D. M ULTI -A GENT P ARTICLE E NVIRONMENT
We also test MAPSR into another environment, multi-agent particle environment (Lowe et al., 2017) [Table]
The agents are displaced into a 2-dimensional coordinate.
This environment does not assume that all agents have identical action and observation spaces.
We run experiments using a different number of agents on two environments, the predator-prey, and cooperative-push.
We use the same configuration in (Liu et al., 2020a).
The network has the same design as MAMuJoCo in section C.3, except we use
MLP with two hidden layers with the same 128 units respectively, so we do not repeat the description.
In Table
[Table]
we report the hyperparameters for multi-agent particle environments.
In the supplementary materials we provide the code and instructions.
E D ETAILED P ROOFS
We give the proof of Theorem 1 in Appendix E.1, and we first introduce the following Lemmas to prepare the proof.
Definition 2.
Let X 1 . . .
X k be independent random variables of dimensionality d X 1 . . .
d X k such that ∥ X k ∥ < t x k .
Let { ( x 1 j . . .
, x k j ) } Nj =1 be the N i.i.d samples from distribution of X 1 , . . .
, X k , the C X 1 :=
E [ X 1 X T 1 ] and (cid:98) C X 1 = 1 N (cid:80) Nj =1 x 1 j x T 1 j , and C X 1 X 2 := E [ X 1 X T 2 ] and (cid:98) C X 1 ,X 2 = 1 N (cid:80) Nj =1 x 1 j x T 2 j .
Also, we use the · ) , v ( · ) to denote the largest, smallest eigenvalue of a matrix.
Lemma 3 ((Tropp, 2015)) .
Let a j be a finite sequence of independent random, Hermitian matrices with dimension d .
Assume that 0 ≤ v ( a j ) and u ( a j ) ≤ L for each j .
Let S
=
(cid:80)
j a j , then for any η ∈ [0 , 1] , it follows that
[Equation]
Corollary 1.
Let X be a random variable, for any ϵ ∈ (0 , 1) such that N > t 2 x log(2 d X /ϵ ) v ( C X ) the following holds with probability at least 1 − ϵ
[Equation]
In other words, if N large enough, then (cid:98) C X and C X will be close enough.
Proof.
Define S j = 1 /Nx j x Tj .
Then it follows that u ( S j ) ≤ L = t 2 x /N and define ϵ := 2 d X e − σNv ( C X ) /t
2 x , which implies that σ = t 2 x log(2 d X /ϵ ) v ( C X ) N .
Then it follows from Matrix Chernoff
Inequality in Lemma 3 that P r ( v ( (cid:98) C X ) ≤ (1 − σ ) v ( C X )) ≤ ϵ
Lemma 4 ((Tropp, 2015)) .
A finite sequence { a j } of independent, random matrices with common dimensions a × b , and assume that E [ a j ] = 0 and ∥ a j ∥ ≤ L for each j , let S =
(cid:80)
j a j as a random matrix.
Let V ar ( S ) be the variance statistics such that V ar ( S ) =
[Equation]
{ (cid:13)(cid:13) [Equation] [Equation] } , then
[Equation]
Corollary 2.
With at least probability 1 − ϵ that
[Equation]
where L = t y t x + ∥ C Y X ∥ ≤ 2 t y t x and V ar = max { t 2 y ∥ C X ∥ , t 2 x ∥ C Y ∥} + ∥ C Y X ∥ 2 ≤ 2 t 2 y t 2 x
Proof.
Let X, Y be two random variables, and let a finite sequence { a j } of independent random matrices to satisfy a j =
y j x
Tj − C Y X .
So the a j will have dimensions d X × d Y .
Let random matrix
S =
(cid:80)
j a j .
It follows that E [ a j ] = 0 and ∥ a j ∥ =
(cid:13)(cid:13)
y j x Tj − C Y X (cid:13)(cid:13) ≤ ∥
y k ∥
∥ x k ∥ + ∥ C Y X ∥ ≤ t y t x + ∥ C Y X ∥
[Equation]
Similarly, (cid:13)(cid:13) E [ SS T ] (cid:13)(cid:13) ≤ N ( t 2 y ∥
C X ∥ + ∥ C Y X ∥ 2 ) .
By applying lemma 4, we have ϵ =
Pr ( ∥ S ∥ ≥
Nc ) ≤ ( d X + d Y ) e
[Equation]
[Equation]
+ Lc/ 3 (cid:19) and therefore, it implies that
[Equation]
N
Corollary 3.
For random variable X with dimensionality d X and ∥ X ∥ ≤ t x , with probability 1 − ϵ , it follows that
[Equation]
where L = t 2 x √ v ( C X ) + t x
Proof.
The proof is similarly to the the proof of corollary 2, define a j = (cid:80) − 1 / 2 X
[Equation]
1 / 2 X ,
[Equation]
= (cid:80) a j then it follows that E [ a j ] = 0 and ∥ a j ∥ ≤
[Equation]
Applying lemma 4 to get
[Equation]
) it follows that
[Equation]
Lemma 5.
For two random variables X, Y , let (
cid:98) C Y X
=
C Y X + ∆ Y X , and (cid:98) C X
= C X + ∆ X where
E [∆ Y X ] and E [∆ X ] are not necessarily zero and (cid:98) C X is symmetric positive semidefinite.
Define
A = C Y X C − 1 X and (cid:98)
A = (cid:98)
C Y X ( (cid:98) C X + λ ) − 1 .
Then it follows that:
[Equation]
Proof.
[Equation]
It follows that
[Equation]
For M 1 , by using facts U − 1 − V − 1 =
U − 1 ( V − U )
V − 1 and C Y X
=
C 1 / 2 Y PC 1 / 2 X , where P is a correlation matrix with ∥ P ∥ ≤ 1 ,
[Equation]
Corollary 4.
Let { ( x k , y k ) }
Nk =1 be i.i.d samples from two random variables
X, Y with dimensions d X and d Y and (uncentered) covariances C X and C Y .
Assume ∥ X ∥ ≤ t x and ∥ Y ∥ ≤ t y .
Define
A = C Y X C − 1 X and (cid:98)
A = (cid:98)
C Y X ( (cid:98) C X + λ ) − 1 .
For any ϵ ∈ (0 , 1) such that N > t 2 x log(2 d X /ϵ ) v ( C X ) the following holds with probability at least 1 − 3 ϵ [Equation] ≤ [Equation]
v
[Equation]
where
[Equation]
Proof.
It follows by applying Corollaries 1,2,3 to Lemma 5.
By union bound, each condition has probability 1 − ϵ , so the total events are bounded
[Equation]
Lemma 6.
For two random variables X, Y , let (
cid:98) C Y X
=
C Y X + ∆ Y X , and (cid:98) C X
= C X + ∆ X where
E [∆ Y X ] and E [∆ X ] are not necessarily zero and (cid:98) C X is symmetric but not positive semidefinite.
Define A = C Y X C − 1 X and (cid:98)
A = (cid:98) C Y X (cid:98) C X ( (cid:98) C 2 X + λI ) − 1 .
Then it follows that:
[Equation]
Proof.
[Equation]
For M 1 , by using facts U − 1 − V − 1 =
U − 1 ( V − U )
V − 1 and C Y X
=
C 1 / 2 Y PC 1 / 2 X , where P is a correlation matrix with ∥ P ∥ ≤ 1 , it follows that
[Equation]
Therefore,
[Equation]
Corollary 5.
Let { ( x k , y k ) }
Nk =1 be i.i.d samples from two random variables
X, Y with dimensions d X and d Y and (uncentered) covariances C X and C Y .
The E [∆ Y X ] and E [∆ X ] is not necessarily zero and C x is symmetric but not necessarily positive semidefinite.
Assume ∥ X ∥ ≤ t x and ∥ Y ∥ ≤ t y .
Define A = C Y X C − 1 X and (cid:98)
A = (cid:98) C Y X (cid:98) C X ( (cid:98) C
2 X + λ ) − 1 .
For any ϵ ∈ (0 , 1) such that N > t 2 x log(2 d X /ϵ ) v ( C X ) the following holds with probability at least 1 − 3 ϵ
[Equation]
where
[Equation]
[-(cid:107) (cid:107) ∆ X (cid:107) v ( C X ) 2 (1 − γ ) 
-]
Proof.
It follows by applying Corollaries 1,2,3 to Lemma 6.
Also by union bound, so the total events are bounded 1 − 3 ϵ
Theorem 2.
Grabner 1997 (3.3) (Grabner & Prodinger, 1997) Consider there are n independent copies X 1 , . . .
X n i.i.d negative binomial random variables, with parameters defined as NB ( b, p ) , and our goal is to calculate the expectation of the maximum of these N random variables E n = E { max( X 1 , . . . , X n ) } then we have following asymptotic solution [Equation]
[Equation]
(where F is a periodic C ∞ − function of period 1 and mean value 0 whose Fourier-coefficients are given by ˆ F ( k ) = −
[Equation]
for k ∈ Z \ { 0 } , and q
= − p )
We omit the proof, interested readers could go to (Grabner & Prodinger, 1997) for details.
E. P ROOF OF T HEOREM
We first prove the bound for ∥ (cid:98) q i,j − q i,j ∥ .
Proposition 2.
Let π Θ be a data collection policy and H is the range of π Θ on joint histories.
If
Equation 2 used, then for all h ∈ H and any ϵ ∈ (0 , 1) , (cid:13)(cid:13)(cid:98) q i,j ( ψ h ) − q i,j ( ψ h ) (cid:13)(cid:13) is bounded as below with probability at least 1 − 3 ϵ .
[Table]
where ∆ 1 follows the bound 44 and ∆ 2 follows the bound 42, and γ = t 2 Aj log(2 d Aj /ϵ ) v (cid:16) C
[Table]
Proof.
Let T i,j is the tensor such that C ψ oi ψ aj | ψ hi,j = T i,j × h ψ hi,j , and U i,j is the tensor such that C ψ aj | ψ hi,j = U i,j × h ψ
hi,j and for simplicity without loss meaning, we use C ψ aj | ψ hi,j to denote
C ψ aj ψ aj | ψ hi,j .
Then we have
[Equation]
We finish the above proof by proofing the ∥ T i,j − T i,j ∥ and ∥ U i,j − U i,j ∥ are bounded by using
Corollary 4.
[Equation]
where
[Equation]
and
[Equation]
where
[Equation]
Then using the equation 2 and corollary 5 to obtain the bound for q i,j
[Equation]
where ∆ 1 follows the bound 44 and ∆ 2 follows the bound
[Equation]
and γ = t 2 Aj log
[Equation]
Now we start to prove our theorem 1.
Proof.
The equation 1 says:
[Equation]
here we assume the agents are homogeneous, in other words, each pair of { ( o i , a i ) } ni =1 coming from the same spaces O , A .
They are permutation invariant and their identities do not matter.
Thus, the bound of q i,j is invariant to agents.
Under the assumption of static fully complete graph, for any agent q i = (cid:80) nj =1 q i,j , thus
[Equation]
E. P ROOF OF L EMMA
Here we prove Lemma 1.
Proof.
Equation 4 says
[Equation]
where I i,j is an indicator function to denote if two agents are connected.
Under the assumption that the static non-complete graph has maximum of number of degrees k and the agents are homogeneous, we have (cid:80)
j I i,j < = k .
Thus,
[Equation]
E. I NFORMAL P ROOF OF L EMMA
Here we make some intuitions for the proof of Lemma 2.
As we already give the proof sketch in our main paper.
Lemma 2 is a direct application of Theorem 2.
Here our random variable
J 1 , · · · , J n ∼ NB ( r, p ) , where { ( J i ) } ni =1 represents the number of time points node i needs before it meets node j number of r times.
For the complete static graph, we need at least N sample for the bound in equation 3 to be valid; in other words, we need the trajectory to run at least N time points to collect enough data to estimate our conditional operator accurately q i,j .
For the dynamic graph, each time t , the two nodes are randomly connected with probability p , if it connects, then we can obtain a valid sample to estimate q i,j ; if not, then we skip to the next time step.
For the dynamic graph node i , if we take the union set of the nodes i connected over the trajectory path, then the union set could form a static complete graph.
We say the two graphs are equivalent.
The number of time points needed by i until the N th connection with j for each of the pairs ( i, j ) follows the same distribution J ∼ NB ( N, p ) .
Then we are interested in the expectation of the maximum of J 1 , . .
, J n − 1 , we denote it as J { 1 ,...,n − 1 } .
E { J { 1 ,...,n − 1 } } means on average, how many time points ( N ′ )
we need for all nodes other than i at least meets N times with i .
Obviously, this N ′ >
= N since p ∈ [0 , 1] .
The calculation of this expectation is solved by (Grabner & Prodinger, 1997).
And we also put their result in Theorem 2.