Lifting in Multi-agent Systems under Uncertainty
Abstract
A decentralised partially observable Markov decision problem (DecPOMDP) formalises collaborative multi-agent decision making.
A solution to a DecPOMDP is a joint policy for the agents, fulﬁlling an optimality criterion such as maximum expected utility.
A crux is that the problem is intractable regarding the number of agents.
Inspired by lifted inference, this paper examines symmetries within the agent set for a potential tractability.
Speciﬁcally, this paper contributes (i) speciﬁcations of counting and isomorphic symmetries, (ii) a compact encoding of symmetric DecPOMDPs
as partitioned DecPOMDPs, and (iii) a formal analysis of complexity and tractability.
This works allows tractability in terms of agent numbers and a new query type for isomorphic DecPOMDPs.
INTRODUCTION
Decentralised partially observable Markov decision problems (DecPOMDPs) allow for formalising an ofﬂine decision making problem for multiple agents collaborating for a joint reward in a stochastic environment.
However, in the worst case, computation time in DecPOMDPs depends exponentially on the number of agents, which means that scalability is limited and crude approximations might be necessary to solve problem instances at the expense of accuracy.
There are applications, though, that require accuracy.
One such application lies in nanoscale medical systems in which a swarm of indistinguishable nanodevices with limited online computation power collaborate on, e.g., a treatment or diagnosis [Lau et al., 2019].
The number of agents goes into the hundreds of thousands, making the exponential dependency punishing.
The question is what to do when faced with large agent sets and whether we are able to do something about the exponential dependency.
In probabilistic inference, the notion of lifting refers to efﬁciently handling sets of indistinguishable objects using representatives [Poole, 2003]. Lifting enables reducing the theoretical dependency from exponential to polynomial, making the problem of probabilistic inference tractable, w.r.t.
these set sizes [Niepert and Van den Broeck, 2014].
Therefore, this paper examines indistinguishability within the agent set and its potential for making DecPOMDPs
tractable regarding agent numbers, i.e., solving them no longer depends on agent numbers exponentially.
The paper analyses two types of symmetries for indistinguishability inspired by the main lifting tools of counting and isomorphism in lifted variable elimination (LVE)
[Taghipour, 2013]. The results rely on partitioning the set of N agents into K partitions with K (cid:28) N , K typically being in the order of magnitude 1 . In nanoscale systems, e.g., the standard setting comprises a small number of agent types (e.g., K = 4 ) of which large numbers of indistinguishable agents are built. Speciﬁcally, this paper contributes (i) an analysis of counting and isomorphic symmetries and their connection, (ii) an encoding of those symmetries as partitioned DecPOMDPs, and (iii) a formal analysis and discussion.
The last part covers representation size, policy space, evaluation cost, and expressiveness.
The results show that counting requires no further assumption but indistinguishability and leads to a polynomial dependence on the agent numbers except for the policy space.
The isomorphic case involves a stricter symmetry, which allows for tractability in agent numbers.
In addition, it enables using existing solution methods and deﬁning new query types.
Speciﬁcally, the isomorphic case allows for formalising a new optimisation problem, asking for the number of agents needed to satisfy a particular goal.
Lifting has been successfully used for query answering [Kersting et al., 2009, Van den Broeck et al., 2011, Braun and
Möller, 2018, Holtzen et al., 2019] and online decision making [Nath and Domingos, 2009, Apsel and Brafman, 2011, Gehrke et al., 2019a,b], which uses decision and utility nodes in a relational or ﬁrst-order probabilistic graphical model (PGM).
In ofﬂine decision making, lifting has been used in calculations for relational descriptions of the state space in (PO)MDPs: First-order MDPs (FOMPDs)
[Boutilier et al., 2001] have a representation based on the situation calculus [McCarthy, 1963].
Factorised FOMDPs additionally assume a factorised representation of the state space [Sanner and Boutilier, 2007].
Sanner and Kersting [2010] use lifting for pruning indistinguishable policies in
FOPOMDPs.
Open-universe FOPOMDPs have an openuniverse assumption about the ﬁrst-order representation, using Bayesian logic as a basis [Srivastava et al., 2014].
Another ﬁrst-order representation is independent choice logic that also allows for a set of agents [Poole, 1997].
To the best of our knowledge, we are the ﬁrst to consider lifting for the agent set.
We leave the state space in this paper as is and look at its structuring as an exciting avenue for future work.
As this paper covers an analysis of the theoretical problem, we do not cover DecPOMDP solution methods as related work.
Please refer to Oliehoek and Amato [2016].
This paper is structured as follows: Section 2 recaps DecPOMDPs
and their complexity results.
Section 3 covers the symmetries, the compact encodings in partitioned DecPOMDPs, and the formal analysis, followed by a discussion.
Section 4 concludes the paper.
BACKGROUND AND NOTATION
This section recaps DecPOMDPs.
It is set to be selfcontained, but familiarity with POMDPs helps.
2.1 DECENTRALISED POMDPS
A DecPOMDP refers to a set of agents working jointly towards a common goal.
The underlying principle is that of maximum expected utility (MEU) in which a utility function represents preferences over states.
We look through a
PGM lens for deﬁnitions, which are based on the work by
Oliehoek and Amato [2016] and Russell and Norvig [2020], using random variables, V , which can take discrete values, referred to as range, ran ( V )
= { v 1 , . .
. , v m } .
So-called decision random variables have actions as ranges.
Setting V to a value v ∈ ran ( V ) (an event ) is denoted as V = v or v for short if V is clear from its context.
We denote sequences over a discrete time interval [ t s , t e ] with subscript t s : t e , e.g., ( V t s , . .
, V t e ) =
V t s : t e .
We use ◦ to denote concatenation.
Deﬁnition 1.
A model M is a tuple ( I , S, A , T, R, O , Ω) ,
I a set of N agents, • S a random variable with a set of states as range, •
A =
{ A i } i ∈ I a set of decision random variables
A i , each with a set of local actions as range, with ran ( A ) =
× i ∈ I ran ( A i ) the set of joint actions , • T ( S (cid:48) , S, A ) =
P ( S (cid:48) | S, A ) a transition function, with T ( S 0 , ., . )
=
P ( S 0 ) referring to a state prior, • R ( S, A ) a reward function, • O =
{ O i } i ∈ I a set of random variables
O i , each with a set of local observations as range, with ran ( O ) =
× i ∈ I ran ( O i ) the set of joint observations , and • Ω( O , S ) =
P ( O | S ) a sensor function.
Optional are a ﬁnite horizon τ , a discount factor γ ∈ [0 , 1] (default 1 ), and an error margin (cid:15) > 0 .
Each agent i ∈
I has a local policy
π i : ran (( O i, 0: t ))
(cid:55)→ ran ( A i ) mapping observation histories o i, 0: t , t ≤ τ − 1 , to actions a , with π =
( π i )
i ∈ I a joint policy .
The semantics of M is given by all possible joint policies, referred to as Π M .
The
DecPOMDP asks for the joint policy π ∗ that yields the maximum expected utility U M ( π ∗ ) in M (with horizon τ ):
[Equation]
where U M ( π ) is calculated recursively over τ steps and weighted according to the prior T ( S 0 , ., . ) :
[Equation]
with U π M ( s τ − 1 , o 0: τ − 1 ) =
γ τ − 1 R ( s τ − 1 , π ( o 0: τ − 1 )) as the stopping point and o 0:
t +1 = o 0: t ◦ o t +1 .
Each agent has its own set of actions and observationswhereas state and reward are joint.
The joint state is usually assumed to not be fully observable, even if combining all local observations.
If the joint state were observable, the
DecPOMDP would simplify to a DecMDP.
The joint reward function encodes that the agents receive a reward as a team.
If the joint state and reward function can be split up into independent subspaces per agent, the DecPOMDP decomposes into a set of POMDPs that can be solved individually.
In this paper, we do not let the problem dissolve into subproblems that can be solved independently.
Rather, we keep a joint state and reward function, while the agent set shows certain degrees of independence.
The generalisation of a DecPOMDP is a partially observable stochastic game (POSG) in which each agent has its own reward function R i , which may conﬂict with other agents’ rewards.
The sensor function may also depend on the joint action, which does not change the problem in a major way [Russell and Norvig, 2020].
A naive solution approach is to solve Eq.
(1) directly, i.e., to generate all possible joint policies, evaluate them according to Eq.
(2), and pick the policy with highest value.
2.2 COMPLEXITY RESULTS
DecPOMDPs depend exponentially on N , the number of agents, in the worst case, which manifests itself in the space requirements of a model M of Def. 1, the size of the policy space spanned by M , i.e., | Π M | , and the cost of evaluating a joint policy π , given a horizon τ .
The size of the policy space together with the cost of evaluating a policy in the space provides an upper bound on any runtime as it bounds the effort of the naive solution approach to the DecPOMDP.
Formally, the sizes T , R , and O of the functions T ( S (cid:48) , S, A ) , R ( S, A ) , and Ω( O , S ) , respectively, lie in T
[Equation]
O
[Equation]
with s = | ran ( S ) | , a = max i ∈{ 1 ,...,N } | ran ( A i ) | , and o = max i ∈{ 1 ,...,N } | ran ( O i ) | , which are all exponential in N .
They follow directly from the range sizes of the inputs.
The cost C of evaluating a joint policy and the size P of the policy space lie in [Oliehoek and Amato, 2016]:
[Equation]
Both depend on N exponentially.
The complexity of C comes from evaluating in each of the s states a joint policy of size o
Nτ where o τ bounds the size of an agent’s observation history.
The complexity of P comes from each possible action at the end of each possible observation history, of which there are ( o τ − 1) /
( o − 1) (histories form trees: geometric series).
SYMMETRIC DECPOMDPS
The idea of partitioning the agent set follows the insight that in systems with regularities in their agent set I , e.g., through types, partitions emerge, in which agents have the same available actions, observations, and behaviour, i.e., they are indistinguishable.
Borrowing from the ﬁeld of lifted inference, we refer to these DecPOMDPs
as symmetric since the indistinguishable behaviour materialises itself by symmetries in the transition, sensor, and reward functions.
This section formalises indistinguishability through symmetries and analyses their relationship and effect on complexity.
We introduce partitioned DecPOMDPs
as a framework.
3.1 PARTITIONING OF THE AGENT SET
For lifting to apply, a DecPOMDP has to fulﬁl a basic requirement: The set of agents I partitions into K sets I k , i.e., I = (cid:83) K k =
I k , I k
(cid:54) = ∅ , and ∀ k, l ∈ { 1 , . . .
, K } , k (cid:54) = l : I k
∩ I l = ∅ , and for each I k , it holds for all pairs i, j ∈ I k :
[Equation]
Given Eq.
(6), it is sufﬁcient to keep K decision and observation random variables instead of N variables
A i and O i .
To still represent the agents, one may add a logical variable X k that represents all agents in I k to A k and O k , forming parameterised random variables (PRVs) A k ( X k ) and
O k
( X k ) , where the individual agents no longer appear explicitly but hidden behind the logical variable.
See, e.g., the work by Taghipour [2013] for details on PRVs.
Equation (6) is a necessary and sufﬁcient condition for using PRVs to list observations and actions available for a partition but it is only a necessary condition for lifting:
Further symmetries have to hold in the transition, reward, and sensor functions for lifting to apply, of which we investigate two types inspired by the lifting tools of counting and isomorphism.
Before we turn to lifting, we set up partitioned
DecPOMDPs
as the general framework for DecPOMDPs with partitioned agent sets fulﬁlling Eq.
Deﬁnition 2.
A partitioned model ¯ M is a tuple (¯ I , ¯ S, ¯ A , ¯ T, ¯ R, ¯ O , ¯Ω) , with
• ¯ I
a partitioning { I k } Kk =1 of agents, n k
= |
I k | and | ¯ I | = (cid:80) k n k = N , • ¯ S a random variable with a set of states as range, • ¯ A
= { ¯ A k } Kk =1 a set of decision random variables ¯ A k , each with possible partition actions as range, with ran ( ¯ A ) =
× Kk =1 ran ( ¯ A k ) the set of joint actions, • ¯ T ( ¯ S (cid:48) , ¯ S, ¯ A ) =
P ( ¯ S (cid:48) | ¯ S, ¯ A ) a transition function, • ¯ R ( ¯ S, ¯ A ) a reward function, • ¯
O
=
{ ¯ O k } Kk =1 a set of random variables ¯
O k , each with a set of partition observations as range, with ran ( ¯ O ) =
× Kk =1 ran ( ¯ O k ) the set of joint observations, and • ¯Ω( ¯ O , ¯ S ) =
P ( ¯ O | ¯ S ) a sensor function.
Optional are a ﬁnite horizon τ , a discount factor γ ∈ [0 , 1] (default 1 ), and an error margin (cid:15) > 0 .
Each partition I k has a partition policy ¯ π k : ran ( ¯ O k ) (cid:55)→ ran ( ¯ A k ) mapping observation histories ¯
o k, 0: t , t ≤ τ − 1 to a partition action ¯ a k , with ¯ π =
(¯ π k )
Kk
=1 a joint policy .
The semantics of ¯ M is given by all possible joint policies Π ¯ M .
The partition
DecPOMDP asks for a joint policy ¯ π ∗ that maximises the expected utility as deﬁned in Eq.
(1) with Π ¯ M , ¯ π , ¯ a , ¯ o as Π M , π , a , o .
This formalism already bears beneﬁts since it enables domain-speciﬁc compound actions or observations for partitions.
It also allows for the same action and observation set to be available to each partition agent (Eq.
(6)), i.e., ¯ A k =
A k ( X k ) =
{ A k ( x k, 1 ) , . . . ,
A k ( x k,n k ) } with A k ( x k,i ) the decision random variable for agent i in partition
I k (analogously for ¯ O k ).
If ∀ k : n k = 1 , i.e., N = K , ¯ M is ground, i.e., ¯ M = M .
The complexity results as given in Eqs.
(4) and (5) for DecPOMDPs of Def.
1 translate directly to partitioned DecPOMDPs of Def. 2, referring to the range sizes of partition actions and observations instead.
Speciﬁcally, the respective function sizes ¯ T , ¯ R , and ¯ O as well as the cost ¯ C and policy space size ¯ P in a partitioned DecPOMDP lie in
[Equation]
with ¯ s = | ran ( ¯ S ) | , ¯ a = max k ∈{ 1 ,...,K } | ran ( ¯ A k ) | , and ¯ o = max k ∈{ 1 ,...,K } | ran ( ¯ O k ) | .
Assuming that M and ¯ M model the same state space, then s = ¯ s .
With N = K ,
Eqs.
(4) and (5) and Eqs.
(7) and (8) coincide.
With nonexistent or unused symmetries, the same holds even with Eq.
Next, we specify two types of symmetries for T , R , and Ω to fulﬁl, specialise the partitioned DecPOMDP accordingly for a compact representation, and discuss their effect on the complexity results, starting with the more general symmetry.
3.2 COUNTING SYMMETRY
The basic assumption of lifting is indistinguishability between certain objects or individuals, or as in this paper, agents in DecPOMDPs.
Next to the partitioning with Eq.
(6), another consequence of the assumption is that it does not matter which agent performs a particular action within a partition, only how many (analogously for observations).
E.g., if there are 10 agents in a partition of which ﬁve agents do one action and the rest do another action, then there exist (cid:0) 10! 5!5! (cid:1) = 252 ways to have the 10 agents perform the 5 and 5 actions with the same outcome (multinomial coefﬁcient, refer to Taghipour [2013] for details).
The effect is that the transition, reward, and sensor functions map to identical values for these permutations of inputs within partitions.
In the example, all of those 252 inputs map to some number p .
We refer to this symmetry as counting symmetry .
Given counting symmetry, the question becomes how one can use it for a compact encoding in partitioned DecPOMDPs.
In the example, it would be enough to store the number for each action, e.g., in a histogram of absolute numbers like [5 , 5] , and map it to p .
This type of intra-function symmetry has also been spotted in probabilistic inference and compactly encoded using histograms as range values of a so-called counting random variable (CRV, refer to Milch et al.
[2008] for details).
Formally, there are r k = | ran ( A k ) | different possible actions per partition
I k , meaning, a partition action can be encoded using a histogram: { [Equation] a l , n l ) } r k l =1 , a l ∈ ran ( A k ) , n [Equation]
N 0 , n
[Equation] = [Equation] n
[Equation]
9) or [ n 1 , . . . , n r k ] for short. CRVs bring us back to encoding identical action and observation sets with logical variables. Given a PRV such as A k ( X k ) , we count how often a particular range value is assigned to any grounding of A k ( X k ) , leading to a CRV # X k
[ A k ( X k )] , with histograms of Eq.
(9) as a range (analogously for O k
( X k ) ).
Hence, in the partitioned model ¯ M of Def. 2, ¯ A k =
# X k
[ A k ( X k )] and ¯ O k =
# X k
[ O k ( X k )] with the domain of X k being dom ( X k ) =
I k .
A joint action then turns into a = { h
[Equation]
The same holds for joint observations.
The transition, reward, and sensor function, T , R , and Ω respectively, receive CRVs as inputs:
[Equation]
which allows for using a single mapping of a histogram to a number, e.g., [5 , 5] (cid:55)→ p , for permutations of inputs.
We refer to this instantiation of a partitioned DecPOMDP (model ¯ M ) as a counting DecPOMDP (model ¯ M c ).
The following results show that each counting model ¯ M c has an equivalent ground model M exhibiting counting symmetry, in which the same solutions are optimal.
This leads us to the ﬁrst result on agent tractability: A counting DecPOMDP depends on agent numbers exponentially in terms of the policy space but we achieve polynomial dependence in terms of representation size and cost.
Lemma 1.
A counting model ¯ M c has an equivalent grounded model M with counting symmetries, i.e., gr ( ¯ M c )
=
M .
Proof.
Lifting is based on the grounding semantics [Sato, 1995], meaning an equivalence between lifted and ground operations.
Thus, it also applies to CRVs.
Grounding ¯ M c means replacing each CRV #
X k
[ A k ( X k )] with decision random variables { A k ( x k,i ) } , i ∈ I k , analogously for observations, grounding the CRVs in the transition, reward, and sensor functions using so-called expansion
[Taghipour, 2013], and setting S to ¯ S , yielding a grounded model.
Because of expansion inverting the encoding of a counting symmetry in CRVs, gr ( ¯ M c ) exhibits counting symmetries.
Theorem 1.
The optimal solution ¯ π ∗ of a counting DecPOMDP in model ¯ M c is also optimal in model gr ( ¯ M c ) .
Proof.
By Lemma 1, ¯ M c is equivalent to gr ( ¯ M c ) .
The difference when solving the problems on ¯ M c and gr ( ¯ M c ) with counting symmetry, respectively, lies in the policies searched.
Namely, a search for ¯ π ∗ in ¯ M c skips those policies that have permutations of agents perform the same actions and perceive the same observations, which does not have an effect in the transition, reward, and sensor functions, and therefore have the same expected utility.
Thus, the search for ¯ π ∗ leads to a policy that is also optimal in gr ( ¯ M c ) .
Theorem 2.
A counting model allows for representation and cost to depend polynomially on N (number of agents).
Proof.
The worst case complexities in terms of function sizes, evaluation cost, and policy space size of a counting
DecPOMDP of model ¯ M c with an equivalent ground DecPOMDP of model M = gr ( ¯ M c ) lie in
[Equation]
with s = | ran ( S ) | , n = max k n k , a = max k | ran ( A k ) | , and ¯ o = max k | ran ( O k ) | , k ∈ { 1 , . . .
, K } .
Because of the equivalence, ¯ s = s .
For ¯ a (and ¯ o analogously), it holds that ¯ a = max k ∈{ 1 ,...,K } | ran (# X k
[ A k ( X k )]) | .
The range size of a CRV # X
[ R ( X )] with | ran ( R ) | = r and | dom ( X ) |
= d is given by (cid:0) d + r − 1 r − 1 (cid:1) ≤ d r [Milch et al., 2008].
Therefore, ¯ a is replaced by n a and ¯ o by n o in Eqs.
(7) and (8).
As we assume K (cid:28) N , n is not much smaller than N .
As such, the policy space still depends exponentially but representation size and cost only polynomially on n < N .
The n represents the largest partition size, which is still large given K is assumed to be small.
An effect is that the naive solution of a brute-force search through the policies for an exact solution still depends exponentially on n and therefore, the problem remains intractable even with counting symmetry.
Solution techniques will have to do some further tricks, use heuristics, or stochastic methods to not depend on n exponentially.
Although the search space remains large here, it is not surprising that we cannot solve the combinatorial problem at once and it opens the door for interesting future work for a stochastic search in histogram spaces.
However, there is also a positive side to this result as we do achieve a polynomial representation size even in the worst case.
Next, we consider isomorphic symmetry, which is actually able to render the problem tractable in agent numbers.
3.3 ISOMORPHIC SYMMETRY
This part starts with the model speciﬁcation to see the origin of this symmetry and then infers the symmetry from it.
A natural next step in lifting is to consider PRVs as inputs instead of CRVs in the transition, reward, and sensor functions shown in Eqs. (10) to (12).
The partitioned DecPOMDP and model is specialised with the PRV versions for actions and observations, i.e., ¯ A k =
A k ( X k ) and ¯ O k
=
O k ( X k ) with dom ( X k ) =
I k , yielding the following transition, reward, and sensor functions:
[Equation]
We call this form of partitioned DecPOMDP (model ¯ M ) an isomorphic DecPOMDP (model ¯ M i ).
The grounding semantics also applies: ¯ T , ¯ R , and ¯Ω turn into combinations over all groundings D = ×
Kk =1 dom ( X k ) , each grounding with identical mappings as of ¯ T , ¯ R , and ¯Ω .
E.g., the result for ¯ T is ¯ T ( ¯ S (cid:48) , ¯ S, ◦ Kk =
A k ( x k,i k )) for all groundings ( x 1 ,i 1 , . . .
, x K,i K ) ∈ D , which are combined using multiplication.
Rewards are additive, i.e., combined using summation, which implies a preferential independence [Russell and Norvig, 2020] over which agent incurs a reward, which is reasonable under indistinguishability.
A joint distribution factorising into such identical functions is the main symmetry used in LVE, encoded using logical variables in PRVs.
Grounded out, subgraphs coming from the same parameterised function are isomorphic.
Therefore, we refer to this symmetry as isomorphic symmetry .
Isomorphic models translate into counting models, which we show next, meaning Theorem 1 also holds in this case.
Lemma 2.
An isomorphic model ¯ M i has an equivalent counting model ¯ M c .
Proof.
A model with isomorphic symmetry has a corresponding isomorphic DecPOMDP based on the grounding semantics with PRVs as inputs in the functions ¯ T , ¯ R , and ¯Ω .
All PRVs are count-convertible as each contains only one logvar that does not occur in any other PRV and the domains are mutually exclusive, automatically fulﬁlling the preconditions of count-conversion.
For ¯ T and ¯Ω , standard count-conversion can be used [Taghipour, 2013].
Because of ¯ R having additive semantics, an additive but otherwise unchanged count-conversion is necessary; see the supplementary material for a formal deﬁnition.
The countconversions result in functions of Eqs.
(10) to (12), i.e., a counting model.
Corollary 2.1.
If a model M exhibits isomorphic symmetry, it also exhibits counting symmetry.
Proof.
By the grounding semantics, such a model M can be encoded in an isomorphic model ¯ M i , which has an equivalent counting model by Lemma 2, which has a ground model with counting symmetry by Lemma 1.
The reverse is not necessarily true: One can build a counter example by turning an isomorphic model into a counting one and then changing the mapped value of one input sequence, destroying the factorisation.
Thus, isomorphic symmetry is a stricter symmetry but it allows for cutting down the policy space.
The reason lies in the PRV version implying a form of independence on the agent level.
Each agent within a partition operates independent of other agents in the partition.
As the agents of a partition are indistinguishable, each agent has the same optimal local policy in this setting.
Thus, one only needs to consider a representative agent and search through its possible actions and observations of A k and O k to ﬁnd its optimal policy, which also applies to all other agents of the partition.
Therefore, we can trim the search space to cover only ran ( A k ) for isomorphic DecPOMDPs
instead of ran (# X k
[ A k ( X k )]) for equivalent counting DecPOMDPs.
Formally, we have the following lemma.
Lemma 3.
In an isomorphic DecPOMDP with model ¯ M i , partition policies are deﬁned over ran ( A k ) and ran ( O k ) .
Proof sketch.
The model ¯ M i is basically a parameterised probabilistic sequential model with three parametric factors, i.e., factors with PRVs as inputs, namely, ¯ T i , ¯ R i , and ¯Ω i .
The inputs are time-stamped by t , i.e., S t , ¯ A t , and ¯ O t .
A policy prescribes an action a to perform for each possible observation history o 0: τ of length τ for each agent.
To determine a joint action a for a joint observation history o 0: τ , o 0: τ becomes evidence in ¯ M i .
Since there are only a bounded number of histories possible in a partition, the agents in a partition can be partitioned again based on the histories.
That is o 0: τ is turned into lifted evidence and the partitions are split on the evidence using the split operator [Taghipour, 2013].
So, each of the K partitions has its own logical variable X k,j with a domain that is disjoint from all other domains and the same evidence for the agents of X k,j .
Next, we unroll the model for τ time steps, i.e., instantiate the split functions by replacing t with each value in { 0 , . .
. , τ } , to handle the evidence over τ time steps.
Then, the model absorbs the evidence in each sub-partition, eliminating O k ( X ) from each sub-partition and each time step, using the lifted absorption operator [Taghipour, 2013].
For the different time steps, we now have ¯ T i ( ¯ S (cid:48) , ¯ S, ◦
Kk =1 ◦ j
A k ( X j )) , ¯ R i ( ¯ S, ◦
Kk =1 ◦ j A k
( X j )) , and ¯Ω i ( S ) where j iterates over all existing sub-partitions of a partition.
Next, we follow the semantics of parameterised models to show that considering only ran ( A k ) is correct.
The semantics prescribe the following: (i) Ground the model.
(ii)
Join all instantiated factors into one large factor.
(iii) Sum out the state variables S t , S (cid:48) t for all instantiated t .
(iv) Pick the MEU actions for the different observation histories.
The task in terms of the proof is to show that the same arg max actions are chosen for agents from the same sub-partition.
Step (i) makes every constant covered by a PRV explicit by having each constant appear explicitly in a grounding (in contrast to the X , which hides the constants).
Making the constants explicit, i.e., no longer hiding them behind an X , can also be achieved by turning each PRV into a CRV, where each constant appears in the counts of the histograms, after which no constants are hidden anymore.
So, we use countconverting instead of grounding in (i) for the purpose of this proof.
(Count-conversions are applicable as the domains are disjoint with no inequalities between logical variables and no PRVs or logical variables recurring in the same function.)
Then, we need to show that peak-shaped histograms (in Eq.
(9), n l = n k , for l (cid:48)
(cid:54) =
l : n l (cid:48) = 0 ) are chosen as arg max actions, which means that agents of the same subpartition perform the same action.
To show that only peak-shaped histograms are relevant, we look at the count-conversions of the ﬁrst step.
Consider a minimum example of a function φ i ( S, A ( X )) and its ground and count-converted versions φ and φ c with | dom ( X ) |
= 2 :
S
A ( X ) φ i sapsapsapsap
S
A ( x 1 )
A ( x 2 ) φ saapsaap 1 · psaap 2 · psaapsaapsaap 3 · psaap 4 · psaap {+
+}
S
# X
[ A ( X )] φ c s[0 , 2] p 02 · ps[1 , 1] p 12 · ps[2 , 0] p 22 · ps[0 , 2] p 04 · ps[1 , 1] p 14 · ps[2 , 0] p 24 · p
The versions φ and φ c show exactly how a CRV is simply another encoding of the same information stored in a grounded version.
It also highlights that an isomorphic symmetry is always also a counting symmetry.
The important part for the proof is that whatever the actual values of the different p ’s are, when the PRVs are count-converted (or grounded), the maximum value for a given state s will always occur where the exponent e is largest as one of the p ’s will be the largest and it will bring the most to take this p to the power of the largest number possible.
And the largest exponent possible occurs in a peak-shaped histogram where one position takes all the available elements.
With the additive semantics of rewards, the rewards are added up (and not multiplied) with the exponent being a factor but the argument is the same: the maximum value occurs where the largest p meets the largest e , which occurs in a peak-shaped histogram.
Non-peak-shaped histograms can only catch up to peakshaped histograms, namely, if p ’s are equal.
Then, the solution will not be unique.
As we are not interested in all solutions, focusing on peak-shaped histograms is still correct.
So, we have that one of the inputs with a peak-shaped histogram maps to the largest value, i.e., a peak-shaped histogram is the arg max action at the moment.
In the functions of a DecPOMDP model, we have K PRVs to count-convert.
However, the result still remains the same:
For each of the count-conversions individually, it holds with the arguments above that the arg max action is one of the peak-shaped his- tograms.
Since the count-conversions are applied iteratively, each CRV still has the maximum value where a peak-shaped histogram occurs, and these maximum values occur for those inputs in which the previous count-conversion had maximum values, which were also peak-shaped.
So, after count-converting every logical variable in ¯ T
i and ¯ R i of each sub-partition ( ¯Ω i does not contain any logical variable as we eliminated them using the observation histories as evidence), the lines with the largest values will be those where peak-shaped histograms occur together.
Step (ii) says to join all functions.
The largest values occur for each possible state where peak-shaped histograms meet as they bring with them the largest current p .
Step (iii) requires summing out all non-decision CRVs, i.e., all state variables over the different time steps.
Summing out adds up values that occur for the different states given the same input sequence of the CRVs, which again does not change any arg max actions as non-peak-shaped inputs cannot catch up with those peak-shaped inputs, where the largest values reside and are now added up.
Step (iv) is already the decision on the arg max action, where we pick the input mapping to the largest value, which has peak-shaped histograms by the arguments above.
Peakshaped histograms prescribe the same action for all agents in a sub-partition, meaning it is enough to consider ran ( A k ) in each sub-partition.
The remaining part of the proof has to show that it is also enough to consider ran ( O k ) and to not consider each possible sub-partition as we have done so far.
Each sub-partition has peak-shaped histograms as arg max actions.
An effect is that the arg max actions do not depend on partition sizes.
Only the size of the peak changes.
Therefore, for all the other possible observation histories o (cid:48) 0: τ , we again get subpartitions that have observation histories as evidence like with o 0:
τ but with different sub-partition sizes.
However, the result in terms of the arg max actions per sub-partition will be the same.
Therefore, it is enough to consider what happens for one representative agent for each possible observation history of that agent as looking at n k,j agents in a sub-partition does not change the outcome compared to looking at one agent.
That is, considering only ran ( O k ) and the histories possible with ran ( O k ) is sufﬁcient.
This concludes our proof.
Corollary 3.1.
Partition sizes do not inﬂuence a decision, but only the overall expected utility in isomorphic models.
The proof of Lemma 3 only works because the CRVs come from PRVs.
If one would have a CRV in the original model, a non-peak-shaped histogram could get such a high reward that it will outweigh all others and be the arg max action.
Corollary 3.1 follows directly from the proof.
Thus, one can ignore partition sizes during policy evaluation if only interested in a ranking.
Lemma 3 brings us to our second main result, which is an overwhelmingly positive one: Isomorphic
DecPOMDPs allow for agent tractability and a complete independence of agent numbers if one ranks policies.
Theorem 3.
An isomorphic model leads to tractability regarding the number of agents N .
Proof.
Complexity-wise, Lemma 3 has the effect that ¯ [Equation] | and ¯ o = max k | ran [Equation]
O k ) | , k ∈ { 1 , . .
, K } in Eqs.
(7) and (8), which is equal to the sizes a and o of Eqs.
(4) and (5).
Thus, the sizes ¯ T i , ¯ R i , and ¯ O i , the evaluation cost ¯ C i , and the policy space size ¯ P i lie in:
[Equation]
[Equation]
16) with s = | ran ( S ) | ( = ¯ s ) and n = max k n k .
The log n for ¯ C
i appears when including partition sizes into the evaluation (they occur as exponents and exponentiation has a complexity of log 2 n ).
Then, the naive solution of a bruteforce search through the policy space no longer depends on
N exponentially, making the problem tractable for models with isomorphic symmetries regarding agent numbers.
3.4 DISCUSSION
This part discusses expressiveness, partition-level symmetries and independence, identifying symmetries, and the connection to LVE, with a concluding note on going from joint policy to agents in symmetric DecPOMDPs.
Expressiveness Models with counting symmetry allow for encoding outcomes based on proportions of agents acting or observing.
And even though the worst case shows an exponential dependence on the number of agents N , domainspeciﬁc knowledge may help to prune the search space as only certain limits may apply (e.g., at least 30 , 000 actions of a certain type) or histograms in increments of 1 are not sensible (cf.
[0 , 50 . 000] and [1 , 49 .
Isomorphic models cannot express proportions, which are not always necessary.
If each agent’s action only has local effects (consider navigating on a grid), the model may be isomorphic.
A joint reward may only depend on S , i.e.,
R ( S ) , which is independent of any action and therefore automatically isomorphic.
If one can assume that partition agents work in close proximity or in a comparable setup, observations may often be identical, even if noisy, opening up future work on approximating observations.
Consider the well-known DecTiger benchmark [Nair et al., 2003] as an example.
Please ﬁnd a full speciﬁcation and discussion in the supplementary material. Agents have the same set of actions and observations available.
The functions T , R , and Ω exhibit a counting symmetry, whereas only Ω is also isomorphic.
T and R do not factorise accordingly.
In case of T , the reason lies in the automatic reset built into it.
For R , one could adjust its mapping of four histograms to achieve a factorisation.
However, the adjusted R would not be able to capture that both agents agreeing on an action even though it opens the door to the tiger costs them less ( − 50 ) than opening different doors ( − 100 ).
Here, one would need counting.
Nonetheless, an isomorphic model would exclude any policy where one agent opens a door while the other agent either listens or opens the other door, which are not optimal (and in a sense stupid) action combinations.
Let us brieﬂy consider the nanoscale medical system for robust diagnosis with, e.g., four types of agents.
Consider two possible actions, releasing or not releasing a load, and one possible observations of detecting a virus signature.
In such a setting, we have four partitions, each with the same actions and observations. Preliminary experiments have shown that each partition may have around 64 ,agents, making the agent set at least of size 4 · 64 , 000 , which is not computable on the ground level. The question then becomes how to model the transition, reward, and observation functions, i.e., how much does one approximate. Counting symmetries would be the more appropriate choice if one wants to model that a certain fraction of agents has to perform an action for a reward.
Braun et al.
[2021] provide a more detailed description of a nanoscale medical system, especially regarding counting symmetries.
In terms of specifying the model, the required memory would be only polynomial in size.
However, solving such a problem with histograms that represent 4 · 64 , 000 agents is also not computable because of the large search space for policies.
With such large agent numbers, we want to assume isomorphic symmetries, which would involve that each agent acts independently of the other agents in their environment.
Such an assumption is very approximate at ﬁrst glance but since nanoagents have only limited online computation power and memory available, approximating them as acting independently of all other agents appears reasonable.
Therefore, the goal would be to set up an isomorphic model.
Identifying Symmetries To use symmetries, one has to identify them in the problem at hand.
In case of the nanoscale medical system, symmetries lie in the nature of the problem as there are a handful of types of agents but thousands of agents of each type and they can be directly used when specifying the problem instance.
If one does have a full propositional model, then one has to check for the symmetries in the probabilities, i.e., values reoccurring, of the functions after one has checked that there are indeed groups of agents that have the same sets of actions and observations available.
A ﬁrst way would be to start with checking if
Eq.
(6) holds and then looking at the numbers in transition, reward, and sensor functions for symmetries.
The colour passing algorithm by Ahmadi et al.
[2013] is a good starting point for automating the search for symmetries.
The idea is of course to learn a counting or isomorphic model directly to capitalise on the fact that these models are smaller and require fewer parameters to learn.
Like in factored PGMs, one does not want to learn the full joint and then test for independences to get the factorisation and then for symmetries to lift the model.
Partition-level Symmetry:
Mixed Symmetries
The analysis above focuses on complete models exhibiting one type of symmetry for ease of exposition.
However, models can of course display a mix of the two symmetries with certain partitions exhibiting counting symmetry and others isomorphic symmetry, which leads to the following deﬁnition.
Deﬁnition 3.
A model M is symmetric if it can be partitioned with Eq.
(6) and T , R and Ω show one of the following symmetries for each of the resulting partitions:
(1) Counting : T , R and Ω map to the same values for permutations of actions and observations in a
partition
(2) Isomorphic : T , R and Ω factorise into a set of identical functions with representative agents.
On the modelling side, a partitioned DecPOMDP would then consist of PRVs and CRVs for actions and observations and as inputs to the transition, reward, and sensor functions, which would compactly encode the corresponding behaviour depending to the symmetry for each partition.
Of course, the assumptions about independence and symmetry have to hold between the different partitions for mixed symmetries to be able to occur, which may not be realistic for realworld data.
Weakening the assumptions or approximating the symmetries in a bounded way lies ahead in future work.
In terms of complexity, we can distinguish between I isomorphic and C counting partitions ( I + C = K ):
[Equation]
with s = | ran ( S ) | , n = max k n k , a = max k | ran ( A k ) | , and ¯ o = max k | ran ( O k ) | , k ∈ { 1 , . . .
, K } .
Partitions of size 1 belong to isomorphic partitions, i.e., count towards I .
Partition-level Independence It might be reasonable to assume that partitions are only connected through the joint state and therefore act and observe independently from each other.
The consequence for the transition, reward, and sensor function T , R , and Ω of a symmetric model is that they factorise into sets of K functions T k , R k , and Ω k :
[Equation] with ¯ A k =
# X k
[ A k ( X k )] and ¯ O k =
# X k
[ O k ( X k )] in the counting case and ¯ A k =
A k ( X k ) and ¯ O k
=
O k ( X k ) with n k as exponent in the isomorphic case.
The effect on the complexities in both settings is that the exponent K becomes a regular factor.
The effect is not that large as K is assumed to be small.
Nonetheless, this setting presents the best case with the exponent N reduced to a factor K in an isomorphic
DecPOMDP.
It also means that we move towards a POSG as each partition has its own reward function ¯ R k .
Connection to LVE One can draw parallels between counting and isomorphic models on the DecPOMDP side and count-conversion and lifted summing out on the LVE side, respectively.
In terms of complexity, lifted summing out also has a better bound than that of count-conversion.
The term that characterises the effort is the lifted width, which basically bounds the largest possible intermediate result during inference [Taghipour, 2013].
Without any CRVs, i.e., no CRVs in the model and no count-conversions in inference, the size in n , the maximum domain size, is bounded by log 2 n , also coming from exponentiation in inference.
With CRVs, it is polynomial in n .
Here, the problem indeed becomes tractable for both counting and isomorphism regarding domain sizes in contrast to DecPOMDPs where the counting case remains intractable because of the policy space and only the isomorphic case becomes tractable.
Isomorphic Models, DecPOMDP Solvers, & New Query
Types Given an isomorphic model, Lemma 3 allows for adapting any solution technique for a (ground) DecPOMDP like the Joint Equilibrium Search for Policies by Nair et al.
[2003] by using it to ﬁnd an optimal response policy for a representative and incorporating partition sizes into the expected utility calculation if necessary, which leads to a
Nash equilibrium policy.
Empirical evaluations of ground techniques usually work with single-digit agent numbers, which is the assumed order of K .
Since we can transfer the algorithms to the isomorphic setting, the results still hold but now for number of partitions if one ranks policies.
As such, we can now let the number of agents increase signiﬁcantly in isomorphic models.
In addition, this new setting allows for a new query type apart from a joint policy based on partitions, paraphrased as: “ How many agents do I need to reach a minimum? ”
In scenarios, where we are interested in reaching a minimum expected utility U min by possibly investing more agents, we can use any algorithm to solve an isomorphic DecPOMDP instance in model ¯ M i as argued above with given partition sizes, yielding a joint policy ¯ π ∗ with an optimal expected utility U ¯ M i (¯ π ∗ ) .
Then, since the agent numbers are still a part of the model, one can deﬁne an optimisation problem around this where we are interested in the number of agents needed per partition to reach U min >
U ¯ M i (¯ π ∗ ) .
Since we have an isomorphic DecPOMDP, having more agents in a partition does not have any intra-partition effects because of the quasi-independence between agents and therefore does not lead to changes in ¯ π ∗ .
However, having more agents does inﬂuence the U value of the joint policy.
From Policies to Functional Agents When implementing a multi-agent system, the agents can be set up (or built, in the nanoscale system setting) according to the joint policies:
In the isomorphic case, all agents in a partition behave according to the same policy and can be set up with that policy.
In the counting case, the counting policy can be grounded, leading to policies for individual agents.
Grounding is always possible with lifted constructs, which would require the so-called expansion operator by Taghipour [2013] for grounding CRVs. The agents can then be set up according to the (grounded) local policies. If combining (counting) all those local policies, a policy emerges equivalent to the original counting policy. In summary, symmetric DecPOMDPs allow for partitions and a compact representation of actions, observations, and policy.
In addition, the isomorphic variant allows for using existing algorithms with large agent sets in partitions and solving new queries for partition sizes.
CONCLUSION & FUTURE WORK
Inspired by the symmetries of counting and isomorphism in lifting, this paper analyses the effect of symmetries on
DecPOMDPs.
To this end, it speciﬁes what counting and isomorphic symmetry look like in ground models, allowing for a compact encoding using a partitioned DecPOMDP, which enables an explicit representation of partitions in the problem speciﬁcation and the solution.
The formal analysis shows that the symmetries allow for reducing the representation complexity and the cost of policy evaluation dependency to polynomial.
There still is an exponential dependence on the agent numbers in the policy space for the counting case, which is unsurprising given the combinatorial explosion and the symmetry used.
In the case of isomorphism, however, the problem becomes tractable regarding agent numbers and even allows for reusing existing ground solution methods while also enabling new query types.
Future work focuses on stochastic search for symmetric
DecPOMDPs, especially counting DecPOMDPs.
Other interesting avenues include continuous state and observation considerations, communication as well as combining lifting for agents with lifting on the state space as in FOMDPs.
Acknowledgements
[-li or 0 .0 .0 .
-]
[-ol li 0 .0 .0 .
-]
[-ol ol 0 .0 .0 .
-]
[-ol or 0 .0 .0 .0 .tl or li 0 .0 .0 .
-]
[-.tl or ol 0 .0 .0 .
-]
[-li or 0 .0 .
-]
[-ol li 0 .0 .0 .
-]
[-0 .tr ol or 0 .0 .0 .0 .tr or li 0 .0 .0 .0 .tr or ol 0 .0 .0 .0 .tr or or 0 .0 .0 .
-]
The research of Marcel Gehrke was funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC’Understanding Written Artefacts: Material, Interaction and Transmission in Manuscript Cultures’, project no.
The research was conducted within the scope of the Centre for the Study of Manuscript Cultures (CSMC) at Universität Hamburg.